Hadoop MapReduce: Appropriate input files size?
file, hadoop, mapreduce, size
Solution
If you're not compressing the files then hadoop will process your large files (say 10G), with a number of mappers related to the block size of the file.
Say your block size is 64M, then you will have ~160 mappers processing this 10G file (160*64 ~= 10G). Depending on how CPU intensive your mapper logic is, this might be an acceptable blocks size, but if you find that your mappers are executing in sub minute times, then you might want to increase the work done by each mapper (by increasing the block size to 128, 256, 512m - the actual size depends on how you intend to process the data).
A larger blocks size will reduce the number of mappers used to process the 10G file. You can of course increase the minimum split size used by the TextInputFormat, but then you'll most probably run into lower data locality as the mapper may be processing 2 or more blocks, which may not all reside locally on that node.
As for output, this again depends on what your processing logic is doing - can you partition just by introducing more reducers? This will create more output files, but what partitioning logic do you require for these files (by default they will be hash partitioned by your key)
Problem
I have data sets in a magnitude of 3-digit GBs or even 1 or 2-digit TB. The input files are therefore a list of files, each sized like 10GB. My map reduce job in hadoop processes all these files and then gives only one output file (with the aggregated information). My questions are: What is the appropriate file size for tuning up the hadoop/mapreduce framework from Apache? I hear that bigger file sizes are more preferred than the small ones. Have any ideas? The only thing I know for sure is that hadoop reads blocks, each with 64MB by default. So it would be good if the file size is kind of multiplicator of 64MB. At the moment, my application is writing the output file into only one file. The file size is then of course 3-digit gigabit. I am wondering how efficiently I can partition the file. Of course I can just use some unix tools to do this job. But is it preferred to do this directly in hadoop? Thx for your comments! P.S.: I am not compressing the files. The file format of the input files is text/csv.