Why is there a limit on the number of counters allowed in a Hadoop map reduce job?
counter, hadoop, mapreduce
Solution
See the section on Counters in this post.
Counters
Counters represent global counters, defined either by the Map/Reduce framework or applications. Applications can define arbitrary Counters and update them in the map and/or reduce methods. These counters are then globally aggregated by the framework.
Counters are appropriate for tracking few, important, global bits of information. They are definitely not meant to aggregate very fine-grained statistics of applications. Counters are very expensive since the JobTracker has to maintain every counter of every map/reduce task for the entire duration of the application.
Problem
I am using Hadoop map-reduce and got a CountersExceededException when I tried to programatically create many job counters. I know that I can increase the number of counters that are allowed through the configuration files, but does anyone know: a) why there is a limit on map-reduce counters and b) if it is a good/bad idea to increase the max number of map-reduce counters? I am using hadoop 0.20.2