More than 120 counters in hadoop

counter, hadoop, limit

Solution

You can override that property in `mapred-site.xml` on your JT, TT, client nodes but make sure that this will be a system-wide modification:

<configuration>
  ...
  <property>
    <name>mapreduce.job.counters.limit</name>
    <value>500</value>
  </property>
  ...
</configuration>

Then restart the mapreduce service on your cluster.

Problem

There's a limit for Hadoop counter size. It's 120 by default. I try to use the configuration "mapreduce.job.counters.limit" to change that, but it doesn't work. I've seen the source code. It's like the instance of JobConf in class "org.apache.hadoop.mapred.Counters" is private. Have anybody seen that before? What's your solution? THX :)

Original source