measure the duration of the whole map phase per node in Hadoop
hadoop, mapreduce, time
Solution
You can use the User Defined Java Counters to derived the time taken by all Mappers per Node.
In the Mapper implementation you need to do the following, 1. override the setup and record the starttime. `long startTime = System.currentTimeMillis();` 2. override the cleanup method
`long endTime = System.currentTimeMillis(); String hostname = java.net.InetAddress.getLocalHost().getHostName(). context.getCounters(hostname,"time consumed").increment(endTime - startTime); `
Problem
At the moment, I know that jobtracker can show you the time that it takes per each map task, but what I want is not per map task, but the time it takes since the first map task till the last one of one node in a cluster. For example: 1 Map takes 2 seconds, but how can I measure the time per node when you have 100 map tasks and not all can be executed in parallel? Is it possible to know the time that it takes to execute the whole map phase (all the map tasks) per node?