Getting existing mapreduce job from cluster (the job could be running or completed)
apache, hadoop, java, mapreduce
Solution
The problem was with with a recent yarn upgrade that required enabling MR history server on my system. This fixed the issue. I recently upgraded from MR v1 to v2 and in that upgrade, all completed jobs are now moved to the history server.
Problem
Previously, I was using `org.apache.hadoop.mapred.JobClient#getJob(org.apache.hadoop.mapred.JobID)` to get the `RunningJob` . This call was made from the job completion callback method, however, seems to me that there is a timing issue where if the job is already completed then the above `getJob()` method cannot find it and returns null. I can confirm that the job was completed from the cluster UI. Keeping the `RunningJob` apart, is there a way to get the `org.apache.hadoop.mapreduce.Job` object of a mapred job given the `org.apache.hadoop.mapreduce.JobID` , regardless whether the job is currently running or is completed? I tried to code up something like: `Cluster cluster = jobClient.getClusterHandle(); Job job = cluster.getJob(JobID.forName(jobId)); log.info("Trying to get actual job with id {} , found {} on cluster {}", JobID.forName(jobId), job, cluster); ` I can see the right jobId, and can also see the cluster object.. but the `cluster.getJob()` method returns null, so the job itself is null. Is there something that I'm missing out here?