Override hadoop's mapreduce.fileoutputcommitter.marksuccessfuljobs in oozie
hadoop, hive, mapreduce, oozie
Solution
Hive unfortunately overrides this capability by setting it's own NullOutputComitter:
conf.setOutputCommitter(NullOutputCommitter.class);
see
src/shims/src/0.20/java/org/apache/hadoop/hive/shims/Hadoop20Shims.java
src/shims/src/common-secure/java/org/apache/hadoop/hive/shims/HadoopShimsSecure.java
Looks like you'll have to create the flag manually. We have filed HIVE-3700 for this.
Problem
``` <property> <name>mapreduce.fileoutputcommitter.marksuccessfuljobs</name> <value>false</value> </property> ``` I want to override the above property to true. The property needs to be false for the rest of the jobs on the cluster, but I need, in my oozie workflow, hadoop to create _SUCCESS file in the output directory after the completion of job. Its a hive action in the workflow which writes output. Please help.