What is the proper value of HADOOP_HOME and PIG_CLASSPATH for APACHE HADOOP version 2.8.0?

Question

I have the problem executing the `Hadoop` command from the `PIG` command line. The command and the error stack is below My instructor suspects that it is because `HADDOP_HOME` and `PIG_CLASSPATH` are incorrect. I am on the `HADOOP version 2.8.0`. So, originally I had `HADOOP_HOME` as ``` HADOOP_HOME=/hadoop/2.8.0/ ``` Then I switched the following setup: ``` HADOOP_HOME=/hadoop/2.8.0/libexec/etc/hadoop ``` `PIG_CLASSPATH` is defined as `$HADOOP_HOME` Commands I used in pig: ``` A = LOAD '/Users/anarinsky/Downloads/loaddata1.txt'; B = MAPREDUCE '/Users/anarinsky/workspace/wordcount/target/wordcount-1.jar' STORE A INTO '/Users/anarinsky/Downloads/tempwrite2' LOAD '/Users/anarinsky/Downloads/tempwrite2' AS (word:chararray, count:int) `com.systemskills.hadoop.wordcount.WordCountDriver /wordcountdata /Users/anarinsky/Downloads/pigoptdir`; ``` Pig Stack Trace ``` ERROR 2025: Expected leaf of reduce plan to always be POStore. Found PONative org.apache.pig.…

Accepted Answer

Alex! Unfortunately, it's not related to Pig paths (tried it on my configured hadoop cluster) with same result. The error you get refers to the fact that Physical plan compiler has a bug in compile method. So in order to make your attempt work you have two possibilities

Run native MR job using hadoop and after it finishes process it's results in pig

Edit pig source code and compile your own version. You'll need to edit `org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler#compile` method and replace

```
for (PhysicalOperator op : leaves) {
    if (!(op instanceof POStore)) {
        int errCode = 2025;
        String msg = "Expected leaf of reduce plan to " +
            "always be POStore. Found " + op.getClass().getSimpleName();
        throw new MRCompilerException(msg, errCode, PigException.BUG);
    }
}
```

with

```
    for (PhysicalOperator op : leaves) {
        if (!(op instanceof POStore) && !(op instanceof PONative)) {
            int errCode = 2025;
            String msg = "Expected leaf of reduce plan to " +
                "always be POStore. Found " + op.getClass().getSimpleName();
            throw new MRCompilerException(msg, errCode, PigException.BUG);
        }
    }
```

What is the proper value of HADOOP_HOME and PIG_CLASSPATH for APACHE HADOOP version 2.8.0?

Solution

Problem