How to fix the "Illegal partition" error in hadoop?
hadoop, java, mapreduce
Solution
The calculated partition number by your custom `Partitioner` has to be non-negative. Try:
public int getPartition(weburl_compositeKey key, Text value, int numPartitions)
{
return (key.hashCode() & Integer.MAX_VALUE) % numPartitions;
}
Problem
I have written a custom partitioner. When I have number of reduce tasks greater than 1, the job is failing. This is the exception which I'm getting: ``` java.io.IOException: Illegal partition for weburl_compositeKey@804746b1 (-1) at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:930) at org.apache.hadoop.mapred.MapTask$OldOutputCollector.collect(MapTask.java:499) ``` The code which I have written is ``` public int getPartition(weburl_compositeKey key, Text value, int numPartitions) { return (key.hashCode()) % numPartitions; } ``` This the `key.hashCode()` equals `-719988079` and mod of this value is returning `-1`. Appreciate your help on this. Thanks.