How to Define Custom partitioner for Spark RDDs of equally sized partition where each partition has equal number of elements?
apache-spark, hadoop, scala
Solution
`Partitioner`s work by assigning a key to a partition. You would need prior knowledge of the key distribution, or look at all keys, to make such a partitioner. This is why Spark does not provide you with one.
In general you do not need such a partitioner. In fact I cannot come up with a use case where I would need equal-size partitions. What if the number of elements is odd?
Anyway, let us say you have an RDD keyed by sequential `Int`s, and you know how many in total. Then you could write a custom `Partitioner` like this:
class ExactPartitioner[V](
partitions: Int,
elements: Int)
extends Partitioner {
def getPartition(key: Any): Int = {
val k = key.asInstanceOf[Int]
// `k` is assumed to go continuously from 0 to elements-1.
return k * partitions / elements
}
}
Problem
I am new to Spark. I have a large dataset of elements[RDD] and I want to divide it into two exactly equal sized partitions maintaining order of elements. I tried using `RangePartitioner` like ``` var data = partitionedFile.partitionBy(new RangePartitioner(2, partitionedFile)) ``` This doesn't give a satisfactory result because it divides roughly but not exactly equal sized maintaining order of elements. For example if there are 64 elements, we use `Rangepartitioner`, then it divides into 31 elements and 33 elements. I need a partitioner such that I get exactly first 32 elements in one half and other half contains second set of 32 elements. Could you please help me by suggesting how to use a customized partitioner such that I get equally sized two halves, maintaining the order of elements?