KafKa partitioner class, assign message to partition within topic using key

apache-kafka

Solution

This is what I've found so far .. Define our own custom partitioner class by implementing the kafka Partitioner interface. The implemented method will have two arguments, first the key that we provide from the producer and next the number of partition available. So we can define our own logic to set which key of message goes to what partition.

Now while creating the producer we can specify our own partitioner class using the "partitioner.class" attribute

    props.put("partitioner.class", "path.to.custom.partitioner.class");

If we don't mention it then Kafka will use its default class and try to distribute message evenly among the partitions available.

Also inform Kafka how to serialize the key

    props.put("key.serializer.class", "kafka.serializer.StringEncoder");

Now if we send some message using a key in the producer the message will be delivered to a specific partition (based on our logic written on the custom partitioner class), and in the consumer (SimpleConsumer) level we can specify the partition to retrieve the specific messages.

In case we need to pass a String as a key, the same should be handled in the custom partitioner class ( take hash value of the key and then take first two digit etc )

Problem

I am new to kafka so apology if I sound stupid but what I understood so far is .. A stream of message can be defined as a topic, like a category. And every topic is divided into one or more partitions (each partition can have multiple replicas). so they act in parallel From the Kafka main site they say The producer is able to chose which message to assign to which partition within the topic. This can be done in a round-robin fashion simply to balance load or it can be done according to some semantic partition function (say based on some key in the message). Does this mean while consuming I will be able to choose the message offset from particular partition? While running multiple partitions is it possible to choose from one specific partition i.e partition 0? In Kafka 0.7 quick start they say Send a message with a partition key. Messages with the same key are sent to the same partition. And the key can be provided while creating the producer as below ``` ProducerData<String, String> data = new ProducerData<String, String>("test-topic", "test-key", "test-message"); producer.send(data); ``` Now how do I consume message based on this key? what is the actual impact of using this key while producing in Kafka ? While creating producer in 0.8beta we can provide the partitioner class attribute through the config file. The custom partitioner class can be perhaps created implementing the kafka partitioner interface. But m little confused how exactly it works. 0.8 doc also does not explain much. Any advice or m i missing something ?

Original source