How many producers to create in kafka?
apache-kafka, java
Solution
In general, a single producer for all topics will be more network efficient.
If the kafka client sees more than one topic+partition on the same Kafka Node, it can send messages for both topic+partitions in a single message. Kafka optimizes for message batches so this is efficient.
In addition, your web servers only need to maintain at-most one tcp connection to each Kafka node, instead of one connection per producer, per node.
For more info on Kafka's design: https://kafka.apache.org/documentation.html#design
As you mention in comments, lock contention may become a limiting factor, YMMV.
Problem
In a high volume real time java web app I'm sending messages to apache kafka. Currently I'm sending to a single topic, but in the future I might need to send messages to multiple topics. In this case I'm not sure weather to create a producer per topic or should I use a single producer to all my topics? Here is my code: ``` props = new Properties(); props.put("zk.connect", <zk-ip1>:<2181>,<zk-ip3>:<2181>,<zk-ip3>:<2181>); props.put("zk.connectiontimeout.ms", "1000000"); props.put("producer.type", "async"); Producer<String, Message> producer = new kafka.javaapi.producer.Producer<String, Message>(new ProducerConfig(props)); ProducerData<String, Message> producerData1 = new ProducerData<String, Message>("someTopic1", messageTosend); ProducerData<String, Message> producerData2 = new ProducerData<String, Message>("someTopic2", messageTosend); producer.send(producerData1); producer.send(producerData2); ``` As you can see, once the producer has been created I can use it to send data to different topics. I wonder what is the best practice? If my app sends to multiple topics (each topic gets different data) can/should I use a single producer or should I create multiple producers? When (generaly speaking) should I use more than a single producer?