How can I send large messages with Kafka (over 15MB)?
apache-kafka, exception, java
Solution
You need to adjust three (or four) properties:
- Consumer side:`fetch.message.max.bytes` - this will determine the largest size of a message that can be fetched by the consumer.
- Broker side: `replica.fetch.max.bytes` - this will allow for the replicas in the brokers to send messages within the cluster and make sure the messages are replicated correctly. If this is too small, then the message will never be replicated, and therefore, the consumer will never see the message because the message will never be committed (fully replicated).
- Broker side: `message.max.bytes` - this is the largest size of the message that can be received by the broker from a producer.
- Broker side (per topic): `max.message.bytes` - this is the largest size of the message the broker will allow to be appended to the topic. This size is validated pre-compression. (Defaults to broker's `message.max.bytes`.)
I found out the hard way about number 2 - you don't get ANY exceptions, messages, or warnings from Kafka, so be sure to consider this when you are sending large messages.
Problem
I send String-messages to `Kafka V. 0.8` with the Java Producer API. If the message size is about 15 MB I get a `MessageSizeTooLargeException`. I have tried to set `message.max.bytes`to 40 MB, but I still get the exception. Small messages worked without problems. The exception appears in the producer, I don't have a consumer in this application. How can I get rid of this exception? My example producer config ``` private ProducerConfig kafkaConfig() { Properties props = new Properties(); props.put("metadata.broker.list", BROKERS); props.put("serializer.class", "kafka.serializer.StringEncoder"); props.put("request.required.acks", "1"); props.put("message.max.bytes", "" + 1024 * 1024 * 40); return new ProducerConfig(props); } ``` Error-Log: ``` 4709 [main] WARN kafka.producer.async.DefaultEventHandler - Produce request with correlation id 214 failed due to [datasift,0]: kafka.common.MessageSizeTooLargeException 4869 [main] WARN kafka.producer.async.DefaultEventHandler - Produce request with correlation id 217 failed due to [datasift,0]: kafka.common.MessageSizeTooLargeException 5035 [main] WARN kafka.producer.async.DefaultEventHandler - Produce request with correlation id 220 failed due to [datasift,0]: kafka.common.MessageSizeTooLargeException 5198 [main] WARN kafka.producer.async.DefaultEventHandler - Produce request with correlation id 223 failed due to [datasift,0]: kafka.common.MessageSizeTooLargeException 5305 [main] ERROR kafka.producer.async.DefaultEventHandler - Failed to send requests for topics datasift with correlation ids in [213,224] kafka.common.FailedToSendMessageException: Failed to send messages after 3 tries. at kafka.producer.async.DefaultEventHandler.handle(Unknown Source) at kafka.producer.Producer.send(Unknown Source) at kafka.javaapi.producer.Producer.send(Unknown Source) ```