Kafka in distributed system
apache-kafka
Solution
Adding on to the previous answer by user2720864
Let us assume that Kafka system with below configuration is needed.
7 Kafka nodes
3 Zoo keepers
To achieve this install 7 Kafka instances, in 7 different server/vm(instances), and in each of these instances set a different broker-id, this will let the zookeeper identify the different kafka nodes for bookkeeping, maintenance. broker.id=X (/config/server.properties)
To start zookeepers, you can use 3 of the previous kafka instances or can use new servers to start zookeepers. Once the servers on which zookeepers run are decided, change the /config/server.properties to specify zookeepers.
zookeeper.connect=hostname1:port1,hostname2:port2
In a distributed environment its nice to have 3 zoo keepers. While there is only one zookeeper which acts as a true master, other 2 zookeepers act as fail overs. When the master fails one of the two ZKs will take over as master.
I found this link to be very useful, it helped me clarify a lot of things about kafka architecture.
This is a good reference for all the configurations on the property files in kafka.
Hope this helps!
Problem
I am new to kafka , i am running kafka in a single machine as of now. I want to run kafka in an distributed environment on multiple machines. There is no proper documentation for this. Any documentation or suggestion on this will be really helpful.