Distributed RabbitMQ Nodes don't recognize each other

connectivity, distributed, erlang, rabbitmq

Solution

So I had multiple issues with connecting my two RabbitMQ nodes- I'll add that my nodes are hosted on rackspace, and so don't have a default exposable hostname, and require iptables since there is no DMZ or built in security group concept like amazon.

Problems: 1. Cookie- Not sure how or why but I had multiple instances of `.erlang.cookie` (in `/root`, in my home directory and in `/var/lib/rabbitmq/`) I kept only the one in rabbitmq and verified all nodes have the same cookie. 2. IPTables- In order for the nodes to communicate I needed to open the `epmd` port and the range of ports for the actual communication `inet_dist_listen_min` `inet_dist_listen_max`.

/sbin/iptables -A INPUT -i eth1 -p tcp --dport ${epmd} -s ${otherNode} -j ACCEPT
/sbin/iptables -A INPUT -i eth1 -p tcp --dport ${inet_dist_listen_min}:${inet_dist_listen_max} -s ${otherNode} -j ACCEPT  

`empd` is the usuall 4369 port and for the other range use whatever range you want. `${otherNode}` is the ip of my other node. I also needed to configure erlang through rabbitmq to use these ports (see config file at end) 3. HostName- Seeing as I don't have a hostname I needed to edit the rabbit scripts to use `-name` and not `-sname` (the first tells erlang to take the whole name, the latter stands for short name and thus appends an @ symbol and the hostname). This was accomplished by editing: `/usr/lib/rabbitmq/bin/rabbitmqctl` Added at the beginning the definition of the `RABBITMQ_NODE_IP_ADDRESS` property

DEFAULT_NODE_IP_ADDRESS=auto
DEFAULT_NODE_PORT=5672
[ "x" = "x$RABBITMQ_NODE_IP_ADDRESS" ] && RABBITMQ_NODE_IP_ADDRESS=${NODE_IP_ADDRESS}
[ "x" = "x$RABBITMQ_NODE_PORT" ] && RABBITMQ_NODE_PORT=${NODE_PORT}

[ "x" = "x$RABBITMQ_NODE_IP_ADDRESS" ] && [ "x" != "x$RABBITMQ_NODE_PORT" ] && RABBITMQ_NODE_IP_ADDRESS=${DEFAULT_NODE_IP_ADDRESS}
[ "x" != "x$RABBITMQ_NODE_IP_ADDRESS" ] && [ "x" = "x$RABBITMQ_NODE_PORT" ] && RABBITMQ_NODE_PORT=${DEFAULT_NODE_PORT}

and in the actual erl command I changed `-sname ${RABBITMQ_NODENAME} \` to `-name ${RABBITMQ_NODENAME}@${RABBITMQ_NODE_IP_ADDRESS}\`. This made rabbitmq listen only on the specified ip address (specified in the config file at the end) and load with that ip instead of the usuall hostname.

edited `/usr/lib/rabbitmq/bin/rabbitmq-server` Changed the actual erl command from `-sname ${RABBITMQ_NODENAME} \` to `-name ${RABBITMQ_NODENAME}@${RABBITMQ_NODE_IP_ADDRESS}`\

Added a rabbit conf (`/etc/rabbitmq/rabbitmq-env.conf`) file with-

#the ip address which rabbit should use, this is to limit rabbit to only use internal rackspace communication and not publicly accessible ports  
NODE_IP_ADDRESS=myIpAdress  
#had to change the nodename becaue otherwise rabbitmq used rabbit@Hostname and not only rabbit  
NODENAME=myCompany
#This instructed rabbit to instruct erlang which ports it should use for its communications with other nodes  
export SERVER_ERL_ARGS="$SERVER_ERL_ARGS -kernel inet_dist_listen_min somePort -kernel inet_dist_listen_max someOtherBiggerPort"

Some resources which helped me along the way: RabbitMQ Clustering Guide Clustering RabbitMQ servers for High Availability rabbitmq-env.conf(5) manual page Node communication by public IP address erlang mailing list (The middle post) Configuring RabbitMQ Cluster on Cloud

Hope this will help anyone else.

EDIT: Not sure how I was mistaken but it seemed my erlang-rabbit port instructions were not taken into consideration or were not enough. Ended up having to allow all communications between the two nodes...

Problem

Original source