Why is my Cassandra node stuck joining the ring?
cassandra
Solution
In this case, I turned on alllll the debug logging in `/etc/cassandra/log4j-server.properties`, and sifted through the reams of output until I noticed a couple of statements about `OutboundTCP: attempting to connect to *host*` that were appearing for certain hosts but not others. Further investigation revealed that there was limited connectivity between certain nodes in the cluster, due to a networking configuration which was my own fault: the new node could connect to only 3 out of the 5 other hosts in the cluster.
After fixing this, the node was able to transfer additional data, and I was free to pursue a life of religious fulfillment.* (Also, I ran `nodetool repair` all over the place out of paranoia.)
Problem
I have a Cassandra node which is joining an existing cluster. It begins to bootstrap itself, streams a bunch of data (monitored via `nodetool netstats`), and then seems to finish - but it doesn't actually join the ring. What's the deal? How do I figure out what's going on? There are no errors in the logs at all, it's been plenty of time, and the Cassandra process is doing very little.