Cassandra node is unable to start after a HDD failure
cassandra, datastax, recovery
Solution
Fixed the issue by these steps:
physically removed files related to `system` keyspace: cassandra was able to start and recreated it, but without any metadata about other keyspaces.
ran `nodetool resetlocalschema`, which synchronized keyspace schema from other nodes.
Problem
I have a 5 node Cassandra 2.0.7 cluster, each node has 4 HDDs. Recently one of these HDDs on node3 had failed and was replaced by a new shiny empty drive. After the replacement cassandra on this node was unable to start with this exception: ``` INFO [main] 2014-06-02 12:45:17,232 ColumnFamilyStore.java (line 254) Initializing system.paxos INFO [main] 2014-06-02 12:45:17,236 ColumnFamilyStore.java (line 254) Initializing system.schema_columns INFO [SSTableBatchOpen:1] 2014-06-02 12:45:17,237 SSTableReader.java (line 223) Opening /mnt/disk2/cassandra/system/schema_columns/system-schema_columns-jb-310 (25418 bytes) INFO [main] 2014-06-02 12:45:17,241 ColumnFamilyStore.java (line 254) Initializing system.IndexInfo INFO [main] 2014-06-02 12:45:17,245 ColumnFamilyStore.java (line 254) Initializing system.peers INFO [SSTableBatchOpen:1] 2014-06-02 12:45:17,246 SSTableReader.java (line 223) Opening /mnt/disk3/cassandra/system/peers/system-peers-jb-25 (20411 bytes) INFO [main] 2014-06-02 12:45:17,253 ColumnFamilyStore.java (line 254) Initializing system.local INFO [SSTableBatchOpen:1] 2014-06-02 12:45:17,254 SSTableReader.java (line 223) Opening /mnt/disk3/cassandra/system/local/system-local-jb-35 (80 bytes) INFO [SSTableBatchOpen:2] 2014-06-02 12:45:17,254 SSTableReader.java (line 223) Opening /mnt/disk3/cassandra/system/local/system-local-jb-34 (80 bytes) ERROR [main] 2014-06-02 12:45:17,361 CassandraDaemon.java (line 237) Fatal exception during initialization org.apache.cassandra.exceptions.ConfigurationException: Found system keyspace files, but they couldn't be loaded! at org.apache.cassandra.db.SystemKeyspace.checkHealth(SystemKeyspace.java:532) at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:233) at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:462) at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:552) ``` Because of cassandra node being unable to start, I cannot use `nodetool repair`. The only way I see to recover the node is to remove all data and bootstrap it from nearly bare metal. Is there a shorter way to recover in a typical HDD failure scenario?