Cassandra EOFException thrown on some queries after a power failure
cassandra
Solution
Instead of deleting the whole CF, you can just delete the corrupt SSTable then run repair to recover the data from the replicas. The problem is the error doesn't print the filename in the log so it's hard to find out which file is corrupt. You can get it though by running Cassandra in debug mode (edit log4j-server.properties). See this other SO post:
Error messages while starting Cassandra node
for how they did it.
Problem
``` `INFO 11:44:29,874 Listening for thrift clients... ERROR 11:47:01,471 Exception in thread Thread[ReadStage:33,5,main] java.lang.RuntimeException: org.apache.cassandra.io.sstable.CorruptSSTableExcept ion: java.io.EOFException: EOF after 2 bytes out of 29545 at org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(Stora geProxy.java:1582) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Caused by: org.apache.cassandra.io.sstable.CorruptSSTableException: java.io.EOFE xception: EOF after 2 bytes out of 29545 at org.apache.cassandra.db.columniterator.SimpleSliceReader.<init>(Simpl eSliceReader.java:89) at org.apache.cassandra.db.columniterator.SSTableSliceIterator.createRea der(SSTableSliceIterator.java:68) at org.apache.cassandra.db.columniterator.SSTableSliceIterator.<init>(SS TableSliceIterator.java:44) at org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnItera tor(SliceQueryFilter.java:101) at org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(Q ueryFilter.java:68) at org.apache.cassandra.db.CollationController.collectAllData(CollationC ontroller.java:275) at org.apache.cassandra.db.CollationController.getTopLevelColumns(Collat ionController.java:65) at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFa milyStore.java:1363) at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamil yStore.java:1220) at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamil yStore.java:1132) at org.apache.cassandra.db.Table.getRow(Table.java:355) at org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadComm and.java:70) at org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThr ow(StorageProxy.java:1052) at org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(Stora geProxy.java:1578) ... 3 more Caused by: java.io.EOFException: EOF after 2 bytes out of 29545 at org.apache.cassandra.io.util.FileUtils.skipBytesFully(FileUtils.java: 350) at org.apache.cassandra.utils.ByteBufferUtil.skipShortLength(ByteBufferU til.java:382) at org.apache.cassandra.db.columniterator.SimpleSliceReader.<init>(Simpl eSliceReader.java:71) ... 16 more ERROR 11:47:12,900 Exception in thread Thread[ReadStage:35,5,main] java.lang.RuntimeException: org.apache.cassandra.io.sstable.CorruptSSTableExcept ion: java.io.EOFException: EOF after 2 bytes out of 29545 at org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(Stora geProxy.java:1582) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Caused by: org.apache.cassandra.io.sstable.CorruptSSTableException: java.io.EOFE xception: EOF after 2 bytes out of 29545 at org.apache.cassandra.db.columniterator.SimpleSliceReader.<init>(Simpl eSliceReader.java:89) at org.apache.cassandra.db.columniterator.SSTableSliceIterator.createRea der(SSTableSliceIterator.java:68) at org.apache.cassandra.db.columniterator.SSTableSliceIterator.<init>(SS TableSliceIterator.java:44) at org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnItera tor(SliceQueryFilter.java:101) at org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(Q ueryFilter.java:68) at org.apache.cassandra.db.CollationController.collectAllData(CollationC ontroller.java:275) at org.apache.cassandra.db.CollationController.getTopLevelColumns(Collat ionController.java:65) at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFa milyStore.java:1363) at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamil yStore.java:1220) at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamil yStore.java:1132) at org.apache.cassandra.db.Table.getRow(Table.java:355) at org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadComm and.java:70) at org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThr ow(StorageProxy.java:1052) at org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(Stora geProxy.java:1578) ... 3 more Caused by: java.io.EOFException: EOF after 2 bytes out of 29545 at org.apache.cassandra.io.util.FileUtils.skipBytesFully(FileUtils.java: 350) at org.apache.cassandra.utils.ByteBufferUtil.skipShortLength(ByteBufferU til.java:382) at org.apache.cassandra.db.columniterator.SimpleSliceReader.<init>(Simpl eSliceReader.java:71) ... 16 more` ``` And in Hector i get a `HTimedOutException` which I am assuming is because of the SSTable corruption in cassandra because it was working well before the power failure. How do I fix this? Also this is development database and I have had this problem before and ended up dropping some Column Families but I don't want this to be the only solution every time power cuts or system crashes. I am using cassandra 1.2.2