Elasticsearch is not working after full disk
elasticsearch
Solution
Bigfoot's solution is the only one that seems to work.
While the stack trace observed seems to be similar to: https://github.com/elastic/elasticsearch/issues/12055
This pull request is supposed to fix the issue: https://github.com/elastic/elasticsearch/pull/9797
But upgrading to v1.5.0 also does not do the trick.
Thus the only thing that works:
find /var/lib/elasticsearch/elasticsearch/nodes/ -name "*.recovering"
And delete all recovering files. Of course this has side-effects.
Problem
I made space, restarted the service (no good) and then rebooted but I still get this in the elasticsearch.log: ``` [2015-02-16 13:35:19,625][WARN ][cluster.action.shard ] [Server] [logstash-2015.02.16][1] sending failed shard for [logstash-2015.02.16][1], node[PFamB-ZJS7CwSdyyAcP_8A], [P], s[INITIALIZING], indexUUID [tZ3I9HZ6TDaZSicIuGWRWQ], reason [Failed to start shard, message [IndexShardGatewayRecoveryException[[logstash-2015.02.16][1] failed to recover shard]; nested: TranslogCorruptedException[translog corruption while reading from stream]; nested: ElasticsearchIllegalArgumentException[No version type match [83]]; ]] [2015-02-16 13:35:19,625][WARN ][cluster.action.shard ] [Server] [logstash-2015.02.16][1] received shard failed for [logstash-2015.02.16][1], node[PFamB-ZJS7CwSdyyAcP_8A], [P], s[INITIALIZING], indexUUID [tZ3I9HZ6TDaZSicIuGWRWQ], reason [Failed to start shard, message [IndexShardGatewayRecoveryException[[logstash-2015.02.16][1] failed to recover shard]; nested: TranslogCorruptedException[translog corruption while reading from stream]; nested: ElasticsearchIllegalArgumentException[No version type match [83]]; ]] [2015-02-16 13:35:43,570][DEBUG][action.index ] [Server] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m] [2015-02-16 13:36:10,757][DEBUG][action.index ] [Server] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m] ``` What should I do?