Couchbase loses data after cluster node failure simulation
couchbase
Solution
Firstly, did you failover the node when it went out of the cluster? Until you failover, the replica on the other nodes will not be promoted to active (and hence any replica data will not be accessible).
Secondly, are you checking the return value from your `insert` operations? If a node is inaccessible (but before a failover) operations will return an exception (likely "timeout") - you should ensure the application retries the insert.
Thirdly, by default most CRUD operations on Couchbase return as soon as the update has occurred on the master node for maximum performance. As a consequence if you do loose a node it's possible that the replica hasn't been written yet - so there would be no replica even if you did perform a failover. To prevent this you can use the `observe` operation to not report the operation "complete" until a replica node has a copy - see Monitoring Items using observe.
Note that using observe will result in a performance penalty, but this may be an acceptable tradeoff for modifications you particularly care about.
Problem
I have 3 nodes in Couchbase cluster with number of replicas set to 1. While performing a multithreaded insert of 1M documents, I restart one of the nodes couple of times. The result is that at the end of insert operations, I am missing about 15% of the data. Any idea how to prevent the data loss?