MongoDB sharded cluster 25 slower than standalone node

mongodb, performance, replication, sharding

Solution

Let's do the math first: how big are your documents? Keep in mind that they have to be transferred over the net multiple times depending on your write concern.

May be you are experiencing this because of the indices which have to be build.

Please try this:

- Disable all indices except the one for `_id` (which is not possible anyway, iirc)

- Load your data

- Reenable indices.

- Enable sharding and balancing if not done already

This is the suggested way for importing data into a shared cluster anyway and should speed up your import considerably. Some (cautious !) fiddling with `storage.syncPeriodSecs` and `storage.journal.commitIntervalMs` might help, too.

The delay can occur even when storing the data on the primary shard. Depending on the size of your indices, they may slow down bulk operations considerably. You might also want to have a look at the `replication.secondaryIndexPrefetch` config option.

Another thing might be that your oplog simply gets filled faster than the replication can take place. Problem here: once it is created, you can not increase it's size. I am not sure wether it is safe to delete and recreate it in standalone mode and then reshare the replica set, but I doubt it. So the safe option would be to have the instance actually leave the replica set, reinstall it with a more appropriate oplog size and add the instance to the replica set as if it were the first time. If you don't care for the data, simply shut the replica set down, adjust the oplog size in the config file, delete the data dir and restart and reinitialize the replica set. Thinking of your problem twice, this sounds like the best bet to me, since the opllog isn't involved in standalone mode, iirc.

If you still have the same performance issues, my bet is on problems with disk or network IO.

You have a fairly standard setup, your `mongos` instance is running on a different machine than your `mongod` (be it a standalone or the primary of a replica set). You might want to check a few things:

- Name resolution latency for resolving the name of your primary and secondary shards from the machine running your `mongos` instance. I can not count the times installing nscd improved performance for various operations.

- network latency from your `mongos` instance to your primary shard. Assuming you have a firewall between your AppServer and your cluster, you might want to talk to the respective administrator.

- In case you are using external authentication, try to measure how long it takes.

- When using some sort of tunneling (e.g. stunnel or encryption like SSL/TLS), make sure you disable name resolution. Please keep in mind that encrypting and decrypting may take a relatively long time.

- Measure random disk IO on the `mongod` instances

Problem

Original source