Fastest way to remove duplicate documents in mongodb
duplicates, mongodb, optimization, performance
Solution
Assuming you want to permanently delete docs that contain a duplicate `name` + `nodes` entry from the collection, you can add a `unique` index with the `dropDups: true` option:
db.test.ensureIndex({name: 1, nodes: 1}, {unique: true, dropDups: true})
As the docs say, use extreme caution with this as it will delete data from your database. Back up your database first in case it doesn't do exactly as you're expecting.
UPDATE
This solution is only valid through MongoDB 2.x as the `dropDups` option is no longer available in 3.0 (docs).
Problem
I have approximately 1.7M documents in mongodb (in future 10m+). Some of them represent duplicate entry which I do not want. Structure of document is something like this: ``` { _id: 14124412, nodes: [ 12345, 54321 ], name: "Some beauty" } ``` Document is duplicate if it has at least one node same as another document with same name. What is the fastest way to remove duplicates?