How to remove duplicates based on a key in Mongodb?
duplicates, key, mongodb, optimization
Solution
This answer is obsolete : the `dropDups` option was removed in MongoDB 3.0, so a different approach will be required in most cases. For example, you could use aggregation as suggested on: MongoDB duplicate documents even after adding unique key.
If you are certain that the `source_references.key` identifies duplicate records, you can ensure a unique index with the `dropDups:true` index creation option in MongoDB 2.6 or older:
db.things.ensureIndex({'source_references.key' : 1}, {unique : true, dropDups : true})
This will keep the first unique document for each `source_references.key` value, and drop any subsequent documents that would otherwise cause a duplicate key violation.
Important Note: Any documents missing the `source_references.key` field will be considered as having a null value, so subsequent documents missing the key field will be deleted. You can add the `sparse:true` index creation option so the index only applies to documents with a `source_references.key` field.
Obvious caution: Take a backup of your database, and try this in a staging environment first if you are concerned about unintended data loss.
Problem
I have a collection in MongoDB where there are around (~3 million records). My sample record would look like, ``` { "_id" = ObjectId("50731xxxxxxxxxxxxxxxxxxxx"), "source_references" : [ "_id" : ObjectId("5045xxxxxxxxxxxxxx"), "name" : "xxx", "key" : 123 ] } ``` I am having a lot of duplicate records in the collection having same `source_references.key`. (By Duplicate I mean, `source_references.key` not the `_id`). I want to remove duplicate records based on `source_references.key`, I'm thinking of writing some PHP code to traverse each record and remove the record if exists. Is there a way to remove the duplicates in Mongo Internal command line?