Trying to do a bulk upsert with Mongoose. What's the cleanest way to do this?
mongodb, mongoose, node.js
Solution
(mongoose@4.9.1, mongodb@3.4.2)
TL;DR
await GasStation.collection.bulkWrite([ // <<==== use the model name
{
'updateOne': {
'filter': { 'id': '<some id>' },
'update': { '$set': { /* properties to update */ } },
'upsert': true, // <<==== upsert in every document
}
},
/* other operations here... */
]);
Long story:
After struggling with Mongoose API poor documentation, I solved the bulk upsert tweaking `updateOne:{}` operation in the `bulkWrite()` method.
A couple of undocumented things to consider:
// suppose:
var GasStation = mongoose.model('gasstation', gasStationsSchema);
var bulkOps = [ ];
// for ( ... each gasStation to upsert ...) {
let gasStation = { country:'a', localId:'b', xyz:'c' };
// [populate gasStation as needed]
// Each document should look like this: (note the 'upsert': true)
let upsertDoc = {
'updateOne': {
'filter': { 'country': gasStation.country, 'localId': gasStation.localId },
'update': gasStation,
'upsert': true
}};
bulkOps.push(upsertDoc);
// end for loop
// now bulkWrite (note the use of 'Model.collection')
GasStation.collection.bulkWrite(bulkOps)
.then( bulkWriteOpResult => {
console.log('BULK update OK');
console.log(JSON.stringify(bulkWriteOpResult, null, 2));
})
.catch( err => {
console.log('BULK update error');
console.log(JSON.stringify(err, null, 2));
});
The two key things here are incomplete API documentation issues (at the time of writing, at least):
- `'upsert': true` in each document. This is not documented in Mongoose API (), which often refers to node-mongodb-native driver. Looking at updateOne in this driver, you could think to add `'options':{'upsert': true}`, but, no... that won't do. I also tried to add both cases to the `bulkWrite(,[options],)` argument, with no effect either.
- `GasStation.collection.bulkWrite()`. Although Mongoose bulkWrite() method claims it should be called `Model.bulkWrite()` (in this case, `GasStation.bulkWrite()`), that will trigger `MongoError: Unknown modifier: $__`. So, `Model.collection.bulkWrite()` must be used.
Additionally, note:
- You don't need to use the `$set` mongo operator in the `updateOne.update` field, since mongoose handles it in case of upsert (see bulkWrite() comments in example).
- Note that my unique index in the schema (needed for upsert to work properly) is defined as:
`gasStationsSchema.index({ country: 1, localId: 1 }, { unique: true });`
Hope it helps.
==> EDIT: (Mongoose 5?)
As noticed by @JustinSmith, the `$set` operator added by Mongoose doesn't seem to be working anymore. Maybe it's because of Mongoose 5?
In any case, using `$set` explicitly should do:
'update': { '$set': gasStation },
Problem
I have a collection that holds documents that contains three fields: first_name, last_name, and age. I'm trying to figure out what query in Mongoose I can use to do a bulk upsert. My app is occasionally receiving a new array of objects with those same three fields. I want the query to check if the first AND last name already exist within a document, and if they do - update the age if it's different. Otherwise, if the first and last name don't exist, insert a new document. Currently, I'm only doing the import - and haven't yet built out the logic for this upsert piece. ``` app.post('/users/import', function(req, res) { let data = req.body; let dataArray = []; data.forEach(datum => { dataArray.push({ first: datum.first, last: datum.last, age: datum.age }) }) User.insertMany(dataArray, answer => { console.log(`Data Inserted:`,answer) }) ``` ` And my User model looks like this: ``` const mongoose = require('mongoose'); const Schema = mongoose.Schema; const userSchema = new Schema({ first: String, last: String, age: Number, created_at: { type: Date, default: Date.now } }); var User = mongoose.model('User', userSchema); module.exports = User; ```