Fastest way to query collection for ALL documents

Question

My collection has approximately 500 documents, which will be double that in a few weeks: How can I make getting all the documents faster? I'm currently using db.registrations.find(), so that I can have all the documents available for searching, sorting, and filtering data. Using skip/limit makes the query display quickly, but you can't search all the registrations for players, and that's necessary. My schema: ``` var mongoose = require('mongoose'); var Schema = mongoose.Schema; var playerSchema = new mongoose.Schema({ first: { type: String, required: true }, last: { type: String, required: true }, email: { type: String, required: true }, phone: { type: String, required: true }, address: { address: String, city: String, state: String, zip: String, country: { type: String, "default" : "USA" }, adult: Boolean } }); var registrationsSchema = new mongoose.Schema({ event : { type: String, required: true }, day : { type: String, required: true }, group : { type: String, required: true }, fie…

Accepted Answer

There is no big deal to load 1000 records from mongodb using mongoose. I did it in the past (2-3k records) and it worked well as long as I respected this 2 rules:

Don't load all the mongoose stuff

Use lean query. This way it won't load all the mongoose methods / attributes and it will get just the data of your objects. You can't use `.save()` or other methods but it's way faster to load.

Use stream to load your data.

Streams are a good way to load large dataset with nodejs/mongoose. It will read the data block by block from mongodb and send them to your application for usage. You will avoid the tipical case :

- I wait 2 seconds my data and my server is idle

- My server is 100% CPU during 2 seconds to process the data I got and the db is idle.

With streams, in this example your total time will be ~2s instead of 2+2=4s

To load data from stream with mongoose use the .cursor() function to change your request to a nodejs stream.

Here is an example to load all your players fast

```
const allPlayers = [];

const cursor = Player.find({}).lean().cursor();
cursor.on('data', function(player) { allPlayers.push(player); });
cursor.on('end', function() { console.log('All players are loaded here'); });
```

Fastest way to query collection for ALL documents

Solution

Problem