CouchDB Views: How much processing is acceptable in map reduce?
couchdb, database, mapreduce, nosql
Solution
Lots of expensive processing is acceptable in CouchDB map-reduce.
CouchDB views (map-reduce) are more like `CREATE INDEX` than they are `SELECT FROM`.
Specifically, CouchDB guarantees that a map function runs only once per document, ever. (Well, actually once per document change ever.) That is what the "iterative map-reduce" is.
Therefore, suppose you had 10,000 documents and they take 1 second each to process (which is way higher than I have ever seen). That is 10,000 seconds or 2.8 hours to completely build the view. However once the view is complete, querying any row (`?key=...`) or row slice (`?startkey=...&endkey=...`) takes the same time as querying for documents directly. Lookup time is O(log n) for the document count.
In other words, even if it takes 1 second per document to execute the map, it will take a few milliseconds to fetch the result. (Of course, the view must build first, since it is actually an index.)
Problem
I've been toying around with Map Reduce with CouchDB. Some of the examples show some possibly heavy logic within the map reduce functions. In one particular case, they were performing for loops within map. Is map reduce run on every single possible document before it emits your selected documents? If so, I would think that means that running any kind of iterative processing within the map reduce functions would increase processing burden by an order of magnitude, at least. Basically it boils down to the following question: how much logic can be performed within map reduce before its an unreasonably expensive query?