Optimizing the design of a backend search engine
elasticsearch, mongodb, optimization, redis
Solution
Summary from comments:
You don't need Redis for the caching as EL takes care of all those and it's scalable the same way. http://www.elasticsearch.org/guide/en/elasticsearch/guide/current/filter-caching.html
You should not store the entire HTML, you don't need that phase, as EL responds in under 50ms with your results.
For hot/trending static pages, you could have easily a Varnish before your chosen web server, and that could carry 10k ops/sec
Use CDN, Only if CDN costs are less than paying for a instance which can generate it on the fly.
Don't put in the same bucket time of templating and time of accessing from storage. While the first one I think will be under 1ms, the later - on a good cluster could be 50ms. Maybe you will have another 50ms from all the other routes (DNS, load-balancer, logging), so I think that this way you can serve under 110ms the whole static thing.
- if you will go with Elasticsearch, you don't need MongoDB at all, as EL is also a document storage.
Problem
TL;DR: this question is about `optimizing the design of a backend search engine`: the idea is to return a fully-formed html page that displays search results as quickly as possible. I figured elasticsearch and redis would be suitable candidates for the job, though I have not decided yet. The server will be node.js, the database mongoDB. No frontend framework will be used, plain html will be returned by the server. I came up with the following server-side design*: (*) NOTE: I have little experience in server-side designing, so my approach is probably naive A first search would: ``` - run a server function that makes an elasticSearch query and returns some json; - create scraps of HTML from the returned json and store them in Redis (for caching purposes); - store the search keywords and the keys of the html scraps in Redis. ``` Whereas subsequent searches would just: ``` - recognize the search keywords, and get the html scraps keys from Redis; - get the html scraps values; ``` The server would then: ``` - build the html page from the scraps; - Return the html page. ``` Let's elaborate further with an example: Imagine you have a rather large collection of articles, say 100,000, which you intend to sell on your website. `All articles are stacked in a MongoDb database` and have multiple keys (title, categories, reviews, pictures, etc...) To implement search, the article collection has been `indexed in elasticSearch`. A first elasticSearch query would return a `json containing a list of articles`. Subsequent queries to elasticSearch, with the same search terms, would return the same json, but would now `extract it from the elasticSearch cache`. But if you're NOT using frontend technology and want `plain html` returned from the server, you would still need to `insert that json into a templator, create an html page, then return to the end user`. If you cached the `whole html page into a superfast key/value database like Redis`, neither the templator nor elasticSearch would be used on subsequent searches: Redis would recognise the search terms, and return the html straight away. Or if, as suggested in the comments/answers, Redis was skipped, the `html could be stored into a cdn` after it has been computed by the templator. The question: Please specify which steps of suggested design are unnecessary/missing, and why. Thank you for your help!