Lucene: Why the search cost increase with the page increase?
elasticsearch, lucene
Solution
In order to page that far, Elasticsearch has to retrieve all previous pages, just to discard them at the end. A search with `from: 1000, size: 100` is equivalent to a search with `from: 0, size: 100100`, only that you just get the last 100 results. That also means, that every document has to be scored, which is a potentially expensive operation.
There was a recent optimization, that should improve performance in such cases, when you use a scroll search, see this github issue.
Problem
I am using ElasticSearch cluster with 1G cache for each machine. I configured the searcher that 5000/perpage. When I search page 3, it cost about 400ms. But when I search page 300, the cost increased to more than 5000ms! About 60% cache was free at this situation. Why the cost increased more than 10 times?