How to: Increase Lucene .net Indexing Speed

indexing, lucene.net

Solution

I wrote a terrible post on how to parallelize a Lucene Index. It's truly terribly written, but you'll find it here (there's some sample code you might want to look at).

Anyhow, the main idea is that you chunk up your data into sizable pieces, and then work on each of those pieces on a separate thread. When each of the pieces is done, you merge them all into a single index.

With the approach described above, I'm able to index 4+ million records in approx. 2 hours.

Hope this gives you an idea of where to go from here.

Problem

Original source