Interesting Lucene.net Exception
.net, c#, lucene.net, multithreading
Solution
The source for MMapDirectory shows that this class does not use memory-mapped files, as expected. It loads all index files into memory using MemoryStream objects, and I would guess that those streams are the cause of the problem when different threads seeks and reads.
You can get a memory-based index by loading it into a RAMDirectory. This passes your test. (But it does what MMapDirectory currently does, not necessarily what you expect it to do...)
var fsDirectory = FSDirectory.Open(directoryInfo);
var directory = new RAMDirectory(fsDirectory);
Problem
According to this or this, I used the same indexsearcher by multiple thread. But when I switched from FsDirectory to MMapDirectory, I got interesting exceptions. This work fine: ``` static void Main(string[] args) { DirectoryInfo directoryInfo = new DirectoryInfo(@"C:\Users\Tams\Desktop\new\"); var directory = FSDirectory.Open(directoryInfo); var indexSearcher = new IndexSearcher(directory); const int times = 100; const int concurrentTaskCount = 5; var task = new Task[concurrentTaskCount]; for (int i = 0; i < concurrentTaskCount; i++) { task[i] = new Task(() => Search(indexSearcher, times)); task[i].Start(); } Task.WaitAll(task); } static void Search(IndexSearcher reader, int times) { List<Document> docs = new List<Document>(10000); for (int i = 0; i < times; i++) { var q = new TermQuery(new Term("title", "volume")); foreach (var scoreDoc in reader.Search(q, 100).ScoreDocs) { docs.Add(reader.Doc(scoreDoc.Doc)); } } } ``` But with this: ``` static void Main(string[] args) { DirectoryInfo directoryInfo = new DirectoryInfo(@"C:\Users\Tams\Desktop\new\"); var directory = new MMapDirectory(directoryInfo); // CHANGED var indexSearcher = new IndexSearcher(directory); const int times = 100; const int concurrentTaskCount = 5; var task = new Task[concurrentTaskCount]; for (int i = 0; i < concurrentTaskCount; i++) { task[i] = new Task(() => Search(indexSearcher, times)); task[i].Start(); } Task.WaitAll(task); } static void Search(IndexSearcher reader, int times) { List<Document> docs = new List<Document>(10000); for (int i = 0; i < times; i++) { var q = new TermQuery(new Term("title", "volume")); foreach (var scoreDoc in reader.Search(q, 100).ScoreDocs) { docs.Add(reader.Doc(scoreDoc.Doc)); } } } ``` I get various exceptions like: ``` System.ArgumentOutOfRangeException: Index was out of range. Must be non-negative and less than the size of the collection. Parameter name: index at System.ThrowHelper.ThrowArgumentOutOfRangeException() at System.Collections.Generic.List`1.get_Item(Int32 index) at Lucene.Net.Index.FieldInfos.FieldInfo(Int32 fieldNumber) in d:\Lucene.Net\FullRepo\trunk\src\core\Index\FieldInfos.cs:line 378 at Lucene.Net.Index.FieldsReader.Doc(Int32 n, FieldSelector fieldSelector) in d:\Lucene.Net\FullRepo\trunk\src\core\Index\FieldsReader.cs:line 234 at Lucene.Net.Index.SegmentReader.Document(Int32 n, FieldSelector fieldSelector) in d:\Lucene.Net\FullRepo\trunk\src\core\Index\SegmentReader.cs:line 1193 at Lucene.Net.Index.DirectoryReader.Document(Int32 n, FieldSelector fieldSelector) in d:\Lucene.Net\FullRepo\trunk\src\core\Index\DirectoryReader.cs:line 686 at Lucene.Net.Index.IndexReader.Document(Int32 n) in d:\Lucene.Net\FullRepo\trunk\src\core\Index\IndexReader.cs:line 732 at Lucene.Net.Search.IndexSearcher.Doc(Int32 i) in d:\Lucene.Net\FullRepo\trunk\src\core\Search\IndexSearcher.cs:line 162 at PerformanceTest.Program.Search(IndexSearcher reader, Int32 times) in c:\Users\Tams\Documents\Visual Studio 2012\Projects\BookCatalog\PerformanceTest\Program.cs:line 28 at PerformanceTest.Program.<>c__DisplayClass2.<Main>b__0() in c:\Users\Tams\Documents\Visual Studio 2012\Projects\BookCatalog\PerformanceTest\Program.cs:line 43 at System.Threading.Tasks.Task.InnerInvoke() at System.Threading.Tasks.Task.Execute() ``` Or ``` System.IO.IOException: read past EOF at Lucene.Net.Store.BufferedIndexInput.Refill() in d:\Lucene.Net\FullRepo\trunk\src\core\Store\BufferedIndexInput.cs:line 179 at Lucene.Net.Store.BufferedIndexInput.ReadByte() in d:\Lucene.Net\FullRepo\trunk\src\core\Store\BufferedIndexInput.cs:line 41 at Lucene.Net.Store.IndexInput.ReadVInt() in d:\Lucene.Net\FullRepo\trunk\src\core\Store\IndexInput.cs:line 88 at Lucene.Net.Index.FieldsReader.Doc(Int32 n, FieldSelector fieldSelector) in d:\Lucene.Net\FullRepo\trunk\src\core\Index\FieldsReader.cs:line 230 at Lucene.Net.Index.SegmentReader.Document(Int32 n, FieldSelector fieldSelector) in d:\Lucene.Net\FullRepo\trunk\src\core\Index\SegmentReader.cs:line 1193 at Lucene.Net.Index.DirectoryReader.Document(Int32 n, FieldSelector fieldSelector) in d:\Lucene.Net\FullRepo\trunk\src\core\Index\DirectoryReader.cs:line 686 at Lucene.Net.Index.IndexReader.Document(Int32 n) in d:\Lucene.Net\FullRepo\trunk\src\core\Index\IndexReader.cs:line 732 at Lucene.Net.Search.IndexSearcher.Doc(Int32 i) in d:\Lucene.Net\FullRepo\trunk\src\core\Search\IndexSearcher.cs:line 162 at PerformanceTest.Program.Search(IndexSearcher reader, Int32 times) in c:\Users\Tams\Documents\Visual Studio 2012\Projects\BookCatalog\PerformanceTest\Program.cs:line 28 at PerformanceTest.Program.<>c__DisplayClass2.<Main>b__0() in c:\Users\Tams\Documents\Visual Studio 2012\Projects\BookCatalog\PerformanceTest\Program.cs:line 43 at System.Threading.Tasks.Task.InnerInvoke() at System.Threading.Tasks.Task.Execute() ``` The last code work fine, with setting the concurrentTaskCount variable to 1. Am I missing something? I cant figure out what that is. Actually, I dont have the path d:\Lucene.Net\FullRepo\trunk\src\core\Store\BufferedIndexInput.cs I don't even have a drive with letter "d"