Regex search pattern in very large file

Question

I'd like to search pattern in very large file (f.e above 1 GB) that consists of single line. It is not possible to load it into memory. Currently, I use `BufferedReader`to read into buffers (1024 chars). The main steps: - Read data into two buffers - Search pattern in that buffers - Increment variable if pattern was found - Copy second buffer into first - Load data into second buffers - Search pattern in both buffers. - Increment variable if pattern was found - Repeat above steps (start from 4) until EOF That algorithm (two buffers) lets me to avoid situation, where searched piece of text is split by chunks. It works like a chram unless pattern result is smaller that two buffers length. For example I can't manage with case, when result is longer - let's say long as 3 buffers (but I've only data in two buffers, so match will fail!). What's more, I can realize such a case: - Prepare 1 GB single line file, that consits of "baaaaaaa(....)aaaaab" - Search for pattern `ba*b`. - The whole fi…

Accepted Answer

I think the solution for you would be to implement `CharSequence` as a wrapper over very large text files.

Why? Because building a `Matcher` from a `Pattern` takes a `CharSequence` as an argument.

Of course, easier said than done... But then you only have three methods to implement, so that shouldn't be too hard...

EDIT I took the plunge and I ate my own dog's food. The "worst part" is that it actually works!

Regex search pattern in very large file

Solution

Problem