How to access directly and efficiently on very large text file?
c, c++, file-io, mpi
Solution
If your file isn't otherwise indexed, there is no direct way.
Indexing it might be worth it (scan it once to find all the line endings, and store the offsets of each line or chunk of lines). If you need to process the file multiple times, and it does not change, the cost of indexing it could be offset by the ease of using the index for further runs.
Otherwise, if you don't need all the jobs to have exactly the same number of lines/items, you could just fudge it. Seek to a given offset (say 1G), and look for the closest line separator. Repeat at offset 2G, etc. until you've found enough break points.
You can then fire off your parallel tasks on each of the chunks you've identified.
Problem
I have a very large text files (+10GB) which i want to read for some data mining technics. To do that, i use parallel technics with MPI so many processes can access together to the same file. In fact, i want that each process read N number of lines. Since the file is not structured (same number of fields but each field can contain different number of characters), i'm in the obligation to parse the file and that is not parallel and it takes a lot of time. Is there any way to access directly to a specific number of line withount parsing and counting the lines? Thank you for you help.