How to process huge text files that contain EOF / Ctrl-Z characters using Python on Windows?

Question

I have a number of large comma-delimited text files (the biggest is about 15GB) that I need to process using a Python script. The problem is that the files sporadically contain DOS EOF (Ctrl-Z) characters in the middle of them. (Don't ask me why, I didn't generate them.) The other problem is that the files are on a Windows machine. On Windows, when my script encounters one of these characters, it assumes it is at the end of the file and stops processing. For various reasons, I am not allowed to copy the files to any other machine. But I still need to process them. Here are my ideas so far: - Read the file in binary mode, throwing out bytes that equal `chr(26)`. This would work, but it would take approximately forever. - Use something like `sed` to eliminate the EOF characters. Unfortunately, as far as I can tell, `sed` on Windows has the same problem and will quit when it sees the EOF. - Use some kind of `Notepad` program and do a find-and-replace. But it turns out that `Notepad`-type…

Accepted Answer

It's easy to use Python to delete the DOS EOF chars; for example,

```
def delete_eof(fin, fout):
    BUFSIZE = 2**15
    EOFCHAR = chr(26)
    data = fin.read(BUFSIZE)
    while data:
        fout.write(data.translate(None, EOFCHAR))
        data = fin.read(BUFSIZE)

import sys
ipath = sys.argv[1]
opath = ipath + ".new"
with open(ipath, "rb") as fin, open(opath, "wb") as fout:
    delete_eof(fin, fout)
```

That takes a file path as its first argument, and copies the file but without `chr(26)` bytes to the same file path with `.new` appended. Fiddle to taste.

By the way, are you sure that DOS EOF characters are your only problem? It's hard to conceive of a sane way in which they could end up in files intended to be treated as text files.

How to process huge text files that contain EOF / Ctrl-Z characters using Python on Windows?

Solution

Problem