Efficiently reading a csv file with windows newline on linux in Python
csv, python, python-3.x
Solution
Python has builtin support for Windows, Linux and Mac line endings:
f = open(filename, 'rtU')
for line in f:
...
If you really want don't want slow string operations, you should strip the files before processing them. You can either use dos2unix (can be found in the Debian package "tofrodos") or (easier) use FTP text mode which should do that automatically.
Problem
The following is working under windows for reading csv files line by line. ``` f = open(filename, 'r') for line in f: ``` Though when copying the csv file to a linux server, it fails. It should be mentioned that performance is an issue as the csv files are huge. I am therefore concerned about the string copying when using things like strip.