Detect text file encoding
c++, character-encoding, qt
Solution
Turns out that auto-detecting the encoding is impossible for the general case.
However, there is a workaround to at least fall back to the system locale if the text is not valid UTF-8/UTF-16/UTF-32 text. It uses `QTextCodec::codecForUtfText()`, which tries to decode a byte array using UTF-8, UTF-16 and UTF-32, and returns the supplied default codec if it fails.
Code to do it:
QTextCodec *codec = QTextCodec::codecForUtfText(byteArray, QTextCodec::codecForName("System"));
const QString &text = codec->toUnicode(byteArray);
Update
The above code will not detect UTF-8 without BOM, however, as codecForUtfText() relies on the BOM markers. To detect UTF-8 without BOM, see https://stackoverflow.com/a/18228382/492336.
Problem
In my program I load plain text files supplied by the user: ``` QFile file(fileName); file.open(QIODevice::ReadOnly); QTextStream stream(&file); const QString &text = stream.readAll(); ``` This works fine when the files are UTF-8 encoded, but some users try to import Windows-1252 encoded files, and if they have words with special characters (for example "è" in "boutonnière"), those will show incorrectly. Is there a way to detect the encoding, or at least distinguish between UTF-8 (possibly without BOM), and Windows-1252, without asking the user to tell me the encoding?