Charset detection in Android

android, character-encoding, metadata

Solution

As vorrtex noted in his comment above, if your data comes as well formed HTML code, you can know its encoding from `<meta content="...">` tag, and this is the best scenario. You can convert this to Android (or other Java implementation) String with code like:

// assume you have your input data as byte array buf, and encoding
// something like "windows-1252", "UTF-8" or whatever
String str = new String(buf, encoding);
// now your string will display correctly

If you don't know encoding - you receive your data as a raw text in unknown format - you can still try the algorithms to guess it, using statistical language models. I just found the ICU - International Components for Unicode project by IBM, with liberal open source licensing (commercial use OK), at http://site.icu-project.org/

They provide both Java and C++ libraries. I just added their Java JAR ver. 51.2 to my Android project, and it worked like a charm. The code I used to recognize character encoding from text files is:

public static String readFileAsStringGuessEncoding(String filePath)
{
    String s = null;
    try {
        File file = new File(filePath);
        byte [] fileData = new byte[(int)file.length()];
        DataInputStream dis = new DataInputStream(new FileInputStream(file));
        dis.readFully(fileData);
        dis.close();

        CharsetMatch match = new CharsetDetector().setText(fileData).detect();

        if (match != null) try {
            Lt.d("For file: " + filePath + " guessed enc: " + match.getName() + " conf: " + match.getConfidence());
            s = new String(fileData, match.getName());
        } catch (UnsupportedEncodingException ue) {
            s = null;
        }
        if (s == null)
            s = new String(fileData);
    } catch (Exception e) {
        Lt.e("Exception in readFileAsStringGuessEncoding(): " + e);
        e.printStackTrace();
    }
    return s;
}

Lt.d and Lt.e above are just my shortcuts for Log.d(TAG, "blah..."). Worked fine on all test files that I could come up with. I was a little bit concerned only about APK file size - the icu4j-51_2.jar is over 9 MB long, and my entire package was only 2.5 MB before adding it. But it was easy to isolate the CharsetDetector and its dependencies, so I ended up adding no more than 50 kB in the end. The Java classes I needed to copy to my project from ICU sources were all under core/src/com/ibm/icu/text directory, and were:

CharsetDetector
CharsetMatch
CharsetRecog_2022
CharsetRecog_mbcs
CharsetRecog_sbcs
CharsetRecog_Unicode
CharsetRecog_UTF8
CharsetRecognizer

Additionally in CharsetRecog_sbcs.java there is a protected 'ArabicShaping as;' member, which wanted to pull a lot more classes, but turns out that for the charset recognition it's not needed, so I commented it out. That's all. Hope it helps.

Greg

Problem

My Android application retrieves SHOUTcast metadata and displays it. I'm having and issue with none-English character sets. Basically, the metadata is displayed as gibberish. How would I perform character encoding detection and display the text properly? Sorry if this is a non-trivial question, I'm not well-versed in this topic. The stream in question is: http://skully.hopto.org:8000

Original source