Cannot parse and display non-utf8 characters read from an http request

encoding, java, json, parsing

Solution

The issue you have is most likely caused by incorrect setting of the character encoding at the point that you are reading in the http response from google. Can you post the code that actually gets URL and parses it into the JSON object?

As an example run the following:

public class Test1 {
  public static void main(String [] args) throws Exception {

    // just testing that the console can output the correct chars
    System.out.println("\"title\":\"مطبخ مطايب - كباب الدجاج والخضار بصلصة الروب");

    URL url = new URL("http://ajax.googleapis.com/ajax/services/search/web?start=0&rsz=large&v=1.0&q=rz+img+news+recordid+border");
    HttpURLConnection connection = (HttpURLConnection) url.openConnection();
    InputStream is  = connection.getInputStream();

    // the important bit is here..........................\/\/\/
    InputStreamReader reader = new InputStreamReader(is, "utf-8");


    StringWriter sw = new StringWriter();

    char [] buffer = new char[1024 * 8];
    int count ;

    while( (count = reader.read(buffer)) != -1){
      sw.write(buffer, 0, count);
    }

    System.out.println(sw.toString());
  }
}

This is using the rather ugly standard `URL.openConnection()` that's been around since the dawn of time. If you are using something like Apache httpclient then you can do this really easily.

For a bit of back ground reading on encoding and maybe an explaination of why `new String (str.getBytes(), "UTF8");` will never work read Joel's article on unicode

Problem

Original source

Related problems

• How to Find the Default Charset/Encoding in Java?