DataContractJsonSerializer.ReadObject sometimes throws: The token " was expected but found 'Â'

.net, datacontractjsonserializer, json, utf-8

Solution

This is due to a bug in the `DataContractJsonSerializer.ReadObject(Stream)` method that's fixed in a patch to .Net 4.0 - the developers have the patch but our users do not, which is why we didn't get it.

The bug appears if you have lots of non-ANSI characters in a single serialised object.

I wrote a simple app to check for the issue:

// Create a JSON string with more non-ANSI characters than can be handled
char test = (char) 0x6cd5;
string content = "\"" + new string(test, 2048) + "\"";

// Use a MemoryStream
byte[] result = Encoding.UTF8.GetBytes(content);
using (var s = new MemoryStream(result))
{
    var outputSerialiser = new DataContractJsonSerializer(typeof(string));

    // This line will throw the exception
    string output = (string) outputSerialiser.ReadObject(s);
}

None of the developer machines throw an exception here, but our clients' PCs do.

The fix is to use a buffered JSON reader instead of a `MemoryStream`:

using (var jsonReader = JsonReaderWriterFactory.CreateJsonReader(result, XmlDictionaryReaderQuotas.Max))
{
    var outputSerialiser = new DataContractJsonSerializer(typeof(string));
    string output = (string) outputSerialiser.ReadObject(jsonReader);
}

Problem

I have an issue that occurs for our customers, but doesn't occur for any of the developers when using the same application version and data. `DataContractJsonSerializer.ReadObject` throws this exception: `Deserialising: There was an error deserializing the object of type {type}.` `The token '"' was expected but found 'Â'.` This exception is not thrown when any of the developers or I attempt to reproduce it, but consistently occurs on the client systems. Everyone is using Windows 7 64bit. My best guess is that this is a text encoding issue, as a UTF-8 byte pair of `0xC2,0x??` would end up as `Â` if converted to Windows 1252 or ISO 8859-1. The conversion to UTF-8 is being done in code: ``` string content = GetSerialised(); byte[] result = Encoding.UTF8.GetBytes(content); using (var s = new MemoryStream(result)) { var outputSerialiser = new DataContractJsonSerializer(typeof(T), null, int.MaxValue, true, null, false); return (T) outputSerialiser.ReadObject(s); } ``` The `content` is displayed with the error message, so we're able to verify that it is valid JSON text. It does contain a `¦` quoted in a JSON string (that's `0xC2,0xA6` in UTF-8, so a likely culprit for the corrupted character). So we have valid JSON in a .Net `string` instance, we convert it using `UTF8.GetBytes`, but when `DataContractJsonSerializer.ReadObject` reads it we get a corruption. Any idea why this is happening? Why does it occur on some machines and not others?

Original source