Serializing supplementary unicode characters into XML documents with Java
java, unicode, xml, xml-serialization, xslt
Solution
Since I didn't see any answer coming, and other people seem to have the same problem, I looked into it further...
To find the origin of the bug, I used the `serializer` source code from `Xalan 2.7.1`, which is also used in `Xerces`.
`org.apache.xml.serializer.dom3.LSSerializerImpl` uses `org.apache.xml.serializer.ToXMLStream`, which extends `org.apache.xml.serializer.ToStream`.
`ToStream.characters(final char chars[], final int start, final int length)` handles the characters, and does not support unicode characters properly (note: `org.apache.xml.serializer.ToTextSream` (which can be used with a `Transformer`) does a better job in the characters method, but it only handles plain text and ignores all markup; one would think that XML files are text, but for some reason `ToXMLStream` does not extend `ToTextStream`).
`org.apache.xalan.transformer.TransformerIdentityImpl` is also using `org.apache.xml.serializer.ToXMLStream` (which is returned by `org.apache.xml.serializer.SerializerFactory.getSerializer(Properties format)`), so it suffers from the same bug.
`ToStream` is using `org.apache.xml.serializer.CharInfo` to check if a character should be replaced by a `String`, so the bug could also be fixed there instead of directly in `ToStream`. `CharInfo` is using a propery file, `org.apache.xml.serializer.XMLEntities.properties`, with a list of character entities, so changing this file could also be a way to fix the bug, although so far it is designed just for the special XML characters (`quot`,`amp`,`lt`,`gt`). The only way to make `ToXMLStream` use a different property file than the one in the package would be to add a `org.apache.xml.serializer.XMLEntities.properties` file before in the classpath, which would not be very clean...
With the default JDK (1.6 and 1.7), `TransformerFactory` returns a `com.sun.org.apache.xalan.internal.xsltc.trax.TransformerImpl`, which uses `com.sun.org.apache.xml.internal.serializer.ToXMLStream`. In `com.sun.org.apache.xml.internal.serializer.ToStream`, `characters()` is sometimes calling `processDirty()`, which calls `accumDefaultEscape()`, which could handle unicode characters better, but in practice it does not seem to work (maybe `processDirty` is not called for unicode characters)...
`com.sun.org.apache.xml.internal.serialize.DOMSerializerImpl` is using `com.sun.org.apache.xml.internal.serialize.XMLSerializer`, which supports unicode. Strangely enough, `XMLSerialize`r comes from `Xerces`, and yet it is not used by `Xerces` when `xalan` or `xsltc` are on the classpath. This is because `org.apache.xerces.dom.CoreDOMImplementationImpl.createLSSerializer` is using `org.apache.xml.serializer.dom3.LSSerializerImpl` when it is available instead of `org.apache.xerces.dom.DOMSerializerImpl`. With `serializer.jar` on the classpath, `org.apache.xml.serializer.dom3.LSSerializerImpl` is used. Warning: `xalan.jar` and `xsltc.jar` both reference `serializer.jar` in the manifest, so `serializer.jar` ends up on the classpath if it is in the same directory and either `xalan.jar` or `xsltc.jar` is on the classpath ! If only `xercesImpl.jar` and `xml-apis.jar` are on the classpath, `org.apache.xerces.dom.DOMSerializerImpl` is used as the `LSSerializer`, and unicode characters are properly handled.
CONCLUSION AND WORKAROUND: the bug lies in Apache's `org.apache.xml.serializer.ToStream` class (renamed `com.sun.org.apache.xml.internal.serializer.ToStream` inside the JDK). A serializer that handles unicode characters properly is `org.apache.xml.serialize.DOMSerializerImpl` (renamed `com.sun.org.apache.xml.internal.serialize.DOMSerializerImpl` inside the JDK). However, Apache prefers `ToStream` instead of `DOMSerializerImpl` when it is available, so maybe it behaves better for other things (or maybe it's just a reorganization). On top of that, they went as far as deprecating `DOMSerializerImpl` in `Xerces 2.9.0`. Hence the following workaround, which might have side effects :
when `Xerces` and Apache's `serializer` are on the classpath, replace "`(doc.getImplementation()).createLSSerializer()`" by "`new org.apache.xerces.dom.DOMSerializerImpl()`"
when Apache's `serializer` is on the classpath (for instance because of `xalan`) but not `Xerces`, try to replace "`(doc.getImplementation()).createLSSerializer()`" by "new `com.sun.org.apache.xml.internal.serialize.DOMSerializerImpl()`" (a fallback is necessary because this class might disappear in the future)
These 2 workarounds produce a warning when compiling.
I don't have a workaround for `XSLT transforms`, but this is beyond the scope of the question. I guess one could do a transform to another DOM document and use `DOMSerializerImpl` to serialize.
Some other workarounds, which might be a better solution for some people :
use `Saxon` with a `Transformer`
use XML documents with `UTF-16` encoding
Problem
I am trying to serialize DOM documents with supplementary unicode characters such as U+1D49C (𝒜, mathematical script capital A). Creating a node with such a character is not a problem (I just set the node value to the UTF-16 equivalent, "\uD835\uDC9C"). When serializing, however, Xalan and XSLTC (with a Transformer) and Xerces (with LSSerializer) all create invalid character entities like "��" instead of "𝒜". I tried the "normalize-characters" parameter for LSSerializer, but it is not supported. Only Saxon gets it right, without using a character entity when the encoding is unicode. I cannot use Saxon in practice (among other reasons, I use Java applets and do not want to load another jar), so I am looking for a solution with the default JDK libraries. Is it possible to get valid XML documents serialized from a DOM document with supplementary unicode characters ? [edit] I found someone else who encountered this problem : http://www.dragishak.com/?p=131 [edit2] actually, it seems to work with LSSerializer when I don't have xerces on the classpath (the class used is com.sun.org.apache.xml.internal.serialize.DOMSerializerImpl). It does not work with a transformer and com.sun.org.apache.xalan.internal.xsltc.trax.TransformerFactoryImpl.