UTF8Encode in Delphi XE2

delphi, delphi-7, delphi-xe2, unicode

Solution

In your Delphi 7 code you probably wrote something like this:

var
  UTF8: string;
  InputString: WideString;//I guess that you used WideString
.....
UTF8 := UTF8Encode(InputString);

This was fine in Delphi 7 where `string` is an alias for `AnsiString`. In XE2 the generic `string` type is now an alias for `UnicodeString` which is UTF-16 encoded. That means that when the code above is compiled by XE2, the UTF-8 encoded buffer returned by `UTF8Encode` is interpreted as UTF-16 encoded text. And that mismatch is what leads to your string full of question marks.

So, if you just wrote

var
  UTF8: AnsiString;
  InputString: string;//aliased to UnicodeString
.....
UTF8 := UTF8Encode(InputString);

then you would have the same behaviour as for your Delphi 7 code.

However, this is not the way to do it in Unicode Delphi. Instead you should use the `UTF8String` type. This is defined as `AnsiString(65001)` which means a string of 8 bit character units with code page `65001`, i.e. the UTF-8 codepage. When you do this you don't need to call `UTF8Encode` at all since the encoding attached to the string type means that the compiler can generated code to convert the string. Now you would simply write:

var
  UTF8: UTF8String;
  InputString: string;//aliased to UnicodeString
.....
UTF8 := InputString;

The principal reference for the Unicode aspects of Delphi 2009 and later is Marco Cantù's white paper: Delphi and Unicode which I recommend that you read before proceeding.

Problem

I'm trying to convert string using ``` Var encode:ansistring; begin encode:=UTF8Encode('اختبار'); showmessage(encode); end; ``` It's working fine in Delphi 7 but in Delphi XE2 it's send Text as question marks Any suggestions?

Original source