Cast from RawByteString to string does automatically invoke UTF8Decode?

delphi, encoding, unicode, utf-8

Solution

In Ansi versions of Delphi (prior to D2009), `UTF8Encode()` returns a UTF-8 encoded `AnsiString`. In Unicode versions (D2009 and later), it returns a UTF-8 encoded `RawByteString` with a code page of `CP_UTF8` (65001) assigned to it.

In Ansi versions, `ShowMessage()` takes an `AnsiString` as input, and the UTF-8 string is an `AnsiString`, so it gets displayed as-is. In Unicode versions, `ShowMessage()` takes a UTF-16 encoded `UnicodeString` as input, so the UTF-8 encoded `RawByteString` gets converted to UTF-16 using its assigned `CP-UTF8` code page.

If you actually wrote the `blob` data directly to the database you would find that it may or may not be UTF-8 encoded, depending on how you are writing it. But your approach is wrong; the use of `RawByteString` is incorrect in this situation. `RawByteString` is meant to be used as a procedure parameter only. Do not use it as a local variable. That is the source of your problem. From the documentation:

The purpose of RawByteString is to reduce the need for multiple overloads of procedures that read string data. This means that parameters of routines that process strings without regard for the string's code page should typically be of type RawByteString.

RawByteString should only be used as a parameter type, and only in routines which otherwise would need multiple overloads for AnsiStrings with different codepages. Such routines need to be written with care for the actual codepage of the string at run time.

For Unicode versions of Delphi, instead of `RawByteString`, I would suggest that you use `TBytes` to hold your UTF-8 data, and encode it with `TEncoding`:

var
  utf8: TBytes;
  str: string;
...
str := ...;
utf8 := TEncoding.UTF8.GetBytes(str);

You are looking for a data type that does not perform implicit text encodings when passed around, and `TBytes` is that type.

For Ansi versions of Delphi, you can use `AnsiString`, `WideString` and `UTF8Encode` exactly as you do.

Personally however, I would recommend using `TBytes` consistently for your UTF-8 data. So if you need a single code base that supports Ansi and Unicode compilers (ugh!) then you should create some helpers:

{$IFDEF Unicode}
function GetUTF8Bytes(const Value: string): TBytes;
begin
  Result := TEncoding.UTF8.GetBytes(Value);
end;
{$ELSE}
function GetUTF8Bytes(const Value: WideString): TBytes;
var
  utf8str: UTF8String;
begin
  utf8str := UTF8Encode(Value);
  SetLength(Result, Length(utf8str));
  Move(Pointer(utf8str)^, Pointer(Result)^, Length(utf8str));
end;
{$ENDIF}

The Ansi version incurs more heap allocations than are necessary. You might well choose to write a more efficient helper that calls `WideCharToMultiByte()` directly.

In Unicode versions of Delphi, if for some reason you don't want to use `TBytes` for UTF-8 data, you can use `UTF8String` instead. This is a special `AnsiString` that always uses the `CP_UTF8` code page. You can then write:

var
  utf8: UTF8String;
  str: string;
....
utf8 := str;

and the compiler will convert from UTF-16 to UTF-8 behind the scenes for you. I would not recommend this though, because it is not supported on mobile platforms, or in Ansi versions of Delphi (`UTF8String` has existed since Delphi 6, but it was not a true UTF-8 string until Delphi 2009). That is, amongst other reasons, why I suggest that you use `TBytes`. My philosophy is, at least in the Unicode age, that there is the native `string` type, and any other encoding should be held in `TBytes`.

Problem

Original source