Support for UTF-8 encoded string in TRestClient Delphi XE5

delphi, delphi-xe5, twitter, utf-8

Solution

Twitter's `1.1/statuses/update.json` URL expects data to be encoded in `application/x-www-form-urlencoded` format, so you need to set the `TRESTClient.ContentType` property to `ctAPPLICATION_X_WWW_FORM_URLENCODED` (it is set to `ctNone` by default).

As for UTF-8, `TRESTClient` uses Indy internally, and Indy supports encoding outbound data using user-specified charsets, but it does not appear that Embarcadero added that feature to its `TRESTClient` interface (it does handle charsets in responses, though). I do not know why Embarcadero would omit such an important feature. It is not enough to just encode the string data as UTF-8 (which you are not doing correctly, BTW), but you also have to tell Twitter that the data has been UTF-8 encoded (via the `charset` attribute of the `Content-Type` REST header), and `TRESTClient` does not allow you to do that, as far as I can see. I don't know if `TRESTClient` sends REST requests with a default charset specified, but looking at its source, I don't think it does, but I have not tried it.

At the very least, you need to fix your `EncodeAsUTF8()` function. It does not produce a `UnicodeString` that holds UTF-8 encoded octets, like you think it does. It produces a UTF-8 encoded `AnsiString` and then converts that to a UTF-16 encoded `UniodeString` using the RTL's default Ansi codepage, so you are invoking a data conversion that loses the UTF-8 data. Try this instead:

function TTwitterApi.EncodeAsUTF8(UnicodeStr: string): string;
var
  UTF8Str: UTF8String;
  I: Integer;
begin
  UTF8Str := UTF8String(UnicodeStr);
  SetLength(Result, Length(UTF8Str));
  for I := 1 to Length(UTF8Str) do
    Result[I] := Char(Ord(UTF8Str[I]));
end;

That should allow `TRESTClient` to url-encode the correct UTF-8 data in its POST data, at least. But you still have to deal with the issue of the missing `charset` attribute in the `Content-Type` request header (unless Twitter defaults to UTF-8 when no `charset` is specified).

Now, with all of that said, if you find that working around the `TRESTClient` problems does not work out for you, then I would suggest switching to Indy's `TIdHTTP` component instead (which has a more accurate `application/x-www-form-urlencoded` implementation than `TRESTClient` is using), eg:

procedure TTwitterApi.Send(Tweet: string);
var
  Params: TStringList;
begin
  Reset;

  Params := TStringList.Create;
  try
    FParams.Add('status=' + Tweet);
    FIdHTTP.Request.ContentType := 'application/x-www-form-urlencoded';
    FIdHTTP.Request.Charset := 'utf-8';
    FIdHTTP.Post('https://api.twitter.com/1.1/statuses/update.json', Params, IndyTextEncoding_UTF8);
  finally
    Params.Free;
  end;
end;

Problem

I want to send tweets using the new TRest components in Delphi XE5. I am looking for a way to UTF8 encode my tweet which contains IS0-8859-1 characters. The code below works, but involves codepage conversion etc. Is the a better way? Anyone? ``` procedure TTwitterApi.Send(Tweet: string); begin Reset; // Encode as UTF8 within (UTF-16 Delphi) string Tweet := EncodeAsUTF8(Tweet); FRestRequest.Resource := '1.1/statuses/update.json'; FRestRequest.Method := rmPOST; FRestRequest.Params.AddItem('status', Tweet, pkGETorPOST); FRestRequest.Execute; end; function TTwitterApi.EncodeAsUTF8(UnicodeStr: string): string; var UTF8Str: AnsiString; TempStr: RawByteString; begin TempStr := UTF8Encode(UnicodeStr); SetLength(UTF8Str, Length(TempStr)); Move(TempStr[1], UTF8Str[1], Length(UTF8Str)); Result := UTF8Str; end; ```

Original source