Characters defined using '\uxxxx' format display the wrong character

c++, character-encoding, unicode

Solution

Universal Character Names (UCNs) are a method of communicating to the compiler the character that you want to represent. As long as you can get the basic source characters to a compiler then every compiler will see the same UCN and therefore will see that you are representing the same character.

This is as opposed to writing the character literally in the source:

char a = '☹';

Since compilers are only required to support the basic source characters a compiler may not even be able to process this code. And what it actually sees depends on the source encoding the compiler uses. One compiler may see the character you want while another compiler sees

char a = 'Â☐¹';

However, simply because UCNs are able to specify the character to the compiler does not mean that:

- the compiler's execution charset contains that character or

- the datatype `char` can represent that character value

In your case the primary problem is that the execution character set is one of Windows' code pages (probably CP1252) which does not have the character '☹'. So when the compiler converts the character '☹' into the execution character set, the conversion produces '?' instead of what you want.

The execution character set for my compiler does include the character '☹', but it happens to have a multi-byte representation so my compiler says:

error: character too large for enclosing character literal type
    char a = '☹';
             ^

To really understand this topic you need to understand encodings, character sets, how these play into the C++ phases of translation, and how that relates to the compiler's processing of character and string literals. Also, locales really have nothing to do with any of this; locales deal with runtime behavior whereas your problem is entirely with your compiler's compile-time handling of encodings.

On a platform that uses UTF-8 everywhere the following works:

#include <iostream>

int main() {
    std::cout << "☹\n";
}

Note that the above uses a string literal rather than a character literal, so that the character can expand to its multi-byte representation.

Unfortunately Windows does not support Unicode this way. On Windows it's more complicated:

#include <Windows.h>
#include <cwchar>

int main() {
    wchar_t const *a = L"\u2639\n";
    DWORD numOfCharsWritten;
    WriteConsole(GetStdHandle(STD_OUTPUT_HANDLE), a, wcslen(a), &numOfCharsWritten, NULL);
}

Unfortunately even the above code is unlikely to display what you want, because the console on Windows typically is not configured to be able to display the Unicode character '☹'. Instead you might want to take a look at the OEM encoding used by your console (probably CP437), look up the encoding for a character you want, and then print out that value. For example CP437 has the character '☺' instead, and you could print that out like this:

#include <iostream>

int main() {
    std::cout << "\x01\n"; // ☺ has the value 0x01 in CP 437
}

Problem

Original source