How to sort an array of UTF-8 strings?

arrays, php, sorting, utf-8

Solution

Eventually this problem cannot be solved in a simple way without using recoded strings (UTF-8 → Windows-1252 or ISO-8859-1) as suggested by ΤΖΩΤΖΙΟΥ due to an obvious PHP bug as discovered by Huppie. To summarize the problem, I created the following code snippet which clearly demonstrates that the problem is the strcoll() function when using the 65001 Windows-UTF-8-codepage.

function traceStrColl($a, $b) {
    $outValue=strcoll($a, $b);
    echo "$a $b $outValue\r\n";
    return $outValue;
}

$locale=(defined('PHP_OS') && stristr(PHP_OS, 'win')) ? 'German_Germany.65001' : 'de_DE.utf8';

$string="ABCDEFGHIJKLMNOPQRSTUVWXYZÄÖÜabcdefghijklmnopqrstuvwxyzäöüß";
$array=array();
for ($i=0; $i<mb_strlen($string, 'UTF-8'); $i++) {
    $array[]=mb_substr($string, $i, 1, 'UTF-8');
}
$oldLocale=setlocale(LC_COLLATE, "0");
var_dump(setlocale(LC_COLLATE, $locale));
usort($array, 'traceStrColl');
setlocale(LC_COLLATE, $oldLocale);
var_dump($array);

The result is:

string(20) "German_Germany.65001"
a B 2147483647
[...]
array(59) {
  [0]=>
  string(1) "c"
  [1]=>
  string(1) "B"
  [2]=>
  string(1) "s"
  [3]=>
  string(1) "C"
  [4]=>
  string(1) "k"
  [5]=>
  string(1) "D"
  [6]=>
  string(2) "ä"
  [7]=>
  string(1) "E"
  [8]=>
  string(1) "g"
  [...]

The same snippet works on a Linux machine without any problems producing the following output:

string(10) "de_DE.utf8"
a B -1
[...]
array(59) {
  [0]=>
  string(1) "a"
  [1]=>
  string(1) "A"
  [2]=>
  string(2) "ä"
  [3]=>
  string(2) "Ä"
  [4]=>
  string(1) "b"
  [5]=>
  string(1) "B"
  [6]=>
  string(1) "c"
  [7]=>
  string(1) "C"
  [...]

The snippet also works when using Windows-1252 (ISO-8859-1) encoded strings (of course the mb_* encodings and the locale must be changed then).

I filed a bug report on bugs.php.net: Bug #46165 strcoll() does not work with UTF-8 strings on Windows. If you experience the same problem, you can give your feedback to the PHP team on the bug-report page (two other, probably related, bugs have been classified as bogus - I don't think that this bug is bogus ;-).

Thanks to all of you.

Problem

I currentyl have no clue on how to sort an array which contains UTF-8 encoded strings in PHP. The array comes from a LDAP server so sorting via a database (would be no problem) is no solution. The following does not work on my windows development machine (although I'd think that this should be at least a possible solution): ``` $array=array('Birnen', 'Äpfel', 'Ungetüme', 'Apfel', 'Ungetiere', 'Österreich'); $oldLocal=setlocale(LC_COLLATE, "0"); var_dump(setlocale(LC_COLLATE, 'German_Germany.65001')); usort($array, 'strcoll'); var_dump(setlocale(LC_COLLATE, $oldLocal)); var_dump($array); ``` The output is: ``` string(20) "German_Germany.65001" string(1) "C" array(6) { [0]=> string(6) "Birnen" [1]=> string(9) "Ungetiere" [2]=> string(6) "Äpfel" [3]=> string(5) "Apfel" [4]=> string(9) "Ungetüme" [5]=> string(11) "Österreich" } ``` This is complete nonsense. Using 1252 as the codepage for `setlocale()` gives another output but still a plainly wrong one: ``` string(19) "German_Germany.1252" string(1) "C" array(6) { [0]=> string(11) "Österreich" [1]=> string(6) "Äpfel" [2]=> string(5) "Apfel" [3]=> string(6) "Birnen" [4]=> string(9) "Ungetüme" [5]=> string(9) "Ungetiere" } ``` Is there a way to sort an array with UTF-8 strings locale aware? Just noted that this seems to be PHP on Windows problem, as the same snippet with `de_DE.utf8` used as locale works on a Linux machine. Nevertheless a solution for this Windows-specific problem would be nice...

Original source

Related problems