Unicode normalization (form C) in R : convert all characters with accents into their one-unicode-character form?
encoding, latin, r, unicode, unicode-normalization
Solution
Ok, it appears that a package has been developed to enhance and simplify the string manipulation toolbox in R (finally!). It is called stringi and looks very promising. Its documentation is very well written, and in particular I find the pages about encodings and locales much more enlightening than some of the standard R documentation on the subject.
It has Unicode normalization functions, as I was looking for (here form C):
> stri_trans_nfc('\u00e9') == stri_trans_nfc('\u0065\u0301')
[1] TRUE
It also contains a smart comparison function which integrates these normalization questions and lessens the pain of having to think about them:
> stri_compare('\u00e9', '\u0065\u0301')
[1] 0
# i.e. equal ;
# otherwise it returns 1 or -1, i.e. greater or lesser, in the alphabetic order.
Thanks to the developers, Marek Gągolewski and Bartek Tartanus, and to Kurt Hornik for the info!
Problem
In Unicode, letters with accents can be represented in two ways: the accentuated letter itself, and the combination of the bare letter plus the accent. For example, é (+U00E9) and e´ (+U0065 +U0301) are usually displayed in the same way. R renders the following (version 3.0.2, Mac OS 10.7.5): ``` > "\u00e9" [1] "é" > "\u0065\u0301" [1] "é" ``` However, of course: ``` > "\u00e9" == "\u0065\u0301" [1] FALSE ``` Is there a function in R which converts two-unicode-character-letters into their one-character form? In particular, here it would collapse `"\u0065\u0301"` into `"\u00e9"`. That would be extremely handy to process large quantities of strings. Plus, the one-character forms can easily be converted to other encodings via `iconv` -- at least for the usual Latin1 characters -- and is better handled by `plot`. Thanks a lot in advance.