Python: testing for utf-8 character in string

python, right-to-left, unicode

Solution

Every unicode character has a "bidirectional" class. You can find the bidirectional class using unicodedata.bidirectional. The function returns a string, e.g. 'L', 'R', 'AL', etc. with the following meaning:

| L   | Left_To_Right           | any strong left-to-right character                                |
| LRE | Left_To_Right_Embedding | U+202A: the LR embedding control                                  |
| LRO | Left_To_Right_Override  | U+202D: the LR override control                                   |
| R   | Right_To_Left           | any strong right-to-left (non-Arabic-type) character              |
| AL  | Arabic_Letter           | any strong right-to-left (Arabic-type) character                  |
| RLE | Right_To_Left_Embedding | U+202B: the RL embedding control                                  |
| RLO | Right_To_Left_Override  | U+202E: the RL override control                                   |
| PDF | Pop_Directional_Format  | U+202C: terminates an embedding or override control               |
| EN  | European_Number         | any ASCII digit or Eastern Arabic-Indic digit                     |
| ES  | European_Separator      | plus and minus signs                                              |
| ET  | European_Terminator     | a terminator in a numeric format context, includes currency signs |
| AN  | Arabic_Number           | any Arabic-Indic digit                                            |
| CS  | Common_Separator        | commas, colons, and slashes                                       |
| NSM | Nonspacing_Mark         | any nonspacing mark                                               |
| BN  | Boundary_Neutral        | most format characters, control codes, or noncharacters           |
| B   | Paragraph_Separator     | various newline characters                                        |
| S   | Segment_Separator       | various segment-related control codes                             |
| WS  | White_Space             | spaces                                                            |
| ON  | Other_Neutral           | most other symbols and punctuation marks                          |

For instance:

In [3]: import unicodedata as UD
In [5]: UD.bidirectional(u'\u0688')
Out[5]: 'AL'

In [6]: UD.bidirectional(u'\u200f')
Out[6]: 'R'

In [7]: UD.bidirectional(u'H')
Out[7]: 'L'

So you might be able to guess if a string is right-to-left by determining if it is composed mainly of characters whose bidirectional class is `R` or `AL`.

For example,

# coding: utf-8
import unicodedata as UD

texts = ['ڈوگرى'.decode('utf-8'),
         u'Hello']
for text in texts:
    x = len([None for ch in text if UD.bidirectional(ch) in ('R', 'AL')])/float(len(text))
    print('{t} => {c}'.format(t=text.encode('utf-8'), c='RTL' if x>0.5 else 'LTR'))

yields

ڈوگرى => RTL
Hello => LTR

Regarding the first question:

Q: What is the correct syntax to test for the occurrence of a single non-ASCII character in a string? Python 2.6 and I can't use 3.

Your method for testing if a character is in a `unicode` is correct. If `u'\u200f' in str.decode('utf-8')` neither complains nor works, then `u'\u200f'` is not in the `unicode`.

Problem

I need to test whether a string that has already been encoded with str.encode('utf-8') is right-to-left. I tried ``` if u'\u200f' in str.decode('utf-8'): print 'found it' ``` It neither complains nor works. Q: What is the correct syntax to test for the occurrence of a single non-ASCII character in a string? Python 2.6 and I can't use 3. Q: I remember reading that predominantly right-to-left characters default to RTL even without an explicit RML. Does anyone know a way to test such a string without knowing which language to expect (i.e. the string can be in Arabic, Hebrew or any other RTL language)? Thanks for all help.

Original source