Confused about backslashes in regular expressions
python, regex
Solution
The confusion is due to the fact that the backslash character `\` is used as an escape at two different levels. First, the Python interpreter itself performs substitutions for `\` before the `re` module ever sees your string. For instance, `\n` is converted to a newline character, `\t` is converted to a tab character, etc. To get an actual `\` character, you can escape it as well, so `\\` gives a single `\` character. If the character following the `\` isn't a recognized escape character, then the `\` is treated like any other character and passed through, but I don't recommend depending on this. Instead, always escape your `\` characters by doubling them, i.e. `\\`.
If you want to see how Python is expanding your string escapes, just print out the string. For example:
s = 'a\\b\tc'
print(s)
If `s` is part of an aggregate data type, e.g. a list or a tuple, and if you print that aggregate, Python will enclose the string in single quotes and will include the `\` escapes (in a canonical form), so be aware of how your string is being printed. If you just type a quoted string into the interpreter, it will also display it enclosed in quotes with `\` escapes.
Once you know how your string is being encoded, you can then think about what the `re` module will do with it. For instance, if you want to escape `\` in a string you pass to the `re` module, you will need to pass `\\` to `re`, which means you will need to use `\\\\` in your quoted Python string. The Python string will end up with `\\` and the `re` module will treat this as a single literal `\` character.
An alternative way to include `\` characters in Python strings is to use raw strings, e.g. `r'a\b'` is equivalent to `"a\\b"`.
Problem
I am confused with the backslash in regular expressions. Within a regex a `\` has a special meaning, e.g. `\d` means a decimal digit. If you add a backslash in front of the backslash this special meaning gets lost. In the regex-howto one can read: Perhaps the most important metacharacter is the backslash, `\`. As in Python string literals, the backslash can be followed by various characters to signal various special sequences. It’s also used to escape all the metacharacters so you can still match them in patterns; for example, if you need to match a `[` or `\`, you can precede them with a backslash to remove their special meaning: `\[` or `\\`. So `print(re.search('\d', '\d'))` gives `None` because `\d` matches any decimal digit but there is none in `\d`. I now would expect `print(re.search('\\d', '\d'))` to match `\d` but the answer is still `None`. Only `print(re.search('\\\d', '\d'))` gives as output `<_sre.SRE_Match object; span=(0, 2), match='\\d'>`. Does someone have an explanation?