How to declare a byte array contains non-ascii characters without escape in python 3

python, unicode

Solution

Finally I found a solution to do this, here is the example:

def to_bytes(string):
    result = b""
    need_eval = ""
    need_encode = ""
    for char in string:
        if char <= "\x7f":
            # if char is ascii, eval it with in b'...'
            if need_encode:
                result += need_encode.encode()
                need_encode = ""
            if char == "'":
                n = 0
                # get how many \ in the end of need_eval 
                for v in reversed(need_eval):
                    if v != "\\":
                        break
                    n += 1
                # if ' is not escaping
                if n % 2 == 0:
                    char = "\\'"
            need_eval += char
        else:
            # if char is non-ascii, encode it to utf-8
            if need_eval:
                result += ast.literal_eval("b'" + need_eval + "'")
                need_eval = ""
            need_encode += char
    result += need_encode.encode()
    result += ast.literal_eval("b'" + need_eval + "'")
    return result

b = to_bytes(r"\x00\x00\x00\x02α\x00\x00\x00\x13привет мир\x00\x00\x00\x02±")

print(repr(b))
print(to_bytes(r"±\xb1"))
print(to_bytes(r"\x90asdfg\\'\''\r\n\xff\u0001"))

Point is take the non-ascii part to encode and take other part to eval (because there no string_escape encoding on python 3, I had to use ast.literal_eval).

The pain is it's not efficient.

Problem

Here is an example I wrote in python2 ``` #!/usr/bin/env python # -*- coding: utf-8 -*- from __future__ import print_function import sys, struct def pack(*s_list): return b"".join(struct.pack(">I", len(s)) + s for s in s_list) if __name__ == "__main__": print(sys.version) a = pack("α", "привет мир", "±") b = b"\x00\x00\x00\x02α\x00\x00\x00\x13привет мир\x00\x00\x00\x02±" print(a == b) ``` And the converted code for python 3, ``` #!/usr/bin/env python3 # -*- coding: utf-8 -*- import sys, struct def pack(*s_list): return b"".join(struct.pack(">I", len(b)) + b for b in (s.encode() for s in s_list)) if __name__ == "__main__": print(sys.version) a = pack("α", "привет мир", "±") b = b"\x00\x00\x00\x02α" + "α".encode() + b"\x00\x00\x00\x13" + "привет мир".encode() + b"\x00\x00\x00\x02" + "±".encode() print(a == b) ``` I notice use `b = b"\x00\x00\x00\x02α\x00\x00\x00\x13привет мир\x00\x00\x00\x02±"` in python 3 will get an exception `SyntaxError: bytes can only contain ASCII literal characters.`, so I had to escape them all or use many + like previous code. Is there a better way to declare a byte array contains non-ascii character in python3?

Original source