Write data structure not read correct with struct.pack and struct.unpack_from

python, python-2.7

Solution

You're running into padding problems. As the docs say:

Padding is only automatically added between successive structure members. No padding is added at the beginning or the end of the encoded struct.

See what's happening:

>>> struct.pack("B", 0)
'\x00'
>>> struct.pack("I", 1)
'\x01\x00\x00\x00'
>>> struct.pack("BI", 0, 1)
'\x00\x00\x00\x00\x01\x00\x00\x00'

So you can't pack items separately and then unpack them together, unless you add the padding yourself...

>>> struct.pack("B0I", 0)
'\x00\x00\x00\x00'

or turn off padding altogether:

>>> struct.pack("=BI", 0, 1)
'\x00\x01\x00\x00\x00'

Problem

I have this problem in get data in binary file ``` # Write data f = open(path, 'wb') start_date = [2014, 1, 1, 0, 0, 0, 0] end_date = [2014, 2, 1, 0, 0, 0, 0] for x in range(10): f.write(struct.pack('B', 0)) f.write(struct.pack('I', x)) f.write(struct.pack('HBBBBBH', *start_date_binary)) f.write(struct.pack('HBBBBBH', *end_date_binary)) f.close() # Read data f = open(path, 'rb') for x in range(10): data_structure = struct.unpack_from("BIHBBBBBHHBBBBBH", f.read(FILE_INDEX_STRUCTURE)) print(data_structure) f.close() ``` Output is ``` (0, 0, 2014, 1, 1, 0, 0, 0, 0, 56832, 7, 2, 1, 0, 0, 0) (0, 17292800, 1, 0, 0, 0, 0, 0, 2014, 258, 0, 0, 0, 0, 0, 0) (7, 257, 0, 0, 0, 222, 7, 2, 1, 0, 0, 0, 0, 0, 3, 0) (0, 0, 56832, 7, 2, 1, 0, 0, 0, 0, 0, 4, 0, 0, 0, 2014) (0, 131989504, 258, 0, 0, 0, 0, 0, 0, 5, 0, 0, 222, 7, 1, 1) (222, 66055, 0, 0, 0, 0, 0, 6, 0, 56832, 7, 1, 1, 0, 0, 0) (1, 0, 0, 0, 7, 0, 0, 0, 2014, 257, 0, 0, 0, 0, 0, 56832) (0, 0, 8, 0, 0, 222, 7, 1, 1, 0, 0, 0, 0, 222, 7, 258) (0, 2304, 56832, 7, 1, 1, 0, 0, 0, 0, 222, 7, 2, 1, 0, 0) ``` Expected output is that ``` (0, 0, 2014, 1, 1, 0, 0, 0, 0, 2014, 2, 1, 0, 0, 0, 0) (0, 1, 2014, 1, 1, 0, 0, 0, 0, 2014, 2, 1, 0, 0, 0, 0) (0, 2, 2014, 1, 1, 0, 0, 0, 0, 2014, 2, 1, 0, 0, 0, 0) (0, 3, 2014, 1, 1, 0, 0, 0, 0, 2014, 2, 1, 0, 0, 0, 0) (0, 4, 2014, 1, 1, 0, 0, 0, 0, 2014, 2, 1, 0, 0, 0, 0) (0, 5, 2014, 1, 1, 0, 0, 0, 0, 2014, 2, 1, 0, 0, 0, 0) (0, 6, 2014, 1, 1, 0, 0, 0, 0, 2014, 2, 1, 0, 0, 0, 0) (0, 7, 2014, 1, 1, 0, 0, 0, 0, 2014, 2, 1, 0, 0, 0, 0) (0, 8, 2014, 1, 1, 0, 0, 0, 0, 2014, 2, 1, 0, 0, 0, 0) (0, 9, 2014, 1, 1, 0, 0, 0, 0, 2014, 2, 1, 0, 0, 0, 0) ``` EDIT Getting type of structure of item where `'B'` is 1 and `'H'` is 2. Using those types in the same `unpack` function the types are confused and in example where `'BH'` is 3 but return 4. ``` >>> struct.unpack_from("B", '') ... struct.error: unpack_from requires a buffer of at least 1 bytes >>> struct.unpack_from("H", '') ... struct.error: unpack_from requires a buffer of at least 2 bytes >>> struct.unpack_from("BH", '') ... struct.error: unpack_from requires a buffer of at least 4 bytes ```

Original source