Substrings in Python. Copies in memory?

immutability, python, regex, string

Solution

From looking at the Python 2.7.3 source code, taking a slice of a string makes a copy of the character data:

`Objects/stringobject.c`:

`string_slice()` calls the following function, `PyString_FromStringAndSize()`:

/* Inline PyObject_NewVar */
op = (PyStringObject *)PyObject_MALLOC(PyStringObject_SIZE + size);
if (op == NULL)
    return PyErr_NoMemory();
PyObject_INIT_VAR(op, &PyString_Type, size);
op->ob_shash = -1;
op->ob_sstate = SSTATE_NOT_INTERNED;
if (str != NULL)
    Py_MEMCPY(op->ob_sval, str, size);
op->ob_sval[size] = '\0';

Here, `str` is a pointer to the character data, and `size` is the length. Note the malloc and the memcpy.

Different Python implementations (and indeed different versions of CPython) might behave differently. For example, Jython probably uses `java.lang.String`, which doesn't make a copy.

Problem

Say I have a string `my_string` in Python and that I tokenize it according to `some_pattern`: ``` match.re.search(some_pattern, my_string) string_1 = match.group(1) string_2 = match.group(2) .... ``` Are `string_1` and `string_2` ("deep") copies of the substrings in `my_string` or references to the same location in memory? Do `string_1` and `string_2` allocate memory for full copies of the characters in `my_string`? Please note that I am not asking about the immutability of the strings. If `my_string` is very long, I would like to know what is the hit in memory that I take by tokenizing my strings. I don't need to know exactly how much memory is re-used, but it would certainly be useful to know if a tokenization of a string ends up duplicating memory.

Original source