Python recursive setattr()-like function for working with nested dictionaries

algorithm, nested, python, recursion, setattr

Solution

I have separated this out into two steps. In the first step, the query string is broken down into a series of instructions. This way the problem is decoupled, we can view the instructions before running them, and there is no need for recursive calls.

def build_instructions(obj, q):
    """
    Breaks down a query string into a series of actionable instructions.

    Each instruction is a (_type, arg) tuple.
    arg -- The key used for the __getitem__ or __setitem__ call on
           the current object.
    _type -- Used to determine the data type for the value of
             obj.__getitem__(arg)

    If a key/index is missing, _type is used to initialize an empty value.
    In this way _type provides the ability to
    """
    arg = []
    _type = None
    instructions = []
    for i, ch in enumerate(q):
        if ch == "[":
            # Begin list query
            if _type is not None:
                arg = "".join(arg)
                if _type == list and arg.isalpha():
                    _type = dict
                instructions.append((_type, arg))
                _type, arg = None, []
            _type = list
        elif ch == ".":
            # Begin dict query
            if _type is not None:
                arg = "".join(arg)
                if _type == list and arg.isalpha():
                    _type = dict
                instructions.append((_type, arg))
                _type, arg = None, []

            _type = dict
        elif ch.isalnum():
            if i == 0:
                # Query begins with alphanum, assume dict access
                _type = type(obj)

            # Fill out args
            arg.append(ch)
        else:
            TypeError("Unrecognized character: {}".format(ch))

    if _type is not None:
        # Finish up last query
        instructions.append((_type, "".join(arg)))

    return instructions

For your example

>>> x = {"a": "stuff"}
>>> print(build_instructions(x, "f[0].a"))
[(<type 'dict'>, 'f'), (<type 'list'>, '0'), (<type 'dict'>, 'a')]

The expected return value is simply the `_type` (first item) of the next tuple in the instructions. This is very important because it allows us to correctly initialize/reconstruct missing keys.

This means that our first instruction operates on a `dict`, either sets or gets the key `'f'`, and is expected to return a `list`. Similarly, our second instruction operates on a `list`, either sets or gets the index `0` and is expected to return a `dict`.

Now let's create our `_setattr` function. This gets the proper instructions and goes through them, creating key-value pairs as necessary. Finally, it also sets the `val` we give it.

def _setattr(obj, query, val):
    """
    This is a special setattr function that will take in a string query,
    interpret it, add the appropriate data structure to obj, and set val.

    We only define two actions that are available in our query string:
    .x -- dict.__setitem__(x, ...)
    [x] -- list.__setitem__(x, ...) OR dict.__setitem__(x, ...)
           the calling context determines how this is interpreted.
    """
    instructions = build_instructions(obj, query)
    for i, (_, arg) in enumerate(instructions[:-1]):
        _type = instructions[i + 1][0]
        obj = _set(obj, _type, arg)

    _type, arg = instructions[-1]
    _set(obj, _type, arg, val)

def _set(obj, _type, arg, val=None):
    """
    Helper function for calling obj.__setitem__(arg, val or _type()).
    """
    if val is not None:
        # Time to set our value
        _type = type(val)

    if isinstance(obj, dict):
        if arg not in obj:
            # If key isn't in obj, initialize it with _type()
            # or set it with val
            obj[arg] = (_type() if val is None else val)
        obj = obj[arg]
    elif isinstance(obj, list):
        n = len(obj)
        arg = int(arg)
        if n > arg:
            obj[arg] = (_type() if val is None else val)
        else:
            # Need to amplify our list, initialize empty values with _type()
            obj.extend([_type() for x in range(arg - n + 1)])
        obj = obj[arg]
    return obj

And just because we can, here's a `_getattr` function.

def _getattr(obj, query):
    """
    Very similar to _setattr. Instead of setting attributes they will be
    returned. As expected, an error will be raised if a __getitem__ call
    fails.
    """
    instructions = build_instructions(obj, query)
    for i, (_, arg) in enumerate(instructions[:-1]):
        _type = instructions[i + 1][0]
        obj = _get(obj, _type, arg)

    _type, arg = instructions[-1]
    return _get(obj, _type, arg)


def _get(obj, _type, arg):
    """
    Helper function for calling obj.__getitem__(arg).
    """
    if isinstance(obj, dict):
        obj = obj[arg]
    elif isinstance(obj, list):
        arg = int(arg)
        obj = obj[arg]
    return obj

In action:

>>> x = {"a": "stuff"}
>>> _setattr(x, "f[0].a", "test")
>>> print x
{'a': 'stuff', 'f': [{'a': 'test'}]}
>>> print _getattr(x, "f[0].a")
"test"

>>> x = ["one", "two"]
>>> _setattr(x, "3[0].a", "test")
>>> print x
['one', 'two', [], [{'a': 'test'}]]
>>> print _getattr(x, "3[0].a")
"test"

Now for some cool stuff. Unlike python, our `_setattr` function can set unhashable `dict` keys.

x = []
_setattr(x, "1.4", "asdf")
print x
[{}, {'4': 'asdf'}]  # A list, which isn't hashable

>>> y = {"a": "stuff"}
>>> _setattr(y, "f[1.4]", "test")  # We're indexing f with 1.4, which is a list!
>>> print y
{'a': 'stuff', 'f': [{}, {'4': 'test'}]}
>>> print _getattr(y, "f[1.4]")  # Works for _getattr too
"test"

We aren't really using unhashable `dict` keys, but it looks like we are in our query language so who cares, right!

Finally, you can run multiple `_setattr` calls on the same object, just give it a try yourself.

Problem

Original source

Related problems

• Finding a key recursively in a dictionary
• Is it possible to index nested lists using tuples in python?
• Is it possible to index nested lists using tuples in python?