Find dependencies in a python source/script

abstract-syntax-tree, dependencies, python

Solution

You can tokenize Python source code using the tokenize module from the standard library. This will allow you to find all variable names used in the script.

Now suppose we define a "non-dependency" as any variable name that comes immediately before an `=` sign. Then, depending on how simple your script code really is (see the Caveats below), you may be able to determine the variable names which are not non-dependencies this way:

import tokenize
import io
import token
import collections
import keyword

kwset = set(keyword.kwlist)
class Token(collections.namedtuple('Token', 'num val start end line')):
    @property
    def name(self):
        return token.tok_name[self.num]

source = '''
C = A+B
D = C * 4
'''

lastname = None
names = set()
not_dep = set()
for tok in tokenize.generate_tokens(io.BytesIO(source).readline):
    tok = Token(*tok)
    print(tok.name, tok.val)
    if tok.name == 'NAME':
        names.add(tok.val)
        lastname = tok.val
    if tok.name == 'OP' and tok.val == '=':
        not_dep.add(lastname)

print(names)
# set(['A', 'C', 'B', 'D'])
print(not_dep)
# set(['C', 'D'])

deps = dict.fromkeys(names - not_dep - kwset, 1)
print(deps)
# {'A': 1, 'B': 1}

Caveats:

If your scripts contain statements other than simple assignments, then `names` may become populated with undesired variable names. For example,

import numpy

would add both `'import'` and `'numpy'` to the set `names`.

If your script contains an assignment that makes use of left-hand side tuple unpacking, such as

E, F = 1, 2

then the naive code above will only recognize that `F` is not a dependency.

Problem

I have a bunch of simple scripts in Python with simple expressions[1] like : ``` C = A+B D = C * 4 ``` I need to execute them, but most importantly I need to know what are the objects I depend on; in the previous case, the object A and B are outer dependencies. Eg. given i have the former code in a var called source, i wanna be able to: ``` deps = { "A" : 1 , "B": 2} exec source in deps ``` so it's strictly necessary to know how to build the dict deps. I've lurked into the ast Python module but I had no clue. [1] simple math aggregations, to an extent `for` cycles, nothing more.

Original source