Emulating Bash 'source' in Python

Question

I have a script that looks something like this: ``` export foo=/tmp/foo export bar=/tmp/bar ``` Every time I build I run 'source init_env' (where init_env is the above script) to set up some variables. To accomplish the same in Python I had this code running, ``` reg = re.compile('export (?P\w+)(\=(?P.+))*') for line in open(file): m = reg.match(line) if m: name = m.group('name') value = '' if m.group('value'): value = m.group('value') os.putenv(name, value) ``` But then someone decided it would be nice to add a line like the following to the `init_env` file: ``` export PATH="/foo/bar:/bar/foo:$PATH" ``` Obviously my Python script fell apart. I could modify the Python script to handle this line, but then it'll just break later on when someone comes up with a new feature to use in the `init_env` file. The question is if there is an easy way to run a Bash command and let it modify my `os.environ`?

Accepted Answer

The problem with your approach is that you are trying to interpret bash scripts. First you just try to interpret the export statement. But when people think they can use bash syntax they will. They will use variable expansion, conditionals, process substitutions. In the end you will have a full blown bash script interpreter with a gazillion bugs. Don't do that.

Let Bash interpret the file for you and then collect the results.

Here is a minimal example how to do so:

```
#! /usr/bin/env python

import os
import pprint
import shlex
import subprocess

command = shlex.split("bash -c 'source init_env && env'")
proc = subprocess.Popen(command, stdout = subprocess.PIPE)
for line in proc.stdout:
  (key, _, value) = line.partition("=")
  os.environ[key] = value
proc.communicate()

pprint.pprint(dict(os.environ))
```

Make sure that you handle errors. see here for how: "subprocess.Popen" - checking for success and errors

Also read the documentation on subprocess.

this will only capture variables set with the `export` statement, as `env` only prints exported variables. you can add `set -a` to treat all variables as exported.

```
command = shlex.split("bash -c 'set -a && source init_env && env'")
                                ^^^^^^
```

note that this code will not handle multi line variables. it will also not handle bash function definitions.

perhaps better than calling bash source from inside python is to first let bash source the file and then run the python script

```
#!/bin/bash
source init_env
/path/to/python_script.py
```

here bash will `source init_env` with all the power and glory and quirks of bash. the python script will inherit the updated environment.

note that again only exported variables will be inherited. you can force all variable assignments to be exported with `set -a`

```
#!/bin/bash
set -a
source init_env
/path/to/python_script.py
```

another approach would be to tell the users that they can strictly only do `key=value` without any bash power. then use python configparser.

this will have the advantage of simple `init_env` syntax and a rigorously tested config parser. but the disadvantage that the `init_env` will no longer be as expressive as bash config files can be.

Emulating Bash 'source' in Python

Solution

Problem

Related problems