set env var in Python multiprocessing.Process

environment-variables, multiprocessing, python

Solution

Yes, that's the right way to do it. While the child will inherit its initial environment from the parent, subsequent changes to `os.environ` made in the child will not affect the parent, and vice-versa:

import os
import multiprocessing


def myfunc(q):
    print "child: " + os.environ['FOO']
    os.environ['FOO'] = "child_set"
    print "child new: " + os.environ['FOO']
    q.put(None)
    q.get()
    print "child new2: " + os.environ['FOO']


if __name__ == "__main__":
    os.environ['FOO'] = 'parent_set'
    q = multiprocessing.Queue()
    proc = multiprocessing.Process(target=myfunc, args=(q,))
    proc.start()
    q.get()
    print "parent: " + os.environ['FOO']
    os.environ['FOO'] = "parent_set_again"
    q.put(None)

Output:

child start: parent_set
child after changing: child_set
parent after child changing: parent_set
child after parent changing: child_set

If you need to pass an initial environment to the child, you would just pass it in the `args` or `kwargs` list:

def myfunc(env=None):
    time.sleep(3)
    if env is not None:
        os.environ = env
    print os.environ['FOO']


if __name__ == "__main__":
    child_env = os.environ.copy()

    for i in range(3):
        child_env['FOO'] = "foo%s" % (i,)
        proc = multiprocessing.Process(target=myfunc, kwargs ={'env' : child_env})
        proc.start()

Output:

foo0
foo1
foo2

Note that if you're using a `multiprocessing.Pool`, you can use the `initializer`/`initargs` keyword arguments to just set the correct environment once at the start of each process in the pool:

def init(env):
    os.environ = env

def myfunc():
    print os.environ['FOO']


if __name__ == "__main__":
    child_env = os.environ.copy()
    child_env['FOO'] = "foo"
    pool = multiprocessing.Pool(initializer=init, initargs=(child_env,))
    pool.apply(myfunc,())

Output:

foo

Problem

In the subprocess Python 2 module, Popen can be given an `env`. Seems that the equivalent way to do it with Process in multiprocessing module is to pass the `env` dictionnary in `args` or `kwargs`, and then use `os.environ['FOO'] = value` in the `target`. Is it the right way? Is it safe? I mean, no risk that the environment in the parent process or other child processes can be modified? Here is an example (that works). ``` import multiprocessing import time import os def target(someid): os.environ['FOO'] = "foo%i" % someid for i in range(10): print "Job %i: " % someid, os.environ['FOO'] time.sleep(1) if __name__ == '__main__': processes = [] os.environ['FOO'] = 'foo' for someid in range(3): p = multiprocessing.Process(target=target, args=(someid,)) p.start() processes.append(p) for i in range(10): print "Parent: ", os.environ['FOO'] time.sleep(1) for p in processes: p.join() ```

Original source