working in different directories (os.chdir) in the same time (parallel threading)

Question

I want to sync all of my vcs directories in parallel. I'm going to directory and run special command line scripts to sync git or mercurial repositories. It's slow process so I want to try to make it parallel. But there is trouble my parallel threads fight for "current directory" so I need some trick to work in different directories in the same time. Current solution: ``` def syncrepos(repos): for r in repos.split("
"): if r: print("------ repository: ", r) thrd = ThreadingSync(r) thrd.setDaemon(True) thrd.start() ``` where ThreadingSync is ``` class ThreadingSync(threading.Thread): def __init__(self, repo): threading.Thread.__init__(self) self.repo = repo def run(self): r = self.repo.split("-t") path = (r[0]).strip() if len(r) < 2: vcs = VCS.git else: vcs = { 'git' : VCS.git, 'git git' : VCS.git_git, 'git hg' : VCS.git_mercurial, 'git svn' : VCS.git_subversion, 'git vv' : VCS.git_veracity, 'hg hg' : VCS.hg_hg}[(r[1]).strip()] os.chdir(path) if vcs == VCS.git: checkGitModifications()…

Accepted Answer

Create a pool of workers to run your subroutine:

http://docs.python.org/2/library/multiprocessing.html#using-a-pool-of-workers

In your case perhaps something like:

```
from multiprocessing import Pool
import os

def gitSync(repo):
    print "I am", repo, "and my cwd is:", os.getcwd()
    os.chdir(repo)
    print "I am", repo, "and my cwd is:", os.getcwd()

if __name__ == '__main__':
    dir = os.getcwd()
    repos = [item for item in os.listdir(dir) if os.path.isdir(os.path.join(dir, item))]
    print repos
    pool = Pool(maxtasksperchild=1)
    pool.map(gitSync, repos)
    pool.close()
    pool.join()
```

Note that the pool can make debugging a bit difficult as the parent usually doesn't reveal much more than -one of my children died-, so get it working single threaded first.

Edit: Well that was interesting to appreciate - note the new argument to the Pool `maxtasksperchild=1`. The process is not `rebooted` between invocations so when you change the directory in one invocation, you're still in that directory when the process gets reused. Here I've solved it simply by telling the pool to kill processes after every single invocation.

```
john:captcrunch john$ python foo.py 
['.git', '.idea', 'fixtures', 'lib', 'obj', 'raw', 'tests']
I am .git and my cwd is: /Users/john/code/linz/src/captcrunch
I am .git and my cwd is: /Users/john/code/linz/src/captcrunch/.git
I am .idea and my cwd is: /Users/john/code/linz/src/captcrunch
I am .idea and my cwd is: /Users/john/code/linz/src/captcrunch/.idea
I am fixtures and my cwd is: /Users/john/code/linz/src/captcrunch
I am fixtures and my cwd is: /Users/john/code/linz/src/captcrunch/fixtures
I am lib and my cwd is: /Users/john/code/linz/src/captcrunch
I am lib and my cwd is: /Users/john/code/linz/src/captcrunch/lib
I am obj and my cwd is: /Users/john/code/linz/src/captcrunch
I am obj and my cwd is: /Users/john/code/linz/src/captcrunch/obj
I am raw and my cwd is: /Users/john/code/linz/src/captcrunch
I am raw and my cwd is: /Users/john/code/linz/src/captcrunch/raw
I am tests and my cwd is: /Users/john/code/linz/src/captcrunch
I am tests and my cwd is: /Users/john/code/linz/src/captcrunch/tests
```

working in different directories (os.chdir) in the same time (parallel threading)

Solution

Problem