How to use multiprocess in python on a class object

Question

I am fairly new to Python, and my experience is specific to its use in Powerflow modelling through the API provided in Siemens PSS/e. I have a script that I have been using for several years that runs some simulation on a large data set. In order to get to finish quickly, I usually split the inputs up into multiple parts, then run multiple instances of the script in IDLE. Ive recently added a GUI for the inputs, and have refined the code to be more object oriented, creating a class that the GUI passes the inputs to but then works as the original script did. My question is how do I go about splitting the process from within the program itself rather than making multiple copies? I have read a bit about the mutliprocess module but I am not sure how to apply it to my situation. Essentially I want to be able to instantiate N number of the same object, each running in parallel. The class I have now (called Bot) is passed a set of arguments and creates a csv output while it runs until it fin…

Accepted Answer

Create a list of configurations:

```
configurations = [...]
```

Create a function which takes the relevant configuration, and makes use of your `Bot`:

```
def function(configuration):
    bot = Bot(configuration)
    bot.create_csv()
```

Create a `Pool` of workers with however many CPUs you want to use:

```
from multiprocessing import Pool
pool = Pool(3)
```

Call the function multiple times which each configuration in your list of configurations.

```
pool.map(function, configurations)
```

For example:

```
from multiprocessing import Pool
import os

class Bot:
    def __init__(self, inputs):
        self.inputs = inputs

def create_csv(self):
        pid = os.getpid()
        print('TODO: create csv in process {} using {}'
              .format(pid, self.inputs))

def use_bot(inputs):
     bot = Bot(inputs)
     bot.create_csv()

def main():
    configurations = [
        ['input1_1.txt', 'input1_2.txt'],
        ['input2_1.txt', 'input2_2.txt'],
        ['input3_1.txt', 'input3_2.txt']]

pool = Pool(2)
    pool.map(use_bot, configurations)

if __name__ == '__main__':
    main()
```

Output:

```
TODO: create csv in process 10964 using ['input2_1.txt', 'input2_2.txt']
TODO: create csv in process 8616 using ['input1_1.txt', 'input1_2.txt']
TODO: create csv in process 8616 using ['input3_1.txt', 'input3_2.txt']
```

How to use multiprocess in python on a class object

Solution

Problem