Interact with a MPI binary via a (Non-MPI) python script

interaction, mpi, python

Solution

Perhaps the most elegant solution would be to make the Python code part of your MPI application. Then it would be able to directly send data (via MPI messages) to the rest of the MPI application as it will be part of it. There are two different approaches here:

1) Insert the Python binary as rank 0 in your MPI job. In order to exclude it from participation in collective operations in `mpibinary`, you would have to make a subcommunicator that excludes rank 0 and use it for all further collective communication in `mpibinary`. The first step is the easy part. In Open MPI you would do:

mpirun --hostfile hosts -np 1 pythonbinary args : -np 32 mpibinary args

This is called MPMD (multiple programs multiple data) launch and it will start one copy of `pythonbinary` that will become rank 0 and also 32 copies of `mpibinary` that will become rank 1, rank 2, ... up to rank 32 (33 processes in total). Other MPI implementations also provide very similar mechanisms for MPMD launch. Then you'd use `MPI_Comm_split()` in order to create a new communicator that does not include the Python program. Splitting a communicator is a collective operation. That's why you have to call it both in your Python code and your C++ application. `MPI_Comm_split()` takes a "color" and a key and splits the communicator in multiple subcommunicators according to the different colors. Processes with the same color are then sorted based on the key value. You will most likely want to call it like this:

in Python:

python_comm = mpi.mpi_comm_split(mpi.MPI_COMM_WORLD, 0, 0)

in C++:

int rank;
MPI_Comm c_comm;

MPI_Comm_rank(MPI_COMM_WORLD, &rank);
MPI_Comm_split(MPI_COMM_WORLD, 1, rank, &c_comm);

By using the `rank` as key one guarantees that the order of the processes in `c_comm` will be the same as it was before the split, i.e. rank 1 from `MPI_COMM_WORLD` will become rank 0 in `c_comm`, rank 2 will become rank 1, etc.

From now on the C++ application can use `c_comm` to perform collective operations as usual. In order to communicate between the Python and the C++ code, you still have to use `MPI_COMM_WORLD` and the Python code would still be rank 0 in it.

2) Use the MPI-2 process management facilities. First you would run an MPI job that consists of the Python binary only:

mpirun --hostfile hosts -np 1 pythonbinary args

Then the Python binary would spawn the other MPI binary directly using `MPI_Comm_spawn()` with the desired number of new processes. The newly spawned processes will have their own `MPI_COMM_WORLD` and you would not need to use `MPI_Comm_split()`. Also the spawn operation will establish an intercommunicator that will allow the Python code to send messages to the other part of the MPI application.

In both cases the `hosts` file would contain definition of all execution hosts that can execute the MPI binaries. You would also need to use one of the available Python MPI bindings.

Note that you only need to add some MPI calls to your Python script like `MPI_Init`, `MPI_Finalize`, `MPI_Comm_split` and the relevant `MPI_Send`/`MPI_Recv`. You do not need to make it parallel. MPI is quite versatile in that allows you to not only use it for parallel worksharing but also as a general messaging framework. But please note that the Python bindings should use the same MPI library as the rest of the program.

Another solution would be to use some message queuing library or file pooling (which is really a crude MQ implementation).

Problem

Original source