Boost.MPI: What's received isn't what was sent!

boost, boost-mpi, c++, mpi

Solution

Ok, I think I have the answer, which requires some knowledge of the underlying C-style MPI calls. Boost's 'isend' function is essentially a wrapper around 'MPI_Isend', and it does not protect the user from needing to know some of the details about how 'MPI_Isend' works.

One parameter of 'MPI_Isend' is a pointer to a buffer that contains the information you wish to send. Importantly, however, this buffer CANNOT be reused until you know that the message has been received. So consider the following code:

// Get solution numbers from the solutions and store in a vector
vector<int> solutionNums(numSolutions);
for (int sol = 0; sol < numSolutions; ++sol)
{
  solutionNums[sol] = solutions[sol]->solutionNum();
}

// Send solution numbers
for (int sol = 0; sol < numSolutions; ++sol)
{
  world.isend(sol + 1, 0, false);  // Indicates that we have not finished, and to expect a solution representation
  cout << "Sending solution no. " << solutionNums[sol] << " to node " << sol + 1 << endl;
  world.isend(sol + 1, 1, solutionNums[sol]);
}

This works perfectly, as each solution number is in its own location in memory. Now consider the following minor adjustment:

// Create solutionNum array
vector<int> solutionNums(numSolutions);
for (int sol = 0; sol < numSolutions; ++sol)
{
  solutionNums[sol] = solutions[sol]->solutionNum();
}

// Send solutions
for (int sol = 0; sol < numSolutions; ++sol)
{
  int solNum = solutionNums[sol];
  world.isend(sol + 1, 0, false);  // Indicates that we have not finished, and to expect a solution representation
  cout << "Sending solution no. " << solNum << " to node " << sol + 1 << endl;
  world.isend(sol + 1, 1, solNum);
}

Now the underlying 'MPI_Isend' call is provided with a pointer to solNum. Unfortunately, this bit of memory is overwritten each time around the loop, so while it may look like the number 4 is sent to node 5, by the time the send actually takes place, the new contents of that memory location (19 for example) are passed instead.

Now consider the original code:

// Send solutions
for (int sol = 0; sol < numSolutions; ++sol)
{
  world.isend(sol + 1, 0, false);  // Tells the slave to expect work
  cout << "Sending solution no. " << solutions[sol]->solutionNum() << " to node " << sol + 1 << endl;
  world.isend(sol + 1, 1, solutions[sol]->solutionNum());
}

Here we are passing a temporary. Again, the location of this temporary in memory gets overwritten each time around the loop. Again, the wrong data is sent to the slave nodes.

As it happens, I've been able to restructure my 'real' code to use 'send' instead of 'isend'. However, if I need to use 'isend' in future, I'll be a little more careful!

Problem

I am relatively new to using Boost MPI. I have got the libraries installed, the code compiles, but I am getting a very odd error - some integer data received by the slave nodes is not what was sent by the master. What is going on? I am using boost version 1.42.0, compiling the code using mpic++ (which wraps g++ on one cluster and icpc on the other). A reduced example follows, including the output. Code: ``` #include <iostream> #include <boost/mpi.hpp> using namespace std; namespace mpi = boost::mpi; class Solution { public: Solution() : solution_num(num_solutions++) { // Master node's constructor } Solution(int solutionNum) : solution_num(solutionNum) { // Slave nodes' constructor. } int solutionNum() const { return solution_num; } private: static int num_solutions; int solution_num; }; int Solution::num_solutions = 0; int main(int argc, char* argv[]) { // Initialization of MPI mpi::environment env(argc, argv); mpi::communicator world; if (world.rank() == 0) { // Create solutions int numSolutions = world.size() - 1; // One solution per slave vector<Solution*> solutions(numSolutions); for (int sol = 0; sol < numSolutions; ++sol) { solutions[sol] = new Solution; } // Send solutions for (int sol = 0; sol < numSolutions; ++sol) { world.isend(sol + 1, 0, false); // Tells the slave to expect work cout << "Sending solution no. " << solutions[sol]->solutionNum() << " to node " << sol + 1 << endl; world.isend(sol + 1, 1, solutions[sol]->solutionNum()); } // Retrieve values (solution numbers squared) vector<double> values(numSolutions, 0); for (int i = 0; i < numSolutions; ++i) { // Get values for each solution double value = 0; mpi::status status = world.recv(mpi::any_source, 2, value); int source = status.source(); int sol = source - 1; values[sol] = value; } for (int i = 1; i <= numSolutions; ++i) { world.isend(i, 0, true); // Tells the slave to finish } // Output the solutions numbers and their squares for (int i = 0; i < numSolutions; ++i) { cout << solutions[i]->solutionNum() << ", " << values[i] << endl; delete solutions[i]; } } else { // Slave nodes merely square the solution number bool finished; mpi::status status = world.recv(0, 0, finished); while (!finished) { int solNum; world.recv(0, 1, solNum); cout << "Node " << world.rank() << " receiving solution no. " << solNum << endl; Solution solution(solNum); double value = static_cast<double>(solNum * solNum); world.send(0, 2, value); status = world.recv(0, 0, finished); } cout << "Node " << world.rank() << " finished." << endl; } return EXIT_SUCCESS; } ``` Running this on 21 nodes (1 master, 20 slaves) produces: ``` Sending solution no. 0 to node 1 Sending solution no. 1 to node 2 Sending solution no. 2 to node 3 Sending solution no. 3 to node 4 Sending solution no. 4 to node 5 Sending solution no. 5 to node 6 Sending solution no. 6 to node 7 Sending solution no. 7 to node 8 Sending solution no. 8 to node 9 Sending solution no. 9 to node 10 Sending solution no. 10 to node 11 Sending solution no. 11 to node 12 Sending solution no. 12 to node 13 Sending solution no. 13 to node 14 Sending solution no. 14 to node 15 Sending solution no. 15 to node 16 Sending solution no. 16 to node 17 Sending solution no. 17 to node 18 Sending solution no. 18 to node 19 Sending solution no. 19 to node 20 Node 1 receiving solution no. 0 Node 2 receiving solution no. 1 Node 12 receiving solution no. 19 Node 3 receiving solution no. 19 Node 15 receiving solution no. 19 Node 13 receiving solution no. 19 Node 4 receiving solution no. 19 Node 9 receiving solution no. 19 Node 10 receiving solution no. 19 Node 14 receiving solution no. 19 Node 6 receiving solution no. 19 Node 5 receiving solution no. 19 Node 11 receiving solution no. 19 Node 8 receiving solution no. 19 Node 16 receiving solution no. 19 Node 19 receiving solution no. 19 Node 20 receiving solution no. 19 Node 1 finished. Node 2 finished. Node 7 receiving solution no. 19 0, 0 1, 1 2, 361 3, 361 4, 361 5, 361 6, 361 7, 361 8, 361 9, 361 10, 361 11, 361 12, 361 13, 361 14, 361 15, 361 16, 361 17, 361 18, 361 19, 361 Node 6 finished. Node 3 finished. Node 17 receiving solution no. 19 Node 17 finished. Node 10 finished. Node 12 finished. Node 8 finished. Node 4 finished. Node 15 finished. Node 18 receiving solution no. 19 Node 18 finished. Node 11 finished. Node 13 finished. Node 20 finished. Node 16 finished. Node 9 finished. Node 19 finished. Node 7 finished. Node 5 finished. Node 14 finished. ``` So while the master sends 0 to node 1, 1 to node 2, 2 to node 3 etc, most of the slave nodes (for some reason) receive the number 19. So rather than producing the squares of the numbers from 0 to 19, we get 0 squared, 1 squared and 19 squared 18 times! Thanks in advance to anyone who can explain this. Alan

Original source