Creating an MPI_Datatype for a structure containing pointers

c, mpi

Solution

MPI is designed to work with arrays of structures rather that with structures of arrays.

The `MPI_Hindexed` that @suszterpatt proposed is a terrible hack. It will only allow you to send one element of the structure type and only the element that was used to define the MPI data type. For other variables of the same structure type it is mostly guaranteed that the computed offsets will be wrong. Besides Hindexed types use one and the same MPI data type for all elements and thus does not allow you to send both ints and doubles.

The wise thing to do is to transform your program to use arrays of structures:

 typedef struct
 {
    int i;
    double z;
 } point;

 typedef struct
 {
    point *A;
    int nz;
 } column;

Now you can create an MPI structured type `point_type` and use it to send `nz` elements of that type giving `column.A` as the buffer address:

 int lens[3];
 MPI_Aint base, disps[2];
 MPI_Datatype oldtypes[2], point_struct, point_type;

 MPI_Get_address(&point, disps);
 MPI_Get_address(&point.z, disps+1);
 base = disps[0];

 lens[0] = 1; disps[0] = MPI_Aint_diff(disps[0], base); oldtypes[0] = MPI_INT;
 lens[1] = 1; disps[1] = MPI_Aint_diff(disps[1], base); oldtypes[1] = MPI_DOUBLE;
 MPI_Type_create_struct(2, lens, disps, oldtypes, &point_struct);
 MPI_Type_create_resized(point_struct, 0, sizeof(point), &point_type);
 MPI_Type_commit(&point_type);

 MPI_Send(column.A, column.nz, point_type, ...);

This first creates an MPI datatype `point_struct` that describes the layout of the structure members, but does not account for any padding at the end and therefore cannot be used to reliably send an array of such structures. Therefore, a second datatype `point_type` with the correct extent is created using `MPI_Type_create_resized`.

On the receiver side you would peek the message with `MPI_Probe`, extract the number of elements with `MPI_Get_count` with a type of `point_type` (that goes straight to the `nz` field), allocate the `A` field and use it in `MPI_Recv` to receive the `nz` elements:

 MPI_Status status;
 MPI_Probe(source, tag, comm, &status);
 MPI_Get_count(&status, point_type, &column.nz);
 if (nz == MPI_UNDEFINED)
   ... non-integral message was received, do something
 column.A = (point *)malloc(column.nz*sizeof(point));
 MPI_Recv(column.A, column.nz, point_type, source, tag, comm, MPI_STATUS_IGNORE);

If that code change is impossible you can still go through the intermediate step of transforming your structure before sending it, a process usually called (un-)marshaling. In your case do something like this (I assume that you store the number of array elements in both `Ai` and `Ax` in the `nz` field):

 point *temp = (point *)malloc(nz*sizeof(point));
 for (int i = 0; i < column.nz; i++)
 {
   temp[i].i = column.Ai[i];
   temp[i].z = column.Az[i];
 }
 MPI_Send(temp, nz, point_type, ...);
 free(temp);

On the receiver side you must do the opposite: allocate a large enough buffer that can hold the structure, receive the message in it and then do the opposite transformation.

Once again, you do not need to transmit the actual value of `nz` since it can be easily extracted from the length of the message using `MPI_Get_count`.

Problem

I have the following structure. ``` typedef struct { int *Ai; double *Ax; int nz; }column; ``` I want to transfer this structure using `MPI_Send` and `MPI_Receive`. How do I create an `MPI_Datatype` for this structure?

Original source