Different MPI_Datatypes for Sender and Receiver
With MPI, is it possible to send and receive data with different types of MPI_Datatypes that are derived from the same base type and have the same total length?
Consider two processes MPI A and B . A has an array double a[n]
and B has an array double b[m]
. Both processes know that A wants to send double characters to B k
, which are somehow distributed in a
(only A has knowledge of this distribution). B (and only B ) knows how he wants to double k
in b
. So both are created (via MPI_Type_indexed
and MPI_Type_commit
) a datatype which for A corresponds to the elements it wants to send from a
, and for Bmatches where he wants to store these items in b
. A calls MPI_Isend(a, 1, A_datatype, ...)
and B calls MPI_Irecv(b, 1, B_datatype, ...)
.
It works? If so, is this the standard way to solve the problem? If not, how does this exchange usually take place?
source to share
MPI types in the mapping of send and receive operations must be congruent. Congruence is a weaker form of equivalence. This means that the two data types may not be equal, but their type signatures must match.
Each MPI data type is a list of tuples in the form (offset[i], basic_type[i])
, where offset[i]
is the offset of the i
th element in bytes from the beginning of the data buffer (it can also be negative) and basic_type[i]
is the data type of the element's language. The list of these tuples is called a type mapping. The list of basic types is itself called a type signature.
Imagine you are declaring an indexed data type with 5 blocks of one element each:
// Data type 1
MPI_Datatype type1;
int b_lens[5] = { 1, 1, 1, 1, 1 };
int b_offs[5] = { 1, 2, 3, 5, 7 };
MPI_Type_indexed(5, b_lens, b_offs, MPI_DOUBLE, &type1);
type1
has the following type mapping: {(8, double), (16, double), (24, double), (40, double), (56, double)} and a signature of type {double, double, double, double, double} ...
Now another continuous type is created with the same number of basic elements:
// Data type 2
MPI_Datatype type2;
MPI_Type_contiguous(5, MPI_DOUBLE, &type2);
type2
has the following type mapping: {(0, double), (8, double), (16, double), (24, double), (32, double)}. This type map is obviously not the same as that for type1
, because the offsets are different. But the type signature is the same: {double, double, double, double, double}. Therefore, type1
they type2
are called congruent.
The following code snippet is a perfectly valid MPI program:
double array1[10];
double array2[5];
double array3[10];
if (rank == rank_sender)
{
MPI_Send(array1, 1, type1, rank_receiver, 0, MPI_COMM_WORLD);
MPI_Send(array1, 1, type1, rank_receiver, 0, MPI_COMM_WORLD);
}
else if (rank == rank_receiver)
{
MPI_Status status;
int num_elems;
MPI_Recv(array2, 1, type2, rank_sender, 0, MPI_COMM_WORLD, MPI_STATUS_IGNORE);
MPI_Recv(array3, 10, MPI_DOUBLE, rank_sender, 0, MPI_COMM_WORLD, &status);
MPI_Get_count(&status, MPI_DOUBLE, &num_elems); // num_elems will be set to 5
}
The call MPI_Send
will fetch the 1st, 2nd, 3rd, 5th, and 7th elements array1
from the sender, and the call MPI_Recv
will put them into a contiguous array of 5 elements array2
in the receiver. Also, the 10-element receive with the original data type is MPI_DOUBLE
consistent with the data type of the sent message, although only the first 5 elements of the buffer will be filled. A call MPI_Get_count
with a datatype MPI_DOUBLE
will then return 5
.
Basically, matching both types means that the same number of data items of the same base type (s) and in the same order are packed into and then unpacked from the message.
The MPI standard requires this consistency in the right applications. But in reality, many MPI implementations do not convey the type signature in the message, and therefore congruence is never applied. For example, both Open MPI and MPICH allow use MPI_DOUBLE
at the sender side and MPI_LONG
(assuming the system is LP64) at the receiver side. The two data types are the same size in bytes, but converting double
to is long
pointless. There are MPI runtime validators such as MUST that can detect such problems in your code.
source to share