MPI_Comm_spawn and MPI_Reduce
I have two programs. A "master" that spawns "workers" who do some calculations, and I want the master to get the results from the workers and save the amount. I'm trying to use MPI_Reduce to collect results from workers, and workers are using MPI_Reduce to send MPI_Comm to the masters. I'm not sure if this is correct. Here are my programs:
Teacher:
#include <mpi.h>
#include <iostream>
using namespace std;
int main(int argc, char *argv[]) {
int world_size, universe_size, *universe_sizep, flag;
int rc, send, recv;
// intercommunicator
MPI_Comm everyone;
MPI_Init(&argc, &argv);
MPI_Comm_size(MPI_COMM_WORLD, &world_size);
if (world_size != 1) {
cout << "Top heavy with management" << endl;
}
MPI_Attr_get(MPI_COMM_WORLD, MPI_UNIVERSE_SIZE, &universe_sizep, &flag);
if (!flag) {
cout << "This MPI does not support UNIVERSE_SIZE. How many processes total?";
cout << "Enter the universe size: ";
cin >> universe_size;
} else {
universe_size = *universe_sizep;
}
if (universe_size == 1) {
cout << "No room to start workers" << endl;
}
MPI_Comm_spawn("so_worker", MPI_ARGV_NULL, universe_size-1,
MPI_INFO_NULL, 0, MPI_COMM_SELF, &everyone,
MPI_ERRCODES_IGNORE);
send = 0;
rc = MPI_Reduce(&send, &recv, 1, MPI_INT, MPI_SUM, 0, everyone);
// store result of recv ...
// other calculations here
cout << "From spawned workers recv: " << recv << endl;
MPI_Finalize();
return 0;
}
Working:
#include <mpi.h>
#include <iostream>
using namespace std;
int main(int argc, char *argv[]) {
int rc, send,recv;
int parent_size, parent_id, my_id, numprocs;
// parent intercomm
MPI_Comm parent;
MPI_Init(&argc, &argv);
MPI_Comm_get_parent(&parent);
if (parent == MPI_COMM_NULL) {
cout << "No parent!" << endl;
}
MPI_Comm_remote_size(parent, &parent_size);
MPI_Comm_rank(parent, &parent_id) ;
//cout << "Parent is of size: " << size << endl;
if (parent_size != 1) {
cout << "Something wrong with the parent" << endl;
}
MPI_Comm_rank(MPI_COMM_WORLD, &my_id) ;
MPI_Comm_size(MPI_COMM_WORLD, &numprocs) ;
cout << "I'm child process rank "<< my_id << " and we are " << numprocs << endl;
cout << "The parent process rank "<< parent_id << " and we are " << parent_size << endl;
// get value of send
send = 7; // just an example
recv = 0;
rc = MPI_Reduce(&send, &recv, 1, MPI_INT, MPI_SUM, parent_id, parent);
if (rc != MPI_SUCCESS)
cout << my_id << " failure on mpi_reduce in WORKER" << endl;
MPI_Finalize();
return 0;
}
I compiled both and executed like this (mpic ++ for osx):
mpic++ so_worker.cpp -o so_worker
mpic++ so_master.cpp -o so_master
mpirun -n 1 so_master
Is this the right way to run a wizard that spawns workers?
In the master, I always get 0 back from MPI_Reduce. Can I use MPI_reduce from intercommunicators or use MPI_Send from workers and MPI_Recv from master? I'm really not sure why it doesn't work.
Any help would be greatly appreciated. Thank!
source to share
MPI_Comm_get_parent
returns the parent intercommunicator, which includes the original process and all spawned ones. In this case, the call MPI_Comm_rank(parent, &parent_id)
does not return the rank of the parent, but rather the rank of the current process in the local intercommunicator group:
I'm child process rank 0 and we are 3 The parent process **rank 0** and we are 1 I'm child process rank 1 and we are 3 The parent process **rank 1** and we are 1 I'm child process rank 2 and we are 3 The parent process **rank 2** and we are 1
(note how the highlighted values differ - one would expect the rank of the parent to be the same, right?)
This is why the call MPI_Reduce()
will not succeed as all worker processes set different values for the root rank. Since there was originally one master process, its rank in the remote group parent
would be 0
, and therefore, all workers must point 0
as root to MPI_Reduce
:
//
// Worker code
//
rc = MPI_Reduce(&send, &recv, 1, MPI_INT, MPI_SUM, 0, parent);
This is only half of the problem. The other half is that entrenched collective operations (for example MPI_Reduce
) work a little differently with intercommunicators. First you need to decide which of the two groups will contain the root. After defining the root group, the root process must go through MPI_ROOT
as value root
in MPI_Reduce
, and all other processes in the root group must go through MPI_PROC_NULL
. That is, the processes in the receiving group do not participate at all in the root collective operation. Since the master code is written so that there can be only one process in the main group, then it would be enough to change the call MPI_Reduce
in the main code to:
//
// Master code
//
rc = MPI_Reduce(&send, &recv, 1, MPI_INT, MPI_SUM, MPI_ROOT, everyone);
Note that the wizard is also not involved in the pruning operation itself, for example. the value sendbuf
( &send
in this case) doesn't matter as the root will not send data to be minified - it just collects the pruning result, performed on the values from the processes in the remote group.
source to share