What is the exact behaviour of MPI_Wait following an MPI_Isend?

I have read all the MPI documentation and tutorials and Stack Overflow questions that I could find relevant to this, but I still do not fully understand how MPI_Wait behaves when "completing" an MPI_Isend. Can we summarize it succinctly? Does it

• A. Return when the buffer used in the corresponding Isend is usable again (I think yes, but this doesn't tell me everything I want to know).
• B. Return when the corresponding Recv completes (I think not necessarily, but maybe sometimes if the message is large?)
• C. Guarantee that the Recv can be completed by the receiving process at some later time after it returns

I ask because I am trying to implement a kind of non-blocking broadcast (since MPI_Ibcast is an MPI 3 thing and doesn't seem to exist in any of the implementations I actually encounter). The sequence I am currently following on each process is this:

1. MPI_Isend from every process to every other process
2. Do some other work
3. MPI_Wait for all the Isends to 'complete', whatever that means exactly
4. MPI_Recv all the messages

This seems to work fine in practice, but I don't know if it is guaranteed to work, or if I am just lucky because my messages are small (they are just one int each, so I suspect that they get rapidly shuffled off by MPI into some internal buffers or whatever). I don't know whether this would create deadlocks if the messages were bigger (I worry that in this case not all the MPI_Waits would return, because some might be deadlocked waiting for MPI_Recvs to happen on another process).

If this is not guaranteed to work in general, is it at least guaranteed to work for very small messages? Or is even this not necessarily true? I am really not sure what I can count on here.

If this is not guaranteed to work then how can a non-blocking broadcast be implemented? Perhaps there is some clever sequence of performing the Waits and Recvs that works out? Like first rank 0 Waits for rank 1 to Recv, then rank 1 Waits for rank 0 to Recv? Is some kind of switched pairing arrangement like that more correct?

1 Answer

Your conditions are met by the following:

MPI_Isend followed by MPI_Wait both complete:

A. The buffer used in the corresponding Isend is usable again.

If you were to use MPI_Issend and MPI_Wait:

almost B. Return when the corresponding Recv is posted.

The following is true right after MPI_Send:

C. Guarantee that a matching Recv can be completed by the receiving process at some later time after it returns.

In your suggested non-blocking broadcast, you would have to allow to swap 3. and 4., otherwise there is a deadlock. That means, there must not be a strict condition that 4. happens after 3. Since 3 happens on the root, and 4 happens on all other ranks, it may not even be an issue.

I would, however suggest to instead a cleaner approach:

1. MPI_Isend on the root to all processes
2. MPI_Irecv on **all88 processes from the root
3. Do some other work (on all processes)
4. MPI_Waitall for all posted requests (both send/recv) on all ranks

That should be clean to implement and work just fine. Of course that is not an optimized collective... but that is another topic.