MPI.NET Tutorial: Point-to-point Communication

In This Section

• Point-to-Point Communications
• Ring Around the Network
• Data Types and Serialization

Point-to-point communication is the most basic form of communication in MPI, allowing a program to send a message from one process to another over a given communicator. Each message has a source and target process (identified by their ranks within the communicator), an integral tag that identifies the kind of message, and a payload containing arbitrary data. Tags will be discussed in more detail later.

There are two kinds of communication for sending and receiving messages via MPI.NET's point-to-point facilities, blocking and non-blocking. The blocking point-to-point operations will wait until a communication has completed on its local processor before continuing. For example, a blocking Send operation will not return until the message has entered into MPI's internal buffers to be transmitted, while a blocking Receive operation will wait until a message has been received and completely decoded before returning. MPI.NET's non-blocking point-to-point operations, on the other hand, will initiate a communication without waiting for that communication to be completed. Instead, a Request object, which can be used to query, complete, or cancel the communication, will be returned. For our initial examples, we will use blocking communication.

For our first example of point-to-point communication, we will write a program that sends a message around a ring. The message will start at one of the processes--we'll pick the rank 0 process--then proceed from one process to another, eventually ending up back at the process that originally sent the data. The figure below illustrates the communication pattern, where is process is a circle and the arrows indicate the transmission of a message.

To implement our ring-communication application, we start with the typical skeleton of an MPI program, and give ourselves an easy way to access the world communicator (via the variable comm). Then, since we have decided that process 0 will initiate the message, we give rank 0 a different code path from the other processes in the MPI program.

using System;
using MPI;

class Ring
{
    static void Main(string[] args)
    {
        using (new MPI.Environment(ref args))
        {
            Communicator comm = Communicator.world;
            if (comm.Rank == 0)
            {
                // program for rank 0
            }
            else // not rank 0
            {
                // program for all other ranks
            }
        }
    }
}

This pattern of giving one of the processes (which is often called the "root", and is typically rank 0) a slightly different code path than all of the other processes is relatively common in MPI programs, which often need to perform some coordination or interaction with the user.

Rank 0 will be responsible for initiating the communication, by sending a message to rank 1. The code below initiates a (blocking) send of a piece of data. The three parameters to the Send routine are, in order:

• The data to be transmitted with the message. In this case, we're sending the string "Rosie".
• The rank of the destination process within the communicator. In this case, we're sending the message to rank 1. (We are therefore assuming that this program is going to run with more than one process!)
• The tag of the message, which will be used by the receiver to distinguish this message from other kinds of messages. We'll just use tag 0, since there is only one kind of message in our program.

if (comm.Rank == 0)
{
    // program for rank 0
    comm.Send("Rosie", 1, 0);

    // receive the final message
}

Now that we have initiated the message, we need to write code for each of the other processes. These processes will wait until they receive a message from their predecessor, print the message, then send a message on to their successor.

else // not rank 0
{
    // program for all other ranks
    string msg = comm.Receive<string>(comm.Rank - 1, 0);

    Console.WriteLine("Rank " + comm.Rank + " received message \"" + msg + "\".");

    comm.Send(msg + ", " + comm.Rank, (comm.Rank + 1) % comm.Size, 0);
}

The Receive call in this example states that we will be receiving a string from the processor with rank comm.Rank - 1 (our predecessor in the ring) and tag 0. This receive will match any message sent from that rank on tag zero; if that message does not contain a string, the program will fail. However, since the only Send operations in our program send strings with tag 0, we will not have a problem. Once a process has received a string from its successor, it will print that to the console and send another message on to its successor in the ring. This Send operation is much like rank 0's Send operation: most importantly, it sends a string over tag 0. Note that each process will add its own rank to the message string, so that we get an idea of the path that the message took.

Finally, we return to the special-case code for rank 0. When the last process in the ring finally sends its result back to rank 0, we will need to receive that result. The receive for rank 0 is similar to the receive for all of the other processes, although here we use the special value Communicator.anySource for the "source" process of the receive. anySource allows the Receive operation to match a message with the appropriate tag, regardless of which rank sent the message. The corresponding value for the tag argument, Communicator.anyTag, allows a Receive to match a message with any tag.

if (comm.Rank == 0)
{
    // program for rank 0
    comm.Send("Rosie", 1, 0);

    // receive the final message
    string msg = comm.Receive<string>(Communicator.anySource, 0);

    Console.WriteLine("Rank " + comm.Rank + " received message \"" + msg + "\".");
}

We can now go ahead and compile this program, then run it with 8 processes to mimic the communication ring in the figure at the beginning of this section:

C:\Ring\bin\Debug>mpiexec -n 8 Ring.exe
Rank 1 received message "Rosie".
Rank 2 received message "Rosie, 1".
Rank 3 received message "Rosie, 1, 2".
Rank 4 received message "Rosie, 1, 2, 3".
Rank 5 received message "Rosie, 1, 2, 3, 4".
Rank 6 received message "Rosie, 1, 2, 3, 4, 5".
Rank 7 received message "Rosie, 1, 2, 3, 4, 5, 6".
Rank 0 received message "Rosie, 1, 2, 3, 4, 5, 6, 7".

In theory, even though the processes are each printing their respective messages in order, it is possible that the lines in the output could be printed in a different order (or even produce some unreadable interleaving of characters), because each of the MPI processes has its own "console", all of which are forwarded back to your command prompt. For simple MPI programs, however, writing to the console often suffices.

At this point, we have completed our "ring" example, which passes a message around a ring of two or more processes and print the results. Now, we'll take a quick look at what kind of data can be transmitted via MPI.

MPI.NET can transmit values of essentially any data type via its point-to-point communication operations. The way in which MPI.NET transmits values differs from one kind of type to another. Therefore, it is extremely important that the sender of a message and the receiver of a message agree on the exact type of the message. For example, sending a string "17" and trying to receive it as an integer 17 will cause your program to fail. It is often best to use different tags to send different kinds of data, so that you never try to receive data of the wrong type.

There are three kinds of types that can be transmitted via MPI.NET:

Primitive types

These are the basic types in C#, such as integers and floating-point numbers.

Public Structures

C# structures with public visibility. For example, the following Point structure:

public struct Point
{
    public float x;
    public float y;
}

Serializable Classes

Any class that is serializable. A class can be made serializable by attaching the Serializable attribute, as shown below; for more information, see Object Serialization using C#.

[Serializable]
public class Employee
{
    // ...
}

As mentioned before, MPI.NET transmits different data types in different ways. While most of the details of value transmission are irrelevant to MPI users, there is a significant distinction between the way that .NET value types are transmitted from the way that reference types are transmitted. The differences between value types and reference types are discussed in some detail in .NET: Type Fundamentals. For MPI.NET, value types, which include primitive types and structures, are always transmitted in a single message, and provide the best performance for message-passing applications. Reference types, on the other hand, always need to be serialized (because they refer to objects on the heap) and (typically) are split into several messages for transmission. Both of these operations make the transmission of reference types significantly slower than value types. However, reference types are often necessary for complicated data structures, and provide one other benefit: unlike with value types, which require the data types at the sender and receive to match exactly, one can send an object for a derived class and receive it via its base class, simplifying some programming tasks.

MPI.NET's point-to-point operations also provide support for arrays. As with transmitting objects, arrays are transmitted in different ways depending on whether the element type of the array is a value type or a reference type. In both cases, however, when you are receiving an array you must provide an array with at least as many elements as the sender has sent. Note that we provide the array to receive into as our last argument to Receive, using the ref keyword to denote that the routine will modify the array directly (rather than allocating a new array). For example:

if (comm.Rank == 0)
{
    int[] values = new int [5];
    comm.Send(values, 1, 0);
}
else if (comm.Rank == 1)
{
    int[] values = new int [10];
    comm.Receive(0, 0, ref values); // okay: array of 10 integers has enough space to receive 5 integers
}

MPI.NET can transmit most kinds of data types used in C# and .NET programs. The most important rule with sending and receiving messages, however, is that the data types provided by the sender and receiver must match directly (for value types) or have a derived-base relationship (for reference types).

Previous: Hello, World!

Next: Collective Communication