Research | Partition Improvement | Predictive Load Balance | Communication Libraries | MapReduce Experiments | Scaling Results
The Inter-Processor Communication Manager (IPComMan) is a general-purpose communication package built on top of MPI, which aims to reduce data exchange costs by exploiting communications in a local neighborhood for each processor. The neighborhood is the subset of processors exchanging messages with each other during a specific communication round, which in many applications is bounded by a constant, typically under 40, independent of the total number of processors. The basic idea of the library is to keep the message-passing within subdomains, and eliminate, or reduce, the number of collective calls needed. The library provides several useful features including: i) automatic message packing, ii) management of sends and receives with non-blocking MPI functions, iii) asynchronous behavior unless the other is specified, and iv) support of dynamically changing neighborhoods during communication steps.
Each processor has its own neighborhood. While initializing a library object, each processor specifies the neighbors it is going to communicate with. From that point, the library delivers messages between a processor and its neighbors only. There are no collective calls during each communication round. There are situations when it is not possible to a priori define all the neighbors for processors, i.e., new neighbors may be encountered during a communication step. As new neighbors needing to communicate will be unaware of each other, one collective call at the end of the communication round is performed to support neighborhood changes.
To measure the performance of mesh migration, an essential part of parallel adaptive unstructured mesh procedures, IPComMan was compared with the Autopack communication utility. Initial results on IBM Blue Gene/L show that IPComMan's ability to localize communication to neighborhoods and its use of nonblocking functions leads to large decreases in communication time as the number of processors increases. Even though the differences in communication times between Autopack and IPComMan are not substantial at 128 processors, IPComMan is from 3 to 7 times faster in the 4096 processor case.