iMeshP Documentation
iMeshP.h -- ITAPS Parallel Mesh Interface
Release 1.0; Updated April 2009
ITAPS' iMeshP interface provides functions to create, query, and modify a parallel distribution of a mesh. Our primary target for iMeshP is distributed memory applications, but the interface is also compatible with shared memory or global address space applications. In addition, backward compatibility of the serial iMesh interface is supported, with iMesh working as an application would expect within a single process, in shared memory, and in global address space systems.
The iMeshP interface defines a model for partitioning the entities of a mesh among distinct processes. It describes the distribution of and relationships among entities on different processes. In the model, a partition is a high-level description of the distribution of mesh entities. A parallel communication abstraction is used to manage communication among entities and processes in a partition. A partition assigns entities to subsets called parts. The partition maps each part to a process such that each process may have zero, one, or many parts.
Parts are identified globally by unique part IDs and, within a process, by opaque part handles. To simplify the iMeshP interface, we allow part handles to be substituted for entity-set handles in all serial iMesh functions. Thus, operations such as adding entities to parts and querying the number of entities in a part can be achieved using the same interface as adding entities to and querying entity sets. Additional iMeshP functions provide information about part boundaries and neighboring parts.
Each entity is owned by only one part; ownership imbues the right to modify. Entities on part boundaries are shared. One part owns the entities while other part(s) have copies of the entities. Copies of additional, non-boundary entities (called ghost entities) may be requested by an application to enable efficient computation. The iMeshP data model defines rules for the amount of information about copies that an implementation must manage. For example, an entity's owner must store information about all copies of the entity, and a copy must store information about the entity's owner.
Functions to easily create and modify partitions, create ghost entities, retrieve ghost and owner entity tag data, and determine an entity's ownership status are defined in iMeshP. In addition, the iMeshP interface supports parallel operations needed for efficient computation, load balancing and mesh modification. By necessity, these operations involve parallel communication; both synchronous and asynchronous parallel operations are supported. This design enables operations such as updates of tag data in ghost entities during computation, large- or small-scale entity migration for dynamic load balancing or edge swapping, updates of vertex coordinates in non-owned vertices for mesh smoothing, and coordination in the creation of new entities along part boundaries for mesh refinement.
Below, we more carefully define these concepts. We also review iMesh concepts that are relevant to the iMeshP interface.
- The term "mesh" refers to an abstraction in the data model; it does not imply a serial or parallel distribution.
- The term "partition" refers to an assignment of a set of entities to subsets; like a "mesh," it does not imply a serial or parallel implementation.
- An application may use one or more meshes.
- A paritition can create subsets of entities from a mesh.
- Meshes can be subdivided by one or more partitions.
- Partitions contain parts. Parts contain the subsets of entities in the partition.
- A "process" can be thought of as an MPI process. The number of processes can be considered to be the result of MPI_Comm_size. The rank of a process can be thought of as the result of MPI_Comm_rank. We will think in terms of processes rather than processors. Initial implementations of the parallel interface will likely use MPI terminology directly; future implementations may accommodate other communication paradigms and libraries.
- Partitions have communicators associated with them. These communicators can be thought of as MPI communicators.
- "Global" operations are operations performed with respect to a partition's communicator.
- "Local" operations are operations performed with respect to a part or a mesh instance within a process.
- Part A "neighbors" Part B if Part A has copies of entities owned by Part B and/or if Part B has copies of entities owned by Part A.
- Each process has one or more "mesh instances." A mesh instance can be thought of as a mesh database. An implementation should support the existence of more than one mesh instance per process (e.g., it should always associate mesh data with a mesh instance). However, we expect applications would most often use only one mesh instance per process.
- There is one root set per mesh instance.
- Many iMeshP functions are analogous to functions in iMesh. Examples include iMeshP_getNumOfType, iMeshP_getNumOfTopo, iMeshP_getEntities, and iMeshP_initEntIter, but there are many others. These functions accept both an entity set handle and a part handle, to allow operations like returning all entities in a given part (represented by a part handle) with a given boundary condition (represented by an entity set). When the entity set is the root set, users may either pass the root set to these functions or pass the part handle as the entity set in the analogous iMesh function.
- Similarly, iMeshP file I/O closely aligns with iMesh file I/O. The major change is the addition of a iMeshP_PartitionHandle argument to both iMeshP_loadAll and iMeshP_saveAll, enabling I/O from parallel processes and the initialization of the partition describing the data distribution.
- Generation and management of global IDs for entities is not included in the iMeshP interface. It can be provided as a service above the iMeshP interface. Uniqueness of global IDs is managed at the partition level.
- A partition assigns entities from one mesh instance to parts.
- Each process may have one or more partition handles.
- Entities in a mesh instance can be partitioned by one or more partitions. Mesh instances know which partitions they contain.
- Users build partitions by
- (1) creating a partition handle on each process to be included in the partition;
- (2) adding parts to the partition handle within the process;
- (3) populating the parts with entities; and
- (4) calling iMeshP_syncPartitionAll to allow the implementation to compute global data for the partition.
- Implementations of iMeshP_loadAll will perform these steps for the application using the given partition handle.
- Each part is wholly contained within a process.
- A process may have zero, one or multiple parts.
- Parts are uniquely identified globally by part IDs of type iMeshP_Part. Local parts can also be accessed by part handles that provide more direct access to a part.
- Functions accepting part handles operate correctly on only local parts (parts on the calling process); they will return an error for remote (off-process) parts.
- Functions that return entity information for a part, set or mesh instance return the information for all entities (including copies and ghosts) in that part, set or mesh instance. Applications can check whether an entity is owned or a ghost using iMeshP_isEntOwner or iMeshP_getEntStatus.
- For each entity that is copied onto remote parts, the owning part knows both the remote part ID and remote entity handle of all copies.
- All parts with copies of a boundary entity know the remote part ID and remote entity handle of all copies of the entity.
- All parts with copies of any entity know the part ID and entity handle corresponding to the owner of the entity.
- Many iMesh functions that accept an iBase_EntitySetHandle are also useful in the context of a iMeshP_PartHandle. These functions are reinterpreted so that they can accept either an iBase_EntitySetHandle or an iMeshP_PartHandle.
- In particular, entities are added to and removed from local parts via the same functions that are used to manipulate entity sets. That is, given a mesh instance, an entity handle, and a part handle, the entity is added to or removed from the part via calls to the following functions with the part handle passed as the entity set handle:
- Add entity to part --> iMesh_addEntToSet
- Remove entity from part --> iMesh_rmvEntFromSet
- Add array of entities to part --> iMesh_addEntArrToSet
- Remove array of entities from part --> iMesh_rmvEntArrFromSet
- iMeshP does not provide the capability to move entire parts to new processes; instead, the new process must create the part in its partition handle and then receive the entities to populate the part. In other words, parts can be added to only a local partition handle.
- Each function description includes its communication requirements. The options are described here:
- COMMUNICATION: Collective -- the function must be called by all processes in the partition's communicator. (These functions have the suffix "All" to indicate collective communication is done.)
- COMMUNICATION: Point-to-Point -- communication is used, but the communication is from one process to only one other process. The receiving process must issue an appropriate receive call to receive the message.
- COMMUNICATION: None -- the function does not use communication; only local operations are performed.
- COMMUNICATION: None++ -- no communication is done; the values are precomputed by iMeshP_syncPartitionAll or iMeshP_syncMeshAll.
- Non-blocking calls for off-processor mesh-modification return a request that indicates whether or not the operation has completed. The request is more than an MPI request; it encapsulates both the MPI information and the mesh operations that were requested. If non-blocking calls are used, appropriate calls to iMeshP "wait" or "poll" functions must be used to handle and satisfy requests.
