Example Applications Built with the Charm++ ParFUM Framework

1. Example Applications Built with the Charm++ ParFUM Framework 1) Parallel Spacetime Discontinuous Galerkin 2) Rocrem: Remeshing a Rocket Isaac Dooley Parallel Programming Lab University Illinois Urbana-Champaign http://charm.cs.uiuc.edu

2. Parallel Spacetime Discontinuous Galerkin

3. Overview • My background in parallel programming • How the Spacetime Discontinuous Galerkin Method utilizes unstructured meshes. • Requirements to parallelize SDG • Existing functionality in ParFUM which satisfies some SDG requirements • New functionality which has been added to ParFUM to support the rest of the SDG requirements LACSI 2005

4. Parallel Programming Lab • Our focus is parallel programming, especially in frameworks and dynamic or adaptive applications • We are not Computational Geometers, nor Mathematicians. • We try to build general purpose reusable high performance frameworks • Charm++ and AMPI • Focus on Migratable Objects and Virtualization • Multiple Platforms (Clusters, SMPs, BlueGene/L) LACSI 2005

5. Spacetime Discontinuous Galerkin • Collaboration with: • Bob Haber, Jeff Erickson, Mike Garland, … • NSF funded center • SDG Motivation: Spatial adaptivity is needed in structural dynamics applications. Why shouldn’t we also adapt in the time dimension? LACSI 2005

6. Spacetime Discontinuous Galerkin • Mesh generation is an advancing front algorithm called Tent Pitcher. • Adds a set of new elements called patches to the mesh, then solves them, thus advancing the front. • Each patch depends only on inflow elements. LACSI 2005

7. 1-d Mesh Generation Unsolved Patches Time Tent Pole Space LACSI 2005

8. 1-d Mesh Generation Unsolved Patches Time Solved Patches LACSI 2005

9. 1-d Mesh Generation Refinement Unsolved Patches Time Solved Patches LACSI 2005

10. Adaptive SDG • Method described in • Abedi, Zhou, et. al.; Spacetime meshing with adaptive refinement and coarsening; 2004 • Tent poles are not just pitched above existing space nodes • Entire space-time mesh or frontier is built as a mesh. Non-adaptive SDG can store patches as attributes of nodes in original mesh. LACSI 2005

11. 2-d Adaptive Mesh Generation LACSI 2005

12. 2-d Adaptive Mesh Generation LACSI 2005

13. 2-d Adaptive Mesh Generation LACSI 2005

14. 2-d Adaptive Mesh Generation LACSI 2005

15. 2-d Adaptive Mesh Generation LACSI 2005

16. Courtesy: Shuo-Heng and Michael Garland LACSI 2005

17. Master/Slave Parallelization of SDG • The first parallelization of the SDG method was based on the observation that each patch could be solved independently. • Thus the space mesh is not partitioned, but maintained on one master processor. • Workers request patches to solve from the master processor. • This method resulted in a bottleneck at the processor holding the entire space mesh. LACSI 2005

18. Must Parallelize the Geometry! • Do not want a single processor bottleneck. • We have an initial mesh, we’ll partition the geometric space mesh. • We need consistent ghost element layer. • We need a locking mechanism for updating ghost values at appropriate times to ensure we have a consistent mesh. • We will need the ability to incrementally add/remove elements to/from the mesh, maintaining consistency across all processors. LACSI 2005

19. Required for non-adaptive space-time meshing ParFUM - New Features • Incremental updates to ghost layers of adjacent processors • Locking of individual elements or nodes. • No global synchronization. • New adjacency data structures • Element to element • Node to node • Node to element LACSI 2005

20. Non-adaptive SDG Program Initial Results

21. ParFUM Support For Adaptivity • Load balancing is required in any efficient framework for adaptive SDG, since mesh partitions can differ in size by orders of magnitude. • We have already extended ParFUM to provide parallel incremental mesh modification primitives. • The primitives allow simple coding of incremental mesh modification, refinement, coarsening, and repair routines. LACSI 2005

22. Rocrem: Remeshing a Rocket

23. Rocrem: Remeshing a Rocket • Old mesh distorted by burning and pressure • Create new mesh from old by remeshing • Using Simmetrix for sequential tetrahedral meshes • Performs variable mesh sizing • Repartition new mesh • Transfer solution data in parallel from deformed mesh to new mesh • Uses accurate conservative transfer (volume-weighted averaging) for cell-centered solution data of Rocflu fluids mesh • Uses linear interpolation for node-centered solution data • Restart simulation using new mesh and solution data LACSI 2005

24. Serial Remeshing • ParFUM: stitch together partitioned surface mesh to form a serial mesh • Use Simmetrix to remesh surface and create volume • Uses regional sizing • Internally coarse volume must be refined LACSI 2005

25. Parallel Solution Transfer • ParFUM: partition mesh for parallel transfer • Collision Detection Library: match up old mesh with overlaid new mesh • Data Transfer Library: conservative, accurate volume-to-volume transfer LACSI 2005

26. Special Features • Scalable parallel solution transfer • Supports both node- and cell-centered solution data • Supports node- and face-centered boundary conditions • Preserved through remeshing and data transfer process • Boundary extrusion • Complete HDF to HDF conversion with restart files LACSI 2005

27. Boundary Extrusion • After remeshing, new mesh boundary does not perfectly coincide with old mesh boundary • Curved surface yields points on new surface beyond surface of old mesh • Solution: extrude boundary faces of old mesh to cover new mesh • All volumes of new mesh now covered by old mesh • Proceed with solution transfer LACSI 2005

28. Thank-you for listening to my talk! Questions or Comments? LACSI 2005