Terascale Simulation Tools and Technologies Center

1. Terascale Simulation Tools and Technologies Center Jim Glimm (BNL/SB), Center Director David Brown (LLNL), Co-PI Ed D’Azevedo (ORNL), Co-PI Joe Flaherty (RPI), Co-PI Lori Freitag (ANL), Co-PI Patrick Knupp (SNL), Co-PI Mark Shephard (RPI), Co-PI Harold Trease (PNNL), Co-PI TSTT

2. TSTT will develop interoperable meshing and discretization technology components • Meshing and Discretization Research and Development • high-quality, hybrid mesh generation for complex domains • adaptive mesh update procedures • high-order discretization techniques • algorithms for terascale computing • Software interoperability is a pervading theme • initial design will account for interoperability at all levels • encapsulate research into software components • define interfaces for plug-and-play experimentation • Application deployment and testing is paramount • SciDAC collaborations in accelerator design, fusion, climate and chemically reacting flows • existing DOE application collaborations in biology, mixing fluids, and many more TSTT-2

3. Existing Tools for Mesh Generation A wide variety of tools exist for the generation of … • … structured meshes • Overture- high quality predominantly structured meshes on complex CAD geometries (LLNL) • Variational and Elliptic Grid Generators (ORNL, SNL) • … unstructured meshes • MEGA (RPI) - primarily tetrahedral meshes, boundary layer mesh generation, curved elements, AMR • CUBIT (SNL) - primarily hexahedral meshes, automatic decomposition tools, common geometry module • NWGrid(PNNL) - hybrid meshes using combined Delaunay, AMR and block structured algorithms These tools all meet particular needs, but they do not interoperate to form hybrid, composite meshes Overture Diesel Engine Mesh (LLNL) MEGA Boundary Layer Mesh (RPI) TSTT-3

4. Given Geometry Specification Level 0 Domain Decomposition Level 1 Mesh Components Level 2 Parallel Decomposition Pc P8 P4 P0 P5 P9 P1 Pd Level 3 Pb P3 Pf P7 P6 Pa P2 Pe Geometric Hierarchy Required to • provide a common frame of reference for all tools • facilitate multilevel solvers • facilitate transfer of information in discretizations • Level 0: Original problem specification via high level geometric description • Level 1/2: Decomposition into subdomains and mesh components that refer back to Level 0 • Level 3: Partitioning TSTT-4

5. Geometry Information (Level A) Full Geometry Meshes (Level B) Mesh Components (Level C) Mesh Data Hierarchy • Level A: Geometric description of the domain • Accessed via tools such as CGM (SNL) or functional interfaces to solid modeling kernels (RPI) • Level B: Full geometry hybrid meshes • mesh components • communication mechanisms that link them (key new research area) • allows structured and unstructured meshes to be combined in a single computation • Level C: Mesh Components TSTT-5

6. Access to Mesh Data Hierarchy... • … as a single object (high-level common interfaces) • TSTT will develop functions that provide, e.g., • PDE discretization operators • adaptive mesh refinement • multilevel data transfer • Prototype provided by Overture and Trellis frameworks • Enables rapid development of new mesh-based applications • … through the mesh components (low-level common interfaces) • TSTT will provide, e.g., • element-by-element access to mesh components • callable routines that return interpolation coefficients at a single point (or array of points) • Facilitates incorporation into existing applications TSTT-6

7. Common Interface Specification • Initially focus on low level access to static mesh components (Level C) • Data: mesh geometry, topology, field data • Efficiency though • Access patterns appropriate for each mesh type • Caching strategies and agglomerated access • Appropriateness through working with • Application scientists • TOPS and CCA SciDAC ISICs • Application scientists program to the common interface and can than use any conforming tool without changing their code • High level interfaces • to entire grid hierarchy which allows interoperable meshing by creating a common view of geometry • mesh adaptation including error estimators and curved elements • All TSTT tools will be interface compliant TSTT-7

8. Start with a set of component meshes... … Stitch together to form a hybrid mesh … Cut holes... Mesh Data Hierarchy Construction • Level 0 to Level 1 geometry • Leverage existing TSTT tools that provide graphical interfaces to decompose the initial geometry into subdomains • CGM (SNL), Overture (LLNL) • Level 1 mesh components • Leverage existing mesh generation tools • Level C to Level B hybrid meshes • Stitching algorithms • Overlapping meshes CUBIT Geometry Decomposition (SNL) Overture Stitching Algorithm (LLNL) TSTT-8

9. Enhancing Mesh Generation Capabilities • Will leverage most existing TSTT technology “as is” • Provisions for • Creating interface compliant tools • Improving mesh generation capabilities on complex geometries for high order elements • Curvilinear elements • Geometry approximations • Interoperability of appropriate tools • e.g., ORNL elliptic and variational mesh generators with Overture • Mesh quality control for hybrid meshes Linear coarse elements verses high-order, curvilinear P elements in MEGA (RPI) TSTT-9

10. Unstructured mesh quality research and development is provided by MESQUITE (SNL, ANL) optimization-based smoothing reconnection schemes development of quality metrics for high order methods a posteriori quality control using error estimators PDE-solution based mesh optimization will be investigated for overlapping and hybrid meshes Improved mesh 8x error reduction by selecting optimal mesh generation parameters Mesh Quality Control TSTT-10

11. Dynamic Mesh Evolution • Geometry evolves due to • Adaptive mesh refinement • Internally tracked interfaces (e.g., shocks) • Motion of the domain boundary MEGA Rayleigh-Taylor Simulation (RPI) Overture simulation of Hele-Shaw flow TSTT-11

12. TSTT Research in Mesh Evolution • Requires evolution of both the hierarchy and the individual mesh components • TSTT will provide interfaces that allow • the mesh tools to access the changing geometry • the application programmer to access the changing mesh • local or global modifications • New techniques will address • Curvilinear geometries to preserve convergence rates of high order discretizations • abstraction of adaptive techniques to provide “plug and play” • adaptive techniques that use multiple criteria to extend applicability • automatic selection and application of optimal strategies TSTT-12

13. Combining TSTT technologies will improve front tracking techniques • Improve conservation properties and accuracy at the front by inserting a surface determined by front tracking into a volume mesh • Results in a front-adaptive space-time discretization FronTier interface representation TSTT-13

14. TSTT will ease the use of high order discretization methods • Observation: Complexities of using high-order methods on adaptively evolving grids has hampered their widespread use • Tedious low level dependence on grid infrastructure • A source of subtle bugs during development • Bottleneck to interoperability of applications with different discretization strategies • Difficult to implement in general way while maintaining optimal performance • Result has been a use of sub-optimal strategies or lengthy implementation periods • TSTT Goal: to eliminate these barriers by developing a Discretization Library TSTT-14

15. The Discretization Library Will... • … contain numerous mathematical operators • Start with +, -, *, /, interpolation, prologation • Move to div, grad, curl, etc. • Both strong and weak (variational) forms of operators when applicable • … contain numerous discretization strategies • Finite Difference, Finite Volume, Finite Element, Discontinuous Galerkin, Spectral Element, Partition of Unity • Emphasize high-order and variable-order methods • various boundary condition operators • … be independent of the underlying mesh infrastructure • Utilizes the common low-level mesh interfaces • All TSTT mesh tools will be available • … be extensible to allow user-defined operators and boundary conditions TSTT-15

16. Additional Functionalities • Support for Temporal Discretization • Method of Lines formulation (time steps and temporal methods are spatially independent) • Local Refinement Methods (time steps and methods vary in space) • Space-Time techniques (unstructured meshes are used in both space and time) • Support for Adaptive Methods • Error estimators • Richardson’s extrapolation (meshes of different resolution) • P-refinement estimators • Solution gradient and vorticity metrics • Optimal strategies for mesh enrichment (combinations of p- and h- adaptivity) • Combined with work on mesh quality improvement • Support for Interpolation • Between meshes and operators • Local conservation when mapping between meshes TSTT-16

17. Performance of Discretization Library • Kernel operations imply good performance is critical • Single Processor Performance • Compile time optimization of user-defined high level abstractions via ROSE (LLNL) • Consider hierarchical memory performance and cache usage • Terascale Computing • scalability of local operations requires good partitioning strategies • Efficiency determined by the size of the partition boundary relative to the partition volume • Will leverage the experience of • LLNL’s Overture project that supports structured mesh topologies • RPI’s Trellis project for variational discretization TSTT-17

18. Benefits of the Discretization Library • Lowers the time, cost, and effort to effectively deploy modern discretization tools • High-level access for new application development on TSTT Level B meshes • Mid- and low-level access for insertion into existing technology • Increases reliability of application codes by eliminating a common source of coding errors • Enhances software reuse • Permits easy experimentation with various combinations of discretization strategies and mesh technologies for a given application TSTT-18

19. Issues in Terascale Computing • Observation: Many tools exist that utilize hierarchical design principles to achieve good performance at the terascale • e.g., multi-level partitioners, multigrid solvers, multiresolution visualization tools • Barrier: Their union is not optimized • often difficult to take advantage of the multiresolution representations from one solution stage to the next • TSTT Goal: To design our hierarchy and tools so that downstream tools can take advantage of the multi-resolution information • Actively consider trade-offs across the entire simulation • Allow preservation of information as desired • e.g., subdomain decompositions used in creating a hybrid mesh may be similarly useful in preconditioning of iterative solvers TSTT-19

20. Load Balancing • Use existing tools for partitioning • Chaco and Metis for static partitioning • Zoltan library (SNL) for dynamic partitioning • Develop and provide interfaces from TSTT software to Zoltan to ensure seamless operation • Augment Zoltan • Research methods to accommodate hierarchical machine models and heterogeneous parallel computers • different processor speeds, memory capacities, cache structures, networking speeds • RPM (RPI) and PADRE (LLNL) serve as prototypes • Load balancing strategies for adaptive, structured, overlapping grids • MLB (LLNL) serves as a prototype TSTT-20

21. TSTT / TOPS Interactions • Interactions of mesh adaptation and solvers • The mesh and resulting equations to be solved evolve as solution process proceeds • Not effective to re-construct system from scratch on each mesh up-date • Can easily describe a good starting solution on most adaptively refined meshes • Critical to determine how to optimize the entire adaptive solution process - sub-optimization of specific components may yield poor overall efficiency • Consistent control of parallel partitions critical - adaptive methods employ dynamic load balancing • Absolutely must avoid “serial bottlenecks” TSTT-21

22. TSTT / TOPS Interactions • Interaction of mesh adaptations and multilevel solvers • Mesh-based coarseners • Defining adaptive multiscale levels for solution process • Taking advantage of the natural hierarchies available with p-version adaptivity • Dealing with high-order, mixed and other equation discretization complexities • High order discretizations have different numerical conditioning and sparseness patterns • Mixed methods have conditioning issues • Mixed methods have natural partitioning that can be important to dealing with conditioning issues • Temporal adaptivity • Adaptive control of local time steps critical to many methods TSTT-22

23. TSTT / TOPS Interactions Contact Information • Joe Flaherty • Rensselaer Polytechnic Institute • Flaherje@cs.rpi.edu 518-276-6305 • Mark Shephard • Rensselaer Polytechnic Institute • Shephard@rpi.edu 518-276-8044 • Lori Freitag • Argonne National Laboratory • freitag@mcs.anl.gov 630-252-7246 TSTT-23