1 / 121

Charm++ FEM Framework Tutorial

Charm++ FEM Framework Tutorial. Orion Sky Lawlor olawlor@uiuc.edu 2003/10/20. Roadmap. Why use FEM? FEM Concepts FEM Basic Calls FEM Advanced Calls Extra Features. Why use FEM?. Why use the FEM Framework?. Makes parallelizing a serial code faster and easier Handles mesh partitioning

nieve
Télécharger la présentation

Charm++ FEM Framework Tutorial

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Charm++ FEM FrameworkTutorial Orion Sky Lawlor olawlor@uiuc.edu 2003/10/20

  2. Roadmap • Why use FEM? • FEM Concepts • FEM Basic Calls • FEM Advanced Calls • Extra Features

  3. Why use FEM?

  4. Why use the FEM Framework? • Makes parallelizing a serial code faster and easier • Handles mesh partitioning • Handles communication • Handles load balancing (via Charm) • Allows extra features • IFEM Matrix Library • NetFEM Visualizer • Collision Detection Library

  5. Why not use the FEM Framework? • Does not help you write serial code • But it does help with parallel • Another thing to learn • But it has a manual and examples • Another thing to break • But it runs on native MPI as well as AMPI

  6. Scalability of FEM Framework

  7. “Overhead” of Multipartitioning

  8. Load balancer in action Automatic Load Balancing in Crack Propagation 1. Elements Added 3. Chunks Migrated 2. Load Balancer Invoked

  9. FEM Framework Users: Frac3D • Frac3D fracture mechanics • Basic structures code, but with “cohesive elements” for fracture • Authors: Scott Brietenfeld, Philippe Geubelle

  10. FEM Framework Users: CSAR • Rocflu fluids solver, a part of GENx • Finite-volume fluid dynamics code • Uses FEM ghost elements • Author: Andreas Haselbacher Robert Fielder, Center for Simulation of Advanced Rockets

  11. FEM Framework Users: DG • Dendritic Growth • Simulate metal solidification process • Solves mechanical, thermal, fluid, and interface equations • Implicit, uses BiCG • Adaptive 3D mesh • Authors: Jung-ho Jeong, John Danzig

  12. NetFEM Client: pretty pictures • Wave dispersion off a crack (simplified frac3d)

  13. FEM Concepts

  14. FEM Basics • FEM programs manipulate elements and nodes • Element is a portion of problem domain, surrounded by nodes • Node is one point in the domain

  15. Serial FEM Mesh

  16. Partitioned Mesh

  17. FEM Parallel Model: Shared Nodes • “Shared Node” model • Element computations based on values of surrounding nodes • Node values are sum of surrounding elements • Example: Mechanical Simulation • Element stresses are computed from locations of element’s nodes • Sum up forces at nodes

  18. FEM Mesh: Node Communication Summing forces from other processors only takes one call: FEM_Update_field Adds values from shared nodes

  19. FEM Parallel Model: Ghosts • “Ghost” model • Element computations based only on values of surrounding nodes and elements • Example: Fluid Dynamics • Element pressures and velocities come from neighboring element pressures and velocities, and node locations

  20. FEM Mesh: Ghosts Serial Mesh 1 2 3 4 Partitioning Right Chunk Left Chunk 1 2 Ghost of 3 Ghost of 2 3 4 Communication

  21. FEM Program Structure read mesh, connectivity, boundary conditions time loop           element loop- Element deformation applies forces to surrounding nodes           node loop- Forces and boundary conditions change node positions end time loop write out mesh data for postprocessing

  22. Parallelization • Partition the FEM Mesh into multiple chunks • Distribute elements, replicate shared nodes and/or add ghosts • Keep track of communication • Partition so that communication is minimized

  23. Parallel FEM Program read/get chunk mesh data, connectivity, shared nodes chunk time loop           element loop- Element deformation applies forces to surrounding nodes  <update forces on shared nodes>           node loop- Forces and boundary conditions change node positions end time loop write chunk mesh data for postprocessing

  24. FEM Framework Program • Consists of at least two user-written subroutines • init • driver • initis called on processor 0 • driver is called on every chunk

  25. init subroutine init          read the serial mesh and configuration data inform the framework about the mesh end subroutine

  26. driver subroutine driver          get local mesh chunk           time loop                FEM computations communication               more FEM computations           end time loop end subroutine

  27. Structure of an FEM Application init() Update driver Update driver Update driver

  28. A Serial Program Processing Input Output

  29. A Parallel Framework Program Parallel Processing Input Output Parallel Infrastructure

  30. Real Names of Pieces Driver Init Finalize or Mesh_updated Charm++ FEM Framework

  31. FEM Mesh Access Calls New and Improved!

  32. Old FEM Mesh Access Routines FEM_Set_elem FEM_Set_elem_data FEM_Set_elem_conn FEM_Set_node FEM_Set_node_data FEM_Set_sparse FEM_Set_sparse_elem FEM_Get_elem FEM_Get_elem_data FEM_Get_elem_conn FEM_Get_elem_ghost FEM_Get_node FEM_Get_node_data FEM_Get_node_ghost FEM_Get_sparse_length FEM_Get_sparse FEM_Get_sym Also, F90 versions of each Missing: FEM_Get_sparse_ghost

  33. New FEM Mesh Access Routine (!) void FEM_Mesh_data( int mesh, int entity, int attr, void *data, int first, int length, int datatype,int width ); Get/Set, and multi-mesh support NODE, ELEM, SPARSE (+GHOST) DATA, CONN, SYM, GLOBALNO,… User data: width x length array Apply to first…first+length-1 User data formatting

  34. New Mesh Access: Example FEM_Mesh_data( mesh, FEM_NODE, FEM_DATA+23, coord, 0,nNodes, FEM_DOUBLE,3 ); e.g., FEM_Mesh_default_read() We’re changing node values User data (tag 23) An array of 3 x nNodes doubles Change all the nodes 3 doubles/node 3 X Y Z X Y Z X Y Z nNodes X Y Z X Y Z

  35. New Mesh Access: Advantages • Novice users have only one routine to learn • User’s read and write routines can be similar or even identical (just like PUP) • Smart users can easily abstract over parameters • E.g., “higher-order function” that sets both ghosts and real elements for any entity type • Library developers can easily, naturally add: • New entities: FEM_SPARSE, FEM_GHOST • New data: FEM_COORD, FEM_NODE_PRIMARY

  36. New Mesh Access: Disadvantages • 8 parameters is too many to keep straight • Invalid combinations detected at runtime, instead of compile time as with the old way • E.g., FEM_NODE doesn’t have FEM_CONN, so there’s no FEM_Set_node_conn • Solution: Keep the old routines • But implement them in terms of the new routine • Mix and match with new routine • Needed for backward compatability, too

  37. FEM Communication Calls

  38. Node Fields • Framework handles combining data for shared nodes and keeps them in sync • Framework does not understand meaning of node fields, only their location and types • Framework needs to be informed of locations and types of fields • Create_field once, Update_field every timestep

  39. Create a Field integer function FEM_Create_simple_field( datatype, len) integer, intent(in) :: datatype, len

  40. Update Field: Shared Nodes subroutine FEM_Update_Field(fid,nodes) integer, intent(in) :: fid varies, intent(inout) :: nodes

  41. FEM Ghost Layers

  42. Ghost Elements: Overview • Most FEM programs communicates via shared nodes, using FEM_Update_field • Some computations require read-only copies of remote elements—“ghosts” • Stencil-type finite volume computations • The push form of matrix-vector product • Many kinds of mesh modification • Ghosts are a recent addition to the FEM framework

  43. Ghosts: 2D Example Serial Mesh 1 2 3 4 Right Chunk Left Chunk 1 2 Ghost of 3 Ghost of 2 3 4

  44. Building Ghosts: • Add ghost elements layer-by-layer from init • A chunk will include ghosts of all the elements it is connected to by “tuples”—sets of nodes • For 2D, a tuple might be a 2-node edge • For 3D, a tuple might be a 4-node face • You specify a ghost layer with FEM_Add_ghost_layer(tupleSize,ghostNodes) • ghostNodes indicates whether to add ghost nodes as well as ghost elements.

  45. Building Ghosts: • FEM_Add_ghost_elem(e,t,elem2tuple) • e is the element type • t is the number of tuples per element • elem2tuple maps an element to its tuples: • A tupleSize by t array of integers • Contains element-local node numbers • Repeat this call for each ghost element type

  46. Ghosts: Node adjacency 1 2 0 /* Node-adjacency: triangles have 3 nodes */ FEM_Add_ghost_layer(1,0); /* 1 node per tuple */ const static int tri2node[]={0,1,2}; FEM_Add_ghost_elem(0,3,tri2node);

  47. Ghosts: Edge adjacency 1 2 0 /* Edge-adjacency: triangles have 3 edges */ FEM_Add_ghost_layer(2,0); /* 2 nodes per tuple */ const static int tri2edge[]={0,1, 1,2, 2,0}; FEM_Add_ghost_elem(0,3,tri2edge);

  48. Extracting and Using Ghosts • Ghosts are always given larger numbers than non-ghosts—that is, ghosts are at the end • FEM_Get_node_ghost() and FEM_Get_elem_ghost(e) • Return the index of the first ghost node or element • FEM_Update_ghost_field(fid,e,data) • Obtain other processor’s data (formatted like fid) for each ghost element of type e 0 g e

  49. Update Field: Ghosts subroutine FEM_Update_ghost_field(fid,elType,elts) integer, intent(in) :: fid,elType varies, intent(inout) :: elts

  50. Ghost Example: Mesh

More Related