TeraScale Supernova Initiative

1. TeraScale Supernova Initiative Anthony Mezzacappa SciDAC PI Meeting, Charleston, SC

2. TeraScale Supernova Initiative www.tsi-scidac.org Explosions of Massive Stars • Relevance: • Element Production • Cosmic Laboratories • Driving Application • 11 Institution, 21 Investigator, 34 Person, Interdisciplinary Effort • ascertain the core collapse supernova mechanism(s) • understand supernova phenomenology • e.g.: (1) element synthesis, (2) neutrino, gravitational wave, and gamma ray signatures • provide theoretical foundation in support of OS experimental facilities • (RHIC, RIA, NUSEL, …) • develop enabling technologies of relevance to many applications • e.g. 3D, multifrequency, precision radiation transport • serve as computational science testbed • drive development of technologies in simulation “pipeline” • (data management, networking, data analysis, and visualization) With ISIC and other collaborators: 89 people from 28 institutions involved. Anthony Mezzacappa SciDAC PI Meeting, Charleston, SC

3. Anatomy of a Supernova • Core collapse supernovae are • radiatively (neutrino) driven. • Fluid instabilities, rotation, and • magnetic fields will play a role. • Strong (Einsteinian) rather than • weak (Newtonian) gravity. Bruenn, DeNisco, and Mezzacappa (2001) • Components of a Supernova Model 1. Accurate (Boltzmann) Neutrino Transport 2. Turbulent Stellar Core Flow 3. Stellar Core Magnetic Fields 4. Gravity (Einsteinian) 5. Nuclear (Stellar Core) and • Weak Interaction (Neutrino) Physics Anthony Mezzacappa SciDAC PI Meeting, Charleston, SC

4. TSI’s 3 Paths to 2D/3D Supernova Models • 1) 2D/3D Hydrodynamics/MHD with Radial Ray Transport • Use existing 1D codes along each ray. • Transport parallelized trivially (one ray per processor). • 2) 2D/3D Hydrodynamics/MHD with 2D/3D Multigroup Flux-Limited Diffusion (MGFLD) • Fully 2D/3D. • Sophisticated approximation to Boltzmann transport. • Less computationally intensive than Boltzmann transport. • Access to 3D science on TeraScale (versus PetaScale) platforms. • 3) 2D/3D Hydrodynamics/MHD with 2D/3D Boltzmann Transport • Final word. • Explosion mechanism extremely sensitive to transport treatment. • MGFLD compares well with Boltzmann in 1D. What about 2D? 3D? Anthony Mezzacappa SciDAC PI Meeting, Charleston, SC

5. TSI members first to perform 1D models with Boltzmann transport. • Mezzacappa and Bruenn (1993) • Mezzacappa et al. (2001) • Progress on 2D (and 3D) Boltzmann transport progressing rapidly. • Development of formalism for • conservative (energy and lepton number) • general relativistic • neutrino transport (analytical tour de force). • Cardall and Mezzacappa (2003) • Development of finite differencing. • Construction of GenASiS. • Completion of test problems. • Initiation of realistic 2D supernova studies. Without this, supernova simulations difficult to interpret. What makes neutrino transport difficult? 1. Difficult to develop number- and energy- conservative differencing for these “observer corrections (aberration, frequency shift).” 2. Difficult to handle “advection” terms when neutrinos and matter are in equilibrium. 3. Memory and CPU requirements. Anthony Mezzacappa SciDAC PI Meeting, Charleston, SC

6. 2D Boltzmann Neutrino Transport Test Problem Development of radiation field stationary state in nonspherical fixed medium: Density Distribution Radiation Field Energy Density and Flux Anthony Mezzacappa SciDAC PI Meeting, Charleston, SC

7. Solve for first moment of neutrino distribution (truncation of 2N-1 moments obtained with Boltzmann solution). Observer Corrections Advection Terms 2D MGFLD Equations Anthony Mezzacappa SciDAC PI Meeting, Charleston, SC

8. First 2D simulations with multifrequency neutrino transport, • advection terms, and observer corrections: • Scientific Target: Development of Proto-Neutron Star Instabilities • Close coupling of matter and neutrinos requires fully 2D transport • for an accurate assessment. • What impact do the neutrinos have on the development of these instabilities? • Running on 1024 • processors at • NERSC. • Scaling now to 2048. • Fully implicit solve. Swesty and Myra (2004) Anthony Mezzacappa SciDAC PI Meeting, Charleston, SC

9. Initial shock location/strength depend on knowledge of nuclear states and their occupation during core collapse. This is a challenge in nuclear computation being addressed by TSI’s nuclear theorists. This challenge is exacerbated by the fact that nuclei increase in size (neutron and proton number) /complexity (population of states, collective excitations) during collapse. Anthony Mezzacappa SciDAC PI Meeting, Charleston, SC

10. Significant change in initial shock location and strength and stellar core profiles when • state of stellar core nuclei computed with more realistic nuclear models and when this • new nuclear physics is included in the supernova models. • Hix et al. 2003, Physical Review Letters, 91, 201102. • Langanke et al. 2003, Physical Review Letters, 90, 241102. Merger of two fields at their respective states of the art. (SciDAC enabled.) Anthony Mezzacappa SciDAC PI Meeting, Charleston, SC

11. In addition to physics of nuclei during collapse, supernova mechanism depends on • high-density physics of “nuclear matter.” • Complex nuclear many-body problem. • How sensitive is the supernova mechanism to changes in this physics? Nuclei Nuclear Matter • TSI using several “equations of state” (EOSs). • 1. Lattimer-Swesty (LS, Industry Standard) • 2. Wilson • 3. Stone-Newton (SN, New) • LS and SN EOSs are both phenomenological at some level, • but fundamentally different. • Will allow us to explore sensitivities to high-density • EOSs despite uncertainties. Anthony Mezzacappa SciDAC PI Meeting, Charleston, SC

12. 2 fundamentally new instabilities discovered by TSI (computationally): Stationary Accretion Shock Instability (SASI) • Supernova shock wave may become unstable. • Instability will • help drive explosion, • define explosion’s shape. • Operates between the proto-neutron star and • supernova shock wave. • Blondin, Mezzacappa, and DeMarino (2003) Lepto-Entropy Fingers • Operates in the proto-neutron star. • Instability may help boost neutrino luminosities, • which power the explosion. • Bruenn, Raley, and Mezzacappa (2004) Anthony Mezzacappa SciDAC PI Meeting, Charleston, SC

13. SASI Visualized with EnSight Anthony Mezzacappa SciDAC PI Meeting, Charleston, SC

14. r-process Element Synthesis • Responsible for half the elements heavier than iron. • Believed to occur above proto-neutron star after • explosion in a neutron-rich, neutrino-driven wind. • In past, difficult to obtain “right” conditions in • supernova simulations. • Neutrinos driving wind also drive it toward • proton richness. • TSI Discovery: An r-process can occur under such conditions… Near “Symmetric” (equal numbers of protons and neutrons) …for rapid wind velocities Meyer 2002, Physical Review Letters, 89, 231101 Anthony Mezzacappa SciDAC PI Meeting, Charleston, SC

15. Correspondence between structure of integro-PDE and underlying linear systems... • …Leads to Nonlinear Algebraic Equations • Linearize • Solve via Multi-D Newton-Raphson Method • Solve Large Sparse Linear Systems • Implicit Time Differencing… • Extremely Short Neutrino-Matter Coupling Time Scales • Neutrino-Matter Equilibration • Neutrino Transport Time Scales Tera- to Peta-Scale Systems Anthony Mezzacappa SciDAC PI Meeting, Charleston, SC

16. Progress (in conjunction with TOPS): • 2D/3D MGFLD • Sparse Approximate Inverse Preconditioner • Saylor, Smolarski, and Swesty (2004) • Successfully implemented in 2D MGFLD code (V2D). • 2D/3D Boltzmann Transport • “ADI” Preconditioner • D’Azevedo et al. (2004) • Successfully implemented in 1D Boltzmann code (AGILE-BOLTZTRAN). • Dense LU factorization was used for dense blocks (D’Azevedo). • Being implemented in 2D/3D Boltzmann code (GenASiS). • Sparse incomplete LU factorization for dense blocks (D’Azevedo, Eijkhout). Anthony Mezzacappa SciDAC PI Meeting, Charleston, SC

17. TSI Radiation Magnetohydrodynamics Code Suite • Hydrodynamics • VH-1 (PPM) • ZEPHYR (Second Order Finite Difference) • Zeus-MP (Second Order Finite Difference) • Magnetohydrodynamics • Zeus-MP(Second Order Finite Difference) • Neutrino Transport • MGFLD_TRAN: 1D General Relativistic Hydrodynamics • with 1D General Relativistic Multifrequency Flux-Limited Diffusion • AGILE-BOLTZTRAN: 1D General Relativistic Adaptive Mesh Hydrodynamics • with 1D General Relativistic Boltzmann Transport • V2D/V3D: 2D/3D MGFLD Transport Code (Under Development) • GenASiS: 2D/3D Boltzmann Code (Under Development) • Zeus-MP • Single-CPU performance boosted by factor of 2. • Highest stat ever seen at NERSC (30% of peak). • AGILE-BOLTZTRAN • Parallel port, scalable linear solve, PERC analysis • reduced run times from weeks to days. Analysis by PERC recently begun. Extensively analyzed by PERC. Anthony Mezzacappa SciDAC PI Meeting, Charleston, SC

18. As TSI enters “production mode” managing its Workflows has become a paramount issue. • Ideally, we would like to automate these workflows. • Data Management • Networking • Visualization These must be viewed together. • Collaboration between: • SDM • Arie Shoshani • Nagiza Samatova • Guru Kora • Ian Watkins • Mladen Vouk • Networking • Beck • Atchley • Moore • Rao • Visualization (TSI) • Blondin • Toedte Anthony Mezzacappa SciDAC PI Meeting, Charleston, SC

19. Addressing Bulk Data Transfer Needs: Current Data Generation Rate: 500 Mbps (10 TB in 2 days). • Logistical Networking • Light Weight • Low Level • Deployable … Solution • New Paradigm • Integrate storage and networking. • Multi-source, multi-stream. • Easy for TSI members to share data. • Data transfer rates 200-300 Mbps using TCP/IP! • Limit set by ORNL firewall. • Greater rates expected • outside firewall, • other protocols (e.g., Sabul). • Direct impact on TSI’s workflow! Anthony Mezzacappa SciDAC PI Meeting, Charleston, SC Atchley, Beck, and Moore (2003)

20. TSI serving as a testbed for an NSF-funded • network (Cheetah) designed to develop • provisioning technologies for dedicated • channels. • PI: Rao • TSI serving as a testbed for a proposed effort • to bring together UltraNet and Cheetah. • PI: Rao • TSI is testbed for a proposed effort to allow • a broad research community to access and • use UltraNet. • PI: Beck Anthony Mezzacappa SciDAC PI Meeting, Charleston, SC

21. Visualization • TSI members played a major role in • development of Exploratorium/Powerwall • at ORNL. • Successfully deployed EnSightfor • production, • remote, and • collaborative visualization. • Development of representations for • higher dimensional data for 2D • supernova models. • Close coupling of • data management, • networking, and • visualization. • New Rendering Techniques/Quantitative, Comparative Visualization • [Kwan-Liu Ma (UC Davis), Pat McCormick, Jim Ahrens (LANL)] • Rapid rendering. • Rendering in hardware. • Interactivity. • High-resolution, high-quality rendering. • Quantitative visualization. • Multiple variables, gradients and functions of variables. • Computation done in hardware. • Interactivity. Kwan-Liu Ma (UC Davis) Anthony Mezzacappa SciDAC PI Meeting, Charleston, SC

22. Kwan-Liu Ma (UC Davis) Anthony Mezzacappa SciDAC PI Meeting, Charleston, SC

23. Conclusions A number of scientific discoveries have been made in TSI’s first two years of operation. A number of technical and technological breakthroughs have enabled our science. Execution of the next major stage in our science has begun: Delivery of 2D supernova models. Continue to serve as a testbed for enabling technologies of relevance to other SciDAC applications. Anthony Mezzacappa SciDAC PI Meeting, Charleston, SC