Evatt R. Hawkes, Ramanan Sankaran, James C. Sutherland, Joseph C. Oefelein, Jacqueline H. Chen

1. Direct Numerical Simulations of Turbulent Combustion: Fundamental Insights Towards Predictive Models Evatt R. Hawkes, Ramanan Sankaran, James C. Sutherland, Joseph C. Oefelein, Jacqueline H. Chen Combustion Research Facility Sandia National Laboratories Livermore CA Supported by Division of Chemical Sciences, Geosciences, and Biosciences, Office of Basic Energy Sciences, DOE SciDAC Computing: LBNL NERSC, ORNL CCS/NLCF, PNNL MSCF, SNL HPCNP Infiniband test-bed, SNL CRF BES Opteron cluster Visualization: Kwan-Liu Ma and Hiroshi Akiba UC Davis

2. Outline • DNS of turbulent combustion – challenges and opportunities • Sandia S3D DNS capability – terascale simulations on Office of Science platforms • New combustion science to advance predictive models: • 2D DNS of HCCI combustion. • 3D simulations of turbulent jet flames with detailed chemistry. • Layering Large Eddy Simulation and DNS approaches. Vorticity fields in DNS of a turbulent jet flame (volume rendering by Kwan-Liu Ma and Hiroshi Akiba)

3. Turbulent combustion is a grand challenge! • Stiffness : wide range of length and time scales • turbulence • flame reaction zone • Chemical complexity • large number of species and reactions (100’s of species, thousands of reactions) • Multi-Physics complexity • multiphase (liquid spray, gas phase, soot, surface) • thermal radiation • acoustics ... • All these are tightly coupled Diesel Engine Autoignition, Soot Incandescence Chuck Mueller, Sandia National Laboratories

4. O(4) Range Continuum O(4) Several decades of relevant scales • Typical range of spatial scales • Scale of combustor: 10 – 100 cm • Energy containing eddies: 1 – 10 cm • Small-scale mixing of eddies: 0.1 – 10 mm • Diffusive-scales, flame thickness: 10 – 100 m • Molecular interactions, chemical reactions: 1 – 10 nm • Spatial and temporal dynamics inherently coupled • All scales are relevant and must be resolved or modeled Terascale computing: ~3 decades in scales (cold flow)

5. Increasing cost and resolution for fixed physical problem Combustion CFD Approaches to tackledifferent length scale ranges • Reynolds–Averaged Navier–Stokes (RANS) • Coarse meshes, full range of dynamic scales modeled • Empirical closure, bulk approximation, current engineering CFD • Large Eddy Simulation (LES) • Energetic scales resolved, sub-grid scale dynamics modeled • combustion closure required, captures large-scale unsteadiness • Direct Numerical Simulation (DNS) • No sub-grid modeling required but limited on range of scales • Building-block configurations, research tool

6. Oxidizer Fuel HO2 CH4 CH3O O Role of Direct Numerical Simulation (DNS) • A tool for fundamental studies of the micro-physics of turbulent reacting flows • A tool for the development and validation of reduced model descriptions used in macro-scale simulations of engineering-level systems • Physical insight into chemistry turbulence interactions • Full access to time resolved fields DNS Engineering-level CFD codes (RANS and future LES) Physical Models DNS Piston Engines

7. S3D code characteristics: Solves compressible reacting Navier-Stokes F90/F77, MPI, domain decomposition. Highly scalable and portable 8th order finite-difference spatial 4th order explicit RK integrator hierarchy of molecular transport models detailed chemistry multi-physics (sprays, radiation and soot) from SciDAC TSTC (see Hong Im’s talk) 70% parallel efficiency on 4096 processors on NERSC Terascale computations => need scalar optimization customized to architecture CCS, NERSC consultants S3D MPP DNS capability at Sandia S3D is a state-of-the-art DNS code developed with 13 years of BES sponsorship. s3d scales up to 1000s of processors… and beyond?

8. Outline • DNS of turbulent combustion – challenges and opportunities • Sandia S3D DNS capability – terascale simulations on Office of Science platforms • New combustion science to advance predictive models: • 2D DNS of HCCI combustion • 3D simulations of turbulent jet flames with detailed chemistry • Layering Large Eddy Simulation and DNS approaches Vorticity fields in DNS of a turbulent jet flame (volume rendering by Kwan-Liu Ma and Hiroshi Akiba)

9. HCCI combustion: Effect of temperature stratification on combustion mode and impact for predictive models Evatt R. Hawkes, Ramanan Sankaran, Jacqueline H. Chen, Hong G. Im Calculations on: NERSC Seaborg, SNL HPCND Catalyst Inifiniband test-bed.

10. HCCI combustion: motivation • Ground transportation accounts for 2/3 of petroleum usage in the US. • Exceeds domestic production – gap projected to widen  energy security. • Worldwide petroleum supplies are limited and worldwide demand for transportation is increasing rapidly. • Concerns about green-house gases and global warming  limit CO2. • Strong worldwide demand for engines that are Highly Efficient. • Low Toxic Emissions: NOX, Particulates, Hydrocarbons, & CO. • Strong international competition to build these clean and efficient engines. • American companies must be competitive in this global market. (economic security)

11. HCCI Engine Gasoline Engine Diesel Engine (Homogeneous Charge Compression Ignition) (spark-ignition) (compression-ignition) spark plug fuel injector Hot flame region: NOx & soot Hot flame region: NOx Low temperature combustion: ultra-low emissions (<1900 K) What Is HCCI? • Offers diesel-like high efficiency and low emissions. Drawing from Gerald Coleman, Caterpillar Inc.

12. 364°CA 359°CA 360°CA 365°CA 367°CA Chemiluminescence images show that natural thermal stratification causes inhomogeneous combustion modes Images of chemiluminescence in HCCI cylinder • Progression of images suggests local appearance of combustion fronts resulting from thermal stratification (John Dec and Magnus Sjöberg, SNL) 358 - 376°CA

13. Pressure traces in HCCI engine (Magnus Sjöberg and John Dec) High Load Low Load HCCI control problems • Ignition timing • no simple control mechanism like a spark or injection • Knock at high loads • rapid combustion event causes acoustics in cylinder • Can these problems be solved by deliberate stratification? • spread out heat release • Stratification will result in new combustion modes

14. Goals of study • Fundamental understanding of combustion modes in an auto-igniting thermally stratified charge. • What numerical diagnostic techniques can be applied to characterize the combustion mode? • Control issue=> need for models. What is required to model HCCI combustion? • Tested a multi-zone model by Aceves et al. (2000) SAE 2000-01-0327 • used in conjunction with RANS CFD by engine developers • accounts for compression heating effects • neglects molecular transport effects (valid for weak stratification)

15. Specified temperature stratification interacts with random 2D flow field Ignition+Periodic BC leads to compression heating Baseline: H2-Air  = 0.1 P0 = 41 atm T0 = 1070 K u’ = 0.5m/s L = 0.41cm What is the effect of different thermal stratification? Variation of amplitude of temperature stratification, T’ T’ = 3.75K T’ = 7.5 K T’ = 15.0 K T’ = 30 K Study performed- first ever DNS targeted at HCCI • Computing: • from 32-576 processors • NERSC • SNL Inifiniband test-bed (Initial Temperature Field)

16. Progress of ignition • Flow field strains initial temperature field • Ignition occurs first in hot kernels, moderated by scalar dissipation • Fronts form and propagate, causing significant heating of the remainder of the charge • Fronts annihilate and end-gas is consumed

17. T` = 3.75K T` = 7.50K T` = 15.0K T` = 30.0K DNS of HCCI combustion reveals combustion mode • Numerical Experiment: Effect of thermal stratification on controlling burn rate under homogeneous charge compression ignition conditions Increasing stratification 1.0 0.0 (Contours of heat release for different cases at maximum total heat release time.) • Low thermal stratification: volumetric combustion • High thermal stratification: flame-front like structures

18. T` = 3.75K T` = 7.50K T` = 15.0K T` = 30.0K DNS of HCCI combustion reveals effect of stratificationon heat release Increasing stratification • Rapid heat release at high load can be bad (a design issue) • Thermal stratification spreads out heat release - good for decreasing rapid heat release

19. T` = 3.75K T` = 7.50K T` = 15.0K T` = 30.0K DNS of HCCI combustion reveals effect of stratification and validates zonal model • Increasing thermal stratification promotes more flame-like structures and zonal model deteriorates with increased stratification.

20. LES, DNS provide complimentary data Which of these cases represents which part of an engine???? • DNS – treats a canonical configuration (assumed BC’s, IC’s), cannot afford whole engine • LES - treats the full geometry and can provide a good estimate of the real parameter space in the engine

21. New findings - HCCI combustion • Thermal stratification can change the combustion mode • Thermal stratification spreads out heat release • good for decreasing damaging rapid pressure rise at high load • Multi-zone model • works well for low levels of thermal stratification • mixing causes higher peak H.R. and shorter burn durations • good news – we could predict how well the model would perform by identifying the combustion mode • LES needed for real geometry – coupling with DNS to ensure relevant parameter space

22. Outline • DNS of turbulent combustion – challenges and opportunities • Sandia S3D DNS capability – terascale simulations on Office of Science platforms • New combustion science to advance predictive models: • 2D DNS of HCCI combustion • 3D simulations of turbulent jet flames with detailed chemistry • Layering Large Eddy Simulation and DNS approaches Vorticity fields in DNS of a turbulent jet flame (volume rendering by Kwan-Liu Ma and Hiroshi Akiba)

23. The effect of chemistry-turbulence interactions in turbulent nonpremixed jet flames: mixing of passive and reacting scalars Evatt Hawkes, Ramanan Sankaran, James Sutherland, and Jacqueline Chen Calculations on: PNNL MPP2, NERSC Seaborg, ORNL CCS Phoenix Cray X1, X1E

24. Air Mixing, Reaction Fuel Mixing, Reaction Air Description of runs- Temporally evolving non-premixed plane jet flames • A canonical flow with shear-driven turbulence Heat release iso-contours

25. Large computing allocations are enabling new science runs INCITE at NERSC, capability computing at NLCF ORNL, ERCAP at PNNL, NERSC Detailed H2/CO chemistry (17 d.o.f., Li et al. 2005) Parameters selected to maximize Re Case A: Re 6000 40 million grid points 480 processors on PNNL MPP2 ~ 1 tb data Case B: Re 8000 100 million grid points 1728 processors on Seaborg 192/240 processors on ORNL CCS Cray X1/E ~ 2.5 tb data INCITE calculation: to be run this summer/fall Re 8000 ~200-250 million grid points ~2000 Seaborg processors 2.5 million hours total ~ 5tb data DNS data-sets of turbulent nonpremixed H2/CO flames

26. Community data sets • How to maximize the impact of these large data-sets? • TNF workshop: International Collaboration of Experimental and Computational Researchers

27. Non-premixed combustion concepts • Mixture fraction Z: the amount of fluid from the fuel stream in the mixture • Z is a conserved (passive) scalar – (no reactive source term) • Scalar dissipation, a measure of local molecular mixing rate:

28. Example development of the jet:40 million grid PNNL run • Left: Vorticity. Right: Simultaneous volume rendering of mixture fraction, scalar dissipation and OH radical. (Rendering by Hiroshi Akiba and Kwan-Liu Ma, UC Davis)

29. 100 million grid run on NERSC Seaborg and ORNL Cray X1: Re = 8000 Vorticity Scalar Dissipation

30. 100 million grid run on NERSC Seaborg and ORNL Cray X1: Re = 8000 HO2 dissipation OH dissipation

31. Mixing timescales • Models for molecular mixing are required in the PDF approach to turbulent combustion (Pope 1985), a sub-grid model used in RANS and LES approaches. • TNF workshop – CFD predictions are dependent on mixing timescale choice. • Models assume that scalar mixing timescales are identical for all scalars and determined by the turbulence timescale. • scalars with different diffusivities? • reactive scalars?

32. is assumed to be order unity in most models is assumed to be the same for all scalars Definitions • Mechanical time-scale: • Scalar time-scale: • Time-scale ratio:

33. Mixture fraction to mechanical timescale ratio • Confirmation that mixture fraction to mechanical time scale ratio is order unity. • Average value about 1.5, similar to values reported by experiments, simple chemistry DNS, and used successfully in models. Timescale Ratio rZ

34. Effect of diffusivity Increasing diffusivity rφ • Smaller, more highly diffusive species do have faster mixing timescales

35. Finite-rate chemistry effects on mixing rφ • HO2 and H2O2 have faster mixing times in the middle of the simulation, while OH and O are lower • Diffusivity trend does not appear to hold for HO2 and H2O2 versus O and OH. • What is going on?

36. Radical production and destruction in high dissipation regions Color scale: mass fraction White contours:  OH HO2 • OH is consumed while HO2 is produced in high dissipation regions

37. Dissipation of passive and reactive scalars • Blue: Z, Green: OH, Red: HO2 • Dissipation fields of Z and HO2 are co-incident and aligned with principal strain directions • OH dissipation occurs elsewhere, more in the centre of the jet • OH is reduced in the high dissipation regions, leading to longer mixing times, opposite for HO2 • At later times, mixing rates relax, OH returns and HO2 decreases

38. 100 million grid run on Seaborg / Cray X1 • We need to establish any Re dependence • Need parametrics at larger Re • Run in progress… but seems to be showing same effects

39. Conclusions - mixing timescales • New finding: detailed transport and chemistry effects can alter the observed mixing timescales • Models may need to incorporate these effects • a poor mixing model could lead to incorrectly predicting a stable flame when actually extinction occurs • This type of information cannot be determined any other way at present • ambiguities in a-posteriori model tests • too difficult to measure • need 3D and detailed chemistry to see this

40. Outline • DNS of turbulent combustion – challenges and opportunities • Sandia S3D DNS capability – terascale simulations on Office of Science platforms • New combustion science to advance predictive models: • 2D DNS of HCCI combustion • 3D simulations of turbulent jet flames with detailed chemistry • Layering Large Eddy Simulation and DNS approaches Vorticity fields in DNS of a turbulent jet flame (volume rendering by Kwan-Liu Ma and Hiroshi Akiba)

41. Layering of LES and DNS approaches(the future...) Joseph C. Oefelein, Evatt R. Hawkes, Jacqueline H. Chen

42. DNS RANS-LES-DNS at High Re Swirl-Stabilized Premixed Combustion Azimuthal Velocity Component • LES gets lots more than RANS • High Re is too expensive for DNS (at present) • can afford 1000s now useful but… • would like 10000s and more LES RANS

43. Why LES? • LES treats the large scales directly • Large scales are geometry dependent • LES can couple directly with high Re experiments • But – LES needs sub-grid models

44. Applications Layering LES and DNS • Better BCs, ICs • Identify modelling questions • Identify relevant parameter space • Moderate Re: • Canonical problems • Cleverly designed experiments LES DNS High Re Experiments • Sub-grid models

45. Multiphysics capabilities and algorithmic framework – Oefelein • Theoretical framework (LES  DNS) • Fully-coupled conservation equations • Compressible, chemically reacting system • Real-fluid EOS, thermodynamics, transport • Detailed chemistry, multiphase flow, sprays • Dynamic subgrid-scale modeling • Numerical framework • Dual-time, all-Mach-number formulation • Generalized preconditioning methodology • Complex geometry, generalized coordinates • Massively-parallel (MPI), highly-scalable • Ported to all major platforms • Over 10 years development • Fine-grain scalability (NERSC IBM SP – Seaborg) • Grid size fixed, number of processors increased

46. Magnitude of vorticity (normalized) DNS (all scales) 40 million grid, 120,000 hours LES (large scales) 0.6 million grid, 600 hours

47. Summary • S3D is a state-of-the-art DNS capability for turbulent combustion simulations that scales to thousands of processors and is ported to all Office of Science platforms. • DOE supercomputing facilities are enabling new combustion science. • DNS of HCCI combustion revealed new combustion modes and validated use of engine models for designing high efficiency, low emissions compression ignition engines. • 3D DNS of detailed finite-rate chemistry effects in turbulent jets provides new insights and data for combustion modeling: • mixing of reactive scalars can be very different from conserved scalars • layering LES and DNS approaches will open new avenues of exploration with great potential for the future

48. 100 million grid run on NERSC Seaborg and ORNL Cray X1: Re = 8000 HO2 dissipation OH dissipation