HPC in the UK: An Update

HPC in the UK: An Update Alan Gray, EPCC, The University of Edinburgh HPC User Forum, EPFL, October 2009

Contents EPCC UK National Services: HECToR and HPCx Case Studies Environmental Modelling Computational Materials Chemistry Fractal Generated Turbulent Flows Interactive Biomolecular Modelling FireGrid: Next Generation Emergency Response Systems

EPCC The University of Edinburgh founded EPCC in 1990 to act as the focus for its interests in simulation Today, EPCC is a leading centre for computational science in Europe ~70 permanent staff Working in academia and industry Managing both UK national HPC facilities HECToR: Cray XT5h HPCx: IBM Power5 eServer HPC Research Facilities Technology Transfer Training European Coordination Visitor Programme 4

HECToR • HECToR: High End Computing Terascale Resource • 6 year service, funded by UK government. Commenced 2007 • Used for wide variety of apps across academia and industry • Located at University of Edinburgh, managed & operated by EPCC (with help from Daresbury Laboratory staff) Cray XT5h • XT4: 5664 quad-core Opterons • peak performance 208 TFlops • X2: 112 Cray Vector Processors 6

HECToR Upgrade path Currently in Phase 2a Q1 2010: Cray ‘Baker’ (Phase 2b, stage 1) 20 cabinets; 3612 AMD ‘Magny Cours’ 12-core chips, 44,544 cores total estimated peak performance of 338 TFlops. ~30 cabinets of XT4 will be retained. Q4 2010: upgrade Baker to Gemini network (Phase 2b, stage 2) Q3 2011: Phase 3. ????? No Hardware provision contract at this time 7

HPCx • Located at Daresbury, managed and operated by EPCC and Daresbury • 160 IBM e-Server p575 nodes • 16 Power5 1.5 GHz cores per node: 2560 cores total • IBM HPS interconnect (aka Federation) • 12.9 TFLOP/s Linpack • UK policy is to have overlapping HPC services • HPCx: previously main service, now in operation as secondary service 8

Complementarity Managing both HECToR and HPCx simultaneously provided unique opportunity to maximise benefits for UK research. Run as “Complementary services”: HECToR is our leading HPC facility HPCx is our "National Supercomputer", trading overall utilisation in favour of a more flexible service Very long jobs Interactive use (computational steering, visualisation, debugging, etc) Advanced reservations … Gain experience to advantage of future HPC services. 9

Environmental modelling Lois Steenman-Clark, University of Reading HIGEM: seven UK academic groups plus UK Met Office Aim: achieve a major advance in developing an Earth System model of unprecedented resolution capable of performing multi-century simulations. Increasing horizontal resolution of Earth System models allows capture of climate processes and weather systems in much greater detail. but scientifically challenging new model created, tested, analysed, assessed, tuned and optimised The control experiment, 115 model years of HIGEM, run on HPCx 11

Environmental modelling • sea surface temperature anomalies associated with El-Nino events from • a) an observational climatology • b) the HIGEM control run • c) standard climate resolution UM experiments • HIGEM model now regularly used in current research projects • will be used for some very high resolution experiments as part of the input to the next IPCC (International Panel on Climate Change) report due in 2013. 12

Computational Materials Chemistry Richard Catlow and Scott Woodley, University College London Materials Chemistry Consortium, comprises over 25 research groups extensive applications portfolio, energy and environmental materials, catalysis and surface science, quantum devices, nano-science biomaterials Highest users of HPCx by project over lifetime of service. Currently heavily utilising HECToR 13

Computational Materials Chemistry Energy and Environmental Materials Modelling radiation damage in materials using DL_POLY MD code assessment and design of materials for use in nuclear reactors. Implemented effects of electronic stopping and electron–ion interactions within radiation damage simulations of metals, investigated the evolution of the damage on annealing for SiO2, GeO2, TiO2, Al2O3, and MgO, Simulation of damage created by 50 keV recoil atom in quartz. 14

Computational Materials Chemistry Biomaterials explored fundamental factors relating to the structure of bone, in particular the interface between apatite and collagen. MD simulation of the nucleation of hydroxyapatite in an aqueous environment at a collagen template, showing the clustering of the calcium and phosphate ions around the collagen functional groups. 15

Computational Materials Chemistry Nano-Chemistry and Nucleation rapidly expanding area: exploiting computational tools to develop models for the structures, properties and reactivities of nano-particulate matter. explored possible structures and properties of such nanoparticles, as well as how particularly stable particles can be employed as building blocks Stable octahedral clusters are connected to create microporous crystals 16

Fractal-Generated Turbulent Flows Professor Christos Vassilicos, Imperial College London New industrial fluid flow solutions urgently needed to meet unprecedented requirements Increase energy savings, reduce environmental impacts Industries which want to create or minimise turbulence Aerospace and automotive industries Reduce noise, fuel consumption, pollutant emissions. Chemical and process industries use turbulence for mixing New flow concept originating from UK: turbulent flows generated by fractal grids create intense turbulence with very little effort or power input only need very small changes to the grid to have enormous effect. Size of simulations is so large that they are impossible without HPC 17

Fractal-Generated Turbulent Flows • Fractal Square Grid • Fluids pass through grid, turbulence created 18

Fractal-Generated Turbulent Flows • Streamwise velocity in one of the planes normal to a turbulence-generating fractal square grid. (From Laizet & Vassilicos 2009.) • The first ever successful simulations of turbulence generated by fractal grids performed on HECToR in 2008 and 2009. • Used Incompact3d code 19

Interactive Biomolecular Modeling Carmen Domene, University of Oxford Aim: further understanding of “ion channel“ proteins within nervous system These regulate ion flow through the cellular membrane, exerting control on electrical signals in cells dysfunction can cause diseases in muscles, kidney, heart or bones. Improved understanding may lead to better drugs and treatment Use Computational Steering to manipulate simulation by hand, to create specific starting configurations of interest. IMD: VMD (run on user PC) connects to NAMD (on back end of HPCx). Can manipulate molecules by hand. Allows simulation of rare, but possible, mechanisms 20

Interactive Biomolecular Modeling Work has demonstrated that alternative pathways for ion conduction to the one already proposed in the literature are possible Considering these results, it would be also interesting to revisit many of the models proposed in the literature which did not successfully agree with experimental data 21

FireGrid: Next Generation Emergency Response systems Retrospective analysis of every emergency poses question: Was the response adequate? Almost invariably the answer is: Better information would have led to more effective response. • Brutally illustrated when emergency crews continued operations oblivious to impending collapse of WTC1 and WTC2. 22

FireGrid: Next Generation Emergency Response systems Large databases Emergency Response Dense Sensor network for early detection and monitoring Alerts Super Real Time Simulation of fire growth and structure response Incident Commander Building Command and Control 23

FireGrid: Next Generation Emergency Response systems Partners: The University of Edinburgh: R&D for all areas of the project EPCC, the Institute for Infrastructure and Environment, the Institute for Digital Communication, the National e-Science Centre, and the Artificial Intelligence Applications Institute BRE (Building Research Establishment) project leader and also provided the state-of-the-art experimental facilities that housed the fire ABAQUS UK Limited and ANSYS-CFX (structural mechanics and CFD software) Xtralis expertise on active fire protection systems, as well as sensor equipment in support of experiments; the London Fire Brigade: principal user and guided the development of the command and control interface. Initial project has completed prototype Integrated Emergency Response System Successful live demos (with real fires) utilised HPCx and local Edinburgh University cluster HPC resources. 24

References EPCC: www.epcc.ed.ac.uk HECToR: www.hector.ac.uk HPCx: www.hpcx.ac.uk Environmental Modelling “Moving the capability boundaries”, Lois Steenman-Clark, HPCx Capability Computing Issue 13, http://www.hpcx.ac.uk/about/newsletter/ Computational Materials Chemistry “Computational materials Chemistry on HPCx”, Richard Catlow and Scott Woodley, HPCx Capability Computing Issue 13, http://www.hpcx.ac.uk/about/newsletter/ Fractal Generated Turbulent Flows To appear under “Casestudies” section at www.hector.ac.uk Interactive Biomolecular Modelling “Interactive Biomolecular Modeling with VMD and NAMD at HPCx”, Carmen Domene,HPCx Capability Computing Issue 13, http://www.hpcx.ac.uk/about/newsletter/ FireGrid: Next Generation Emergency Response Systems http://www.epcc.ed.ac.uk/research-collaborations/casestudies/firegrid ; http://www.firegrid.org/

HPC in the UK: An Update