Running OpenFOAM in parallel on the grid: CFD in the study of cardiovascular disease

1. Running OpenFOAM in parallel on the grid: CFD in the study of cardiovascular disease 1Bogdan Ene-Iordache, 1Massimo Rizzi, 2Daniele Cesini, 2Emidio Giorgio, 2Luciano Gaido and 2Giuseppe La Rocca 1Mario Negri Institute for Pharmacological Research, Italy; 2INFN and IGI, Italy

2. The Mario Negri Institute for Pharmacological Research is a not-for-profit biomedical research organization founded in 1961 • The Institute’s research programs span from the molecular level to the whole human being, and the findings help build up the basis for developing new drugs or making existing ones more effective • Main research headings are the battle against cancer, nervous and mental illnesses, cardiovascular and kidney diseases, rare diseases and the toxic effects of environmental contaminants, mother and child’s health

3. Mario Negri Institute is organized in Departments, Laboratories and Research Units Department of Biomedical Engineering Conducts research and development in biomedicine, both at experimental and clinical level The ongoing studies are related to four main areas: • study of the mechanisms involved in the progression of chronic nephropathy • studies on the role of haemodynamics in the development of vascular diseases • development of laboratory techniques for tissue engineering • development of information systems to manage clinical data and images generated in the context of controlled clinical trials and in routine clinical practice

4. CLINICAL BACKGROUND Introducing atherosclerosis • Atherosclerosis is the thickening of the arterial wall due to formation of an atherosclerotic plaque Carotid Artery Disease • Complex pathological process in the walls of blood vessels that develops over many years • Fatty material and cholesterol are deposited inside the lumen of blood vessels. These deposits cause stenosis of the blood vessel • The plaque can trigger formation of blood clot Coronary Heart Disease • Atherosclerosis leads to heart attack, stroke, both with high mortality Source: National Heart Lung and Blood Institute http://www.nhlbi.nih.gov/health/health-topics/topics/atherosclerosis/ Peripheral Arterial Disease

5. CLINICAL BACKGROUND Global burden of Cardiovascular Disease • An estimated 17 million people die of cardiovascular diseases, particularly heart attacks and strokes, every year. • Coronary heart disease kills more than 7 million people each year, and strokes kills nearly 6 million. Most of these deaths are in developing countries. What about the future ? WHO - The Atlas of Heart Disease and Stroke (http://www.who.int/cardiovascular_diseases/resources/atlas/en/) Year 2002

6. CLINICAL BACKGROUND Haemodynamics and atherosclerosis • The entire arterial tree is exposed to atherogenic effects of systemic risk factors • However, the atherosclerotic lesions form at specific arterial sites • The haemodynamic conditions play a decisive role in vascular remodelling and in the development of atherosclerosis • Understanding of the patho-biologic processes responsible for atherosclerosis might allow identification of new therapies

7. BACKGROUND Mechanical forces acting on the vessel wall Pressure (p) is the force per unit area normal to the vessel wall, and results in circumferential stretching of the vessel wall Shear stress (t) is the tangential force per unit area of the vessel wall and is exerted in the direction of blood flow u(r) u t = Shear stress Shear rate du r ) t(r) = m ( [ dynes/cm2] dr m = blood viscosity C Hahn* and MA Schwartz, NatRev, 2009

8. BACKGROUND Effect of different blood flow patterns Endothelial cells (EC) act as mechanoreceptors detecting and responding to shear stress After mechanoreceptor activation, a complex network of intracellular pathways is triggered (mechanotransduction) DISTURBED FLOW LAMINAR FLOW Regions in which complex flow patterns develop (oscillatory flow) Regions in which flow is always in the same direction and patterns are laminar C Hahn* and MA Schwartz, NatRev, 2009

9. BACKGROUND Athero-prone and athero-protective waveforms • Athero-prone shear stress waveform induces: • Dysregulation of EC cytoskeletal and • junctional proteins • Proinflammatory phenotype • Cytokine inducible cell surface expression of adhesion molecules associated with atherosclerosis Dai G et al., PNAS, 2004

10. BACKGROUND CFD in the study of arterial diseases Malek et al, JAMA, 1999 (review)

11. BACKGROUND “Disturbed-flow” indicators (haemodynamic wall parameters) TAWSS = Time AveragedWallShear Stress Lee, J BiomechEng, 2009 OSI = Oscillatory Shear Index quantifies the deviation of WSS from its “natural” direction during the cardiac cycle. Varies from 0 e 0.5. He and Ku, J Biomech Eng, 1996 RRT = Relative Residence Time is a combination of both TAWSS and OSI H. A. Himburg, Am J Physiol Heart CircPhysiol, 2004

12. OpenFOAM: overview • OpenFOAM (Open Field Operation and Manipulation) CFD Toolbox is a free, open source CFD software package produced by a commercial company, OpenCFD Ltd • (latest news: was acquired by ESI group on Sept 12) and distributed by the OpenFOAM Foundation (www.openfoam.org) • Large user base across most areas of engineering and science, from both commercial and academic organizations • OpenFOAM has an extensive range of features to solve from complex fluid flows involving chemical reactions, turbulence and heat transfer, to solid dynamics and electro-magnetics www.openfoam.com

13. Installing OpenFOAM on grid infrastructure • Thanks to the NGI_IT User Support team, OpenFOAM (v. 2.0.1) with openmpi support (ver. 1.5.3) and ParaView (ver. 3.10.1), has been successfully installed on the NGI_IT computing infrastructure • Due to the peculiar directory structure and the need to appropriate script sourcing on all the grid nodes before to start the analyses, the installation and the configuration of OpenFOAM on the grid infrastructure was not easy • Additional SW packages were requested: • OpenFOAM-2.0.1 builds on many Linux distributions but the ParaView-3.10.1 supplied in ThirdParty required the following add-ons: • cmake-2.8.6 or higher; • Qt-4.6.4; • Compiler GNU ver. 4.0.0. • Defined a new MPI wrapper for OpenFOAM: • It has been created in /opt/i2g/etc/mpi-start/, the following MPI wrapper /opt/i2g/etc/mpi-start/openmpi_openfoam.mpi to run OpenFOAM in parallel • Post-configuration on the LSF master node • In order to properly set-up the OpenFOAM environment on all WNs dedicated to the parallel execution, the following setting . /opt/exp_soft/gridit/OpenFOAM/OpenFOAM-2.0.1/etc/bashrc has been included in the /etc/bashrc of the LSF master node

14. OpenFOAM in grid: more details • The open-source software has been installed for the GRIDIT VO For technical information about installation and configuration of OpenFOAM in grid, please refer to the following wiki: https://wiki.italiangrid.it/twiki/bin/view/UserSupport/OpenFoam

15. (serial case x 1.0) 1 CPU speed-up x 1.3 4 CPU speed-up x 1.1 8 CPU 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 (serial case x 1.0) 1 CPU speed-up x 3.0 4 CPU speed-up x 3.9 8 CPU 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 OpenFOAM in grid: validation • To test the installation of OpenFOAM, we have used OpenFOAM tutorial cases pitzDaily 2-D mesh - 12,200 cells simpleFoamsolver steady state case • To validate the installation of OpenFOAM in a grid cluster, we have used parallel jobs with very big cylindrical meshes (1,824,200 cells) 3-D big mesh - 1,824,200 cells icoFoam solver transient case - 100 timesteps 15

16. Application’s workflow From a technical point of view, a parallel CFDrun is performed in three different steps: • Initially, the master node prepares the case by using OpenFOAM’s domain decomposition decomposePar utility: processorNN directories for each CPU are created with decomposed mesh, fields, solution controls, model choice and discretization parameters; distribution of input data to each slave node is subsequently guaranteed by the grid infrastructure (mpi_start) • The CFD case is then solved with icoFoam solver on a per-processor basis, where each CPU node uses its local disk space to store results data in time directories • At the end of the run, results are collected from the slave nodes to the master node, and the resulting archive is then saved on a grid SE using user-defined post hooks The numerical and graphical post-processing is performed locally after using the reconstructPar utility of OpenFOAM to recreate the case to a single CPU

17. 3.0 2.5 2.0 1.5 1.0 0.5 0 0.2 0.4 0.6 0.8 1 CFD simulation on Carotid Bifurcation METHODS • UNSTEADY CFD RUNS simulation of 3 cardiac cycles icoFoam solver -parallel implicit, backward timeintegration velocity waveform in CCA: ECA ICA 3-D mesh ~ 200,000 cells Q/Qmean t/T Ford MD, et al., Physiol Meas, 2005 • BLOOD PROPERTIES CCA Density ρ= 1.045 g/cm3 m = 0.033 Poise (Newtonian) Viscosity

18. 3.0 3.0 2.5 2.5 2.0 2.0 1.5 1.5 1.0 1.0 0.5 0.5 1 1 0 0 0.2 0.2 0.4 0.4 0.6 0.6 0.8 0.8 Simulation on Carotid Bifurcation RESULTS: WSS [dyne/cm2] Diastole WSS Peak systolic WSS

19. Simulation on Carotid Bifurcation RESULTS: TAWSS [dyne/cm2]

20. Simulation on Carotid Bifurcation RESULTS: OSI

21. Simulation on Carotid Bifurcation RESULTS: RRT

22. CONCLUSIONS • Low and oscillating wall shear stress is a powerful stimulus for • atherogenesis • progression of early atherosclerotic plaques • differentiation to high-risk plaque • Patient specific CFD models can be used for describing the haemodynamics in those arteries prone to atherogenesis and for predicting potential risks of developing and sustaining atherosclerosis • NGI_IT infrastructure has been successfully set up and used to efficiently run parallel CFD simulations in large arteries • Future improvements of our numerical simulations will include setting of fluid-structure interaction (FSI) for blood vessels, a new challenge for the cardiovascular biomechanics research

23. ACKNOWLEDGEMENTS Thank you !