SlideServe
Télécharger Obtenir Pro
Menu
Accueil Pro
Préférences
ApparenceLight Aide et commentaires
© SlideServe · v2026

Optimizing Tera-Scale Data Handling for Global Change Prediction

carly-saunders avatar
carly-saunders
23 diapositives · 362 Views · Oct 27, 2014

Explore possible measures and strategies for efficient data management in tera-scale numerical simulations for accurate global change prediction. Current targets include high-resolution models and data assimilation to enhance accuracy and performance. Funding from leading institutions supports ongoing research on climate variations, ecosystem changes, and more. Earth Simulator's cutting-edge infrastructure and development projects play a vital role in understanding and predicting global changes. The emphasis is on optimizing I/O performance and handling extensive output data generated by large-scale simulations.

#optimizing tera-scale data handling for global change prediction #optimizing tera-scale data handling for global change prediction ppt #powerpoint #presentation
Optimizing Tera-Scale Data Handling for Global Change Prediction - slide 1
01 / 23Optimizing Tera-Scale Data Handling for Global Change Prediction
1 / 17

Télécharger la présentation

Optimizing Tera-Scale Data Handling for Global Change Prediction
Une image / un lien ci-dessous est fourni (tel quel) pour télécharger la présentation

Politique de téléchargement : le contenu du site vous est fourni TEL QUEL pour votre information et votre usage personnel ; il ne peut être vendu, concédé sous licence ni partagé sur d’autres sites sans l’accord de son auteur.Le contenu vous est fourni TEL QUEL, pour votre information et votre usage personnel uniquement. Téléchargez la présentation en cliquant sur ce lien. Lors du téléchargement, si vous ne parvenez pas à télécharger une présentation, l’éditeur a peut-être supprimé le fichier de son serveur.Pendant le téléchargement, si vous ne parvenez pas à obtenir une présentation, le fichier a peut-être été supprimé par l’éditeur.

F I N

Plus de contenu associé

Transcription de la présentation

  1. Possible foreseeable measures for tera-scale data handling Kazutoshi Horiuchi*1 Keiko Takahashi*1 Hirofumi Sakuma*1 Shigemune Kitawaki*2 *1 Frontier Research System for Global Change *2 Earth Simulator Research and Development Center

  2. Global Change Prediction by an Integrated Three-in-one Research FORSGC Observation Assimilation data for validation Accurate & spatially representative data Accurate & spatially representative data Optimal monitoring plan High performance computing Numerical Simulation Process Study & Modeling ESRDC Sophisticated high resolution model FRSGC

  3. Frontier Research System for Global Change (Project) (Common) Goal Global Change Prediction Activities • Process Study • Model Development Funding Bodies • Japan Marine Science and Technology Center (JAMSTEC) • National Space Development Agency (NASDA)

  4. On-Going Process Studies • Climate Variations Research • Hydrological Cycle Research • Global Warming Research • Atmospheric Composition Research • Ecosystem Change Research • Research of International Pacific Research Center • Research of International Arctic Research Center

  5. Current Target of Model Development Group • Coupled Model(now based on CCSR/NIES, MOM3) • for Climate Change Experiment • Cloud Cluster Resolving Ultra High Resolution Model • for Prediction of Typhoon/Baiu Evolution • Coupled Chemistry - Global Climate Model • for Prediction of Atmospheric Composition Change • Next Generation Model(Cubic/Icosahedral Grid, CIP method) • 4DVAR Ocean Data Assimilation Model • (based on MOM3)

  6. Current Target of Coupled Model Developmenton ES • High Resolution: • Atmosphere Model part: T213L50 • Ocean Model part: 1/10 deg. 53 layers • High Performance • Estimation of Acceleration ratio • Atmosphere Model: (under estimation) • Ocean Model: 300 ~ 400 times (480PEs;60Ns) 5 days for 100 years integration

  7. Earth Simulator Research Development Center (Project) (Common) Goal Global Change Prediction Activities • Development of High Speed Parallel Computer • Understanding and Prediction of Global Change Funding Bodies • Japan Atomic Energy Research Institute (JAERI) • National Space Development Agency (NASDA) • Japan Marine Science and Technology Center (JAMSTEC)

  8. Characteristics of Earth Simulator • Peak Performance: 40TFLOPS • Number of Processor Nodes: 640 • Number of PEs: 5120 (8PEs/Node) • Interconnection Network: 16GB/s • Total Memory: 10TB • Total Secondary Storage: 600-700TB • Total Mass Storage: 1PB(84Drives)

  9. 16 Nodes 16 Nodes 16 Nodes … 16 Nodes MM MM MM MM MM MM VP 1 VP 1 VP 1 VP 1 VP 1 VP 1 VP 7 VP 7 VP 7 VP 7 VP 7 VP 7 VP 0 VP 0 VP 0 VP 0 VP 0 VP 0 … … … … … … Fiber Channel Switch … … Fiber Channel Switch … Architecture of Earth Simulator Interconnection Network WAN TSS Cluster *1 Batch Cluster *39 WS WS … WS Mass Storage System FS 84 Drives …

  10. I/O Model in Distributed Memory Parallel Computer communication … … P P P P P P … P P P (Unix) File (Data-Distributed Unix) Files Parallel File

  11. Parallel File System on ES To handle distributed data as a logically single file has advantages to develop application softwares and to process post processings. Distribution Mechanism 1 D1 P1 1 4 7 1 2 3 M1 M2 2 P2 2 5 8 D2 4 5 6 3 Proc no.: Np Dist. Size: Sd Dist. Pattern: P= BLOCK/CYCLIC Disk no.: Nd Striping Size: Ss 4 P3 3 6 9 7 8 9 D3 5 6 7 8 P 9 1 2 3 P File Image 4 5 6 P 7 8 9 Processors Disks

  12. Support for Parallel File on Several Levels User Program Compiler C F90 HPF Library FAL F90-RTP MPI-IO HPF-RTP PFS UFS PFS UFS PFS Operating System (with PFS) Hardware …… Unix File Parallel File

  13. Review of Model Development Flow Results of Process Study Improvement of Model Execution with Model Evaluation of Results Input Data Output Data • Analysis • Visualization Resources for Process Study

  14. Is it Satisfactory about I/O performance? • The faster super computers are, the larger the amount of the output data generated by large-scale simulations. • The large amount of data is stored to secondary storages and/or mass storages whose devices are slower. • Is it satisfactory about I/O performance ?

  15. Amount of Input/Output Data -Coupled Model • To answer the question, the following cases are investigated . • Case I: 1000 Years Integration for the Prediction of Global Warming, Decadal Variability, etc. • Case II: 50 Years Integration for the Analysis of El nino, Dipole Mode Events, Asian Monsoon, etc.

  16. Amount of Input/Output Data -Coupled Model • Atmosphere Model part • Ocean Model part NOTE: The amount of output data is estimated as 2 byte integer elements

  17. Estimated I/O time - Coupled Model NOTE: Time is estimated only on drive’s I/O rate. Multiple drives are assumed to be independent.

  18. Summary of I/O Performance (from the viewpoint of Model Development) • Disk I/O time might be satisfactory. • 0.2% of the simulation time • Less than 2 hours as a total • Tape I/O time might be conspicuous. • 11-35% of the simulation time for 8 tape drives • 1-6 days for 8 tape drives This inefficiency might be critical for iterative works such as model development

  19. How to Shorten Turn Around Time of Model Development • Give up outputting numerical data. • Output necessary minimum data. • Output full data, with executing tape I/O and simulations in parallel, and with tape I/O library being able to extract necessary minimum data for post processing.

  20. Shortening of TAT by Giving up Outputting Numerical Data Elapsed time for the concurrent visualization with RVSLIB in the batch processing mode on SX-4 1.6M grid (169x92x101) 6.2M grid (337x183x101) Elapsed time (h) Time increase (%) Elapsed time (h) Time increase (%) － － CFD solver only 14.5 67.5 + fixed camera 14.7 1.9 68.0 0.79 + moving camera 14.9 3.0 68.1 0.98 • The number of computational time steps was 10000. • Contour and tracers were displayed at every 10 time steps and visualized animation was stored in a file. • Time integration for moving the tracers was done at every time step for greater accuracy. * This result was provided by NEC

  21. Shortening of TAT by Outputting Necessary Minimum Data This may be Know-Hows of using ES • “Browse sets”, into which the large amount of output data is abstracted (spatially and/or temporally) within simulations, should be stored. • Specific regions of output data should be stored.

  22. Shortening of TAT by Enhancement of Tape I/O Library for Full Output Data • Tape I/O should be executed with simulations in parallel. • In mass storages, output data should be re-organized, and small subsets which are needed for post processing should be able to be extracted. This may be requirements for the improvement of ES

  23. Conclusion • I/O performance was roughly estimated and I/O problem was apprehended. • I/O problem would be avoided with concurrent visualization and/or know-how of the usage. • However we would like to examine the efficient technique for handling the large amount of data continuously to realize comfortable environment for global change prediction.