1 / 19

Virtual Geophysics Laboratory (VGL): Monopolising on HPC and Cloud

Virtual Geophysics Laboratory (VGL): Monopolising on HPC and Cloud. Minerals down under. Ryan Fraser, Terry Rankine, Robert Woodcock, Josh Vote, Pavel Golodoniuc (CSIRO) Lesley Wyborn, Richard Chopping (GA) Joseph Antony , Ben Evans (NCI). Different starting point: AuScope Grid and SISS.

lsteele
Télécharger la présentation

Virtual Geophysics Laboratory (VGL): Monopolising on HPC and Cloud

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Virtual Geophysics Laboratory (VGL): Monopolising on HPC and Cloud Minerals down under Ryan Fraser, Terry Rankine, Robert Woodcock, Josh Vote, Pavel Golodoniuc (CSIRO) Lesley Wyborn, Richard Chopping (GA) Joseph Antony , Ben Evans (NCI)

  2. Different starting point: AuScope Grid and SISS • AuScope Grid • Establishing governance and sustainability of services • Delivering solutions to research and government organisations • Interoperability • Spatial Information Services Stack (SISS) • An open-source, open-standards SDI • Achieves Interoperable data exchange • Standardises on 3 components – Format, Content, Tools

  3. V(what)GL Done • VEGL – Virtual Exploration Geophysics Laboratory • One primary science collaboration • One primary workflow • One collection of geophysical data sets • VGL - Virtual Geophysics Laboratory • National eResearch Collaboration Tools and Resources funded activity • Collaboration with multiple partners (CSIRO, NCI, GA, UQ, Monash, ANU) • Supporting multiple workflows • Model Library • New data sets • New data types

  4. Scientific workflow – Virtual Geophysics Laboratory (VGL) • Automates and massively expands Geophysicists computational capacity via the Cloud • Amazon – EC2 (and others using a compliant API - Eucalyptus) • OpenStack • Collaboration between CSIRO, GA, NCI, Monash, UQ, and ANU. • Web based GUI • Leverages SISS data providers, and cloud technologies • Non-Restrictive Open-source • Workflow engine • Simulation codes

  5. A Workflow: Watch Closely Hardcopy of data Intrepid MATLAB Get handed field data Visualise data Transform to a regular grid Download results Crop data to area of interest Reformat data for processing SSH Client Configure job and start processing Upload data to NCI

  6. There seems to be a problem… • Reproducibility? • What was the input of your model? • What transformations occurred? • Where are the inputs? • Where is the code/libs/os/system it was run on? • Would you get the same results? Would you draw the same conclusions? • It’s a manual process. • Time consuming • Error prone • Expensive • Licensing costs for 3 commercial packages, plus ‘connective tissue’ Limited by size

  7. Our solution • Virtual Geophysics Laboratory - http://vgl.auscope.org/ • Provenance (“You want to do WHAT with the result?”) • All input data is saved and then published with the final data product • VGL automates large portions of the workflow • Allowing scientists to focus on science • Share their workflows, and or results • Integrate someone else output as their input! • Built entirely on open source tools • No licensing costs

  8. CLOUD

  9. Data Discovery

  10. Data Selection

  11. Script Builder

  12. Job Monitoring

  13. Published provenance records

  14. From this… Hardcopy of data Intrepid MATLAB Get handed field data Visualise data Transform to a regular grid Download results Crop data to area of interest Reformat data for processing SSH Client Configure job and start processing Upload data to NCI

  15. …to this Virtual Geophysics Laboratory Select spatial bounds Build “science” from existing libraries Discover raw data Run job Collect and publish results

  16. VGL - Summary • The VL presented can be re-used for other scientific workflows. It’s built from many ‘generic’ components that can be repurposed • https://svn.auscope.org/subversion/AuScopePortal/ • It can produce actual scientific data products ready for others • GA’s National Grav Map could have been produced using it. • Is capable of integrating data from ANY SISS data provider. • The power lies with the underlying services, accessed using standardised protocols

  17. Future Work • Supporting multiple workflows • Enhancing provenance workflows to increase transparency and reusability • New data types (3D models, base survey data instead of grids) • New data sets (at least 6 regional 3D models, GA geophysical survey data) • Model results library (results stored in a spatially enabled registry) • More resources? • Already has the capability to use more HPC resources (NCI, Pawsey) • The workflow engine will also drive: NeCTAR cloud, Commercial Clouds.

  18. Memes to Explain our Collaboration Team! Computational Geologist (puts Geology Processes into Code) Field Geologist (acquires Data and defines processes to model) HPC Expert (makes Code run On HPC) Desk Geologist (User) – Answer science questions SW Eng – Interface between HPC Expert and Client – creates Client Programs

  19. Thank you and for more information: http://vgl.auscope.org/ https://www.seegrid.csiro.au/wiki/NeCTARProjects/VGLProjectOverview CSIRO Earth Science & Resource Engineering Ryan Fraser Project Leader Phone: +61 8 6436 8760 Email:ryan.fraser@csiro.au Web:www.csiro.au siss.auscope.org CSIRO Earth Science & Resource Engineering Terry Rankine Solutions Architect Phone: +61 8 6436 8601 Email:terry.rankine@csiro.au Web:seegrid.csiro.au www.csiro.au

More Related