1 / 27

Peter Bajcsy, Rob Kooper, Luigi Marini, Barbara Minsker and Jim Myers

CyberIntegrator: A Meta-Workflow System Designed for Solving Complex Scientific Problems using Heterogeneous Tools. Peter Bajcsy, Rob Kooper, Luigi Marini, Barbara Minsker and Jim Myers National Center for Supercomputing Applications (NCSA) University of Illinois at Urbana-Champaign (UIUC)

yahto
Télécharger la présentation

Peter Bajcsy, Rob Kooper, Luigi Marini, Barbara Minsker and Jim Myers

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. CyberIntegrator: A Meta-Workflow System Designed for Solving Complex Scientific Problems using Heterogeneous Tools Peter Bajcsy, Rob Kooper, Luigi Marini, Barbara Minsker and Jim Myers National Center for Supercomputing Applications (NCSA) University of Illinois at Urbana-Champaign (UIUC) POC: Peter Bajcsy, email: pbajcsy@ncsa.uiuc.edu

  2. Outline • Problem Formulation • Meta-Workflow Definitions • Past Work • Design • Workflow Requirements Driven by Environmental Observatories • Architecture of NCSA Meta-workflow Prototype Called CyberIntegrator • Implementation • Key Capabilities of CyberIntegrator • Use Cases • Environmental and Hydrological Engineering • Summary

  3. Problem Formulation

  4. Science Problem Formulation

  5. System Problem Formulation

  6. Work Flow Problem Formulation

  7. Meta-Workflow Definition • Meta-workflow (MWF) definitions in the past: • (1) Workflow aspect: a workflow is an aggregation of tasks, a meta-workflow is an aggregation of workflows or a hierarchy of workflows • (2) Process management aspect:large activities have to be integrated, executed and evaluated in a process of conducting electronic commerce • Our meta-workflow definition includes multiple of its dimensions: • (1) hierarchical structure and organization of software, • combinatorial explosion of module connection • (2) heterogeneity of software tools and computational resources, • the number of different engines and software applications used by people for a reason • (3) usability of tool and workflow interfaces, • (4) community sharing of fragments and user friendly security, • (5) community knowledge and provenance, • (6) execution and built-in fault-tolerance, etc

  8. Previous Work • Other efforts: • Business process workflow architectures - FlowMark, WSFL and BPEL: serving business community • Scientific workflow architectures - DAGMan, Taverna, SciFlo, Kepler, D2K, OGRE, CCA, Pegasus, GridFlow and Grid Ant, Triana and GSFL • Comparison: • Our work focuses on the simplicity of end user interactions with information technologies while utilizing all execution mechanisms transparently (workflow by example). • Our work creates provenance to recommendation pipelines for the benefit of a community (recommendations based on provenance information).

  9. Research Topics • Data Translations: Semantic and syntactic mapping of data structures • Provenance Information: Granularity of gathered provenance information for recommendations, auditing and re-construction • HCI: User interface design issues and community dependencies • Meta-Data: Federation of distributed (data, tool, computational resource) registries • Execution: Just in time data delivery wrt. remote computing; Cost benefit analysis of data transfer vs. CPU requirements; Execution triggered by streaming data

  10. Design

  11. Design Goals • Make scientific discoveries easier • Workflow by example (step-by-step experimentation) • Design friendly user interfaces • Build seamless access to heterogeneous data/tools/resources • Provide data and process provenance information • Recommend data, tools and computational resources • Derive higher level semantic tools

  12. Meta-workflow Architecture

  13. Implementation

  14. Meta-Workflow Features • Workflow by example • Support of heterogeneous executors • Workflows: GeoLearn, D2K, Kepler/Ptolemy • Applications: MS Excel, Im2Learn, ArcGIS • Web services: D2KWS • Provenance • Gathering & Meta-data repositories • Recommendations

  15. Meta-workflow Editor

  16. Use Cases

  17. Meta-Workflow R&D Drivers • Community drivers: • Environmental Science: CLEANER • Hydrological Science: CUAHSI • Science drivers: • Environmental Modeling of Nutrient Distribution • Monte Carlo simulations of maximum amount of pollution that a water body can receive each day and still retain its uses • Understanding the Dynamic Evolution of Land-Surface Variables in the Illinois River Basin • Data-driven analyses of multi-variable relationships from remote sensing data • Technology drivers: • Collaboratory Cyberenvironments

  18. Summary • The problem of designing a highly interactive scientific meta-workflow system is very complex • Key capabilities of our meta-workflow prototype implementation called CyberIntegrator were demonstrated with two use cases. • We plan on building and deploying a practical tool for multiple communities. • Publications: • Image Spatial Data Analysis Group at NCSA: • URL: http://isda.ncsa.uiuc.edu • Questions: • Peter Bajcsy; Email: pbajcsy@ncsa.uiuc.edu

  19. Hydro-informatics

  20. Backup

  21. Meta-workflow System Information

  22. Terminology • Engines are stand-alone environments and applications that are used by many tools • Examples: Matlab, MS Excel, D2K, Im2Learn, ArcGIS, Kepler • Tools are solutions specific to a problem and consist of several algorithms • Examples: Image Calculator in Im2Learn, Pie chart visualization in MS Excel, … • Algorithms are code fragments that perform a specific operation in a tool • Examples: image addition operation in Image Calculator

  23. Environmental Science

  24. Hydrological Science

More Related