The GriPhyN Planning Process

1. The GriPhyN Planning Process All-Hands Meeting ISI 15 October 2001

2. Meeting Goals • Clarify our vision and direction • Know how we can make a difference! • Map that vision to each experiment • Develop concrete examples of what we want to build • Coordinate our research programs • Shape toolkit to challenge-problem needs • Coordinate overlapping technologies • Organize as coordinated subteams with specific missions and defined points of interaction

3. Meeting Plan • Meeting Goals • Discuss our needs; detail our year 2 plan • Create visible, demonstrable, useful results this (project) year • Review of Architecture • An in-depth look at virtual data and request mgmt • Breakouts to focus on technologies • Breakouts to focus on experiments • Determine toolkit and testbed needs • Shape concrete challenge problems • ask: when we’re done building what we propose, what value will we bring to the experiments? • create deliverables that will capture the imagination (and commitment) of the experiments

4. Refining the Project Plan • Current draft is only a framework • team members need to create the real plan • Approach: • Vision > VDT > Experiment Plans > CS • Proposal: • Vision, framework: needs your input • Miron

5. Project Plan Responsibilities • VDT: Miron • ATLAS: Jenny • CMS: Rick • LIGO: Albert • SDSS: Jim • Testbed: tbd • CS Research: tbd • GOAL: An accepted plan by Fri Nov 2

6. Project Flow

7. Project Activities • Vision Focusing • Experiment Analysis • Use cases, statistics, distributions, data flow patterns, tools, data types, HIPO • Architectural Refinement • Research • Attach both specific and general problems • Focus and coordinate • Toolkit design and development • Testbed construction • Two types: GR and GD • Challenge Problems • Result Integration – the goal!

8. Architecture • Identify new functional modules that need to be created for GriPhyN • Monitoring service, • Request Planner • Virtual Data Catalog service • Define how we will connect and integrate our solutions, e.g.: • Virtual data language • DAGman graphs

9. Technology • Focus Talks • Breakouts • Monitoring • Virtual data • Replica location • Job execution

10. Technology: Virtual Data • Catalog Structures • Catalog manipulation language (VDL) • Complex Semantic issues • Transformation signatures • Derivation generation • Fuzzy dependencies and data equivalence

11. Technology: Replica Location • Explore schema • Explore distributed replica location service • Explore interfaces to metadata databases

12. Technology: Job Planningand Execution • Connections between Condor and Globus • Roles of RSL and ClassAds • Role of DAGman job desc. language • Where planner fits in – key issue • Requirements for data staging and storage management – tied to execution plan

13. Monitoring • Drives research • Drives planning modules • One of the keys to scalability

14. Experiments • Analysis goals • Identify existing data sources • Identify gaps and how to fill them • Identify one person responsible • Challenge Problems • Identify a sequence for Y2 and items for Y3-5 • Planning goals • Identify a project coordinator responsible for the coordination and plan production: one person

15. Experiment Specifics • ATLAS: Using data challenge progressions; focus on analysis phase rather than simulation-production • CMS: Virtual data for MOP, then high throughput and automated planning • LIGO: Need to move into more complex LDAS operations and start tracking virtual data • Explore issues of tracking data dependencies that are derived from extensive searches • SDSS: starting with cluster finding application • Goldmine of use-case info waiting for analysis of processing to date – need some CS detectives!

16. Experiment Plan Template • Analysis of Experiment IT process • Testbed / infrastructure construction • VDT deployment • Challenge Problem Cycles • Identify and analyze the problem • Communicate and negotiate VDT needs • Design and construct the solution • Negotiate and perform integration

17. Experiment plan details • Analysis documents • Infrastructure plans • VDT feature requests • VDT deployments • Challenge problem analysis • Challenge solution designs • Challenge result (research publications) • Challenge solution integration, usage, and support plans

18. Research • VD Representation, provenance, transformation signatures, data generation paradigms • Request Planning and Execution • Striving for increasingly greater resource leverage with increasing both ower AND transparency • Monitoring – tieing it all together • Fault tolerance – keeping it all running! • User interfaces • How to access the power of the datagrid • Scalability: more data, faster

19. Infrastructure / testbeds • Need to create a testbed team within GriPhyN • Hope to leverage resources from the GRIDS center • Need to create a separate project plan for this activity • In this meeting, we should identify requirements for this effort

20. Meeting Tips • Designate both a leader and note taker for all sessions • Share the bandwidth / airtime! • Call out items that should be noted in the minutes • Identify action items • Track issues to take offline or devote later sessions to; identify a path to resolution • We *can* adjust the schedule as makes sense – suggest this if you see the need • Email your slides to avery@phys.ufl.edu