A Plan for a Plan

1. FSP: Validation Program PlanningPresented by Martin GreenwaldFSP Kick-off MeetingPrinceton, 7/15/2009

2. A Plan for a Plan • Refine our concept of what needs to be done • Enumerate a concrete set of tasks and deliverables • Who does which tasks? • How will they be carried out? • Estimate resource requirements (for FSP definition phase) • Define a preliminary schedule

3. Work Elements for FSP Validation Program Definition • Build team • Gap analysis • Outreach to fusion community • Assess and document lessons learned from other communities • Define validation planning tasks • Detail tasks and roles for FSP validation • Define mechanisms for collaboration with experimental groups • Coordinate with code groups • Plan and prototype documentation strategy • Plan and execute pilot projects as needed

4. Team Building • Successful validation program for FSP will require unprecedented level of collaboration between theory, modeling, experiments and diagnostics • Approximately 1 FTE explicitly funded for project definition • Will need to leverage other resources and build extended team from both inside and outside FSP • Define responsibilities for verification and validation within FSP • Identify and recruit collaborators from outside • Provide modes of communication, documentation • Web site, Wiki, etc.

5. Outreach Within MFE Community • Questions: • Assess state of art and plans for validation in MFE • How does community view priorities? • How would they approach collaboration with FSP? • How to best marshal necessary resources (outside FSP)? • How to coordinate? • Avenues for information gathering • BPO/TTF validation task force • ECC • Other topical groups? • ITER modeling team • International fusion programs • Informal get-togethers at meetings • Site visits • Build formal and informal collaborations

6. Assess Lessons Learned from Other Communities • Communities • Climate • Combustion • ASCII • Other CFD • Others? • Approaches • Literature search • Contacts (especially team members with outside contacts) • Site visits?

7. Perform Gap Analysis • Combine info from inside and outside MFE • Identify critical gaps in capabilities or methodologies, for example • Post-processing and Visualization tools • Analysis and synthetic diagnostics • Other software requirements • Statistical techniques • Metrics • Experimental capabilities • Measurement capabilities • Manpower and other resource estimates (for FSP itself)

8. Detail Tasks and Roles for FSP Validation • Tasks would include • Definition of critical physics tests, priorities • Design of validation experiments including diagnostic/experimental requirements • Code predictions • Experiment execution • Analysis/tool development • Documentation • Feedback into code development effort • Define roles for theory, code groups, analysts, experimentalists, diagnosticians • Software support • Coordination and management

9. Define Methodologies for FSP Validation • Summarize best practices • Elements include: • Roles and responsibilities • Jointly designed experiments – principles and practice • Hierarchy of experimental tests • Documentation of code predictions • Measurement and documentation requirements for experiments • Quantification of errors and uncertainties • Quantification of comparisons • Documentation methodologies

10. Critical Need: Analysts • In “high-consequence” applications, V&V is led by dedicated analysts • Weather prediction and climate modeling • Turbomachinery • Airframe design • Not tied to code development groups • Can serve as honest brokers, providing unbiased and dispassionate assessments • Requires specialized skills • Analysts would have leading role in designing and analyzing validation experiments • Close collaboration with theorists, computationalists and experimentalists • Help marshal the computational and experimental resources • Help develop post-processors, synthetic diagnostics • What is FSP role in developing/training?

11. Define Mechanisms for Collaboration with Experimental Groups • How to marshal necessary resources and build engagement and commitment • How to impact long term planning and development of capabilities • Machine operations (heating, fueling, current drive, etc) • Diagnostic development and deployment • How to organize experimental programs and interact with experimental planning • How to jointly design and execute validation experiments, including careful documentation • Identify experimental time, manpower and other resources required • Develop guidelines for publication and other IP issues • Define approach for coordination and management

12. Coordination with Code Groups (Inside and Outside FSP) • Identify critical physics for testing • Identify validation requirements at various “levels” of physics integration • Define validation experiments and measurements required • Coordinate with verification efforts • make sure this has sufficient priority and is carried out in timely manner • “Regularize” interactions • Validation results should guide code development • Ensure adequate computational resources for verification and validation • Common infrastructure • Documentation – approach, schema • Data structures, API • Post-processors, analysis tools, visualization tools and synthetic diagnostics

13. Embody V & V Planning Within FSP • As FSP plan solidifies, map out research needs and directions in detail • Ask some important questions • How will model predictions be used? • Which applications? • What are the impacts of predictions? of errors in predictions? • Assess Status • Which areas are well understood? Where are they uncertain or controversial? • What new developments in physics or methodology are required? • Define requirements for validation • When are models “ready”? • Which are the critical elements of each model? – priorities for testing • What are the experimental and diagnostic requirements?

14. Plan and Prototype Documentation Strategy (1) • What needs to be saved? • (Everything) • Include both experimental and modeling data • Raw and processed data • All auxiliary data and inputs, calibrations, assumptions, geometry, boundary and initial conditions, etc. • Estimations of errors and uncertainties • Include metadata for every data item. Create a complete, coherent, self-descriptive structure • Results of all analysis • Textual information to describe methods, physics, commentary. • Everything time stamped and attributed. • Make this list complete, specific then prototype

15. Plan and Prototype Documentation Strategy (2) • Characteristics for data storage and access • Needs to be shareable, easy to use (API critical), archival • Dynamic and interactive – able to be updated, annotated, appended • Queryable - searchable by content or by address • Browsable • Linked to publications • ? • Refine functional and non-functional requirements, define approach and prototype • (With other groups) define data access rules

16. Plan and Execute Pilot Projects • How could pilot projects aid FSP? (more or less in order of importance) • Help address open issues or questions • Validate approaches and methodology • Start to build needed collaborations • Demonstrate ROI of FSP • Pilot projects chosen to meet above criteria • Workshops? • Encourage and solicit proposals?

17. Some Short-Term Actions • Build team and define roles • Provide tools for collaboration and documentation (Should be FSP-wide, but structured?) • Enumerate deliverables • Define coordination with verification efforts • Define preliminary schedule

18. End

19. Elements of Thrust: Relation of Processes Imperfectly and Incompletely Measured! Schlesinger 1979