WEAG – THALES – JP 11.20 - PowerPoint PPT Presentation

slide1 n.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
WEAG – THALES – JP 11.20 PowerPoint Presentation
Download Presentation
WEAG – THALES – JP 11.20

play fullscreen
1 / 74
WEAG – THALES – JP 11.20
71 Views
Download Presentation
iokina
Download Presentation

WEAG – THALES – JP 11.20

- - - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript

  1. WEAG – THALES – JP 11.20 REVVA WE 0201 – PKIT2 Date : November2004 Slide : 1 / 74

  2. Presentation Outline Project facts ; the consortium Background and rationale The REVVA concepts The REVVA generic process The REVVA products Techniques and support REVVA technical achievements REVVA future WE 0201 – PKIT2 Date : November2004 Slide : 2 / 74

  3. Common Validation, Verification and Accreditation Framework for Simulation (REVVA) Project Facts Purpose: The THALES JP 11.20 is a Western European Union / WEAG research program funded by 5 nations: France (lead nation), Denmark, Italy, the Netherlands, Sweden. Objectives: To develop a basis for a common methodological framework for the Verification, Validation and Accreditation (VV&A) of data, models and simulations. To be a starting point for a draft standard defining VV&A taxonomy, process and methodology. Project timeline: March 2003 – September 2004 WE 0201 – PKIT2 Date : November2004 Slide : 3 / 74

  4. The consortium The THALES JP 11.20 is conducted by a multi-national consortium led by ONERA (France). FOI – FMV (SW) UNI-C (DK) TNO (NL) DATAMAT (IT) ONERA – ALEKTO (FR) WE 0201 – PKIT2 Date : November2004 Slide : 4 / 74

  5. REVVA background and rationale WE 0201 – PKIT2 Date : November2004 Slide : 5 / 74

  6. DMSO ITOP Exchange SISO Capitalization Information JP 11.20 REVVA Capitalization NATO Exchange AFDRG RTP 11.13 REVVA Background (1) WE 0201 – PKIT2 Date : November2004 Slide : 6 / 74

  7. REVVA Background (2) • VV&A state-of-the-art: • DMSO, SISO, NATO, ITOP. • M&S scientific contributions: • Systems Theory (B. Zeigler) • Contributions on the concept of validation coming from the epistemology of simulation WE 0201 – PKIT2 Date : November2004 Slide : 7 / 74

  8. VV&A rationale (1) Doing (only) what makes sense for customers and users in their search for trust. • Trust: • identification of trust components, • support (based on project knowledge, consolidation by shared domain knowledge) WE 0201 – PKIT2 Date : November2004 Slide : 8 / 74

  9. VV&A Informal Formal VV&A rationale (2) M&S supplier skillsand reputation Audit trail of experiments and uses Formal analysis ofproducts, processes andorganisation Trust WE 0201 – PKIT2 Date : November2004 Slide : 9 / 74

  10. Needs Purpose Organisation Products Processes VV&A rationale (3) Validation Verification FormalAnalysis Acceptance WE 0201 – PKIT2 Date : November2004 Slide : 10 / 74

  11. VV&A rationale (4) Activities driven and prioritised by: • Requirements (acceptance requirements) • Identification / perception of risks • Organisational needs, risks and benefits • Costs WE 0201 – PKIT2 Date : November2004 Slide : 11 / 74

  12. REVVA Concepts WE 0201 – PKIT2 Date : November2004 Slide : 12 / 74

  13. REVVA Terminology (1) • Decomposition of VV&A in Verification, Validation and Acceptance; (temporal) independence of each concept. • Verification: Model Verification. • Validation: Model Validation within an Experimental Frame (EF) • Acceptance: Before use / After use Focus on Acceptance and Validation WE 0201 – PKIT2 Date : November2004 Slide : 13 / 74

  14. REVVA Terminology (2) • Correctness: • The property of a simulation model to comply with formal rules and bodies of reference information for its representation and for the transformation of its representation into another one. • Verification: • The process which is used to construct, under a set of time, cost, skills, and organisational constraints a justified belief about model correctness. WE 0201 – PKIT2 Date : November2004 Slide : 14 / 74

  15. REVVA Terminology (3) • Validity: • The property of a simulation model to have, within a specific experimental frame, a behaviour which is indistinguishable under a set of validation criteria from the behaviour of the System of Interest. • Validation: • The process which is used to construct, under a set of time, cost, skills, and organisational constraints a justified belief about model validity. WE 0201 – PKIT2 Date : November2004 Slide : 15 / 74

  16. REVVA Terminology (4) • Acceptance: • The informed decision of a M&S customer to use the results of a simulation for a specific purpose. • The informed decision of a M&S customer to use a simulation, within an experimental frame, for a specific purpose WE 0201 – PKIT2 Date : November2004 Slide : 16 / 74

  17. REVVA Assumptions • Acceptance (and Validation) should: • be purpose oriented, • be development neutral, • be technology neutral, • recognise the impact of organisations, • recognise the impact of timeframe of M&S and VV&A, • rely on data and knowledge management. WE 0201 – PKIT2 Date : November2004 Slide : 17 / 74

  18. A comprehensive view The "M&S Product World" The "Real World" The "Problem World" The "M&S World" System Requirements M&S Questions Objectives Needs Problem Formalisation Decisions Actions M&S Enabling Products Problem solution M&S Answers WE 0201 – PKIT2 Date : November2004 Slide : 18 / 74

  19. Methodological hints • Independence • Incremental nature of activities and results • Focus on the intended purpose • Objective assessment (Acceptance Criteria) • Careful construction of a justified belief • Tailoring WE 0201 – PKIT2 Date : November2004 Slide : 19 / 74

  20. EF M Tailoring Question Answer The EF used for Acceptance (and for Validation) is derived from the Acceptance Requirements • All the activities are planed, designed and controlled by them. • The Acceptance Requirements are not (necessarily) the M&S • Requirements ; they are specific of each use and purpose WE 0201 – PKIT2 Date : November2004 Slide : 20 / 74

  21. Before M&S use1 AfterM&S use1 BeforeM&S use2 | | | t t1 t2 t3 Validation t1 Validation t2 Validation t3 Impact of time • Verification is (mostly/hopefully) a one-shot process • Validation is a permanent/recurrent issue • Acceptance and Validation are purpose oriented and tailored WE 0201 – PKIT2 Date : November2004 Slide : 21 / 74

  22. The REVVA Generic Process WE 0201 – PKIT2 Date : November2004 Slide : 22 / 74

  23. Intended purpose Acceptance Recommendation Develop ToA Evaluate V&V Report The REVVA Generic Process ToA Acquire Information Product flow Model information and system knowledge V&V report Back step Develop ToVV Assess Evidence Integration Phase Assessed items of evidence Product ToVV ToA: Target of Acceptance Assess Evidence Conduct V&V ToVV: Target of Verification and Validation V&V Items WE 0201 – PKIT2 Date : November2004 Slide : 23 / 74

  24. Phase Overlaps Develop ToA Acquire Information Develop ToVV Conduct V&V Assess Items of Evidence Assess Evidence Integration Evaluate V&V Report Main effort VV&A Endeavour Duration WE 0201 – PKIT2 Date : November2004 Slide : 24 / 74

  25. Develop ToA • Performed by the Acceptance Agent in close cooperation with the user • Develop Acceptability Criteria (AC), including Validity Criteria (VC) • Document rationale for development WE 0201 – PKIT2 Date : November2004 Slide : 25 / 74

  26. Develop ToA (2) Intended Purpose Objectives and Sub-Objectives (”Questions”) (Fictive) Experimental Frames Precision Requirements for the Behavior Characteristics WE 0201 – PKIT2 Date : November2004 Slide : 26 / 74

  27. Intended Purpose Sub-Objective 1 Sub-Objective 2 Sub-Objective n ToA SO1.1 SO1.1 SO1.1 AC2 AC3 Acceptability Criterion m AC1 WE 0201 – PKIT2 Date : November2004 Slide : 27 / 74

  28. Acquire Information • Led by the V&V Agent, performed by the V&V Team • Get overview over the data, information, and knowledge (likely to become) available • about the Simulation Executable Model • about the System of Interest • Create data, information, and knowledge repository • Identify additional sources WE 0201 – PKIT2 Date : November2004 Slide : 28 / 74

  29. Develop ToVV • Performed by the V&V Agent, supported by the V&V Team • Develop an approach to demonstrate that the Validation Criteria are passed or failed, with the data, information, and knowledge available • Identify data, information, and knowledge desirable for more efficient V&V WE 0201 – PKIT2 Date : November2004 Slide : 29 / 74

  30. Intended Purpose Sub-Objective 1 Sub-Objective 2 Sub-Objective n ToA SO1.1 SO1.1 SO1.1 AC2 AC3 Acceptability Criterion m AC1 Sub-Criterion m.1 SC m.1.1 SC m.1.2 SC m.1.3 Task ToVV Task Task WE 0201 – PKIT2 Date : November2004 Slide : 30 / 74

  31. Conduct V&V • Led by the V&V Agent, performed by the V&V Team • Choose techniques and tools • Run the technique, e.g., • Use 2-test to measure deviation • Use SME to assess structural decomposition • Record technique, referrenced information, and result as V&V item • Modify ToVV, if creation of V&V item fails WE 0201 – PKIT2 Date : November2004 Slide : 31 / 74

  32. Assess Evidence • Performed by the V&V Agent • Review documented V&V items • Assess the possibility of an erroneous result • Accept V&V item as assessed item of evidence, or reject it • Direct feedback loop WE 0201 – PKIT2 Date : November2004 Slide : 32 / 74

  33. Assess Evidence Integration • Performed by the Acceptance Agent • Assemble items of evidence according to most recent ToVV • Derive which VC are passed or failed • Intuitively, subjectivly • Based on methods for dealing with uncertainty • Assess possibility of misperception WE 0201 – PKIT2 Date : November2004 Slide : 33 / 74

  34. Intended Purpose Sub-Objective 1 Sub-Objective 2 Sub-Objective n ToA SO1.1 SO1.1 SO1.1 AC2 AC3 Acceptability Criterion m AC1 Sub-Criterion m.1 ToVV SC m.1.1 SC m.1.2 SC m.1.3 Task Task Task Item of Evidence IoE IoE WE 0201 – PKIT2 Date : November2004 Slide : 34 / 74

  35. Evaluate V&V Report • Performed by the Acceptance Agent • Review expressiveness and completeness of AC with newly gained knowledge • Weigh impact of misperception according to possibility of misperception • Make acceptance recommendation WE 0201 – PKIT2 Date : November2004 Slide : 35 / 74

  36. The REVVA Products WE 0201 – PKIT2 Date : November2004 Slide : 36 / 74

  37. Intended purpose Acceptance Recommendation Develop ToA Evaluate V&V Report The REVVA Generic Process ToA Acquire Information Model information and system knowledge V&V report Product flow Back step Develop ToVV Assess Evidence Integration Phase Assessed items of evidence ToVV Product Assess Evidence Conduct V&V V&V Items WE 0201 – PKIT2 Date : November2004 Slide : 37 / 74

  38. Target of Acceptance (ToA) Intended purpose • Precise specification of the Acceptability Criteria (coming from validity related functional requirements) • Stakeholder oriented • “What exactly needs to be assessed?” • Important elements in the ToA nodes: requirement, MoE, constraint, level of impact, decomposition argument Develop ToA ToA Acquire Information Develop ToVV WE 0201 – PKIT2 Date : November2004 Slide : 38 / 74

  39. ToA (2) • Building • Getting the requirements • Deriving MoE and constraints • Determining the level of impact • Validation of the ToA • Reuse • Using • Part of contract • Basis for assessment report • Maintaining • Due to changes in science, stakeholder changing mind, corrections in ToA found during V&V work, V&V findings may indicate more limited validity range WE 0201 – PKIT2 Date : November2004 Slide : 39 / 74

  40. Model information and system knowledge • Information used to determine the V&V strategy for demonstrating the model’s correctness and validity • Identifies all sources of information and knowledge • Important elements: • design documents, code, modeller’s perception of the real world, behaviour data (of model and sub-models), conceptual model. • Real world data (referent): data on (related or similar) systems,expert opinions Acquire Information Model information and system knowledge Develop ToVV Assess Evidence Integration Conduct V&V WE 0201 – PKIT2 Date : November2004 Slide : 40 / 74

  41. Target of Verification and Validation (ToVV) Develop ToVV Assess Evidence Integration • V&V items: evidence required to substantiate the ToA Acceptability Criteria • When necessary: further decomposition of criteria (extension of ToA) ToVV Conduct V&V • Important additional elements in the ToVV: method to use for collecting evidence, required rigor, cost estimate, aggregation instructions • Tasks WE 0201 – PKIT2 Date : November2004 Slide : 41 / 74

  42. V&V Items • A V&V item consists of: • relevant information, • the evaluation objective, • reference information, • an evaluation technique, • evaluation result. • Must be reproducible Assess Evidence Conduct V&V V&V Items WE 0201 – PKIT2 Date : November2004 Slide : 42 / 74

  43. Assessed items of evidence • Assessed items of evidence: • V&V agent assessment of the V&V result, • the acceptance or rejection of each V&V item, • probative force (depending on method or technique and reference information) Assess Evidence Integration Assessed items of evidence Assess Evidence WE 0201 – PKIT2 Date : November2004 Slide : 43 / 74

  44. V&V report • The assessed items of evidence assembled and integrated based on the ToVV • Links the assessed items of evidence to the Acceptability Criteria • Indicates if • evidence is considered sufficient to pass an acceptability criterion • V&V items for a criterion are missing or too weak, • disproving evidence was found Evaluate V&V Report V&V report Assess Evidence Integration WE 0201 – PKIT2 Date : November2004 Slide : 44 / 74

  45. Acceptance Recommendation • The final recommendation on acceptance • Based on the ToA • Residual uncertainty after V&V • from probative forces, • use level of impact to determine recommendation • The recommendation states (with consequences for M&S product’s utility): • if required evidence is missing or too weak, • if a weakness in the ToA was detected Acceptance Recommendation Evaluate V&V Report WE 0201 – PKIT2 Date : November2004 Slide : 45 / 74

  46. A VV&A project management view The "3 pillars" concept Purpose Organisation Products Objectives Process WE 0201 – PKIT2 Date : November2004 Slide : 46 / 74

  47. The M&S and VV&A roles • M&S stakeholders • M&S suppliers • VV&A agents • Experts WE 0201 – PKIT2 Date : November2004 Slide : 47 / 74

  48. Levels of independence • DV&V: • dependent V&V; V&V performed by suppliers and accepted as it is • IA: • independent assessment; V&V performed by supplier and assessed by the V&V agent • IV&V: • independentV&V; V&V performed under the control of the V&V agent WE 0201 – PKIT2 Date : November2004 Slide : 48 / 74

  49. Feasibility assumptions • High-level organisational decision • Standards of V&V documentation • Incremental accumulation of knowledge: • project vs corporate knowledge management • Process for knowledge revision WE 0201 – PKIT2 Date : November2004 Slide : 49 / 74

  50. The REVVA Techniques and support WE 0201 – PKIT2 Date : November2004 Slide : 50 / 74