1. Measurements & Validation R G Stamp Head of Department of ATM Research National Air Traffic Services Ltd, UK

2. What is Validation? • EATMS Validation Strategy (EVAS) “The process through which a desired level of confidence in the ability of a deliverable to operate in a real-life environment may be demonstrated against a pre-defined level of functionality, operability and performance”

3. What is Validation? • EATMS Validation Strategy (EVAS) “The process through which a desired level of confidence in the ability of a deliverable to operate in a real-life environment may be demonstrated against a pre-defined level of functionality, operability and performance”

4. What is Validation? • EATMS Validation Strategy (EVAS) “The process through which a desired level of confidence in the ability of a deliverable to operate in a real-life environment may be demonstrated against a pre-defined level of functionality, operability and performance”

5. What is Validation? • EATMS Validation Strategy (EVAS) “The process through which a desired level of confidence in the ability of a deliverable to operate in a real-life environment may be demonstrated against a pre-defined level of functionality, operability and performance”

6. What is Validation? • EATMS Validation Strategy (EVAS) “The process through which a desired level of confidence in the ability of a deliverable to operate in a real-life environment may be demonstrated against a pre-defined level of functionality, operability and performance” • Validation builds confidence that the system is fit for purpose

7. How to Validate

8. How to Validate Analytic modelling Fast-time Field test simulation Shadow-mode Small-scale trial real-time simulation Operational trial Operations Large-scale real-time simulation

9. Where to Validate Small-scale real-time simulation

10. Where to Validate NATS DLR CENA NLR EEC real-life environment Small-scale real-time simulation

11. Validation Methodology • Consistent Application of Validation Methodology Across All Sites

12. Validation Methodology • Consistent Application of Validation Methodology Across All Sites

13. Validation Methodology • Consistent Application of Validation Methodology Across All Sites

14. Validation Methodology • Consistent Application of Validation Methodology Across All Sites

15. Validation Methodology • Consistent Application of Validation Methodology Across All Sites

16. Validation Methodology • Consistent Application of Validation Methodology Across All Sites pre-defined levelof functionality, operability and performance

17. Measures of Merit • MoM’s Chosen to Test Hypotheses • Controller Workload • Airspace Capacity • Quality of Service • Usability • Safety • Measurements Taken: • e.g. Workload: ISA, TLX, R/T, ….. • Objective; Subjective; Observed

18. Analysis • Descriptive Statistics

19. Analysis • Descriptive Statistics • Inferential Statistics • significant at 5% confidence level

20. Analysis • Descriptive Statistics • Inferential Statistics • significant at 5% confidence level • Backed up by Controller Subjective Comments • “I can still handle the aircraft while doing the crossword!”

21. Analysis • Descriptive Statistics • Inferential Statistics • significant at 5% confidence level • Backed up by Controller Subjective Comments • “I can still handle the aircraft while doing the crossword!” desired level of confidence

22. Effectiveness of PHARE Validation • Core Validation Group with Representation from all Participating Sites • Common Methodology Crucial • Methodology Improved with Experience • Validation Conducted by Individual Sites • Lessons Re-learnt &/or Lost • Effectiveness Diminished • Validation Should be Applied Throughout Project - Not Just at the End

23. Summary • Validation Methodology Major Contributor to Credibility of PHARE • Forms the Backbone for much of the Validation Activities within European ATM R&D

24. Summary • Validation Methodology Major Contributor to Credibility of PHARE • Forms the Backbone for much of the Validation Activities within European ATM R&D