Teaching an Advanced Simulation Topic Verification and Validation of Simulation Models

1. Teaching an Advanced Simulation TopicVerification and Validation of Simulation Models Stewart Robinson School of Business and Economics WSC 12, Berlin

2. Session Aim Develop an understanding of the concepts of verification, validation and confidence in a model Understanding some of the methods that can be used in V&V • Aimed at: • Specialists: undergraduate and graduate students on a simulation course; industrial training in simulation • Management students: e.g. MBA

3. Session Outline • Define V&V • V&V in the modelling life-cycle • Difficulties in performing V&V • Impossibility of validating a model! • (Techniques of V&V) • Role-play illustrating V&V

4. Verification and Validation Verification: The model design (conceptual model) has been satisfactorily converted into a computer model Validation: The model is sufficiently accurate for the purpose at hand

5. Real world (problem) Solutions/ understanding Computer model V&V in the Modelling Process Implementation Conceptual modelling Solution validation Conceptual model validation validation validation Conceptual Data validation model Experimental validation Black-box White-box Verification Experimentation Model coding

6. Conceptual Model Validation: determining that the content, assumptions and simplifications of the proposed model are sufficiently accurate for the purpose at hand. Data Validation: determining that the contextual data and the data required for model realisation and validation are sufficiently accurate for the purpose at hand. White-Box Validation: determining that the constituent parts of the computer model represent the corresponding real world elements with sufficient accuracy for the purpose at hand. Black-Box Validation: determining that the overall model represents the real world with sufficient accuracy for the purpose at hand. Experimentation Validation: determining that the experimental procedures adopted are providing results that are sufficiently accurate for the purpose at hand. Solution Validation: determining that the results obtained from the model of the proposed solution are sufficiently accurate for the purpose at hand.

7. Implications for V&V Verification and Validation needs to be performed continuously throughout the modelling process. Key point Since the modelling process is iterative in nature, so too verification and validation need to be iterated and reiterated from the point of model conception to the implementation of the results.

8. Difficulties in Performing V&V 1. There is no such thing as general validity 2. There may be no real world to compare against 3. Which real world? 4. Often the real world data are inaccurate 5. There is not enough time

9. Implications for V&V Key points It is impossible to validate a model! Model validation is a process of increasing confidence in a model – to the point where there is a willingness to use it for decision-making. When validating a model the aim is to demonstrate that the model is in fact invalid. The more tests that can be performed in which it cannot be proved that a model is invalid, the greater the confidence that can be placed in that model.

10. Natland Bank Natland Bank: Planning a New Bank Branch Question: How many ATMs are required (95% of customers queue for less than 3 minutes)? Proposed model ATM 1 Customers(Arrival rate) Queue ATM 2 Service time Simplifications: 1. No breakdowns of ATMs 2. No customers balk or leave

11. Natland Bank: Confidence Check Conceptual Model Validation High Medium Low

12. Natland Bank: Data Customer Arrivals

13. Natland Bank: Data Service Time

14. Natland Bank: Confidence Check Data Validation High Medium Low

15. Natland Bank White-Box Validation (also performed in verification) Watch the model animation: face validation Inspect the model code: correct entry of data Extreme value testing: very high service time

16. Natland Bank: Confidence Check White-Box Validation High Medium Low

17. I O R R I O S S Black-Box Validation Comparison with the real system Real system Simulation model H : If I =I  O then O 0 S R S R

18. I O A A I O S S Black-Box Validation Comparison with other models Alternative model Simulation model H : If I =I  O then O 0 S A S A

19. Black-Box Validation Comparison with other models Extreme approach is to make the simulation deterministic Accuracy derived from complexity  Alternative model Simulation

20. Natland Bank Black-Box Validation: Comparison with Another (Simpler) Model Deterministic model comparison: Arrival rate = 100/hour 2 tellers: service time = 1 minute Customers served/hour = 60 x 2 = 120 Expected teller utilisation = 100/120 = 83.3%

21. Natland Bank Black-Box Validation: Comparison with Another (Simpler) Model Full model comparison: Mean arrival rate = 157.14/hour 2 tellers: mean service time = 40.45 seconds Mean customers served/hour = 89.00 x 2 = 178.00 Expected teller utilisation = 157.14/178.00 = 88.28%

22. Natland Bank: Confidence Check Black-Box Validation High Medium Low

23. Will you use my model to determine the number of ATMs in the bank?