Dynamic Runtime Testing for Cycle-Accurate Simulators

1. Dynamic Runtime Testing for Cycle-Accurate Simulators Saša Tomić, Adrián Cristal,OsmanUnsal, Mateo Valero Barcelona Supercomputing Center (BSC) UniversitatPolitecnica de Catalunya (UPC)

2. Can we trust the simulator-based evaluations? • Typical simulator evaluation: Make a simulator REPEAT { Debug Simulate } UNTIL: the results make sense (intuition!) • Discard and ignore the failed simulations • Are there any bugs left?

3. Verifying the simulators • Verification is important! • Industry puts significant resources • Testing and Verification 50-70% of the costs • Mission critical application even 90% of the costs • Academia puts less resources • Why do we have bugs? • Simulators are complex • Proposed extensions are often complex • The extensions may uncover existing bugs

4. Simulator bugs • Timing bugs • Incorrect estimation of the execution time • Simulation terminates without obvious errors • Needs other types of testing and verification • Functional bugs • Incorrect implementation of functional units • Simulations may or may not terminate correctly • Our target

5. Outline • Examples of functional bugs • An overview of the Dynamic Simulator Testing methodology • Use Cases of Dynamic Simulator Testing • Performance evaluation • Conclusions

6. Example: a bug in the cache coherence protocol Bug: X should be X = 10+20 = 30 X += 10 X += 20 X=0 time processor 1 processor 2 Proc 1 X+=10 Proc 2 X+=20 X=0 X=10 X=0 X=20 simulator of multi-level coherent caches X=30 X=0 X=20 shared memory

7. Example: a bug in the HTM X = 0; Atomic { X += 10; } X += 10 processor X=0 X=10 HTM simulator Bug: not committed X = 10 X=0 nothing? shared memory

8. Detecting functional bugs • The functionality of the simulators is often simple • Can be emulated with simple emulators • The emulators can be fast and stable • Can we take an advantage of the emulators?

9. Dynamic Testing Methodology • Add a simple, no-timing emulator • Execute each operation in the simulator and then in the emulator • Compare the executions • We compared only the memory accesses • The execution must be identical during entire simulation

10. An overview of dynamic testing input output output input timing simulator simpleno-timing emulator • Use the same input • Compare the outputs • Repeat for every operation! input output output input

11. Dynamic testing a cache coherence protocol Check failed: should be X=10 X += 10 X += 20 processor 1 processor 2 X=20 X=0 X=10 X=0 X=10 X=0 X=10 output input output input output input timing simulator of multi-level coherent caches STL map X=0 X=10 input output output input X=0 X=0 X=20 shared memory

12. Dynamic testing of an HTM processor TX_Begin; X += 10; TX_Commit; X=10 X=0 X=0 X=10 input output output input timing simulator of an HTM STL map per TX X=10 X=0 input output output input Check failed: should commit X=10 X=0 X=??? X=10 X=0 shared memory

13. Other Use Cases • Out-Of-Order or pipelined processor • With a processor emulator, e.g., QEMU • Complex memory hierarchy • With an STL map • Incoherent multilevel memory hierarchy • W/ multiple STL maps, one per memory hierarchy • System-On-Chip, Routing Protocols, etc. • Simple emulators of the functionalities

14. Performance Evaluation • Implemented on 4 HTMs with lazy and eager version management • Implemented for a directory-based cache-coherence protocol • Baseline: M5 full-system simulator

15. Performance evaluationOS booting

16. Performance evaluationapplications

17. Our experience with Dynamic Testing • Reduced the time spent on writing tests • Faster debugging • Detects most bugs “in minutes” • Eliminating a bug takes tens of minutes instead of hours/days/weeks/… • Shortened the total simulator development from12-18 months to 3-4 months

18. Conclusions • Presented the Dynamic Simulator Testing • Detects the functional bugs in Cycle-Accurate Simulators • Modest reduction of simulator performance

19. Thanks! SashaTomić sasa.tomic@bsc.es Dynamic Runtime Testing for Cycle-AccurateSimulators

20. Cache/HTM emulator implementation address line data STL map (dictionary) address line data address line data address line data . . . . . .