What is the Cost of Determinism?

1. What is the Cost of Determinism? Cedomir Segulja, Tarek S. Abdelrahman University of Toronto

2. Source: [Intel] Source: [Youtube]

3. Non-Determinism • Same program + same input ≠ same output • This is bad for … • Testing • Too many interleaving to test • Debugging • Hard to debug when behavior is not repeatable • Selling • CAD tools users expect each run to produce the same circuit

4. Determinism • Is good, but costly • What is the fundamental cost of determinism? • What is this cost across various execution environments? • “Determinism in the field” 1 2 Source: [Bergan et al. 2011] and the respective papers *Only to show that determinism comes at a cost, and not to be used for a direct comparison (different features, benchmarks, # threads, etc.)

5. What is Determinism? • Property that requires observing the same output whenever program runs with the same input • SyncOrder determinism [Lu and Scott 11] • Require the same program result and same order of synchronization • More flexible than internal determinism • Still greatly eases testing [Cui et al. 13] • We assume data-race-freedom • Determinism during debugging is needed • But the cost of determinism matters the most in production • All data races are bugs [Boehm 2008, S. Adve 2010, Marino et al. 2010, Lucia et al. 2010, …] • Data races in general do not help performance [Boehm 12] • External • SyncOrder • Internal

6. What is the impact of enforcing a fixed synchronization order on program execution time?

7. Schedule-Record-Replay Framework 1 2 application application schedule thread1 thread2 scheduler replayer serial hybrid round-robin perturber idle small perturbations architectures dynamic-A dynamic-S NUMA background processes recorder DVFS

8. Replayer • Force threads to wait only when absolutely necessary under the schedule • And do so with as little overhead as possible Non-deterministic execution vs. Non-deterministic execution with the replayer’s overhead

9. Schedules • When does a thread pass its turn? • At the end – serial • After each synchronization operation – round-robin • After each instruction/store – dynamic-A/dynamic-S • After N instructions – hybrid • N = 100,000 • No “reduced serial mode”

10. Platform • 8-core Xeon E5-2660 • 24 SPLASH-2 and PARSEC benchmarks, 8 threads • Deterministic slowdown • Data races in general do not help performance [Boehm 12] • 15 benchmarks had races, performance degradation in only 3 • barnes (11%), radiosity (5%), raytrace_parsec (8%)

12. For this set of benchmarks and our platform, and implementation overhead set aside, the fundamental cost of determinism is small.

13. What is the performance cost of insisting on the same schedule across different environments?

14. Schedule-Record-Perturb-Replay Framework 1 2 application application schedule thread1 thread2 scheduler replayer serial hybrid round-robin perturber idle small perturbations architectures dynamic-A dynamic-S NUMA background processes recorder DVFS

15. Perturber • Small perturbations (context switches, thread migrations, page faults) • Simulate first order effects by inserting small delays (μs and ms) • Background processes • Spawn additional threads and control their work to sleep ratio • Dynamic voltage and frequency scaling (DVFS) • Use Linux’s cpufreq system to explore different DVFS policies • Non-uniform memory access (NUMA) • Spread threads over two NUMA nodes • Asymmetric architectures • Use DVFS to create asymmetry [Shelepov et al. 2009]

16. Metric • Deterministic slowdown • Same conditions during both runs, for example

18. Insisting on the same schedule in the presence of skewed conditions can slow down execution by a factor of almost 2x.

19. Conclusions • Employed the schedule-record-replay framework to divorce implementation overhead from the fundamental cost of enforcing deterministic execution • Fundamental cost of determinism is small (4% on avg., 33 % max.) • There is room for lowering overheads in current deterministic systems • Measured this fundamental cost across a range of execution environments • The cost of raises to almost 2x when threads face skewed conditions • Do we need a more relaxed definition of determinism? • Quantified various sources of non-determinism • Deterministic logical clocks are not deterministic (not only due to the performance counters imperfections [Weaver et al. 2013])

20. Thank you!