Causal Models for Performance Analysis of Computer Systems

1. Causal Models for Performance Analysis of Computer Systems Jan Lemeire TELE lab May 24th 2006 Causal Performance Models

2. Statistics/Causality • Philosophy • Machine Learning • Performance Modeling Causal Performance Models

3. What can be learnt about the world from observations? • We have to look for regularities • & model them Causal Performance Models

4. MDL-approach to Learning • Occam’s Razor “Among equivalent models choose the simplest one.” • Minimum Description Length (MDL) “Select model that describes data with minimal #bits.” model = shortest program that outputs data length of program = Kolmogorov Complexity Learning = finding regularities = compression Causal Performance Models

5. Randomness vs. Regularity • 0110001101011010101 random string=incompressible=maximal information • 010101010101010101 regularity of repetitionallows compression Separation by the Two-part code Causal Performance Models

6. Ex.: Numberplate Recognition Noise fiercely hinders recognition algorithms Shortest program? + Separation! Two-part code: ‘MWV735’ + letter style + + + drop size variance + drop frequency + random information Causal Performance Models

7. Conclusions Part I • Extensions to Shannon (information content of a message): • Algorithmic Information Theory • & Kolmogorov Complexity • Fundamental! • But not practical… • No algorithm can exist that outputs the shortest program and Kolmogorov Complexity of an object. Causal Performance Models

8. II Model of Multivariate Systems • Variables • Experimental data Probabilistic model of joint distribution with minimal description length? Causal Performance Models

9. 1 variable • Average code length = Shannon entropy of P(x) • Multiple variables • With help of other, P(xi|x1…xi-1) (CPD) • Factorization • Mutual information decreases entropy of variable Causal Performance Models

10. Conditional Independence • Two variables A and B are independent if: • P(A|B)=P(A) • Qualitative property: • Quality of my speech is independent of chance of rain today • P(rain|speech)=P(rain) ? Causal Performance Models

11. A.Conditional independencies • Reduction of factorization complexity • Bayesian Network • Minimal factorization = MDL • B.Faithfulness Joint Distribution Directed Acyclic Graph Conditional independencies  d-separation Theorem: if faithful graph exists, it is the minimal factorization. Causal Performance Models

12. C.Causal Interpretation • Definition through interventions, otherwise only correlation • V-structure <> Markov Chain • Motivation: Causal models describe all relational regularities in a canonical form Causal Performance Models

13. Reductionism • Causality = reductionism • Building block = P(Xi|parentsi) • Unique, minimal, independent • Whole theory based on it, like asymmetry of causality • Intervention • = change of block Causal Performance Models

14. But… Engineers use causal models all the time! Causal Performance Models

15. Causal model is MDL of joint distribution if Incompressible Incompressible (random distribution) Contribution 1: MDL interpretation of causal models Causal Performance Models

16. Learning Algorithms • Construct causal model from experimental data • Directly related variables cannot become independent by conditioning on other variables • Undirected graph • V-structures determine orientation • Directed graph Causal Performance Models

17. Part III: When do causal models become incorrect? • By other regularities! Causal Performance Models

18. A. Lower-level regularities • Compression of the distributions Causal Performance Models

19. B. Better description form • Pattern • in figure Causal model? • Other models are better • Why? Graph is compressible & blocks (CPDs) are related Causal Performance Models

20. C. Interfere with independencies X and Y independent by cancellation of X→U → Y and X → V → Y • dependency of both paths • = regularity Causal Performance Models

21. Deterministic relations • Y=f(X1, X2) • Y becomes unexpectedly independent from Z conditioned on X1 and X2 Solution: augmented model - add regularity to model - adapt inference algorithms • Learning algorithm: • variables possibly contain equivalent information • Choose simplest relation Causal Performance Models

22. Moral • Occam’s Razor works • Describe all regularities Contribution 2: Faithful representation of deterministic relations Causal Performance Models

23. Part IV: Performance Analysis • High-Performance computing parallel system 1 processor Performance Questions: • Performance prediction • Parameter-dependency? • Reasons of bad performance? • System-dependency? • Effect of Optimizations? Causal Performance Models

24. Causal models (cf. COMO lab) • Representation form • Close to reality • Learning algorithms • TETRAD tool Causal Performance Models

25. No magic bullet!! Complexity of real data • Mix of continuous and discrete variables • Non-linear relations • Deterministic relations • Context-specific variables and relations Frederik Verbist Joris Borms Causal Performance Models

26. Causal Performance Model • Computation time of a quicksort algorithm Contribution 3: Formal definition of causal performance models Causal Performance Models

27. Integrated in statistical analysis • Statistical characteristics • Regression analysis Iterative process • Perform additional experiments • Extract additional characteristics • Indicate exceptions • Analyze the divergences of the data points with the current hypotheses Contribution 4: Performance modeling tool (EPDA) Causal Performance Models

28. Results so far 1. Learning of non-trivial models Iterative algorithm for solving differential equation in parallel (Aztec benchmark Library) Now: expert can input background knowledge Causal Performance Models

29. 2. Point-to-point communications flight time = latency + message size/bandwidth?? Causal Performance Models

30. 3. Explanations for outliers 4. Effects of optimizations … Causal Performance Models

31. Conclusions • Theoretical foundations for performance models • Practical use: a lot of tuning • integration, tests, extensions, … • Occam’s Razor works • Choice of simplest model • models close to ‘reality’ • but what is reality? • Atomic description of regularities that we observe? Papers, references and demos: http://parallel.vub.ac.be Causal Performance Models