Dynamic Program Analysis for Real Deadlock Detection in Multi-Threaded Applications

1. A Randomized Dynamic Program Analysis for Detecting Real Deadlocks KoushikSen CS 265

2. What is a Deadlock? • An unintended condition in a shared-memory, multi-threaded program in which: • a set of threads blocks forever • because each thread in the set waits to acquire alock being held by another thread in the set • This work: ignore other causes (e.g., wait/notify) • Example // thread t2 sync (l2) { sync (l1) { … } } l1 // thread t1 sync (l1) { sync (l2) { … } } t1 t2 l2

3. Motivation • Today’s concurrent programs are rife with deadlocks • 6,500/198,000 (~ 3%) of bug reports in Sun’s bug database at http://bugs.sun.com are deadlocks • Deadlocks are difficult to detect • Usually triggered non-deterministically, on specific thread schedules • Fail-stop behavior not guaranteed (some threads may be deadlocked while others continue to run) • Fixing other concurrency bugs like races can introduce new deadlocks • Our past experience with reporting races: developers often ask for deadlock checker

4. Our Goal • Build a deadlock detection tool that • Scales to very large programs • Finds deadlocks quickly • Has no false positives • Shows a real execution exhibiting a deadlock

5. Related Work: Program Analysis • Static program analysis (e.g., Engler & Ashcraft SOSP’03; Williams-Thies-Ernst ECOOP’05, Naik et al. ICSE’09) + Examines all possible program behavior • Often reports many false positives • Type systems (e.g., Boyapati-Lee-Rinard OOPSLA’02) - Annotation burden often significant • Model checking (e.g., SPIN, Verisoft, Java Pathfinder) - Does not currently scale beyond few KLOC - Not “directed” • Dynamic program analysis (e.g. GoodlockHavelund et al., Agrawal et al.) + Usually reports lesser false positives - Has false negatives

6. Related Work: Testing • Testing + Easy to deploy + Scales to large programs + No false positives • Same schedule gets tested many times • No effort to control thread scheduler - Often subtle thread schedules that exhibit deadlocks are not tested

7. In Summary • Static and dynamic program analyses have false positives • Testing is simple • No false positives • But, may miss subtle thread schedules that result in deadlocks

8. In Summary • Static and dynamic program analyses have false positives • Testing is simple • No false positives • But, may miss subtle thread schedules that result in deadlocks • Can we leverage program analysis to make testing quickly find real deadlocks?

9. Our Approach • Active Testing • Phase 1: Use imprecise static or dynamic program analysis to find “abstract” states where a potential deadlock can happen • Phase 2: “Direct” testing (or model checking) based on the “abstract” states obtained from phase 1

10. Example Thread2 foo(o2,o1,false) void foo(Object l1, Object l2, boolean flag) { if(flag) { // Long running computations s1: f1(); s2: f2(); } s3: synchronized(l1){ s4: synchronized(l2){ } } } Thread1 foo(o1,o2,true)

11. Testing Thread 1 Thread 2 Thread2 foo(o2,o1,false) Lock(o2) f1() Lock(o1) void foo(Object l1, Object l2, boolean flag) { if(flag) { // Long running computations s1: f1(); s2: f2(); } s3: synchronized(l1){ s4: synchronized(l2){ } } } Unlock(o1) f2() Unlock(o2) Thread1 foo(o1,o2,true)

12. Testing Thread 1 Thread 2 Thread2 foo(o2,o1,false) Lock(o2) f1() Lock(o1) void foo(Object l1, Object l2, boolean flag) { if(flag) { // Long running computations s1: f1(); s2: f2(); } s3: synchronized(l1){ s4: synchronized(l2){ } } } Unlock(o1) f2() Unlock(o2) Lock(o1) Lock(o2) Unlock(o2) Unlock(o1) Thread1 foo(o1,o2,true)

13. Testing Thread 1 Thread 2 Thread2 foo(o2,o1,false) Lock(o2) f1() Lock(o1) void foo(Object l1, Object l2, boolean flag) { if(flag) { // Long running computations s1: f1(); s2: f2(); } s3: synchronized(l1){ s4: synchronized(l2){ } } } Unlock(o1) f2() No deadlock detected Unlock(o2) Lock(o1) Lock(o2) Unlock(o2) Unlock(o1) Thread1 foo(o1,o2,true)

14. Deadlock Directed Testing Thread 1 Thread 2 Thread2 foo(o2,o1,false) Lock(o2) f1() Lock(o1) void foo(Object l1, Object l2, boolean flag) { if(flag) { // Long running computations f1(); f2(); } synchronized(l1){ synchronized(l2){ } } } Paused f2() Thread1 foo(o1,o2,true)

15. Deadlock Directed Testing Thread 1 Thread 2 Thread2 foo(o2,o1,false) Lock(o2) f1() Lock(o1) void foo(Object l1, Object l2, boolean flag) { if(flag) { // Long running computations f1(); f2(); } synchronized(l1){ synchronized(l2){ } } } Paused f2() Lock(o1) Lock(o2) Paused Thread1 foo(o1,o2,true)

16. Deadlock Directed Testing Thread 1 Thread 2 Thread2 foo(o2,o1,false) Lock(o2) f1() void foo(Object l1, Object l2, boolean flag) { if(flag) { // Long running computations f1(); f2(); } synchronized(l1){ synchronized(l2){ } } } f2() Lock(o1) Lock(o1) Lock(o2) Paused Paused Thread1 foo(o1,o2,true)

17. Deadlock Directed Testing Thread 1 Thread 2 Thread2 foo(o2,o1,false) Lock(o2) f1() void foo(Object l1, Object l2, boolean flag) { if(flag) { // Long running computations f1(); f2(); } synchronized(l1){ synchronized(l2){ } } } f2() Deadlock detected ! Lock(o1) Lock(o1) Lock(o2) Paused Paused Thread1 foo(o1,o2,true)

18. Preempting threads • How do we know where to pause a thread ?

19. Preempting threads • How do we know where to pause a thread ? • Use existing static or dynamic analyses to find potential deadlock cycles • Note that these analyses may report false deadlock cycles • Use “information” recorded for a deadlock cycle to decide where to pause a thread • We use a modified version of the Goodlock algorithm (iGoodlock) [Havelund et al, Agarwal et al]

20. iGoodlock • We consider dynamic instances of the following statements • c: Acquire(t,l) • c: Release(t,l) • c: Call(t,m) • c: Release(t,m) • c: o = new (t,o’,T)

21. iGoodlock • We consider dynamic instances of the following statements • c: Acquire(t,l) • c: Release(t,l) • c: Call(t,m) • c: Release(t,m) • c: o = new (t,o’,T)

22. Lock Dependency Relation • D is a subset of (T X 2L X L X C*) • (t,L,l,C) is in the lock dependency relation if

23. Lock Dependency Relation • D is a subset of (T X 2L X L X C*) • (t,L,l,C) is in the lock dependency relation if • thread t acquires lock l while holding the locks in the set L, and C is the sequence of labels of Acquire statements that were executed by t to acquire the locks in L ∪ {l}.

24. Lock Dependency Chain (t1,L1,l1,C1), (t2,L2,l2,C2), …, (tm,Lm,lm,Cm) is a lock dependency chain if

25. Lock Dependency Chain (t1,L1,l1,C1), (t2,L2,l2,C2), …, (tm,Lm,lm,Cm) is a lock dependency chain if • for all distinct i, j∈[1, m], ti ≠ tj • for all distinct i, j∈[1, m], li ≠ lj • for all i ∈ [1,m − 1], li ∈ Li+1, • for all distinct i,j ∈ [1,m], Li ∩ Lj = ∅.

26. Lock Dependency Chain (t1,L1,l1,C1), (t2,L2,l2,C2), …, (tm,Lm,lm,Cm) is a lock dependency chain if • for all distinct i, j∈[1, m], ti ≠ tj • for all distinct i, j∈[1, m], li ≠ lj • for all i ∈ [1,m − 1], li ∈ Li+1, • for all distinct i,j ∈ [1,m], Li ∩ Lj = ∅. • A chain is a deadlock if

27. Lock Dependency Chain (t1,L1,l1,C1), (t2,L2,l2,C2), …, (tm,Lm,lm,Cm) is a lock dependency chain if • for all distinct i, j∈[1, m], ti ≠ tj • for all distinct i, j∈[1, m], li ≠ lj • for all i ∈ [1,m − 1], li ∈ Li+1, • for all distinct i,j ∈ [1,m], Li ∩ Lj = ∅. • A chain is a deadlock if • lm∈ L1

28. Compute D on this trace Thread 1 Thread 2 Thread2 foo(o2,o1,false) f1() Acquire(o2) void foo(Object l1, Object l2, boolean flag) { if(flag) { // Long running computations s1: f1(); s2: f2(); } s3: synchronized(l1){ s4: synchronized(l2){ } } } Acquire(o1) Release(o1) f2() Release(o2) Acquire(o1) Acquire(o2) Release(o2) Release(o1) Thread1 foo(o1,o2,true)

29. Computing lock dependency chains • Compute Dk where D1 = D • for each d ∈D and τ ∈ Di • if d, τ is a lock dependency chain and a deadlock cycle then report a potential deadlock • if d, τ is a lock dependency chain and not a deadlock cycle • add d, τ to Di+1

30. Preempting threads • T1, o1, T2, o2 are runtime objects • How do we identify them across executions ?

31. Preempting threads • T1, o1, T2, o2 are runtime objects • How do we identify them across executions ? • Runtime addresses?

32. Preempting threads • T1, o1, T2, o2 are runtime objects • How do we identify them across executions ? • Runtime addresses? • No, they change across executions !

33. Preempting threads • T1, o1, T2, o2 are runtime objects • How do we identify them across executions ? • Runtime addresses? • No, they change across executions ! • Allocation sites where they were created ? • Yes, but different objects created at the same allocation site will all be treated as the same object

34. Preempting threads • T1, o1, T2, o2 are runtime objects • How do we identify them across executions ? • Runtime addresses? • No, they change across executions ! • Allocation sites where they were created ? • Yes, but different objects created at the same allocation site will all be treated as the same object • Can we do better?

35. Abstractions • Based on k-object sensitivity [Milanova et al] • Based on light-weight execution indexing [Xin et al]

36. Abstractions • 1 class C { • 2 void foo() { • 3 A a = new A(); • 4 a.bar(); • } • 6 } • 7 class A { • 8 void bar() { • 9 for(int j = 0; j < 5; j++) • 10 Object l = new Object(); • 11 } • 12 } • 13 main() { • 14 C c = new C(); • 15 for(inti = 0; i < 5; i++) • 16 c.foo(); • 17 }

37. Abstractions • 1 class C { • 2 void foo() { • 3 A a = new A(); • 4 a.bar(); • } • 6 } • 7 class A { • 8 void bar() { • 9 for(int j = 0; j < 5; j++) • 10 Object l = new Object(); • 11 } • 12 } • Abstraction of last • object created (k- • object sensitivity) : [10,3, 14] • Abstraction of last • object created • (execution indexing) : • [(10,5),(4,1),(16,5)] • 13 main() { • 14 C c = new C(); • 15 for(inti = 0; i < 5; i++) • 16 c.foo(); • 17 }

38. Implementation • Implemented in a prototype tool called DEADLOCKFUZZER for Java • Part of the CALFUZZER framework • Active testing [Sen PLDI 08, Park et al. FSE 08, Joshi et al. CAV 09] of concurrent programs • Instrument Java bytecode to • observe events during execution • control scheduler

39. Evaluation

40. Evaluation

41. Evaluation

42. Limitations • If DEADLOCKFUZZER does not reproduce a deadlock, we cannot say if the deadlock is a false positive or not • Jigsaw • Deadlocks reported by iGoodlock : 283 • Deadlocks reproduced by DEADLOCKFUZZER : 29 • Deadlocks confirmed as false positives : 18 • Rest : 236

43. Limitations • If DEADLOCKFUZZER does not reproduce a deadlock, we cannot say if the deadlock is a false positive or not • Jigsaw • Deadlocks reported by iGoodlock : 283 • Deadlocks reproduced by DEADLOCKFUZZER : 29 • Deadlocks confirmed as false positives : 18 • Rest : 236 Cannot say if they are real deadlocks or false warnings

44. Conclusion • DEADLOCKFUZZER is a practical deadlock detection tool that • Scales to large programs • Finds deadlock quickly • Finds real deadlocks • Complements static and dynamic analyses • Automatically confirms some of the real deadlocks