Yogi – Part 3

1. Yogi – Part 3 Engineering the tool and parallelization

2. Windows Device Drivers do { //get the write lock KeAcquireSpinLock(&devExt->writeListLock); nPacketsOld = nPackets; request = devExt->WriteListHeadVa; if(request && request->status){ devExt->WriteListHeadVa = request->Next; KeReleaseSpinLock(&devExt->writeListLock); irp = request->irp; if(request->status > 0){ irp->IoStatus.Status = STATUS_SUCCESS; irp->IoStatus.Information = request->Status; } else{ irp->IoStatus.Status = STATUS_UNSUCCESSFUL; irp->IoStatus.Information = request->Status; } SmartDevFreeBlock(request); IoCompleteRequest(irp, IO_NO_INCREMENT); nPackets++; } } while (nPackets != nPacketsOld); KeReleaseSpinLock(&devExt->writeListLock); Unlocked U L Locked U L Error

3. Read for understanding Drive testing tools API Usage Rules (SLIC) New API rules Defects Software Model Checking 100% path coverage Rules Static Driver Verifier Development Testing Source Code

4. Read for understanding Drive testing tools API Usage Rules (SLIC) New API rules Defects Software Model Checking 100% path coverage Rules Static Driver Verifier Development Testing Source Code

5. Architecture of Yogi C Program slamcl Slam.li database SLIC property slicc Initial functionsummaries Instrumented Slam.li database Alias and mod-ref information li2yir polymorphic region graphs Yogi IR (yir) YSim YAbsMan Proof that program satisfies property Test case that exposes a bug Z3 theorem prover

6. Implementation • ~6K lines of Ocaml • Z3theorem prover • Integrated with SDVfor Windows • Engineering effort: 3 person years • F# version available with SDVRP • http://research.microsoft.com/yogi

7. Optimizations • Share our experiences in making Yogi robust, scalable and industrial strength • Several of the implemented optimizations are folklore • Very difficult to design tools that are bug free evaluating optimizations is hard! • Our empirical evaluation gives tool builders information about what gains can be realistically expected from optimizations • Details in “An empirical study of optimizations in Yogi, ICSE ’10“ • Vanilla implementation of algorithms: • (flpydisk, CancelSpinLock) took 2 hours • Algorithms + engineering + optimizations: • (flpydisk, CancelSpinLock) took less than 1 second!

8. Optimizations • Initial abstraction from property predicates • Relevance heuristics for predicate abstraction • Suitable predicates (SP) • Control dependence predicates (CD) • Summaries for procedures • Thresholds for tests

9. Evaluation setup • Benchmarks: • 30 WDM drivers and 83 properties (2490 runs) • Anecdotal belief: most bugs in the tools are usually caught with this test suite • Presentation methodology: • Group optimizations logically such that related optimizations are in the same group • Total time taken, total number of defects found for every possible choice of enabling/disabling each optimization in the group

10. Initial abstraction state { enum {Locked = 0, Unlocked = 1} state = Unlocked; } KeAcquireCancelSpinlock.Entry { if (state != Locked) { state = Locked; } else abort; } KeReleaseCancelSpinlock.Entry { if (state == Locked) { state = Unlocked; } else abort; } 0 0 0 1 1 1

11. Empirical results 16%

12. Relevance heuristics (SP) Irrelevant? • Avoid irrelevant conjuncts A B C D A B C C D

13. Relevance heuristics (CD) • Abstract assume statements that are not potentially relevant by skip statements • If Yogi proves that the program satisfies property, we are done. • Otherwise, validate the error trace and refine the abstraction by putting back assume statements, if the error trace is spurious

14. Example: SP heuristic int x; void foo() { bool protect = true; … if (x > 0) protect = false; … if (protect) KeAcquireCancelSpinLock(); for (i = 0; i < 1000; i++) { a[i] = readByte(i); } if (protect) KeReleaseCancelSpinLock(); } A B C D A B C C D

15. Example: SP heuristic int x; void foo() { bool protect = true; … if (x > 0) protect = false; … if (protect) KeAcquireCancelSpinLock(); for (i = 0; i < 1000; i++) { a[i] = readByte(i); } if (protect) KeReleaseCancelSpinLock(); } A B C D A B C C D

16. Example: CD heuristic int x; void foo() { bool protect = true; … if (x > 0) protect = false; … if (protect) KeAcquireCancelSpinLock(); for (i = 0; i < 1000; i++) { a[i] = readByte(i); } if (protect) KeReleaseCancelSpinLock(); }

17. Empirical results 10%

18. Empirical results 16%

19. Empirical results 25%

20. Interprocedural analysis • Yogi performs a compositional analysis • : Is it possible to execute starting from state and reach state ? • Global modification analysis • Compositional May-Must analysis (POPL 2010)

21. Example A B C D A B C C foo(…) D

22. Empirical results 32%

23. Empirical results 28%

24. Empirical results 42%

25. Testing • Yogi relies on tests for “cheap” reachability • Long tests • avoiding several potential reachability queries • results in too many states and thus memory consumption • Test thresholds: time vs. space tradeoff

26. Empirical evaluation

27. Modeling the environment if (DestinationString) { DestinationString->Buffer = SourceString; // DestinationString->Length should be set to the // length of SourceString. The line below is missing // from the original stub SDV function DestinationString->Length = strlen(SourceString); } if (SourceString== NULL) { DestinationString->Length = 0; DestinationString->MaximumLength = 0; }

28. Summary • Described optimizations implemented in Yogi • Evaluated optimizations on the WDM test suite • Empirical data used to decide which optimizations to include in Yogi • We believe that this detailed empirical study of optimizations will enable tool builders to decide which optimizations to include and how to engineer their tools • Download:http://research.microsoft.com/yogi

29. Bolt • Bolt: a generic framework that uses MapReduce style parallelism to scale top-down analysis Intraprocedural analysis Bolt Queries Yogi Querying summaries sumDB

30. Interprocedural analysis Y not-may summary: perform refinement must summary : generate test and extend frontier can we parallelize this algorithm?

31. Main idea intmain (int y){ if(*) x = foo(y); else x = bar(y); if (x < 0) error(); } Analyze queries in parallel!

32. Is it worth it?

33. Introducing Bolt Intraprocedural analysis Bolt Queries Yogi Querying summaries sumDB

34. Modification to intraprocedural Yogi Q1 Done (add summary to sumDB) Q1 Q1 Yogi Blocked (add new sub-queries) Q1 Ready (may add new sub-queries) sumDB

35. Lifecycle of a query Reduce blocked ready Reduce done

36. In a nutshell … • Bolt uses a MapReduce-style parallelization: • Map stage: Run each ready query on a separate instance of Yogi • Reduce stage: Resolve dependencies between queries • Terminate when the main query has an answer in sumDB

37. Example void main (inti) { if (i> 0) x = foo(i); else if (j > -10) x = bar(i); else x = baz(j); y = x + 5; if (y <= -5) error(); } Yogi Map sumDB initially empty

38. Example void main (inti) { if (i> 0) x = foo(i); else if (j > -10) x = bar(i); else x = baz(j); y = x + 5; if (y <= -5) error(); } Yogi Map Reduce sumDB

39. Example void main (inti) { if (i> 0) x = foo(i); else if (j > -10) x = bar(i); else x = baz(j); y = x + 5; if (y <= -5) error(); } Map Yogi Yogi Yogi sumDB

40. Example void main (inti) { if (i> 0) x = foo(i); else if (j > -10) x = bar(i); else x = baz(j); y = x + 5; if (y <= -5) error(); } Map Yogi Yogi Yogi Reduce sumDB

41. Implementation let bolt () = while (initQuery.isNotDone()) do worklist := Async.AsParallel [for q in worklist -> async{return yogi q}]; reduce (); done;

42. Evaluation • Benchmarks • 45 Windows device drivers and 150 safety properties • Picked 50 driver+property pairs taking 1000+ seconds on sequential version • Experimental setup • 8 processor cores • Artificial throttle: number of threads per Map stage

43. Evaluation • Cumulative results on 50 checks, 8-cores, 64 threads

44. In-depth analysis Driver:toastmon(25KLoC)

45. In-depth analysis Driver:toastmon(25KLoC)

46. In-depth analysis 2 Driver:toastmon(25KLoC) Property: PnpIrpCompletion

47. In-depth analysis 4 Driver:toastmon(25KLoC) Property: PnpIrpCompletion

48. In-depth analysis 8 Driver:toastmon(25KLoC) Property: PnpIrpCompletion

49. In-depth analysis 8 Driver:toastmon(25KLoC) Property: PnpIrpCompletion

50. In-depth analysis 16 Driver:toastmon(25KLoC) Property: PnpIrpCompletion