Regression Verification: Proving the equivalence of similar programs

1. Regression Verification: Proving the equivalence of similar programs Benny Godlin Ofer Strichman Technion, Haifa, Israel

2. Functional Verification • The main pillar of the grand challenge [H’03]. • Suppose we ignore completeness. • Still, there are two major problems: • Specification • Complexity

3. Words of wisdom “For every problem that you cannot solve, there is an easier problem that you cannot solve” * * George Polya, in How To Solve It

4. A more modest challenge: Regression Verification • Develop a method for formally verifying the equivalenceoftwosimilar programs. • Pros: • Default specification = earlier version. • Computationally easier than functional verification. • Ideally, the complexity should depend on the semanticdifference between the programs, and not on their size. • Cons: • Defines a weaker notion of correctness.

5. Regression-based technologies • In Testing: Regression testing is the most popular automatic testing method • In hardware verification: Equivalence checking is the most used verification method in the industry

6. …and 6 pages later: Hoare’s 1969 paper has it all. . . .

7. Previous work • In the theorem-proving world (mostly @ ACL2 community): • Not industrial programming languages • Not utilizing the similarity between the two programs • Industrial / realistic programs: • Code free of: loops, recursion, dynamic-memory allocation • microcode @ Intel [AEFMMSSTVZ-05], • embedded code @ Feng & Hu [FH-05], • symbolic simulation @ Matsumoto et al. [TSF-06]

8. Goals • Develop notions of equivalence • Develop corresponding proof rules • Present a prototype for verifying equivalence of C programs, that • incorporates the proof rules • sensitive to the magnitude of change rather than the magnitude of the programs

9. Notions of equivalence (1 / 6) Partial equivalence • Executions of P1 and P2 on equal inputs • …which terminate, • result in equal outputs. • Undecidable

10. Notions of equivalence (2 / 6) Mutual termination • Given equal inputs, • P1 terminates ,P2 terminates • Undecidable

11. Notions of equivalence (3 / 6) Reactive equivalence • Let P1 and P2 be reactive programs. • Executions of P1 and P2 • …which read the same input sequence, • emit the same output sequence. • Undecidable

12. Notions of equivalence (4 / 6) k-equivalence • Executions of P1 and P2 on equal inputs • …where loops and recursions are restricted to k iterations, • result in equal output. • Decidable

13. Notions of equivalence (5 / 6) Full equivalence • P1 and P2 are partially equivalent and mutually terminate • Undecidable

14. Notions of equivalence (6 / 6) Total equivalence • P1 and P2 are partially equivalent and both terminate • Undecidable

15. Notions of equivalence: summary • Partial equivalence • Mutual termination • Reactive equivalence • k-equivalence • Full equivalence* = Partial equivalence + mutual termination • Total equivalence** = partial equivalence + both terminate * Appeared in Luckham, Park, and M. Paterson [LPP-70] / Pratt [P-71] ** Appeared in Bouge and D. Cachera [BC-97]

16. Partial equivalence • Consider the call graphs: • … where A,Bhave: • same prototype • no loops • Prove partial equivalence of A, B • How shall we handle the recursion ? A B Side 1 Side 2

17. Hoare’s Rule for Recursion LetAbe a recursive function. “… The solution... is simple and dramatic: to permit the use of the desired conclusion as a hypothesis in the proof of the body itself.” [H’71]

18. Hoare’s Rule for Recursion // {p} A( . . . ) { . . . // {p} call A(. . .); // {q} . . . } // {q}

19. //in[A] A( . . . ) { . . . //in[call A] call A(. . .); //out[call A] . . . } //out[A] Rule 1: Proving partial equivalence //in[B] B( . . . ) { . . . // in[call B] call B(. . .); //out[call B] . . . } //out[B] A B

20. Q: How can a verification condition for the premise look like? A: Replace the recursive calls with calls to functions that over-approximateA, B,and are partially equivalent by construction Natural candidates: Uninterpreted Functions Rule 1: Proving partial equivalence

21. Proving partial equivalence • Let AUF , BUFbe A,B, after replacing the recursive call with a call to (the same) uninterpreted function. • We can now rewrite the rule: The premise is decidable

22. a, b) y) x, z; g; Using (PART-EQ-1): example unsigned gcd1UF (unsigned a, unsigned b) { unsigned g; if (b == 0) g = a; else { a = a % b; g = gcd1(b, a); } return g; } unsigned gcd2UF (unsigned x, unsigned y) { unsigned z; z = x; if (y > 0) z = gcd2(y, z % y); } return z; } ? = U U Tgcd1Tgcd2 a,b x,y g z

23. Rule 1: example Equal inputs Equal outputs

24. Partial equivalence: Generalization • Assume: • no loops; • 1-1 mapping mapbetween the recursive functions of both sides • Mapped functions have the same prototype • Define: • For a function f, UF(f) is an uninterpreted function such that • f and UF(f) have the same prototype • (f,g) 2map , UF(f) = UF(g).

25. Partial equivalence: Generalization • Definition: is called in A]

26. f f ’ UF UF = g’ g UF UF = Partial equivalence: Example (1 / 3) {(g,g’),(f,f’)} 2 map g’ g f f ’ Side 2 Side 1 Need to prove:

27. UF UF Partial equivalence: Example (2 / 3) • An improvement: • Find a map that intersects all cycles, e.g., (g,g’) • Only when calling functions in this map replace with uninterpreted functions g’ g f f ’ Side 2 Side 1

28. Partial equivalence: Example (3 / 3) Connected SCCs… • Prove bottom-up • Abstract partially-equivalent functions • Inline g g’ f f ’ h h’ UF UF Side 2 Side 1

29. RVT: Decomposition algorithm Legend: Equivalent pair Equivalence undecided yet Could not prove equivalent Syntactically equivalent pair check Unpaired function B: A: check f1() f1’() f2’() f2() U U f5’() f5() f7’() f3() f4() f3’() f4’() f6() U U U U

30. RVT: Decomposition algorithm (with SCCs) Legend: Equivalent pair Equivalence undecided yet Could not prove equivalent Syntactically equivalent pair Equivalent if MSCC B: A: check f1() f1’() f2’() f5’() f2() f5() U U U U U U f6’() f3() f4() f3’() f4’() f6() U U U U

31. PART-EQ: Soundness • Proved soundness for a simple programming language (LPL) • Covers most features of modern imperative languages • …but does not allow • call by reference, and • address manipulation.

32. PART-EQ: Completeness • We show a sufficient condition for completeness.

33. PART-EQ: Completeness • Definition: A,B are reach-equivalent if… (by example) A() { … if (cond1) f(a1,b1) else … … } B() { … if (cond2) g(a2,b2) else … .. }

34. PART-EQ: Completeness • Definition: A,B are reach-equivalent if… • For equal inputs: • For callees F,G s.t. (F,G) 2 map: • F is called ,G is called • F and G are called with the same arguments. • Completeness: (PART-EQ) is complete for reach-equivalent functions.* * under some “reasonable” assumptions (e.g. no dead-code)

35. What (PART-EQ) cannot prove... • Not reach-equivalent: calling with different arguments. returns n + nondet() returns n + n -1 + nondet()

36. What (PART-EQ) cannot prove... • Not reach-equivalent: calling under different conditions: when n == 1 : returns 1 returns 1 + nondet()

37. Notions of equivalence: summary • Partial equivalence • Mutual termination • Reactive equivalence • k-equivalence • Full equivalence* = Partial equivalence + mutual termination • Total equivalence** = partial equivalence + both terminate * Appeared in Luckham, Park, and M. Paterson [LPP-70] / Pratt [P-71] ** Appeared in Bouge and D. Cachera [BC-97]

38. Rule 2: Proving Mutual termination Prove with (PART-EQ)

39. a, b) y) x, (y, z % y); (b, a); Using (M-TERM): example unsigned gcd1UF (unsigned a, unsigned b) { unsigned g; if (b == 0) g = a; else { a = a % b; g = gcd1(b, a); } return g; } unsigned gcd2UF (unsigned x, unsigned y) { unsigned z; z = x; if (y > 0) z = gcd2(y, z % y); } return z; } ? = (b == 0) (y > 0) U U

40. Using (M-TERM): example • Proving reach-equiv(gcd1UF, gcd2UF) • Valid. gcd1,gcd2mutually terminate. Equal inputs Equal guards if called then equal arguments

41. The Regression Verification Tool (RVT) • Given two C programs: • loops recursive functions. • Map functions, globals, etc. • After that: • Decompose to the granularity of pairs of functions • Use a C verification engine (CBMC) to discharge

42. The Regression Verification Tool (RVT) • CBMC: a C bounded model checker by Daniel Kroening • Our use: • No loops or recursion to unroll... • Use “assume(…)” construct to enforce equal inputs. • Use assert() to demand equal outputs. • Uninterpreted functions are implemented as C functions: • Return consistent nondeterminisitic values.

43. The Regression Verification Tool (RVT) • The premise of (PART-EQ) requires comparing arguments. • What if these arguments are pointers ? • What our system does: • Dynamic structures: creates an unrolled nondeterministic structure • Arrays: attempts to find references to cells in the array.

44. . . . P1: exp1 exp2 . . . P2: exp’1 exp’2 = = = = RVT: User-defined equivalence specification • The user can define pairs of ‘checkpoints’: side 1: <label1, cond1, exp1>side 2:<label2, cond2, exp2> • In each side: • update an array with the value of expeach time it reacheslabelandconditionholds. • Assert that when executed on the same input…, • … these arrays are equivalent.

45. RVT Version A Version B • result • counterexample RVT • rename identical globals • enforce equality of inputs. • assert equality of outputs • add checkpoints • Supports: • Decomposition • Abstraction • some static analysis • … C program feedback CBMC

47. RVT: Experiments We tested the Regression Verification Tool (RVT) with: • Automaticallygenerated sizable programs with complex recursive structures and loops. • up-to thousands of lines of code • Limited-size industrial programs: • Parts of TCAS - Traffic Alert and Collision Avoidance System. • Core of MicroC/OS- real-time kernel for embedded systems. • Matlab examples: parts of engine-fuel-injection simulation.

48. Testing RVT on programs: Conclusions • For equivalent programs, partial-equivalence checks were very fast: • proving equivalence in minutes. • For non-equivalent programs: • RVT attempts to prove partial-equivalence but fails • then RVT tries to provek-equivalence

49. Summary • Regression verification is an important problem • A solution to this problem has a better chance to succeed in the industry than functional verification • A grand challenge by its own right… • Lots of future research...

50. More Challenges • Q1: How can we generalize counterexamples ? • Q2: What is the ideal gap between two versions of the same program, that makes Regression Verification most effective ? • Q3: How can functional verification and equivalence verification benefit from each other ?