Enhancing Symbolic Execution with Veritesting

Enhancing Symbolic Execution with Veritesting ThanassisAvgerinos, AlexandreRebert, Sang Kil Cha and David Brumley Carnegie Mellon University ICSE 2014

Background Background Symbolic Execution • Use symbols to represent variables • Concrete execution • Symbolic execution x = y + 1 z = x * 2 + 3 y = 1  z = 7 • y = in_y z = (in_y + 1) * 2 + 3

Background Symbolic Execution (2) x = input() if(x > 0) y = x; else y = -x; z = y; x = input() x > 0 ? T F y = x y = -x z = y Test case generation x > 0  SMT solver  input

Background Symbolic Execution (3) x = input() if(x > 0) y = x; else y = -x; z = y; x = input() x > 0 ? T F y = x y = -x z = y Program verification: z = |x| z = ite(x>0, x, -x)  SMT solver  Valid?

Problem Approaches • Dynamic symbolic execution (DSE) - testing • Path-based formulas • Easy-to-solve • Hard-to-generate (Path explosion) • Static symbolic execution (SSE) - verification • Property-based formulas • Hard-to-solve (solver blowup) • Easy-to-generate • Easy-to-generate & Easy-to-solve ?

Method Veritesting • Alternates between SSE andDSE. • Twice as many Bugs • Orders of magnitude more paths • Higher code coverage DSE SSE DSE SSE DSE

Method DSE w/o Veritesting x = input() x > 0 ? S <- Ø T F y = x y = -x z = y ……………

Method • CFGRecovery • CFGReduce • StaticSymbolic • Finalize

Method(1) CFGRecovery • Generate a partial CFG • (S) Symbolic branch • (E) Any hard-to-handle inst • ret • syscall • unknown  exit node

Method(2) CFGReduce • Transition points • Immediate postdominator of entry node • Predecessors of Exit • Unrolling loops • Switch to concrete value • User-defined bound

Method(3) StaticSymbolic if(x > 1) y = 1; else if(x < 42) y = 17;

Method(4) Finalize x = input() • Create new executor • For each distinct transition point • CFG accurate • Overestimation • Underestimation • Incremental Deployment x > 0 ? T F y = x y = -x z = y …………… 12

Implementation MergePoint

Evaluation Evaluation • Metrics • Number of bugs • Node coverage • Path coverage • Benchmarks • GNU coreutils • BIN suite (1,023 programs) • Debian packages (33,248 programs)

Evaluation (1) Bug finding • BIN: 63 + 85 • coreutils: 2 new bugs • 9 years old, time zone parser in Gnulib

Evaluation (2) Node Coverage 27% more coverage than S2E on coreutils

Evaluation (3) Path Coverage • Three estimations • Time to complete test • 46 programs, 73% faster • Multiplicity • For bin, 1.4 x 10290(average), 1.8 x 1012(median) • For coreutils, 1.4 x 10199 (average), 4.4 x 1011 (median) • Fork rate • Reduce average by 65% • Reduce median by 44%

Evaluation (4) Debian benchmark

Conclusion Conclusion • Veritesting: enhance the DSE with SSE • MergePonit: infrastructure testing programs • Large value evaluation and results

Discussion • Why is it faster? • SSE introduces overhead for formula-solving • Reduces the number of duplicated paths  Benefits > cost • Insight into the SMT solver • Exploit generation • Other bugs

Thanks

Enhancing Symbolic Execution with Veritesting