Concolic Modularity Testing

1. Concolic Modularity Testing Derrick Coetzee University of California, Berkeley CS 265 Final Project Presentation

2. Concolic Testing • A white-box improvement on random testing • Observe how inputs are used at runtime • Compute new inputs designed to force program to execute a different path • Effective in testing program correctness on a variety of inputs

3. More than correctness • Analogy with type systems: • Some types prevent bad inputs • Some types prevent abstraction violations • Can concolic testing also detect abstraction violations? • Detect implementation dependence • Find portability issues • Help enforce modularity

4. Detect implementation dependence • Forgetting to check for error return codes • e.g. Assuming malloc() never returns NULL • Relying on the behavior of a particular implementation • e.g. Assuming result of malloc() is always word-aligned • Not coping with improbable events • e.g. Assuming hash functions always return distinct values on distinct inputs.

5. A simple approach • Introduce a new symbolic variable whenever the function is invoked • Depending on its value, modify the result void* my_malloc(size_t size) { void* result = malloc(size); intsimulate_out_of_memory; CREST_int(simulate_out_of_memory); if (simulate_out_of_memory) { result = NULL; } return result; }

6. A simple approach • Doesn’t always work • May lead to an infeasible sequence of results intmy_hash(char* s) { int result; CREST_int(result); return result; } my_hash("hello") → 0 /* OK */ my_hash("world") → 0 /* OK */ my_hash("hello") → 1 /* Not OK */

7. Generalizing the approach • In general, the behavior of functions is constrained by their specification • e.g. Hash functions must be pure • With concolic testing, invariants can be enforced directly in the source language • e.g. By adding an input/output history

8. Generalizing the approach string_int_maphistory; intmy_hash(char* s) { if (contains_key(history, s)) { return lookup(history, s); } else { int result; CREST_int(result); add_pair(history, s, result); return result; } }

9. A more complex example • A sort function invokes a comparator (≤,=) • Comparator is constrained to be a total order • a = a • if a = b then b = a • if a = b and b = c then a = c • if a ≤ b and b ≤ a then a = b • if a ≤ b and b ≤ c then a ≤ c • Can use input/output history to enforce all these properties in the source language

10. Liveness properties • Specifications may constrain sequences of operations • e.g. if malloc() returns NULL, it will continue to return NULL for arguments of the same size or larger until free() is called • Easy to support by maintaining state across calls • May even help reduce the search space

11. Experiments: malloc test • Took the CREST grep 2.2 example • Instrumented all malloc() calls • Possibly return NULL • CREST enumerated 5621 tests • 2 were real bugs • Adding the liveness constraint: • Reduced the number of tests to 2869 (49% less) • Still found the 2 bugs

12. Experiments: sort • Selection sort using a comparator function • The comparator was constrained only to represent a total order • For a list of 5 elements, CREST enumerated 541 different comparators • If any of the constraints are removed, CREST finds an explicit comparator function that causes the sort to fail

13. Experiments: hash • grep 2.2 uses an internal hash function for fast state matching • Constrained hash function only to be pure • Enumerated 1472 tests on test input • grep was robust against hash collisions • Removing pure constraint did not produce errors but produced extra states