University of Virginia Computer Science

1. Automatically Inferring Temporal Properties for Program EvolutionJinlin Yang and David Evans15th IEEE International Symposium on Software Reliability Engineering 5 November 2004Saint-Malo, France University of Virginia Computer Science

2. Temporal Properties • Constrain ordering of events • Essential for program correctness (file open before read) • Reveal important differences between programs: S P  not a chef • But, hard for humans to document correctly [Holzmann’s FSE 2002 keynote] • Can we infer useful temporal properties automatically? This talk argues: Yes! www.cs.virginia.edu/ipa

3. Dynamically Inferring Properties • Inherently unsound: guessing properties of all executions by looking at a small number • Value-based invariants • Daikon [Ernst, TSE, Feb 01] • Temporal properties • Mining specification [Ammons, POPL 02] • Extracting component interfaces [Whaley, ISSTA 02] • We focus on relationship between 2 or 3 events: automation, scalability www.cs.virginia.edu/ipa

4. System Overview Shared Properties Program Version 1 Inferred Properties 1 Dynamic Inference Difference Analyzer Program Version 2 Inferred Properties 2 Different Properties … … www.cs.virginia.edu/ipa

5. Shared Properties Program Version 1 Inferred Properties 1 Dynamic Inference Difference Analyzer Program Version 2 Inferred Properties 2 Different Properties … … Property Inference Program Inferred Properties Instrumented Program Execution Traces Inference Instrumentation Testing Test Suite Candidate Property Patterns www.cs.virginia.edu/ipa

6. Candidate Property Patterns • Response pattern [Dwyer, ICSE ‘99] [-P]* (P [-S]* S [-P]*)* • Filtered version (all non P/S events removed): S*(P+S+)* SPPSPS  SPSP  • Too weak for our purposes www.cs.virginia.edu/ipa

7. Partial Order of Patterns Alternating (PS)* 4 Derived Patterns Λ MultiCause (P+S)* EffectFirst S*(PS)* MultiEffect (PS+)* Stricter Λ Λ Λ OneCause S*(PS+)* CauseFirst (P+S+)* OneEffect S*(P+S)* 3 Primitive Patterns Response S*(P+S+)* www.cs.virginia.edu/ipa

8. For all pairs of events, P and S: Check which (if any) of OneCause, CauseFirst and OneEffect are satisfied Infer the strictest property satisfied if more than one primitive is Inferring Properties Alternating (PS)* Λ MultiCause (P+S)* EffectFirst S*(PS)* MultiEffect (PS+)* Λ Λ Λ OneCause S*(PS+)* CauseFirst (P+S+)* OneEffect S*(P+S)* www.cs.virginia.edu/ipa

9. Traces Example For any two events, determine the strictest pattern satisfied by all traces www.cs.virginia.edu/ipa

10. Traces Example For any two events, determine the strictest pattern satisfied by all traces www.cs.virginia.edu/ipa

11. Traces Example For any two events, determine the strictest pattern satisfied by all traces CauseFirst  OneEffect MultiCause www.cs.virginia.edu/ipa

12. Implementation Program Inferred Properties Instrumented Program Execution Traces Inference Instrumentation Testing Test Suite Candidate Property Patterns • Prototype (described in paper) • 900 lines of Perl • New implementation • 8K lines of Java • Improved performance • Analyses, ranking heuristics • Automated for Java using JRat • Method entry/exit events • Manually for C code or output • (today’s examples) www.cs.virginia.edu/ipa

13. Experiments • Hypotheses • We can automatically extract interesting temporal properties • Differences in inferred temporal properties among multiple versions can reveal interesting things • Target programs • Tour bus simulator (8 student submissions) • OpenSSL (0.9.6, 0.9.7-0.9.7d) www.cs.virginia.edu/ipa

14. Tour Bus Simulator • Bus and each passenger are a separate thread • Assignment in Fall 2003 graduate-level course (before we started this project) • 8 submissions from the instructor (all believed to be correct) www.cs.virginia.edu/ipa

15. Testing N, the number of people C, the capacity of the bus T, the number of trips Executed each submission with 100 randomly generated inputs, where 20 < C ≤ 40 C+1 ≤ N ≤ 2C 1 ≤ T ≤ 10 > cville_bus –N 2 –C 1 –T 2 Bus waiting for trip 1 Passenger 0 boards Bus drives around Charlottesville Passenger 0 exits Bus waiting for trip 2 Passenger 1 boards Bus drives around Charlottesville Passenger 1 exits Bus stops for the day Event traces extracted from program output (no need to instrument program) www.cs.virginia.edu/ipa

16. Differences Reveal Problems • waitdrives not Alternating • Bus drives around before all passengers exit • Bug in locking code (misplaced synchronization) • drivesexits, waitexits not MultiEffect • Bus drives around twice before letting passengers exit • Missing synchronization www.cs.virginia.edu/ipa

17. OpenSSL • Widely used implementation of the Secure Socket Layer protocol • 6 versions [0.9.6, 0.9.7, 0.9.7a-d] between Sept 2000 and March 2004 • We focus on the handshake protocol • 38 different event types www.cs.virginia.edu/ipa

18. Server Client BEFORE+ACCEPT SR_CLNT_HELLO SW_SRVR_HELLO SW_CERT SW_KEY_EXCH SW_CERT_REQ SW_SRVR_DONE SW_FLUSH SR_CERT SR_KEY_EXCH SR_CERT_VRFY SR_FINISHED SW_CHANGE SW_FINISHED SW_FLUSH OK www.cs.virginia.edu/ipa

19. Testing • Manually instrumented server to record handshake events • Executed each version of server with 1000 randomly generated clients • Client modified advance to a randomly selected state with 5% probability www.cs.virginia.edu/ipa

20. Inferred Alternating Patterns Documented change: ignore unrequested client certificates Fixed bug causing server crashes Race condition (present in all versions) 7 alternating patterns same for all versions www.cs.virginia.edu/ipa

21. Partitioning Traces All Traces Correct Clients (never jump to random state) Faulty Clients Follows SSL specification No Server Error Server Error Missing Alternating patterns: allows handshake cycle Segmentation fault (<0.9.7d) www.cs.virginia.edu/ipa

22. Performance • Prototype implementation (described in paper) • Up to 3-4 minutes for examples (Pentium 4, 3GHz, 1G RAM) • Too slow (several days) for larger programs • Scales approximately as trace length events2 • New implementation • Scales approximately as trace length events • Analyze these experiments in < .5s • Bus: 102 events, 100 traces, 222 events/trace: 0.4s • OpenSSL: 35 events, 1000 traces, 18 events/trace: 0.3s • Feasible to analyze large systems • Thousands of different events • Traces with millions of events • Preliminary results from experiments with JBoss www.cs.virginia.edu/ipa

23. Summary of Experiments • Useful in program evolution • Reveal interesting changes in OpenSSL • Identify unexpected differences • Revealed bugs • Tour bus: identified faulty implementation • Multiple implementations are rare, but multiple representations are common (i.e. design, model, code) • OpenSSL: • Differences between versions revealed • Flaws found by partitioning traces www.cs.virginia.edu/ipa

24. Future Work/Research Questions • Can we make this feasible in practice? • Automatically identify interesting events • Heuristics to identify important patterns • Scalability and automation • Can we make it more useful? • More expressive patterns: more events, combine with data-flow • Understand impact of different testing strategies www.cs.virginia.edu/ipa

25. Conclusion • Automatically inferring temporal properties is feasible • Even very simple property patterns reveal interesting program properties www.cs.virginia.edu/ipa

26. Questions? http://www.cs.virginia.edu/ipa This work is funded in part by the National Science Foundation. Thanks: Marty Humphrey for providing the student submissions, Chengdu Huang for help with OpenSSL, Joel Winstead, the anonymous reviewers, and anonymous CS650 students for writing buggy code. www.cs.virginia.edu/ipa