1 / 72

via Cooperative Testing and Analysis

Improving Software Dependability. via Cooperative Testing and Analysis. Tao Xie Department of Computer Science North Carolina State University. Software Dependability Matters. Loss of Money: Software faults costed the U.S. economy about $59.5 billion each year (0.6% GDP) [NIST 02]

ciro
Télécharger la présentation

via Cooperative Testing and Analysis

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Improving Software Dependability via Cooperative Testing and Analysis Tao Xie Department of Computer Science North Carolina State University

  2. Software Dependability Matters • Loss of Money: Software faults costed the U.S. economy about $59.5 billion each year (0.6% GDP) [NIST 02] • Loss of Life: Faulty medical devices caused 30,000 deaths and 600,000 injuries (1985-2005), with likely 8% due to software faults [FDA 06] • …

  3. Improving Software Dependability Titles of Major Conference Pubs (2005-Present)

  4. Improving Software Dependability Major Conference Pubs (2005-Present) Testing & Analysis Analytics Reliability ICSE 11, ICSE 10a, ICSE 10b ICSE 09b, ICSE 07 FSE 10 , FSE 12c ISSTA 11, ISSTA 10, ISSTA 09 ASE 11a, ASE 08b, ASE 06 OOPSLA 11, ECOOP 06 ICSE 12a, ICSE 09a ICSE 08, ICSE 05 FSE 09, FSE 07, FSE 12b ASE 11b, ASE 10, ASE 09a ASE 09b, ASE 08a, ASE 07 ECOOP 09, WWW 13 Security/Privacy FSE 11, SIGMETRICS 08 WWW 07 FSE 12a Performance ICSE 12b 10 ICSE, 7 FSE 3 ISSTA, 9 ASE 3 OOPLSA/ECOOP SIGMETRICS 08

  5. Improving Software Dependability Major Conference Pubs (2005-Present) • Artifacts Under Analysis • DB apps • GUI apps • Web/SOA apps • Mobile apps • Cloud apps • Analytics systems • AC/Firewall policies • API docs • Bug reports • Requirements doc • Execution traces • … Testing & Analysis Analytics Reliability ICSE 11, ICSE 10a, ICSE 10b ICSE 09b, ICSE 07 FSE 10 , FSE 12c ISSTA 11, ISSTA 10, ISSTA 09 ASE 11a, ASE 08b, ASE 06 OOPSLA 11, ECOOP 06 ICSE 12a, ICSE 09a ICSE 08, ICSE 05 FSE 09, FSE 07, FSE 12b ASE 11b, ASE 10, ASE 09a ASE 09b, ASE 08a, ASE 07 ECOOP 09, WWW 13 Security/Privacy FSE 11, SIGMETRICS 08 WWW 07 FSE 12a Performance ICSE 12b 10 ICSE, 7 FSE 3 ISSTA, 9 ASE 3 OOPLSA/ECOOP SIGMETRICS 08

  6. Impact/Leadership: Software Testing • We produce; others use • We lead; others follow Keynotes 2010 Tutorials Programming Contests Call for Proposals

  7. Redundant Test Detectionfor ParasoftJtest Rostra identified 90% tests generated by ParasoftJtest 4.5 to be redundant. Parasoft fixed issue in later versions after seeing our results. ASE 2004 7

  8. Fitnex Path-Exploration Strategy for Pex in Pex Released since 2008 • Download countsinitial 20 months of releaseAcademic: 17,366 • Industrial: 13,022 • Total: 30,388 Pex detected various bugs (including a serious bug) in a core .NET component (already been extensively tested over 5 years by 40 testers) , used by thousands of developers and millions of end users. “It has saved me two major bugs (not caught by normal unit tests) that would have taken at least a week to track down and fix normally plus a few smaller issues so I'm a big proponent of Pex.” 8

  9. Coding Duel Games forPex for Fun Released since 2010 www.pexforfun.com 1,129,019 clicked 'Ask Pex!' “I used to love the first person shooters and the satisfaction of blowing away a whole team of Noobies playing Rainbow Six, but this is far more fun.” X “I’m afraid I’ll have to constrain myselfto spend just an hour or so a day on this really exciting stuff, as I’m really stuffed with work.” “It really got me *excited*. The part that got me most is about spreading interest in teaching CS: I do think that it’s REALLY great for teaching | learning!”

  10. Access Control Policy Tool (ACPT) Released since 2009 Access Control Policy Tool (ACPT) beta release being beta-tested with >130 users/organizations “There are many valuable features in the ACPT and we hope to recommend it to our vendors to verify and validate the policies they author.” Beta-users: NSA, MITRE, DISA, NOAA, SAIC, DNI, Pacific Northwest National Lab, Fermi Lab, BAE system, Lockheed Martin, Raytheon, Boeing, SMI, VA government, John Hopkins U., … “ACPT provides all the adequate functionality for the verification of access control policies against static constraints.” 10

  11. Impact/Leadership: Software Testing • We produce; others use • We lead; others follow Scalable Unit Test Generation Testing+Human Factors Keynotes 2010 Tutorials Programming Contests Call for Proposals

  12. Impact/Leadership: Software Analytics • We produce; others use • We lead; others follow XIAO StackMine 2014 2007 7 years of ICSE Tutorials 2013 … 2013

  13. StackMine: Performance Debugging in the Large StackMine 13

  14. XIAO: Code Clone Detectionfor Security + Refactoring XIAO available in Visual Studio 2012 XIAO Clone Search integrated in workflow @Microsoft Security Response Center (MSRC) “run [XIAO] for every MSRC case to find any instance of the vulnerable code in any shipping product. This system is the one that found several of the copies of CVE-2011-3402 that we are now addressing with MS12-034.” -MS Security Research & Defense blog Searching similar snippets for fixing bug once Finding refactoring opportunity 2012

  15. Impact/Leadership: Software Analytics • We produce; others use • We lead; others follow Software Analytics (insightful and actionable info for software practitioners) XIAO StackMine 2014 2007 7 years of ICSE Tutorials 2013 … 2013

  16. Cooperative Testing and Analysis Testing & Analysis Analytics Reliability ICSE 11, ICSE 10a, ICSE 10b ICSE 09b, ICSE 07 FSE 10 , FSE 12c ISSTA 11, ISSTA 10, ISSTA 09 ASE 11a, ASE 08b, ASE 06 OOPSLA 11, ECOOP 06 ICSE 12a, ICSE 09a ICSE 08, ICSE 05 FSE 09, FSE 07, FSE 12b ASE 11b, ASE 10, ASE 09a ASE 09b, ASE 08a, ASE 07 ECOOP 09, WWW 13 Security/Privacy FSE 11, SIGMETRICS 08 WWW 07 FSE 12a Performance ICSE 12b ICSE 12b SIGMETRICS 08

  17. Global Trend: Replace Human orGet Human Out of the Loop IBM Watson as Jeopardy! player Google’s driverless car Microsoft's instant voice translation tool

  18. Unit Test Generation: Replace Human or Get Human Out of the Loop Class Under Test 00: classGraph { … 03: public void AddVertex(Vertex v) { 04: vertices.Add(v); 05: } 06: public Edge AddEdge(Vertex v1, Vertex v2) { … 15: } 16: } Manual Test Generation: Tedious, Missing Special/Corner Cases, … Generated Unit Tests • void test2() { • Graph ag = new Graph(); • Vertex v1 = new Vertex(0); • AddEdge(v1, v1); • } void test1() { Graph ag = new Graph(); Vertex v1 = new Vertex(0); } … 18 18

  19. State-of-the-Art/Practice Test Generation Tools Running Symbolic PathFinder ... … ====================================================== results no errors detected ====================================================== statistics elapsed time: 0:00:02 states: new=4, visited=0, backtracked=4, end=2 search: maxDepth=3, constraints=0 choice generators: thread=1, data=2 heap: gc=3, new=271, free=22 instructions: 2875 max memory: 81MB loaded code: classes=71, methods=884 …

  20. Challenges Faced by Test Generation Tools • Ex: Dynamic Symbolic Execution (DSE) /Concolic Testing [Godefroid et al. 05][Sen et al. 05][Tillmann et al. 08] • Instrument code to explore feasible paths • Challenge: path explosion When desirable receiver or argument objects are not generated Total block coverage achieved is 50%, lowest coverage 16%. • object-creation problems (OCP) - 65% • external-method call problems (EMCP) – 27%

  21. Example Object-Creation Problem 00: classGraph { … 03: public void AddVertex(Vertex v) { 04: vertices.Add(v); // B1 } 06: public Edge AddEdge(Vertex v1, Vertex v2) { 07: if (!vertices.Contains(v1)) 08: throw new VNotFoundException(""); 09: // B2 10: if (!vertices.Contains(v2)) 11: throw new VNotFoundException(""); 12: // B3 14: Edge e = new Edge(v1, v2); 15: edges.Add(e); } } //DFS:DepthFirstSearch 18: classDFSAlgorithm{ … 23: public void Compute (Vertex s) { ... 24: if (graph.GetEdges().Size() > 0) { // B4 25: isComputed = true; 26: foreach (Edge e ingraph.GetEdges()) { 27: ... // B5 28: } 29: } } } [OOPSLA 11] • A graph example from QuickGraph library • Includes two classes Graph DFSAlgorithm • Graph AddVertex AddEdge: requires both vertices to be in graph 21 21

  22. Example Object-Creation Problem • Test target: Cover true branch (B4) of Line 24 • Desired object state: graph should include at least one edge • Target sequence: • Graph ag = new Graph(); • Vertex v1 = new Vertex(0); • Vertex v2 = new Vertex(1); • ag.AddVertex(v1); • ag.AddVertex(v2); • ag.AddEdge(v1, v2); • DFSAlgorithmalgo = new DFSAlgorithm(ag); • algo.Compute(v1); 00: classGraph { … 03: public void AddVertex(Vertex v) { 04: vertices.Add(v); // B1 } 06: public Edge AddEdge(Vertex v1, Vertex v2) { 07: if (!vertices.Contains(v1)) 08: throw new VNotFoundException(""); 09: // B2 10: if (!vertices.Contains(v2)) 11: throw new VNotFoundException(""); 12: // B3 14: Edge e = new Edge(v1, v2); 15: edges.Add(e); } } //DFS:DepthFirstSearch 18: classDFSAlgorithm{ … 23: public void Compute (Vertex s) { ... 24: if (graph.GetEdges().Size() > 0) { // B4 25: isComputed = true; 26: foreach (Edge e ingraph.GetEdges()) { 27: ... // B5 28: } 29: } } } [OOPSLA 11] 22 22

  23. Challenges Faced by Test Generation Tools • Ex: Dynamic Symbolic Execution (DSE) /Concolic Testing [Godefroid et al. 05][Sen et al. 05][Tillmann et al. 08] • Instrument code to explore feasible paths • Challenge: path explosion • Typically DSE instruments or explores only methods @ project under test; • Third-party API external methods (network, I/O, ..): • too many paths • uninstrumentable Total block coverage achieved is 50%, lowest coverage 16%. • object-creation problems (OCP) - 65% • external-method call problems (EMCP) – 27%

  24. Example External-Method Call Problems (EMCP)

  25. Challenges Faced by Test Generation Tools • Ex: Dynamic Symbolic Execution (DSE) /Concolic Testing [Godefroid et al. 05][Sen et al. 05][Tillmann et al. 08] • Instrument code to explore feasible paths • Challenge: path explosion Total block coverage achieved is 50%, lowest coverage 16%. • object-creation problems (OCP) - 65% • external-method call problems (EMCP) – 27%

  26. What to Do Next? 2010 Dagstuhl Seminar 10111 Practical Software Testing: Tool Automation and Human Factors

  27. Conventional Wisdom: Improve Automation Capability @NCSU ASE • Tackling object-creation problems • Seeker [OOSPLA 11] , MSeqGen[ESEC/FSE 09] Covana[ICSE 2011], OCAT [ISSTA 10]Evacon[ASE 08], Symclat[ASE 06] • Still not good enough (at least for now)! • Seeker (52%) > Pex/DSE (41%) > Randoop/random (26%) • Tackling external-method call problems • DBApp Testing [ESEC/FSE 11], [ASE 11] • CloudApp Testing [IEEE Soft 12] • Deal with only common environment APIs

  28. Unconventional Wisdom: Bring Human in the Loop Ironies of Automation “The increased interest in human factors among engineers reflects the irony that the more advanced a control system is, so the more crucial may be the contribution of the human operator.” Malaysia Airlines Flight 124 @2005 Lisanne Bainbridge, "Ironies of Automation”, Automatica 1983 .

  29. Example Object Creation Problem (OCP) • Test target: Cover true branch (B4) of Line 24 • Desired object state: graph should include at least one edge • Target sequence: • Graph ag = new Graph(); • Vertex v1 = new Vertex(0); • Vertex v2 = new Vertex(1); • ag.AddVertex(v1); • ag.AddVertex(v2); • ag.AddEdge(v1, v2); • DFSAlgorithmalgo = new DFSAlgorithm(ag); • algo.Compute(v1); 00: classGraph { … 03: public void AddVertex(Vertex v) { 04: vertices.Add(v); // B1 } 06: public Edge AddEdge(Vertex v1, Vertex v2) { 07: if (!vertices.Contains(v1)) 08: throw new VNotFoundException(""); 09: // B2 10: if (!vertices.Contains(v2)) 11: throw new VNotFoundException(""); 12: // B3 14: Edge e = new Edge(v1, v2); 15: edges.Add(e); } } //DFS:DepthFirstSearch 18: classDFSAlgorithm{ … 23: public void Compute (Vertex s) { ... 24: if (graph.GetEdges().Size() > 0) { // B4 25: isComputed = true; 26: foreach (Edge e ingraph.GetEdges()) { 27: ... // B5 28: } 29: } } } 29 29

  30. Human Can Help! Object Creation Problems (OCP) Tackle object-creation problems with Factory Methods

  31. Human Can Help!External-Method Call Problems (EMCP) Tackle external-method call problems with Mock Methods or Method Instrumentation Mocking System.IO.File.ReadAllText

  32. Automation in Software Testing 2010 Dagstuhl Seminar 10111 Practical Software Testing: Tool Automation and Human Factors

  33. Automation in Software Testing Human Factors Dagstuhl Seminar 10111 Practical Software Testing: Tool Automation and Human Factors

  34. CooperativeSoftware Testing and Analysis • Human-AssistedComputing • Driver: tool Helper: human • Ex. Covana[ICSE 2011] • Human-CentricComputing • Driver: human  Helper: tool • Ex. Pex for Fun[ICSE 2013 SEE] Interfaces are important. Contentsare important too!

  35. Example Problems Faced by Tools Symptoms all non-primitive program inputs/fields object-creation problems (OCP) external-method call problems (EMCP) (Likely) Causes • all executed external-method calls

  36. Technical Challenges • Causal analysis: tracing between symptoms and (likely) causes • Reduce cost of human consumption • reduction of #(likely) causes • diagnosis of each cause • Solution construction: fixing suspected causes • Reduce cost of human contribution • measurement of solution goodness • Inner iteration of human-tool cooperation!

  37. Black-Box Systematic Debugging Not Feasible Symptoms • Given symptom s • foreach (c in LikelyCauses) { • Fix(c); if (IsObserved(s)) RelevantCauses.add(c) } object-creation problems (OCP) external-method call problems (EMCP) (Likely) Causes

  38. White-Box Causal Analysis: Covana [ICSE 11] • Goal: Precisely identify problems (causes) faced by a tool for causing not to cover a statement (symptom) • Insight: Partially-covered conditional has data dependency on a real problem From xUnit

  39. ECMP with Data Dependency on Program Inputs [Inputs  EMCP] • Consider only EMCPs whose arguments have data dependencies on program inputs • Fixing such problem candidates facilitates test-generation tools Data Dependencies From xUnit

  40. Symptom with Data Dependency on EMCP [EMCP Symptom] Symptom Expression: return(File.Exists) == true Element of EMCP Candidate: return(File.Exists) • Partially-covered conditionals have data dependencies on EMCP candidates Conditional in Line 1 has data dependency on File.Exists

  41. Example EMCP being Filtered [EMCP !Symptom] From xUnit

  42. Tool Architecture of Covana Problem Candidate Identification Program Generated Test Inputs Forward Symbolic Execution Runtime Events Problem Candidates [Inputs  EMCP] Coverage Runtime Information Data Dependence Analysis Identified Problems [EMCP Symptom]

  43. Evaluation – Subjects and Setup • Subjects: • xUnit: unit testing framework for .NET • 223 classes and interfaces with 11.4 KLOC • QuickGraph: C# graph library • 165 classes and interfaces with 8.3 KLOC • Evaluation setup: • Apply Pex to generate tests for program under test • Feed the program and generated tests to Covana • Compare baseline solution and Covana

  44. Evaluation – Research Questions • RQ1: How effective is Covana in identifying the two main types of problems, EMCPs and OCPs? • RQ2: How effective is Covana in pruning irrelevant problem candidates of EMCPs and OCPs?

  45. Evaluations - RQ1: Problem Identification • Covana identifies • 43 EMCPs with only 1 false positive and 2 false negatives • 155 OCPs with 20 false positives and 30 false negatives.

  46. Evaluation –RQ2: Irrelevant-Problem-Candidate Pruning • Covana prunes • 97%(1567 in 1610) EMCP candidates with 1 false positive and 2 false negatives • 66% (296 in 451) OCP candidates with 20 false positives and 30 false negatives

  47. CooperativeSoftware Testing and Analysis • Human-AssistedComputing • Driver: tool Helper: human • Ex. Covana[ICSE 2011] • Human-CentricComputing • Driver: human  Helper: tool • Ex. Pex for Fun[ICSE 2013 SEE] Interfaces are important. Contentsare important too!

  48. Coding Duel Games forPex for Fun Released since 2010 www.pexforfun.com 1,129,019 clicked 'Ask Pex!' “I used to love the first person shooters and the satisfaction of blowing away a whole team of Noobies playing Rainbow Six, but this is far more fun.” X “I’m afraid I’ll have to constrain myselfto spend just an hour or so a day on this really exciting stuff, as I’m really stuffed with work.” “It really got me *excited*. The part that got me most is about spreading interest in teaching CS: I do think that it’s REALLY great for teaching | learning!”

  49. Behind the Scene of Pex for Fun behavior Secret Impl== Player Impl Player Implementation class Player { public static int Puzzle(int x) { return x; } } Secret Implementation class Secret { public static int Puzzle(int x) { if (x <= 0) return 1; return x * Puzzle(x-1); } } class Test { public static void Driver(int x) { if (Secret.Puzzle(x) != Player.Puzzle(x)) throw new Exception(“Mismatch”); } }

  50. Coding Duel Competition@ICSE 2011 http://pexforfun.com/icse2011

More Related