Upcoming Schedule

1. Upcoming Schedule • Friday: Homework #3 due at 5:00pm • Saturday: last chance to submit Homework #3 (noon) • Monday: Midterm exam!! • Next Wednesday: Still. More. Testing!

2. Exam Logistics • When: Monday, October 28, 4:30pm • Where: DRL A1 • The exam is open-book but you may not use electronic devices • Questions from previous midterms available in Files section of Canvas • Solutions now available, too! • Study guide linked from course website

3. What does “open-book” mean? • You may use: • Printouts of the assigned papers • Lecture notes (yours, or from website) • The study guide and previous exam solutions • Graded quizzes and homework solutions • You may not use: • Electronic devices • The papers of the person sitting next to you • “Secret” notes that you are sharing with your friends and stashed in the men's room trash bin

4. What's on the exam? • Software testing basics • Testing strategies & test case generation • Black-box testing • White-box testing • Fault-based testing • Software quality • Design concepts • Design complexity metrics • Code smells • Refactoring patterns

5. What's NOT on the exam? • “All I Really Need to Know (about Software Engineering) I Learned in Kindergarten” • Process models, software configuration management • Material presented by guest speakers • You will not be asked about the results of any experiments discussed in the papers • You will not be asked about the implementation details of any software presented in the papers

6. Exam format • Multiple choice questions based on definitions • Quantitative problem-solving questions • Calculating design metric values • Applying refactoring patterns • Identifying black-box test cases • Using symbolic execution to identify path conditions • Generating test cases using mutation testing • Measuring test coverage according to various criteria • Qualitative short-answer “analysis” questions

7. Preparing for the exam • Review the lecture notes on course website • Make sure you know (or can quickly look up) the definitions of terms from the study guide • Make sure you understand the code analysis examples we did in class • Review the study guide and previous exam questions; make sure you more or less know the answers and how to solve those problems • Find other students who are smarter than you and convince them to let you study with them

8. How to prepare for anopen-book exam • Make sure you study anyway!! • Have a “cheat sheet” of definitions • Put the papers in a sensible order • Create an index mapping concepts to papers/notes • Highlight the important parts of the papers

9. How to take a Chris Murphy exam • The number of points a question is worth is (roughly) equal to the number of minutes you should be spending on it • You will have 75 minutes to complete the exam • There are 60 total points on the exam • Multiple choice questions are always first • Don't spend a lot of time on these (1 point each) at the expense of more complicated problems

10. Any questions about the format of the exam?

11. Part 1: Design and Refactoring

12. Software Quality: ISO 9126 • External Quality • Functionality, Reliability, Efficiency, Usability, Portability • Internal Quality (Maintainability) • Stability, Changeability, Testability, Analyzability (Readability, Understandability) • Internal quality affects external quality!

13. “No Silver Bullet” • What are the “essential difficulties” of software development? • Which past advances in software engineering have addressed “accidental difficulties”? • What does Brooks say is the most important thing we can do to improve software?

14. Software Design Concepts • Organization • Modularity • Functional Independence • Cross-cutting Concerns (Aspects) • Exposure of Functionality • Abstraction & Refinement • Information Hiding (Encapsulation) • How do these relate to internal quality?

15. Inter-Component Code Complexity • Henry-Kafura structural complexity (fan-in, fan-out) • Object-oriented complexity metrics • Depth of Inheritance Tree • Number of Children • Afferent & Efferent Coupling • Pairwise Coupling • Instability • Response for a Class

16. Intra-Component Code Complexity • Weighted Methods per Class • McCabe Cyclomatic Complexity • Chapin Data Complexity • Lack of Cohesion of Methods

17. Refactoring • What is refactoring? • Why should you refactor? • When should you refactor? • How should you refactor? • Possible problems caused by refactoring

18. Code Smells • Duplicate code • Long method • Large class • Primitive obsession • Message chain

19. Refactoring Patterns • Extract Method • Pull Up Method • if duplicate code is found in two separate classes • Extract Class, Extract Superclass • composition vs. inheritance • Hide Delegate • to break a message chain

20. Part 2: Testing

21. Software Testing Basics • What is the definition of “software testing”? • Executing a program in an attempt to determine whether it behaves as expected • Goal is to find bugs, not prove correctness • Failure: when there is a difference between the actual output and the expected output (as reported by the test oracle) • Error: deviation in internal state (from correct state) that led to failure • Fault: static defect in code that led to error

22. Test Oracles • False positive: thinking there’s a bug when really there isn’t • False negative: thinking there’s no bug when really there is • Sound: no false positives • Complete: no false negatives • Accuracy: percent of “true” results • Precision: percent of “true” positive results

23. Test Case Generation • Exhaustive testing: all possible inputs • Generally not feasible • Random testing: choose inputs randomly • Easy to automate • No indication of progress • Hard to know expected outputs • Coverage-based: attempt to cover a certain criteria • Black-box testing: cover the specification • White-box testing: cover the code • Fault-based: show that program does not exhibit certain types of faults

24. Coverage-Based Testing • Amount of testing to be done is stated in terms of measurable criteria • A test set (collection of individual test cases) covers all or some part of the criteria • The percentage of criteria that are covered is the coverage level • Testing continues until an adequate level is achieved

25. Black-Box Testing • Criteria: how much of the specification is covered • Assumption #1: if a failure is revealed for a given value of input variable v, then it is likely to be revealed for similar values of v • As a result of this assumption, we can split up the specification space into equivalence classes • Assumption #2: if a failure is revealed for a given value of input variable v, then it is likely to be revealed regardless of the value of other variables (single fault assumption)

26. Black-Box Coverage Criteria • Weak normal: what percentage of the separate equivalence classes are covered? • Pairwise normal: what percentage of the pairs of equivalence classes are covered? • Strong normal: what percentage of the combinations of equivalence classes are covered?

27. White-Box Testing • “Treat the code as a graph and try to cover it” • How do you know that this code works if you haven’t tested it? • Coverage metrics: Statement, Branch, Path • Path coverage subsumes statement and branch coverage • If you’ve covered 100% of the paths, you’ve necessarily covered 100% of the statements and branches

28. Symbolic Execution • Replace all variables with symbols • For each path, determine its path condition as an expression in terms of the input • Any input that satisfies that path condition will cause that path to be covered • If the path condition is not satisfiable, the path is infeasible

29. White-Box Adequacy Criteria • Structural Coverage Criteria • Path/Statement/Branch coverage • Edge-Pair coverage • Prime Path coverage • Data Flow Coverage Criteria • Def-Use, All-Uses • These criteria can be used to generate a test set or to measure a test set (regardless of how it was generated)

30. Fault-Based Testing • We can’t show that the program is free of all faults but we can try to show that it is free of certain faults • Given a program P, we create a faulty version P’ • If a test input produces the correct output for P and an incorrect output for P’ then it shows that P != P’ and thus does not contain that fault

31. Assumptions • Competent Programmer Hypothesis: the program we’re testing is nearly correct and any bugs are likely to be small variations from the correct program • Coupling Effect Hypothesis: a test case that is able to find small/subtle faults is likely to find more complex/egregious ones

32. Mutation Analysis • Systematically insert faults into the code by making a single, localized change (“mutation”) • AOIU: arithmetic operator insertion (unary) • AORB: arithmetic operator replacement (binary) • COI: conditional operator insertion • COR: conditional operator replacement • ROR: relational operator replacement • For each fault, if a test case passes in the original version and fails in the mutated version, we say that the mutant is killed

33. Mutation Analysis vs. Testing • Mutation Analysis: given an existing test set, determine the percentage of mutants that it kills • Mutation Testing: given a set of mutations, derive a test set that kills all of them

34. Mutation Testing • To identify a test case that will kill a given mutant, represent P and P’ as expressions written as implications in terms of the input and output • Then find inputs that satisfy the conjunction of those expressions such that the outputs of P and P’ are different • A mutant may survive if it is not covered or if it is an equivalent mutant

35. That's it!

36. Reminder • When: Monday, October 28, 4:30pm • Where: DRL A1 • The exam is open-book but you may not use electronic devices • Solutions to past exam questions are now available in Canvas • Solutions to Homework #3 will be available Saturday around 12pm