Object-Oriented and Classical Software Engineering Eighth Edition, WCB/McGraw-Hill, 2011 Stephen R. Schach

1. Object-Oriented and Classical Software EngineeringEighth Edition, WCB/McGraw-Hill, 2011Stephen R. Schach

3. CHAPTER 6 TESTING

4. Overview • Quality issues • Non-execution-based testing • Execution-based testing • What should be tested? • Testing versus correctness proofs • Who should perform execution-based testing? • When testing stops

5. Revise Terminology • When humans make a MISTAKE, a FAULT is injected into the system • FAILURE is the observed incorrect behavior of a product as a consequence of a FAULT. • ERROR is the amount by which a result is incorrect • DEFECT is a generic term for fault. • QUALITY the extent to which a product satisfies its specifications.

6. IEEE Glossary of terms 610.12, 1990 ERROR • (1) The difference between a computed, observed, or measured value or condition and the true, specified, or theoretically correct value or condition. For example, a difference of 30 meters between a computed result and the correct result. • (2) An incorrect step, process, or data defini- tion. For example, an incorrect instruction in a computer program.

7. IEEE Glossary of terms 610.12, 1990 FAULT • A human action that produces an incorrect result. • A defect in a hardware device or component; for example, a short circuit or broken MISTAKE FAILURE • The inability of a system or component to perform its required functions within specified performance requirements.

8. IEEE Glossary of terms 610.12, 1990 • The fault tolerance discipline distinguishes between a human action (a mistake), its manifestation (a hardware or software fault), the result of the fault (a failure), and the amount by which the result is incorrect (the error).

9. Testing • There are two basic types of testing • Execution-based testing • Non-execution-based testing

10. Program testing • Testing is intended to show that a program does what it is intended to do and to discover program defects before it is put into use. • When you test software, you execute a program using artificial data. • You check the results of the test run for errors, anomalies or information about the program’s non-functional attributes. • Can reveal the presence of errors NOT their absence. • Testing is part of a more general verification and validation process, which also includes static validation techniques.

11. Program testing goals • To demonstrate to the developer and the customer that the software meets its requirements. • For custom software, this means that there should be at least one test for every requirement in the requirements document. For generic software products, it means that there should be tests for all of the system features, plus combinations of these features, that will be incorporated in the product release. • To discover situations in which the behavior of the software is incorrect, undesirable or does not conform to its specification. • Defect testing is concerned with rooting out undesirable system behavior such as system crashes, unwanted interactions with other systems, incorrect computations and data corruption.

12. Validation and defect testing • The first goal leads to validation testing • You expect the system to perform correctly using a given set of test cases that reflect the system’s expected use. • The second goal leads to defect testing • The test cases are designed to expose defects. The test cases in defect testing can be deliberately obscure and need not reflect how the system is normally used.

13. Testing process goals • Validation testing • To demonstrate to the developer and the system customer that the software meets its requirements • A successful test shows that the system operates as intended. • Defect testing • To discover faults or defects in the software where its behaviour is incorrect or not in conformance with its specification • A successful test is a test that makes the system perform incorrectly and so exposes a defect in the system. Chapter 8 Software testing

14. An input-output model of program testing

15. Verification vs validation • Verification: "Are we building the product right”. • The software should conform to its specification. • Validation: "Are we building the right product”. • The software should do what the user really requires.

16. Revise Terminology V & V • Verification • Process of determining whether a workflow has been correctly carried out • Takes place at the end of each workflow • Validation • Intensive evaluation that determines if the product as a whole satisfies its requirements • Takes place just before the product is to be delivered to the client

17. V & V confidence • Aim of V & V is to establish confidence that the system is ‘fit for purpose’. • Depends on system’s purpose, user expectations and marketing environment • Software purpose • The level of confidence depends on how critical the software is to an organisation. • User expectations • Users may have low expectations of certain kinds of software. • Marketing environment • Getting a product to market early may be more important than finding defects in the program. Chapter 8 Software testing

18. Inspections and testing • Software inspectionsConcerned with analysis of the static system representation to discover problems (static verification) • May be supplemenedtby tool-based document and code analysis. • Discussed in Chapter 15. • Software testingConcerned with exercising and observing product behaviour (dynamic verification) • The system is executed with test data and its operational behaviour is observed. Chapter 8 Software testing

19. Inspections and testing Chapter 8 Software testing

20. Software inspections • These involve people examining the source representation with the aim of discovering anomalies and defects. • Inspections not require execution of a system so may be used before implementation. • They may be applied to any representation of the system (requirements, design,configuration data, test data, etc.). • They have been shown to be an effective technique for discovering program errors. Chapter 8 Software testing

21. Advantages of inspections • During testing, errors can mask (hide) other errors. Because inspection is a static process, you don’t have to be concerned with interactions between errors. • Incomplete versions of a system can be inspected without additional costs. If a program is incomplete, then you need to develop specialized test harnesses to test the parts that are available. • As well as searching for program defects, an inspection can also consider broader quality attributes of a program, such as compliance with standards, portability and maintainability. Chapter 8 Software testing

22. Inspections and testing • Inspections and testing are complementary and not opposing verification techniques. • Both should be used during the V & V process. • Inspections can check conformance with a specification but not conformance with the customer’s real requirements. • Inspections cannot check non-functional characteristics such as performance, usability, etc. Chapter 8 Software testing

23. 6.1 Software Quality • Not “excellence” • The extent to which software satisfies its specifications • Every software professional is responsible for ensuring that his or her work is correct • Quality must be built in from the beginning

24. 6.1.1 Software Quality Assurance • The members of the SQA group must ensure that the developers are doing high-quality work • At the end of each workflow • When the product is complete • In addition, quality assurance must be applied to • The process itself • Example: Standards

25. Responsibilities of SQA Group • Development of standards to which software must conform • Establishment of monitoring procedures for ensuring compliance with standards

26. 20 1 5 12 18 9 4 14 13 1 1 6 10 8 16 15 2 7 19 17 3 6.1.2 Managerial Independence • There must be managerial independence between • The development team • The SQA group • Neither group should have power over the other Problem: SQA team finds serious defects as the deadline approaches! What to do? Who decides?

27. Managerial Independence (contd) • More senior management must decide whether to • Deliver the product on time but with faults, or • Test further and deliver the product late • The decision must take into account the interests of the client and the development organization

28. 6.2 Non-Execution-Based Testing • Testing software without running test cases • Reviewing software, carefully reading through it • Analyzing software mathematically • Underlying principles • We should not review our own work • Other people should do the review. We cannot see our own mistakes. • Group synergy • Review using a team of software professionals with a brad range of skills

29. 6.2.1 Walkthroughs • A walkthrough team consists of from four to six members • It includes representatives of • The team responsible for the current workflow • The team responsible for the next workflow • The SQA group • The client representative • The walkthrough is preceded by preparation • Lists of items • Items not understood • Items that appear to be incorrect Experienced Senior technical Staff members

30. 6.2.2 Managing Walkthroughs • The walkthrough team is chaired by the SQA representative • In a walkthrough we detect faults, not correct them • A correction produced by a committee is likely to be of low quality • The cost of a committee correction is too high • Not all items flagged are actually incorrect • A walkthrough should not last longer than 2 hours • There is no time to correct faults as well

31. Managing Walkthroughs (contd) • Walkthrough can be • Participant driven : reviewers present their lists • Document Driven : person/team responsible from the document walks through it • A walkthrough must be document-driven, rather than participant-driven • Walkthrough leader elicits questions and facilitates discussion • Verbalization leads to fault finding • A walkthrough should never be used for performance appraisal

32. 6.2.3 InspectionsAn inspection has five formal steps • Overview : • Given by a person responsible from producing the document • Document distributed at the end of the session • Preparation • Understand the document in detail • Use statistics of fault types • Inspection • One participant walks through the document • Every item must be covered • Every branch must be taken at least once • Fault finding begins • Within 1 day moderator (team leader) produces a written report • Rework • Person responsible resolves all faults and problems in the written document • Follow-up • Moderator ensures that all issues mentioned in the report are resolved • List faults that were fixed. • Clarify incorrectly flagged items

33. 6.2.3 InspectionsAn inspection has five formal steps

34. Inspections (contd) • An inspection team has four members • Moderator • A member of the team performing the current workflow • A member of the team performing the next workflow • A member of the SQA group • Special roles are played by the • Moderator • Reader • Recorder

35. Fault Statistics A fault that causes termination of the program • Faults are recorded by severity • Example: • Major or minor • Faults are recorded by fault type • Examples of design faults: • Not all specification items have been addressed • Actual and formal arguments do not correspond • In general interface and logic errors

36. Fault Statistics (contd) • For a given workflow, we compare current fault rates with those of previous products • Early warning!!!! • If disproportionate number of a certain fault type is discovered in 203 code artifacts • Check other artifacts for the same fault type • We take action if there are a disproportionate number of faults in an artifact • Redesigning from scratch is a good alternative • We carry forward fault statistics to the next workflow • We may not detect all faults of a particular type in the current inspection

37. Statistics on Inspections • IBM inspections showed up • 82% of all detected faults (1976) • 70% of all detected faults (1978) • 93% of all detected faults (1986) • Switching system • 90% decrease in the cost of detecting faults (1986) • JPL • Four major faults, 14 minor faults per 2 hours (1990) • Savings of $25,000 per inspection • The number of faults decreased exponentially by phase (1992)

38. Statistics on Inspections (contd) • Warning • Fault statistics should never be used for performance appraisal • “Killing the goose that lays the golden eggs” Another problem:

39. 6.2.4 Comparison of Inspections and Walkthroughs • Walkthrough • Two-step, informal process • Preparation • Analysis • Inspection • Five-step, formal process • Overview • Preparation • Inspection • Rework • Follow-up

40. 6.2.5 Strengths and Weaknesses of Reviews • Reviews can be effective • Faults are detected early in the process • Reviews are less effective if the process is inadequate • Large-scale software should consist of smaller, largely independent pieces • The documentation of the previous workflows has to be complete and available online

41. 6.2.6 Metrics for Inspections • Inspection rate (e.g., design pages inspected per hour) • Fault density (e.g., faults per KLOC inspected) • Fault detection rate (e.g., faults detected per hour) • Fault detection efficiency (e.g., number of major, minor faults detected per hour)

42. Metrics for Inspections (contd) • Does a 50% increase in the fault detection rate mean that • Quality has decreased? Or • The inspection process is more efficient?

43. 6.3 Execution-Based Testing • Organizations spend up to 50% of their software budget on testing • But delivered software is frequently unreliable • Dijkstra (1972) • “Program testing can be a very effective way to show the presence of bugs, but it is hopelessly inadequate for showing their absence”

44. 6.4 What Should Be Tested? • Definition of execution-based testing • “The process of inferring certain behavioral properties of the product based, in part, on the results of executing the product in a known environment with selected inputs” • This definition has troubling implications

45. 6.4 What Should Be Tested? (contd) • “Inference” • We have a fault report, the source code, and — often — nothing else • Input data => Desirable output • “Known environment” • We never can really know our environment • Is the problem due to memory? OS? • “Selected inputs” • Sometimes we cannot provide the inputs we want • Simulation is needed • Ex:” avionics software: how well can you simulate the phyical behavior outside the aircraft?

46. 6.4 What Should Be Tested? (contd) • We need to test correctness (of course), and also • Utility • Reliability • Robustness, and • Performance

47. 6.4.1 Utility • The extent to which the product meets the user’s needs • Examples: • Ease of use • Useful functions • Cost effectiveness • Compare with competitors

48. 6.4.2 Reliability • A measure of the frequency and criticality of failure • Mean time between failures • Mean time to repair • Time (and cost) to repair the results of a failure • Recover gracefully and quickly

49. 6.4.3 Robustness • A function of • The range of operating conditions • The possibility of unacceptable results with valid input • The effect of invalid input • A robust product should NOT crash when it is NOT used under valid conditions. • Ex: enter ?@#$?@#$ as student id!

50. 6.4.4 Performance • The extent to which space and time constraints are met • Response time, main mem requirements • Real-time software is characterized by hard real-time constraints • Nuclear reactor control system. • If data are lost because the system is too slow • There is no way to recover those data