INFO 631 Prof . Glenn Booker

INFO 631Prof. Glenn Booker Week 4 – Testing and Object-Oriented Metrics INFO631 Week 4

Software Testing Overview • Software testing is the process of executing a software system to determine if it matches its specification and executes correctly in its intended environment • Requires running an executable program with real or simulated inputs to determine its response INFO631 Week 4

Software Testing Overview • Incorrect software behavior leads to a failure • Produces incorrect outputs • Code executes too slowly • Code uses too much memory • Failures are caused by faults INFO631 Week 4

Testing Prerequisites • Code reviews – uncompiled source code is read and analyzed statically • May use Formal methods to provide proof of correctness (Z, Larch) • Specification – defines correct behavior so that incorrect behavior is easier to identify • If you don’t know what it should do, when is it broken? INFO631 Week 4

Software Testing by Phase • Unit testing tests individual software components (modules) • To test, need to define the input domain for unit in question and ignore rest of system • Integration (or component) testing tests combinations of units that have each passed previous unit testing • Focus is on a larger subset of domain, representing communication between units INFO631 Week 4

Software Testing by Phase • Regression testing tests code containing bug fixes or new features to ensure original functions still work (may be used during development and maintenance) • System testing tests a collection of components which are regarded as a deliverable product INFO631 Week 4

# test cases Time Testing S Curve • A common method for tracking test cases is to count the total number of test cases planned over time • Often looks like an s-shaped curve INFO631 Week 4

Testing S Curve • Then, plot the number of test cases attempted, and successfully completed; all also over time • Main purpose is to help estimate test completion INFO631 Week 4

Defect Arrival • As a project progresses, defects are identified in documents, code, etc. • A typical mechanism for tracking these defects is a Problem Tracking Report (PTR) • The rate of testing, and quality of the code, determine the rate at which new PTRs will be discovered INFO631 Week 4

Defect Arrival • Generally want more PTRs found early in testing, so there are few defects found near an intended release or product shipment • Can track trends in PTR discovery across several releases • Also key to track backlog number of PTRs remaining to be fixed INFO631 Week 4

Product Size • Product size is critical to estimation of development effort and schedule, making it the most common basic measurement of the product • Can track the number of lines of code released over time (so each key event is a release date and its size) INFO631 Week 4

Other Testing Measures • Measures during testing could also include • Percent of CPU utilization • Number of system crashes and hangs per week • Average time between crashes or hangs (text refers to unplanned IPL (Initial Program Load = booting)) INFO631 Week 4

Severity of Problems • As problems are identified through PTRs, their severity or criticality is important to assess • Resolving PTRs which cause complete system failure or severe trouble can preempt all other activity • Release criteria can include specific numbers of PTRs by severity INFO631 Week 4

Testing Process Measurement • Often helps to use calendar time to measure process-related metrics, instead of life cycle phase • Time-based measurements are best presented relative to the release date, to help focus attention on the goal • Qualify measurements as good/bad or red/yellow/green INFO631 Week 4

Defect Cause Classification • Look for patterns of defect arrival, and what kind of defects are identified • Patterns such as by testing phase (unit, component, regression, or system test) • Kind of defect, such as per the ODC or some other classification scheme INFO631 Week 4

Measures That Work • No set of testing measures are “The” correct set, so monitor the effectiveness of your measures in providing meaningful information • Change measures slowly, to allow for significant data to be collected for each set INFO631 Week 4

Measures That Work • Likewise, specific measures and goals for them need to be defined: • In order to evaluate a vendor-developed product, or • Determine when a release is ready to go INFO631 Week 4

Testing Test Case Type • Testing may be done for structural and/or functional test cases • Structural testing focuses on correctness of the code structure • Functional testing focuses on the software meeting its requirements • Both types are generally used; some more in certain life cycle phases INFO631 Week 4

Structural Testing • Structural testing – inputs based solely on structure of the source code or its data structures • Also called “code based testing,” or “white box testing” • Structural testing may focus on • Control flow – which module is currently being run? • Data flow – software moves data from one location to another INFO631 Week 4

Structural Test Cases • Criteria (measures) for structural testing completeness • Has testing covered all common programming errors? • Has all source code been exercised? • Statements, branches, independent paths • Has all internal data been initialized and used? INFO631 Week 4

Structural Test Cases • Example: Control flow based • Draw control flow graph for code • Determine its cyclomatic complexity • Determine basis set of linearly independent paths (also used for regression) • Prepare test cases that will force execution of each path in basis set INFO631 Week 4

Predicate Testing (Structural) • What can go wrong with predicates: • Boolean operator error (not, and, or) • Boolean variable error (wrong variable) • Boolean parentheses error (unmatched) • Relational operator error (<, <, etc.) • Arithmetic expression error (+, -, etc.) INFO631 Week 4

Predicate Testing (Structural) • If (Cond1 or cond2) then … • Cover all possible T/F conditions; e.g. if there are two T/F conditions, check: • Cond1 true, cond2 true • Cond1 true, cond2 false • Cond1 false, cond2 true • Cond1 false, cond2 false INFO631 Week 4

Loop Testing (Structural) • Simple loop: Let N be the maximum number of allowable passes through loop • Define test cases which will: • Skip loop entirely • One pass through loop • Two passes through loop • Some typical middle number of passes through loop • N-1 passes through loop • N passes through loop • N+1 passes through loop INFO631 Week 4

Functional Testing • Functional testing – test cases are selected without regard to source code structure • Based on attributes of the specification or operational environment • Also called “black box testing,” “behavioral testing,” or “specification based testing” INFO631 Week 4

Functional Test Selection • Criteria for functional testing completeness: • Have ways that software can fail been thought through? • Have test cases been selected that show it doesn’t fail? • Have all of the inputs been applied? • Have all possible software states been explored? • Have all user scenarios been run? INFO631 Week 4

Functional Test Selection • Design tests to answer the following questions: • How is functional validity tested? • What classes of inputs will make good test cases? • Is the system sensitive to certain input values? • How are boundaries of a data class isolated? INFO631 Week 4

Functional Test Selection • What data rates and data volume can the system tolerate? • What effect will specific combinations of data have on system operation? INFO631 Week 4

Unit Testing Strategy • Focus testing effort on small unit of design • Module consists of a Function, Subroutine, Subprogram, Class or Package • Often unit testing is white box (structural) oriented INFO631 Week 4

Unit Testing - Strategy • Develop test cases to check • Interfaces • Boundary conditions • Local data structures • Independent paths • Error handling paths • Measure percent of each covered during testing INFO631 Week 4

Driver Module Test Cases Stub1 Stub2 … Unit Testing Procedures • Module is generally invoked by or invokes other modules • Calls or is called by subprograms • Uses or is used by classes, packages INFO631 Week 4

Test Module Test Module Data Stub Stub Display Trace Message Display a Passed Parameter The Idea of Stubs Stubs are the start of a module, used as a placeholder INFO631 Week 4

Test Module Test Module Data Data Data Stub Stub Return value from table or external file Do table search for input parameter, return associated output parameter The Idea of Stubs INFO631 Week 4

Integration Testing – Top Down Integration • Typical Steps (see next slide) • Main module (m1) used as test driver • Use stubs for subordinates (m2, m3, m4) • Replace subordinate stubs one at a time with actual modules • Tests are conducted as each module is integrated • Continue to replace stubs with modules and test • Regression test to ensure no new errors introduced INFO631 Week 4

m1 m2 m3 m4 m5 m6 m7 … m8 Integration Testing – Top Down Integration Code modules INFO631 Week 4

Integration Testing –Bottom Up Integration • Typical Steps (see next slide) • Low level modules are combined into builds (clusters) • Driver (higher level stubs) are written to control test case I/O (input and output) • Cluster is tested • Drivers are removed, clusters are combined moving upward INFO631 Week 4

A B D C Driver Driver Driver Driver Data Data Data Data Test Module Test Module Test Module Test Module Invoke Subordinate Send param from table or external file Display param Combo of drivers B and C Integration Testing–Bottom Up Integration First test A, then B & C separately, then integrate into D INFO631 Week 4

Regression Testing • Regression testing occurs when • Software faults are repaired, and/or • Software is enhanced • Given some version N of software • Collection of faults to be repaired are bundled with a collection of enhancements to be made • This modified software constitutes version N+1 INFO631 Week 4

Regression Testing • How much retesting is required of the N+1 version to determine if faults repaired or no new faults are added in the enhanced portions of software? • For faults which are repaired • Can fix fault that was reported • Fail to fix the fault • Fix fault, but injected a new fault • Failed to fix fault, and injected a new fault INFO631 Week 4

Adequacy of Test Coverage • Fault seeding – insert faults intentionally into the code • Based on capture/recapture models in biology • Apply test cases, discover what fraction of seeded faults are discovered • Assume that the fraction of unseeded faults are in the same proportion • Given numbers of seeded faults vs. the seeded faults uncovered, and the number of unseeded faults found, can then estimate the number of unseeded faults remaining. INFO631 Week 4

Object Oriented Metrics • Many of the traditional software metrics can also be applied to object-oriented software • Lines of code • Cyclomatic complexity • Percent of comments • In addition, several sets of metrics specifically for OO development exist INFO631 Week 4

OO Counting Metrics • Often counting basic characteristics is helpful, such as • Number of classes • Average number of LOC per class • Average number of methods per class • Average number of LOC per method INFO631 Week 4

OO Productivity • Productivity during object oriented development is measured much like any other code creation process • Typical measures are LOC/hour, function points per person-month (PM), or number of classes per person-year (PY) • Any kind of size measure per unit time could be used INFO631 Week 4

OO Quality • Quality measures can include the number of defects per class, but defects per KLOC are much more commonly accepted • Quality measures are typically used to identify problem areas of code, much like procedural quality measures INFO631 Week 4

Object Oriented Metrics • The six “CK” metrics are a common foundation for OO measurement • Weighted methods per class (WMC) • Response for a class (RFC) • Lack of cohesion of methods (LCOM) • Coupling between objects (CBO) • Depth of inheritance tree (DIT) • Number of children (NOC) INFO631 Week 4

Sample Class Diagram INFO631 Week 4

Weighted methods per class (WMC) • WMC is defined for each class • WMC could be defined two ways • The number of methods (messages) for each class • E.g. class Triangle has WMC=2 (for methods area and perimeter), but Quadrilateral only has WMC=1 (perimeter) • The sum of complexities for all methods for each class • Code specific INFO631 Week 4

Response for a class (RFC) • RFC is the sum of how many ways all methods may be used by the class to implement all messages • Class Polygon has five objects which may invoke perimeter differently • Class Triangle has three objects which may invoke area differently • Hence RFC for Polygon is 5+3=8 INFO631 Week 4

Lack of cohesion of methods (LCOM) • Measures how isolated classes are by how often they use the same attributes • One method: • Find percent of methods which use a given attribute • Average those percentages for the class • Subtract from 100% • Low resulting value means higher cohesion (a good thing usually) INFO631 Week 4

Lack of cohesion of methods (LCOM) • Another method: • Look for pairs of methods which share an attribute • Count how many different groups of pairs were found • Low number means less cohesion (bad) INFO631 Week 4

INFO 631 Prof . Glenn Booker