1 / 30

Requirement-based Testing

Requirement-based Testing. A Presentation by Harry Sneed for the John von Neumann Society Budapest, 27 May 2009. The many Dimensions of Software Testing. Test Objects Units (Modules/ Classes ) Components Subsystems Systems Test Types Functional Testing Performance Testing

gasha
Télécharger la présentation

Requirement-based Testing

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Requirement-based Testing A Presentation by Harry Sneed forthe John von Neumann Society Budapest, 27 May 2009

  2. The many Dimensions of Software Testing Test Objects • Units (Modules/Classes) • Components • Subsystems • Systems Test Types • FunctionalTesting • Performance Testing • LoadTesting • UsabilityTesting Test Approaches • Random Testing • Creative Testing • SystematicTesting Test Bases • Code-basedtesting • Interface-basedtesting • Reference-basedtesting • Model-basedtesting • Require.-basedtesting

  3. A Test is always a Test against Something Oracle Software Units Components Systems Functional Specification (Oracle) In 1978 Bill Howden introduced the notion of a test oracle in connection with Functional Testing. It means testing the System against the Requirement Specification (functional & non-functional Requirements)

  4. Code-based Testing = Testing the Program against itself Specification Expected results Expected paths SpecifiedPaths compare Results & Paths Potential Paths Test inputs Code Test results & Actual paths Corresponds to a Unit Test (White-Box)

  5. Data-based Testing = Testing the Program against the Data Specification Expected results Specified Results Representative Values compare Data Test inputs Code Test results Interfaces/Files/Tables Interfaces/Files/Tables Interfaces/Databases Corresponds to an Integration Test (Grey-Box)

  6. Requirement-based Testing = Testing the Program against the Specs Specification Expected results Specified Results compare Specified Inputs Results Test inputs Code Test results Files/Tables/Panels Files/Tables/Panels Corresponds to a System Test (Black-Box)

  7. The Mother of all Test Tools = RXVP Requirement Evaluation & Verification Package Dr. Edward Miller from General Research Corporation Correct Correct Correct Correct FORTRAN SyntaxErrors Semantic Errors Exec Errors Struct Errors Source Program Syntax Analysis Static Analysis Execution Test Verf. Cond. Generator Program with Assert "Correct" Source Prog. Indented Program Listing+Diagnostics Graph Checking+Call Checking+Units Consistency+Mode Checking+Asserted/Actual Use+Set/Use Checking Test Results+Execution Errors+Coverage Report+Assertion Exceptions Annotated Listing+VC'S+Symbolic Execution of Expressions+Simplified VC'S Assertions taken from Specs(RSL) The idea was to put dirty software into a washing maschine and get out cleaned software .

  8. P P P P 1 2 3 4 Test Cases were extracted from the Code Test Object = Procedure or Module Test Case = Path thru Test Object 1. Path 2. Path Preconditions Postconditions Before State Entry After State 3. Path Exit 4. Path 5. Path

  9. The Budapester Testlabor • The Budapester Test Laboratory was setupby Harry Sneedand Dr. Ed Miller from Software Research Inc in cooperationwiththe SZKI andSzamok in May, 1978. This was thefirstknownattemptto outsource thetestingactivity. • The customer was Siemens, Munich, which was developing a large scale, distributedsoftwaresystemforthe German Railroad – the Integrated Transport System (ITS).

  10. Test Process in the Test Labor • Compilable Modules weredeliveredby Siemens everyweek on a magnetictape. • Sourceswerestaticallyanalyzedtoproduceflowandstructurediagrams. • Pathsweremarkedthrutheflowdiagramswithmagicmarkers. • Conditional variables wereassignedvaluestoconfirmornegateeverydecision. • Values wereassignedattest time tothetestdriverbeforeeachtestcaseexecution. • Resultswereloggedtobecheckedlateragainsttheexpectedresults.

  11. Experience with the Test Labor • In thefirst half year 128 moduleswith 60881 SPL statementspassedthroughthetestlabor. • 4378 codebranchesweretestedwith a coverageof 89.7 % by 1544 testcases. • 192 programerrorswerediscovered via dynamicanalysis. • 1396 design andcodedeficiencieswerediscovered via staticanalysis. • The costsofthequalityassuranceremainedbelow 4 DM pro statement. This was lessthan 10% ofthedevelopmentcosts.

  12. Summary of first 6 Months

  13. Resume on Module Testing • In oneyear 281 Modules with 134 K stmtsweretestedwith 3864 testcasesat a costof DM 75,- per testcase = DM 289.800 • The minimumbranchcoverage was 85% • 403 Errors werereportedat a costof DM 150,- per error = DM 60.450 • Later in thesystemtestbythecustomer, another 386 errorswereuncovered. • The intensive moduletesthaduncoveredonly 51% ofthe total errorsfound. • Thisdemonstratesthefactthatmoduletestingofthecodeagainstitselfis not sufficient. The softwareas a whole must betestedagainsttherequirements.

  14. RXVP progressed to test the systems against Requirements Nets R-Net Engine Monitoring Receive Signals ASSERT Report Signal Error Store Measure Check Measure ASSERT ASSERT Invalid Measure Valid Measure Report Engine ok send Alarm Validation Points ASSERT ASSERT Out of safe Range Safe Range

  15. Extracting Test Cases from Requirement Nets R-Net-Entry Requirement Specificationwith RSL Measuringdevice Engine Monitoring System Receive Signal invalid Measure or Valid Measureand & Report Measure Error Store Measure Check Measure Engine ok or Engine problem System Display Message Log 1. test case Report ok Alarm Nurse on Duty 2. test case Alarm Display 3. test case

  16. Action Description If Account_Balance minus the withdrawal is more than the credit limit allow the withdrawalIn case the withdrawal causes the Account_Balance to fall below the credit limit refuse the withdrawal. Extracting Test Cases from Business Rules Service Function Test Case Generation Test Case 1 2 3 4 Account_Balance minus the withdrawal is more than the credit limit Y Y N N Withdrawal causes the Account_Balance to fall below the credit limit Y N Y N Allow the withdrawal - X - - Refuse the withdrawal X - X - Each Condition is treatedseparately and combinedwith the other conditionsin the same action to givethe number of testcases = nr_conditions2 Conditional keywords arescanned.

  17. Extracting Test Cases from Use Case Descriptions One Use Case can have one or more Test case Normally, there are at least 2 Test Cases Positive outcome Negative outcome Cash Withdrawal Use Case = Invalid Card Wrong PIN Number Not enough cash Account overdrawn Cash delivered Test Cases = Use Case Paths

  18. Extracting Test Cases from Requirement Documents Objects Actions States Conditions Test Cases Functional Non-Functional Requirements Text

  19. Grammar types indicating Declarations to be tested Actions upon objects: The customer account should be deleted. The invoice should be sent within 30 days. Object states: The customer account is overdrawn. The customer is an employee. Rules: If the account is overdrawn a notice should be sent. Whenever someone fails to pay within 30 days, a reminder should be sent. Customer accounts should be deleted, if they are overdrawn more than 3 months. In case of a system crash, the recovery routine should be started.

  20. Sample Requirements Text [RF 41] Notification can be divided into action and non-action (=information) notification. [RF 42] In case of an “action needed” notification, the system provides a link to the place where the action is needed. [RF 43] Notifications are added to a notification interface, which contains information about The reason of the notification The kind of the notification The type of the notification The date/time of the notification If action needed: A link to the place where the action is needed. [RF 44] The user is able “take” a notification (locked for others), do the action and mark the notification as “finished”. If the action is finished, the user name is stored by the notification. [RF 45] If at least one provider changed his feed data e.g. changing starting time, a notification is added to the notification table (see [RF 43]) so that each user can see the notification. The notification is only added to the notification table if the data is unequal to the current event data (see also [RF 30]).

  21. Test Cases extracted from the Requirement Text TestCase-Id BaseType Requirement Relation TestType TestCase Description -------------------------------------------------------------------------------------------------------------------------------------------------------- DFI0163 Require RF_43 perform action Notifications are added to a notification interface, which contains information about: The reason of the notification The kind of the notification The type of the notification The date/time of the notification DFI0164 Require RF_43 checks rule If action is needed a link to the place where the action is needed. -------------------------------------------------------------------------------------------------------------------------------------------------------- TestCase-Id BaseType Requirement Relation TestType TestCase Description -------------------------------------------------------------------------------------------------------------------------------------------------------- DFI0165 Require RF_44 perform action The user is able “take” a notification (locked for others), do the action and mark the notification as “finished”. DFI0166 Require RF_44 checks rule If the action is finished, the user name is stored in the notification. -------------------------------------------------------------------------------------------------------------------------------------------------------- TestCase-Id BaseType Requirement Relation TestType TestCase Description -------------------------------------------------------------------------------------------------------------------------------------------------------- DFI0167 Require RF_45 checks rule If at least one provider changed his feed data changing starting time, a notification is added to the notification table (see [RF_43]) so that each user can see the notification. DFI0168 Require RF_45 checks rule The notification is only added to the notification table if the data is unequal to the current event data (see also [RF_30]). DFI0169 Require RF_45 checks rule The user choose Provider A as primary source, so the event time is the time of Provider A. DFI0170 Require RF_45 checks rule If Provider B updates his time to the time of the event, no notification is added. DFI0171 Require RF_45 checks rule If Provider B sends a time different to the event time, a notification is sent.

  22. Sample Non Functional Requirements The order entry system must be able to handle at least 1000 customer web pages at one time. A customer order must be processed within 2 seconds. The article database may contain up to 10.000 articles. The system should be able to produce as many as 25.000  bills in one billing run. A billing run should not take more than 2  hours. If the system crashes it should be back up and in operation within 15 minutes. Database backups should take place every 4 hours. Customers should be checked both by number and by password. Passwords should be rejected after 3 tries, customer number should be rejected after 2 tries.

  23. Typical Non-Functional Test Cases • 999 customer orders at one time • >1000 customer orders at one time • response time per order < 2 sec. • response time per order >2 sec. • 9.999 article records • > 10.000 article records • 24.999 bills in one billing • >25.000 bills in one billing • run time of billing < 2 hours • run time of billing >2 hours • If system crash, recovery < 15 minutes • If > 4 hours, database secured • Submit 3 passwords where 3rd password is correct • Submit 3 passwords where all 3 are incorrect • Submit 2 customer number where 2nd number is correct • Submit 2 customer number where both are incorrect 

  24. Properties of a Test Case • A Test Case: • has a unique identifier • serves a purpose • refers to a Testobject • refers to a requirement or use case • has a precondition state • has a postcondition state • has a set of required input parameters • has a set of expected results • has a predecessor test case • has a given environment • has an owner • has a status • has a date of execution An international standard is required for the minimum Properties of test cases!!!

  25. Functional Test Case Specification TestCase ID: IC-Order-7 Test Case – Purpose: to test a fulfilled order Test-Object: Customer – Order Precondition: Ordered Article is on stock. Article. Amount is > Order.Position.Amount (Article. Amount - Order. Position.Amount ) > minimum – Amount Postcondition: Article.Amount = a Pre.Article.Amount – Order.Position.Amount No supply order Inputs: Order.Cust-Nr = 200100 Customer.Credit = ’A’ Order.Position-Nr = 2 Order.Position.Art – Nr = 4711 Order.Position.Amount = 3 Article.Amount = 10 Minimum – Amount = 7 Results: Text-Msg = „Article order fulfilled” Environment: MS – Internet, Windows – XP Usecase tested: Customer.Order.Processing Status: tested Date: 29. 8. 2005.

  26. From Requirements via Test Cases to Test Scripts Tester enhances & refines Test Cases Text Analysis Test Cases Repository Loader Test Case DB Text Analyzer generates Test Cases Test Test Scripts Repository Unloader Test Script Generator generates Test Scripts for Test Automation Tester executes Test Cases

  27. Object List TextAnal = Tool for extracting Test Cases from English and German Language Documents Text Pre Processor Text Processor TxtAnal TxtScan Object Model Test Cases Rule Violations Spec Metrics Repository Entries • Objects • Use Cases • - Actions • - States • - Rules Text Excel XML Format Rules Rupp Rules Size Quantity Complexity • Sections • Use Cases • - Objects • - Actions • - States • - Rules

  28. Requirements Analysis Demo • Bet&WinRequirementDocument • Setting Key Words • Scanning for Objects • AnalyzingtheDocument • Extracting Test Cases • CollectingRequirementMetrics • Documenting Cross References • Generating a test design

  29. Experience with Requirement-based Testing of the Public Internet Portal for the state of Saxony Egovernment Web Site with threeapplicationsand circa 33 K Java Stmts 1844 Test cases were specified of which 1272 were generated 1495 Test cases were executed = 81% Test Case coverage 352 Test cases discovered at least one error = 24% Hit rate 450 Errors were reported by testers = 13 per KDSI 56 Errors were reported by users after release = 11% of all errors Systemtest discovered 89% of all reported errors = test efficiency Systemtest lasted 4 months from January to May 2005 392 Test days were booked against the project = 3.8 Test cases per Testertag = 1.1 Errors reported per tester day Customer was satisfied with the test result.

  30. Conclusions • Test methodshave not reallyimprovedsincethe 1970s. • Weare still usingthebasicapproachesintroduced back then. • Test toolshavehoweverimprovedsignificantly. • Today therearemoreandbettertestingtools. • The future lies in testautomation. • Onlythroughautomationcanwereallyreducetestcostsandincreasethetest ROI. • Test ROI = (TestBenefits – TestCosts) / TestCosts

More Related