Components of UAT

1. User Acceptance Testing Components of UAT 1 Introduction 2 The Need 3 Components a) Key elements 4 The UAT team 5 Monitoring 6 Measuring

2. Key components of UAT 1 2 3 User Acceptance Testing 6 4 5 Contents Planning & documenting User Acceptance Criteria Setting up and maintaining the UAT environment Designing User Acceptance Tests Using static test techniques for UAT

3. Requirements are key to the success of the project Requirement faults can be very expensive to fix Reasons for project failures Only 26% of software projects succeed Standish Group, CHAOS Report,1998 • Lack of user involvement • Incomplete requirements & specifications • Changing requirements & specifications

4. 1000 100 10 1 Req Des Test Use Cost of fixing faults “The results show as much as a 1000:1 cost ratio between finding faults in the requirements and maintenance stages of the software lifecycle” Average cost ratio 14:1 Grady 1989

5. What is a requirement? • Requirements are capabilities and objectives to which the system you are building must conform (IEEE 830) • requirements-based testing: Designing tests based on objectives derived from requirements for the software component (e.g., tests that exercise specific functions or probe the non-functional constraints such as performance or security). (BS7925-1)

6. Warning : The more documents we have - the more likely we are to miss potential tests Where are requirements found? • Product specifications • Marketing documents • Design documents • Business/Statutory rules • Functional specifications • Memos/e-mail • Prototypes • People’s heads!

7. Idea Feature The problem Detailed Requirement Requirement process

8. The challenges for UAT • No detailed requirements/varying levels of detail • Requests come from many sources • Requests (and Requirements) are difficult to express in words • Features/Requirements have different priorities - not usually specified • Large numbers of requirements are more difficult to manage and control

9. How would you test this spec? • A computer program plays chess with one user. It displays the board and the pieces on the screen. Moves are made by dragging pieces.

10. Requirement testing traceability matrix

11. Acceptance criteria checklist For each project

12. What is quality? “Quality, like beauty, is in the eye of the beholder.” A system is a combination of functions and attributes. Attributes include qualities such as reliability, usability, maintainability, etc. To know when you have it, you have to test for it.To test it, you have to measure it.How do you measure these “unmeasurables”? Gilb’s Law: “Anything can be made measurable in a way which is superior to not measuring it at all” Tom Gilb, Principles of Software Engineering Management,Addison-Wesley, 1988

13. new clients account updates queries work_capacity likeability uncertainty mis-use help Usability completeness ease of access consistency accuracy manuals on-line help desk Hierarchies of objectives • attributes may be expressed hierarchically • a name tag is equivalent to a set of sub-attributes • any number of sub-levels is OK • achieving all the sub-attributes = achieving the high level attribute

14. Quality attribute template • main entries: • concept: description of attribute • scale: ‘what’ is being measured • test: ‘how’ it is to be measured • must do: worst measure that is acceptable • plan: measure to be achieved • other entries (for comparison purposes) • now: current situation • best: state of the art

15. Quality of work_capacity (performance)

16. Quality of on-line help (documentation)

17. Exactness, accuracy, correctness • exact:precise description or definition (e.g. specific numbers) • correct:reflecting the truth, true state of things • accurate:correct and “reasonably” exact, but not perfect • using exact numbers does not imply exact knowledge • quality attribute technique is accurate but not exact • numbers gives expression to ideas behind woolly statements It is better to know about the state of your knowledge Exercise

18. Key components of UAT 1 2 3 User Acceptance Testing 6 4 5 Contents Planning & documenting User Acceptance Criteria Setting up and maintaining the UAT environment Designing User Acceptance Tests Using static test techniques for UAT

19. The environment Is the environment we create a good breeding ground for our bugs to live in?

20. The technical environment HARDWARE TESTWARE NETWORK SOFTWARE

21. The technical environment • software • bugs survive when we have poor ‘de-bugging’! (approximately 2.4 bugs per 1000 source statements)* (1 in 7 faults are not fixed correctly) ** • testware • bugs survive if we have poor quality tests • it’s difficult to distinguish between test bugs and software bugs * Boris Beizer : Software Testing Techniques ** Michael Fagan

22. The technical environment • hardware • bugs survive when we ignore the hardware architecture • networks • bugs survive when we fail to acknowledge the network’s infrastructure

23. How do the bugs survive? • No control • configuration management problems • No maintenance • out of date and unmanageable • No representative data • realistic • size • poor fault tracking • closing faults before they are fixed!

24. Quote... “if the environment is wrong, out of date or not representative - testing can’t succeed” Bob Bartlett SIGIST 16th July 1999

25. The office environment • Parkinson’s Law • “work expands to fill the time allocated for it” • People • poor work environment for dev = more bugs • poor work environment for test = fewer bugs found • Noise • what difference does a little noise make? • Interruptions • short term memory problems

26. Problem bottlenecks could be created, preventing further testing being carried out development do not use a realistic environment Solution monitor amount of ‘down time’ for your environment and raise with management assist development in creating/coping a representative environment Potential problems & solutions Development using the UAT environment

27. Problem violation of the Data Protection Act too large and un-manageable Solution introduce standards for using ‘live data’ take a copy of the ‘live’ and change construct a ‘life-like’ environment (if possible) Potential problems & solutions Over reliance on ‘live data’

28. Problem not a full/representative data set Solution introduce standards for using ‘live data’ take a copy of the ‘live’ and change construct a ‘life-like’ environment (if possible) Potential problems & solutions Under reliance on ‘live data’

29. Problem reliance on development to perform upgrades UAT testers waiting for system manual intensive Solution develop skills within the UAT team less optimistic! better planning monitor & analyse process time to consider automation Potential problems & solutions Maintenance of environment is a long process

30. UAT environment checklist • UAT environment strategy • identify skills required within the team • consider automation • balance between ‘self-contained’ and ‘baselined’ tests • regular reviews of the environment • security considerations • independent environment desirable

31. Key components of UAT 1 2 3 User Acceptance Testing 6 4 5 Contents Planning & documenting User Acceptance Criteria Setting up and maintaining the UAT environment Designing User Acceptance Tests Using static test techniques for UAT

32. The need for techniques • Exhaustive testing (use of all possible inputs and conditions) is impossible • must use a subset of all possible test cases • must have high probability of detecting faults • Need thought processes that help us select test cases more intelligently • test case design techniques are such thought processes

33. What is a testing technique? • a procedure for selecting or designing tests • based on a structural or functional model of the software • successful at finding faults • 'best' practice • a way of deriving good test cases • a way of objectively measuring a test effort Testing should be rigorous, thorough and systematic

34. Three types of technique Static (non-execution) • examination of documentation,source code listings, etc. Behavioural (Black Box) • based on behaviour /functionality of software Structural (White Box) • based on structureof software

35. etc. Reviews Behavioural Static Analysis Inspection Non-functional Functional Structural Walkthroughs etc. Desk-checking etc. Equivalence Partitioning Usability Control Flow Data Flow Performance Boundary Value Analysis etc. etc. Statement Symbolic Execution Decision Tables Arcs Branch Experience Based Condition LCSAJ Definition -Use State Transition Cond Comb Scenario / Use Case Testing Test techniques hierarchy Dynamic Static

36. Advantages of techniques for UAT • People: standardisation • ease of maintenance of tests • Effective testing: • focus attention on specific types of test • know you're testing the right thing • Efficient testing: • avoids duplication of tests • avoids wasted test effort Using techniques makes testing much more effective

37. Which technique is best? • no one technique is best • a diversity of techniques is best • the best set of techniques for any one situation depends on things such as: • level of testing • technology of software • criticality of software • environment • experience and knowledge available The best single technique isno single technique

38. Key components of UAT 1 2 3 User Acceptance Testing 6 4 5 Contents Planning & documenting User Acceptance Criteria Setting up and maintaining the UAT environment Defining User Acceptance Tests Using static test techniques for UAT

39. People techniques Informal Review (undocumented) • useful and cheap, helpful if nothing else done Peer Review • includes peer and technical experts, normally documented, fault-finding. Often subjective. Walkthrough • lead by author, all gain understanding of work Inspection: • most formal, uses sources, rules and checklists, entry and exit criteria. Leader must be trained & certified, metrics required.

40. Benefits of reviews • Development productivity improvement • Reduced development timescales • Reduced testing time and cost • Lifetime cost reductions • Reduced fault levels • Improved customer relations • etc.

41. Reviews in general • Objectives / goals • V&V, consensus, process improvement • Activities • planning, preparation, follow-up, metrics • Roles and responsibilities • Leader, Author, Inspectors/Reviewers, Editor • Deliverables • Edits, Requests, Improvement Ideas, Metrics

42. Review pitfalls • lack of training in the technique (especially Inspection, the most formal) • lack of or quality of documentation - what is being reviewed / Inspected • Lack of management support - “lip service” - want them done, but don’t allow time for them to happen in project schedules • Failure to improve processes (gets disheartening just getting better at finding the same thing over again)

43. Reviews are cost-effective • 10 times reduction in faults reaching test, testing cost reduced by 50% to 80% • Freedman & Weinberg, Handbook of Walkthroughs, Inspections & Technical Reviews • reduce faults by a factor of 10 • Yourdon, Structured Walkthroughs • 25% reduction in schedules, remove 80% - 95% of faults at each stage, 28 times reduction in maintenance cost, many others • Gilb & Graham, Software Inspection

44. Costs of reviews • Rough guide: 5%-15% of development effort • half day a week is 10% • Effort required for reviews • planning (by leader / moderator) • preparation / self-study checking • meeting • fixing / editing / follow-up • recording & analysis of statistics / metrics • process improvement (should!)

45. Accept. Test Requirements System Test Functions Integration T Design Unit Test Code What to review / Inspect? Tests Tests Tests Tests

46. Other static techniques to consider • Brainstorming • Workshops • Interviews • Questionnaires • Web based reviews • Storyboard • Prototype • Use Cases

47. Key components of UAT 1 2 3 User Acceptance Testing 6 4 5 Summary: Key Points the importance of clear, unambiguous acceptance criteria the test environment a breeding ground for our bugs? using systematic and experience based techniques when designing our acceptance tests reviews are effective throughout the lifecycle

48. Software Testing Essentials EXERCISE 3