How to test Level-of-Service Requirements

1. How to test Level-of-Service Requirements CS 577b Software Engineering II Supannika Koolmanojwong

2. Level of Service Requirements Non-functional requirements Performance Requirements Quality Attributes -ilities Non-behavioral requirements Quality goals

3. Examples of LOS requirements • Security • Usability • Testability • Maintainability • Scalability • Backup • Compliance • Reliability • Resource constraints (speed, memory, bandwidth) • Extensibilities • Fault Tolerance • Interoperability • Portability • Stability • Response Time • Disaster Recovery • Escrow (ensure maintenance of SW) • Privacy

4. Non-functional requirements • Can be categorized into two main categories • Execution qualities - observable at run time • Evolution qualities – embodied in the static structure of the software system Security, usability Maintainability, scalability http://en.wikipedia.org/wiki/Non-functional_requirement

5. How to test your LOS requirements • Start with how to write your LOS requirements • Make it SMART • Specific – clear, consistent and concise • Measurable • Attainable – achievable given what they know • Realizable – can be done given the constraints • Traceable – need to understand why we need this http://www.win.tue.nl/~wstomv/edu/2ip30/references/smart-requirements.pdf

6. Specific • “The mission planning system shall support several planning environments for generating the mission plan” • Specific ? • What do you mean by several? • Have you defined “planning environment” and “mission plan”?

7. Specific • Avoid obviously, clearly, certainly • Avoid some, several, many • Avoid etc., so on, such as • When numbers are specified, identify the units • Ensure pronouns are clearly referenced • When module A calls B, its message history file is updated • Use glossary

8. Measurable • “The system shall produce a plan optimized for time.” • The only way to measure is to compare with an absolute optimum, which may not be available. • Should have a fixed performance against a predefined set of test cases for which the absolute optimum is known. • Think about • Can this requirement be verified? • Can the test be conducted on one site? • Can this requirement be tested in isolation?

9. Attainable • Possible physically for the system to exhibit that requirements • “The system shall be 100% reliable and 100% available” • “The system shall have a minimum response to a query of 1 second irrespective of system load” • Must be very expensive to meet these requirements or will never be accepted • Think • Is there a theoretical solution to the problem? • Done before? Any feasibility study? • Any physical constraints ? Memory size

10. Realizable • Achievable given what is known about the constraints • 99% reliability • But limited budget, short schedule • Think • Sufficient resources / time / budget? • Afford to manage them? • Use status, “desirable” at the beginning, after prototyping / has evidence, then update the status

11. Traceable • Traceable to/from concepts / spec / design / implementation / test • Need to know why we need this; business justification • Verify that it is implemented

12. Testing LOS requirements http://scaledagileframework.com/nonfunctional-requirements/

13. Check list for LOS requirements • Security  • Login requirements - access levels, CRUD levels  • Password requirements - length, special characters, expiry, recycling policies  • Inactivity timeouts – durations, actions • Audit  • Audited elements – what business elements will be audited?  • Audited fields – which data fields will be audited?  • Audit file characteristics - before image, after image, user and time stamp, etc • Performance  • Response times - application loading, screen open and refresh times, etc  • Processing times – functions, calculations, imports, exports  • Query and Reporting times – initial loads and subsequent loads  http://leadinganswers.typepad.com/leading_answers/2009/03/nonfunctional-requirements-minimal-checklist.html

14. Check list for LOS requirements • Capacity  • Throughput – how many transactions per hour does the system need to be able to handle?  • Storage – how much data does the system need to be able to store?  • Year-on-year growth requirements • Availability  • Hours of operation – when is it available? Consider weekends, holidays, maintenance times, etc  • Locations of operation – where should it be available from, what are the connection requirements? • Reliability  • Mean Time Between Failures – What is the acceptable threshold for down-time? e.g. one a year, 4,000 hours  • Mean Time To Recovery – if broken, how much time is available to get the system back up again? http://leadinganswers.typepad.com/leading_answers/2009/03/nonfunctional-requirements-minimal-checklist.html

15. Integrity  • Fault trapping (I/O) – how to handle electronic interface failures, etc  • Bad data trapping - data imports, flag-and-continue or stop the import policies, etc  • Data integrity – referential integrity in database tables and interfaces  • Image compression and decompression standards • Recovery  • Recovery process – how do recoveries work, what is the process?  • Recovery time scales – how quickly should a recovery take to perform?  • Backup frequencies – how often is the transaction data, set-up data, and system (code) backed-up?  • Backup generations - what are the requirements for restoring to previous instance(s)? • Compatibility  • Compatibility with shared applications – What other systems does it need to talk to?  • Compatibility with 3rd party applications – What other systems does it have to live with amicably?  • Compatibility on different operating systems – What does it have to be able to run on?  • Compatibility on different platforms – What are the hardware platforms it needs to work on? http://leadinganswers.typepad.com/leading_answers/2009/03/nonfunctional-requirements-minimal-checklist.html

16. Check list for LOS requirements • Maintainability  • Conformance to architecture standards – What are the standards it needs to conform to or have exclusions from?  • Conformance to design standards – What design standards must be adhered to or exclusions created?  • Conformance to coding standards – What coding standards must be adhered to or exclusions created? • Usability  • Look and feel standards - screen element density, layout and flow, colours, UI metaphors, keyboard shortcuts  • Internationalization / localization requirements – languages, spellings, keyboards, paper sizes, etc • Documentation  • Required documentation items and audiences for each item

17. http://www.ibm.com/developerworks/webservices/library/ws-soa-nonfunctional/index.htmlhttp://www.ibm.com/developerworks/webservices/library/ws-soa-nonfunctional/index.html

18. Compatibility Testing • Hardware platform (IBM 360, HP 9000) • Peripherals (printer, DVD) • OS (Linux, Windows) • DB (Oracle, SQL server, MySQL) • Browser (Chrome, Firefox) • Carrier (Verizon, Sprint) • Hardware (different phones)

19. Compliance / Conformance Testing • Conform to standards • Often performed by external organization • Local / global standards

20. Soak / Endurance / Load / Stress Testing • Test beyond the maximum ratings for a longer period of time • Test for robustness, availability, error handling, memory leaks, denial of service, response time, throughput • Spike test – suddenly increase the number of load generated

21. Recovery Testing • How well an application is able to recover from crashes, hardware failures • Forced failure of the software • suddenly restart the computer, unplug the network cable

22. Security Testing • 6 basic security concepts that need to be covered • Confidentiality • Protecting the disclosure of information • Integrity • Correctness, as intended • Authentication • Confirming the identify, correct label • Availability • Ready when expected • Authorization • Access control, right service to right person • Non-repudiation • Ensure that the transferred mesg has been sent and received by the parties claiming to have sent and received. Can not deny that the party was the one who sent. http://en.wikipedia.org/wiki/Security_testing

23. Monkey Test • Unit test with no specific test • Random strings / actions

24. Usability Testing • Measure ease of use • Goals • Learnability – how easy to accomplish a task first time • Efficiency – how much time, how many steps • Memorability – remember afterwards • Errors – how many mistakes • Satisfaction – confident, stressed ? Recommend to a friend ? • Task first, design second

25. How to improve usability • Use representative users • Ask users to perform representative tasks • Observe and listen • Testing 5 users is typically enough

26. Usability testing for the website • What your visitors think the first time they see your site. • Whether your visitors "get" what your site is about. • What visitors think about your site's look and feel. • Whether your visitors can read the content easily and understand what they see. • Whether your visitors can perform your set "tasks" easily, and if not, why not. • Where visitors get "stuck"... and click away. • What visitors think about the product, pricing and offer. http://www.usabilitytesting.tv/articles/9-what-is-usability-testing

27. More usability checklists for website • http://www.usereffect.com/topic/25-point-website-usability-checklist • http://www.pantos.org/atw/35317.html

28. Other Methods • Cognitive Walkthrough • Evaluating the user interaction of a prototype / final product. • Core-Capability Drivethrough • Benchmarking • Create/ Use standardized test materials for a specific type of design. • E.g. time to do core task, time to fix errors, time to learn applications http://en.wikipedia.org/wiki/Usability_requirements

29. Persona • Need to know exactly who your customers are • Help product developers and product designers understand how to optimize the product around the ways the user will want to use the product • As detail as possible in all aspects that are related to your system • Generally 10 persona

30. User Stories vs Persona • “As a mom, I want to take and share videos of the kids so that I can share important moments with grandparents, aunts, uncles, and friends” • User Stories – Left brain, focus on functionality • Persona – Right brain, focus on outcomes http://www.agilemarketing.net/user-stories-agile-marketing-part-2/

31. User Stories vs Persona http://www.agilemarketing.net/agile-marketing-user-stories/

32. http://www.agilemarketing.net/agile-marketing-user-stories/

33. http://becubed.me/2007/06/08/download-an-example-persona-used-in-the-design-of-a-web-application/http://becubed.me/2007/06/08/download-an-example-persona-used-in-the-design-of-a-web-application/ http://blog.ocad.ca/wordpress/gdes1b26-fw2010-19/2011/02/18/exercise-eight-ocad-student-personas/

35. http://www.dylanux.com/case-study-1.shtml#.URVOcaV9Jbo

36. Finding the users • Building a hypothesis • Verification • Finding Patterns • Constructing personas • Body • Psyche • Background • Emotions and attitudes, • Personal traits • Defining Situations • Validation and Buy-in • Dissemination of Knowledge • Creating Scenarios • Ongoing Development http://www.hceye.org/HCInsight-Nielsen.htm

37. Workshop • Identify possible roles & their description • Pick one role and define the persona • Picture • Basic Demographic • Age, occupation, hometown, marital status • Attributes • Description • User Scenario • Goals & Aspirations • One wild card (anything that would be good for your project, such as brands, device usage) • Present in front of the class • For off-campus student, submit as in-class quiz