Test process and test process development

1. Testprocess and testprocessdevelopment Jussi Kasurinen, Software Engineering Laboratory

2. What were MASTO and ESPA? • ESPA was a 3-year academic project conducted by LUT and Aalto University, funded by Tekes and a group of companies and organizations. • The focus on the project was on software testing in all different organizational levels. • MASTO at LUT was the part focusing on test process and organizational aspects. • SQUID at Aalto was the part focusing on the testing work and project-level aspects. Software-producing organization Organizational level, one company has one set of these Project-level work, one company may have several of these Pof these Projecese Jussi Kasurinen, Software Engineering Lab. LUT Information Technology

3. Test Standard and MASTO Publications Journal article “Software Test Automation in Practice: Empirical Observations” Conference paper “How Test Organizations Adopt New Testing Practices and Methods?” Conference paper: “A Study on Agility and Testing Processes in Software Organizations” Covered in article “Software Test Automation in Practice: Empirical Observations” Conference paper: “Test Case Selection and Prioritization: Risk or Design-based?” Conference paper “A Self-Assessment Framework for Finding Improvement Objectives with ISO/IEC 29119 Test Standard” Conference paper: “Exploring Quality Concepts in Software Organizations” Conference paper “Analysis of Problems in Testing Practices” Test strategy constructs (and layers)

4. Self-assessment framework • Combination of Test Improvement Model (TIM) (Ericson et al. 1997) maturity levels and ISO/IEC 29119 processes. • Two results: • General maturity and conformance with the standard model. • Process improvement objectives to develop test process. • Here is an example of the self-assessment results from one case organization: Maturity levels from TIM Processes from ISO/IEC 29119 General maturity/conformance estimation Individual assessment of each process area, development ideas Jussi Kasurinen, Software Engineering Lab. LUT Information Technology

5. Other Major implications for industry and research • 1. The test strategy establishes a framework for testing work at the project level. The following hypotheses promote the development of a test strategy to address the factors important for the testing work: • The development of a test plan can be characterised as applying two stereotypical approaches. The first approach promotes a design-based approach, in which the testing work focuses on validating the object under testing. The second approach promotes the risk-based approach, where the testing work focuses on minimising the potential losses caused by the object under testing. • There is only a loose association between development methods and test processes. The applied development method does not restrict the practical testing work to any large degree, or require compromises in the test process definitions. • The most important aspects in the test process which have positive association with the perceived end-product quality are trust between the customer and producer, a test process which conforms to the self-optimising processes similarly as defined in the ISO/IEC 29119 standard and the communication of the preferred quality characteristics to all of the process stakeholders. • In the test process resourcing, the organisations have an average of 70% of their self-defined ―optimal amount of resources and dedicate on average 27% of the total project effort to testing. Based on the study results presented in the literature and the survey data, the test process hindrances are also based on the efficiency factors and test management, in addition to simple resourcing issues. Jussi Kasurinen, Software Engineering Lab. LUT Information Technology

6. Other Major implications for industry and research • 2. The ISO/IEC 29119 test standard is a feasible process model for a practical organisation with the following limitations as regards for applicability: • The standard model can be characterised as being overtly detailed in the definition of roles and activities. In the practical test organisations, the boundaries of different levels, processes and roles are less organised than the model presents. • The process model is top heavy and places a considerable emphasis on the organisational aspects of the test process. Based on the qualitative analysis, the model defines several responsibilities for the upper management, many of which are performed, in real-life organisations, at the project-level management or at least not as systematically as defined in the model. • The current standard model does not include a roadmap or phased process for adopting the model. This hinders the applicability of the model in organisations, as the organisations had difficulties in creating an approach which their existing process could adopt for the concepts presented in the standard. Jussi Kasurinen, Software Engineering Lab. LUT Information Technology

7. Other Major implications for industry and research • 3. The organisations do not actively try out new test methods and prefer a status quo in their test process. The following hypotheses relate to the test process development at the organisational level: • The organisations do not test out new testing tools or apply new testing methods unless they have a significant external incentive to do so. Based on the qualitative analysis, these incentives are things like the current state of the existing process or business needs in the operating domain. • The organisational test process may have feedback processes in place to allow continuous development, but in practice, the organisations tend to disregard the evidence of process enhancement needs if the existing process still performs at least at an acceptable efficiency. • In test process development, the organisations need a way of relating their existing process to the proposed changes to understand the objectives, and more pragmatically, the requirements the process improvement needs to succeed. Jussi Kasurinen, Software Engineering Lab. LUT Information Technology

8. Software Testing in the Cloud Leah Riungu-Kalliosaari

9. Background Empirical study aimed at understanding how organizations can successfully use the cloud for testing Cloud computing: A model for enabling convenient, on-demand network access to a shared pool of configurable computing resources (e.g. networks, servers, storage, application and services) that can be rapidly provisioned and released with minimal management effort or service provider interaction (NIST definition) STaaS: A model for software testing whereby testing of an application is provided as a service across the internet. Cloud-based testing is provided on-demand and billed on pay-per-use basis, so that the user pays only for the resources they have used

10. Current Situation The trend towards cloud services is growing Traditional desktop applications  online software services e.g. SaaS, StaaS Mixed reactions  Generally positive Positive – due to e.g. cost reduction, flexibility, performance, access to global markets Negative – due to concerns regarding e.g. security, domain knowledge, test data management Neutral – mainly explorative approach

11. Implications on Testing With the cloud becoming more common, the quality expectations are growing New issues and complexities surrounding these services will need to be addressed Testing will become more important Need for testing increases

12. 3. Testing the cloud 2. Testing environments in the cloud 1b. Non-SaaS software 1a. SaaS software Testing in the Cloud • The system or application under test is available online • Testing infrastructure and platforms are hosted in the cloud (Including crowdsourcing/Human as a Service-(Haas)) • Testing of the cloud itself Facets of testing in the cloud

13. What next? To better understand how cloud based services can be achieved with intended quality Identify the most important quality requirements for cloud-based services Provide recommendations for achieving overall intended quality of cloud-based services Difference between intended and achieved quality What are the tradeoffs, and what is the optimal balance. HOW Through interviews during Autumn Collect data to uncover the needs, problems, and practices requiring improvement Learn how OUs develop software, test software and build quality

14. Questions and comments What would you like to achieve during this project? What is most important for your organization? What kind of results? What kind of collaboration do you expect? Suggestions/improvement proposals regarding our research approach so far.

15. Software testing and development for intended quality Tero Pesonen

16. Objective To show how software development, software testing and intended quality depend on one another. The research area includes, in addition to traditional software development and service models, also the emerging XaaS (Everything as a Service) architectures, technologies and service models. The project results help the participating companies in improving the efficiency of their quality management and software testing and hence the efficiency of their software development as a whole. Testing techniques Testing as a service

17. Research Problem Intended Software Quality Software Development • Software Products • ISO/IEC 12207 • ISO/IEC 15504 • New Services • SaaS ISO/IEC 25010, Software Product Quality Software Testing ISO/IEC 29119, Testing Spice, IEEE Std 1012 StaaS Assessment framework through which each OU is evaluated

18. Research Problem The three-level view to software development: Software quality Specify the intended quality Quality attributes & metrics according to the ISO/IEC 25010 quality model. Software development Product or service oriented development Traditional or SaaS software Software development methods, software life cycle models ISO/IEC 12207, 15504 software development process improvement

19. Research Problem Software testing Ensure that the software development process produces the targeted quality Testing strategy & policy → testing processes → testing techniques ISO/IEC 29119, IEEE Std 1012, Testing process assessment, Testing SPICE

20. Research Problem The framework used as a ”grid” through which each OU is assessed. Intended quality vs. achieved quality. Top-down and bottom-up analysis (testing work  project management  organizational level). The assessment framework is defined through international software engineering standards. We need to anchor the assessment to a specific, concrete view on what quality, software testing and software development are. Based on standards, the model remains coherent throughout the assessment. Project results and usability; contribution to standardization work. The results include improvement proposals for the OU’s. Data collection methods include interviews and surveys.

21. Work division, entire project 3 year project 12 months / work package; WP 1 starts 8/2011 WP1: Requirements for intended quality Identifies the quality requirements and attributes that the OU's perceive as key for achieving the intended quality. Defining intended quality, the importance of different quality characteristics in achieving it, traditional vs. SaaS WP2: Building the intended quality Achieved vs. Intended quality vs. the software development process Action research WP3: Assessment How the OU’s can improve their quality management and software testing to achieve the intended quality more efficiently?

22. WP 1 Begins in August 2011 Quality requirements and the intended quality, software development and software testing Study units are OU's. What are the quality requirements, and hence the quality attributes, that the OU perceives as key for achieving the intended quality (level & characteristics)? Are some quality attributes more important than others when determining whether the targeted quality has been achieved? Do OU's or projects differ here? Does SaaS development emphasize different kind of targeted quality (level, characteristics) to traditional software development? The data are collected from the participating companies Interviews Additionally, supplementary data from the previous MASTO project may also be used. The findings are published as scientific publications during the WP.