Agile Software Development and ISO/IEC Software Quality Standards: Measuring Economic Benefits and Calculating Quantitative Yields Anca-Juliana Stoica, Margus Nael Royal Institute of Technology, KTH-ITC, Sweden 25th International Forum on Systems, Software, and COCOMO Cost Modeling University of Southern California, Los Angeles, CA November 2-5, 2010
Outline • Research Background • Agility vs. Discipline in Software Development • ISO/IEC Software Quality Standards • Research Objectives • Research Design • Case Studies • Techniques and Features Used • Framework for assessing a project • Metrics and Characteristics • Quality Metrics • Integrated Models for Measuring Economic Benefits and Calculating Quantitative Yields • Results • Conclusions
Agile methods: Increased customer satisfaction Lower defect rates Faster development times Solution to rapidly changing requirements Examples: XP(Extreme Programming), Cristal Methods, ASD(Adaptive Software Development), DSDM, FDD, Scrum Plan-driven methods: Predictability Stability High assurance Examples: Cleanroom, PSP, TSP, CMM methods (Capability Maturity Models: Software CMM, Integrated CMM(CMMI)), ISO/IEC Software Quality Standards (9001 and the recent SQuaRE standards) Examples of Software Engineering Methods
Challenge Balance the two approaches to take advantage of their strengths in a given situation while compensating for their weaknesses
Two suggested solutions • Software risk management approach • Software quality management approach, for structuring projects to incorporate both agile and plan-driven features in proportion to project’s needs.
Trends for the 21st Century Software Systems • For large – to very large scale software intensive systems, the Emerging Scalable Spiral process model • For small – to medium scale software systems, hybrid methods, using some other concepts in addition to software risk management, like: software quality management. Our focus here is on small – to medium scale software systems that need to balance agility and quality assurance.
ISO/IEC SQuaRE Series • Software product Quality Requirements and Evaluation • SQuaRE series of International Standards is based on 2 predecessors: • ISO/IEC 9126 (Software product quality) • ISO/IEC 14598 (Software product evaluation) • New general reference model consisting of 5 quality divisions: • Quality Management Division • Quality Model Division • Quality Measurement Division • Quality Requirements Division • Quality Evaluation Division.
SQuaRE Common Models • SQuaRE general reference model - navigation guide as a function of user’s task(s) • Software product LC model – views of internal, external software quality and software quality in use during software LC • Quality model structure –categorisation of software quality attributes into characteristics, sub-characteristics, and quality attributes. Remarks • Measuring and evaluating characteristics will assure quality • Organization must support the project with necessary equipment and knowledge • Create Quality Evaluation Project Plan
Research Objectives • Study if there are any conflicts between the application of agility principles and the ISO/IEC SQuaRE series of software quality standards • Guidelines on how to apply SQuaRE to agile (hybrid) software development • Real-life case studies for proving major benefits: • Software quality • Economic benefits.
Research Design • Data collection • Software project assessment framework based on agile/plan-driven characteristics • Experience base • Guidelines for applying ISO/IEC SQuaRE • Principles • How they apply to a specific project/development environment • Use of inductive approach • Real-life case studies with/without using SQuaRE • Software quality improvements • Economic benefits and other quantitative yields • Data analysis
Real-Life Case Studies Focus on two separate but similar real-life software development projects: • Project A -project without ISO/IEC SQuaRE • Project B -project with ISO/IEC SQuaRE Applied comparative analysis of projects and software metrics
Real-Life Case Studies Projects’ characteristics: • Modern Web-based small- sized software business systems • Stakeholder committment based on Theory W principles • Incremental development and delivery • Balancing agility and discipline • Software quality management done through the whole life-cycle • Use of a tailored software engineering method.
ProjectA/Project B • To publish fresh publications, e.g. news articles from news providers, via WAP or GPRS to the customers of a tecommunication company • Publications are delivered to application by RSS and server creates pages that are suitable for mobile stations to read. • To publish Web Logs (Blogs) via WAP or GPRS to the custommers of a telecommunication company • Publications are delivered to application by RSS and server creates pages that are suitable for mobile stations to read.
Company vision • capture a larger market share • become more attractive to customers by • developing better software quality products • managing better software development processes.
Project A Used techniques: • Feature Driven Development • Small team • Osmotic communication • Spiking, Walking skeleton • Information radiators • Simple design • Access to expert user • Estimation using comparison and analogy • Frequent delivery • Documentation is not so important • Kick-off meeting • Status reporting
Project A Home ground polar chart
Project A Framework for project assessment Included in the experience base
Project B Used techniques: • Feature Driven Development • Small team • Osmotic communication • Spiking, Walking skeleton • Information radiators (cont.)
Project B Used techniques: • Simple design • Acess to expert user • Estimation using comparison and analogy • Frequent delivery • Kick-off meeting • Status reporting • Code reviews • Measurements of quality characteristics • Burn charts
Project B Home ground polar chart
Project B Framework for project assessment:
Project A Quality Model Dark colors represent quality characteristics of the project mm n
Project B Quality Model Remark:We have two quality models,Project A and Project B. We just have to find a wayto measure it and prove that Project B is better.
Metrics and Characteristics • Project metrics: Personnel. • Product metrics: Lifetime, Downtime, Number of defects, Complexity. • Product quality metrics: Mean time to failure, Defect density, Customer satisfaction. • Other metrics: Lost amount of money to customer, Lost amount of money to developer. • Quantitative economic yields of improving software quality Remarks • Software quality metrics measured with Research Standard Metrics (RSM) tool • Quantitative economic yields of improving software quality are calculated (estimated) using a value-based framework.
Quantitative yields of improving software engineering methods Stoica, A.J. (2007). Exploring Quantitative Yields of Software Engineering Methods in the Software Life-Cycle. The first IEEE Computer Society EQUITY International Conference. 19-21 March 2007, Amsterdam, the Netherlands.
Motivation The problem of exploring quantitative yields of a new (improved) software engineering method in the software life-cycle has to be addressed in a systemic manner • relevant metrics and models have to be selected • their interrelashionships have to be expressed in a formal manner to detect possible inconsistencies and/or incompletenesses. The Model Systems (MS) concept is applied to • analyze multiple facets of software development • derive a value-based framework (VF) for exploring the quantitative yields of a new (improved) software engineering method in the software life-cycle.
Value-Based Framework (VF) Interrelated software system metrics and models that address: • Costs (CM) • Quality (QM) • Productivity (WM) • Benefits (BM) from applying better software project quality management • Resulting value estimation metrics (VM), evaluated over the software life-cycle.
Quantitative Models for Estimating Costs, Benefits, and Value Applicable for software engineering methods that: • Improve software quality via structured and facilitated group reviews • Certify companies to adhere to regulations for international trade • Evaluate, qualify, certify suppliers to government agencies • Perform individual and group-based project management via technical challenging approaches • Use sets of criteria designed for supplier selection and establishment of priorities for internal software process improvement.
VF Instantiation Example • Improving an agile (hybrid) software engineering method is achieved as a result of applying ISO/IEC software quality (SQuaRE) standards • Metrics and models are applied to: i) existing, and ii) new (improved) software development process in order to calculate all the possible quantitative yields • Improvements to an agile (hybrid) software engineering process are in similar to applying Software Inspection Method (SIM) • Guidance of practical use that ISO/IEC SQuaRE quality standard contains is also quite similar.
Main results • The real return on investment is 4700%. • The real gross benefit is 49701,6 Eur. • Real net benefit when you apply Software inspection Method (SIM) 48666,15 Eur. • Real benefit to cost ratio i.e. the ratio between benefits and costs is 48. • Real breakeven point is 24288,90 Eur, or 202,12 hours.
Summary • Our research shows that there are no conflicts between applying ISO/IEC SQuaRE software quality standards and agile software development • Economic and quality benefits are demonstrated using a value framework as well as software metrics and measurements • Simple guide to SQuaRE practical use with real-life example is presented • Evaluation framework for software system projects to decide on the balance between agile and plan-driven approaches is outlined • Experience base for analysing software system projects is created • Our work is a step forward in reconciling agile approaches and formal software product quality evaluation that is discussed in ISO/IEC software quality standards.
Conclusions • We showed that creatively using software quality standards in agile software development is possible for • combining the best of agility and discipline • creating quality software • generating economic benefits. • During the last years, agile software development has become a work facet in small and medium software development organizations, but its lack of theoretical foundations require practical methods, models, metrics, and/or processes to relate it to software engineering research and application as well as to the associated engineering standards. • Our work supports this effort and offers theoretical and practical support for future advances in educational and software development organizations.