Mathematical Formulation and Validation of the Impact of Requirements Volatility on Systems Engineering Effort

1. Mathematical Formulation and Validation of the Impact of Requirements Volatility on Systems Engineering Effort November 2nd, 2011 Mauricio E. Peña Dr. Ricardo Valerdi

2. Outline • Motivation and introduction • Implications to COSYSMO • Research methods • Observations from exploratory research • Proposed model of the impact of requirements volatility on Systems Engineering effort • Preliminary model validation • Conclusions and next steps

3. Importance of Understanding Requirements Volatility • Requirements volatility has been identified by numerous research studies as a risk factor and cost-driver of systems engineering projects1 • Requirements changes are costly, particularly in the later stages of the lifecycle process because the change may require rework of the design, verification and deployment plans2 • The Government Accountability Office (GAO) concluded in a 2004 report on the DoD’s acquisition of software-intensive weapons systems that missing, vague, or changing requirements are a major cause of project failure3 System developers often lack effective methods and tools to account for and manage requirements volatility Source: 1- Boehm (1991), 2- Kotonya and Sommerville (1995), 3- GAO-04-393

4. Requirements Volatility is Expected • Changes to requirements are a part of our increasingly complex systems & dynamic business environment • Stakeholders needs evolve rapidly • The customer may not be able to fully specify the system requirements up front • New requirements may emerge as knowledge of the system evolves • Requirements often change during the early phases of the project as a result of trades and negotiations Requirements volatility must be anticipated and managed Sources: Kotonya and Sommerville (1995); Reifer (2000)

5. Requirements Volatility Definitions • Requirements volatility is typically defined as the change in requirements (added, deleted, and modified) over a given time interval • Also known as: • Requirements creep: An increase in scope and number of system requirements • Requirements churn: Instability in the requirements set – requirements are modified or re-worked without necessarily resulting in an increase in the total number of requirements Source: MIL-STD-498 (1994)

6. CSSE Parametric Cost Models • The Constructive Systems Engineering Cost Model (COSYSMO) was developed by the USC Center for Software and Systems Engineering (CSSE) in collaboration with INCOSE and Industry affiliates • COSYSMO is the first generally-available parametric cost model designed to estimate Systems Engineering effort • Built on experience from COCOMO 1981, COCOMO II • During the development of COSYSMO, volatility was identified as a relevant adjustment factor to the model’s size drivers Source: 7th Annual Practical Software and Systems Measurement Conference. COSYSMO Workshop, Boehm

7. COSYSMO 2.0 Operational Concept Proposed Volatility Factor Source: Fortune (2009)

8. Life Cycle Phase Definition • Conceptualize phase focuses on identifying stakeholder needs, exploring different solution concepts, and proposing candidate solutions. • The Development phase involves refining the system requirements, creating a solution description, and building a system. • The Operational Test & Evaluation Phase involves verifying/validating the system and performing the appropriate inspections before it is delivered to the user • The Transition to Operation Phase involves the transition to utilization of the system to satisfy the users’ needs. Source: Valerdi (2008); ISO/IEC 15288 (2002)

9. Research Methods In Progress Literature Review & 6 Workshops completed to date Source: Boehm et al (2000)

10. Observations from the Literature and Exploratory Research • Requirements volatility is caused by an identifiable set of project and organizational factors. • The level of requirements volatility is a function of the system life cycle phase. • Requirements volatility leads to an increase in project size and cost • The cost and effort impact of a requirements change increases the later the change occurs in the system life cycle • The impact of requirements volatility varies depending on the type of change: added, deleted, or modified

11. Life Cycle Effort Penalty due to Volatility (n = 27) Systems Engineering Effort Penalty Due to Volatility Operational Test & Evaluation Transition to Operation Development Conceptualize Data collected from two workshops: 25th Annual USC CSSE COCOMO and the 2011 USC-CSSE Annual Research Review

12. Estimated Life Cycle Effort Penalty per Change Category (n = 9) Estimated SE Effort Penalty due to Volatility Operational Test & Evaluation Transition to Operation Development Conceptualize Results from a two-round Delphi Survey held at the 2011 Practical Software and Systems Measurement Conference

13. Mathematical Formulation of Requirements Volatility ( 1 of 2) Where, R0 = Baseline number of requirements Req = Equivalent number of requirements wv= Requirements Volatility weighting factor • The effective increase in the number of requirements would result in an associated increase in systems engineering effort Sources: Boehm, B., et al (2000)

14. Mathematical Formulation of Requirements Volatility ( 2 of 2) Where wv = Requirements volatility weighting factor wx,l = Weighting factor for added, deleted, or modified requirements Θx,l = % of total requirements changes that were added, deleted or modified l = lifecycle phases Pena (2010)

15. Preliminary Model Validation • Data were collected from nine projects from a space systems application domain • Number of requirements, interfaces, algorithms operational scenarios (estimated systems engineering size) • Added, modified, and deleted requirements over time • The cost estimation accuracy of COSYSMO was compared to the accuracy of the model with the requirements volatility size driver adjustment factor • The models were tested using the coefficient of determination (R2) and predictive accuracy levels • The relationship between requirements volatility and systems engineering size was also examined

16. Baseline Coefficient of Determination (n = 9) * Due to proprietary reasons only the analysis of the model accuracy is shown, not the data itself

17. Coefficient of Determination with Volatility Factor (n = 9)

18. Prediction Accuracy Level The prediction accuracy at a particular level (l) is defined as Where, n = set of projects k = number of projects in the set whose Magnitude of Relative Error is ≤ l Source: Conte, Dunsmore, and Shen (1986)

19. Relationship Between Requirements Volatility and Sys. Engr. Size

20. Relationship Between the Volatility Adjustment Factor and Sys. Engr. Size

21. Conclusions – Next Steps • Observations from the literature and workshops were used to develop a mathematical framework for quantifying the impact of requirements volatility on systems engineering effort • A preliminary validation of the model was performed by comparing its prediction accuracy against COSYSMO • The preliminary results indicate an improvement in systems engineering effort prediction accuracy when a requirements volatility factor is added to the model • The correlation between the requirements volatility factor and the estimated systems engineering size was examined – the results were not conclusive • Additional project data will be collected to complete the model validation

22. References • B. Blanchard and W. Fabrycky, Systems engineering & analysis, Prentice Hall, New York, NY, 1998. • B. Boehm, Software risk management: Principles and practices, IEEE Software 1 (1991), 32-41. • B. Boehm, C. Abts, A.W. Brown, S. Chulani, B. Clark, E., Horowitz, R. Madachy, D.J. Reifer, and B. Steece, Software Cost Estimation with COCOMO II, Prentice Hall, New York, NY, 2000 • S. Conte, H. Dunsmore, and V. Shen. Software Engineering Metrics and Models. Benjamin/Cummings Publishing Company, 1986. • Department of Defense, “Instruction 5000.02. Operation of the Defense Acquisition System,” 2008. • S. Ferreira, J. Collofello, D. Shunk and G. Mackulak, Understanding the Effects of Requirements Volatility in Software Engineering by Using Analytical Modeling and Software Process Simulation, Journal of Systems and Software 82 (2009) 1568-1577. • General Accounting Office, "Stronger management practices are needed to improve DoD’s software-intensive weapon acquisitions (GAO-04-393)," 2004. • D. Houston, "A software project simulation model for risk management," Ph.D. Dissertation, Arizona State University, 2000. • ISO/IEC, "ISO/IEC 15288:2002 (e) systems engineering - system life cycle processes," 2002. • C. Jones, Assessment and Control of Software Risks, Prentice Hall, Inc. Englewood Cliffs, New Jersey, 1994. • C. Jones, Strategies for managing requirements creep, IEEE Computer, Vol 20 (1996), pp. 92-94 • G. Kotonya and I. Sommerville, Requirements engineering: Processes and techniques, John Wiley & Sons, New York, NY, 1998. • G.P. Kulk, and C. Verhoef, Quantifying Requirements Volatility Effects, Science of Computer Programming, 72, 136-175, 2008. • Y. Malaiya, and J. Denton, Requirements Volatility and Defect Density, Proceedings of the International Symposium on Software Reliability Engineering, 1999 • MIL-STD-498, "Software development and documentation," 1994. • S. Nidumolu, Standardization, Requirements Uncertainty and Software Project Performance, Information and Management. Vol 31 (No. 3) (1996), pp 135-150 • M. Pena, Characterizing the Impacts of Requirements Volatiliy, Proceedings of the 25th Annual COCOMO Forum, 2010 • D. Pfahl, and K. Lebsanft, Using Simulation to Analyze the Impact of Software Requirements Volatility on Project Performance, Information and Software Technology, Vol 42 (No. 14) (2000), pp 1001-1008. • D. Reifer, Requirements management: The search for nirvana, IEEE Software 17(3) (2000), 45-47 • D. Rhodes, R. Valerdi and G. Roedler, Systems engineering leading indicators for assessing program and technical effectiveness, Systems Engineering 12(1) (2009), 21-35. • R. Valerdi, The Constructive Systems Engineering Cost Model (COSYSMO): Quantifying the Costs of Systems Engineering Effort in Complex Systems, Saarbrücken, Germany, VDM Verlag, 2008. • D. Zowghi and N. Nurmuliani, A study of the impact of requirements volatility on software project performance, Proceedings of the Ninth Asia-Pacific Software Engineering Conference, 2002.