A study of the Causes of Requirements Volatility and its Impact on Systems Engineering Effort

1. A study of the Causes of Requirements Volatility and its Impact on Systems Engineering Effort COSYSMO WorkshopCenter for Software and Systems Engineering, Annual Research ReviewMarch 11, 2010 Mauricio E. Peña – Ph.D. Student, USC Industrial and SystemsEngineering Department

2. Motivation “Requirements are the foundation of the project. They form the basis for design, manufacture, test and operations….changes in requirements later in the development cycle can have a significant cost impact, possibly resulting in cancellation.” INCOSE Systems Engineering Handbook (2006)

3. Objectives • Provide and overview of requirements volatility and its importance to engineering projects • Introduce the research questions & research plan • Discuss the state of the art and implications to COSYSMO • Solicit feedback: • On a requirements volatility causal model • On a requirements volatility survey

4. Requirements Volatility Overview Requirements volatility is the change in requirements (added, deleted, and modified) over a given time interval [MIL-STD-498, 1994, Ferreira, 2002] Also known as: Requirements creep: An increase in scope and number of system requirements Requirements churn: Instability in the requirements set – requirements are modified or re-worked without necessarily resulting in an increase in the total number of requirements CDR [Hammer et al, 1998]

5. Importance of Understanding Requirements Volatility • Requirements volatility has been identified by numerous research studies as a risk factor and cost-driver of systems engineering projects [Ferreira 2002, Boehm 1991] • Requirements changes are costly, particularly in the later stages of the lifecycle process because the change may require rework of the design, verification and deployment plans [Kotonya and Sommerville, 1995] • Requirements volatility trends are considered leading indicators of project performance as defined by the Lean Advancement Initiative Systems Engineering Leading Indicators Guide (2007) • The Government Accountability Office (GAO) concluded in a 2004 report on the DoD’s acquisition of software-intensive weapons systems that missing, vague, or changing requirements are a major cause of project failure

6. Research Questions • Research Question 1: What causes requirements volatility? • Research Question 2: How does requirements volatility impact systems engineering effort?

7. Research Plan Completed In-progress Planned Literature Review Tested Hypothesis Observations Statistical Analysis Pilot RV survey Causal Model Industry Data COSYSMO Workshop Discussion Updated Causal Model (hypothesis) Delphi Survey Delphi Survey RV survey Considerations for requirements volatility LAI Knowledge Exchange Survey Results

8. Literature Background • Most of the requirements volatility research to date has been focused on software systems • Various research methods have been utilized to investigate the causes and effects of requirement volatility – a methodological breakdown of the studies reviewed to date is below • However, there still a lack of empirical data to determine the quantitative impact of requirements volatility on systems engineering effort for a broader base of engineering projects

9. Implications to COSYSMO • Volatility was identified as a size driver adjustment factor in COSYSMO [Valerdi, 2005] • It is currently a placeholder in the model Reuse Categories Volatility Factor # Requirements # Interfaces # Scenarios # Algorithms Size Drivers COSYSMO Effort Effort Multipliers • Application factors • 8 factors • Team factors • 6 factors Calibration [Fortune, 2009]

10. Impact of volatility after the requirements have been baselined PROJECT WITH VOLATILITY PROJECT WITH LOW VOLATILITY # Req. # Req. Planned # Req Actual # Req Planned # Req Actual # Req SRR SRR SE Hours SE Hours Delta SE effort Delta SE effort SRR SRR

11. Observations • Requirements volatility is correlated with an increase in the size drivers of systems engineering effort [Valerdi, 2005; Houston,2000] • Requirements Volatility is expected during the early stages of a project (conceptualize / requirements phase). It becomes a concern when it ocurrs after the requirements phase is complete because it is likely to result in re-work of engineering products [Kotonya and Sommerville (1995); Rosenberg and Hyatt (1996); Ferreira, (2002)] • Requirements volatility is correlated to increases in cost, schedule duration and re-work [Ferreira et al, 2009; Zowghi and Nurmuliani, 2002] • Requirements volatility may have a compounding effect on systems engineering effort because the changes may not only cause rework but also trigger increases in the number of lower-level requirements [Kulk and Verhoef, 2008] • The use of volatility metrics and thresholds can help determine the adequacy of SE processes and staff and trigger corrective actions to mitigate further volatility (Roedler and Rhodes, 2007) • Requirements volatility is caused by a number of factors that are both external (e.g. environmental, customer priority changes) and internal (process capability, organizational policies) Based on the review of the literature a causal model was developed to investigate both research questions

12. Causal Model (normative) Case 1 research question: what causes requirements volatility? Contextual / Environmental changes Changes in org. structure and process Technology Maturity Requirements. Re-use SE Process Maturity Experienced staff - (Ferreira, 2002) (GAO, 2004) (GAO, 2004) - - (Jones 1994) (Curtis et al. 1988) Zowghi et al. (2000) - (Ferreira 2002) (Ferreira 2002) + + (Kotonya and Sommerville 1998) Customer-Driven Changes + (Jones 1994) (Curtis et al. 1988) (Kotonya and Sommerville 1998) (Valerdi 2005) Requirements volatility Number of Sys Requirements +/- SE Productivity +/- (Houston, 2000) Zowghi et al. (2000) (Ferreira, et al 2009) + (Ferreira, 2002) (Ferreira, et al 2009) (Valerdi 2005) (Houston, 2000) (Ferreira 2002) (Charette et al,2003) + +/- +/- (Valerdi 2005) (Charette et al,2003) SE Project Effort Project Schedule + Quality Re-work - +/- (Ferreira, et al 2009) (Houston, 2000) (Ferreira 2002) (Charette et al,2003) +/- + (Valerdi 2005) + +/- Project Cost SE Cost (Valerdi 2005) - - Customer Satisfaction Dependent Variable Independent Variable

13. Causal Model (normative) Case 2 research question: how does requirements volatility impact systems engineering effort? Contextual / Environmental changes Changes in org. structure and process Technology Maturity Requirements. Re-use SE Process Maturity Experienced staff - (Ferreira, 2002) (GAO, 2004) (GAO, 2004) - - (Jones 1994) (Curtis et al. 1988) Zowghi et al. (2000) - (Ferreira 2002) (Ferreira 2002) + + (Kotonya and Sommerville 1998) Customer-Driven Changes + (Jones 1994) (Curtis et al. 1988) (Kotonya and Sommerville 1998) (Valerdi 2005) Requirements volatility Number of Sys Requirements +/- SE Productivity +/- (Houston, 2000) Zowghi et al. (2000) (Ferreira, et al 2009) + (Ferreira, 2002) (Ferreira, et al 2009) (Valerdi 2005) (Houston, 2000) (Ferreira 2002) (Charette et al,2003) + +/- +/- (Valerdi 2005) (Charette et al,2003) SE Project Effort Project Schedule + Quality Re-work - +/- (Ferreira, et al 2009) (Houston, 2000) (Ferreira 2002) (Charette et al,2003) +/- + (Valerdi 2005) + +/- Project Cost SE Cost (Valerdi 2005) - - Customer Satisfaction Dependent Variable Independent Variable

14. Moderating impact of expected volatility / thresholds Org Improvement Actions + + +/- SE Process Maturity Experienced staff (Ferreira, 2002) (GAO, 2004) - (Ferreira 2002) - Volatility Metrics / Thresholds Requirements volatility +/- (Houston, 2000) Zowghi et al. (2000) (Ferreira, et al 2009) +/- (Kulk and Verhoef 2008 Zowghi and Nurmuliani (1998) (Kulk and Verhoef 2008) Number of Requirements +/- +/- +/- (Valerdi 2005) (Ferreira, et al 2009) SE Project Effort Project Schedule +/- +/- (Valerdi 2005) SE Cost Dependent Variable Independent Variable Moderating Variable

15. Questions for discussion • Are there other important causes of volatility missing in the model? • In what cases is the relationship between requirements volatility and # of systems requirements a positive one, and in what cases is it a negative one? • Should the impact of requirements volatility on the # of system requirements be adjusted based on the criticality/coupling of the requirements that are added/modified/deleted? • Does volatility have a direct impact on productivity? • Should volatility thresholds vary depending on the size and duration of a project? • How does the lifecycle phase affect the expected level of volatility? • Feedback on the survey -

16. Next Steps • Utilize recommendations/suggestions from this workshop to update the causal model and survey • Administer the survey to LAI workshop participants • Need to develop an approach to generate cost driver weight factors based on requirements volatility • Start the industry outreach to reach agreement on data-sharing

17. References • Boehm, B. (1991). Software Risk Management: Principles and Practices. IEEE Software 8, 1. Pp 32-41. • Ferreira, S., Collofello, J., Shunk, D., and Mackulak, G. (2009). “Understanding the effects of requirements volatility in software engineering by using analytical modeling and software process simulation.” The Journal of Systems and Software. Vol. 82, pp 1568-1577. • Fortune, J. (2009). Estimating systems engineering reuse with the constructive systems engineering cost model (COSYSMO 2.0). Doctoral Dissertation. University of Southern California, Industrial and Systems Engineering Department. • GAO-04-393 (2004). Report to the Committee on Armed • Services, U.S. Senate. Defense Acquisitions. Stronger Management Practices Are Needed to Improve DOD’s Software-Intensive Weapon Acquisitions • Hammer, T., Huffman, L., and Rosenberg, L. (1998). “Doing requirements right the first time.” Crosstalk, the Journal of Defense Software Engineering. Pp 20-25. • Houston, Dan X. (2000). A Software Project Simulation Model for Risk Management, Ph.D. Dissertation, Arizona State University • INCOSE Systems Engineering Handbook, Version 3, INCOSE, June 2006 • ISO/IEC (2008). ISO/IEC 15288:2008 (E) Systems Engineering - System Life Cycle Processes. • Kotonya, G., Sommerville, I., (1998). Requirements Engineering: Processes and Techniques. John Wiley and Sons, Ltd. • Kulk, G, and Verhoef, C. (2008). “Quantifying requirements volatility effects”. Science of Computer Programming. Vol 72. pp 136–175 • MIL-STD-498. 1994. Software Development and Documentation. U.S. Department of Defense. • Roedler, G. and Rhodes, D. (2007). Systems engineering leading indicators guide. Version 1. Massachusetts Institute of Technology, INCOSE, and PSM. • Rosenberg, L. and Hyatt L., (1996). A Software Quality Model and Metrics for Identifying Project Risks and Assessing Software Quality. (retrieved from http://satc.gsfc.nasa.gov/support/STC_APR96/qualtiy/stc_qual.html) • Schunn, C. (2008) Engineering Educational Design. Educational Designer, 1(1) • Valerdi, R. (2005). The constructive systems engineering cost model (COSYSMO). Doctoral Dissertation. University of Southern California, Industrial and Systems Engineering Department. • Zowghi, D. and Nurmuliani, N. (2002). A Study of the Impact of Requirements Volatility on Software Project Performance. Proceedings of the Ninth Asia-Pacific Software Engineering Conference