COSYSMO CO nstructive SYS tems Engineering Cost MO del

1. COSYSMO COnstructive SYStems Engineering Cost MOdel INCOSE IW Workshop Tampa, FL February 3, 2003 Dr. Barry Boehm Ricardo Valerdi University of Southern California Center for Software Engineering (CSE)

2. Outline • Workshop Agenda • Background on CSE, COSYSMO, and COCOMO II • COSYSMO Overview • Operational concept and scope • Prototype demo • Model Progress to Date • Front end sizing and drivers • Full life cycle sizing and drivers • New proposed drivers • Action items

3. Workshop Agenda • Proposed drivers • Process maturity • # of recursive levels of design • Technology risk (different ratings on the viewpoints) • Length of life cycle • Quality Attributes (ie., documentation) • Manufacturability/Producibility • Degree of Distribution • Discontinuity of members • ISO/IEC 15288 expertise in the working group • Preparation for Delphi Round 2 • Preliminary data collection

4. USC Center for Software Engineering (CSE) • Researches, teaches, and practices CMMI-based Software engineering • Systems and software engineering fully integrated • Focuses on better models to guide integrated systems and software engineering • Success models: stakeholder win-win, business cases • Product models: requirements, architectures, COTS • Process models: spiral extensions, value-based RUP extensions • Property models: cost, schedule, quality • Applies and extends research on major programs (DARPA/Army, FCS, FAA ERAM, NASA Missions)

5. USC-CSE Affiliates (34) • Commercial Industry (15) • Daimler Chrysler, Freshwater Partners, Galorath, Group Systems.Com, Hughes, IBM, Cost Xpert Group, Microsoft, Motorola, Price Systems, Rational, Reuters Consulting, Sun, Telcordia, Xerox • Aerospace Industry (6) • Boeing, Lockheed Martin, Northrop Grumman, Raytheon, SAIC, TRW • Government (8) • DARPA, DISA, FAA, NASA-Ames, NSF, OSD/ARA/SIS, US Army Research Labs, US Army TACOM • FFRDC’s and Consortia (4) • Aerospace, JPL, SEI, SPC • International (1) • Chung-Ang U. (Korea)

6. INCOSE Companies actively involved with COSYSMO • Commercial Industry (1) • Galorath • Aerospace Industry (5) • Lockheed Martin, Northrop Grumman*, Raytheon, SAIC, TRW • Government (1) • US Army Research Labs • FFRDC’s and Consortia (2) • Aerospace, SPC *Most recent addition

7. Karen Owens, Marilee Wheaton Evin Stump Garry Roedler, Gary Hafen Gary Thomas, John Rieff Tony Jordano, Don Greenlee Chris Miller Cheryl Jones Barry Boehm, Elliot Axelband, Don Reifer, Ricardo Valerdi Key Members of the COSYSMO Working Group Aerospace Corp. Galorath LMCO Raytheon SAIC SPC US Army/PSSM USC underline = will participate in Monday’s Workshop Italics = SE experience

8. USC-CSE Cost, Schedule, and Quality Models • Build on experience with COCOMO 1981, COCOMO II • Most widely used software cost models worldwide • Developed with Affiliate funding, expertise, data support • Collaborative efforts between Computer Science (CS) and Industrial Systems Engineering (ISE) Depts. • 3 CS PhD’s, 2 ISE PhD’s to date • Valerdi an ISE PhD student • Boehm joint appointment in CS, ISE • COCOMO Suite of models • Cost, schedule: COCOMO II, CORADMO, COCOTS • Quality: COQUALMO • Systems Engineering: COSYSMO • Uses mature 7-step model development methodology

9. 7-step Modeling Methodology Analyze Existing literature Perform Behavioral Analysis 1 Identify Relative Significance 2 Perform Expert- Judgement, Delphi Assessment 3 4 A-PRIORI MODEL + SAMPLING DATA = A-POSTERIORI MODEL Gather Project Data Determine Bayesian A-Posteriori Update 5 Gather more data; refine model 6 7

10. Parametric Cost Model Critical Path Usual # Months* Critical Path Task 6 Converge on cost drivers, WBS 6 Converge on detailed definitions and rating scales 12 Obtain initial exploratory dataset (5-10 projects) 6 Refine model based on data collection & analysis experience 12+ Obtain IOC calibration dataset (30 projects) 9 Refine IOC model and tool *Can be shortened and selectively overlapped

11. COSYSMO: Overview • Parametric model to estimate system engineering costs • Covers full system engineering lifecycle • Focused on use for Investment Analysis, Concept Definition phases estimation and tradeoff analyses • Input parameters can be determined in early phases

12. COSYSMO Operational Concept # Requirements # Interfaces # Scenarios # Algorithms + Volatility Factor Size Drivers Effort COSYSMO Effort Multipliers Duration • Application factors • 5 factors • Team factors • 7 factors • Schedule driver Calibration WBS guided by EIA/ANSI 632 & ISO/IEC 15288

13. Previous COSYSMO Evolution Path Oper Test & Eval Inception Elaboration Construction Transition 1. COSYSMO-IP IP (Sub)system 2. COSYSMO-C4ISR C4ISR System Physical Machine System 3. COSYSMO-Machine System of Systems (SoS) 4. COSYSMO-SoS

14. Revised View of COSYSMO Evolution Path(Results from last week’s meeting) Operate, Maintain, or Enhance Transition to Operation Replace or Dismantle Oper Test & Eval Conceptualize Develop Global Command and Control System 1. COSYSMO-IP Include ISO/IEC 15288 Stages 2. COSYSMO-C4ISR Satellite Ground Station 3. COSYSMO-Machine Joint Strike Fighter 4. COSYSMO-SoS Future Combat Systems Initiate data collection for all and let the amount of data received determine what is included.

15. System life Process description ISO/IEC 15288 EIA/ANSI 632 High level practices Level of detail Input to 632/1220 IEEE 1220 Detailed practices Operate, Maintain, or Enhance Transition to Operation Replace or Dismantle Conceptualize Develop Purpose of the Standards: ISO/IEC 15288 - Establish a common framework for describing the life cycle of systems EIA/ANSI 632 - Provide an integrated set of fundamental processes to aid a developer in the engineering or re-engineering of a system IEEE 1220 - Provide a standard for managing systems engineering Breadth and Depth of Key SE Standards Source : Draft Report ISO Study Group May 2, 2000

16. ISO/IEC 15288 Key Terms • System • a combination of interacting elements organized to achieve one or more stated purposes • System-of-Interest • the system whose life cycle is under consideration in the context of this International Standard • System Element • a member of a set of elements that constitutes a system • NOTE: A system element is a discrete part of a system that can be implemented to fulfill specified requirements • Enabling System • a system that complements a system-of-interest during its life cycle stages but does not necessarily contribute directly to its function during operation • NOTE: For example, when a system-of-interest enters the production stage, an enabling production system is required Source: ISO/IEC 15288.

17. ISO/IEC 15288 System of Interest Structure Make or buy Source: ISO/IEC 15288.

18. Outline • Workshop Agenda • Background on CSE, COSYSMO, and COCOMO II • COSYSMO Overview • Operational concept and scope • Prototype demo • Model Progress to Date • Front end sizing and drivers • Full life cycle sizing and drivers • New proposed drivers • Action items

19. Recent developments • Developed a project plan • Reduced drivers from 24 to 18 • Expanded the model to cover the later phases of the life cycle • Replaced EIA 632 with ISO 15288 • Introduced new drivers to reflect full life cycle scope • Changed name from COSYSMO-IP (Information Processing) to COSYSMO • Revised the evolution path to allow the data to drive the scope of the model

20. 4 Size Drivers • Number of System Requirements • Number of Major Interfaces • Number of Operational Scenarios • Number of Unique Algorithms • Each weighted by complexity, volatility, and degree of reuse

21. Number of System Requirements This driver represents the number of requirements that are typically taken from the system or marketing specification. A requirement is a statement of capability containing a normative verb such as “shall” or “will.” It may be functional, performance, feature, or service-oriented in nature depending on the methodology used for specification. System requirements can typically be quantified by counting the number of applicable “shall’s” or “will’s” in the system or marketing specification.

22. Number of Major Interfaces This driver represents the number of shared major physical and logical boundaries between system components or functions (internal interfaces) and those external to the system (external interfaces). These interfaces typically can be quantified by counting the number of interfaces identified in either the system’s context diagram and/or by counting the significant interfaces in all applicable Interface Control Documents.

23. Number of Operational Scenarios This driver represents the number of operational scenarios that a system must satisfy. Such threads typically result in end-to-end test scenarios that are developed to validate the system satisfies all of its requirements. The number of scenarios can typically be quantified by counting the number of end-to-end tests used to validate the system functionality and performance. They can also be calculated by counting the number of high-level use cases developed as part of the operational architecture.

24. Number of Unique Algorithms This driver represents the number of newly defined or significantly altered functions that require unique mathematical algorithms to be derived in order to achieve the system performance requirements. As an example, this could include a complex aircraft tracking algorithm like a Kalman Filter being derived using existing experience as the basis for the all aspect search function. Another example could be a brand new discrimination algorithm being derived to identify friend or foe function in space-based applications. The number can be quantified by counting the number of unique algorithms needed to support each of the mathematical functions specified in the system specification or mode description document (for sensor-based systems).

25. 12 Cost Drivers Application Factors (5) • Requirements understanding • Architecture complexity • Level of service requirements • Migration complexity • Technology Risk

26. Requirements understanding This cost driver rates the level of understanding of the system requirements by all stakeholders including the systems, software, hardware, customers, team members, users, etc…

27. Architecture complexity This cost driver rates the relative difficulty of determining and managing the system architecture in terms of platforms, standards, components (COTS/GOTS/NDI/new), connectors (protocols), and constraints. This includes tasks like systems analysis, tradeoff analysis, modeling, simulation, case studies, etc…

28. Level of service requirements This cost driver rates the difficulty and criticality of satisfying the Key Performance Parameters (KPP). For example: security, safety, response time, the “illities”, etc…

29. Migration complexity This cost driver rates the complexity of migrating the system from previous system components, databases, workflows, etc, due to new technology introductions, planned upgrades, increased performance, business process reengineering etc…

30. Technology Risk The maturity, readiness, and obsolescence of the technology(ies) being implemented. This may include We need to supply guidelines on how to compose the readiness and obsolescence into a single factor.

31. 12 Cost Drivers (cont.) Team Factors (7) • Stakeholder team cohesion • Personnel capability • Personnel experience/continuity • Process maturity • Multisite coordination • Formality of deliverables • Tool support

32. Stakeholder team cohesion Represents a multi-attribute parameter which includes leadership, shared vision, diversity of stakeholders, approval cycles, group dynamics, IPT framework, team dynamics, trust, and amount of change in responsibilities. It further represents the heterogeneity in stakeholder community of the end users, customers, implementers, and development team.

33. Personnel capability Systems Engineering’s ability to perform in their duties and the quality of human capital. Personnel experience/continuity The applicability and consistency of the staff over the life of the project with respect to the customer, user, technology, domain, etc…

34. Revisit, should include more detail Under process levels, process improvement commitment Process maturity Maturity per EIA/IS 731, SE CMM or CMMI. Multisite coordination Location of stakeholders, team members, resources (travel). Add Collaboration barriers row (time zone, security, export restrictions, language, culture, competition, Intellectual property, contractual)

35. Right size of documentation to the life cycle needs The Formality of deliverables The breadth and depth of documentation required to be formally delivered. Breadth depth Right size Reviews? Tool support Use of tools in the System Engineering environment.

36. Outline • Workshop Agenda • Background on CSE, COSYSMO, and COCOMO II • COSYSMO Overview • Operational concept and scope • Prototype demo • Model Progress to Date • Front end sizing and drivers • Full life cycle sizing and drivers • New proposed drivers • Action items

37. # and diversity of installations/platforms The number of different platforms that the system will be hosted and installed on. The complexity in the operating environment (space, sea, land, fixed, mobile, portable, information assurance/security). For example, in a wireless network it could be the number of unique installation sites and the number of types of fixed clients, mobile clients, and servers. Include phase out

38. # of recursive levels in the design New cost driver. Can capture the number of boundaries in the SOI (System of Interest). Number of recursive levels that require SE.For example, a system with a single level of design may have lower numbers of recursive levels in the design. On the other hand, a system with a system integrator (system of systems level), a prime (system level), a subcontractor (segment level), second-tier subcontractor (subsystem level), and third-tier contractor (component/box level) would have 5 levels of recursive design. Includes internal design/subsystem levels. Note: this driver depends on what level your enabling system lies. Revision: Has system-wide multiplicative effect vs. incremental additive effect. Better handled in two cost drivers: Architecture Complexity (technical aspects) & Multisite Coordination (management aspects).

39. # and diversity of installations/platforms New size driver. Can capture the number of varied and multiple installations and/or platforms. This is in light of the discussion to cover the full system life cycle. This driver is especially important in later implementation/deployment stages. Revision: Its effect on effort is additive but the amount of added effort per type of installation/platform is a multiplier of the baseline effort. Probably best covered by a formula like: (# of diverse installation/platform types) * (Avg extra % of systems engineering effort per type)

40. The number of different processing platforms that the system will be hosted and installed on. For example, in a network it could be the number of unique installation sites and the number of workstations and servers.

41. # of years in operational life cycle Revision: Another additive factor where the added effort is a multiplier of the baseline effort. Probably best covered by a formula like: (# of years in operational life cycle) * (Avg annual % of continuing systems engineering effort) • Evolutionary acquisition • # of independently evolving interoperating systems • Upgrade to prior models

42. # and diversity of installations/platforms phased out Revision: Its effect on effort is additive but the amount of added effort per installation/platform being phased out is a multiplier of the baseline effort. Probably best covered by a formula like: (# of diverse installation/platform types) * (Avg extra % of systems engineering effort per type being phased out)

43. Quality Attributes Rationale: Reliability, availability and maintainability requirements for certain classes of systems often drive their costs. This is true for military as well as commercial systems. A cost driver aimed at assessing the impact of varying these requirements is therefore needed. Quality Attributes: Represents a measure of the extent to which the system must perform its intended requirements (reliability), be available (availability) and facilitate change, modification and repair (maintainability) over time.

44. Manufacturability/Producibility Rationale: The ability to manufacture/produce the system can drive the costs especially when advances in manufacturing technology are needed to field the system. Manufacturability/Producibility: Represents the ability of contractors to manufacture/produce the system in specific quantity to meet the market demand. DFMA Design for “x” Design trades Revisit

45. Degree of Distribution Rationale: The degree of partitioning and distribution of the system impacts its cost. This cost driver how interoperable the system must be. Degree of Distribution: Characterizes how distributed the architecture is.

46. Levels & complexity of V&V Revisit Might be under # of Requirements (under VH)

47. Size Drivers vs. EIA/ANSI 632 & ISO/IEC 15288 Stages Late in the Life Cycle Legend Bold = existing driver Italics = proposed addition

48. Cost Drivers vs. EIA/ANSI 632 & ISO/IEC 15288 Stages Late in the Life Cycle Legend Bold = existing driver Italics = proposed addition

49. Cost Drivers vs. EIA/ANSI 632 & ISO/IEC 15288 Stages Late in the Life Cycle Legend Bold = existing driver Italics = proposed addition

50. Action Items from Last WG Meeting • Develop a project Plan • Address technology maturity/obsolescence • Refine driver definitions to incorporate ISO/IEC 15288 definitions • Incorporate System and People idea • Refine drivers applicability matrix • Develop data collection strategy • Generate Data Collection Form • Update Stakeholder Cohesion to include diversity, identification and trust