Systems Engineering Cost Estimation Curriculum Development Workshop Debrief

1. Systems Engineering Cost Estimation Curriculum Development Workshop Debrief Ray Madachy, Naval Postgraduate School Ricardo Valerdi, University of Arizona

2. Attendees • Mauricio Pena (USC/Boeing) • Anca Juliana-Stoica (Uppsala University) • Andy Nolan (Rolls-Royce) • Ed Colbert (USC) • Jorge Aguilar (UPAEP) • Mary Boghosian (Aerospace Corp.) • Sue Koolmanojwong (USC) • MonvorathPhonpaibul (IBM/Thammasat University) • Hunter Deane (ICONIX) • Doug Rosenberg (ICONIX) • NupulKukreja (USC) • Warren Reid (WSR Consulting) • Ray Madachy (Naval Postgraduate School) • Ricardo Valerdi (University of Arizona)

3. Workshop Overview • Goal: Engage stakeholders from the cost community interested in education and training to learn and share best practices related to curriculum development and delivery.   • Our discussion guided by • Links between cost estimation curriculum and ongoing efforts to update the systems engineering body of knowledge (BKCASE) and software engineering reference curriculum (GSwERC) • Sharing of cost estimation course syllabi from the Naval Postgraduate School, University of Arizona, and other universities • Review of Systems Engineering Cost Estimation Workbook being developed by Madachy and Valerdi • Obtained feedback to move the cost estimation profession forward.

4. Participant Teaching Experiences Overview • Thammasat University in Thailand • cost estimation is part of the Project Management class (undergraduate and graduate students in software engineering) • Uppsala University in Sweden • software and systems engineering classes (Masters and PhD students in IT Project Management), deals with both HW and SW • Rolls-Royce • three courses: 1) how to estimate, 2) how to develop tools, 3) how to be a good recipient of an estimate. Management of uncertainty is a significant part. • University of Southern California • graduate level classes for Software Engineering Management & Economics, and Software Engineering contain cost estimation topics

5. Curricula Recommendations • Students should learn that cost estimation does not live in isolation • E.g. business concerns, stakeholder management • Start with model development, statistical analysis and data collection • Include feedback loop for students to compare their estimates with actuals, adjust and improve estimation skills

6. Workbook Discussion • Include example systems, learning curve problems, life cycle cost problems • Beproblem-based vs. model-specific • Add “problem domain” dimension to coverage matrix • Include problems with size expansion ratios, DODAF, estimation risk/confidence • Add bottom-up and top-down cost estimates • Change reuse questions for students to figure out how much of the system is new/modified/adopted vs. handing the values • Tradeoff exercises between cost and performance by negotiating scope; scope creep exercises to test cost model sensitivity

7. Workbook Discussion (Cont.) • Introduce workbook with everyday estimation problems • Cover art and science of estimation • Introduce case studies to include lessons learned • Sync cost estimation course with another project course developing software so that students can estimate that development; then compare estimates vs. actuals • Have examples include multi-criteria decision making (cost, schedule, risk, performance, value, etc.) • Add annotated bibliography and glossary

8. Backups from Workshop

9. GSWERC • A reference curriculum for a Masters Degree in Software Engineering based on Software Engineering Body of Knowledge SWEBoK • Joint industry, academic, and government initiative • Modified SWEBOK in several places, including adding Engineering Economics and Risk Management to SWE Management Knowledge Area

10. GSWERC Outcomes and Knowledge Areas 5. Understand the relationship between software engineering and systems engineering and be able to apply systems engineering principles and practices in the engineering of software. • I.3. Risk Management 7. Be able to reconcile conflicting project objectives, finding acceptable compromises within limitations of cost, time, knowledge, risk, existing systems, and organizations. • I.1. Software Project Planning • I.2. Risk Management • I.7. Engineering Economics

11. GRCSE • The Body of Knowledge and Curriculum to Advance Systems Engineering (BKCASE) project is defining a Systems Engineering Body of Knowledge (SEBoK) and developing an advanced Graduate Reference Curriculum for Systems Engineering (GRCSE) based on the (SEBoK) • Joint industry, academic, and government initiative • See v.0.5 at bkcase.org • SEBoK Knowledge Areas involving cost estimation • Project Planning, Measurement, Risk Management, Systems Analysis, others

12. Our Courses • NPS SE3011: Engineering Economics and Cost Estimation • Offered multiple times per year in Monterey, CA (and virtually) • UA SIE 464/564: Cost Estimation • Offered every Spring in Tucson, AZ (and virtually)

13. NPS MSSE Outcome Performance Measures • Demonstrates proficiency in techniques, skills, and tools of systems engineering. • Demonstrate proficiency in the assessment of a system. • Demonstrate proficiency in economic and life-cycle analysis.

14. NPS Course Learning Outcomes • Understand how cost is used in choosing between alternatives • Understand basics of cost behavior and cost allocation • Understand how costs change over the life-cycle of a system • Choose between alternatives using economic reasoning • Understand cost issues related to program management • Develop life-cycle cost models • Choose between alternatives using net present value analysis • Apply cost estimating techniques to develop cost estimates • Incorporate uncertainty into cost estimates • Quantify and evaluate cost risk • Develop and analyze system cost

15. NPS MSSE Degree Program Outcome Coverage • Demonstrate the ability to identify, formulate, and solve operational, technical, and engineering problems in systems engineering and related disciplines using the techniques, skills, and tools of modern practice, including modeling and simulation. These problems may include issues of research, design, development, procurement, operation, maintenance or disposal of systems and processes for military applications. • Demonstrate proficiency in the systems engineering process, including defining requirements, conducting functional analysis, designing and architecting a system, analyzing it against requirements, allocation of requirements to sub-systems, conducting trade-off studies, determining the cost of the system, integrating human factors into the system, designing logistical supportability, and planning for its testing and evaluation. • Demonstrate proficiency in core skills of systems analysis, to include deterministic and stochastic modeling of systems, optimization, decision analysis, risk analysis, economic models, and lifecycle supportability analysis. This includes familiarity with combat simulations and combat modeling. • Demonstrate the ability to work as a team member or leader in a large systems engineering project, and to provide leadership in the systems engineering management process. The graduate must be able to interact with personnel from other services, industry, laboratories and academic institutions. • Demonstrate competence in the planning and management of large systems engineering projects. • Demonstrate proficiency in written and oral presentation of technical material.

16. Workbook Objectives/Targets • Provide a flexible, one-stop resource for companies and Universities who provide education/training in cost modeling • Example University courses being taught by • Young Kwak, George Washington University • Peggy Brouse, George Mason University • Dan Nussbaum, Naval Postgraduate School • John Farr, West Point • Sue Koolmanojwong, USC • Etc.

17. Workbook Contents • Systems Engineering • Software Engineering • Hardware Engineering • Human Systems Integration • Systems of Systems • Product Lines • Cost model extensions (quality, multiple modules, increments, etc.) • Exercises in all areas • Spreadsheets and web-based estimation tools • Instructors version

18. Workbook Boundary • IN • Equations for parametric models • Cost driver rating scales and effort multipliers • Tables to decompose estimates by phases and activities • Equations for other decision metrics such as ROI, PV, etc. • Worked out examples and student exercises for all models • Calibration equations for models (linear and logarithmic) • Instructor version includes all answers plus master tool to compose and solve modified problems • OUT • Background on how the models were developed • Details of the phenomena behind the cost driver values • See COSYSMO book, COCOMO II book, SOSE dissertation, COSYSMO reuse dissertation, etc. for above

19. Example Exercise Mapping

20. Discussion Topics • Application domains for examples • Appropriate level of detail, decomposition, hierarchy, lifecycle coverage, etc. contained in each exercise • Decision scenarios (i.e., risk management, ROI, tradeoffs, etc.) • Any missing topics?

21. Example Exercise • Coverage: Systems engineering reuse; equivalent size; tradeoff analysis Three system configurations are under consideration that employ systems engineering reuse in varying degrees. The reuse categories are from the COSYSMO reuse model. Option 1: All New Development Option 2: Adapt Legacy System 50% Deleted from the legacy system The system will then be comprised of: 50% New 50% Modified Option 3: Product Line Approach 25% Designed for Reuse (across product line) 25% Modified 50% Managed

22. Example Exercise • Coverage: Software cost; software reuse; tradeoff analysis; military application A top Army project is for an application that enables soldiers to find the location of resources for Forward Operating Bases (FOBs). To achieve the Geo-Mapping capability, a geographic software vendor is offering a tailored version of Google Maps at an up-front license price of $500K. However, an alternate proposal is to use a dedicated team with expertise in geographic applications and Google Maps to build a better and cheaper map system. Prepare a COCOMO estimate of the custom development cost using these inputs: Adapt the Google Map software of 80 KSLOC with the reuse parameters: % Design Modified = 35 % Code Modified = 20 % Integration Required = 60 Assessment and Assimilation = 0 Software Understanding = 10 Unfamiliarity = 0.2

23. Example Exercise (Cont.) • Coverage: Software cost; software reuse; tradeoff analysis; military application Cost drivers are all Nominal except for the following: • Programmer Capability and Analyst Capability: 75th percentile • Personnel Continuity: stable staff with 3% annual turnover • Multi-site Development: entire team is co-located together • Use of Software Tools: strong toolset, moderately integrated The cost per person-month is $8K. Additionally, the manager believes that they would have no case if they delivered the software system too late. Thus they must have the software completed 6 weeks ahead of the deadline, which would suppress the schedule to 85% of the nominal schedule. Determine the estimated cost of adapting the Geo-Mapping capability for the application. Is it a better option from a cost standpoint than the vendor’s tailored product?

24. Example Exercise • Coverage: Software and hardware lifecycle costs; software reuse; time value of money Develop an integrated software and hardware lifecycle cost over time using the equipment procurement from #1, the mini-drone first unit cost from problem #2, and the following for software labor. Software Portion The newly developed software size is 250,000 SLOC and modified software size is 90,000 SLOC. - the schedule compression ratio is 85%- the effect of a software failure is a large equipment loss- average language and tool experience is 3 years- there is expected to be a major change in the operating system once per year- use of a comprehensive toolset fully integrated across all activities. For the modified software:- 40 out of 200 design modules were modified- 30 KSLOC out of the total 90 KSLOC were modified- the integration of the modified code is estimated to be 2/3 of the effort compared to if it was all new- only basic module search and documentation was required for its assessment and assimilation - the team did not develop the software being modified and are unfamiliar with it. - to understand the software, it is described as: structure - high cohesion, low couplingapplication clarity - moderate correlation between program and applicationself-descriptiveness - moderate level of comments, headers, documentation.

25. Example Exercise (cont.) • Coverage: Software and hardware lifecycle costs; software reuse; time value of money Create a monthly budget profile of costs for the full project including labor by software engineering phase, the first drone development costs and non-labor items for the testing equipment. The drone development will dovetail with the software completion. Assume its monthly costs are constant for the duration of software Inception through Construction (i.e. spread evenly over this time). It finishes when software Transition starts. Assume a nominal annual interest rate of 5% and labor rate of $10K/Person-Month.  Your final output should provide the costs for every month of the project with separate items for the individual software engineering phases, hardware development, and other procurement costs. Provide the lifecycle totals for software, hardware, procurement and grand total cost.

26. Example Exercise • Coverage: Systems engineering process improvement; ROI In a systems engineering organization, automated tools for architectural development and verification have been developed along with other process improvements. Their re-calibrated COSYSMO equation shows a reduced size exponent from 1.06 to 1.03: Effort (Person-Months) = .254* System Requirements1.03 On average, the organization develops 500 Equivalent System Requirements per year at $10,000/Person-Month. What is their estimated cost savings per year compared to before the improvements? What is their ROI over time for 5 years if the improvements cost 2 person-years?

27. Example Exercise • Coverage: Systems and software lifecycle costs; equivalent size; time value of money; tradeoff analysis; military application The Navy desires to develop information dominance capabilities to satisfy national security strategy objectives for Maritime Domain Awareness (MDA). These MDA capabilities will be developed through the use of new and emerging commercial software technologies. By combining these technologies, different Courses of Action (COAs) for the capability of “Develop Awareness” were developed for the OPNAV Capability Based Assessment. COA 1 assumes that the technologies will be inserted within 13 existing systems. COA 2 creates a new architecture that combines the systems into one overall system, so the software to be developed is dramatically reduced. These ground rules and assumptions for these options are listed below. Ground Rules and Assumptions for the Materiel Solutions 1. The wage inflation rate is 3% per year for all years beyond FY12. 2. Annual maintenance costs are 25% of the total software cost and do not start until the software development is complete. 3. Use $10,000 cost per person month for systems and software engineers (constant year dollars). 4. COAs are costed independently. 5. Assume both COAs start at beginning of FY12.

28. Example Exercise (Cont.) • Coverage: Systems and software lifecycle costs; equivalent size; time value of money; tradeoff analysis; military application

29. Example Exercise (Cont.) • Coverage: Systems and software lifecycle costs; equivalent size; time value of money; tradeoff analysis; military application Use the above information to estimate an annual cost of COAs 1 and 2 for the years of FY12 to FY17. List the costs per year in constant (FY 12) and future (then year) dollars. Assume that systems and software engineering are sequential activities, with software engineering not starting until the systems engineering is complete. Only the software Elaboration and Construction phases are to be estimated, as requirements will already be defined and transition starts immediately. Software maintenance costs start incurring when software engineering is complete (after Construction). The annual wage inflation rate is 3% (as FY12 dollars are used, start inflation in FY13). It can be assumed that all systems engineering costs will occur in Year 1 (FY12) for both COAs.