DoD Systems Engineering and Management Implications for Evolutionary Acquisition of Major Defense Systems

DoD Systems Engineering and Management Implications for Evolutionary Acquisition of Major Defense Systems A DoD SERC Quick-Look Study and CSER 2010 Invited Presentation Barry Boehm, Jo Ann Lane, USC-CSSE March 17, 2010 Charts with notes

Summary: Evolutionary Acquisition (EvA) • EvA is the preferred DoD strategy for MDAPs (DoDI 5000.02) • It is more responsive to DoD’s current and future challenges • There are several forms of EvA • With different strengths and shortfalls • EvA requires significant advances in current practices • In systems engineering (SE) and development processes, particularly to avoid unscalable, easiest-first, sunny-day architectural commitments • In rapid, adaptive integration of SE and acquisition management (AM) • In financial and human resource allocation • In workforce capability and empowerment • In research and technology priorities and transition speedup • Initiatives are needed to provide these advances in time for EvA to succeed 3/1/2010

What is “Evolutionary Acquisition”? • An acquisition approach that involves • Delivering mission capability in increments • Continuing reassessment of future-increment priorities • It departs from the outdated assumptions underlying traditional acquisition • System requirements can be specified in advance • System requirements are largely stable • Full operational capability cost estimation is feasible • Full-development, up-front Integrated Master Plans, Integrated Master Schedules, and Earned Value Management Plans are feasible • Its successful application requires rethinking traditional systems engineering (SE) and acquisition practices 3/1/2010

Future DoD Challenges: Asymmetric Conflict and OODA Loops • Adversary • Picks time and place • Little to lose • Lightweight, simple systems and processes • Can reuse anything • Defender • Ready for anything • Much to lose • More heavy, complex systems and processes • Reuse requires trust Observe new/updated objectives, constraints, alternatives Orient with respect to stakeholders priorities, feasibility, risks Decide on next-cycle capabilities, architecture upgrades, plans • Act on plans, specifications Development Commitment Milestone for Cycle 3/1/2010

Average workdays to process changes Average Change Processing Time: Two Complex Systems of Systems Incompatible with turning within adversary’s OODA loop 3/1/2010

There is No One-Size-Fits-All EvA Model Time phasing terms: Scoping; Architecting; Developing; Producing; Operating (SADPO) Prespecified Sequential: SA; DPO1; DPO2; DPO3; … Evolutionary Sequential: SADPO1; SADPO2; SADPO3; … Evolutionary Overlapped: SADPO1; SADPO2; SADPO3; … Evolutionary Concurrent: SA; D1 ; PO1… SA2; D2 ; PO2… SA3; D3; PO3 … 3/1/2010

Evolutionary Acquisition (EvA) Decision Table * Example enablers: Technology maturity; External-system capabilities; Needed resources 3/1/2010

Evolutionary Overlapped Form per DoDI 5000.02Defers the start of a new increment until its technology is mature Figure 2.Requirements and Acquisition Process Flow. 3/1/2010

Evolutionary Concurrent: Rapid Change with High Assurance Unforeseeable Change (Adapt) Rapid Change Agile Rebaselining for Future Increments Future Increment Baselines Short Development Increments Deferrals Foreseeable Change (Plan) Short, Stabilized Development of Increment N Increment N Transition/ Operations and Maintenance Increment N Baseline Stable Development Increments Artifacts Concerns High Assurance Future V&V Resources Current V&V Resources Verification and Validation (V&V) of Increment N Continuous V&V 3/1/2010

“Evolution” and “Maintenance” • EvA often includes maintenance of fielded increments • Changes to fielded increments must be coordinated with development of new increments • Even if a separate organization handles maintenance • As above, there is no one-size-fits-all process for this • Can involve an evolutionary next-increment with a Milestone B gate • As in USD(AT&L) Designation of MDAP Subprograms memo [Carter, 2009] • Can also involve pools of maintainers performing low levels of as-needed maintenance not requiring Milestone Bs • Can’t have predetermined process for change traffic • Best available process shown in next chart • Situation even more complex with systems of systems • May need to integrate all types of evolution processes 3/1/2010

Agile Change Processing and Rebaselining Stabilized Increment-N Development Team Agile Future- Increment Rebaselining Team Future Increment Managers Change Proposers Defer some Increment-N capabilities Propose Changes Proposed changes Recommend handling in current increment Assess Changes, Propose Handling Recommend no action, provide rationale Discuss, revise, defer, or drop Negotiate change disposition Handle in current rebaseline Accept changes Formulate, analyze options in context of other changes Handle Accepted Increment-N changes Recommend deferrals to future increments Discuss, resolve deferrals to future increments Prepare for rebaselined future-increment development Rebaseline future-increment Foundations packages 3/1/2010

Current and Future DoD Challenges: Systems of Systems SoS SE Level* Constituent System n (pre-existing)       Increment m Increment m+1 ● ● ● Constituent System B (pre-existing)       Increment n-1 Increment n Increment n+1 MS C MS A MS B PD New System A    O&S    MSA TD EMD * DoD Systems Engineering Guide for SoS, 2008

EvA Differences: SE and Development Processes • EvA not a “one-size-fits-all” process • As shown in charts above • No clear boundary between “development,” “operations and maintenance” • Don’t release your key SE people at PDR • Short prioritized increments; Schedule as independent variable (SAIV) • Architect to add or drop borderline-priority features to meet schedule • Exceptions for non-subsettable deliverables • Earlier testing, verification, and validation (V&V) • Architecture, infrastructure V&V to avoid unscalable easiest-first increments • Earlier/continuous test, V&V • Rapid, concurrent, pro-active SE and acquisition management • Of product and process; requirements and solutions; development and evolution; hardware/software/human factors • Proactive vs. reactive with respect to technology, NDI, external interfaces • Evidence/risk as decision criterion, first class deliverable • Process for evidence preparation • Risk/opportunity-driven: avoid overkill, underkill (balance risk, opportunity) • Competitive prototyping as a way of buying information to reduce risk 3/1/2010

The SAIV* Process Model—Cross Talk, January 2002 (http://www.stsc.hill.af.mil/crosstalk) 1. Shared vision and expectations management 2. Feature prioritization 3. Schedule range estimation and core-capability determination - Top-priority features achievable within fixed schedule with 90% confidence 4. Architecting for ease of adding or dropping borderline-priority features - And for accommodating post-IOC directions of growth 5. Incremental development - Core capability as increment 1 6. Change and progress monitoring and control - Add or drop borderline-priority features to meet schedule • *Schedule As Independent Variable; Feature set as dependent variable • Also works for cost, schedule/cost/quality as independent variable • Also known as timeboxing, time-determined development, time-certain development 3/1/2010

Rapid, Adaptive Acquisition Management • Need to rework traditional guidance documents • No longer prespecified, total-package, sequential IMP, IMS , WBS, EVMS • Concurrently engineered with evidence-based decision milestones • No “one-size-fits-all” contracting • E.g., for build-to-spec, agile rebaselining, V&V teams • SE budgets a function of system size, criticality, requirements volatility • Award fees rewarding collaborative, effective change adaptation • Continuous monitoring: INCOSE leading indicators; SERC effectiveness measures, or equivalent • Evidence preparation; schedule; exit criteria • Success-critical stakeholders participation; validation by experts • Competitive prototyping critical success factors • Acquisition corps empowerment • Next generation of tools, education, and training; career paths 3/1/2010

Types of Milestone Reviews • Schedule-based reviews (contract-driven) • We’ll hold the PDR on April 1 whether we have a design or not • High probability of proceeding into a Death March • Event-based reviews (artifact-driven) • The design will be done by June 1, so we’ll have the review then • Large “Death by PowerPoint and UML” event • Hard to avoid proceeding with many unresolved risks and interfaces • Evidence-based commitment reviews (risk-driven) • Evidence provided in Feasibility Evidence Description (FED) • A first-class deliverable • Based on concurrently engineered ConOps, specs, and plans • Shortfalls in evidence are uncertainties and risks • Should be covered by risk mitigation plans • Stakeholders decide to commit based on risks of going forward 3/1/2010

Content of Evidence-Based Reviews • Evidenceprovided by developer and validated by independent expertsthat: • If the system is built to the specified architecture, it will • Satisfy the specified operational concept and requirements • Capability, interfaces, level of service, and evolution • Be buildable within the budgets and schedules in the plan • Generate a viable return on investment • Generate satisfactory outcomes for all of the success-critical stakeholders • Shortfalls in evidence are uncertainties and risks • Should be resolved or covered by risk management plans • Assessed in increasing detail at major anchor point milestones • Serves as basis for stakeholders’ commitment to proceed • Serves to synchronize and stabilize concurrently engineered elements Can be used to strengthen current schedule- or event-based reviews 3/1/2010

Scalable remotely controlled operations 3/1/2010

Total vs. Incremental Commitment – 4:1 RPV • Total commitment • Agent technology demo and PR: Can do 4:1 for $1B • Winning bidder: $800M; PDR in 120 days; 4:1 capability in 40 months • PDR: many outstanding risks, undefined interfaces • $800M, 40 months: “halfway” through integration and test • 1:1 IOC after $3B, 80 months • Incremental commitment [number of competing teams] • $25M, 6 mo. to MDD [4]: may beat 1:2 with agent technology, but not 4:1 • $75M, 8 mo. to Milestone A [3]: agent technology may do 1:1; some risks • $225M, 10 mo. to Milestone B [2]: validated architecture, high-risk elements • $675M, 18 mo. to IOC [1]: viable 1:1 capability • 1:1 IOC after $1B, 42 months 3/1/2010

Finance, Human Resource Allocation • Need balance of short-term, long-term budget commitments • Budget for long-term continuity but build in short-term off-ramp options • E.g., for technology maturity as in the DoDI 5000.02 example of EvA in chart 9 • Prespecified total-package capabilities, budgets, and schedules generally infeasible • Higher up-front SE schedules, costs for larger, more critical programs • Earlier infrastructure and test plans and preparations • Heavier penalties for late rework; generally high payoffs for competitive prototyping • Program-specific budgets, schedules, staffing established via mix of parametric models and expert judgment • Need to address total cost of ownership, lifecycle ROI (DOTMLPF) • Lifecycle success-critical stakeholder involvement • Mission effectiveness analysis with respect to alternatives; • Use to help prioritize increments • Use as basis for monitoring progress vs. plans • Continuing update of resources required 3/1/2010

Risk of Delaying System-Specific Knowledge Blanchard-Fabrycky, 1998, pg. 37 Improvement Opportunities 3/1/2010

Effect on Size on SE Investment Payoff 3/1/2010

Effect of Volatility and Criticality on SE Investment Payoff 3/1/2010

COSYSMO Operational Concept # Requirements # Interfaces # Scenarios # Algorithms + Volatility Factor Reuse Factor Size Drivers COSYSMO Effort Effort Multipliers • Application factors • 8 factors • Team factors • 6 factors • Schedule driver Calibration WBS guided by ISO/IEC 15288 3/1/2010

Workforce, Capabilities, And Empowerment • Increasing gaps in DoD SE Knowledge, Skills, and Abilities (KSAs) • Workforce gaps: domain experts retiring • Needed balance of specialists and systems thinkers • Hardware/software /human factors combined expertise needs • Needed to avoid commitments to unscalable , easiest-first EvA • Needs for life-long learning; learning how to learn • Applies to acquisition corps as well as SEs • Need to anticipate trends in workforce capabilities • Tool empowerment • Life cycle coverage, interoperability, domain tailorability • Tailorability to new and wider ranges of warfighter skills 3/1/2010

EvA Research and Technology Needs • Incremental vs. start-over tools and methods (e.g., for formal methods) • Ambiguity-tolerant methods, processes, and tools • Powerful, flexible, composable models and simulations • Rapid change-impact analysis • Scalable methods, processes, and tools • Concurrent engineering support • Collaboration technology; quality attribute tradeoff analysis tools • Value-based architecting and design methods, processes, and tools • Continuous V&V; early, continuous testing; reliable autonomy • Use of multi-core computing processors for rapid multiple-option analysis • Integration of hardware, software , and human factors solution approaches and architectures • Next-generation process maturity models • Evolutionary acquisition contract and incentive structures • Rapid technology maturation and adoption 3/1/2010

System/Software Architecture Mismatches- Maier, 2006 • System Hierarchy • Part-of relationships; no shared parts • Function-centric; single data dictionary • Interface data flows • Static functional-physical allocation • Software Hierarchy • Served-by relationships; layered multi-access • Data-centric; class-object data relations • Interface protocols; concurrency challenges • Dynamic functional-physical migration 3/1/2010

Examples of Architecture Mismatches … … … Sensor 1 Sensor 2 Sensor 3 Sensor n SDMS1 SDMS2 SDMS3 SDMSn • Fractionated, incompatible sensor data management • “Touch Football” interface definition earned value • Full earned value taken for defining interface dataflow • No earned value left for defining interface dynamics • Joining/leaving network, publish-subscribe, interrupt handling, security protocols, exception handling, mode transitions • Result: all green EVMS turns red in integration 3/1/2010

References - I Baldwin, C. and Clark, K., Design Rules: The Power of Modularity, MIT Press, 2000. Blanchard, B. and Fabrycky, W., Systems Engineering and Analysis. Upper Saddle River: Prentice Hall, 1998. Boehm, B., Ingold, D., Dangle, K., Turner, R., P. Componation et al., “Early Identification of SE-Related Program Risks,” SERC Technical Report and TR USC-CSSE-2009-518, Sept. 2009. Boehm, B., Port, D., Huang, L., and Brown, A.W., “Using the Spiral Model and MBASE to Generate New Acquisition Process Models: SAIV, CAIV, and SCQAIV,” Cross Talk, January 2002, pp. 20-25. Boehm, B., Brown, A. W., Basili, V, and Turner, R. “Spiral Acquisition of Software-Intensive Systems of Systems,” CrossTalk, May 2004, pp. 4-9. Boehm, B., Valerdi, R., and Honour, E., "The ROI of Systems Engineering: Some Quantitative Results for Software-Intensive Systems," Systems Engineering, Fall 2008, pp. 221-234. Boehm, B., and Lane, J., "Guide for Using the Incremental Commitment Model (ICM) for Systems Engineering of DoD Projects, v0.5,“ USC-CSSE-TR-2009-500. Carter, A. , “Designation of Subprograms for Major Defense Acquisition Programs, USD(AT&L) Memorandum, June 23, 2009. Cusumano, M., and Selby, R., Microsoft Secrets, Free Press, 1995. Department of Defense (DoD), Defense Acquisition Guidebook, version 1.6, http://akss.dau.mil/dag/,2006. Department of Defense (DoD), Systems Engineering Guide for System of Systems, June 2008. Department of Defense (DoD), Instruction 5000.02, Operation of the Defense Acquisition System, December 2008. Fortune, J., “Estimating systems engineering reuse with the constructive systems engineering cost model (COSYSMO),” USC Ph.D. Dissertation, December 2009. 3/1/2010

References - II Gelosh, D., “Systems Engineering Workforce Development Update,” Proceedings, NDIA SE Conference , October 2009. Hall, E.T., Beyond Culture, Anchor Books/Doubleday, 1976. INCOSE Systems Engineering Handbook v.3.1 INCOSE-TP-2003-002-03.1, 2007. Johnson , J., My Life Is Failure, Standish Group, 2006. Maier, M., “System and Software Architecture Reconciliation,”Systems Engineering 9 (2), 2006, pp. 146-159. Maranzano, J., et al. 2005. Architecture Reviews: Practice and Experience. IEEE Software, Mar./Apr. 2005. Parnas, D., “Designing Software for Ease of Extension and Contraction,” IEEE Trans. SW Engr., March 1979, pp. 128-137. Pew, R. W., and Mavor, A. S., Human-System Integration in the System Development Process: A New Look, National Academy Press, 2007. Schroeder, T., “Integrating Systems and Software Engineering: Observations in Practice, 22nd International Annual Forum on COCOMO and Systems/Software Cost Modeling, 2007. U.S. Congress, “Weapon System Acquisition Reform Act of 2009,” January 6, 2009. U.S. General Accountability Office, “Defense Acquisitions: Assessments of Selected Major Weapon Programs,” March 2008. Valerdi, R, Systems Engineering Cost Estimation with COSYSMO, Wiley, 2010 (to appear). Young, J., “Prototyping and Competition,” USD(AT&L) Memorandum, September 19, 2007 3/1/2010

DoD Systems Engineering and Management Implications for Evolutionary Acquisition of Major Defense Systems