17 July, 2003

1. ONR Advanced Distributed Learning USN EDO Application Greg Chung Allen Munro Bill Bewley Quentin Pizzini Girlie Delacruz USC/BTL Terry Vendlinski Zenaida Aguirre-Munoz UCLA/CRESST 17 July, 2003 2003 Regents of the University of California

2. CRESST/BTL/USN EDO Collaboration • Research to develop new kinds of assessment and instructional tools to support USN EDO school • Initial domain: Risk management within the integrated case study • Tool prototypes: • Knowledge mapping (assessment) • Decision-analysis (instruction)

3. Assessment Focus • Knowledge mapping as a way to measure problem solving • Examine students’ understanding of the decision analysis process • Examine the extent to which domain-independent links can be used to measure problem solving processes

4. Context • Integrated Case Study (ICS) • Problem-based learning • Teams are given broad requirements • Deliverable is a professional-level briefing • Problem solving, teamwork, communication, prior job experience • Concepts covered in instruction are put into practice in ICS

5. Domain • Risk Management • Complex — multiple variables to consider, broad requirements, imperfect information, changing conditions • No “correct” answer — trade-offs among cost, schedule, and performance variables • Reasoning and analytical processes are critical to success

6. The Tools • Knowledge maps • Students depict their understanding of the decision process underlying the refueling at sea (RAS) “hiccup” • Students’ maps can be compared to an expert’s map for assessment purposes • Decision-analysis • Tool permits quantitative or qualitative views of problem solution alternatives • Instruction can be delivered in the context of the tool

7. Options Domain Variables Selection Decision Analysis Template Environment Analysis

8. Options Environment Cost Selection Risk Management Schedule Performance Analysis

9. Cost Selection Risk Management: Refueling at Sea Hiccup Environment • Requirements • Regulations • Legal issues Options • Existing system • Recompete • Full inventory buy • Technical data plan buy • Full + TDP buy • No. ships • Time constraints Schedule • Config. Mgt • Manufacturing • Contracting • Supportability • T&E Performance Analysis

10. Assessment Tools • How do we measure an EDO’s problem solving processes within the risk management domain? • Knowledge maps • Online processes and decisions with decision-analysis simulation tool

11. Refueling At Sea (RAS) Scenario

12. Refueling At Sea (RAS) Scenario Options Cost Analyze Schedule Environment Performance Select

13. Refueling At Sea (RAS) Scenario

14. Current Status • Instructor consensus knowledge map created for RAS scenario • Domain-independent links • Harder for processes (vs. declarative) • Level of abstraction issues • Domain-independent: context • Domain-dependent: requirement, legal, regulation

15. Next Steps • Pilot test at EDO with EDO instructor and EDO class (July/Aug) • Evaluate and revise for next class

16. Decision Aiding Tool http://btl.usc.edu/WS/rasProb/rasProb.jnlp

17. Decision Making and Risk Management • A major aspect of EDO education • The Refueling at Sea (RAS) problem • Evaluating options systematically • A tool that promotes systematic assessments of risks and remediations • A tool that teaches about itself and about risk and decision-making

18. Decision Aiding Tool Demo T&E = Testand Evaluation RAS = Refuelingat Sea system TDP = Technical Data Package RAS = Refuelingat Sea system User can set estimatedprobabilities, utilities. User created nodeshave arbitrary names. Decision Thresholds Controller

19. Decision Aiding Tool Instruction

20. Analyzing the User • Application of Chung’s ‘Click-through’ interface.

21. General Application • Instructional approach can provide opportunities for students to engage in systems engineering thinking • Assessment approach can provide information on what people know and understand about processes