Intelligent Approaches to Lessons Learned Processes

1. Intelligent Approaches to Lessons Learned Processes Rosina O Weber University of Wyoming Navy Center for Applied Research in AI, Naval Research Lab

2. Intelligent Approaches to Lessons Learned Processes Collaborators: David W. Aha Hector Munoz Avila Len Breslow Nabil Sandhu R.O.Weber Calgary 3 Aug 2001

3. Outline • Introduction • Context • Problems with lessons learned systems • Methodology • Monitored Distribution • Case Representation • Lesson Elicitation Tool • Next Steps R.O.Weber Calgary 3 Aug 2001

4. Context • Knowledge management context • Lessons learned systems (LLS) • Organizations adopting LLS • Lessons learned definition, representation and example • Lessons learned process R.O.Weber Calgary 3 Aug 2001

5. Knowledge management context • Three types of KM initiatives* • knowledge repositories • knowledge access and transfer • knowledge environment • Knowledge repositories • Internet • industry oriented (alert systems) • organization oriented (lessons learned systems) R.O.Weber Calgary 3 Aug 2001 5

6. KNOWLEDGE ARTIFACTS Lessons learned systems • Lessons learned systems are knowledge repositories of knowledge artifacts • Examples of knowledge artifacts are lessons, alerts, best practices, reports, video clips, etc. R.O.Weber Calgary 3 Aug 2001 6

7. Lessons learned systems in organizations • Aha & Weber (Eds.)Intelligent lessons learned Systems. Papers from the AAAI 2000 Workshop(Technical Report WS-00-03) AAAI Press • Weber, Aha & Becerra-Fernandez (survey) Intelligent lessons learned Systems. 2001 International Journal of Expert Systems Research & Applications, Vol. 20, No. 1., 17-34. R.O.Weber Calgary 3 Aug 2001 7

8. government Construction Industry Inst. Honeywell GM Hewllet Packard Bechtel Jacobs Company Lockheed Martin E. Sys, Inc DynMcDermott Petroleum Co. Xerox IBM BestBuy Siemens int’l US European Space Agency Italian (Alenia) French (CNES) Japanese (NASDA) United Nations Air Force Army Coast Guard Joint Forces Marine Corps Navy int’l US Department of Energy: SELLS NASA (Ames, Goddard) Canadian Army Lessons Learned Centre non-government non-military military 8

9. Lessons learned definition… …or organizational lessons, lessons, lessons identified Definition: A lesson learned is a knowledge or understanding gained by experience. The experience may be positive, as in a successful test or mission, or negative, as in a mishap or failure. A lesson must be significant in that it has a real or assumed impact on operations; valid in that is factually and technically correct; and applicable in that it identifies a specific design, process, or decision that reduces or eliminates the potential for failures and mishaps, or reinforces a positive result.” (Secchi et al., 1999) Definition: A lesson learned is a knowledge or understanding gained by experience. The experience may be positive, as in a successful test or mission, or negative, as in a mishap or failure. A lesson must be significant in that it has a real or assumed impact on operations; valid in that is factually and technically correct; and applicable in that it identifies a specific design, process, or decision that reduces or eliminates the potential for failures and mishaps, or reinforces a positive result.” (Secchi et al., 1999) Definition: A lesson learned is a knowledge or understanding gained by experience. The experience may be positive, as in a successful test or mission, or negative, as in a mishap or failure. A lesson must be significant in that it has a real or assumed impact on operations; valid in that is factually and technically correct; and applicable in that it identifies a specific design, process, or decision that reduces or eliminates the potential for failures and mishaps, or reinforces a positive result.” (Secchi et al., 1999) Definition: A lesson learned is a knowledge or understanding gained by experience. The experience may be positive, as in a successful test or mission, or negative, as in a mishap or failure. A lesson must be significant in that it has a real or assumed impact on operations; valid in that is factually and technically correct; and applicable in that it identifies a specific design, process, or decision that reduces or eliminates the potential for failures and mishaps, or reinforces a positive result.” (Secchi et al., 1999) Definition: A lesson learned is a knowledge or understanding gained by experience. The experience may be positive, as in a successful test or mission, or negative, as in a mishap or failure. A lesson must be significant in that it has a real or assumed impact on operations; valid in that is factually and technically correct; and applicable in that it identifies a specific design, process, or decision that reduces or eliminates the potential for failures and mishaps, or reinforces a positive result.” (Secchi et al., 1999) R.O.Weber Calgary 3 Aug 2001 9

10. Lessons learned example applicable task Installing custom stereo speakers. conditions for applicability The car is the Porsche Boxster. lesson suggestion Make sure you distinguish the wires leading to the speakers from the wires leading to the side airbag. Rationale Somebody has cut the wrong wire because they look alike and the airbag went off with explosive force. This means spending several thousand dollars to replace the airbag in addition to be a potential hazard. From article “Learning from Mistakes” about Best Buy in Knowledge management magazine, April 2001. R.O.Weber Calgary 3 Aug 2001 11

11. Lessons learned process R.O.Weber Calgary 3 Aug 2001 12

12. Lessons distribution sub process R.O.Weber Calgary 3 Aug 2001 20

13. Lesson distribution methods Broadcasting bulletins, doctrine Passive Standalone repository Active casting list servers, information gathering tools Pull Push R.O.Weber Calgary 3 Aug 2001 20

14. Problems with lesson distribution methods • Distribution is divorced from targeted organizational processes. • Users maynot know or be reminded of the repository, as they need to access a standalone tool to search for lessons. • Users maynot beconvinced of the potential utility of lessons. • Users maynot have the time and skills to retrieve and interpret relevant lessons. • Users maynotbe able to apply lessons successfully. R.O.Weber Calgary 3 Aug 2001 21

15. Organization’s members Repository of lessons learned Here is a gap Organizational processes R.O.Weber Calgary 3 Aug 2001

16. Organization’s members Repository of lessons learned How to bridge this gap? Organizational processes R.O.Weber Calgary 3 Aug 2001

17. Organization’s members Repository of lessons learned How to bridge this gap? Organizational processes R.O.Weber Calgary 3 Aug 2001

18. Organization’s members Repository of lessons learned How to bridge this gap? Organizational processes R.O.Weber Calgary 3 Aug 2001

19. Organization’s members Repository of lessons learned How to bridge this gap? Organizational processes R.O.Weber Calgary 3 Aug 2001

20. Organization’s members Repository of lessons learned How to bridge this gap? Organizational processes R.O.Weber Calgary 3 Aug 2001

21. Organization’s members Repository of lessons learned How to bridge this gap? Organizational processes R.O.Weber Calgary 3 Aug 2001

22. Organization’s members Repository of lessons learned How to bridge this gap? Organizational processes R.O.Weber Calgary 3 Aug 2001

23. Organization’s members Repository of lessons learned How to bridge this gap? Organizational processes R.O.Weber Calgary 3 Aug 2001

24. Organization’s members Repository of lessons learned How to bridge this gap? Organizational processes R.O.Weber Calgary 3 Aug 2001

25. Organization’s members Repository of lessons learned Monitored distribution Organizational processes R.O.Weber Calgary 3 Aug 2001

26. Organization’s members Repository of lessons learned Monitored distribution Lesson repository is integrated with targeted processes …and lessons aredistributed when and where they are needed. Organizational processes R.O.Weber Calgary 3 Aug 2001 22

27. Problems with lessons learned systems • Technological • Human • Managerial R.O.Weber Calgary 3 Aug 2001 13

28. Technological issues Standalone distribution outside context of reuse Lessons disseminated in context of reuse Low precision and recall in text databases Case retrieval for lesson dissemination • cases indexed by applicability Convert lessons into cases/Collect cases Requirement: machine recognizable format Textual representation of lessons Case strutcture for lessons R.O.Weber Calgary 3 Aug 2001 14

29. Human issues Lesson authors Lesson validators Lesson (re)users • Lack of training/instructions: content and format • Hard to validate textual descriptions • Have to access the repository in another context • Have to accept the potential benefit of lessons • Have the skills to search for lessons • Have to interpret textual lessons R.O.Weber Calgary 3 Aug 2001 15

30. Managerial issues • Determine, communicate and enforce standards for lesson collection and representation • Define structured format • Embed knowledge in targeted processes • Monitor knowledge transfer • Oversee knowledge reuse Knowledge collection Knowledge validation Knowledge reuse R.O.Weber Calgary 3 Aug 2001 16

31. Monitored distribution Case retrieval Lessons as structured cases Lesson elicitation tool that embeds instructions for lesson submission and converts them into the structured format Technological, Human, Managerial issues Standalone repository Retrieval method Textual format of lessons Collection method embed instructions for lesson submission R.O.Weber Calgary 3 Aug 2001 17

32. Methodology Prescriptive KM infra structure for frameworks human users • Monitored distribution • Case representation • Lesson elicitation tool that embeds instructions for lesson submission and converts them into the structured format LET organizational processes lesson repository for monitored distribution Case base case of lessons base R.O.Weber Calgary 3 Aug 2001 18

33. Frameworks • Monitored Distribution • Case represebtation • Lesson Elicitation Tool R.O.Weber Calgary 3 Aug 2001 19

34. Monitored Distribution

35. Problems & Solutions Lessons are integrated to targeted organizational processes. Users don’t need to be reminded of the repository because they don’t need to access a standalone tool. No additional time or skills are required. Case retrieval of disambiguated knowledge increase recall and precision Whenever possible, an ‘apply’ button allows the lesson to be automatically executable. Distribution is divorced from targeted organizational processes. Users maynot be reminded of the repository, as they need to access a standalone tool to search for lessons. Users maynot have the time and skills to retrieve relevant lessons. Text databases have low levels of precision and recall Users maynotbe able to apply lessons successfully. 20

36. Monitored distribution characteristics • Distribution is tightly integrated to the targeted processes and distribute lessons when and wherethey are needed. • Represent lessons as form-like cases. • Distribute lessons using case-retrieval/ and retrieve lessons based on similarity. • Additional benefits are: • Case representation facilitates interpretation. • Users access the lesson rationale to evaluate its potential utility. • Cases are retrieved in the context of similar experiences. R.O.Weber Calgary 3 Aug 2001 21

37. Noncombatant Evacuation Operations: Military operations to evacuate noncombatants whose lives are in danger and rescue them to a safe haven R.O.Weber Calgary 3 Aug 2001

38. Assembly Point Campaign headquarters Intermediate Staging Base . safe haven NEO site

39. Example in HICAP • HICAP is a plan authoring tool suite http://www.aic.nrl.navy.mil/hicap • Muñoz-Avila et al., 1999 • Users interact with HICAP by refining an HTN (hierarchical task network) through decompositions R.O.Weber Calgary 3 Aug 2001 24

40. safe haven NEO site R.O.Weber Calgary 3 Aug 2001

41. Selecting the Suggested Case… R.O.Weber Calgary 3 Aug 2001

42. Expanding yields… R.O.Weber Calgary 3 Aug 2001

43. And the user is notified of a lesson RATIONALE: TYPE: advice Clandestine SOF should not be used alone WHY: The enemy might be able to infer that SOF are involved, exposing them. RATIONALE: TYPE: advice Clandestine SOF should not be used alone WHY: The enemy might be able to infer that SOF are involved, exposing them. RATIONALE: TYPE: advice Clandestine SOF should not be used alone WHY: The enemy might be able to infer that SOF are involved, exposing WHY: The enemy might be able to infer that SOF are involved, exposing R.O.Weber Calgary 3 Aug 2001

44. After applying the lesson R.O.Weber Calgary 3 Aug 2001

45. Evaluation • Hypothesis • Using lessons will improve plan quality • Methodology • Simulated HICAP users generated NEO plans with and without lessons • NEO executor implemented plans • Plan total duration • Plan duration before medical assistance • Casualties: evacuees, FF, enemies R.O.Weber Calgary 3 Aug 2001 30

46. Plan evaluator • non deterministic (100 plans 10 times each) • 30 variables: 12 random • length of plans 18 steps • size of planning space 3,000,000 • 13 lessons • Actions and their influences R.O.Weber Calgary 3 Aug 2001 31

47. Plan evaluator: actions • Plans where evacuees were transported by land modes have an increased chance of being attacked by enemies. • Plans that combined weather with too strong wings have a small chance of helicopter crash. • When an attack or crash happens it increases the number of casualties among evacuees and FF (in proportion to # of evacuees). R.O.Weber Calgary 3 Aug 2001 32

48. Plan evaluator: lessons Conditions for applicability: There are representatives of different branches assigned to participate. Lesson suggestion: Assign representatives of all forces to plan. Rationale:Lack of representatives prevent good communication causing delays and miscommunication. • Conditions for applicability: • There are hundreds or more evacuees as to justify a security effort. • Lesson suggestion: • Assign EOD personnel. • Rationale:An evacuee once asked what to do with their weapons. R.O.Weber Calgary 3 Aug 2001 33

49. Results *The resulting values are averages no lessons with lessons reduction NEO plan total duration* 32h48 18 % 39h50 duration until medical assistance* 24h13 18 % 29h37 casualties among evacuees 24 % 11.48 8.69 casualties among friendly forces 6.57 30 % 9.41 casualties among enemies -2 % 3.08 3.14 R.O.Weber Calgary 3 Aug 2001 34

50. Case representation