German Workshop on Experience Management 2003

1. German Workshop on Experience Management 2003 Experience Management within Project Management Processes Maya Kaner and Reuven Karni Industrial Engineering and Management Technion – Israel Institute of Technology

2. Experience Management within Project Management Processes Project management is a knowledge intensive activity. Managers use their skills and experience to make decisions during the execution of a development process. Better results in terms of meeting schedule budget and functionality are usually achieved due to the manager’s accumulated experiences and knowledge. (Barros, Werner and Travassos)

3. Aims of the research • To propose an process oriented architecture for project management experience management • To define metrics for interpreting knowledge and transferring experience along the process • To study the effectiveness of a knowledge-based architecture in terms of these metrics

4. Definitions • Project management • Project management is the application of knowledge, skills, tools and techniques to project activities in order to meet or exceed stakeholder expectations (PMBOK) • Project management processes • Those processes within a project which are concerned with planning, organizing, managing and controlling a project in the areas of integration, scope, time, cost, quality, human resources, communications, risk and procurement (PMBOK)

5. The Project Management Body of Knowledge • The PMBOK provides a knowledge perspective of project management • It divides project knowledge into an hierarchy of project knowledge areas and project management processes • It details nine knowledge areas and 39 project management processes • This stresses the wide range of knowledge and experience required by the project manager

6. Definitions • Experience management • A special form of knowledge management which deals with task-based and experiential knowledge that is both gained and applied when carrying out business-related tasks (Minor) • Experience-based knowledge • Knowledge is not so much a capacity for specific action, but the capacity to use information, learning and experience resulting in an ability to interpret information and ascertain what information is necessary in decisionmaking (Watson)

7. Definitions • Knowledge focus • The compilation and correlation of knowledge relevant to the execution of a knowledge-intensive activity • Metric • A measure of a particular characteristic of performance or efficiency • In our case this is the ability to interpret knowledge and transfer experience

8. Scope • Business-related tasks • Our specific tasks are those concerned with knowledge application in decisionmaking within project management • Project-related experience • The specific experience investigated is that obtained and employed by project managers when making decisions (our “target”)

9. Research theory • Based on a schema proposed by March and Smith • “Design and natural science research in information technology” • Research deliverables or outputs are • Constructs – concepts or vocabulary of the domain • Model – propositions expressing relationships amongst the constructs • Method – a set of steps to perform a task

10. Research theory • Research activities include • Build • Evaluate • “We build an artifact to perform a specific task. The basic question is: does it work? • We evaluate artifacts to determine if we have made any progress. The basic question is: how well does it work? • Evaluation requires the development of metrics and the measurement of artifacts according to those metrics”

11. Research theory • Research metrics include • Constructs – completeness, simplicity, understandability, ease of use • Model – completeness, level of detail, consistency • Method – operability, efficiency, generality, ease of use

12. Our research outputs – constructs Knowledge area level (K) Process level (P) Issue level (I) Case level (C) Entity level (E) Value level(V)

13. Our research outputs – constructs Knowledge area level (K) Project management body of knowledge (PMBOK) Process level (P) Issue level (I) Case level (C) Case base Entity level (E) Value level(V)

14. Our research outputs – model (I) (processes, issues and cases) K P I/C E V Human resources Staff acquisition Select Candidate System Experience Budget Beginner C Risk mgt. Risk response Corrective action Problem Response Experience Delay R Time mgt. Activity definition Duration estimating Duration Cost 1.5 – 3 wks 30 – 40K E

15. Our research outputs – model (II) (dyads and paths) (+,+,+,+,-,-) R 14 (+,+,+,+,+,+) E 18 C 16 (+,+,+,+,+,+) E 18 (+,+,+,+,-,-) +valuerecorded - novalue

16. Our research outputs – method • (creation of a knowledge focus) • Five case-based and path-based activities • Define case retrieval input from a decision scenario using “case-by-example” • Retrieve all corresponding cases • Select relevant cases to retrieve paths • Retrieve all path dyads and interpret them • Synthesize cases and path dyads to create a knowledge focus related to the decision scenario • Carry out any feedback required to retrieve • other cases or paths

17. Our research – analysis of a decision scenario • (creation of a knowledge focus) • Scenario (text) • "The manager of an information systems implementation project is required to select the best candidate for analyzing a human resource management system. He can choose a beginner, someone with experience, or an expert. The decision should be based on knowledge from past projects, concentrating on the trade-off between inexperience risks versus salary costs"

18. Knowledge and experience metrics • Mappability • Definition – the ability to map project processes (PMBOK) into issues • Experience – associating project process outputs with the decisions (usually selection) needed to create them • Measure – percentage of project process outputs (PMBOK) mapped into issues

19. Knowledge and experience metrics • Compliance • Definition – the ability to map a textual decisionmaking situation into cases-by-example expressed by the issue-case-entity lexicon • Experience – associating textual formats with ontological formats • Measure – number of cases-by-example created from the decisionmaking situation description

20. Knowledge and experience metrics • Accordance • Definition – the ability to map a graphical representation of a path into a textual interpretation of the relationships expressed by the path • Experience – associating nodes and arcs with specific successive actions • Measure – graph traverse mapped into causality

21. Knowledge and experience metrics • Convergence • Definition – the ability to map a set of retrieved cases and paths onto a knowledge focus • Experience – synthesizing knowledge from a set of retrieved sources • Measure – cases and paths mapped into knowledge focus

22. Knowledge and experience metrics • Cover • Definition – the knowledge resources compiled to create a knowledge focus • Experience – the ability to retrieve and compile knowledge (completeness) • Measure – non-redundant cases and paths compiled towards a knowledge focus

23. Knowledge and experience metrics • Efficiency • Definition – the time required to create a knowledge focus • Experience – the ability to retrieve and compile knowledge (speed) • Measure – the time required to create a knowledge focus

24. Measures (scoring) • We score performance by counting the number of errors in the “knowledge items” required to successfully carry out each step in the process. • In this way we can learn from the experiences of the subjects in order to reduce errors and thus improve the methodology

25. Measures – Mappability • (issues connected with process outputs) • Errors: outputs not mapped into issues • (out of scope of this presentation)

26. Measures – Compliance • (decomposition of a text into issues) • Errors: faulty search for relevant cases • Redundant search – CBE values not directed • unneeded knowledge • Incomplete search – CBE values not relevant • unneeded knowledge • Constrained search – too many CBE values • missing knowledge • Null search – case/issue not identified missing knowledge

27. Measures – Accordance • (interpretation of a path graph) • Errors: arcs or nodes overlooked or misinterpreted • Null – arc overlooked • missing causality • Null – case overlooked missing knowledge • Incomplete – arc misinterpreted missing causality • Incomplete – case misinterpreted missing knowledge

28. Measures – Convergence • (creation of a knowledge focus) • Errors: cases or paths overlooked or misinterpreted • Null – focus component overlooked • missing focus • Incomplete – focus component misinterpreted misleading focus

29. Measures – Cover • (cases/paths required for knowledge focus) • Errors: excessive cases or paths • Cases – too many cases retrieved • blurred knowledge focus • Paths – too many paths retrieved • blurred knowledge focus

30. Measures – Efficiency • (total time required to derive knowledge focus) • Errors: excessive time required • Time – extra time required • misdirected knowledge focus

31. Experiment – Compliance • (decomposition of a text into issues) • Scenario (text) • "The manager of an information systems implementation project is required to select the best candidate for analyzing a human resource management system. He can choose a beginner, someone with experience, or an expert. The decision should be based on knowledge from past projects, concentrating on the trade-off between inexperience risks versus salary costs"

32. Experiment – Compliance • (decomposition of a text into issues) • Solution (cases-by-example) • C (candidates) • Specialization = “Analysis” • System = “Human resource management” • Aim = Experience and salary of past HRM • candidates • E (time and cost estimates) • Specialization = “Analysis” • System = “Human resource management” • Aim = If and when original estimates updated • R (risks) • Category = “Project team member(s)” • Type = “Experience” • Aim = Risks of team inexperience and responses

33. Experiment – Compliance • (decomposition of a text into issues) • Experiment • Groups • Control given case lexicon at start of experiment • Treatment given case lexicon before start of experiment • Results – average errors (p < 0.01) • Control n = 30; average = 3.8; variance = 2.9 • Treatment n = 30; average = 2.0; variance = 1.0 • Inference • Treatment group makes fewer errors • The experience of studying the lexicon contributes to better • performance

34. Experiment – Compliance • (decomposition of a text into issues) • Experiment • Errors – misdirected input • Redundant search 46% • Constrained search 24% • Null search 20% • Incomplete search 9% • Errors – cases involved • Risks 87% • Estimates 9% • Candidates 3% • Interpretation of main errors • Fear of missing knowledge leads to redundancy • Complexity of risk case causes problem with decomposition

35. Experiment – Accordance • (interpretation of a path graph) • Scenario (graph) (+,+,+,+,-,-) R 14 (+,+,+,+,+,+) E 18 C 16 (+,+,+,+,+,+) E 18 (+,+,+,+,-,-) +value recorded - no value

36. Experiment – Accordance • (interpretation of a path graph) • Solution (textual interpretation) • C 16 (+,+,+,+): A beginner with salary 10-15K was selected for coding the maintenance system • E 18 (+,+,+,+,-,-): This led to an estimate of the corresponding activity duration (1.5 – 2 wks) and cost (30 – 40K) • R 14 (+,+,+,+,-,-): This also led to anticipating a problem of lack of experience which could be triggered by a change in requirements • E 18 (+,+,+,+,+,+): Foreseeing this problem then led to a revision of the estimated activity duration (2 – 3 wks) and cost (50 – 60K) • R 14 (+,+,+,+,+,+): This in turn led to an expectation of project delay

37. Experiment – Accordance • (interpretation of a path graph) • Experiment • Groups • Control none • Treatment given case details • Results – average errors • Treatment n = 30; average = 1.9; variance = 1.6 • Inference • The graph represents “experience” in linking decisions; the subjects were require to reproduce this experience

38. Experiment – Accordance • (interpretation of a path graph) • Experiment • Errors – incorrect case interpretation • Risks 16% • Estimates 9% • Candidates 9% • Errors – incorrect arc interpretation/overlook • Arc E – R 30% • Other arcs (3) 12% • Candidates 3% • Interpretation of main errors • Inability to interpret the evolution of the risk case along the path sequence • Inability to interpret the apparent “loop” in the path

39. Experiment – Convergence • (creation of a knowledge focus) • Scenario (text) • "The manager of an information systems implementation project is required to select the best candidate for analyzing a human resource management system. He can choose a beginner, someone with experience, or an expert. The decision should be based on knowledge from past projects, concentrating on the trade-off between inexperience risks versus salary costs"

40. Experiment – Convergence • (creation of a knowledge focus) • Five case-based and path-based activities • Define case retrieval input from the decision scenario using “case-by-example” • Retrieve all corresponding cases • Select relevant cases to retrieve paths • Retrieve all path dyads and interpret them • Synthesize cases and path dyads to create a knowledge focus related to the decision scenario • Carry out any feedback required to retrieve • other cases or paths

41. Experiment – Convergence • (control – no guidance given) • Four CBR and PBR actions (icons) • Define case retrieval input from the decision scenario using “case-by-example” • Retrieve all corresponding cases • Select relevant cases to retrieve paths • Retrieve all path dyads and interpret them • Press any icon to activate the corresponding action

42. Experiment – Convergence • (treatment – process map given) • Four linked CBR and PBR actions (icons) • Define case retrieval input from the decision scenario using “case-by-example” • Retrieve all corresponding cases • Select relevant cases to retrieve paths • Retrieve all path dyads and interpret them • Carry out any feedback required to retrieve • other cases or paths

43. Experiment – Convergence • (creation of a knowledge focus) • Experiment • Groups • Control carry out any of the activities, in any order • Treatment carry out all four activities in a given order • (process) • Results – average errors (p < 0.01) • Control n = 15; average = - 0.9; variance = 0.5 • Treatment n = 15; average = - 0.5; variance = 0.3 • Inference • The treatment group is better able to synthesize the knowledge retrieved and to relate it to the decision to be made • A systematized process contributes to better performance • Experience – knowledge about the sequence in which to retrieve and interpret knowledge – contributes to better performance

44. Experiment – Efficiency • (total time required to derive knowledge focus) • Experiment • Groups • Control carry out any of the activities, in any order • Treatment carry out all four activities in a given order • (process) • Results – average errors (p < 0.05) • Control n = 15; average = 23 minutes; variance = 37 • Treatment n = 15; average = 19 minutes; variance = 21 • Inference • The treatment group is faster in generating the knowledge focus • A systematized process contributes to faster and more uniform performance • Experience – knowledge about the sequence in which to retrieve and interpret knowledge – contributes to better performance

45. Experiment – Cover (cases) • (cases required for knowledge focus) • Experiment • Groups • Control carry out any of the activities, in any order • Treatment carry out all four activities in a given order • (process) • Results – average errors (lower bound = 3) (p < 0.05) • Control n = 15; average = + 1.47; variance = 2.6 • Treatment n = 15; average = + 0.40; variance = 1.4 • Inference • The treatment group requires less to find a meaningful input • Systematization gvies more effective and uniform performance • Systematization gives more confidence and thus less scope • Experience – knowledge about the sequence in which to retrieve and interpret knowledge – contributes to better performance

46. Experiment – Cover (paths) • (paths required for knowledge focus) • Experiment • Groups • Control carry out any of the activities, in any order • Treatment carry out all four activities in a given order • (process) • Results – average errors (lower bound = 3) • Control n = 15; average = + 3.27; variance = 2.8 • Treatment n = 15; average = + 3.47; variance = 3.3 • Inference • Both groups tend to search the same number of paths • However, the treatment group retreieved these pathns in an organized fashion • Experience – knowledge about the sequence in which to retrieve and interpret knowledge – contributes to better performance

47. Conclusions • Business process oriented knowledge management (BPOKM) means that knowledge is structured according to process requirements • Experience oriented knowledge management means that knowledge is created and updated through the experience of those applying the knowledge • The project manager looks for suitable candidates (process) and seeks the effects of previous decisions (experience) on time, costs and risks as stored in the knowledge base

48. Conclusions • Our aim is to create an effective knowledge focus – the right number, identification and interpretation of the retrieved cases and paths • At each step of the process, knowledge focus errors may occur • These have been measured by a uniform metric – a count of the errors incurred • Errors reflect redundant or missing or misinterpreted knowledge elements (cases or paths) as compared to an “ideal” search for the knowledge required

49. Conclusions • In general we find that systematization of the decision process and the link between process and knowledge required leads to better performance • Our methodology allows us to locate areas and reasons for non-performance, such that the project management decisionmaking process can be improved

50. To sum up … • Knowledge bases can be structured by adopting a process description as one specific foundation for the schema of the knowledge base. This facilitates access since in a process oriented application the user knows his exact position in the process and can therefore easily navigate the knowledge base. • Since the process description functions as a structuring criterion for a knowledge base, all the relevant information and knowledge for a process step – that goes beyond the capability of a conventional process model – can be captured. • (Jablonski, Horn and Schlundt)