1. Latent Problem Solving Analysis (LPSA): A computational theory of representation in complex, dynamic problem solving tasks

2. Complex problem solving (CPS) definition • dynamic, because early actions determine the environment in which subsequent decision must be made, and features of the task environment may change independently of the solver’s actions; • time-dependent, because decisions must be made at the correct moment in relation to environmental demands; and • complex, in the sense that most variables are not related to each other in one-to-one manner

3. ‘Despite 10 years of research in the area, there is neither a clearly formulated specific theory nor is there an agreement on how to proceed with respect to the research philosophy. Even worse, no stable phenomena have been observed’ (Funke, 1992, p. 25)

4. "How similar are two participant's solutions?" • For CPS there is no common, explicit theory to explain why a complex, dynamic situation is similar to any other situation or how two slices of performance taken from a problem solving task can possibly be compared quantitatively. • This lack of formalized, analytical models is slowing down the development of theory in the field.

5. Example of a complex, dynamic task: Firechief (Omodei and Wearing 1995)

6. No. Command/ Gen Perf App. App. Position Destination/ Landscape/ Event Code Type Upper Left Lower Right Wind Change 0 100.00 Wind Strength = 6 Wind Direction = East Mature Fire 0 100.00 (10, 10) Mature Fire 0 100.00 (6, 9) Mature Fire 0 100.00 (6, 8) Mature Fire 0 100.00 (9, 10) 1 Move 17 100.00 4 Copter (11, 4) (11, 9) Forest 2 Move 31 100.00 2 Truck (4, 11) (17, 7) Clearing 3 Drop Water 38 100.00 4 Copter (11, 9) Forest 4 Move 54 100.00 3 Copter (8, 6) (10, 11) Forest 5 Move 70 99.77 1 Truck (4, 14) (18, 10) Forest 6 Drop Water 77 99.42 3 Copter (10, 11) Forest 7 Move 99 99.18 4 Copter (11, 9) (21, 8) Dam 8 Move 113 99.18 3 Copter (10, 11) (12, 14) Dam 9 Control Fire 122 98.95 2 Truck (17, 7) Clearing 10 Control Fire 131 98.95 1 Truck (18, 10) Forest 11 Move 152 98.95 4 Copter (21, 8) (12, 10) Clearing 12 Drop Water 177 98.71 4 Copter (12, 10) Clearing 13 Move 187 98.71 3 Copter (12, 14) (11, 11) Clearing 14 Move 222 98.48 4 Copter (12, 10) (21, 8) Dam 15 Move 236 98.48 3 Copter (11, 11) (12, 14) Dam 16 Move 267 98.25 3 Copter (12, 14) (10, 12) Forest 17 Drop Water 273 98.01 3 Copter (10, 12) Forest 18 Move 296 98.01 2 Truck (17, 7) (8, 5) Forest 19 Move 319 96.85 4 Copter (21, 8) (7, 6) Forest 20 Move 341 96.61 3 Copter (10, 12) (12, 7) Forest 21 Drop Water 347 96.50 4 Copter (7, 6) Forest 22 Drop Water 352 96.50 2 Truck (8, 5) Forest 23 Drop Water 361 96.26 3 Copter (12, 7) Forest

7. Problems with the classic 'problem space’ approach! Most of the theories about cognitive skill acquisition and procedural learning are based in two principles: • The problem space hypothesis • Representation of procedures as productions

8. Problems with the classic 'problem space’ approach! • The problem with the ‘generation’ of the problem space • The utility of the state space representation for tasks with inner dynamics is reduced because in most CPS environments it is not possible to undo the actions, and prepare a different strategy:

9. Problems with the classic 'problem space’ approach! • The classic problem solving theory used mainly verbal protocols as data. However, TALK ALOUD INTERFERES PERFORMANCE IN COMPLEX DYNAMIC TASKS (Dickson, McLennan & Omodei, 2000) • Independence (or very short-term dependences) of actions/states is assumed in some of the methods for representing performance. That is, the features that represent performance are local

10. What is LPSA and how it relates to these problems and other theories

11. Latent Problem solving Analysis(LPSA) • m(trial) = f{m(sa1), m(sa2),….. m(san), context} • Simplifying assumptions:m(trial1) = m(sa11) + m(sa21) +….. + m(san1) m(trial2) = m(sa12) + m(sa22) +….. + m(san2)…. m(trialk) = m(sa1k) + m(sa2k) +….. + m(sank) • Where sa is a ‘state or action’

12. Latent Problem solving Analysis(LPSA) • Complexity reduction: Reducing the number of dimensions in the space reduces the noise

13. Cow Cheetah calf LSA LPSA The problem space is a metric space, wherestates andtrials are represented as vectors

14. LPSA as a theory of representation in CPS tasks • Applications: Automatic landing technique assessment • Expertise effects of amount of practice • Expertise effects of amount of environmental structure • human similarity judgments • ‘Strategy’ changes

15. Approaches to complexity: The ant and the beach parable (Simon, 1967,1981)

16. Approaches to complexity: The ant and the beach parable (Simon, 1967,1981)

17. Approaches to complexity: The ant and the beach parable (Simon, 1967,1981)

18. Approaches to complexity: The ant and the beach parable (Simon, 1967,1981) ?

19. Unsupervised learning • Empirical adjustment of a problem space • Definition of a productivity mechanism and a similarity measure. • LPSA: addition and cosine.

20. LPSA solutions for the problems with the classic 'problem space’ approach • The problem with the ‘generation’ of the problem space • The utility of the state space representation for tasks with inner dynamics is reduced because in most CPS environments it is not possible to undo the actions, and prepare a different strategy: LPSA proposes a mechanism to generate automatically the problem space

21. LPSA solutions for the problems with the classic 'problem space’ approach • The classic problem solving theory used mainly verbal protocols as data. However, TALK ALOUD INTERFERES PERFORMANCE IN COMPLEX DYNAMIC TASKS (Dickson, McLennan & Omodei, 2000) • Independence (or very short-term dependences) of actions/states is assumed in some of the methods for representing performance. That is, the features that represent performance are local LPSA uses log files and human judgments as data, but not concurrent verbal protocols LPSA does not assume independence or short dependences between states/actions. Indeed, it uses the dependences of all of them simultaneously to derive the problem space. The features that represent performance are global

22. Theoretical surroundings of Latent Problem Solving Analysis

23. Mental representations Perceptual symbols Propositions Rules and theories Similarity based (varying in the amount of structure represented) Continuous features Set theoretic models Structural alignment models Transformational distance

24. Encoding processes Processes of internal transformation Decoding processes Anderson (1978)

25. LPSA applied to model human judgments

26. Main equivalence: Actions : Move_4_Copter_11_4_11_9_Forest_ Words 1 Move_4_Copter_11_4_11_9_Forest_ 2 Move_2_Truck_4_11_17_7_Clearing_ 3 Drop_Water_4_Copter_11_9_Forest___ 4 Move_3_Copter_8_6_10_11_Forest_ 5 Move_1_Truck_4_14_18_10_Forest_ 6 Drop_Water_3_Copter_10_11_Forest___ 7 Move_4_Copter_11_9_21_8_Dam_ 8 Move_3_Copter_10_11_12_14_Dam_ 9 Control_Fire_2_Truck_17_7_Clearing___ 10 Control_Fire_1_Truck_18_10_Forest___ 11 Move_4_Copter_21_8_12_10_Clearing_ Participants’ trials Docs

27. Firechief corpus • Data from the experiments described in experiments 1 and 2 in Quesada et al. (2000), and Canas et al. (2003). • Total: 3441 trials, 75.575 different actions • The first 300 dimensions where used

28. Trial 1 Trial 2 Trial 3 log files containing series of actions Action 1 Action 2 57000 actions 3400 log files actions

29. Three examples of performance • 8 first actions in a trial 2 1 RELATED NON RELATED 3

30. 1 0 CONTROL FIRE 1 2 3 4 5 6 7 8 9 10 11 CONTROL FIRE 12 13 14 15 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24

31. 2 0 1 2 3 4 5 6 7 8 DROP WATER 9 10 11 CONTROL FIRE 12 CONTROL FIRE 13 14 15 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24

32. 3 CONTROL FIRE 0 1 2 CONTROL FIRE 3 4 5 6 7 8 9 10 11 12 13 14 15 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24

34. Possible way of comparison: Exact matching of actions • Exact matching: count the number of common actions in two files. The higher this number, the more similar they are

37. Possible way of comparison: Transitions between actions • count the number transitions between actions in two files. Create matrices, and correlate them

42. Possible way of comparison: Transitions between actions

44. Exact matching is not sensitive to similarity differences • (exigent criterion). Since Transitions between actions is blind to most of the information in the logs, it fails because declares as similar performances that are not • LSA has correctly inferred that the remaining actions, • although different, are functionally related

45. LPSA - Human Judgments correlation

46. Human Judgment correlation • if LSA captures similarity between complex problem solving performances in a meaningful way, any person with experience on the task could be used as a validation • To test our assertions about LSA, we recruited 15 persons and exposed them to the same amount of practice as our experimental participants, so they could learn the constraints of the task.

47. 0.06 0.12 0.53 0.60 0.75 0.90 G F D E A B Human Judgment correlation • Replay trials, with different similarities • People watched a randomly ordered series of trials, in a different order for each participant, which were selected as a function of the LSA cosines (pairs A, B, C, D, E, F, G with cosines 0.75, 0.90, 0.53, 0.60, 0.12 and 0.06 respectively)

48. Human Judgment correlation One of the pairs was presented twice to measure test-retest reliability. That is, for example, pair G was exactly the same as pair A for one participant, the same as pair F for another participant, etc. Filling out a form that presented all the possible pairings of ‘stimuli pairs’ were presented

49. Human Judgment correlation FULL-SCREEN REPLAY OF THE TRIAL SELECTED, 8 TIMES FASTER THAN NORMAL SPEED

50. Human Judgment correlation: Results