Learning and Reasoning with Qualitative Models of Physical Behavior

1. Learning and Reasoning with Qualitative Models of Physical Behavior Scott Friedman, Ken Forbus, Jason Taylor Qualitative Reasoning Group Northwestern University

2. Novices & Qualitative Physics • Students have misconceptions about force and motion (McCloskey, 1983; diSessa, 1993) • Often similar to: impetus theory, Aristotelian models… • Systematicity and cohesiveness of intuitive models are heavily debated in the literature • Intuitive physics models (including misconceptions) are qualitative & tenacious • Good performance in mathematics is not indicative of good performance in qualitative physics (Halloun & Hestenes, 1985) • Qualitative physics misconceptions exist after physics instruction (Clement, 1982)

3. Novices & Qualitative Physics • Intuitive models are learned from experience • Misconceptions may occur from improperly generalizing or contextualizing (Smith, diSessa, & Roschelle, 1994) • How are naïve qualitative models learned from experience? • Our idea: temporal encoding of exemplars, analogical generalization, and parameterization

4. Our System • Given: • Multimodal input stimuli • Set of target concepts • Learn intuitive models of the target concepts • Perform similar to naïve students on two reasoning tasks: • Brown (1994) • Hestenes et al (1992) Comic-strip stimuli New Scenarios Reasoning NL NL List of target concepts {pushing, moving, blocking}

5. Multimodal Stimuli Encapsulated Histories Comic-strip stimuli SEQL Generalizations Temporally encoded CycL exemplars Reasoning List of target concepts {pushing, moving, blocking} New Scenarios def-EH: participants: … conditions: … consequences: … def-EH: participants: … conditions: … consequences: … NL NL

6. Multimodal Stimuli • Stimuli are multi-state comic strips • States divided by qualitative behaviors • Temporal relations between states • Stimuli contain 1+ example of target concepts • English blurbs • CogSketch(Forbus et al, 2008) processes sketched input • EA NLU (Tomai & Forbus, 2009) processes simplified English • Both use extended OpenCyc KB • 17 stimuli used for experiments “The child child-13 is playing with the truck truck-13.”

7. Encoding Exemplars Encapsulated Histories Comic-strip stimuli SEQL Generalizations Temporally encoded CycL exemplars Reasoning List of target concepts {pushing, moving, blocking} New Scenarios def-EH: participants: … conditions: … consequences: … def-EH: participants: … conditions: … consequences: … NL NL

8. Temporally Encoded Exemplars “The child child-13 is playing with the truck truck-13.” Target Concepts {pushing, moving, blocking}

9. Temporally Encoded Exemplars temporallyCoterminal temporallySubsumes Temporal relations add structure holdsIn startsAfterEndingOf

10. SEQL Generalization Encapsulated Histories Comic-strip stimuli SEQL Generalizations Temporally encoded CycL exemplars Reasoning List of target concepts {pushing, moving, blocking} New Scenarios def-EH: participants: … conditions: … consequences: … def-EH: participants: … conditions: … consequences: … NL NL

11. Contextualized Generalizations Pushing All exemplars describing the associated concept are automatically generalized within the context via SEQL • Target Concepts • Pushing • Moving • Blocking Moving Blocking

12. Generalizing with SEQL Pushing Exemplar enew • Identify destination context(s) Moving Blocking

13. Generalizing with SEQL Pushing • Structure Mapping Engine (SME) • Input • Base and Target representations • Computes mappings • Entity & relation correspondences • Structural evaluation (similarity) score • Candidate inferences • Used for merging representations Exemplar enew 2. Compare enew to each generalization g via SME (Falkenhainer et al, 1989) …merge enew with the first generalization where: SME-Similarity-Score(enew, g) > assimilation-threshold Base Target Mapping

14. Generalizing with SEQL Pushing Exemplar enew 3. If unmerged, compare enew to each unassimilated exemplar e via SME …create a new generalization from enew and the first exemplar e where: SME-Similarity-Score(enew, e) > assimilation-threshold

15. Generalizing with SEQL Pushing Exemplar enew 4. If structurally dissimilar to all exemplars and generalizations: Add enew to the list of unassimilated exemplars.

16. Result of Generalizing with SEQL Pushing 17 stimuli (50 concept exemplars) 10 generalizations + 12 unassimilated exemplars Moving Blocking

17. Generalization Anatomy • Generalizations are probabilistic abstractions of exemplars • Entities that correspond analogically become generalized entities (GenEntFn) in the generalization • Facts have probabilities reflecting their frequency • We want to capture the central behavior in a formal model

18. Building Models Encapsulated Histories Comic-strip stimuli SEQL Generalizations Temporally encoded CycL exemplars Reasoning List of target concepts {pushing, moving, blocking} New Scenarios def-EH: participants: … conditions: … consequences: … def-EH: participants: … conditions: … consequences: … NL NL

19. Encapsulated Histories • Represent categories of behavior over a span of time • Refer to multiple qualitative states or events • Can describe causal or temporal relationships between concepts • Used here as descriptive models of typical behavior • Agent is agnostic to the underlying mechanisms of change • If the participants (entities) and conditions of an EH hold, it is activated by the agent, and its consequences are assumed to hold • Automatically generated by parameterizing SEQL generalizations

20. Filtering Generalizations • Some generalizations are poor for causal reasoning • They don’t capture the relationship between concepts • We can filter these out • Find highly-correlated concepts within a context: • For probability threshold t, find concepts C where P(cC) > t in at least one generalization in the context • Filter generalizations that are indecisive w/r/t cC • Using binary entropy function H, exclude generalizations where: H(P(cC)) > H(t)

21. Models from Generalizations • Filter facts below a probability threshold t • Use temporal relations to hypothesize causal role of each fact f to the concept c • If f starts with or before c, f might cause c • If f starts with or after c, c might cause f • If f temporally subsumes c, f might be a condition for c

22. Generating Encapsulated Histories define-encapsulated-history Push05 Participants: Entity(?P1) Entity(?P2) PushingAnObject(?P3) Direction(?dir1) Direction(?dir2) Conditions: providerOfMotiveForce(?P3, ?P1) objectActedOn(?P3, ?P2) dir-Pointing(?P3, ?dir1) touches(?P1, ?P2) dirBetween(?P1, ?P2, ?dir1) dirBetween(?P2, ?P1, ?dir2) Consequences: Normal-Usual(and(PushingAnObject(?P3) providerOfMotiveForce(?P3, ?P1) objectActedOn(?P3, ?P2))) causes-SitProp(Push05, exists(?M1, and(MovementEvent(?M1) objectMoving(?M1, ?P1) motionPathway(?M1, ?dir1))) • Resulting EHs are more general than the initial stimuli • Low probability attributes and relations have been generalized and filtered • Only high probability facts remain • Used for reasoning about scenarios with same ontology as learning stimuli • EH activation identifies modeled behaviors as “Normal” or typical • PushingAnObject, at left • Failure to activate any EH on a concept instance means the instance is anomalous • Effective for simple counterfactual reasoning & indirect proofs

23. Resulting Encapsulated Histories • 10 generalizations total • 4/10 automatically filtered out for being causally irrelevant • 6/10 automatically parameterized into encapsulated histories • Blocking: • E1 touches & pushes E2 in dir d, which is blocked by E3 to dir d • Pushing: • E1 touches & pushes E2 in direction d, causing E2 to move in dir d • E1 touches & pushes E2 in dir d, caused by E1 moving in dir d, which results in E2 moving in d • Movement • E1 moves in dir d, caused by E2 moving toward E1 and pushing it in dir d • …same as above, but on a surface above the ground • E1 moves in dir d along a surface, caused by E2 pushing it in dir d

24. Reasoning with Learned Models Encapsulated Histories Comic-strip stimuli SEQL Generalizations Temporally encoded CycL exemplars Reasoning List of target concepts {pushing, moving, blocking} New Scenarios def-EH: participants: … conditions: … consequences: … def-EH: participants: … conditions: … consequences: … NL NL

25. Problem Solving Experiments • Two problem solving tasks • Brown (1994): Does the table pushes up against the book? • Hestenes et al (1992): which direction will this moving puck will go when given an instantaneous “kick?” • Problem scenarios were provided in same modalities as learning stimuli • The simulation reasoned through both scenarios by activating EHs • Contradiction detection • Compare with human results

26. Results from Brown (1994) • 73 high school students asked whether a table exerts an upward force against a book resting on its surface

27. Results from the Force Concept Inventory • High school & college students given this scenario: a puck is moving with constant velocity along on a frictionless surface. If given an instantaneous “kick,” which choice best describes its path?

28. Summary of Learning Results • Given • 17 multistate multimodal stimuli • Target concepts: {pushing, movement, blocking} • SEQL assimilation threshold = 0.5 • Parameterization probability threshold = 0.9 • Learning & reasoning on Companions Cognitive Architecture • Output from learning • After encoding: 50 concept exemplars • After generalization: 10 generalizations • After parameterization: 6 encapsulated histories • System reasoned with these learned models…

29. Simulation Results: Brown • System found two active instances of an EH in the scenario: Using EH: E1 pushes E2 in dir d, E2 pushes E3 in dir d and E2 is blocked by E3 • E1: gravity; E2: book; E3: table; d: down • E1: gravity; E2: table; E3: ground; d: down • System infers that the book pushes the table and the table pushes the ground, and that both are blocked. • The system could not conclude that the table pushed the book: • Assumed that the table did push against the book, and arrived at an indirect proof: Using EH: E1 touches & pushes E2 in direction d, causing E2 to move in dir d • The consequence of the table (E1) pushing the book (E2) upwards (d) is the book moving upwards, which is not visible in the scenario.

30. Simulation Results: FCI • System observed a branch after the second state and instantiated EHs to determine which following state(s) are feasible • Activated an EH instance on choice (a): Using EH: E1 touches & pushes E2 in direction d, causing E2 to move in dir d • E1: foot; E2: puck; d: up • Could not activate EHs on other choices, due to direction mismatches

31. Conclusion • Simulation that learns naïve qualitative models • Temporal encoding of concept exemplars • Analogical generalization • Qualitative model generation from probabilistic abstractions • Resulting explanations are compatible with those of physics-naïve students • Evidence that naïve physics models can be learned via analogical generalization

32. Future Work • Incorporate other physical concepts to model human reasoning over all of the Force Concept Inventory (Hesteneset al, 1992) • Automatically induce physical process models to develop deeper domain theories • No more batch mode: shift to an anomaly response model of concept learning • Incorporate this into the Companions Cognitive Architecture • Incorporate into a larger model of conceptual change

33. Related Work • Using sketched & NL input for learning • Lockwood, K. and Forbus, K. (2009). Multimodal Knowledge Capture from Text and Diagrams.  To appear in Proceedings of the Fifth International Conference on Knowledge Capture. • Computational models of conceptual change • Ram, A. (1993). Creative conceptual change. Proceedings of CogSci 1993. • Esposito, F., Semeraro, G., Fanizzi, N., & Ferilli., S. (2000). Conceptual Change in Learning Naive Physics: The Computational Model as a Theory Revision Process. In E. Lamma and P. Mello (Eds.), AI*IA99: Advances in Artificial Intelligence, Lecture Notes in Artificial Intelligence 1792, 214-225, Springer: Berlin. • Concept learning & causal learning • Fisher, D. H. (1987). Knowledge acquisition via incremental concept clustering. Machine Learning, 2,139-172. • Gopnik, A., Glymour, C., Sobel, D., Schulz, L., Kushnir, T., Danks, D. (2004). A Theory of Causal Learning in Children: Causal Maps and Bayes Nets. Psychological Review, 111(1), 3-32.

34. Acknowledgments This work was funded by the Office of Naval Research under grant N00014-08-1-0040