Innovation, Environment and Expertise

1. Innovation, Environment and Expertise Kenneth Kotovsky Department of Psychology Carnegie Mellon University

3. Introduction • Goal: to link three somewhat disparate ideas to make an argument about creative invention or innovation, its contexts and development: • Some research done in collaboration with Jonathan Cagan on the engineering design process and its environmental context, on the assumption that a deeper process understanding enhances our chances of creating both process and educational interventions to stimulate creative innovation. • Some classic findings with regard to the acquisition of expertise. • A movement in higher education that can be put in service of the above.

4. 1. Understanding the process: problem solving search within multiple representations(with Jonathan Cagan)

5. Basic model applied to individual, group and computational design agent process

6. RAT Problems (20) Lexical Decision task (50) RAT Repeated + New (20) Stimulating the Design Loop(with Jarrod Moss & Jonathan Cagan ) • What allows environmental input to be assimilated? • Typical Experiment • Remote Associate Test (RAT) problems (fox peep man)  hole • Answers for both new and previously unsolved problems presented in intervening task Moss, J., K. Kotovsky, and J. Cagan, “The Influence of Open Goals in the Acquisition of Problem Relevant Information”, Journal of Experimental Psychology: Learning, Memory, and Cognition, Vol. 33, No. 5, pp. 876-891, 2007.

7. Effect of hint on solution time (solved problems)

8. Experiment 2: Hint Timing fixation No fixation

9. Conclusions • Hints more effective in presence of open goals. • People carry unsolved problems with them and are implicitly sensitized to relevant environmental cues. • Timing matters-hint effective after exploration & dev. of open goal but before fixation (protocol studies). • Method for impacting the design process by delivering useful hints, perhaps even automatically.

10. Transfer to Engineering Design(with Ian Tseng Jarrod Moss & Jonathan Cagan) How do open goals and hint assimilation affect engineering design? Generate as many time-keeping devices as possible using a list of 14 household objects. Explored early/late and close/distant hints Tseng, I., Moss, R. J., Cagan, J. and Kotovsky, K. Overcoming Blocks in Conceptual Design: the Effects of Open Goals and Analogical Similarity on Idea Generation. 2008 ASME International Design Engineering Technical Conferences and Computers and Information in Engineering Conference, IDETC/CIE August 3-6 2008, New York.

11. Presented Hints Similar Information - Three Clocks • Distantly Related Information - Three Devices

12. Effects of Open Goals • Examples are influential in extending both the range and novelty of solutions, with devices (distant/varied examples) most effective • Open goals facilitated recognizing analogies in distantly related information

13. Transfer to Group Design & Tracking Representations (with Katherine Fu & Jonathan Cagan) • This study examined: • How engineering design teams converge to a common understanding of a design problem and its solution using LSA to track representation development. • Effect of good and bad examples on convergence and quality • of produced solutions. • The problem was to design an automatic peanut shelling device for a poor isolated village. Fu, K., Cagan, J. & Kotovsky, K. Design Team Convergence: The Influence of Example Solution Quality. Proceedings of the ASME 2009 International Engineering Design Technical Conferences & Computers and Information in Engineering Conference IDETC/CIE 2009 August 30 - September 2, 2009, San Diego, California, USA

14. AVERAGE LEVEL OF SEMANTIC CONVERGENCE WITHIN GROUPS; ERROR BARS SHOW ± ONE STANDARD ERROR AVERAGE LEVEL OF SEMANTIC CONVERGENCE TO FINAL DESIGN SOLUTION DESCRIPTION TEXT; ERROR BARS SHOW ± ONE STANDARD ERROR Results: Convergence within Groups and Convergence to Final Design Solution • Poor example produced decreased convergence • Good example condition and control not significantly different • All increase over sessions: example effect is immediate and lasting

15. AVERAGE QUALITY OF FINAL DESIGN SOLUTIONS; ERROR BARS SHOW ± ONE STANDARD ERROR Results: Quality of Final Designs • Good example produced best quality solutions. • Poor example produced lower quality solutions. • Convergence and quality not totally equivalent.

16. Emerging Conclusions From Work on Design Process • The design environment or context matters: • Hints, both cs and ucs and other people have an impact • Representations can be tracked and do converge • Timing matters • Representation tracking potentially allows for a fine-grained analysis of multiple issues in creative invention: • -- Brainstorming, • -- Team composition effects, • -- Expertise effects, • -- Monitoring impact of environmental input. • Other work: • Automate the generation of fruitful examples • Develop a model of the control systems by which designers can optimally control their activity and resist fixation • Turn this developing process knowledge into teachable-learnable skills.

17. 2. Expertise: A long time coming • Hayes & others: ten year rule for world class expertise in many domains (music composition, music performance, chess, etc). • Simon & Gilmartin : On the order of 50,000-100,000 memory chunks (for chess expertise). • Ericsson: targeted deliberate practice (not just amount) leads to expertise. • Expertise development involves many cognitive skills and functions: memory & strategy (ex. chess, problem-solving), perception (chess) and representation and knowledge (physics, engineering). Simon H. A. & Gilmartin, K. (1973). A simulation of memory for chess positions. Cognitive Psychology, 5, 29 -46. Chase, W. G., & Ericsson, K. A. (1982) Sill and working memory. In G. H. Bower (Ed.), The psychology of learning and motivation, (Vol. 16). New York: Academic Press. Kotovsky, K., Hayes, J.R. & Simon, H. A. (1985). Why are some problems hard? Evidence from Tower of Hanoi, Cognitive Psychology, 9, 52-76. Chi, M. T. H., Feltovich, P. J., & Glaser, R. (1981) Categorization and representation of physics problems by experts and novices, Cognitive Science, 5, 121-152. Chase, W. G., & Simon, H. A. (1973). The mind’s eye in chess. In W. G. Chase (Ed.), Visual information processing. New York: Academic Press. Moss, J., K. Kotovsky, and J. Cagan, “Expertise Differences in the Mental Representation of Mechanical Devices in Engineering Design”, Cognitive Science, Vol. 30, No. 1, pp. 65-93, 2006.

18. Power Law of Practice • Skill learning often seems to asymptote relatively quickly over training trials. • In actuality, keeps increasing, with additional increments in performance requiring exponentially more practice. • Time = Practice –b • log (Time) = -b log (Practice)

19. Generating Geometry Proofs Neves, D. M., & Anderson, J. R. (1981) Knowledge Compilation: Mechanisms for the automatication of cognitive skills. In J. R. Anderson (Ed.) Cognitive Skills and Their Acquisition. Hillsdale, NJ: Erlbaum.

20. Time to Perform Mental Addition Crossman, E. R. F. W. (1959). A theory of the acquisition of speed-skill. Ergonomics, 2, 153-166.

21. 3. The Broader Context: Educational Implications • If expertise requires vast time and results in continuous cognitive improvement in many diverse areas, why not innovative engineering design as well—particularly as we come to better understand the specific processes and skills via empirical study? • This is somewhat in contrast to short-term focused creativity training courses. • Focus on in-situ, learning by doing & concomitant training of skill and knowledge often within the milieu of higher education.

22. The Carnegie Commission Boyer Report-Reinventing Undergraduate Education: a Blueprint for America’s Research Universities • Educate through research, making research-based education the standard. • Beginning in the freshman year. • Inquiry-based courses should allow for joint projects & collaborative efforts. • Internships can turn inquiry-based learning into practical experience. • Combining a group of students with a combination of faculty and graduate students for a semester or a year of study of a single complicated subject or problem. • Doesn’t focus on the issue of creativity. • Does have a tilt toward universality (whole school approach).

23. The Carnegie Plan • An older version of some of the main ideas of the Boyer Report • Generated by a forward thinking former president of my own university, Robert Doherty, who was president from 1936 to 1950. • The Carnegie Plan provided a well-rounded "liberal/professional" education in the context of an engineering school. • Emphasis on being able to do things in addition to knowing things • Students taught to apply fundamental knowledge to solve practical problems • Forerunner to today's focus on an interdisciplinary, problem-solving oriented university curriculum. • Enhanced by Herbert Simon with focus on science of design and incorporation of meta-cognitive skills.

24. Example: Engineering Education • Some Carnegie Plan-like engineering curricular examples that in combination can lead toward expertise. • First year engineering courses: design, not analysis. • Research internships over multiple years. • Senior design and Integrated product design courses • Focus on multiple interdisciplinary perspectives with faculty from three colleges and diverse students. • Company sponsorship and involvement directed toward real-world challenges and resultant patents which are often obtained. • Necessary focus on innovation and creative solutions. (Similar approach in psychology curriculum but issue of creativity)

25. Conclusions • The basic argument is that creative invention and innovation is at least in part a skill set that students can acquire expertise in if- • we understand it at a process level, • we are clear about the desired behavior and our goals of teaching it, • Students are trained and motivated to do it via deliberate training, coaching and sustained (motivated) effort over long periods of time • we all maintain a continued focus on creative solutions rather than simply skilled performance. • The policy recommendation is that part of what do in this area is to not only foster research on creative processes but also the translation of the results of that research to methods for teaching invention and innovation.

26. The Larger Context “Mediocrity, It takes a lot less time, and most people won't notice the difference until it's too late." “Underachievement, The tallest blade of grass is the first to be cut by the lawnmower.”

27. Acknowledgements Funding NSF under grants DMI-0627894 and BCS0717957 People • Lauren Burakowski • Steven Benders • Dhruv Datta • Pasha Gill