Programming by Example • Henry Lieberman • Media Laboratory • Massachusetts Institute of Technology • Cambridge, MA, USA • email@example.com • http://www.media.mit.edu/~lieber • PBE Web site: http://www.media.mit.edu/~lieber /PBE/
Your Wish is My Command:Programming by Example • Morgan Kaufmann, 2001
Programming should be like teaching • User plays the role of a teacher • Computer plays the role of a student
Programming by Example • (Also called "Programming by Demonstration") • Agent "watches what you do" in an interactive interface • Records sequence of operations, data • Generalizes program so that you can use an analogous procedure in new examples • "Macros on Steroids"
Programming in the User Interface • The same interface used by end-users can be used for programming • Empowers end-users to do programming • No switching between user and programmer interfaces • Empowers "visual thinkers"
PBE and the "Intelligent Agent" movement • There's been a recent movement toward "intelligent agents" -- proactive, assistive, (possibly anthropomorphic) interfaces • PBE can be used to make agents instructible • Agent personalities: • Butler/Servant/Secretary • Teacher/Student, Master/Apprentice
Issues in Programming by Example • The Generalization Problem (= Data Description) • How do I describe actions and data? • How similar do the actions/data have to be next time? • Feedback • How does the system show the user what it's learned? • Advice • How can the user influence or modify what the system learns?
Issues in Programming by Example • How should we record the user's actions? • At what granularity should things be recorded? • Can the user interact with the agent during recording? If so, how? • Do we represent the agent directly in the interface? • Do we anthropomorphize the agent? • Can users give advice during recording? • Turning recording on and off
Issues in Programming by Example • How should the recorded program be represented? • Should the user see the recorded program? • What should it look like? • Should the user interact directly with the recorded program? If so, how? • How to represent examples? • How to represent generalizations?
Issues in Programming by Example • How should the user edit an already-recorded program? • How do we keep the representations consistent during editing? • Can the user edit previous generalizations? • How can users debug programs? • Can users step/trace/break programs?
Machine learning in Programming by Example • Lots of work in machine learning • Careful: Watch for hidden assumptions • Few examples, learns fast, explainable • Explanation-based Generalization • Case Based reasoning • Lots of examples, slow but more robust • Neural networks, PAC learning, data-mining
Users of the world, unite! • You have nothing to lose but your error messages! • Almost "political" message of empowerment of users • Users should not be at the mercy of software providers • User procedures and data should not be held hostage by software companies
"Poor man's"Programming by Example • Macro recorders in applications, AppleScript • Emacs keyboard macros • Interface builders / Scripting languages • Quickeys / OneClick …
Hooking PBE systems to conventional applications • PBE systems would like to use conventional interfaces as tools rather than re-implement apps • Conventional interfaces aren't set up to be operated by an external program • Issues: Access to application data, sharing the interface, parallelism • Article: Lieberman, IUI 98 "Attaching Interface Agents to Conventional Applications"
Watch What I Do • Allen Cypher, ed. MIT Press 1993
A Small Matter of Programming • Bonnie Nardi, MIT Press, 1994. • End-User Programming • Scripting languages • Application-specific languages • "Gardeners"
Eager - Allen Cypher • Programming by example for Hypercard • Notices patterns in user actions • Anticipation highlighting of predicted actions • Anthropomorphization of learning agent • Example: Collect subject lines from messages
Mondrian: An Instructible Graphical Editor • Mondrian = • An object-oriented graphical editor + • An agent that records user interface actions + • Learning and generalization heuristics
Wizard of Oz Experiment • Human simulates agent • What kinds of communication between human and agent are plausible? • What kind of knowledge does the agent need to know to perform the task?
Wizard of Oz experiment task • Reformat a Bibliography • Lieberman, H., The One True Path to Intelligent Agents, International Conference on Buzzwords and Hype, Tahiti, 1996. • [Lieberman 96] The One True Path to Intelligent Agents, Int. Conf. on Buzzwords and Hype, Tahiti, (1996). • Oz, Wizard Of, Pay No Attention to the Man Behind the Curtain, Journal of Smoke and Mirrors, August 1942, Emerald City, Oz.
Wizard of Oz results • Users can make mistakes • Information can be incomplete • Agents can ask the user if they encounter an error situation • Agent could ask the user to supply missing information • Agent could ask the user to edit the example • Skip steps that don't work
Wizard of Oz experiment method • User performs task once • “Thinking aloud”: goals and methods • Agent (secretly) constructs description of actions • In terms of simple “implementable” operations • Agent offers help • Agent performs help according to action description • Agent gives user feedback • Agent accepts advice from user
Survey of "Your Wish" systems • Astonishingly wide variety of application domains: • Kids, Games, CAD, GIS, Text editing, Graphical editing, Web browsing, Animation, Databases, etc. • Illustrates variety of design decisions in PBE systems • Different reps for programs, agents, etc. • How much inference, generalization, initiative?
Stagecast Creator - Smith, Cypher • Programming for kids, grid game world • If you see this pattern of squares, replace it with that pattern of squares • Viz rep of rules, editor/stepper • Generalization on object types, introduce variables • Commercially available, success with thousands of users
Toontalk - Ken Kahn • Programming language is a video game • Most radical programming model • Concurrent logic programming • Abstraction by removing detail • Commercially available, success with kids in schools
How much intelligence?- Brad Myers, Rich McDaniel • CMU group one of most active and consistent in PBE • Multitude of systems, "gems" of PBE • Peridot, Lapidary, Jade, Gilt, Tourmaline, C32, Pursuit, Gold, Marquise, Katie, Turquoise, Topaz, Gamut • Experience with range of conservative to radical PBE systems • "Spectrum of Instructibility" • Tradeoff between conveience and control
Internet Scrapbook - Sugiura • Everybody's favorite application area - the Web! • Automates Browse & Cut & Paste applications • How do you describe a Web page selection? • Partial matching, Web design heuristics • Commercially available - preloaded on NEC PCs
TriAS - Bauer, Meyer, Dengler, Paul • Automating Browse & Copy & Paste, like Scrapbook • Collaborative, mixed-initiative dialog • At any time, agent or user can propose selections, generalizations • What Next? Can I Stop? • Integrates wrapper description language HyQL, InfoBeans dynamic Web sources
PBE for GIS: CSprl - Traynor, Williams • Study of how end users use Geographical Information Systems • "Gardeners", "Surrogate Users" • Comic-strip PBE language
PBE for Computer Aided Design -Girard, Pierra • True "Industrial Strength" PBE, probably the champ • CAD users are sophisticated in application domain, willing to specify a lot • Provides complete editing and debugging • Key issues: Naming, Ambiguity in spec/solutions, general control structures: conditionals, loops
Demonstrating "Hidden Features" - McDaniel, Myers • PBE for animated computer games • WYSIWYG - "What you get is all you have" • Demonstrate "hidden features" by reifying them • Guides, e.g. Arrows for directions • Sophisticated heuristics, lots of inference • Stop This! • Do Something!
Report Generation - Masuishi • Mundane, but hey, could save a lot of time • Format generalization • Application-specific iteration • Distributed to wide user community
Dynamic Macro - Masui • Notices loops in text editing commands • Simple, automatic: No start/stop recording • Wide community of users • Also: POBox predictive input for PDAs, phones, etc.