Personalized Interface Agents for Virtual Webpages

1. Personalized Interface Agentsfor Virtual Webpages Third International Conference on Autonomous Agents Agents 99 Seattle, May 3. 1999 Wolfgang Wahlster German Research Center for Artificial Intelligence, DFKI GmbH Stuhlsatzenhausweg 3 66123 Saarbruecken, Germany phone: (+49 681) 302-5252/4162 fax: (+49 681) 302-5341 e-mail: wahlster@dfki.de WWW:http://www.dfki.de/~wahlster

2. Outline l What is a Virtual Webpage? l What are Personalized Interface Agents? l Using Life-Like Characters for the Personalization of User Interfaces l The Combination of Retrieved and Generated Media Objects for the Generation of Virtual Webpages l PET: A Persona-Enabling Toolkit l Commercial Applications in Advanced WebCommerce l Psychological Effects of Personas l Our Research Agenda for Agent-based User Interfaces l Multiple Interface Agents l Conclusion

3. Three Generations of Web Sites First Generation Second Generation Third Generation Virtual Web Sites Interactive Web Sites Static Web Sites Netbots, Information Extraction, Presentation Planners JavaScripts and Applets User Modeling, Machine Learning, Online Layout Database Access and Template-based Generation Fossils cast in HTML Dynamic Web Sites Adaptive Web Sites

4. What is a Virtual Web Page? Virtual Memory, Virtual Relation, Virtual Reality... A Virtual Web Page l is generated on the fly as a combination of various media objects from multiple web sites or as a transformation of a real web page. l looks like a real web page, but is not persistently stored. l integrates generated and retrieved material in a coordinated way. l can be tailored to a particular user profile and adapted to a particular interaction context. l has an underlying representation of the presentation context so that an Interface Agent can comment, point to and explain its components.

5. AiA: Information Integration for Virtual Webpages PAN Travel Agent Andi Car Route Planner Yahoo News Server Yahoo Weather Server Gault Millau Restaurant Guide Hotel Guide

6. Virtual Webpage Retrieved from 5 Different Servers

7. Virtual Webpage Augmented by Persona

8. What are Agent-based User Interfaces? l appear as life-like characters l plan interactive behavior autonomously l can initiate interaction ACTIVE l understand the user‘s requests l answer clarification questions l allow mixed initiative dialogs l respond immediately to interruptions l criticism and clarification questions l direct manipulation INTERFACE AGENTS RE- ACTIVE INTER- ACTIVE PROACTIVE l anticipate the user's needs l adopt the user's goals l provide unsolicited comments

9. Intelligent Web Services Provider Netbot Consumer sells  Information  Goods  Services buys  Information  Goods  Services  Intelligent Parallel Retrieval  Information Extraction and Summarization  Personalized Presentation  Matchmaking  Teleshopping Assistance  Telemarketing Assistance  Translation Services  Data Mining Services Knowledge about:  Usage Patterns  User Models  Consumer Profiles Web Sites

10. Enhancement of User Interfaces through Personalization System is able to flexibly tailor presentations to the individual user and the current situation. An animated character serves as “Alter Ego” of the presentation system. Personalized Presenters at DFKI

11. Persona as a Personal Travel Consultant

12. The Personal Picture Finder

13. A Netbot for Portrait Photos: The Personal Picture Finder http://finder.dfki.de:7000 Portrait Photo <Name> Netbot: Personal Picture Finder Parallel Meta- Search of Webpages for <Name> Parallel Search in Picture Archives Home Pages Filtering of Logos, Graphics, ... Extraction of Images WWW Knowledge Sources Applications: Journalism, Contact Preparation, Tracing Criminals

14. Personalized Package Presentation by an Animated Agent  Personalization = adaption of system behavior according to a user model  Personalization = the “agent” appears as animated character which presents, explains and comments an offer, and to which the user can talk to.

15. Video Character Presents the Interior of a Boeing 757

16. Reactive Behavior of the Persona Agent

17. PPP’s Persona Server implements a generic Presentation Agent that can be easily adapted to various applications Visual Appearances Behaviors Hand-drawn Cartoon l Presentation Gestures Bitmaps l Reactive Behaviors l Idle-time actions l Navigation actions Persona Server Auditory Characteristics Video Bitmaps l Sound effects, auditory icons l Voice: male, female Generated Bitmaps from 3D-Models

18. The frames of the visual appearance of persona can be cartoon-like images or video-frames which show real persons More than 200 cartoon frames were drawn by a professional artist. A real persona was filmed with a video-camera and the pictures were digitized with a frame grabber.

19. The Persona Editor

20. Context-Sensitive Decomposition of Persona Actions High-Level take-position (t t ) point-to (t t ) 1 2 3 4 Persona Actions Context-Sensitive Expansion move-to (t t ) r-stick-pointing (t t ) 1 2 3 4 (including Navigation Actions) Decomposition r-turn (t t ) 1 21 r-hand-lift (t t ) into 3 31 r-step (t t ) Uninterruptable 21 22 Basic Postures f-turn (t t ) r-stick-expose (t t ) 22 2 31 4 Bitmaps ... ... ... ...

21. PPP System Architecture Multimedia Multimedia Generation Server Text Presentation Production Generator Layout Acts Acts Presentation Manager Graphics Planner Generator (PREPLAN) Gesture Generated Signals + Generator KR & Material Events Reasoning Animation (RAT) Generator Persona Music Server Generator

22. Extensions of the Representation Formalism Distinction between production and presentation acts (i.e. Persona- or display acts) Explicit representation of qualitative and quantitative constraints Production Act Presentation Act Introduce S-Show S-Position Elaborate-Parts Create- Graphics S-Wait Label Label S-Create- S-Depict Window S-Speak S-Point S-Speak S-Point Qualitative constraints: Create-Graphics meets S-Show, ... Metric constraints: 1 <= Duration S-Wait <= 1, ...

23. Presentation Strategies in PPP contain qualitative and metric constraints (cf. [André/Rist 97]) (define-plan-operator :HEADER ( A0 (INTRODUCE P A ?object ?window)) :INFERIORS (( A1 (CREATE-GRAPHICS P A ?window ?object)) ( A2 (S-SHOW P A ?window ?object)) ( A3 (S-POSITION P A ?window)) ( A4 (S-WAIT P A)) ( A5 (ELABORATE-PARTS P A ?object ?window))) :QUALITATIVE ((A1 (m) A2) (A3 (s) A2) (A3 (m) A5) (A5 (m) A4) (A4 (f) A2)) :METRIC ((10 <= DUR A2) (2 <= DUR A4 <= 2)) :START A1 :FINISH A2 )

24. The Combination of Retrieved and Generated Media Objects for Virtual Webpages Multi-Domain Problem Specs NETBOT Multiple Data Sources Distributed Information Information Structures l Relations, Lists l KR Terms Media Objects l Texts, Sounds, Videos l Pictures, Maps, Animations Retrieved Results

25. The Combination of Retrieved and Generated Media Objects for Virtual Webpages Information Structures l Relations, Lists l KR Terms Media Objects l Texts, Sounds, Videos l Pictures, Maps, Animations Retrieved Results Select Canned Media Objects Design New Media Objects Coordinate Media Objects Transform Media Objects l Icons, Clip Art l Frames, Sounds l Graphics, Animation l Text, Speech, Mimic l Temporal Synchroni- zation l Spatial Layout l Clip, Convert, Abstract l Zoom, Pan, Transition Effects Select & Design Reuse & Transform

26. Persona Presents an Automatically DesignedBusiness Chart

27. Virtual Webpage with Animation Effects Based on a Single GIF Image

28. Transition Effects in a Series of Retrieved Pictures

29. The Generation of Virtual Webpages with PAN and AiA Address Hotel Agent Map Agent AiA Presentation Planner Pictures and Graphics Netbot PAN Pieces of Text Components of virtual Webpages Virtual Web Presentation Coordinates for Pointing Gestures Trip Data Input for Speech Synthesis Persona Server Icons for Hyperlinks Constraint- based Online Layout Weather Agent Train & Flight Scheduling Agent Major Event Agent

31. PET: Persona-Enabling Toolkit Objective: l Enable non-professional computer users to populate their web pages with lifelike characters PET comes with: l a set of characters and basic gestures l an easy-to-learn Persona markup language Developer’s PET will include: la character design tool which enables users to build their own characters Technical Realization: l Based on standard XML, JavaScript, Java and DHTML

32. The Persona Markup Language Specification of the character to be used Specification of Persona actions <html> <head> <title> Persona Test </title> </head> <body> <persona bitmap=“cartoon” ...> <uselib url= .../> <do name=“greet”/> <do name =“speak” args=“hello”/> </persona> </body> </html> Features: • XML-based • easy to learn

33. Functional View of PET PET Parser PET Generator Behavior Monitor Event Handler Character Composer Bitmaps Webpage with Reference to Java Applet URL of Webpage with Persona Tag <html> ... <APPLET archive=“personaplayer.jar”...</APPLET> ...</html> PET Application Server <html> <head> <title> Persona Test </title> </head> <body> <persona bitmap=“cartoon” ...> <uselib url= .../> <do name=“greet”/> <do name=“standard”/> <do name =“speak” args=“hello”/> </persona> </body> </html> Persona Scripts waitscreen 4 gesture greet 0 0 null gesture laugh 0 0 null ... Persona Engine Audio Data

34. Tasks of PET PET-Parser l Preprocessing (required for non-standard HTML pages) l Identification of Persona Tags • PET-Generator • l Replacement of Persona Tag with Java Applet • l Determination of parameters for Java Applet • l Realization of Persona active elements as JavaScript • l Generation of DHTML layers (transparency of Applet box) • l Generation of data, e.g. audio files for speech output • l Generation of scripts

35. The Bidirectional Control Flow onPersona-Enabled Webpages l Mouse Clicks l Mouse Movements Triggers actions of the Persona l Text Input l Speech Input l Menu Input l Direct Manipulation Input Triggers operations on elements of the webpage Web Persona

37. Persona Active Elements (PAE) <ACTIVEIMAGE SRC=“image” HREF=“url” NAME=“image name” STATUS=“status message” ALT=“tooltip” CACTION=“persona action onClick” OACTION=“persona action on MouseOver” ...> <PDIV DIVNAME=“name of the element” DVFRAME=“frame name” TOP=“anchor-y” LEFT=“anchor-x”>some HTML elements</PDIV> • l Active Images • An active image starts a persona action when clicked. • l Addressable Objects • An addressable object is an object which can be addressed and • manipulated by Persona via its name and its position.

38. Use of a Life-like Character for Electronic Commerce

39. Use of a Life-like Character for Electronic Commerce

40. Personified Agents Increase the User's Trust in the System's Presentation Experimental evidence for effects of modality on the user's trust (van Mulken, 1999) The system gives recommendations, which turn out to be wrong in some cases. How much does a user trust the system's advice depending on the modality of a presentation? 1.0 0.8 0.7 0.6 0.5 Self-animated Persona, Speech, Gesture, Facial Expression, Pointing Speech, Graphical Highlighting Text, Graphical Highlighting

41. Impact of the modality of a Presentation on the User's Trustfulness Result:Persona > Speech > Text Conclusion: If the presentation is more human-like, recommendations are more readily followed For l decision support systems l tutoring systems l recommendation systems l virtual sales agents personified interface agents have a clear advantage: They increase the user's trust in the information presented by the system

42. Sending Interface Agents to Clients: Plug-Ins or Applets? Plug-Ins Applets l Add features (character players) to browser l Download triggered by user l Requires disk space on client l Unrestricted access to client l Less appropriate for WebCommerce, Guides l Agents integrated in 3D environments l Appropriate for Entertainment Examples: lExtempo's Jennifer James (Hayes-Roth et al. 98) l PFMagic's virtual petz l Java animation code sent over the net l Automatic loading l Requires no disk space on client l Restricted access to client l Appropriate for WebCommerce, Guides l Agents integrated in 2D environments l Less appropriate for Entertainment Examples: l DFKI's Web Persona (Müller et al. 98) l ISI's Adele (Johnson et al 98) New in AiA/PAN: Balanced combination of Applets and Servelets Efficient distribution of client-side Java and server-side Java for driving the Interface Agent

43. SmartKom: Intuitive Multimodal Interaction DAIMLERCHRYSLER Uinv. Of Munich European Media Lab Project Budget: $ 34 M Project Duration: 4 years The SmartKom Consortium: Main Contractor Project Management Testbed Software Integration DFKI Saarbrücken MediaInterface Saarbrücken Berkeley Dresden Univ. of Stuttgart Heidelberg Univ. of Erlangen Munich Stuttgart Ulm Aachen

44. SmartKom-Public:A Multimodal Communication Booth Loudspeaker Smartcard/ Credit Card for authentication and billing Room microphone Face-tracking camera Docking station for PDA/Notebook/ Camcorder high speed and broad bandwidth Internet connectivity Virtual touchscreen protected against vandalism High-resolution scanner Multipoint video conferencing

45. SmartKom-Mobile: A Handheld Communication Assistant GSM for Telephone, Fax, Internet Connectivity GPS Camera Wearable Compute Server Stylus-Activated Sketch Pad Microphone MOBILE Biosensor for Authentication & Emotional Feedback Loudspeaker Docking Station for Car PC

46. SmartKom-Home/Office: A Versatile Agent-based Interface SpeechMike Natural Gesture Recognition Virtual Touchscreen

47. Our Research Agenda for Agent-based Interfaces(Wahlster, André, Rist, Müller, Graf etc; www.dfki.de/imedia) Personalized Presentation Agents (limited user interaction) 1 WIP: 1989-1993 Personalized Interface Agents (full user interaction) 2 PPP:1994-1996 Multiple Interface Agents (agent-agent and user interaction) 3 l Multiple Presentation Agents in one scene (eg. pros and cons) l Multiple Role-Taking (eg. Travel Assistance vs. Comparison Shopper) l Multiple Interface Agents (eg. human-computer, human-human interaction) AiA: 1997-2000

48. Use of a Life-like Character for Electronic Commerce Digital Assistant Selector

49. Current and Future Work: Multiple Interface Agents for User-adaptive Decision Support User-Adaptive Search Planning weighted propositions ... ... Multiple Decision Support Agents Spare parts for this car are rather expensive! But, it’s fast!

50. Multiple Interface Agents l Multiple Presentation Agents l Convey different attitudes, beliefs and biases (e.g. proponent vs. opponent) l Generate argumentatives dialogs among a group of agents l Multiple Interface Agents l human and virtual agents in collaborative browsing environments l different dialog roles (e.g. moderator, advisor, sales agent) l Multiple Role-Taking of One Interface Agent l e.g. travel assistant vs. comparison shopper (use of a different outfit of the character to signal a role change)