1 / 45

Reactive and Responsive Intelligent Environments

Reactive and Responsive Intelligent Environments. Kevin Quigley aire group MIT AI Lab. Reactive and Responsive Environments. We are trying to build pervasive, perceptually enabled human-centered environments

myrna
Télécharger la présentation

Reactive and Responsive Intelligent Environments

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Reactive and Responsive Intelligent Environments Kevin Quigley aire group MIT AI Lab

  2. Reactive and Responsive Environments • We are trying to build pervasive, perceptually enabled human-centered environments • Such an environment must respond in reasonable ways to high level requests from its users. • It should be up to the system to figure out a reasonable way to implement the request, translating goals to plans that meet the users needs and that utilize available resources. • E.g. I might ask to light the room up; the system responds by opening the drapes. • Such an environment should react to events in the environment even when there is no explicit user request. • E.g. when I walk into my room in the morning, the system should light up the room. • Reactions and responses should both be contextually sensitive. • Both must show human levels of adaptivity Kevin Quigley — MIT Artificial Intelligence Lab

  3. 5 Challenges • Providing a practical level of knowledge representation that enables groups interactions and grounding in the real world of space and time • Providing run-time composable services in a multi-user environment that make optimal use of the currently available resources • Recovering from equipment failures, information attacks, misestimates of sensors, etc. • Coordinating and fusing information from many sensors and modalities • Capitalizing on and recognizing context (task, location, personal style & state) 6. Maintaining security and privacy and trading these off against other goals Kevin Quigley — MIT Artificial Intelligence Lab

  4. Challenge 1: Grounding in Real-World Semantics • We want to build applications that service many individuals and groups of individuals • These people will move among many physical spaces • The devices and resources they use change as time progresses • The context shifts during interactions • The relevant information base evolves as well. • The system is required to respond dynamically Kevin Quigley — MIT Artificial Intelligence Lab

  5. Research Agenda: Knowledge Representations • Information nodes • Topic area, place in ontology, format • Services • Methods, parameter bindings, resource requirements • Agents • Capabilities, society, acting on behalf of whom • Events • E.g. Person identification, motion into a new region of space, gestures • Qualitative Changes in any of the properties in the KR • People • Interests, skills, responsibilities, organizational role • Organizations • Members, structure • Spaces • Location • Subspaces • Devices and resources • Resources Kevin Quigley — MIT Artificial Intelligence Lab

  6. Challenge 2: Adaptive Resource Management • In most systems, applications are written in terms of specific resources • (e.g. The left projector in Michael’s office, or worse yet, a physical address). • This is in conflict with • Portability across physical contexts • Changes in equipment availability across time • Multiple applications demanding similar resources • Need to take advantage of new resources • Need to integrate mobile devices as they migrate into a space • Need to link two or more spaces • What is required is a more abstract approach to resources in which no application needs to be tied to a specific device. Kevin Quigley — MIT Artificial Intelligence Lab

  7. An Example of Service Requests • When I come into my office in the morning it’s dark • The lights are out • The drapes are closed • I ask the office to light up the room • It’s a sunny day, it opens the drapes • If I had asked it to turn on the lights, it wouldn’t have opened the drapes • It’s a cloudy day, it turns on the lights Kevin Quigley — MIT Artificial Intelligence Lab

  8. Service Mapper Plans Resource Pool Resource Allocator Actions Responsive, Goal-Directed Processing Goals Service mapping is provided by the resource management component Kevin Quigley — MIT Artificial Intelligence Lab

  9. Each Plan Requires Different Resources Each Method Binds the Settings of The Control Parameters in a Different Way Resource1,1 Service Control Parameters Resource Cost Function Resource1,2 User’s Utility Function Abstract Service Plan1 Resource1,j Plan2 The Binding of Parameters Has a Value to the User User Requests A Service With Certain Parameters The Resources Used by the Method Have a Cost Plann Each Service Can Be Provided by Several Plans Net Benefit Services are Dynamically Mapped to Plans The System Selects the Plan Which Maximizes Net Benefit Kevin Quigley — MIT Artificial Intelligence Lab

  10. Challenge 3: Robustness and Recovery From Failures • Breakdowns are inevitable • Resources sometimes fail while being used • The system acts on sensor data which has uncertainty • The system renders services by translating them into plans • A plan-monitor watches over the execution of a plan. • Each plan step accomplishes sub-goals needed by succeeding steps • Each sub-goal has some way of monitoring whether it has been accomplished • These monitoring steps are also inserted into the plan • If a sub-goal fails to be accomplished, model-based diagnosis isolates and characterizes the failure • A recovery is chosen based on the diagnosis • It might be as simple as “try it again”, we had a network glitch • It might be “try it again, but with a different selection of resources” • It might be as complex as “clean up and try a different plan” Kevin Quigley — MIT Artificial Intelligence Lab

  11. I need to ask a question of a systems wizard The Plan Monitor Manages Recovery From Failures I don’t see light on the screen • Plan 1: • Locate a systems wizard in the E21 • Monitor: check that person is still there • Turn on the selected projector • Monitor: check that projector turned on • Project the message • Done • Monitor: check that the person noticed the message Plan Breakdown Projector-1 must be broken. We’ll try again, but using Projector-3. I see Sally by the screen Kevin Quigley — MIT Artificial Intelligence Lab

  12. Step-B Step-A requires achieves Condition-1 Condition-1 prerequisite Making the System Responsible for Achieving Its Goals Diagnostic Service Service Request Plan-for Localization & Characterization Repair Plan Selector alerts Scope of Recovery Selection of Alternative Rollback Designer Concrete Repair Plan Plan Monitor Resource Allocator Resource Plan A plan is a partially ordered collection of steps Each step achieves a subgoal Some steps establish pre-requisites for others Enactment Kevin Quigley — MIT Artificial Intelligence Lab

  13. Goals Service Mapper Diagnosis & Recovery Plan Monitor Plans Resource Pool Resource Allocator Actions Self-AdaptiveGoal Directed Processing Kevin Quigley — MIT Artificial Intelligence Lab

  14. Challenge 4: Context Awareness • The context should influence how the system behaves: • Task Structure • Location • Emotional State • Personal Style • Perception can help determine the context • The system should choose its reactions to events based on the context • Perceptual interpretation should be biased by context • E.g. a person near the White Board, is likely to start drawing • Estimation of utility should be influenced by context Kevin Quigley — MIT Artificial Intelligence Lab

  15. A Reactive System Responds to Events If somebody enters the room Then illuminate the room Kevin Quigley — MIT Artificial Intelligence Lab

  16. Events Structure of REBA • A Reaction maps an event to a goal • Reactions are grouped into behavioral bundles • Sets of reactions that are always activated as a unit • The Context is determined by the task within a plan • As well as location, people present, ... • Each context activates a set of behavioral bundles • Contexts have Sub-Contexts, activities that occur within other activities • Watching a video within a meeting • The active reactions of a sub-context override the reactions of the parent context Goal2 Sub-context-1 Goal1 Context Stack Sub-context-2 Reactions Kevin Quigley — MIT Artificial Intelligence Lab

  17. Context Awareness Conditions Reactions If somebody enters the room Then illuminate the room But not if a movie is being watched Kevin Quigley — MIT Artificial Intelligence Lab

  18. Goals Reactive Manager Service Mapper Diagnosis & Recovery Plan Monitor Context Plans Resource Pool Resource Allocator Actions Events Context Sensitivity Reactive Processing is provide by the REBA MetaGlue Component Kevin Quigley — MIT Artificial Intelligence Lab

  19. Challenge 5: Perceptual Integration • We want to separate the implementation of perceptual tasks from the uses to which perception is put • Some modules advertise the class of “behavioral events” they are capable of recognizing and signaling • These events are organized into a taxonomy • The same event can be signaled by quite different perceptual modules (e.g. both face and voice recognition can localize a person). • Other modules register their interest in certain classes of events • Requests at a higher level in the taxonomy subsume lower level events • Modules which receive low-level events may register for and collate many different classes of events • They combine these and signal higher-level events • Modules may request perceptual services when they are uncertain of their conclusions Kevin Quigley — MIT Artificial Intelligence Lab

  20. Interested in Face Location I Signal the Location of Individuals Face Recognition A Dynamic Event Bus For Perceptual Integration Interested in the Location of Individuals Signal People Approaching the Whiteboard White Board Context Manager Interested in Body Motion Signal the Location of Individuals Interested in Body Motion I Signal Face Location Voice Identification Face Spotter Signal Body Motion Visual Tracker • A “Blackboard” System • Publishers & Subscribers Are “Knowledge Sources” • Events Are the Blackboard Data Items • Highly Distributed • Use of Bayesian Techniques Kevin Quigley — MIT Artificial Intelligence Lab

  21. intent sentence word Syllable phonemic acoustic Blackboards: Using Context Context: Discussion of Oil Tanker Crash Fragility of Environment It’s not Hard to Wreck a Nice Beach Wreck A Nice Beach Recognize Speech Wreck A NICE Recognize A NIS Wreck Rec Og IZE R eck SP Og N I Z Kevin Quigley — MIT Artificial Intelligence Lab

  22. Integration of Reactive and Goal Directed Processing Goals Reactions Diagnosis & Recovery Service Mapper Plan Monitor Context Plans Events Resource Pool Resource Allocator Blackboard Sensory Systems Actions Kevin Quigley — MIT Artificial Intelligence Lab

  23. Summary • Responsive processing dynamically maps goals to plans • Plans are selected by balancing the benefit to the user against the cost of resources • Plan monitoring recovers from plan breakdown • Access control is handled as part of the cost benefit analysis • Reactive processing dynamically maps event to goals • Events are handled within context • Perceptions are maps to events within context Kevin Quigley — MIT Artificial Intelligence Lab

  24. Reactive and Goal Directed Processing with Security Goals Reactions Diagnosis & Recovery Service Mapper Plan Monitor Context Plans Events Resource Pool Resource Allocator Blackboard Access Policies Sensory Systems Actions Kevin Quigley — MIT Artificial Intelligence Lab

  25. The Metaglue System Software agents for intelligent spaces

  26. Smart Environments Distributed Components Dynamic Changes Highly Varied Equipment New Modes of Interaction Frequent Failures Abundance of Information Metaglue Communication Resource Management Customization Multi-modal HCI Agent Recovery Persistent Storage Computational needs Kevin Quigley — MIT Artificial Intelligence Lab

  27. The Metaglue System • What is it? • A communication infrastructure for building systems of distributed software agents • A software architecture for creating adaptive applications for Intelligent Environments • Why for Intelligent Environments? • dynamic adaptation of the applications based on • the availability of resources (hardware/software) in the current system/environment • security controls of the participants • preferences of the participants Kevin Quigley — MIT Artificial Intelligence Lab

  28. Metaglue Software Infrastructure • What do you get with Metaglue? • Multi-modal Human-Computer Interaction • Spoken Natural Language • Perceptual (Vision) • Direct manipulation (Graphical) • Persistent storage • Multiple communication paths • Remote procedure call (Java RMI) • Publish/Subscribe message passing • Configurable setting for asserting preferences • Start on demand of Agents • Automatic recovery / handling of direct communication errors • Resource Management and Service mapping Kevin Quigley — MIT Artificial Intelligence Lab

  29. What Are Software Agents? Our definition: • A software agent is any software object capable of communication by exposing functionality to other agents running within the network. Metaglue is a Multi-Agent System where agents perform individually specialized, (usually) simple tasks but connect in a web of intercommunication to cooperate on more complex tasks. Kevin Quigley — MIT Artificial Intelligence Lab

  30. More than Java Objects • Metaglue Agents can be as complex or simple as necessary • Larger programs can be used in Java thru JNI or other interface • ViaVoice, ASSIST (a sketching tool) • Most Metaglue Agents are simpler and single-purpose • Projector display or light control • Complex agents are those that control other simpler agents • ReBa – Reactive Behavioral System • SPIe – Self-adaptive Plan-based Intelligent Environment • Metaglue provides to agents • identity and occupation • the entity this agent represents • location and the ability to change locations • intrinsic communication to other agents Kevin Quigley — MIT Artificial Intelligence Lab

  31. Device Control lights, drapes, fans, sensors DVD & CD players, MP3 video multiplexers, projectors Cameras, TVs, VCRs, Audio Data Organization Systems Blackboard agents – data flow START system – NLP & KB Presence & Location agents Newswall – visual data organization and presentation Agent Systems Applications ReBa – interactive behaviors SPIe and Planlet – plan monitoring Web info display Debugging and Logging Agent testing, Simulated devices Log & Catalog Monitors Notification listeners Recreation Checkers, Boggle, RPG, HexaCheckers, Crosswords, ELIZA clone What are some of the agents in Metaglue? Kevin Quigley — MIT Artificial Intelligence Lab

  32. System Organization - Societies • Clusters of agents that operate on behalf of a real-world entity (space or person) are called Societies • Societies allow the same agent to exist with different customized attributes. • Agents can talk to agents in other societies as easily as their own society • Societies look like agents when viewed from the outside • They exposed functionality to higher level resource management through Hyperglue Kevin Quigley — MIT Artificial Intelligence Lab

  33. System Organization - Catalog • The Catalog Agent is the central component which knows about all running agents Kevin Quigley — MIT Artificial Intelligence Lab

  34. System Organization - MetaglueVM • The Metaglue Virtual Machine (MVM) runs as a base platform for all other agents in the Metaglue system. • It handles all registration of the agent with the Catalog Agent • Provides methods for direct communication (RMI) to other agents on the current catalog To Catalog Kevin Quigley — MIT Artificial Intelligence Lab

  35. Calling on Other Agents – agent present • A reliesOn call by Agent B will create a chain of events to get the RMI stub for Agent A • The MVM takes the call and passes it to the Catalog Agent • The Catalog checks the internal table of agents to see if there is one matching the requested description • If the agent exists, the RMI stub for Agent A is returned Kevin Quigley — MIT Artificial Intelligence Lab

  36. Calling on Other Agents – agent not present 3. The Catalog Agent checks the internal table of agents to see if there is one matching the requested description • If the agent does not exist, it must be started locally (calling MVM) • The new Agent A registers its stub • That stub is then returned to Agent B Kevin Quigley — MIT Artificial Intelligence Lab

  37. Automatic Agent Recovery Failure recovery through proxies

  38. Without error handling Kevin Quigley — MIT Artificial Intelligence Lab

  39. Without error handling Kevin Quigley — MIT Artificial Intelligence Lab

  40. Without error handling Kevin Quigley — MIT Artificial Intelligence Lab

  41. Without error handling Kevin Quigley — MIT Artificial Intelligence Lab

  42. Error handling with proxy objects Kevin Quigley — MIT Artificial Intelligence Lab

  43. Error handling with proxy objects Kevin Quigley — MIT Artificial Intelligence Lab

  44. Error handling with proxy objects Kevin Quigley — MIT Artificial Intelligence Lab

  45. Statistics on Metaglue • 10 Tons of fun: • There are over 450 agents that exist within Metaglue • Between 50 and 80 agents are running the intelligent room • You are using more than 10 agents just while running the X10BasicLightControl • Test it! Use agentland.util.LogMonitor • Metaglue has been in development since 1998 • The system is used in several offices and homes including the office of the AI lab director, Rodney Brooks • There are 2 full spaces at MIT (a 3rd is coming soon!) and one space in Australia running Metaglue • Why not get your own? Kevin Quigley — MIT Artificial Intelligence Lab

More Related