Honeywell Labs

1. Honeywell Labs Achieving Intelligent Real-Time Control Dr. David J. Musliner David.Musliner@honeywell.com (612) 951-7599

2. Outline • Motivation for this research and this talk. • Approach: • Overview of CIRCA, the Cooperative Intelligent Real-Time Control Architecture. • Brief details on executive. • Details on planner. • Plan verification.

3. Characteristics of Motivating Problems • UAVs, UGVs, UUVs, spacecraft, rovers… • Complex dynamic environments: • Require intelligence. • Critical environments: • Require predictable performance. • Logical correctness. • Timeliness guarantees. • Resource limitations: • Bounded reactivity. • Bounded rationality. • Sometimes distributed.

4. CIRCA Motivation: Real-Time Intelligent Control • Time-critical, hazardous situations --- Possibly catastrophic. • CIRCA guarantees that it will respond in a timely way to threats in its environment. • Flexible systems --- Many alternative actions/behaviors. • CIRCA automatically selects behaviors and reconfigures itself while it is operating. • Limited resources --- Bounded reactivity & rationality. • CIRCA dynamically synthesizes controllers (plans) for only the immediately relevant parts of the situation. CIRCA does this introspectively, reasoning about resource limits.

5. Component Technologies • Reactive controller execution to ensure real-time responsiveness and opportunistic behavior. • Automatic controller synthesis (planning) to adapt to changing conditions. • This work formed the motivation and precursors to Häkan’s thesis. • Task-based negotiation to distribute roles/responsibilities. • Decision-theoretic deliberation scheduling to optimize benefits of negotiation and synthesis activity.

6. CIRCA Architecture Adaptive Mission Planner: Divides an overall mission into multiple control problems, with limited performance goals designed to make the controller synthesis problem solvable with available time and available execution resources. Deliberation scheduling. Controller Synthesis Module: For each control problem, synthesizes a real-time reactive controller according to the constraints sent from AMP. Planning. Real Time Subsystem: Continuously executes synthesized control reactions in hard real-time environment; does not “pause” waiting for new controllers. Adaptive Mission Planner Controller Synthesis Module Real Time Subsystem

7. Break down mission Adaptive Mission Planner Generate controller Generate controller Generate controller State Space Planner Execute controller if (state-1) then action-1 if (state-2) then action-2 ... Real Time System

8. Adaptive Mission Planner Roles, Goals Planned Actions, Planned Negotiations Controller Synthesis Module Real-Time Reactions Real Time System Extending Performance Guarantees to Multi-Agent Teams Adaptive Mission Planner: Explicitly manages complexity of planning and negotiation processes that dynamically distribute roles/responsibilities. Controller Synthesis Module: Builds controllers that include coordinated actions by multiple agents. Real Time Subsystem: Executes coordinated controllers predictably, including distributed sensing and acting. Only system to guarantee end-to-end multi-agent coordinated behaviors Adaptive Mission Planner Controller Synthesis Module Real Time System

9. CIRCA Architecture - RTS Adaptive Mission Planner Feedback Data Subgoals, Configurations Controller Synthesis Module Feedback Data Reactive Plans Real Time System The World

10. Real Time Subsystem (RTS) • The RTS executes loops of Test-Action Pairs (TAPs). • Each TAP takes a snapshot of world state and then executes test expression. • Testi tests for a group of states in which Actioni was planned. • Actioni may take up to its worst-case execution time to execute. • The Controller Synthesis Module is responsible for building the TAP loop. test1 action1 test2 action2 test1 action1 test3 action3 test1 action1 test4 action4

11. test1 action1 test2 action2 test1 action1 test3 action3 test1 action1 test4 action4 Key RTS Features • The RTS executes in parallel with the other CIRCA modules. • Parallel execution permits re-planning using computationally-expensive algorithms while preserving platform safety. • Special-purpose TAPs used to download and switch to next controller. • RTS includes multiple TAP schedule caches to hold controllers before they are activated.

12. (light green) (status stopped) (light green) (status moving) Accel Light- changes (light red) (status moving) (light red) (status stopped) Accel (ghost) Cautionary Example: Ghosting Can Ruin Your Day • All real executives take non-zero time to select and perform a planned action: the “sense-act gap”. • CIRCA’s RTS takes a “snapshot” of consistent world state, executes boolean test expression, and performs action if expression returns true. • If the world changes between snapshot and action, the action may occur in a different state than originally intended. • Planner must understand this “ghosting”.

13. CIRCA Architecture - CSM Adaptive Mission Planner Feedback Data Subgoals, Configurations Controller Synthesis Module Feedback Data Reactive Plans Real Time System The World

14. Controller Synthesis Module State Space Planner Verifier TAP Compiler Scheduler Controller Synthesis Module (CSM) Available actions “Non-volitional” transitions Timed Automata Controller Design & Executable Reactive Controller Goal state description Initial state description Transition-based input model similar to classical planners, but with temporal characteristics and non-volitional transitions.

15. Controller Synthesis Module State Space Planner Verifier TAP Compiler Scheduler CSM Functionality • State Space Planner predicts future threats and opportunities, plans actions with timing constraints for future states. • Verifier reasons about complex temporal model to ensure that all failures are preempted. • TAP compiler reduces timed automata controller model to time-constrained reactions (Test-Action Pairs). • Scheduler builds executable cycle of TAPs to meet time constraints.

16. Domain Models • Domain models implicitly describe a state-space world model by describing its transitions. • Transitions are STRIPS-like (pre- and post-conditions). • Transitions may be controllable (actions) or uncontrollable. • Transitions have associated time bounds: • Lower bounds for uncontrollable transitions; • Upper bounds for actions; • Both for “reliable temporals” used to capture continuous processes.

17. Threatened Safe Failure Objectives of CIRCA plans • Maintain safety of agent while attempting to achieve goals. • Exert supervisory control of continuous processes in the environment and the plant. • Maintain safety by preempting bad events. OK Build and execute these plans (controllers) on the fly.

18. Cassini Spacecraft Example • Saturn orbit insertion was a mission-critical, one-shot engine burn opportunity. • Dual Inertial Reference Units (IRUs). • If one IRU fails during engine burn, system has only milliseconds to switch to a hot backup. • IRUs take much longer to “warm up”. • Hence both IRUs must be warmed up well in advance of critical engine burns.

19. Cassini Spacecraft Example (make-instance 'action :name "start_IRU1_warm_up" :preconds '((IRU1 off)) :postconds '((IRU1 warming)) :delay (make-range 0 1)) (make-instance 'temporal :name "fail_if_burn_with_broken_IRU1" :preconds '((engine on) (active_IRU IRU1) (IRU1 broken)) :postconds '((failure T)) :delay (make-range 5 ) (make-instance 'reliable-temporal :name "warm_up_IRU1" :preconds '((IRU1 warming)) :postconds '((IRU1 on)) :delay (make-range 45 60))

20. A Simple Preemption State Space Planner (SSP) has planned action select_IRU2 to preempt temporal transition to failure.

21. Temporal Reasoning Issue Safe Threatened Preparing • A single action may not “solve” a temporal transition to failure; TTF may persist across multiple states. • Non-Markovian representation of temporal information. • Path-dependent nature of transitions: how long was this transition enabled in prior states? • Need for efficient representation of only critical timing value distinctions leads to timed automata theory. • Approach: use timed automata methods to handle almost all temporal reasoning. Failure

22. Why Use Non-Markovian? • Efficient representation of temporal, concurrent, partially-controllable worlds. • All processes/actions have temporal extent and constraints. • Exogenous processes run concurrently with own actions. • They interact. • To write down in Markovian form, need to write state of all different process clocks into the state space. • Using compact non-Markovian representation allows planner to make decisions that span broad swaths of clock space. • But planner (and exec) cannot make time-dependent decisions that are not flagged by some discretized process state change.

23. Verifying Controllers • Translate state-space model of current plan into a timed automata model. • Introduce failure transitions for real failures. • Introduce failure transitions for states which the planner believes it has preempted. • Treat any as yet unplanned states as safe sink states. • Invoke verifier to test for the reachability of failure from the initial state. • Verifier enumerates all possible futures in a timed simulation. • Verification assures controller avoids known failures when given a correct domain model.

24. What are Timed Automata? • State machines with transitions that depend on synchronously-updated timers (called “clocks”). • All clocks update at same rate. • States can have “invariant” expressions on clocks that force automata to leave state before invariant would become false. • Transitions can have “guard” expressions that restrict when they can be followed (preconditions). • Transitions can have “reset” expressions that set some of the clocks back to zero.

25. Timed Automata for Preemption FAILURE Fail-if-burn-with-broken-IRU1 Guard: Cr > 5 IRU fails Reset: Cr=0 IRU1 broken Active-IRU IRU1 Engine on Invariant: Cr < 3 This planned action causes this invariant Select-IRU2

26. Multi-Machine Mapping • Base machine captures structure of SSP state space using edges labeled with synchronization tags. • Each SSP transition has separate machine moving between enabled and disabled states. • Base machine synchronizes with individual transition machines via tags/labels. • Model checker builds “cross product”. Base machine Planner’s state space machine Transition machines

27. Base Model for State Space Base model captures states of world and links parallel models for each planned or uncontrollable transition. Labels sync parallel automata.

28. Model for Uncontrollable Event Edges with shared labels can fire only when both models are able to traverse edge at same time. No timing info: event has zero delay from system’s perspective.

29. Model for Temporal Transition to Failure Clock Ct constrains when failure can happen.

30. Executive Model for Action Here’s where ghosting happens Clock Ca times planned select-IRU2 action. Executive may latch sensors and test in any state of world (base model); only commits if in state 41. Constraint on Clock Ca enforces preemption.

31. Model Checking Verifiers • Use advanced techniques to find equivalence regions in the space of continuous clock values. • Exhaustively enumerate the possible system traces, modulo clock region equivalence. • Produce example system trace to failure when verification fails. • Kronos (from VERIMAG). • CIRCA-Specific Verifier (HTC): • Optimized for CIRCA problems, implicit transition-based representation of state spaces. • Incremental version saves verifier state during plan generation and revision, reducing effort dramatically.

32. CSM Algorithm in a Nutshell • A search algorithm that • Assigns an action to each reachable state that: • Preserves safety and • If possible, moves towards a goal state; • Re-computes the set of reachable states as actions are chosen and uncontrolled processes projected forward. • Invokes a timed-automaton verifier after each decision to determine whether safety is preserved (is a failure state reachable?). • If partial plan is unsafe, use verifier trace information to guide backjumping. • Similar to timed game-theoretic approaches [Asarin, Maler, Pneuli]: choose a move for each discrete state that will avoid a victory by nature.

33. CIRCA Architecture - AMP Adaptive Mission Planner Feedback Data Subgoals, Configurations Controller Synthesis Module Feedback Data Reactive Plans Real Time System The World

34. Subdividing the Mission

35. Adaptive Mission Planner • Divide mission into phases, subdividing them as necessary to handle resource restrictions. • Build problem configurations for each phase, to drive CSM. • Modify problem configurations, both internally and via negotiation with other AMPs, to handle resource limitations. • Capabilities (assets). • Bounded rationality: deliberation resources. • Bounded reactivity: execution resources.

36. Deliberation Scheduling • AMP directs and monitors controller synthesis for each mission phase. • Attempts to maximize expected utility of total mission by selecting and ordering planning tasks. • Ensures timeliness by ordering planning tasks so that a safe plan is ready for the next phase. • Currently deliberation scheduling performed based on myopic approximate solutions to optimal MDP policies; ongoing research.

37. Adaptive Mission Planner Roles, Goals Planned Actions, Planned Negotiations Controller Synthesis Module Real-Time Reactions Real Time System Extending Performance Guarantees to Multi-Agent Teams Adaptive Mission Planner: Explicitly manages complexity of negotiation processes that dynamically distribute roles/responsibilities. Controller Synthesis Module: Builds controllers that include coordinated actions by multiple agents. Real Time Subsystem: Executes coordinated controllers predictably, including distributed sensing and acting. Adaptive Mission Planner Controller Synthesis Module Real Time System

38. AMP Negotiation • Contract Net style system for allocation of threats and goals to highest bidder. • Contracts arrive at any site. • Multiple concurrent negotiations. • Bids are commitments. • CN cycle: announce, bid, award, succeed/fail.

39. Multi-Agent Self-Adaptive CIRCA Approach: • Automatic synthesis and adaptation of guaranteed real-time controllers. Performance: • Reactive control responses to threats and contingencies in milliseconds. • Coordinated multi-agent behaviors in tens of milliseconds. • Dynamic reconfiguration of team mission plan in less than 10 seconds. • Demonstrations in simulated UAV team domains: coordinated defense, dynamic replanning for contingencies. Impact: • Robust UAVs that rebuild their own control systems in response to contingencies (e.g., damage, target of opportunity). • Smart UAV teams that actively coordinate distributed capabilities/resources to maximize mission effectiveness. • Sponsor: NSF; Honeywell; DARPA: SAFER, ANTS, MICA. Goal: Adaptive real-time coordination and control of multi-UAV teams.

40. Attacks, intrusions Security Tradeoff Planner Computing services Networks, Computers Controller Synthesis Module Intrusion Assessment Active Security Controller Executive CIRCADIA CIRCADIA: Synthesizing Security Control Systems Approach: • Use CIRCA controller synthesis to automatically generate security controllers. • Tailor responses automatically according to available resources, varying threat levels & security policies. Performance: • Fully autonomous operations defeating attacks in milliseconds. • Rapid reconfiguration for dynamic network assets, security state, threat profile. • Demonstrations in real computer networks. Impact: • Real-time responses defeat manual and automated attack scripts. • Automatic tradeoffs of security vs. service level and accessibility. • System derives responses for novel attacks built from known components. Sponsor: DARPA: CyberPanel. Goal: Automatic real-time response to computer security intrusions.

41. Summary If your planner doesn’t really understand your executive, it doesn’t really understand its plans. Planning for real world autonomy requires consideration of time, exogenous processes, and concurrency. Reliable autonomy is hard and important.

42. Related Work • Executive models: not much. • Planning in timed automata worlds: • SimPlan (Kabanza). • Synthesis of controllers via backwards fixpoint computations: Asarin, Maler, Pneuli, Sifakis. • Least restrictive controller design. • Efficient representations of state spaces using BDDs, planning as model checking w/ BDDs: Giunchiglia, Traverso. • Lots of work on timed automata verification (Yovine, Dill, …). • Also recent work on more complex forms of hybrid automata and relatively small amount of synthesis work for them: Henzinger, Sastry, Wong-Toi ...

43. Honeywell Technology Center THE END

44. Incremental Verification • CIRCA calls verifier many times to check partial plans. • Problem: Model checkers can take a long time to explore all possible paths. Most model checkers designed for batch operations. • Key idea: Retain information about prior verification runs to make subsequent verification runs more efficient. • Issues: • Dependency-directed backtracking changes plan and verification model in non-monotonic fashion. • Solution: invalidate verifier cache and start from scratch on backtrack.

45. Incremental Verification Results • Extremely valuable in low-backtracking domains: up to 97% speedup. • Low overhead to invalidate cache. • Patent application filed October 2001. 41% average speedup on 21 regression test domains. 17.5 minutes --> 1.4 minutes

46. Planning with Accurate Executive Model • Theme: pay attention to the plan executive… if your planner doesn’t understand the execution semantics, its plans may be incorrect. • Approach: • Simplify executive. • Model executive explicitly in planner. • Use increasing model resolution as plan nears completion. • Result: • Planner really understands what its plans will do, accounting for uncontrollable events and time. • Performance guarantees.

47. (light green) (status stopped) (light green) (status moving) Accel Light- changes (light red) (status moving) (light red) (status stopped) Accel (ghost) Cautionary Example: Ghosting Can Ruin Your Day • All real executives take non-zero time to select and perform a planned action: the “sense-act gap”. • CIRCA’s RTS takes a “snapshot” of consistent world state, executes boolean test expression, and performs action if expression returns true. • If the world changes between snapshot and action, the action may occur in a different state than originally intended. • Planner must understand this “ghosting”.

48. Inflammatory Position • Plan executives are getting increasingly complicated to account for world dynamics and uncertainty, and to encode human procedural knowledge conveniently. • Examples include RAPS/3T, Remote Agent Exec. • But, • Executives with multiple methods and persistent goals can have unmodeled side effects during plan execution. • Resource consumption, clobbering. • The whole point of planning is to give system the base knowledge to plan its way out of unusual situations. • Reactivity is relatively easy to plan. • So, as planner scalability improves, let’s get back to using strong planning with a simpler execution model. • And be sure planner understands its executive!

49. Executive Models to the Rescue • Verify plans using models of executive, representing sense-act gap explicitly. • Model checking efficiently examines all possible executions including those with ghosting effects.

50. CIRCA’s Controller Synthesis Module Available actions Controller Synthesis Module State Space Planner Uncontrollable transitions TAP Compiler Automata Controller Design including timing constraints Scheduler Initial state descriptions Verifier Goal state descriptions