Modern Planning Techniques Part II

1. Modern Planning TechniquesPart II Jőrg Hoffmann Albert-Ludwigs-University Freiburg, Germany

2. Outline • Modern Planners (since 1995) • Part I: ID PLANMIN • Reachability • Regression Search • Non-directional Search • Part II: Greedy PLANSAT • Ignoring Delete Lists • Local Search Topology • Numeric and Temporal Extensions • Summary

3. Greedy PLANSAT Directly consider plan existence problem, without plan length bound Greedy element: simply assume that there is a plan, and (heuristically) go look for (an arbitrary)one + we can use non-admissible heuristics – techniques (per se) completely useless for proving unsolvability, or finding provably optimal plans

4. FF vs. IPP in Gripper ([Helmert 03]: most benchmarks have lower complexity in sub-optimal case)

5. Part II: Greedy PLANSAT • Ignoring Delete Lists • A Popular Relaxation • Approximating Relaxed Plan Length • Heuristic Search • Systems, and Open Questions • Local Search Topology • Heuristic Landscapes • h+ in the Planning Benchmarks • Open Questions

6. Relaxations • General principle for deriving heuristics: Define a simplification (aka relaxation) of the problem, take solution to the simplified problem as heuristic • Eg, straight-line distance as a heuristic in a road map is based on the simplification that one needs no roads • Eg, k-reachability is based on the simplification that we only need to achieve the hardest k-subset • Another possible relaxation in planning: ignore the negative effects (aka delete lists) of the operators

7. Ignoring Delete Lists • Technically: pre => add, del pre => add • Example Drive-truck: at(T,L1) => at(T,L2), ¬at(T,L1) • Set of true facts increases monotonically under relaxed action application • Simplifies task because preconditions and goals are all positive

8. Example T P1 P2 • Relaxed Plan: Load T P1 Left Drive T Left Right Unload T P1 Right Load T P2 Right Unload T P2 Left

9. Ignoring Delete Lists, ctd. • We focus on sequential plans in the following, more generally on action count heuristics • Optimal relaxed plan length is an admissible heuristic Bylander [94]: relaxed PLANSAT is in P, relaxed PLANMIN is NP-complete => Use approximations of optimal relaxed plan length to obtain, in general non-admissible, heuristics • Percentage of domain-independent competition planners using this idea: 20%'98, 42%'00, 63%'02 => ??% '04

10. Relaxed Plan Length • Say we want to know, in the current search state, the number of actions needed to get from fact set F to fact set F' (concrete search schemes: see below) • Admissible estimate: h+(F,F') := length of optimal (shortest) relaxed plan from F to F' • h+ hard to compute => we approximate

11. Forward Chaining • Bonet & Geffner [97,99,01]: (HSP1/2, HSPr) h(F,f) := 0, if fF; else, mina,fadd(a) (1+ppre(a) h(F,p)) h(F,F') := fF'h(F,f) • ie, estimate „cost“ of fact set by sum of fact costs • Assumes that facts must be achieved independently • ignores positive interactions • can overestimate

12. Forward Chaining, ctd. • For all f : h(F,f) := 0, if fF; h(F,f) := ∞, else • While ( changes occured ) • for all a, ppre(a) h(F,p) < ∞: for all f, fadd(a): h(F,f ) := min(h(F,f ), 1+ ppre(a) h(F,p)) • Note: • this is a value iteration technique... • we might have to re-compute this for each state, depending on the search scheme (see later)

13. Example T P1 P2 • Fact costs: (F = initial state, F' = goal) • at(T,Left) 0, at(P1,Left) 0, at(P2,Right) 0 • at(T,Right) 1, in(P1,T) 1 • at(P1,Right) 3, in(P2,T) 2 • at(P2,Left) 3 • h+(F,F') = 5; h(F,F') = 6(Drive T Left Right counted twice)

14. Backward Chaining • McDermott [96,99]: (Unpop) „Greedy Regression-Match Graphs“ • Roughly: • start with single open subgoal F' • regress (ignoring interactions) each open subgoal O through all action sets A s.t. reg(O,A)  F is maximal • leaf nodes are O  F • cost of leaf O is 0, else it‘s minA (|A| +cost of reg(O,A))

15. Example T P1 P2 • Backwards graph: (F = initial state, F' = goal) • at(P1,Right), at(P2,Left) • Unload T P1 Right, Unload T P2 Left • in(P1,T), at(T,Right), in(P2,T) • Load T P1 Left, Drive T Left Right, Load T P2 Right -- OR ... • at(T,Right) • Drive T Left Right • h+(F,F‘) = 5; h(F,F‘) = 6 (Drive T Left Right counted twice)

16. Backward Chaining, ctd. • Computing greedy regression-match graphs appears to be very time-consuming • Can ignore positive interactions • Refanidis & Vlahavas [99,01]: (GRT) „Greedy Regression Tables“ • If F' is constant during search, one can partly pre-compute the backchaining process (see later) • …and take some positive interactions into account

17. Fwd./Bwd. Chaining • Hoffmann & Nebel [00,01]: (FF) Chain forward to compute fact costs; use fact costs for efficient backward chaining • Chain forward to compute parallel reachability in the relaxation (one can simply replace HSP-“” by “max”) • Chain backwards in reachability layers, and select achieving actions below • Is the same as a relaxed Graphplan execution, returns a relaxed plan

18. Example, Fwd. T P1 P2 • Fact costs: (F = initial state, F' = goal) • at(T,Left) 0, at(P1,Left) 0, at(P2,Right) 0 • at(T,Right) 1, in(P1,T) 1 • at(P1,Right) 2, in(P2,T) 2 • at(P2,Left) 3 • Iterations == relaxedplanning graph layers

19. Example, Bwd. T P1 P2 • Backwards action selection: (F = initial state, F' = goal) • 3: at(P2,Left) • Unload T P2 Left • 2: at(P1,Right), in(P2,T) • Unload T P1 Right, Load T P2 Right • 1: in(P1,T), at(T,Right) • Load T P1 Left, Drive T Left Right • Iterations == relaxed regression

20. SearchSchemes • Search Scheme (S, s0, br, Sol): • space of all search states S • start state s0S • branching rule br:S  ((S)) (maps a state to a set of branching points; each branching point is a set of search states) • solution states Sol S (s.t. below each branching point there are the same solutions) • Transition-cost c:S x S  N0, path-cost g(s0 ... s) • Remaining cost heuristic h: S  N0 ? Heuristic search algorithm? • In what follows, branching heuristic not considered; we simply assume a set of successors to each state

21. Forward Search • Search Scheme (S, s0, br, Sol) where • S == space of all fact sets F, ie world states • s0 == initial state • br(s) == a single branching point, containing the results of executing the actions (or parallel action sets) applicable in F • Sol == fact sets that contain the goal • c(s,s') here, uniform 1, ie action count (step count) • Easy to understand, captures reachability; needs to be informed about relevance

22. Backward/Regression Search • Search Scheme (S, s0, br, Sol) where • S == space of all fact sets F, ie sub-goals • s0 == original goal set • br(s) == a single branching point, containing F‘s regression through all possible actions (or parallel action sets) • Sol == fact sets contained in original initial state • c(s,s') here, uniform 1, ie action count (step count) • Captures relevance; needs to be informed about reachability (in particular, “spurious” states are possible)

23. Partial-Order Search • Search Scheme (S, s0, br, Sol) where • S == space of all partially ordered ID‘ed actions • s0 == ({0:aI, 1:aG}, {0<1}) where aI/aG are dummy actions that add the initial state/require the goal • br(s) == branching points are the flaws (open conditions or threats), each point contains the set of possible flaw repairs • Sol == search states without flaws • c(s,s') here, 1 if flaw repair added an action, 0 else (ie action count) • More flexible than either forward or backward search, maybe better for temporal planning; captures relevance, spurious states possible • Branching heuristics known as „flaw selection strategies“

24. From F to F' • The semantics: • In forward search, F is the current search state, and F' is the goal • In backward search, F is the initial state, and F' is the current subgoal • In partial-order search, F is the initial state and F' are the currently open conditions • If F' is the same throughout search then heuristic backchaining can be (partially) pre-computed  GRT • If F is the same throughout search then heuristic forward chaining can be (partially) pre-computed  HSP

25. Global Search Algorithms Expand all search states in an order that depends on their heuristic values • Weighted A*: combine path costs and remaining costs by f(s) := w * g(s) + (1-w) * h(s) and expand search states by increasing f value • First found solution optimal (wrptoc) if w = 0.5, and heuristic admissible (wrptoc) (valid irrespective of branching heuristic) (eg. partial order) • Greedy with w < 0.5, extreme case w = 0,Best-first search, used in many planners

26. Local Search Algorithms Look at local surroundings of search states, and try to make improvements on h value • Hill-climbing: • Start in s0 • When in s, select one successor s' s.t. h(s') minimal • „Enforced Hill-climbing“: (used in FF) • Start in s0 • When in s, do breadth-first search for s' s.t. h(s') < h(s) • Note: algorithms of this kind can‘t prove unsolvability

27. Systems, Overview STAN4, LPG, AltAlt „special cases“...

28. Systems • Unpop [McDermott]: fwd search, bwd h, best-first/hill-climbing • HSP1 [Bonet&Geffner]: fwd search, fwd h, hill-climbing • HSPr [Bonet&Geffner]: bwd search, fwd h (pre-computation), best-first • GRT [Refanidis&Vlahavas]: fwd search, bwd h (pre-computation), best-first • FF [Hoffmann&Nebel]: fwd search, fwd/bwd h, enforced hill-climbing/best-first; pruning technique using relaxed plans • HSP2 [Bonet&Geffner]: configurable; fwd search, fwd h, best-first • AltAlt [Kambhampati et.al.]: bwd search, fwd h (various versions, including non-relaxed plan based ones), weighted A*

29. Systems, ctd. • Mips [Edelkamp&Helmert]: hybrid; fwd search, fwd/bwd h, weighted A* • STAN4 [Fox&Long]: hybrid; fwd search, fwd/bwd henhanced with generic types-knowledge, hill-climbing • Repop [Nguyen&Rao]: partial-order search, fwd/bwd h, weighted A* • SAPA [Do&Rao]: temporal&numeric; fwd search, fwd/bwd h, weighted A* • VHPOP [Younes&Simmons]: partial-order search, fwd h, weighted A* • LPG [Gerevini&Serina&al.]: search in the space of „action graphs“ (partial planning graphs); amongst others, a heuristic extracting relaxed plans for preconditions in the planning graph (fwd/bwd in that sense); hill-climbing style algorithm • Metric-FF [Hoffmann]: relaxation extended to numeric variables

30. Open Questions (I) How can we develop heuristics that yield good performance in those cases where current planners are bad? • Improve on relaxed plan estimate using additional information (STAN4, LPG) • Use different relaxations (eg pattern databases [Edelkamp 01])

31. Open Questions (II) How can we speed up the computations that estimate relaxed plan length in each search state? • Bottleneck in most systems (easily 80% of runtime) • Remove actions that are irrelevant within relaxation [Hoffmann&Nebel 01] • Re-use computation made in previous search state [Liu et al 02] • Avoid operator instantiation?

32. .bbl • T.Bylander. The Computational Complexity of Propositional STRIPS Planning. AIJ’94. • D.McDermott, A Heuristic Estimator for Means-Ends Analysis in Planning. AIPS’96. • B.Bonet, H.Geffner, Planning as Heuristic Search. AI’01. • I.Refanidis, I.Vlahavas, GRT: A Domain-Independent Heuristic for STRIPS Worlds Based on Greedy Regression Tables. ECP’99. • J.Hoffmann, B.Nebel, The FF Planning System: Fast Plan Generation Through Heuristic Search. JAIR’01. • M. Fox, D. Long, STAN4: A Hybrid Planning Strategy Based on Subproblem Abstraction. AIM’01. • X.Nguyen, S.Kambhampati, Reviving Partial Order Planning. IJCAI‘01 • M.Do, S.Kambhampati, Sapa: A Domain-Independent Heuristic Metric Temporal Planner. ECP’01.

33. .bbl, ctd. • X.Nguyen, S. Kambhampati, Extracting Effective and Admissible State Space Heuristics from the Planning Graph. AAAI’00. • J.Hoffmann, Extending FF to Numerical State Variables. ECAI’02. • H.Younes, R. Simmons, On the Role of Ground Actions in Refinement Planning. AIPS’02. • A. Gerevini, A. Saetti, I. Serina, Planning through Stochastic Local Search and Temporal Action Graphs in LPG. JAIR’03. • S.Edelkamp, Planning with Pattern Databases. ECP’01. • J.Hoffmann, B.Nebel, RIFO Revisited: Detecting Relaxed Irrelevance. ECP’01. • Y.Liu, S.Koenig, D.Furcy, Speeding Up the Calculation of Heuristics for Heuristic-Search-Based Planning. AAAI’02. • M. Helmert, Complexity Results for Standard Benchmark Domains in Planning, AIJ’03.

34. Heuristic Quality => Search Performance • A common-sense wisdom: The performance of a heuristic search algorithm is largely determined by the quality of the heuristic function • Extreme cases: + h(s) exact -- solution is found in linear time –h(s) constant -- no heuristic information, search trivializes to breadth/depth-first

35. Problem Structure => Heuristic Quality • Another common-sense wisdom: The quality of a heuristic function is largely determined by the structure of the specific problem instance • Extreme cases: + when driving X-over is possible, straight-line distance is exact – eg, getting from Hurghada to Sharm-el-Sheik

36. Local Search Topology • Search space with heuristic forms a landscape: states, and state transitions == road map heuristic values == height of states • Heuristic search moves along this landscape, searching for a lowest point (== solution) • Measure heuristic quality by form of landscape, its “local search topology” • Formally, landscape is annotated directed graph • Eg, [Frank et. al., JAIR-97] (for random SAT)

37. Local Minima “Must go upwards to reach a solution“

38. Local Minima Maximal exit distance Exit “Must go upwards to reach a solution“

39. Benches “All neighbour states look the same“

40. Benches Exit Maximal exit distance Exit “All neighbour states look the same“

41. Dead Ends “There is no path to a solution“

42. Unrecognised Dead Ends Recognised “There is no path to a solution“

43. Local Search Topology in Planning Benchmarks • We look at: (1)forward state space search (2)with the h+ heuristic • Why (1)? – simple to look at,used in many systems (in particular, my own) • Why (2)? – approximated by state-of-the-art sytems, can obtain hard(provable) results for it

44. A Planning Domain Taxonomy:(proved) Topology under h+ In lowermost classes, FF‘s search algorithm is polynomial!

45. Structure => Topology • There are 3 widely spread patterns of structure: Pattern 1 => no dead ends Patterns 1 & 2 => no local minima Patterns 1 & 2 & 3 => maximal benchexit distance <= c

46. Patterns of Structure - I • (At least) invertible actions: actions a to which there exists an inverse action undoing (at least) a‘s effects • Irrelevant negative effects: not needed anywhere • Static add effects: not deleted by any action • Example: • Drive T Left Right / Drive T Right Left • Load / Unload

47. Patterns of Structure - II • Actions that are respected by the relaxation: if a starts an optimal plan from (any) s, then a also starts an optimal relaxed plan from s (=> construct relaxed plan for successor state by either removing a, or replacing it with the inverse action) • Example: • Drive T Left Right: truck is needed at right side, to load/unload package • Load T P1 Left: P1 must be at right side • Unload T P1 Left: P1 must be at left side

48. Patterns of Structure - III • Actions whose negativeeffects are irrelevant upon execution: once a has been applied on an optimal solution path, a‘s negative effects are no longer needed (neither real nor relaxed) • Example: • Load T P1 Left: P1 must be at right side, delete is at(P1 Left) • Unload T P1 Left: P1 must be at left side, delete is in(P1 T) • Other actions (example: Drive) need to be applied on an optimal solution path <= c times in a row (example: 1)

49. Structure => Topology Pattern I Patterns I&II Patterns I&II&III

50. Open Questions (I) Is this (the quality of h+) a good or a bad result for AI planning? • Are the observed patterns of structure a phenomenon of the real world, or only a phenomenon of the, unrealistically simple, benchmarks? • Hard to answer, just like that. Best, get more realistic benchmarks