5th International Planning Competition: Results of the Deterministic - - PowerPoint PPT Presentation

5th International Planning Competition: Results of the Deterministic Track

Alfonso Gerevini DEA – University of Brescia, Italy gerevini@ing.unibs.it

IPC-5 Organizing Committee:

• Y. Dimopoulos, A. Gerevini (chair), P. Haslum, A. Saetti

Talk Outline

• General Organization of IPC-5
• New Language for the Deterministic Part
• Benchmark Domains for the Deterministic Part
• Competing Planners and Evaluation Criteria
• Samples and Summary of the Results
• Awards and Best Performing Planners

2

General Organization (Deterministic Part)

• Organizing Committee:

Y.Dimopoulos, A.Gerevini (chair), P.Haslum, A.Saetti

• Consulting Committee: S.Edelkamp, M.Fox, J.Hoffmann,

D.Long, D.McDermott, L.Schubert, I.Serina, D.Smith, D.Weld

• General Goals of IPC:

– analyzing and advancing the state-of-the-art – providing new benchmarks and data sets to the community – emphasizing new research issues in planning – promoting applicability of planning technology.

• Focus of the 5th IPC: plan quality (“traditional” quality mea-

sures + new measures related to the new planning language).

3

The Planning Language of IPC-5: PDDL3

Developed with D. Long. Extends previous versions of PDDL with

• Soft Goals: desired goals (don’t have to be necessarily achieved)
• State Trajectory Constraints: constraints on the plan struc-

ture using a LTL-like language – Strong: must be satisfied in any valid plan – Soft: don’t have to be necessarily satisfied

• Preferences: Soft goals and constraints with penalty weights
• Plan Metric: includes preference penalties to be minimized

– satisfying all goals and constraints can be infeasible – tradeoff between computational cost and plan quality

4

Example of Benchmark Domain: Travelling Purchaser Problem (TPP)

M4 M5 M2 D1 D2 M1 M3

Given (1) a set of different types of goods (2) a set of markets (M) selling different types and amounts of goods at different prices, (3) a demand of each type of goods to be purchased and transported by trucks to some depot (D),

⇒ satisfy the demand minimizing the routing cost of the trucks

and the purchasing cost 6 different PDDL formulations with simplifications and extensions

5

Examples of trajectory constraints in TPP

Each market is visited at most once by a truck: (forall (?m - market ?t - truck) (at-most-once (at ?t ?m))) At most one truck at a market at the same time: (forall (?m - market ?t1 ?t2 - truck) (always (imply (and (at ?t1 ?m) (at ?t2 ?m)) (= ?t1 ?t2)))) Each truck should be used (loaded with some goods): (forall (?t - truck) (sometime (exists (?g - goods) (> (load ?g ?t) 0)))) Whenever goods3 are loaded, they should be in a depot within 100 units: (forall (?t - truck) (always-within 100 (> (loaded goods3 ?t) 0) (= (loaded goods3 ?t) 0))) We start storing goods2 in a depot only after we have stored the requested amount of goods1: (sometime-before (> (stored goods2) 0) (>= (stored goods1) (request goods1)))

6

Benchmark Domains of IPC-5

5 new domains + 2 from IPC-3/4: 36 variants, 978 problems

• TPP: traveling and buying goods at selected markets minimizing costs

(from OR with variants, NP-hard)

• Openstacks: combinatorial optimization problem in production schedul-

ing (from CSP benchmarks, NP-hard)

• Storage: moving and storing crates of goods by hoists from containers to

depots with spatial maps

• Pathways: finding a sequence of biochemical (pathways) reactions in an
• rganism producing certain substances
• Trucks: moving packages between locations by trucks under certain spatial

constraints and delivering deadlines

• Rovers (IPC-3), PipesWorld (IPC-4).

7

Subtracks & Domain Categories

Subtracks: Optimal Planning and Satisficing (sub-optimal) Planning Domain Categories:

• Propositional: ADL or (compiled) STRIPS domains
• Metric-Time: PDDL2.2 features (IPC-3/4), no derived effects
• Simple Preferences: propositional domains with soft goals
• Qualitative Preferences: propositional domains with soft

trajectory constraints

• Constraints: Metric-Time domains with strong trajectory

constraints

• Complex Preferences: Metric-Time domains with soft trajec-

tory constraint and/or soft goals.

8

Competing Planners (optimal track)

• CPT2 (V. Vidal and S. Tabary)

Partial-order causal-link planning and constraint satisfaction

• FDP (S. Grandcolas and C. Pain-Barre)

CSP techniques and planning graphs

• IPPLAN-1SC (M. van den Briel, S. Kambhampati and T. Vossen)

Planning as integer programming

• Maxplan (Z. Xing, Y. Chen and W. Zhang)

Planning as propositional satisfiability with problem decomposition

• MIPS-BDD (S. Edelkamp)

Symbolic planning based on BDDs

• SATPLAN (H. Kautz, B. Selman, and S. Neph)

Planning as propositional satisfiability (new encoding)

• SATPLAN.IPC4 and CPT.IPC4 (reference planners – IPC-4 winners)

9

Competing Planners (suboptimal track)

• Downward-sa (M. Helmert)

Planning based on heuristic search

• IPPLAN-G1SC (M. van den Briel, S. Kambhampati and T. Vossen)

Planning as integer programming

• MIPS-XXL (S. Edelkamp, S. Jabbar and M. Nazih)

Planning based on heuristic search and domain compilation techniques

• SGPlan5 (C. Hsu, B. W. Wah, R. Huang and Y. Chen)

Planning based on problem partitioning and heuristic search

• HPlan-P (J. Baier, F. Bacchus and S. McIlraith)

Planning based on heuristic search and domain compilation techniques

• YochanPS (J. Benton, S. Kambhampati and M. Do)

Techniques for Partial satisfaction planning and heuristic search

• Downward.IPC4 and SGPlan.IPC4 (reference planners – IPC-4 winners)

10

General Evaluation Criteria

• Different evaluation/prizes for optimal and suboptimal planners
• For optimal planners: number of solved problems and CPU-

time (CPU-time limit: 30 minutes)

• For satisficing planners:
• 1. Number of solved problems and plan quality
• 2. CPU-time (secondary measure)
• Planner ranking by domain category (as in IPC-4):

– for each domain in a category we assign 1st/2nd places; – in each category, all 1st/2nd places are then summed

• IPC-4 best planners as reference for performance improvements.

11

Sample of Results: TPP-prop. (speed optimal planners)

30 problems. Largest problem solved by SATPLAN: 163 actions, 11 levels

12

Sample of Results: Pathways-prop. (speed optimal planners)

30 problems. Largest problem solved by Maxplan: 135 actions, 20 levels

13

Sample of Results: Storage-prop. (quality suboptimal planners)

14

Sample of Results: Openstacks-time (quality suboptimal planners)

Plan quality: makespan

15

Sample of Results: TPP-SimplePref. (quality suboptimal planners)

Plan quality: linear combination of preference violation penalties Only soft goals. Not all preferences can be satisfied

16

Sample of Results: Openstacks-QP (quality suboptimal planners)

Plan quality: linear combination of preference violation penalties Strong and soft goals. Not all preferences can be satisfied

17

Sample of Results: Openstacks-QP (speed suboptimal planners)

Sample of Results: Pathways-ComplexP. (quality suboptimal planners)

Plan quality: preference violation penalties, chemical substances, makespan Only soft goals. Not all preferences can be satisfied

18

Summary of 1st/2nd Places

(optimal planners with at least one 1st or 2nd place) IPC-5 Category CPT2 MIPS-bdd SATPLAN Maxplan FDP Prop. 0/1 1/1 3/2 3/2 0/3 Time 2/0 IPC-4 Category SATPLAN.ipc04 CPT.ipc04 Prop. 0/2 Time 0/2

19

Summary of 1st/2nd Places

(suboptimal planners with at least one 1st or 2nd place) IPC-5

Category Downward Mips-bdd Mips-xxl SGPlan.5 HPlan-P YochanPS Propositional 1/4 0/1 5/2 0/1 MetricTime 0/3 8/1 1/3 SimplePref. 0/1 0/4 6/0 0/4 QualPref. 5/0 0/5 Constraints 0/3 3/0 ComplexPref. 0/3 5/0

IPC-4

Category Downward.ipc04 SGPlan.ipc04 Propositional 3/4 MetricTime 0/5

20

IPC-5 Prizes (deterministic part)

• Optimal planning:
• 1st Prize: best propositional planner of IPC-5
• Distinguished performance in temporal domains
• Suboptimal (satisficing) planning:
• 1st Prize: best satisficing planner of IPC-5
• Some 2nd prizes for distinguished performance in the new do-

main categories (soft goals, qualitative preferences, strong/soft constraints)

21

And the Winner is....

• Optimal planning:
• Suboptimal (satisficing) planning:

22

And the Winner is....

• Optimal planning:

Distinguished performance in temporal domaions: CPT2

• Suboptimal (satisficing) planning:

23

And the Winner is....

• Optimal planning:

1st Prize: SATPLAN and Maxplan (propositional domains) Distinguished performance in temporal domains: CPT2

• Suboptimal (satisficing) planning:

24

And the Winner is....

• Optimal planning:

1st Prize: SATPLAN and Maxplan (propositional domains) Distinguished performance in temporal domains: CPT2

• Suboptimal (satisficing) planning:

Distinguished performance:

• Mips-xxl (Simple/Complex Preferences, Constraints)
• HPlan-P (Qualitative Preferences)
• YochanPS (Simple Preferences)

25

And the Winner is....

• Optimal planning:

1st Prize: SATPLAN and Maxplan (propositional domains) Distinguished performance in temporal domains: CPT2

• Suboptimal (satisficing) planning:

1st Prize: SGPLAN5 (best overall performance) Distinguished performance:

• Mips-xxl (Simple/Complex Preferences, Constraints)
• HPlan-P (Qualitative Preferences)
• YochanPS (Simple Preferences)

26

Conclusions: Overall Results

• New language for modeling preferences and soft constraints/goals
• A large set of new benchmarks
• 12 competing planners (5 of them handle PDDL3 features).

Significant advances in both the optimal and suboptimal tracks!

• Suboptimal planners evaluated by plan quality (other criteria

may reveal other improvements and different evaluation results).

• An archive of all data (soon available on the IPC-5 website) to

be used as reference for the community.

27