The Role of Higher-order Models in Robotics and its Reasoning - - PowerPoint PPT Presentation

1 RobMoSys Review: WP2

• Feb. 20, 2018, Luxembourg

The Role of Higher-order Models in Robotics and its Reasoning Challenges

Herman Bruyninckx, RobMoSys project KU Leuven – TU Eindhoven MODELS 2019, 17 September, 2019, M¨ unchen

RobMoSys’ five levels of modelling

• 1. Abstraction: guidance for humans, by following harmonized

interpretation of abstractions.

• 2. Reuse & Flexibility: reuse and customization of robotics software

assets, via data sheets.

• 3. Predictability:

composition is correct by construction.

• 4. Automation: automate labor-intensive stuff:

Validation & Verification, code generation,. . .

• 5. Autonomy: models at run-time.

self-configuration & -adaptation, explanation,. . .

The Role of Higher-order Models in Robotics and its Reasoning Challenges

• H. Bruyninckx, RobMoSys project, KU Leuven – TU Eindhoven

MODELS 2019, 17 September, 2019, M¨ unchen 1

What is “higher-order modelling”? model:

• set of entities connected by relations.
• data structure for each entity & relation.

→ property graph (or“entity-relation” graph)

higher-order model:

• set of relations on top of other relations,
• with (partially ordered) hierarchy in the semantics of the

relations. → one sees the hierarchy in the directed graph structure, and in the properties of the relations.

Relevant for RobMoSys’ modelling levels 3–5.

The Role of Higher-order Models in Robotics and its Reasoning Challenges

• H. Bruyninckx, RobMoSys project, KU Leuven – TU Eindhoven

MODELS 2019, 17 September, 2019, M¨ unchen 2

Added value higher-order modelling?

• knowledge representation of the domain:
• data → information → knowledge
• composability and compositionality:

to combine pieces of knowledge in “the right way”

• reasoning: to explain, to generate, to monitor
• sofware brings knowledge representation too:
• configuration: gazillions of “magic numbers” to be combined when

composing components.

• coordination: generate task state machines at runtime, because

gazillions of contextual requirements require different interaction behaviour of components.

The Role of Higher-order Models in Robotics and its Reasoning Challenges

• H. Bruyninckx, RobMoSys project, KU Leuven – TU Eindhoven

MODELS 2019, 17 September, 2019, M¨ unchen 3

Major example

M0–M3 meta model of model-driven engineering

metametamodel1 RobMoSys GEO "meta model" URDF meta model b RobMoSys robot GEO UR5 model PR2 model corridor model conforms-to conforms-to

M3 M2 M1 M0

realisation of

Real-world systems

metametamodel 2 QUDT metametamodel 2 Provenance meta model b RobMoSys world GEO

The higher order represented here is:

• M1–M3 relations are relative; “hierarchy” can be

extended “upwards” indefinitely.

• level n models the constraints that must be

satisfied in a model at level n − 1.

• allows translation between (meta) models, for

their conforming parts.

• typically, that knowledge is used by humans using

a tool chain.

• dream of robotics: develop robots that can use

that knowledge, themselves.

The Role of Higher-order Models in Robotics and its Reasoning Challenges

• H. Bruyninckx, RobMoSys project, KU Leuven – TU Eindhoven

MODELS 2019, 17 September, 2019, M¨ unchen 4

Robotics example: motion stack

Most abstract model: mereo-topology

joint0

body0

body0 body1 b

• d

y 2 body3 joint0 joint1 joint2 camera

body1

joint1

body2

joint2

body3

attach

camera

The model represents:

• parts in the model, and
• connections between those parts.

The Role of Higher-order Models in Robotics and its Reasoning Challenges

• H. Bruyninckx, RobMoSys project, KU Leuven – TU Eindhoven

MODELS 2019, 17 September, 2019, M¨ unchen 5

One more concrete level: motion constraint relation

joint0

body0 body1

joint1

b0.fr0 b0.fr1 pose0 b1.fr0 b1.fr1 pose1

...

The extra higher-order model represents:

• joint is a motion constraint between robot’s links
• at every moment in time, two links have a relative pose whose

properties depend on the type of the joint constraint → mathematical constraints between positions on connected body points.

The Role of Higher-order Models in Robotics and its Reasoning Challenges

• H. Bruyninckx, RobMoSys project, KU Leuven – TU Eindhoven

MODELS 2019, 17 September, 2019, M¨ unchen 6

Yet one more concrete level: pose measurement type

joint0

body0 body1

joint1

b0.fr0 b0.fr1 pose0 b1.fr0 b1.fr1 pose1

...

meas0 sensorX

dt-part

• f QUDT

meas1 sensorY

dt-part

• f QUDT

The extra higher-order model represents:

• the pose is measured by sensors
• it has a dimension and type
• QUDT is a standard meta model for this purpose

The Role of Higher-order Models in Robotics and its Reasoning Challenges

• H. Bruyninckx, RobMoSys project, KU Leuven – TU Eindhoven

MODELS 2019, 17 September, 2019, M¨ unchen 7

Yet one more concrete level: pose measurement values

joint0

body0 body1

joint1

b0.fr0 b0.fr1 pose0 b1.fr0 b1.fr1 pose1

...

meas0 sensorX

dt-part

• f QUDT

meas1 sensorY

dt-part

• f QUDT

value0 value1

qu-part

• f QUDT

qu-part

• f QUDT

The extra higher-order model represents:

• measurement of pose gives numerical values.
• those quantities have physical units
• QUDT is a standard meta model for this purpose

The Role of Higher-order Models in Robotics and its Reasoning Challenges

• H. Bruyninckx, RobMoSys project, KU Leuven – TU Eindhoven

MODELS 2019, 17 September, 2019, M¨ unchen 8

Queries on such higher-order models

• raise an event when camera speed is below “motion blur” limit
• can my software components exchange velocity data with yours?
• if not, which software component can provide the missing

translation between both coordinate representation?

• generate the composite kinematics solver when Arm xyz is put on

top of MobileBase 123

• generate a dynamics solver that adapts to a 1kHz torque control

loop around it, and to the accuracy of the sensors

The Role of Higher-order Models in Robotics and its Reasoning Challenges

• H. Bruyninckx, RobMoSys project, KU Leuven – TU Eindhoven

MODELS 2019, 17 September, 2019, M¨ unchen 9

Robotics example: semantic map

elevator Door/Elevator, i.e., topological links to other maps Traffic Lane Semantic area constraining specific behaviour C1 D1

D11

D12 D13 D14 D15 C2 J1 J11 J12 J13 J14 J15 C11 E1 E11 E12 E13 E14 E15 E18 E17 E16 N1 N2 N3 N4 N5 N6 Map nodes, to compose into areas C12 J16 room corridor corridor corridor corridor intersection

The Role of Higher-order Models in Robotics and its Reasoning Challenges

• H. Bruyninckx, RobMoSys project, KU Leuven – TU Eindhoven

MODELS 2019, 17 September, 2019, M¨ unchen 10

Queries on such higher-order models

• raise event if robot is near high-risk area
• can my software component interpret all semantic tags relevant for

a given task?

• if not, which software component can provide the missing

translation between the map’s semantics and my software components data sheet?

• generate the task graph to move from Area xyz to Area 123,

while maintaining safe behaviour against all expected other users of the building

The Role of Higher-order Models in Robotics and its Reasoning Challenges

• H. Bruyninckx, RobMoSys project, KU Leuven – TU Eindhoven

MODELS 2019, 17 September, 2019, M¨ unchen 11

We’re not there yet. . . ! Nevertheless:

• problem investigated for 50 years already. . .
• the market pull is tremendous. . .
• all researchers claim they provide solutions. . .

Why?

• mainstream focus: sofware only.
• higher-order modelling is tough;

developing query solvers even more.

• software components: too limited compositionality via models.
• . . .

The Role of Higher-order Models in Robotics and its Reasoning Challenges

• H. Bruyninckx, RobMoSys project, KU Leuven – TU Eindhoven

MODELS 2019, 17 September, 2019, M¨ unchen 12

Mechanism of higher-order modelling

Entity-Relation models

Rel Arg1 Arg2 Arg3

properties properties properties properties

• entities are the arguments in relations.
• supporting data structure: property

graph: every node in the graph has:

• a property data structure.
• a list of outgoing edges.
• a list of incoming edges.
• a Semantic ID: its own ID + IDs of its

meta model + list of meta meta model IDs

The Role of Higher-order Models in Robotics and its Reasoning Challenges

• H. Bruyninckx, RobMoSys project, KU Leuven – TU Eindhoven

MODELS 2019, 17 September, 2019, M¨ unchen 13

Mechanism (continued) Reasoning

• query is a property graph in itself:
• sub-graph of entities one wants information about,
• when constrained by specified relations in the graph,
• and extra constraint relations introduced by the query.

For example: “Give all places on the map close to the robot, and

• bservable by its sensors”.
• solver base on graph traversal:
• represents the knowledge required to travel through the graph in the “right

way” to find the answer to the query.

• is also property graph, of higher order!
• state of the art: close to nowhere, still. . .

RobMoSys: develops platform infrastructure.

The Role of Higher-order Models in Robotics and its Reasoning Challenges

• H. Bruyninckx, RobMoSys project, KU Leuven – TU Eindhoven

MODELS 2019, 17 September, 2019, M¨ unchen 14

Major modelling errors

• using is-a instead of conforms-to.

E.g.: a mobile robot is not a robot, but it shares a lot of the behaviour

• using property (has-a) instead of attribute (property of argument

in relation) E.g.: a robot does not have a position value, because that is the property of a relation between the robot and its environment

• reification is not a first-class citizen:
• every relation becomes an entity in itself

→ can become part of higher-order relation itself! Examples of “modelling” languages that fail here: UML, StateCharts, SysML, OWL, Prolog, Lisp,. . .

The Role of Higher-order Models in Robotics and its Reasoning Challenges

• H. Bruyninckx, RobMoSys project, KU Leuven – TU Eindhoven

MODELS 2019, 17 September, 2019, M¨ unchen 15

Developments in RobMoSys

Cgraph library (from graphviz ecosystem):

• efficient C-library for property graphs

→ in-memory reasoning possible.

• query formulation and graph traversal solving are not supported. . .

Application focus, for now:

• geometric world model, with semantic labels.
• kinematic and dynamic solvers, with control and estimation around

it.

• semantic localization at “platform” level.

JSON-LD: identified as (non-exclusive!) best fit for purpose of serialisation and file format:

• supports Semantic ID out of the box.

The Role of Higher-order Models in Robotics and its Reasoning Challenges

• H. Bruyninckx, RobMoSys project, KU Leuven – TU Eindhoven

MODELS 2019, 17 September, 2019, M¨ unchen 16

Conclusions

• robotics has a lot of context, hence higher-order modelling is a

must. → surprisingly few results in that direction. . .

• modelling remains an art.

querying and solving queries even more so!

• especially for the higher-order models,
• connected to system-wide dependencies.

→ those are where the money is! → business models feasible, even on top of fully open source models!

The Role of Higher-order Models in Robotics and its Reasoning Challenges

• H. Bruyninckx, RobMoSys project, KU Leuven – TU Eindhoven

MODELS 2019, 17 September, 2019, M¨ unchen 17

Thank you for your attention

The Role of Higher-order Models in Robotics and its Reasoning Challenges

• H. Bruyninckx, RobMoSys project, KU Leuven – TU Eindhoven

MODELS 2019, 17 September, 2019, M¨ unchen 18