Programming by Demonstration with Situated Semantic Parsing Yoav - PowerPoint PPT Presentation

Programming by Demonstration with Situated Semantic Parsing Yoav Artzi, Maxwell Forbes, Kenton Lee, Maya Cakmak University of Washington

Programming by Demonstration • Teaching the robot new skills • Kinesthetic demonstrations • Natural language interface [Cakmak and Takayama 2014]

Semantic Parsing at the chair, move forward three steps past the sofa λ a.pre ( a, ι x.chair ( x )) ∧ move ( a ) ∧ len ( a, 3) ∧ dir ( a, forward ) ∧ past ( a, ι y.sofa ( y ))

Situated Semantic Parsing “close left hand” λ a.close ( a ) ∧ patient ( a, A ( λ x.hand ( x ) ∧ left ( x ))) close-left-hand

Situated Semantic Parsing “close” λ a.close ( a ) close-right-arm ? close-right-hand close-action close-left-hand …

Situated Semantic Parsing Use the system state to disambiguate “close” λ a.close ( a ) close-right-arm close-right-hand close-action close-left-hand …

Situated Semantic Parsing Use the system state to disambiguate “close” • The close action can only be applied to hand objects λ a.close ( a ) close-right-arm close-right-hand close-action close-left-hand …

Situated Semantic Parsing Use the system state to disambiguate “close” • The close action can only be applied to hand objects λ a.close ( a ) • Infer salient objects from recent actions close-right-arm close-right-hand close-action close-left-hand …

Situated Semantic Parsing Bi-modal interaction “close” Verbal Physical Instruction Manipulation λ a.close ( a ) Robot command close-right-arm close-right-hand close-action close-left-hand …