Object schemas for grounding language in a responsive robot

• Main Authors: Vosoughi, Soroush (Contributor), Kubat, Rony Daniel (Contributor), Roy, Deb K. (Contributor), Hsiao, Kai-yuh 1977- (Author), Tellex, Stefanie 1980- (Author)
• Other Authors: Massachusetts Institute of Technology. Media Laboratory (Contributor), Program in Media Arts and Sciences (Massachusetts Institute of Technology) (Contributor), Hsiao, Kai-yuh (Contributor), Tellex, Stefanie (Contributor)
• Format: Article
• Language: English
• Published: 2014-05-14T16:00:29Z.
• Subjects: Article
• Online Access: Get fulltext

An approach is introduced for physically grounded natural language interpretation by robots that reacts appropriately to unanticipated physical changes in the environment and dynamically assimilates new information pertinent to ongoing tasks. At the core of the approach is a model of object schemas that enables a robot to encode beliefs about physical objects in its environment using collections of coupled processes responsible for sensorimotor interaction. These interaction processes run concurrently in order to ensure responsiveness to the environment, while co-ordinating sensorimotor expectations, action planning and language use. The model has been implemented on a robot that manipulates objects on a tabletop in response to verbal input. The implementation responds to verbal requests such as 'Group the green block and the red apple', while adapting in real time to unexpected physical collisions and taking opportunistic advantage of any new information it may receive through perceptual and linguistic channels.
National Science Foundation (U.S.) (NSF Graduate Research Fellowship)