Overview Esta es una naranja atrac1va: Adventures in Adap1ng an - - PDF document

SLIDE 1

4/22/19 1 Esta es una naranja atrac1va: Adventures in Adap1ng an English Language Grounding System to Non-English Data

By: Caroline Kery CommiGee:

• Dr. Cynthia Matuszek
• Dr. Frank Ferraro
• Dr. Timothy Oates

1

Overview

• Research Goal
• Introduc1on to Grounded Language
• Prior Works
• Methods
• Results
• Conclusion and Future Work

2

Research Goal

Take a grounded language acquisi1on system and adapt it to non-English data

3

What is Grounded Language Acquisi1on?

• Tying language to the real world

What does “cat” mean?

hGp://www.petmd.com/sites/default/files/what-does-it-mean-when-cat-wags-tail.jpg hGps://images.immediate.co.uk/vola1le/sites/4/2018/08/iStock_000044061370_Medium-fa5f8aa.jpg? quality=45&crop=5px,17px,929px,400px&resize=960,413 hGp://www.royalcanin.ca/~/media/Royal-Canin-Canada/Product-Categories/cat-adult-landing-hero.ashx hGps://www.akc.org/wp-content/themes/akc/component-library/assets/img/welcome.jpg

4

Why is it important?

• Robots can learn from users
• Adaptable to new situa1ons

5

The English-centric Problem

• A common problem in Natural Language

Processing (NLP), systems are oien designed with English in mind

• Lots of materials available for English systems,

not as much for others

6

SLIDE 2

4/22/19 2 The English-Centric Problem

• Robo$c assistants should be accessible to

non-English-speakers!

7

Related works

• Grounded language

– Grounding ac1ons (e.g. Kollar et al.), direc1ons (e.g. Matuszek et al. 2012), some1mes mul1lingual (e.g. Chen et al. 2010)

• Computer Vision

– Object recogni1on (e.g. Bo et al. 2011), image cap1oning (e.g. Gella et

• al. 2017)
• Mul1lingual Natural Language Processing

– Machine transla1on (e.g. Wu et al. 2016), system adapta1ons (e.g. Poesio et al. 2010)

8

My Research Goal

Take a grounded language acquisi1on system and adapt it to non-English data

9

The Grounded Language System

(Pillai et al. RSS 2016)

10

Methods

• Analysis with Spanish and Hindi

Map from the Washington Post Website: hGps://www.washingtonpost.com/ pbox.php?url=hGp:// www.washingtonpost.com/blogs/ worldviews/files/2015/04/Screen- Shot-2015-04-23-at-9.04.22- AM.png&w=1484&op=resize&opt=1&filter=a n1alias&t=20170517

A Romance Language An Indo-Iranian language

11

Methods

• Analysis with Spanish and Hindi

– Started with Google Translate data

• Iden1fied adapta1ons (primarily preprocessing)

– Collected new crowd-sourced descrip1ons

• Analyzed differences across languages with real data

12

SLIDE 3

4/22/19 3 Google Translated Data

• Checked transla1on accuracy: back-transla1on

13

Google Translated Data

• Overall scores comparable

14

Adjec1ve-Noun Agreement

Spanish Hindi

15

Necessary modifica1on: Stemming

• Lemma1zer:

baked -> bake baking -> bake runs -> run

• (Simple) Stemmer:

running -> run baked -> bak baking -> bak runs -> run running -> runn

Lemma$zers are hard to find outside of English

16

Impact of Stemming on GT Data

17

Real Data Collec1on

• Google translate data is an approxima1on
• Doesn’t necessarily reflect real language data

18

SLIDE 4

4/22/19 4 Real Data Collec1on: Amazon Mechanical Turk

• “Give 1 to 2

sentences describing the

• bject”
• No sample

descrip1ons.

19

Data Collec1on: Results

• Around 6,000 descrip1ons were collected for each language

20

Data Collec1on: Results

• Final counts for Spanish and Hindi were smaller due

to problema1c workers

21

Results: lots of overlap but also some variety!

22

Overall Scores

23

Analysis: Some proper1es that could impact scores

• Token count
• Stop words
• Nega1ve/Posi1ve Examples

24

SLIDE 5

4/22/19 5 Token Count

• More tokens used in more specific contexts can raise the
• verall scores

Scores when problema1c workers who used lots of unrelated terms were not removed from the Hindi dataset

25

Stop words

• Generic and low IDF (Inverse Document Frequency)

Both Low IDF stop word only General stop word only

25 26

Stop words

Both Low IDF stop word only General stop word only

27

Stop words: Scores

28

Par1cular Tokens and posi1ve/nega1ve Examples

English stemmed Count F1 Score Spanish stemmed Count F1 Score cabbag 237 0.9297 col 28 0.8352 cabbag

• repoll

113 0.8294 29

Par1cular Tokens and posi1ve/nega1ve Examples

English stemmed Count F1 Score Spanish stemmed Count F1 Score yellow 562 0.8449 amarill 648 0.933 30

SLIDE 6

4/22/19 6 Bringing it all together

• Can this system be adapted to other

languages?

– Yes! For languages similar to Hindi and Spanish, the grounded language system works with minimal adapta1ons

31

Future Work

• Lots!
• More complex preprocessing system

– Spelling correc1on, en1ty recogni1on – Introduces addi1onal possible issues with languages

• More complex model

– Logis1c regression might not be the best one – Neural nets

• Larger and more complex dataset

– In the works. Would mi1gate many of the sensi1vity problems.

32

Special thanks to my advisors Dr. Matuszek and Dr. Ferraro for your guidance as well as Dr. Oates for serving on my commiGee. I also thank Nisha Pillai for developing the original grounded language system, and Rishabh Sachdeva for his help with the Hindi analysis. Finally, thank you to all of my family and friends for your love and support!

33

Appendix: References

• [1] Joost Broekens, Marcel Heerink, Henk Rosendal, et al. Assis1ve social

robots in elderly care: a review. Gerontechnology, 8(2):94–103, 2009.

• [2] Cynthia Matuszek, Nicholas FitzGerald, Evan Herbst, Dieter Fox, and Luke
• ZeGlemoyer. Interac1ve learning and its role in pervasive robo1cs. In ICRA

Workshop on The Future of HRI, St. Paul, MN, 2012.

• [3] Raymond Mooney. Learning to connect language and percep1on. In

Proceedings of the 23rd AAAI Conference on Ar1ficial Intelligence (AAAI), pages 1598–1601, Chicago, IL, 2008.

• [4] United States Census Bureau, US Department of Commerce. American

community survey, 2017. Data collected from 2012-2016.

• [5] David Chen, Joohyun Kim, and Raymond Mooney. Training a mul1lingual

sportscaster: Using perceptual context to learn language. J.Ar1f.Intell.Res. (JAIR), 37:397–435, 01 2010.

• [6] Muhannad Alomari, Paul Duckworth, David C Hogg, and Anthony G Cohn.

Natural language acquisi1on and grounding for embodied robo1c systems. In Thirty-First AAAI Conference on Ar1ficial Intelligence (AAAI-17), 2017.

• [7] Cynthia Matuszek, Nicholas FitzGerald, Luke ZeGlemoyer, Liefeng Bo, and

Dieter Fox. A joint model of language and percep1on for grounded aGribute

• learning. In

Proceedingsoihe2012Interna1onalConferenceonMachineLearning,Edinburg h, Scotland, 2012.

• [8] Nisha Pillai and Cynthia Matuszek. Unsupervised selec1on of nega1ve

examples for grounded language learning. In Proceedings of the 32nd Na1onal Conference on Ar1ficial Intelligence (AAAI), New Orleans, USA, 2018.

• [9] Ranjay Krishna, Yuke Zhu, Oliver Groth, Jus1n Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen,

Yannis Kalan1dis, Li-Jia Li, David A Shamma, et al. Visual genome: Connec1ng language and vision using crowdsourced dense image annota1ons. Interna1onal Journal of Computer Vision, 123:32–73,

• 2017. 46
• [10] Haoyuan Gao, Junhua Mao, Jie Zhou, Zhiheng Huang, Lei Wang, and Wei Xu. Are you talking to a

machine? dataset and methods for mul1lingual image ques1on. In Advances in neural informa1on processing systems, pages 2296–2304, 2015.

• [11] Thomas Kollar, Stefanie Tellex, Deb Roy, and Nick Roy. Grounding verbs of mo1on in natural

language commands to robots. In Experimental Robo1cs. Springer Tracts in Advanced Robo1cs, Springer, Berlin, Heidelberg, 2014.

• [12] Chen Yu and Dana H. Ballard. A mul1modal learning interface for grounding spoken language in

sensory percep1ons. In ACM Transac1ons on Applied Percep1on, pages 57–80, 2004.

• [13] Jake Brawer, Olivier Mangin, Alessandro Roncone, Sarah Widder, and Brian Scassella1. Situated

human–robot collabora1on: predic1ng intent from grounded natural language. In2018IEEE/RSJ Interna1onal Conference on Intelligent Robots and Systems (IROS), pages 827–833, 2018.

• [14] Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey,

Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. Google’s neural machine transla1on system: Bridging the gap between human and machine transla1on. In CoRR, 2016.

• [15] Jawharah Alasmari, J Watson, and ES Atwell. A compara1ve analysis between arabic and english
• f the verbal system using google translate. In Proceedings of

IMAN’20164thInterna1onalConferenceonIslamicApplica1onsinComputerScience and Technologies, Khartoum, Sudan, 2016.

• [16] Ekta Gupta and Shailendra Shrivastava. Analysis on transla1on quality of english to hindi online

transla1on systems- a review. In Interna1onal Journal of Computer Applica1ons, 2016.

• [17] Hadis Ghasemi and Mahmood Hashemian. A compara1ve study of” google translate”

transla1ons: An error analysis of english-to-persian and persian-to-english transla1ons. English Language Teaching, 9:13–17, 2016.

• [18] Joachim Daiber, Max Jakob, Chris Hokamp, and Pablo N Mendes. Improving efficiency and

accuracy in mul1lingual en1ty extrac1on. In Proceedings of the 9th Interna1onal Conference on Seman1c Systems, pages 121–124. ACM, 2013.

• [19] Michael Gamon, Carmen Lozano, Jessie Pinkham, and Tom ReuGer. Prac1cal experience with

grammar sharing in mul1lingual nlp. From Research to Commercial Applica1ons: Making NLP Work in Prac1ce, 1997.

Appendix: References

• [20] Craig Macdonald, Vassilis Plachouras, Ben He, Chris1na Lioma, and Iadh Ounis.

University of glasgow at webclef 2005: Experiments in per-field normalisa1on and language specific stemming. In Workshop of the Cross-Language Evalua1on Forum for European Languages, pages 898–907, 2005.

• [21] Massimo Poesio, Olga Uryupina, and Yannick Versley. Crea1ng a coreference resolu1on

system for italian. In Interna1onal Conference on Language Resources and Evalua1on, ValleGa, Malta, 2010.

• [22] Joonatas Wehrmann, Willian Becker, Henry EL Cagnini, and Rodrigo C Barros.

Acharacter-basedconvolu1onalneuralnetworkforlanguage-agnos1ctwiGersen1mentanalysis. In2017Interna1onalJointConferenceonNeuralNetworks(IJCNN), pages 2384–2391. IEEE, 2017.

• [23] Emily M Bender. Linguis1cally na¨ıve!= language independent: why nlp needs linguis1c
• typology. In Proceedings of the EACL 2009 Workshop on the Interac1on between Linguis1cs

and Computa1onal Linguis1cs: Virtuous, Viciousor Vacuous?, pages 26–32, 2009.

• [24] Joseph Le Roux, Benoit Sagot, and Djam´e Seddah. Sta1s1cal parsing of spanish and

data driven lemma1za1on. In ACL 2012 Joint Workshop on Sta1s1cal Parsing and Seman1c Processing of Morphologically Rich Languages(SP-Sem-MRL2012), pages 6–pages, 2012.

• [25] Ferran Pla and Llu´ıs-F Hurtado. Poli1cal tendency iden1fica1on in twiGer using

sen1ment analysis techniques. In Proceedings of COLING 2014, the 25th interna1onal conference on computa1onal linguis1cs: Technical Papers, pages 183–192, 2014.

• [26] Nisha Pillai, Francis Ferraro, and Cynthia Matuszek. Op1mal seman1c distance for

nega1ve example selec1on in grounded language acquisi1on. Robo1cs: Science and Systems Workshop on Models and Representa1ons for Natural Human-Robot Communica1on, 2018.

• [27] Liefeng Bo, Kevin Lai, Xiaofeng Ren, and Dieter Fox. Object recogni1on with hierarchical

kernel descriptors. In Computer Vision and PaGern Recogni1on, 2011.

• [28] Kevin Lai, Liefeng Bo, Xiaofeng Ren, and Dieter Fox. Rgb-d object recogni1on: Features,

algorithms, and a large scale benchmark. In Consumer Depth Cameras for Computer Vision: Research Topics and Applica1ons, pages 167–192, 2013.

• [29] Mar1nF.Porter. Snowball: A language for stemming algorithms. Retrieved

March, 1, 01 2001.

• [30] Ananthakrishnan Ramanathan and Durgesh D Rao. A lightweight stemmer

for hindi. In the Proceedings of EACL, 2003.

• [31] India: Office of the Registrar General & Census Commissioner. Compara1ve

speakers’ strength of scheduled languages -1971, 1981, 1991 and 2001, 2015. Archived 2007-11-30.

• [32] Shilpi Srivastava, Mukund Sanglikar, and DC Kothari. Named en1ty

recogni1on system for hindi language: a hybrid approach. Interna1onal Journal

• f Computa1onal Linguis1cs (IJCL), 2(1):10–23, 2011.
• [33] Chetana Thaokar and Latesh Malik. Test model for summarizing hindi text

using extrac1on method. In 2013 IEEE Conference on Informa1on & Communica1on Technologies, pages 1138–1143. IEEE, 2013.

• [34] Vishal Gupta. Hybrid algorithm for mul1lingual summariza1on of hindi and

punjabi documents. In Mining Intelligence and Knowledge Explora1on, pages 717–727. Springer, 2013.

• [35] Manjula Subramaniam and Vipul Dalal. Test model for rich seman1c graph

representa1on for hindi text using abstrac1ve method. Interna1onal Research Journal of Engineering and Technology (IRJET), 2(2), 2015.

• [36] Sifatullah Siddiqi and Adi1 Sharan. Construc1on of a generic stopwords list

for hindi language without corpus sta1s1cs. Interna1onal Journal of Advanced Computer Research, 8(34):35–40, 2018.

• [37] Spandana Gella, Rico Sennrich, Frank Keller, and Mirella Lapata. Image

pivo1ng for learning mul1lingual mul1modal representa1ons. arXiv preprint arXiv:1707.07601, 2017.