SLIDE 1
Modeling Conceptual Understanding in Image Reference Games
Rodolfo Corona*
Stephan Alaniz* Zeynep Akata
*Equal contribution
Modeling Conceptual Understanding in Image Reference Games Rodolfo - - PowerPoint PPT Presentation
Modeling Conceptual Understanding in Image Reference Games Rodolfo - - PowerPoint PPT Presentation
Modeling Conceptual Understanding in Image Reference Games Rodolfo Corona * Stephan Alaniz * Zeynep Akata *Equal contribution Yellow Yellow Yellow Spotted Listener Clusters Misunderstands Misunderstands Colors Shapes 7 Speaker
SLIDE 2
SLIDE 3
SLIDE 4
Yellow
SLIDE 5
Yellow
SLIDE 6
Yellow Spotted
SLIDE 7
7
Listener Clusters
Misunderstands Shapes Misunderstands Colors
SLIDE 8
8
Speaker Listener
SLIDE 9
9 Red beak Cone beak Yellow feet Red beak Cone beak Yellow feet Red beak Cone beak Red beak Yellow feet Cone beak Yellow feet
Predicting attributes
Speaker Listener
SLIDE 10
10
- Cone beak
Yellow feet Red beak
Selecting one attribute
Red beak Cone beak Yellow feet Red beak Cone beak Yellow feet Red beak Cone beak Red beak Yellow feet Cone beak Yellow feet
Speaker Listener
= Cone beak
SLIDE 11
11 Red beak Yellow feet Cone beak
Listener guesses
- Cone beak
Yellow feet Red beak Red beak Cone beak Yellow feet Red beak Cone beak Yellow feet Red beak Cone beak Red beak Yellow feet Cone beak Yellow feet
Speaker
= Cone beak
Listener
SLIDE 12
12
Reward
“It’s image ”
+1 -1
+1 match, -1 mismatch
Red beak Yellow feet Cone beak
- Cone beak
Yellow feet Red beak Red beak Cone beak Yellow feet Red beak Cone beak Yellow feet Red beak Cone beak Red beak Yellow feet Cone beak Yellow feet
Speaker
= Cone beak
Listener
SLIDE 13
13
Agent Embedding
Speaker incorporates game into history
Reward
“It’s image ”
+1 -1
Red beak Yellow feet Cone beak
- Cone beak
Yellow feet Red beak Red beak Cone beak Yellow feet Red beak Cone beak Yellow feet Red beak Cone beak Red beak Yellow feet Cone beak Yellow feet
Speaker
= Cone beak
Listener
SLIDE 14
Parameterized attribute selection policies incentivised to adapt faster to listeners.
CUB
- Avg. Eval Reward
Number of Training Games
SLIDE 15
CUB
- Avg. Eval Reward
Number of Training Games
Parameterized attribute selection policies incentivised to adapt faster to listeners.
SLIDE 16
CUB
- Avg. Eval Reward
Number of Training Games
Parameterized attribute selection policies incentivised to adapt faster to listeners.
SLIDE 17
CUB
- Avg. Eval Reward
Number of Training Games
Parameterized attribute selection policies incentivised to adapt faster to listeners.
SLIDE 18
CUB
Modeling listener crucial for maximizing game performance.
- Avg. Eval Reward
Number of Training Games
SLIDE 19
CUB VI
Parameterized policies embed agents with greater correspondence to ground truth clusters.
Number of Training Games
SLIDE 20
Discrim. Chosen
Game 1
Brown back Brown back Blue underparts Blue underparts Rufous belly Rufous belly Yellow wing Yellow wing
SLIDE 21
Discrim. Chosen
Game 1
Brown back Brown back Blue underparts Blue underparts Rufous belly Rufous belly Yellow wing Yellow wing Yellow belly Solid belly pattern Discrim. Chosen
Game 10
Orange leg Spotted belly pattern Yellow belly Spotted back pattern Rufous crown Rufous crown
SLIDE 22
Discrim. Chosen
Game 1
Brown back Brown back Blue underparts Blue underparts Rufous belly Rufous belly Yellow wing Yellow wing Yellow belly Solid belly pattern Discrim. Chosen
Game 10
Orange leg Spotted belly pattern Yellow belly Spotted back pattern Rufous crown Rufous crown White belly Forked tail shape Yellow belly Has eyebrow Orange beak Duck-like shape Yellow wing Solid belly pattern Discrim. Chosen
Game 100
SLIDE 23
Modeling Conceptual Understanding in Image Reference Games
*Equal contribution