Using evidence accumulation to bridge the gap between neural - - PowerPoint PPT Presentation

▶
using evidence accumulation to bridge the gap between
SMART_READER_LITE
LIVE PREVIEW

Using evidence accumulation to bridge the gap between neural - - PowerPoint PPT Presentation

Dec 30, 2023 280 likes •676 views

Using evidence accumulation to bridge the gap between neural networks and symbolic cognitive control Modeling Associative Recognition with large-scale Neural Networks Jelmer Borst & Terry Stewart Theories of Associative Recognition

slide-1
SLIDE 1

Using evidence accumulation to bridge the gap between neural networks and symbolic cognitive control

Jelmer Borst & Terry Stewart

Modeling Associative Recognition with large-scale Neural Networks

slide-2
SLIDE 2
slide-3
SLIDE 3
slide-4
SLIDE 4
slide-5
SLIDE 5

Theories of Associative Recognition

Global Matching Dual-process ACT-R

Encoding Matching Response Encoding Response Familiarity Recollection Encoding Response Associative retrieval Decision

(e.g., Anderson, 2007; Anderson & Reder, 1999; Schneider & Anderson, 2012) (e.g., Gillund & Shiffrin, 1984; Hintzman, 1988; Murdock, 1993; Wixted & Stretch, 2004) (e.g., Diana et al., 2006; Malmberg, 2008; Rugg & Curran, 2007; Yonelinas, 2002)

slide-6
SLIDE 6

The Magnetic Fan Experiment

Borst, Ghuman, & Anderson, NeuroImage, 2016

slide-7
SLIDE 7
slide-8
SLIDE 8

−1000 −800 −600 −400 −200 200 −3 −2 −1 1 2 3 4 5 6 x 10

−13

MEG1143

Time (ms) Activity Left Right

Sensors and sources

Sensor space Source space

slide-9
SLIDE 9

Sensors and sources

Source space

Data - Response Generation _lh

Time (ms) Estimated current (x 10^-11 Am)

  • 800
  • 700
  • 600
  • 500
  • 400
  • 300
  • 200
  • 100

0.0 0.5 1.0 1.5 2.0 2.5

left right

Data - Response Generation _rh

Time (ms) Estimated current (x 10^-11 Am)

  • 800
  • 700
  • 600
  • 500
  • 400
  • 300
  • 200
  • 100

0.0 1.0 2.0

left right

slide-10
SLIDE 10

Associative Recognition Task

COMFORT – MUSTARD FLAME – CAPE METAL – SPARK EXCHANGE – HARVEST JELLY – MOTOR DUNGEON – GODDESS DRUNKARD – HARVEST CAPE – DECK Study Phase Test Phase COMFORT – MUSTARD FLAME – DECK BERRY – CREAM DRUNKARD - HARVEST METAL – MOTOR EXCHANGE – HARVEST FINANCE – TOURIST JELLY – MOTOR …

slide-11
SLIDE 11

Associative Recognition Task

COMFORT – MUSTARD FLAME – CAPE METAL – SPARK EXCHANGE – HARVEST JELLY – MOTOR DUNGEON – GODDESS DRUNKARD – HARVEST CAPE – DECK Study Phase Test Phase COMFORT – MUSTARD FLAME – DECK BERRY – CREAM DRUNKARD - HARVEST METAL – MOTOR EXCHANGE – HARVEST FINANCE – TOURIST JELLY – MOTOR …

Target vs Re-paired Foil vs New Foil

slide-12
SLIDE 12

Associative Recognition Task

COMFORT – MUSTARD FLAME – CAPE METAL – SPARK EXCHANGE – HARVEST JELLY – MOTOR DUNGEON – GODDESS DRUNKARD – HARVEST CAPE – DECK Study Phase Test Phase COMFORT – MUSTARD FLAME – DECK BERRY – CREAM DRUNKARD - HARVEST METAL – MOTOR EXCHANGE – HARVEST FINANCE – TOURIST JELLY – MOTOR …

short vs long words

slide-13
SLIDE 13

Associative Recognition Task

COMFORT – MUSTARD FLAME – CAPE METAL – SPARK EXCHANGE – HARVEST JELLY – MOTOR DUNGEON – GODDESS DRUNKARD – HARVEST CAPE – DECK Study Phase Test Phase COMFORT – MUSTARD FLAME – DECK BERRY – CREAM DRUNKARD - HARVEST METAL – MOTOR EXCHANGE – HARVEST FINANCE – TOURIST JELLY – MOTOR …

associative fan of 1 or 2

slide-14
SLIDE 14

Test Phase

Fixation

COMFORT MUSTARD

Probe

+

Feedback

Correct

ITI 500 ms (jitter) Until Response 1000 ms 500 ms Time

slide-15
SLIDE 15

Behavior

Target RP Foil New Foil

Response Time EEG

RT (ms) 500 1000 1500 2000

Fan 1 / Short Fan 1 / Long Fan 2 / Short Fan 2 / Long

slide-16
SLIDE 16

100 200 300 400 500 600 700 800 900 1000 1100 1200 1300 Stimulus

  • nset

Time (ms)

Response Fan Word length Indicated by: Probe Response hand

MEG Results

Borst, Ghuman, & Anderson, NeuroImage, 2016

slide-17
SLIDE 17

100 200 300 400 500 600 700 800 900 1000 1100 1200 1300 Stimulus

  • nset

Time (ms)

Response

Visual Encoding

Fan Word length Indicated by:

Lexical and Semantic Access

Probe Response hand

MEG Results

Borst, Ghuman, & Anderson, NeuroImage, 2016

slide-18
SLIDE 18

100 200 300 400 500 600 700 800 900 1000 1100 1200 1300 Stimulus

  • nset

Time (ms)

Response

Visual Encoding

Fan Word length Indicated by:

Lexical and Semantic Access Familiarity Recollection Representation

Probe Response hand

MEG Results

Borst, Ghuman, & Anderson, NeuroImage, 2016

slide-19
SLIDE 19

100 200 300 400 500 600 700 800 900 1000 1100 1200 1300 Stimulus

  • nset

Time (ms)

Response

Visual Encoding

Fan Word length Indicated by:

Lexical and Semantic Access Familiarity Recollection Representation

Probe Response hand

Decision

MEG Results

Borst, Ghuman, & Anderson, NeuroImage, 2016

slide-20
SLIDE 20

MEG Results

100 200 300 400 500 600 700 800 900 1000 1100 1200 1300 Stimulus

  • nset

Time (ms)

Response

Visual Encoding

Fan Word length Indicated by:

Lexical and Semantic Access Familiarity Recollection Representation

Probe Response hand

Decision Response

Borst, Ghuman, & Anderson, NeuroImage, 2016

slide-21
SLIDE 21

Large-scale neural networks: Nengo

slide-22
SLIDE 22

Nengo

Terry Stewart

slide-23
SLIDE 23

Neural ensembles represent values Ensembles can encode multiple dimensions Represents symbols as multidimensional vectors Basal Ganglia coordinate cognition

Nengo

But, now we have to deal with dynamics…

slide-24
SLIDE 24

Visual

Occipital Cortex

Semantic

Dorsal Temporal

Visual Scratchpad

Declarative Memory

Representation

Prefrontal Cortex

Cognitive Control

Anterior Cingulate?

Motor

Precentral

Decision

Posterior Parietal

1 2

So, what about the dynamics?

Thalamus Basal Ganglia

slide-25
SLIDE 25

New Foil

slide-26
SLIDE 26

Target

slide-27
SLIDE 27

Re-paired Foil

slide-28
SLIDE 28

Behavior

Target RP Foil New Foil

Response Time EEG

RT (ms) 500 1000 1500 2000 Target RP Foil New Foil

Response Time Model

RT (ms) 500 1000 1500 2000

Fan 1 / Short Fan 1 / Long Fan 2 / Short Fan 2 / Long

slide-29
SLIDE 29

Visual

Occipital Cortex

Semantic

Dorsal Temporal

Visual Scratchpad

Declarative Memory

Representation

Prefrontal Cortex

Cognitive Control

Anterior Cingulate?

Motor

Precentral

Decision

Posterior Parietal Thalamus Basal Ganglia

pixels

alien

low-dimensional representation

Gabor Filters as Tuning Curves

Visual

Data - Visual Encoding

Time (ms) Estimated current (x 10^-11 Am) 100 200 300 400 500 600 700 800 1 2 3 4 5 6

short long

Model - Visual Encoding

Time (ms) Estimated current 100 200 300 400 500 600 700 800 0e+00 4e+04 8e+04

Length Short Length Long

slide-30
SLIDE 30

Visual

Occipital Cortex

Semantic

Dorsal Temporal

Visual Scratchpad

Declarative Memory

Representation

Prefrontal Cortex

Cognitive Control

Anterior Cingulate?

Motor

Precentral

Decision

Posterior Parietal Thalamus Basal Ganglia

Accumulator of Summed Similarity

alien?

Familiarity

Data - Familiarity _lh

Time (ms) Estimated current (x 10^-11 Am) 100 200 300 400 500 600 700 800 1 2 3 4

target foil Familiarity Recollection

Model - Familiarity

Time (ms) Estimated current 100 200 300 400 500 600 700 800 0.0 0.5 1.0 1.5 2.0

PairType Target PairType RPFoil PairType NewFoil

slide-31
SLIDE 31

Visual

Occipital Cortex

Semantic

Dorsal Temporal

Visual Scratchpad

Declarative Memory

Representation

Prefrontal Cortex

Cognitive Control

Anterior Cingulate?

Motor

Precentral

Decision

Posterior Parietal Thalamus Basal Ganglia

Recollection

Data - Recollection _lh

Time (ms) Estimated current (x 10^-11 Am) 100 200 300 400 500 600 700 800 0.0 0.5 1.0 1.5 2.0

fan1 fan2 Familiarity Recollection

Model - Recollection

Time (ms) Estimated current 100 200 300 400 500 600 700 800 0.0 0.2 0.4 0.6 0.8

Fan 1 Fan 2

slide-32
SLIDE 32

Visual

Occipital Cortex

Semantic

Dorsal Temporal

Visual Scratchpad

Declarative Memory

Representation

Prefrontal Cortex

Cognitive Control

Anterior Cingulate?

Motor

Precentral

Decision

Posterior Parietal Thalamus Basal Ganglia

Representation

Data - Representation _lh

Time (ms) Estimated current (x 10^-11 Am)

  • 800
  • 700
  • 600
  • 500
  • 400
  • 300
  • 200
  • 100

0.0 0.4 0.8 1.2

fan1 fan2 Recollection Representation

Model - Representation

Time (ms) Estimated current

  • 800
  • 700
  • 600
  • 500
  • 400
  • 300
  • 200
  • 100

0e+00 4e+05 8e+05

Fan 1 Fan 2

slide-33
SLIDE 33

Visual

Occipital Cortex

Semantic

Dorsal Temporal

Visual Scratchpad

Declarative Memory

Representation

Prefrontal Cortex

Cognitive Control

Anterior Cingulate?

Motor

Precentral

Decision

Posterior Parietal Thalamus Basal Ganglia

Motor

Data - Response Generation _lh

Time (ms) Estimated current (x 10^-11 Am)

  • 800
  • 700
  • 600
  • 500
  • 400
  • 300
  • 200
  • 100

0.0 0.5 1.0 1.5 2.0 2.5

left right

Data - Response Generation _rh

Time (ms) Estimated current (x 10^-11 Am)

  • 800
  • 700
  • 600
  • 500
  • 400
  • 300
  • 200
  • 100

0.0 1.0 2.0

left right

Model - Left Motor

Time (ms) Estimated current

  • 800
  • 700
  • 600
  • 500
  • 400
  • 300
  • 200
  • 100

0.0 0.2 0.4 0.6 0.8

Hand Left Hand Right

Model - Right Motor

Time (ms) Estimated current

  • 800
  • 700
  • 600
  • 500
  • 400
  • 300
  • 200
  • 100

0.0 0.2 0.4 0.6 0.8

Hand Left Hand Right

slide-34
SLIDE 34

So, what about the dynamics?

No production rules that are on or off No memory retrieval that is done

slide-35
SLIDE 35

Symbolic Model

Anderson, Zhang, Borst, & Walsh, Psychological Review, 2016 Production Visual Retrieval Problem State Manual Encoding Associative Retrieval Decide & Respond Familiarity

ACT-R Model

Borst, Schneider, Walsh, & Anderson, JOCN, 2013 Borst & Anderson, NeuroImage, 2015 Zhang, Walsh, & Anderson, JOCN, 2017

slide-36
SLIDE 36

Accumulators as a solution?

slide-37
SLIDE 37

Discussion

Visual

Occipital Cortex

Semantic

Dorsal Temporal Visual Scratchpad

Declarative Memory

Representation Prefrontal Cortex Cognitive Control Anterior Cingulate?

Motor

Precentral

Decision

Posterior Parietal Thalamus Basal Ganglia

  • Do we need on/off symbolic production rules?

Or can we do goal-directed control with dynamics?

  • What is missing in symbolic architectures

like ACT-R/PRIMs?

– Low level visual effects – Dynamics to account for continuous neural data

  • What is missing in neural networks

as cognitive architecture?

– A semi-fixed architecture – Controlled cognition

slide-38
SLIDE 38

Using evidence accumulation to bridge the gap between neural networks and symbolic cognitive control

Jelmer Borst & Terry Stewart www.jelmerborst.nl Modeling Associative Recognition with large-scale Neural Networks