Overview of neuro-symbolic processing in Neural Blackboard - - PowerPoint PPT Presentation

Overview of neuro-symbolic processing in Neural Blackboard Architectures

Frank van der Velde University of Twente, The Netherlands f.vandervelde@utwente.nl

Dagstuhl Seminar 17192: Human-Like Neural-Symbolic Computing 8-12 May 2017

Ø Combinatorial structures and processing in neuronal manner Ø Comparing with human behavior Ø Satisfying constraints on cognition and brain:

• Grounded, ‘in situ’ representations,

also in combinatorial structures

• Connection paths as basis for behaviour
• Content addressable combinatorial structure

Ø Modeling of neuronal activity (Marc de Kamps) Ø Incremental sentence processing Ø Competition in NBA as tool (e.g., ambiguity resolution) Ø Architecture for parallel computing

2

Aims of Neural Blackboard Architectures (NBAs)

• Grounded, ‘in situ’, develop over time
• Hebbian-like assemblies, with (long-term) relations

(combinations of distributed and more local representations)

Assumptions concept representation in brain

Neural circuits & populations (interacting exc and inh). u Associative connections u Conditional connections (labeled connections)

Content addressable

3 FvdV (2015). Neural Networks

Grounded representations: Not copied and pasted (e.g., like symbols) Without grounding: Central population has no meaning (no ‘neural code’).

Assumptions concept representation in brain

4 FvdV (2015). Neural Networks

• Grounded, ‘in situ’, develop over time
• Hebbian-like assemblies, with (long-term) relations

(combinations of distributed and more local representations)

FvdV (2015). Neural Networks

Combinatorial structures (e.g., short-term)

• Relations between concepts in specified ‘neural blackboards’.
• Interaction blackboards via ‘in situ’ concepts.

5

Role blackboard (e.g., sentence structure):

• Allow productive (novel) combinations
• Provide co

connect ctio ion path between sensory and motor activation, needed for behaviour

Neural blackboard architecture for sentence structures

6

Nx

Sy

Vz

cat

Nu

sees dog

/ca/ /t/ /do/ /g/

Phonology Blackboard Sentence Blackboard

FvdV (2015). Neural Networks

Cognition:

• ‘Delayed reflex’
• But not just reflexive behaviour
• Increased complexity in evolution
• But all behaviour depends on some connection path

FvdV (2015). Neural Networks

Importance connection paths

7

Feldman (2013, Cogn Neurodyn): ‘‘if I tell you that my granddaughter Sonnet is brilliant, you have a new person to consider as a possible filler for variable roles and also a number of new facts for use in inference.’’

Argument against Neural Blackboards with ‘fixed’ connection structures for novel bindings:

Importance connection paths

8

Not: If If ‘ ‘fixed’ connection structures for novel bindings. But: what kin kind?

Importance connection paths

Dedicated sentence representations: Content addressable. But: No novel bindings

9 FvdV & MdK (2015). Cognitive Neurodynamics

Not: If If ‘ ‘fixed’ connection structures for novel bindings. But: what kin kind?

Importance connection paths

Universal machine: Maximal novel binding. But: Not content addressable

10 FvdV & MdK (2015). Cognitive Neurodynamics

Language:

• Two tier structure (at least)
• Phonology neural blackboard: new words
• Sentence neural blackboards: novel sentences
• But with constraints: familiar language; based on development

Importance connection paths

‘Small-world’ like connection structure for binding:

• Content addressable
• Forms of novel binding

11 FvdV & MdK (2015). Cognitive Neurodynamics

N1 n i X di Conditional connections: Control circuit WM circuit Binding Syntax

12

NBA: structure neural blackboard

S1 n V1 v N2 t n v t

sees dog

Structure assemblies

N1

cat

S1 = Main assemblies v = Sub assemblies

Structure assemblies:

FvdV & MdK (2006). Behavioral and Brain Sciences

‘in situ’ word assemblies

N1 n i X di Conditional connections: Control circuit WM circuit Binding Syntax

13

NBA: structure neural blackboard

S1 n V1 v N2 t n v t

sees dog

Structure assemblies

cat

N1 S1 = Main assemblies v = Sub assemblies

Structure assemblies:

FvdV & MdK (2006). Behavioral and Brain Sciences

‘in situ’ word assemblies

S1 n V1 v N2 t n v t

sees dog

NBA: duplicating words, multiple sentences

Structure assemblies

cat

N1 S2 n N3 v n v V2

likes

N4 t t

bird

14

• Same ‘in situ’ word assembly
• Different structure assemblies

S1 = Main assemblies v = Sub assemblies

Structure assemblies:

FvdV & MdK (2006). Behavioral and Brain Sciences

S1 n V1 v N2 t n v t

sees dog

Structure assemblies

cat

N1 S2 n N3 v n v V2

likes

N4 t t

bird

15

S1 = Main assemblies v = Sub assemblies

Structure assemblies:

FvdV & MdK (2006). Behavioral and Brain Sciences

NBA: Content addressable

cat sees?

Illustrated with question:

• Structure of question
• Effect of question in blackboard
• Evolutionary pressure: fast (direct) and informative

S1 n V1 v N2 t n v t

sees dog

NBA: Content addressable

cat

N1 S2 n N3 v n v V2

likes

N4 t t

bird

FvdV & MdK (2006). Behavioral and Brain Sciences 16

cat sees?

Illustrated with question:

• Structure of question
• Effect of question in blackboard
• Evolutionary pressure: fast (direct) and informative

S1 = Main assemblies v = Sub assemblies

Structure assemblies:

• Based on binding competition in Connection Matrices (CMs)
• CM: matrix of ‘connection nodes’
• Connection node: gating circuits and WM
• WM: neuronal population with sustained activity

Connection Matrix (CM)

NBA: Binding

Connection Node

17

= inhibition: Internal CM competition (binding restriction)

=

N2-t V1-t

Yj Xi

i

Xin Yin

I

WM

i I

Good (optimal) Pilot simulation of development of connection matrix (CM):

• Initial random connections between assemblies and nodes in CM
• Process of hebbian and anti-hebbian learning
• Able to produce selective CMs

Not good: Confusion Good (not optimal)

NBA: Binding

18

Frankland & Greene, PNAS, 2015 NBA: A Connection Matrix for each specific binding:

• Agent ‘ field’
• Theme ‘field’
• Other ‘fields’

Binding in the brain?

19

NBA:

• Massively parallel architecture
• Extensive computation:

More complex processing with more (but similar) neural ‘hardware’ lmSTC: left mid-superior temporal cortex

NBA: Incremental sentence processing

FvdV & MdK (2010). Cognitive Systems Research

Interaction control circuits and blackboard activity

20

• UPA1a. Bill knows John.
• UPA1b. Bill knows John likes fish.

NBA: Ambiguity resolution

21

Lewis (1993) An Architecturally-based Theory of Human Sentence Comprehension Ø A collection of (31) unproblematic ambiguities (UPA) Ø A collection of (26) garden path (GP) constructions

V1-t N2-t V1-c C1-c

NBA: Ambiguity resolution

= inhibition: Between CM competition (Constraints on binding)

FvdV & MdK (2010). Cognitive Systems Research

Connection Matrix Connection Matrix

22

• UPA1a. Bill knows John.
• UPA1b. Bill knows John likes fish.

NBA: Ambiguity resolution

23

• UPA1a. Bill knows John.
• UPA1b. Bill knows John likes fish.

NBA: Ambiguity resolution

24

• UPA3a. Ron believes the linguistics professor.
• UPA3b. Ron believes the linguistics professor he had met

the week before in Prague disliked him.

25

NBA: Ambiguity resolution

• UPA3a. Ron believes the linguistics professor.
• UPA3b. Ron believes the linguistics professor he had met

the week before in Prague disliked him.

26

NBA: Ambiguity resolution

• UPA3a. Ron believes the linguistics professor.
• UPA3b. Ron believes the linguistics professor he had met

the week before in Prague disliked him.

27

NBA: Ambiguity resolution

• UPA3a. Ron believes the linguistics professor.
• UPA3b. Ron believes the linguistics professor he had met

the week before in Prague disliked him.

28

NBA: Ambiguity resolution

• UPA3a. Ron believes the linguistics professor.
• UPA3b. Ron believes the linguistics professor he had met

the week before in Prague disliked him.

29

NBA: Ambiguity resolution

• UPA3a. Ron believes the linguistics professor.
• UPA3b. Ron believes the linguistics professor he had met

the week before in Prague disliked him.

30

NBA: Ambiguity resolution

• UPA3a. Ron believes the linguistics professor.
• UPA3b. Ron believes the linguistics professor he had met

the week before in Prague disliked him.

31

NBA: Ambiguity resolution

• UPA3a. Ron believes the linguistics professor.
• UPA3b. Ron believes the linguistics professor he had met

the week before in Prague disliked him.

NBA: Ambiguity resolution

John believes Ron (who) believes the linguistics professor he had met the week before in Prague disliked him hated him. (Resembles Garden Path #2 in Lewis 1993)

• UPA3b. Ron believes the linguistics professor he had met

the week before in Prague disliked him.

32

NBA: Ambiguity resolution

John believes Ron (who) believes the linguistics professor he had met the week before in Prague disliked him hated him (Resembles Garden Path #2 in Lewis 1993)

33

• UPA3b. Ron believes the linguistics professor he had met

the week before in Prague disliked him.

NBA: Ambiguity resolution

34

NBA: Ambiguity resolution

35

NBA: Ambiguity resolution

36

NBA: Ambiguity resolution

37

NBA: Ambiguity resolution

38

NBA: neural blackboard for relations

Reaso sonin ing

39 FvdV (2016). Frontiers in Neurorobotics

• 1. Competition within and between blackboards: illustrations

(works as in language NBA, e.g. ambiguity resolution: CoCoNIPS workshop Montreal 2015)

• 2. Reservoir for selective sequential control of processing (basic simulations)

Conditional Connections:

feature space In sit situ co conce cepts s Phonology blackboard Relations blackboard

• ffice

milk room go get drop

(Other blackboards) Sequence blackboard Sentence blackboard

localizer location

A1 John A2 A3 A4 A5 O1 V1 V2 V3 V4 V5 O2 O3 O4 O5

kitchen

• ffice

milk room go get drop

location localizer

NBA: neural blackboard for relations

40 FvdV (2016). Frontiers in Neurorobotics

BaBi Reasoning tasks: John go kitchen John get milk John go office John drop milk John go room Where re is is Jo John? A: room Where re is is milk? milk? A: office

In sit situ co conce cepts s Ai = Agent Vi = Verb Oi = Object

A1 John A2 A3 A4 A5 O1 V1 V2 V3 V4 V5 O2 O3 O4 O5

kitchen

• ffice

milk room go get drop

location localizer

NBA: neural blackboard for relations

41 FvdV (2016). Frontiers in Neurorobotics

BaBi Reasoning tasks: John go kitchen John get milk John go office John drop milk John go room Where re is is Jo John? A: room Where re is is milk? milk? A: office

In sit situ co conce cepts s Ai = Agent Vi = Verb Oi = Object

A1 John A2 A3 A4 A5 O1 V1 V2 V3 V4 V5 O2 O3 O4 O5

kitchen

• ffice

milk room go get drop

location localizer

NBA: neural blackboard for relations

42 FvdV (2016). Frontiers in Neurorobotics

BaBi Reasoning tasks: John go kitchen John get milk John go office John drop milk John go room Where re is is Jo John? A: room Where re is is milk? milk? A: office

In sit situ co conce cepts s Ai = Agent Vi = Verb Oi = Object

A1 John A2 A3 A4 A5 O1 V1 V2 V3 V4 V5 O2 O3 O4 O5

kitchen

• ffice

milk room go get drop

location localizer

NBA: neural blackboard for relations

43 FvdV (2016). Frontiers in Neurorobotics

BaBi Reasoning tasks: John go kitchen John get milk John go office John drop milk John go room Where re is is Jo John? A: room Where re is is milk? milk? A: office

In sit situ co conce cepts s

Where re is is ? + + Ag Agent = = lo loca catio ion

Ai = Agent Vi = Verb Oi = Object

A1 John A2 A3 A4 A5 O1 V1 V2 V3 V4 V5 O2 O3 O4 O5

kitchen

• ffice

milk room go get drop

location localizer

NBA: neural blackboard for relations

44 FvdV (2016). Frontiers in Neurorobotics

BaBi Reasoning tasks: John go kitchen John get milk John go office John drop milk John go room Where re is is Jo John? A: room Where re is is milk? milk? A: office

In sit situ co conce cepts s Ai = Agent Vi = Verb Oi = Object

Where re is is ? + + Ag Agent = = lo loca catio ion

O1 O2 O3 O4 O5

kitchen

• ffice

milk room

location

NBA: neural blackboard for sequence order

45 FvdV (2016). Frontiers in Neurorobotics

BaBi Reasoning tasks: John go kitchen John get milk John go office John drop milk John go room Where re is is Jo John? A: room Where re is is milk? milk? A: office

In sit situ co conce cepts s

S1 S2 S3 S4 S5 A1 John A2 A3 A4 A5

Sequence blackboard

O1 O2 O3 O4 O5

kitchen

• ffice

milk room

location

NBA: neural blackboard for sequence order

46 FvdV (2016). Frontiers in Neurorobotics

BaBi Reasoning tasks: John go kitchen John get milk John go office John drop milk John go room Where re is is Jo John? A: room Where re is is milk? milk? A: office

In sit situ co conce cepts s

S1 S2 S3 S4 S5 A1 John A2 A3 A4 A5

Sequence blackboard

O1 O2 O3 O4 O5

kitchen

• ffice

milk room

location

NBA: neural blackboard for sequence order

47 FvdV (2016). Frontiers in Neurorobotics

BaBi Reasoning tasks: John go kitchen John get milk John go office John drop milk John go room Where re is is Jo John? A: room Where re is is milk? milk? A: office

In sit situ co conce cepts s

Sequence blackboard

S1 S2 S3 S4 S5 A1 John A2 A3 A4 A5

A1 John A2 A3 A4 A5 O1 V1 V2 V3 V4 V5 O2 O3 O4 O5

kitchen

• ffice

milk room go get drop

location localizer

NBA: neural blackboard for relations

48 FvdV (2016). Frontiers in Neurorobotics

BaBi Reasoning tasks: John go kitchen John get milk John go office John drop milk John go room Where re is is Jo John? A: room Where re is is milk? milk? A: office

In sit situ co conce cepts s Ai = Agent Vi = Verb Oi = Object

A1 John A2 A3 A4 A5 O1 V1 V2 V3 V4 V5 O2 O3 O4 O5

kitchen

• ffice

milk room go get drop

location localizer

NBA: neural blackboard for relations

49 FvdV (2016). Frontiers in Neurorobotics

BaBi Reasoning tasks: John go kitchen John get milk John go office John drop milk John go room Where re is is Jo John? A: room Where re is is milk? milk? A: office

In sit situ co conce cepts s Ai = Agent Vi = Verb Oi = Object

A1 John A2 A3 A4 A5 O1 V1 V2 V3 V4 V5 O2 O3 O4 O5

kitchen

• ffice

milk room go get drop

location localizer

NBA: neural blackboard for relations

50 FvdV (2016). Frontiers in Neurorobotics

BaBi Reasoning tasks: John go kitchen John get milk John go office John drop milk John go room Where re is is Jo John? A: room Where re is is milk? milk? A: office

In sit situ co conce cepts s Ai = Agent Vi = Verb Oi = Object

A1 John A2 A3 A4 A5 O1 V1 V2 V3 V4 V5 O2 O3 O4 O5

kitchen

• ffice

milk room go get drop

location localizer

NBA: neural blackboard for relations

51 FvdV (2016). Frontiers in Neurorobotics

BaBi Reasoning tasks: John go kitchen John get milk John go office John drop milk John go room Where re is is Jo John? A: room Where re is is milk? milk? A: office

In sit situ co conce cepts s Ai = Agent Vi = Verb Oi = Object

A1 John A2 A3 A4 A5 O1 V1 V2 V3 V4 V5 O2 O3 O4 O5

kitchen

• ffice

milk room go get drop

location localizer

NBA: neural blackboard for relations

52 FvdV (2016). Frontiers in Neurorobotics

BaBi Reasoning tasks: John go kitchen John get milk John go office John drop milk John go room Where re is is Jo John? A: room Where re is is milk? milk? A: office

In sit situ co conce cepts s Ai = Agent Vi = Verb Oi = Object

A1 John A2 A3 A4 A5 O1 V1 V2 V3 V4 V5 O2 O3 O4 O5

kitchen

• ffice

milk room go get drop

location localizer

NBA: neural blackboard for relations

53 FvdV (2016). Frontiers in Neurorobotics

BaBi Reasoning tasks: John go kitchen John get milk John go office John drop milk John go room Where re is is Jo John? A: room Where re is is milk? milk? A: office

In sit situ co conce cepts s Ai = Agent Vi = Verb Oi = Object

A1 John A2 A3 A4 A5 O1 V1 V2 V3 V4 V5 O2 O3 O4 O5

kitchen

• ffice

milk room go get drop

location localizer

NBA: neural blackboard for relations

54 FvdV (2016). Frontiers in Neurorobotics

BaBi Reasoning tasks: John go kitchen John get milk John go office John drop milk John go room Where re is is Jo John? A: room Where re is is milk? milk? A: office

In sit situ co conce cepts s Ai = Agent Vi = Verb Oi = Object

Where re is is ? + + Obje ject ct = = lo loca calize lizer

A1 John A2 A3 A4 A5 O1 V1 V2 V3 V4 V5 O2 O3 O4 O5

kitchen

• ffice

milk room go get drop

location localizer

NBA: neural blackboard for relations

55 FvdV (2016). Frontiers in Neurorobotics

BaBi Reasoning tasks: John go kitchen John get milk John go office John drop milk John go room Where re is is Jo John? A: room Where re is is milk? milk? A: office

In sit situ co conce cepts s Ai = Agent Vi = Verb Oi = Object

Where re is is ? + + Obje ject ct = = lo loca calize lizer

A1 John A2 A3 A4 A5 O1 V1 V2 V3 V4 V5 O2 O3 O4 O5

kitchen

• ffice

milk room go get drop

location localizer

NBA: neural blackboard for relations

56 FvdV (2016). Frontiers in Neurorobotics

BaBi Reasoning tasks: John go kitchen John get milk John go office John drop milk John go room Where re is is Jo John? A: room Where re is is milk? milk? A: office

In sit situ co conce cepts s Ai = Agent Vi = Verb Oi = Object

Where re is is ? + + Obje ject ct = = lo loca calize lizer

A1 John A2 A3 A4 A5 O1 V1 V2 V3 V4 V5 O2 O3 O4 O5

kitchen

• ffice

milk room go get drop

location localizer

NBA: neural blackboard for relations

57 FvdV (2016). Frontiers in Neurorobotics

BaBi Reasoning tasks: John go kitchen John get milk John go office John drop milk John go room Where re is is Jo John? A: room Where re is is milk? milk? A: office

In sit situ co conce cepts s Ai = Agent Vi = Verb Oi = Object

Where re is is ? + + Ag Agent = = lo loca catio ion

Reservoir:

• Random and sparse (fixed) connectivity.
• S nodes affect blackboard control (learning).
• Input from question and blackboard
• More elaborate: S node = column

NBA: Reservoir for sequential control

Reservoir of S

(sequence) nodes S node = column

58

NBA: Reservoir for sequential control

Reservoir of S

(sequence) nodes S node = column

59

Reservoir column:

• Circuits for control
• Delay gives temporal control.
• Stop prevents reactivation (in same sequence)

NBA: Reservoir for sequential control

Where is John Agent

• Each color is different set of S nodes (columns)
• With Stop to prevent reactivation (in same sequence)

60

Selective response to sequence: Where is John Agent (= location)

Ø Extended sentence processing and neuronal activation Ø Learning linguistic constructions (e.g., Goldberg, 1995, 2003) Ø Forms of reasoning Ø Integrating with geometrical concept relations Ø Integrating sentence NBA with a phonology NBA Ø Parallel implementation (TrueNorth would perhaps be suitable for NBA)

NBA: Further developments