Topographic Organization of Receptive Fields in RecSOM or RecSOM - - PowerPoint PPT Presentation

Topographic Organization of Receptive Fields in RecSOM

• r

RecSOM as nonlinear IFS

Peter Tiˇ no University of Birmingham, UK Igor Farkaˇ s Slovak University of Technology, Slovakia Jort van Mourik NCRG, Aston University, UK

Receptive fields in RecSOM

Some motivations

◗ Most approaches to topographic map formation operate on the assumption that the data points are members of a finite- dimensional vector space of a fixed dimension. ◗ Recently, there has been an outburst of interest in extending topographic maps to more general data structures, such as se- quences or trees. ◗ Modified versions of SOM that have enjoyed a great deal of interest equip SOM with additional feed-back connections that allow for natural processing of recursive data types. ◗ No prior notion of metric on the structured data space is im- posed, instead, the similarity measure on structures evolves through parameter modification of the feedback mechanism and recursive comparison of constituent parts of the structured data.

• P. Tiˇ

no, I. Farkaˇ s and J. van Mourik 1

Receptive fields in RecSOM

Motivations cont’d

◗ Typical examples: Temporal Kohonen Map (Chappell, 1993), recurrent SOM (Koskela, 1998), feedback SOM (Horio, 2001), recursive SOM (Voegtlin, 2002), merge SOM (Strickert, 2003) and SOM for structured data (Hagenbuchner, 2003) ◗ At present, there is no general consensus as to how best to process sequences with SOMs. ◗ Representational capabilities of the models are hardly under- stood. ◗ The internal representation of structures within the models is unclear. ◗ First major theoretical study within a unifying framework in (Hammer, 2004).

• P. Tiˇ

no, I. Farkaˇ s and J. van Mourik 2

Receptive fields in RecSOM

Recursive Self-Organizing Map - RecSOM

s(t) w c

i i

map at time t map at time (t−1)

• P. Tiˇ

no, I. Farkaˇ s and J. van Mourik 3

Receptive fields in RecSOM

RecSOM - weights

Each neuron i ∈ {1, 2, ..., N} in the map has two weight vectors associated with it:

• wi ∈ Rn – linked with an n-dim input s(t) feeding the network at

time t

• ci ∈ RN – linked with the context

y(t − 1) = (y1(t − 1), y2(t − 1), ..., yN(t − 1)) containing map activations yi(t − 1) from the previous time step.

• P. Tiˇ

no, I. Farkaˇ s and J. van Mourik 4

Receptive fields in RecSOM

RecSOM - neuron activations

The output of a unit i at time t is computed as yi(t) = exp(−di(t)), (1) where di(t) = α · s(t) − wi2 + β · y(t − 1) − ci2 (2) α > 0 and β > 0 are model parameters that respectively influence the effect of the input and the context upon neuron’s profile.

• P. Tiˇ

no, I. Farkaˇ s and J. van Mourik 5

Receptive fields in RecSOM

RecSOM - learning

The weight vectors are updated using the same form of learning rule ∆wi = γ · hik · (s(t) − wi), (3) ∆ci = γ · hik · (y(t − 1) − ci), (4) k is an index of the best matching unit at time t, k = argmin

i∈{1,2,...,N}

di(t) = argmax

i∈{1,2,...,N}

yi(t), γ > 0 is the learning rate, hik is a (Gaussian) neighborhood func- tion.

• P. Tiˇ

no, I. Farkaˇ s and J. van Mourik 6

Receptive fields in RecSOM

RecSOM - fixed-input dynamics

Under a fixed input vector s ∈ Rn, the time evolution becomes di(t + 1) = α · s − wi2 + β · y(t) − ci2. (5) After applying a one-to-one coordinate transformation yi = e−di, yi(t + 1) = e−αs−wi2 · e−βy(t)−ci2, (6) where y(t) = (y1(t), y2(t), ..., yN(t)) =

• e−d1(t), e−d2(t), ..., e−dN(t)

.

• P. Tiˇ

no, I. Farkaˇ s and J. van Mourik 7

Receptive fields in RecSOM

RecSOM - fixed-input dynamics cont’d

Gaussian kernel of inverse variance η > 0, acting on RN: for any u, v ∈ RN, Gη(u, v) = e−ηu−v2. (7) The fixed-input dynamics written in a vector form: y(t + 1) = Fs(y(t)) =

• Fs,1(y(t)), ..., Fs,N(y(t))
• ,

(8) where Fs,i(y) = Gα(s, wi) · Gβ(y, ci), i = 1, 2, ..., N. (9)

• P. Tiˇ

no, I. Farkaˇ s and J. van Mourik 8

Receptive fields in RecSOM

RecSOM - Contractive IFS

Study the conditions under which the map Fs becomes a con- traction. Then, by the Banach Fixed Point theorem, the autonomous Rec- SOM dynamics y(t + 1) = Fs(y(t)) will be dominated by a unique attractive fixed point ys = Fs(ys). A mapping F : RN → RN is said to be a contraction with contrac- tion coefficient ρ ∈ [0, 1), if for any y, y′ ∈ RN, F(y) − F(y′) ≤ ρ · y − y′. (10) F is a contraction if there exists ρ ∈ [0, 1) so that F is a contraction with contraction coefficient ρ.

• P. Tiˇ

no, I. Farkaˇ s and J. van Mourik 9

Receptive fields in RecSOM

Contractive IFS - Theorem

Collection of activations coming from the feed-forward part of RecSOM: Gα(s) = (Gα(s, w1), Gα(s, w2), ..., Gα(s, wN)). (11) Theorem: Consider an input s ∈ Rn. If for some ρ ∈ [0, 1), β ≤ ρ2 e 2 Gα(s)−2, (12) then the mapping Fs is a contraction with contraction coefficient ρ.

• P. Tiˇ

no, I. Farkaˇ s and J. van Mourik 10

Receptive fields in RecSOM

Theorem - Proof

The proof follows the worst case analysis of the distances Fs(y) − Fs(y′) under the constraint y − y′ = δ: Dβ(δ) = sup y,y′;y−y′=δ Fs(y) − Fs(y′). The analysis is quite challenging, because Dβ(δ) can be expressed

• nly implicitly.
• P. Tiˇ

no, I. Farkaˇ s and J. van Mourik 11

Receptive fields in RecSOM

Theorem - Proof cont’d

It is possible to prove that, for a given β > 0:

• 1. limδ→0+ Dβ(δ) = 0,
• 2. Dβ is a continuous monotonically increasing concave function of

δ.

• 3. limδ→0+ dDβ(δ)

dδ

=

• 2β

e .

0.5 1 1.5 2 2.5 3 3.5 4 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 D(delta) delta beta=2 beta=0.5

• P. Tiˇ

no, I. Farkaˇ s and J. van Mourik 12

Receptive fields in RecSOM

Theorem - Proof cont’d

Therefore, Dβ(δ) ≤ δ

• 2β

e . (13) Using (6) we get that if δ2 2β e

N

• i=1

G2α(s, wi) ≤ ρ2 δ2, (14) then Fs will be a contraction with contraction coefficient ρ. In- equality (14) is equivalent to 2β e Gα(s)2 ≤ ρ2. (15)

• P. Tiˇ

no, I. Farkaˇ s and J. van Mourik 13

Receptive fields in RecSOM

Experiment

Natural language data - ”Brave New World” by Aldous Huxley. Removed punctuation symbols, upper-case letters were switched to lower-case, the space between words was represented by ’-’. Length: 356606 symbols. Letters of the Roman alphabet were binary-encoded using 5 bits. RecSOM with 20 × 20 = 400 neurons was trained for two epochs using the following parameter settings: α = 3, β = 0.7, γ = 0.1 and σ : 10 → 0.5. Radius of the neighborhood function reached its final value at the end of the first epoch and then remained constant to allow for fine-tuning of the weights.

• P. Tiˇ

no, I. Farkaˇ s and J. van Mourik 14

Receptive fields in RecSOM

Receptive Fields (RF)

RF of a neuron is defined as the common suffix of all sequences for which that neuron becomes the best-matching unit.

n− n− h− ad− d− he− he− a− ag . in ig . −th −th −th th ti an− u− − l− nd− e− re− −a− ao an ain in . l t−h th . . y− i− g− ng− ed− f− −to− o− en un −in al −al h wh ty

• t−

at− p− −a− n−

• n− m−
• −

−an n rn ul ll e−l e−h gh x y to t− es− as− er− er− mo

• −to

−on ion .

• l

e−m m . ey t− ut− s− is−

• r−

ero t−o

• lo

ho

• n
• n
• .
• m

um im am ai ry ts tw ts− r− r− ro wo io e−o −o e−n

• n

−m t−m si ai ri e−s he−w −w t−w no so tio −o ng−o −o −n −l −h e−i di ei ni ui he−s e−w w nw

• ng

no ak k −k −− −o . −l −h −i t−i −wi −hi −li −thi ns rs ing ng nf e−k j e−c −s −g −m −y −i −i i li hi s us uc e−g g if e−f e−b −c −s −w −w −e . −a −a n−a ia la ha is c nc f

• f

−f −f −b −u −u −d d−a t−a na da . −ha as ac ic ib b .

• c

−v . −p g−t −t −d −e −q e−a a wa era ra ac ir e−r .

• s

−r −p −t s−t .

• w

sa

• re

re ar ar hr r tr

• r
• p
• v

−v t−t d−t −t

• t
• d

. u se we ere pe es er her z p e−p p av d−t n−t e−t

• t
• u

au −se be ue me es . her ter ap . mp v st rt −st tt ut

• ut

lu tu e e−e ce −he ew ev . q ea . . at t

• −t

ent

• nt

ind d dd de te e he the− e− e− em ec . at −at ht −it nt −and rd e−d ne −the the he− e− eo . . ee ed ed ad it it id

• nd

nd and ud ld le −the he

• P. Tiˇ

no, I. Farkaˇ s and J. van Mourik 15

Receptive fields in RecSOM

Investigating fixed-input dynamics

Context activations y(0) initialized in 10,000 different positions within the state space (0, 1]N. For each initial condition y(0), we checked asymptotic dynamics of Fs by monitoring L2-norm of the activation differences (y(t) − y(t − 1)) and recording the limit set (after 1000 pre-iterations).

10 20 30 40 50 60 70 80 90 100 0.5 1 1.5 2 2.5 3 3.5 4

a e h i l m n

• p

r s t −

Iteration L2−norm of activity difference

• P. Tiˇ

no, I. Farkaˇ s and J. van Mourik 16

Receptive fields in RecSOM

Induced dynamics on the map

y(t) = Fs(y(t − 1)) induces a dynamics of the winner units on the map: is(t) = argmax

i∈{1,2,...,N}

yi(t) (16)

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

a a b c d e e g h i i j k l m n

• p

q r s t t u v w x y z − − f

• P. Tiˇ

no, I. Farkaˇ s and J. van Mourik 17

Receptive fields in RecSOM

On the importance of being contractive

When the fixed-input dynamics for s ∈ A is dominated by a unique attractive fixed point ys, the induced dynamics on the map settles down in neuron is, corresponding to the mode of ys, is = argmax

i∈{1,2,...,N}

ys,i. The neuron is will be most responsive to input subsequences ending with long blocks of symbols s. Receptive fields of neurons on the map will be organized with respect to closeness of neurons to the fixed input winner is.

• P. Tiˇ

no, I. Farkaˇ s and J. van Mourik 18

Receptive fields in RecSOM

Markovian suffix-based RF organization

Assuming a unimodal character of the fixed point ys, as soon the symbol s is seen, the mode of the activation profile y will drift towards the neuron is. The more consecutive symbols s we see, the more dominant the attractive fixed point of Fs becomes and the closer the winner position is to is. In this manner, a Markovian suffix-based RF organization is cre- ated.

• P. Tiˇ

no, I. Farkaˇ s and J. van Mourik 19

Receptive fields in RecSOM

Bounds on β

Symbols with bounds above β = 0.7 (used in the experiment) are guaranteed to lead to attractive dynamics and hence to locally Markovian organizations of RF.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 2

Symbol index Beta bound

a b c d e f g h i j k l m n

• p

q r s t u v w x y z −

beta Ups1

• P. Tiˇ

no, I. Farkaˇ s and J. van Mourik 20

Receptive fields in RecSOM

Markovian suffix-based RF organization - cont’d

Assume that for each input symbol s ∈ A, Fs is a contraction with contraction coefficient ρs. Set ρmax = maxs∈A ρs. For a sequence s1:n = s1...sn−2sn−1sn over A and y ∈ (0, 1]N, define Fs1:n(y) = Fsn(Fsn−1(...(Fs2(Fs1(y)))...)) = (Fsn ◦ Fsn−1 ◦ ... ◦ Fs2 ◦ Fs1)(y). (17) If two prefixes s1:p and s1:r of a sequence s1...sp−1sp...sr−1sr... share a common suffix of length L, Fs1:p(y) − Fs1:r(y) ≤ ρL

max

√ N, (18) where √ N is the diameter of the RecSOM state space (0, 1]N.

• P. Tiˇ

no, I. Farkaˇ s and J. van Mourik 21

Receptive fields in RecSOM

Markovian suffix-based RF organization - cont’d

For sufficiently large L, the activations y1 = Fs1:p(y) and y2 = Fs1:r(y) will be close enough to have the same location of the mode i∗ = argmax

i∈{1,2,...,N}

y1

i =

argmax

i∈{1,2,...,N}

y2

i ,

and the two subsequences s1:p and s1:r yield the same best match- ing unit i∗ on the map, irrespective of the position of the subse- quences in the input stream. All that matters is that the prefixes share a sufficiently long com- mon suffix. We say that such an organization of RFs on the map has a Markovian flavour, because it is shaped solely by the suffix structure of the processed subsequences, and it does not depend

• n the temporal context in which they occur in the input stream.
• P. Tiˇ

no, I. Farkaˇ s and J. van Mourik 22

Receptive fields in RecSOM

Markovian vs. Non-Markovian RF organization

Periodic (beyond period 1), or aperiodic dynamics of autonomous systems y(t) = Fs(y(t − 1)) can result in a ‘broken topography’ of RFs and embody a potentially unbounded memory structure - current position of the winner neuron is determined by the whole series of processed inputs, and not only by a history of recently seen symbols. In many experiments on various data sets we observed purely Markovian organizations of RF. Markovian organizations can be achieved using standard SOM

• perating on top of simple fixed contractive affine IFS!
• P. Tiˇ

no, I. Farkaˇ s and J. van Mourik 23

Receptive fields in RecSOM

A question instead of a conclusion

What is the principal motivation behind building topographic maps

• f sequential data?
• better understanding of cortical signal representations -

then a considerable effort should be devoted to mathematical analysis of the scope of potential temporal representations and conditions for their emergence within the given model.

• data exploration or data preprocessing -

then we need to strive for a solid understanding of the way tem- poral contexts get represented on the map and in what way such representations fit the bill of the task we aim to solve.

• P. Tiˇ

no, I. Farkaˇ s and J. van Mourik 24

Receptive fields in RecSOM

A challenge

It is crucial to understand exactly what structures that are more powerful than Markovian organization of RFs are desired and why. To go beyond mere commenting on empirical observations, one needs to address issues such as

• what properties of the input stream are likely to induce periodic

(or aperiodic) fixed input dynamics leading to context-dependent RF representations in SOMs with feedback structures,

• what periods for which symbols are preferable,
• what is the learning mechanism (e.g. sequence of bifurcations of

the fixed input dynamics) of creating more complicated context dependent RF maps.

• P. Tiˇ

no, I. Farkaˇ s and J. van Mourik 25