In p ro T K in A c tio n : O p e n -S o u rc e S o ftw a re fo r - - PowerPoint PPT Presentation

In p ro T K in A c tio n : O p e n -S o u rc e S o ftw a re fo r B u ild in g G e rm a n -S p e a k in g In c re m e n ta l S D S s

T im o B a u m a n n , O k k o B u ß , D a v id S c h la n g e n tim o @ lin g .u n i- p o ts d a m .d e h ttp ://w w w .lin g .u n i- p o ts d a m .d e /~ tim o

S p o k e n D ia lo g u e S y ste m s

d o m a in A S R re sp o n se g e n e ra to r h isto ry dialogue manager N LU T T S v isu a l

• u tp u t

S p o k e n D ia lo g u e S y ste m s

d o m a in A S R re sp o n se g e n e ra to r h isto ry dialogue manager N LU T T S v isu a l

• u tp u t

S p o k e n D ia lo g u e S y ste m s

d o m a in A S R re sp o n se g e n e ra to r h isto ry dialogue manager N LU T T S v isu a l

• u tp u t

S p o k e n D ia lo g u e S y ste m s

d o m a in A S R re sp o n se g e n e ra to r h isto ry dialogue manager N LU T T S v isu a l

• u tp u t
• m o d u le s s ta rt a fte r th e ir p re d e c e s s o rs h a v e fin is h e d

In c re m e n ta l In c re m e n ta l S p o k e n D ia lo g u e S y ste m

d o m a in A S R re sp o n se g e n e ra to r h isto ry dialogue manager N LU T T S v isu a l

• u tp u t
• p a r tia l r e su lts

p a r tia l r e su lts a re be in g p ro c e s s e d im m e d ia te ly

• re a c tio n is q

u ic k e r, in te ra c tio n m o re n a tu ra l

B e n e fits o f In c re m e n ta l S p o k e n D ia lo g u e S y ste m s

1 .re a c t m o re q u ic k ly a s m o d u le s p ro c e s s in p u t d u rin g a s p e a k e r's tu rn :

U: Ich möchte am Samstag von Berlin nach Hamburg fahren. S: Ok, um wieviel Uhr möchten Sie fahren? U: Ich möchte am Samstag von Berlin nach Hamburg fahren. S: Ok, um wieviel Uhr möchten Sie fahren?

(C ra fte d e xa m p le s fo r a n im a g in a ry tra in tim e ta ble in fo rm a tio n s ys te m .)

B e n e fits o f In c re m e n ta l S p o k e n D ia lo g u e S y ste m s

1 .re a c t m o re q u ic k ly a s m o d u le s p ro c e s s in p u t d u rin g a s p e a k e r's tu rn :

U: Ich möchte am Samstag von Berlin nach Hamburg fahren. S: Ok, um wieviel Uhr möchten Sie fahren? U: Ich möchte am Samstag von Berlin nach Hamburg fahren. S: Ok, um wieviel Uhr möchten Sie fahren?

s u ffic ie n t in fo rm a tio n : S a tu rd a y, B H H

→

B e n e fits o f In c re m e n ta l S p o k e n D ia lo g u e S y ste m s

1 .re a c t m o re q u ic k ly a s m o d u le s p ro c e s s in p u t d u rin g a s p e a k e r's tu rn 2 .g iv e fe e d ba c k d u rin g a s p e a k e r's tu rn :

• fe e d ba c k m ig h t be v is u a l in a m u lti- m o d a l s ys te m

U: Ich möchte am Samstag mit dem ICE Nummer, äh … warten sie … 798 … S: ja? ok. U: Ich möchte am Samstag mit dem ICE Nummer, äh … warten sie … 798 … S: ja? ok.

B e n e fits o f In c re m e n ta l S p o k e n D ia lo g u e S y ste m s

1 .re a c t m o re q u ic k ly a s m o d u le s p ro c e s s in p u t d u rin g a s p e a k e r's tu rn 2 .g iv e fe e d ba c k d u rin g a s p e a k e r's tu rn 3 .e v e n in te rru p t a s p e a k e r's tu rn :

U: Ich möchte am Samstag mit dem ICE Nummer 798 nach, äh … S: Entschuldigung, ICE 798 verkehrt nicht samstags, wohin möchten Sie denn fahren? U: Ich möchte am Samstag mit dem ICE Nummer 798 nach, äh … S: Entschuldigung, ICE 798 verkehrt nicht samstags, wohin möchten Sie denn fahren?

B e n e fits o f In c re m e n ta l S p o k e n D ia lo g u e S y ste m s

1 .re a c t m o re q u ic k ly a s m o d u le s p ro c e s s in p u t d u rin g a s p e a k e r's tu rn 2 .g iv e fe e d ba c k d u rin g a s p e a k e r's tu rn 3 .e v e n in te rru p t a s p e a k e r's tu rn

→ → a ll th e s e c a p a bilitie s m a k e th e S D S m o r e sim ila r m o r e sim ila r to a h u m a n in te rlo c u to r

Co n te n t:

✔ A d v a n ta g e s o f in c re m e n ta l S D S s

➔ R e q

u ire m e n ts fo r in c re m e n ta l S D S s

• O u r m o d e l o f in c re m e n ta l p ro c e s s in g
• O u r im p le m e n ta tio n : In p ro T K

 O v e rv ie w o f th e a rc h ite c tu re  P re d e fin e d M o d u le s

• E xa m p le s ys te m s

R e q u ire m e n ts fo r In c re m e n ta l S D S s

• S ys te m fu lly e m bra c e s in c re m e n ta lity

 it's v e ry h a rd to a d a p t a p re - e xis tin g S D S

to tu rn it in to a n in c re m e n ta l s ys te m

• 1

00 % in c re m e n ta l m o d u le s

 j

u s t o n e n o n - in c re m e n ta l m o d u le bre a k s th e p ip e lin e

• P ro c e s s in g d e la ys a re m in im ize d (bu ffe rin g , e tc .)

 a c ro s s th e bo a rd – a ll p ro c e s s in g d e la ys a d d u p !  o th e rw is e to o s lo w fo r re a lly in te re s tin g a p p lic a tio n s

R e q u ire m e n ts (II): D e a lin g w ith Un c e rta in ty

• in te rm e d ia te h yp o th e s e s c h a n g e w ith tim e

 w e m a y g e t th in g s w ro n g in te rm itte n tly:

„Hamburg“ /hamb →

Urg/

• In cre m e n ta lly th is w ill lo o k to

s p e e c h re c o g n itio n a s fo llo w s …

R e q u ire m e n ts (II): D e a lin g w ith Un c e rta in ty

• in te rm e d ia te h yp o th e s e s c h a n g e w ith tim e

 w e m a y g e t th in g s w ro n g in te rm itte n tly:

„Hamburg“ / → hambUrg/

• A S R :
• N LU :

th is s o u n d s lik e „ H a m m “ th e y m u s t be ta lk in g a bo u t [ c ity:H a m m (W e s tfa le n )]

R e q u ire m e n ts (II): D e a lin g w ith Un c e rta in ty

• in te rm e d ia te h yp o th e s e s c h a n g e w ith tim e

 w e m a y g e t th in g s w ro n g in te rm itte n tly:

„Hamburg“ / → hambUrg/

• A S R :
• N LU :

th is s o u n d s lik e „ H a m bu rg “ th e y m u s t be ta lk in g a bo u t [ c ity:H a m bu rg ]

R e q u ire m e n ts (II): D e a lin g w ith Un c e rta in ty

• in te rm e d ia te h yp o th e s e s c h a n g e w ith tim e

 w e m a y g e t th in g s w ro n g in te rm itte n tly:

„Hamburg“ / → hambUrg/

• C o u ld n 't th e A S R j

u s t la g be h in d a little bit?

R e q u ire m e n ts (II): D e a lin g w ith Un c e rta in ty

• C o u ld n 't th e A S R j

u s t la g be h in d a little bit?

• Y e s , bu t:

 lo n g - d is ta n c e d e p e n d e n c ie s  th e re w ill a lw a ys be lo c a l a m big u itie s  a ll d e la ys w ill a d d u p ➔ h e n c e , p re v io u s h yp o th e s e s m u s t be c h a n g e a b le

R e q u ire m e n ts (II): D e a lin g w ith Un c e rta in ty

• C o u ld n 't th e A S R j

u s t la g be h in d a little bit?

• Y e s , bu t:
• e . g . g a rd e n - p a th

s e n te n c e s , …

➔ h e n c e , p re v io u s h yp o th e s e s m u s t be c h a n g e a b le

„The horse raced past the barn

W ill s h e s a y „fell“ o r n o t?

a n in c re m e n ta l p a rse r:

Co n te n t:

✔ A d v a n ta g e s o f in c re m e n ta l S D S s ✔ R e q

u ire m e n ts fo r in c re m e n ta l S D S s

➔O u r m o d e l o f in c re m e n ta l p ro c e s s in g

• O u r im p le m e n ta tio n : In p ro T K

 O v e rv ie w o f th e a rc h ite c tu re  P re d e fin e d M o d u le s

• E xa m p le s ys te m s

O u r M o d e l o f In c re m e n ta l Pro c e ssin g

• A s ys te m c o n s is ts o f s e v e ra l c o n n e c te d m o d u le s
• In cre m e n ta l M o d u le s a re c o m p o s e d o f

 a le ft b u ffe r , a p ro ce sso r , a n d a rig h t b u ffe r

• a p ro c e s s o r ta k e s in p u t fro m th e le ft bu ffe r a n d

p ro v id e s o u tp u t in its rig h t bu ffe r

(S c h la n g e n a n d S k a n tze , 2 009 )

le ftb u ffe rA p ro c e sso rA rig h tb u ffe rA

In te r-M o d u le Co m m u n ic a tio n

• A m o d u le 's rig h t bu ffe r m a y be s u p e rim p o s e d to
• th e r m o d u le s ' le ft bu ffe r to s h a re th e s a m e c o n te n t
• m o d u le s c o m m u n ic a te by p ro bin g c o n te n t

a n d a d d in g c o n te n t in th e ir bu ffe rs

le ftb u ffe rB p ro c e sso rB rig h tb u ffe rB le ftb u ffe rA p ro c e sso rA rig h tb u ffe rA

(S c h la n g e n a n d S k a n tze , 2 009 )

In c re m e n ta l Un its

• C o n te n t is s h a re d in th e fo rm o f In c r e m e n ta l U n its

(IU s ), w h ic h a re s m a lle s t 'c h u n k s ' o f in fo rm a tio n

• Lin k s be tw e e n IU s :

 g r o u n d e d -in lin k s (g rin ) to d e n o te a n c e s try  sa m e -le v e l lin k s (sll) fo r in fo rm a tio n o f th e s a m e typ e

le ftb u ffe rB p ro c e sso rB rig h tb u ffe rB IU4 le ftb u ffe rA p ro c e sso rA rig h tb u ffe rA IU1 IU2 IU3 IU1 IU2 IU3 IU4 IU4 g rin sll

(S c h la n g e n a n d S k a n tze , 2 009 )

IU N e tw o rk

• a ll IU s a re c o n n e c te d th ro u g h (sll a n d g rin ) lin k s

 th is n e tw o rk c o n ta in s a ll th e in fo rm a tio n be lie v e d

by th e s ys te m a t a c e rta in p o in t in tim e

 th e n e tw o rk is h ig h ly d yn a m ic , w ith ch a n g e s to th e

n e tw o rk re fle c tin g th e s ys te m 's in te rn a l s ta te o v e r tim e

• M o d u le s re a c t to th re e ba s ic c h a n g e s :

 n e w IU s a re a d d e d  e rro n e o u s ly h yp o th e s ize d IU s a re r e v o k e d  IU s a re c o m m itte d , i. e . w o n 't be c h a n g e d a n ym o re

In p ro T K : O v e rv ie w

• O u r to o lk it In p ro T K is a n im p le m e n ta tio n
• f o u r m o d e l o f in c re m e n ta l p ro c e s s in g

 m o d u la r a rc h ite c tu re  e v e n t- ba s e d c o m m u n ic a tio n be tw e e n m o d u le s

• w ritte n in J

A V A , in te g ra te d w ith S p h in x- A S R

 ric h s p e e c h re c o g n itio n , p ro s o d ic p ro c e s s in g

• e xte n s ible , o p e n - s o u rc e , s o m e w h a t d o c u m e n te d
• w w w .lin g .u n i- p o ts d a m .d e /~ tim o /c o d e /in p ro tk /

In p ro T K : A v a ila b le M o d u le s

d o m a in A S R re sp o n se g e n e ra to r h isto ry dialogue manager N LU T T S v isu a l

• u tp u t

Flo o r T ra c k e r

✔ ✔ ✔ ✔ ✔

m o n ito rin g , d e bu g g in g , a n d a n a lys is c o m p o n e n ts

✔

In p ro T K : In c re m e n ta l A S R

• in te g ra te s w ith S p h in x- 4

 s u p p o rts J

S GF- g ra m m a rs , S LM s , fo rc e d - a lig n m e n t …

 in p u t fro m m ic ro p h o n e , file , R T P

• c u rre n t h yp o th e s is is u p d a te d a fte r e v e ry fra m e
• f a u d io c o n s u m e d by th e re c o g n ize r

 h yp o th e s is s m o o th in g to re d u c e „

j itte r“ a t th e c o s t o f s o m e tim e lin e s s

• (s h o w v id e o )

(B a u m a n n e t a l., 2 009 )

In p ro T K : Flo o r-T ra c k in g

• tu rn - ta k in g is (a lm o s t) triv ia l in c o n v e n tio n a l S D S

 th e u s e r's tu rn is o v e r w h e n s h e s to p s fo r 5

00 m s

• in th e in c re m e n ta l c a s e , w e w a n t to be q

u ic k w h e n w e c a n , bu t n o t in te rru p t w h e n w e s h o u ld n 't

➔ a s p e c ific c o m p o n e n t th a t h a n d le s th is c o m p le xity

 th e flo o r tra c k e r e m its s ig n a ls lik e „

e n d o f tu rn (ris in g / fa llin g /… )“ , „ u s e r is h o ld in g “ , „ B C o p p o rtu n ity“ , e tc .

 th e d ia lo g u e m a n a g e r c o n s u m e s th e s e s ig n a ls

In p ro T K : In c re m e n ta l N LU

• w o rd s a re a s s ig n e d a ttrib u te-v a lu e p a irs (A V P s )
• c o m p le x s e m a n tic s a re re p re s e n te d a s

a ttrib u te -v a lu e m a trices (A V M s )

• firs t s te p : c o m p o s in g A V P s to

u n d e rs p e c ifie d A V M s

• s e c o n d s te p : re s o lv in g A V M s a g a in s t (fu lly

s p e c ifie d ) e n titie s in th e d o m a in

In p ro T K : D ia lo g u e M a n a g e m e n t

• in fo rm a tio n -sta te u p d a te (IS U ) m e c h a n is m
• ba s e d o n q u estio n s u n d er d iscu ssio n (Q U D )
• IS c o m bin e s s e m a n tic s lo ts , a c tio n p la n n in g a n d

in fo rm a tio n g ro u n d in g

• th is is v e ry m u c h w o rk in p ro g re s s
• a ls o , th e re is a s im p le E c h o D ia lo g u e M a n a g e r

(B u s s a n d S c h la n g e n , 2 01 0)

Exa m p le A p p lic a tio n :

• s h o w v id e o 1

Co n c lu sio n

• I h o p e to h a v e c o n v in c e d yo u th a t …

 in c re m e n ta l p ro c e s s in g is v ita l

fo r m o re n a tu ra l d ia lo g u e s ys te m s

 im p le m e n tin g s u c h s ys te m s

is a w o rth w h ile e n d e a v o u r

 yo u s h o u ld g o a h e a d a n d bu ild o n e yo u rs e lf

… p re fe ra bly u s in g o u r to o lk it!

T h a n k y o u !

A c k n o w le d g e m e n ts : O k k o B u ß a n d D a v id S c h la n g e n , m y c o lla bo ra to rs . D FG fo r fu n d in g (E m m y N o e th e r p ro g ra m m e )