CUIML: A Language For the Generation of Multimodal Human-Computer - - PowerPoint PPT Presentation

CUIML: A Language For the Generation of Multimodal Human-Computer Interfaces

Christian Sandor

sandor@cs.tum.edu

Technische Universit¨ at M¨ unchen Chair for Applied Software Engineering

Abstract

DWARF Project at the Technische Universit¨ at M¨ unchen UIML complied with some of our re- quirements Extension of UIML was developed and will now be presented

2

What is DWARF?

Distributed Wearable Augmented Reality Framework DWARF movie

Augmented Reality Wearable Computer

3

The Problem

Framework for Presentation Layer of Wearable Computers

• An application should be platform independent

4

The Problem

Framework for Presentation Layer of Wearable Computers

• An application should be platform independent
• Interaction with the user is multimodal

Voice Input Actions Actions Mouse Input trigger

4

The Problem

Framework for Presentation Layer of Wearable Computers

• An application should be platform independent
• Interaction with the user is multimodal

Voice Input Mouse Input Actions Actions trigger

5

CUIML – a solution

Cooperative User Interfaces Markup Language Based on UIML

• Separation of document structure and

presentation

• Transformation to markup languages
• Display in browsers for VRML, VoiceXML, HTML

(Views)

New concepts:

• Controller
• Manipulators

6

System Design – Views

UIML:

• Adresses requirement of platform independence
• No support for multiple cooperating Views

A B C D E HTML H1 H2 H3 H2 H2

UIML document

Generation Process 1. 2. Mapping Generic Structure A = W1 C = W2 B = W2 D = W7 E = W7 WML W1 W2 W2 W7 W7 A = H1 C = H3 B = H2 D = H2 E = H2 HTML WML

7

System Design – Views (2)

Multimodal Human-Computer Interfaces consist of multiple Views

VRML WML VRML 7 8 4 5 6 1 2 3 HCI 2 B E C F G D I H Generation Process 1. 2. HTML 1 2 3 4 5 HCI 1 VoiceXML

CUIML document

6 7 8 Generic Structure A Mapping

• 2. VRML WML

C = 1 G = 3 F = 2 D = 4 H = 5 I = 6 E = 8 B = 7 E = 2 B = 1 C = 3 G = 5 F = 4 D = 6 H = 7 I = 8

• 1. VRML HTML VoiceXML

8

System Design – Controller

A central component is needed

• Describes state of the HCI
• Synchronizes views

View View View

??

9

System Design – Controller

A central component is needed

• Describes state of the HCI
• Synchronizes views

Deterministic Finite Automaton with XML based Configuration

View View View DFA

10

System Design – Controller (2)

CUIML extended with Controller

VRML B E C F G D I H VoiceXML 6 7 8 Controller Controller Generation Process 1. 2. HTML 1 2 3 4 5 HCI 1

CUIML document

Generic Structure A Mapping

• 2. VRML WML

C = 1 G = 3 F = 2 D = 4 H = 5 I = 6 E = 8 B = 7 E = 2 B = 1 C = 3 G = 5 F = 4 D = 6 H = 7 I = 8

• 1. VRML HTML VoiceXML

Controller Config = WML VRML 7 8 4 5 6 1 2 3 HCI 2

11

Controller Example

State transitions of a View

VRML View

PrintJobQuestion Confirmed Rejected Yes No Printer available

Controller

12

Controller Example

State transitions of a View

VRML View

PrintJobQuestion Confirmed Rejected Yes No Printer available

Controller

13

Controller Example

State transitions of a View

VRML View

PrintJobQuestion Confirmed Rejected Yes No Printer available

Controller

14

System Design – Manipulators

Views have to be modified by the Controller

• Access mechanisms for Views differ

Controller View ??

15

System Design – Manipulators

Views have to be modified by the Controller

• Access mechanisms for Views differ
• Every View needs an appropriate Manipulator
• Described in behaviour section

Controller View Manipulator

16

System Design – Manipulators (2)

Example of a Manipulator

A B C D E C’ D’ E’ A’ B’ Manipulator Manipulator A’ B’ C’ D’ E’ Controller

CUIML document

Structure Behaviour Controller Event "X"

=>

A B C D E Generation Process Controller Controller Event "X" 17

Benefits of CUIML

Eases the task of developing multimodal HCIs Dynamic reconfiguration of the HCI to preserve functionality Incorperates the advantages of UIML

• Separation of development tasks
• Platform independence

18

Future Work

Rewrite renderers with JavaCC Adapt the Human-Computer Interface at runtime

19

Future Work

Rewrite renderers with JavaCC Adapt the Human-Computer Interface at runtime Establish standard metaphors

http://www.cg.tuwien.ac.at/research/vr/pip/ http://www.csl.sony.co.jp/person/rekimoto/cube/ 19