THE STATE OF SPEECH IN HCI: TRENDS, THEMES & CHALLENGES - - PowerPoint PPT Presentation

▶

Nov 19, 2022 405 likes •740 views

THE STATE OF SPEECH IN HCI: TRENDS, THEMES & CHALLENGES @lmhclark @cogsis LEIGH CLARK @hci_ucd UNIVERSITY COLLEGE DUBLIN The CogSIS Project UPCOMING EVENTS Measuring and designing trust in Human-Agent Interaction workshop Before HAI

SLIDE 1

THE STATE OF SPEECH IN HCI: TRENDS, THEMES & CHALLENGES

LEIGH CLARK UNIVERSITY COLLEGE DUBLIN

@lmhclark @cogsis  @hci_ucd

The CogSIS Project

SLIDE 2

Conversational user interface (CUI) conference August 2019 Dublin, Ireland Details TBC Measuring and designing trust in Human-Agent Interaction workshop Before HAI 2018 Conference: 15th December Southampton, UK https://sites.google.com/view/mdt-hai2018/

UPCOMING EVENTS

SLIDE 3

Ben Cowan Philip Doyle Diego Garaialde

SLIDE 4

Emer Gilmartin Trinity College Dublin Stephan Schlögl MCI Centre Innsbruck Jens Edlund KTH Stockholm Matthew Aylett CereProc Ltd João Cabral Trinity College Dublin

Cosmin Munteanu University of Toronto Mississauga

SLIDE 5

SLIDE 6

SLIDE 7

SLIDE 8

https://www.amazon.com/AmazonBasics-Microwave-Compact-Works-Alexa/ dp/B07894S727

SLIDE 9

MAP OUT: PUBLICATION TRENDS RESEARCH METHODS RESEARCH THEMES

RESEARCH AIMS

SLIDE 10

speech interface; voice user interface; voice system; human computer dialog*; human machine dialog*; natural language dialog* system; natural language interface; conversational interface; conversational agent; conversational system; conversational dialog* system; automated dialog* system; interactive voice response system; spoken dialog* system; spoken human machine interaction; human system dialog*; intelligent personal assistant

+ ABSTRACT, TITLE & KEYWORD SEARCH

SEARCH TERMS & DATABASES

SLIDE 11

INCLUSION/EXCLUSION CRITERIA

1181 68

INCLUDE EXCLUDE

Speech focused Full conference / journal papers English Embodiment No interaction evaluation Non-full / non- peer reviewed

SLIDE 12

RESEARCH METHODS

SLIDE 13

DIRECTION OF COMMUNICATION User-system dialogue (44) User input only (16) System output

nly (12)

SLIDE 14

User attitudes 36 Task performance 33 Lexis & syntax 20 Perceived usability 18 System usage 15 User recall 7 Physiological data 3 Other 11

CONCEPTS MEASURED

SLIDE 15

RESEARCH THEMES

SLIDE 16

Synthesis 8 Content 7

SYSTEM SPEECH PRODUCTION

SLIDE 17

Keyboard and/or mouse 10 Digital pen 3

MODALITY COMPARISON

SLIDE 18

General production 3 Addressee identification 2 Alignment 1

USER SPEECH PRODUCTION

SLIDE 19

ASSISTIVE TECHNOLOGY & ACCESSIBILITY

https://www.nationaldeafcenter.org/topics/assistive-technology

Tabletop designs - physicians, deaf patients & interpreters Mobile interface - limited hand dexterity Voiced-based browser plugin - blind users

SLIDE 20

DESIGN INSIGHT Early design insight - speech to access GUI-based software Interface for a large-scale game

SLIDE 21

IPA EXPERIENCE Disparity between people’s mental models of IPAs & reality of interaction Human likeness can negatively affect IUX Embarrassment of public use Structure of multiple user interaction w/ Siri

SLIDE 22

CHALLENGES & ONGOING RESEARCH

SLIDE 23

CHALLENGES & ONGOING RESEARCH

MORE THEORETICAL UNDERSTANDING FOR: 1. LANGUAGE PRODUCTION TO SYSTEMS 2. PERCEPTION OF SYSTEMS 3. DESIGN IN LIGHT OF THESE

SLIDE 24

Global Partner models

SLIDE 25

Knowledge Trust Style Universal Functional Reliability Formal Colloquial Social Reliability Casual

Local Partner models

SLIDE 26

Proliferation of humanlike voices in non-human artefacts can create unrealistic expectations of capabilities

Moore, R. K. (2017). Appropriate Voices for Artefacts: Some Key Insights. In 1st International Workshop on Vocal Interactivity in-and-between Humans, Animals and Robots.

SLIDE 27

SLIDE 28

POLITENESS & FACE

Politeness linked to concept of face (Goffman, 1952; 1967) Social self-image dependent on societal norms and rules Usually best interest to save face

SLIDE 29

No politeness

Connect… Give each piece a twist… Attach… …so it’s in line with the feet Locate… …so the end is closest to the top of the body

Politeness

Just connect…. Just give each piece a little bit of a twist… Basically, attach…. …so it’s more or less in line with the feet Now just locate…. …the end should be closest to the top of the body

EXAMPLE

SLIDE 30

https://fineartamerica.com/featured/i-dont-care-if-she-is-a-tape-dispenser-i-love-sam-gross.html

SLIDE 31

KEY POINTS

1. Speech HCI fragmented
2. More theoretical development/application

can help improve cohesion in the field

3. Theories can help explain & understand,

but can also be redefined and re- conceptualised in HCI

4. Current work at HCI @ UCD looking at UCD

looking design choices affecting partners models, language production, user perception

SLIDE 32

RELEVANT PAPERS leigh.clark@ucd.ie

Clark, L., Doyle, P., Garaialde, D., Gilmartin, E., Schlögl, S., Edlund, J., ... & Cowan, B. (2018). The State

f Speech in HCI: Trends, Themes and Challenges. arXiv preprint arXiv:1810.06828.

Murad, C., Munteanu, C., Clark, L., & Cowan, B. R. (2018). Design guidelines for hands-free speech

interaction. Mobile HCI 2018 Adjunct (pp. 269-276). ACM.

Clark, L. (2018). Social boundaries of appropriate speech in HCI: a politeness perspective. BCS HCI 2018. Clark, L., Cabral, J. & Cowan, B.R. (2018). The CogSIS Project: Examining the Cognitive Effects of Speech Interface Synthesis. BCS HCI 2018. Large, D. R., Clark, L., Quandt, A., Burnett, G., & Skrypchuk, L. (2017). Steering the conversation: a linguistic exploration of natural language interactions with a digital assistant during simulated

driving. Applied Ergonomics, 63, 53-61.

Clark, L., Ofemile, A., Adolphs, S. & Rodden, T. (2016). A Multimodal Approach to Assessing User Experiences with Agent Helpers. ACM Transactions on Interactive Intelligent Systems (TIIS), 6(4) 29. Clark, L. M. H., Bachour, K., Ofemile, A., Adolphs, S. & Rodden, T. (2014). Potential of Imprecision: Exploring Vague Language in Agent Instructors. HAI 2014. Tsukuba, Japan, ACM: 339-344.

@lmhclark @cogsis  @hci_ucd