New Techniques for Usability Evaluation of Mobile Systems Jesper - - PowerPoint PPT Presentation

New Techniques for Usability Evaluation of Mobile Systems

Jesper Kjeldskov & Jan Stage Department of Computer Science Aalborg University Denmark

2 Kjeldskov & Stage, New Techniques for Usability Evaluation of Mobile Systems

Background

Mobile technologies and systems…

PDAs, wearables, mobile phones, tablet computers

… challenge usability testing methods

Users are physically mobile during use Use involve activities in physical surroundings Use context can be difficult to recreate realistically

This paper explore new techniques for usability

testing mobile systems in laboratory settings

3 Kjeldskov & Stage, New Techniques for Usability Evaluation of Mobile Systems

Two Evaluation Approaches

Field experiments

Realistic use context Difficult to control Complicated data collection Complex and time consuming Safety and ethical issues

Laboratory experiments

Experimental control High quality data collection Lack of realism

4 Kjeldskov & Stage, New Techniques for Usability Evaluation of Mobile Systems

Laboratory vs. Field

Most usability evaluations of mobile systems are

currently conducted in laboratory settings

A recent literature study revealed that…

41% of mobile HCI research involve evaluation 71% of this is done in laboratory settings

It is a widely adopted point of view that mobile

systems require field evaluations, but…

It is difficult to conduct field evaluations The added value of testing in the field is unknown Additional problems come at a high cost (time & effort)

5 Kjeldskov & Stage, New Techniques for Usability Evaluation of Mobile Systems

Overall Research Question

• How can new techniques for usability tests of

mobile systems increase realism of use in a laboratory setting? … while facilitating systematic data collection in a controlled environment

• Previous studies: Specialized use contexts
• This study: Physical mobility during use

6 Kjeldskov & Stage, New Techniques for Usability Evaluation of Mobile Systems

Two Experiments

• Two experiments comparing techniques for lab- and field-

based usability testing of mobile systems were conducted

• Experiments explored different techniques requiring…

1.

Different levels of physical movement

2.

Divided cognitive attention

• Example application: use of Short Message Service (SMS)
• n PDAs and mobile phones
• Narrow focus on interaction rather than broad focus on use

7 Kjeldskov & Stage, New Techniques for Usability Evaluation of Mobile Systems

Five Laboratory Techniques…

Attention needed to navigate

• 5. Walking at

varying speed on a changing track

• 3. Walking on

a treadmill with varying speed Varying

• 4. Walking at

constant speed on a changing track

• 2. Walking on a

treadmill with constant speed Constant n/a

• 1. Sitting at a table
• r standing

None Body Motion Conscious None

The experiment: measuring the relative strengths

and weaknesses with reference to field testing

8 Kjeldskov & Stage, New Techniques for Usability Evaluation of Mobile Systems

The Laboratory Experiments

5 conditions (6 test subjects per condition) Number of usability problems Performance (task completion time) Subjective workload (NASA TLX)

9 Kjeldskov & Stage, New Techniques for Usability Evaluation of Mobile Systems

The Field Experiment

1 condition: walking in a pedestrian street (6 test subjects) Number of usability problems Performance (task completion time) Subjective workload (NASA TLX)

10 Kjeldskov & Stage, New Techniques for Usability Evaluation of Mobile Systems

Findings (1)

Mean number of usability problems identified by each technique

Problems detected

6 ,3 5,2 10 ,8 7,5 6 ,7 6 ,7 2 4 6 8 10 12 14 1 2 3 4 5 6

Technique

• 1. Sitting at a table
• 2. Walking on a treadmill with constant speed
• 3. Walking on a treadmill with varying speed
• 4. Walking at constant speed on a changing track
• 5. Walking at varying speed on a changing track
• 6. walking in a pedestrian street

More problems found on average when seated at a table Statistical significance Lab techniques with physical movement comparable to

field evaluation

11 Kjeldskov & Stage, New Techniques for Usability Evaluation of Mobile Systems

Findings (2)

Number of identified usability problems categorized by severity

53 23 18 21 20 23 34 Total 32 12 6 8 8 8 19 Cosmetic 17 8 9 9 9 11 11 Serious 4 3 3 4 3 4 4 Critical Field Lab 5 Lab 4 Lab 3 Lab 2 Lab 1 Total Techniques

No technique identified all problems Most problems found when seated at table (34) Comparable numbers of critical problems found (3-4)

12 Kjeldskov & Stage, New Techniques for Usability Evaluation of Mobile Systems

Findings (2)

Number of identified usability problems categorized by severity

53 23 18 21 20 23 34 Total 32 12 6 8 8 8 19 Cosmetic 17 8 9 9 9 11 11 Serious 4 3 3 4 3 4 4 Critical Field Lab 5 Lab 4 Lab 3 Lab 2 Lab 1 Total Techniques

More than double the number of cosmetic problems were

found while seated compared to the other lab techniques

13 Kjeldskov & Stage, New Techniques for Usability Evaluation of Mobile Systems

Findings (2)

Number of identified usability problems categorized by severity

53 23 18 21 20 23 34 Total 32 12 6 8 8 8 19 Cosmetic 17 8 9 9 9 11 11 Serious 4 3 3 4 3 4 4 Critical Field Lab 5 Lab 4 Lab 3 Lab 2 Lab 1 Total Techniques

Only 3 out of 4 critical usability problems were identified

• n basis of the field technique

14 Kjeldskov & Stage, New Techniques for Usability Evaluation of Mobile Systems

Findings (3)

Subjective experience of workload with the different techniques

54 48 55 48 35 27 Overall workload 186 178 228 106 163 52 Effort 194 127 118 112 117 92 Physical demands 148 185 126 204 75 29 Mental demands Field Lab 5 Lab 4 Lab 3 Lab 2 Lab 1 Techniques

Sitting at a table (lab 1) required significantly less mental

activity compared to all other techniques but lab 2

Overall, sitting or walking at constant speed is experienced

significantly less demanding than any other technique

15 Kjeldskov & Stage, New Techniques for Usability Evaluation of Mobile Systems

Discussion (1)

Seating test subjects at a table… superior?

Supported the identification of most usability problems Difference mostly accounted for by cosmetic problems Less workload facilitated more thinking-aloud Relevance of cosmetic problems can be questioned

Increasing workload in lab

Helped approximating the field condition but resulted in

fewer problems identified compared to when being seated

Better focus missing vital problems??

16 Kjeldskov & Stage, New Techniques for Usability Evaluation of Mobile Systems

Discussion (2)

Usability problems and mobility

Physical motion in the lab triggered unique interaction

problems also found in the field

Added value in relation to e.g. layout and button sizes

Data collection in the field

All field tests were recorded with

a camcorder

Difficult to capture good images

• f screen

The “bodyguard” effect Changing the role of the test monitors? Mounting small cameras on test subject and device?

17 Kjeldskov & Stage, New Techniques for Usability Evaluation of Mobile Systems

Future work

Cost/benefit of different techniques and settings

Time and effort spent per problem found

When should lab or field tests ideally be applied? How can field test techniques be improved?

Improving data collection? Enforcing more experimental control? Doing something completely different?