Felicitous Computing David S. Rosenblum School of Computing - - PowerPoint PPT Presentation

▶
felicitous computing
SMART_READER_LITE
LIVE PREVIEW

Felicitous Computing David S. Rosenblum School of Computing - - PowerPoint PPT Presentation

Apr 06, 2024 151 likes •1.26k views

Felicitous Computing David S. Rosenblum School of Computing National University of Singapore From UCI to NUS From UCI to NUS From UCI to NUS From UCI to NUS Singapore Singapore Singapore Ubiquitous Computing The Ideal The most

slide-1
SLIDE 1

Felicitous Computing

David S. Rosenblum School of Computing National University of Singapore

slide-2
SLIDE 2

From UCI to NUS

slide-3
SLIDE 3

From UCI to NUS

slide-4
SLIDE 4

From UCI to NUS

slide-5
SLIDE 5

From UCI to NUS

slide-6
SLIDE 6

Singapore

slide-7
SLIDE 7

Singapore

slide-8
SLIDE 8

Singapore

slide-9
SLIDE 9

Ubiquitous Computing

The Ideal “The most profound technologies are those that disappear.”

[Mark Weiser, Scientific American, 1991]

Weiser envisioned computing being “an integral, invisible part of people’s lives”, where “the computers themselves ... vanish into the background”

slide-10
SLIDE 10

Ubiquitous Computing

The Research Vision

slide-11
SLIDE 11

Ubiquitous Computing

The Research Vision

slide-12
SLIDE 12

Ubiquitous Computing

The Research Vision

slide-13
SLIDE 13

Ubiquitous Computing

The Research Vision

slide-14
SLIDE 14

Ubiquitous Computing

The Research Vision

slide-15
SLIDE 15

The Reality

Fading into the Background?

slide-16
SLIDE 16

The Reality

Fading into the Background?

slide-17
SLIDE 17

The Reality

Google Android Market (early 2012)

  • The average price of the top 50 paid

applications is just US$3.79 [modymi.com]

slide-18
SLIDE 18

The Reality

Google Android Market (early 2012)

  • The average price of the top 50 paid

applications is just US$3.79 [modymi.com]

  • 79.3% of paid applications have been

downloaded less than 100 times [Distimo]

slide-19
SLIDE 19

The Reality

Google Android Market (early 2012)

  • The average price of the top 50 paid

applications is just US$3.79 [modymi.com]

  • 79.3% of paid applications have been

downloaded less than 100 times [Distimo]

  • Only 0.1% of paid applications have been

downloaded 50,000 times or more [Distimo]

slide-20
SLIDE 20

The Reality

Stuff Just Doesn’t Work Right

slide-21
SLIDE 21

The Reality

Stuff Just Doesn’t Work Right

slide-22
SLIDE 22

The Reality

Stuff Just Doesn’t Work Right

slide-23
SLIDE 23

The Reality

Stuff Just Doesn’t Work Right

slide-24
SLIDE 24

The Reality

Stuff Just Doesn’t Work Right

slide-25
SLIDE 25

The Reality

Stuff Just Doesn’t Work Right

slide-26
SLIDE 26

The Reality

Stuff Just Doesn’t Work Right

slide-27
SLIDE 27

The Reality

Stuff Just Doesn’t Work Right

slide-28
SLIDE 28

The Reality

Stuff Just Doesn’t Work Right

slide-29
SLIDE 29

The Reality

Stuff Just Doesn’t Work Right

slide-30
SLIDE 30

The Reality

Stuff Just Doesn’t Work Right

slide-31
SLIDE 31

The Reality

Stuff Just Doesn’t Work Right

slide-32
SLIDE 32

My Next Appointments Calendar?

slide-33
SLIDE 33

Felicitous Computing Institute

Goal: to realize the original ideals of ubiquitous computing A new multi-disciplinary research institute hosted in the NUS School of Computing Strongly driven by challenge problems in a variety of application domains

slide-34
SLIDE 34

Felicitous Computing

Definition fe•lic•i•tous, adj., well chosen or suited to the circumstances; pleasing and fortunate

[Oxford American Dictionary]

  • Computing that is not poorly chosen, ill-

suited, displeasing or unfortunate!

  • An overarching philosophy of technology

development and evaluation

slide-35
SLIDE 35

Felicitous Computing

Some Key Elements

slide-36
SLIDE 36

Felicitous Computing

Some Key Elements context-awareness

slide-37
SLIDE 37

Felicitous Computing

Some Key Elements intelligent, unobtrusive processing

slide-38
SLIDE 38

Felicitous Computing

Some Key Elements robustness

slide-39
SLIDE 39

Felicitous Computing

Some Key Elements multi-modal interaction

slide-40
SLIDE 40

Felicitous Computing

Some Key Elements

  • Beyond these technical characteristics ...

Natural Useful Realistic Beneficial to Users

slide-41
SLIDE 41

Felicitous Computing

Current Research Directions

✓Context-Aware Adaptation ✓Multi-Modal Interaction ✓Emotion Sensing and Inference ✓Software Engineering for Mobile Systems

slide-42
SLIDE 42

Two Example Projects

  • 1. Automated Fault Detection in Context-Aware

Adaptive Applications (CAAAs)

  • 2. Context-Aware Mobile Music

Recommendation (CAMMR)

✓ Solved problems of both design and robustness ✓ Revealed new, interesting research challenges

slide-43
SLIDE 43

CAAAs

Context-Aware Adaptive Applications

slide-44
SLIDE 44

CAAAs

Context-Aware Adaptive Applications

slide-45
SLIDE 45

CAAAs

Context-Aware Adaptive Applications

slide-46
SLIDE 46

CAAAs

Context-Aware Adaptive Applications

slide-47
SLIDE 47

CAAAs

Context-Aware Adaptive Applications

slide-48
SLIDE 48

Adaptation in CAAAs

Environment Application

  • M. Sama, D.S. Rosenblum, Z. Wang and S. Elbaum, “Multi-Layer Faults in the Architectures of Mobile,

Context-Aware Adaptive Applications”, Journal of Systems and Software,

  • Vol. 83, Issue 6, Jun. 2010, pp. 906–914.
slide-49
SLIDE 49

Adaptation in CAAAs

Physical Context

Environment Application

  • M. Sama, D.S. Rosenblum, Z. Wang and S. Elbaum, “Multi-Layer Faults in the Architectures of Mobile,

Context-Aware Adaptive Applications”, Journal of Systems and Software,

  • Vol. 83, Issue 6, Jun. 2010, pp. 906–914.
slide-50
SLIDE 50

Adaptation in CAAAs

Physical Context

Environment

Context Manager

Application

Adaptation Manager

Middleware

  • M. Sama, D.S. Rosenblum, Z. Wang and S. Elbaum, “Multi-Layer Faults in the Architectures of Mobile,

Context-Aware Adaptive Applications”, Journal of Systems and Software,

  • Vol. 83, Issue 6, Jun. 2010, pp. 906–914.
slide-51
SLIDE 51

Adaptation in CAAAs

Physical Context Sensed Context

Environment

Context Manager

Application

Adaptation Manager

Middleware

  • M. Sama, D.S. Rosenblum, Z. Wang and S. Elbaum, “Multi-Layer Faults in the Architectures of Mobile,

Context-Aware Adaptive Applications”, Journal of Systems and Software,

  • Vol. 83, Issue 6, Jun. 2010, pp. 906–914.
slide-52
SLIDE 52

Adaptation in CAAAs

Physical Context Sensed Context Inferred Context

Environment

Context Manager

Application

Adaptation Manager

Middleware

  • M. Sama, D.S. Rosenblum, Z. Wang and S. Elbaum, “Multi-Layer Faults in the Architectures of Mobile,

Context-Aware Adaptive Applications”, Journal of Systems and Software,

  • Vol. 83, Issue 6, Jun. 2010, pp. 906–914.
slide-53
SLIDE 53

Adaptation in CAAAs

Physical Context Sensed Context Inferred Context Presumed Context

Environment

Context Manager

Application

Adaptation Manager

Middleware

  • M. Sama, D.S. Rosenblum, Z. Wang and S. Elbaum, “Multi-Layer Faults in the Architectures of Mobile,

Context-Aware Adaptive Applications”, Journal of Systems and Software,

  • Vol. 83, Issue 6, Jun. 2010, pp. 906–914.
slide-54
SLIDE 54

Adaptation in CAAAs

Physical Context Sensed Context Inferred Context Presumed Context

Environment

Context Manager

Application

Adaptation Manager

Middleware

3rd-Party Libraries

  • M. Sama, D.S. Rosenblum, Z. Wang and S. Elbaum, “Multi-Layer Faults in the Architectures of Mobile,

Context-Aware Adaptive Applications”, Journal of Systems and Software,

  • Vol. 83, Issue 6, Jun. 2010, pp. 906–914.
slide-55
SLIDE 55

Adaptation in CAAAs

Physical Context Sensed Context Inferred Context Presumed Context

Environment

Context Manager

Application

Adaptation Manager

Middleware

3rd-Party Libraries Rule Engine

  • M. Sama, D.S. Rosenblum, Z. Wang and S. Elbaum, “Multi-Layer Faults in the Architectures of Mobile,

Context-Aware Adaptive Applications”, Journal of Systems and Software,

  • Vol. 83, Issue 6, Jun. 2010, pp. 906–914.
slide-56
SLIDE 56

Validation of CAAAs

Environment

Context Manager

Application

Adaptation Manager

Middleware

Rule Engine

slide-57
SLIDE 57

Validation of CAAAs

Environment

Context Manager

Application

Adaptation Manager

Middleware

Rule Engine

Rules are strongly interdependent and have multiple priorities which makes reasoning difficult even for a small number of rules

slide-58
SLIDE 58

Validation of CAAAs

Environment

Context Manager

Application

Adaptation Manager

Middleware

3rd-Party Libraries

slide-59
SLIDE 59

Validation of CAAAs

Environment

Context Manager

Application

Adaptation Manager

Middleware

3rd-Party Libraries

Context is sensed periodically from multiple sources at varying rates

slide-60
SLIDE 60

Approach

  • 1. Derive Adaptation Finite-State Machine

(A-FSM) from rule logic

  • 2. Explore state space of A-FSM to discover

potential faults

✓Enumerative algorithms ✓Symbolic algorithms

  • 3. (Confirm existence of discovered faults)
  • M. Sama, S. Elbaum, F. Raimondi and D.S. Rosenblum, “Context-Aware Adaptive Applications: Fault Patterns and Their

Automated Identification”, IEEE Transactions on Software Engineering,

  • Vol. 36, No. 5, Sep./Oct. 2010, pp. 644-661.
slide-61
SLIDE 61

PhoneAdapter

slide-62
SLIDE 62

PhoneAdapter

normal, vibrate silent, vibrate loud, vibrate silent, divert to voicemail loud, divert to hands-free

slide-63
SLIDE 63

PhoneAdapter

normal, vibrate silent, vibrate loud, vibrate silent, divert to voicemail loud, divert to hands-free

slide-64
SLIDE 64

PhoneAdapter A-FSM

Office Driving Fast Meeting Driving Sync

General

Home Outdoor Jogging

slide-65
SLIDE 65

PhoneAdapter A-FSM

ActivateMeeting DeactivateMeeting

Office Driving Fast Meeting Driving Sync

General

Home Outdoor Jogging

slide-66
SLIDE 66

PhoneAdapter A-FSM

checking location implies GPS is on locations are mutually exclusive speeds monotonically increase a meeting’s end time is later than its start time

Global constraints:

ActivateMeeting DeactivateMeeting

Office Driving Fast Meeting Driving Sync

General

Home Outdoor Jogging

slide-67
SLIDE 67

Example Faults in PhoneAdapter

Office

General

Home

slide-68
SLIDE 68

Example Faults in PhoneAdapter

User’s phone discovers office PC at home (or vice versa)

Office

General

Home

slide-69
SLIDE 69

Example Faults in PhoneAdapter

Nondeterminism!

Office

General

Home

slide-70
SLIDE 70

Example Faults in PhoneAdapter

General

slide-71
SLIDE 71

Example Faults in PhoneAdapter

User leaves home

General

Outdoor

slide-72
SLIDE 72

Example Faults in PhoneAdapter

User starts driving before Bluetooth detects hands-free system

Driving

General

Outdoor

slide-73
SLIDE 73

Example Faults in PhoneAdapter

Activation hazard!

Driving

General

Outdoor Jogging

slide-74
SLIDE 74

Example Faults in PhoneAdapter

Activation hazard!

Driving

General

Outdoor Jogging

slide-75
SLIDE 75

Faults in CAAAs

  • Behavioral Faults

Nondeterminism Dead rule Dead state Unreachable state Activation race Activation cycle

slide-76
SLIDE 76

Faults in CAAAs

  • Behavioral Faults

Nondeterminism Dead rule Dead state Unreachable state Activation race Activation cycle

  • Hazards

Hold hazard Activation hazard Priority inversion hazard

slide-77
SLIDE 77

PhoneAdapter Results

Behavioral Faults: Enumerative, Symbolic

State Nondeterministic Dead Adaptation Unreachable Adaptations Predicates Races Cycles States General 37 1 45 13 Outdoor 3 135 23 Jogging 97 19 Driving 36 13 DrivingFast 58 19 Home 76 19 Office 29 1 Meeting 32 1 Sync 27 5 1

slide-78
SLIDE 78

PhoneAdapter Results

Hazards: Enumerative

s Context Hazards le Paths Hold Activ. Prior. 13 14085 11 3182 23 161 52 19 2 13 16 2 2 4 19 2 19 104 8 13 1 82634 1828 368 2164 1 5 2 2 State General Outdoor Jogging Driving DrivingFast Home Office Meeting Sync

slide-79
SLIDE 79

CAAAs

Summary

✓Rule-based CAAAs can be extremely fault-

prone, even with a small set of rules and context variables

✓The fault detection algorithms find many

actual faults, with different tradeoffs

✓Some alternative to rule-based adaptation

is needed ...

slide-80
SLIDE 80

CAMMR

Context-Aware Mobile Music Recommendation

✓Users’ short-term music needs are driven

by their current activity

✓Fully automated music recommendation

requires solving the cold-start problem: Which existing user will like a new song? Which existing songs will a new user like?

  • X. Wang, D.S. Rosenblum and
  • Y. Wang, “Context-Aware Mobile Music Recommendation for Daily

Activities”, Full Paper, Proc. ACM Multimedia 2012 (ACMMM 2012), Nara, Japan, Oct.–Nov. 2012, pp. 91–108.

slide-81
SLIDE 81

CAMMR

Functionality

slide-82
SLIDE 82

CAMMR

Functionality

slide-83
SLIDE 83

CAMMR

Functionality

slide-84
SLIDE 84

CAMMR

Functionality

slide-85
SLIDE 85

CAMMR

Key Characteristics

✓Real-time sensor-driven activity inference

Running, Walking, Sleeping, Working, Studying, Shopping

✓Offline low-level audio content analysis ✓Personalization of recommendations

slide-86
SLIDE 86

CAMMR

Supervised Learning

✓Machine learning, not handcrafted rules!

Ground truth: Activity: Manually tagged sensor streams Music: Activity-tagged Grooveshark playlists Coupled with incremental learning of individual preferences

slide-87
SLIDE 87

CAMMR

Architecture

Music& Database&

Back End Front End

slide-88
SLIDE 88

CAMMR

Architecture

Music& Database&

Running& Walking& Sleeping& Working&

Studying&

Shopping&

Binary&classifiers& (Adaboost)&

Binary Classifiers (Adaboost) Audio Feature Extraction

Back End Front End

slide-89
SLIDE 89

CAMMR

Architecture

Music& Database&

Sensor'signal' features'

Running& Walking& Sleeping& Working&

Studying&

Shopping&

Binary&classifiers& (Adaboost)&

Binary Classifiers (Adaboost) Audio Feature Extraction Sensor Stream Feature Extraction

Back End Front End

slide-90
SLIDE 90

CAMMR

Architecture

Music& Database&

Sensor'signal' features'

Classification Results

Running& Walking& Sleeping& Working&

Studying&

Shopping&

Binary&classifiers& (Adaboost)&

Binary Classifiers (Adaboost) Audio Feature Extraction Sensor Stream Feature Extraction Microphone, Accelerometer and Clock Features

Back End Front End

ACACF$ Probabilistic$Graphical$Model$ (Naive ¡Bayes)

slide-91
SLIDE 91

CAMMR

Architecture

Music& Database&

Sensor'signal' features'

Classification Results

Running& Walking& Sleeping& Working&

Studying&

Shopping&

Binary&classifiers& (Adaboost)&

Binary Classifiers (Adaboost) Audio Feature Extraction Recommendation Sensor Stream Feature Extraction Microphone, Accelerometer and Clock Features Music Play

Back End Front End

ACACF$ Probabilistic$Graphical$Model$ (Naive ¡Bayes)

slide-92
SLIDE 92

CAMMR

Architecture

Music& Database&

Sensor'signal' features'

Classification Results

Running& Walking& Sleeping& Working&

Studying&

Shopping&

Binary&classifiers& (Adaboost)&

Binary Classifiers (Adaboost) Audio Feature Extraction Recommendation User Feedback Sensor Stream Feature Extraction Microphone, Accelerometer and Clock Features Music Play

Back End Front End

ACACF$ Probabilistic$Graphical$Model$ (Naive ¡Bayes)

slide-93
SLIDE 93

Results

Inter-Subject Agreement on Music Preferences

  • 10 subjects
  • Manual activity tagging of 1200

Grooveshark and YouTube songs

  • p < 0.0001

Activity Kappa Agreement Percent Agreement Running Working Sleeping Walking Shopping Studying

0.27 0.35 0.03 0.02 0.29 0.28 0.03 0.03 0.07 0.17 0.09 0.11

slide-94
SLIDE 94
  • 10 subjects, 6 activities, 30 minutes/session
  • Naive Bayes provides very good precision

and efficiency for smartphones

Results

Precision of Activity Inference

Activity AdaBoost C4.5 LR NB SVM KNN Running Working Sleeping Walking Shopping Studying OVERALL

0.974 0.976 0.975 0.841 0.974 0.970 0.933 0.932 0.921 0.876 0.929 0.922 0.999 0.999 0.999 0.994 0.999 0.993 0.961 0.960 0.955 0.909 0.960 0.953 0.972 0.972 0.948 0.953 0.965 0.955 0.854 0.867 0.835 0.694 0.860 0.855 0.951 0.952 0.941 0.893 0.950 0.943

slide-95
SLIDE 95
  • 10 subjects, 6 activities, 30 minutes/session
  • Naive Bayes provides very good precision

and efficiency for smartphones

Results

Precision of Activity Inference

Activity AdaBoost C4.5 LR NB SVM KNN Running Working Sleeping Walking Shopping Studying OVERALL

0.974 0.976 0.975 0.841 0.974 0.970 0.933 0.932 0.921 0.876 0.929 0.922 0.999 0.999 0.999 0.994 0.999 0.993 0.961 0.960 0.955 0.909 0.960 0.953 0.972 0.972 0.948 0.953 0.965 0.955 0.854 0.867 0.835 0.694 0.860 0.855 0.951 0.952 0.941 0.893 0.950 0.943

slide-96
SLIDE 96
  • Precision@K for top K songs
  • Baselines are random rankings

Results

Retrieval Performance

slide-97
SLIDE 97
  • 10 subjects, divided into experimental and control group
  • R2 vs R1: p = 0.0478
  • R3 vs R1: p = 0.0001
  • R3 vs R2: p = 0.1374

Results

Accuracy of Music Recommendation

1" 1.5" 2" 2.5" 3" 3.5" 4" 4.5" 5" Baseline" Automatic"mode" Manual"mode" Average'Rating'

5-point Likert scale

Traditional CAMMR CAMMR Auto Mode Manual Mode (R1) (R2) (R3)

slide-98
SLIDE 98
  • 2 subjects, continuous usage for one week

Results

Effectiveness of Incremental Adaptation

0" 0.1" 0.2" 0.3" 0.4" 0.5" 0.6" 0.7" 0.8" 0.9" 1" Context"Inference" Recommendation" Before"Adaptation" After"Adaptation"

Precision

slide-99
SLIDE 99

CAMMR

Summary

✓CAMMR is the first automated solution for

short-term music listening needs

✓Provides a complete solution to the cold-

start problem

✓Employs machine learning for more robust

adaptation

slide-100
SLIDE 100

Other Projects

Emotion Sensing Valence Arousal

+

⬆

–

⬇

slide-101
SLIDE 101

Other Projects

Emotion Sensing Valence Arousal

+

⬆

–

⬇

slide-102
SLIDE 102

Other Projects

Emotion Sensing

  • Sensing from Mobile & Wearable Sensors

Microphone: speech Camera: facial expressions, eye tracking Accelerometer: movement, orientation MS Kinect: gesture GPS: location GSR: skin conductivity HRM: pulse

slide-103
SLIDE 103

Other Projects

Emotion Sensing

  • Many Obstacles and Limitations

Lack of empirical evidence for biological signatures of emotions Much variability in experiencing emotions

➡ Short-term situations vs. long-term mood ➡ Between and within cultures and languages ➡ By the same individual

Difficulty of inducing spontaneous, genuine emotion in controlled experimental settings

slide-104
SLIDE 104

Other Projects

Emotion Sensing

  • Research Agenda

✓Multimodal sensing of core affect ✓Contextualization of emotion sensing ✓Computational platform for sensing and

processing

✓Realistic empirical study designs

slide-105
SLIDE 105

Felicitous Computing

Software Engineering Challenges

slide-106
SLIDE 106

Felicitous Computing

Software Engineering Challenges

“There are known knowns; there are things we know we know. We also know there are known unknowns; that is to say, we know there are some things we do not know. But there are also unknown unknowns – the ones we don’t know we don’t know.”

— Donald Rumsfeld

slide-107
SLIDE 107

Felicitous Computing

Software Engineering Challenges

Validation of ubiquitous computing systems is riddled with uncertainty

➡ Unpredictable ambient environments ➡ Imprecision of context inference ➡ Disagreement among users and/or observers ➡ Slippery slope between “known unknowns” and faults

Similar in spirit (but not in character) to Weyuker’s “non-testable programs” May need to employ relative quality comparison of systems rather than absolute quality assessment

slide-108
SLIDE 108

Conclusion

✓ The technology for felicitous computing is here ✓ But building felicitous computing systems

remains challenging

✓ We still lack a clear body of design and

engineering principles

✓ At NUS we are pursuing research

breakthroughs to make felicitous computing an integrated and invisible part of people’s lives

slide-109
SLIDE 109

Felicitous Computing

David S. Rosenblum School of Computing National University of Singapore

Thank you!