A System for Speech and 3D Facial Image Acquisition, Modeling and - - PowerPoint PPT Presentation

Elmar Nöth, Tobias Bocklet, Arnd Gebhard

A System for Speech and 3D Facial Image Acquisition, Modeling and Analysis

Wednesday, 30 May 2012

Outline

• Motivation: Long-term goal of the project
• Patient groups:
• Parkinson’s disease (PD) patients
• Stroke patients and patients with facial paresis
• Speech technology
• Facial analysis technology
• Results
• Summary

Necessity of Evaluation

• Diagnosis
• How intelligible is the patient?

(holistic impression)

• How strongly does the patient nasalize?

(distinct aspect)

• Therapy control
• Has the situation of the patient improved during therapy?
• Comparison of therapy methods
• Which therapy method leads to the best results for a group of

patients?

• Screening
• Is the quality of a child’s speech according to its age?
• Computer-assisted therapy
• Did the patient perform the exercise correctly?

Motivation

Necessity of Evaluation

• Diagnosis
• How intelligible is the patient?

(holistic impression)

• How strongly does the patient nasalize?

(distinct aspect)

• Therapy control
• Has the situation of the patient improved during therapy?
• Comparison of therapy methods
• Which therapy method leads to the best results for a group of

patients?

• Screening
• Is the quality of a child’s speech according to its age?
• Computer-assisted therapy
• Did the patient perform the exercise correctly?

Motivation

Long-term Goal of the Project

• Provide a telemedical rehabilitation unit for clinical/home use
• Support speech analysis and analysis of facial gestures and …

(gait, cognitive abilities  open, flexible platform)

• Patient groups:
• Parkinson’s disease (PD) patients
• Stroke patients and patients with facial paresis
• Instruct the patient what to do
• Evaluate the exercises
• Compare with previous sessions
• Summarize exercises for therapist

Motivation

Outline

• Long-term goal of the project
• Patient groups:
• Parkinson’s disease (PD) patients
• Stroke patients and patients with facial paresis
• Speech technology
• Facial analysis technology
• Results
• Summary

Parkinson’s Disease

• Degenerative disorder of the central nervous system
• Death of dopamine-containing cells in the substantia nigra
• Cause of cell-death is unknown
• Second most common neurodegenerative disorder

(after Alzheimer's disease)

• Prevalence ≈ 0.3% (whole population)
• More common in the elderly:

1% of > 60 years, 4% of > 80 years

• Incidence of PD ≈ 8 - 18 per 100,000 people
• Onset in most cases > 50 years, mean onset ≈ 60 years

Patient Groups

Speech-related Symptoms of PD

• Hypophonia (soft speech)
• Monotonic speech: Speech quality tends to be soft, hoarse, and

monotonous

• Festinating speech: excessively rapid, soft, poorly-intelligible

speech

• Drooling: most likely caused by a weak, infrequent swallow
• Dysphagia (impaired ability to swallow)
• Dysarthria

Patient Groups

Dysarthria

• A speech disorder affecting the coordination of muscles in the

vocal tract, face, larynx, and respiratory system (dysarthrophonia)

• Mostly results from a neurological injury,

such as a stroke or other kind of brain injury

Patient Groups

Dysarthria

• A speech disorder affecting the coordination of muscles in the

vocal tract, face, larynx, and respiratory system (dysarthrophonia)

• Mostly results from a neurological injury,

such as a stroke or other kind of brain injury

Patient Groups

Outline

• Long-term goal of the project
• Patient groups:
• Parkinson’s disease (PD) patients
• Stroke patients and patients with facial paresis
• Speech technology
• Facial analysis technology
• Results
• Summary

Speech Technology

• Automatic speech processing methods
• Word and phoneme recognition
• Acoustic speaker modeling
• Prosodic analysis
• Model of excitation signal
• Evaluation measures

features

classi- fication words with highest probability

word chain

Word and Phoneme Recognition

• Off-the-shelf technology
• Semi-continuous HMMs
• Easier to adapt with small amounts of data
• Comparable results with continuous models
• 11 Mel cepstrum coefficients + energy + 1. derivative

Speech Technology

Acoustic Speaker Modeling

• Idea:
• Acoustic space of speakers can be modeled
• Space represents the multidimensional characteristics of voice of a

speaker

• Degree of pathology varies in acoustic space
• Find characteristics of degree of speech disorder
• Approach:
• Acoustics modeled by Gaussian Mixture Models (GMMs)
• Train Universal Background Model (UBM) with normal speakers
• Train GMM of path. speakers and transform into vector
• Perform a classification/regression (depends on the task)

Speech Technology

Acoustic Speaker Modeling

Gaussian density of UBM feature dimension 1 feature dimension 2 features of healthy speakers Gaussian density of speaker model features of a path. speaker

• Variations of speakers with different degrees of pathology
• Can be modeled by adaptation from UBM to GMM

Speech Technology

Gaussian densities (i = 1,.., N) of speaker model defined by mean values(mi) und covariance matrices (Ki)

feature dimension 1 feature dimension 2

Concatenation

• f elements of densities

m1 m2 m3 m4 m5 m6 K1 K2 K3 K4 K5 K6

ms = Ks =

m1 K1 m2 K2 m3 K3 m4 K4 m5 K5 m6 K6

Acoustic Speaker Modeling

Speech Technology

Acoustic Speaker Modeling

speakers with pathology type 2

points correspond to supervectors (SVs)

speakers with pathology type 1

• Discriminate between different types of pathology
• Create SVs of speakers
• Train some classifier on labeled SVs
• Create SV of test speaker
• Classify SV of test speaker

Speech Technology

Acoustic Speaker Modeling

supervector space

Train a regression (linear/SVR) Create SV for a test speaker Estimate degree of pathology

degree of pathology

• Estimate degree of pathology

Speech Technology

Prosodic Analysis

• Prosody: rhythm, intonation, stress, and related attributes
• Computation of prosodic features on word level, across several words or

across syllable nuclei or across voiced segments

• Computation across several words requires ASR
• Computation across syllable nuclei requires syllable detection
• Local features:
• Pauses before/after segments, signal energy,

segment duration, and F0

• Calculation of mean, max., min., and std. dev.
• Global features: jitter, shimmer, voiced/unvoiced characteristics

 ≈100-200 features per test utterance

Speech Technology

Two-Mass Model of the Vocal Folds

Speech Technology

Two-Mass Model of the Vocal Folds

Speech Technology

Evaluation

• Word accuracy (WA) and word correctness (WC)
• Calculated features
• Features of acoustic speaker models
• Features of prosodic analysis
• Features of 2-mass model
• Correlation (Pearson & Spearman) based on calculated

features or WA, WC with human listener

• Classification based on calculated features
• Interpretation of relevant features after feature selection

Speech Technology

Outline

• Long-term goal of the project
• Patient groups:
• Parkinson’s disease (PD) patients
• Stroke patients and patients with facial paresis
• Speech technology
• Facial analysis technology
• Results
• Summary

• PD: Increasing inability to express emotions with facial gestures

(important for communication)

• Dysarthric speech often accompanied by other physical impairments
• Facial paresis
• Motor handicaps
•  Analysis of facial gestures
• Reduced mobility requires therapist to come to patient
• High costs
• Waste of therapist’s time

 Telemedical therapy

Analysis of Facial Gestures

Facial Analysis Technology

Anger vs. Joy

Facial Analysis Technology

Showing Emotions

Reduced Ability to Vary Facial Expressions with PD

Unstressed look Lip pursing Closing of eyes Showing the teeth

Ability to Analyze Sequence of Movements

Dynamic Facial Expressions for Facial Paresis

Grading of Facial Paresis

Facial Analysis Technology

• Different Grading Systems are used
• Most prominent: Grading System by House&Brackmann

[J. House and D. Brackmann: Facial nerve grading system in Otolaryngolocical Head and Neck Surgery, 1985]

• 6 Grades:
• House I

→ healthy person

• House VI

→ completely paralyzed half of the patient's face

• Grading is performed on (subjective) observations by expert
• Problem: Objective tracking of cure processes
• Solution: Automatic System for diagnosis support

3D Camera: Principle

Dynamic Analysis of Facial Gestures

Time-of-Flight (ToF) 3D Camera

• Up to 50 Hz
• More than 25k 3D points

(176*144 pixels)

• Eye-safe infrared light /

no exposure

• Precision for facial images
• 40 cm: +/-

1mm

• 80 cm: +/-

5mm

• 120 cm: +/- 15mm

Dynamic Analysis of Facial Gestures

Dynamic Analysis of Facial Gestures

Principles of Kinect

Dynamic Analysis of Facial Gestures

Principles of Kinect

Control image for the patient

Prototype

Illumination Stereo microphones TOF camera Webcam

Dynamic Analysis of Facial Gestures

Telemedical System

Framework

Measuring the Precision

Measuring the Precision

Dynamic Parameters: Interface for Therapist

Dynamic Parameters: Interface for Therapist

Dynamic Parameters: Interface for Therapist

Dynamic Parameters: Interface for Therapist

Dynamic Parameters: Interface for Therapist

Outline

• Long-term goal of the project
• Patient groups:
• Parkinson’s disease (PD) patients
• Stroke patients and patients with facial paresis
• Speech technology
• Facial analysis technology
• Results
• Summary

Hoehn and Yahr Scale

• Stage 0:

No signs of disease.

• Stage 1:

Unilateral symptoms only

• Stage 1.5:

Unilateral and axial involvement

• Stage 2:

Bilateral symptoms; no impairment of balance

• Stage 2.5:

Mild bilateral disease with recovery on pull test

• Stage 3:

Balance impairment; mild to moderate disease physically independent

• Stage 4:

Severe disability, still able to walk or stand unassisted

• Stage 5:

Needing a wheelchair or bedridden unless assisted

Parkinson’s Disease

Czech Speech Data

Speech Analysis for Parkinson’s Disease

Classification PD vs. Control Group

Speech Analysis for Parkinson’s Disease

• 46 Czech speakers
• 23 with PD (Hoehn & Yahr 1-2)
• 23 as control group
• Age matched

ROC-Curve for T4 and Prosody

Speech Analysis for Parkinson’s Disease

Screening of 100 000 people >= 60  ≈ 1000 people with PD 1% more people found developing PD ≈ 10 people 1% more FA of people w/o PD ≈ 1000 people  Need for cheap robust screening (e.g. automatic telephone system) + more detailed screening + detailed exam %PD % FA

Intelligibility

• “The North Wind and the Sun”:
• 107 words (71 disjoint)
• Contains all German phonemes
• Commonly used by speech therapists and

phoneticians

• 28 patients with dysarthria
• recorded during post-stroke rehabilitation
• 39 to 76 years old
• Speech Technology: word recognition, prosody

Speech Analysis for Dysarthric Speech

1 Rater vs. Avg. of Other 3 Raters

Speech Analysis for Dysarthric Speech

Human raters vs. Word Recognition

Correlation r = −0. 84 Speech Analysis for Dysarthric Speech

Automatic Graduation of Paresis (2001)

Speech Analysis for Dysarthric Speech

4 grades of facial paresis were defined:

• G1: healthy person (corresp. to House I)
• G2: weak paresis (corresp. to House II + III)
• G3: strong paresis (corresp. to House IV + V)
• G4: paralysis (corresp. to House VI)

Result:

New Study of Patients with Paresis (2011)

Speech Analysis for Dysarthric Speech

• In 2011 a new study of patients with facial paresis was started
• Cooperation with HNO-Clinic Erlangen
• Goal: > 10 patients of every House class to be acquired and

analyzed

• Current state: 15 patients of different classes acquired

Gait Analysis – Sensor Platform

• Acceleration sensors
• Gyroscopes
• Wireless transmission of the sensor data

Gait Analysis – Sensor Platform

Gait Analysis: 10 m Walking (4 Repetitions)

Summary

• Provide a telemedical rehabilitation unit for clinical/home use
• Patient groups:
• Parkinson’s disease (PD) patients
• Stroke patients and patients with facial paresis
• Speech technology
• Facial analysis technology (still images and sequences)
• Results
• Up to 91% classification PD vs. control group & 0.97 AUC
• Correlation of -0.84 between automatic classification and

human raters for dysarthric speech using WC & prosody

Thank you for your attention

Supported by Deutsche Forschungsgemeinschaft (DFG) Wilhelm-Sander-Stiftung noeth@ informatik.uni-erlangen.de