Mobile Video Quality Assessment Database Lark Kwon Choi with Anush - - PowerPoint PPT Presentation

▶
mobile video quality assessment database
SMART_READER_LITE
LIVE PREVIEW

Mobile Video Quality Assessment Database Lark Kwon Choi with Anush - - PowerPoint PPT Presentation

Feb 02, 2024 166 likes •364 views

Mobile Video Quality Assessment Database Lark Kwon Choi with Anush K. Moorthy, Prof. Alan C. Bovik, and Prof. Gustavo de Veciana Outline Introduction LIVE Mobile VQA Database Subjective Study Source videos and distortion simulation Test

slide-1
SLIDE 1

Mobile Video Quality Assessment Database

Lark Kwon Choi

with Anush K. Moorthy, Prof. Alan C. Bovik, and

  • Prof. Gustavo de Veciana
slide-2
SLIDE 2

2

Introduction LIVE Mobile VQA Database Subjective Study

Source videos and distortion simulation Test methodology

Evaluation of Subjective Opinion Discussion and Conclusion

Outline

slide-3
SLIDE 3

3

Growth in Mobile Video Traffic

More devices, higher bit rates contents

78 % Global Mobile Data Traffic Growth by 2016

Mobile Video Traffic 70.5 %

How?

Need for more video capacity, viewer’s QoE

slide-4
SLIDE 4

4

Promising Direction

“Perceptual optimization” of video networks

Humans are final “receivers” of videos

Video Compression

Wireless Networks

Transmission Visual Perception

Feedback

To understand human’s opinion and behavior on visual quality,

HD Mobile VQA Database & Subjective Analysis

slide-5
SLIDE 5

5

Previous Subjective Studies

Subjective studies Focus Limits [K. Seshadrinathan et al., 2010] [A.K. Moorthy et al., 2010] [VQEG VQA Phase I and II, 2000, 2003] [VQEG Multimedia Phase I, 2008]

  • Large displays
  • Distortion:

Compression, IP/wireless loss

  • Results cannot be

translated into small mobile devices [S.R. Gulliver et al., 2007] [Q. Huynh‐Thu et al., 2006]

  • Delayed and

jitter [A. Eichhorn et al., 2009] [H. Knoche et al., 2005] [S. Jumisko‐Pyykko et al., 2005, 2008] [M. Ries et al., 2007] [S. Winkler et al., 2003]

  • Mobile devices
  • Small datasets
  • Insufficient distortions
  • Unknown source
  • Small resolution
  • Lack of publicity

LIVE Mobile VQA Database

To aid the development of perceptually optimized algorithms for wireless video transmission,

slide-6
SLIDE 6

10 Reference, 200 distorted videos, and >50 subjects

slide-7
SLIDE 7

7

Source Videos

Digital Cinematographic Camera

12bit REDCODE RAW 2K (2048x1152), 30/60 fps

RED ONE

10 Actual Study 2 Training Downsampled 720p (1280x720), Uncompressed YUV, 15 sec, Reference videos

slide-8
SLIDE 8

8

Distortion Simulations

H.264 Compression Frame freezes Rate adaptation Temporal dynamics Wireless Packet Loss

slide-9
SLIDE 9

9

Distortions

4 Compression + 4 Wireless Packet Loss

‐ JM H.264 SVC / R1 < R2 < R3 < R4 / 0.7 ~ 6M using fixed QP encoding Perceptual separation

  • f video quality

source rate SVC layers R1 R2 R3 R4 Q1 Q2 Q3 Q4 quality

IEEE 802.11 Wireless channel Simulator

‐‐‐‐‐‐‐‐‐‐‐‐‐‐ QAM OFDM modulation

slide-10
SLIDE 10

10

Time Varying Distortions

4 Frame freezes 3 Rate adaptation 5 Temporal dynamics

1sec short stored video freezing 4sec long stored video freezing 4sec long LIVE video freezing

Time (15sec) Layers (R1 ~ R4)

slide-11
SLIDE 11

11

Test Methodology

Single‐stimulus continuous quality evaluation (SSCQE) with hidden‐reference Phone (Motorola Atrix)

‐ 200 videos, 36 subjects, 18 ratings, 2 rejects

Real‐time quality evaluation End‐of‐video quality evaluation

slide-12
SLIDE 12

12

Result of Subjective Study

DMOS scores and corresponding histogram

We assume that the DMOS scores have a Gaussian distribution

slide-13
SLIDE 13

Evaluation of Subjective Opinion

Statistical analysis of human behavior

slide-14
SLIDE 14

14

Compression

Conducted t‐test by using DMOS values

‐ 95 % confidence level ‐ compared in separate contents

R1 R2 R3 R4 R1

‐ ‐ ‐ ‐ ‐ ‐ ‐ ‐ ‐ ‐ 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

R2

1 1 1 1 1 1 1 1 1 1 ‐ ‐ ‐ ‐ ‐ ‐ ‐ ‐ ‐ ‐ 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

R3

1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 ‐ ‐ ‐ ‐ ‐ ‐ ‐ ‐ ‐ ‐ 0 0 0 0 0 0 0 0 0 0

R4

1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 ‐ ‐ ‐ ‐ ‐ ‐ ‐ ‐ ‐ ‐

  • ‘ 1 ’ : row algorithm is ’statistically better’ than the column algorithm
  • ‘ 0 ’ : ‘worse’ and ‘ ‐ ’ : ‘ identical ’
  • Each entry of the matrix represent 10 reference videos

Higher rate is better

slide-15
SLIDE 15

15

Frame Freeze

Choppy freezing is worse Frame lost is worse

Freeze – 1sec Freeze – 2sec Freeze – 4sec Real‐time Freeze 4 sec Freeze – 1sec

‐ ‐ ‐ ‐ ‐ ‐ ‐ ‐ ‐ ‐ 0 0 0 ‐ 0 0 0 ‐ 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

Freeze – 2sec

1 1 1 ‐ 1 1 1 ‐ 1 1 ‐ ‐ ‐ ‐ ‐ ‐ ‐ ‐ ‐ ‐ 0 0 0 0 0 0 0 0 0 0 0 0 0 ‐ 1 ‐ 1 0 ‐ 0

Freeze – 4sec

1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 ‐ ‐ ‐ ‐ ‐ ‐ ‐ ‐ ‐ ‐ 1 1 1 1 1 1 1 1 1 1

Real‐time Freeze 4 sec

1 1 1 1 1 1 1 1 1 1 1 1 1 ‐ 0 ‐ 0 1 ‐ 1 0 0 0 0 0 0 0 0 0 0 ‐ ‐ ‐ ‐ ‐ ‐ ‐ ‐ ‐ ‐

1sec short stored video freezing 4sec long stored video freezing 4sec long LIVE video freezing

slide-16
SLIDE 16

16

Rate Adaptation

R1 ‐ R4 ‐ R1 R2 ‐ R4 ‐ R2 R3 ‐ R4 ‐ R3 R1 ‐ R4 ‐ R1

‐ ‐ ‐ ‐ ‐ ‐ ‐ ‐ ‐ ‐ 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

R2 ‐ R4 ‐ R2

1 1 1 1 1 1 1 1 1 1 ‐ ‐ ‐ ‐ ‐ ‐ ‐ ‐ ‐ ‐ 0 0 0 0 0 0 0 0 0 0

R3 ‐ R4 ‐ R3

1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 ‐ ‐ ‐ ‐ ‐ ‐ ‐ ‐ ‐ ‐

Higher rate is better Average stable constant rate is better

R1 R2 R3 R4 R1 ‐ R4 ‐ R1

1 1 1 1 1 1 1 1 1 1 0 0 0 ‐ 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

R2 ‐ R4 ‐ R2

1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

R3 ‐ R4 ‐ R3

1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1‐ 0 1 ‐ 0 1 0 0 0 0 0 0 0 0 0 0 0

slide-17
SLIDE 17

17

Temporal Dynamics

R1 ‐ R4 ‐ R1 R1 ‐ R4 ‐ R1 ‐ R4 ‐ R1 R1 ‐ R4 ‐ R1

‐ ‐ ‐ ‐ ‐ ‐ ‐ ‐ ‐ ‐ 0 ‐ ‐ ‐ 0 0 0 0 1 ‐

R1 ‐ R4 ‐ R1 ‐ R4 ‐ R1

1 ‐ ‐ ‐ 1 1 1 1 0 ‐ ‐ ‐ ‐ ‐ ‐ ‐ ‐ ‐ ‐

Multiple rate switches are better Ending on a higher quality is preferable

R1‐R4‐R1‐R4‐R1 R1‐R2‐R4 R4‐R2‐R1 R1‐R3‐R4 R4‐R3‐R1 R1‐R4‐R1‐R4‐R1

‐ ‐ ‐ ‐ ‐ ‐ ‐ ‐ ‐ ‐ ‐ 0 0 0 1 1 0 0 0 0 1 1 ‐ 1 1 1 1 1 1 1 0 0 0 0 ‐ 0 0 0 0 0 1 1 0 0 1 1 1 1 1 1

R1‐R2‐R4

‐ 1 1 1 0 0 1 1 1 1 ‐ ‐ ‐ ‐ ‐ ‐ ‐ ‐ ‐ ‐ 1 1 1 1 1 1 1 1 1 1 0 0 ‐ 0 0 0 ‐ 0 0 0 1 1 1 ‐ 1 1 1 1 1 1

R4‐R2‐R1

0 0 ‐ 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ‐ ‐ ‐ ‐ ‐ ‐ ‐ ‐ ‐ ‐ 0 0 0 0 0 0 0 0 0 0 1 ‐ 0 0 0 0 ‐ 0 1 0

R1‐R3‐R4

1 1 1 1 1 1 1 1 1 1 1 1 1 ‐ 0 ‐ 0 1 ‐ 1 0 0 0 0 0 0 0 0 0 0 ‐ ‐ ‐ ‐ ‐ ‐ ‐ ‐ ‐ ‐ 1 1 1 1 1 1 1 1 1 1

R4‐R3‐R1

0 0 1 1 0 0 0 0 0 0 0 0 ‐ 0 0 0 0 0 0 0 ‐ 1 1 1 1 ‐1 0 1 0 0 0 0 0 0 0 0 0 0 ‐ ‐ ‐ ‐ ‐ ‐ ‐ ‐ ‐ ‐

slide-18
SLIDE 18

18

Discussion and Conclusion

LIVE Mobile VQA Database

10 reference, 200 distorted videos, and Over 50 subjects

4 compression + 4 wireless packet loss + 4 frame freezes + 3 rate adaptation + 5 temporal dynamics

Invite further analysis of human behavior Analysis of human behavior

Higher, stable bit rates, and multiple efforts Getting better (ending) quality Choppy freezing, frame lost

slide-19
SLIDE 19

19

Questions ?

A.K. Moorthy, L‐K. Choi, A.C. Bovik, and G. de Veciana, “Video Quality Assessment on Mobile Devices: Subjective, Behavioral and Objective Studies,” IEEE Journal of Selected Topic in Signal Processing. Special Issue on New Subjective and Objective Methodologies for Audio and Visual Signal Processing, 2011. (re‐submitted after revision)

For more explanations, Acknowledgment ‐ NSF CCF‐0728748 ‐ Intel & Cisco VAWN program