Maximum Differentiation Competition: Direct Comparison of - - PowerPoint PPT Presentation

▶
maximum differentiation competition direct comparison of
SMART_READER_LITE
LIVE PREVIEW

Maximum Differentiation Competition: Direct Comparison of - - PowerPoint PPT Presentation

Jun 12, 2023 123 likes •435 views

Maximum Differentiation Competition: Direct Comparison of Discriminability Models Zhou Wang & Eero P. Simoncelli Howard Hughes Medical Institute, Center for Neural Science, and Courant Institute for Mathematical Sciences New York

slide-1
SLIDE 1

Wang & Simoncelli, VSS-2005

Maximum Differentiation Competition: Direct Comparison of Discriminability Models

Zhou Wang & Eero P. Simoncelli

Howard Hughes Medical Institute, Center for Neural Science, and Courant Institute for Mathematical Sciences New York University

slide-2
SLIDE 2

Which model best accounts for perceived image quality? Image Quality Assessment

reference distorted

Wang & Simoncelli, VSS-2005

slide-3
SLIDE 3

Which model best accounts for perceived image quality? Image Quality Assessment

reference distorted

Wang & Simoncelli, VSS-2005

slide-4
SLIDE 4

Which model best accounts for perceived image quality? Image Quality Assessment

SSIM MSE

reference distorted

Wang & Simoncelli, VSS-2005

slide-5
SLIDE 5

Wang & Simoncelli, VSS-2005

MSE: Mean Squared Error SSIM: Structural Similarity [Wang, et. al. ‘04]

– local cross-correlation measure: – pooling

Example Models

where

s(x, y) = (2µxµy + C1)(2σxy + C2) (µ2

x + µ2 y + C1)(σ2 x + σ2 y + C2)

S(X, Y) =

  • i w(xi, yi)s(xi, yi)
  • i w(xi, yi)

w(x, y) = log2(1 + σ2

x/C) + log2(1 + σ2 y/C)

E(X, Y) = 1 N

  • i

(xi − yi)2

slide-6
SLIDE 6

Wang & Simoncelli, VSS-2005

  • Procedure
  • Difficulties

– Subjective experiments expensive – “Curse of dimensionality”: impossible to cover image space

Conventional Method

  • 1. Choose set of reference and distorted images
  • 2. Perform subjective tests
  • 3. Compare model prediction with subjective responses
slide-7
SLIDE 7

Wang & Simoncelli, VSS-2005

Distortion: JP2(1) JP2(2) JPG(1) JPG(2) Noise Blur Error # images: 87 82 87 88 145 145 145 MSE 0.934 0.895 0.902 0.914 0.987 0.774 0.881 SSIM 0.968 0.967 0.965 0.986 0.971 0.936 0.944

Conventional Method: MSE vs. SSIM

SSIM Mean Subject Rating Mean Subject Rating

  • log(MSE)

“LIVE” image database, UT Austin

slide-8
SLIDE 8

Wang & Simoncelli, VSS-2005

Distortion: JP2(1) JP2(2) JPG(1) JPG(2) Noise Blur Error # images: 87 82 87 88 145 145 145 MSE 0.934 0.895 0.902 0.914 0.987 0.774 0.881 SSIM 0.968 0.967 0.965 0.986 0.971 0.936 0.944

Conventional Method: MSE vs. SSIM

SSIM Mean Subject Rating Mean Subject Rating

  • log(MSE)
slide-9
SLIDE 9

Wang & Simoncelli, VSS-2005

Proposed Method: MAximum Differentiation (MAD) Competition

slide-10
SLIDE 10

Wang & Simoncelli, VSS-2005

Proposed Method: MAximum Differentiation (MAD) Competition

  • Let two models compete
slide-11
SLIDE 11

Wang & Simoncelli, VSS-2005

Proposed Method: MAximum Differentiation (MAD) Competition

  • Let two models compete
  • ... by synthesizing optimal stimuli
slide-12
SLIDE 12

Wang & Simoncelli, VSS-2005

Proposed Method: MAximum Differentiation (MAD) Competition

  • Let two models compete
  • ... by synthesizing optimal stimuli
  • ... that maximally differentiate the models
slide-13
SLIDE 13

Wang & Simoncelli, VSS-2005

Geometric Description in Image Space

slide-14
SLIDE 14

Wang & Simoncelli, VSS-2005

Geometric Description in Image Space

all images with same MSE

slide-15
SLIDE 15

Wang & Simoncelli, VSS-2005

all images with same SSIM

reference image

Geometric Description in Image Space

slide-16
SLIDE 16

Wang & Simoncelli, VSS-2005

Geometric Description in Image Space

worst MSE

reference image

slide-17
SLIDE 17

Wang & Simoncelli, VSS-2005

Geometric Description in Image Space

best MSE worst MSE

reference image

slide-18
SLIDE 18

Wang & Simoncelli, VSS-2005

Geometric Description in Image Space

best SSIM

slide-19
SLIDE 19

Wang & Simoncelli, VSS-2005

Geometric Description in Image Space

worst SSIM

slide-20
SLIDE 20

Wang & Simoncelli, VSS-2005

Geometric Description in Image Space

reference image

slide-21
SLIDE 21

Wang & Simoncelli, VSS-2005

MAD Competition: MSE vs. SSIM

reference add noise

slide-22
SLIDE 22

Wang & Simoncelli, VSS-2005

reference best SSIM worst SSIM

MAD Competition: MSE vs. SSIM

slide-23
SLIDE 23

Wang & Simoncelli, VSS-2005

reference best MSE worst MSE

MAD Competition: MSE vs. SSIM

slide-24
SLIDE 24

Wang & Simoncelli, VSS-2005

reference best SSIM worst SSIM best MSE worst MSE

MAD Competition: MSE vs. SSIM

slide-25
SLIDE 25

Wang & Simoncelli, VSS-2005

  • Subjects: 5 (4 naïve, 1 author)
  • Images: 10 reference, viewed at 16 pixels/degree
  • Trials: 20 per distortion-level per subject

2AFC Experiment

distortion level (MSE) 22

23

24 25 26 27 28

initial image best SSIM worst SSIM

slide-26
SLIDE 26

Wang & Simoncelli, VSS-2005

  • Subjects: 5 (4 naïve, 1 author)
  • Images: 10 reference, viewed at 16 pixels/degree
  • Trials: 20 per distortion-level per subject

2AFC Experiment

distortion level (MSE) 22

23

24 25 26 27 28

initial image best SSIM worst SSIM

slide-27
SLIDE 27

Wang & Simoncelli, VSS-2005

Psychometric Functions

% correct initial distortion level (MSE)

best/worst SSIM best/worst MSE

slide-28
SLIDE 28

Wang & Simoncelli, VSS-2005

Psychometric Functions

all 5 subjects chose top 1 chose top twice 2 chose bottom twice 2 gave 1-1 tie best/worst SSIM best/worst MSE

% correct initial distortion level (MSE)

slide-29
SLIDE 29

Wang & Simoncelli, VSS-2005

Summary

  • MAximum Differentiation (MAD) Competition

– Let two models compete – ... by synthesizing optimal stimuli – ... that maximally differentiate the models

  • Advantages

– Optimized images maximize opportunity for model failure – Efficient (minimal # of 2-alternative comparisons) – Images reveal model weaknesses => potential improvements

  • To Do

– Full experiment, with more reference images – Application to other discriminable quantities – Physiology