Leveraging Frequency Analysis for Deep Fake Image Recognition Joel - - PowerPoint PPT Presentation

▶
leveraging frequency analysis for deep fake image
SMART_READER_LITE
LIVE PREVIEW

Leveraging Frequency Analysis for Deep Fake Image Recognition Joel - - PowerPoint PPT Presentation

Oct 03, 2022 225 likes •594 views

Leveraging Frequency Analysis for Deep Fake Image Recognition Joel Frank , Thorsten Eisenhofer, Lea Schnherr, Asja Fischer, Dorothea Kolossa, Thorsten Holz Which Face is Real? Which Face is Real? Which Face is Real? N 1 1 N 2 1 I x ,

slide-1
SLIDE 1

Leveraging Frequency Analysis for Deep Fake Image Recognition

Joel Frank, Thorsten Eisenhofer, Lea Schönherr, Asja Fischer, Dorothea Kolossa, Thorsten Holz

slide-2
SLIDE 2

Which Face is Real?

slide-3
SLIDE 3

Which Face is Real?

slide-4
SLIDE 4

Which Face is Real?

Dkx,ky =

N1−1

∑

x=0 N2−1

∑

y=0

Ix,y cos[ π N1 (x + 1 2)kx] cos[ π N2 (y + 1 2 )ky] .

slide-5
SLIDE 5

Which Face is Real?

Dkx,ky =

N1−1

∑

x=0 N2−1

∑

y=0

Ix,y cos[ π N1 (x + 1 2)kx] cos[ π N2 (y + 1 2 )ky] .

slide-6
SLIDE 6

Which Face is Real?

Dkx,ky =

N1−1

∑

x=0 N2−1

∑

y=0

Ix,y cos[ π N1 (x + 1 2)kx] cos[ π N2 (y + 1 2 )ky] .

slide-7
SLIDE 7

Specific to StyleGAN?

slide-8
SLIDE 8

Specific to StyleGAN?

BigGAN ProGAN SN-DCGAN StyleGAN Stanford Dogs

slide-9
SLIDE 9

Specific to StyleGAN?

LSUN Bedrooms Nearest Neighbour Bilinear Binomial BigGAN ProGAN SN-DCGAN StyleGAN Stanford Dogs

slide-10
SLIDE 10

Advantages of the Frequency Domain

slide-11
SLIDE 11

Advantages of the Frequency Domain

Domain Accuracy Image 75.78% Frequency 100.00%

slide-12
SLIDE 12

Advantages of the Frequency Domain

  • Experiments on corrupted data

Domain Accuracy Image 75.78% Frequency 100.00%

slide-13
SLIDE 13

Advantages of the Frequency Domain

  • Experiments on corrupted data
  • Blurring, cropping, jpeg compression, noise, combination

Domain Accuracy Image 75.78% Frequency 100.00%

slide-14
SLIDE 14

Advantages of the Frequency Domain

  • Experiments on corrupted data
  • Blurring, cropping, jpeg compression, noise, combination
  • Frequency representation performs better (bar one exception)

Domain Accuracy Image 75.78% Frequency 100.00%

slide-15
SLIDE 15

Advantages of the Frequency Domain

  • Experiments on corrupted data
  • Blurring, cropping, jpeg compression, noise, combination
  • Frequency representation performs better (bar one exception)
  • When trained on corrupted data, frequency representation

recovers higher accuracy

Domain Accuracy Image 75.78% Frequency 100.00%

slide-16
SLIDE 16

Frequency Domain

Dkx,ky =

N1−1

∑

x=0 N2−1

∑

y=0

Ix,y cos[ π N1 (x + 1 2)kx] cos[ π N2 (y + 1 2 )ky] .

slide-17
SLIDE 17

Frequency Domain

Discrete Cosine Transform

slide-18
SLIDE 18

Frequency Domain

slide-19
SLIDE 19

Frequency Domain

Dkx,ky =

N1−1

∑

x=0 N2−1

∑

y=0

Ix,y cos[ π N1 (x + 1 2)kx] cos[ π N2 (y + 1 2 )ky] .

slide-20
SLIDE 20

Frequency Domain

Dkx,ky =

N1−1

∑

x=0 N2−1

∑

y=0

Ix,y cos[ π N1 (x + 1 2)kx] cos[ π N2 (y + 1 2 )ky] .

slide-21
SLIDE 21

Frequency Domain

Dkx,ky =

N1−1

∑

x=0 N2−1

∑

y=0

Ix,y cos[ π N1 (x + 1 2)kx] cos[ π N2 (y + 1 2 )ky] .

slide-22
SLIDE 22

Frequency Domain

Dkx,ky =

N1−1

∑

x=0 N2−1

∑

y=0

Ix,y cos[ π N1 (x + 1 2)kx] cos[ π N2 (y + 1 2 )ky] . Frequencies in x direction Frequencies in y direction

slide-23
SLIDE 23

Specific to StyleGAN?

Stanford dogs BigGAN ProGAN SN-DCGAN StyleGAN

slide-24
SLIDE 24

Specific to GANs?

Wang, et al., "CNN-generated images are surprisingly easy to spot... for now", CVPR 2020 Cascaded Refinement Networks Implicit Maximum Likelihood Estimation

slide-25
SLIDE 25

Upsampling?

Latent 4x4 ... 1024x1024 8x8 16x16 32x32 64x64

slide-26
SLIDE 26

Upsampling?

Latent 4x4 ... 1024x1024 8x8 16x16 32x32 64x64 Odena, et al., "Deconvolution and Checkerboard Artifacts", Distill 2016

slide-27
SLIDE 27

Upsampling?

Latent 4x4 ... 1024x1024 8x8 16x16 32x32 64x64 Odena, et al., "Deconvolution and Checkerboard Artifacts", Distill 2016 Strided Transposed Convolution → Upsampling + Convolution

slide-28
SLIDE 28

Upsampling?

Latent 4x4 ... 1024x1024 8x8 16x16 32x32 64x64 Durall, et al., "Watch your Up-Convolution: CNN Based Generative Deep Neural Networks are Failing to Reproduce Spectral Distributions", CVPR 2020 Odena, et al., "Deconvolution and Checkerboard Artifacts", Distill 2016 Strided Transposed Convolution → Upsampling + Convolution

slide-29
SLIDE 29

Advantages of the Frequency Domain

Domain Accuracy Image 75.78% Frequency 100.00%

slide-30
SLIDE 30

Advantages of the Frequency Domain

  • Frequency domain enables linear separability
slide-31
SLIDE 31

Advantages of the Frequency Domain

Nearest Neighbour Bilinear Binomial

slide-32
SLIDE 32

Advantages of the Frequency Domain

  • Frequency domain enables linear separability
  • Still artifacts for more elaborate upsampling techniques
slide-33
SLIDE 33

Advantages of the Frequency Domain

  • Frequency domain enables linear separability
  • Still artifacts for more elaborate upsampling techniques
  • For existing source attribution tasks, we can reduce the error rate

by up to 75%

slide-34
SLIDE 34

Advantages of the Frequency Domain

  • Frequency domain enables linear separability
  • Still artifacts for more elaborate upsampling techniques
  • For existing source attribution tasks, we can reduce the error rate

by up to 75%

  • Neural network training is easier and needs less training data
slide-35
SLIDE 35

Advantages of the Frequency Domain

  • Frequency domain enables linear separability
  • Still artifacts for more elaborate upsampling techniques
  • For existing source attribution tasks, we can reduce the error rate

by up to 75%

  • Neural network training is easier and needs less training data
  • Experiments on corrupted data