GradNet: Unsupervised Deep Screened Poisson Reconstruction for - - PowerPoint PPT Presentation

▶
gradnet unsupervised deep screened poisson reconstruction
SMART_READER_LITE
LIVE PREVIEW

GradNet: Unsupervised Deep Screened Poisson Reconstruction for - - PowerPoint PPT Presentation

Aug 28, 2022 455 likes •964 views

GradNet: Unsupervised Deep Screened Poisson Reconstruction for GradientDomain Rendering Jie Guo 1 Mengtian Li 1 Quewei Li 1 Yuting Qiang 1 Bingyang Hu 1 Yanwen Guo 1 LingQi Yan 2 1 State Key Lab for Novel Software 2 University of California,

slide-1
SLIDE 1

sa2019.siggraph.org

GradNet: Unsupervised Deep Screened Poisson Reconstruction for Gradient‐Domain Rendering

Jie Guo1 Mengtian Li1 Quewei Li1 Yuting Qiang1 Bingyang Hu1 Yanwen Guo1 Ling‐Qi Yan2

1State Key Lab for Novel Software

Technology, Nanjing University

2University of California, Santa Barbara
slide-2
SLIDE 2 CONFERENCE 17‐20 November 2019 ‐ EXHIBITION 18‐20 November 2019 ‐ BCEC, Brisbane, AUSTRALIA

SA2019.SIGGRAPH.ORG

Path tracing

Diffuse Diffuse Mirror Light

slide-3
SLIDE 3 CONFERENCE 17‐20 November 2019 ‐ EXHIBITION 18‐20 November 2019 ‐ BCEC, Brisbane, AUSTRALIA

SA2019.SIGGRAPH.ORG

Path tracing

error / 2 = samples * 4

slide-4
SLIDE 4 CONFERENCE 17‐20 November 2019 ‐ EXHIBITION 18‐20 November 2019 ‐ BCEC, Brisbane, AUSTRALIA

SA2019.SIGGRAPH.ORG

Gradient‐domain Rendering

Diffuse Diffuse Mirror Light

Base Path Offset Path

slide-5
SLIDE 5 CONFERENCE 17‐20 November 2019 ‐ EXHIBITION 18‐20 November 2019 ‐ BCEC, Brisbane, AUSTRALIA

SA2019.SIGGRAPH.ORG

Gradient‐domain Rendering

slide-6
SLIDE 6 CONFERENCE 17‐20 November 2019 ‐ EXHIBITION 18‐20 November 2019 ‐ BCEC, Brisbane, AUSTRALIA

SA2019.SIGGRAPH.ORG

Gradient‐domain Rendering

6

slide-7
SLIDE 7 CONFERENCE 17‐20 November 2019 ‐ EXHIBITION 18‐20 November 2019 ‐ BCEC, Brisbane, AUSTRALIA

SA2019.SIGGRAPH.ORG

Screened Poisson Reconstruction

  • For a base image and gradients rendered by any gradient‐

domain algorithms, we can reconstruct the final image by solving the following optimization problem

7

slide-8
SLIDE 8 CONFERENCE 17‐20 November 2019 ‐ EXHIBITION 18‐20 November 2019 ‐ BCEC, Brisbane, AUSTRALIA

SA2019.SIGGRAPH.ORG

Screened Poisson Reconstruction

  • For a base image and gradients rendered by any gradient‐

domain algorithms, we can reconstruct the final image by solving the following optimization problem

8

Data term Gradient term

slide-9
SLIDE 9 CONFERENCE 17‐20 November 2019 ‐ EXHIBITION 18‐20 November 2019 ‐ BCEC, Brisbane, AUSTRALIA

SA2019.SIGGRAPH.ORG

Screened Poisson Reconstruction

  • Lp norm

9

p = 1 means L1 reconstruction p = 2 means L2 reconstruction

slide-10
SLIDE 10 CONFERENCE 17‐20 November 2019 ‐ EXHIBITION 18‐20 November 2019 ‐ BCEC, Brisbane, AUSTRALIA

SA2019.SIGGRAPH.ORG

Regularization

  • A regularized version of the screened Poisson solver can

be written as Regularizer

9

slide-11
SLIDE 11 CONFERENCE 17‐20 November 2019 ‐ EXHIBITION 18‐20 November 2019 ‐ BCEC, Brisbane, AUSTRALIA

SA2019.SIGGRAPH.ORG

Regularized Screen Poisson Reconstruction

1 Rendering‐specific features

slide-12
SLIDE 12 CONFERENCE 17‐20 November 2019 ‐ EXHIBITION 18‐20 November 2019 ‐ BCEC, Brisbane, AUSTRALIA

SA2019.SIGGRAPH.ORG

Our Idea

1 Rendering‐specific features

slide-13
SLIDE 13 CONFERENCE 17‐20 November 2019 ‐ EXHIBITION 18‐20 November 2019 ‐ BCEC, Brisbane, AUSTRALIA

SA2019.SIGGRAPH.ORG

Related Work

METHOD DEEP LEARNING BASED AUXILIARY BUFFERS PERFORMANCE L1 × × ≈0.45s GPU CV [Rousselle et al. 2016] × × ≈2s CPU LTS [Ha et al. 2019] × × ≈1.7s CPU NFOR [Bitterli et al. 2016] × √ ≈200s CPU REG [Manzi et al. 2016] × √ ≈60s GPU KPCN [Bako et al. 2017] √ Supervised √ ≈1.7s GPU [Kettunen et al. 2019] √ Supervised √ ≈0.3s GPU

4

slide-14
SLIDE 14 CONFERENCE 17‐20 November 2019 ‐ EXHIBITION 18‐20 November 2019 ‐ BCEC, Brisbane, AUSTRALIA

SA2019.SIGGRAPH.ORG

Related Work

METHOD DEEP LEARNING BASED AUXILIARY BUFFERS PERFORMANCE L1 × × ≈0.45s GPU CV [Rousselle et al. 2016] × × ≈2s CPU LTS [Ha et al. 2019] × × ≈1.7s CPU NFOR [Bitterli et al. 2016] × √ ≈200s CPU REG [Manzi et al. 2016] × √ ≈60s GPU KPCN [Bako et al. 2017] √ Supervised √ ≈1.7s GPU [Kettunen et al. 2019] √ Supervised √ ≈0.3s GPU

4

slide-15
SLIDE 15 CONFERENCE 17‐20 November 2019 ‐ EXHIBITION 18‐20 November 2019 ‐ BCEC, Brisbane, AUSTRALIA

SA2019.SIGGRAPH.ORG

Related Work (REG [Manzi et al. 2016])

Feature Bases via Truncated SVD. Solved by an iteratively reweighted least squares (IRLS) approach

slide-16
SLIDE 16 CONFERENCE 17‐20 November 2019 ‐ EXHIBITION 18‐20 November 2019 ‐ BCEC, Brisbane, AUSTRALIA

SA2019.SIGGRAPH.ORG

Related Work (KPCN)

slide-17
SLIDE 17 CONFERENCE 17‐20 November 2019 ‐ EXHIBITION 18‐20 November 2019 ‐ BCEC, Brisbane, AUSTRALIA

SA2019.SIGGRAPH.ORG

Related Work ([Kettunen et al. 2019])

  • 1. Using gradients as an additional feature
  • 2. Adopting a new perceptual loss
slide-18
SLIDE 18 CONFERENCE 17‐20 November 2019 ‐ EXHIBITION 18‐20 November 2019 ‐ BCEC, Brisbane, AUSTRALIA

SA2019.SIGGRAPH.ORG

Our Method

  • Deep learning based
  • Unsupervised
  • Fast to reconstruct high‐quality image

1

slide-19
SLIDE 19 CONFERENCE 17‐20 November 2019 ‐ EXHIBITION 18‐20 November 2019 ‐ BCEC, Brisbane, AUSTRALIA

SA2019.SIGGRAPH.ORG

  • Replace the traditional optimization in screened Poisson

reconstruction with GradNet

Our Solution

10

slide-20
SLIDE 20 CONFERENCE 17‐20 November 2019 ‐ EXHIBITION 18‐20 November 2019 ‐ BCEC, Brisbane, AUSTRALIA

SA2019.SIGGRAPH.ORG

Network Architecture

  • Multi‐branch auto‐encoder with dual skip connection

20 Low‐frequency contents High‐frequency details

slide-21
SLIDE 21 CONFERENCE 17‐20 November 2019 ‐ EXHIBITION 18‐20 November 2019 ‐ BCEC, Brisbane, AUSTRALIA

SA2019.SIGGRAPH.ORG

Network Architecture

  • Multi‐branch auto‐encoder with dual skip connection

21 Data branch G‐branch for generating derivative Gradient branch

slide-22
SLIDE 22 CONFERENCE 17‐20 November 2019 ‐ EXHIBITION 18‐20 November 2019 ‐ BCEC, Brisbane, AUSTRALIA

SA2019.SIGGRAPH.ORG

Impact of the Branches

22 One‐branch encoder weaken the effects of sparse image gradients

slide-23
SLIDE 23 CONFERENCE 17‐20 November 2019 ‐ EXHIBITION 18‐20 November 2019 ‐ BCEC, Brisbane, AUSTRALIA

SA2019.SIGGRAPH.ORG

Dynamic Range Compression

  • We employ the μ‐law transformation to compress HDR

data

  • The μ‐law transformation makes the training process

easier than naïve log transformation

23

slide-24
SLIDE 24 CONFERENCE 17‐20 November 2019 ‐ EXHIBITION 18‐20 November 2019 ‐ BCEC, Brisbane, AUSTRALIA

SA2019.SIGGRAPH.ORG

Loss Function

  • The loss function contains 3 items
  • data item, gradient item and first‐order item
  • Data item:

24

slide-25
SLIDE 25 CONFERENCE 17‐20 November 2019 ‐ EXHIBITION 18‐20 November 2019 ‐ BCEC, Brisbane, AUSTRALIA

SA2019.SIGGRAPH.ORG

Loss Function

  • The loss function contains 3 items
  • data item, gradient item and first‐order item
  • Gradient item:

25

slide-26
SLIDE 26 CONFERENCE 17‐20 November 2019 ‐ EXHIBITION 18‐20 November 2019 ‐ BCEC, Brisbane, AUSTRALIA

SA2019.SIGGRAPH.ORG

Impact of the Gradient Loss

26 Ib Idx Idy

Reference

  • w. gradient loss

w.o. gradient loss

slide-27
SLIDE 27 CONFERENCE 17‐20 November 2019 ‐ EXHIBITION 18‐20 November 2019 ‐ BCEC, Brisbane, AUSTRALIA

SA2019.SIGGRAPH.ORG

Loss Function

  • The loss function contains 3 items
  • data item, gradient item and first‐order item
  • The first‐order item:

27

slide-28
SLIDE 28 CONFERENCE 17‐20 November 2019 ‐ EXHIBITION 18‐20 November 2019 ‐ BCEC, Brisbane, AUSTRALIA

SA2019.SIGGRAPH.ORG

First‐order Regularization

  • The first‐order regularization defines as follow

28

1 𝑂 𝑂 𝑥, 𝐽

  • 𝐽

𝐻

𝐺 𝐺 ∈

  • Neighboring pixels

around index i Difference of auxiliary features Derivative of Ii with respect to Fi

slide-29
SLIDE 29 CONFERENCE 17‐20 November 2019 ‐ EXHIBITION 18‐20 November 2019 ‐ BCEC, Brisbane, AUSTRALIA

SA2019.SIGGRAPH.ORG

First‐order Regularization

  • The first‐order regularization encourages nearby pixels to

lie on a hyper‐plane parameterized by G

29

G

slide-30
SLIDE 30 CONFERENCE 17‐20 November 2019 ‐ EXHIBITION 18‐20 November 2019 ‐ BCEC, Brisbane, AUSTRALIA

SA2019.SIGGRAPH.ORG

Impact of the First‐order Loss

w.o. the first-order loss

  • w. the first-order loss

reference

slide-31
SLIDE 31 CONFERENCE 17‐20 November 2019 ‐ EXHIBITION 18‐20 November 2019 ‐ BCEC, Brisbane, AUSTRALIA

SA2019.SIGGRAPH.ORG

Bias of Network

μ will introduce bias to the reconstructed image

Reference (mean: 0.204) μ = 16 (mean: 0.187) μ = 128 (mean: 0.176) μ = 1024 (mean: 0.163)

slide-32
SLIDE 32 CONFERENCE 17‐20 November 2019 ‐ EXHIBITION 18‐20 November 2019 ‐ BCEC, Brisbane, AUSTRALIA

SA2019.SIGGRAPH.ORG

Post‐Processing

  • We find a simple post‐processing step can reduce the bias

32

Gaussian filter with r = 45 and σ = 15 Without P.P. With P.P.

slide-33
SLIDE 33 CONFERENCE 17‐20 November 2019 ‐ EXHIBITION 18‐20 November 2019 ‐ BCEC, Brisbane, AUSTRALIA

SA2019.SIGGRAPH.ORG

Training Details

  • Adam optimizer
  • β1 = 0.5 and β2 = 0.999
  • Initial learning rate = 0.0001 and decays with the power of

0.95 for every other epoch

  • λ follows the schedule
  • Train main branches and G‐branch alternatively for 50

epochs with 32 mini‐batches

33

slide-34
SLIDE 34 CONFERENCE 17‐20 November 2019 ‐ EXHIBITION 18‐20 November 2019 ‐ BCEC, Brisbane, AUSTRALIA

SA2019.SIGGRAPH.ORG

Dataset

  • Randomly perturb 9 base scenes and render to 900 high‐

resolution images with 64 spp

34

slide-35
SLIDE 35 CONFERENCE 17‐20 November 2019 ‐ EXHIBITION 18‐20 November 2019 ‐ BCEC, Brisbane, AUSTRALIA

SA2019.SIGGRAPH.ORG

Dataset

  • 12654 patches with 256x256 resolution are extracted from

these 900 high‐resolution images

35

slide-36
SLIDE 36 CONFERENCE 17‐20 November 2019 ‐ EXHIBITION 18‐20 November 2019 ‐ BCEC, Brisbane, AUSTRALIA

SA2019.SIGGRAPH.ORG

Results

36

slide-37
SLIDE 37 CONFERENCE 17‐20 November 2019 ‐ EXHIBITION 18‐20 November 2019 ‐ BCEC, Brisbane, AUSTRALIA

SA2019.SIGGRAPH.ORG

Comparison with traditional methods

37

Path L1 CV LTS Manzi NFOR Ours

slide-38
SLIDE 38 CONFERENCE 17‐20 November 2019 ‐ EXHIBITION 18‐20 November 2019 ‐ BCEC, Brisbane, AUSTRALIA

SA2019.SIGGRAPH.ORG

Comparison with traditional methods

38

Path L1 CV LTS Manzi NFOR Ours

slide-39
SLIDE 39 CONFERENCE 17‐20 November 2019 ‐ EXHIBITION 18‐20 November 2019 ‐ BCEC, Brisbane, AUSTRALIA

SA2019.SIGGRAPH.ORG

Comparison with traditional methods

39

Path L1 CV LTS Manzi NFOR Ours

slide-40
SLIDE 40 CONFERENCE 17‐20 November 2019 ‐ EXHIBITION 18‐20 November 2019 ‐ BCEC, Brisbane, AUSTRALIA

SA2019.SIGGRAPH.ORG

Comparison with traditional methods

slide-41
SLIDE 41 CONFERENCE 17‐20 November 2019 ‐ EXHIBITION 18‐20 November 2019 ‐ BCEC, Brisbane, AUSTRALIA

SA2019.SIGGRAPH.ORG

Comparison with traditional methods

  • Comparison with the regularized reconstruction method
  • f Manzi et al. [2016]
  • Time: >1min (REG) vs. 0.16s (ours)
slide-42
SLIDE 42 CONFERENCE 17‐20 November 2019 ‐ EXHIBITION 18‐20 November 2019 ‐ BCEC, Brisbane, AUSTRALIA

SA2019.SIGGRAPH.ORG

Comparison with traditional methods

42

slide-43
SLIDE 43 CONFERENCE 17‐20 November 2019 ‐ EXHIBITION 18‐20 November 2019 ‐ BCEC, Brisbane, AUSTRALIA

SA2019.SIGGRAPH.ORG

Comparison with KPCN

43

slide-44
SLIDE 44 CONFERENCE 17‐20 November 2019 ‐ EXHIBITION 18‐20 November 2019 ‐ BCEC, Brisbane, AUSTRALIA

SA2019.SIGGRAPH.ORG

Comparison with KPCN

44

slide-45
SLIDE 45 CONFERENCE 17‐20 November 2019 ‐ EXHIBITION 18‐20 November 2019 ‐ BCEC, Brisbane, AUSTRALIA

SA2019.SIGGRAPH.ORG

Comparison with KPCN

45

slide-46
SLIDE 46 CONFERENCE 17‐20 November 2019 ‐ EXHIBITION 18‐20 November 2019 ‐ BCEC, Brisbane, AUSTRALIA

SA2019.SIGGRAPH.ORG

Impact of the Training Datasets

46

1/4 dataset 1/2 dataset Full dataset

slide-47
SLIDE 47 CONFERENCE 17‐20 November 2019 ‐ EXHIBITION 18‐20 November 2019 ‐ BCEC, Brisbane, AUSTRALIA

SA2019.SIGGRAPH.ORG

Conclusion

  • The first unsupervised deep learning solution to

screened Poisson reconstruction.

  • A multi‐branch auto‐encoder allowing extracting both

low‐frequency contents and high‐frequency details.

  • A novel reconstruction loss function incorporating

auxiliary feature buffers.

slide-48
SLIDE 48 CONFERENCE 17‐20 November 2019 ‐ EXHIBITION 18‐20 November 2019 ‐ BCEC, Brisbane, AUSTRALIA

SA2019.SIGGRAPH.ORG

Future Work

  • Adding an adversarial loss to further enhance

important local structures of reconstructed images.

  • Extend to the temporal domain by introducing

temporal finite differences.

  • Combine our technology with adaptive sampling.
slide-49
SLIDE 49

sa2019.siggraph.org

Thank you

Jie Guo guojie@nju.edu.cn