Joint SVBRDF Recovery and Synthesis From a Single Image using an - - PowerPoint PPT Presentation

▶
joint svbrdf recovery and synthesis from a single image
SMART_READER_LITE
LIVE PREVIEW

Joint SVBRDF Recovery and Synthesis From a Single Image using an - - PowerPoint PPT Presentation

Oct 04, 2022 339 likes •796 views

EGSR 2020 Joint SVBRDF Recovery and Synthesis From a Single Image using an Unsupervised Generative Adversarial Network Yezi Zhao 1 , Beibei Wang 2 , Yanning Xu 1 , Zheng Zeng 1 , Lu Wang 1 and Nicolas Holzschuch 3 1 School of Software, Shandong

slide-1
SLIDE 1

Joint SVBRDF Recovery and Synthesis From a Single Image using an Unsupervised Generative Adversarial Network

Yezi Zhao1, Beibei Wang2, Yanning Xu1, Zheng Zeng1, Lu Wang1 and Nicolas Holzschuch3

1 School of Software, Shandong University 2 School of Computer Science and Engineering, Nanjing University of Science and Technology 3 Univ. Grenoble Alpes, Inria, CNRS, Grenoble INP, LJK

1

EGSR 2020

EGSR 2020

slide-2
SLIDE 2

2

EGSR 2020

Motivation

Substance by Adobe normal diffuse roughness specular

slide-3
SLIDE 3

3

EGSR 2020

Our Goal

normal diffuse specular roughness

A lightweight method for recovering real-world material

slide-4
SLIDE 4

4

EGSR 2020

State of the art

Deschaintre et al., SIGGRAPH 2018

Single-Image SVBRDF Capture with a Rendering-Aware Deep Network

Li et al., SIGGRAPH 2017

Modeling surface appearance from a single photograph using self-augmented convolutional neural networks normal diffuse specular normal diffuse roughness specular

Aittala et al., SIGGRAPH 2016

Reflectance Modeling by Neural Texture Synthesis normal diffuse glossiness specular

slide-5
SLIDE 5

5

EGSR 2020

State of the art

Deschaintre et al., SIGGRAPH 2018

Single-Image SVBRDF Capture with a Rendering-Aware Deep Network

Li et al., SIGGRAPH 2017

Modeling surface appearance from a single photograph using self-augmented convolutional neural networks normal diffuse specular normal diffuse roughness specular

Aittala et al., SIGGRAPH 2016

Reflectance Modeling by Neural Texture Synthesis normal diffuse glossiness specular

slide-6
SLIDE 6

6

EGSR 2020

State of the art

Deschaintre et al., SIGGRAPH 2018

Single-Image SVBRDF Capture with a Rendering-Aware Deep Network

Li et al., SIGGRAPH 2017

Modeling surface appearance from a single photograph using self-augmented convolutional neural networks normal diffuse specular normal diffuse roughness specular

Aittala et al., SIGGRAPH 2016

Reflectance Modeling by Neural Texture Synthesis normal diffuse glossiness specular

slide-7
SLIDE 7

7

EGSR 2020

State of the art

Deschaintre et al., SIGGRAPH 2018

Single-Image SVBRDF Capture with a Rendering-Aware Deep Network

Li et al., SIGGRAPH 2017

Modeling surface appearance from a single photograph using self-augmented convolutional neural networks normal diffuse specular normal diffuse roughness specular

Aittala et al., SIGGRAPH 2016

Reflectance Modeling by Neural Texture Synthesis normal diffuse glossiness specular

slide-8
SLIDE 8

8

EGSR 2020

State of the art

Deschaintre et al., SIGGRAPH 2018

Single-Image SVBRDF Capture with a Rendering-Aware Deep Network

Li et al., SIGGRAPH 2017

Modeling surface appearance from a single photograph using self-augmented convolutional neural networks normal diffuse specular normal diffuse roughness specular

Aittala et al., SIGGRAPH 2016

Reflectance Modeling by Neural Texture Synthesis normal diffuse glossiness specular

Low-resolution

slide-9
SLIDE 9

9

EGSR 2020

State of the art

Gao et al., SIGGRAPH 2019

Deep Inverse Rendering for High-resolution SVBRDF Estimation from an Arbitrary Number of Images normal diffuse specular input roughness

… …

slide-10
SLIDE 10

10

EGSR 2020

State of the art

Gao et al., SIGGRAPH 2019

Deep Inverse Rendering for High-resolution SVBRDF Estimation from an Arbitrary Number of Images normal diffuse specular input roughness

… …

Rely on a plausible starting point of optimizing

slide-11
SLIDE 11

11

EGSR 2020

State of the art

Synthesize [Zhou et al. 2018] Zhou et al., SIGGRAPH 2018

Non-stationary texture synthesis by adversarial expansion

slide-12
SLIDE 12

12

EGSR 2020

State of the art

Synthesize Separately Zoom-in [Zhou et al. 2018] Synthesize Separately [Zhou et al. 2018] Zhou et al., SIGGRAPH 2018

Non-stationary texture synthesis by adversarial expansion

slide-13
SLIDE 13

13

EGSR 2020

State of the art

Synthesisze Seperatly Zoom-in [Zhou et al. 2018] Synthesisze Seperatly [Zhou et al. 2018] Zhou et al., SIGGRAPH 2018

Non-stationary texture synthesis by adversarial expansion

Inconsistency between SVBRDF maps

slide-14
SLIDE 14

14

EGSR 2020

Method overview

normal diffuse specular roughness Reference Captured image Render Our method

slide-15
SLIDE 15

15

EGSR 2020

Imaging setup

Imaging setup Captured image

𝑦 𝑦𝑑

slide-16
SLIDE 16

16

EGSR 2020

Generator Generator SVBRDF maps Discriminator

Re-render Input

Generative Adversarial Network (GAN)

slide-17
SLIDE 17

17

EGSR 2020

Generator Generator SVBRDF maps Discriminator

Captured image

𝑦

slide-18
SLIDE 18

18

EGSR 2020

Generator Generator SVBRDF maps Discriminator

𝑦

Captured image

slide-19
SLIDE 19

19

EGSR 2020

Generator Generator SVBRDF maps Discriminator

𝑦

Captured image

slide-20
SLIDE 20

20

EGSR 2020

Generator SVBRDF maps Discriminator

𝑦

Captured image

slide-21
SLIDE 21

21

EGSR 2020

Untrained encoder Visualization of the first 4 layers latent vector from encoder

Generator

slide-22
SLIDE 22

22

EGSR 2020

Generator

Two decoders

slide-23
SLIDE 23

23

EGSR 2020

Discriminator

“Real data” “Fake data”

Discriminator

slide-24
SLIDE 24

24

EGSR 2020

Loss function

slide-25
SLIDE 25

25

EGSR 2020

Guessed diffuse map

Guessed diffuse map Input image

Computed as in [AAL16]

[AAL16] Aittala et al. Reflectance modeling by neural texture synthesis.

slide-26
SLIDE 26

26

EGSR 2020

Loss function

—

1

Discriminator( , )

Generated diffuse Guessed diffuse Tile 𝑦 Re-render 𝑧

Adversarial loss L1 loss

slide-27
SLIDE 27

27

EGSR 2020

Loss function

—

1

Discriminator( , )

Tile 𝑦 Re-render 𝑧

Adversarial loss L1 loss

Tile 𝑦 Re-render 𝑧

slide-28
SLIDE 28

28

EGSR 2020

Loss function

Input Diffuse Diffuse Highlights Input Highlights Highlights

slide-29
SLIDE 29

29

EGSR 2020

Results

Input image： 1632×1224 SVBRDF & Render: 3264×2448

slide-30
SLIDE 30

30

EGSR 2020

Results

Input image： 1632×1224 SVBRDF & Render: 3264×2448

slide-31
SLIDE 31

31

EGSR 2020

Results

Input image： 1632×1224 SVBRDF & Render: 3264×2448

slide-32
SLIDE 32

32

EGSR 2020

Results

×2 ×4 ×8 ×1

slide-33
SLIDE 33

33

EGSR 2020

Results

512 × 512 Input 1024×1024 SVBRDF maps normal roughness diffuse specular Render 1024×1024

slide-34
SLIDE 34

34

EGSR 2020

Results

Render 1024×1024 1024×1024 SVBRDF maps normal roughness diffuse specular 512 × 512 Input

slide-35
SLIDE 35

35

EGSR 2020

Results

2048×2048 SVBRDF maps normal diffuse specular roughness

slide-36
SLIDE 36

36

EGSR 2020

Results

2048×2048 SVBRDF maps normal diffuse specular roughness 2048×2048 Render (omit)

slide-37
SLIDE 37

37

EGSR 2020

Results

4096×4096 SVBRDF maps normal diffuse specular roughness

slide-38
SLIDE 38

38

EGSR 2020

Results

Rendered with Arnold

……

Captured images

slide-39
SLIDE 39

39

EGSR 2020

Network Analysis – loss function

slide-40
SLIDE 40

40

EGSR 2020

Network Analysis – generator

slide-41
SLIDE 41

41

EGSR 2020

Limitation

  • Our method failed to synthesis textures when the input image has a global structure.
slide-42
SLIDE 42

42

EGSR 2020

Limitation

  • Our method failed to synthesis textures when the input image has a global structure.
  • Each input for our method requires individual training, which costs about 3 hours.
slide-43
SLIDE 43

43

EGSR 2020

Conclusion and Future work

  • An unsupervised GAN for joint SVBRDF recovery and synthesis without a large training dataset.
  • A two-stream generator to enhance specular component.
  • A novel joint loss function for high-quality novel view renderings.
  • Introduce existing knowledge about the material.

Conclusion Future work

slide-44
SLIDE 44

44

EGSR 2020

Thank you for your attention

EGSR 2020 Yezi Zhao, Beibei Wang, Yanning Xu, Zheng Zeng, Lu Wang and Nicolas Holzschuch The code is available: https://github.com/mengshu1996/SVBRDF-GAN