SLIDE 1
What’s a Pixel?!
source: Molecular Expressions!
photon to electron converter! ! photoelectric effect!!
wikipedia!
Noise, Image Reconstruction with Noise ! EE367/CS448I: Computational - - PowerPoint PPT Presentation
Noise, Image Reconstruction with Noise ! EE367/CS448I: Computational - - PowerPoint PPT Presentation
Noise, Image Reconstruction with Noise ! EE367/CS448I: Computational Imaging and Display ! stanford.edu/class/ee367 ! Lecture 10 ! Gordon Wetzstein ! Stanford University ! Whats a Pixel? ! photon to electron converter ! ! photoelectric effect! !
SLIDE 2
SLIDE 3
What’s a Pixel?!
source: Molecular Expressions!
- !
microlens: focus light on photodiode!
- !
color filter: select color channel!
- !
quantum efficiency: ~50%!
- !
fill factor: fraction of surface area used for light gathering!
- !
photon-to-charge conversion and ADC includes noise!!
SLIDE 4
ISO (“film speed”)!
sensitivity !
- f sensor !
to light – ! digital gain!
bobatkins.com!
SLIDE 5
Noise
- noise is (usually) bad!
- many sources of noise: heat, electronics, amplifier gain, photon to
electron conversion, pixel defects, read, …
- let’s start with something simple: fixed pattern noise
SLIDE 6
Fixed Pattern Noise!
- !
dead or “hot” pixels, dust, …! !
- !
remove with dark frame calibration:! ! ! ! !
- n RAW image!!
not JPEG (nonlinear)! !
Emil Martinec!
I = Icaptured ! Idark Iwhite ! Idark
SLIDE 7
Noise
- ther than that, different noise follows different statistical
distributions, these two are crucial:
- Gaussian
Gaussian
- Poisson
Poisson
SLIDE 8
Gaussian Noise!
- !
thermal, read, amplifier! !
- !
additive, signal-independent, zero-mean! ! ! !
+ =
SLIDE 9
Gaussian Noise!
- !
with i.i.d. Gaussian noise:!
(independent and identically-distributed) !
additive Gaussian noise!
! ! N 0," 2
( )
b = x +!
x ! N x,0
( )
SLIDE 10
Gaussian Noise
- with i.i.d. Gaussian noise:
(independent and identically-distributed)
η ∼ N 0,σ 2
( )
b = x +η
x ∼ N x,0
( )
b ∼ N x,σ 2
( )
p b | x,σ
( ) =
1 2πσ 2 e
− b−x
( )
2
2σ 2
SLIDE 11
Gaussian Noise!
- !
with i.i.d. Gaussian noise:!
(independent and identically-distributed) !
! ! N 0," 2
( )
b = x +!
x ! N x,0
( )
b ! N x,! 2
( )
p b | x,!
( ) =
1 2"! 2 e
# b#x
( )
2
2! 2
- !
Bayes’ rule:!
SLIDE 12
Gaussian Noise - MAP!
- !
with i.i.d. Gaussian noise:!
(independent and identically-distributed) !
! ! N 0," 2
( )
b = x +!
x ! N x,0
( )
b ! N x,! 2
( )
p b | x,!
( ) =
1 2"! 2 e
# b#x
( )
2
2! 2
- !
Bayes’ rule:!
- !
maximum-a-posteriori estimation: !
SLIDE 13
Gaussian Noise – MAP “Flat” Prior!
p x
( ) = 1
- !
trivial solution (not useful in practice):!
SLIDE 14
Gaussian Noise – MAP Self Similarity Prior!
- !
Gaussian “denoisers” like non-local means and other self- similarity priors actually solve this problem:!
SLIDE 15
General Self Similarity Prior!
- !
generic proximal operator for function f(x):!
- !
proximal operator for some image prior!
SLIDE 16
General Self Similarity Prior!
- !
can use self-similarity as general image prior (not just for denoising)!
- !
proximal operator for some image prior!
SLIDE 17
General Self Similarity Prior!
- !
can use self-similarity as general image prior (not just for denoising)!
- !
proximal operator for some image prior!
! 2 = " /! ! = " /!
(h parameter in most NLM ! implementations is std. dev.)!
SLIDE 18
Image Reconstruction with Gaussian Noise
- with i.i.d. Gaussian noise:
(independent and identically-distributed)
η ∼ N 0,σ 2
( )
b = Ax +η
x ∼ N Ax,0
( )
SLIDE 19
Image Reconstruction with Gaussian Noise
- with i.i.d. Gaussian noise:
(independent and identically-distributed)
η ∼ N 0,σ 2
( )
b = Ax +η
x ∼ N Ax,0
( )
b ∼ N Ax,σ 2
( )
p b | x,σ
( ) =
1 2πσ 2 e
− b−Ax
( )
2
2σ 2
SLIDE 20
Image Reconstruction with Gaussian Noise!
- !
with i.i.d. Gaussian noise:!
(independent and identically-distributed) !
! ! N 0," 2
( )
b = Ax +!
x ! N Ax,0
( )
b ! N Ax,! 2
( )
p b | x,!
( ) =
1 2"! 2 e
# b#Ax
( )
2
2! 2
- !
Bayes’ rule:!
- !
maximum-a-posteriori estimation:!
- !
regularized least squares (use ADMM) !
SLIDE 21
Scientific Sensors!
- !
e.g., Andor iXon Ultra 897: cooled to -100° C!
- !
scientific CMOS & CCD!
- !
reduce pretty much all noise, except for photon noise !
SLIDE 22
What is Photon (or Shot) Noise?
- noise caused by the fluctuation when the incident light is
converted to charge (detection)
- also observed in the light emitted by a source, i.e. due to particle
nature of light (emission)
- same for re-emission à cascading Poisson processes are also
described by a Poisson process [Teich and Saleh 1998]
SLIDE 23
Photon Noise
- noise is signal dependent!
- for N measured photo-electrons
- standard deviation is , variance is
- mean is
- Poisson distribution
σ = N σ 2 = N N
f (k;N) = N ke−N k!
SLIDE 24
Photon Noise - SNR!
N photons: ! 2N photons: ! nonlinear!!
! = N
! = 2 N
signal-to-noise ratio
SNR = N N
N
SNR in dB
1,000 10,000
SLIDE 25
N photons: ! 2N photons: ! nonlinear!!
wikipedia!
! = N
! = 2 N
signal-to-noise ratio
SNR = N N
Photon Noise - SNR!
SLIDE 26
Maximum Likelihood Solution for Poisson Noise!
- !
image formation:!
- !
probability of measurement i:!
- !
joint probability of all M measurements (use notation trick ):!
SLIDE 27
- !
log-likelihood function:! ! ! ! !
- !
gradient:!
Maximum Likelihood Solution for Poisson Noise!
SLIDE 28
Maximum Likelihood Solution for Poisson Noise!
- !
Richardson-Lucy algorithm: iterative approach to ML estimation for Poisson noise!
- !
simple idea: ! 1.! at solution, gradient will be zero! 2.! when converged, further iterations will not change, i.e. ! !
SLIDE 29
- !
equate gradient to zero! !
- !
rearrange so that 1 is on one side of equation!
Richardson-Lucy Algorithm!
= 0
SLIDE 30
- !
equate gradient to zero! !
- !
rearrange so that 1 is on one side of equation!
- !
set equal to !
Richardson-Lucy Algorithm!
= 0
SLIDE 31
Richardson-Lucy Algorithm!
- !
for any multiplicative update rule scheme: !
- !
start with positive initial guess (e.g. random values)!
- !
apply iteration scheme!
- !
future updates are guaranteed to remain positive!
- !
always get smaller residual! !
- !
RL multiplicative update rules:!
SLIDE 32
Richardson-Lucy Algorithm - Deconvolution!
Blurry & Noisy Measurement! RL Deconvolution!
SLIDE 33
Richardson-Lucy Algorithm
- what went wrong?
- Poisson deconvolution is a tough problem, without priors it’s pretty
much hopeless
- let’s try to incorporate the one prior we have learned: total variation
SLIDE 34
- !
log-likelihood function:! ! !
- !
gradient:!
Richardson-Lucy Algorithm + TV!
SLIDE 35
- !
gradient of anisotropic TV term! ! !
- !
this is “dirty”: possible division by 0! ! !
Richardson-Lucy Algorithm + TV!
SLIDE 36
- !
follow the same logic as RL, RL+TV multiplicative update rules:! !
- !
2 major problems & solution “hacks”:! 1.! still possible division by 0 when gradient is zero! 2.! multiplicative update may become negative!! !
Richardson-Lucy Algorithm + TV!
! set fraction to 0 if gradient is 0 ! only work with (very) small values for lambda
SLIDE 37
(A Dirty but Easy Approach to) Richardson-Lucy with a TV Prior!
RL!
RL+TV, lambda=0.005! RL+TV, lambda=0.025!
Measurements! Log Residual! Mean Squared Error!
SLIDE 38
Signal-to-Noise Ratio (SNR)
SNR = mean pixel value standard deviation of pixel value = µ σ
signal signal noise noise
= PQet PQet + Dt + Nr
2
P = incident photon flux (photons/pixel/sec) Qe = quantum efficiency t = eposure time (sec) D = dark current (electroncs/pixel/sec), including hot pixels Nr = read noise (rms electrons/pixel), including fixed pattern noise
SLIDE 39
Signal-to-Noise Ratio (SNR)!
SNR = mean pixel value standard deviation of pixel value = µ !
signal! noise!
= PQet PQet + Dt + Nr
2
P = incident photon flux (photons/pixel/sec) Qe = quantum efficiency t = eposure time (sec) D = dark current (electroncs/pixel/sec), including hot pixels Nr = read noise (rms electrons/pixel), including fixed pattern noise signal-dependent! signal-independent!
SLIDE 40
Next: Compressive Imaging!
- !
single pixel camera!
- !
compressive hyperspectral imaging!
- !
compressive light field imaging!
Wakin et al. 2006! Marwah et al., 2013!
SLIDE 41
References and Further Reading
- https://en.wikipedia.org/wiki/Shot_noise
- L. Lucy, 1974 “An iterative technique for the rectification of observed distributions”. Astron. J. 79, 745–754.
- W. Richardson, 1972 “Bayesian-based iterative method of image restoration” J. Opt. Soc. Am. 62, 1, 55–59.
- N. Dey, L. Blanc-Feraud, C. Zimmer, Z. Kam, J. Olivo-Marin, J. Zerubia, 2004 “A deconvolution method for confocal microscopy with total
variation regularization”, In IEEE Symposium on Biomedical Imaging: Nano to Macro, 1223–1226
- M. Teich, E. Saleh, 1998, “Cascaded stochastic processes in optics”, Traitement du Signal 15(6)
- please also read the lecture notes, especially for the “clean” ADMM derivation for solving the maximum likelihood estimation of Poisson
please also read the lecture notes, especially for the “clean” ADMM derivation for solving the maximum likelihood estimation of Poisson reconstruction with TV prior! reconstruction with TV prior!