Signals, Information and Sampling Steve McLaughlin University of - - PowerPoint PPT Presentation

▶
signals information and sampling steve mclaughlin
SMART_READER_LITE
LIVE PREVIEW

Signals, Information and Sampling Steve McLaughlin University of - - PowerPoint PPT Presentation

Dec 16, 2023 536 likes •1.04k views

Signals, Information and Sampling Steve McLaughlin University of Edinburgh, 16 th January, 2008 Page 1 of 49 Signals, Information and Sampling Steve McLaughlin Signals, Information and Sampling or Some new directions in

slide-1
SLIDE 1

Page 1 of 49

“Signals, Information and Sampling”

University of Edinburgh, 16th January, 2008

Steve McLaughlin

slide-2
SLIDE 2

Page 2 of 49

“Signals, Information and Sampling”

University of Edinburgh, 16th January, 2008

Steve McLaughlin

“Signals, Information and Sampling”

  • r

Some new directions in signal processing and communications

Steve McLaughlin

IDCOM, School of Engineering & Electronics

with thanks to Mike Davies, Bernie Mulgrew, John Thompson

slide-3
SLIDE 3

Page 3 of 49

“Signals, Information and Sampling”

University of Edinburgh, 16th January, 2008

Steve McLaughlin

What are signals?

A signal is a time (or space) varying quantity that can carry

  • information. The concept is broad, and hard to define precisely.

(Wikipedia)

slide-4
SLIDE 4

Page 4 of 49

“Signals, Information and Sampling”

University of Edinburgh, 16th January, 2008

Steve McLaughlin

What is signal processing?

Signal processing is the analysis, interpretation, and manipulation of signals. Signals of interest include sound, images, biological signals such as ECG, radar signals, and many others. Processing of such signals includes storage and reconstruction, separation of information from noise (e.g., aircraft identification by radar), compression (e.g., image compression), and feature extraction (e.g., speech-to-text conversion). (Wikipedia)

slide-5
SLIDE 5

Page 5 of 49

“Signals, Information and Sampling”

University of Edinburgh, 16th January, 2008

Steve McLaughlin

What is information?

I nformation theory is a branch of applied mathematics and engineering involving the quantification of information. Historically, information theory developed to find fundamental limits on compressing and reliably communicating data. Since its inception it has broadened to find applications in statistical inference, networks other than communication networks, biology, quantum information theory, data analysis , and other areas, although it is still widely used in the study of communication. (Wikipedia)

slide-6
SLIDE 6

Page 6 of 49

“Signals, Information and Sampling”

University of Edinburgh, 16th January, 2008

Steve McLaughlin

What is informatics?

I nformatics includes the science of information, the practice

  • f information processing, and the engineering of information
  • systems. Informatics studies the structure, behaviour, and

interactions of natural and artificial systems that store, process and communicate information. I nformatics is broader in scope than information theory. (Wikipedia)

slide-7
SLIDE 7

Page 7 of 49

“Signals, Information and Sampling”

University of Edinburgh, 16th January, 2008

Steve McLaughlin

Part I: Sparse Representations and Coding

slide-8
SLIDE 8

Page 8 of 49

“Signals, Information and Sampling”

University of Edinburgh, 16th January, 2008

Steve McLaughlin

What are signals made of?

The Frequency viewpoint (Fourier):

Signals can be built from the sum of harmonic functions (sine waves)

Joseph Fourier

50 100 150 200 250 −1.5 −1 −0.5 0.5 1 1.5 50 100 150 200 250 −1.5 −1 −0.5 0.5 1 1.5 50 100 150 200 250 −1.5 −1 −0.5 0.5 1 1.5 50 100 150 200 250 −1.5 −1 −0.5 0.5 1 1.5 50 100 150 200 250 −1.5 −1 −0.5 0.5 1 1.5 50 100 150 200 250 −1.5 −1 −0.5 0.5 1 1.5

∑

=

k k k

t c t x ) ( ) ( φ

signal Fourier coefficients Harmonic functions

slide-9
SLIDE 9

Page 9 of 49

“Signals, Information and Sampling”

University of Edinburgh, 16th January, 2008

Steve McLaughlin

Sampling and the digital revolution

Today we are more familiar with discrete signals (e.g. audio files, digital images). This is thanks to: Whittaker–Kotelnikov–Shannon Sampling Theorem: “Exact reconstruction of a continuous-time signal from discrete samples is possible if the signal is bandlimited and the sampling frequency is greater than twice the signal bandwidth.” Sampling below this rate introduces aliasing

slide-10
SLIDE 10

Page 10 of 49

“Signals, Information and Sampling”

University of Edinburgh, 16th January, 2008

Steve McLaughlin

0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 2 2.2 −0.4 −0.2 0.2 0.4 0.6

time

2 4 6 8 10 12 14 16 18 20 20 40 60 80 100 120 140 160 180

frequency

Audio representations

Which representation is best: time or frequency?

slide-11
SLIDE 11

Page 11 of 49

“Signals, Information and Sampling”

University of Edinburgh, 16th January, 2008

Steve McLaughlin

a Gabor ‘atom’

Time and

Audio representations

Frequency (Gabor)

“Theory of Communication,” J. I EE (London) , 1946 “… a new method of analysing signals is presented in which time and frequency play symmetrical parts…” Frequency (Hz) Time (s)

slide-12
SLIDE 12

Page 12 of 49

“Signals, Information and Sampling”

University of Edinburgh, 16th January, 2008

Steve McLaughlin

Gabor and audio coding

Modern audio coders owe as much to Gabor’s notion of time- frequency analysis as it does to Shannon’s paper of a similar title, two years later, that heralded the birth of information and coding theory.

“A Mathematical Theory of Communication,” Bell System Technical Journal, 1948.

  • C. E. Shannon

“Theory of Communication,” J. I EE (London) , 1946 “…In Part 3, suggestions are discussed for compressed transmission and reproduction of speech or music…”

Time and Frequency (Gabor)

slide-13
SLIDE 13

Page 13 of 49

“Signals, Information and Sampling”

University of Edinburgh, 16th January, 2008

Steve McLaughlin

Image representations

… Space and Scale: the wavelet viewpoint:

Images can be built of sums of wavelets. These are multi- resolution edge-like (image) functions.

“Daubechies, Ten Lectures on Wavelets,” SI AM 1992

slide-14
SLIDE 14

Page 14 of 49

“Signals, Information and Sampling”

University of Edinburgh, 16th January, 2008

Steve McLaughlin

Transform Sparsity

What makes a good transform?

50 100 150 200 250 50 100 150 200 250

“TOM” image Wavelet Domain

Good representations are efficient - Sparse!

slide-15
SLIDE 15

Page 15 of 49

“Signals, Information and Sampling”

University of Edinburgh, 16th January, 2008

Steve McLaughlin

50 100 150 200 250 50 100 150 200 250 50 100 150 200 250 50 100 150 200 250 50 100 150 200 250 50 100 150 200 250

Compressed to 3 bits per pixel

50 100 150 200 250 50 100 150 200 250 50 100 150 200 250 50 100 150 200 250 50 100 150 200 250 50 100 150 200 250

Compressed to 2 bits per pixel

50 100 150 200 250 50 100 150 200 250 50 100 150 200 250 50 100 150 200 250 50 100 150 200 250 50 100 150 200 250

Compressed to 2 bits per pixel Compressed to 1 bits per pixel

50 100 150 200 250 50 100 150 200 250 50 100 150 200 250 50 100 150 200 250 50 100 150 200 250 50 100 150 200 250

Compressed to 0.5 bits per pixel

50 100 150 200 250 50 100 150 200 250 50 100 150 200 250 50 100 150 200 250 50 100 150 200 250 50 100 150 200 250

Compressed to 0.1 bits per pixel

What is the difference between quantizing a signal/image in the transform domain rather than the signal domain?

Quantization in wavelet domain Tom’s nonzero wavelet coefficients Quantization in pixel domain

Coding signals of interest

slide-16
SLIDE 16

Page 16 of 49

“Signals, Information and Sampling”

University of Edinburgh, 16th January, 2008

Steve McLaughlin

Coding can be done by covering the set with ε-balls.

Coding signals of interest

An important question is: what are the signals of interest? If we digitize (via sampling) each signal is a point in a high dimensional vector space. e.g. a 5 Mega pixel camera image lives in a 5,000,000 dimensional space. What is a good signal model?

Geometric Model I Consider the set of finite energy signals: the L2 ball (an n-sphere). The L2 ball is NOT a good signal model! Almost all signals look like this…

slide-17
SLIDE 17

Page 17 of 49

“Signals, Information and Sampling”

University of Edinburgh, 16th January, 2008

Steve McLaughlin

Efficient transform domain representations implies that our signals of interest live in a much smaller set. These sets can be covered with much fewer ε-balls and require much fewer ‘bits’ to approximate.

Coding signals of interest

Sparse signal model L2 ball (not sparse)

slide-18
SLIDE 18

Page 18 of 49

“Signals, Information and Sampling”

University of Edinburgh, 16th January, 2008

Steve McLaughlin

For images (Olshausen and Field, Nature, 1996): For Audio (Abdallah & Plumbley, Proc. ICA 2001):

Learning better representations

) (

∑

=

k k k

t c x(t) ϕ

Recent efforts have been targeted at learn better representations for a given set of signals, x(t): That is, learn dictionaries of functions that represent signals of interest with only a small number of significant coefficients, ck.

) (t

k

ϕ

slide-19
SLIDE 19

Page 19 of 49

“Signals, Information and Sampling”

University of Edinburgh, 16th January, 2008

Steve McLaughlin

Build bigger dictionaries

Another approach is to try to build bigger dictionaries to provide more flexible descriptions. Consider the following test signal: Heisenberg’s uncertainty principle implies that a Time-Frequency analysis has:

100 200 300 400 500 600 700 800 900 1000

either good time resolution and poor frequency resolution

20 40 60 80 100 120

  • r good frequency resolution and

poor time resolution

slide-20
SLIDE 20

Page 20 of 49

“Signals, Information and Sampling”

University of Edinburgh, 16th January, 2008

Steve McLaughlin

Multi-resolution representations +

good frequency representation good time representation

Combined representation New uncertainty principles for sparse representations Heisenberg only applies to time-frequency analysis NOT time-frequency

  • synthesis. Consider a TF synthesis representation with a combination of

long (40 msec.) atoms and short (5 msec.) atoms. Finding the sparse coefficients is now a nonlinear (and potentially expensive)

  • peration.
slide-21
SLIDE 21

Page 21 of 49

“Signals, Information and Sampling”

University of Edinburgh, 16th January, 2008

Steve McLaughlin

Analysis versus synthesis

For invertible representations (e.g. wavelets) Analysis is equivalent to

  • Synthesis. For redundant representations they are not:

Analysis

Synthesis

Invertible Representations

∑ ∑

= ⇔ =

k k k n k k

t c t x n x(n) c ) ( ) ( ) ( φ ϕ

∑ ∑

= ⇔ =

k k k n k k

t c t x n x(n) c ) ( ) ( ) ( φ ϕ

Redundant Representations

X

Sparse approximation in redundant dictionaries requires a nonlinear

  • peration.

This may require an exhaustive search of all possibilities (not practical). So currently we use ‘greedy’ iterative methods.

Sparse signal set

Nonlinear Approximation Linear synthesis transform

slide-22
SLIDE 22

Page 22 of 49

“Signals, Information and Sampling”

University of Edinburgh, 16th January, 2008

Steve McLaughlin

Part I Review

Sparse Representations and Coding

  • How we represent signals is very important;
  • Sparse representations provide good compression;
  • Recent efforts have targeted bigger and better

representations;

  • Despite the linear representations nonlinear approximations

play an important role.

slide-23
SLIDE 23

Page 23 of 49

“Signals, Information and Sampling”

University of Edinburgh, 16th January, 2008

Steve McLaughlin

Part II: Sparse Representations and Sampling (compressed sensing)

slide-24
SLIDE 24

Page 24 of 49

“Signals, Information and Sampling”

University of Edinburgh, 16th January, 2008

Steve McLaughlin

Compressed sensing

Traditionally when compressing a signal (e.g. speech or images) we take lots of samples (sampling theorem) move to a transform domain and then throw most of the coefficients away!

Why can’t we just sample signals at the “Information Rate”?

  • E. Candès, J. Romberg, and T. Tao, “Robust Ucertainty principles: Exact

signal reconstruction from highly incomplete frequency information,” IEEE Trans. Information Theory, 2006

  • D. Donoho, “Compressed sensing,” IEEE Trans. Information

Theory, 2006

This is the philosophy of Compressed Sensing

slide-25
SLIDE 25

Page 25 of 49

“Signals, Information and Sampling”

University of Edinburgh, 16th January, 2008

Steve McLaughlin

Compressed sensing

The Compressed Sensing principle:

  • 1. Take a small number of linear observations of a signal

(number of observations << number of samples/pixels)

  • 2. Use nonlinear reconstruction to estimate the signal via a

transform domain in which the signal is sparse

Theoretical results

We can achieve an equivalent approximation performance to using the M most significant coefficients for an signal/image (in a sparse domain) by a fixed number of non-adaptive linear observations as long as:

  • No. of observations ~ M x log N (N, the size of full signal) and
  • for almost all (random) observations and
  • can be achieved with practical reconstruction algorithms
slide-26
SLIDE 26

Page 26 of 49

“Signals, Information and Sampling”

University of Edinburgh, 16th January, 2008

Steve McLaughlin

Compressed sensing principle

sparse “Tom”

4

Invert transform

X =

2

Observed data roughly equivalent

Wavelet image

Nonlinear Reconstruction

3

50 100 150 200 250 50 100 150 200 250
  • riginal “Tom”

Sparsifying transform

Wavelet image

50 100 150 200 250 50 100 150 200 250

1 Note that 1 is a redundant representation

  • f 2
slide-27
SLIDE 27

Page 27 of 49

“Signals, Information and Sampling”

University of Edinburgh, 16th January, 2008

Steve McLaughlin

Linear analysis transform coefficients (Method of Frames) are generally not sparse.

Sparse signal representations

Instead use sparse linear synthesis transform and invert – using various nonlinear methods from Chen & Donoho 1995

slide-28
SLIDE 28

Page 28 of 49

“Signals, Information and Sampling”

University of Edinburgh, 16th January, 2008

Steve McLaughlin

50 100 150 200 250 300 350 400 450 500 20 40 60 80 100 120 140

Sparse Subband representations

If we consider a oversampled subband analysis/synthesis model (e.g. STFT) then we ideally want to go from this to… something like this

50 100 150 200 250 300 350 400 450 500 20 40 60 80 100 120 140

slide-29
SLIDE 29

Page 29 of 49

“Signals, Information and Sampling”

University of Edinburgh, 16th January, 2008

Steve McLaughlin

Part III: Open Research Problems in Compressed Sensing

slide-30
SLIDE 30

Page 30 of 49

“Signals, Information and Sampling”

University of Edinburgh, 16th January, 2008

Steve McLaughlin

Let define our over-complete (redundant) dictionary (M > N). We want an approximate over-complete representation: such that s is sparse and e is a small approximation error One approach is to solve a penalized least squares problem (e.g. ‘Basis Pursuit De-Noising’ – Chen et al 1995) Direct solution can be computationally expensive.

Sparse signal representations enforces sparsity standard least squares term

slide-31
SLIDE 31

Page 31 of 49

“Signals, Information and Sampling”

University of Edinburgh, 16th January, 2008

Steve McLaughlin

Overcomplete Sparse Approximations

Signal space ~ RM

Set of signals of interest, say, L1 ball

Transform domain ~ RN N>M Sparse Approximation uses a nonlinear Approximation to construct a sparse representation.

Nonlinear Approximation Linear analysis transform Linear synthesis transform

slide-32
SLIDE 32

Page 32 of 49

“Signals, Information and Sampling”

University of Edinburgh, 16th January, 2008

Steve McLaughlin

Problems of Interest

Sparse Approximations

1. Overcomplete dictionary design - how overcomplete should/can a dictionary be, particularly when constrained for example to represent Time-Frequency tiling? 2. How efficient can overcomplete representations be for coding? 3. Provably good algorithms for finding optimal, or near optimal, sparse representations. We know L1 regularization and we know “Greedy” algorithms work under certain conditions. Also we know that the general problem is NP-hard. Is there a gap to be filled? For example, empirically stochastic search techniques work well, such as MCMC, however as far as I know there are no provable results for these within a given complexity. 4. relationship between sparse redundant representations, compressed sensing and super-resolution.

slide-33
SLIDE 33

Page 33 of 49

“Signals, Information and Sampling”

University of Edinburgh, 16th January, 2008

Steve McLaughlin

Current algorithms are provably good when a Restricted Isometry Property holds… For δk < ½ it has been shown that the following linear programme provides exact reconstruction (for k-exact sparse signals): and random matrices have been shown to satisfy this when:

What are good observation matrices?

slide-34
SLIDE 34

Page 34 of 49

“Signals, Information and Sampling”

University of Edinburgh, 16th January, 2008

Steve McLaughlin

Problems of Interest

Compressed Sensing

  • What are good observation matrices (currently only random is provably good).

How do we go about designing good CS observation matrices, particularly when there may be constraints on the form of the observation (as in MRI).

  • provably good decoding/approximation schemes for Compressed

Sensing/Sparse Approximation (same as for Sparse Approximations)

  • Faster good reconstruction algorithms.
  • Extensions of CS theory beyond sparsity
slide-35
SLIDE 35

Page 35 of 49

“Signals, Information and Sampling”

University of Edinburgh, 16th January, 2008

Steve McLaughlin

Part IV: Applications (Measurements and Bits)

slide-36
SLIDE 36

Page 36 of 49

“Signals, Information and Sampling”

University of Edinburgh, 16th January, 2008

Steve McLaughlin

Compressed sensing applications

Compressed Sensing provides a new way of thinking about signal acquisition. Applications areas include:

  • Medical imaging
  • Distributed sensing
  • Remote sensing
  • Very fast analogue to digital conversion

(DARPA A2I research program)

Still many unanswered questions… Coding efficiency? Restricted

  • bservation domains? Etc.
slide-37
SLIDE 37

Page 37 of 49

“Signals, Information and Sampling”

University of Edinburgh, 16th January, 2008

Steve McLaughlin

Compressed Sensing Hallmarks

Compressed Sensing changes the rules of data acquisition game by exploiting a- priori signal sparsity information (signal is compressible)

  • Hardware:

Universality – same random projections / hardware for any compressible signal class – simplifies hardware and algorithm design

  • Processing:

Information scalability – random projections ~ sufficient statistics – same random projections for range of tasks

  • reconstruction > estimation > recognition > detection

– far fewer measurements required to detect/recognize

  • Next generation data acquisition

– new imaging devices and A/D converters [DARPA] – new reconstruction algorithms – new distributed source coding algorithms [Baron et al.]

slide-38
SLIDE 38

Page 38 of 49

“Signals, Information and Sampling”

University of Edinburgh, 16th January, 2008

Steve McLaughlin

Compressed Sensing example: Magnetic Resonance Imaging

Compressed Sensing ideas can be applied to reduced sampling in Magnetic Resonance Imaging:

  • MRI samples lines of spatial frequency
  • Each line takes time and Energy
  • Each line heats up the patient!

The Logan-Shepp phantom image illustrates this: Logan-Shepp phantom We sample in this domain Spatial Fourier Transform

Haar Wavelet Transform

Logan-Shepp phantom Sparse in this domain Haar Wavelet Transform Logan-Shepp phantom Sub-sampled Fourier Transform ≈ 7 x down sampled (no longer invertible) …but we wish to sample here

slide-39
SLIDE 39

Page 39 of 49

“Signals, Information and Sampling”

University of Edinburgh, 16th January, 2008

Steve McLaughlin

Compressed Sensing in Practice: Magnetic Resonance Imaging

– Original MRI of a Mouse

slide-40
SLIDE 40

Page 40 of 49

“Signals, Information and Sampling”

University of Edinburgh, 16th January, 2008

Steve McLaughlin

Compressed Sensing in Practice: Magnetic Resonance Imaging

–MRI of Mouse with 25% Nyquist sampling and an L2 reconstruction

slide-41
SLIDE 41

Page 41 of 49

“Signals, Information and Sampling”

University of Edinburgh, 16th January, 2008

Steve McLaughlin

–MRI of Mouse with 25% Nyquist sampling and CS reconstruction

Compressed Sensing in Practice: Magnetic Resonance Imaging

slide-42
SLIDE 42

Page 42 of 49

“Signals, Information and Sampling”

University of Edinburgh, 16th January, 2008

Steve McLaughlin

Single Pixel Camera

  • Encoder integrates sensing, compression, processing

Directly acquires random projections of a scene without first collecting the pixels/voxels, employing a digital micromirror array to optically calculate linear projections of the scene onto pseudorandom binary patterns. Ability to obtain an image or video with a single detection element while measuring the scene fewer times than the number of pixels/voxels. Since the camera relies on a single photon detector, it can also be adapted to image at wavelengths where conventional CCD and CMOS imagers are blind.

slide-43
SLIDE 43

Page 43 of 49

“Signals, Information and Sampling”

University of Edinburgh, 16th January, 2008

Steve McLaughlin

Second Image Acquisition

slide-44
SLIDE 44

Page 44 of 49

“Signals, Information and Sampling”

University of Edinburgh, 16th January, 2008

Steve McLaughlin

A/D Conversion Below Nyquist Rate

  • Challenge:

– wideband signals (radar, communications..) – currently impossible to sample at Nyquist rate

  • Proposed CS-based solution:

– sample at “information rate” – simple hardware components – good reconstruction performance reported

slide-45
SLIDE 45

Page 45 of 49

“Signals, Information and Sampling”

University of Edinburgh, 16th January, 2008

Steve McLaughlin

Slepian-Wolf Theorem (Distributed lossless coding)

  • Consider a communication system with two correlated signals, X and Y Assume

they come from separate sources that cannot communicate, so the signals are encoded independently or are “distributed”. The receiver can see both encoded signals and can perform joint decoding.

  • A sensor system composed of low-complexity spatially separated sensor nodes,

sending correlated information to a central processing receiver, is an example of such system.

  • What is the minimum encoding rate required such that X and Y can still be

recovered perfectly?

slide-46
SLIDE 46

Page 46 of 49

“Signals, Information and Sampling”

University of Edinburgh, 16th January, 2008

Steve McLaughlin

Slepian-Wolf Theorem (Distributed lossless coding)

  • The Slepian-Wolf Theorem, established in 1971, shows it is possible to

transmit X and Y without any loss if all of the following hold true: R1 > H(X1|X2) (conditional entropy) R2 > H(X2|X1) (conditional entropy) R1+R2 > H(X1,X2) (joint entropy)

slide-47
SLIDE 47

Page 47 of 49

“Signals, Information and Sampling”

University of Edinburgh, 16th January, 2008

Steve McLaughlin

CS Approach- Measure separately, reconstruct jointly

  • Zero collaboration, trivially scalable, robust
  • Low complexity, universal encoding
  • Ingredients

– models for joint sparsity – algorithms for joint reconstruction – theoretical results for measurement savings

  • The power of random measurements

– single-signal: efficiently capture structure without performing the sparse transformation – multi-signal: efficiently capture joint structure without collaborating or performing the sparse transformation

slide-48
SLIDE 48

Page 48 of 49

“Signals, Information and Sampling”

University of Edinburgh, 16th January, 2008

Steve McLaughlin

Summary

Sparse representations provide a powerful mathematical model for many natural signals in signal processing and are a basis for:

  • good compression;
  • good source separation and;
  • efficient sampling

There is an interesting interplay between linear representations and nonlinear approximation Compressed sensing is only in its infancy… For more info see:

http://www.dsp.ece.rice.edu/cs/