In the name of Allah the compassionate, the merciful D IGITAL V IDEO - - PowerPoint PPT Presentation

In the name of Allah the compassionate, the merciful

DIGITAL VIDEO SYSTEMS

• S. Kasaei

Room: CE 307 Department of Computer Engineering Sh if U i it f T h l Sharif University of Technology E-Mail: skasaei@sharif.edu Webpage: http://sharif.edu/~skasaei Webpage: http://sharif.edu/ skasaei

• Lab. Website: http://ipl.ce.sharif.edu

ACKNOWLEDGMENT ACKNOWLEDGMENT

Most of the slides used in this course have been provided by: Prof Yao Wang (Polytechnic provided by: Prof. Yao Wang (Polytechnic University, Brooklyn) based on the book: Video Processing & Communications g by: Yao Wang, Jom Ostermann, & Ya-Oin Zhang Prentice Hall, 1st edition, 2001, ISBN: 0130175471. [SUT Code: TK 5105 2 W36 2001] [SUT Code: TK 5105 .2 .W36 2001].

CHAPTER 14

Error Control in Video Communications

OUTLINE OUTLINE

Necessity & challenge for error control

Kas

y g

Characteristics of typical applications & networks Overview of techniques

saei

Error resilient encoding Error concealment Encoder-decoder-network interactive error control

6

VIDEO COMMUNICATION SYSTEM VIDEO COMMUNICATION SYSTEM

Kasaei

A typical video communication system.

7

CHALLENGE FOR VIDEO COMMUNICATIONS COMMUNICATIONS

Effective video communications include:

Kas

Reduction of video data rate. Handling errors & losses in communication networks.

Data communications are not usually subject to

saei

Data communications are not usually subject to

strict delay constraints.

Thus can be handled using network protocols (that use

resubmission to ensure error-free delivery. resubmission to ensure error free delivery.

Transmission error categories are:

Random bit errors, caused by imperfections of physical

, y p p y channels; bit inversion/insertion/deletion.

Erasure errors, caused by packet lost in packet networks,

long burst errors in storage media due to physical defects, system failures/link downs random bit errors in VLC system failures/link downs, random bit errors in VLC coded streams.

8

CHALLENGE FOR VIDEO COMMUNICATIONS COMMUNICATIONS

Real networks are unreliable

Kas

Wireless networks: random bit errors, long burst

errors, & possibly link downs.

Internet: packet loss & variable delay due to

saei

Internet: packet loss & variable delay due to network congestion.

Excessive delay = loss for real-time applications. Real networks are heterogeneous in

bandwidth & reliability.

Video data are delay-sensitive: One cannot rely on retransmission for error

control because of the stringent delay control because of the stringent delay requirement!

9

CONVENTIONAL SOURCE CODING TECHNIQUE

IS NOT GOOD ENOUGH IS NOT GOOD ENOUGH

Conventional source coding techniques

Kas

g q difficulties:

Optimal performance is obtained only for fixed rate &

perfect channels.

saei

It results in a poor reconstruction quality when parts

• f the coded data are lost.

10

CONVENTIONAL SOURCE CODING TECHNIQUE

IS NOT GOOD ENOUGH IS NOT GOOD ENOUGH

Compressed video data is very sensitive to

Kas

transmission errors because of:

Variable length coding. Temporal predictive coding.

S ti l di ti di

saei

Spatial predictive coding. All contribute to error propagation within the same frame

as well as the following frames:

1 bit error or packet loss can render the following 1 bit error or packet loss can render the following

received data useless.

11

SPATIAL/TEMPORAL ERROR PROPAGATION SPATIAL/TEMPORAL ERROR PROPAGATION

Kasaei

Illustration of spatiotemporal error propagation Illustration of spatiotemporal error propagation.

12

DRIFT PROBLEM (REFERENCE MISMATCH) (REFERENCE MISMATCH)

Motion compensated temporal prediction

Kas

Motion compensated temporal prediction

should be retained to preserve the coding efficiency.

saei

efficiency.

Loss in a previous frame can cause

mismatch between the reference frame used in the encoder & that in the decoder:

Encoder & decoder out of sync.

13

DRIFT PROBLEM (REFERENCE MISMATCH) (REFERENCE MISMATCH)

Kasaei

Distortion

referen

Satellite dish

refere nce

transmission

ence

14 encoder

decoder

EFFECT OF TRANSMISSION ERRORS EFFECT OF TRANSMISSION ERRORS

Kasaei

Coded, no loss 3% 5% 10%

15

Reconstructed video frames using a H.263 coded sequence, subject to packet losses.

CHANNEL CODING BASICS CHANNEL CODING BASICS

Channel coding; forward error correction

Kas

C g; f (FEC):

Adding redundancy bits on compressed source

bit to e able e

• detectio & co

ectio

saei

bits to enable error detection & correction.

Simple example:

Adding a parity check bit at the end of a block of data

stream, can detect all single bit errors.

Channel coding rate:

For every k source bits, add l channel bits, to create

n=k+l bits channel coding rate r=k/n.

Well designed code (e.g., Reed-Solomon (RS) code) can

correct t=l/2 error bits in each n-bit block.

16

CHANNEL CODING BASICS CHANNEL CODING BASICS

Classical Shannon information theory states that

l d i h & h l

Kas

• ne can separately design the source & channel

coders, to achieve error-free delivery of a compressed bit stream, as long as the source is represented by a rate below the channel capacity

saei

represented by a rate below the channel capacity.

Source coding minimizes the bit rate necessary to satisfy a

distortion criterion (Shannon rate-distortion theory).

Channel coding adds just enough redundancy bits to

g j g y reduce the raw channel error rate to the permitted level.

Such ideal error-free delivery requires infinite

d l i i l i (FEC) delays in implementing (FEC).

Only valid for stationary source & channel, requires

processing of infinitely long blocks of data (delay = infinity!) infinity!).

17

FEC FOR VIDEO TRANSMISSION FEC FOR VIDEO TRANSMISSION

For wireless networks, FEC is necessary to reduce

Kas

raw bit error rates.

For the Internet, errors are mainly due to congestion-

caused packet losses, FEC can be applied across

saei

packets to correct/detect packet losses.

Unequal error protection (UEP): using stronger

channel codes (r=k/n smaller) for more “important” bit t (b l ) B t i l t d ith t bit streams (base layer). Best implemented with rate- compatible punctured convolutional (RCPC) code.

18

TRANSMISSION ERROR RESILIENCE TRANSMISSION ERROR RESILIENCE

Joint source & channel coding is often a

Kas

g more viable scheme (which allocates a total amount of redundancy between the source & channel coding).

saei

so ce & c a e co g).

All error resilient encoding methods work

d thi i under this promise.

This is usually accomplished by carefully This is usually accomplished by carefully

designing both the predictive coding loop & the VLC, to limit the extend of error propagation propagation.

19

ERROR CONCEALMENT ERROR CONCEALMENT

When a part of image is missing due to

i i h d d i h

Kas

transmission errors, the decoder estimates them based on surrounding received samples (using the inherent correlation among spatially & temporally adjacent samples)

saei

adjacent samples).

It does not employ any additional bit rate. ☺

But adds computational complexity at the decoder But adds computational complexity at the decoder.

To facilitate error concealment in the decoder, the

codec & the network transmission protocol must p

• cooperate. For example:

Important parts should be assigned a more stringent set

• f QoS parameters.

N t k id f db k h l ( th d

Network may provide a feedback channel (so the encoder

does not use this part for prediction).

20

TYPICAL VIDEO APPLICATIONS & COMMUNICATION NETWORKS & COMMUNICATION NETWORKS

Some basic definitions:

Kas

Latency: maximally allowed average end-to-end delay. Jitter: delay variation.

R l ti d li d d t t f d t

saei

Real-time delivery: compressed data are transferred at

a speed that matches the coded video source rate.

21

QOS REQUIREMENTS

OF TYPICAL VIDEO APPLICATIONS OF TYPICAL VIDEO APPLICATIONS

Video applications consist of:

Kas

pp

Interactive two-way visual communications. One-way video streaming.

O id d l di

saei

One-way video downloading.

No difference from file downloading.

22

INTERACTIVE TWO-WAY VISUAL COMMUNICATIONS COMMUNICATIONS

Examples include: teleconferencing, video

telephony virtual classrooms

Kas

telephony, virtual classrooms.

They have very stringent delay requirements.

saei

The latency & jitter must be kept within a certain

limit.

For intercontinental telephone conversations (&

video conferencing), the ITU-T G.114 standard recommends:

<=150 ms (one way) desired <=150 ms (one way) desired. 150-400 ms can be acceptable. >400 ms not acceptable. Audio & video must be in sync to maintain lip

h i ti synchronization.

Both encoding & decoding must be completed in real-time.

23

INTERACTIVE TWO-WAY VISUAL COMMUNICATIONS COMMUNICATIONS

Only low-to-intermediate video quality is

Kas

O y q y required: ☺

QCIF at 5-10 fps acceptable for video telephony.

CIF f i f f id

saei

CIF at 10-20 fps satisfactory for video

conferencing.

Moderate amount of compression/transmission

p artifacts can be tolerated.

R

id h li it d ti i t

Raw video has limited motion

easier to code & conceal errors.

24

ONE-WAY VIDEO STREAMING ONE WAY VIDEO STREAMING

Examples include: TV broadcasting,

Kas

p g, Multicast of a conference/event, Video streaming over the Internet.

saei

Except for live broadcast/multicast/

unicast, it can precompress the video, but decoding must be done in real-time decoding must be done in real-time.

Initial play-out delay can be up to a few

seconds: seconds:

Receiver uses a large smoothing buffer to store

several seconds of video frames before starting to display the first received frame to display the first received frame.

25

ONE-WAY VIDEO STREAMING ONE WAY VIDEO STREAMING

Bit rate (video quality) can vary widely

Kas

( q y) y y depending on the application.

saei

Recipients of the same video source may

be connected to the network with different access links (e g wireless modem to 100 access links (e.g., wireless modem to 100 mbps fast Ethernet) & the receiving terminal may have varying computing power (palm vs. laptop vs. desktop):

Scalable coding is desired.

26

ONE-WAY VIDEO DOWNLOADING ONE WAY VIDEO DOWNLOADING

Precoded video (& audio) is downloaded to a

Kas

( ) destination.

saei

Receiving terminal starts to play the

material when the entire video has arrived.

It is least demanding in terms of delay

requirement although excessive delay may requirement, although excessive delay may cause the receiver to abort the downloading.

27

ONE-WAY VIDEO DOWNLOADING ONE WAY VIDEO DOWNLOADING

Scalable coding & progressive transmission

Kas

g p g are desirable.

saei

Retransmission can be used to handle any

lost or erroneously delivered part of the data data.

No special error control mechanisms are No special error control mechanisms are

necessary, beyond those typically used for data transfer.

28

MAJOR TYPES

OF COMMUNICATION NETWORKS OF COMMUNICATION NETWORKS

Plain old telephone system (PSTN):

Kas

p y ( )

Modem link, 56 kbps, QCIF. Asymmetric digital subcarrier loop (ADSL):

saei

Asymmetric digital subcarrier loop (ADSL): Streaming MPEG-1 & MPEG-2 movies through

video-on-demand services.

Integrated service digital network (ISDN): circuit switching (a particular conversation once circuit switching (a particular conversation once

the connection is established), very reliable, px64 kbps (p=1-24), QCIF @ 5-10 fps for videophone, CIF@ 15-30 fps for video conferencing CIF@ 15-30 fps for video conferencing.

29

MAJOR TYPES

OF COMMUNICATION NETWORKS OF COMMUNICATION NETWORKS

Broadband ISDN (B-ISDN):

Kas

( )

Virtual ATM packet-switching using fixed-size small cells,

quite reliable.

Cell loss rate (CLR) 10-6 -10 -4, >=155 mbps, real-time

saei

applications.

The Internet:

Is composed of multiple physical networks interconnected

by computers (routers).

Most widely used physical network for supporting local

area network (LAN) is the Ethernet which has a rate of area network (LAN) is the Ethernet which has a rate of 10-100 mbps.

Interconnected LANs in a local region form a metropolitan

area network (MAN). area network (MAN).

Several interconnected MANs form a wide area network

(WAN).

30

MAJOR TYPES

OF COMMUNICATION NETWORKS OF COMMUNICATION NETWORKS

Transport of packets is governed by the TCP/IP

Kas

p p g y protocol suite which includes:

Internet protocol (IP). User datagram protocol (UDP); for unreliable connectionless

saei

g p ( ) packet-delivery service.

Transmission control protocol (TCP); for reliable stream

service, guarantees delivery by using automatic t i i t (ARQ) retransmission request (ARQ).

IP is the lower layer protocol on top of which UDP & TCP

• perate.

31

MAJOR TYPES

OF COMMUNICATION NETWORKS OF COMMUNICATION NETWORKS

The Internet & its associated TCP/IP protocol was

Kas

• riginally designed for data communications which are

not delay sensitive.

For real-time applications the retransmission

h i i TCP i t i t

saei

mechanism in TCP is not appropriate.

New protocols have been developed for real-time data

transport:

R l ti t t t l (RTP) & it i

Real-time transport protocol (RTP) & its companion

real-time control protocol (RTCP).

Real-time streaming protocol (RTSP).

Et

Etc.

32

MAJOR TYPES

OF COMMUNICATION NETWORKS OF COMMUNICATION NETWORKS

Wireless networks:

Kas

Cellular networks: mobile voice, wireless

modems, under 20 kbps, can be connected to a packet-based wireless data network (64-384

saei

packet based wireless data network (64 384 kbps) [vehicular].

Wireless LAN: mobile IP, wireless ATM, IEEE

802 11b: 11 mbps 802 11a: 56 mbps [indoor] 802.11b: 11 mbps, 802.11a: 56 mbps [indoor].

Broadband wireless IP networks: access to the

Internet through small rooftop antennas i ti t b t ti i communicating to a base station using microwave radio, multichannel multipoint distribution service (MMDS): 1mbps, local l i i di ib i i (LMDS) 100 600 multipoint distribution service (LMDS): 100-600 mbps [indoor].

33

CHARACTERISTICS OF MAJOR VIDEO COMMUNICATIONS APPLICATIONS COMMUNICATIONS APPLICATIONS

Kasaei

34

H.262: MPEG-2, H.222.0: MPEG2 systems, BER: bit-error rate.

ERROR CONTROL TECHNIQUES

FOR VIDEO FOR VIDEO

Transport-level error control techniques

Kas

Transport level error control techniques

(applied on coded video streams):

Forward error correction: error detection & correction through

FEC; increases transmission overhead.

saei

Error-resilient packetization & multiplexing: error can be

isolated within a small region, unequal-sized packets.

FEC & error-resilient are low level error control mechanisms.

35

ERROR CONTROL TECHNIQUES

FOR VIDEO FOR VIDEO

Delay-constrained retransmission: retransmission of lost

k f l i d i i i

Kas

packets; for non-real-time data transmission, automatic retransmission request (ARQ).

For real-time data transmission: Delay-constrained retransmission.

C bi i f l d di i h i i i d i i

saei

Combination of layered coding with prioritized retransmission

(by placing base layer- packets in front of the transmission queue to increase the number of retransmission trials for the base layer).

Sending multiple copies of a lost packet.

g p p p

Unequal error protection: picture header & other side

information are protected to be delivered with a much lower error rate.

Retransmission & strong FEC can be applied for the base layer.

g y

Using different transport protocols.

36

ERROR CONTROL TECHNIQUES

FOR VIDEO FOR VIDEO

Error resilient encoding (ERE) techniques:

Kas

Error-resilient encoding (ERE) techniques: Source coding methods (robust to transmission

errors):

saei

)

Basic idea: intentionally insert redundancy in source

coding to help the decoder to recover bit errors & packet from transmission errors. p

Design goal: minimize the redundancy to achieve a

desired level of resilience.

These techniques include: These techniques include:

Error isolation (part of H.263/MPEG-4 standard):

inserting resynchronization markers, data partitioning.

Robust binary encoding (part of H.263/MPEG-4

y g (p standard): error resilient entropy coding (EREC), reversible VLC (RVLC).

37

ERROR CONTROL TECHNIQUES

FOR VIDEO (ERE) FOR VIDEO (ERE)

Kasaei

RVLC codewords can be parsed in both forward & backward directions (making it possible to recover more data from a corrupted data stream). MPEG-4 syntax is assumed in the figure, but the basic principle holds true also for

38

• ther RVLC-coded data.

ERROR CONTROL TECHNIQUES

FOR VIDEO (ERE) FOR VIDEO (ERE)

Error-resilient prediction (part of H.263/MPEG-4

Kas

standard): insertion of intra-blocks or frames, independent segment prediction (spatiotemporal prediction within same segments).

Joint source & channel coding; quantization & entropy

saei

Joint source & channel coding; quantization & entropy

coder design at the source coder & FEC & modulation schemes at the channel coder.

39

ERROR CONTROL TECHNIQUES

FOR VIDEO (ERE) FOR VIDEO (ERE)

Layered coding with unequal error protection:

Kas

y g q p

The most promising approach for combating channel

errors.

Base layer provides acceptable quality, enhancement saei

layer refines the quality.

Base layer transmitted in error-free channel (strong

FEC + ARQ).

Good for a network with differentiated service (do

NOT exist today over Internet, may become part of emerging wireless standards).

Problems: Problems:

• Any error in the base layer causes severe

degradation.

• Repetitive ARQ may incur unacceptable delay, strong

Repetitive ARQ may incur unacceptable delay, strong FEC may be too complex or cause extra delay.

• The enhancement layer is useless by itself.

40

ERROR CONTROL TECHNIQUES

FOR VIDEO (ERE) FOR VIDEO (ERE)

Kasaei

Block diagram of a system using layered coding & prioritized transport.

41

ERROR CONTROL TECHNIQUES

FOR VIDEO (ERE) FOR VIDEO (ERE)

Multiple-description (bit stream) coding (MDC); several

l d d i i (bi ) f h

Kas

correlated descriptions (bit streams) of the same source signal are generated & transmitted over separate channels.

Any description provides low but acceptable quality.

Additi l d i ti id i t l

saei Additional descriptions provide incremental

improvements.

No retransmission required -> low delay ☺ However, correlation -> reduced coding efficiency Design goal: maximize the robustness to channel

errors at a permissible level of redundancy.

Assumptions:

• Existence of multiple channels between source & destination.
• Independent error & failure events.
• Probability that all channels fail simultaneously is low.
• Good model for the Internet & wireless networks when data

are properly packetized & interleaved.

42

GENERIC TWO DESCRIPTION CODER GENERIC TWO DESCRIPTION CODER

Kasaei

Decoder 1

Decoded Signal S1

MDC Encoder Channel 1 Decoder Ch l 2

Source Signal

from S1 (D1). D d d S1 (R1)

Decoder Channel 2

Decoded Signal from S1 & S2 (D0). S2 (R2)

Decoder 2

Decoded Signal from S2 (D2) Balanced MDC: R1=R2, D1=D2. R: bit rate, D: distortion, S: signal. 43 (D2).

MDC Decoder

R: bit rate, D: distortion, S: signal.

REDUNDANCY RATE-DISTORTION REDUNDANCY RATE DISTORTION

Kas

D0 D1

saei

Rate-Distortion (RD) Function

D1

Redundancy Rate-Distortion (RRD) Function

D0

*

D R

*( )

(SDC) D R

0 ( ) (MDC)

ρ D D

1

( ; )

*

ρ Design criteria for MD coders:

R R* R

D R

0 ( ) (SDC)

ρ ρ Design criteria for MD coders:

Minimize D1 for a given ρ, for fixed R* or D0

*

(minimizing the average distortion given channel loss rates for a given total rate) loss rates, for a given total rate).

Can easily vary the ρ vs. D1 trade-off to match

network conditions.

44

CHALLENGE IN DESIGNING MD VIDEO CODER CHALLENGE IN DESIGNING MD VIDEO CODER

To achieve high coding efficiency, the

Kas

g g y, encoder should retain the temporal prediction loop. Sh ld h di i b b d

saei

Should the prediction be based on

reconstruction from both descriptions or individual descriptions? individual descriptions?

Prediction based on two-description

reconstruction:

Higher prediction efficiency. Mismatch problem at decoder.

45

CHALLENGE IN DESIGNING MD VIDEO CODER CHALLENGE IN DESIGNING MD VIDEO CODER

Prediction based on single-description

Kas

g p reconstruction:

Lower prediction efficiency.

N i h bl

saei

No mismatch problem. How to provide a proper trade-off between

prediction efficiency & mismatch: prediction efficiency & mismatch:

Predict based on two-description

reconstruction, but explicitly code the i t h mismatch error.

46

VIDEO REDUNDANCY CODING

IN H.263+ IN H.263+

Coding even frames & odd frames as separate

Kas

Coding even frames & odd frames as separate

threads:

High redundancy (~30%) due to reduced prediction

i b f l di t b t f

saei

gain; because of longer distance between frames.

Hard to vary the redundancy based on channel loss

characteristics.

Even frames.

47

Odd frames.

MULTIPLE DESCRIPTION MOTION COMPENSATION (WANG & LIN, 2001) COMPENSATION (WANG & LIN, 2001)

A description contains even (or odd) frames only,

but each frame is predicted (central predictor)

Kas

but each frame is predicted (central predictor) from both even & odd past frames.

Code the central prediction error:

saei

Code the central prediction error:

Sufficient if both descriptions are received.

To avoid mismatch a side predictor for even To avoid mismatch, a side predictor for even

frames predicts only from the past even frame, & the mismatch signal (difference between central & side prediction) is also coded.

The predictors & the mismatch error quantizer

control the redundancy of the coder, & can be designed based on the channel loss designed based on the channel loss characteristics.

48

SPECIAL CASE: TWO-TAP PREDICTOR

Kas

MV1

1

a MV2 ,

2

a

saei

~ ) ( ˆ ) ( : error prediction Central ) 2 ( ~ ) 1 ( ~ ) ( ˆ : predictor Central

2 1

→ − = − + − = (n) e n n (n) e n a n a n ψ ψ ψ ψ ψ MV1 ,

1

a MV2 MV1, , ~ , ~ : Send ~ ) ( ˆ ) ( ˆ : error Mismatch ) 2 ( ~ ) ( ˆ : predictor Side

1 1 1 1 1 3 1

→ − − = − = (n) e (n) e (n) e (n) q n n (n) e n a n ψ ψ ψ ψ

n-3 n-2 n-1 n

1 , 1 : predictor leaky

• Non

, , ,

3 2 1 1

= = + a a a ( ) ( )

)) 2 ( ) 1 ( b th (h i d d i ti b th If

MV2 ,

3

a

)) 2 (

• nly

(have received is n descriptio

• ne

If ) ( ) ( ~ ) ( ˆ ) ( )) 2 ( ), 1 ( both (have received ns descriptio both If

1

n n q n (n) e n n n n − + = + = − − ψ ψ ψ ψ ψ ψ

49

) ( ) ( ~ ~ ) ( ˆ ) (

1 1 1 1

n q n (n) e (n) e n n + = + + = ψ ψ ψ

RRD PERFORMANCE

OF VRC & MDMC OF VRC & MDMC

Kasaei

50

SAMPLE RECONSTRUCTED FRAMES

(10% RANDOM PACKET LOSS TOP: MDMC BOTTOM: VRC) (10% RANDOM PACKET LOSS, TOP: MDMC, BOTTOM: VRC)

Kasaei

51

ERROR CONTROL TECHNIQUES

FOR VIDEO FOR VIDEO

Decoder error concealment (DEC):

Kas

Decoder error concealment (DEC):

Due to various constrains (coding delay,

implementation complexity, availability of a good d l) d d id bi ill

saei

source model) a coded video bit stream still possesses a certain degree of statistical redundancy.

Error concealment recovers lost/damaged regions

based on image/video characteristics & human visual system (HVS) properties at the decoder. y ( ) p p

Decoder optimization issue, is not a part of video

coding standard! D d th k t diff i th i

Decoders on the market differ in their error

concealment capabilities.

52

SAMPLE ERROR CONCEALMENT RESULTS SAMPLE ERROR CONCEALMENT RESULTS

Kasaei

Without concealment. With concealment.

53

ERROR CONTROL TECHNIQUES

FOR VIDEO (DEC) FOR VIDEO (DEC)

Basic idea:

Kas Recover damaged regions by interpolating from

surrounding (in the same frame & in nearby frames) regions.

Basic error concealment techniques include: saei Basic error concealment techniques include:

Recovery of texture information: motion-compensated

temporal interpolation, spatial interpolation, maximally smooth recovery (spatiotemporally smoothly connected pels). pels).

Motion-compensated temporal interpolation:

• Replaces a damaged MB by its corresponding MB in the

reference frame. If the MV is also lost it has to estimate the MV first;

• If the MV is also lost, it has to estimate the MV first;

typically by copying the MV of the MB above.

• Simple & quite effective, if the data was appropriately

partitioned.

Maximally smooth recovery (Wang/Zhu 1993): Maximally smooth recovery (Wang/Zhu, 1993):

• Estimates the missing DCT coefficients in a block so

that a combination of spatial & temporal smoothness measure is maximized.

54

ERROR CONTROL TECHNIQUES

FOR VIDEO (DEC) FOR VIDEO (DEC)

Recovery of coding modes & motion vectors: mean, median,

Kas

set to zero.

Syntax-based repair: directly detects & repairs errors in the

bit stream (due to few valid combinations of codewords).

saei

55

ERROR CONTROL TECHNIQUES

FOR VIDEO FOR VIDEO

Encoder-decoder-network interactive error control

Kas

Encoder decoder network interactive error control

(EDNIER):

Sender & receiver cooperate in the process of error

control using a backward channel from the decoder to

saei

g the encoder.

It can be realized at either the source coding or the

transport-level.

At the source coder, coding parameters can be adapted

based on the feedback information from the decoder.

At the transport-level, the feedback information can be

employed to change the percentage of the total employed to change the percentage of the total bandwidth used for FEC or retransmission.

Feedback message is not a part of the video syntax but

is transmitted in a different layer of the protocol stack (in which control information is exchanged) (in which control information is exchanged).

Feedback channel not necessarily involves extra

coding delay.

56

ERROR CONTROL TECHNIQUES

FOR VIDEO FOR VIDEO

B i d d d t k i t ti t l

Kas

Basic encoder-decoder-network interactive error control

techniques include:

Coding-parameter adaptation based on

h l di i i di ll i &

saei

channel conditions: periodically estimate & update the QoS parameters (bandwidth, delay, packet loss rates), appropriately y, p ), pp p y determine the coding parameters (intra- block rate, frequency of synchronization markers range of prediction) markers, range of prediction).

Changes intra-rate based on average loss rates.

57

ERROR CONTROL TECHNIQUES

FOR VIDEO (EDNIER) FOR VIDEO (EDNIER)

Reference picture selection based on feedback

Kas

Reference picture selection based on feedback

information (part of H.263/MPEG-4 standard): decides to code the next P-frame relative not to the most recent frame: F ll i g d g d f (f db k i f f

saei Following a damaged frame (feedback info from

receiver), uses undamaged previous frame as reference frame for temporal prediction.

Error tracking based on feedback information:

determines which MBs are affected following a lost MB (feedback info), avoids using those MBs as reference pixels.

Requires a feedback channel, not necessarily involving

q , y g extra coding delay.

Retransmission without waiting. Multiple path transport with multiple bit stream coding.

58

ERROR CONTROL TECHNIQUES

FOR VIDEO (EDNIER) FOR VIDEO (EDNIER)

Kasaei

Illustration of error propagation when error tracking is used & damaged blocks are intra-coded upon the receipt of the negative acknowledgement (NACK) information .

59

VIDEO TRANSPORT USING PATH DIVERSITY VIDEO TRANSPORT USING PATH DIVERSITY

Kas

Wireless ad hoc networks

saei

Wireless ad hoc networks, cellular/WLAN with MIMO antennas, Internet with different relay servers.

L a p t o p c o m p u t e r L a p t o p c o m p u t e r

A

MD/Layered Encoder

L a p t o p c o m p u t e r L a p t o p c o m p u t e r L a p t o p c o m p u t e r

A B

MD/Layered Decoder

60

L a p t o p c o m p u t e r

WHY USING MULTIPLE PATHS WHY USING MULTIPLE PATHS

Enables sending high bandwidth signal

Kas

g g g (video) that exceeds the capacity of a single path (e.g., low bandwidth wireless links).

saei

s).

Can withstand individual path failures in

i l t k i d l d wireless networks or excessive delay due to congestion on a particular path in the Internet.

Enables traffic dispersion & load

balancing which in turn helps to reduce balancing, which in turn helps to reduce congestion & consequently packet losses.

61

WHY USING MULTIPLE PATHS WHY USING MULTIPLE PATHS

If one path is known to be better than the

Kas

If one path is known to be better than the

• ther, or if it is feasible to set up such a

path, use layered coding.

saei

If the paths are symmetric in QoS & no

p y Q reliable path can be set up, use MDC.

62

VIDEO OVER AD HOC NETWORKS

JOINT WORK WITH SHIVENDRA PANWAR SHUNAN LIN SHIWEN MAO JOINT WORK WITH SHIVENDRA PANWAR, SHUNAN LIN, SHIWEN MAO Wireless ad hoc networks:

Kas

Has no fixed infrastructure, peer to peer. Neighboring nodes are continuously changing.

saei

Multiple paths exit between two end users &

many proposed routing protocols return multiple paths to destination.

A path may become invalid during a

connection due to a link down.

Proposed solution: Integration of multistream

coding with multipath transport.

63

VIDEO OVER AD HOC NETWORKS

JOINT WORK WITH SHIVENDRA PANWAR SHUNAN LIN SHIWEN MAO JOINT WORK WITH SHIVENDRA PANWAR, SHUNAN LIN, SHIWEN MAO

Scope of the Project:

Kas

p j

Set-up/update an active path set (Haas,

Cornell).

QoS monitoring of existing paths (Panwar

saei

QoS monitoring of existing paths (Panwar,

Poly).

MDC/LC based on QoS parameters of active

th (W P l ) paths (Wang, Poly).

Testbed development.

64

PROPOSED SOLUTION:

MULTISTREAM CODING + MULTIPATH TRANSPORT

Kasaei

65

HOW TO GENERATE MULTIPLE SOURCE

STREAMS? STREAMS?

Trade-off between coding efficiency & error-

Kas

g y resilience.

We examine three types of coding & transport

saei

yp g p schemes:

Multiple description coding. Layered coding with selective ARQ.

y g Q

Reference picture selection based on channel feedback. The three schemes differ in:

Requirement for a feedback channel.

q

Delay. Buffer requirement. Video quality under different channel loss patterns.

66

LAYERED CODING WITH SELECTIVE ARQ LAYERED CODING WITH SELECTIVE ARQ

Kasaei

better path lost base layer

Base layer is sent on the better path, lost base layer packets are retransmitted over the enhancement layer path, while dropping the corresponding enhancement layer

67

• packets. Redundancy is due to prediction using base layer reconstruction only.

REFERENCE PICTURE SELECTION REFERENCE PICTURE SELECTION

Kasaei

not referred to Even/odd frames are sent on separate paths. Predicts damaged frames based on NACK on each path, & uses undamaged frames as reference pictures

68

frames as reference pictures. Compatible with the RPS option in H.263+ standard. ☺

SIMULATION RESULTS

BASED ON MARKOV LINK MODELS BASED ON MARKOV LINK MODELS

Path model: Each path consists of three links,

Kas

p , chosen randomly from an available link pool.

Each link is modeled by a 3-state Markov chain (down,

poor, good).

saei

The path is updated frequently.

We run a large set of tests to compare the three

schemes under different packet loss patterns. schemes under different packet loss patterns.

We investigate the influence of the following

channel characteristics:

Mea lo ate

Mean loss rate. Mean burst length. Symmetric vs. asymmetric paths.

69

When no feedback or retransmission is allowed, MDC

is significantly better than LC.

Otherwise, LC+ARQ & RPS are better:

RPS is best at low loss rates.

LC ARQ i b tt t hi h l t ( t th f

LC+ARQ is better at higher loss rates (at the expense of

extra delay).

Simulation Results based on Markov Link Models

The performance of all three schemes improves gradually

Kas

The performance of all three schemes improves gradually

when the burst length increases:

When the loss rate is the same, longer bursts means fewer

frames are effected (bursty error is better than random error

saei

frames are effected (bursty error is better than random error for compressed video!)

Longer burst lengths increase the diversity gain from using

two paths.

This trend reverses when the burst length exceeds more

than one frame time.

When the average loss rate is the same the three When the average loss rate is the same, the three

schemes perform similar in the symmetric vs. asymmetric cases.

71

OPNET SIMULATION RESULTS OPNET SIMULATION RESULTS

Kas

• 16 nodes, 600 m by 600 m, 10 m/s, transmission range=250 m, multipath

ti d i ti (DSR)

saei

routing dynamic source routing (DSR):

72

Mobility can actually help!

A WIRELESS VIDEO STREAMING TESTBED A WIRELESS VIDEO STREAMING TESTBED

Kasaei

73

The current testbed consists of 4 mobile nodes: 1 source, 1 destination, 2 relays.

MORE ON THE TESTBED MORE ON THE TESTBED

Testbed Setup:

Kas

Testbed Setup:

Each mobile node is a Thinkpad notebook computer

equipped with IEEE 802.11b cards working in the ad hoc mode.

saei

The source node sends a pre-encoded compressed bit

stream (MDMC or LC) to the destination node via two separate relays. Th d ti ti d bl th i d th k t

The destination node assembles the received the packets

from two paths, decodes, displays the video in real-time (10 fps).

With the LC scheme the destination node detects & With the LC scheme, the destination node detects &

requests retransmission of lost BL packets, the source node resend appropriate BL packets & drops corresponding EL packets.

74

MORE ON THE TESTBED MORE ON THE TESTBED

Results:

Kas

Results:

Tested when the nodes are placed in a Polytechnic

building.

With slow moving (walking) of the nodes & limited

saei

g ( g) interference from other traffic, we can get good video quality within both 2s & 300 ms playout delay.

Results are fairly consistent with the simulations results

i th M k d l i ll ith th MDMC using the Markov model, especially with the MDMC scheme.

75

SUMMARY OF THE VIDEO OVER AD HOC PROJECT AD HOC PROJECT

Multiple path transport is feasible &

Kas

p p p effective for increasing both throughput & reliability in ad hoc networks.

saei

The right source coding strategy depends

• n application requirement & path

pp q p conditions:

MDC is effective when:

O t h diff ti l t t t i th

One cannot have differential treatment in the

network layer.

Delay constraint is very stringent & round trip delay

is long is long.

76

SUMMARY OF THE VIDEO OVER AD HOC PROJECT AD HOC PROJECT

Layered coding is effective when the base layer

Kas

y g y can be transported reliably, through:

A reliable physical channel, if available. Retransmission, if delay is acceptable. saei

, y p

RPS is effective when a feedback channel is

available & channel loss rates are not too high.

But not suitable for video streaming applications But not suitable for video streaming applications

where video is pre-encoded.

Reference: S Mao S Lin YWang & S Panwar ``Video Transport over Ad Reference: S.Mao, S.Lin, Y.Wang, & S.Panwar, Video Transport over Ad

Hoc Networks: Multistream Coding with Multipath Transport,'' IEEE J.

• Select. Areas Commun., Dec. 2003.

77

SUMMARY SUMMARY

Transport-level error control:

Kas

p

Guarantees a basic level of quality. Retransmission is effective within the delay

t i t

saei

constraint.

Unequal error protection has practical

difficulties against separation of source coding & transport layer.

Error resilient encoding: Error-resilient encoding: Trade off coding efficiency for error resilience. Some techniques are only useful for bit-error

q y dominated channels.

78

SUMMARY SUMMARY

Error concealment:

Kas

Does not involve extra redundancy, motion-

compensated temporal concealment is simple & yet offers visible improvements

saei

& yet offers visible improvements.

Encoder-decoder-network interactive error

control:

Requires feedback info, may not be available. Choice of techniques depends on

underlying application & network. underlying application & network.

79

HOMEWORK 11 HOMEWORK 11

Reading assignment:

Kas

Reading assignment:

• Chap. 14.

Computer assignment:

saei

Co pute ass g e t:

Course project.

80

THE END