1 / 48

Scalable Joint Source-Network Coding of Video

Scalable Joint Source-Network Coding of Video. Yufeng Shan Supervisors: Prof. Shivkumar Kalyanaraman Prof. John W. Woods April 2007. Outline. Problem statement and motivations Our proposed techniques Fine grain adaptive FEC (FGA-FEC) Generalized FGA-FEC for wireless networks

jill
Télécharger la présentation

Scalable Joint Source-Network Coding of Video

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Scalable Joint Source-Network Coding of Video Yufeng Shan Supervisors: Prof. Shivkumar Kalyanaraman Prof. John W. Woods April 2007

  2. Outline • Problem statement and motivations • Our proposed techniques • Fine grain adaptive FEC (FGA-FEC) • Generalized FGA-FEC for wireless networks • Overly multi-hop FEC scheme • Distributed FGA-FEC over multihop network • Conclusion and future work

  3. Problem statement and challenges • Problem: • Simultaneously streaming video to diverse users, such as powerful PCs, laptops and handset devices, over heterogeneous networks. • Challenges: • Different users may have different video frame rate /resolution /quality preferences. • May be in different networks, such as high speed wired network, multi-hop ad hoc wireless network or cellular network. • Neither the network nor the video application can provide quality assurances working independent of each other.

  4. Conventional Approaches • Windows media, Real player • Send separate copies of the same bitstream to different users, bandwidth utilization is inefficient (unicast to each user). • Layered multicast (by Steven McCanne) • Send different video layer to different multicast groups, limited by number of video layers, no efficient error correction approach. • Proxy-based streaming • Caches video content at local proxy disk and transcodes it for different users, delay and computation inefficiency.

  5. Our Techniques Scalable joint source-network coding: Scalable video + Scalable FEC + in network adaptation Video Performance • FGA-FEC • Generalized FGA-FEC • OM-FEC • Distributed FGA-FEC • Header CRC/FEC • Cross-layer FEC Encoding G-FGA-FEC Video FGA-FEC De/re-code server Adaptation G-FGA-FEC DSN DSN Adaptation DSN Router DSN Router G-FGA-FEC DSN Router DSN Router Video Performance Router Router FGA-FEC De/re-code

  6. Outline • Problem statement and motivations • Our proposed techniques • Fine grain adaptive FEC (FGA-FEC); • Generalized FGA-FEC for wireless networks; • Overly multi-hop FEC scheme; • Distributed FGA-FEC over multihop network • Conclusion and future work

  7. Background -- scalable video We treat three types of scalability: • SNR (aka quality, bitrate) scalability • Spatial (resolution) scalability • Temporal (frame-rate) scalability

  8. SNR (quality) scalability Embedded compressed file Decoding more data corresponds to better quality

  9. Resolution scalability Embedded compressed file Decoding more data corresponds to larger picture size

  10. Frame-rate scalability Original video One GOP - 16 frames Half frame rate – 8 frames ¼ frame rate – 4 frames 1/8 frame rate – 2 frames Begin Embedded compressed file End Decoding more data corresponds to displaying frames at higher rate

  11. Frame Rate Resolution Quality A(4,0,2) A(4,1,2) A(4,2,2) A(4,3,2) A(4,4,2) A(4,0.1) A(4,2,1) A(4,3,1) A(4,1,1) A(4,4,1) A(0,0,0) A(0,0,0) A(0,0,0) A(0,1,0) A(0,0,0) A(0,0,0) A(0,0,0) A(2,3,0) A(0,0,0) A(0,0,0) A(0,0,0) A(0,0,0) A(0,2,0) A(2,2,0) A(0,3,0) A(0,0,0) A(2,1,0) A(0,0,0) A(0,0,0) A(0,0,0) A(0,0,0) A(1,0,0) A(1,1,0) A(1,2,0) A(1,3,0) A(1,4,0) A(2,0,0) A(0,0,0) A(0,0,0) A(0,0,0) A(0,4,0) A(0,0,0) A(0,0,0) A(0,0,0) A(0,0,0) A(0,0,0) A(0,0,0) A(0,0,0) A(0,0,0) A(0,0,0) A(0,0,0) A(0,0,0) A(3,4,0) A(0,0,0) A(0,0,0) A(0,0,0) A(0,0,0) A(0,0,0) A(0,0,0) A(0,0,0) A(0,0,0) A(3,0,0) A(3,1,0) A(3,2,0) A(3,3,0) A(0,0,0) A(0,0,0) A(0,0,0) A(0,0,0) A(0,0,0) A(0,0,0) A(0,0,0) A(0,0,0) A(2,4,0) A(0,0,0) A(0,0,0) A(4,4,0) A(4,2,0) A(4,1,0) A(4,0,0) A(0,0,0) A(0,0,0) A(0,0,0) A(0,0,0) A(4,3,0) 3-D representation of bitstream Digital items in view of three forms of scalability (quality / resolution / frame-rate). Adaptation of the bitstream is achieved by choosing a subset of these items along user’s preferred adaptation order: For example: SNR -> Temporal -> Spatial

  12. Next, we need scalable FEC To protect the scalable video bitstream, we have these basic requirements for channel coding: • High adaptivity: If part of the video bitstream is actively dropped (scaled), FEC protecting that part of data should also be removed • Accuracy: near that of robust non-scalable video • Efficiency: Low computational cost

  13. Fine Grain Adaptive FEC Goal: • To encode a video to facilitate efficient and precise adaptation of the encoded bitstream at intermediate overlay nodes for diverse users. Main idea: Extend existing approaches (PET, MD-FEC) to work with scalable video and perform the adaptation in the network.

  14. Multiple Description FEC (MD-FEC) by Puri and Ramachantran MSB LSB If any i out of N packets are received, the decoder can decode up to Ri. Our work extends this concept to provide fine-grained scalability of FEC, applied to scalable video

  15. A B G F C E D Server DSN DSN FGA-FEC overview FGA-FEC uses overlay infrastructure to assist video streaming to heterogeneous users simultaneously by providing light weight support at intermediate overlay nodes. Adaptation Adaptation 30/C/1M 30/C/1M 30/C/2M 30/C/3M 15/Q/384k DSN Encoding 15/C/384k 15/Q/1M 30/C/3M Adaptation A-G : users and their requirements (frame rate/resolution/bitrate) DSN : Data service node, which aggregates video requirements to server and performs data adaptation for users, without transcoding.

  16. R R R R R R 1 i N - 1 N 0 i - 1 Enlarged view of Section S i S ... S ... S 1 i N B ... B ... B ... ... B ... 1 i 2 i mi ... Sect . 1 Sect . 2 Sect . i Sect . N ... S ( 1 ) B B B ... S ( 1 ) S ( 1 ) ... S ( 1 ) ... S ( 1 ) Description 1 1 i + 1 2 i + 1 i 1 2 i N ... S ( 2 ) B B ... ... FEC S ( 2 ) ... S ( 2 ) ... S ( 2 ) Description 2 2 i + 2 i 2 i N g ... S ( 3 ) B B ... ... FEC FEC ... S ( 3 ) ... S ( 3 ) n 3 i + 3 i Description 3 i N i ... ... ... ... ... d ... ... ... ... ... ... ... o c ... n S ( i ) FEC FEC ... S ( i ) ... S ( i ) B B B B Description i e i 2 i 3 i mi i i N ... S FEC FEC ... FEC ... S ( i + 1 ) FEC FEC FEC FEC Description i + 1 R N ... ... ... ... ... ... ... ... ... FEC FEC FEC FEC FEC FEC ... FEC ... S ( N ) Description N N FGA-FEC Encoding • Bitstream is divided into N sections from MSB to LSB • Each section is further split into small blocks • RS(N,i) codes are applied at block level to section Si, vertically • Each block column is independently accessible • one description = one network packet

  17. FGA-FEC – optimal rate allocation FEC assignment is optimized by minimizing the expected distortion over the channel with packet loss probability p, available bandwidth B. Subject to: The result of the optimization is the video source-rate breakpoints Ri where qi is the probability of receiving i out of N packets, Rtotal is the total bandwidth occupied by both FEC and video data. Also ri is the source rate of section Si. N is the number of descriptions encoded for each GOP

  18. At DSN One description Adapted description one GOP FGA-FEC adaptation at intermediate nodes Shorten & drop Two adaptation methods are used in FGA-FEC : • Direct truncation: only shorten each packet in a GOP by removing unwanted blocks. • FGA-FEC adaptation: adapt the encoded GOP by a combined shortening and/or dropping packets. • An algorithm is proposed to near optimally adapt the encoded GOP for available bandwidth and user preference • No video or FEC transcoding, only a packet shorting/dropping

  19. 6 7 Simulation-- FGA-FEC adaptation vs. optimal decoding/recoding • Comparison of FGA-FEC, optimal decoding/recoding solution, and direct truncation at different available bandwidths; • In FGA-FEC, MC-EZBC encoded GOP #7, Foreman CIF is adapted from 1100 Kbps to satisfy different users; • FGA-FEC optimization is based on 15% loss rate at source, N=64.

  20. ns-2 Simulation-- FGA-FEC vs. MD-FEC transcoding MD-FEC transcoding: FEC decode/recode at each intermediate node FGA-FEC adaptation: Encode at server, adaptation in the intermediate nodes Want to see the received video quality at User 4 and User 12, Link loss probability is 0.01, N=64, Foreman, CIF

  21. ns-2 Simulation-- FGA-FEC vs. MD-FEC transcoding Receiver 12 FGA-FEC is 0.4 dB lower Receiver 4 FGA-FEC is 0.01dB lower The quality loss is due to the adaptation precision.

  22. ns-2 Simulation-- FGA-FEC vs. Layered multicast Layered multicast: Base layer bit rate 0.6, and enhancement layers 0.32, 0.42, 0.32 Mbps, respectively FGA-FEC adaptation: Encode at server, adaptation in the intermediate nodes Want to see the received video quality at User 5 and User 7, Link loss probability is 0.01, N=64, Foreman, CIF

  23. ns-2 Simulation-- FGA-FEC vs. Layered multicast Receiver 7 Layered multicast can subscribe up to three layers Receiver 5 Layered multicast can subscribe up to two layers The quality loss of layered multicast is due to coarser available layers.

  24. Outline • Problem statement and motivations • Our proposed techniques • Fine grain adaptive FEC (FGA-FEC); • Generalized FGA-FEC for wireless networks; • Overly multi-hop FEC scheme; • Distributed FGA-FEC over multihop network • Conclusion and future work

  25. Generalized FGA-FEC for wireless In addition to congestion related packet losses, in wireless must deal with: • channel bit errors due to channel fading and noise; • large bandwidth fluctuations, and • intermediate node computational capability constraint • limited maximum transmission unit (MTU) size.

  26. R R R R R R 0 1 k N - 1 N k - 1 Embedded bitstream S ... S ... S 1 k N ... ... Sect . 1 Sect . 2 Sect . k Sect . N BCH CRC FGA - FEC + BCH / CRC Encoded Packet 1 S ( 1 ) S ( 1 ) ... S ( 1 ) ... S ( 1 ) Description 1 l e 1 2 k N BCH v e CRC FGA - FEC + BCH / CRC Encoded Packet 2 FEC S ( 2 ) ... S ( 2 ) ... S ( 2 ) Description 2 l 2 k N k BCH c FEC FEC ... S ( 3 ) ... S ( 3 ) o CRC FGA - FEC + BCH / CRC Encoded Packet 3 Description 3 l k N b … ... ... ... n i d BCH e FEC FEC ... S ( k ) ... S ( k ) Description k CRC FGA - FEC + BCH / CRC Encoded Packet k d k N o BCH c FEC FEC ... FEC ... S ( k + 1 ) n Description k + 1 CRC FGA - FEC + BCH / CRC Encode Packet k + 1 N e ... ... ... … S R BCH FEC FEC ... FEC ... S ( N ) Description N CRC FGA - FEC + BCH / CRC Encoded Packet N N Generalized FGA-FEC for wireless + Bit level protection FGA-FEC Generalized FGA-FEC = A kind ofProduct codes (extendSherwood and Zeger’s codes): • Column codes: Reed Solomon for packet loss • Row codes: CRC + BCH for bit errors A BCH code is represented as: BCH(n,k,t): n=2m-1, n-k<=mt

  27. Optimal product code assignment Find a concatenated column RS code assignment ccand row BCH code assignment cr from a set of RS codes CRS and BCH codes CBCH BCH(n,k,t), such that the end-to-end video distortion over a lossy channel is minimized. Subject to channel rate constraint: where Rs , RRS , RCRC and RBCH are rates allocated to the video source, RS parity bits, CRC and BCH check bits, respectively. Here B denotes the maximum available channel bandwidth

  28. Fast optimization method n=8191, m=13 Given a BCH code BCH(n,k,t) Probability of correctly decoding BCH code Near optimal points, search starts here, itcan find the optimal solution within a few iterations Exhaustive search results GOP #7 of Foreman, CIF, N= 64, pdrop=0.05, BER pb varies

  29. Multi-cluster descriptions coding • Single-cluster description coding: one GOP  N descriptions • Multi-cluster descriptions coding: one GOP  Multiple of N descriptions due to: • Limited network packet size; • Easier adaptation, such as encode each temporal layer as one cluster as below. One GOP of scalable bitstream Low frame rate High frame rate Mid frame rate Example of encoding one GOP into 3 clusters, each cluster is coded with product codes

  30. Simulation – FGA-FEC vs. MD-FEC in wireless Channel changes over time between node 2 and 3 Consider two adaptation orders: SNR->Temporal, SNR-> Spatial pdrop=0.05 at node2 Resolution scaling Frame rate scaling

  31. Outline • Problem statement and motivations • Our proposed techniques • Fine grain adaptive FEC (FGA-FEC); • Generalized FGA-FEC for wireless networks; • Overly multi-hop FEC scheme; • Distributed FGA-FEC over multihop network • Conclusion and future work

  32. Segment 1 Segment 3 Segment 2 N1 N5 N2 N3 N4 Motivations -- an example Given an overlay path: (B6 , P6 , RTT6 ) (B1, P1, RTT1 ) Receiver Sender OM-FEC vs. Traditional FEC The parameter of each hop

  33. i M L … … … … Ni Nj Nh Nn N1 OM-FEC algorithm OM-FEC partitions a given overlay path into segments: By solving the following: minimize(Nsegement) Subject to: The i-th node is boundary node, if the two inequalities hold: min{Bstart, …, Bi}  (Bdata + BFEC(start>i)) min{Bstart, …, Bi+1}< (Bdata +BFEC(start>(i+1)) FEC that should be added to the Segment is BFEC(start ->i),

  34. Video Simulations Topology: loss rate on each hop (L1–L10) random within [0.4%-1.2%] • H.263+ encoded Foreman CIF, at bitrate 1530 Kbps; • Available bandwidth for each hop: 1656 Kbps Average number of segments partitioned is only 2 vs. 10 in hop-by-hop FEC

  35. video.testbed.ecse.rpi.edu planetlab1.cs.cornell.edu planetlab1.flux.utah.edu planetlab-1.cmcl.cs.cmu.edu nima.eecs.berkeley.edu PlanetLab Experiments • Video: Foreman QCIF, H.263+, 512 Kbps, 30 fps, I frame every second • Packet-loss rate from Utah to CMU is set to 5%, other links are set to 1%. • The available bandwidth from Utah to CMU is also upper bounded to 550 Kbps. • Two paths: 1): with nodes shown at map 2): Add Node 4: planet1.ecse.rpi.edu at last hop with 1% loss rate

  36. OM-FEC experimental results Video PSNR of OM-FEC scheme vs. e2e scheme (4 virtual links) Video PSNR of OM-FEC scheme vs. e2e scheme (5 virtual links) • Results show: • OM-FEC outperforms e2e FEC scheme • As more congested links are encountered, gain due to OM-FEC increases • Large PSNR loss of the e2e scheme is due to the dependency of bitstream, • one packet drop could cause later packets of the same GOP useless .

  37. Outline • Problem statement and motivations • Our proposed techniques • Fine grain adaptive FEC (FGA-FEC); • Generalized FGA-FEC for wireless networks; • Overly multi-hop FEC scheme; • Distributed FGA-FEC over multihop network • Conclusion and future work

  38. Problem statement and our solution • In FGA-FEC We assume no congestion between DSNs • Remaining problem: • What if there is congestion between DSNs? For example, in a rate constrained multihop ad hoc wireless network. • Our solution – distributed FGA-FEC: - FEC coding and optimization algorithm will be run at DSNs in a distributed way to serve the diverse users.

  39. The optimization algorithm Problem: Solution: Use Lagrange multiplier method: Minimize the e2e distortion for each user: Subject to: where Given one value of λ, corresponds to the point on the D(R) curve with slope equal to Solution can be found by searching over λ

  40. The Distributed FGA-FEC Algorithm To reduce the overall computation while maintain the best possible video quality, we propose two approaches: • A coordination method between optimization processes running at adjacent nodes to reduce the optimization computation. • Full search, search with previous GOP, search with neighbor • Applied the idea of OM-FEC to reduce the number of FGA-FEC decode/recode nodes, do FEC computation at selected DSNs

  41. Simulation - reduce optimization computation Test number of λiterations need to reach optimization stop points, Foreman, CIF. The proposed coordination between adjacent nodes can reduce the optimization computation requirement.

  42. Simulation– distributed FGA-FEC vs. Hop-by-hop FEC decode/recode Hop-by-hop FGA-FEC decode/recode: FGA-FEC decode/recode at each intermediate node Distributed FGA-FEC: Identify the congested links and select appropriate nodes to do FGA-FEC decode/recode. Want to see the received video quality at User 5 and User 12 Link loss probability is 0.01, N=64, Foreman, CIF

  43. Simulation– distributed FGA-FEC vs. Hop-by-hop FGA-FEC decode/recode Receiver 12 Receiver 5 The two schemes deliver similar video quality, but distributed FGA-FEC uses fewer FEC computation nodes 3 vs. 6 at FEC decode/recode

  44. Measured CPU time We measure the CPU time using FGA-FEC adaptation and FGA-FEC decode/recode in a Dell PC with 1.6 GHZ CPU, 256 M memory, running Red Hat linux 8.2, N=64. Distributed FGA-FEC can greatly reduce the computation burden, while can deliver near optimal video quality

  45. Conclusions • Fine grain adaptive FEC (FGA-FEC) • to encode and adapt a scalable video • Generalized FGA-FEC for wireless networks • to do encoding and adaptation over wireless networks • Overly multi-hop FEC scheme • to efficiently utilize one congested path • Distributed FGA-FEC over multihop network • to stream video over a congested heterogeneous networks • Header error correction • To improve the effective throughput of wireless networks • Cross-layer FEC • To joint optimize protocol with application layer FEC

  46. Future work • Evaluate FGA-FEC over DCT-based standard coders such as H.264/AVC and SVC • Extend the distributed FGA-FEC to work with Spatial and Temporal scalability • Extend FGA-FEC idea to multi-point video conferencing

  47. Publication and submissions [1] Yufeng Shan, Ivan Bajic, John W. Woods and Shivkumar Kalyanaranman “ Scalable video streaming with fine grain adaptive forward error correction” submitted to IEEE trans. CSVT, 2006 [2] Su Yi, Yufeng Shan, Shivkumar Kalyanaraman and Babak Azimi-Sadjadi, "Video streaming over 802.11 Ad Hoc wireless ntworks with header error protection", submitted to Ad Hoc Networks, 2006 [3] Yufeng Shan, Ivan V. Bajic, Shivkumar Kalyanaraman and John W. Woods, “Overlay multi-hop FEC scheme for video streaming,” Signal Processing: Image Communications Vol. 20/8, 2005 [4] Yufeng Shan, John. W. Woods and Shivkumar Kalyanaraman “Fine grain adaptive FEC over wireless networks”, submitted to ICIP 2007 [5] Su Yi, Yufeng Shan, Shivkumar Kalyanaraman and Babak Azimi-Sadjadi, "Header error protection for multimedia data transmission in wireless AdHoc networks", ICIP, 2006 [6] Yufeng Shan, Ivan Bajic, Shivkumar Kalyanaraman, and John W. Woods, "Joint source-network error control coding for ccalable overlay streaming," ICIP, 2005 [7] Yufeng Shan, Su Yi, Shivkumar Kalyanaraman and John.W. Woods, "Two-Stage FEC scheme for scalable video transmission over wireless networks" SPIE Communications/ITCom, Multimedia Systems and Applications, Oct. 2005 [8] Yufeng Shan, Ivan Bajic, Shivkumar Kalyanaraman, and John W. Woods, "Overlay multi-hop FEC scheme for video streaming over peer-to-peer networks," ICIP, 2004 [9] Yufeng Shan and Shivkumar Kalyanaraman “Hybrid video downloading/streaming over peer-to-peer networks,” ICME,2003 

  48. Thanks

More Related