Understanding Video Compression: Techniques and Human Vision Limitations

1. Digital Media Dr. Jim Rowan ITEC 2110 Video Part 2

2. Coping with Video Size Consider human vision limitations 1) Chrominance sub-sampling Compression - two versions 2) Spatial 3) Temporal differencing vectoring

3. Coping with Video Size Consider human vision limitations 1) Chrominance sub-sampling Compression - two versions 3) spatial 4) Temporal differencing vectoring

4. Chrominance sub-sampling • Humans can’t distinguish changes in color as well as they can distinguish luminance changes • http://en.wikipedia.org/wiki/Chroma_subsampling • In our cameras… • Of every 4 frames • store the luminance for each frame • only store a proportion of the color info • 4:2:0

5. Chrominance sub-sampling http://dougkerr.net/pumpkin/articles/Subsampling.pdf

6. Chrominance sub-sampling Luminance, Cr, Cb http://en.wikipedia.org/wiki/Chroma_subsampling#Sampling_systems_and_ratios

7. Coping with Video Size • Aside from screen size and frame rate... • Consider human vision limitations 1) Chrominance sub-sampling • Compression - two versions 2) spatial 3) Temporal differencing vectoring

8. Coping with Video Size • Spatial compression • Individual images can be compressed using the techniques discussed in the bitmapped section • Doesn’t result in very much compression for video • Doesn’t take into consideration the other frames that come before or after it • Our video cameras do this, compressing each frame to jpeg

9. Coping with Video Size • Aside from screen size and frame rate... • Consider human vision limitations 1) Chrominance sub-sampling • Compression - two versions 2) spatial 3) Temporal differencing vectoring

10. Temporal Compression differencing • Use the Difference in two frames • A naive approach can result in good compression • Works well for a small amount of movement • Security cameras spend most of their time “seeing” the same thing all night long • A Tarantino film? not so much… • Most pixels change with nearly every frame Saved Frame 2 Saved Frame 1 Captured Frame 1 Captured Frame 2

11. Image Differencing • To subtract one image from the next • Do it one pixel at a time • red minus red • green minus green • blue minus blue • Store the difference • To play it back • Play frame one 1 • Add frame 2 to frame 1 • Next, an example in black & white

12. Example 1, the difference of two identical images

13. Example 1, the difference of two identical images The result ===>

14. Example 2, the difference of two similar images

15. Example 2, the difference of two similar images The result ===>

16. Temporal Compressionvectoring • When an OBJECT moves • compute its trajectory • fill in the resulting exposed background Captured Frame 1 Captured Frame 2 Stored Background Stored Object Movement Vector • BUT there’s a problem... • why isn’t this an easy thing to do?

17. More on differencing • The differencing can happen in a forward manner and a backward manner • It might be more economical (in data size) to create a frame from a frame that follows it...

18. MPEG-2 iFrame pFrame bFrame GOP http://en.wikipedia.org/wiki/Inter_frame http://en.wikipedia.org/wiki/I-frames#Intra_coded_frames_.28or_slices_or_I-frames_or_Key_frames.29

19. iFrame: -a keyframe -spatially compressed (a fully specified image) pFrame -predicted frame -contains only the difference between the current frame and the previous iFrame (smaller in size than iFrame) bFrame -bi-predicted frame -contains difference between current and both the preceding and following iFrames -even smaller in size than iFrame

20. 3 mpeg 2 video streams http://www.spiritalchemy.com/blog/wp-content/uploads/2009/10/IPB_frame_example.jpg Group(s) Of Pictures (GOPs) The largest All spatially (intra-frame) compressed Spatially compressed and predictive (difference) Smaller… but more computation Spatially compressed forward and backward predictive (difference) Smallest… but more computation and it is transported out of order! Play sequence: 1 2 3 4 5 6 7 8 9 10 Transmit sequence: 1 4 2 3 7 5 6 10 8 9

21. Mpeg compression technique 1 Mpeg compression technique 2 Mpeg compression technique 3 Frame capture sequence: 1 2 3 4 5 6 7 8 9 10

22. So… How does this ===> happen?

24. Mpeg encoding http://www.100fps.com/codec_quality_comparison.htm • It’s COMPLICATED… • Images are broken up and transmitted as macroblocks • They can be a variety of sizes • Typically 8X8 or 16X16 • When missing (or there are video errors) • May use a lower resolution block (that was previously received) • May just use a solid color block instead

25. Video CompressionWhat does this? • http://en.wikipedia.org/wiki/Video_codec • Coder/Decoder - Codec • encodes and decodes video • Can be symmetric • it takes as long to compress as decompress • Can be asymmetric • it takes longer to compress or decompress than it does to decompress to compress

26. Video Compression Can be very complex Each image is just jpeg compressed Each image is jpeg compressed; the p-frames are the result of frames being subtracted from the i-frames Each image is jpeg compressed; the p-frames are the result of frames being subtracted from the i-frames; b-frames are the result of frames being subtracted from i-frames and future p-frames

27. A final worry... • Codecs compress video to make it smaller • There are a variety of codecs to do this • Which to choose? • It is a tradeoff between compression technique, its computational complexity and its artifacts

28. So... How do codecs vary? • compression and decompression complexity • affects the artifacts that are created • affects the time required to carry them out • affects the volume of the data stream created • affects the type and expense of the equipment used • affects whether or not it can be implemented in hardware of software • WMV, DivX, Cinepak, Intel Indeo & Sorenson

29. Which is the original? Notice the artifacts? http://www.100fps.com/codec_quality_comparison.htm

30. Center one is the original Left is “sharpened” Right is “blurred” http://www.100fps.com/codec_quality_comparison.htm

31. That’s It!