Video Compression

1. Video Compression Fall 2011 Hongli Luo

2. Video Compression • Image compression • To reduce spatial redundancy • Video compression • spatial redundancy exists in each frame as in images • Temporal redundancy exists between frames and can be used for compression • Video compression reduces spatial redundancy within a frame and temporal redundancy between frames • Each video frame can be encoded differently depending on whether to exploit spatial redundancy or temporal redundancy • Intraframe • Interframe

3. Intraframe and Interframe • Intraframe • Each frame is encoded as an individual image • Use image compression technique, e.g., DCT • Interframe • Predictive Encoding between frames in the temporal domain • Instead of coding the current frame directly, the difference between the current frame and a prediction based on previous frames • Use motion compensation

4. Intraframe Coding • The frames are compressed using • Lossy compression, e.g., DCT or subsampling and quantization • Lossless entropy compression, e.g. huffman or arithmetic • MPEG/ITU standard compress the intraframe similar to JPEG image standard • Get 8 x 8 blocks • DCT transformation on each block • Quantization of the coefficients • AC zigzag • DPCM on DC coefficients • Runlength coding on AC coefficients • Huffman or arithmetic coding

5. Interframe Coding • How does a pixel value change from one frame to the next frame? • No change, e.g., background • Slight changes due to quantization • Changes due to motion of the object • Changes due to motion of the camera • Changes due to environment and lighting • No changes – no need to code • Changes due to motion of object or camera • Predict how the pixel has moved • Encoding the changing vector

6. Video Compression with MotionCompensation • Consecutive frames in a video are similar - temporal redundancy exists. • Temporal redundancy is exploited so that not every frame of the video needs to be coded independently as a new image. • The difference between the current frame and other frame(s) in the sequence will be coded - small values and low entropy, good for compression. • Steps of Video compression based on Motion Compensation (MC): • Motion Estimation (motion vector search). • Motion Compensation based Prediction. • Derivation of the prediction error, i.e., the difference.

7. Video Compression Based on Motion Compensation • Each image is divided into macroblocks of size N x N. By default • For luminance images, N = 16 • For chrominance images, N = 8 if 4:2:0 chroma subsampling is adopted • Motion compensation is at the macroblock level • The current image frame is referred to as Target Frame. • A match is sought between the macroblock in the Target Frame and the most similar macroblock in previous and/or future frame(s) (referred to as Reference frame(s)). • The displacement of the reference macroblock to the target macroblock is called a motion vector MV.

8. Assume color of (x, y) is the same or very similar to (x0,y0) • Displacement or motion vector d = (dx, dy) • (x, y) = (x0+dx, y0+dy) • d = (dx, dy) = (x-x0, y-y0) = (x,y) – (x0,y0) dx = x-x0 dy=y-y0

9. Motion Estimation and Compensation • Motion Estimation • For a certain macroblock of pixels in the current frame (referred to as target frame) , find the most similar macroblock in a reference frame (previous or future frame), within specified search area. • Search for the Motion Vector - MV search is usually limited to a small immediate neighborhood – both horizontal and vertical displacements in the range [−p, p] • Motion Compensation • The target macroblock is predicted from the reference macroblock • Use the motion vectors to compensate the picture

10. Simple Motion Example • Consider a simple block of a moving circle. Instead of coding the current frame, code the difference between 2 frames. The difference needs fewer bits to encode. From Multimedia CM0340 David Marshall

11. Estimate Motion of Blocks • Estimate the motion of the object, encode the motion vectors and difference picture. From Multimedia CM0340 David Marshall

12. Decode Motion of Blocks • Use the motion vector and difference picture for decoding. From Multimedia CM0340 David Marshall

13. Motion Estimation and Compensation • Advantage • Motion estimation and compensation reduce the video bitrates significantly • After the first frame, only the motion vectors and difference macroblocks need be coded. • Introduce extra computational complexity • The motion estimation is the most computation expensive part of a video encoder • Need to buffer reference pictures – previous frames or future frames

14. Video Compression Standard • Image, Video and Audio compression standards have been specified by two major groups since 1985 • ISO (International Standards Organization) • JPEG • MPEG • MPEG-1, MPEG-2, MPEG-4, MPEG-7, MPEG-21 • ITU (International Telecommunications Union) • H.261 • H.263 • H.264 – by Joint Video Team (JVT) of ISO/IEC MPEG and ITU-T VCEG.

15. H.261 • H.261: An earlier digital video compression standard, its principle of MC-based compression is retained in all later video compression standards. • The standard was designed for videophone, video conferencing and other audiovisual services over ISDN. • The video codec supports bit-rates of p x 64 kbps, where p ranges from 1 to 30 (Hence also known as p x 64). • Require that the delay of the video encoder be less than 150 msec so that the video can be used for real-time bidirectional video conferencing.

16. ITU Recommendations & H.261 Video Formats • H.261 belongs to the following set of ITU recommendations for visual telephony systems: • H.221 - Frame structure for an audiovisual channel supporting 64 to 1,920 kbps. • H.230 - Frame control signals for audiovisual systems. • H.242 - Audiovisual communication protocols. • H.261 - Video encoder/decoder for audiovisual services at p x 64 kbps. • H.320 - Narrow-band audiovisual terminal equipment for p x 64 kbps transmission.

19. H.261 Frame Sequence • Two types of image frames are defined: Intra-frames (I-frames) and Inter-frames (P-frames): • I-frames are treated as independent images. Transform coding method similar to JPEG is applied within each I-frame, hence “Intra”. • P-frames are not independent: coded by a forward predictive coding method (prediction from a previous P-frame is allowed - not just from a previous I-frame). • Temporal redundancy removal is included in P-frame coding, whereas I-frame coding performs only spatial redundancy removal. • To avoid propagation of coding errors, an I-frame is usually sent a couple of times in each second of the video.

20. Intra-frame (I-frame) Coding • Macroblocks are of size 16 x16 pixels for the Y frame, and 8 x8 for Cb and Cr frames, since 4:2:0 chroma subsampling is employed. • A macroblock consists of four Y, one Cb, and one Cr 8 x 8 blocks. For each 8 x 8 block a DCT transform is applied, the DCT coefficients then go through quantization zigzag scan and entropy coding.

21. Inter-frame (P-frame) Predictive Coding • Figure 10.6 shows the H.261 P-frame coding scheme based on motion compensation: • For each macroblock in the Target frame, a motion vector is allocated by one of the search methods discussed earlier. • After the prediction, a difference macroblock is derived to measure the prediction error. • Each of these 8 x 8 blocks go through DCT, quantization, zigzag scan and entropy coding procedures.

22. Inter-frame (P-frame) Predictive Coding • The P-frame coding encodes the difference macroblock (not the Target macroblock itself). • Sometimes, a good match cannot be found, i.e., the prediction error exceeds a certain acceptable level. • The MB itself is then encoded (treated as an Intra MB) and in this case it is termed a non-motion compensated MB. • In fact, even the motion vector is not directly coded. • The difference, MVD, between the motion vectors of the preceding macroblock and current macroblock is sent for entropy coding: MVD = MVPreceding− MVCurrent(10:3)

24. H.263 • H.263 is an improved video coding standard for video conferencing and other audiovisual services transmitted on Public Switched Telephone Networks (PSTN). • Aims at low bit-rate communications at bit-rates of less than 64 kbps. • Uses predictive coding for inter-frames to reduce temporal redundancy and transform coding for the remaining signal to reduce spatial redundancy (for both Intra-frames and inter-frame prediction).

25. MPEG-1 • MPEG: Moving Pictures Experts Group, established in 1988 for the development of digital video. • MPEG-1 adopts the CCIR601 digital TV format also known as SIF (Source Input Format). • MPEG-1 supports only non-interlaced video. Normally, its picture resolution is: • 352 x240 for NTSC video at 30 fps • 352 x288 for PAL video at 25 fps • It uses 4:2:0 chroma subsampling

26. Motion Compensation in MPEG-1 • Motion Compensation (MC) based video encoding in H.261 works as follows: • In Motion Estimation (ME), each macroblock (MB) of the Target P-frame is assigned a best matching MB from the previously coded I or P frame - prediction. • prediction error: The difference between the MB and its matching MB, sent to DCT and its subsequent encoding steps. • The prediction is from a previous frame - forward prediction.

27. The MB containing part of a ball in the Target frame cannot find a good matching MB in the previous frame because half of the ball was occluded by another object. • A match however can readily be obtained from the next frame.

28. Motion Compensation in MPEG-1 (Cont'd) • MPEG introduces a third frame type - B-frames, and its accompanying bi-directional motion compensation. • The MC-based B-frame coding idea is illustrated in Fig. 11.2: • Each MB from a B-frame will have up to two motion vectors (MVs) (one from the forward and one from the backward prediction).

30. Group of Picture (GOP): starts with a I-frame, followed by B and P frames This GOP has 10 frames, with the structure: IBBPBBPBB

31. MPEG-1 Frames • Coding mechanism similar to H.261 • Three types of frames: • I-frames, coded in intra-frame mode • P-frames, coded with motion compensation using a previous I or P frame as reference) • B-frames, coded with bidirectional motion compensation based on a previous or a future I or P frames

32. B-frames • Advantages: Coding efficiency. • Most B frames use less bits. • Better Error propagation: B frames are not used to predict future frames, errors generated will not be propagated further within the sequence. • Disadvantage: • Frame reconstruction memory buffers within the encoder and decoder must be doubled in size to accommodate the 2 anchor frames.

33. Other MPEG • MPEG-2: For higher quality video at a bit-rate of more than 4 Mbps. • Originally designed as a standard for digital broadcast TV • Also adopted for DVDs • MPEG-3: Originally for HDTV (1920 x 1080), got folded into MPEG-2 • MPEG-4: very low bit-rate communication • The bit-rate for MPEG-4 video now covers a large range between 5 kbps to 10 Mbps. • MPEG-7: Main objective is to serve the need of audiovisual content-based retrieval (or audiovisual object retrieval) in applications such as digital libraries. • MPEG-21: New standard • The vision for MPEG-21 is to define a multimedia framework to enable transparent and augmented use of multimedia resources across a wide range of networks and devices used by different communities.

34. MPEG-4 Part10/H.264 • The H.264 video compression standard, formerly known as “H.26L”, is being developed by the Joint Video Team (JVT) of ISO/IEC MPEG and ITU-T VCEG. • Preliminary studies using software based on this new standard suggests that H.264 offers up to 30-50% better compression than MPEG-2, and up to 30% over H.263+ and MPEG-4 advanced simple profile. • The outcome of this work is actually two identical standards: ISO MPEG-4 Part10 and ITU-T H.264.

35. H.264 • H.264 is currently one of the leading candidates to carry High Definition TV (HDTV) video content on many potential applications. • H.264 is adopted by Apple QuickTime 7 • delivers high quality at remarkably low data rates. • Generate bit stream across a broad range of bandwidths, • 3G mobile devices, iPod • Video on demand, video streaming (MPEG-4 Part 2) • video conferencing (H.263) • HD for broadcast (MPEG-2) • DVD (MPEG-2)