1. ECEC-453Image Processing Architecture 3/11/2004 Exam Review Oleh Tretiak Drexel University

2. Please complete course survey (on the web)

3. Announcements • Final on March 20 • Cumulative • Extra credit problem - write plugin for ImageJ (everybody does a different plugin)

4. Architecture for the DCT • Separable DCT • Options • Fast DCT ~ conventional computer • Vector DCT ~ parallel hardware • 8x8 1-D DCT • Unit operation: Multiply 8x8 matrix with 8x1 matrix ~ 64 ops

5. Computational Complexity • 1D DCT • N input and output samples ~ N2=64 operations (additions + multiplications) • 2D DCT - direct implementation • M = N2 input values, M output values -> M2 = N4 • 2D DCT - separable implementation, Y = TXTT = ZTT,where Z = TX, all matrices are NxN -> 2N3operations • For N = 8 • 2D DCT direct — 4096 operations, 64 operations per pixel • 2D DCT separable — 1024 operations, 16 ops/pixel • Big savings due to separable transform • Inverse DFT — same story.

6. DCT: Encoding in JPEG, MPEG • Take 8x8 blocks of pixels • Subtract range mean value • Compute 8x8 DCT • Quantize the DCT coefficients • Typically, many of the samples are equal to zero • Lossless entropy coding of the quantized samples • Different quantization step is used for different DCT coefficients • ykl — DCT coefficients, qkl — quantizer steps • zkl— quantized values

7. Optimized (fast) DCT • 1-D Chen DCT diagram. Dashed lines indicate subtraction, — multi-plication by a constant, — multiplication by 0.5 (shift). Characteristics of optimized DCT algorithms

8. JPEG Classic

9. Huffman Coding - Block Diagram

10. Coding AC Coefficients • AC coefficients are coded in zig-zag (called ZZ in standard) order to maximize possible runs of zeros. • Code unit consist of run lengthfollowed by coefficient size. • Baseline coding of size category is the same as for DC differences (Table 2.9) • Example: run of 6 zeros, size = -18. In the table, -18is in category 5. Code is(6/5, 01101). If the Huffmancode for 6/5 is 1101, codeword = 110101101

11. Example of JPEG compression Very high quality: compression = 2.33 Photoshop Image Very low quality: compression = 115 Produced by MATLAB with Quality = 0

12. Compression = 64 JPEG JPEG2000

13. Decoder Encoder Predictive Coding of Video • E(x, y, t) = I(x, y, t) - P(x, y, t) • I ~ image, P ~ prediction, E ~ error • P(x, y, t+1) = P(x, y, t) + Code(E(x, y, t)) • At receiver, Ie(x, y, t) = P(x, y, t+1) • Ie(x, y, t) ~ estimate of image at time t

14. Generic Encoder - simplified Review

15. Motion Estimation Methods No compensation Full search logarithmic search 3 level hierarchical Review

16. Full-Search Method • Compute for (2p+1)2 values of (i, j). • Each location requires 3MN operations • Picture dimensions IxJ, F pictures per second • 3IJF(2p + 1)2 operations per second • I = 720, J = 480, F = 30, p = 15 —> 30 GOPS • Guaranteed to find best (MAE) displacement • How to do it? • Special computers • Smaller p • Faster (suboptimal) algorithm

17. Hierarchical Search • Prepare downsampled versions of current and reference images • Full macroblock 16x16 • Down 2 macroblock 8x8 • Down 4 macroblock 4x4 • Full search in Down 4 reference image • 16 x speedup, smaller macroblock • 16 x speedup, fewer displacement vectors • p = ±16, p’ = ±4 • Around point of best match, do local search in Down 2 reference image (3x3 search zone) • Repeat for Full reference image (3x3 search zone) Full Down 2 Down 4

18. Comparison

19. MPEG-1: ‘1.5’ Mbps • Sample rate reduction in spatial and temporal domains • Spatial • Block-based DCT • Huffman coding (no arithmetic coding) of motion vectors and quantized DCT coefficients • 352 x 340 pixels, 12 bits per pixel, picture rate 30 pictures per second —> 30.4 Mbps • Coded bit stream 1.15 Mbps (must leave bandwidth for audio) • Compression 26:1 • Quality better than VHS! • Temporal • Block-based motion compensation • Interframe coding (two kinds)

20. Picture Types • MPEG-1 is designed to support random access & editing • I — intraframe coding only • P — predictive coding • B — bi-directional coding

21. Picture of Layers

22. Coding Image Blocks • B pictures • Inter or intra? • Forward, backward, interpolational? • Code block or skip? • Quantization step?

23. MPEG-1 Wrap-up • Data below for decoder, SIF pictures, 2 B pictures per P • IDCT must be precise, because of inter-frame coding • MPEG-1 does not deliver quality acceptable for broadcast —> MPEG-2

24. Typical MPEG coding parameters • Typical sequence • IPBBPBBPBBPBBPBB (16 frames)

25. MPEG-2 Goals • Compatibility with MPEG-1 • Good picture quality • Flexibility in input format • Random access capability (I pictures) • Capability for fast forward, fast reverse play, stop frame • Bit stream scalability • Low delay for 2-way communications (videoconferencing) • Resilience to bit errors

26. MPEG-2 profiles • A profile is a subset of the entire MPEG-2 bit-stream syntax • Simple • Main • 4:2:2 • SNR • Spatial • High • Multiview • Each profile has several levels (resolution quality) • Low — MPEG1 • Main — CCIR 601 • High-1440 (Video Editing) • High (HDTV)

27. MPEG2 - Alternate Scan Zig-zag scan Alternate scan

28. MPEG2 — Subsampling • Suppose picture is 720x480 • 4:4:4 • Luminance and chrominance @ 720x480 • 4:2:2 • Luminance @ 720x480, chrominance 360x480 • 4:2:0 • Luminance 420x480, chrominance 360x240 • Weird terminology

29. Teleconferencing Standards • Digital video areas • Broadcast television • Recorded programs • Two-way communications

30. Review: Video Telephone System H.261 H.221 H.200/AV.250 -Series H.320

31. Review: H.261 Features • Common Interchange Format • Interoperability between 25 fps and 30 fps countries • 252 pix/line, 288 line, 30 fps noninterlace • Terminal equipment converts frame and line numbers • Y Cb Cr components, color sub-sampled by a factor of 2 in both directions • Coding • DCT, 8x8, 4 Y and 2 chrominance per masterblock • I and P frames only, P blocks can be skipped • Motion compensation optional, only integer compensation • (Optional) forward error correction coding

32. Picture Formats for H.263

33. 10% 63% 10% Encoder: Where’s the meat?

34. Advanced Video Coding • H.263 and MPEG-4 based on ~1995 technology • After 1995, MPEG and VCEG (video coding) started working on a new low-rate standard (H.26L) • Rec H.264 released in September 2002 • Information on http://www.vcodex.com/ (some is on our web site) • Site maintained by Ian Richardson, who has written books about video coding

35. New Features • Prediction in I pictures • Different block transform • Different Block Sizes • Changes in motion compensation • VLC and arithmetic coding

36. I Picture Prediction • System operates with 4x4 blocks and 16x16 macroblocks

37. 9 Prediction Modes for 4x4 Blocks

38. 4 Modes for 16x16 Macroblocks • Mode 0: Vertical, extrapolate from upper samples • Mode 1: Horizontal, extrapolate from left samples • Mode 2: DC, mean of upper and left-hand samples • Mode 3: Plane, linear fit to left and upper samples

39. Different Block Transform • Basically, 4x4 DCT • Scanning sequence for 16x16 macroblock is shown below • 4x4 and 2x2 DC coefficients transformed (again)

40. 4x4 DCT Tricks • Y = AXAT • a = 1/2, b = 0.707 cos(π/8), c = cos(3π/8) • Trick: Y = (CXCT).*E

41. Motion Compensation Ideas • Adaptive motion compensation blocks: • 16x16, 16x8, 8x16, 8x8, 8x4, 4x8, 4x4

42. Coding Ideas • Constant quantizer value • Zig-zag scan with novel run-length code • Arithmetic coding an option • Motion vectors to 1/4 pixel

43. Loop Filter • Concept to overcome block artifacts • Average across inter-block lines if differenceis too big • Difference threshold depends on codingmode (intra or inter) and quantization stepsize

44. Example of Loop Filter