1. Ai-Mei Huang and Truong Nguyen Image Processing (ICIP), 2009 16th IEEE International Conference on MOTION VECTOR PROCESSING USING THE COLOR INFORMATION

2. CONTENTS • INTRODUCTION • COLOR INFORMATION • MV PROCESSING FOR MCFI USING THE COLOR INFORMATION • SIMULATIONS • CONCLUSIONS

3. Introduction(1/2) MB (Macro Block) • Color information has been shown to be effective in the object edges detection • Due to its insensitivity on specular reflection • Prevent false edge detection as compared to luminance-based methods 8 16 8 8 8 16 Luminance(intensity) Chrominance(color information)

4. Introduction(2/2) • Color has sharper and more consistent variations between object boundaries • Applications often take the color information to assist the image segmentation process. • In our previous work [8] • The color information was found very useful for the unreliable MV detection • Especially in the areas where the luminance component tends to distribute uniformly. • In this paper, we would like to • Examine the color information • Analyze how the chrominance components can be used to assist the MV processing in MCFI. [8] A.-M. Huang and T. Nguyen, “A novel multi-stage motion vector processing method for motion compensated frame interpolation,” in Proc. ICIP’07, pp. 389–392, 2007.

5. Color Information(1/5) • The luminance components have stronger intensity distribution than the chrominance components • Conventional motion estimation often ignore the color information due to the complexity. Cb Y Cr

6. Color Information(2/5) • Color characteristics are distinct from luminance • Such as the insensitivity in highlight or shadow areas • Used in preventing the false edge detection • The chrominance improves the edge identification for the static text, face features, the cap, and the shirts

7. Color Information(3/5) • If the moving objects have sharp edges, the ambiguous motions seem more unlikely to appear. • From Fig. 1(b), we can observe that the luminance has very smooth variations around the face and shirts areas. • Since the motion is mainly determined using the luminance difference, the motion can be easily wrong in these areas.

8. Color Information(4/5) • Many artifacts around the nose and the shirts in (a) • MVs around the shirtscan only be detected in (c) • The luminance have uniformly distribution so that the encoder always chooses the face motion. • The color has strong gradients.

9. Color Information(5/5) • Generally, chrominance residual distribution is similar to the luminance components. • The pavement and lawn have very similar intensity. • The color difference will become relatively large.

10. Motion Vector Analysis • The residual energy with color consideration be represented as follows: • rY(i, j), rCb(i, j), and rCr(i, j) are the reconstructed residual signals of Y, Cb and Cr components of the 8×8 block, bm,n • α is the weight used to emphasize the degree of color difference.Empirically set α=8 for 4:2:0 YUV • The residues are embedded in the reconstructed signals during the decoding process.

11. MV classification process • Compare Em,nto a predefined threshold, ε1, based on the combined residual information. • The adjacent MBs will be merged as a group using the residual energy distribution. MB 16 16

12. Motion Vector Correction using the Color Information • Minimizing the absolute bidirectional prediction difference (ABPD) between forward and backward predictions.

13. SIMULATIONS • Two video sequences, FOREMAN and FORMULA 1 • CIF frame resolution • all encoded using H.263 • with even frames skipped and the skipped frames are interpolated

14. Visual Comparisons(1/2) • Fig. 4(c), the artifacts around the nose and the eye are reduced. • These artifacts are removed in Fig. 4(d). • Since the chrominance information sharpens the residual energy. • Unreliable MVs around the shirts and face areas can be identified and be corrected accordingly.

15. Visual Comparisons(2/2) • The intensity between grass and pavement is very similar. • So the motion estimation easily fails on the white lines areas.

16. CONCLUSIONS • We present using color information for the MV reliability classification. • Unreliable MVs with small luminance difference can be effectively detected.