Manipulation and Compositing of MC-DCT Compressed Video

1. Manipulation and Compositing of MC-DCT Compressed Video S-F. Chang D.G. Messerschmitt Presented by Alex Li

2. Outline • Motivations • MC-DCT based compression system • Manipulations in DCT domain • Manipulation of MC-compressed video • Manipulation of MC-DCT compressed video • Results • Conclusion

3. Motivations • Video data is compressed for storage and transmission • Spatial domain video manipulations can be computationally expensive • involves full decoding and encoding that brings higher data rate and more computations • Manipulations in compressed domain is proposed • NO full decoding and encoding, thus lower data rate and fewer computations • low-end computing/communication devices can ignore less significant components without much degradation (reduced computation and bandwidth)

4. MC-DCT based compression system

5. Manipulations in DCT domain • Overlap • Spatial: Pnew(i,j) = α Pa(i,j)+(1-α) Pb(i,j) • DCT: DCT(Pnew) = α DCT(Pa)+(1-α) DCT(Pb) • Pixel Multiplication • Spatial: Pnew(i,j) = α(i,j) Pa(i,j)+(1-α(i,j)) Pb(i,j) • need to figure out DCT of Pnew(i,j) = Pa(i,j) Pb(i,j) • 2N-point circular convolution of DCT(Pa) DCT(Pb)

6. Manipulations in DCT domain • Translation • blockwise translation only replaces the DCT blocks • pixelwise translation is not trivial • Spatial: B’ = B13 + B24 + B31 + B42 • DCT: DCT(B’) = DCT(B13)+DCT(B24)+DCT(B31 )+DCT(B42) • How do we extract subblocks? • B24 = H1B4H2 where H1 and H2 are extraction matrices

7. Manipulations in DCT domain • Translation (cont’d) • DCT: DCT(B24) = DCT(H1)DCT(B4)DCT(H2) • DCT(B’) = Σi [ DCT(Hi1)DCT(Bi)DCT(Hi2) ] • Linear Filtering, Scaling, and Shearing • Different matrix multiplications, similar operations • Spatial: Y = Σi [Gi Xi Hi] • DCT: DCT(Y) = Σi [DCT(Gi) DCT(Xi) DCT(Hi)] • Rotation is likely too cumbersome

8. Manipulation of MC-compressed video foreground Indirectly affected area Directly affected area background Problem: * Blocks in DAA could reference blocks in FG, need to find new reference blocks. This leads to recalculations. Solution: * Reduce number of recalculations by restricting them to blocks in DAA. Turns out 10% to 15% of the blocks in DAA need recalculations.

9. Manipulation of MC-compressed video foreground D Indirectly affected area D2 D1 B Directly affected area background Problem: *Require considerable amount of computations to find best-matched block again for B. Solution: *Jain and Jain’s algorithm -- D1 and D2 are likely the less distorted blocks *Just pick between D1 and D2, saves a LOT of computations

10. Manipulation of MC-DCT compressed video Original: Prec(t,x,y) = DCT-1(DCT(e(t,x,y))) + Prec(t-1,x-dx,y-dy) Now: DCT(Prec(t,x,y)) = DCT(e(t,x,y)) + DCT(Prec(t-1,x-dx,y-dy))

11. Results • 10% to 30% faster than spatial domain • However, only blockwise translation

12. Results (cont’d) • 3 to 6 times faster than spatial domain

13. Conclusion • 2nd quantization degrades image quality. • DCT-domain manipulation produces variable throughput. • Can skip high-order coefficients. • More zero motion vectors during encoding can reduce computations further (10% to 20% speedup). • Combine multiple processes into one through precomputed matrices. • Future work involves compromising video compression and manipulation flexibilities.

14. Questions ?