Video Mosaics

1. Video Mosaics CS 294/283 Final Project Steve Martin U.C. Berkeley, Fall 2003

2. Overview • Previous Work • Method Overview • Metrics for Comparing Video • Preliminary Results • Future Work • Questions?

3. Previous Work on Mosaics • Image Mosaics • 1970’s: Chuck Close/Salvadore Dali make image mosaics by hand • 1990’s: Papers at SIGGRAPH and commercial programs create image mosaics • Video Mosaics • 2002: Video Mosaic program created at Princeton, work published at NPAR 2002 • Example. . . • 2003: Wells Fargo commercial (done by hand) • Example. . .

4. Method Overview • Step 1: Get a LOT of source video • Quality of results highly dependant on quality and amount of component clips! • In our case, problem solved! • Got stock video companies donate video to the war effort • We have 200GB of high quality video to play with (more than 3000 clips)! • Step 2: Preprocess video • Create a database of feature values for all video. • Each clip further subdivided into tiles • Size determined by the user. • With 20 by 20 pixel tiles, each movie has 864 tiles. • Metrics are run on each tile, values recorded • This is done with a separate program that crawls the video repository

5. Method Overview (2) • Step 3: Create video mosaic • Load in original video • Parse the previously generated database • For each clip in the original video • Run the same metrics as was used to generate the database • Search the database for the closest tile • Replace the original tile with one from another movie, continue • Note that we can’t just brute force compare video or we’ll be computing forever. • How to come up with numbers that describe tiles?

6. 0.5 * + 0.5 * = Video Comparison Metric 1: Average Color • Want replacement clips to have close to or same average color as original tile. • To calculate: • Just add color values of pixels in movie tile and divide. • Actually not terrible by itself. . . • As tiles get smaller, approach pixel size, eye blends them • However, note that average color can be ‘fooled’ • i.e. a clip that is half red and half green averages to olive brown: • Need something else. . .

7. Vs. Vs. Video Comparison Metric 2: Color Histograms • In addition to average color, want some kind of measure of color frequency. • Helps solve previous problem, although not entirely • To calculate: • For each pixel of each frame of each tile in movie: • Convert from RGB to HSB • Take hue value and histogram • Create histograms on a per-tile basis • Number of histogram buckets is changeable • Usually use around 10.

8. θ Video Comparison Metric 3: Edge Histograms • Also would like to have some kind of edge comparison between original and replacement tile. • To calculate: • Foreach frame of each tile • Convert frame to black and white • Don’t care about color—want edge data • Resize the image down to 10 x 10 • We want major edges only • Use a bicubic resize to blur the image slightly • For each pixel in the new image, • Calc intensity gradient in the x and y directions • Take the arctangent and histogram. • Do for each frame, create histogram. • Example of this metric at work:

9. Finding the Best Match • Currently, a linear search through a large unordered flat file. • Parse this into a simple row/column data structure, keep it in memory. • Search not as bad as it sounds; takes about a second with 259200 entries. • Coming soon: some tricks to make this much better. • For each tile of the source movie, compare with each tile entry in the database. • Get the absolute difference between average colors • Get the distance between color histograms • Get the distance between edge histograms • Add difference values to get a value for how good a match this tile is for the original. • Can weight different metric to emphasize different features • Choose entry that minimizes this.

10. Video vs. Photo Mosaics • Why bother using video? • Could mosaic each frame individually, and then string the frames together. • Example. . . • Would like to exhibit some kind of temporal consistency • I.e. each tile is recognizable as a movie instead of a compression artifact. • Neat effect; less distracting for the eye.

11. Preliminary Results

12. Preliminary Results

13. What’s Next? • Need to implement some way of dividing tile clips into arbitrary length. • Approximation: subdivide tiles into groups of frames. • Index the rest of the video so I can leverage the full library. • Still working on good ways to preserve temporal locality • i.e. place video tiles in a manner that makes sense over time in addition to just per frame • Include metrics that look at neighboring tiles to avoid drastically different edges • These just end up looking like compression artifacts • Princeton’s video has this problem • Different sized tiles • I think a ‘patchwork’ effect would look interesting, albeit harder to compute. • Optimize, optimize, optimize! • Takes forever to make one of these!

14. References • Klien, Grant, Finkelstein, and Cohen. “Video Mosaics” Proceedings of NPAR 2002. • Microsoft Video For Windows developer’s site • Apple Quicktime SDK developer resources • James O’Brien, Alexander Berg, Charless Fowlkes Questions? Comments?