1 / 53

Advancements in Spatiotemporal Regularity Flow (SPREF) for Image and Video Processing

This document discusses the Spatiotemporal Regularity Flow (SPREF) model developed at the Computer Vision Lab, University of Central Florida. The SPREF model focuses on analyzing spatiotemporal consistency in image and video data through innovative features like color histograms, wavelet coefficients, and SIFT. It serves applications such as video inpainting, object removal, and video compression. By modeling both spatial and temporal regularities, SPREF enhances understanding of pixel behaviors across multiple frames, providing insights into the construction of seamless visuals.

ikia
Télécharger la présentation

Advancements in Spatiotemporal Regularity Flow (SPREF) for Image and Video Processing

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Spatiotemporal Regularity Flow (SPREF) Mubarak Shah Computer Vision Lab School of Electrical Engineering & Computer Science University of Central Florida Orlando, FL 32765

  2. What are good features? • Color Histograms • Eigen vectors • Wavelet Coefficients • Edges • Spatiotemporal Surfaces of edges • XY, XT, YT slices • Spatial/spatiotemporal tensors • SIFT • Optical Flow

  3. SPREF • New Spatiotemporal feature for VACE • Generalization of Isophotes, Optical flow,… • Can be computed when gradient is zero • It analyzes whole region instead of a single pixel • Applications • Image and Video In-painting • Object removal • Video Compression • Tracking, Segmentation, …

  4. Spatiotemporal Regularity • Definition:A spatiotemporal volume is regular along the directions, in which the pixels change the least. • SPatiotemporal REgularity Flow (SPREF) • 3D vector field ζ • models the directions of regularity • No motion (Spatial Regularity) • Depends on the regularity of a single frame • Presence of motion (Temporal Regularity) • Global motion • Single regularity model • Local motion • Multiple regularity models

  5. Estimating SPREF • …gives the directions, along which the sum of the gradients is minimum: where F is the spatiotemporal volume, and H is a regularizing filter (Gaussian)

  6. The energy function is modified according to the flow type: x-y Parallel: ζ(c1'[t], c2'[t],1) y-t Parallel: ζ(1,c2'[x], c3'[x]) x-t Parallel: ζ(c1'[y],1, c3'[y]) The SPREF Energy Functions

  7. Solving for the SPREF • Approximate each flow component, cm'[p], with a 1D spline • Incorporates multiple frames in the solution. • Quadratic minimization of the energy functions • Solve for the spline parameters

  8. Solving T-SPREF Equation

  9. The original synthetic sequence (8 frames) x-y Parallelism: ζ(c1'[t], c2'[t],1) y-t Parallelism: ζ(1,c2'[x], c3'[x]) x-t Parallelism: ζ(c1'[y],1, c3'[y]) There are three types of planar parallelism constraints.

  10. The SPREF Curves • … define the actual paths, along which the GOF is regular.

  11. T-SPREF - An Overview • Demo

  12. x-y Parallel SPREF

  13. y x y t x y-t Parallel SPREF ζ(1,c2'[x], c3'[x])

  14. y y x t x x-t Parallel SPREF ζ(c1'[y],1, c3'[y])

  15. T-SPREF Results (Flower Sequence) Oblique View Top View Side View

  16. T-SPREF Results (Alex Sequence) Oblique View Top View Side View x y y t x t t

  17. The Affine SPREF (A-SPREF) • When the directions of regularity depend on multiple axes (zooming, rotation and etc.) • Precision of T-SPREF goes down • Translational flow model to Affine flow model • Affine (A-)SPREF Flow energy equation:

  18. Comparison of T- and A- SPREF 1st row: A synthetic sequence from the Lena image. 2nd row: T-SPREF approximation to the underlying directions of regularity. 3rd row: A-SPREF approximation of the directions of regularity.

  19. T-SPREF A-SPREF T-SPREF A-SPREF More examples

  20. Comparison of T- and A- SPREF

  21. Optical Flow Vs SPREF • SPREF carries similar but not necessarily the same information as the optical flow. • SPREF captures both the spatial and temporal regularity • Optical flow only cares about motion information in temporal direction. • When motion exists, the directions of xy parallel SPREF depend on direction of motion. • If the motion is globally translational, then xy-parallel SPREF converges to the optical flow.

  22. Optical Flow Vs SPREF • Optical Flow is not well-defined where the spatiotemporal gradients are insignificant. • Spline-based formulation of SPREF minimizes over multiple frames. • The true optical flow usually lacks plane parallelism.

  23. Optical Flow Vs SPREF

  24. Applications of SPREF

  25. Inpainting • Filling in the regions of missing data • Image Inpainting • Missing regions create spatial holes • Inpainting the missing region in the SPREF direction • Video Inpainting • Missing regions create spatiotemporal holes • Inpainting these holes require using the information from temporal neighbors.

  26. Image Inpainting

  27. Video Inpainting • Requires understanding the temporal behavior of the pixels. • The temporal behavior of the undamaged pixels gives clues about the behavior of the damaged pixels • Temporal behavior • Modeled explicitly by x-y Parallel SPREF

  28. Video Inpainting • The algorithm (cont’d) • Estimate the x-y Parallel SPREF curves using the non-missing regions. • The pixels along the SPREF curves vary smoothly • Fit a spline to the non-missing pixels along each flow curve. • Interpolate the values of the missing pixels from the splines

  29. Results • Big Bounce (Before)

  30. Results • Big Bounce (Flow)

  31. Results • Big Bounce (After)

  32. Supervised Removal of Objects from Videos

  33. Motivation • Object removal from videos • Preceding step to video inpainting • Manual selection of the object from each frame is required. • Time consuming • Use x-y Parallel SPREF to decrease the amount of manual work • Removal along the SPREF curves

  34. Algorithm • Given a group of frames (GOF): • Compute the x-y Parallel SPREF, and the SPREF curves • Remove the object from the first and the last frames of the GOF • Remove the pixels along the curves, whose first and last pixels have been removed.

  35. Results • Golden Eye (Final) • 86% reduction in manual work!

  36. Video Compression Using SPREF

  37. 3D Wavelet Decomposition • Problem • The spatiotemporalregularity of the GOF is not taken into account • Solution • Decompose the GOF along the SPREF directions • Entropy along these directions is lower: • Higher compression rate

  38. SPREF-based Video Compression • Warping the wavelet basis along the flow curves • x-y Parallel : G(x,y,t) = (x+c1[t], y+c2[t], t) • y-t Parallel : G(x,y,t) = (x,y+c2[x],t+c3[x]) • x-t Parallel : G(x,y,t) = (x+c1[y], y, t+c3[t])

  39. Choosing the correct SPREF type • The correct SPREF type is the one that minimizes the compression cost : Di + λRi • Di: Reconstruction error • λ: Lagrange multiplier • Ri: Bit cost of the bandelet and flow coefficients

  40. Fi Segmentation for Optimal Compression • Find the segmentation of the GOF (F) into subGOFs (Fi), such that the total compression cost is minimized:

  41. Oct-tree Segmentation • Recursively partition the GOF (F) into rectangular prisms (cuboids), Fi. • Compute the best flow and the compression cost for each cuboid. • Use split/merge algorithm to achieve the final segmentation. • Merge the child nodes if:

  42. Compression results for frames 98-105 of the Alex sequence at 1000kbps

  43. Compression results for frames 11-18 of the Akiyo sequence at 480kbps

  44. Compression results for frames 99-106 of the Mobile sequence at 350kbps

  45. Compression results for frames 26-33 of the Foreman sequence at 500kbps

  46. Compression Results (b) (a) The bit-rate vs PSNR plots of (a) Alex, (b) Akiyo. Both SPREF-based compression and LIMAT framework are shown in the results. LIMAT framework, Secker and Taubman, IEEE TIP, 2004.

  47. Compression Results (cond.) (b) (a) The bit-rate vs PSNR plots of (a) Foreman, (b) Mobile. Both SPREF-based compression and LIMAT framework are shown in the results.

  48. Summary • SPREF • New Spatiotemporal Feature • Computes direction of regularity simultaneously in space & Time • Similar to optical flow, edge direction.. • SPREF is plane parallel (xy, xt, yt) • SPREF is computed using region/image information instead of a single pixel • SPREF is defined even when gradient is zero

  49. Summary • Applications • Image and Video In-painting • Object Removal • Video Compression • Tracking • Segmentation

  50. Orkun Alatas August 16th, 1977 - September 3rd, 2005

More Related