Optimizing Content-Preserving Projections for Wide-Angle Images

1. Optimizing Content-Preserving Projections for Wide-Angle Images ACM SIGGRAPH 2009 Robert Carroll (University of California, Berkeley) ManeeshAgrawal (University of California, Berkeley) AseemAgarwala (Adobe Systems, Inc.)

2. Outline • Introduction • Wide-angle projection • Approach • Results • Future work

3. Introduction Perspective Mercator Stereographic Paper result

4. The space of wide-angle projections • Wide-angle projections should maintain the following properties: • Shape constancy • Orientation constancy • There is no wide-angle projection that can simultaneously preserve all of these properties

5. Perspective projection • The viewing sphere is projected onto a tangent plane through lines emanating from the center of the sphere. • Orientation constancy • Not conformal projection • Robust for fields of view less than about40° • Field of view approaches 180° the stretching becomes infinite

6. Mercator projection The Mercator projection is a cylindrical projection that is designed to maintain conformality • Conformal projection • It can handle a complete 360 horizontal field of view, but stretches to infinity as the vertical field of view approaches 180. • useful for panoramic images with large horizontal fields of view

7. Stereographic projection The viewing sphere is projected onto a tangent plane through lines emanating from the pole opposite the point of tangency. • Conformal projection • Like perspective projection, stereographic projection stretches objects toward the periphery

8. Approach

9. Approach Select lines • Click on the two endpoints of the linear structure to specify the constrain • general line constraint • fixed orientation line constraint (modify the general line constraint) endpoints Drawn line Line in the scene endpoints

10. Approach • The general line constraint • Keep linear structures in the scene from bending • The fixed orientation line constraint • Let linear structures map to straight lines at a user-specified orientation in output images (user can choose vertical or horizontal)

11. Approach • Given these line constraints our algorithm computes a mapping from the viewing sphere to the image plane.

12. Mathematical setup • Notations Viewing sphere maps to plane Mapping function: ,vector form: longitude : latitude : vector form:

13. Mathematical setup • Local properties of this mapping • Differential vector • Conformal mapping , spherical coordinates are non-Euclidean equal steps in travel different distances on the sphere depending on (1) (2) , (3) Cauchy-Riemann equations for mapping a sphere to a plane [Hilbert and Cohn-Vossen 1952; Snyder 1987]

14. Mathematical setup • Discretize the mapping • In our case, we can’t derive a close-form solution • We discretize the mapping by sampling a uniform grid in () indexed by integers () Quad : entire set of vertices () that fall in the field of view of the input image

15. Conformality • We form conformality constraints on the mesh by discretizing the Cauchy-Riemann equations (3), giving , (3) (4) (5)

16. Conformality • All quads on the viewing sphere are not equal in size • We weight the constraints by (6)

17. Straight lines • We define : the set of all line constraints marked by user :orientation line constrain • Virtual vertex • Midpoint of line-quad intersection • We define a virtual vertex as abilinear interpolation of the surrounding vertices. points are collinear on the image plane points lie on a line (line is on the viewing sphere) Sphere Virtual vertex

18. Straight lines • We compute theposition of a virtual vertex on the sphere, and its bilinear interpolationcoefficients (a, b, c, d), as shown in Figure on which we place our line constraints.

19. Straight lines • : two line endpoints • For the rest of this section we drop the superscript and assume the u variables correspond to virtual vertices for a particular line

20. Straight lines • Distance • We therefore define the line energy for a constrained line as (7) (8) This energy function is non-linear, so we simplify the line energy in two ways, each of which can be solved linearly

21. Straight lines • We can express the energy function as another way : Equation (10) is the normalized length of the projection of () onto ()

22. Straight lines (8) • Two ways to simplify the line energy • By fixing the normal vector in equation (8)、(9)

23. Smoothness (13) (14)

24. Spatially-varying constraint weighting • Line endpoint weights : • Salience weights : • Face detection weights : • face detection algorithm of Viola and Jones [2004], as implemented in OpenCV [Bradski and Kaehler2008] • Total weight

25. Total energy and Optimization • Total energy function • The quadratic energy function at each iteration of our algorithm results in a sparse linear system Ax= 0 • PARDISO sparse direct solver • + (16)

26. Results

27. Future work • Developing a completely automatic system that identifies salient linear structures using line detection algorithms • Improved by using a more sophisticated salience measure