1. Viewpoint Invariant Features from Single Images Using 3D Geometry Yanpeng Cao(post-doc) and John McDonald (Co-PI) Overview Motivation: To obtain good quality local features from single images for robust matching over widely separated views. Provides a core component in various computer vision applications, such as Structure from Motion (SFM), pose estimation, and object recognition. Challenge: When images are taken at significantly changed viewpoints, the state of the art feature techniques (e.g. SIFT, MSER, etc.) cannot to produce satisfactory matching results because perspective effect will add severe distortions to the image , resulting in large variation in the descriptors. Proposed solution: We combine recent advances in 2D interest point detection and description with 3D viewpoint normalization: (1) Retrieve the 3D building layout from single images; (2) normalize the 2D local region of interest with respect to the recovered 3D geometry to achieve viewpoint invariance. Viewpoint Invariant Features Urban environments usually contain many structured regularities, so that the images of those environments contain straight lines (parallel in 3D space) resulting in vanishing points in image space. We present an effective scheme to recover 3D planar surfaces using the extracted line segments and their associated vanishing points. Step 3: Viewpoint normalization Each line segment from a horizontal vanishing direction provides a strong indication of the existence of a 3D plane in that direction. Based on the distribution of line segments, we divide the whole image into several vertical strips with each one corresponding to a different 3D plane (building walls) in the world. For the detected vertical strip (a 3D plane), we warp the original image to a normalized front-parallel view where perspective distortion is removed. Segmentation result after assigning each strip a single direction. Each line segment gives a vertical strip to support a 3D plane in its direction Step 1: Tilt rectification Transform the original image to a rectified view where the vertical tilt effect is removed (i.e. vertical world lines become parallel and upright in the image). Building boundaries will appear vertical Choose four lines to construct a quadrilateral Result of viewpoint normalization Step 2: Line grouping Group the non-vertical lines into dominant vanishing directions by identifying their associated vanishing points. The search of possible vanishing points can be restricted to a small horizontal strip by exploiting the fact that the “horizon” (the line connecting horizontal vanishing points) will appear horizontal in the rectified image. Step 4: Feature extraction and matching Viewpoint invariant features are then extracted on these rectified images. The resulting features provide useful information such as patch scale, dominant orientation, and plane label for efficient feature matching. The initial line extraction Line grouping result Viewpoint invariant features are extracted in the rectified image Efficient matching over widely separated images Experiments Quantitative evaluation across variable viewpoints Wide baseline image matching The viewpoint invariant features can handle the large viewpoint changes for which SIFT and MSER do not work (even hard for human vision) The performances of local feature extraction and matching can be significantly improved by taking into account the underlying 3D geometry . Future works To facilitate an end-to-end vision based navigation system Lidar + vision sensor fusion More general 3D recovery from single images via ML Research presented in this poster was funded by a Strategic Research Cluster Grant (07/SRC/I1168) by Science Foundation Ireland under the National Development Plan. The authors gratefully acknowledge this support.