Advanced 3D Reconstruction Techniques for Moving Objects and Canal Surfaces

1. Computer Vision Projects (:,5) Vincenzo Caglioti Giacomo Boracchi, Simone Gasparini, Alessandro Giusti, Pierluigi Taddei

2. Computer Vision Group Team • Vincenzo Caglioti • Giacomo Boracchi, Simone Gasparini, Alessandro Giusti, Pierluigi Taddei

3. Reconstructing Canal Surfaces, trajectories and spin of moving balls We can reconstruct circular-cross section canal surfaces from a single image (main algorithms are already implemented in well-packaged Java methods).

4. Explore possibilities for a 3D input device (maybe real-time?) with a webcam + flexible tube. Develop demo application (OpenGL or Java3D). Example: virtual flexible stick-figure. Example: 3D modeling of flexible objects. Be creative! :) Requisites: learn principles of 3D visualization Quantitatively evaluate performance of current algorithms, and test possible improvements. Implement camera autocalibration. Prerequisite: knowledge of camera calibration and projective geometry (Image Analysis and Synthesis classes). Project proposals

5. Other applications With the same algorithm we can reconstruct the trajectory of a moving ball from a long-exposure photograph (no frame rate issues, can handle very fast games, any lighting condition, inexpensive equipment). • Reconstruct the trajectory of a real moving ball. Evaluate reconstruction accuracy. Can we also measure nonparabolic trajectories (e.g. Pirlo’s penalty kick, spinning table tennis shots, volleyball “floater” serves…)? Mostly implementation work, some interesting possible optimizations. • Implement an automatic refereeing system for table tennis. • Implement a system for detecting the exact bounce position of a fast-moving ball from its blurred trail (“in or out?”). • Augment low-frame rate videos of table tennis matches (think of the new Shell’s TV advertisement). For example: • Draw ball “shadow” on the table (3D reconstruction) • Draw ball velocity vector • Predict the remaining trajectory portion • Manipulate the ball trail opacity and/or color (image processing) • Estimate ball speed • Synthethize a 100 fps slow motion replay from a 20 fps video…

6. Ball spin from a single image (ongoing research) If the ball surface is textured: analyze the trail and find the ball spin (axis and rotational speed) – ongoing research • 3D geometry issues • Image processing issues • Find ball spin axis and speed from traces left by dots on the ball surface (long-exposure). • Find ball spin axis and speed from blur of the ball’s surface features (short-exposure) -- joint work with Giacomo Boracchi. • Find ball spin speed from orthographic images (long-exposure)  look for periodic color patterns.

7. Tram transit detection and notification • A webcam will be placed on the DEI building pointing at Via Edoardo Bassini. • The video stream will be analized in order to detect the transit of any ATM tram, identifying its number and registering the transit time • A web framework will be then implemented in order to predict the next tram transit • The system will be exploited by the department employees in order to leave the office at the last usefull moment • Next predicted transits: • heading to Duomo: 5’ 23’’ • heading to Lambrate: 2’12’’

8. Structure from Motion • The aim is to reconstruct the 3D structure of a scene and the camera motion using as input only the video sequence caputred

9. Project 1: Surface fitting • Given the 3D points and the initial image frames identify reliable surface patches onto which the initial images can be mapped Coordinators: Caglioti, Taddei

10. Project 2: Feature Tracking for Structure from Motion • The features tracked may disappear due to occlusion or to wrong matches between images • A good tracking algorithm should compensate for these effects and extract as many features as possible Coordinators: Caglioti, Taddei

11. Project 3: Paper Like surfaces • The object recorded is assumed to be a paper-like surface that is represented by a particular family: developable surfaces • The project will be aimed to build a framework to generate sintetic datas to test the alghoritms Coordinators: Caglioti, Taddei

12. Motion Estimation from a Single Blurred Image • Application: 3D reconstruction from a single image • Local motion extraction from blurred details (Corners/Texture) • Exploit Global Camera Movement

13. Motion Estimation from a Single Blurred Image • Image Restoration: De-Blurring • Build a “Blur Map” • Adapt Existing De-blurring Techniques to real blurred images

14. Robbery detection • Objective: automatically detect holdup situations (“Hands Up!”) from video-surveillance sequences on a dsp-equipped camera. Background subtraction Detection of “hands up” pose Color-based skin segmentation

15. Robbery detection Face and hands detection

16. Robbery detection - Available projects • Pose recognition • Develop an approach based on silhouette extraction and pose recognition • Face detection • Improve performance and robustness of face detector, test on a larger training set • Hands detection • Develop a new detection algorithm (similar to the face detection one) and test performance (hit rate and computational speed) • Skin segmentation • Improve performance of the skin detector using a voting system involving three color spaces RGB, YCbCR, HUV. Coordinators: Caglioti, Boracchi, Gasparini, Giusti, Taddei

17. Rectification of perspective images • Objective: removing perspective effect from images Perspective Image Rectified Image

18. Natural image recognition through compression level analysis • Natural image recognition • Recognition of natural image (e.g. leaves, flowers) by compressing the contour image and matching the compression levels Coordinator: Caglioti Original image Edge image

19. Rectification of perspective images • Available project: • 3D reconstruction of urban scene from uncalibrated images for virtual tour Coordinator: Caglioti

20. Calibration of catadioptric camera • Catadioptric camera: a perspective camera placed in front of a curved mirror Catadioptric camera Catadioptric images Mirror Camera

21. Calibration of catadioptric camera • Calibration procedure • Develop a new calibration procedure for catadioptric cameras from single image exploiting the silhouette of the mirror and the alignment constraints deriving from the image of straight lines. Coordinator: Caglioti, Gasparini, Taddei

22. License Plate Recognition • Objective: automatically detect and recognize license plate from video sequences on a dsp-equipped camera. YZH 4025

23. License Plate Recognition – Available Projects • New Starting Project • Probably Strict Deadlines • ONLY THE BRAVES! • License Plate Detection Module • Develop a module that detect the license plate in image according to color and shape • License Plate Recognition Module • Given the license plate image, develop a module that recognize characters (e.g. using a neural network) and provide the license number • Both projects require good C-programming skills • Coordinators: Caglioti, Gasparini, Taddei

24. Robot mapping • Objective: built a map of the environment collecting laser scans while robot is moving • Scanning while moving algorithm • Implement and test on a real robot (Mo.Ro 2) the mapping algorithm • Coordinators: Caglioti, Gasparini Map built by collecting scans Laser scanner

25. Thank you… Further informations avaialble at www.elet.polimi.it/people/caglioti/ {caglioti | boracchi | gasparini | giusti | taddei}@elet.polimi.it