Motion from image and inertial measurements (additional slides)

1. Motion from image and inertial measurements (additional slides) Dennis Strelow Carnegie Mellon University

2. Outline Robust image feature tracking (in detail) Lucas-Kanade and real sequences The “smalls” tracker Motion from omnidirectional images Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, 20052

3. Robust image feature tracking: Lucas-Kanade and real sequences (1) • Combining image and inertial measurements improves our situation, but… • we still need accurate feature tracking tracking • some sequences do not come with inertial measurements Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, 20053

4. Robust image feature tracking: Lucas-Kanade and real sequences (2) • better feature tracking for improved 6 DOF motion estimation • remaining results will be image-only Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, 20054

5. Robust image feature tracking: Lucas-Kanade and real sequences (3) • Lucas-Kanade has been the go-to feature tracker for shape-from-motion • minimizes a correlation-like matching error • using general minimization • evaluates the matching error at only a few locations • subpixel resolution Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, 20055

6. Robust image feature tracking: Lucas-Kanade and real sequences (4) Additional heuristics used to apply Lucas-Kanade to shape-from-motion: Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, 20056

7. Robust image feature tracking: Lucas-Kanade and real sequences (5) But Lucas-Kanade performs poorly on many real sequences… Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, 20057

8. Robust image feature tracking: the “smalls” tracker (1) • smalls is a new feature tracker targeted at 6 DOF motion estimation • exploits the rigid scene assumption • eliminates the heuristics normally used with Lucas-Kanade • SIFT is an enabling technology here Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, 20058

9. Robust image feature tracking: the “smalls” tracker (2) • First step: epipolar geometry estimation • use SIFT to establish matches between the two images • get the 6 DOF camera motion between the two images • get the epipolar geometry relating the two images Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, 20059

10. Robust image feature tracking: the “smalls” tracker (3) Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, 200510

11. Robust image feature tracking: the “smalls” tracker (4) Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, 200511

12. Robust image feature tracking: the “smalls” tracker (5) • Second step: track along epipolar lines • use nearby SIFT matches to get initial position on epipolar line • exploits the rigid scene assumption • eliminates heuristic: pyramid Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, 200512

13. Robust image feature tracking: the “smalls” tracker (6) • Third step: prune features • geometrically inconsistent features are marked as mistracked and removed • clumped features are pruned • eliminates heuristic: detecting mistracked features based on convergence, error Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, 200513

14. Robust image feature tracking: the “smalls” tracker (7) • Fourth step: extract new features • spatial image coverage is the main criterion • required texture is minimal when tracking is restricted to the epipolar lines • eliminates heuristic: extracting only textured features Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, 200514

15. Robust image feature tracking: the “smalls” tracker (8) Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, 200515

16. Robust image feature tracking: the “smalls” tracker (9) left: odometry only right: images only • average error: 1.74 m • maximum error: 5.14 m • total distance: 230 m Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, 200516

17. Robust image feature tracking: the “smalls” tracker (10) • Recap: • exploits the rigid scene and eliminates heuristics • allows hands-free tracking for real sequences • can still be defeated by textureless areas or repetitive texture Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, 200517

18. Outline Robust image feature tracking (in detail) Motion from omnidirectional images Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, 200518

19. Motion from omnidirectional images (1) Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, 200519

20. Motion from omnidirectional images (2) Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, 200520

21. Motion from omnidirectional images (3) Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, 200521

22. Motion from omnidirectional images (4) Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, 200522

23. Motion from omnidirectional images (5) left: non-rigid camera right: rigid camera squares: ground truth points solid: image-only estimates dash-dotted: image-and-inertial estimates Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, 200523

24. Motion from omnidirectional images (6) • In this experiment: • omni images • conventional images + inertial • have roughly the same advantages • But in general: • inertial has some advantages that omni images alone can’t produce • omni images can be harder to use Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, 200524