1 / 29

CS 395/495-25: Spring 2004

CS 395/495-25: Spring 2004. IBMR: The Fundamental Matrix Combines Two Cameras Jack Tumblin jet@cs.northwestern.edu. Basic Camera P 0 : P 3 P 2 ( or camera R 3 ). Basic Camera P 0 is a 3x4 matrix: Non-square pixels? change scaling ( x ,  y )

adriel
Télécharger la présentation

CS 395/495-25: Spring 2004

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. CS 395/495-25: Spring 2004 IBMR: The Fundamental Matrix Combines Two Cameras Jack Tumblin jet@cs.northwestern.edu

  2. Basic Camera P0: P3P2(or camera R3) • Basic Camera P0 is a 3x4 matrix: • Non-square pixels? change scaling (x, y) • Parallelogram pixels? set nonzero skew s K matrix: “(internal) camera calib. matrix” yc xc yc zc 1 xfspx0 0 yfpy00 0 1 0 x y z = y y’ p C z K (3x3 submatrix) zc f y' P0 X = x x' xc [K 0] = P0

  3. ONE Camera Matrix P links P3P2: xfspx0 0 yfpy00 0 1 0 y Input:X 3D World Space Output: x 2D Camera Image x X (world space) z yc C zc xc x(camera space) y x • Basic camera: x = P0 X whereP0 = [K | 0] = • World-space camera: translate X (origin) to camera location C, then rotate: x = PX = (P0·R·T) X = K·R·[I C] X • Rewrite as: P = K[R -RC] • Redundant notation:P = [M p4]M = RKp4 = -K·R·C ~ ~ ~ ~

  4. ONE Complete Camera Matrix P ~ ~ C y C x X (world space) z yc C zc xc y x • K matrix: “internal camera calib. matrix” • R·T matrix: “external camera calib. matrix” • T matrix: Translate world to the cam. origin • R matrix: 3D rotate world to fit cam. axes Combine: write (P0·R·T·X = x or K[R -RC]·X = x or P·X = x Input:X 3D World Space Output: x 2D Camera Image ~ x(camera space)

  5. TWO Cameras: Epipolar Geometry Basic idea: • Given ONLY images from 2 cameras C, C’ • Different views of same objects X, butwe don’t know world-space points X. • If we choose an x, how can we find x’ ? • How are x, x’ linked? X x’ x C’ C

  6. TWO Cameras: Epipolar Geometry Basic idea: • 2 cameras located at C, C’ in world space. • Find ‘baseline’ through camera centers C, C’ • Baseline hits image planes at ‘epipoles’ Notice baseline and Xform a plane... (OTHER planesthru baseline for other X...) X x’ x baseline C’ C

  7. TWO Cameras: Epipolar Geometry Basic idea: • 2 cameras located at C, C’ in world space. • Find ‘baseline’ through camera centers C, C’ • Baseline hits image planes at ‘epipoles’ • Family of planes thru baseline are all the ‘epipolar planes’ • Image of planes = lines =‘epipolar lines’ • Lines intersect at epipolar points in both images. X x’ x baseline C’ C

  8. TWO Cameras: Epipolar Geometry Summary: • Connect cameras C, C’ with a baseline, which hits image planes at epipoles e, e’. • Chose any world pt X, then  everything is coplanar! epipolar plane includes image points x, x’, and these connect to epipoles e,e’ by epipolar lines L, L’ X epipolar plane  epipolar line L’ epipolar line L x’ x baseline C’ e’ e C

  9. Epipolar Geometry Useful properties: • Every image point x maps to an epipolar line L’also: • Epipoles e,e’ = each camera’s view of the other • All epipolar lines L’ pass through epipole e’ • Epipolar Line L’ is (image of the CX ray...) • Epipolar Line L’ links (image of C) to (image of X) X epipolar plane  epipolar line L’ epipolar line L x’ x baseline C’ e’ e C

  10. Fundamental Matrix: Fx = L’ One Matrix Summarizes ALL of Epipolar Geometry! Fundamental Matrix F: 3x3, rank 2. • Maps image point x to image point x’: x’T F x = 0 but F is only Rank 2 – given only x, F cannot find x’ for you!! • Maps image point x to epipolar line L’: F x = L’ X epipolar plane  epipolar line L’ epipolar line L x’ x baseline C’ e’ e C

  11. Fundamental Matrix: Fx = L’ One Matrix Summarizes ALL of Epipolar Geometry! Fundamental Matrix F: 3x3, rank 2. • Maps image point x to image point x’: (x’T F) x = 0 (We know x is on line L, thus (LT) x = 0, so • Maps image point x to epipolar line L’: F x = L’ X epipolar plane  epipolar line L’ epipolar line L x’ x baseline C’ e’ e C

  12. Fundamental Matrix: Fx = L’ • 2) What is F if we DON’T know the cameras P, P’,but we DO know some corresp. point pairs (x, x’)? • F finds epipolar line L’ from point x : Fx = L’ • (Recall that if (any) point x’ is on line a L’, then x’T L’= 0) • Substitute Fx for L’: x’TF x = 0 AHA! we can find F using DLT-like method! see Chap. 10 X epipolar plane  epipolar line L’ epipolar line L X() x’ x baseline C’ e’ C

  13. Derive the Fundamental Matrix: • Recall Pseudo-Inverse: P+ = PT ( P PT)-1 and P P+= I (pg148) • Write world-space ray XC as: X() = P+x + C • Other camera’s image of the ray is its epipolar line L’: L’() = P’X() = P’P+x + P’C . • But P’C = e’; it is the epipole of the other camera, so L’() = P’P+x + e’ X epipolar plane  epipolar line L’ epipolar line L X() x’ x baseline C’ e’ C

  14. Derive the Fundamental Matrix: • But P’C = e’; it is the epipole of the other camera, so L’() = P’P+x + e’ . • Rewrite epipolar line L’ using points P’P+x and e’ • (Recall: find line between two P2 points with cross product: L= x1 x2) • To get L’ = e’P’ P+ x or L’ = F x X epipolar plane  epipolar line L’ epipolar line L X() x’ x baseline C’ e’ C

  15. Derive the Fundamental Matrix: • But P’C = e’; it is the epipole of the other camera, so L’() = P’P+x + e’ . • Rewrite epipolar line L’ using points P’P+x and e’ • (Recall: find line between two P2 points with cross product: L= x1 x2) • To get L’ = (e’) (P’ P+ x) or L’ = F x THUS F = [e’]P’ P+ X epipolar plane  epipolar line L’ epipolar line L X() x’ x baseline C’ e’ C Hunh? What Is This?!?!

  16. Derive the Fundamental Matrix: F = [e’]P’ P+But what’s this? A NEW TRICK: • Cross Product written as matrix multiply (Zisserman pg. 554) a b = = = = [a] · b • So write: a b = -b a = [a]·b = (aT·[b])T X epipolar plane  epipolar line L’ epipolar line L X() x’ x baseline C’ e’ C b1 b2 b3 a2b3 – a3b2 a3b1 – a1b3 a1b2 – a2b1 0 -a3 a2 a3 0 -a1-a2 a1 0 b1 b2 b3 a1 a2 a3 a ‘skew symmetric’ matrix

  17. Fundamental Matrix: Fx = L’ • Given a point pair (x,x’) Matrix F is unique! • Cool! still works even for different cameras! • F tied DIRECTLY to corresp. point pairs (x, x’): • F finds epipolar line L’ from point x : Fx = L’ • (Recall that if (any) point x’ is on line a L’, then x’T L’= 0) • Substitute Fx for L’: x’TF x = 0 X epipolar plane  epipolar line L’ epipolar line L X() x’ x baseline C’ e’ C F = [e’] P’ P+

  18. Fundamental Matrix: Fx = L’ 1) How do we findF? If we know the camera matrices P and P’ (we almost never do), we showed (Zisserman pg 224) F = [e’]P’ P+ X epipolar plane  epipolar line L’ epipolar line L X() x’ x baseline C’ e’ C ( Recall: a b = -b a = [a]·b = (aT·[b])T ) (Recall: P+ = PT(PPT)-1, the pseudo-inverse)

  19. Fundamental Matrix: Fx = L’ • 2) What is F if we DON’T know the cameras P, P’,but we DO know some corresp. point pairs (x, x’)? • F finds epipolar line L’ from point x : Fx = L’ • (Recall that if (any) point x’ is on line a L’, then x’T L’= 0) • Substitute Fx for L’: x’TF x = 0 AHA! we can find F using DLT-like method! see Chap. 10 X epipolar plane  epipolar line L’ epipolar line L X() x’ x baseline C’ e’ C

  20. Fundamental Matrix Summary (pg. 226) F is 3x3 matrix, maps P2P2, rank 2, 7-DOF • If world space pt X image space pts. x and x’ then x’TF x = 0 • Every image pt has epipolar line in the other image:Fx = L’FTx’ = L • Baseline pierces image planes at epipoles e, e’ : Fe = 0 FTe’ = 0

  21. Fundamental Matrix Summary (pg. 226) F is 3x3 matrix, maps P2P2, rank 2, 7-DOF • Given camera matrices P, P’, find F matrix by: F = [e’] P’ P+ (recall: e’ is image of C: e’ = P’C) • F is unaffected by any proj. transforms done on BOTH cameras(PH, P’H) has same F matrix as (P, P’) for any full-rank H (e.g. F measures camera C vs. Camera C’ only, no matter where you put them)

  22. Fundamental Matrix Uses tx ty tz Special case: camera translate only (no rotations) • Camera matrices are P= K[ I | 0 ], P’ = K[I | t ] • where K is internal calib., t is 3D translation vector • F matrix simplifies to F = [e’] • Epipolar lines are all parallel to direction t • x,x’ displacement depends only on t & 3D depth z: x’ = x + (Kt)(1/z)

  23. Fundamental Matrix Uses Can we find a camera matrix K from motion + fundamental F? • Let one camera position define the world’s coords:P = P0 = K[ I | 0 ], and define the other as P’ = [M | m] = K[R | -RC] where K is internal calib., R is rotation C is world-space position • F matrix simplifies to F = [m]M • If we know how we moved the camera (R, C matrices) then find F by correspondence, and then solve for K. (pg 237) ~ ~ ~

  24. Fundamental Matrix Uses General movement? • Recall: rotations don’t change image content (camera rotate a P3 homography matrix H) • ANY cameras, ANY movements can then be warped to remove rotations, THUS • Can ALWAYS get parallel epipolar lines! • Easier to find correspondences • Easier to find depth values z • ‘Parallel Epipolar Lines’==‘Rectified Image Pair’

  25. Fundamental Matrix Properties (pg. 226) Why bother with F? • Can find it from image pt. correspondences only • Works even for mismatched cameras (example: 100-year time-lapse of Eiffel tower) • Choose your own world-space coordinate system. • SVD lets us recover P, P’ camera matrices from F • (4-way ambiguity; what is front/back of C and C’?) pg 240 • BUT WE DON’T NEED TO! • Complete 2-camera mapping from worldimage • 2 images + corresponding point pairs (xi,x’i)F • Let camera coords == 3D world coords, then (xi,x’i)Xi

  26. Correspondence Problem: Where Computer Vision, IBMR part ways: Few Images: (image pairs, trios…) FIND Fundamental Matrix + FIND Corresponding point pairs (x,x’) ************Versus************** Many Images: GATHER enough for P2-only work: ignore or simplify the correspondence?

  27. Conclusions • P2, P3 matrix forms give elegant, principled notation for ALL image geometry • Cameras, lights, points, lines, planes, conics, quadrics, twisted cubics, … • Matrix form makes everything reversible: 3D from (2D)*! • We can find shapes from images,BUT ONLY if we have enough reliable point correspondences…And THAT’s what we usually CAN’T find!

  28. END

  29. Another way to describe P3 lines! • Given world-space camera position C =[a b c 1] and • a world-space direction D (found from a pixel xd, as shown on prev. slide), • even though D is a ‘point at infinity’, we can use it to make a parametric line in world space:line(t) = C + Dt =[a b c 1] + [d e f 0]t • Note that ‘t’ is position along the line, and it’s in world-space coords! • If you have two cameras C0, C1, and found two corresponding image points p0 and p1, you can use method on previous slide to find a world-space direction D0, D1 for each image point, then make two parameterized lines L0(t0), L1(t1) and solve for their world-space intersection point X. THAT’S how you can do triangulation!.

More Related