Dezhen Song Department of Computer Science and Engineering Texas A&M University

1. CPSC 643 Aligning Windows of Live Video from an Imprecise Pan-Tilt-Zoom Robotic Camera into a Remote Panoramic Display Dezhen Song Department of Computer Science and Engineering Texas A&M University Supported in part by

2. Network PTZ Robotic Camera for Nature Observation • Panosonic HCM 280 • PTZ Robotic Camera: • 350° Pan, 120° Tilt, 42x Zoom • 200° per second servo speed • Network Video Camera: • Built-in streaming server • 640x480 pixels video • >30 frames per second • Low power consumption: <5 Watt • Affordable price: $ 1.2 K

3. Panorama Tilt Pan Frame sequence Tilt Updated Part in Panorama Panorama Live frame sequence Time Real Time Panoramic Video

5. Related Work • Multiple fixed cameras • [Swaminathan and Nayar 2000] • [Tan et al. 2004] • [Foote et al. 2000, 2001] • Single wide angle camera • [Baker and Nayar 1999] • [Nayar 1997] • [Xing and Turkowski 1997]

6. Related Work: Image Alignment • Direct Method • Use pixel intensity value • Sensitive to luminance change • Need good guess for initial parameters input • Existing work • [Shum and Szeliski 1997] [Szeliski 1994, 1996] • [Coorg and Teller 2000] [Kang and Weiss 1997] • Frequency Domain Registration • Existing work • [Castro and Morandi 1987] [Reddy and Chatterji 1996]

7. Related Work: Image Alignment • Feature-based Image Registration • Use feature points: Harris corner point, SIFT • Robust to luminance change • Faster than direct method • Existing work • [Torr and Zisserman 1997] [Brown and Lowe 2003] • [Zoghlami et al. 1997] [Hu et al. 2001] [Cho et al. 2003] • [Kanazawa and Kanatani 2002] [Zhang et al. 2002]

8. Comparison: Panoramic Video

9. Assumptions • Pan-tilt camera with a fixed base • Known intrinsic camera parameters • Calibrated camera before deployment • Inaccurate pan-tilt readings • May deteriorate over time • Standard video camera with HFOV ≤ 45o

10. Review: Perspective Projection [Tsai86, 87] Intrinsic Parameters Extrinsic Parameters

11. Problem Definition Re-projection: Project image B onto image A plane: Image alignment:

12. Excessive Computation in Image Alignment Speed slow down caused by coupling re-projection and SSD: • Extensive float point computation • Coupled with Sum of Squared Difference (SSD) operation • A naive search takes O(km) re-projection operations • k: number of candidate pan/tilt pairs over feasible solution set. • m: number of overlapping pixels • Proposed solution: decouple re-projection and SSD • Spherical re-projection • Cell-based Alignment • Constant time alignment

13. CY O v Image plane f t p CX CZ u Spherical Projection • Project image onto a spherical surface • Image =(p, t) on local spherical coordinate system { }

14. 400 0.8 200 v 0 0.4 -200 t 0 -400 -0.4 -1000 -600 -200 200 u -0.8 -1.5 -1 -0.5 0 0.5 p Distortion under Re-projection Planar Spherical Two poses have 30o pan difference with the same 30o tilt value

15. 0.8 0.4 t 0 -0.4 -0.8 -1.5 -1 -0.5 0 0.5 p Invariant under Spherical Re-projection

16. Re-projection after Spherical Projection Define conversion between camera coordinate system and local spherical coordinate system Re-projection function between two local spherical coordinate system

17. Lemma 1: If the spherical cell is small , define point and its corresponding point we have • Rc is a 2x2 rotation matrix • Cell distortion under re-projection is negligible.

18. Introduce coefficient matrix H Radius f remains same Proof for Lemma 1:

19. HFOV≤45o and VFOV ≤34o  0.956 ≤ cos(t) ≤ 1 Dropping cos(t) introduces ≤5% distortion for 20x20 cell Continue: Proof for Lemma 1 ∆f=0  substitute [m13, m23, m33]T

20. Continue: Proof for Lemma 1

21. Lemma 2: Rotation angle Θ of Rc can be approximated by where αis the dot product of Z axis of {CA} and {CB}

22. Cj ε(Cj ) Algorithm

23. Select kc cells from the overlapping region in O(1) • Sphere projection O(1) • Feature detection in the cell and searching regions O(1) • For each(δpB, δtB) O(1) • For each cell O(1) • Compute Cj • Compute , j=1, …, kc • Compute SSD between and • End For • Report sum of SSD across all cells • End For • Output solution with the minimum SSD Cell based Image Alignment

24. Experiment and Results Speed test: • 881 milliseconds to align 21 320x240 images • 4 seconds for Autostitch program on same data set • Up to 25fps on a laptop PC

26. Thank You! • Align video images from a camera that only differ in pan and tilt settings into a panorama at 25 frames per second. • Alignment is performed on a spherical surface to avoid excessive distortion caused by homographic transformation. • A constant time algorithm pre-rotates small pre-sampled squared patches on spherical surface for matching. • Experiments show that the alignment speed is 4.5x faster than the best method available. live video window