Seamless Multi-Projector Display on Curved Screens

1. Seamless Multi-Projector Display on Curved Screens Jeroen van Bar, Thomas Willwacher, Srinivas Rao, Ramesh Raskar Mitsubishi Electric Research Labs Cambridge, MA USA

2. Curved Screen Displays Multiple overlapping projectors on curved screens Goal : • Replace single-proj • Higher resoltn/brightness • Sub-pixel auto-alignment • Parametric solution • Low cost infrastructure Markets Planetarium Curved screens

3. Dome Projection Techniques Edge-Blended (Tiled/Mosaic) Display • Sub-Frames w/Spherical Mapping & Edge-Blends

4. 1 Dome Screen 2 4 3

5. OutlineSeamless Curved Display • Multi-projector low cost method • Simplified Quadric Image Transfer • Calibration with camera-pair • Parametric Rendering solution

6. Related Work • Conventional Displays • Manual alignment, expensive infrastructure • [Jupiter,Trimensions, CAVE, Planetaria, Flight Simulators] • Planar Screens • Camera in loop, auto calibration, low cost • Exploit homography parameters • [Raskar98,Surati99,Chen00,Brown02 ……] • Curved Screens • Non-parametric solutions • [Jarvis97,Raskar98,Yang01 …] • Parametric • ?, Siggraph 2003

7. Parametric Approach • Advantages • Lower camera resolution • Tolerance for pixel localization errors • Faster calibration • Efficient well-defined warping • Avoid look up tables

8. X j i Parametric Image Transfer X i j Planar Homography Quadric Transfer

9. Planar projective transfer What is homography ? • Two images of 3D points on a plane are related by a 3x3 matrix M i j = A3 x 3 i j Proj 1 Proj 2

10. What is homography ? Two images of 3D points on a plane Related by a 3x3 matrix ~ j = A3 x 3 i A3 x 3 a1 a2 a3 b1 b2 b3 c1 c2 c3 jx jy 1 ix iy 1 = k j i jx = (a •i)/ (c •i) Proj 1 Proj 2 jy = (b •i)/ (c •i)

11. 1 2 3 1 4 2 3 4 Planar Displays Current Multi-Cube System MERL Projector Planar Mosaic

12. Curved ScreensView for a Sweet-spot Projector Sweet spot (Static user)

13. Calibration for a Sweet-spot Projector Camera at Sweet spot

14. Discretized non-parametric approach Projector Image p1 p6 Projector c1 Camera at Sweet spot c6 Camera Image Desired Image =

15. Off-Axis Spherical Distortion Offset Viewpoint Ideal Viewpoint

16. Fish-eye ProjectionPlanetaria and Digital Dome Theaters • Immersive Production Software • Spitz - PolyDome™ • SkySkan - DigiDome™

17. OutlineSeamless Curved Display • Multi-projector low cost method • Simplified Quadric Image Transfer • Calibration with camera-pair • Parametric Rendering solution

18. Ruled quadrics: hyperboloids of one sheet Degenerate ruled quadrics: cone two planes Curved projective transferQuadric classification Projectively equivalent to sphere: sphere ellipsoid paraboloid hyperboloid of two sheets

19. Quadrics For 3D points X on Quadric X Q : 4x4 symmetric matrix, Nine d.o.f Q In general 9 points in 3D define quadric

20. Quadric Image Transfer • Quadratic image transfer function [Shashua97] X Quadric written as x’ x 21 params, 4 more than necessary !

21. Simplified Quadric Image Transfer Our Solution X Based on observation .. x’ x 17 param warp

22. Simplified Quadric Image Transfer X 17 param warp x’ x Planar homography: 4 corresponding pixels Quadric transfer: 9 corresponding pixels

23. OutlineSeamless Curved Display • Multi-projector low cost method • Simplified Quadric Image Transfer • Calibration with camera-pair • Parametric Rendering solution

24. Calibration of Quadric Screens

25. Approach Calibration • At each projector i, • Project structured pattern • View with stereo camera • Finding camera to projector quadric transfer, Run-time • At each projector i, • Pre-warp input image using

26. CalibrationFinding relationship between camera and projector Low-res Camera 640x480 images But each Projector 1024x768

27. Non-linear Refinement Linear Estimation Error ~10 pixels NonLinear Refinement Error ~ 1.0 pixels

28. Before Blending

29. After Blending

30. Intensity Correction in Overlap Projector Framebuffers

31. Projector Framebuffers

32. Projector Framebuffer Intensity Weights

33. OutlineSeamless Curved Display • Multi-projector low cost method • Simplified Quadric Image Transfer • Calibration with camera-pair • Parametric Rendering solution

34. Rendering a 3D Scene Steps at each projector (Pre-distort vertex 3D location) • For each triangle T with vertices {Mj} • For each vertex M • Find pixel m via VirtualViewProjection( M ) • Find warped pixel m’ via quadricTransferof m • Replace M with m’

35. Vertex Shader for Quadric Transfer in Cg vertout main( appin IN, uniform float4x4 modelViewProj, uniform float4 constColor, uniform float3x3 A, uniform float3x3 E, uniform float3 e) { vertout OUT; float4 m1 = float4(IN.position.x, IN.position.y, IN.position.z, 1.0f ); float4 m, mi ; float3 m2,mp; float scale; m = mul( modelViewProj, m1); m2.x = m.x/m.w; m2.y = m.y/m.w; m2.z = 1; scale = mul(m2, mul(E,m2)); mp = mul(A,m2) + sqrt(scale)*e; mi.x = m.w * (mp.x)/(mp.z); mi.y = m.w * (mp.y)/(mp.z); mi.zw = m.zw; OUT.position = mi; OUT.color0 = IN.color0; // Use the original per-vertex color specified return OUT; } ParametricWarp

36. Rendering 2D + 3D scene Concave Dome Convex Dome

37. Details I Skipped .. • Estimating camera and projector params • Internal and External params • Issue with near-planar 3D points • Finding pixels weights for blending • Non-linear optimization • Rendering • Warping and Depth buffer issues

38. Seamless Curved Display • Multi-projector low cost method • Simplified Quadric Image Transfer • Complete Parametric calib+rendering solution More info :www.raskar.com/Projector/

39. Projector Mailing List majordomo@cs.unc.edusubscribe projector Projector bibliography www.raskar.com/Projector/

40. Advantages • Parametric warp • Lower camera resolution • Tolerance for pixel localization errors • Faster calibration • Efficient well-defined warping