function).

1. http://cameraculture.info MITMedia Lab |Camera Culture | University ofWaikato Coded Time of Flight Cameras: Sparse Deconvolutionto Resolve Multipath Interference AchutaKadambi,Refael Whyte,AyushBhandari, Lee Streeter,Christopher Barsi, Adrian Dorrington,RameshRaskarin ACM Transactions on Graphics 2013 (SIGGRAPH Asia) Howcan we create new time of flight camerasthat canrange translucent objects, look through diffusers, resolve multipath artifacts, and create time profile movies? Conventional Time ofFlight Cameras Time of FlightCameras(ToF) utilize the Amplitude ModulatedContinuous Wave(AMCW) principle to obtain fast, real-time rangemapsof a scene.Thesetypesof camerasare increasingly popular – the new Kinect2 is a ToFcamera–and applicationareas includegesturerecognition,robotic navigation, etc. Hardware Prototype.AnFPGAis usedfor readoutandcontrollingthe modulationfrequency. The FPGA isinterfaced witha PMD sensor whichallowsfor external controlof the modulation signal.Finally, laser diode illuminationissyncedtothe illumination controlsignal fromthe FPGA. Comparing Different Codes Application 1: Light SweepImaging Application 2: Looking Through Diffuser A time of flight camerasendsanoptical codeand measures the shift in the codeby sampling the cross-correlationfunction. Conventional ToFCamerasuse Square or Sinusoidal Codes. ForwardModel: Convolution with the Environment The cross-correlationfunction isconvolved withthe environmentresponse (here, noted as a delta Here light is visualized sweeping across a checkered wallat labelled time slots. Colorsrepresent different time slots. Since weknow the size ofthe checkers,we canestimatetime resolution. (left): The Measured AmplitudeImage.(right): The Component Amplitude Application 3: Ranging of Translucent Objects Application 4: Correcting Multi-Path Artifacts Aconventional time offlight camera measuresincorrect phase depthsofa scene with edges (red). Our correction is able toobtain the correct depths (green). We comparedifferent codes.The codessent to the FPGA are inthe blue column. Goodcodesfordeconvolutionhave a broadbandspectrum (green).The autocorrelation of the blue codesis inred.Finally, the measured autocorrelation function(gold)is the low pass versionofthe red curves. The low pass operator representsthe smoothing ofthe correlation waveform dueto therise/falltime ofthe electronics. function). y For asimple object, suchas awall, the environment responsecanbe modelledas a single dirac. The codeweuse is the m-sequence, whichhas strong autocorrelation properties. The codethat conventional cameras use isthe square code,which approximates a sinusoid when smoothed. A wall further away would havea shifted dirac. y (left) rangemaptaken by a conventional time offlight camera.(middle) We can“refocus”on the foregrounddepth, or(right) the background depth. Comparing Different Sparse Programs ForwardModel: Convolutionwith Multi-Path Environment Consider ascene with mixedpixels, e.g., a translucentsheetin front of a wall. Multi-Pathscenarios occurat anyedge. In equation form,we express the resulting convolution: For thissystem, true sparsityis definedas: Theresulting environmentresponse is no longer 1-sparse. In this figure it is modelled as 2-sparse. y y To solve thisproblemwe consider greedyapproaches,such as OrthogonalMatchingPursuit (OMP).We maketwo modificationsto the classicOMP thataretailored for our problem: Whenconvolvedwitha sinusoid (toprow), the resulting measurement, y, is another sinusoid, which results ina unicity problem. Modification 1: Nonnegativity. Consider only positiveprojections when searching for the next atom. Whenupdating the residual, usea solver to impose positivity on the coefficients (weuse CVX). Perhaps the solution lies in creatinga custom correlationfunction (bottomrow). y We comparedifferent programs fordeconvolvingthe measurement(upper-left) into the constituentdiracs. A naïve pseudo-inverse resultsina poorsolution. Tikhonov regularization isbetter, butlacks thesparsity in theoriginal signal. The LASSO problemis decent, but has manyspurious entries. Finally, the modified orthogonal matching pursuit approach provides a faithful reconstruction(bottom-right). Wecan express this ina Linear Algebra Framework: EnvironmentProfiles are oftensparse. At left is aone-sparse environmentfunction,atmiddle isthe environment responseresulting froma transparency. Modification 2: ProximityConstraints. RelatedWork: Velten,Andreas, etal. "Femto-photography:Capturing andvisualizingthe propagationoflight." ACMTrans. Graph32 (2013). Heide, Felix, etal."Low-budgetTransient Imagingusing Photonic MixerDevices."Technical Paperto appearat SIGGRAPH2013(2013). Raskar, Ramesh,Amit Agrawal,and JackTumblin."Coded exposurephotography: motiondeblurringusingfluttered shutter." ACM Transactionson Graphics(TOG). Vol.25.No.3. ACM,2006. Because wehaveexpressed this asa linear inverse problem, wehave accesstotechniquestosolve sparse linear inverse problems. Atrightis anon-sparse environment profile.Torecoverthese, a Tikhonov Deconvolution is used .