Remote E ngineered S uper Resolved I maging

1. Remote Engineered Super Resolved Imaging Zeev Zalevsky Faculty of Engineering, Bar-Ilan University, 52900 Ramat-Gan, Israel

2. Outline • Introduction • The “SW Adaptation” Process • Diffractive type Super Resolution • Geometrical type Super Resolution • Hearing with light • Conclusions

3. Outline • Introduction • The “SW Adaptation” Process • Diffractive type Super Resolution • Geometrical type Super Resolution • Hearing with light • Conclusions

4. Introduction-Diffraction Limitation What is Resolution? • Resolution is finest spatial feature that an imaging system can resolve. • Resolution of optical systems is restricted by diffraction (Lord Rayleigh, Abbe), by the geometry of the detector and by the noise equivalence of its pixels.

5. Introduction-Diffraction Limitation Diffraction limitation of resolution is proportional to the F number of the imaging optics.

6. Introduction- Geometrical Limitation Geometrical resolution is limited by the number of detector’s pixels and their size.

7. Introduction- Noise Equivalent Resolution Noise equivalent resolution is originated by the internal noises existing within each pixel of the detector (electronic noises, shot noises etc).

8. Outline • Introduction • The “SW Adaptation” Process • Diffractive type Super Resolution • Geometrical type Super Resolution • Hearing with light • Conclusions

9. No, if A Priori information on the object is available!!! SW Adaptation Process If not resolved, is it hopeless? Types of a priori information: • A single dimensional object • Polarization restricted information • Temporally restricted signal • Wavelength restricted signal • Object shape

10. Adapt the SW of the signal to the acceptance SW of the system SW Adaptation Process- cont. The Suggested Solution: Having a priori knowledge of the signal may lead to super resolution using an SW (space-bandwidth) adaptation process:

11. Outline • Introduction • The “SW Adaptation” Process • Diffractive type Super Resolution • Geometrical type Super Resolution • Hearing with light • Conclusions

12. Ap. Obj. Img. G1 G2 Synchron. moving gratings The structure of the rotated grating (for 2-D S.R. effect) Diff. SR- Time Multiplexing Time Multiplexing: Conversion of temporal degrees of freedom to spatial domain (diffraction) Recent improvements: • Automatic synchronization (one grating, transmitted twice) • 2-D objects, 2-D gratings • Dammann gratings

13. Diff. SR- Time Multiplexing

14. With clear aperture Without time multiplexing With time multiplexing Diff. SR- Time Multiplexing

15. Remote Diff. SR Open aperture Reconstruction Projected grating Closed aperture

16. Diff. SR- Speckle Projection Closed aperture Reconstruction Coherent Incoherent

17. Remote Diff. SR- via Background Open aperture Closed aperture Reconstruction

18. Open aperture Background Reconstruction Closed aperture Reconstruction Closed aperture sequence Remote Diff. SR- via background, cont.

19. Remote Diff. SR- from satellite Numerical simulations Experimental results

20. Remote Diff. SR- via rain/droplets Reconstruction Open aperture Closed aperture

21. Spatial DLP based SR Two possible experimental setups. FOV SR.

22. FOV improvement 3X3 Raw images (different DMD positions) Sensor size image

23. FOV improvement 3X3 Two types of algorithms (LSQR, L1) N = 8 DMDs positions L1 N=8 LSQR N=8 Original FOV = middle square only

24. Spatial DLP based SR The experimental setup. (a). (b). (c). Experimental results: (a). Image captured with open iris. (b). Image captured with semi closed iris. (c). Reconstructed super resolved image (demonstrating optical zooming of 3x).

25. Optical SAR- I Left: Schematic sketch of the proposed configuration. Right: The proposed iterative algorithm.

26. Optical SAR- II, Simulations (a). (b). (c). Numerical simulations. (a). Original image. (b). Its reconstruction. (c). The type of reconstruction that is obtained when the phase is wrongly reconstructed. The unwrapped phase of the Fourier of the original image and the reconstructed phase. Left: Original object. Middle: Original object Fourier transform. Right: Blurred image.

27. Optical SAR- III, Experimental results

28. Outline • Introduction • The “SW Adaptation” Process • Diffractive type Super Resolution • Geometrical type Super Resolution • Hearing with light • Conclusions

29. (a). (b). Geometrical SR- Intermediate plane mask (a). High resolution ref. (b). Low resolution image without SR (b). (c). (a). Reconstruction: (a). Field of view border condition. (b). High resolution mask in the intermediate image plane. (c). Low resolution mask in the intermediate image plane. Mask+sensor are shifted for the micro scanning process

30. Geometrical SR- Improved Technique (a). (b). (a) A random mask with size of . (b) Optical configuration including the binary mask. The mask has to be in an intermediate imaging position. The mask can be moved only in one direction to get different images each time.

31. Geometrical SR- Improved Technique Improvements: • The mask is the only part being shifted (instead of mask+sensor). • The SR is achieved, without any spatial loss of information. • The reconstruction has a reduced sensitivity to noise. • Although the movement of the mask is in 1-D, the obtainable SR is 2-D. • The movement of the mask does not have to be in sub pixel steps. • The recovery time is improved (the reconstruction process each pixel can be treated separately & simultaneously).

32. Geometrical SR- Simulations vs. SR factor Simulation results depicting the algorithm dependence on the super resolution factor. Image Size: 256256, Number of images taken during the process = 2×SR factor, percentage of the image covered by the random mask = 50%, Noise variance = 0.001. (a). Reference image. Low resolution images ((b), (d), (f), (h)) and their corresponding high resolution reconstructions ((c), (e), (g), (i)) for SR factor of 4X4, 8X8, 12X12 and 16x16 respectively. Dependence of the algorithm and the runtime on the super resolution factor. Number of images taken during the process = 2× SR factor, the size of the ring image is 256×256 pixels. It has been obtained by performing matrix inversions. The size of each matrix is 2×SR factor rows, and SR factor columns.

33. Geometrical SR- Experimental results Spherical Mirror Auto Collimator Auto Collimator Detector Aperture Stop Manual micrometer Relay Lens Folding Mirror Binary Mask The experimental setup

34. Geometrical SR- Experimental results Upper row: The left image: Central part of a low resolution image. The right image: Resulting reconstructed higher resolution image. Lower row: The cross sections.

35. Outline • Introduction • The “SW Adaptation” Process • Diffractive type Super Resolution • Geometrical type Super Resolution • Hearing with light • Conclusions

36. Opto-Phone: Hearing with Light • Hearing with Light:Features • The ultimate voice recognition system compatible to “hear” human speech from any point of view (even from behind). • There is no restriction on the position of the system in regards to the position of the sound source. • Capable of hearing heart beats and knowing physical conditions without physical contact for measuring.

37. Opto-Phone: Hearing with Light • Features- cont. • Works clearly in noisy surroundings and even through vacuum. • Allows separation between plurality of speakers and sounds sources. • Works through glass window. • Simple and robust system ( does not include interferometer in the detection phase).

38. Opto-Phone: Hearing with Light

39. Cell phone Back part of neck Counting…1,2,3,4,5,6 Counting…5,6,7 Face (profile) Heart beat pulse taken from a throat Counting…5,6 All recordings were done in a very noisy constriction site at distance of more than 80m. Let’s listen…from 80m

40. Results: Detection of occluded objects I (a). (b). (c). (a). Camouflaged object. (b). Camouflage without the object. (c). The object (upper left part) and the low resolution camouflaged scenery. (d). (f). (e). (d). The spectrogram of the camouflaged object with its engine turned on. (e). The spectrogram of the object with its engine turned on and without the camouflage. (f). The spectrogram of the camouflaged object without turning on its engine.

41. Results: Detection of occluded objects II (a). (b). (a). The scenario of the experiment. (b). Experimental results: upper recording is of the camouflaged subject. Lower recording is the same subject without the camouflage.

42. Outline • Introduction • The “SW Adaptation” Process • Diffractive type Super Resolution • Geometrical type Super Resolution • Hearing with light • Conclusions

43. Conclusions: • Resolution of optical system is restricted by various terms. • SW Adaptation process is a useful tool for designing super-resolution systems. • A generalization for handling more types of resolution restrictions was introduced for large variety of applications. • Examples of achieving super resolution effects were viewed. • New approach for “hearing” with light was demonstrated.