LightSync : Unsynchronized Visual Communication over Screen-Camera Links

1. LightSync: Unsynchronized Visual Communication over Screen-Camera Links Wenjun Hu Hao Gu Qifan Pu

2. 2D barcodes everywhere

3. (“Linking” to online information) http://www.windowsphone.com/zh-cn

4. A video of barcodes instead • “Transmitting” information (vs linking) • E.g., information on Gibraltarian attractions • E.g., movie show times for nearby theaters Receiver Transmitter

5. Visual communications on phones • No need for additional connectivity • Free, directional, lightweight links • “Optical” software-defined radio • Run as apps on commodity devices

6. Many challenges • Frame synchronization • Poor image quality • Limited computation power on phones • Aesthetics…

7. In the rest of the talk… • Frame synchronization challenges • LightSync design • Implementation and evaluation

8. Frame synchronization challenges

9. Frame rate mismatch Screen side (TX) Camera side (RX) 15 – 30 fps in practice Variable Lighting conditions Device capability API/system factors • 30fps video playback • Fairly stable High rate, steady Low rate, unsteady

10. Received frame patterns Original frame

11. Received frame patterns Mixing pattern varies by line Original frame Single frame 2-frame mix

12. Received frame patterns Due to Rolling Shutter Mixing pattern (& #components) varies by line CMOS camera sensor scans one line (of pixels) at a time 2-frame mix

13. Received frame patterns Mixing pattern varies by line Original frame Single frame 2-frame mix Consecutive frames indicating a missing frame 3-frame mix

14. Display frame rate Occasional missed frames 4-frame mixes 2C C – Frame capture rate Occasional missed frames 3-frame mix 3-frame mix C C/2

15. Synchronized communication Display frame rate 2C C – Frame capture rate (of worst camera) Cannot be used Under-utilized and inefficient! C The only decodable pattern Framesynchronized Carries information But undecodable and dropped C/2

16. Unsynchronized communication Display frame rate (fps) 30 Works with any display and capture rates Higher camera capture rate, better performance 0 15 Supported frame capture rate (fps) 30

17. Requirements • Need to be able to decode mixed frames • In-frame color tracking per line • Need to guard against lost frames/lines • Inter-frame erasure coding • Essentially a temporal code across frames • Works with any (monochrome) per-frame code

18. Our solution: LightSync • Need to be able to decode mixed frames • In-frame grayscale tracking per line • Need to guard against lost frames/lines • Inter-frame erasure coding • Essentially a temporal code across frames • Works with any (monochrome) per-frame code

19. LightSync Demo

20. LightSync • Need to be able to decode mixed frames • In-frame grayscale tracking per line • Need to guard against lost frames/lines • Inter-frame erasure coding

21. Per-line tracking: Mixing patterns • 3 frames mixing → 3 blocks mixing • Bl or w per block → 8 mixing possibilities Frame 1 Frame 2 Frame 3 → ? ? ? RX TX

22. Per-line tracking: Mixing patterns Frame 1 TX Frame 2 Frame 3 Line 1 RX Line 2 Line 3

23. Per-line tracking: Mixing patterns • Trivial cases Frame 1 TX Frame 2 Frame 3 Line 1 RX Line 2 Line 3

24. Per-line tracking: Mixing patterns • Easy cases Frame 1 TX Frame 2 Frame 3 Line 1 RX Line 2 Line 3

25. Per-line tracking: Mixing patterns • Tricky cases, to be tracked per line Frame 1 TX Frame 2 Frame 3 Line 1 RX Line 2 Line 3

26. Per-line tracking: Reference blocks Corner Corner Code area Corner Corner Tracking bars Scanning direction Tracking bar orientation

27. Per-line tracking: Reference blocks Tracking bar 1 Tracking bar 2 Tracking bar 3 Tracking bar 4 4 reference blocks per line Txframe1 Tx frame 2 Tx frame 3 Tx frame 4 Tx frame 5 (repeats)

28. Per-line tracking: Reference blocks Tracking bar 1 Tracking bar 2 Tracking bar 3 Tracking bar 4 Txframe1 Tx frame 2 Tx frame 3 Tx frame 4 Tx frame 5 (repeats)

29. Per-line tracking: Reference blocks Tracking bar 1 Tracking bar 2 Tracking bar 3 Tracking bar 4 Txframe1 Tx frame 2 Reference colors of mixes Tx frame 3 Tx frame 4 Decoding by comparison Tx frame 5 (repeats)

30. LightSync • Need to be able to decode mixed frames • In-frame grayscale tracking per line • Need to guard against lost frames/lines • Inter-frame erasure coding

31. Inter-frame coding • Simple pairwise XOR Group of 3 original frames Coded frames

32. Inter-frame coding • Looping display on screen … All original frames … Coded frames Camera stops frame capture after message decoded

33. Performance

34. Implementation Inter-frame intervals Current frame capture rate

35. Performance • Goal recap: Works with any display/capture rate combinations • Metric: Effective frame capture rate • Actual throughput scales with per-frame capacity

36. Schematic performance For a 20fps phone camera

37. Schematic performance For a 20fps phone camera Ideal Ideal Peaks at the camera rate Level at higher display rates

38. Schematic performance For a 20fps phone camera Ideal Peaks at ~ half the camera rate Drops to 0 at higher display rates Conventional

39. Schematic performance For a 20fps phone camera Ideal LightSync Peaks at the camera rate Slight drops at higher display rates Conventional

40. LightSync performance Nokia Lumia 920 (~30 fps)

41. LightSync performance Nokia Lumia 920 (~30 fps) HTC One X (~20 fps)

42. LightSync performance Nokia Lumia 920 (~30 fps) HTC One X (~20 fps) Works with any display and capture rates Higher camera capture rate, better performance Huawei Ascend C8812E (~15 fps)

43. Conclusion and outlook • LightSync: Unsynchronized visual communication • Primitive for similar scenarios • Lots of potential in the space of visual communication

44. Thank you! wenjun@microsoft.com

45. Related work • Temporal barcode design (inter-frame) • Unsynchronized 4D Barcodes • Basically slows down display, only for TX up to 10fps • Spatial barcode design (per-frame layout) • COBRA: Optimized for phone-to-phone • PixNet: High-end camera, computationally intensive, offline processing • Other work on visible light communications