Interactive BRDF Estimation for Mixed-Reality Applications

1. Interactive BRDF Estimation for Mixed-Reality Applications Martin Knecht, Georg Tanzmeister, Christoph Traxler, Michael Wimmer Institute of Computer Graphics and Algorithms Vienna University of Technology

2. Motivation • Goal of our mixed reality framework • Light interaction between real and virtual objects  Materials of real objects must be known Martin Knecht

3. Problem Statement • Material estimation should not need any preprocessing  Use Kinect sensor and fish-eye lense camera for data acquisition • Should run at interactive framerates  Use GPU wherever possible • Should estimate Phong parameters Used in mixed reality framework Martin Knecht

4. Estimation Pipeline • Similar to pipeline of Zheng et al. 2009 Martin Knecht

5. Input Data for Estimation Martin Knecht

6. Highlight Removal Martin Knecht

7. Highlight Removal Martin Knecht

8. Diffuse Reflectance Estimation Martin Knecht

9. Diffuse Reflectance Estimation Inverse shading: Martin Knecht

10. Clustering Martin Knecht

11. Clustering 1/5 • Assumption: similar color  same material • Same material  same specular parameters • Clustering executed on the diffuse estimation • Novel hybrid CPU/GPU K-Means 1) Initialize cluster centers 2) Assign pixel to nearest cluster center 3) Calculate new cluster centers 4) Repeat steps 2 & 3 Martin Knecht

12. Clustering 2/5 • 1) Initialize cluster centers • Random cluster centers • Exploit temporal coherence • Reuse of cluster centers of previous frame Martin Knecht

13. Clustering 3/5 • 2) Assign pixel to nearest cluster center RGBC1 RGBC2 ... Cluster Shader ... Cluster 1 Cluster 2 Cluster 6 Bitmask 1 Bitmask 2 ... Martin Knecht

14. Clustering 4/5 • 3) Calculate new cluster centers • 1x1 Mipmap is the average over all pixel • New cluster center: TRGBD 1x1 RGBD T* 1x1 Bitmask Martin Knecht

15. Clustering 5/5 • 4) Repeat steps 2 & 3 • Repeat until no pixel changes cluster  Standard stopping criteria too conservative • Max. 20 iterations • Check variance change of distances Martin Knecht

16. Clustering 5/5 Martin Knecht

17. Specular Reflectance Estimation Martin Knecht

18. Specular Reflectance Estimation • Done on a per cluster basis  same material • CPU based nonlinear function solver • Variables: • Specular parameter • Light positions • Evaluation of objective function done on GPU • Similar mipmap method used as for clustering Martin Knecht

19. Specular Reflectance Estimation Martin Knecht

20. Results - Estimation + Diffuse component Specular component Phong shaded image Martin Knecht

21. Results – Timings • BRDF estimation runs at ~2.8 fps • Two tasks with major impact  K-Means clustering  Specular estimation < 0.5 % ~ 11 % ~ 88,5 % Martin Knecht

22. Differential Instant Radiosity Martin Knecht

23. Results – Mixed Reality Integration Martin Knecht

24. Limitations • Kinect sensor does not work everywhere • Bright objects are discarded from estimation • Shadows are not considered • No estimation of optimal amount of clusters • No integration of data over time • Simplifications lower quality of estimation Martin Knecht

25. Conclusion & Future work • BRDF estimation without any preprocessing • Hybrid CPU/GPU K-Means implementation • Runs at interactive framerates • Future work • Improve speed  specular estimation • Improve quality BRDF estimation • Exploit temporal coherence more often Martin Knecht

26. Thank you for your attention! Supported by grand from the FFG-Austrian Research Promotion Agency under the Program “FIT-IT Visual Computing” Project Nr.: 820916