Interruptible Rendering for Balancing Complexity and Interactivity

2. Interruptible Rendering Cliff Woolley, U.Va. David Luebke, U.Va. Benjamin Watson, NWU

3. Motivation

4. Motivation • Balancing complexity with interactivity • Improving on traditional LOD approaches • Unified error metrics Interruptible Rendering

5. Balancing complexity with interactivity • The age-old detail vs. frame rate tradeoff • Is a high, constant frame rate good enough? • Maintaining the frame rate can be tricky Interruptible Rendering

6. Improving on traditional LOD • Traditional approaches seek to create a mesh that can be rendered “in time” • Selecting a mesh for rendering takes time in and of itself • The only goal is usually “maintain high, constant frame rate” Interruptible Rendering

7. Unified error metrics • Meruvia’s “IO differencing” is a good start • Combines notions of temporal and spatial error (i.e., “lateness” versus “coarseness”) • Using screen-space distances as a unifying metric, we extend IO differencing with a progressive refinement system Interruptible Rendering

8. Unified error metrics • With interruptible rendering, IO differencing can answer this key question: At what point does further refinement of the current frame become pointless? Interruptible Rendering

9. Methodology

10. Methodology • Refine a stream of continuous LODs • Monitor input frequently • Minimize visual error Interruptible Rendering

11. Refine a stream of continuous LODs • Render progressive refinements on top of each other until “out of time” • Guaranteeing containment of coarser LODs allows interruption at any time Interruptible Rendering

12. Monitor input frequently • Ideally, input would be monitored continuously; realistically, it can be checked every x ms (many times per frame) • This allows quick reaction when sudden changes in input occur Interruptible Rendering

13. Minimize visual error • When temporal error (lateness) exceeds spatial error (coarseness) for the current frame, further refinement is pointless • When total error (spatial + temporal) in the front buffer is greater than total error in the back buffer, a swap should occur • In either case, we swap buffers Interruptible Rendering

14. Minimize visual error • Comparing spatial and temporal error in this way allows a balanced approach to the detail vs. frame rate question • High frame rate does no good if the input is changing slowly Interruptible Rendering

15. Implementation

16. Implementation • Preprocess – stream generation • Runtime – two interacting state machines Interruptible Rendering

17. Stream generation • Our preliminary system uses a simple octree-based hierarchy of splats • Any node of the octree that is fully contained in the original mesh is rendered as a splat • Rendering will occur breadth-first, so we pre-order the stream of splats breadth-first Interruptible Rendering

18. Stream generation Interruptible Rendering

19. Stream generation • Streams of many types possible • Splats • Progressive hulls (fig. from Sander et al. 2000) • Interactive ray tracing (future work) Interruptible Rendering

20. Runtime state machines • Error monitoring state machine • Progressive refinement state machine Interruptible Rendering

21. Error monitoring • Calculate spatial error s : find screen-space size of coarsest geometry still being displayed • Calculate temporal error t: project the bounding box of the object and find the max screen-space distance any corner has moved • Compare s and t to decide whether to continue refinement Interruptible Rendering

22. Error monitoring • If current frame is still worth refining, check its total error against total error in front buffer • If total error is less than in the front buffer, go ahead and swap buffers • … and continue refining in the front buffer Interruptible Rendering

23. time start Start refining a new image toward most current input in the back buffer. clear back buffer Iback = Icurrent clear front buffer Ifront = Icurrent Refine the current image in the back buffer. improve iback tback = Icurrent - Iback sback = Iback – iback Refine the current image in the front buffer. improve ifront tfront = Icurrent - Ifront sfront = Ifront – ifront tback > sback? no yes tfront = Icurrent – Ifront efront = sfront + tfront eback = sback + tback It is pointless to continue refining. swap buffers Ifront = Iback sfront = sback tfront > sfront? efront >= eback? no yes no yes Back buffer now closer to Icurrent than front is. swap buffers Ifront = Iback Rendering to Front Buffer Rendering to Back Buffer

24. time start Clear back buffer Clear front buffer Refine back buffer Refine front buffer tback > sback? no yes Compute total error Swap buffers tfront > sfront? efront >= eback? no yes no yes Swap buffers Rendering to Front Buffer Rendering to Back Buffer

25. Progressive refinement • Keeps track of what has been rendered so far in the current frame • Progresses through the stream a “chunk” at a time • Hands the next set of refinements off to the GPU while CPU does the next set of error checks • Note: CPU/GPU synchronization required Interruptible Rendering

26. Demo

27. Benefits & Implications

28. Benefits & Implications • Ensures low latency in the face of unpredictable input • Displays as much detail as the hardware can handle in the time allowed by those changes • Interesting “one-and-a-half buffered” approach renders sometimes to the back buffer and sometimes to the front buffer Interruptible Rendering

29. Questions?