School of Computer Science and Software Engineering

1. Monash University School of Computer Science and Software Engineering Design Issues in Human Visual Perception Experiments on Region Warping Yang-Wai Chow Ronald Pose Matthew Regan

2. Overview • Background • Address Recalculation Pipeline • Priority Rendering • Large object segmentation • Scene tearing artefacts • Region Warping • Related Work • Image/Video compression • Level-of-Detail (LOD) management • Visual attention models • Design Issues • Conclusions

3. Background • Address Recalculation Pipeline • Priority Rendering • Large object segmentation • Scene tearing artefacts • Region Warping

4. The Address Recalculation Pipeline (ARP) Head Tracking Image Creation Buffer Swap Image Valid Average latency to head rotations WITHOUT pipeline Head Tracking Image Valid Average latency to head rotations WITH pipeline The Address Recalculation Pipeline (ARP) was designed to reduce the end-to-end latency due to user head rotations for immersive Head Mounted Display (HMD) virtual reality systems • Latency is a major factor that plagues the designing of immersive Head Mounted Display (HMD) virtual reality systems • End-to-end latency is defined as the time between a user’s actions and when those actions are reflected by the display

5. Priority Rendering Priority Rendering allows different section of the scene that surrounds the user’s head to be rendered onto separate display memories and therefore can updated at different update rates • Different section of the scene can be rendered onto separate display memories before being combined to form an image of the whole scene

6. Large object segmentation Segmentation of large virtual world object in conjunction with Priority Rendering can further reduce the overall rendering load • Fractal terrain example – A fractal terrain typically consists of thousands of polygons. If the terrain were to be segmented for priority rendering, different sections of the fractal terrain could be updated at different update rates

7. Scene tearing artefacts However the implementation of object segmentation in conjunction with priority rendering introduces a potential scene tearing problem • Scene tearing artefacts will completely destroy the illusion of reality, and therefore have to be addressed before object segmentation can be used effectively with priority rendering

8. Region Warping Region Warping was designed to hide the scene tearing artefacts resulting from object segmentation with Priority Rendering • Region Warping essentially involves the perturbation of object vertices in order to hide the tearing artefacts

9. Scene tearing An example showing the scene tearing effect

10. Region Warping results The exact same frame, this time with Region Warping

11. Initial region warping experiments and analysis Initial Region Warping experiments and analysis do not portray the human perception or response to the warping • Some initial experiments on region warping have been conducted (Please refer to our paper entitled “Region Warping in a Virtual Reality System with Priority Rendering” – IADIS Applied Computing 2005) • However the analysis that was performed were mathematical analysis of the distortions and did not portray the human visual perception to region warping • Therefore further experimentation involving human subjects are necessary

12. Related Work • Image/Video compression • Level-of-Detail (LOD) management • Visual attention models

13. Related work A number of other computer graphics techniques have also been developed that take advantage of the limitations in the psychophysical aspects of the human visual perception • The human visual system has a finite perception to detail • A variety of computer graphics techniques attempt to reduce the computational effort and time required in the rendering of scenes or generating of images without greatly affecting a user’s perception of the displayed image quality • Examples of human visual perception exploits can be found in lossy image/video compression techniques • JPEG and MPEG coding are lossy compression techniques that attempt to decrease the amount of data in the representation of an image/video

14. Level of Detail (LOD) Level of Detail (LOD) management used in computer graphics attempts to decrease the amount of detail in a scene without the user noticing the difference • Level of Detail (LOD) management have been used by degrading the peripheral detail in a HMD virtual reality system’s display, with no adverse effects to the user task performance • LOD methods also include the use of object meshes with varying degrees of complexity • A number of criteria for switching between varying LODs have been proposed and implemented • Simpler object meshes can be switched into the display without the user noticing any alterations in the scene

15. Visual attention models Others have experimented with visual attention models to selectively concentrate computational effort on important sections of a scene • A user will typically focus on prominent or conspicuous objects in a scene and will pay less attention to the details contained in the rest of the environment • This is a fundamental feature of the human visual system known as inattention blindness • Some researchers have conducted visual perception experiments based on visual attention models in order to predict where the user will be looking in a scene and focus rendering effort on those sections of the scene

16. Design Issues • Stereoscopy versus Monoscopy • Distance from the user • Head tracked displays • Scene complexity • Virtual environment characteristics • Scene familiarity • Ability to interact with surrounding environment

17. Stereoscopy versus monoscopy As part of the human visual system, stereo vision is one of the main factors that contribute to a user’s perception of realism within a virtual world • The human visual system is based on stereo vision • A human perceives information such as depth based on the disparity between left and right eyes • It is therefore conceivable that a user might perceive differently the effects of the distortions caused by region warping in a virtual world viewed through a stereoscopic display or a monoscopic display

18. Distance from the user A human perceives less detail in objects that are located further away compared to closer objects • Objects that are further from the users will appear to be smaller from the user’s viewpoint • Human binocular disparity/stereo perception fails after a distance of more than 30 meters • It is therefore possible that distortions that are concentrated at a distance away from the user will be less noticeable

19. Head tracked displays Upon user head movements or rotations, based on feedback from the motoric senses the human brain will expect the scene to change in certain ways • The human eye is less sensitive to details that move rapidly over the retina • Perception to distortion might vary between head tracked virtual reality systems and non-head tracked systems

20. Scene complexity The human visual system depends on the retina to cope with the wealth of information contained within the scene • The user visual attention will typically be focus or drawn to certain interesting areas of the scene • High levels of detail contained within the scene will distract and possibly overwhelm the user’s perception

21. Virtual environment characteristics The makeup of the virtual world might also potentially affect a user’s perception to the region warping distortions • Distortions will appear less obvious on scenes containing random nature objects compared to scenes containing rigid structured architectures • The reason for this is because highly structured scenes contain straight lines can might noticeably bend as a result of the warping

22. Scene familiarity The human brain might unconsciously correct perceived errors when presented with a familiar scene • The human visual system is governed by real world experiences, as the brain interprets images presented to the retina based on these experiences • The human brain might overlook errors contained in familiar scene • Alternatively the converse might apply, in that errors might appear more prominent when presented with a familiar scene

23. Ability to interact with surrounding environment A user’s focus of attention will be divided with allowed to interact with the surrounding environment • Interactability with the surrounding environment adds to the user’s feeling or sense of presence in the virtual world • When a user is immersed in a virtual world, his/her mindset and focus of attention will be drawn towards trying to manipulate objects or to navigate through the virtual world

24. Conclusions • Region warping attempts to take advantage of the limitations in the human visual perception, to reduce the overall rendering load of a ARP virtual reality system with priority rendering • However while region warping appears to works well on computer monitors and perform well by analytical measures, the real test can only be with real human perception • Therefore before this technique can be used effectively, it is important to characterize and understand the level of distortions that might be perceivable to the user • This paper has outlined our work and future approaches

25. Questions or Suggestions?