Virtual Environments : System Architectures

Virtual Environments: System Architectures Anthony Steed Simon Julier Department of Computer Science University College London http://www.cs.ucl.ac.uk/teaching/VE

Overview • Problem Statement & Requirements • Data Representations (Contents) • Execution Models (Dynamics) • Latency

User Synthetic Environment User Environment Mediated Medium Real Environment Reminder - VE is an Immersive,Mediated Communication Medium Interface Devices

Ellis’s Content, Geometry, Dynamics Model • Contents is data • Dynamics is code or rules to change content • Today we will look at common data and code representations

Sources and Sinks of Data • Input “Data” • Data from trackers (e.g. 3D position plus 3D rotation for head and hand = 12D data) • Data from input devices (e.g. 2D joystick, buttons) • Possibly audio, haptic, physiological input • Output “Data” • Displays systems, audio, video

Displays Have Different Requirements • Video (N copies – for stereo and multiple screens) • Maintain copy of visual state • Render as fast as possible (~60Hz) • Synchronise with other renders • Audio • Maintain copy of audio state • Render without glitches (requires fast interrupt) • Haptics • Maintain copy of haptic dataRender as fast as possible (~1000Hz)

“Under the Hood” of the Code • We will find code modules for many different tasks: • Managing data and assets • Reading devices • Audio rendering • Video rendering • User input • Networking • Complex calculations such as physics (see seminar tomorrow!)

Graphs • A graph consists of vertices and edges • Vertices define the “state” information • Edges define “relationships” • Scene-graphs are directed and acyclic Directed acyclic graph Arbitrary graph Directed graph

In a scene-graph, vertices are often called nodes Store state information Can include arbitrary property information All graphs have a root node which defines the base of the tree All other nodes divided into two types: Group nodes Leaf Nodes Root node Group nodes Leaf nodes Scene-graphs

Group Nodes • Group nodes have multiple nodes as children • Child nodes can be other group nodes or leaf nodes • Applies common state information to multiple objects • State information propagates down the graph • Examples include: • Transformations • Switch nodes • Effects • Bump mapping, scribing, specular highlights • (In recent times) Shader programs

Examples (OpenSceneGraph) Anisotropic Lighting Bumpmapping Cartoon Scribing

Leaf Nodes • Leaf nodes cannot have children • State information relates to the appearance of specific objects • Examples include: • Geometry • Image based rendering • Billboards • Impostors

Examples (OpenSceneGraph) Impostors Billboards

Dynamics • These are the rules of interaction between the contents • These can be: • Differential equations of Newtonian dynamics to describe kinematic and dynamic relationships • Grammatical rules for pattern-matched triggered actions • Many different ways of doing this from imposing numerical approximations to Newtonian physics, through to plain old C++ / Java / XVR coding

Dynamics Representation Model • Separate from the contents, how do we want to represent our dynamics? • Its leads to quite critical computer science questions about the separation between code and data • From a scene-authors point of view two basic models • Standalone process model • Within-scene-graph model

Implementing Dynamics as Standalone Processes • Dynamics implemented as separate processes / threads • Can change state of the graph in arbitrary ways • Change values of nodes • Add / remove nodes Dynamics

Problems of Standalone • Treats scene-graph as a black-box, makes initial scene-creation very easy (load nodes in to scene-graph, changes a few fields on nodes) • Composing scenes together, or instantiating existing objects is very hard • End up with quite complicated code • The code probably has variables in local scope which “should” be in the scene-graph • Couples together rendering and animation to the same frame rate

Implementing Dynamics Within the Scene-Graph • Fairly “autonomous” dynamics can be achieved by embedding dynamics within the scenegraph • Animations are group nodes which apply state changes to their children • Examples include: • Animation paths • Particle systems Animation Node Animated nodes

Example Animation and Particle System

Problems with Within-Scene-Graph Model • Difficult to get all the code in there • e.g Difficult to write code that interfaces to other applications • Some valiant attempts such as VRML97/X3D • Does provide for easy composition of scenes • Different parts of the scene do need an explicit way of talking to each other • In VRML/X3D there are routes • In many other systems there is an event system • Many systems provide a scripting language support • Python and Lua are very common

Execution Model Ties Everything Together • So far we we’ve talked about a disparate set of systems: • A master environment • Separate representations for different output modes • User interfaces for controlling the environment • The execution model “glues” all these parts together • Closely related to distributed systems as well

Execution Models Tying Things Together • Example: • The position on an object is changed • The update needs to be reflected in: • The master database • The different scenegraphs • Over the network (if connected) • How can all of this be coordinated? • Two main models: • Kernel model • Actor model

Simple Kernel Model • Treats a VR application like a traditional graphical application: • while(true) • { • read_trackers(); • set_body_position();//Changes scene graph • do_animation(); //Changes scene graph • render_left_eye(); • render_right_eye(); • render_sound(); • poll_trackers(); • }

do_animation • After the user’s position is set by the read_trackers and set_body_position (see later in course) … • … do_animation manipulates the scene • As we have noted, is responsible for: • Animations, physics, interaction • This is where the standalone or all the within-scene-graph dynamics are updated

Problems with Simple Kernel Model • Everything happens in serial order • Rendering only happens at a fixed rate, so if part of the animation slows down, the rendering slows down (very noticeable in many video games!) • What if there are different output requirements such as haptics (1000Hz)? • What if the input rates are much higher (e.g. 200Hz)

Modified Kernel Model (1) • Has a fast loop which calls different functions at different frame rates: • while(true) • { • fast_function(); • if (elapsed>30ms) slow_function(); • } • fast_function() • {read_trackers(), do_haptics();} • slow_function() • {render_left_eye();render_right_eye(); • render_sound();}

Modified Kernel Model (2) • Uses a simple main loop, for the video rendering, but “background” threads for reading devices • Main thread is very much like the Simple Kernel Model, but the function read_trackers and poll_trackers are now in a separate thread, and the main thread just copies information • This is a good match to how operating systems actually schedule non-blocking input and output

Pros / Cons of the Kernel Model • Advantages: • Simple to understand • Application programmer keeps their own data structures within the loop • Disadvantages: • Implementation needs care because of different update rates • Usually requires some awareness of parallel programming issues • Lots of complexity ends up in the do_animation() method

Actor Model • Virtual environment is realised by a set of collaborating asynchronous processes (actors) • Actors send messages to one another • Processes share a common database • Database is a central repository of the scene graph Tracking Speech Database Collision Audio Application Video2 Video1

Setting Object State Using in the Actor Model • Setting the object state is often achieved using the subject-observer design pattern • The object in the database is the subject • Different renderers / networking systems are the observers • When the subject’s state is updated, the observers are automatically notified

Pros / Cons of the Actor Model • Advantages: • Application program does not care about distribution / what rendering systems used • Update rates are locally very good • Scales well to multiple cores • Disadvantages: • Difficult to code efficiently • Difficult to query between code modules in different actors, needs very clear separation of responsibilities • End to end latency difficult to control (see next section)

Overview • Problem Statement & Requirements • Data Representations • Execution Models • Latency

End-to-End Latency • Total time from head movement to scene movement Mine, Mark (1993). Characterization of End-to-End Delays in Head-Mounted Display Systems, UNC Chapel Hill Computer Science Technical Report TR93-001.

End-to-end latency

Problems • Single process • For the main CPU, tracker processing is actually mostly just “waiting”, could schedule something else IF and only IF (IFF) we know how long it will take • Previously we did poll and read separately • If there are multiple video streams, we could do them in parallel IFF rendering changes no state in the scene graph (depending on your scene graph this isn’t a good bet!)

Example Timing Problem Ideal case Animate Render Animate Render T0 = tracker poll T1 = video output Worst case Animate Render Animate Render T1 = video output T0 = tracker poll

Summary • Representing the environment is difficult • The representation has to be rich enough to capture the contents, geometry and dynamics • Each display mode requires its own form of the environment to optimise the display • Want to make content as rich as possible to support dynamic models • Otherwise behaviour is expressed only in code. • At run-time there are logically concurrent processes (rendering, collision, audio etc…) • Execution models need to reflect this concurrency

Virtual Environments : System Architectures