Assisting Visualization System

1. Assisting Visualization System ZhonghuaQu and OvidiuDaescu December24, 2009 University of Texas at Dallas

2. Outline • Introduction • Assisting Visualization System • Implementation • Examples • Conclusion

3. Outline • Introduction • Visualization and Application • Implementing Visualization • Integration Challenges • Our Goal • Assisting Visualization System • The Implementation • Experiment Result • Conclusion

4. Introduction: Visualization and Applications • Computer Aided Design • IKEA Home Planner

5. Introduction: Visualization and Applications • Computer Games • First Person Shooting Games

6. Introduction: Implementing Visualization • Three layers of visualization programs • Program Model • Visualization Layer • Graphics Engine

7. Introduction: Implementing Visualization • Program Model • Logic of Programs For example, in car simulation: when a driver turns the wheel, the orientation of the car should change correspondingly. • Numerical Data For example, the width and height of a piece of furniture, or the coordinate of a vehicle. They are abstract and only different by the definition given.

8. Introduction: Implementing Visualization • Visualization and Graphics Engine • 3D Graphics Rendering Includes the detailed implementation of how the object inside the program model should be rendered on the screen. For example, achair objectis rendered based on its size, type, material etc. • Graphics Engine Implementation of rendering algorithms. Provides more abstract control over the object rendered.

9. Introduction:Concerns about Visualization Layer • Steep Learning Curve To use the graphics engine for visualization requires significant 3D graphics knowledge including matrix transformation, lighting etc. Graphics Engine Visualization Code Matrix Transformation Lighting Model Texture Format …. ….

10. Introduction: Concerns about Visualization Layer • Not Flexible Once the visualization layer is built the visualization schema cannot be changed without modifying the code. Visualization Layer Visualization Layer Program

11. Introduction: Concerns about Visualization Layer • Not reusable Changing the underlying program forces changes in the definition of the visualization. This implies rewriting most of the code. Visualization Layer Program Program

12. Introduction: Our Goal • Provide a visualization system • Doesn’t require Graphics Knowledge • Flexibleto change the visualization schema • Reusable between applications

13. Outline • Introduction • Assisting Visualization System • Overview • Visual Component • Binding with Program • Implementation • Examples • Conclusion

14. Assisting Visualization System: Overview • Achieve Visualization without actually programming graphics • Flexible and easy to adjust the visualization schema • Highly reusable visualization layer

15. Assisting Visualization System: Overview • Achieve Visualization Without Programming • Usean approach called Binding • Treat visualizations as components • No visualizationCodein program • Bind visualization at runtime • Binding Defined Through XML

16. Assisting Visualization System: Overview • Achieve Visualization without actually programming graphics • Flexible and easy to adjust the visualization schema • Highly reusable visualization layer

17. Assisting Visualization System: Overview • No hard coded visualization layer • Define Visualization Through XML • No source code modification required • Doesn’t need recompilation • Easy to check the visualization definition

18. Assisting Visualization System: Overview • Achieve Visualization without actually programming graphics • Flexible and easy to adjust the visualization schema • Highly reusable visualization layer

19. Assisting Visualization System: Overview • Reuse visualization acrossprograms • Reusable Visual Component • Integrated through XMLbinding • No data structure dependency • Reusable through changing the binding

20. Assisting Visualization System: Overview • Achieve Visualization without actually programming graphics • Flexible and easy to adjust the visualization schema • Highly reusable visualization code

21. Assisting Visualization System: Visual Component • Aggregation of Visualization Code • Interacts with Graphics Engine • Nottiedto any application • Reusable across applications • Ready to use component developed by thirdparty

22. Assisting Visualization System: Binding with Program • What is the basis for binding? • Object Oriented Programming Language • One visual component perclass Each class can bind to one visual component. During the execution of the program, this class will be visualized by the visual component. • One visualized entity per object instance Whenever a new instance of the class is created, an instance of the visual component which is tied to the program instance is created and rendered on the screen.

23. Outline • Introduction • Assisting Visualization System • Implementation • Overview • Program Loader • Code Execution Monitor • Rendering System • Binding Definition Format • Examples • Conclusion

24. Implementation: Overview • Implementation Language: Java • Cross Platform • Applets run in browser on Internet • Low development cost • Third party graphics engine: JmonkeyEngine • Written in Java • UsingOpenGLfor rendering • Opensourceproject • Used in many commercial software

25. Implementation: Overview • Major Components • Program Loader Loadsand modifiesthe target java program to integrate itwith the visualization framework. • Code Execution Monitor • Monitors the execution of the target program and realizesthe visualization binding. • Rendering System • Managesthe life cycle of visualcomponentinstances that tie with underlying target program. • Rendersthe visual components using Jmonkey Graphics Engine.

26. Implementation: Program Loader • Customized Java ClassLoader • Loads class binary from file into memory for Java virtual machine • Codeinjectionat load time • Inject binary code at runtime before the class is used by Java virtual machine • Interact with CodeExecutionMonitor • Register target class information and call handler function inside execution monitor

27. Implementation: Code Execution Monitor • Monitors the target program execution • New instancecreation Call Render System to create visual component instance and tie to the underlying object • Object garbagecollectionRemove the object entry from the monitor • Object valuechanged Update the binding property in its tied visual component correspondingly

28. Implementation: Rendering System • Visual Component Management • Load visual component Search for visual component java class file for the visual component • Create, remove visual component Manage the visual component based on outside system call • Render visual components Use Graphics Engine to finally render the visual component

29. Implementation: Rendering System • Jmonkey Graphics Engine • Functions • Create User Interface Create visualization window in which all visual components are rendered • Model Loading Load 3D models from files on disk and store in memory as an object • Scene graph Management Object oriented way of accessing the 3D objects in the rendered environment

30. Implementation: Overview • Graphics Engine: Jmonkey Engine • Rendering Output

31. Implementation: Overview • Graphics Engine: JmonkeyEngine • Rendering Output

32. Implementation: Binding Definition File • XML Format • Simplicity, easily accessible to developers • Open format, many tools available • Binding XMLSchema • Define the format of binding xml • Document for people writing the binding XML • Automated parsing inside visualization system

33. Implementation: Binding Definition File • XML Parsing • Apache XMLBeans library Open source XML parsing library under Apache foundation. • Automatic java parsing code generation base on Schema File Generate source code for parser based on the schema file. Visit the content of the XML file using an object oriented approach.

34. Outline • Introduction • Binding Approach • Assisting Visualization System • Implementation • Examples • QuickSort • Growing Tree • Conclusion

35. Experimental Results: QuickSort • Visualization of QuickSort on 80 elements

36. Examples: QuickSort • Visual Component Binding with Java Program • Java Class: SortElement • Visual Component: BarVO • Bindings • 1 ClassBind • 3 PropertyBind

37. Examples: QuickSort • Visual Component Binding with Java Program • ClassBind – Bind SortElement with BarVO • Binding XML • <ClassBind> • <TargetClassname=“SortElement”> • <Constructor signature="()V" /> • </TargetClass> • <VisualObjectname=“BarVO” > • <Parameter key="scaleX" value="10" /> • <Parameter key="scaleZ" value="50" /> • </VisualObject> • </ ClassBind >

38. Examples: QuickSort • Visual Component Binding with Java Program • PropertyBind – SortElement.value to BarVO.scaleY • Binding XML • <PropertyMapclassName="SortElement" toName="scaleY"> • <Field name="value*500" /> • </PropertyMap>

39. Examples: QuickSort • Visual Component Binding with Java Program • PropertyBind – SortElement.index to BarVO.x • Binding XML • <PropertyMapclassName="SortElement" toName="x”> • <InterpolateParam key="adaptFactor" value="0.0" /> • <InterpolateParam key="estimateInterval" value="0.7" /> • <Field name="(float)index*20.0" /> • </PropertyMap>

40. Examples: QuickSort • Visual Component Binding with Java Program • PropertyBind – to BarVO.z • Binding XML • <PropertyMapclassName="demo.sort.SortElement" toName="z“> • <InterpolateParam key="adaptFactor" value="0.0" /> • <InterpolateParam key="estimateInterval" value="0.1" /> • <CodeoutVar="offset"> • float offset=0.0f; • if(pivot) offset=100f; • else offset=0f; • </Code> • </PropertyMap>

41. Examples: QuickSort • Visual Component Binding with Java Program • Resulting Visualization:

42. Examples: QuickSort • Changing the Visualization for QuickSort • Use color instead of height to represent value ofSortElement • Binding XML • Before • <PropertyMapclassName="demo.sort.SortElement" toName="scaleY"> • <Field name="value*500" /> • </PropertyMap> • After • <PropertyMapclassName="demo.sort.SortElement" toName="colorRed"> • <Field name="value" /> • </PropertyMap>

43. Examples: QuickSort • Changing the Visualization for QuickSort • Resulting Visualization:

44. Examples: QuickSort • Changing the Visualization for QuickSort • Display index of SortElement on a curve • Binding XML • Before • <PropertyMapclassName="demo.sort.SortElement" toName="x“> • <Field name="(float)index*20.0" /> • </PropertyMap> • After • <PropertyMapclassName="demo.sort.SortElement" toName="rotateW“> • <Field name="Math.cos(-((float)index/(float)maxIndex*Math.PI- • Math.PI/2f)*0.5f)" /> • </PropertyMap> • <PropertyMapclassName="demo.sort.SortElement" toName="rotateY"> • <Field name="Math.sin(-((float)index/(float)maxIndex*Math.PI- • Math.PI/2f)*0.5f)" /> • </PropertyMap>

45. Examples: QuickSort • Changing the Visualization for QuickSort • Resulting Visualization:

46. Examples: Growing Tree • Visualization of a growing Tree • Growing Tree with branch factor 3

47. Examples: Orbiting Mass • Visualizatoin for trajectory simulation • Simulation of the movment of 80 elements with gravity influence on each other

48. Outline • Introduction • Binding Approach • Assisting Visualization System • Implementation • Examples • Conclusion

49. Conclusion: • Introduced visualization through binding approach. • Implemented Assisting Visualization System • Decoupled from underlying program • Flexible to change visualization schema • Reusable visual component • Created a prototype system that can be adopted in languages other than Java.