Visualization Middleware for e-Science UK e-Science Core Programme Open Call gViz Project

1. Visualization Middleware for e-Science UK e-Science Core Programme Open Call gViz Project

2. Visualization is key component in e-Science Increasingly so, as size of datasets and scale of simulations continues to increase gViz project aims to: Grid-enable an existing visualization system (IRIS Explorer) Broaden out to: Study XML applications for visualization Look at computational steering in depth Grid-enable another visualization system, pV3 gViz partners: Academic: Leeds, Oxford, Oxford Brookes, CLRC/RAL Industrial: NAG, IBM UK and Streamline Computing International: Caltech, MIT IRIS Explorer IRIS Explorer gViz - Visualization for e-Science Grid-enable Comp Steering XML Appl’n pV3

3. Visualization represented as pipeline: Read in data Construct a visualization in terms of geometry Render geometry as image Realised as modular visualization environment IRIS Explorer is one example Visual programming paradigm Extensible – add your own modules Others include IBM Open Visualization Data Explorer Toolkits such as vtk have similar underlying model visualize data render Starting Point: Dataflow Visualization Systems

4. Grid-enabled IRIS Explorer • As we move to a Grid world… • Dataflow paradigm remains entirely relevant • … but we want to be able to distribute modules across the Grid • … so can we simply extend NAG’s IRIS Explorer to work in this way?

5. Select remote host Harnessing Remote Compute Resources Explorer on multiple hosts Explorer on single host • Automatic authentication using: • Globus certificate • SSH Key pair

6. Secure Distributed IRIS Explorer • Users can manually launch modules from remote Librarians • IRIS Explorer maps can link specific modules to specific remote hosts • ‘Design’ the dataflow for the Grid

7. Simple Computational Steering • Extensibility of IRIS Explorer allows user code (such as a simulation) to be included as a module in the dataflow • Demonstrator created for an environmental crisis scenario • Dangerous chemical escapes! • Model dispersion using system of PDEs and solve numerically over mesh • Visualize mesh elements where concentration exceeds threshold • What happens when the wind changes? • ‘faster-than-real-time’

8. Computational Steering within IRIS Explorer • Finite volume simulation code as module in IRIS Explorer • Exploit secure distributed IRIS Explorer to run simulation remotely • Steer and visualize on the desktop – simulate on the Grid resource • Other IRIS Explorer features can be brought into action – such as collaborative working…. between numerical modeller and meteorologist for example

9. Initiate collaborative session Collaborative Visualization - Bring in the Meteorologist Meteorologist with wind information (linking in remotely) Numerical modeller studying error details

10. This gives us Grid-enabled Collaborative Visualization and Computational Steering Why is this not the whole answer? IRIS Explorer is just one system It is designed primarily for visualization – not computational steering Broaden out from IRIS Explorer Study XML application for visualization, independent of any specific system XML for data too Study computational steering Decouple the simulation from the visualization Look in depth at grid-enabling another system, pV3 First we revisit the reference model… Moving ahead…

11. Visualization pipeline described independently of software or hardware resources For Grid environments we extend to progressively bind in resources…. Conceptual: intent of the visualization Show me isosurface of constant temperature Logical: bind in the software system Use IRIS Explorer (or vtk, or whatever) Physical: bind in the resources to be used Run the isosurface extraction on particular Grid resource visualize data render Extending the Model for Grid Environments

12. Dataflow consists fundamentally of: a map containing links between ports on modules which have parameters This leads us to a simple XML application for visualization: called skML Here a data reader is linked to an isosurfacer <?xml version="1.0"?> <skml> <map> <link> <module name="ReadLat” out-port="Output"> <param name="Filename"> testVol.lat </param> </module> <module id=“iso” name="IsosurfaceLat" in-port="Input"> <param name="Threshold" min="0" max="27"> 1.8</param> </module> </link> … Developing an XML Language for Visualization: skML

13. Resources can be bound in by adding RDF annotations to describe software and hardware requirements Here we run ‘iso’ remotely and say that we are requiring IRIS Explorer <rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#" xmlns:v="http://www.gviz.org/skML/"> <rdf:Description about="iso"> <v:Type>IRISExplorer</v:Type > <v:PhysicalLocation rdf:resource=”http://www.gviz.org/ Mars101” /> </rdf:Description> </rdf:RDF> Binding in Resources

14. SVG Map Editor • skML gives us an XML application for visualization • In addition to language representation, a diagrammatic representation has been created in SVG – so we can do dataflow programming in a web browser • A new IRIS Explorer module can read skML and generate corresponding map • skML can also be turned into an IBM Open Visualization Data Explorer network <?xml version="1.0"?> <skml> <map> <link> <module name="ReadLat” out-port="Output"> <param name="Filename"> testVol.lat </param> </module> <module id=“iso” name="IsosurfaceLat" in-port="Input"> <param name="Threshold" min="0" max="27"> 1.8</param> </module> </link> …

15. Taming dataset structure for visualizing over the Grid • At present: • Multiplicity of data sources (say, m) • Multiple visualization systems (say, n) • Suppose a VO is being assembled where that diversity is present and may vary? • Would like to move from Axmxn to Bxm+Cxn+D (where we would like B,C,D not to be too big) • Investigating an approach where: • Intermediate XML-based language for data structure is used • Structure is processed by sequence of straightforward filters • Bulk data (e.g. resequencing) is processed as late as possible • Existing tools used as far as possible (e.g. XSLT transformers) • Metadata (including provenance) is not lost! • Interested to gain access to other data sources to test approach

16. Experience in building steering applications with IRIS Explorer taught us: Better to decouple simulation from visualization system… … simulation runs autonomously, scientist uses visualization system as front-end control panel to monitor and steer Useful to be able to distribute the visualization – move some code close to the simulation Scientist should be able to connect/disconnect from long-running simulations Led us to build the gViz steering library Code to link simulation and visualization Some design criteria: Lack of intrusion Minimize performance loss Breadth of scope Exploit service-oriented concepts Exploit existing visualization systems Manage different rates of producer-consumer Support collaboration Computational Steering

17. visualize render gViz-lib Simulation code Simulation code Register Parameters Data Launch code on Selected resource Retrieve resource information Locate any running simulations Data Parameters Pass Location gViz-lib gViz-lib control getData Communications using Web Services and Proxies • Use GIIS to locate grid resources. Grid resources Grid Information • Use web service tool, gVizDS, as directory service of running simulations. gVizDS gVizProxy • Use proxy tool, gVizProxy, if no direct connection possible between simulation and visualization system Discover/ Launch Locate Desktop

18. Pollution Simulation Using the gViz Library and IRIS Explorer

19. gViz with other visualization systems • Any visualization system can potentially act as the front-end control panel • SCIRun from University of Utah • vtk toolkit

20. IRIS Explorer IRIS Explorer Exploitation through future releases from NAG Open Overlays project (Fundamental CS for e-science) Lancaster, Ox Brookes Ontologies for Visualization -> E-Viz project with Bangor, Swansea and Manchester Apply within Integrative Biology project and compare/contrast with RealityGrid Mike Giles talk Where Next? Grid-enable Comp Steering XML Appl’n pV3

21. Acknowledgements The gViz project team has involved many people: • Leeds University: Ken Brodlie, Jason Wood, Chris Goodyer, Martin Thompson, Mark Walkley, Haoxiang Wang, Ying Li • Oxford Brookes University: David Duce, Musbah Sagar • Oxford University: Mike Giles, David Gavaghan • CLRC/RAL: Julian Gallop • NAG: Steve Hague, Jeremy Walton • Streamline Computing: Mike Rudgyard • IBM UK: Brian Collins, Alan Knox, John Illingworth • CACR, Caltech: Jim Pool, Santiago de Lombeyda, John McCorquadale • MIT: Bob Haimes Development environment at Leeds: White Rose Grid – e-Science Centre of Excellence

22. Use of Web Services for Remote Distributed-Memory Visualization Mike Giles Bob Haimes giles@comlab.ox.ac.uk Oxford University Computing Laboratory

23. Goal of Oxford Work Starting from existing pV3 software from MIT, • Post-processing or co-processing of data on distributed-memory parallel systems • Support for collaborative visualization and steering • MPI used for communication with parallel system Replace PVM message-passing to remote workstation by secure web services – much better suited for collaborative visualization • Cleaner security model – restricted access to data, no user account required on parallel system • Easy handling of firewalls and NAT

24. Existing pV3 Architecture

25. Web Service Version

26. Achievements and Conclusions • Web service implementation used C/C++ gSOAP toolkit • Very easy to use with extensive feature set: OpenSSL, keepalive, gzip compression • Main technical challenge was mimicking symmetric message-passing using asymmetric web service RPC • Has been tested successfully on a PC cluster with NAT firewall, running the concentrator on the front-end node • Performance is comparable to PVM message passing