SIGGRAPH 2010: X3D Medical Birds Of a Feather

1. SIGGRAPH 2010:X3D Medical Birds Of a Feather Web3D Consortium President Nicholas F. Polys, Ph.D., Virginia Tech Web3D Consortium Medical Working Group Chairs Michael Aratow M.D. FACEP, San Mateo Medical Center Nigel John Ph.D., Bangor University Wales

2. Session Topics • What is this BOF anyway? • What is X3D Medical? • Web3D Consortium • Extensible 3D (X3D) • Updates and Activities: • Recent conferences and workshops • Specification progress • Implementation progress • Demos • Future Forward

3. Medical Birds Of a Feather • Describe, demonstrate and discuss the latest trends and developments in medical graphics and visualization • Define value proposition and new technology in standards roadmaps • Gather community interest and engagement in: • specification development • specification adoption

4. Convergence in Medical IT • New opportunities in improving quality-of-care and reducing cost-of-care: • Standards enable new analytics • Reproducible Interactive presentations and reports • Leveraging the web multimedia ecology (W3C) • Semantics (FMA, SNOMED CT) • Standards enable new levels of care • Consistent communication between providers and organizations

5. Medical Imaging & Informatics • Interoperability challenges for the presentation of medical images are well-recognized • DICOM Vendors • Stakeholder Groups (AMA, GOV, MIL, you!) • Why risk a new wheel? • The internet information space is an ecology that functions through a number of specifications • Unifying these standards catalyzes significant value • Proven and unencumbered scene graph abstractions yield a quantum jump for DICOM: the ‘Hopscotch Strategy’

6. Scope • Consistent and Interoperable presentation states for medical image data • Hospitals, Experts, Patients • Well-developed use cases • Broad impact across the spectrum of care: • Training • Acute & Chronic Care • Prevention • Rehabilitation

7. X3D: Platform Independence

8. X3D Medical Working Group • A Working Group chartered to catalyze reproducible medical visualizations across platforms and over the network • Develop Volume visualization standards that meet the industry’s greatest common denominator • Work with DICOM on the n-Dimensional Presentation State work item (WG11)

9. Use Cases Accessibility outside the radiology suite: • Surgical Planning • Informed Consent • Custom Prostheses • Radiation Therapy • Anatomy Education, Surgical Education …

10. Requirements Healthcare is ‘Mission Critical’ • An n-D Presentation must include: • Structured and interactive virtual environment display (2D & 3D objects and time series) * • Platform-independent, royalty-free technology to enable vendor innovation • Can be rendered with stereoscopy • Openly-published

11. Questions? • Comments?

12. Web3D Collaboration & Convergence ISO W3C - XML - HTML 5 - SVG • OGC • - Web3D MS • Web3D FS • KML Web3D Consortium - X3D - H-Anim - VRML Interoperability & Access Across Verticals IETF - Mime types Open Source Multi-D Random Access Filesystems - HDF5 - NetCDF DICOM - N-D Presentation State - Volume data Khronos - OpenGL - COLLADA

13. The Web3D Consortium A not-for-profit organization that develops and maintains ISO standards for the description and transmission of 4D data over telecommunication networks. The X3D, H-Anim and VRML standards specify both a file format and a runtime specification that are open and royalty-free www.web3d.org

14. Web3D & X3D • Proven Innovation • Real time interactive graphics • Lights, cameras, text, meshes, textures and materials • Animations, sensors, volumes & style • Interoperability with web • distributed resources referred via URL/URI • E.G. images, sounds, videos, models, worlds • Durability • ISO Specifications: • Abstract Spec (scene graph) • Encodings • Bindings • H-Anim http://www.web3d.org/x3d/specifications/

15. X3D Highlights • Abstract scene graph with multiple encodings and API bindings • Well-defined runtime semantics • ISO (SC-24), Royalty-Free • Multiple open-source and commercial implementations (approx 24) • Demonstrated capability for medical applications: • covers significant majority of DICOM requirements • remainder being specified through WG11 work item

16. X3D Encodings and Bindings An equivalent scene graph can be: encoded in and manipulated by multiple Languages ! http://www.x3dgraphics.com

17. X3D Components and Profiles • Components • Families of related nodes • Levels • Of support • (fields and functionality) • Profiles • Well-defined nodesets • for vertical application functionality

18. Recent Events • SPIE Medical Imaging • WG 11, WG 2, WG 12 • DICOM Update • IEEE VR Medical Workshop • Keynote by Henry Fuchs, UNC • Papers & presentations covered volume rendering, haptics and surgical simulation, therapy and training • Web3D 2010

19. Medical X3D Specification Development • History and Prior work • X3D Volume Rendering • Relationship w/ DICOM standards • Current activities

20. US ARMY TATRC Contract • Successfully completed in 2007 • MedX3D Draft Specification available to members • Implementations: • SenseGraphicsH3D: Open Source Haptics • Fraunhofer IGD Instant Reality • Semantic scene graph inline or referenced • Volume Rendering Demos www.H3D.org www.InstantReality.org

21. Opacity Map ISO Surface MIP Map Composed Volume Segmented Volume MedX3D Volume rendering styles with default fields (TATRC deliverable, H3D) RESULTS AND REPORT – Publically Published on Web3D.org !!! Spec development is currently toward a 3.3 Profile.

22. X3D Presentations Reproducible interactive virtual environments (over a network) • X3D scene graph is Transformation graph and Behavior graph: • Core, Grouping, Navigation • 2D & 3D Geometry, Appearance, Text • Rendering, Lighting • Event Utilities • Time, Interpolators, Sensors • Environmental Effects, Layers

23. X3D Volume Rendering Component • Specified and developed w/ US Army TATRC; • two implementations (one open-source): • General transfer function • Shading styles • Blinn-Phong(including advanced features such as shadows and light scattering) • Cartoon shading • Gooch shading(tone mapped) • Enhancement styles (enables enhancement of important features in an easy way) • Boundary enhancement • Edge enhancement • Silhouette enhancement • Maximum , Minimum and Average intensity projection • Iso-surface rendering • Segmented data rendering (render different parts of volume data using different styles)

24. Questions? • Comments?

25. Web3D + DICOM Recent History • Started in WG-17 • Web3DC contributed to 3D Surface data structures (compatible with ISO 19775 (X3D)) • Now primarily in WG11 • Scoped Work Item on n-Dimensional Presentation States • Web3D Consortium Medical WG presenting specs and royalty-free technology meeting (and exceeding) requirements

26. Timeline Recap

27. DICOM: Critical Mass • Continue to gather momentum, requirements, alignments with DICOM & Vis stakeholders (you!) • Web3D Consortium brings a time-proven solution: a standardized scene graph to meet the requirements of the healthcare enterprise • Recognizing the value of consistent renderings across caregivers and stakeholders, the US Army TATRC provided critical support (spec draft and open source implementations) • We are pleased to report: • Significant progress on this work item!

28. Specification Strategy:Medical X3D 2011 Profile • Specify additional nodes and components for X3D abstract spec: • Render styles • Octree, Annotation • Create MedX3D Profile for X3D abstract specification; [X3D 3.3] • Encoding • Assess X3D XML, X3D Binary and DICOM encodings of X3D Medical 2010 Profile

29. DICOM WG 11 Direction for n-D Presentation States • Functional Profile (node set): • X3D: Medical 2011 Profile • X3D Interactive+ • Volumes & Rendering styles • Cutting/ clipping (i.e. for Multi-planar Reconstruction (MPR)) • Scene graph could be encoded in X3D Binary Encoding scene graph profile in DICOM

30. X3D Medical 2011 Concepts • Include some set of components for meeting requirements: • Grouping: Transformation, Switch, LOD • Viewpoints (Perspective and Ortho) • Lights (Directional, Point, Spot) • 3D, 2D surfaces, lines and point geometries • including Shape Appearance and Materials • Per-vertex data such as RGBA and ImageTexture maps • Annotation (Text, Layers) • Interactivity • Sensors • Event model, animation • Scripting?

31. Specification Progress: Volumetric Presentation • Volume Rendering • MedX3D styles have necessary and sufficient coverage • Masks, subvolumes & segments • MedX3D voxel data structure and access per WG 2 and WG 12 requirements • Addressing voxels within volume • Volume time series • User Interface • e.g. Clipping & MPR • Planes • Curved surfaces • 3D volumes

32. Questions? • Comments?

33. Implementations & Demos www.H3D.org www.InstantReality.org

34. Demos

35. R&D Opportunities • Informatics • Interoperation, mapping, integration • Electronic Medical Records • Graphics • Rendering • Display • Perception • Interfaces and Interaction • Medical Cognition & HCI • 3DUI

36. Going Forward • Unifying ISO scene graph technology yields a quantum leap in health care interoperability • Immediate value in cross-platform transmission and rendering of medical image presentations • Continue development of DICOM+MedX3D Profile: specification (X3D 3.3) including annotations, implementations, conformance examples • Consider standards for soft deformable bodies (tetrahedral meshes for haptic simulation)

37. Ways to get Engaged • Web3D Consortium Membership • Web3D Working Group Activity (email lists, phone & skype conferences) • Upcoming Conferences: • SuperComputing 2010 • MMVR • SPIE Medical Imaging • Web3D 2011

38. Contacts • Nicholas Polys : npolys@vt.edu • Michael Aratow : maratow@noegenesis.com http://www.web3d.org IEEE VR papers: http://www.hpv.cs.bangor.ac.uk/vr10-med/