Experiences from Cyberinfrastructure Development for Multiuser Remote Instrumentation

1. Experiences from Cyberinfrastructure Development for Multiuser Remote Instrumentation Prasad Calyam, Ph.D. David Hudak, Ph.D. Ashok Krishnamurthy, Ph.D. Karen Tomko, Ph.D. Cyberinfrastructure and Software Development Group, OSC IEEE e-Science Conference, Indianapolis, December 10th 2008

2. Overview • Multi-user Remote Instrumentation? • Application Scenario and Components • OSC-developed Solutions • Deployment Case Studies

3. Remote Instrumentation Overview • Academia and Industry use scientific instruments • E.g., Electron Microscopes, Telescopes, Spectrometers • Used for research and training/classroom purposes • Such instruments are expensive to buy and maintain • $450K - $ 4Million initial purchase + Staff $ to maintain • Remote Instrumentation • Remote access of instruments, related devices and their data resources via the Internet • Benefits • Access for remote students and researchers • Return on Investment (ROI) for instrument labs • Avoids duplication of instrument investments for funding agencies

4. Remote Instrumentation Application Scenario

5. OSC’s Remote Instrumentation Program • RI cyberinfrastructure development for Ohio-based universities involves using OSC’s state-wide resources • Networking, HPC, Storage, Analytics • Pilot program funded by the Ohio Board of Regents • Goal: “Leverage Ohio’s investments in scientific instruments, wide area networking, high-performance computing, and data storage to foster academia-industry collaborations involving remote instrumentation”

6. OSC’s Remote Instrumentation Partners • The Ohio State University • Scanning Electron Microscope, Material Science Engg. Dept. • Raman Spectrometer, Chemistry Dept. • McGraw-Hill Telescope, Astronomy Dept. • Miami University • NMR Spectrometer, Chemistry and Biochemistry Dept. • Unipulsed EPR Spectrometer, Chemistry and Biochemistry Dept. • Transmission Electron Microscope, Geography Dept. • Ohio University • Nuclear Accelerator, Physics and Astronomy Dept.

7. Remote Instrumentation Components • Instrument Lab Site • Resource Scheduling • Sample Handling • Use Policy • Usage Billing • Remote User Site • Remote Observation • Remote Operation • Voice/Text Chat • Lab Notebook • OSC • Web-portal Development • Data Storage Management • Real-time Analytics • Network and Data Security

8. Technical Challenges • Network bandwidth • Last-mile bottlenecks lead to improper operation – can cause expensive instrument damage • Frame rate and video quality tradeoffs • Communications • Remote User(s) and Operator co-ordination – VNC/RDP, VoIP, Videoconference, Presence, Control passing, Web cam • Simultaneous multi-device views for user workflows • Dead man's switch • Fail-safe method to stop the service in case instrument Operator becomes incapacitated • File system access – Web services • User accounts, Data read on instrument file system, Data read/write for analytics on mass storage file system • Network Security • VPN, Ports, Firewall rules, Encryption/Authentication

9. Policy Challenges • Scheduling Policy • Prioritizing users – PI/Co-PI, Graduate Students, Industry • Synchronizing calendars of devices and personnel • Sample handling • Licensing • Remote users observation/operation and analytics • Service Level Agreements • Vendor • ISP/ASP • Safe-use Policy • Expert privileges, Novice privileges • Billing • Setup surcharge, Fee/hr, Fee/session, Resource units

10. Case Studies • Account of OSC experiences with cyber-enabling various kinds of scientific instruments • Solutions evaluated • Solutions developed • Open issues • Three Case Studies • OSU Material Science and Engg. Dept. – Electron Microscopes • OSU Chemistry Dept. – Raman Spectrometer • MU Bio-Chemistry Dept. – 850 MHz NMR

11. Case Study-I: OSU CAMM • OSU Center for Accelerated Maturation of Materials (CAMM) has acquired high-end Electron Microscopes • Used for materials modeling studies at sub-angstrom level • OSC providing networking, analytics and storage support for remote microscopy • Permanent console at Stark State for Timken access • Hardware-based (ThinkLogical) KVMoIP solution • Image processing of samples (automation with MATLAB) for Analytics service • Lab Notebook for image management • Remote Microscopy Demonstrations • Supercomputing, Tampa, FL (Nov 2006) • Internet2 Fall Member Meeting, Chicago, IL (Dec 2006) • Stark State University, Canton, OH (Mar 2007)

12. Network Connection Quality Vs User Control • Higher TCP throughput (i.e., mouse and keyboard activity) on poor network connections Increased user effort with keyboard and mouse on poor connections • “Congestion begets more congestion” 1 Gbps LAN – Expert Task-1 Task-2 Task-3 60 B/s Public 100 Mbps LAN – Expert 100 Mbps WAN – Expert Task-1 Task-2 Task-3 Task-1 Task-2 Task-3 900 B/s 1400 B/s 60 s 100 s User expends minimum effort with keyboard and mouse to complete use-case User expends notably more effort with keyboard and mouse to complete use-case 140 s User expends a “lot” of effort with keyboard and mouse to complete use-case

13. Image Processing • Automated Matlab processing of electron microscope images • Alternate to Adobe Photoshop (Fovea Pro 4 plug-in) filters that take multiple days to process • Filters: E.g., Image Blurring/De-Blurring, Image Dilation/Opening • Matlab GUI development for sample filters testing before batch jobs Processed Image Inputs Gaussian Blur Dilate Image

14. Case Study-II: OSU Chemistry • Recent purchase of a Combined Raman - FTIR Microprobe • To get complimentary Raman-IR information about chemicals • OSC custom developed a Remote Instrumentation Collaboration Environment (RICE) software • Enable local students and researchers to work from comfort of their offices or homes • Access for remote  collaborators: California State University, Dominguez Hills, CA; Oakwood University, Huntsville, AL • Active RICE testing and concurrent development in progress • “Pink Screen” GPU problem • Dual-screen resolution issues • Overlay error issues in Vendor software

15. OSU Chemistry Instrument Lab 4XEM Webcam Live View on Web browser; Dual monitor PC withadvanced network & multimedia cards Cyber-enabled Instrument

16. “Pink Screen” GPU Problem • Software-Software VNC issue • OS not aware of GPU video processing • Solution: Use Hardware-Software KVMoIP (e.g. Adder), or Hardware-Hardware KVMoIP (e.g. Thinklogical) Remote VNC Client View

17. Dual-screen Resolution Problem • Default VNC (i.e., Ultra VNC) distribution issue with dual monitors and extended desktop Remote VNC Client View Right Monitor Left Monitor

18. Dual-screen Resolution OSC-Solution • Solution: OSC-patch with increased image geometry Right Monitor Left Monitor Remote RICE Client View

19. Overlay Error Issue in Vendor Software • Remote VNC client refresh causes local video overlay error • Solution: Adhoc trials - notified vendor support Right Monitor Local View with error message Left Monitor

20. OSU RICE Solution Screenshot

21. OSU RICE Solution Features • “Network-aware” video encoding • Optimizes frame rates based on available network bandwidth • Manual video-quality adjustment slider • “Network-aware” action blocking • Warns user of network congestion • Blocks user-actions during high network congestion scenarios • Collaboration tools for Multi-user support • Peer-to-Peer VoIP/ Multi-user VoIP • Multi-user Colored-text Chat • Multi-user Presence • Multi-user Control-lock passing • Multiple display resolutions • Small screen • Full screen • Dual screen (remote site with dual monitors and extended desktop)

22. OSU RICE Solution Demo

23. Case Study-III: MU Biochemistry • Recent purchase of 850 MHz NMR – first of its kind in North America • For studying supra molecular architectures and functional materials • RICE integrated with web-portal for management of remote instrumentation sessions, user collaboration and data • Access for remote  collaborators: Bowling Green State University, Ohio University, Muskingum College, Talawanda High School

24. MU-NMR System Deployment

25. Web-portal User Work-flows

26. Web-portal Features • User Accounts and Privileges • Management of Instruments, Projects, Samples, Sessions, Experiments • KVMoIP and RICE access control • Asynchronous chat for remote monitoring of experiment progress • Experiment data archival at OSC storage • Analysis of stored data sets using OSC-hosted Topspin software

27. Web-portal Architecture

28. Web-portal and RICE Screenshots

29. RICE use-cases for Research and Training • Remote participants can view expert (also remote!) controlling a scientific instrument • Efficiently: Multi-party VoIP, Presence and Chat collaboration • Reliably: Network awareness mitigates instrument damage • Expert can pass control to remote participants • Train students to operate the instrument during class • Allow another expert to study the sample under study • Researchers and Students can conduct experiments at their assigned slots on the instruments

30. Conclusion • Developing cyberinfrastructures for RI requires: • Understanding and overcoming multi-disciplinary challenges to develop solutions (b) Developing reconfigurable-and-integrated solutions that need to be tailored on a per-instrument basis (c) Close collaborations between instrument labs, infrastructure providers, and application developers

31. Future Directions • “Reconfigurable-and-Integrated” Tools and Web-portals • RICE, Wikis, Lab Notebook, Mailing lists, Calendar, … • Human-centered Remote Instrumentation solutions for “at-the-instrument” experience • Human-aware Codec Adaptation • ROI Video Encoding

32. Thank you!

33. MU RICE Solution Demo