Portals and User Interfaces for Data Management and Grid Computing

1. Portals and User Interfaces for Data Management and Grid Computing Rob Allan Leader of the Grid Technology Group http://www.grids.ac.uk 9th November 2007

2. Making the Grid easier to use… • Why are we doing it? • What solutions are we investigating? • What has research given us? • How is it being used?

3. Institutions need Autonomy and Security Host – client relationship Example solution suggested by Web server - browser Communication must be initiated by client because of firewall around client’s institution. Can use a proxy or gateway.

4. The Grid “Client Problem” Many clients want to access a few Grid-enabled resources Grid Core Consumer clients: PC, TV, video, AG Middleware e.g. Globus Workplace: desktop clients Portable clients: phones, laptop, pda, data entry… Grid Core

5. Grid Infrastructure Deployment(adapted from Foster and Kesselman)

6. How to Deliver e-Science Services? • Provide heavyweight functionality (Globus, Condor, SRB?), but only on Grid-enabled hosts; • Implied need for client-server software architecture, e.g.: • Web-based portal with familiar browser • Client programming library, API in C, C++ Java, Perl, Python, R etc. • Ability to link to existing applications/ GUIs • Command-based shell interface • Drag and Drop interface • Need a published set of services on Grid hosts – OGSA model needs registry and semantics; • Need easy development and deployment framework for applications and client tools . International workshops: Portals and Portlets, NeSC, 14-17th July2003 Portals and Portlets, NeSC, 17-19th July2006

7. Lightweight Grid Computing • Concept: • Aim was to develop “lightweight” client interfaces to the Grid using Web services via an intermediate server. Lessons learned from portal development which has a similar architecture. The server is “trusted” and can have ports open enabling Globus and other Grid middleware to be used at the back end. • Research and outcomes: • GROWL – developed as a JISC-funded Virtual Research Environment prototype • MCS/ RMCS – developed in the NERC-funded e-Minerals project: Environment from the Molecular Level • AgentX – developed with e-Science core funding to support data interoperability for Collaborative Computational Project (CCP) applications

8. Architecture

9. Users of Lightweight Interfaces(adapted from Foster and Kesselman)

10. Opportunities • GROWL, RMCS and AgentX now constitute a computational framework which is being deployed for application users in the CCPs as a collaboration with CSED, NceSS: the National Centre for e-Social Science via the ESRC e-Infrastructure project and for large-scale facilities such as the Diamond Light Source. • We organised a workshop in May 2005 to compare solutions for lightweight Grid computing • in addition to RMCS, GROWL and AgentX, there is AHE, GridSAM, GridSite, WSRF::Lite, … • Also considered SAGA from GGF as a standard interface • There could be a significant chance for the UK to establish a lightweight Grid client toolkit combining the best aspects of these solutions • A joint bid to OMII was unsuccessful http://www.grids.ac.uk/GROWL/Lightweight

11. Portals • Concept: • Aim was to develop Web-based interfaces using technology accessible to standard browsers. Embrace all aspects of this, including Web 2.0. Best practice from existing tools for on-line learning, information management, collaboration and research. Use current Java language standards for interoperabilty: JSR-168, JSR-170, JSR-286, WSRP, AJAX, … • Research and outcomes: • DataPortal – example of pilot project for data management • HPCPortal/ InfoPortal – examples of pilot projects for Grid computing • NGS Portal – evolved from HPCPortal and InfoPortal into a generic interface for the National Grid Service • e-HTPX – management of protein crystallography pipeline, data reduction and structure analysis • Sakai – development of a framework and tools as a Virtual Research Environment for different communities JSR-168 is the key Java portlet technology. Announced shortly after our first international conference in 2003.

12. Users of Portals(adapted from Foster and Kesselman)

13. NGS Portal • NGS Portal is a generic portal for HPC applications on Grid clusters • Embraces new ideas and standards, e.g. JSDL • Goal: to build communities, sharing job descriptions and best practice • Portlets tried and tested - can be used by new e-Science projects (available on CD?) • Opportunities to collaborate to enhance the suite of portlets and develop a repository, e.g. for workflow • Additional funding from OMII

14. Virtual Research Environments • Sakai: Open Source, Java, Collaborative Learning Environment. Enterprise-level - can support 10,000s of users • We we funded by JISC to carry out the Sakai Evaluation Exercise (2005)‏ • I developed much of the background material for the JISC VRE-1 Programme – realisation that VLE and VRE are similar • We were funded by JISC to develop a Virtual Research Environment based on Sakai (2005-7)‏ • Groups at Cambridge and Hull/ East Anglia were funded to apply Sakai to targeted research areas (2005-7)‏ • We were funded by JISC to carry out a review of how the Information Environment could meet the needs of researchers using portals (2006)‏ • We are currently further developing the CREE and SPP portlets for access and cross-search of open archival institutional repositories (2007)‏

15. Some Questions about VRE Development • How can existing tools be re-purposes to rapidly meet the needs of e-Research? – collaboration, information management, training • How can we make use of the best Java technology available? – JSR-168, Spring, Hibernate, AJAX, etc. • How can we include legacy code, e.g. Perl, C++? – WSRP, bridges • Can we federate services underpinning a portal, e.g. using remote portlets? - WSRP • Can we develop portlets using service registries? – UDDI, WSDL • Can portlets truly be made re-usable in different frameworks? • Can we support a repository of such portlets? • Can we include commercial portlets and frameworks (IBM, Sun, BEA, Oracle)? • How can we include Workflow, Semantics, Web 2.0?

16. Emerging Architecture

17. Some Questions about VRE Usage • Deployment and evaluation of such a VRE tests and extends our understanding of practical ICT-based support for research in the following areas: • How can portal frameworks be configured to best suit the expectations and work practices of different research user communities and institutional or organisational contexts? • Can tools from multiple institutions and organisations be brought together coherently to enable sharing of information, processes and collaboration? • Can community-specific tools be integrated meaningfully alongside generic and remotely-hosted Web tools? • Can a portal based approach provide the flexibility to enable effective use by both researchers and administrators? • At what points are desktop tools or those provided by a mobile platform, more effective? • How might these be best integrated within a meaningful user experience? • How can we engage HCI and CSCW specialists

18. Science Gateways • Sakai is now the preferred delivery framework for portals to support growing communities of users who want to manage data and information, access Grid resources, and collaborate on-line using Web 2.0 style tools. The Generic portal and VRE tools are being customised to their needs. We refer to these as “science gateways”. • ReDRESS: Resource Discovery for e-Social Science • NceSS: National Centre for e-Social Science and the ESRC e-Infrastructure users • University of Cambridge – teaching and research • University of Manchester – research services • CCP9 and Psi-k: large EU network of excellence (1,900 users)‏ • NW-GRID: project management and users • Diamond Light Source is investigating Sakai-based tools

19. Opportunities • Hosting Science Gateways for growing number of communities • may require special-purpose servers to enhance the Grid infrastructure, e.g. for data pre-processing, visualisation or statistical analysis • engage the community in contributing to a research portlet repository, e.g. via NGS WP6 • provide CD version of pre-configured portals for new projects, they can develop and add their own tools • engage further with international community, Sakai, GridSphere, uPortal, etc. • Growth of Campus Grids, requires easy access for non-experts

20. Summary • Demonstrated need to make distributed resources accessible to end users engaged in research activities • Address needs to integrate and manage access to underlying services • Delivery to a variety of users with different needs • Common architecture emerging • Addressing issues of rapid deployment and re-usability of software • Engaging with activities of NeSC, NIEeS and NCeSS for outreach and training • Beginning to address usability issues • An ongoing programme of targetted research to address the above has produced outcomes - papers, best practice and software – all available from STFC e-Science Centre.