Grids: How do we define success?

1. Grids: How do we define success? Vicky White, Fermilab April 27, 2004 ISGC-2005 , Taiwan

2. Europe U.S Physics Computing Lab Funding Agency Builder Manager Views of a Straddler Verb: to strad·dle • To stand or sit with a leg on each side of; bestride: straddle a horse. • To be on both sides of; extend over or across: a car straddling the centerline..

3. Hopefully not a Straggler Strag"gler1. One who straggles, or departs from the direct or proper course, or from the company to which he belongs; one who rambles without any settled direction.

4. What is a Grid? • So many definitions e.g. • Coordinated resource sharing and problem solving in dynamic, multi-institutional virtual organizations. [Ian Foster] • So many analogies – but most are not very useful • Power Grid • The next big thing after the WWW ? • Seamless virtual environment • So many technologies and types of Grid • Globus almost a standard – but not universal • Data Grid, Computational Grid, Campus Grid, National Grid, Climate Grid, and so on……. • So much hype – and such high expectations ! • Confusion with • Public Resource Computing – e.g. SETI@home • Plain old distributed computing, commercial enterprise computing

5. Global GridCommunity

6. Or ship power to where work needs to be done? The (Dubious) Power Grid Analogy Do we ship work to the power source? On-demand access to and integration of resources & services, regardless of location Source: Ian Foster

7. The GRID: networked data processing centres and ”middleware” software as the “glue” of resources. Researchers perform their activities regardless geographical location, interact with colleagues, share and access data Scientific instruments and experiments (and simulations) provide huge amount of data Grid Vision Federico.Carminati@cern.ch

8. Grid Vision : • Useful, able to support/promote new science • Usable(accessible, robust, easy-to-use) • High-capacity(rich in resources) • High capability(rich in options) • Evolutionary(able to adapt to new technologies and uses) • Persistent (usable by community in the long term) • Stable(usable by community in the short term) • Scalable(growth must be a part of the design) • Integrative(promoting end-to-end solutions) Source: Fran Berman, SDSC

9. Grid Vision = CyberInfrastructure?

10. Why do we want to use Grids for Science? • Cost ? • Reliability ? • Ease of use ? • Scalability with growing datasets ? • Attracts funding? • ?

11. Large Hadron Collider(LHC) at CERN

12. Is “Grid” really about costs and technology? • The way to get massive computing cycles? • The CMS detector alone cost several hundred million CHF and the LHC Accelerator several billion CHF • The total estimated cost of the Tier0 and Tier 1 center at CERN for LHC is in the tens of millions of CHF • Total estimated cost for computing might be 3 times this • Power and cooling for all this is a problem but…. • The only way to store multi-petabyte datasets ? • Today at Fermilab we have robotic storage that houses >2.5 PB of data • Potential for 13 PB with today’s tape technology and probably at least double that by 2008 • The way to reasonably share the costs ?

13. Is Grid about scavanging unused computing cycles? • NOT Public Resource Computing • seti@home • Einstein@home • folding@home • LHC@home • You-name-it@home

14. Why do companies jump on “Grid” • To make money • To control markets • To leverage open source into market leadership For Science and Computer Science these are not our goals or the ways in which we determine success.

15. The Value of Grid Computing:IBM Perspective Increased Efficiency Higher Quality of Service Increased Productivity & ROI Reduced Complexity & Cost Improved Resiliency

16. switchfabric compute storage Grids: HP Perspective computing utility or GRID virtual data center value programmable data center grid-enabled systems UDC Tru64, HP-UX, Linux clusters Open VMS clusters, TruCluster, MC ServiceGuard today shared, traded resources

17. Grid - its really about collaboration! • It’s about sharing and building a vision for the future • And it’s about getting connected • It’s about the democratization of science

18. Cathedral and Bazaar • “Linux overturned much of what I thought I knew. I had been preaching the Unix gospel of small tools, rapid prototyping and evolutionary programming for years. But I also believed there was a certain critical complexity above which a more centralized, a priori approach was required. I believed that the most important software (operating systems and really large tools like Emacs) needed to be built like cathedrals, carefully crafted by individual wizards or small bands of mages working in splendid isolation, with no beta to be released before its time. “ • “The Linux community seems to resemble a great babbling bazaar of different agendas and approaches….. out of which a coherent and stable system could seemingly emerge only by a succession of miracles” Eric S. Raymond, The Cathedral and the Bazaar

19. The Grid Bazaar • Many Grids and Grid environments • An evolutionary process • An environment for innovation and prototyping ways of achieving the grand vision • A place where seemingly miracles are needed to make it all work together • But really it is smart people and the will to do so • Interoperability of seemingly divergent efforts will become the norm • Standards must only come later and not suppress the possibility of achieving the grander goals and visions

20. For Success: Define a Destination? or look for a pot of gold? “To travel hopefully is a better thing than to arrive”. Robert Louis Stevenson

21. Borromean Rings These rings have the interesting feature that, while no two rings are linked, the set of rings is linked. Remove any one ring, and the system falls apart. Computer and Information Science Behavioral and Organizational Sciences, (Policy, Government, Economics) Scientific Research and Education

22. Collaboration • EMSL Collaboratory • The logo of the Collaboratory Project at the Environmental Molecular Sciences Laboratory uses the rings to symbolise collaboration between the researchers. Their website expains: The Borromean Rings are three symmetric loops. Although the rings are interwoven and cannot be pulled apart, no two rings are interlinked - removing (breaking) one allows the other two to slide apart. This is the nature of collaborative work as well - people working together create something that is more than the simple sum of their individual efforts. This synergism is the principle behind the Collaboratory concept.

23. Pasteur’s Quadrant Creation of knowledge: basic, curiosity-driven research Application of knowledge Classic Linear Research Model Pasteur’s Quadrant Research Model Pasteur Yes Bohr Focus on New Knowledge Creation? No Edison No Yes Focus on Application? Source: Dan Atkins

24. Many possible Grid Quadrants • Information Technology v. Science Applications • Local economic, network and IT development v. Global Science collaborations and projects • Physics v. Biology • Earth Observation v. ? • Education v. Scientific Research

25. Fermilab and Collaboration “Active” experiments from the 2003 Research Program Workbook • Of 213 Institutions involved 114 of them are non-US • Of 1916 physicists, 753 are non-US • Of 699 students, 234 are non-US

26. Taiwan-Fermilab Collaboration on physics • Incredibly valuable contributions to the CDF experiment over many years • Now we travel the Grid road together also

27. Open Science Grid – a Collaboration

28. Working towards the Grid vision- a building process, not a deployment activity • How many physicists does it take to change a light bulb • Answer: One to hold the lightbulb, but more than 100 to turn the building • How many physicists does it take to BUILD a Grid • Two typical types of answer : • (1) None, because they all know that someone else is deploying it for them • (2) Three times as many as it would take to get their work done without using a Grid

29. Grids: Metrics for Success • How many jobs are run ? • With what level of errors? • How much data is stored/moved/read/written? • How many people are using a specific Grid ? • How many press releases are there about a Grid? • How much money do governments allocate to a Grid? • How easy to use is a Grid? • How much more science gets done from use of a Grid? • How effectively are resources being shared and well used?

30. Grids: Metrics for Success • All of the previous metrics are great indicators, but not the real measure for success • Too many spins and interpretations

31. Grid: Interoperable set of cyber infrastructures (Grids) A few problems still to solve • Economy • Payment • Value of contributions • Currencies • Policy • Who can use? • Who is responsible? • Who can decide? • Security (of course)

32. If we can work together • To make a contribution to all of the following: • Science and Scientific Research • Society and its prosperity and harmony • Education of the next generation • Creating a dynamic collaborative environment for problem solving and innovation • Then we will have succeeded

33. Contributing and Communicating • If we can in some small way contribute to peace and prosperity in the world • If we can help to close the digital divide and make contributing to science and technology and innovation a more democratic process (between the haves and the have-nots) • If we can “speak” to our governments and have them better understand the “Grid Quadrants” and the Borromean rings I spoke of earlier • Then we will have succeeded

34. Success on a Worldwide scale • If we can bring together people from all over the world (whether they be physicists, biologists, computer scientists, climate researchers or ….) and they • Want to be part of building the “cyber infrastructure” or Grid environments or “e-science environments” for the future • Actively participate • Get benefit from the collaboration Then we will be succeeding

35. Success on a Worldwide scale • The work will get done • Scalability issues will get addressed • Robustness will come • Interoperability will be made to work • Even more diversity than we see today Because people working together can and will make things work – if they are involved and getting benefit for themselves, their institutions, their science, their nation – in a broad way

36. Success is when we clearly communicate • To each other • To governments and funders • To the public • To our communities And the message we give is clear and positive

37. The Message • Collaboration is essential on a worldwide scale • To carry out many of the ambitious endeavors we want to undertake – such as LHC and LHC experiments and after that the International Linear Collider and of course many other large international science endeavors in fields other than physics • Diversity is the norm and healthy • Networks are fundamental and high speed connectivity is essential • Data is pervasive and ever growing and we need to analyze it, organize it, mine it, catalog it, curate it, understand it for all purposes ranging over • Wallmart inventory tracking • Climate and Earthquake predictions • Finding the Higgs? And understanding the fundamental nature of matter • Mapping the human genome • Fighting cancer • And much more…..

38. Success • When Grid – the vision and the way of working - helps us get this message out • Without finger pointing • Without complaining about technology and its imperfections – it will be gone before its perfect • With balance between what must work today and what might help us tomorrow

39. ISGC-2005 • This workshop is one of the very signs of success • The program is really excellent and diverse with strong participation from Asia-Pacific, Europe and the Americas • Let’s shut down our email and listen to the speakers and hear how their solutions to problems (in the small) fit in and each play a part in building a Grid environment (in the large) • Let’s communicate by listening

40. Thank you • To Academia Sinica for hosting us and to Simon Lin and all of his staff for organizing this workshop • To all of you who have been and will be collaborators on both Science and Grid projects • Thank you for your energy and willingness to work together despite financial and cultural pressures that make collaboration hard • To all of you for listening