PRAGMA Grid Research Projects in Universiti Sains Malaysia

1. PRAGMAGrid Research Projects in Universiti Sains Malaysia Compute Power Market / P2P e-Science Grid Presented by : Chan Huah Yong Gian Chand Sodhy Grid Computing Research Lab Universiti Sains Malaysia

2. PRAGMAGrid Research Projects in Universiti Sains Malaysia Compute Power Market / P2P

3. Compute Power Market/Peer to Peer • Compute Power Market (CPM) • computational market in peer-to-peer computing style, • introduces resource trading on idle computational power across the Internet.

4. CPM/P2P Framework Layer 4: Applications Applications Portals Scheduler Layer 3: CPM Utilities Market Server CPM Agent Trader Dispatcher Accounting Job management Layer 2: P2P Middleware JXTA Protocols (Peer Groups, Peer Pipes, Monitoring etc) JXTA Security Layer 1: P2P Fabric Internet USM Melbourne

5. CPM Design Architecture Three basic entities: • resource provider • resource consumer • market. These entities are communicating with each other through a set of utilities and services that will cater for resource searching, pricing negotiation, resource allocation, job deployment, transaction accounting and so on.

6. CPM Design Architecture Two key players driving the Compute Power Market: • Compute Power Market Agent • Market Server Like the conventional marketplace, CPM Agent works on behalf of both resource owner and the resource consumer, while the market server provides the necessary infrastructure including communication, information, security, transport access to remote resources, market repository and information services that able to bring these 2 parties together for resources trading.

7. Market Repository EntitiesInteraction Market Market Server 1) Publish Resource Information 2) Discover Resource Information CPM Agent CPM Agent 3) Contract Negotiation 4) Job execution 5) Accounting Consumer Provider

8. CPM Market Server Consists of two major components: • Membership Control • maintains the access authentication of the market by applying membership among the provider and consumer. • Resource Discovery component • manipulates the resource publishing and discovery process. • The resource publishing service allows the resource provider to advertise their resource and the resource discovery service provides consumer the facility to discover resource available in the market.

9. CPM Agent Application that comprises the fundamental services of CPM, includes: • Scheduler– schedule and dispatch job • Trader- handles the trading and negotiation between the provider and consumer. • Job management - handles the job deployment and execution on the provider computer device and returns the result back to the consumer. • Accounting- handles the fees and pays for the trading using a resource meter   to keep track of the resources that have been consumed.

10. CPM Agent (cont…) Users interact with their CPM Agent for • searching (search ideal resource provider base on consumer preferences), • managing(negotiating with CPM Agent from the provider for a particular resources trading) and • running (executing and scheduling job) resources in the CPM. • CPM Agent will represent resource owners and do the arrangement and management of the resources and play an important role in trading negotiation. When the job finished, CPM Agent completes the accounting process and bill the consumer for the charge of resource usage.

11. Work Completed • CPM/P2P Version 1.0 • Trader • Job Management • Accounting • Generic Components • Basic components • Already uploaded to JXTA CVS on September 2002.

12. Benchmark + Retrieve + Publish Info waiting for contract Discover + Join Market Receive + Accept/Reject Contract Update log database Receive + Execute job Calculate Charge + prepare bill Pack + Return Result + Bill Complete System Flow for CPM Agent on Provider Trader provider Generic Components [using JXTA] Accounting Job management

13. Set Trading condition Discover resource Discover + Join Market Generate + send Contract Update consumer account Send job receive Result + Bill Check Bill + Pay provider Complete System Flow for CPM Agent on Consumer Trader consumer Generic Components [using JXTA] Reject Accept Accounting Job management

14. Work In Progress • Scheduler • Market Repository • Provide advertisement manipulation on Market repository • Multi Rendezvous Model

15. CPM Agent CPM/P2P Scheduler (super peer) … … … Scheduler Accounting Submit job Receive result User Trader Job Management Master Worker

16. Scheduler • The scheduler has a Hierarchical Master-Worker architecture. • Master (metascheduler) schedules job based on global scheduling policies. • Worker (local scheduler) schedules and dispatches job based on local system policies. • Job scheduling is based on dateline and budget constraint.

17. CPM Agent Market Repository Market Server • Discovery • - Membership Market Repository User • Market repository. • Searching algorithm (advertisement) • Index manipulation (add new, update, delete) • Market-to-market Interaction Resources

18. Updating Market Repository • Once peer leave the network • Update Repository indicate peer has left (delete the service offered by that peer) • If there are new services to be shared by existing peer • Update the old information (services) in Repository

19. Future Work • Trading Policy • The resource usage policy (such as resource reservation, resource availability) • Security • Handle the security issues in CPM/P2P, include secure communications, intrusion detection. • Multi Market Model • Adopt various market model (auction, bargain) into CPM/P2P trader. Currently, only Commodity (Flat Price) market model is being adopted.

20. CPM Project Schedule

21. PRAGMAGrid Research Projects in Universiti Sains Malaysia e-Science Grid

22. “A meeting point for scientific communities to run scientific applications through distributed global collaborations enabled by the Internet and grid engine, using distributed resources and high performance platform.” What is an e-Science grid Portal?

23. Simulations Of Materials Scientific Informatics Mathematical & Theoretical Simulations • • • Phase II Compute- Intensive Data-Intensive Visualization & Collaboration Collaborative Resource Management Resource Monitoring Phase I Uniform Resource Access Uniform Data Access Authentication & authorization Security services Grid Information Service Brokering Co-scheduling • • • Data Storage Special Instrument Fast-Ethernet based PC Cluster e-Science Grid The Complete Framework Layer 4: Portal & Application Iterative Solver Parallel Molecular Modeling Layer 3: Application Toolkit Data Management Security Layer 2: Core Grid Engine Globus Layer 1: Infrastructure Short-Area- Network based PC Cluster Existing Server/ Super Computer

24. Conceptual Model of e-Science grid • Service-oriented grid • Provider – organization provides services to the grid • Economic-based grid • Based on the compute power market • Contract – agreement on services usage between provider and consumer • Marketplace – intermediate/ communication medium for both parties (organizations)

25. I O I I I O O Market Market O Level 1 G G Level 2 I I I Difference between P2P and grid E-Science GRID CPM/p2p O = Organization, etc USM, UKM, UTM G = Group (group by Individuals in an organization) = Individual user, etc: grid007.cs.usm.my

26. Visit us: grid.cs.usm.my/portal/index.jsp

27. Myproxy Server Compute Resources with Globus deployed Information Services LDAP servers Client browser Web Server Portal Framework GSI HTTPS HTTPS, Globus Services LDAP Tomcat, Java, COG kit, GPDK

28. Applications Replica Catalog e-Science Portal Replica Management Service File Transfer Job Manager Intrusion Detection Service Scheduler Specific plug-in Bill Directory Service E-Science Grid Architecture User Mobile Agent Resources Account Manager Resource Usage Tracking Agent

29. RESOURCE MONITORING Resource Monitor GRID RESOURCE ALLOCATION Overall Framework RESOURCE ALLOCATION Resource Broker Available resources? Grid Scheduler Cost? Bill? Resource Usage? Trader Metering Replica Management Data? (by agent) Select resources Sends agents RESOURCES ON GRID

30. Framework (newly proposed) grid.cs.usm.my - Resource Broker - GRAM - DUROC Job Submission request via http GRAM Gatekeeper Job manager Condor GRAM Gatekeeper Job manager Condor Condor pool PBS pool Standalone PC’s GRAM Gatekeeper Job manager fork

31. Resource Broker Framework Resource broker User request to RSL mapper Job Description in RSL Resource request Resource Monitoring Matchmaker Available resources Job + resource description RSL Resource Reservation Dispatch job Dispatch job Dispatch job Condor pool PBS pool Individual resources

32. Consumer Provider Administrator Metering Design Cost? Budget/Deadline Charging Trading Accept? No Yes Credit >= Budget? Yes No Job Scheduling Message top-up Done? No Usage Tracking Message failure Yes Accounting Billing

33. REPLICA MANAGEMENT Overall Framework Information Services (4) List of available resources (5) Conf. of each resources Replica Catalog (2) List of proteins Replica Selection (1) List of proteins to dock (3) List of available resources to a ligand Not replicated protein files Selected replicas Compute Resources Selected replicas Set of replicated PDB Grid Environments

34. Resource Monitoring User Access Visualization Metering (Project 2) e-Sciences Grid Portal (Dynamic Information Services) Grid Scheduler (Project 2) Other Applications (Project 5) Processed Data Authorized Prediction Platform Type, Operating System, CPU, Memory, Network, File System, Job Status Resource Monitor Event Publication Information Dispatch Agent Directory Service Mobile Agents Facility Invitation/Correction Event Publication Information Resources

35. Focused Area Resource Monitoring Status Monitoring Event Archive Resource Monitor Job Monitoring Mobile Agents Facility Reverse Registering Directory Service Resource Discovery Fault Correction Resources

36. Prediction Module Network Weather Service Network Monitoring Name Server network measurement Sensor Persistent State Forecaster e-Science Grid forecasts forecasts E-Science Portal Client

37. Milestones • Phase I(June - August 2002)– Final Model and Design • Phase II(Sep 2002 - May 2003)– Major Implementation & Integration

38. Manpower • 4 lecturers • 9 research officers • 5 MSc. students • 5 BSc. students

39. Resources (Machines) • 16-node clusters x 2 • Pentium-4 dual-processor servers x 2 • Pentium-4 high-end workstations x 8 • Pentium-4 PCs x 29 • Pentium-3 PCs x 3

40. Visit us at: http://grid.cs.usm.my

41. Thank You