Design and Evaluation of a Resource Selection Framework for Grid Applications

1. Design and Evaluationof a Resource Selection Frameworkfor Grid Applications University of Chicago

2. Problem • Co-selection of resources • Resource Selection • Resource configuration • Mapping workload to resources

3. Outline • Problem • > Related work • Set Match • Resource Selection Service • Experiments • Summary • Future work

4. Related work • Networked Batch Queue System • PBS, LSF, NQE • Application-specific scheduler • AppLeS, etc. • Condor

5. Outline • Problem • Related work • > Set Match • Resource Selection Service • Experiments • Summary • Future work

6. Set Match • Resource Set: • Several resources with similar characteristics that may be located in different sites. • Set Match • A Set Match occurs between a resource request and a resource set. If the resource set fulfill the resource request and the resource request fulfill resource’s management policy, they match each other.

7. Set Match • Matchmaker

8. Set Match • ClassAds Language The classads language is a language for expressing and evaluating attributes. An attribute is a named expression. A classad is a set of named expressions. Expression similar to those found in C/C++. B=[ name=”foo”; type="machine"; cpuspeed=800M; memory=512M] A=[ Requirements = other.type = = "machine" && other.cpuspeed > 500M; Rank = other.cpuspeed ]

9. Set Match • Set-Extended ClassAds language • Aggregation functions • Max(), Min(), Sum(), SizeOfSet() • Other functions • suffix(S, L), Prefix(S, L)

10. Set Match An Example of the Set Extended ClassAds [ Type=”Set”; Domainlist={ucsd.edu, cs.uiuc.edu}; requirements = Sum(other.memory) > 1G && other.cpuspeed > 200M && suffix(other.Hostname, Domainlist); rank = Min(other.cpuspeed)*SizeOfSet() ]

11. Resource Selection mechanism

12. Resource Selection mechanism

14. Outline • Problem • Related work • Set Match • > Resource Selection Service • Experiments • Summary • Future work

15. Resource Selection Service • Help applications to choose a good Resource Set in Grid environment. • Synchronous and asynchronous service • Mapping application workload to resources if needed.

16. Resource Selection Service • Framework

17. Resource Request • Owner: The sender of this request. • Type of Service: Synchronous or asynchronous. • Job description: The characteristics of the job to be run, for example, problem size, problem specification and the performance model. • Job resource requirements: User resource requirements, for example, memory capability, type of operating system, software packages installed, etc. • Mapper: The kind of mapper algorithm to be used. • Rank: The criteria to rank the matched resources.

18. Resource Request: An example Type of Service 1. [ Service = "InstantService"; 2. iter=100; alpha=100; x=100; y=100; z=100; 3. computetime = x*y*alpha/other.cpuspeed*370; 4. comtime= ( other.RLatency+ y*x*254/other.RBandwidth +other.LLatency+y*x*254/other.Lbandwidth); 5. exectime=(computetime+comtime)*iter+startup; • Mapper = [type ="dll"; libraryname="cactus"; function="mapper"]; 7. requirements = Sum(other.MemorySize) >= (1.757 + 0.0000138*z*x*y) && suffix(other.machine, domains); 8. domains={ cs.utk.edu, ucsd.edu}; 9. rank=Min(1/exectime) ] Job description Mapper Resource Requirements Rank

19. Resource Selection Result <virtualMachine> <result statusCode="200" statusMessage="OK"/> <machineList> <machine dns="torc2.cs.utk.edu" processor= 2 x= 20> <machine dns="torc3.cs.utk.edu" processor= 2 x= 15> <machine dns="torc6.cs.utk.edu" processor= 2 x= 15> </machineList> </virtualMachine>

20. Outline • Problem • Related work • Set Match • Resource Selection Service • > Experiments • Summary • Future work

21. Cactus application • Application performance model • ExecTime= (Communication time + Computation time) * slowdown • Application mapping algorithm • 1 Dimension mapping R 1

22. Cactus application • Application performance model • ExecTime= (Communication time + Computation time) * slowdown • Application mapping algorithm • 1 Dimension mapping R 1 R 2

23. Cactus application • Application performance model • ExecTime= (Communication time + Computation time) * slowdown • Application mapping algorithm • 1 Dimension mapping R 1 R 2 R 3

24. Cactus application • Application performance model • ExecTime= (Communication time + Computation time) * slowdown • Application mapping algorithm • 1 Dimension mapping R 1 R 2 R 3

25. Execution Time Prediction Time (S) Torc1,3, 5, o.ucsd.edu Torc1,3, 5 Torc1 Torc5 Torc1,3, 5 Torc1 Torc5 torc1 Torc1 Torc5 Problem Size

26. Resource Selection Algorithm Test • 1. o.ucsd.edu, mystere.ucsd.edu, saltimbanco.ucsd.edu • 2: mystere.ucsd.edu, o.ucsd.edu • 3: o.ucsd.edu, Saltimbanco.ucsd.edu 4. o.ucsd.edu • 5: saltimbanco.ucsd.edu 6: mystere.ucsd.edu

27. Resource Selection Algorithm Test • 1: torc6.cs.utk.edu 2: o.ucsd.edu 3: Saltimbanco.ucsd.edu • 4. Torc6.cs.utk.edu + o.ucsd.edu 5: o.ucsd.edu + saltimbanco.ucsd.edu • 6. o.ucsd.edu, mystere.ucsd.edu, torc6.cs.utk.edu

28. Outline • Problem • Related work • Set Match • Resource Selection Service • Evaluation of the implementation • > Summary • Future work

29. Summary • Extended the ClassAds language to describe the requirement for a Resource Set • Implement a Set Matchmaker • Create a general framework for the Resource Selection Service • Validate our implementation by the Cactus application

30. Future Work • Extend the resource selection algorithm to support other applications. • Gang match • Resource with particular topology • Provide fault tolerance to incomplete or error information • A general matchmaking mechanism

31. Welcome comments and suggestions Thank you!

32. Problem • Co-selection of resources • Dynamic nature of Grid resources • Heterogeneity of resources • The autonomy of resources • The quality of resources depends on the application’s run characteristics