SlideServe
Télécharger Obtenir Pro
Menu
Accueil Pro
Préférences
ApparenceLight Aide et commentaires
© SlideServe · v2026

Cooperative Computing for Data Intensive Science

tamma avatar
tamma
15 diapositives · 287 Views · Sep 17, 2014

Explore the concept of cooperative computing, which combines computing and storage resources to tackle larger scientific problems. This approach allows idle computing power to be utilized effectively, advancing fields such as physics, biology, and computer science. The Cooperative Computing Lab at the University of Notre Dame showcases successful distributed systems designed to address complex research challenges, emphasizing scalability, fault tolerance, and high throughput. Join us in revolutionizing data processing and simulation through collaborative efforts in diverse scientific disciplines.

Cooperative Computing for Data Intensive Science - slide 1
01 / 15Cooperative Computing for Data Intensive Science
1 / 17

Télécharger la présentation

Cooperative Computing for Data Intensive Science
Une image / un lien ci-dessous est fourni (tel quel) pour télécharger la présentation

Politique de téléchargement : le contenu du site vous est fourni TEL QUEL pour votre information et votre usage personnel ; il ne peut être vendu, concédé sous licence ni partagé sur d’autres sites sans l’accord de son auteur.Le contenu vous est fourni TEL QUEL, pour votre information et votre usage personnel uniquement. Téléchargez la présentation en cliquant sur ce lien. Lors du téléchargement, si vous ne parvenez pas à télécharger une présentation, l’éditeur a peut-être supprimé le fichier de son serveur.Pendant le téléchargement, si vous ne parvenez pas à obtenir une présentation, le fichier a peut-être été supprimé par l’éditeur.

F I N

Plus de contenu associé

Transcription de la présentation

  1. Cooperative Computing for Data Intensive Science Douglas Thain University of Notre Dame NSF Bridges to Engineering 2020 Conference 12 March 2008

  2. What is Cooperative Computing? • By combining our computing and storage resources together, we can attack problems larger than we could alone. • I can use your computer when it is idle, and vice versa. (Most computers are idle about 90 percent of the day.) • Also known as… • Grid computing, distributed computing, metacomputing, volunteer computing, etc…

  3. Who Needs Coop Computing? • Many fields of study rely on simulation and data processing to conduct science. • Physics, chemistry, biology, engineering, finance, sociology, computer science. • More Computing == Better Results • NOT High Performance: Speed up one program. • High Throughput: Produce as many results as possible over the next day / week / year.

  4. Cooperative Computing Lab • We design and build distributed systems that helps people to attack BIG problems. • Work directly with end users to make sure that our solutions affect the real world. • Operate a modest computing system as both a production service and a research testbed. • Currently about 500 cpus and 300 disks. • CS Research challenges: scalability, robustness, usability, debugging, and performance. http://www.nd.edu/~ccl

  5. What Makes this Challenging? • The Programming Model • I want to process 10 TB of data on 100 machines, then distribute it across 20 disks, then view the best results on my workstation. • Fault Tolerance • Something is always broken! • Performance Robustness • There is always one slowpoke. • Debugging • My job runs correctly here but not there...!?

  6. An Example Collaboration:Biometrics ResearchandDistributed Systems

  7. F A Common Pattern in Biometrics Sample Workload: 4000 images 256KB each 1s per F 185 CPU-days Future Workload: 60000 images 1MB each 0.1s per F 4166 CPU-days

  8. Try 1: Each F is a batch job. Failure: Dispatch latency >> F runtime. Try 2: Each row is a batch job. Failure: Too many small ops on FS. F F F F F CPU CPU CPU CPU CPU F F F F F F F F F F CPU F CPU F CPU F CPU F CPU F F F F F F HN HN Try 3: Bundle all files into one package. Failure: Everyone loads 1GB at once. Try 4: User gives up and attempts to solve an easier or smaller problem. F F F F F F F F F F CPU F CPU F CPU F CPU F CPU F F F F F F HN Non-Expert User Using 500 CPUs

  9. All Pairs Production System 300 active storage units 500 CPUs, 40TB disk Web Portal F G H 4 – Choose optimal partitioning and submit batch jobs. S T F F F 1 - Upload F and S into web portal. 2 - AllPairs(F,S) F F F All-Pairs Engine 6 - Return result matrix to user. 5 - Collect and assemble results. 3 - O(log n) distribution by spanning tree.

  10. Some Results on Real Workload

  11. Collaboration is Where the Interesting Problems Are! (Cooperative ComputingProvides the Resources)

  12. What Makes a Collaboration Work? • Like a marriage? (old joke.) • First, a show of commitment: go after some low hanging fruit, and publish it. • A proposal for funding only succeeds if you have already started working together. • Need very concrete goals: your partner may not share your idea of an interesting tangent. • Students sometimes need a big push to leave their comfort zone and work together.

  13. For more information… • Douglas Thain • dthain@nd.edu • Cooperative Computing Lab • http://www.nd.edu/~ccl • Apply for Summer 2008 REU: • http://www.nd.edu/~ccl/reu Supported by NSF Grants CCF-0621434 and CNS-0643229.