SlideServe
Télécharger Obtenir Pro
Menu
Accueil Pro
Préférences
ApparenceLight Aide et commentaires
© SlideServe · v2026

YARN

tyson avatar
tyson
15 diapositives · 337 Views · Aug 23, 2014

Nagarjuna K. YARN. Why Next Generation MR. Reliability Availability Scalability - Clusters of 10,000 machines and 200,000 cores, and beyond. Backward (and Forward) Compatibility Ensure customers’ MapReduce applications run unchanged in the next version of the framework.

#yarn #yarn ppt #powerpoint #presentation
YARN - slide 1
01 / 15YARN
1 / 17

Télécharger la présentation

YARN
Une image / un lien ci-dessous est fourni (tel quel) pour télécharger la présentation

Politique de téléchargement : le contenu du site vous est fourni TEL QUEL pour votre information et votre usage personnel ; il ne peut être vendu, concédé sous licence ni partagé sur d’autres sites sans l’accord de son auteur.Le contenu vous est fourni TEL QUEL, pour votre information et votre usage personnel uniquement. Téléchargez la présentation en cliquant sur ce lien. Lors du téléchargement, si vous ne parvenez pas à télécharger une présentation, l’éditeur a peut-être supprimé le fichier de son serveur.Pendant le téléchargement, si vous ne parvenez pas à obtenir une présentation, le fichier a peut-être été supprimé par l’éditeur.

F I N

Plus de contenu associé

Transcription de la présentation

  1. Nagarjuna K YARN nagarjuna@outlook.com

  2. Why Next Generation MR • Reliability • Availability • Scalability - Clusters of 10,000 machines and 200,000 cores, and beyond. • Backward (and Forward) Compatibility • Ensure customers’ MapReduce applications run unchanged in the next version of the framework. • Evolution – Ability for customers to control upgrades to the Hadoop software stack. • Predictable Latency – A major customer concern. • Cluster utilization nagarjuna@outlook.com

  3. Why Next Generation MR • Secondary Requirements • Support for alternate programming paradigms to MapReduce. • Support for short-lived services nagarjuna@outlook.com

  4. ReArchitecure • Need • Separate the tasks of Job Tracker • Resource management • Job Scheduling / Management nagarjuna@outlook.com

  5. So, What did we come up with nagarjuna@outlook.com Resource Manager Node Manager Application Master Container

  6. Resource Manager (RM) Manages the global assignment of compute resources to applications. nagarjuna@outlook.com

  7. Resource Manager (RM) • A pure Scheduler • No monitoring, tracking status of application • No guarantee on restarting failed tasks. nagarjuna@outlook.com

  8. Resource Manager (RM) • Each client/application may request multiple resources • Memory • Network • Cpu • Disk .. • This is a significant change from static Mapper / Reducer model nagarjuna@outlook.com

  9. Application Master • A per – applicationApplicationMaster(AM) that manages the application’s life cycle(scheduling and coordination). •  An application is either a single job in the classic MapReduce jobs or a DAG of such jobs. nagarjuna@outlook.com

  10. Application Master A per – applicationApplicationMaster(AM) that manages the application’s life cycle. nagarjuna@outlook.com

  11. Application Master • Application Master has the responsibility of • negotiating appropriate resource containers from the Scheduler • launching tasks • tracking their status • monitoring for progress • handling task-failures. nagarjuna@outlook.com

  12. Node Manager • The NodeManager is the per-machine framework agent • responsible for launching the applications’ containers, monitoring their resource usage (cpu, memory, disk, network) and reporting the same to the Scheduler. nagarjuna@outlook.com

  13. Gain with New Architecture • Scalability • Availability • Wire-compatibility • Innovation & Agility • Cluster Utilization • Support for programming paradigms other than MapReduce nagarjuna@outlook.com

  14. Gain with New Architecture Scalability Availability Wire-compatibility Innovation & Agility Cluster Utilization Support for programming paradigms other than MapReduce • RM and Job manager segregated • The Hadoop MapReduceJobTracker spends a very significant portion of time and effort managing the life cycle of applications nagarjuna@outlook.com

  15. Gain with New Architecture Scalability Availability Wire-compatibility Innovation & Agility Cluster Utilization Support for programming paradigms other than MapReduce • ResourceManage • Uses ZooKeeper for fail-over. • When primary fails, secondary can quickly start using the state stored in ZK • Application Master • MapReduceNextGen supports application specific checkpoint capabilities for the ApplicationMaster. • MapReduceApplicationMaster can recover from failures by restoring itself from state saved in HDFS. nagarjuna@outlook.com