Batch Scheduling at CERN (LSF)

Batch Scheduling at CERN (LSF) Hepix Spring Meeting 2005 Tim Bell IT/FIO Fabric Services

End User Clusters 950 Batch Servers “lxbatch” lxbatch001 lxbatch001 DNS-like load balancing lxbatch001 lxbatch001 lxbatch001 lxbatch001 lxbatch001 lxbatch001 lxbatch001 lxb0001 lxplus001 lxplus001 80 Tape Servers lxplus001 LSF CLI or Grid UI lxplus001 lxplus001 lxplus001 rfio lxplus001 lxplus001 lxplus001 400 Disk Servers tape001 rfio tpsrv001 44 Interactive Servers “lxplus” disk001 lxfsrk123 Batch Scheduling at CERN Tim.Bell@cern.ch

Hard/software of clusters • Typically we make two acquisitions per year • Interactive login: • 44 lxplus dual 2.8 GHz, 2GB memory, 80GB disk, slc3 • Batch worker nodes - /pool is working space: • 85 lxbatch dual 800MHz, 512MB memory, 8GB /pool,redhat7 • 86 lxbatch dual 1 GHz, 1GB memory, 8GB /pool, slc3 • 538 lxbatch dual 2.4 GHz, 1GB memory, 40GB /pool, slc3 • 226 lxbatch dual 2.8 GHz, 2GB memory, 40GB /pool, slc3 • On order 225 lxbatch dual 2.8GHz, 2GB memory, 120GB disk, slc3 Batch Scheduling at CERN Tim.Bell@cern.ch

Current CERN LSF queues • View queue information • bqueues [options] [queue-name] • % bqueues • QUEUE_NAME PRIO STATUS MAX JL/U JL/P JL/H NJOBS PEND RUN SUSP • grid_dteam 20 Open:Active - - - - 0 0 0 0 • grid_cms 20 Open:Active - - - - 0 0 0 0 • grid_atlas 20 Open:Active - - - - 226 21 205 0 • grid_lhcb 20 Open:Active - - - - 0 0 0 0 • grid_alice 20 Open:Active - - - - 0 0 0 0 • lcgtest 20 Open:Active - - - - 0 0 0 0 • system_test 10 Open:Active - - - - 0 0 0 0 • 8nm 7 Open:Active - 120 - - 3487 3472 15 0 • 1nh 6 Open:Active - 120 - - 2218 2104 114 0 • 8nh 5 Open:Active - 100 - 3 5330 5004 326 0 • cmsprs 5 Open:Active 60 75 - 3 98 86 12 0 • 1nd 4 Open:Active - - - 2 6894 6230 664 0 • 2nd 4 Open:Active - - - 2 3212 2686 526 0 • 1nw 3 Open:Active - - - 1 4623 4257 366 0 • prod100 2 Open:Active - 100 - 2 146 120 26 0 • cmsdc04 2 Open:Active - 600 - 2 0 0 0 0 • prod200 1 Open:Active - 250 - 2 1611 1561 50 0 • prod400 1 Open:Active - 1000 - 2 5773 5301 472 0 Batch Scheduling at CERN Tim.Bell@cern.ch

Queues are based on cputime requirements • 8nm 8 normalised minutes • 1nh 1 normalised hour • 8nh 8 normalised hours • 1nd 1 normalised day • 1nw 1 normalised week • The normalisation changes every few years as machines get faster. We recently converted to kilo-specint2000 units so that one cpu hour on a 2.8GHz farm PC is almost exactly one normalised hour (they are rated at 1.037 KSI2K). • Additional low priority production queues for reserved users with high job concurrency allow 1nw • Grid queues use mapped team accounts and allow 1nw cpu time. Further refinements will be investigated. • All queues can have resource or user group restrictions Batch Scheduling at CERN Tim.Bell@cern.ch

Batch Scheduling at CERN Tim.Bell@cern.ch

Queue issues • The queue definitions are essentially empirically defined to match users requirements for turnaround times where a user could expect many short jobs per day and a few long jobs overnight. • Production queues are for low priority work where specific turnaround is not an issue. The three queues allow higher numbers of concurrent jobs at decreasing priority. • There is a grid queue for each VO but without any cpu-time granularity. • There is a local queue funded by an experiment for fast turnaround of analysis jobs. • Many experiments define subgroups of privileged users which have higher priority within the group and can run higher numbers of concurrent jobs. • We allow to schedule up to 3 shorter jobs per worker node but stop at 2 if cpu load exceeds 90%. Batch Scheduling at CERN Tim.Bell@cern.ch

Resource sharing • CERN experiments and major user groups apply annually for part of centrally funded shared resources and can fund extra capacity giving them a guaranteed share. Any unused shares are available to all. LSF schedules jobs firstly on a per user group basis to deliver the defined shares of cpu time to the group in a rolling 12 hour period. • We have currently defined (arbitrarily) 1800 shares to match our 1800 KSi2K of capacity. • There is currently an allocatable capacity reserve (but which is used in practise) of 300 shares that we use to satisfy short term requests. Batch Scheduling at CERN Tim.Bell@cern.ch

Scheduling policies • LSF supports hierarchies of user groups. • Each level of the hierarchy can have a scheduling policy : • Fairshare is used at the highest group level and by most experiments among their users. Jobs are scheduled to give equal shares in average over a certain period – currently 12+ hours. Queue priorities are ignored and more longer jobs may be started. • FCFS (First Come First Served)(default policy at CERN). Shorter queues are scheduled first. Within a group and queue a single user can block others. • Preemptive/Preemptableused at CERN on an engineering cluster only (for parallel jobs) Batch Scheduling at CERN Tim.Bell@cern.ch

Current CERN shares per user group • [lxplus058.cern.ch] > 75 ~ > bhpart SHARE • HOST_PARTITION_NAME: SHARE • HOSTS: g_share/ • SHARE_INFO_FOR: SHARE/ • USER/GROUP SHARES PRIORITY STARTED RESERVED CPU_TIME RUN_TIME • u_prod 289 95.740 0 0 95.6 0 • u_z2 100 33.333 0 0 0.0 0 • zp_prod 90 1.978 5 0 109780.2 31637 • u_vp 20 0.586 2 0 19698.6 109414 • u_vo 1 0.333 0 0 0.0 0 • harp_dydak 2 0.261 1 0 200.8 8353 • u_slap 75 0.161 11 0 766725.5 1442519 • u_za 16 0.138 10 0 191694.7 236536 • u_DELPHI 2 0.129 1 0 33828.5 14929 • u_wf 3 0.077 0 0 183887.4 0 • u_LHCB 280 0.070 263 0 1268841.6 15357023 • u_HARP 9 0.068 11 0 138163.6 353289 • u_c3 1 0.050 2 0 20586.3 35210 • zp_burst 40 0.048 77 0 1401348.9 1681989 • u_xu 2 0.043 10 0 26846.6 42004 • u_CMS 400 0.035 408 0 18033716.0 34445474 • u_vl 24 0.024 132 0 2220209.0 908173 • u_NA48 6 0.024 34 0 681000.8 85772 • u_coll 75 0.023 426 0 6378126.0 4021268 • others 20 0.016 23 0 1847487.4 4105005 • u_yt 16 0.016 177 0 2208468.0 167261 • u_l3c 5 0.016 65 0 568145.3 53442 • u_vk 1 0.010 20 0 182311.5 29401 • u_OPAL 2 0.009 9 0 124016.9 880953 • u_COMPASS 200 0.008 502 0 42877972.0 82582955 • u_zp 90 0.008 216 0 5662955.5 51679400 • u_z6 2 0.003 10 0 967997.3 1925963 Batch Scheduling at CERN Tim.Bell@cern.ch

Resource Requirement String • Most LSF commands accept a resource requirement string. • Describes the resources required by a job. • Used for mapping the job onto execution hosts which satisfy the resource request. • Queue names imply a cpu (and real time) time limit and are enough for most users. • Real time limit is hardwired per queue at a multiple (3 to 4) of the cputime limit. Batch Scheduling at CERN Tim.Bell@cern.ch

Users can request specific static resources Batch Scheduling at CERN Tim.Bell@cern.ch

Or dynamic resources Batch Scheduling at CERN Tim.Bell@cern.ch

Other dynamic Resources Batch Scheduling at CERN Tim.Bell@cern.ch

Typical batch job requirements • Beyond queue name the most commonly used resource requests are: • “Cpuf > factor” where current values are from 0.2 to 1.0. This is to avoid slower machines with long execution real times. • “Mem > MB” requesting free memory at job start. Avoids jobs running on hosts where they would be killed by our monitors or swap excessively. • “pool > MB” requests current working directory space at job start. We kill jobs that use more than 75% of a nodes work space. • Current grid UI mapping for CPU factor and pool space is unclear Batch Scheduling at CERN Tim.Bell@cern.ch

Further information • User level info is on the web at • http://cern.ch/batch which has links to live status, accounting data and basic user guides. • Live monitoring information which starts at the cluster level but can descend to individual nodes is at: • http://cern.ch/lemon-status (internal access only) Batch Scheduling at CERN Tim.Bell@cern.ch

Batch Scheduling at CERN (LSF)

Télécharger la présentation

Plus de contenu associé