1 / 7

Nikhef High Mem Observations

Investigating high memory problems in Normal LHC jobs at Nikhef, addressing Pvmem limits, resource inconsistencies, and the link between memory and cores. The discussion includes handling out-of-memory signals, requesting more memory, and Maui's capability in handling high memory jobs.

jolene
Télécharger la présentation

Nikhef High Mem Observations

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Nikhef High Mem Observations Jeff Templon GDB 2014.02.12

  2. Which High Mem Problem? • Normal LHC jobs are somewhat high mem … Nikhef limit is 4096mb pvmem Jeff Templon, Multicore TF #2

  3. PVMEM • Pvmem limit? • Translates to ulimit on process • Process gets out-of-mem signal instead of kill • Chance to handle • Room for overhead outside of app space • We set this at queue level … job default, apps don’t have to do anything. Jeff Templon, Multicore TF #2

  4. What if you need more? • Can ask for it • Most ways that should work, don’t • Dennis van Dok's investigations • This does “work”: • CERequirements = "other.GlueHostMainMemoryRAMSize >= 8192” • However, not consistent … Jeff Templon, Multicore TF #2

  5. Inconsistency Resource_List.mem= 8192mb Resource_List.neednodes= 1:ppn=5 Resource_List.nodect= 1 Resource_List.nodes= 1:ppn=5 Resource_List.pvmem= 2936012800b • Pvmemlimis 2800mb from ‘infra’ queue (dteam) data seg size (kbytes, -d) 8388608 max memory size (kbytes, -m) 8388608 cputime (seconds, -t) unlimited virtual memory (kbytes, -v) 8398848 Jeff Templon, Multicore TF #2

  6. Link between mem & cores • Current : ask for “mem” (not vmem) then get (mem / 2GB) cores • No longer so relevant … • 2 / 3 / 4 / 8 GB phys per node in service • Maui is capable of steering high mem jobs to right nodes • Could accept up to 8G / core as long as these jobs are a minority • Currently reviewing situation Jeff Templon, Multicore TF #2

  7. BDJ vs many processes • Maui doesn’t understand 1st one • BDJ needs 32 GB and 8 cores • Maui looks for one node with 256 GB and 8 cores freed (32 GB per core) • Look for info : mostly find other people with same questions • Not clear at this point whether it’s worth source spelunking to figure it out. Jeff Templon, Multicore TF #2

More Related