1 / 11

REcent Developments

Recent developments in GaudiPython bring multiprocessing and synchronization improvements, such as JoinableQueues and Event() class, enhancing parallel processing capabilities. New algorithms and scripts improve support and analysis, with a focus on parallelism for IPC and event streaming. These updates are part of the pytools package version 1.1. Explore the enhanced parallel model and performance boosts for LCG applications.

merrillg
Télécharger la présentation

REcent Developments

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. REcent Developments • GaudiPython / WP8

  2. GaudiPython parallel

  3. lcg developments • multiprocessing module added to LCG: • Available in LCG Externals • part of pytools package version 1.1 • requirements line : use pytools v1.1 LCG_Interfaces • LCG Config may be off... replace 1.0_python2.5 with 1.1_python2.5 in PYTHONPATH • setenv PYTHONPATH ${PYTHONPATH}:/afs/cern.ch/sw/lcg/external/pytools/1.1_python2.5/x86_64-slc5-gcc43-opt/lib/python2.5/site-packages

  4. developments • Move to multiprocessing complete: • JoinableQueues allow for better synchronisation • both ends guaranteed to stay alive until Queue is empty • Event() class allows for better synchronisation • Impossible to hang on a Queue wait • added Syncer class to ensure no hanging • SMAPS : • Add SmapSnapshot Algorithm to GaudiPython Parallel Model

  5. developments • Scripts for full support and analysis : • Reorder.py -p par -r refOptions • Reorders Parallel output based on the original run options • Allows better comparison to serial output for verification • FileCompare.py -p par -s ser • Compares Output Files by contained ROOT TTrees • lists differences in decreasing size order

  6. developments • Documentation for project • model • results : UnitTest script developed for all applications • TESSerializer details

  7. p0 : 0.0002p1 : 5.775e-6 T Event Size (kB) TESSerializer performancebrunel fest09 1000evStandard lxplus node (8core 2.33GHz x86_64-slc5-gcc43-opt)

  8. parallel workshop • Outcomes : • Modify OutputStream/PersistencySvc in Gaudi specially for Event IPC • IPCSvc.h/cpp added to PersistencySvc • modelled on existing OutputStream • objects collected and dumped to a member TBufferFile • IIPCSvc.h added to GaudiKernel • dumpObject() • loadBuffer() • syntax change : --ncpus=N, --ncpus=-1 • Combine 2 approaches : • Athena MP fork-all-workers / GaudiPython single output stream

  9. parallel SCHEMA

  10. GP parallel : cluster • Cluster-version of GaudiPython • Using pp-1.5.7 and pyssh modules (Not LCG) • possibly update from pyssh to pssh (maintained) • Open SSH session to a (given) list of hosts • Event Parallelism and message passing, instead of single Task with return value

  11. Gp parallel : cluster • Test mechanism implemented : • “GaudiServer” running on remote node • modified ppserver... • added Worker Classes/process, modifed processing of incoming data • environment prepared as part of SSH Session setup • GaudiServer forks one/many Worker Processes, but continues to listen on a single socket • Events can be streamed to socket as TBufferFiles, and then queued to Workers • Concept model... just working in scripts!

More Related