1 / 13

Optimizing Sparse Trees in FSRs for Efficient File Handling

This update from R. Lambert details solutions to fix frustrating summary records files in Core Soft. Suggestions include tuning sparse trees, addressing buffer and basket sizes, and optimizing memory usage. The text outlines test cases, buffer optimizations, and results, emphasizing the importance of efficient file management for stability. It concludes with discussions on merging strategies and potential alternatives to improve performance.

gudrun
Télécharger la présentation

Optimizing Sparse Trees in FSRs for Efficient File Handling

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. FSR – frustrating summary records file ^ Update R. Lambert Core Soft, 10th October 2012

  2. Possible Solution • Fixing any of these will fix FSRs for good • Get sparse trees in Root to work properly • Have basket size and optimization tuneable tree-by-tree • Markus realized this is possible in LHCb code • Available in head or Online/RootCnv … let’s try it … 10,000 FSRs Sparse Trees in TES Sparse Trees Persisted ROOT Tree used ROOT can’t handle sparse trees Core Soft, 10th October 2012

  3. Two Parameters • Buffer: • Reserved memory for each leaf in the Tree • Basket: • Supposed sizes ofall the leaves Core Soft, 10th October 2012

  4. Test Cases A B Real Data 13475 FSRs, 4 levels MC Filtered 40 FSRs, 7 levels Lxplus 135 Only me using it My code on local disk LHCb code on cvmfs Write to local disk Read from Castor Verified stability of CPU/Vmem CPU : 10% Vmem: 0.5% File size: 0% Core Soft, 10th October 2012

  5. Buffer A B Core Soft, 10th October 2012

  6. Buffer Original RootCnv A B Core Soft, 10th October 2012

  7. Buffer Markus/Ivan optimized A B Core Soft, 10th October 2012

  8. Buffer Suggested optimum A B Core Soft, 10th October 2012

  9. Basket Completelynot relevant A B Core Soft, 10th October 2012

  10. Results • Only minor improvements possible! SetupProjectDaVinci v32r2p1 time gaudirun.py ~rlambert/public/forMarkus/fsrMemLeak/options.py #100 files with one event each, but 13475 FSRs in total Core Soft, 10th October 2012

  11. Summary • Re-optimizing for FSRs not as good as we hoped • Writing *any* FSRs causes a leap in Vmem and CPU time • Need to keep merging, such that we never have 10k FSRs • Even 40 FSRs are a problem if buried under many levels • Again we can turn to these other options, but they are all very difficult … • Automatic merging • Event-like FSRs • Flattening (Three possible directions) • Abandon trees altogether Core Soft, 10th October 2012

  12. End • Backups are often required Core Soft, 10th October 2012

  13. Discussion Core Soft, 10th October 2012

More Related