1 / 41

WLCG Transfers Dashboard A unified monitoring tool for heterogeneous data transfers .

WLCG Transfers Dashboard A unified monitoring tool for heterogeneous data transfers . Alexandre Beche < Alexandre.beche@cern.ch>. Outlines. Mission History of the data transfers monitoring tools Generic monitoring workflow New WLCG Transfers Dashboard Summary. Mission. ATLAS

taran
Télécharger la présentation

WLCG Transfers Dashboard A unified monitoring tool for heterogeneous data transfers .

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. WLCG Transfers Dashboard A unified monitoring tool for heterogeneous data transfers. AlexandreBeche <Alexandre.beche@cern.ch>

  2. Outlines Mission History of the data transfers monitoring tools Generic monitoring workflow New WLCG Transfers Dashboard Summary Alexandre Beche – White Area

  3. Mission ATLAS FTS + XRootD CMS FTS + XRootD LHCb FTS ALICE XRootD Alexandre Beche – White Area

  4. Mission ATLAS FTS + XRootD CMS FTS + XRootD LHCb FTS ALICE XRootD Alexandre Beche – White Area

  5. Mission ATLAS FTS + XRootD CMS FTS + XRootD LHCb FTS ALICE XRootD Alexandre Beche – White Area

  6. History of data transfers monitoring tools WLCG Transfers Dashboard WEB API / UI WLCG FTS Alexandre Beche – White Area

  7. History of data transfers monitoring tools WLCG Transfers Dashboard WEB API / UI XRootD integration WLCG FTS FAX AAA Alexandre Beche – White Area

  8. History of data transfers monitoring tools WLCG Transfers Dashboard WEB API/UI WEB API/UI WEB API / UI XRootD integration XRootD Detailled Dashboard WLCG AAA FAX FTS FAX AAA Alexandre Beche – White Area

  9. History of data transfers monitoring tools WLCG Transfers Dashboard WEB API/UI WEB API/UI WEB API / UI XRootD integration XRootD Detailled Dashboard WLCG AAA FAX EOS integration FTS FAX EOS AAA EOS Alexandre Beche – White Area

  10. History of data transfers monitoring tools WLCG Transfers Dashboard WEB API/UI WEB API/UI WEB API / UI XRootD integration XRootD Detailled Dashboard WLCG AAA FAX EOS integration ALICE integration FTS FAX EOS AAA EOS ALICE Alexandre Beche – White Area

  11. Generic monitoring workflow AMQ AMQ Raw & Statistics WEB UI WEB API Consumer Alexandre Beche – White Area

  12. From ActiveMQ to Oracle • Load balanced collector • Horizontal scaling Consumer AMQ DB Consumer Alexandre Beche – White Area

  13. From ActiveMQ to Oracle • Load balanced collector • Horizontal scaling Consumer AMQ DB Stompclt* * Developed by the CERN messaging team Alexandre Beche – White Area

  14. From ActiveMQ to Oracle • Load balanced collector • Horizontal scaling Consumer AMQ DB Stompclt* Disk queue* * Developed by the CERN messaging team Alexandre Beche – White Area

  15. From ActiveMQ to Oracle • Load balanced collector • Horizontal scaling Consumer AMQ DB Stompclt* DB inserter • Customizable inserter • Data enhancement • Filtering Disk queue* * Developed by the CERN messaging team Alexandre Beche – White Area

  16. From ActiveMQ to Oracle • Load balanced collector • Horizontal scaling Consumer AMQ DB Simplevisor* Stompclt* DB inserter • Customizable inserter • Data enhancement • Filtering Disk queue* * Developed by the CERN messaging team • Modular architecture • Common building blocks (EPEL) • Reliable Alexandre Beche – White Area

  17. Raw data aggregation • Well optimized storage • Partitioned table • No global indexes (including the PK) Alexandre Beche – White Area

  18. Raw data aggregation • Well optimized storage • Partitioned table • No global indexes (including the PK) • Aggregation of unordered events • Stateless: Full re-computation of touched bins each time • Compute stats from raw data in 10 min bins • Aggregate 10 min stats in daily bins Alexandre Beche – White Area

  19. Aggregation methods Transfers 2pm 3pm 4pm 5pm 6pm 7pm Alexandre Beche – White Area

  20. Aggregation methods Easy method Transfers 2pm 3pm 4pm 5pm 6pm 7pm Alexandre Beche – White Area

  21. Aggregation methods Easy method Transfers 2pm 3pm 4pm 5pm 6pm 7pm Alexandre Beche – White Area

  22. Aggregation methods Easy method Transfers 2pm 3pm 4pm 5pm 6pm 7pm Alexandre Beche – White Area

  23. Aggregation methods Easy method Transfers 2pm 3pm 4pm 5pm 6pm 7pm Adopted method • Both method equivalent if: • Small transfers or many transfers Alexandre Beche – White Area

  24. A well defined web API Raw & Statistics DAO Collection of SQL queries Application specific Generic • Legacy dashboard framework: • Use the MVC design pattern (based on Struts) Alexandre Beche – White Area

  25. A well defined web API Raw & Statistics Run a parameterized query, Return the results in a dict DAO Action Collection of SQL queries Application specific Generic • Legacy dashboard framework: • Use the MVC design pattern (based on Struts) Alexandre Beche – White Area

  26. A well defined web API Raw & Statistics Run a parameterized query, Return the results in a dict DAO Action Collection of SQL queries Views Json, xml, … Python dict to #format# converter Application specific Generic • Legacy dashboard framework: • Use the MVC design pattern (based on Struts) Alexandre Beche – White Area

  27. A well defined web API Raw & Statistics Run a parameterized query, Return the results in a dict DAO Action Controller Router of url+mime To action + view Collection of SQL queries Views Json, xml, … Python dict to #format# converter Application specific Generic • Legacy dashboard framework: • Use the MVC design pattern (based on Struts) Alexandre Beche – White Area

  28. A well defined web API Raw & Statistics Run a parameterized query, Return the results in a dict DAO Action Controller Router of url+mime To action + view Collection of SQL queries Views Json, xml, … Python dict to #format# converter Application specific Configuration List of actions Generic • Legacy dashboard framework: • Use the MVC design pattern (based on Struts) Alexandre Beche – White Area

  29. A well defined web API Raw & Statistics Run a parameterized query, Return the results in a dict Django under investigation DAO Action Controller Router of url+mime To action + view Collection of SQL queries Views Json, xml, … Python dict to #format# converter Application specific Configuration List of actions Generic • Legacy dashboard framework: • Use the MVC design pattern (based on Struts) Alexandre Beche – White Area

  30. Visualization layerDashboard UI Rich web single-page user interface AJAX+JSON communication Alexandre Beche – White Area

  31. Visualization layerDashboard UI Rich web single-page user interface AJAX+JSON communication jQuery + popular plugin technology cocktail Alexandre Beche – White Area

  32. Visualization layerDashboard UI xbrowse • Rich web single-page user interface • AJAX+JSON communication • jQuery + popular plugin technology cocktail • Modular code reuse with xbrowse framework • Lightweight in-house URL-driven MVC (<300 LOC) • Views shared with several other applications • Common look and feel Alexandre Beche – White Area

  33. History of data transfers monitoring tools WLCG Transfers Dashboard WEB API/UI WEB API/UI WEB API / UI XRootD integration XRootD Detailled Dashboard WLCG AAA FAX EOS integration ALICE integration FTS FAX EOS AAA EOS ALICE Alexandre Beche – White Area

  34. WLCG Transfers Dashboardfederated approach WEB API/UI WEB API WEB API/UI WEB API / UI ALICE FTS AAA FAX No duplication anymore EOS could be included FTS FAX EOS AAA EOS ALICE Alexandre Beche – White Area

  35. WLCG Transfers Dashboardfederated approach WLCG Transfers Dashboard API / UI WEB API/UI WEB API WEB API/UI WEB API / UI ALICE FTS AAA FAX No duplication anymore EOS could be included FTS FAX EOS AAA EOS ALICE Alexandre Beche – White Area

  36. Some plots Alexandre Beche

  37. New WLCG Transfers Dashboard Aggregate Main thread Get results Return Async call 1 Async call 2 • Multi-threaded HTTP Call • Thread pool • Async request (std::future c++11) Alexandre Beche – White Area

  38. Not just a re-writing WEB API/UI WEB API / UI FTS FAX FTS FAX EOS Possible re-think of UI Alexandre Beche – White Area

  39. Not just a re-writing ATLAS Monitoring UI WEB API/UI WEB API WEB API / UI WEB API FTS FTS FAX FAX FTS FTS FAX FAX EOS EOS • Possible re-think of UI • VO specific UI Alexandre Beche – White Area

  40. Summary Thanks for your attention • More flexible architecture • Database-less • Based on common building blocks • UI code fully re-used Alexandre Beche – White Area

  41. Useful links • AAA Dashboard • http://dashb-cms-xrootd-transfers.cern.ch • FAX Dashboard: • http://dashb-atlas-xrootd-transfers.cern.ch • FTS Dashboard: • http://dashb-fts-transfers/ui/ • WLG Transfers Dashboard: • http://dashb-wlcg-transfers-new/ui/# • Xbrowse framework: • https://twiki.cern.ch/twiki/bin/view/ArdaGrid/XbrowseFramework Alexandre Beche - ITTF

More Related