1 / 23

Towards an Open Source Project for Online Software

Towards an Open Source Project for Online Software. Bob Jones CERN EP/atd Robert.Jones@cern.ch. Brief overview of ATLAS DAQ/EF Prototype -1 Project Structure and organisation of the Back-end DAQ sub-system Lessons learnt Proposal to the HENP community:

conan-dickerson
Télécharger la présentation

Towards an Open Source Project for Online Software

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Towards anOpen Source Project forOnline Software Bob Jones CERN EP/atd Robert.Jones@cern.ch

  2. Brief overview of ATLAS DAQ/EF Prototype -1 Project Structure and organisation of the Back-end DAQ sub-system Lessons learnt Proposal to the HENP community: An Open Source project for online software Issues involved How to proceed Summary Contents Towards an Open Source Project for Online Software - CHEP2000 - Bob Jones

  3. ATLAS DAQ/EF Prototype -1 • Goal • to produce a prototype DAQ system for evaluating candidate technologies and architectures. 1996-2000 • Organisation • functionality divided into sub-systems • Information • overview, tech. notes, conf. papers etc. • http://atddoc.cern.ch/Atlas/ Detector Interface Data Flow Event Filter Back-end Software Towards an Open Source Project for Online Software - CHEP2000 - Bob Jones

  4. Back-end Sub-system • Excludes the processing and transportation of physics data • Interfaces to: • Data-flow • event builder, ROB and • detector ROD crates • Triggers • Lvl1, Lvl2, event filter • DCS • SCADA Software for configuring, controlling & monitoring the online system It is the software glue of the experiment Towards an Open Source Project for Online Software - CHEP2000 - Bob Jones

  5. Back-end Software Issues • How to... • Start, stop & synchronize 100s programs on 100s processors? • Configure the programs for many data taking modes? • Identify and report problems when running? • Integrate software from many different sub-systems • While... • Minimizing the time taken to start/stop a run • Simplifying the information presented to the operator • Given that... • Rapidly evolving environment (e.g. UNIX/WNT, Java/C++) • Not all software can be produced at CERN Towards an Open Source Project for Online Software - CHEP2000 - Bob Jones

  6. Back-end Organisation Collaboration of some institutes in ATLAS Trigger-DAQ system • Univ. of Innsbruck, Austria • IN2P3 Marseille, France • NIKHEF, Netherlands • LIP, Portugal • IAP Bucharest, Romania • JINR Dubna, Russia • PNPI St. Petersburg, Russia • CERN, Switzerland • Univ. of Geneva, Switzerland Functionality divided into a number of components Institutes responsible for the development and maintenance of components within a well-defined framework Towards an Open Source Project for Online Software - CHEP2000 - Bob Jones

  7. Back-end Components Towards an Open Source Project for Online Software - CHEP2000 - Bob Jones

  8. Back-end Design Approach • Activities • Concentrate development effort on domain specific items • Use external software when possible • Use common solutions across all components • Adhere to relevant standards (e.g. OMG, ANSI etc.) • Design for portability from the start • Use Software engineering principles, OO techniques and follow a well-defined software process Towards an Open Source Project for Online Software - CHEP2000 - Bob Jones

  9. SW Development Process • Not formally defined in a document but web pages • - more dynamic • - less imposing • Basic but practical • Used for all the components Phases Reviewed Deliverables Collect Requirements Requirements Evaluation report(s) Pre-design Investigations High-level Design High-level Design Code Users Guide Test Plan Impl. Note Detailed DesignImplementation Testing &Integration Test programs Test report Software Development Environment Training, Programming and Testing Tools, FrameMaker, html, SRT Configuration Management, StP/OMT CASE TOOLS Towards an Open Source Project for Online Software - CHEP2000 - Bob Jones

  10. Back-end Status • All components developed and deployed (except Evt Dump) • 8 releases • 0.0.0 - Jan’99 … 0.0.7 - Dec’99 • Integrated with data-flow • use in test-beam summer 2000 • Evaluated and accepted by Level-1 & 2 triggers • 0.0.7 (James Bond) • 800K lines of code • 600K external & 200K developed in-house • C++, Java, shells, IDL, CHSM, CLIPS, QRL Towards an Open Source Project for Online Software - CHEP2000 - Bob Jones

  11. Lessons Learnt: Organisation • Component model is effective • Clear interfaces allows parallel development by multiple groups • Small group dedicated to each component (max. 4 developers) • One coordinator for each component • One institute per component works best • Component development based on agreed priority (core components first) • Look for commonality between components - avoid duplication • Centralised integration is best • Need people and tests dedicated to this task • Must have good collaborative tools • Code repository, release management, web site, mailing lists etc. • External software (Corba/ILU, Tools.h++, CmdLine, CHSM etc.) • One developer responsible for each external package Towards an Open Source Project for Online Software - CHEP2000 - Bob Jones

  12. Lessons Learnt: Software Process • Software process is important • do start gently (can’t go from chaos to Nirvana in one step) • do keep it simple and stick with it (don’t abandon ½ way through) • do review/inspect every deliverable (requirements, design, code etc.) • do provide templates, checklists and examples for all deliverables • do get a non-author to perform component testing • do encourage developers to share their work and ideas • Things to avoid • don’t ask developers for a deliverable unless it will be used • don’t underestimate time & effort for sw mgmt, integration & test. Towards an Open Source Project for Online Software - CHEP2000 - Bob Jones

  13. Lessons Learnt: Software Releases • Each one must be better than the last • Iterative development model reduces errors and risks • Use feedback to drive/focus work • Nightly on all supported platforms - build & test • Official - coincide with Back-end meetings: discuss status of last release and contents of the next • More platforms == more work • Implies every developer has access to every platform • Try to keep the list short - agree within experiment • Software librarian != developer • Librarian is not there to fix faults in the software • Web page gives access to build log for each component A release is an important milestone and a lot of work for everybody Towards an Open Source Project for Online Software - CHEP2000 - Bob Jones

  14. Towards an Open Source Project • Motivation • Similar solutions to the same problems in many groups & experiments • Online groups are getting bigger (ATLAS: > 50 institutes from 17 countries) • Commodity hardware (PCs) and software (Linux) now being used in online systems • Open Source projects have proved they can deliver high-quality reliable software Towards an Open Source Project for Online Software - CHEP2000 - Bob Jones

  15. Proposal to HENP Community:Use Back-end as basis of Open Source Project • Justification • Software is general - can be use in other projects & experiments • Open Source project-like culture and organisation - would not require many changes • Participating Institutes want to use the software they have developed on other projects • Modular architecture and copious doc. means developers can learn the internals quickly • Developed using sw. eng. principles, OO techniques and a defined sw. process - a sound base for extensions Towards an Open Source Project for Online Software - CHEP2000 - Bob Jones

  16. Issues: Selection of a license • Why do we need a license? • Guarantees • developers’ contributions are acknowledged • software remains open • no warranty - can’t sue us if it does not work • Which one to choose? • GNU GPL & Lesser GPL, BSD, NPL etc. • LGPL would be my personal choice: • Can be mixed with non-free software - e.g. SCADA • Code that begins free remains free - can’t make mods. private • Cannot be re-licensed by anyone • Must have a legal basis • Need help from the men in suits Towards an Open Source Project for Online Software - CHEP2000 - Bob Jones

  17. Issues: Use of Third Party Software • Each external package has its own license policy • ILU: “unlimited reproduction modification, distribution…” • ACE: open source - being phased out • CHSM: GNU GPL • CLIPS: “freely used and distributed without restrictions…” • CmdLine: “freely copy and redistribute…” • SRT: based on CVS and GNU make (GPL) • Motif: Open Group Master Software License - use LessTif (LGPL) • Tools.h++:cannot distribute source - replace with STL • Objectivity/DB: not in SRT & not distributed - only needed by OBK • Java: Get Internal Noncommercial Use Src. License from Sun if needed • Licenses must be respected and compatible Towards an Open Source Project for Online Software - CHEP2000 - Bob Jones

  18. Issues: Support • Continue to distribute official releases on CD-ROM • Fully tested and documented- for short list of platforms • Support level to be defined by participating experimnts. & institutes • Source code and tools available to build own version • No direct support but participants will assist on a best effort basis • Experiments still need online groups! • Back-end software will not cover all the needs of each experiment • Treat Back-end as a third-party software suite to be integrated with experiment specific software • Online groups remain responsible for ensuring the Back-end software works for them on their experiment Towards an Open Source Project for Online Software - CHEP2000 - Bob Jones

  19. Issues: Organisation • Similar to existing Back-end organisation • Components built from packages • Prefer one institute per component/package • One coordinator per component/package • Handles bug reports, change requests, patch submissions etc. • Only coordinator and trusted developers have write access to the code repository • External package - coordinator tracks and selects best version • Distributed development & centralised integration • All major modifications and additions must: • Follow the software process • Meet the quality criteria • Adopt the same software license Towards an Open Source Project for Online Software - CHEP2000 - Bob Jones

  20. How Can You Participate? • Testing and Problem Reporting • You have access to full doc., test programs and source code • pin-point faults so they get fixed quicker • suggest fixes and improvements • Reading doc. & source code • you are helping to inspect/review the software • Code Development • Extend functionality of existing components • Add new components • Port to new platforms Towards an Open Source Project for Online Software - CHEP2000 - Bob Jones

  21. Why Should You Participate? • For the Boss - i.e. person responsible for online sw. in an experiment • Quick start-up - Working software already available • Long-term survival • Safety-net - source code available • Software license - means it will always be available to you • Less software to be developed by you • Contributing extensions, patches etc. means everyone benefits & improves quality • For the Developer - i.e. person willing to take part in the development • Chance to work on a global project - doing what you enjoy/know best • Merit based organisation - your work is more visible • Easier to get jobs - same sw. used in many institutes/experiments Towards an Open Source Project for Online Software - CHEP2000 - Bob Jones

  22. How to Proceed • Try the James Bond (0.0.7) release - to see how much interest • CD-ROM contains header files, libraries, binaries (Linux, Solaris & LynxOS), full documentation • Source code (via LXR tool) & class diagrams visible on the web where permitted by license • No commitment from you or us • Give feedback - used to improve subsequent official releases • Next Step - if sufficient interest • Seek agreement from relevant bodies • Work on turning the proposal into reality • Training session (mixed hands-on and presentations) • To be held at CERN: 2 days during week of April 10th • Will be put on the web (Web Archive Project) & distributed (CD-ROM) Towards an Open Source Project for Online Software - CHEP2000 - Bob Jones

  23. Summary • Back-end software successfully developed in a distributed, collaborative manner • Open Source Project Proposal to HENP community • Given sufficient interest we are prepared to make the Back-end software the basis of an Open Source project for online software • Issues will be addressed in forthcoming official releases • Steps are incremental improvements to the existing software and organisation - not fundamental changes • Based on level of interest we will seek approval from appropriate bodies • More info: http://atddoc.cern.ch/Atlas/ • Live demo: this evening 18:30 - 19:30 after session F in location P5 • Back-end scalability and performance: Igor Soloviev B329 Thurs. 17:00 • Back-end inspections and reviews: Doris Burckhart F331 Thurs. 17:50 Towards an Open Source Project for Online Software - CHEP2000 - Bob Jones

More Related