Software Review panel

1. ALICE Software Review panel Process, Planning, Training, Milestones René Brun, Federico Carminati, Andreas Morsch for the ALICE collaboration

2. A brief history New software organisation since June 1998 Adoption of a unique framework (Sep 98) Creation of AliRoot (Nov 98) Online & Offline use ROOT (99) Very fast transition to C++/OO All detectors included MDCs, Physics Performance Report (00/01) Software before June 1998 the galice package (Geant3, Zebra, PAW) no coordination between detectors Software Panel Computing Review

3. Some numbers (1) Rapidly growing • AliRoot • more than 200 classes • More than 100,000 lines in C++ • +70,000 automatically generated by rootcint • About 5000 lines in f77 • Geant3 • 130,000 lines f77 (GeanT, GeanG,GeanH) • 10,000 lines f77 (mini Zebra) About to disappear Wrapped in class TGeant3 Software Panel Computing Review

4. Some numbers (2) ALICE sotfware • Geant4 • GGG classes • 500,000 lines C++ • Root • 480 classes • 600,000 lines C++ • FLUKA • 700,000 lines f77 • +Event generators • 100,000 lines f77 Framework Services Experiment Independent Software Panel Computing Review

5. Which elements of the computing and software organisation participate, and interact, to effect the design and development of the software ? • Central team at CERN • 4 staff + 3 PA + 2 fellow + 1 VSNA • infrastructure • framework development • DAQ interface • Decomposition per detector • simulation and reconstruction • cooperation between labs for major detectors Software Panel Computing Review

6. Representatives from Alice Projects + Physics Board Representatives from Institutions Representatives from major Software Work-packages Software project organisation Alice Offline Project Software Panel Computing Review

7. Which parts of the software will physicists write and which parts software engineers ? • Present situation • not many software engineers in Alice, mainly as fellows and cooperants. • Most software has been written by physicists (90 per cent) • About 250 physicists could (intend) to contribute. • Currently more than 200 subscribed to the Offline mailing list • More than 50 actively involved in the process today • Future evolution • Computer scientists involvement in databases, graphics ... Software Panel Computing Review

8. How do you stimulate and control contributions from authors spread worldwide ? Fundamental and successful decision • One Single and Modular framework • persistency, containers • histogramming services • UI, GUI, 2-d, 3-d graphics • interpreter + dynamic compilation and linking • same for interactive and batch • coding rules • Regular collaboration-wide software meetings • Weekly meetings with phone conferencing • Dedicated detector specific meetings Software Panel Computing Review

9. How do you stimulate and control contributions from authors spread worldwide ? • Provide mature and productive framework and clear development plans • Training • on the framework • detector specific • Extensive web based documentation • including pages from detector group • Central CVS server • Coding conventions + checking tool • Planning code reviews • Physics Performance Report coming in 2001 is a challenging milestone Software Panel Computing Review

10. http://AliSoft.cern.ch/offline/ Software Panel Computing Review

11. What mechanisms are you using to assure that the core infrastructure components of the software have broad experiment input, validation and testing? • While the current software evolves towards its final form, all groups are involved and motivated to come with comments and suggestions. • All Alice teams co-operating to this single framework. • Detector representatives are involved in the building of detectors. Software Panel Computing Review

12. How does the interface with physics groups work? How is the responsibility divided between Software and Computing groups and Physics groups? For physics algorithms, physics object definition and identification? • Software developers are a subset of the physics group • Close collaboration between physics, technical and computing board • Physics algorithms and objects are defined by discussions in the offline group and offline computing board • No complicated hierarchy • easier to take decisions • strong integration Software Panel Computing Review

13. What design methodology and design process does the experiment use ? Why? How well does this work ? • A growing and encouraged use of design techniques • UML • ROOT tools for reverse engineering • Joint project with IRST Trento for coding convention checking and reverse engineering AliRoot-aware tools • Combination of macro/micro cycles • Mainly play->try->retry->redesign • Rapid Prototyping • Motivation for high quality software Software Panel Computing Review

14. What design methodology and design process does the experiment use ? Why? How well does this work ? • Tools to show class inheritance in a working program, also class relationships. • Code checking and reverse engineering tools gradually being introduced • Automatically generated browsable code documentation • with embedded UML diagrams, HTML and graphics This process has proved to work very well Software Panel Computing Review

15. Code Checker exampleFirst pass on AliRoot modules Software Panel Computing Review

16. ROOT Quality assurance More about this was presented at the ROOT workshop Software Panel Computing Review

17. How did you arrive at your workplan, or work breakdown structure? What planning process has taken place to map out the work to be done in the next two years? Who is responsible for the work breakdown structure and for keeping it up to date? • Pragmatic approach: • Immediate migration to C++/OO • Discourage Fortran/Zebra/PAW. • Adoption of a unique framework: ROOT • Rapid prototyping • Work breakdown is • per subdetector (simulation, part of reconstruction) • central tasks (framework, global reconstruction) • specific workpackages assigned to interested groups (visualisation, geometry database …) • The idea is to grow the current prototype gradually into a production system. Software Panel Computing Review

18. How have you arrived at your workplan, or work breakdown structure? What planning process has taken place to map out the work to be done in the next two years? Who is responsible for the work breakdown structure and for keeping it up to date? • Important milestones are the Physics Performance Reports foreseen for 2001. • Full detailed simulation for June 2000 • Full sub-detector reconstruction for June 2000 • Global reconstruction for fall 2000 • Mock Data Challenges scheduled at regular intervals to monitor the progress. • MDC1 April 1999 (7 TBytes) • MDC2 March 2000 (100Mb/s + LV3) • MDC3 planned fall 2000 (remote centres involvement) • Offline board + central co-ordination at CERN Software Panel Computing Review

19. What are your milestones in the area of development and how do these interact with other experiment milestones? How will you measure the success of your milestones and evaluate progress in carrying out the work plan ? • The AliRoot software chain has been effective already for the TDRs • AliRoot has been used to take important decisions about the layout of the detector. The effects of various detector configurations to important parameters, like the vertex resolution, the tracking efficiency and the track resolution, have been studied, balancing physical and economical issues • Online and Offline both make use of ROOT • As a framework for Off-line • As a visualisation and monitoring toolkit for On-line Software Panel Computing Review

20. What are your milestones in the area of development and how do these interact with other experiment milestones? How will you measure the success of your milestones and evaluate progress in carrying out the work plan ? • Online and Offline share several key projects • ALICE mass storage • ALICE Mock Data Challenges • EuroStore II (if approved) • Milestones are discussed and reviewed in the Offline computing board and in the technical board. • No interest in toy milestones • No software milestones per se Software Panel Computing Review

21. Which areas of planning and work are the highest areas of risk, in that lateness or poor quality will have far reaching effects? What is being done to mitigate these risks ? • Manpower. • ROOT medium/long term support. • G4: worries about Physics, intermediate and low energy hadronic interactions. • Fluka integration: medium long term support • Coherent handling of geometry between simulation and reconstruction. • HSM/MSS • Design and deployment of the worldwide computing model. Alice is specific for the huge amount of raw data. Software Panel Computing Review

22. What is being done to mitigate these risks ? • Manpower. • Take advantage from a mature and working framework to reduce the start time of newcomers • Use temporary personnel • Decentralise development to collaboration institutes • Keep at CERN: framework development, MSS prototyping and co-ordination • collaboration contracts with external Software providers • ROOT medium/long term support. • Repeated requests to CERN management • Take advantage of the large acceptance of Root in a large number of experiments (cross-fertilisation and OS) Software Panel Computing Review

23. What is being done to mitigate these risks ? • G4: worries about Physics, intermediate and low energy hadronic interactions. • Integration of G3, G4 and Fluka (planned) in our framework • Comparison between the 3 MCs and test beams data (planned) • Fluka support • CERN management has promised CERN support • Coherent handling of geometry between simulation and reconstruction. • Opportunity for a common development • but a decision must be taken soon Software Panel Computing Review

24. What is being done to mitigate these risks ? • HSM/MSS • Development of the Alice mass storage project • Evaluation of CASTOR • Participation in Eurostore II • Design and deployment of the worldwide computing model. Alice is specific for the huge amount of raw data. • Participation in MONARC • Participation in EuroGrid Software Panel Computing Review

25. What is the plan for training? What are the various types of training required -design? Use of tools? C++, other? What have you learned so far about the successes and failures of training programs and what do you intend to do in the next two years? Do you expect that any of this be in common with other experiments? What role do you expect CERN IT to play? • Training courses on AliRoot, the Alice framework. • Typically one training course during Alice software week • At CERN and outside: GSI, Heildelberg • including courses on ROOT • Typically 20 attendants per course (in CERN training room) • These one day courses have been highly successful. • We should do more Software Panel Computing Review

26. What is the plan for training? What are the various types of training required -design? Use of tools? C++, other? What have you learned so far about the successes and failures of training programs and what do you intend to do in the next two years? Do you expect that any of this be in common with other experiments? What role do you expect CERN IT to play? • C++ & OO courses (CERN or home institutions) • These courses FOLLOW and not precede the training on the framework • IT should be more active in C++/OO training and support • Too much focus on Java in IT • More help on C++/Java integration Software Panel Computing Review

27. How will technology choices for languages, tools, database products, etc. be made? What provisions are being made for rapidly changing technology? • Privilege HEP/industry main-stream • C++ is the chosen language • restrict features of C++ to facilitate co-operation with some parts possibly written in Java • Root I/O as basic data base in combination with an RDBMS system (MySQL, Oracle?) for the run catalogue. • Extensive comparison Root vs Objectivity • Intensive testing/prototyping via the ALICE MDCs Software Panel Computing Review

28. What plans do you have for the long-term support of your software? • Our basic framework (ROOT) is an Open Source project. • We need an in house team • Problem well understood at FNAL (3 staff devoted to local ROOT support) • Expect to take advantage of experience and developments in other experiments • Graphics • Geometry packages and data base • Physics algorithms • Parallelism Software Panel Computing Review

29. What plans do you have for the long-term support of your software? • Support for ALICE specific software • See next slide and manpower planning • Support for non ALICE specific framework • Large number of experiments (~30) using ROOT world-wide ensures open-source style support • Request official support of Root at CERN • The large number of users should be an additional motivation for this! • Can be a small team (3 staffs) • But has urgently to be put in place Software Panel Computing Review

30. What quality assurance and control mechanisms are being put in place, and in which stages of the design, implementation and testing processes? All testbeds with production code • Mock Data Challenges essential • Periodic Code reviews planned • Code checking tools (from IRST Trento) • Code systematically reviewed by senior scientists • As much as possible documentation in the code with automatic html generation • Enforce documentation with code checking tools • html and pictures in the C++ code directly • Require portability on different platforms • currently : Linux, Solaris, HP-UX, Alpha/Unix Software Panel Computing Review

31. What decisions on software technology and implementation choices have to be taken in the future and when do you plan to take them? • Major choices already done in 1998 • Language • Framework • Event database and object store • Data analysis package • Run Catalogue and conditions data base • consensus on RDBMS • follow experience of other experiments(STAR, FNAL) • MySQL prototype in MDCs (choice in 2001/2002) • Geometry data base • XML ? Common with other experiments ? Software Panel Computing Review

32. What decisions on software technology and implementation choices have to be taken in the future and when do you plan to take them? • Decision on MSS/HSM • Eurostore II , CASTOR, InterStage? • Prototyping via ALICE Mock Data Challenges • Decision in 2003 • Understanding access patterns • ratio tape/disk • analysis patterns • Event by Event and conventional multi-event Software Panel Computing Review

33. What decisions on software technology and implementation choices have to be taken in the future and when do you plan to take them? • Simulation packages • what is the role of fast simulation • when and if replace GEANT3 with GEANT4 • FLUKA will be integrated in the framework and kept for • comparison with and complement to GEANT4 • having two options for full detector simulation in the same framework and with common geometry, kinematics and hits • detailed detector understanding • radiation studies Software Panel Computing Review

34. What decisions on software technology and implementation choices have to be taken in the future and when do you plan to take them? • WAN data access • LAN and WAN data access tools • We assume a constant network latency and a rapidly growing bandwidth excluding object-level access (the DREAM=illusion) • rootd a possible choice • want to follow/understand GRID middleware • decision to be taken in the scope of MONARC, GRID projects Software Panel Computing Review

35. What decisions on software technology and implementation choices have to be taken in the future and when do you plan to take them? • Parallelism • batch job level parallelism assumed to be solved by LSF/Condor (or like) evolution • Interactive data analysis requires a tight integration between the analysis tools and the data bases • Prototyping based on PROOF (Parallel Root Facility), possibly in the framework of the GRID project • Important to understand the implication on the analysis model and tools (prototyping necessary) • Final choice in 2003 Software Panel Computing Review

36. What decisions on software technology and implementation choices have to be taken in the future and when do you plan to take them? • Role of Java • C++ is our language. We invest on our decisions. • It would be unfair to impose a change in the short/medium term • Timescale & priority depending on the perceived role of Java in and outside HEP • Worries about numerical precision and standardisation process (http://www.cs.berkeley.edu/~wkahan/JAVAhurt.pdf) • We expect complementarity with C++, not an alternative • Our Object model is simple (no templates) • It cannot be an obstacle to a migration • Already compliant with Java (object base class, containers) Software Panel Computing Review

37. What decisions on software technology and implementation choices have to be taken in the future and when do you plan to take them? • Role of Java • step1: Transparent access to C++/Root services via web applets or JRE. (In development with FNAL) • step2: Exchange of objects (data objects only) between C++, Java. Easy to do via the CINT RTTI. Could be very quickly implemented. • Step3: Full Java integration. Implement C++ methods on the Java classes where it makes sense Software Panel Computing Review

38. What decisions on software technology and implementation choices have to be taken in the future and when do you plan to take them? • Possible role of XML • A possible role for detector geometry • Watch developments in other collaborations • Exchange format for low-volume objects • Documentation • Electronic Logbook • ROOT & XML • Trivial for ROOT to generate XML output for objects • Replace ROOT HTML with ROOT XML documentation Software Panel Computing Review

39. How will it evolve? How will it meet demands for parallel processing? distributed processing? security and authentication needs? Language evolution? • Parallel processing will be vital. Thinking to use an evolution of PROOF and rootd. • WAN prototyping with MONARC and Grids • A state of the art authentication package is part of ROOT (SRP from Stanford). • Possible role for the Grid software • It is essential to support multi-threading and asynchronous I/O • Thread package developed jointly with the GSI GO4 project • Developments in Root I/O to support multi-threading. Software Panel Computing Review

40. How do you plan to provide working software and do development at the same time? How do you plan to transition from existing software to final production software? • Open Software spirit • Micro/Macro development cycles • We gradually develop the production system. • At each step, the software is used by the community in production for the day- to- day work. • Not simply a prototype with the idea to start from scratch in two years. • We exploit the power of abstract interfaces • We make sure that enough modularity is built in our system. Software Panel Computing Review

41. Manpower Support for ALICE specific software • No chance to have enough people at CERN to develop the majority of the code of ALICE, as it was the case for LEP. • Planning based on maximum decentralisation. • All detector specific software developed at outside institutes • Around 40 people directly involved today • CERN Off-line team responsible for • central coordination, software distribution and framework development • prototyping, (in particular of MSS in collaboration with IT) Software Panel Computing Review

42. Manpower Support for ALICE specific software • Fulfill this role with Project Associates and Fellows • main worry is continuity -- basic support and coordination more difficult with short-term resident staff • more attention to the project structure and documentation • New people should get acquainted quickly with the structure and be effective in a short time. For this we have setup a web site with all the information on the code and the project structure..... Software Panel Computing Review

43. Activities Software Panel Computing Review

44. ALICE CERN man-power evolution Needed FTEs Current FTEs Current Staff Software Panel Computing Review