1. The software flow The software life cycle Development:tagging A-01-00-03 A-00-01-07 Prod release Prod release CVS Prod release cmt co Any scale Tag registering CMT Tag collector cmt co Kit construction Tag selection Nightly build Nightly build Tag collector cmt co Pacman cache days Integration tests Core tags lock All tags lock cvs ci cvs tag Minor release cycle 8.2.0 8.3.0 cmt co 8.1.0 Register package register tag weeks Work area Major release cycle 10.0.0 9.0.0 8.0.0 pacman -get months pacman -get • Individual packages are tagged randomly • Tags can be submitted to the Tag Collector for the next release until it is locked • Every day the last registered tags are selected for the nightly build • Every three weeks the tag collection is locked during the release build (between the last nightly and the end of the release validation) • Tags for core packages are locked one week before the release build • Twice a year a major release is produced (global tag M.0.0) • Patch releases may be created after any minor or major release (global tag M.m.p) Grid CE Laptop Prod release Kit validation Work area DC job The software structure Prod release Atlas generic data files Prod release Prod release • Packages are first created in a work area, registered to the TagCollector and imported to the CVS repository • Packages are checked out to work area(s) • Tags are declared to the TagCollector, registered for the next release • Nightly builds take all registered tags for the next release • Quality checks and Integration tests results are reported to developers (through Web pages) • Production releases take validated tags • Distribution kit is automatically constructed from production releases • Release kits are installed on Grid CE and laptops • KitValidation jobs are run after installation • Users may develop packages against installed kits Atlas packages Prod release • Atlas External software • Generators • Local installations of other software • Geant4 • Java suites Gaudi packages CMT packages • LCG software • SEAL • POOL • PI • … Containers LCGCMT interfaces Software Policy • HEP or Free software • Root • MySQL • … Interfaces • Managed by CMT • Three sub-projects Atlas, Gaudi and LCGCMT • Each sub-project has its own global version (the release tag) • Atlas selects the release of Gaudi • Gaudi selects the release of LCGCMT • Each release of each sub-project selects the set of versions of individual packages • Both Atlas and Gaudi define their specific centralized installation area • Binaries • Include files • jobOptions • Scripts (shell, python) • Java packages Package semantics • Container packages: • Purely managerial packages • Not involved when using the software • Only provide a CMT requirements file with a set of explicit use statements (no wild card) • Purpose: • Specifies the set of exact version for a release of the software group • Drive the cmt broadcast commands over the complete software group • The top container is AtlasRelease and describes the complete Atlas software release • Every new release of the Atlas software yields a new release of the AtlasRelease container. • Policy packages: • Provide policy statements for a software domain • Include dir strategy • CMT strategies • CMT patterns • Generic symbols (macros, sets, …) • AtlasPolicy is the main policy package, mandatory for all Atlas packages. • Examples: ExternalPolicy, AtlasCxxPolicy, AtlasFortranPolicy The Tag Collector Jérôme Fulachier – Johann Collot • Control the consistency of package versions assigned to any release • Can be locked to prepare the release • Support branch release management • Bug fixes • Late patches • Directly connected to CVS • Perform cross checks with CMT settings • Handle the container hierarchy • Interface packages: • Provide an interface between a software project managed by CMT and a so-called external product (not managed by CMT) • Specify the native version tag defined by the product (as opposed with the CMT version tag) • Macro <package>_native_version • Adapt the policies and conventions to the project’s ones • Eg Include directory • Construct the compiler and linker options for this product • Set the runtime options (PATH, LD_LIBRARY_PATH, etc…) • Specify the material to export when constructing a distribution kit • Examples • MySQL • ROOT • SEAL • Geant4 • Java kits • The LCGCMT project is entirely made of such interface packages • Connect to LCG internal (SEAL, POOL, PI, …) or external products NICOS: the build driver Alex Undrus • http://atlas.web.cern.ch/Atlas/GROUPS/SOFTWARE/OO/ The atlas software page • http://isnpx0168.in2p3.fr/athena/ The tag collector • http://www.cmtsite.org CMT Links • Run quality checks • Run unit and integration tests • Generate documentation • Operate on fast local disks • Then transferred to AFS The Atlas Software Distribution Christian Arnault Alessandro De Salvo Simon George Grigori Rybkine arnault@lal.in2p3.frAlessandro.DeSalvo@roma1.infn.itS.George@rhul.ac.ukgrigori.rybkine@rhul.ac.uk

2. Pacman cache cache cache description='Package IdDictParser (v1)' source = '../kits/8.7.0' download = { '*':'IdDictParser-opt.tar.gz' } depends = [ 'dist-8.7.0-opt', '8.7.0/AtlasPolicy-opt' , '8.7.0/XMLCoreParser-opt' , '8.7.0/IdDict-opt' , '8.7.0/Identifier-opt' ] Pacman –get ATLAS:8.7.0/IdDictParser-opt • Manifest files describe packages • Version • Link to the contents tar ball • Describe dependencies • Specific installation procedure • Contribution to the global post-install setup procedure • Specifies and checks the site configuration • A cache is a web-visible disk space • Packages may be found in different caches • Caches may be virtual or mirrored (snapshots) • Installation transitively and recursively fetches packages from caches • Apply the installation procedures • Generate setup scripts • Saul Youssef • youssef@bu.edu • http://physics.bu.edu/pacman/ Saul Youssef The special case of external packages Operations Cache/<project>/<package>-<platform>.pacman Cache/LCGCMT_26_2c/Expat-opt.pacman > cmt broadcast $CMTROOT/../cmtpacman/scripts/create_kit.sh -platform opt /afs/cern.ch/…/pacman description='Package Expat v1 opt in project=LCGCMT/LCGCMT_26_2c (v1)' url = 'http://atlas.web.cern.ch/Atlas/GROUPS/SOFTWARE/OO/Development/' source = '../kits/LCGCMT_26_2c' download = { '*':'Expat-opt.tar.gz' } depends = [ 'LCGCMT-LCGCMT_26_2c-opt', 'LCGCMT_26_2c/LCG_Settings-opt' , 'Expat-1.95.5-opt' ] install = { '*': ['. $PACMAN_INSTALLATION/CMT/*/mgr/setup.sh; cd $PACMAN_INSTALLATION/LCGCMT/LCGCMT_26_2c/LCG_Interfaces/Expat/cmt; cmt -quiet -no_cleanup config; a=1'] } • Creating the kit: • Define CMTPATH for the subproject • Select a cache directory • Select a platform tag • Launch a cmt broadcast loop over the release package • Will reach all packages of the sub-project • Run the cmtpacman/scripts/create_kit.sh shell script • An external package is described in a CMT Interface package • is characterized by two special macros • <package>_native_version • <package>_export_paths > cd /myinstall > pacman –get ATLAS:8.7.0/AtlasRelease-opt • The tar ball only contains the CMT directory • kits/<project>/<package>-<platform>.tar.gz • kits/LCGCMT_26_2c/Expat-opt.tar.gz • Installing the kit: • Select an installation location • Select a release • Run pacman package Expat use LCG_Settings v* macro Expat_home "$(LCG_external)/expat/$(Expat_native_version)/$(LCG_system)" macro Expat_native_version "1.95.5" macro Expat_export_paths "$(Expat_home)/lib/libexpat.so $(Expat_home)/include/expat.h“ include_path none include_dirs $(Expat_home)/include macro Expat_linkopts "-L$(Expat_home)/Libs -lexpat" \ WIN32 "/LIBPATH:$(Expat_home)/Libs libexpat.lib" path_remove LD_LIBRARY_PATH "/expat/" WIN32 "" path_prepend LD_LIBRARY_PATH "$(Expat_home)/Libs" WIN32 "" path_remove PATH "" WIN32 "\expat\" path_prepend PATH "" WIN32 "$(Expat_home)/Libs" • The pacman kit for the interface package • Related with CMT, • and to the top project (e.g. LCGCMT) > source setup.sh > source dist/8.7.0/Control/AthenaRunTime/*/cmt/setup.sh • Setup the runtime environment: • Source the setup script generated by pacman • Source the setup script for the specific application • Run athena straight from the installed kit > cd /myworkarea > export CMTPATH=`pwd`:$CMTPATH > cmt co DetectorDescription/IdDictParser > cd DetectorDescription/IdDictParser/*/cmt > gmake > … • Develop against an installed kit: • Setup the runtime environment • Select a work area • Checkout a package using client CVS • Build it and run it Cache/<package>-<native_version>-<platform>.pacman Cache/Expat-1.95.5-opt.pacman description='External Package Expat v1 opt (v2)' url = 'http://atlas.web.cern.ch/Atlas/GROUPS/SOFTWARE/OO/Development/' source = '../kits' download = { '*':'Expat-1.95.5-opt.tar.gz' } • Currently we only produce development kits • Source kits can also be produced • Development against the installed kit implies getting the source through CVS • Pacman 3 permits to install several releases in a common location • External packages are shared between releases • The tar ball contains the real package contents • kits/<package>-<native_version>-<platform>.tar.gz • kits/Expat-1.95.5-opt.tar.gz • The pacman kit for the physical contents • Independent of CMT • Fully standalone kit Kit validation • One of the Atlas packages • Available from pacman too • Provide scripts to automate the post-install kit validation • Structured as a set of transformation scripts • Transform data from a state to another state (generation, simulation, etc…) • Launch athena.py over existing job options (picked up from the installed release) • Run various configuration tests • Checking of DLLs • Test the development scenario (building against the installed kit) • Can generate dedicated data files or job options • Construct job parameters • IO filenames • Random seeds • Number of events • Provide a more general framework of batch job submission • Data Challenges productions • Grid job submissions • Format and capture the job output • Can describe the PASS/FAIL conditions in terms of patterns (regular expressions) • Filter PASS/FAILURE conditions according to the patterns • Generate a consolidated report • Can be formatted as a web page (or even Wiki page) ALL:SimpleChecks.kvt:Release Simple Checks ALL:AthExHelloWorld.kvt:Athena Hello World ALL:AthenaPoolTest.kvt:Athena POOL Test ALL:InDetDetDescrExample.kvt:InDetDetDescr Example ALL:MooEventCompile.kvt:MooEvent compilation ALL:EvgenMuon.kvt:Athena Muon Generation ALL:DC2EvgenZee.kvt:DC2 Z -> e e Event Generation ALL:DC2EvgenZmm.kvt:DC2 Z -> mu mu Event Generation ALL:DC2EvgenZtt.kvt:DC2 Z -> tau tau Event Generation ALL:DC2EvgenJJ.kvt:DC2 DiJet Event Generation ALL:DC2EvgenMinBias.kvt:DC2 Minimum Bias Event Generation ALL:DC2EvgenSusy.kvt:DC2 Susy Event Generation ALL:DC2EvgenTop.kvt:DC2 Top Event Generation ALL:DC2EvgenJetCalib.kvt:DC2 Jet Calibration Event Generation ALL:DC2EvgenJetFilter.kvt:DC2 Jet (pt>17) Event Generation ALL:DC2AtlfastZee.kvt:DC2 atlfast Z -> e e Ntuple/Histo production ALL:DC2G4simZee.kvt:DC2 Z -> e e G4 Simulation ALL:DC2G4digitZee.kvt:DC2 Z -> e e Digitization ALL:DC2reconZee.kvt:DC2 Z -> e e Reconstruction The Atlas Software Distribution Christian Arnault Alessandro De Salvo Simon George Grigori Rybkine arnault@lal.in2p3.frAlessandro.DeSalvo@roma1.infn.itS.George@rhul.ac.ukgrigori.rybkine@rhul.ac.uk From the requirements to the pacman kit description='Package IdDictParser (v1)' source = '../kits/8.7.0' download = { '*':'IdDictParser-opt.tar.gz' } depends = [ 'dist-8.7.0-opt', '8.7.0/AtlasPolicy-opt' , '8.7.0/XMLCoreParser-opt' , '8.7.0/IdDict-opt' , '8.7.0/Identifier-opt' ] cache/<project>/<release>/<package>-<platform>.pacman cache/dist/8.7.0/IdDictParser-opt.pacman kits/<project>/<release>/<package>-<platform>.tar.gz kits/dist/8.7.0/IdDictParser-opt.tar.gz • All specifications to create the kit for one package are uniquely extracted from the requirements file of this package • CMT Use statements produce Pacman depends statements • The entire dependency graph is preserved • Dependency cycles are filtered out • Constituent definitions populate the tar ball from the binary installation area • Conventional macros describe special files to bring to the tar ball (relocated to the installation area) • Runtime files • Gaudi jobOptions • Scripts • XML files • Header files • The cmt directory and its supporting directory tree is copied into the tar ball to permit configuration activity against the installed kit • The generated tar ball is relocatable