Frontier rpms

1. Frontier rpms June 2011 David Front Weizmann Institute frontier rpms

2. frontier rpms

3. Why rpms? Once a site (takes the effort and) moves to use rpms • upgrading SW is faster, and easier compared to using tarballs • It should also be more safe … assuming that proper testing of the new rpm release has been done Yum package manager is supported by frontier rpms • With yum, installing, upgrading and removing is easier that via rpm: - yum is aware of rpm dependencies - No need to specify rpm version, only package name • Automatic upgrade possibility: - Periodically do ‘yum upgrade package’ (via crontab) • Note, however, that relocation is not supported by yum frontier rpms

4. Frontier rpms – history and status • Frontier has been written by CMS and distributed as tarballs • FaviaDonno did create initial frontier rpms for ATLAS: • frontier-squid • frontier-tomcat • frontier-servlet • frontier-awstats • A new release of frontier rpms is available, is in use at CERN ATLAS product servers since today, and is planned to be defined as ‘production ready’ soon: • In particular, there is a need for rpms to be tested out of CERN first • Current use of frontier rpms: • ATLAS (and CMS?) has squids installed as rpms • The other frontier rpms are installed only at ATLAS servers at CERN: CMS, and non CERN ATLAS frontier servers are installed via tarballs. frontier rpms

5. Installing frontier rpms: Prerequisites • The (configurable) user to own rpm files should exist prior to rpm installation: Default: • ‘squid’ for squid • ‘dbfrontier’ for the other rpms • If you are installing frontier rpms, instead of existing non rpm (tarball) package/s: First uninstall the existing non rpm SW: • Remove the existing directory in which the software was installed • Take care to remove/undo changes at other files that have been edited manually or by scripts. In particular, undo crontab (manual) changes. • Verify that squid/tomcat service actually stopped: Be aware that: Stopping squid/tomcat may take a long time, or: ‘service frontier-{squid or tomcat} status’ may say by mistake ‘stopped’, while ‘ps –efw|grep{squid or tomcat}’ is not empty. frontier rpms

6. Installing frontier rpms: configuration • Manually take care of per rpm and per site and per site/machine files, if any. For details, consult later slides, and: http://frontier.cern.ch/dist/rpms/_README • Configuration changes: As a rule of thumb, configuration changes are applied to files as follows: • The directory is /etc/{squid|awstats(|tomcat)} • Configuration changes of customize.sh are applied when the appropriate server restarts. • Hence, there should be no need to reinstall an rpm in order to apply (such) configuration changes. • In addition, for frontier-squid, squidconf is applied at install time frontier rpms

7. Installing frontier rpms (cont.) • In order to install frontier rpms, as user root, import the gpg public key once (per machine): rpm --import http://frontier.cern.ch/dist/rpms/cernFrontierGpgPublicKey • Using the command 'rpm': as user root: rpm -Uvh [--prefix <RPM_INSTALL_PREFIX>] http://frontier.cern.ch/dist/rpms/<rpm> • Relocating an rpm: The defaults of RPM_INSTALL_PREFIX are: frontier-squid: / frontier-awstats: / frontier-tomcat: /data/dbfrontier frontier-servlet: /data/dbfrontier/tomcat/webapps/atlr (for servlet atlr) The frontier rpms are relocatable Exception: frontier-tomcat and frontier-servlet rpms can not be relocated to / The plan is that / will be the default for them as well. frontier rpms

8. The current rpm versions At:http://frontier.cern.ch/dist/rpms/: • frontier-awstats-6.0-1.noarch.rpm • frontier-servlet-3.29-1.noarch.rpm • frontier-tomcat-6.0.32-1.noarch.rpm • frontier-squid-2.7.STABLE9-5.3.sl5.x86_64.rpm frontier rpms

9. Installing via yum • get cernFrontier repository file once per machine: wget -O /etc/yum.repos.d/cern-frontier.repo http://frontier.cern.ch/dist/rpms/cern-frontier.repo • Now you may use yum as usual. for example: yum install package1 [package2] [...] yum upgrade package1 [package2] [...] yum remove package1 [package2] [...] frontier rpms

10. Testing the installation: tomcat, squid For a simple test of tomcat and squid servers, using fnget.py: you may adjust and run commands like the following, from bash: wget -O /tmp/fnget.py http://frontier.cern.ch/dist/fnget.py chmod +x /tmp/fnget.py server=atlasfrontier1.cern.ch # Adjust this to your server servlet=atlr # Adjust this to you servlet port=8000 # To test tomcat. Adjust if needed /tmp/fnget.py \--url='http://'${server}':'${port}'/'${servlet}'/Frontier' \--sql="select 'TOMCAT_IS_HAPPY' from dual" \--refresh-cache port=8080 # To test squid. Adjust if needed /tmp/fnget.py \--url='http://'${server}':'${port}'/'${servlet}'/Frontier‘ \--sql="select 'SQUID_IS_HAPPY' from dual“ \--refresh-cache If all went well, you should get records like the following: 'TOMCAT_IS_HAPPY' CHAR 'SQUID_IS_HAPPY' CHAR frontier rpms

11. Testing the installation: awstats, logrotate To see that awstats works, one may wait a while and see that the numbers or the upgraded squid do grow Logrotates: Logs should not grow too much: /data/tomcat_logs/catalina.out /data/tomcat_logs/modifiedqueries.out /data/squid_logs/access.log To see the allowed sizes: • sudo –u <squid/tomcat user> crontab –l • Find there when logrotate should work and sizes are allowed frontier rpms

12. Testing the installation: load (After a while,) monitor the upgraded server/s host load, verifying that it did not rise or drop considerably, as a result of the upgrade: • CERN: use lemon https://lemonweb.cern.ch/ • Else where: ganglia or whatever host monitoring tool is in use frontier rpms

13. Building a frontier rpm • rpms should be built at voatlas137 • The related sources should be checked out from svn, under: /data/$USER/rpms • Build an rpm: cd /data/$USER/rpms/<rpm Name> pushd ./scripts; ../../scripts/rpmbuild_.sh;popd 2) Copy rpm to the web: cd /data/$USER/rpms/<rpm Name> pushd ./scripts; ../../scripts/rpm2web.sh ; popd frontier rpms

14. Plan • Once the new rpms will be defined as production ready,it will besuggested that all ATLAS frontier sites will use these frontier rpm/s instead of existing rpm or tarball/s • Enhance frontier rpms to support CMS • Support more than one servlet at a site.ATLAS does not require this because it has only one servlet per site • Add a /etc/tomcat/servlets.conf configuration file • Add a /etc/tomcat/servlets.passwd configuration file(will reside at sindes at CERN) 3) Avoid file duplication at underlying (CMS) tarball and at the rpms code (mainly at frontier-squid). This will make the maintenance of frontier code simpler 4) TBD: Add more monitoring tools into the rpms frontier rpms

15. frontier rpms

16. Configuration: frontier-squid Optional configuration files: 1) /etc/squid/squidconf: Used at rpm installation Consists of export commands Main usage: to specify user and group for frontier-squid, other than default user and group: squid, squid Example (as of CERN): export FRONTIER_USER=dbfrontier export FRONTIER_GROUP=users 2) /etc/squid/customize.sh Used per restart of the squid server. Consists of specific commands that control the content of squid.conf file. Examples: Appear at /etc/squid/customize.sh.example_* and at https://svnweb.cern.ch/trac/frontier/browser/rpms/frontier-squid/trunk/src/frontier-squid-utils/etc/squid -See also: FlaviaDonno's documentation for previous version of frontier-squid rpms: https://twiki.cern.ch/twiki/bin/view/PDBService/SquidRPMsTier1andTier2 frontier rpms

17. Configuration: frontier-servlet Per servlet configuration file should be edited manually. May be done after installation: <install dir>/tomcat/conf/Catalina/localhost/<servlet name>.xml Example of such a file name: /data/dbfrontier/tomcat/conf/Catalina/localhost/atlr.xml frontier rpms

18. Configuration: frontier-awstats Optional file to control environment variables at install time /etc/awstats/awstats.conf Syntax: export <variable>=<value> Example: export FRONTIER_USER=<something other than dbfrontier> Editing of the followingfiles may be done (before or) after rpm installation: 1) A file for rsync with the password of user $SITE$PROJ (Example: cernatlas) /etc/awstats/password-file 2) An optional per awstats client site/machine config file. For CERN clients, this file is not needed, but for other sites it may be needed. File: /etc/awstats/customize.sh ## WARNING Care should be taken when changing variable in this file, ( and in particular PATH_LOG). Testing of frontier rpms has not been done with changing of these variables The rpm installs an example file: /etc/awstats/customize.sh.example_CERN: https://svnweb.cern.ch/trac/frontier/browser/rpms/frontier-awstats/trunk/src/etc/awstats/customize.sh.example_CERN frontier rpms

19. urls • Documentation of new versions of frontier rpms: http://frontier.cern.ch/dist/rpms/_README • The main url of frontier: http://frontier.cern.ch • Current frontier rpms: http://frontier.cern.ch/dist/rpms/ • To browse svn files of frontier rpms: https://svnweb.cern.ch/trac/frontier/browser/rpms • To checkout a frontier rpms svn directory (example: frontier-tomcat): svn co svn+ssh://svn.cern.ch/reps/frontier/rpms/frontier-tomcat/trunc frontier-tomcat • Atlas frontier documentation by John Destefano: https://www.racf.bnl.gov/docs/services/frontier/ • Documentation of (previous) frontier server rpms, by FlaviaDonno: https://twiki.cern.ch/twiki/bin/view/LCG/WLCGFrontier • Documentation of (previous) versions of frontier-squid rpm, by FlaviaDonno: https://twiki.cern.ch/twiki/bin/view/PDBService/SquidRPMsTier1andTier2 • Previous versions of frontier rpms, created by FlaviaDonno, are at: http://grid-deployment.web.cern.ch/grid-deployment/flavia/?C=M;O=D • CMS has a lot of frontier documentation frontier rpms