1. aims2 Automated Installation Management System

2. Overview of Presentation • Give you an explanation of AIMS • Introduce you to the installation infrastructure at CERN • My Project • AimsRewrite • aims2 • Overview • New Features • The future Aims2 – Linux Automated Installation Management System - 2

3. What is AIMS? • Automated Installation Management System • Perform parallel installations minimising the need for human intervention • Makes use of PXE and ELILO. • The system is based on and extends the Kickstart software from the RedHat distribution. • Supports Linux and Windows. • Around since 2000, Perl code base. • Used throughout CERN • Experiments • Fabric/Grid deployment • General infrastructure user Aims2 – Linux Automated Installation Management System - 3

4. Installation Infrastructure Aims2 – Linux Automated Installation Management System - 4

5. Architectures & Bootloaders • Network boot loaders need to bootstrap the device, load a configuration and the kernel. • I386, x86_64 • Supported by pxelinux.0 loader • Based on syslinux • ELILO • ElILO is the EFI Linux boot loader for IA-64(IPF), IA-32(x86), and x86_64 EFI-based platforms. • Each architecture has its own way of doing things. Aims2 – Linux Automated Installation Management System - 5

6. Installation boot sequence • Client makes DHCP request • DHCP service replies with NEXT_SERVER (lxpxeboot.cern.ch) and location of the network bootstrap file to use, dependant on the client architecture if option client-architecture = 00:00 { # Intel x86PC - in use now. option LINUX.pxelinux-magic F1:00:74:7E; option LINUX.pxelinux-pathprefix "aims2/"; option LINUX.pxelinux-reboottime 50; filename "aims2/loader/pxelinux.0"; } else if option client-architecture = 00:02 { # EFI Itanium - in use now. filename "aims2/loader/elilo64.0"; } else if option client-architecture = 00:06 { # EFI IA32 - future extension (think Intel Apple ..)‏ filename "aims2/loader/elilo32.0"; }

7. Boot sequence cont. (PXE)‏ • pxelinux.0 is invoked and starts to look for client configuration • By client system UID ( XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX )‏ • By hardware address, appending 01 • Client IP address, HEX encoded, stripping one byte and retrying • default • default contains info about the “kernel” to be loaded default main label main kernel /loader/vesamenu.c32 append /pxelinux.cfg/main.conf • vesamenu.32 takes over and builds interactive boot menus from config file

8. Boot sequence cont. (PXE)‏ • ELILO is a little different • Boot program is elilo64.0 • Does not try to load system UID • Appends ia64 to encoded IP address • “default” is elilo-ia64 or elilo.conf image=/aims/boot/SLC4X_IA64/vmlinuz label=slc4X description="Install Scientific Linux CERN 4 on ia64 (graphics console)" read-only initrd=/aims/boot/SLC4X_IA64/initrd append="load_ramdisk=1 maxcpus=1 network keymap=us lang=en_US.UTF-8 ip=dhcp method=http://linuxsoft.cern.ch/cern/slc4X/ia64/" • In both cases, we use the hardware address of the device and use the architecture of the image to decide which configuration to use.

9. aimsRewrite Project • A solution to meet the new modern requirements of CERN and its users • A rethink of what LA is providing as a remote installation service • Move away from AFS dependency • Kickstarts, /tftpboot/ sync'ing • Delegate • PXE image management • Device authorisation • Reduce maintenance/administration overhead • Improved logging and auditing Aims2 – Linux Automated Installation Management System - 9

10. DHCP • Already completed • Useful, and flexible • DHCP/BOOTP behaviour passed back to CS • No messing around with DHCP configurations. • Operating System in LANDB dictates which NEXT_SERVER the client is sent to. • LINUX = lxpxeboot.cern.ch • PXELINTEST = lxpxeboottest.cern.ch Aims2 – Linux Automated Installation Management System - 10

11. Alternatives • AII • Automated Installation Infrastructure • Set of Quattor components • Supports PXE, Kickstart and Jumpstart • Cobbler & Koan • From RedHat • Cobbler is a provissioning server. Koan is the client installer • 'Very' feature rich (Templates, Snippets, WebUI...)‏ • Can manage a lot (TFTP, DHCP, DNS, REPOS...)‏ • Supports Xen, KVM and WMWare installations Aims2 – Linux Automated Installation Management System - 11

12. Introduction to aims2 • A rewrite of AIMS code base making it simpler and more flexible • Perl SOAP Client and Server • Server-side uses a modular approach • Database support provided by Oracle • Improved integration with the CERN environment (LanDB, LDAP, e-groups, CDB)‏ • Improved • Customisation of a users installation • User authentication and authorisation Aims2 – Linux Automated Installation Management System - 12

13. Authentication • Identifying who the user is and whether they are allowed to use the service. • Originally provided by a manually maintained .klogin • Uses Kerberos 5 • Defined service in the KDC • So you can get a ticket for aims2 • Ticket is presented by the client • No KRB5 credentials for CERN, no access. Aims2 – Linux Automated Installation Management System - 13

14. Authorisation • Do you have permission to install X device? • Try to prevent accidental/malicious installations. • Previously manually maintained by AFS ACLs on Kickstart directories. • Difficult to transfer ownership of a device. • In one word - Icky! • Solution is not easy, as I found out. • No global one-stop source of device ownership at CERN. • Need to use multiple sources. Aims2 – Linux Automated Installation Management System - 14

15. How we try to achieve this • Is the USER listed as the OWNER or MAINUSER of the DEVICE? • If the DEVICE is known to CDB, is the USER listed as having root permissions on the DEVICE? (CDB ACL, type=root)‏ • Is the USER a member of Linux Support? (They might be helping you out)‏ • If the device is located within Building 513 or 613, is the USER an FIO sysadmin? • Is the owner or main user something we've been told about? • We can map shared accounts we know about to e-groups. For example, we can map the service account "FS.Administrator@cern.ch" to "fs_installers“. • Explicit deny at the end. • Still not perfect, but following user feedback it is a lot better. • Code used is very flexible so if a new source becomes available, it can be easily accommodated. Aims2 – Linux Automated Installation Management System - 15

16. Kickstart Handling • No more enforced use of AFS directories • New commands added: showks, updateks • Upload or link to your kickstart file. • Linked Kickstart files can be easy re-sync'd if changed • Link sources permitted include AFS and http (not https yet)‏ • Kickstarts rendered to Anaconda on the fly if • Correct device • Device is enabled. • Balance between hiding the Kickstart and rendering Aims2 – Linux Automated Installation Management System - 16

17. PrepareInstall • A (large) script used by the Quattor/Elfms community • Uses device information in CDB to generate Kickstart file for device. • Also deals with SINDES • Only small amount of worked was needed • Uploading of Kickstart rather than writing to AFS • Catching errors thrown from AIMS • Modification of PXE boot targets, with append options. • Many thanks to Jan VE for his help. Aims2 – Linux Automated Installation Management System - 17

18. Kernel append options • Previously maintained templates • bootif, eth0, eth1, with(out) serial console.... • Now allows arbitrary options to be provided with the –kopts option • If you provide ks, it will override aims' ks • No option uses DHCP • NEXT_SERVER/ip-address.ks • Can now very easily deal with “icky” hardware • Smthg new: allowwireless, essid=<essid>... Aims2 – Linux Automated Installation Management System - 18

19. Arbitrary .img management • Smart users can test and deploy their own kernels and/or intitrd.img's • Use case within FIO • u-boot, burn-in-tests, fireware updates, hardware utilities • SLCx • Other operating systems (unsupported)‏ • Client commands • addimage • showimage • remimage • Limited right now to AFS sources Aims2 – Linux Automated Installation Management System - 19

20. Master-Slave configuration • Master-Slave configuration manually maintained in server configuration • Should master fall over, nothing works. Aims2 – Linux Automated Installation Management System - 20

21. Server independence model Aims2 – Linux Automated Installation Management System - 21

22. The Role of the Database • aims2 is Oracle driven • Used for holding device PXE states and PXE boot media for deployment • Centralised logging and server configurations • Service reliability • If a server is lost, the important stuff is safe. • Server states maintained with database daemon • Maintains connection to database, sleeping and waking during downtime. • Polls for changes, maintain directories Aims2 – Linux Automated Installation Management System - 22

23. Audit trail and Traceability • Log “every” action • Tracer bullets everywhere • Logging is centralised • Log the “who, what, when, where and how” • Provide information on each important step of the installation • When the client booted • When the client pulled it's kickstart • When the client got to %POST (pxeoff)‏ • Helps the user understand their installation and Linux Support can follow • Still some work needed on training Linux Support Aims2 – Linux Automated Installation Management System - 23

24. Improvements for the future • Fix bugs, of course • Improve the use of Oracle • More processing “inside” Oracle, but not everything • Improve LOB storage. Right now Oracle is greedy • New commands • updateimage • downloadimage • Showhistory • As time goes on, user's needs will change Aims2 – Linux Automated Installation Management System - 24

25. Evaluation • Writing software is difficult • aims2 tries to provide a more flexible system that can be easily adapted as the environment changes • Benefits should be felt the users first • The installation process is improved • and also felt by Linux Support • Improved debugging tools • Again, the benefits filter back to the user. Aims2 – Linux Automated Installation Management System - 25

26. Thank You and Questions • ALL the users of AIMS • Fantastic feedback (dev and testing)‏ • Patient and willing to talk about new ideas (and accept bumps on the way)‏ • Thanks! • Jarek as my Supervisor • LA for its support • Jan VE for help with PrepareInstall • Many others too! • Over to you now, Question Time. Aims2 – Linux Automated Installation Management System - 26