LANL Case Study: Funding Computing Support NLIT 2008

1. LANL Case Study: Funding Computing SupportNLIT 2008 Michael S. Zollinger CTN-1 Group Leader Computing, Telecommunications, and Networking Division

2. Outline • Overview of Computing Environment at LANL • Historical Overview of Funding Computing LANL • What Led to Decision to Change Funding Methodology • New Funding Model • The Resulting Challenges • The Resulting Benefits • Real Pressure Points • Service Delivery Changes • Lessons Learned • Summary

3. Overview of the LANL Computing Environment • We are big! • Geographically - 42 square miles in size • User Community - 10,000+ employees • Variety of computing environments • Protected Unclassified (Yellow) Network – over 20,000 devices • Various levels of Classified (Red) Networks • Public Interface (Green) - approved public information release • Visitor Network (Gray) • Unclassified Scientific Enclaves - (Turquoise)

4. Historical Overview of Funding Computer Support • To understand where we are at now, a brief history lesson is in order • Previous to FY08 computing support was funded by a mixture of the following: • Centrally provided personnel through a charge back (recharge) mechanism – most prevalent – covered overhead, training, materials and supplies, etc. • Service Level Agreement (SLA) with each customer organization • Demand driven service • Matrix (FORM B) arrangement to provide employee to partners with only care and feeding from central organization • Customer owned and operated personnel (partner organizations / shadows) – directly funded • Institutionally funded services provided by the central support organization – infrastructure tax (levied across all organizations as a tax on labor) Programmatic requirements determined support – standards voluntary

5. What Led to Decision to Change Funding Methodology? • LANL contract awarded to Los Alamos National Security, LLC in 2006 • More corporate style management practices required • Reorganization of LANL management structure • Doubled the number of directorates • Increased number of divisions and groups • Doubled the number of SLA negotiations • Impossible to extract money from customers • Requirement to become more efficient in our service delivery and standardized in our approach • Decreasing budgets, therefore need for more cost effectiveness required • LANL Director desired to fully understand and manage the IT expenditures

6. New Funding Model • After a weighing the alternatives the decision was made to fund support of unclassified standard desktop and server support from Infrastructure Tax • Microsoft Windows • RHEL • Mac OS X • Non-standard (unique programmatic requirements driven) unclassified desktop and server support • Programs who required their own infrastructure and could not use the centrally managed infrastructure i.e., email, DNS, home directory storage • Non-standard or legacy operating systems • IRIX, True64, etc. • All classified computer support to remain funded by charge back mechanism • The funding model to be revisited in FY09

7. Institutional Desktop (Unclassified) Funded – Infrastructure tax Support hours 8 a.m. - 5 p.m. Central services on call Help Desk Remote assistance & problem diagnostics Dispatch additional support Deployed on site computer support (Migration to geographic not organizational, where it makes sense) Configuration management services Guidelines Automated build tools Automated verification tools Automated SW/HW inventory tools Support for IA standard software and OS Computer systems security Patching/Antivirus Standard protection Tools ESD systems access & license management Central home directories storage Institutional strategic planning Servers in support of institutional desktop needs CTN institutional applications servers Print services Program and Classified (Custom support) Funded - Recharge Support hours 8 a.m. – 5 p.m. unless otherwise negotiated Specialized application client/server support Pro E (W,WT) Documentum (TA-55, others) Lotus Notes (LC) MRP (TA-55) 3D Vis (X) Genome Sequencing (B) GIS (ADISS) Support for non IA approved operating systems Diskless computing technologies for client/server CTN-5 like services (Institutional-Consolidation opportunities) LDAP DNS Local Servers (Institutional-Consolidation opportunities) Scanning Web servers Data file Chat services Collaboration Server clusters Human Genome NISAC Special projects JTOT SCADA Data acquisition Additional central project storage Programmatic vs. Infrastructure Dividing Line

8. Budget Facts for FY08 • Approved infrastructure tax budget for standard desktop support • $27.2 M • 90% for time and effort • 10% for operations costs (workplace recharge, training, equipment, etc.) • Number of employees funded • 175 FTE (full-time employees) • $155K (average loaded cost per support employee) • Cost per platform supported • ~ 20,000 computers on unclassified computing network • $1360 per computer per year • Tech to computer ratio average • ~ 1 : 114 • REMEMBER, this is an average

9. Operations Budget – Breaking Down the 10% Slide 9

10. Resulting Challenges • We were not popular around campus • The infrastructure tax rate increased 3.5% • 2% was directly attributable to moving approximately 66% of our cost to that infrastructure budget from the charge back methodology • In discussions with customers we received the entire blame for the 3.5% increase • Our current customers were impacted, but not as severely as our “partner” organizations • They were taxed whether they used our services or not • There was no requirement to use CTN for programmatic computing support • Programs retained most of their staff and transferred a few personnel to do infrastructure funded support • They also transferred their hiring vacancies to us

11. Resulting Challenges - cont • 1st Qtr FY08 LANL was faced with a workforce restructuring • Contractors were trimmed first • A Voluntary Separation Program was offered • Approximately 500 people left • A potential Involuntary Separation (layoff) was averted • Hiring external has been nearly impossible • The initial funding for infrastructure tax funded support has been reduced • All funding for vacancies has been retracted • LANL is under a Security Compliance Order to implement the NAPS on the classified and unclassified networks • No additional resources added

12. Resulting Benefits • One Service Level Agreement (SLA) negotiated with senior management for infrastructure tax funded support! • Instead of 20 to 30 • Funding secured at one level • Programmatic support requirements are more focused • Programmatic customers pay less for same level of support they received previously • Programmatic customers are migrating more of their systems to standard configurations and centralized services • Restructuring of field support to geographic support vs. organizational support is occurring where it makes sense • Support based on organizational affiliation is still happening for scientific multiplatform environments

13. Real Pressure Points • The goal is to drive towards more efficient support • Minimize the need to physically touch computers • Support more centrally • The “Catch 22” is that to become more efficient you need to have the headroom to change your business model • This generally requires an increased budget for a period of time to implement new processes and technologies • For LANL, all the budget pressure has been downward and not upward • Do more with less • This is painful • In addition, contract incentives require us to make changes now

14. Service Delivery Changes • Moving to geographic model where it makes sense • LANL Acquisition Strategy Project will provide one purchasing vehicle for computers • Sometime of central provisioning will take place • A “blind buy” process will be implemented where manufactures and JIT vendors will not know who the final end customer is for the system • An evaluation of what we need as a service desk is in process • Restructuring our current Help Desk approach • Certification and Accreditation of the unclassified networks will now apply to the 20,000+ devices

15. Lessons Learned • The fact that all desktop support was not transferred into one organization created a work overload condition for infrastructure funded support • Director initially wanted to fund all support; programmatic and infrastructure out of infrastructure tax • We advised against this believing that it would lead to unfettered demand for service • There is unfettered demand anyway • May happen in FY09 • The lack of an IT governance model at the inception of this change did not allow for IT portfolio management (prioritization of services) • Which services do we turn off when funding dries up • Managing expectations of customer base is difficult • Relying on infrastructure tax funding puts in competition with: • Roads and grounds • Snow removal • Facilities • Utility bills

16. Summary • Changing our funding model has benefits and challenges • The real payoff may not be realized for a few more years • Streamlining processes and leveraging resources is a priority • Technology can only solve some the problems • Political change is the most difficult to achieve in any enterprise

17. Contact Information Mike Zollinger 505-667-5355 msz@lanl.gov Q&A