An HML white paper: Agile IT

1. An HML white paper: Agile IT A value-driven approach to IT delivery

2. Contents 2: About this paper 3. About HML Introduction 4. The need for change Lean agile roots 5. The journey to Agile IT 7. Overcoming challenges to change 8. The Eight Principles of Agile IT 13. The benefits of Agile IT 14. Agile IT at work: Forbearance Project 15. About the author About this Paper The purpose of this paper is to demonstrate how applying lean and agile principles can create value in software development. It will be of interest to people who are responsible for IT projects, budget control, organisational change or who have an interest in lean and agile thinking. The paper describes how HML has put these principles into practice in a form of software development called ‘Agile IT’, which addresses key business concerns like time to market, responsiveness to changing needs, customer satisfaction and cost reduction. The paper outlines the history of lean and agile approaches and explains how these have been applied to create Agile IT. It sets out Eight Principles that underpin HML’s approach, and how these enable the delivery of the right IT capabilities at the right time to maximise value to customers. “Agile IT has allowed HML to meet the strategic challenges of wasted resource and lengthy change management processes. What might have taken months now takes weeks and our IT department is engaged and able to offer clients tangible results throughout the process. ” Andrew Jones, Chief Executive Officer, HML

3. HML’s success in applying lean and agile principles has resulted in a nomination for the Best Agile Newcomer at this year’s UK Agile Awards. For more information about the Awards seehttp://www.agileawards.co.uk/Awards2012.html About HML HML is a leading financial outsourcer, providing outsourced mortgage, savings and loan administration services to over 50 leading financial institutions in the UK and Ireland. HML operates from three UK locations: Skipton, Londonderry and Glasgow, with a presence in Dublin as well. The company was established in 1988 and manages approximately £43bn of assets and 400,000 customer accounts. Introduction HML’s iConnect platform provides a sophisticated and comprehensive tool to support a diverse range of client requirements. iConnect is constantly updated, driven by HML’s desire to continually improve its offering, but also by the regulatory environment in which HML and its clients operate. In 2008, HML began to apply lean 6-sigma improvement techniques principally within its operational areas. In 2010 it embarked on an ambitious programme to improve iConnect in support of this drive for operational excellence. To meet this challenge, the IT teams needed to develop their ways of working, so Agile IT was born. Agile IT is achieved by the application of eight principles: 1. Make it Visible 2. Make Collaboration Easy 3. Assume Change: Experiment, Discover, Learn, Iterate 4. Focus on Delivering Useful Value Today 5. Build a Sustainable Flow of Value 6. Build Quality In 7. Continuously Improve 8. Agility is Everyone’s Responsibility HEADLINE RESULTS In just 12 months, Agile IT has enabled HML to increase its rate of delivery of new IT capabilities by a factor 4, while reducing the time to deliver a new feature from months to a matter of weeks; all without impacting cost or quality.

4. At the start of 2011, HML recognised that its IT teams needed to evolve their ways of working in order to become more responsive to changing needs and reduce the ‘time to value’ of changes, whilst retaining the improvements in quality. The Need For Change In common with many financial institutions, HML has in the past relied on a sequential software development lifecycle, often known as ‘waterfall’, where projects progress through a series of discrete stages: Request Design Build Test Implement Each stage within this sequence was largely undertaken by a separate specialist functional group and work was handed off from one group to the next with formal documentation providing the communication between groups. This approach worked well for HML’s needs at the time and over five years had resulted in a steady improvement in the stability of production systems, with progressively fewer defects escaping into production, and improved predictability of delivery dates. However, these improvements tended to come at the expense of responsiveness to change and long delivery timescales. At the start of 2011, HML recognised that its IT teams needed to evolve their ways of working in order to become more responsive to changing needs and reduce the ‘time to value’ of changes, whilst retaining the improvements in quality. These changes were to be guided by the adoption of lean and agile software development principles and practices – an approach that complemented HML’s existing use of lean and 6-sigma techniques in its Operational Service Areas. Lean-Agile Roots The term ‘Agile’ as applied to software development was first coined in 2001 to describe a range of methods that emphasised adaptability to change. The Manifesto for Agile Software Development and its associated 12 Principles sets out what being agile means; while a range of methods, such as Scrum, eXtreme programming and DSDM, provide ways of developing software in an agile way. The idea of lean software development began at a similar time. It takes the core principles of lean thinking, e.g. flow, pull, and continuous improvement, and applies them to software development.

5. The timely and sustained delivery of valuable IT capabilities to HML’s internal customers and external clients remains paramount, and is at the core of Agile IT. Lean and agile can be seen as complementary approaches: lean tends to address the wider organisational concerns, while agile focuses at the development team level. The combination of lean and agile principles and methods, together with other approaches such as 6-Sigma, Kanban and the Theory of Constraints provide the foundations for Agile IT at HML. See http://agilemanifesto.org/ See http://www.scrum.org/ and http://www.scrumalliance.org/ See http://www.extremeprogramming.org/ See http://www.dsdm.org/ See http://www.lean.org/WhatsLean/Principles.cfm The Journey To Agile IT Lean and agile software development is not just something that you do; it’s equally about how you think. Changing the ways of working in IT had to address both transformation of its culture and the adoption of new practices and techniques. And while lean and agile principles and techniques have been at the heart of the changes, these have always been treated as a means to an end rather than an end in their own right. The timely and sustained delivery of valuable IT capabilities to HML’s internal customers and external clients remains paramount, and is at the core of Agile IT. At the start of the change transition, HML appointed an experienced agile development practitioner to provide expertise, leadership and support through hands-on coaching. Reporting directly to the Head of IT provided the appropriate executive authority and clearly signalled the importance of the changes. Building awareness of lean and agile approaches, principles and techniques has been a cornerstone of the transition. Alongside formal training and hands-on coaching, peer-based informal ‘learning lunches’ and ‘community of practice’ forums have been used to regularly share experiences and learn new techniques. Task boards - big visible displays that show a team’s flow of work - were one of the first items to be introduced and they remain one of the most visible aspect of the changes to visitors. Impediments – things either slowing down or blocking flow – are quickly identified enabling action to be swiftly taken to resolve.

6. Figure 1: Impediments can be escalated to a Department Level Board HML has borrowed from many of the well known agile methods, such as Scrum, eXtreme Programming (XP) and DSDM, but has deliberately chosen to avoid mandating a particular method in order to encourage adaptation, innovation and ownership by teams in how they work. Symbols used to indicate type of impediment: Stopping progress Slowing progress Significant risk of stop or slow Two other key changes were also introduced early in the transition: co-location of teams and incremental development; where business goals are progressively broken down into a series of features that are developed in short iterations and delivered over a series of releases. Fundamental to the long term success of changing the way of working has been to make continuous improvement and learning a way of life. Fortnightly ‘retrospectives’ enable a team to reflect on what is working well, and what is not; and gives them responsibility for changing their process. Over time, this process has itself been adapted, refined and improved, with both a team and department wide improvement regime. Instead, HML has developed eight principles of Agile IT that can be used by everyone involved, from individual IT practitioners, to teams and stakeholders to guide how they should work. The principles help reinforce the view that Agile IT is primarily cultural, and represents a journey not a destination.

7. Iterative development and incremental delivery creates significant challenges for upstream and downstream processes. Creating an IT capability to build and deliver new features on a just-in-time basis also requires a business organisation that can define and prioritise a flow of desired features, and one that can absorb and apply them usefully. Overcoming Challenges To Change Change is never easy, and HML faced a number of challenges on its Agile IT journey. One of the first challenges HML encountered was the creation of co-located, cross-functional teams. Unsurprisingly, some individuals felt unhappy to have to move desks, and breaking up long standing functional groups risked losing their sense of community. HML worked hard to involve those who needed to move in the planning, and introduced changes progressively. The existing functional line-reporting structure and regular functional team meetings were also retained. Only at the start of 2012 did formal line management change to reflect the new cross-functional structure of teams. Throughout the transition, HML needed to ensure governance was not compromised. Many of the existing governance controls were organised around or were dependent on waterfall stages and functional teams, so the move to cross-functional teams and iterative development potentially reduce these controls’ effectiveness. Close collaboration with the Risk and Compliance teams ensured appropriate oversight and HML is currently introducing Practice Leaders who will be responsible for governing key practice standards. Finally, iterative development and incremental delivery creates significant challenges for upstream and downstream processes. Creating an IT capability to build and deliver new features on a just-in-time basis also requires a business organisation that can define and prioritise a flow of desired features, and one that can absorb and apply them usefully. HML has made some progress in this area, using agile techniques to define business cases for increments rather than whole projects, but this remains an area of development and improvement.

8. Teams own their visual management boards, and this fosters innovation in how boards and the underlying process they represent evolve to meet the team’s needs. The Eight Principles of Agile IT 1. Make it Visible The process of software development is difficult to visualise, and while Gantt charts have a place, they rarely map well to the needs of IT. Agile IT makes use of large, physical display boards that show the flow of work being undertaken by the team, an approach known as Visual Management. Task boards and similar visual management tools help a team visualise the ‘state of play’ of the work they are doing and make it easy to spot and respond to impediments to a team’s development flow, contributing to shorter delivery times. Teams own their boards, and this fosters innovation in how boards and the underlying process they represent evolve to meet the team’s needs. For example, blockages are highlighted with stop signs; avatars of team members are used to indicate who’s doing what; and colour is used to indicate different types of work. FIGURE 2: SHOWING FLOW OF WORK ITEMS, TYPICALLY USER STORIES AND ASSOCIATED TASKS As well as helping the team, visible task and status boards have proved invaluable for updating stakeholders. They’ve also fostered greater team spirit and visibility of the status of the project with the wider business. Avatars show assignment Impediments highlighted with Red Flags

9. Face to face communication is encouraged as the primary method of communication IT projects face two key challenges: customers do not know precisely what they really need; and in today’s competitive business environment those needs are often a moving target. Agile IT assumes that change is inevitable and seeks to exploit this rather than resist it. 2. Make Collaboration Easy IT development is a creative process. Customers and IT professionals have to work together to ensure they deliver what is needed in a background of imperfect knowledge and change. Making collaboration easy is therefore essential for successful projects. With Agile IT, work is undertaken by small co-located, cross-functional teams, known as ‘pods’ who take responsibility for the end-to-end delivery of new software features. As well as bringing together expertise in different IT disciplines, such as test, engineering and analysis, customer representatives and other key stakeholders work closely with teams throughout the development and delivery lifecycle. Face-to-face communication is encouraged as the primary method of communication. Documentation remains important, but the emphasis is on doing just what is necessary and using alternative forms of documentation, such as wikis and other electronic knowledge bases, rather than paper documents. 3. Assume Change: Experiment, Discover, Learn, Iterate IT projects face two key challenges: customers do not know precisely what they really need; and in today’s competitive business environment those needs are often a moving target. The traditional approach to addressing these challenges is to undertake extensive up-front analysis before locking down scope and then limiting subsequent amendments through change control regimes. Unfortunately it’s also common to see these IT projects take a very long time to deliver what turns out to be the wrong thing. Agile IT assumes that change is inevitable and seeks to exploit this rather than resist it. Using an iterative approach to development coupled with incremental delivery ensures that the most valuable features needed today are delivered first whilst others are deferred for later increments. IT projects at HML typically use fortnightly iterations, and are able to deliver new features into production monthly. Stakeholders are able to review working versions of iConnect at least every iteration, and this feedback is used to adjust what features will be developed next.

10. Agile IT seeks to incrementally deliver value by delivering in small, valuable increments. Each increment is focused on delivering the minimum needed to provide benefit to its customers today. 4. Focus on Delivering Useful Value Today Delivering something of value early is generally much more beneficial than delivering a collection of things much later. As well as increasing return on investment, focusing on delivering what’s useful today helps prevent speculative development or ‘gold-plating’. Research by the Standish Group suggests this is far from uncommon, reporting that 45 per cent of software applications’ features are never used. As well as incurring unnecessary cost, building these features means that other more useful features are either delayed or not built at all. Agile IT seeks to incrementally deliver value by delivering in small, valuable increments. Each increment is focused on delivering the minimum needed to provide benefit to its customers today. Stakeholders are actively involved in determining the order of development of features and the team is encouraged to seek out opportunities for delivering increments early. Monthly releases enable new features to be rolled out quickly so it’s possible for a feature to go from idea to delivery in just a few weeks. Delivering incrementally places new pressures on repetitive tasks such as build, deployment and regression testing and HML has steadily focused on automating and refining this process. Similarly, architecture and detailed designs need to support change, and technical practices such as Test Driven Development and Refactoring have been adopted to support this. 5. Build a Sustainable Flow of Value Many organisations seek to maximise the utilisation of staff and the number of projects that are in progress. Unfortunately, this approach tends to result in long delivery times for any one activity and relatively low completion rates. Whilst it can give the appearance of cost efficiency it comes at the expense of time to value and responsiveness to changing needs – qualities that usually far outweigh the savings made by maximising utilisation. Agile IT seeks to take a holistic view of the whole delivery value stream - not just that part in IT - optimising for a fast, steady stream of deliveries of valuable features. Instead of maximising utilisation (and minimising unit cost), Agile IT seeks to minimise cycle time, the elapsed time it takes for valuable increments to be delivered, and so increase throughput, the rate of delivery of benefits.

11. Simply put, Agile IT values finishing over starting; doing less in order to achieve more, and systematically finding and removing anything that disrupts or impedes flow. Agile IT recognises that speed and agility requires attention to quality and technical excellence; it is simply not possible to go fast if the process or product is unreliable. Reducing cycle time enables returns on investments to be achieved sooner and improves responsiveness to changing needs. Increasing throughput maximises the opportunity to deliver what is needed. Simply put, Agile IT values finishing over starting; doing less in order to achieve more, and systematically finding and removing anything that disrupts or impedes flow. Teams at HML use either iterations - short timeboxes of typically two weeks - or explicit work in progress limits to manage flow. Projects are decomposed into small increments which are in turn broken down into small features that can be completed in days rather than weeks. And teams actively identify and resolve impediments to flow, escalating those they cannot quickly resolve to a department-wide daily impediments meeting attended by management to ensure visibility. 6. Build Quality In Deming famously declared that “You can not inspect quality into the product; it is already there.”* Most traditional approaches to software development focus on the (often late) detection of problems, through inspections and testing, and make extensive use of manual processes that inevitably introduce variation and errors. Agile IT recognises that speed and agility requires attention to quality and technical excellence; it is simply not possible to go fast if the process or product is unreliable. Teams at HML employ automation for repetitive tasks, such as integration, build, check and deploy; enabling them to execute these tasks frequently – typically multiple times a day. They also use techniques such as pair programming and test driven development that help prevent errors or catch them as soon as possible after they are introduced. *From Out of the Crisis, by W Edwards Deming (MIT Center for Advanced Engineering Study, 1986)

12. The idea that a single process will work equally well for all time and across all projects and teams is not reasonable, and so Agile IT promotes a culture of continuous adaptation and improvement and advocates that those who do the work are best placed to improve how the work is done. ‘Doing agile’ is important, but ‘being agile’ is essential. Fundamentally, agility is a mindset that everyone in the organisation needs to express. 7. Continuously Improve The need to respond to change applies just as much to the process of development as to the capabilities of the software. The idea that a single process will work equally well for all time and across all projects and teams is not reasonable, so Agile IT promotes a culture of continuous adaptation and improvement and advocates that those who do the work are best placed to improve how the work is done. HML has made knowledge sharing explicit; sponsoring ‘learning lunches’ and the formation of communities of practice, and has recently introduced Practice Leaders to guide and support practice improvement. Frequent and regular retrospectives - workshops focusing on improvements - are undertaken by teams, typically every two weeks, where teams are encouraged and empowered to make small incremental changes to their processes to drive improvement. 8. Agility is Everyone’s Responsibility ‘Doing agile’ is important, but ‘being agile’ is essential. Fundamentally, agility is a mindset that everyone in the organisation needs to express. Agile IT provides an organisation with the potential for becoming an Agile Business. At HML, agility in general and Agile IT specifically, has become central to its ethos.

13. An HML consultant feeds back after a recent project to automatically and securely retain customer card details goes live: “I have to give you this feedback because it is proof of the success of card re-use functionality. We had over 200 calls today and not dropped a single one, on last working day! Normally by now we would have expected to drop calls as it is so busy. It's obvious that there are a lot of card payment only calls and we are flying them. Customer feedback is positive and consultant feedback is very positive.” Benefits of Agile IT Over the first 18 months of its Agile IT journey, HML has significantly improved its IT capability: Increased throughput HML has been able to move from quarterly releases to monthly releases. Delivering more frequently has had two benefits: firstly, it is much easier to release opportunistically – stakeholders, having seen an iteration demo, may realise that the software would be beneficial if deployed as is, and monthly releases allow these opportunities to be exploited. Secondly, more frequent and smaller releases reduce the risks associated with each deployment. Reduced time to deliver value Before the transition to Agile IT, IT projects at HML, even those that used a phased delivery approach, would typically take between three and nine months from approval to their first delivery into production. Agile IT has reduced this ‘time to value’ considerably, with projects typically now delivering benefits within three months, and often considerably sooner. Greater flexibility and responsiveness to changing needs Focusing on achieving a business goal frees teams to explore and adapt their solution so that it best meets that goal, enabling them to respond to changing circumstances and new knowledge about what is important. This flexibility has enabled projects to start and begin to deliver useful benefits despite imperfect information; while allowing new needs to be much more easily accommodated. Improved stakeholder and client satisfaction By working more closely with its stakeholders, building iteratively and acting on their feedback, HML has been able build trusted relationships and a real feeling that they were a part of our product development.

14. INDUSTRY REACTION TO AGILE IT: “I got the impression that [lean-agile principles] were truly embedded in the day-to-day work of the teams.” “I was greatly impressed by the IT department...to see an IT team working so cohesively is something we can only dream of.” Agile IT at Work: Project ForBearance At the start or 2012, responding to a request from one of its clients, HML began a project to implement improvements to iConnect to support anticipated regulatory requirements proposed by the FSA. Speed of response was critical to the client’s needs, but uncertainty around what was really needed was high – HML was seeking to be first to market – and operational usability was essential to ensure minimal impact to customers and call centre consultants. During the start-up phase of the project, the team, through a series of workshops with stakeholders, created a high level vision, a prioritised list of high level requirements and an outline roadmap. Work then proceeded using fortnightly iterations that planned, built and demonstrated new functionality. Regular retrospectives generated frequent improvement ‘experiments’ which drove constant adaptation and improvement in how the team worked. As the project progressed many items originally in the backlog were considered unnecessary, and many new items were added in response to improved understanding of needs identified through the demonstrations. Throughout the process, both the client and internal stakeholders were involved, and this has resulted in high levels of engagement and satisfaction. What the client said: “The projects team's approach throughout has been one of collaboration best demonstrated by their invitation to me, to not only attend, but part-facilitate the two day forbearance training event that was delivered to our Credit Management colleagues.” (Compliance Manager) “It has been a pleasure to work with the Forbearance project team. Communication has been excellent throughout the project, they have been responsive to requirements (including late ones) and all signs point to delivery meeting agreed scope and originally agreed timescale.” (Operations Manager) “I was surprised at just how good the visual management was in IT in terms of its variety and complexity.” • Delegate responses from a meeting of the Lean Service Forum, hosted at HML in April 2012. For more information on the Lean Service Forum see: http://www.oeeuk.com/community-leanforum.asp

15. About the Author: Andy Lawrence, Lean-Agile Practice LeadAndy has over 20 years experience in delivering IT based solutions across a diverse range of industries, and has been leading and coaching software teams in lean and agile software development for the last decade. He has presented at conferences and industry events, and regularly contributes to groups such as Agile Yorkshire and Agile Testing in Finance. Andy joined HML in 2008, initially working with the Business Intelligence team before joining the IT department to lead the lean-agile implementation.Andy has been nominated in the Agile Coach or Mentor of the Year category for UK Agile Awards 2012.