The sins of IT projects and why they fail (sometimes)

1. The sins of IT projects and why they fail (sometimes) Maurice Perks ….. IBM Fellow (maurice_perks@uk.ibm.com) Title slide

2. This lecture is based upon the 2009 BCS Lovelace Lecture given by Maurice Perks Disclaimer Acclaimer This lecture is dedicated to Dr Tony Storey, IBM Fellow, for his contributions to software design and excellence.

3. The landscape is IT projects for large and medium enterprisesPrivate and Public Our industry has more project failures than it should have Function on time and to budget If we can reduce these failures we can improve the efficiencies of many core businesses and our society which are highly dependent upon IT systems If we can identify the risks early and manage them then we can make a real difference Our IT industry has few if any physical laws to guide us but we do have ….. Patterns that we can recognise Mistakes that we can avoid

4. The IT background to large IT projects Often they are big simply because the business they are to support is big in terms of end-users, locations, functions, connections The technologies that we use and deploy change at a fast rate. Few other industries see this rate of change We have a great diversity of skills that range from business experts to chip experts and therefore a wide range of languages with which we communicate Large IT projects are very costly so therefore very visible especially when they don’t deliver on time

5. What are some of the key technical aspects that we should understand and monitor? • The Challenge of Scale • Middleware or Fiddleware? • Method or Madness? • Where are we? • Software development is still an art • Integration, the final frontier • Will it really work? • Complexity • Change our greatest Challenge

6. 1. The Challenge of Scale Small systems often don’t scale to large systems Prototypes are great but you can’t run a bank or a country on a laptop (yet) Our industry has a wonderful and fundamental way of scaling though apply Parallelism For chips (multicore), servers, connectivity, application tasks and threads, databases we can say: if one instance won’t cope then multiple instances will. But there is a hidden pitfall The dreaded Serialisation Point Sometimes the very nature of the problem that we are dealing with means that a bottleneck is inherent in the solution. We need to understand these bottlenecks and design around them. Sin #1 We cannot scale because we don’t recognise where the bottlenecks in the system are.

7. 2. Middleware or Fiddleware? Middleware is that ‘software stuff’ that sits in between the application code (business processes) and the control programme (operating system) Our industry has a very rich set of standard middleware that is crafted by specialised suppliers some of it open Enterprise IT projects should not develop middleware. It can become a major inhibitor to real progress It starts out as a good idea but soon becomes a millstone around the development schedules and subsequent upgrades. Often a few very skilled people start it and then disappear Sin #2 If it’s not real application code it’s middleware and this is a foolish way to proceed unless you really know what you are doing

8. 3. Method or Madness? When we develop and deploy a major IT system we are engineering something that is very complex. You must have a methodical way of managing the complexity of the development. Our industry does not have a standard engineering method by which we can all work with and to for the technical aspects of projects. There are numerous ‘proprietary methods’. They are all similar at the highest levels … Requirements – Design – Build - Deploy But there are often contradictions in what many of the fundamental stage deliverables are e.g. Architecture? Macro Design? Micro Design? System Integration Test? Sin #3a You attempt a large IT project with no proven method Sin #3b You attempt a large IT project with more than one proven method

9. 4. Where are we? Large IT projects are often long and have several stages. It’s hard to know where you are sometimes. SatNavs don’t help So what’s the problem?

33. Le plus grand pont au monde 2460 mètres de long 343 mètres de haut

35. We do not work in a physical world. None of our senses are useful. We do not work with physical ‘things’ like a bridge. Much of what we work with is virtual and can’t be seen e.g. • How components interface • The accuracy of data • The logic inside applications • The execution of an IT process • The real status of a project • The complexity of what we are working with and on The following slide is a picture of an IT project. This is what it looks, smells, sounds and feels like. There is no mistake on this slide!

36. Sin #4: Assuming that you know where you are in a large IT project without really checking

37. 5. Software development is still an art Most large projects develop some custom software The software gets designed and then coded The software then gets tested in units and then multiple units Errors are then identified within the units and between the units The cause of these errors has remained the same through many changes in the software technologies and techniques like assembler, cobol, c, c++, java, structured, object, etc … Lack of real engineering discipline and the use low level languages Sin #5 Assuming that anyone can write defect free software and that testing will be a mere formality.

38. 6. Integration the final frontier (but 1) When IT solutions are big we break them down into parts, components, building blocks, etc.. This is good practice for the development of these parts; often in parallel But in the end they have to be brought together with themselves and any external parts that are used in the whole system. It’s this integration task that is getting more and more challenging. Perfect interfaces are the key to success. Perks’ 1st Law of IT Integration IT components that are not designed to fit together and form a working system only do fit together by chance. Sin #6 Assuming that breaking an IT solution down into a manageable number of parts for development is success. Success is only achieved when they all work together If you think planning is hard try execution!

39. 7. Will it work in practice? There are four main aspects of what has to work • The technology of the system • The people who run and manage the technology of the system • The end users of the business who own the system • The end users outside the business of the system If any of the above do not work correctly then the whole system does not work Sin #7 Forgetting that the whole system has to work, all the technology and all the people involved have to be able to work with it

40. 8. Complexity – the growing enemy Many operational IT systems and infrastructures that support businesses and environments are now mature They are the summation of Hundreds of development projects Millions of changes Aging We try to understand this complexity through flat diagrams that give us static views that show components and connections and static flows. What we see is STATIC COMPLEXITY But many businesses and environments and the supporting IT systems are highly dynamic with changes happening to a times that vary from microseconds (say workload) to daily, weekly, monthly, etc modifications What we experience are the effects of DYNAMIC COMPLEXITY Suddenly we hit bottlenecks and big queues Suddenly a hole in the security system appears Suddenly the person who knows how much of the system really works – leaves We experience unexpected shocks and there can be global consequences We badly need new ways of describing, modelling and predicting the effects of Dynamic Complexity on business or environments and the IT systems that support them. There is a need for a new union of skills There are three components to this union of skills: • Business or environmental knowledge • IT knowledge and how to use joined-up computing power • Mathematics Sin #8 Not understanding how complexity can change and how we need to predict this change better

41. 9. Change the final frontier! Our industry is the most exciting industry many of us can imagine. It gains this excitement because of constant and rapid change. There are two main sources of the changes that get us so excited: • The changes in the businesses that IT supports. These are coming at us faster and faster as the business world changes on a global basis. These changes will continue as we strive to evolve to a smarter planet. • The technologies that we use which grow in power and capability, variety and where and how they can be used. Large IT projects are subjected to many change pressures as they progress. The natural reaction is to resist these changes to maintain stability of the solution design and the plan. But this stance is often self defeating. IT projects have to learn to live better with change. Why?

42. 9. Change the final frontier! A questionable and very debatable hypothesis The more we can accommodate change properly within a large IT project, rather than ignore it or force it back, the greater the chances of real success for that project will be Our basic problem is that we often do not understand change and its effects Who does or did?

43. 9. Change the final frontier! It is not the strongest of the species that survives, nor the most intelligent, but the one most responsive to change …. C. Darwin Patterns

44. Sir Isaac Newton (1642 – 1727)

45. Newton’s First LawEvery body continues in its state of rest, or of uniform motion in a straight line, unless it is compelled to change that state by forces impressed upon it. Newton’s Second LawWhen an external force acts on a body of constant mass then the acceleration produced is directly proportional to the force.Newton’s Third LawFor every action there is an equal and opposite reaction

46. Newton’s First LawEvery body continues in its state of rest, or of uniform motion in a straight line, unless it is compelled to change that state by forces impressed upon it.An IT project proceeds to plan until something changesitNewton’s Second LawWhen an external force acts on a body of constant mass then the acceleration produced is directly proportional to the force.When a change is forced upon an IT project the extent of deviation from the plan is directly proportional to the magnitude of the changeNewton’s Third LawFor every action there is an equal and opposite reactionFor every change applied to an IT project there is an equal effect on the project’s chances of success unless we really understand the change Sin #9 Not listening to Darwin and Newton and their likes and recognising how change comes in predictable patterns

47. What can we conclude? Our IT industry is the most exciting industry many of us can imagine. It gains this excitement because of constant and rapid change and the associated challenges. But our excitement often overrules our common sense and experiences and we induce unknown risk. We do not have any ‘natural laws’ that are our bedrock but we do have recognisable patterns especially bad patterns. Nevertheless we are great at enabling real business change through the deployment of Information Technology. We just need to get better. Sin #10 Overrunning your presentation time .. Thank you