G. S. Kearns Information Resources Management Journal Vol. 17, No. 1, pp. 37-62 Jan-Mar 2004

1. A Multi-Objective, Multi-Criteria Approach for Evaluating IT Investments: Results from Two Case Studies G. S. Kearns Information Resources Management Journal Vol. 17, No. 1, pp. 37-62 Jan-Mar 2004

2. Outline • Introduction • The IT Investment Decision • The Analytic Hierarchy Process • The IT Investment Model • An Information Systems Example • Evidence From Two Case Studies • Results of Investment Decisions • Discussion • Study Contributions • Conclusions

3. Introduction (1/3) • A majority of CEOs • IT investments were economically infeasible • Confidence about the future ability of IT to provide strategic advantages • Economic analysis of IT returns relies on quantitative measures

4. Introduction (2/3) • Traditional approach • Have not proven useful in the economic evaluation of IT-based investments • Single criteria techniques • Discounted cash flow, cost/benefit analysis • Bias towards the tangible benefits • IRR or net present value may ignore the ‘soft’ , qualitative benefits of IT applications • Strategic applications • Require a method • Reliably measure all benefits in a consistent manner that is understood and supported by management

5. Introduction (3/3) • Maximizing returns from IT investments requires a total portfolio planning approach • Can not be accomplished by valuing each investment individually • Mutually exclusive, mutual dependencies • Should not be combined due to the total risk • Combined with integer programming and the Analytic Hierarchy Process • Support a multi-objective, multi-criteria approach • Address several issues hindering the success of IT investments • The purpose of the paper • Demonstrate the MOMC approach to IT investment analysis • The applicability of the proposed model using an illustrative example of five information systems projects • The results of two case studies in which the model was successfully applied

6. The IT Investment Decision (1/3) • There is little persuasive evidence • Investment in IT positively impacts the financial position of the firm or increases productivity • Measurement problem • Time period between investment and realized benefits • The direction of causality is difficult to prove • The study examines a more direct method of influencing business performance • Improving the quality of the IT investment portfolio

7. The IT Investment Decision (2/3) • Traditional financial accounting measures • Past evaluation of IT investments suffer from • Isolation • Difficulty in valuation of benefits • Low explanatory power • Ignore basic investment tenets • All financial measures are sensitive to the valuation of benefits • The approach assumes that each investment stands on its own merits without regard to other investments • Some investments generally have failing marks under ROI and passing marks under net presents value • Such as ERP

8. The IT Investment Decision (3/3) • IT-related investments represent in excess of half the annual capital expenditures for many firms • An agreed-upon approach to measuring IT investments does not exit • Returns on IT investments have been unsatisfactory • The selection of IT-based investments • Produce the highest value for the firm • Value must reflect a combination of both quantitative and qualitative criteria • A decision support process is needed that will incorporate all relevant decision criteria

9. The Analytic Hierarchy Process • AHP applications are numerous • Strategic planning, microcomputer selection, etc • AHP combines with other techniques • multi-dimensional scaling and integer and linear programming • No prior illustrations of this use • The MOMC is an effective measurement process • Rank alternative investments according to criteria • Corporate strategies • The strict time constraints of the planning process • Support consensus among a diverse group of individuals • Reflect investment precedence or exclusivity constraints • Incorporate both quantitative and qualitative criteria • Be understood by management

10. The IT Investment Model (1/7) • Corporate strategies used as project ranking criteria • The importance of linking IT strategies to corporate strategies has been well known • Traditional discounted cash flow techniques lack linkage to corporate strategy • AHP facilitate specification of criteria based upon corporate strategies

11. The IT Investment Model (2/7) • Level of difficulty • Include • The flexibility of the measurement process in reflecting changes • Perform sensitivity analysis • Produce viable alternative solutions • Provide an explanatory trail • AHP methodology • Use a paired-comparisons approach • The criteria indicators represent typical investment alternatives • The sum of each criterion’s value becomes the investment’s global score for final ranking

12. The IT Investment Model (3/7) • Explanatory power • The most valuable feature of AHP • A convenient framework for concise representation • Offer a formal, systematic, consistent approach • When combined with an integer optimizing model • The weights can readily be compared • Managers are able to see into the process

13. The IT Investment Model (4/7) • Creating consensus • AHP is highly effective in distilling information from groups and fostering consensus • By paired comparisons • An important foundation for acceptance • AHP creates quantitative rankings • Use a systematic approach to capture priorities • Measures the consistency of the overall process • Cost, precedence, and exclusivity constraints • Resource constraints limit the number of investments • Precluded investment may be due to overlap in functionality or competition for non-cash resources • Convert the multi-criteria resource allocation problems into integer programming maximization-type problems

14. The IT Investment Model (5/7) • Structuring the AHP hierarchy

15. The IT Investment Model (6/7) • AHP theory • An overall view of the complex relationships • Help the decision-maker assess the importance of the issue • Support meaningful comparisons between attributes • Steps of using • Establish the decision hierarchy • Create input data and make paired-comparisons of the decision elements • Estimate the relative weights of the decision elements • Aggregate the relative weights of decision elements to arrive at a final set of ratings • For the decision alternatives

16. The IT Investment Model (7/7) • Incorporating quantitative and qualitative investments • In practice and theory • No consensus on the appropriate mechanism for ranking IT investments • Objective evaluation method • Net present value, cost-benefit analysis, project risk, value analysis, benchmarking, multiple criteria approach, DSS evaluation, aggregate scoring technique, and anecdotal evidence • Subjective method • Attitude surveys and the opinions of users and analysts

17. An Information Systems Example (1/4) • A simple hierarchy illustration • Includes both financial and non-financial criteria • Compare on the basis of corporate strategies • Investment risk • Revenue enhancing • Operating efficiency • Customer satisfaction • Market growth

18. An Information Systems Example (2/4) • Steps • Define the decision hierarchy • The goal is to rank the decision alternatives • Input the data • Expert ChoiceTM • The input data are manipulated using matrix algebra to produce the relative weights or priorities • Aggregation of all weights to produce a vector of composite relative weights between the criteria and the alternatives

19. An Information Systems Example (3/4)

20. An Information Systems Example (4/4) • Optimizing using integer programming • Maximize the AHP priority weights with the resource constraint • The optimal solution is (1,1,0,1,0) • The objective function value is equal to 0.709 • Higher values signify higher overall returns for the IT investments

21. Evidence From Two Case Studies (1/7) • Research methodology • Two case studies • Use the IT investment model • Contextual conditions could impact the outcomes • The goals • Ascertain the efficacy of the proposed ranking mechanism • Collect and report the attitudes, behaviors, and perceptions • Results were reviewed by the CIOs with minor corrections and revisions • Use multiple cases • Allow the investigator to replicate the results and improves generalizability • The study will show • Management involvement is necessary • Organizational structure affects the success of the ranking process • Hot and Lukewarm

22. Evidence From Two Case Studies (2/7) • Case study background of companies • Two U.S. utility companies • North-central region and southern region • Similarities • Generators of electricity, retail and wholesale markets, sold surplus power, and controlled their own transmission and distribution systems • Both had CIOs committed to IT planning

23. Evidence From Two Case Studies (3/7) • Hot • Smaller company under greater competitive pressure • Highly participative management structure with younger management • Previous experience in non-regulated industries • Highly committed to planning and the strategic use of IT • Lukewarm • Relatively secure markets • Issues of deregulation • Shortly put markets under competitive pressures • With traces of political rivalry • Top management was without experience outside their field • CEO and CIO had previous experience in non-regulated industries • Committed to planning and increasing returns on IT investments

24. Evidence From Two Case Studies (4/7) • IT planning and evaluation - Hot • Interest in IT planning and using IT strategically • Want a system • Satisfy all areas of management • Ask IT management for assistance in identifying technologies • That might allow revision of business processes to improve efficiencies and customer service • A combination of project evaluation tools • ROI, payback, and a corporate model • Useful but probably unreliable

25. Evidence From Two Case Studies (5/7) • IT planning and evaluation - Lukewarm • Delegate all IT planning to the CIO • Complain about the time and cost of implementing systems • IT steering committee • Composed of several senior managers • Rely heavily on the opinion of the CIO • The IT plan contained • A wish list of applications that continually changes with the political climate • Use a cost/benefit and payback approach • Selection of projects depends on • How well managers could creatively assign dollars to benefits

26. Evidence From Two Case Studies (6/7) • The Hot results • The decision criteria and sub-criteria • Originally developed by a team of IT managers • Later modified by other members of management • The participators were familiarity with AHP prior to the session • Use a modified Delphi technique to decide the weights • IT management played an impartial advisory role • The initial analysis was completed • Working with managers from finance, engineering, and marketing • Use both the AHP and integer programming models • Disadvantage • Total time involved in making the paired-comparisons and estimating other parameters • Advantage • Their understanding of the process would help to make future estimates easier and cut the time requirement • Select five IT investments with a capital requirement in excess of $18.5 million

27. Evidence From Two Case Studies (7/7) • The Lukewarm results • Expected to benefit from the results of the Hot experience • Partly implemented and with less success • Less efficient session • A cross-functional management team • Review and refine the comparisons after individual discussions with managers • The team would have final authority • The CEO supported the process but didn’t participate directly • On the advice of the team • The investments identified as strategic, high cost, and high risk were evaluated • 26 investments were analyzed • Many were overlapping and mutually exclusive • 8 investments were selected with a capital cost in excess of $34 million • Problems • Many managers continually requested revisions of the management team • Use a spreadsheet program • Perform a modified ROI analysis on the selected projects • IT managers felt • The direction was an improvement

28. Results of Investment Decisions (1/5) • Acceptance • Managers form both companies • Enthusiasm • The documentation for the methodology improved their understanding and made it easier for new managers to grasp • Hot • The internal environment and organizational structure are more conducive to acceptance of new processes • Lukewarm • Acceptance of the methodology had removed a major burden from IT planning • No longer incurred the wrath of managers who had not been funded • This supports • One of the benefits of the MOMC approach is the balancing of conflicting objectives of different users and stakeholders

29. Results of Investment Decisions (2/5) • Status of the IT investments selected • There was no immediate pressure to cut capital investments • Hot • Lower earnings-per-share • Delay one project to conserve cash and deploy resources to the other projects in order to realize the benefits more quickly • All of the projects were on or under schedule and under budget • Lukewarm • Benefit from reduced political tensions • Most of the projects were on schedule and within budget • The delayed project had suffered from a political tug-of-war about infrastructure issues • IT projects were an outstanding success

30. Results of Investment Decisions (3/5) • Status of selection process • Hot • Managers were continuing to modify and enhance the model • They wanted to be able to analyze individual investments on a stand-alone basis • The use of a program to quickly generate an initial set of paired-comparisons • Two strategic categories emphasized on valuation of intangible benefits • Tow over $1 million categories emphasized on risk analysis • Lukewarm • The CEO had to contend with several presidents of the operating companies • Less time to focus on IT • Time period was not sufficient • Little had been accomplished towards improving the process, primarily documentation of the process and the training of new managers • The CIO was confident • The next round of investment proposals would be handled more expeditiously

31. Results of Investment Decisions (4/5) • Generalizable findings • In one firm • The CEO had greater knowledge of IS • The CEO worked closely with the CIO and other managers followed the lead • In the other firm • The CEO had superficial knowledge • The CEO did not work closely with the CIO • Hot had capitalized on the new process to insure success and reduce the time requirement on management • By extending the model and adding administrative controls • Lukewarm accomplished less • Managers in both firms had an improved attitude • The new process improved the quality of information available to measure investment proposals, increased the involvement of managers, and added credibility to the final results • An investment’s potential return may be reduced • Because of implementation problems • The inability to control quality during software system development

32. Results of Investment Decisions (5/5) • Summary

33. Discussion (1/2) • Benefits • The ability of the model to handle a large number of criteria • The ability to represent both tangible and intangible items • The ability to model exclusivity and dependency of investments • The ability to quickly reflect revisions • The explanatory power of the model • The support for group decision-making • Limitations • The lack of a financial measure of profitability • The overall time requirements for management • The problem of valuing intangibles, although ameliorated, remained

34. Discussion (2/2)

35. Study Contributions (1/2) • Provide a tested process for prioritization and selection of IT investments • Identify benefits and limitations inherent within the process • Identify facilitators and inhibitors and generalizable findings to the approach • Assist the introduction of the process

36. Study Contributions (2/2) • Suggestions for future research • Further case studies • Suggested from different industries • Provide more insights into the completeness of the approach • Examine the impact of contextual variables on the success of the IT investment model • The balancing of investment risk was not tested in this study • The relationship between process credibility and subsequent development and implementation remains unresolved

37. Conclusions • The MOMC approach merits attention as a investment selection and ranking tool • Utilize AHP and integer programming • Improve the IT investment process • Strictly quantitative approaches have not yielded satisfactory results • Subjective approaches lack explanatory power and can not be easily adjusted to reflect new knowledge • Basing selection criteria on business strategies ensures the alignment of IT investments with these strategies