IT Research Seminar February 10, 2003

1. IT Research Seminar February 10, 2003 Measuring IS Success: Quest for the Dependent Variable in IS ResearchThe JourneySpecial IT Research SeminarWilliam H. DeLoneKogod School of BusinessFebruary 8, 2008

2. IS Success Research Stream • UCLA Dissertation – Successful use of IS by SMEs; MISQ 1988 • IS Success: Quest for the Dependent Variable; ISR 1992 = “DeLone & McLean Success Model” • Ten Year Update; JMIS, Spring 2003 • Measuring E-Commerce Success, International Journal of Electronic Commerce, Fall 2004 • IS Success Models, Dimensions, Measures, and Interrelationships, under review European Journal of Information Systems

3. Research Motivation • UCLA measurement course (Mason & Swanson) • Peter Keen’s 5 research challenges for MIS (ICIS 1) – What is the dependent variable? How does MIS establish a cumulative tradition? • Dependent variable for PhD study in SMEs; Use & Impact • If you can’t measure it, you can’t research it

4. Quest for MIS Dependent Variable (ISR, 1992) – “IS Success” • Purposes • Organize & summarize MIS research related to defining the dependent variable • Measure progress on defining the dependent variable • Improve IS research practice • Contribute to “Cumulative Tradition” – compare apples to apples

5. Theoretical Underpinnings • Mason’s 1978 article on measuring information output • Production->Product->Receipt->Influence on Recipient->Influence on System • Mason’s work was based on Shannon & Weaver’s 1949 Theory of Communications book – Levels of communications measurement = technical, semantic, effectiveness • DeLone & McLean Success Categories • System Quality->Information Quality-> Use->Satisfaction->Individual Impact-> Organization Impact

6. Methodolgy • Literature review • IS Articles from 1981 to 1988 • Framework/model for organizing success measures • Empirical measures grouped into six success categories

7. Figure 1 D&M IS Success Model System Quality Use Individual Impact Organizational Impact Information Quality User Satisfaction D & M IS Success Model

8. Results/Conclusions • A simple and parsimonious framework for organizing IS success measures • IS Success - multi-dimensional and interdependent construct • Selection of measures is contingent on objectives and context of study • Reduce IS Success measures; build on existing measures=> “cumulative tradition” – comparison of results • Need for organizational impact measures

9. The Challenge • “ This success model clearly needs further development and validation before it could serve as a basis for the selection of appropriate IS measures.” (p.88)

10. Ten-Year Update (JMIS 2003) • Purposes • Model Utility • Validate the Model – Causal relationships • Update the Model to recognize the changes in IS • Assess progress in IS success measurement

11. Utility of D&M Success Model • Cited by more than 285 refereed journal and proceedings articles between 1993 and 2002 (according to a recent study in CAIS vol. 20, ISR 1992 article is the most cited article in MIS over the last 15 years; > 400 citations) • Most articles used the Model “as a drunkard uses a lamppost for support rather than illumination.” Statement was rejected by editor.

12. Model Validation • Seddon & Kiew (1994) validated 4 of the proposed associations • Rai et. al. validated overall model using goodness of fit tests (ISR 2002) • Fifteen additional empirical studies validated one or more proposed associations

13. Updated Model • IS move from production function to production & service function => importance of service quality • Impacts on whom? Individuals, groups, org., industry, economy => Net Benefits dimension with contextual definition

14. Updated Model

15. Assessment & Conclusions • D&M IS Success Model supported and validated • More careful attention to multidimensionality of IS Success • Confusion between Independent & Dependent variables • Operationalization of the model is contextual (Seddon et. al.) • Progress in parsimonious measure development is slow • System Use is misunderstood and undervalued • Use and satisfaction are not a substitute for Net Benefits measures (Yuthas & Young, 1998)

16. Recent Advances in Success Measurement

17. Application of Model: E-Commerce Success (IJEC 2004) • Premise: E-commerce does not need a new measurement paradigm but some new measures • Apply the D&M IS Success Model for measuring E-Commerce Success • Literature Review and classification of emerging e-commerce effectiveness measures • Case examples of application of IS Success Model to E-Commerce success measurement

18. Application of Model: ERP Success • Article by Sedera & Gable (ICIS 2004) – Government & University ERPs • Most comprehensive empirical test of model • Four dimensions of IS Success – • System quality • Information quality • Individual impact • Organizational impact • ** 27 Item measures

19. Validated Measures for IS Success Source: Sedera and Gable (2004) • System Quality - ease of use, ease of learning, user requirements, system features, system accuracy, flexibility, sophistication, integration, and customization • Information Quality – availability, usability, understandability, relevance, format, and conciseness • Individual Impact – learning, awareness/recall, decision effectiveness, and individual productivity • Organizational Impact - organizational costs, staff requirements, cost reduction, overall productivity, improved outcomes/outputs, increased capacity, e-Government, and business process change • What happened to Use and User Satisfaction?

20. Measuring Systems/Information Usage (Don Marchand) • Don Marchand, Professor of Strategy & Information Management at IMD – Switzerland • 20% of value realization is in deployment: 80% of value realization is in information and IT USAGE • Challenge – Information Usage is difficult to see, measure and manage

21. Measuring System Usage (Burton-Jones & Straub, ISR 2006) • Problem – over-simplified measures of use; e.g. duration of use and breadth of use • Importance of context & purpose • Elements of usage include: systems, user and task • Proposed Dimensions of Systems Usage – Cognitive Absorption (engagement) + Deep Structure (use of system features that support a specific task) • Systems Usage empirically related to task performance

22. Current Research • Measuring IS Success: Models, Dimensions, Measures and Interrelationships under review at European Journal of Information Systems • Assessing the state of IS Success Measurement (via D & M Model) • Summarizing empirical literature, 1992 to 2006 • Contributions • Summarizes measures used for each dimension of success • Validates significant relationships for 10 of the 15 causal relationships in the Updated Success Model based on empirical studies

23. Conclusions • DeLone & McLean IS Success Model remains the most popular, comprehensive framework for guiding the development of the dependent variable in IS research and for comparing results • More IS researchers are using the D&M Model to inform and guide their measurement of the dependent variable rather than to merely justify their choice of measures • Information quality and information use are under studied • Satisfaction is over used as a surrogate for success; therefore much information is lost • Bias toward ease of data collection threatens rigor and understanding