1 / 42

Design Science Research: Rigorous and Relevant

Design Science Research: Rigorous and Relevant. Alan R. Hevner University of South Florida National Science Foundation. NSF Disclaimer.

darrend
Télécharger la présentation

Design Science Research: Rigorous and Relevant

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Design Science Research: Rigorous and Relevant Alan R. Hevner University of South Florida National Science Foundation

  2. NSF Disclaimer • Any opinion, findings, and conclusions or recommendations expressed in this material are those of the author and do not necessarily reflect the views of the National Science Foundation. Arizona Seminars

  3. Outline • Introduction • Design Science Research • Research Impacts • Design Research Guidelines • Three Cycles of Design Research • Relevance • Rigor • Design – Build and Evaluate • Questions and Discussion Arizona Seminars

  4. Research Portfolio • Ph.D. in Computer Science from Purdue • Faculty Member at Minnesota (CS), Maryland (IS), and USF (IS) • Database Systems • Query Optimization on Distributed Database Systems • Query and File Allocation Algorithms • Software Engineering • Cleanroom Software Engineering • Metrics and Software Testing • Information Systems Analysis and Design • Health Care Data Warehousing and Data Mining • Telecommunication Systems Arizona Seminars

  5. Design Science Research • Sciences of the Artificial, 3rd Ed. – Simon 1996 • A Problem Solving Paradigm • The Creation of Innovative Artifacts to Solve Real Problems • Design in Other Fields – Long Histories • Engineering, Architecture, Art • Role of Creativity in Design • Design Research in Information Systems • How to Perform Research in Design Science ! • Formal Design Science Research Theories ? Arizona Seminars

  6. MISQ 2004 Research Essay • A. Hevner, S. March, J. Park, and S. Ram, “Design Science Research in Information Systems,” Management Information Systems Quarterly, Vol. 28, No. 1, March 2004, pp. 75-105. • Historically, the IS field has been confused about the role of ‘technical’ research. • Technical researchers felt out of the mainstream of ICIS/MISQ community. • Formation of Workshop on Information Technology and Systems (WITS) in 1991 • Initial Discussions and Papers • Iivari 1991 – Schools of IS Development • Nunamaker et al. 1991 – Electronic GDSS • Walls, Widmeyer, and El Sawy 1992 – EIS Design Theory • March and Smith 1995 from WITS 1992 Keynote • Encouragement from IS Leaders such as Gordon Davis, Ron Weber, and Bob Zmud • Allen Lee, EIC of MISQ, invited authors to submit essay on Design Science Research in 1998 • Four Review Cycles with multiple reviewers • Published in 2004 Arizona Seminars

  7. MISQ Paper Impacts • Professional Impact • Raised visibility of DSR in IS • Identified interdisciplinary synergies (e.g., CS, Engineering design, management, etc.) • Identified relationships among research paradigms (e.g., behavioral, economic, etc.) • Citation Impact • Over 200 citations on Google Scholar • International Impact • Doctoral Education and Research Impact • Concerns: • Over reliance on seven guidelines for research design • Silence on role of theory in DSR • Pragmatic nature of DSR • Roles of Rigor and Relevance Arizona Seminars

  8. DESRIST Conferences • Design Science Research in Information Technology and Systems • Claremont 2006 • Pasadena 2007 • Atlanta 2008 – May 7-9, 2008 • Interdisciplinary Participation • Website: • http://desrist2008.cis.gsu.edu/ Arizona Seminars

  9. European Conference on ISJune 7-9, 2007 • St. Gallen, Switzerland • ECIS Theme - “Relevant Rigour – Rigorous Relevance” • Keynote Address • Title – “Design Science Research as a Rigorous Approach for Relevant Solutions” • Panel • “Is Design Science Research the Answer to the Relevance Problem?” • AACSB Call for Relevant Business Research in “Impact of Research” Report Arizona Seminars

  10. International Conference on ISDecember 9-12, 2007 • Montreal, Canada • Design Science Track • 40 Submissions – 10 Papers Accepted • At ICIS 2006 in Milwaukee, 52 Submissions – 12 Papers Accepted • Workshop on Information Technology and Systems (WITS) Arizona Seminars

  11. Design Science Research in Major IS Journals • MISQ • Special Issue on Design Science to appear in 2008 • New Senior Editor and Associate Editors for DS Papers • ISR • Encouragement of DS Submissions • Role of ACM and IEEE Publications • Senior Scholars List of Top IS Journals Arizona Seminars

  12. IS Research Framework • Information Systems (IS) are complex, artificial, and purposefully designed. • IS are composed of people, structures, technologies, and work systems. • Two Basic IS Research Paradigms • Behavioral Research – Goal is Truth • Design Research – Goal is Utility Arizona Seminars

  13. IS Research Cycle Arizona Seminars

  14. Design Science • Design is a Artifact (Noun) • Constructs • Models • Methods • Instantiations • Design is a Process (Verb) • Build • Evaluate • Design is a Wicked Problem • Unstable Requirements and Constraints • Complex Interactions among Subcomponents of Problem and resulting Subcomponents of Solution • Inherent Flexibility to Change Artifacts and Processes • Dependence on Human Cognitive Abilities - Creativity • Dependence on Human Social Abilities - Teamwork Arizona Seminars

  15. Environment IS Research Knowledge Base Relevance Rigor • People • Roles • Capabilities • Characteristics • Experience • Organizations • Strategies • Structure • Culture • Processes • Technology • Infrastructure • Applications • Communications Architecture • Development Capabilities • Foundations • Theories • Frameworks • Experimental Instruments • Constructs • Models • Methods • Instantiations • Methodologies • Experimentation • Data Analysis Techniques • Formalisms • Measures • Validation Criteria • Optimization • Develop / Build • Theories • Artifacts Applicable Knowledge Business Needs Assess Refine • Justify / Evaluate • Analytical • Case Study • Experimental • Field Study • Simulation Application in the Appropriate Environment Additions to the Knowledge Base

  16. Guidelines for DS Research in IS • Purpose of Seven Guidelines is to Assist Researchers, Reviewers, Editors, and Readers to Understand and Evaluate Effective Design Science Research in IS. • Researchers will use their creative skill and judgment to determine when, where, and how to apply the guidelines to projects. • All Guidelines should be addressed in the Research. Arizona Seminars

  17. Design Science Guidelines

  18. Design Science Case Studies • Three Exemplars in MISQ Paper • Gavish and Gerdes DSS 1998 • Aalst and Kumar ISR 2003 • Markus, Majchrzak, and Gasser MISQ 2002 • Recent Doctoral Research Project • Monica Tremblay - Uncertainty in the Information Supply Chain: Integrating Multiple Health Care Data Sources • Artifacts: • ISC Metrics – Completeness, Volatility • User Presentations of Metrics • Evaluation – Focus Groups • Details Arizona Seminars

  19. Scandinavian Journal of IS • Upcoming Issue on Design Research • Iivari, “A Paradigmatic Analysis of Information Systems as a Design Science” • Hevner, “A Three Cycle View of Design Science Research” • Other Responses to the Iivari paper Arizona Seminars

  20. Three Cycles of DS Research Design Science Knowledge Base Environment • Foundations • Scientific Theories & Methods • Application Domain • People • Organizational Systems • Technical Systems • Problems & Opportunities Build Design Artifacts & Processes • Experience & Expertise • Rigor Cycle • Grounding • Additions to KB • Relevance Cycle • Requirements • Field Testing Design Cycle Evaluate • Meta-Artifacts (Design Products & Design Processes) Arizona Seminars

  21. The Relevance Cycle • The Application Domain initiates Design Research with: • Research requirements (e.g., opportunity, problem, potentiality) • Acceptance criteria for evaluation of design artifact in application domain • Field Testing of Research Results • Does the design artifact improve the environment? • How is the improvement measured? • Field testing methods might include Action Research or Controlled Experiments in actual environments. • Iterate Relevance Cycle as needed • Artifact has deficiencies in behaviors or qualities • Restatement of research requirements • Feedback into research from field testing evaluation Arizona Seminars

  22. The Rigor Cycle • Design Research Knowledge Base • Design Theories • Engineering Methods • Experiences and Expertise • Existing Design Artifacts and Processes • Research Rigor is predicated on the researcher’s skilled selection and application of appropriate theories and methods for constructing and evaluating the artifact. • Additions to the Knowledge Base: • Extensions to theories and methods • New experiences and expertise • New artifacts and processes Arizona Seminars

  23. Design Theories • Is a kernel ‘Design Theory’ essential for rigorous design research? • I would contend that the answer is No. • Design research can be grounded on: • Design Theory • Opportunities, Problems, Potentialities • Analogies, Metaphors • Creative Inspiration and Insight Arizona Seminars

  24. Design Cycle • Rapid iteration of Build and Evaluate activities • The hard work of design research • Build – Create and Refine artifact design as both product (noun) and process (verb) • Evaluation – Rigorous, scientific study of artifact in laboratory or controlled environment • Continue Design Cycle until: • Artifact ready for field test in Application Environment • New knowledge ready for inclusion in Knowledge Base Arizona Seminars

  25. Design Science Challenges • Inadequate Theory Base for Scientific and Engineering discipline • Insufficient Sets of Constructs, Models, Methods, and Tools in Knowledge Base to Represent real-world Problems and Solutions • Design is still a Craft relying on Intuition, Experience, and Trial-and-Error • Design Science Research is perishable as technology advances rapidly • Rigorous Evaluation Methods are difficult to apply in Design Science Research • Communication of Design Science Results to Managers is Essential but a Major Challenge Arizona Seminars

  26. Questions and Discussion Arizona Seminars

  27. Design as an Artifact • The IT Artifact is the ‘core subject matter’ of the IS field. • Artifacts are innovations that define the ideas, practices, technical capabilities, and products through which the analysis, design, implementation, and use of IS can be accomplished. • Design Science Research in IS must produce an Artifact • Construct, Model, Method, Instantiation • Research Design vs. Routine Design • Innovation vs. Use of Known Techniques Arizona Seminars

  28. Problem Relevance • Research Motivation • The Problem must be real and interesting. • Problem solving is a search process using actions to reduce or eliminate the differences between the current state and a goal state [Simon 1999]. • Design Science Artifact must be relevant and useful to IS practitioners - Utility. Arizona Seminars

  29. Design Evaluation • Rigorous Evaluation of the Utility, Quality, and Beauty (i.e., Style) of the Design Artifact. • Evaluation provides feedback to the Construction phase for improving the artifact. • Design Evaluation Methods Arizona Seminars

  30. Design Evaluation Methods

  31. Research Contributions • What is New and Interesting? • Does the Research make a clear contribution to the business environment, addressing a relevant problem? • The Design Artifact • Exercising the artifact in the problem domain adds value to the IS practice • Foundations • Extend and improve foundations in the design science knowledge base • Methodologies • Creative development and use of methods and metrics Arizona Seminars

  32. Research Rigor • Use of Rigorous Research techniques in both the Build and Evaluate phases • Building an Artifact relies on mathematical foundations to describe the specified and constructed artifact. • Principles of Abstraction and Hierarchical Decomposition to deal with Complexity • Evaluating an Artifact requires effective use of techniques in previous slide. • Research must be both Relevant and Rigorous Arizona Seminars

  33. Design as a Search Process • Good design is based on iterative, heuristic search strategies. • Simon’s Generate/Test Cycle • Problem Simplification and Decomposition • Modeling Means, Ends, and Laws of the Problem Environment • The Search for Optimal Solutions may not be feasible or tractable. • The Search for Satisfactory Solutions may be the best we can do - Satisficing Arizona Seminars

  34. Communication of Research • Technical audiences need sufficient detail to construct and effectively use the artifact. • How do I build and use the artifact to solve the problem? • Managerial audiences need an understanding of the importance of the problem and the novelty and utility of the artifact. • Should I commit the resources (staff, budget, facilities) to adopt the artifact as a solution to the problem? • Research presentation must be fitted to the appropriate audience (e.g., journal). Arizona Seminars

  35. Problem Relevance • Problem Statement:Public policy knowledge workers draw on a set of pre-existing tools when acquiring data from multiple data sources available from the information supply chain. The data acquisition process and the task of correctly combining and manipulating data from multiple data sources in the information supply chain have many challenges: data are unbounded, data definitions and schemas vary, and there is no guarantee of data quality. In most cases, knowledge workers make decisions with available information and use “gut instinct” or experience to choose the correct course of action when data sources conflict or do not match expectations. These challenges are made even more complicated by the knowledge worker’s own judgment biases. Existing tools can aid knowledge workers, yet the lack of integration among these tools aggravate cognitive and behavioral biases and result in missed opportunities for knowledge creation. Arizona Seminars

  36. Research Questions • Can we design metrics and a decision method that will integrate multiple data sources with varying degrees of data quality in the IS supply chain to better support public policy decision makers? • Can we evaluate the utility, quality, and efficacy of the metrics and method? Arizona Seminars

  37. Research Rigor • Designs will be grounded by: • Data Products Foundations [Shankaranarayan et al. 2003] • Data Quality Foundations [Wang et al. 1997] • Behavioral Decision-Making Foundations[Tversky and Kahneman 1982] • Design Evaluation will be performed in two phases: • Field Study of Public Health Decision-Makers • Focus Groups of Experts Arizona Seminars

  38. Design as a Search Process • Two Research Cycle Iterations • Cycle 1: • Initial Designs and Instantiation • Practical Evaluation in Field Study with Feedback • Cycle 2: • Improved Design and Instantiation • Evaluation in Focus Groups to Study Impacts on Behavioral Biases Arizona Seminars

  39. Design as an Artifact • The Artifacts are: • Data Quality Metrics on Data Products • New Algorithms for Calculating Data Quality Metrics on Data Products • New Methods for Comparing and Integrating Data Products • New Human-Computer Interface Presentations to support Decision-Making Arizona Seminars

  40. Design Evaluation • Cycle 1 – Field Study of How Public Health Policy workers use initial metrics to support Decision-Making. • Cycle 2 – Focus Groups on the use of the metrics and methods to study User Biases on Public Health Policy Decision-Making • Experts in the Health Care Field • Scenarios will be drawn from Field Study Arizona Seminars

  41. Research Contributions • The Design Artifacts • Metrics, algorithms, methods and interfaces will add clear value to Public Health Policy environments • Foundations • New models and algorithms for calculating data quality metrics • New methods of integrating multiple data products • New methods of data product presentation to decision-makers Arizona Seminars

  42. Communication of Research • Presentation of Research Results in top-quality IS journals and conferences • Initial study that motivated this project: • Tremblay, Fuller, Berndt, and Studnicki, “Doing More with More Information: Changing Healthcare Planning with OLAP Tools,” Decision Support Systems, In Press, Available on DSS Elsevier Website, 2006. Arizona Seminars

More Related