Research Methods in Distance Education: Design Based Research

1. Research Methods in Distance Education: Design Based Research Terry Anderson PhD Seminar Leicester UK Feb. 2012

2. My Focus: The context of Distance Education Implementation • Disruptive innovation (Christensen, 2008) simpler, not wanted by main stream customers • Rapid gains in functionality • Cheaper • Adaptive • Moving from peripheral to mainstream (blended and online for full time students)

3. Research Paradigms

4. Research Paradigms

5. Research Paradigms Quantitative~ discovery of the laws that govern behavior Qualitative ~understandings from an insider perspective Critical~Investigate and expose the power relationships Design-based ~interventions, interactions and their effect in multiple contexts

6. Paradigm 1Quantitative Research • employs a scientific discourse derived from the epistemologies of positivism and realism.

7. “those who are seeking the strict way of truth should not trouble themselves about any object concerning which they cannot have a certainty equal to arithmetic or geometrical demonstration” • (Rene Descartes) • Inordinate support and faith in randomized controlled studies

8. Quantitative Example 1 – CMC Content Analysis • Anderson, Garrison, Rourke 1997-2003 • http://communitiesofinquiry.com - 9 papers reviewing results focusing on reliable , quantitative analysis • Identified ways to measure teaching, social and cognitive ‘presence’ • Most reliable methods are beyond current time constraints of busy teachers • Questions of validity • Serves as basic research as grounding for AI methods and major survey work of the future • Serves as qualitative heuristic for teachers and course designers

9. Quantitative – Meta-Analysis • Aggregates many effect sizes creating large N’s more powerful results. • Ungerleider and Burns (2003) • Systematic review of effectiveness and efficiency of Online education versus Face to face • The type of interventions studied were extraordinary diverse –only criteria was a comparison group • “Only 10 of the 25 studies included in the in-depth review were not seriously flawed, a sobering statistic given the constraints that went into selecting them for the review.”

10. 1928-2008 • distance delivery modes from correspondence schools, radio, television, video, and now e-learning • when the course materials and teaching methodology are held constant, there are no significant differences (NSD) in learner outcomes http://www.nosignificantdifference.org/ Slide from Tom Reeves

11. Ungerleider, C., & Burns, T. (2003). A systematic review of the effectiveness and efficiency of networked ICT in education. Ottawa: Industry Canada.

12. Is DE Better than Classroom Instruction?Project 1: 2000 – 2004 • Question: How does distance education compare to classroom instruction? (inclusive dates 1985-2002) • Total number of effect sizes: k = 232 • Measures: Achievement, Attitudes and Retention (opposite of drop-out) • Divided into Asynchronous and Synchronous DE Bernard, R. M., Abrami, P. C., Lou, Y. Borokhovski, E., Wade, A., Wozney, L., Wallet, P.A., Fiset, M., & Huang, B. (2004). How does distance education compare to classroom instruction? A meta-analysis of the empirical literature. Review of Educational Research, 74(3), 379-439.

13. Quantitative Research Summary • Can be useful especially when fine tuning well established practice • Provides incremental gains in knowledge, not revolutionary ones • The need to “control” context often makes results of little value to practicing professionals • In times of rapid change too early quantitative testing may mask beneficial positive capacity • Will we ever be able to afford blind reviewed, random assignment studies?

14. Paradigm 2 Qualitative Paradigm • Many different varieties • Generally answer the question ‘why’ rather then ‘what’, ‘when’ or ‘how much’? • Presents special challenges in distributed contexts due to distance between participants and researchers • Currently most common type of DE research (Rourke & Szabo, 2002)

15. Qualitative Example • Dearnley (2003) Student support in open learning: Sustaining the Process • Practicing Nurses, weekly F2F tutorial sessions • Phenomenological study using grounded theory discourse

16. Core category to emerge was “Finding the professional voice” Dearnley and Matthew (2003 and 2004)

17. Qualitative example 2 • Mann, S. (2003) A personal inquiry into an experience of adult learning on-line. Instructional Science 31 • Conclusions: • The need to facilitate the presentation of learner and teacher identities in such a way that takes account of the loss of the normal channel • The need to make explicit the development of operating norms and conventions • reduced communicative media there is the potential for greater misunderstanding • The need to consider ways in which the developing learning community can be open to the other of uncertainty, ambiguity and difference

18. 3rd ParadigmCritical Research • Asks who gains in power? • David Noble’s critique of ‘digital diploma Mills’ most prominent Canadian example • Are profits generated from user generated content or OERs exploitative? • Confronting the “net changes everything” mantra of many social software proponents. • Who is being excluded from online world?

19. See Norm Friesen’s Friesen, N. (2009) Re-thinking e-learning research: foundations, methods, and practices. Peter Lang Publishers

21. Why does Facebook own all the content that we supply? • Does the power of the net further marginalize the non-connected? • Why did the One Laptop Per Child project fail? Or did it?

22. Do Either Qualitative or Quantitative Methods Meet Real Needs of Practicing Distance Educators?

23. But what type of research has most effect on practice? • Kennedy (1999) - teachers rate relevance and value of results from each of major paradigms. • No consistent results – teachers are not a homogeneous group of consumers but they do find research of value • “The studies that teachers found to be most persuasive, most relevant, and most influential to their thinking were all studies that addressed the relationship between teaching and learning.”

24. But what type of research has most effect on Practice? • “The findings from this study cast doubt on virtually every argument for the superiority of any particular research genre, whether the criterion for superiority is persuasiveness, relevance, or ability to influence practitioners’ thinking.” Kennedy, (1999)

25. Design-Based Research • Developed from frustration of the lack of impact of educational research on educational systems.

26. 4th ParadigmDesign-Based Research • Related to engineering and architectural research • Focuses on the design, construction, implementation and adoption of a learning initiative in an authentic context • Related to ‘Development Research’ • Closest educators have to a “home grown” research methodology

27. Design-Based Research Model Phase 2 Phase 4 Phase 1 Phase 3 Development of Solutions Informed by Existin Design Principles and Technological Innovations Analysis of Practical Problems by Research and Practitioners in Collaboration Reflection to Produce “Design Principles” and Enhance Solutions Implementation Iterative Cycles of Testing and Refinement of Solutions in Practice Refinement of Problems, Solutions, Methods, and Design Principles (Reeves, 2006, p. 59) Reeves, T. C. (2006). Design research from the technology perspective. In J. V. Akker, K. Gravemeijer, S. McKenney, & N. Nieveen (Eds.), Educational design research (pp. 86-109). London: Routledge.

28. Design-Based Research Studies • iterative, • process focused, • interventionist, • collaborative, • multileveled, • utility oriented, • theory driven and generative • (Shavelson et al, 2003)

29. Critical characteristics of design experiments • According to Reeves (2000:8), Ann Brown (1992) and Alan Collins (1992): • addressing complex problems in real contexts in collaboration with practitioners, • integrating known and hypothetical design principles with technological affordances to render plausible solutions to these complex problems, and • conducting rigorous and reflective inquiry to test and refine innovative learning environments as well as to define new design-principles.

30. Design-based research • Methodology developed by educators for educators • Developed from American pragmatism – Dewey (Anderson, 2005) • Recent Theme Issues: • The Journal of the Instructional Sciences, (13, 1, 2004), • Educational Researcher (32, 1, 2003) and • Educational Psychologist (39, 4, 2004) • See bibliography at http://cider.athabascau.ca/CIDERSIGs/DesignBasedSIG/ • My article at www.cjlt.ca/abstracts.html

31. Anderson, T., & Shattuck, J. (2012). Design-Based Research: A Decade of Progress in Education Research? Educational Researcher, 41(Jan/Feb.), 16-25. Retrieved from http://edr.sagepub.com/content/41/1/7.full.pdf+html.

32. Anderson, T., & Shattuck, J. (2012). Design-Based Research: A Decade of Progress in Education Research? Educational Researcher, 41(Jan/Feb.), 16-25. Retrieved from http://edr.sagepub.com/content/41/1/7.full.pdf+html.

33. Design Tradition • “Learning and productivity are the results of the designs (the structures) of complex systems of people, environments, technology, beliefs and texts” New London Group 2000 • DBR opens the door for teachers, researchers and learners to become designers, not merely consumers, bosses or observers .

34. Integrative Learning Design(Bannan-Ritland, 2003)

35. “design-based research enables the creation and study of learning conditions that are presumed productive but are not well understood in practice, and the generation of findings often overlooked or obscured when focusing exclusively on the summative effects of an intervention” Wang & Hannafin, 2003

36. Iterative because • ‘Innovation is not restricted to the prior design of an artifact, but continues as artifacts are implemented and used” • Implementations are “inevitably unfinished” (Stewart and Williams (2005) • intertwined goals of (1) designing learning environments and (2) developing theories of learning (DBRC, 2003)

37. Design Based research and the Science of Complexity • Complexity theory studies the emergence of order in multifaceted, changing and previously unordered contexts • This emerging order becomes the focus of iterate interventions and evaluations • Order emerges at the “edge of chaos” in response to rapid change, and failure of previous organization models

38. DBR ExamplesCall Centres At Athabasca: Answer 80% of student inquiries Savings of over $100,000 /year Anderson, T. (2005). Design-based research and its application to a call center innovation in distance education. Canadian Journal of Learning and Technology, 31(2), 69-84

39. Design Based research in Action • Phase 1 Exploration – surveys, talking to faculty and tutors, investigating open source tools, setting research questions • Phase 2. Building the intervention – Elgg through two versions and 85 plugins (on going)

40. Design Based research in Practice • Athabasca Landing • Elgg based • Started in 2008 • 3500 users • Unpaced • Paced Courses • Informal Learning • Staff and alumni networking • Problem of critical mass

41. Phase 3 Evaluation – Before and after survey’s see: • Anderson, T., Poelhuber, B., & McKerlich, R. (2010). Self Paced Learners Meet Social Software. Online Journal of Distance Education Administration • Dr students – Use of past student archives • Ongoing iterations and development of tools • Phase 4 – Design Principles • Development of design principles/patterns Nets and Sets Dron, J., & Anderson, T. (2009). How the crowd can teach. In S. Hatzipanagos & S. Warburton (Eds.), Handbook of Research on Social Software and Developing Community Ontologies.

42. Network Tool Set (example) Text Text Stepanyan, Mather & Payne, 2007

43. Access Controls in Elgg

44. Need to study usability, scalability and innovation adoption within bureaucratic systems • Allow knowledge tools to evolve in natural context through supportive nourishment of staff

45. Conclusion • Education research is grossly under-resourced to meet the magnitude of opportunity and demand • Paradigm wars are unproductive • Design-based research offers a promising new research design model • It can be used for Doctoral dissertations see • Herrington, J., McKenney, S., Reeves, T., & Oliver, R. (2007). Design-based research and doctoral students: Guidelines for preparing a dissertation proposal.

46. Your comments and questions most welcomed! Terry Anderson terrya@athabascau.ca Blog: terrya.edublogs.org Twitter: terguy