Educational Design Research: appropriate to address problems in educational practice

1. Educational Design Research:appropriate to address problemsin educational practice Tjeerd Plomp Professor Emeritus University of Twente Enschede, The Netherlands ICREAMS

2. Van den Akker (1999; director of Netherlands Institute for Curriculum Development) about research in domain of curriculum development and implementation: many ‘traditional’ research approaches such as experiments, surveys, correlational analyses, with their emphasis on description hardly provide prescriptions that are useful for design and development problems in education ICREAMS

3. In this presentation: Research functions and research designs What is design research Research question Quality criteria for interventions Generalizability in design research New book on design research ICREAMS

4. Focus of research project Dependent on research question/goal research may address a need to describe to explain, understand, predict to compare to evaluate to design/develop Called: research functions ICREAMS

5. Research approaches/designsto realize research functions e.g.: Survey: to describe, compare, evaluate Case studies: to describe, to explain Experiments: to explain, to compare Ethnography: to describe, to understand Correlational research: e.g. to describe, to explain Evaluation research: e.g. to determine the effectiveness of a program ICREAMS

6. Research designs, e.g.: Design research: to design/develop an intervention (e.g. program, T/L strategy, process, product, system, etc) with the purpose to improve practice andcontribute to body of knowledge OR to develop a theory (and possibly: to inform decision making and policy development) Note:intervention is container term ICREAMS

7. What is Educational Design Research (EDR)? Distinction between: Development studies aimed at research-based solutions for complex problems in educational practice Validation studies aimed at development or validation of a theory Research on interventions versus research through interventions (McKenney & Reeves, 2012) Often both angles in one study! ICREAMS

8. What is Educational Design Research (EDR)? Educational design research – type development studies- is the systematic analysis, design, development and evaluation of educational interventions with the dual aim of generating research-based solutions for complex problems in educational practice, and advancing our knowledge about the characteristics of these interventions and the process of designing and developing them. ICREAMS

9. What is Educational Design Research (EDR)? Educational design research – type validation studies - is the study of educational interventions – such as learning processes, learning environments and the like – with the purpose to develop and validate theories about such processes and how these can be designed. ICREAMS

10. Educational Design Research encompasses systematic educational design processes. The reverse is not true: Not all systematic educational design can be called research ICREAMS

11. Iterations of systematic design cycles PROBLEM design & develop prototype analysis evaluation Revision: yes? No? STOP

12. ID of problems with practitioners Development of prototype solutions Design research methodology can be expressed as follows: Revision Testing solutions in context Reflection to produce design principles (adapted from Reeves, UoGeorgia, Athens, USA) ICREAMS

13. Susan McKenney, Univ of Twente: ICREAMS

14. Phases in Design Research Preliminary phase: needs and content analysis, review of state-of-art of literature => conceptualisation & design specifications Development or Prototyping phase (iterative design phase + formative evaluation) <=> micro-cycles of research Assessment phase (semi-summative evaluation) ALL Phases: systematic reflection and documentation (resulting in design principles) ICREAMS

15. On terminology: (Educational) Design Research stands for a ‘family’ of related research approaches Design studies, design experiments Design-based (implementation) research Development research Participative action research Formative research Engineering research (e.g. Van den Akker et al., 2006): But a number of characteristics in common! ICREAMS

16. Common characteristics Interventionist Iterative Process oriented Utility oriented: merit of design measured Theory oriented: based upon & contributing to Involvement of practitioners ICREAMS

17. Context Z: A closer look: Intervention X Input => Process Outcomes Y1, Y2, …, Yn In general: Research goal may vary, but always: (i) design/development of intervention (ii) design principles or theory validation ICREAMS

18. ICREAMS

19. Research question in design research If research goal is development of intervention: What are the characteristics of an <intervention X> to realize outcomes Y in context Z OR in case of developing/validating a theory (tentatively): What is an adequate learning and teaching strategy for <topic B> in context C in order to cope with the main difficulties in the learning of this topic ICREAMS

20. Examples of research questions #1 what are the characteristics of an effective in-service programme for mathematics teachers through which they develop the ability to apply student-centred pedagogical methods (Tecle, 2003) ICREAMS

21. Examples of research questions #2 What is an adequate learning and teaching strategy for genetics in upper secondary biology education in order to cope with the main difficulties in learning and teaching genetics, and to promote the acquisition of a meaningful and coherent understanding of hereditary phenomena? ( Knippels, 2002) ICREAMS

22. Quality criteriafor interventions: • Relevance or Content Validity: • need for the intervention • based on contemporary scientific insights 2. Consistency or Construct Validity 3. Practicality - can be used in settings for which developed 4. Effectiveness: - yields desired results - cost-benefit ratio #3&4: Expected versus Actual ICREAMS

23. Shift in emphasis on criteria ICREAMS

24. Formative evaluation e.g.: ICREAMS

25. Yin (2003) about case study research: striving to generalize findings to some broader theory: Generalizability in Design Research: Context Z: Outcomes Y1, Y2, …, Yn Intervention X Input => Process design principles as intervention theory local instruction theory

26. Generalizability in Design Research: Yin (2003) about case study research: striving to generalize findings to some broader theory A theory (design principles or local theories) must be tested through replications of findings in a 2nd, 3rd or more cases: if same results in various cases, then …. the theory (= design principles or local theories) might be accepted for a much larger number of contexts. This replication logic is the same that underlies design research! Analytical generalization. (compare with ‘transferability’ in qualitative research) ICREAMS

27. Editedby Tjeerd Plomp & Nienke NieveenPublishedby: SLO – Netherlands Institutefor Curriculum Development. Enschede; 2013Part A: anIntroductionPart B: Illustrative casesFree available from:http://international.slo.nl/edr ICREAMS

28. Part B: 51 Cases of Design Research Case Selection Tool: Countries Educational Sectors Educational domains (>1) Main aim Educational focus Type of intervention (>1) See: http://international.slo.nl/edr ICREAMS

29. Part B: 51 Cases of Design Research Case Selection Tool: 22 countries: CHI, SIN, IND, KOR – USA (14), NET (11) Educational Sectors: all Educ domains (>1): curriculum (16), learning & instruction (29), subject-related pedagogy (34), school organization (3), instructional technology (7), ICT in education (11) Main aim: development studies (36); validation studies (14); implemtation studies (1) Educational focus Type of intervention (>1) ICREAMS

30. Part B: 51 Cases of Design Research Case Selection Tool: Countries Educational Sectors Educ domains (>1) Main aim Educational focus: teaching-learning methods; curriculum planning; ICT in education; school management & leadership; monitoring quality of educ; professional development; workplace learning Type of intervention:curr unit/course (34), educprogramme (12), learning task (9), monitoring system (4), assessment task (2), other (4) ICREAMS

31. Thank you!(t.plomp@utwente.nl) ICREAMS

32. Quotes illustrate that many researchers in various ‘corners’ of our field belief that there is a need for an alternative research approach that is directly relevant for problems in educational practice. ICREAMS

33. NOTE: One resq =>more than oneres function Example: if aim of research is to design and develop a teaching-learning strategy for acquiring the competency of mathematical modelling (in grade 11 & 12), then researchers may first want to understand and carefully describe what barriers students experience with mathematical modelling. But: there is primary research function ICREAMS

34. Why embark on Design Research? Various reasons – e.g.: Need to solve a complex problem in education practice for which no how-to-do guidelines are available Need to elaborate and/or validate a theory (e.g. whether the theory of Realistic Mathematics Education is valid in a certain context) ICREAMS

35. Problems may exist at all levels - for example: System level: Need for system for e-learning to serve specific group of students Institutional level: What are effective methods for collaborative learning ICREAMS

36. When is Design Research Appropriate? Eamonn Kelly (2009): Initial state(s) unknown or unclear Goal state(s) unknown or unclear Operators to move from initial states to goal states are unknown or how to apply the operators is unclear ICREAMS

37. Generalizability in Design Research: So: design principles must be seen as ‘heuristic’ statements providing guidance and direction, but do not give ‘certainties’. When we give proper weight to local conditions, any generalization is a working hypothesis, not a conclusion! (Lee Cronbach, 1975, cited by Tom Reeves) ICREAMS