Evaluation Research into the effectiveness of university learning environments

1. Evaluation Research into the effectiveness of university learning environments Rob Phillips Academic Consultant Educational Design Project, School of Education, Curtin University, Perth Adjunct Associate Professor, School of Education, Murdoch University, Perth

2. My Background • 1980s • Research scientist – theoretical chemistry and computer science – Germany and Australia • 1990s • Interactive multimedia designer and project manager • 2000s • Educational designer • 2010s • Learning and Teaching Policy work • Academic consultant – mentoring and educational design

3. Introductions • Why are you here?

4. Book advert

5. Disclaimer • The book is largely about e-learning • The ideas apply well to any learning environment • almost all include technology these days

6. Overview • Phenomena • The e-learning lifecycle • e-learning evaluation research • Cyclical evaluation research approaches • Divide and Conquer techniques • Work on own project

7. Fundamentals of Research • Inquiry into the behaviour of phenomena • Exercise of human curiosity • Attempting to solve problems in the world • Attempting to understand the world

8. In Broad Terms • What sort of research do you do? • What do you investigate? • What are the main characteristics of this research? • Work in pairs

9. Fundamentals of Research • Inquiry into the behaviour of phenomena • Exercise of human curiosity • Attempting to understand the world • Attempting to solve problems in the world

10. What is a phenomenon? • Inquiry into the behaviour of phenomena • What is a phenomenon? • an instance of something – an artefact or event that is known through the senses

11. Natural Phenomena • They exist naturally in our world • the seasons, gravity and waves on the ocean • we can observe and measure them • Other natural phenomena cannot be directly observed • the inner workings of atoms and molecules • their behaviour can be measured

12. Artificial Phenomena • Phenomena/ artefacts which we create through a design activity • Tangible – aircraft, cars, etc. • Intangible – computer software

13. Event Phenomena • An artefact or event that is known through the senses • In Education • a coming-together of teacher, learners and various resources in a particular setting, usually following some design activity • Learning Environment

14. Summary • Natural phenomena • 1 phenomenon – many instances • Artificial phenomena • Many phenomena – many instances • Event Phenomena • Many phenomena – single instances

15. Paradigms vs Ideology Paradigms • The set of practices that define a scientific discipline at any particular period of time (wikipedia) • A world view underlying the theories and methodology of a particular scientific subject (http://oxforddictionaries.com)

16. Ideology • The set of beliefs characteristic of a social group or individual (http://oxforddictionaries.com) • A comprehensive and coherent set of basic beliefs about political, economic, social and cultural affairs that is held in common by a sizable group of people within a society. (http://www.socialsciencedictionary.com)

17. Paradigms vs Ideology • Is your research approach based on a paradigm or an ideology?

18. Overview • Phenomena • The e-learning lifecycle • e-learning evaluation research • Cyclical evaluation research approaches • Divide and Conquer techniques • Work on own project

19. E-learning Artefacts and Environments • We design and use e-learning artefacts (intangible artificial phenomena) • learning objects, learning management systems, e-portfolios • e-learning environments (event phenomena)add the human context • an interaction between learners, teachers and resources in a set context

20. Inquiring About the World (IATW) • New science unit for a new primary and early childhood degree • First offered in 2013 • Teaching problem • How to make science interesting and engaging • Design • Authentic learning approach • Students investigate a self-chosen environmental issue over the whole semester

21. Learning Outcomes • Develop lifelong learning skills including: analytical thinking, information management, technology skills and communication skills • Apply processes of inquiry to carry out a scientific investigation • Report on the outcomes of the processes of inquiry using a range of Information and Communication Technologies • Evaluate the outcomes of the inquiry processes in the context of the science and humanities content from the Australian Curriculum

22. Assessment

23. Video • IATW Flash video

24. The Learning Lifecycle Learning lifecycle Phase Baseline Design Develop Evaluate Design Trial learning env’t Learning environment Mature Learning environment

25. The IATW Lifecycle • On- campus and Open Universities Australia (OUA)

26. Overview • Phenomena • The e-learning lifecycle • e-learning evaluation research • Cyclical evaluation research approaches • Divide and Conquer techniques • Work on own project

27. My Invention • The Kugelschreiber • A better way to write!

28. Researching the Kugelschreiber • I know that this technology is better than old, pencil technology • Compare the two, in writing a test • Treatment and control groups

29. Researching the Kugelschreiber • I know that this technology is better than old, pencil technology • Compare the two, in writing a test • Treatment and control groups Kugelschreiber results were slightly worse than pencil results, but the difference was not significant at the 90% confidence level. The Kugelschreiber is no worse than a pencil.

30. Inquiry into Phenomena • Natural • Goal: observing how natural phenomena work • and ultimately understand them • Artificial • Goal: design artefacts, improve their design and use them • and ultimately understand them • Event • Goal: design events, improve their design • and ultimately understand them

31. Goals of Research • Be clear about the phenomenon you are investigating • Clearly define the goals and questions of your research • Choose the research approach accordingly

32. Summary so Far • We have thought about the phenomenon of e-learning • We have different research goals at different stages of the e-learning lifecycle • How will we conduct this research?

33. General Research Approaches • Pure versus applied research • Hard versus softresearch • Research and development • Research then development Becher, T. (1989). Academic tribes and territories: intellectual enquiry and the cultures of disciplines. Buckingham: Society for Research into Higher Education. Jones, C., Zenios, M., & Griffiths, J. (2004). Academic use of digital resources: Disciplinary differences and the issue of progression Proceedings of the 4th Networked Learning Conference. Lancaster University, England, UK.

34. Boyer’s Four Scholarships • Scholarship of discovery • Scholarship of integration • Scholarship of application • Scholarship of teaching Boyer, E. L. (1990). Scholarship reconsidered - Priorities of the professoriate. New Jersey: Carnegie Foundation for the Advancement of Teaching. Boyer, E. L. (1996). The scholarship of engagement. Journal of Public Outreach, 1(1), 11-20.

35. Pasteur’s Quadrant Stokes, D. E. (1997). Pasteur's Quadrant: Basic science and technological innovation. Washington, D.C.: The Brookings Institution.

36. What is evaluation? • The term is used imprecisely • Evaluation implies judgement and decisions • Formative evaluation: • judgements about improving something • Summative evaluation: • judgements about the merit or worth of something

37. Investigating a New Learning Environment • Questions: • Does the environment function as designed? • How can the environment be improved? • These are evaluation judgements • Is it any good???

38. Investigating a Mature Learning Environment • A study of a mature learning environment may shed light on • how learners engaged with the designed learning environment • why some learners achieved at different levels • how some learners used the learning environment to achieve a deeper understanding • These are educational research questions • The goal is increased understanding

39. Evaluationvs Research • Studies of learning environments involve a mixture of • Evaluation: making judgements about improvement and value and worth • Research: increasing understanding • Continuum between judgements and understanding • Does it function as designed? • How can it be improved? • How does this lead to improved student learning? • Similar methods – different goals Judgments Understanding

40. IATW - currently • Pre- and post-test • Confidence with ICT • Knowledge of scientific inquiry processes • Anonymous survey of students • Did they like it?

41. IATW questions to ask • Was the website usable? • What things didn’t work as well as they should? • Was there evidence that students engaged with the authentic task? • Will the learning design scale up for 2000 OUA students?

42. Overview • Phenomena • The e-learning lifecycle • e-learning evaluation research • Cyclical evaluation research approaches • Divide and Conquer techniques • Work on own project

43. The Learning Lifecycle Learning lifecycle Phase Baseline Design Develop Evaluate Design Trial learning env’t Learning environment Mature Learning environment

44. Evaluation Research across the Learning Lifecycle Learning lifecycle Baseline analysis Design Develop Evaluate Design evaluation Formative evaluation Formative evaluation Summative evaluation Effectiveness research

45. e-learning Research Approaches • Need to apply to an artificial phenomenon • Reflect the e-learning life cycle • Cyclical research approaches • Design-based research • Action Inquiry

46. Design-based Research • Focus on a designed learning environment • A cyclical approach that originated in engineering and other design fields • Consistent with • a mixture of evaluation and research • Stokes’ views on quest for understanding and consideration of use

47. Action Inquiry • Focus on personal practice • Local focus • Little concern with theory

48. Drawing the Threads Together • e-learning environments are designed phenomena • they follow a lifecycle • Cyclical research approaches • Mix of evaluation and research • make sure it works first • Now – onto the practical stuff

49. Overview • Phenomena • The e-learning lifecycle • e-learning evaluation research • Cyclical evaluation research approaches • Divide and Conquer techniques • Work on own project

50. Breaking Down the Complexity • How to use all this information: • It’s a complex situation, with • many elements, which • evolves over time • It requires multiple research methods • Use divide and conquer techniques and a template to simply the issue