Evaluating your teaching and learning project

1. Evaluating your teaching and learning project UQ Teaching and Learning Week, 2007 Clair Hughes clair.hughes@uq.edu.au

2. Evaluation • The process of determining the merit and/or worth of things for a range of purposes. • Evaluation is generally concerned with: • Appropriateness - the extent to which stated or implied project goals (or expectations) and implementation processes match the stated or implied needs of nominated target group/s (e.g. students; teachers; coordinators; tutors; program, school, faculty or institutional leaders and administrators; external agencies etc) and the context in which implementation is to be undertaken. • Effectiveness - the extent to which the project or innovation has had an impact on the knowledge, skills, attitudes and/or behaviours of nominated target group/s. • Efficiency - the amount of resources or effort needed to implement the project or innovation. Adapted from Queensland School Curriculum Council, 2000, p34

3. Session overview • Summary of key purposes of evaluation • Consideration of how purpose influences: • identification of stakeholders • framing of evaluation questions • selection of data collection methods.

4. Funding agency requirements Ensure that evaluation is undertaken at multiple points throughout the project and is understood and reported within an evaluation framework. Carrick Competitive Grants program: Guidelines and Supporting Information – 2008 (p.23) …………………………………………………………………………………………………… Must provide a strategy for the evaluation of the project with information on outcomes for students, whether there has been a noticeable improvement or not in the expected outcomes for students and how the project outcomes will be disseminated to others within the discipline. UQ Teaching and Learning Small and Large Grants Scheme Guidelines

5. The main purposes of project evaluation Adapted from “The Learning Partnership” 2007 and W. K. Kellogg Foundation

6. Models of planning for evaluation Implementation Evaluation Implementation Evaluation Dissemination Dissemination Evaluation Implementation Implementation Evaluation Dissemination Dissemination

7. Identifying key stakeholders • Who is your project audience? Who will be interested in the outcomes now? – in the future? • project funding agency? • future funding agency? • students? • academic supervisor/s? • senior management? • teaching colleagues? (supportive and antagonistic) • teachers in related disciplines? • professional association members? • industry or employers? • government organisations? • committees (appointment, promotion, T&L awards? • reviewers (conferences, publications)? • lobby groups? • the media? • other?

8. Frame stakeholder questions • Identify two stakeholders in a project with which you are familiar or which you have in development - in your selection try to represent diverse interests. • List one or more questions each stake-holder may have in relation to the project? • Examples: • What is the impact of the project on student outcomes? success? satisfaction? (Funding agency) • What resources are needed to implement the project innovation more widely? Is it feasible? (Senior manager) • How do students experience the project? Are there any unintended/undesirable effects? (Students) • What contextual factors facilitated or hindered implementation? (Teaching peers)

9. Collecting evidence • How will data be collected? (Methods) • From whom or where (stakeholders and others) will data be collected? (Sources) • What will constitute evidence? (Consider both qualitative and quantitative)

10. The evaluation framework EXAMPLE

11. References • Carrick Institute for Learning and Teaching in Higher Education. (2006). Carrick Institute Dissemination Framework Retrieved from http://www.carrickinstitute.edu.au/carrick/webdav/site/carricksite/users/siteadmin/public/dissemination_carrickframework_2006.pdf • Carrick Institute for Learning and Teaching in Higher Education. (2008). Competitive Grants program: Guidelines and Supporting Information – 2008. Retrieved from http://www.carrickinstitute.edu.au/carrick/webdav/site/carricksite/users/siteadmin/public/grants_competitive_guidelines2008_july07.pdf • Felton, P., Kalish, A., Pingree, A., & Plank, K. (2007). Towards a scholarship of teaching and learning in educational development. In D. Robertson & L. Nilson (Eds.), To Improve the Academy: Resources for Faculty, Instructional, and Organisational Development (pp. 93-108). Boston: Anker. • The Learning Partnership. (2007). Material distributed during Leading and Managing Projects, a workshop sponsored by the Carrick Institute for Learning and Teaching in Higher Education. Melbourne, 17 – 18 September. • Queensland School Curriculum Council. (2000). Procedures for the Evaluation of Approved Council Curriculum Materials. Retrieved 29 October, 2007, from http://www.qsa.qld.edu.au/publications/yrs1_10/ev_app.pdf • The University of Queensland. (2007). UQ Teaching and Learning Small and Large Grants Scheme Guidelines. Retrieved 29 October, 2007, from http://www.uq.edu.au/teaching-learning/download/TLStrategicGrantsGuidelines07.pdf • W. K. Kellogg Foundation. Executive summary: Introducing Program Logic Models. Material distributed during The Professional Development Program for Medical and Healthcare Educators offered jointly by the University of Queensland and Harvard Medical International. 21-25 August, 27-29 November, 2006.