TAPS Forum 12 th November 2013

1. TAPS Forum12th November 2013 Evaluating your work

2. Evaluation Evaluation is the assessment of worth or value Evaluation may be expert-based or evidence-based

3. Evaluation for Improvement Evaluation seeks to assist in the improvement of whatever is being evaluated “The purpose of an evaluation is to assess the effects and effectiveness of something, typically some innovation or intervention, policy, practice or service”

4. Evidence in Evaluation Participation, activity, take-up – how many people turned up? Effectiveness/quality – did it ‘go well’ ? Was it successful? Impact – what happened as a result? What were the outcomes.

5. Evaluation? Research? “ An evaluation is a study which has a distinctive purpose ; it is not a new or different research strategy” “... Evaluations are essentially indistinguishable from other research in terms of design, data collection techniques and methods of analysis” “Education evaluation is typically evidence-based “

6. Evaluation? Danger? “ Evaluation is intrinsically a very sensitive activity where there may be a risk (duty?) of revealing inadequacy or worse.” “ ... Your intentions may be misconstrued and your findings misused or ignored”

7. Your Evaluation • What do you want to know? • What kinds of evidence will you draw on? • Who will be involved and in what ways ? • Who needs to be informed by your evaluation? • What do you hope will happen as a result of your evaluation?

8. Collecting data or evidence • Documents and recorded data • Interviews • Questionnaires and other forms of prompts for responses • Group e.g. group interviews; online approaches • Observation • Using the assessment review tool

9. Q - Sort

10. Approaches to Learning and course experience inventories http://www.etl.tla.ed.ac.uk/publications.html#measurement

11. Research ethics Institutional procedures and approval may include: • How ‘subjects’ will be recruited. Who are they? Informed consent? • Storage of and access to data • Right to withdraw • Research methods that are fit for purpose

12. Real world research Colin Robson London, Wiley Any edition, latest is 3rd [Source of quotations]

13. Slides on Q sorts for HEA core forumNovember 2013 Sally Brown

14. Characteristics of excellent teachers: diamond-9 1 2 3 4 5 6 7 8 9

15. Q Methodology Q is used for establishing patterns within and across individuals, rather than across individual traits. (Barry & Proops, 1999) William Stephenson (1902-1989) Psychologist and Physicist

16. User-friendly: a Q-sort underway Acknowledgement: Thanks to Wendy Stainton Rogers for sharing this graphic, which was adapted from an original paper by StaintonRogers, W. (2011) Social Psychology. OUP.

17. Stages in a Q-study • Identifying and sampling the concourse • Developing a set of statements that is representative of the concourse • Selecting participants for a diversity of views on the issues • Q-sorting and post-sort interviews • Pattern analysis – data reduction and interpretation

18. Example statements -3 -2 -1 0 +1 +2 +3 DisagreeAgree Strongly Strongly Essays and exams should be ‘the gold standard’ in terms of Masters assessment methods. (3) Improving assessment methods requires a shift in how learning is viewed. (41) Writing assessment criteria is an easy job for academics. (26)

19. Findings – system voices • In my Assimilate project, five different viewpoints on the issues were interpreted, relating to different aspects of M-level assessment activity • Areas of consensus or near-consensus among the viewpoints were also interpreted