Download
1 / 13
200 likes | 573 Vues
Principles of Effective Evaluation. Prof Keithia Wilson Academic Leader Student Success & Retention. Importance of Evaluation. Results in evidence-based practice Helps us to identify the most effective & strategic interventions (make a difference or add value)
E N D
Principles of Effective Evaluation Prof Keithia Wilson Academic Leader Student Success & Retention
Importance of Evaluation • Results in evidence-based practice • Helps us to identify the most effective & strategic interventions (make a difference or add value) • Leads to sustainable & enjoyable practice • Provides accountability to School & Group leadership • Contributes to scholarship of the FYE
Kirkpatrick (1994) Model of Evaluation - Overview 4 levels of outcome evaluation – • Level 1 – Reaction • Level 2 – Learning • Level 3 – Behaviour • Level 4 – Results
Level 1 - Reaction Measures participant’s reactions to the training program, including: • reactions to the overall program (outcomes) e.g., * “To what extent did you find the Orientation program useful?” • reactions to specific components of the program (processes) e.g., * What aspects of Orientation did you MOST APPRECIATE and found USEFUL? * What aspects of Orientation did you LEAST APPRECIATE and feel is MOST IN NEED OF IMPROVEMENT?
Level 1 - Reaction • Important to include closed-ended items including rating scales e.g., How effective was the Orientation program overall in helping you to make a transition to university? not at all effective very effective 1 2 3 4 5 6 7 How useful was the initial information session on the predictors of success in first year? not at all useful very useful 1 2 3 4 5 6 7 • As well as open-ended items What else would you have liked included in the Orientation program today?
Level 1 - Reaction Some FYE examples – • Starting@Griffith Survey questions on Orientation (retrospective student survey) • School specific Orientation Program evaluation at the end of O-Day & the Common Time (ongoing Orientation Program) – student surveys, focus groups • Peer Mentoring Program final evaluation by mentors & mentees
Level 2 - Learning Measures what participants have learned from involvement in the program • What is measured needs to relate to what was covered in the program viz. learning objectives • Typically covers knowledge, skills, or attitudes • Needs to include both rating scales & open-ended questions • Can include self-report & tests of actual knowledge
Level 2 - Learning Some examples – • Student self-report * I feel confident that I know what to do to be successful academically this semester not at all confident very confident 1 2 3 4 5 6 7 * What have you learned about the transitions you will need to make as a commencing student in order to be successful? 2. Student knowledge * I know the name of my First Year Advisor yes no (circle one) * I know the difference between a lecture and a tutorial (circle one) Strongly Disagree | Disagree | Neutral | Agree | Strongly Agree
Level 2 - Learning Some FYE examples – • Starting@Griffith Survey data evaluates learning outcomes in terms of knowledge & attitudes (e.g., knowing where to go to get help; perceptions of good teaching) • Outcome evaluation for Peer Mentees (What are your 3 most important learnings from your participation in the Peer Mentoring Program?) • First-Assessment-First-Feedback amber alert intervention (What have you learned as a result of your participation?)
Level 3 - Behaviour Measures the transfer of knowledge, skills & attitudes from the training context to in vivo or real-life contexts (the classroom & study). • Evaluate both before & after the program if practical • Use survey, focus groups, interviews with students, mentors, staff (convenors, sessionals) • Starting@Griffith Survey (Did you attend Orientation? How much time are you spending on study?)
Level 3 - Behaviour Some FYE examples – • Starting@Griffith survey data • Attendance at E-Day, O-Day & Common Time/ASP Program • Attendance at lectures & tutes • Assessment submission rates in courses
Level 4 - Results Measures “return-on-investment”, or the extent to which the program has produced results. Some FYE examples include – • Hard student outcomes such as - * pass rates in courses for assessment items * GPA in semester 1, 2 & FY overall * student retention into year 2 • Soft student outcomes such as – * student satisfaction
Evaluation Key question – What difference do our FYO&E interventions (e.g., Orientation, Peer Mentoring, Attendance, First Assessment Intervention etc.) make overall? Program/Intervention1 Reactions2 Learning 3 Behaviour 4 Overall Productivity/Result Program/InterventionFeelKnowDowhat is the result or difference overall?
