Assessment and Evaluation, Part III: Evaluation Processes

1. Assessment and Evaluation, Part III:Evaluation Processes Week 12

2. SMART Notebook Quizzes

3. SMART quizzes • Bryan will pass out the SMART response clickers • Each of you will be assigned the following number: • Lugassy, Stefanie • Scheckelhoff, Renae • Marr, Clifford • Hoey, Bryan Bryan will have a laptop with the SMART notebook software loaded on it • In the order listed above, you will open your quiz, and have the others take your quiz • After all three quizzes are completed, Bryan will take you on a tour of the data analysis tools in SMART notebook • Discussion: what can the tables, charts, and tools tell you about your students’ performance on your quiz?

4. Today’s agenda • Evaluation and teaching • Task: Create your evaluation process for units • Task: Work on your units

5. What’s due, when, and where? • All “unit plan peer reviews” should have been finished by class today • Each of you should have turned in one lesson plan as a Word doc to Oncourse and to our turnitin website • By Friday at 11:59pm, complete peer reviews of our other two classmates’ LESSON plans in the turnitin website • Remember, all units (plans and materials) are due as part of a Google Site that includes everyone’s units by Thursday 4/12 at 5pm • Sample website from last spring: https://sites.google.com/site/w310webdesignteam/

6. Evaluation: What is it?

7. Evaluation: What is it? • First, let’s clear the air – what evaluation “is” depends on what you are evaluating • Teacher Evaluation? • Course/Curriculum Evaluation? • Technology/Materials Evaluation? • We are going to focus mostly on the second one, with some of the third today

8. What I am doing today • Today is a “fire-hose” approach to talking about evaluation • In the end, you are going to be asked to create your own approach to evaluating any of the units you and your peers will be creating. • I am going to mention some different approaches, and just try to overview a few key points • Overall, I want you to learn by doing – by trying to create a means by which you can determine if a unit/curricular materials are efficient, effective, and quality

9. Remember these questions? How do we know they have learned anything? How do we know the lesson/unit was the way it should have been (to be effective)?

10. This was the second question that we covered in Week 10 – remember that? How do we know the lesson/unit was the way it should have been (to be effective)? • This question is a matter of evaluation • You will self-evaluate as a teacher, and administrators/others will evaluate you? • As a tech coordinator, part of your job will include evaluation, to see how you can best help teachers integrate and use technology in their classrooms – a good evaluation of the status quo is necessary to make recommendations

11. Review: Is there a difference? • Is there a difference between assessment and evaluation? • It depends on who is defining the terms – some make a distinction, some do not. • To some, this is a silly distinction. Regardless, it is a matter of the object (end-goal) of your activity • Much K-12 literature conflates the two, but I am drawing a distinction, because I want you to focus on the object (what is being examined) and its nature • Assessment tends to look at whether or not learning has taken place and to what degree/under what conditions • Evaluation tends to look at the teaching process, how effectively, efficiently, and appropriately teaching and structured learning activities were executed

12. Today’s approach • I want you – after a brief talk – to wrestle with creating an “evaluation plan/instrument” • This can be based on your intuition, our Reading 4, or past experiences • You will create a Google Doc that will detail – in paragraph form – what this plan will be and how you will use it • Chip will present a few different approaches • You can use, combine, modify or replace any of these approaches • These approaches are food for thought – what is the best way to create an evaluation plan/instrument?

13. Perspective 1: Goal-Driven evaluation Models

14. Perspective 1 • In this model, the one creating the curriculum/instructional materials sets the goals, objectives, etc. The evaluation is goal-driven – did the results match the goals? • Very top-down, outcomes-driven. Also well-suited to a test-driven approach • There are many different approaches, and this one is not “right.” It should get you thinking, though, about the different ways and levels at which we can evaluate the effectiveness and quality of a unit or other curricular/instructional materials • Evaluation is seen as a structured means of checking to see if goals and objectives are met in an activity, curriculum, etc

15. Overall… Perspective 1? • What are some examples of each in terms of course/curriculum evaluation? • What does this miss, do you think? • What other dimensions (if any) could we evaluate?

16. Perspective 2: Appreciative Inquiry and evaluation

17. What is AI (appreciative inquiry)? • Traditional, goal oriented evaluation and learning is too negative – focused on what went wrong, what is not “good enough,” and what is not “right.” • AI was developed by a scholar named Cooperrider, who sought a more positive, group-oriented, democratic and “appreciative” approach to research and evaluation

18. Eight assumptions • It has eight basic assumptions (Coughlin, Preskill, and Tzavaras, 2003: • In every society, organization, or group, something works. • What we focus on becomes our reality. • Reality is created in the moment, and there are multiple realities. • The act of asking questions of an organization or group influences the group in some way. • People have more confidence and comfort to journey to the future (the unknown) when they carry forward parts of the past (the known). • If we carry parts of the past forward, they should be what is best about the past. • It is important to value differences. • The language we use creates our reality.

19. The four phases of the AI cycle • AI is a cyclical process, where there is continual growth (much like the PIE approach from the Newby book chapter we have read (“Reading 4”) • There are four phases to the AI cycle/model • Discovery: what works? • Dream: what could be? In this step, the evaluators (everyone involved - students, teachers, etc) think about what possibilities they could achieve • Design: what will it look like? What will the process be, and how - as a group (!) will we get there together? • Destiny: how do we make this great thing endure? (keyword: sustain)

20. Perspective 3:Responsive evaluation

21. Who sets the agenda? • Whereas in #1, the evaluator/instructor/etc sets the goals that they choose/value/must. In #2, the group uses past success to determine what how to best proceed (not improve) • In #3, “Responsive Evaluation” eliminates the teacher’s/instructor’s goals, and has the learners express their rich perspectives on the curriculum/materials/etc • A case study is conducted – but there are two versions: • The strict version: only the learners are considered – anything else would be authority-figures imposing their perception of what the learners should have learned onto the situation • The more used version: learners have priority, but teachers, administrators, etc are used in a case study

22. Who sets the agenda? • The key is – goals do not have a place here in the evaluation. The teacher or maker of the unit can have goals, but the purpose of a responsive evaluation is to gather the interpretations and experiences of others, not to seek evidence that your goals have been met. • The value of the unit is in the perceptions and feedback from others, not in a match-the-results-to-the-goals exercise

23. Responsive evaluation • The key is – goals do not have a place here in the evaluation. The teacher or maker of the unit can have goals, but the purpose of a responsive evaluation is to gather the interpretations and experiences of others, not to seek evidence that your goals have been met. • The value of the unit is in the perceptions and feedback from others, not in a match-the-results-to-the-goals exercise

24. Overall… Perspective 3? • What do you think are the strengths of this approach? • How do you think this "goes over" in a standards-driven, test-driven age? • What are the other difficulties with this approach? • What lessons do you think we can take from a case-study driven, responsive evaluation?

25. Summary of evaluation perspectives

26. Overall view • There are many, many, many more perspectives. These are just three very different ones that we can compare and contrast • Questions • What do these approaches have in common? • What parts are mutually exclusive? • What can we learn from this?

27. Why you should care • As part of your course project, you will be creating a document that outlines your evaluation process for a unit in a CompEd course. • Then, you will use it to evaluate different units that your classmates have made • This will be part of your course project grade • You will be creating this document in the following task

28. Task: Creating an evaluation instrument

29. Task: creating an evaluation plan/instrument • Your units are due in Week 13, but that’s not the end • On Monday of Week 14, I will assign to you one of your classmate’s units • You will apply your “evaluation plan/instrument” and try to evaluate the efficiency, effectiveness, and quality of their unit • At the end of week 14, you will draft a 1-2 page report to the unit’s creator, where you report on what you found in your evaluation (and provide a copy to Chip); this will be a Google Doc, also • You will also provide the person whose unit you are reviewing with your plan/instrument • You will be graded (see syllabus) based on the quality of your plan/instrument and how well you report your findings (which are based on that instrument

30. Task: creating an evaluation plan/instrument • You will be graded (see syllabus) based on the quality of your plan/instrument and how well you report your findings (which are based on that instrument • Once you receive feedback on your own unit (and from Chip) by Monday of Week 15, you will revise your unit, and how much you improve will affect your grade on your unit when you turn it in for the second time on Friday of Week 15 (by 5pm)

31. Task: creating an evaluation plan/instrument • Your evaluation plan/instrument will not be perfect, but this is an opportunity for you to begin to consider how you would judge if something is “good” • You will upload any Google Docs to Oncourse Assignments (“evaluation plan/instrument” assignment) • If you are a tech coach/facilitator, you may have to evaluate the teaching, materials, and work of other teachers – this is an opportunity to begin to formulate a mindset now • I am more than happy to help you shape this approach, but you must ask for my advice

32. Your evaluation Plan/instrument • It is a document that you can format however you want, but you will use it to examine another’s unit and generate a report for that person • It should have two parts (one or two docs – it’s your choice): • Part I: Your evaluation plan • 1-2 paragraphs (as many as necessary) to describe your approach to evaluation. In other words, how do you know that a unit is "good" - that it is quality, and effective. • List out the specific, step-by-step procedures you would use to evaluate one of our units. Someone else is going to have to use this, so do not just write for you or for me. Give instructions as if the person knows nothing about your unit. • Part II: Your evaluation instrument– Include the following in your blank evaluation form: • A space for the evaluator's and evaluatee'snames • A space for a brief description of the unit by the evaluator • A space to describe what all of the pieces of the unit you have (if you are missing something, it is best to tell what you are evaluating, and not assume that you have a full inventory of pieces) • A space for a brief description of the unit's context (target learners, prerequisite knowledge, etc) • Leave a space for a paragraph on the strength of the materials (overall) • Leave a space for a paragraph on the weaknesses of the materials (overall) • Leave a space for VERY specific and general recommendations for how the unit creator should re-work their unit to improve it. That person will need to make those modifications during weeks 14-15, and I will be looking that they have made those modifications (unless the creator makes a good argument to Chip and the evaluator)

33. Task 2: Unit Work timeUse any remaining time to work on your units and or evaluation plans