Black Holes & Gaseous Processes: Really Big Mistakes in Assessment :

1. Black Holes & Gaseous Processes: Really Big Mistakes in Assessment: Lawrence University November 2008 Susan Hatfield Assessment Director Winona State University SHatfield@winona.edu

2. Really Big AssessmentMistakes

3. Assuming it Will Go Away

4. Creating a Culture of Assessment instead of a Culture of Learning

5. Assessment Learning Instrument Driven Outcome Driven National Norms Targets & Goals Trend Lines Relational Data Collection Analysis How do we compare? What does it mean?

6. Still doing assessment like it is 1990

7. 1990 2000 2010 INDIRECT MEASURES DIRECT MEASURES

8. 1990 2000 2010 PROCESS MEASURES OUTCOME MEASURES

9. 1990 2000 2010 CLASSROOM ASSESSMENT PROGRAM ASSESSMENT

10. 1990 2000 2010 INSTITUTIONAL RESPONSIBILITY PROGRAM RESPONSIBILITY

11. 1990 2000 2010 INSTITUTIONAL EFFECTIVENESS STUDENT LEARNING

12. Waiting for everyoneto get on board

13. Attitudes toward Assessment Level of Commitment

14. Attitudes toward Assessment Level of Commitment 70% 15% 15% Hostile Accepting Enthusiastic

15. Forgetting to Make aCurriculum Map

16. Program Level Student Learning Outcomes Course 1 Course 2 Course 3 Course 4 Course 5 K K K K A S A A S S K A K A S A S K A S K K= Knowledge/Comprehension; A= Application / Analysis; S= Synthesis /Evaluation

17. Program Level Student Learning Outcomes Course 1 Course 2 Course 3 Course 4 Course 5 K A S K A S A A S S K A K A S A S K K= Knowledge/Comprehension; A= Application / Analysis; S= Synthesis /Evaluation

18. Program Level Student Learning Outcomes Course 1 Course 2 Course 3 Course 4 Course 5 K S K A S A S S K A K A S S K S K K= Knowledge/Comprehension; A= Application / Analysis; S= Synthesis /Evaluation

19. Program Level Student Learning Outcomes Course 1 Course 2 Course 3 Course 4 Course 5 K A S K A S K S K A K A S A S K K= Knowledge/Comprehension; A= Application / Analysis; S= Synthesis /Evaluation

20. Trying to do too much

21. Program Level Student Learning Outcomes Course 1 Course 2 Course 3 Course 4 Course 5 K A S K A S A A S S K A K A S A S K A S K K= Knowledge/Comprehension; A= Application / Analysis; S= Synthesis /Evaluation

22. Program Level Student Learning Outcomes Course 1 Course 2 Course 3 Course 4 Course 5 K A S K A S A A S S K A K A S A S K A S K K= Knowledge/Comprehension; A= Application / Analysis; S= Synthesis /Evaluation

23. Program Level Student Learning Outcomes Course 1 Course 2 Course 3 Course 4 Course 5 K A S K A S A A S S K A K A S A S K A S K K= Knowledge/Comprehension; A= Application / Analysis; S= Synthesis /Evaluation

24. Program Level Student Learning Outcomes Course 1 Course 2 Course 3 Course 4 Course 5 K A S K A S A A S S K A K A S A S K A S K K= Knowledge/Comprehension; A= Application / Analysis; S= Synthesis /Evaluation

25. Not Aligning Campus Processes withAssessment

26. Process Alignment Course Proposal - learning outcomes? Program Proposal - curriculum map? Program Review - assessment data? Master Syllabi - program outcomes? Reporting Structure - assessment office? Promotion, Renewal, Tenure - rewarded?

27. Choosing Assessment Methods before Defining Outcomes

28. organization P O R T F O L I O √ presentation √ # of submissions √ multi media √ navigation √ captions √

29. Outcomes: Theories & Theorists Assignment Research Methods Research Writing T E S T Data Analysis

30. Outcomes: appropriateness Theories & Theorists P O R T F O L I O √ understanding √ Research Methods application √ accuracy Research Writing √ detail √ Data Analysis language √

31. Too Many Program Level Learning Outcomes

32. Learning Outcomes • NOT a compilation of your course level student learning outcomes • NOT intended to represent everything that your students learn in the program

33. Exertion without Intention

34. Intention without Exertion

35. Intention and Exertion

36. Poorly Written Student Learning Outcomes

37. Student Learning Outcomes • Students should be able to comprehend, interpret, analyze and critically evaluate material in a variety of written and visual formats.

38. Student Learning Outcomes • Students will demonstrate creative and evaluative thinking in the analysis of theoretical and practical issues in the areas of politics, the economy, and the environment.

39. Student Learning Outcomes • FORMAT: Students should be able to <<action verb>> <<something>>

40. Inappropriate Program Level Learning Outcomes

41. COMPREHENSION EVALUATION APPLICATION ANALYSIS SYNTHESIS KNOWLEDGE Associate Classify Compare Compute Contrast Differentiate Discuss Distinguish Estimate Explain Express Extrapolate Interpolate Locate Predict Report Restate Review Tell Translate Analyze Appraise Calculate Categorize Classify Compare Debate Diagram Differentiate Distinguish Examine Experiment Inspect Inventory Question Separate Su rize Test Arrange Assemble Collect Compose Construct Create Design Formulate Integrate Manage Organize Plan Prepare Prescribe ProducePropose Specify Synthesize Write Appraise Assess Choose Compare Criticize Determine Estimate Evaluate Grade Judge Measure Rank Rate Recommend Revise Score Select Standardize Test Validate Cite Count Define Draw Identify List Name Point Quote Read Recite Record Repeat Select State Tabulate Tell Trace Underline Apply Calculate Classify Demonstrate Determine Dramatize Employ Examine Illustrate Interpret Locate Operate Order Practice Report Restructure Schedule Sketch Solve Translate Use Write Lower division course outcomes

42. COMPREHENSION EVALUATION APPLICATION ANALYSIS SYNTHESIS KNOWLEDGE Associate Classify Compare Compute Contrast Differentiate Discuss Distinguish Estimate Explain Express Extrapolate Interpolate Locate Predict Report Restate Review Tell Translate Analyze Appraise Calculate Categorize Classify Compare Debate Diagram Differentiate Distinguish Examine Experiment Inspect Inventory Question Separate Summarize Test Arrange Assemble Collect Compose Construct Create Design Formulate Integrate Manage Organize Plan Prepare Prescribe ProducePropose Specify Synthesize Write Appraise Assess Choose Compare Criticize Determine Estimate Evaluate Grade Judge Measure Rank Rate Recommend Revise Score Select Standardize Test Validate Cite Count Define Draw Identify List Name Point Quote Read Recite Record Repeat Select State Tabulate Tell Trace Underline Apply Calculate Classify Demonstrate Determine Dramatize Employ Examine Illustrate Interpret Locate Operate Order Practice Report Restructure Schedule Sketch Solve Translate Use Write Upper division Course / Program outcomes

43. Student Learning Outcomes • RULE OF THUMB: If you have more than one action verb, keep the one that represents the highest order of thinking.

44. Relying on Indirect Assessment Measures of Learning

45. Direct Measures of Learning • Capstone experience • Standardized tests • Performance on national licensure certification or professional exams • Locally developed tests • Essay questions blind scored by faculty • Juried review of senior projects • Externally reviewed exhibitions performances • Evaluation of internships based upon program learning outcomes

46. Indirect Measures of Learning • Alumni, employer, and student surveys (including satisfaction surveys) • Exit interviews of graduates and focus groups graduate follow up studies • Retention and transfer studies • Length of time to degree • ACT scores • Graduation and transfer rates • Job placement rates

47. 1990 2000 2010 INDIRECT MEASURES DIRECT MEASURES

48. Making Assessment the Responsibility of One Person

49. Allowing Each Faculty Member to “Own” the Program Level Outcomes

50. teacher4 teacher2 teacher1 teacher3 teacher5 Speaking eye contact gestures volume sources transitions style rate poise examples verbal variety appearance evidence conclusion organization attention getter