Beyond Student Assessment

1. Beyond Student Assessment Issues to consider in the assessment of Information Literacy

2. Objectives for this Workshop • By the end of this workshop, participants should be able to • Identify possible assessments strategies that align with the Information Literacy objectives • Identify instructional strategies and activities that align with Information Literacy objectives and support performance on Information Literacy assessments

3. Agenda • Discuss course design triangle • Review Information Literacy objectives from ACRL • Describe issues to consider when planning assessment • Identify potential assessment options • Identify potential learning activities • Practice alignments

4. Assessment • Where to begin? Objectives, objectives, objectives, objectives, objectives, objectives, objectives, objectives, objectives, objectives, objectives, objectives, objectives, objectives, objectives, objectives, objectives, objectives, objectives, objectives, objectives, objectives, objectives, objectives, objectives, objectives, objectives, objectives!

5. The Education Design Triangle Objectives Descriptions of what students should be able to do Assessment Tasks that provide feedback on students’ knowledge and skills Instructional Activities Contexts and activities that foster students’ active engagement in learning

6. When a Link is Broken… Objectives X X Assessment X Instructional Activities

7. The Triangle Exemplified

8. Why Focus on Objectives? • They communicate your intentions clearly to students and to colleagues. • They provide a framework for selecting and organizing content. • They guide you in decisions about assessment and evaluation methods. • They provide a framework for selecting appropriate teaching and learning activities. • They give students information for directing their learning efforts and monitoring their own progress. Based on A.H. Miller (1987), Course Design for University Lecturers. New York: Nichols Publishing.

9. Developing Clear Objectives • Focus on students, e.g., “Students should be able to…” • Break down the task, focus on specific cognitive processes • Use action verbs, e.g., • State theorems • Identify and describe trends… • Evaluate the arguments proposed… • Describe conditions under which techniques apply • Use measurable verbs Hard to measureEasier to measure Obtain a working knowledge Demonstrate Understand Describe, interpret Be aware List, describe Appreciate Identify value Begin to develop competency Perform

10. Cognitive and Behavioral Dimensions of Objectives • Bloom’s taxonomy • Remember • Comprehend • Apply • Analyze • Synthesize • Evaluate • Create

11. ACRL Objectives • Information literate student… • Defines and articulates the need for information

12. An Example to Refine/Revise • Choose a learning objective(s) from your course. Is the objective…? Student centered (i.e., defined in terms of what students do) Broken down into component skills (not too big, not too small) Phrased with an action verb Measurable How well does it… communicate expectations to the student and guide students’ efforts through the course? help you prepare assessments and learning activities?

13. Your Objectives? Global Education Diverse Cultural/Historical Literacy? International Awareness? Global Competency? CFA CIT Heinz HSS MCS SCS TSB Unique objectives Shared objectives

14. What are your Objectives?

15. Break

16. Now we can begin to think about Assessment

17. The Basics • Assessments must map onto objectives • Objectives need to be clearly articulated to students • What should they learn from the experience • What skills should they be developing • Students need to understand how they will be assessed and by who • Product and domain knowledge and skills • Process and group skills and strategies

18. Target Assessments to Objectives

19. Designing and conducting an assessment • Clarifying & operationalizing the objectives • Selecting a method • Selecting a measure • Interpreting the results • Reporting the results • Follow-up Actions

20. Question to ask before you begin • What change in students’ knowledge, skills, attitudes and/or behaviors are you interested in? • What kind(s) of assessment is necessary? • Knowledge-based assessment • Skill-based assessment • Product-based assessment • Group vs. individual • What is the timeframe of the assessment? • Short-term (within course life) • Longer-term (outside course life)? • Interest • Future course selection • Career path

21. How would you measure it? • Method? • Measure?

22. Methodological Issues • What type of information is most useful to YOU? • Observations, informal discussions, drafts are often used early in an assessment process • Surveys and tests, project-based deliverables can take considerable development to be informative, valid, and reliable • What level of Measurement is important to you? • Qualitative Quantitative • How defensible do you want to be? • Personally Institutionally Publicly Formally

23. Qualitative Student/Peer Comments Classroom Observations Logs and journals Focus groups Portfolios Quantitative Problem sets, quizzes, exams, exam questions Performance/product Qualitative & Quantitative Projects Surveys (self-reports) Concept maps Selecting Methods

24. What do you assess? • If application of skill and knowledge is the learning objective then assessment must include • Product • Quality (against some defined criteria) • Overall • By feature • Individual Knowledge (oral, written, visual, multi-modal) • Presentation • Exam Questions • Essay

25. What do you assess? • If application of skills and teamwork are both learning objectives then assessment would include • Process • Product

26. Potential Measures • Process measures • Product measures • Concept inventories • Concept maps • Content inventories

27. Tools for Assessing process • Team Charter Review • Meeting Minutes • Communication transcripts • Journals/logs • Early peer evaluation • Progress reports

28. Tools for Assessing product • Client • Acceptance of product • Recommendations • Strength and weakness analysis • Performance Evaluation: Does it work? • Audience reaction (critic reviews) • Program/product execution: qualities (speed, efficiency) • Rubrics • Dimensions and qualities of product/design/outcome being evaluated

29. Rubrics

30. Inventories • Concept Inventories • Goal: Identify how students reason about and apply critical concepts, principles, decision making strategies, etc. • Frequently used to uncover systematic but incorrect “theories” • Ideal Structure* • Multiple-choice format • Distractor items are common misconceptions or errors • Multiple questions address same concept in different contexts • Depth and breadth • Assessing Results • Pattern of responses across questions indicative of underlying “beliefs” or “theories” • Inconsistent responses across questions dealing with same concept reveal conceptual depth, contextualization, or fragile knowledge *Data driven from open-ended questions

31. (Hestenes & Wells, 1992) The diagram depicts a block sliding along a frictionless ramp. The eight numbered arrows in the diagram represent directions to be referred to when answering the questions. 4. The direction of the acceleration of the block, when in position I, is best represented by which of the arrows in the diagram? 1 B) 2 C) 4 D) 5 E) None of the arrows; the acceleration is zero. 5. The direction of the acceleration of the block, when in position II, is best represented by which of the arrows in the diagram? 1 B) 3 C) 5 D) 7 E) None of the arrows; the acceleration is zero. 6. The direction of the acceleration of the block, (after leaving the ramp) at position III, is best represented by which of the arrows in the diagram? A) 2 B) 3 C) 5 D) 6 E) None of the arrows; the acceleration is zero. 1 8 2 7 3 6 4 5 III I II

32. Concept map on Taste

33. Concept map on Taste

34. Pragmatics: What is practical? • Scope of Assessment • Class • Unit • Lecture • Time investment and resources • Timeframe for results

35. Recommendations • Start collecting and saving data now • Instruments & scores, student feedback, classroom videotaping, etc • Be as rigorous as is possible and practical • Use mixed methods & multiple measures • Defining clear operational goals and being pragmatic is key to good assessment and evaluation.