Approaches, Reproaches: The Joy of Methods

1. May, 2009 AAC&U General Education Institute, Minneapolis, MN 1 Approaches, Reproaches: The Joy of Methods Barbara D. Wright Associate Director, ACSCU/WASC bwright@wascsenior.org

2. May, 2009 AAC&U General Education Institute, Minneapolis, MN 2

3. May, 2009 AAC&U General Education Institute, Minneapolis, MN 3 What is an “assessment method”? It’s how you collect the evidence, direct or indirect, that will tell you about the quality of your students’ learning (step 2 on the loop) and how to improve it. What are my options? Direct Indirect Descriptive data

4. May, 2009 AAC&U General Education Institute, Minneapolis, MN 4 Direct? Indirect? Direct evidence demonstrates your students’ learning directly, in an unfiltered way. Indirect evidence is mediated by the person responding to a questionnaire, interviewer, etc. It is influenced by perceptions, experiences, etc.

5. May, 2009 AAC&U General Education Institute, Minneapolis, MN 5 What do you use when? Direct evidence tells you what your students know and can do, and how well, in relation to your learning outcomes Indirect evidence can reveal why and how students learned what they learned – or didn’t.

6. May, 2009 AAC&U General Education Institute, Minneapolis, MN 6 Quantitative or qualitative? “Not everything that can be counted counts, and not everything that counts can be counted.” – Albert Einstein

7. May, 2009 AAC&U General Education Institute, Minneapolis, MN 7 “Qualitative” versus “quantitative”? Quantitative research focuses on numeric and graphic representations of phenomena. Qualitative approaches focus on language, signs and symbols, observations, and documents. Both are useful to identify patterns. Good assessment needs both.

8. May, 2009 AAC&U General Education Institute, Minneapolis, MN 8

9. May, 2009 AAC&U General Education Institute, Minneapolis, MN 9 Which is which? Direct methods can generate quantitative AND qualitative evidence Quantitative: standardized M/C test Qualitative: essay, project Indirect methods can generate quantitative AND qualitative evidence Quantitative: survey Qualitative: focus groups, ethnographic research

10. May, 2009 AAC&U General Education Institute, Minneapolis, MN 10 Using both . . .

11. May, 2009 AAC&U General Education Institute, Minneapolis, MN 11

12. May, 2009 AAC&U General Education Institute, Minneapolis, MN 12 Shifts in our understanding of assessment Isolated facts, skills Memorization, reproduction Comparing performance against other students A full range of knowledge, skills, dispositions Problem solving, investigating, reasoning, applying, communicating Comparing performance to established criteria

13. May, 2009 AAC&U General Education Institute, Minneapolis, MN 13 Shifts in assessment, cont. Scoring right, wrong answers a single way to demonstrate knowledge, e.g. m/c or short-answer test Simplified evidence Looking at the whole reasoning process Multiple methods & opportunities, e.g., open-ended tasks, projects, observations Complex evidence

14. May, 2009 AAC&U General Education Institute, Minneapolis, MN 14 Shifts in assessment, cont. A secret, exclusive & fixed process Reporting only group means, normed scores Psychometric A filter An add-on open, public & participatory Disaggregation, analysis, feedback Educative A pump Embedded

15. May, 2009 AAC&U General Education Institute, Minneapolis, MN 15 Shifts in assessment, cont. “Teacher-proof” assessment Students as objects of measurement Episodic, conclusive Reliability Highly quantitative Respect, support for faculty & their judgments Students as participants, beneficiaries of feedback Ongoing, integrative, developmental Validity More qualitative

16. May, 2009 AAC&U General Education Institute, Minneapolis, MN 16 Choice of assessment method matters. Students value and learn what we teach and test. How we teach and test matters as much as what What and how we assess also matters. We get more of what we test or assess, less of what we don’t. At a comprehensive, teaching institution like WKU, what students learn is of primary importance. There’s official curriculum and pedagogy, and there’s what students understand as implicit in curriculum and pedagogy. Don’t underestimate the implicit, or the importance of pulling implicit and explicit into alignment for greatest effectiveness.At a comprehensive, teaching institution like WKU, what students learn is of primary importance. There’s official curriculum and pedagogy, and there’s what students understand as implicit in curriculum and pedagogy. Don’t underestimate the implicit, or the importance of pulling implicit and explicit into alignment for greatest effectiveness.

17. May, 2009 AAC&U General Education Institute, Minneapolis, MN 17 Higher-order thinking …( adapted from L. Resnick, 1987) It’s nonalgorithmic, i.e., the path of action is not fully specified in advance. It’s complex, i.e., the total path is not “visible” from any single vantage point. It often yields multiple solutions, each with costs and benefits. It requires nuanced judgment and interpretation It involves application of multiple criteria, which may conflict with one another

18. May, 2009 AAC&U General Education Institute, Minneapolis, MN 18 Higher order thinking, cont … It often involves uncertainty; not everything about the task is known or can be. It requires self-regulation; someone else is not giving directions. It involves making meaning, discerning patterns in apparent disorder. It is effortful: the elaborations and judgments required entail considerable mental work and are likely to take time.

19. May, 2009 AAC&U General Education Institute, Minneapolis, MN 19 Other approaches to higher-order learning . . . Bloom’s taxonomy Perry Scheme of Intellectual Development Biggs’ and Entwistle’s deep versus surface learning

20. May, 2009 AAC&U General Education Institute, Minneapolis, MN 20 So what should we choose? It depends on your question. Best practice: multiple methods Direct evidence is the gold standard Indirect evidence fills out the picture Both are useful at step 3: interpretation Descriptive data are the third major source of evidence and most useful when combined w/ other methods

21. May, 2009 AAC&U General Education Institute, Minneapolis, MN 21 The methods we choose should reflect paradigm shifts in assessment; be educative, engaging for students and educators; promote higher-order learning by asking students to demonstrate higher-order learning; have content, construct, and consequential validity; and be sustainable over the long term.

22. May, 2009 AAC&U General Education Institute, Minneapolis, MN 22