Show Me an Evidential Approach to Assessment Design

1. Show Me an Evidential Approach to Assessment Design Michael Rosenfeld F. Jay Breyer David M. Williamson Barbara Showers

2. A Brief Introduction to Evidence Centered Design (ECD) Michael Rosenfeld, Ph.D. Rosenfeld & Associates

3. What is Evidence Centered Design? • A formal framework for assessment design • Emphasizes assessment as an evidential argument • Encourages models and representations that strengthen validity arguments • Focuses the test development process to better target decisions of interest

4. The Basic Models of ECD • Proficiency Model – What you want to measure • Evidence Model – How to recognize & interpret observable evidence of unobservable proficiencies • Task Models – How to elicit valid and reliable evidence

5. ECD Validity Chain Tasks Task Model Evidence Tasks Task Model Proficiency Evidence Tasks Evidence Task Model Tasks Task Model Tasks

6. What is Gained from ECD? • Better understanding of the domain, for new constructs • Explicit chain of evidence from assessment goals, to models of evidence, to assessment tasks • Strong construct validity argument • Rapid and efficient item generation from models • automatic item generation • Opportunities for innovation, task types other than multiple choice • Philosophical shift from content-authoring to evidential argument

7. Typical Test Development The Job Content Specification Job Analysis Major Dimensions Knowledge & Tasks TestBlueprint Incumbent Survey ItemWriting Content Linking Analysis Statistical Modeling CutScores

8. Typical Test Development and Evidence Centered Design Typical Test Development Evidence Centered Design Content Specification Proficiency Model TestBlueprint Evidence Model ItemWriting Task Model Analysis Tasks CutScores

9. Evidence Centered Design Claims (Criterion) Job Analysis Proficiency Model Reporting Variables Cognitive Task Analysis Knowledge, Strategies & Skills Evidence Model Evidential Argument Statistical Modeling Task Model Task Design Content Linking Tasks

10. Still to Come • Example of the Proficiency Modeling process (Jay) • Illustration of Evidence Modeling and Task Modeling (David) • Commentary from an applied perspective (Barbara)

11. Proficiency Models in the ECD Framework F. Jay Breyer, Ph.D.

12. Proficiency Model • Combination of • Formalization of assessment goals, population and purpose • Formal claims to be made on the basis of the assessment • He/she is competent to operate an automobile • Proficiencies of interest • Knowledge of rules, procedures, and laws • Ability to implement driving regulations in practice • Skill in automobile operation

13. What is a Proficiency? • Proficiencies are knowledge, skills and abilities that provide the basis for making claims about people

14. Building a Proficiency Model: Assessment Goals • Assessment Goals • Protect the public health, safety & welfare • Examinee population • All eligible (age, prior training requirements, etc.) citizens self-selecting into the program • Test users/stakeholders • State regulatory bodies • State resident drivers • Other citizens exposed to driving • Result Usage/Decisions • State issuing the license (to issue or deny) • Means of official identification

15. Building a Proficiency Model: Assessment Claims • Formal Claims are developed for assessment goals • Claims can be course-grained and general • Can safely operate a motor vehicle • These general claims can be be supported by a variety of more specific sub-claims in a hierarchy • Operates a vehicle in conformance with posted signs • Stops at a posted stop sign • Obeys posted speed limits • The final hierarchy of claims drives the evidential argument and the score reporting needs of the test

16. Building a Proficiency Model: Proficiencies • Proficiency variables are the basis of making claims • Proficiency variables, like claims, may be organized hierarchically • Values or levels of proficiency variables may be selected to support claim distinctions • Informed by investigations of the domain and learning within the domain • job analyses, cognitive task analyses, learning models, etc.

17. Example of a Hypothetical Proficiency Model for Driving

18. Still to Come • Illustration of Evidence Modeling and Task Modeling (David) • Commentary from an applied perspective (Barbara)

19. Providing the Evidence in Evidence Centered Design David M. Williamson, Ph.D.

20. Building Evidence Models • Objective: valid inference • Understanding what constitutes evidence and why • Situations that elicit evidential behaviors • Task design • Statistical representation of evidential value of observations • Integration of scoring with design • ECD models – conceptual scoring

21. Establishing Evidence in a “Perfect World” Targets of Inference Links behaviors and inference Behavior Elicitation • Driving Ability • Knowledge of RotR • Psychomotor • Evidence • Knowledge declarations • Performance • Task Requirements • Situation • Outcomes

22. Establishing Evidence: Driver’s Exam • Driving Ability • Knowledge of RotR • Psychomotor • Intersections • Recognizing signs • Specifying intersection traffic behaviors • Following proper procedure • Knowledge of RotR • Weather & Light • Traffic Flow • Intersections • Speed • Psychomotor • Intersections • Speed • Turning • Parking

23. Building Task Models: Why? • Elicits behaviors by predictably manipulating • Focus • Difficulty • Context • Targets what we want to measure • Provides a framework for generating multiple tasks targeting same proficiencies

24. Building Task Models: What? To target elicitation of particular evidence, the task model specifies: • Tools provided • Directions and stimulus materials • Indication of data collected and retained for scoring • Relevance of specified elements to particular evidence, and particular proficiencies

25. Evidence as Inference about Proficiency • Stage 1: Evidence Identification • Identify and summarize response elements • Stage 2: Evidence Accumulation • Using these elements to estimate ability

26. Beneficial Outcomes of ECD • Development direction w evidential focus • Integration of expertise/activities (TD, stat, etc.) in the design process • Chain of reasoning • Integrated design – implications of design changes • Validity argument • Item modeling • AIG • Defensibility • Directed item writing – future forms • Somewhat more expensive (Initially)

27. Still to Come • Commentary from an applied perspective (Barbara)

28. Discussant Barbara Showers, Ph.D. Director, Office of Education and Examinations Wisconsin Department of Regulation and Licensing