Assessment: Measuring the Impact of our Programs on Students

1. Assessment: Measuring the Impact of our Programs on Students Cathy Manduca and Ellen Iverson SERC at Carleton College

2. Why Assess? • “I don’t know what we are doing right” • “We only have 120 hours”

3. Performance Evaluation • Does the program address an important challenge? • What is the quality of the implementation? • What is the utility--are participants using it? • Is it being used in different situations? • Is it being shared by participants? John McLaughlin

4. Why Assess? • “I don’t know what we are doing right” • “We only have 120 hours” • “We will lose majors if we do X” • “Change is accomplished one funeral at a time” • “That would really help me with the dean” • “It must work because other institutions are doing it.”

5. Adoption of a Practice • What results does it yield? • How do you do it? • Where is it flexible? • How much does it cost? • What aspects of context influence use? John McLaughlin

6. Powers of Observation Applied to Complex Systems

7. Experimental Design

8. Assessment and Geoscience Research Experimental design: What do I want to know? How can I obtain this information? What do I think is going on? How can I test this idea? Analysis: Is the experimental apparatus working? Is the experiment yielding the desired information? How can I verify my results? Interpretation: How do I determine causality? Is this the only interpretation?

9. Observing Complex Systems Why are undergraduate research experiences valuable? • What do I think is going on? • What can I observe to test my theory? • How can I quantify my observations?

10. Geologic Strategies in Assessment • Multiple working hypotheses • Observations • Describing, classifying, coding • Inferring process and cause • Probing complex systems

11. Understanding your model through concept maps • Articulate and map change process • Go from your assumptions to desired goal • Present plausible rationale for how program works • Analyze likelihood of success • Clarify what should be measured when, how, and by whom • Depict the links between the resources, activities, or services and desired outcomes

12. Internship logic model example Students gain from program: Who then impact: Which impact students’: Employers perception of program Success in finding a job in geoscience at graduation Longevity at first job Knowledge Skills Attitudes Networks Resources Students participate in internship initiative Other students’ perception of program

13. Are you implementing your model? • Is your model correct? • Is what you are mapping what you think should happen? • What are the impacts? • Are the impacts caused by your planned actions?

14. Qualitative – self report and artifacts Open-ended survey questions Embedded assessment Artifacts such as reports or online discussion Interviews and focus groups Quantitative - counting Data sets Surveys Instruments to measure interventions Observations

15. Why Assess? • “I don’t know what we are doing right” • “We only have 120 hours” • “We will lose majors if we do X” • “Change is accomplished one funeral at a time” • “That would really help me with the dean” • “It must work because other institutions are doing it.” What is the purpose? Who is the audience? What will make a compelling argument?