Assessing Student Learning to Improve Teaching

1. Assessing Student Learning to Improve Teaching • Jeff Bell and Jim Bidlack

2. Assessing Student Learning • This session will provide some suggestions for how your current assessments can be “tweaked” to provide tools to assess learning objects or any change in teaching style or technique

3. Why Assess Student Learning? • To assign grades (not the point of this talk) • To determine the effectiveness of the instruction (the point of this talk) • Assumption: to improve instruction you need feedback on the effectiveness of the instruction

4. Why Not Use Grades? • Grades for the course, an exam, or even a paper usually are affected by too many different factors to give a good indication of the effectiveness of a specific learning task • Purpose is to assign a value to a student, not the instructor or instruction • Need to define very specific learning objectives, and then measure student achievement of those objectives

5. Example Using Grades- The Use of Clickers • Clickers are used in large lectures to quiz the students during the lecture • Jim Bidlack looked at the effectiveness of this strategy in a Biology class

9. Problems • If no statistical difference (the most likely outcome) there is no evidence to guide changes • Were students coming to class? • Did they do better on “clicker questions” but worse on other questions? • Did exam questions cover the same learning objectives as the clicker questions? • Did changes in class size, students, etc., affect the results?

10. Specific Learning Objectives • Specific Learning Objectives (SLOs) - narrowly defined to cover one specific skill or knowledge • Should describe what the student can ‘do’ • Not – “understands probability,” but, “Can use probability theory to calculate the expected frequency of unordered events” instead

11. Genetics Examples • Can use the ratios of phenotypes produced in a monohybrid cross to determine the genotypes of the parents and the mode of inheritance of the trait • Can predict the probable outcome of phenotypes in a monohybrid cross from the genotypes of the parents and mode of inheritance

13. How to Get SLOs • SLOs should be very specific and it should be obvious how to measure them • Examination of your current tests and assignments will provide you with many of the actual SLOs for your courses • If you don’t have any assessments for a SLO then your students may not really be learning that SLO

14. How to Measure SLOs? • Item analysis of exam questions or assignment tasks • Surveys • To measure the effect of an instructional change will also need • Pre-test and post-tests, or • Control groups and experimental groups

15. Item Analysis • Typical exams and assignments have multiple SLOs, but can be designed so that each individual SLO can be assessed • SLO: Can use probability theory to calculate the expected frequency of unordered events • Exam question: The probability that in a family with seven children, three of the children will be girls, is? • Assessment: Fall’05 – 47/49 (96%)

16. SLO: Can use a pedigree, the laws of inheritance for an autosomal gene, and the product, sum and conditional rules to calculate the risk of affected children in a specific mating. • Exam question: An older women who has no genetic disease has a sister with the recessive disease cystic fibrosis; both her parents are normal. How likely is it that she is not a carrier for the genetic disease? • Assessment: Fall’05 – 30/49 (61%)

17. Process • Write the exam so each question covers one SLO • When grading each student keep track on a separate sheet of success or failure for each SLO • Tally the success rate for each SLO • For assignments, use a rubric and follow the same process as above

19. Surveys • You can ask your students what they know • If the question is specific, they usually do know what they can do, or not do • “Before taking this course, I could use a chi-square statistical test to test a hypothesis about expected ratios in a genetic cross and understood what the p value the test returned meant.”

20. Before taking this course, given the probability of the independent individual events I could predict the probability of any sequence or set of outcomes (e.g., the probability of a family with five children having three boys and two girls). • Pre – 2.27 (5=strongly agree) • Post – 4.23 (3 students disagreed) • Exam – 96% (2 students failed)

21. Pre and post test • Test them before instruction and after and then use item analysis • Problem: Can not usually grade them on the pre-test – effort on pre-test is usually not as great as on post-test • Good if you want to always show a positive effect • Does work for low effort recall type questions

22. Spring ‘02

23. Control and Experimental Groups • Best if done at the same time with different sections – logistically difficult and may have a placebo effect • Different semesters – have to get baseline data before trying your innovation • Point of this talk - you need to start now even if you don’t know what you’re going to do next year • Bonus – baseline data may tell you where you really need something new!