Assessing Teaching Quality Student Assessments

1. Charles A. Burdsal and Sandra Ranney Wichita State University Assessing Teaching QualityStudent Assessments

2. Has evolved over a 30+ year period. 39 item (+ 2 validity items) questionnaire. Not self administered. Scales derived using factor analysis. Biases dealt with. SPTE: A quick Overview

3. SPTE Results Front

5. Decide the intent of the instrument. Is it primarily for summative or formative purposes? • If the purpose is formative, almost anything a faculty member finds useful is fine. • If summative; i.e., used for such things as tenure, promotion and salary adjustments, then we have a very different situation. Note that a good summative instrument may also have formative value. Some General Principles

6. While anyone who teaches may offer valuable content for such a questionnaire, it is important to have someone involved with a significant background in questionnaire development. • Scales are more useful for evaluations than single items. • While there are other methods, factor analysis is a very useful tool to develop scales from items. Developing a Good summative Instrument

7. Quality Scores by College/Division

8. Now Let’s Look at Difficulty

9. Biases in Teaching Evaluations Just What Do We Need to Fix?

10. Course Difficulty (72%) Course Workload (60%) Class Size (60%) Instructor Popularity (63%) Student Interest in Subject Before Course (55%) Reason for Taking the Course Grading Leniency (68%) Student’s GPA H. W. Marsh (1984)

11. “Perhaps for each large, representative, well designed study, there is another study, comment, or electronic bulletin-board message that relies on an atypical anecdote or an appeal to popular myth for its impact.” Marsh & Roche (2000) Endurance of Myths

12. We believe that three of Marsh’s biases can be combined in the concept of a priori motivation: • Instructor Popularity • Student Interest in Subject Before Course • Reason for Taking the Course A Priori Motivation

13. Prior to enrolling in this class, I expected it to be of (little or no value . . . . great value) • When I enrolled in this class, I wanted to take this course (very much . . . . not at all) • I took this course because I had to. (strongly agree . . . . strongly disagree) • I took this course because I was interested in the subject. (very much . . . . not at all) S.P.T.E. Motivation Scale (α= .92)

14. Relation of Motivation Scale to Factors (N = 21,532 classes)

15. A priori motivation is not in the control of the instructor. Its effect should be removed from other evaluation scales. S.P.T.E. does so using regression techniques. Recommendations

16. Marsh & Roche (2000), Greenwald (1997) review concerns of the relation of Course Demands & Difficulty on Student Evaluations. The harder the course, the worse the ratings. Course Difficulty

17. The Relationship of Course Demands with:N = 21,532 classes

18. Frequently believed that the larger the class is the poorer the ratings. Class Size

19. Relation of Number of Raters to:N=21,932

20. As most student evaluation instruments are administered before grades are assigned, we are really talking about expected grades. A great deal of concern as to the relation of expected grades to student evaluations is seen in the literature – Greenwald & Gillmore (1997), Greenwald (1997), Marsh & Roche (2000), etc. Expected Grades and Student Evaluations

21. Teaching effectiveness influences both grades and ratings. • Students’ general academic motivation influences both grades and ratings. • Students’ course-specific motivations influences both grades and ratings. • Students infer course quality and own ability from received grades. • Students give high ratings in appreciation for lenient grades. Possible Reasons for the Relationship Between Grades and Student Evaluations(Greenwald & Gillmore, 1997)

22. Correlations of Expected Grade with SPTE Scales (n=21,532)

23. The relationship of expected grades & teaching evaluations is probably not one-dimensional. By removing a priori motivation from the NORMED scales, one noticeably reduces their correlation with expected grades. This probably supports the third of Greenwalds’ model – course expectations of specific courses affects the ratings of that course. The remaining correlations seem rather small and hopefully are related to learning. Conclusions Concerning Bias’s

24. Distribution Issues

25. We decided to examine open ended comments to see if the could help. • The questions asked on the comment sheet: • What could the instructor do to improve the course? • What did you like about the course and/or instructor? • Please comment on the effectiveness of computer-aided instruction, if it was used in the course. • Any additional comments? What We Did about it

26. 250 sections sampled randomly. 24 opted not to participate. 13 excluded because of various problems. Two excluded because we forgot to enter them. Sampling

27. The 250 sections were sampled from the Fall, 2002 administration of SPTE from WSU. • The Social Science Lab staff copied the comments and returned the originals to the faculty member. • They then read all comments eliminating anything in a comment that might identify the faculty member. • Spring & Summer 2003, data unitized, entered in Excel and rated. • Fall 2003, data was analyzed. Data Collection

28. Comments were unitized and entered in Excel. 14,313 comments in all after eliminating “not applicable” After entering, comments reviewed for proper unitizing. SPTE Comments

29. Valences of 1 to 5 were assigned to each comment. 1 the most negative; 5 the most positive. Examples of each follow: Valences

30. Valence 1 Examples • You suck! • I wish I never came to this University • She makes me feel foolish in front of class.

31. Slow down a little bit. Try and set more deadlines and stick to them. Let us know what our grades are at least by mid-semester. Valence 2 Examples

32. Bring treats! Blackboard was used in the course. I type my papers on the computer. Valence 3 Examples

33. His knowledge of the subject is very good. I like the fact that the homework was relevant to the exams. The instructor is enthusiastic. Valence 4 Examples

34. You gave me so much personal attention – THANK YOU! Best accounting class I’ve had at WSU so far! He’s one of the best instructors I’ve had at WSU! Valence 5 Examples

35. Average valence was computed for each section. The mean was 3.4. Mean valences for each section ranged from 2.2 to 4.2. Standard deviation was 0.377 Summary of Valences

36. Uncorrected University PQI .803 Uncorrected Local PQI .758 Corrected University PQI .790 Corrected Local PQI .746 Correlations of Comment Valences with SPTE’s PQI (Overall Score)

37. There is a strong positive relationship between the valence of comments and the overall SPTE score (PQI). Comments do go with the scales. Yet, it still doesn’t feel right? Conclusions?

38. Means PQI for each of the Valence Ranges

39. People are probably being too negative in interpreting SPTE (or probably any SETE). SPTE results need to have some interpretation help to have them reflect the comments. So What Does this Mean to SETE/SPTE?

40. Ranges with Adjectives z (Scale)

41. Some of us are really trying and it seems like it's never ever good enough. I am sorry you don't seem to enjoy what you teach. It must make life rough. I felt I was robbed from this class and what I could gain if another had taught. I believe that I would of gotten more out of it if my dog taught the class. I fear that because of the poor teaching of the instructor, that I am going to struggle through many classes in the future.. I felt that the instructor was rude most of the time. “LOW” Comments

42. She/he's nice Make her lectures more interesting Very friendly When they accuse someone of academic dishonesty have proof! She/he could have graded more effectively. It challenged me to be one step ahead, instead of one step behind. Have more than one major paper. XXXX was very nice and understanding. She/he was fun Be clear about what is on the tests! “Good” Comments

43. Well organized Instructor is very well educated with the subject. The instructor helped the students understand the material through his classroom lectures. Wrote very good notes. I couldn't get detailed enough answers to some of my questions. That she/he was cool The instructor covered the topics supposed to be covered very well with what the topics were suppose to be. The instructor made learning fun and understandable. “V Good” Comments

44. I truly appreciated your ideas and real-life experience. His mastery of the subject matter was superb. (What would you change?) Nothing, she/he did a great job!! I really enjoyed this class and was upset when I had to miss it. She/he is exceptionally clear about what she/he expects from her students. The instructor was wonderful. He/she is enthusiastic and passionate about this subject which helps in focusing the students on the subject of XXX. I don't think there is anything he/she could do to improve the course. “High” Comments

45. Get people skilled in questionnaire development (not just people good with surveys) involved in producing your instrument. All results should be norm based with the emphasis on scales rather than single items. Correct for known sources of bias beyond the instructor’s control. Deal with the issue of negative skewness of quality scales. In Conclusion