Technology Tools for Facilitating Learning Outcomes Assessment

1. Strengthening Student Success Technology Tools for Facilitating Learning Outcomes Assessment Jerry Rudmann, Irvine Valley CollegePat Arlington, Coastline Community CollegeOctober 2007

2. Agenda and Goal • Technology Uses • Technology ToolsExpected Outcome: Be able to select and use technology-based approaches to assess student learning outcomes

3. Assessment Challenges • Assessing Students in Large Classes • Assessing Performance at a Distance • Minimizing Subjectivity in Assessment • Creating Authentic Assessments • Engaging Students in Self-Evaluation • Accommodating Diverse Learning Styles • Assessing Conceptual Thinking

4. Technology Tools • CCC Confer (Web Conferencing) • Online Rubric Builders • eLumen (SLO Assessment/Tracking) • Calibrated Peer Review (CPR) • Classroom Responders (“Clickers”) • Scannable and Online Tests • Cmap Tools (Concept Maps) • ePortfolio

5. Robust SLOs • Developed through faculty dialog • Behavioral/measurable • Real-world • Higher-level • Conditions • Performance Criteria • Global, over-arching • Scored with rubric

6. CCC Confer • Small-group work in project-based learning • Assessment of DL students • Foreign language classes • Speech/oral communication • Information literacy • Score using discussion or presentation rubric http://www.cccconfer.org

7. CCC Confer Screen Shot

8. Online Presentation Rubric Rubric for Online Presentation Excerpted and adapted from Evaluating Student Presentations rubric by Education Development Center, Inc.: http://www.internet4classrooms.com/doc/PBL_germs.doc

9. Discussion Forums • Tools • Integrated discussion forum in CMS or standalone (e.g., WebBoard by O’Reilly) • Uses • Practice/enrichment • Small-group project-based learning • To assess understanding – score with rubric

10. Online Discussion Rubric http://www.uas.alaska.edu/sitka/IDC/resources/onlineDiscussionRubric.pdf

11. Online Rubric Builders • Rubrics to guide and measure learning • Tools • Rubistar • Landmark Rubric Machine • Coastline Rubric Builder

12. Rubistar Art History Rubric Rubistar

13. Rubric Builder Screen Shot

14. Coastline Rubric Builder Coastline Rubric Builder

15. Digital Assessment Suite • Plagiarism • Grading (GradeMark) • Peer Review • GradeBook http://www.turnitin.com

16. Peer Review Screen Shot Return to Turnitin ~ Next Slide

17. How Do You Report Results?

18. eLumen to Assess SLOs • Reduce Time Spent Creating Reports • Assess Course, Program, and/or Degree-Level Outcomes • Share Assessment Rubrics Across Classes and Programs • View Individual or Aggregated Results • Use Online or Offline http://www.elumen.info

19. Use Online or Offline

20. Criterion-Based Assessment • Rubrics are attached to each SLO Excerpted from eLumen: A Brief Introduction by David Shupe, July 2007

21. Rubrics Describe Criteria • Writes prose clearly Excerpted from eLumen: A Brief Introduction by David Shupe, July 2007

22. Student activity Student learning outcome(s) Rubric Link Activity to Rubric • “Link” a specific student activity (e.g. project, paper, test) to the SLO(s) for which the activity can serve as evidence Excerpted from eLumen: A Brief Introduction by David Shupe, July 2007

23. Writes prose clearly Rubric Read and write a response paper for the novel A Lesson Before Dying Critically analyzes a text Rubric Considers ethical aspects of a situation or text Rubric Link to Multiple SLOs • Here, one assignment stands as evidence for 3 different SLOs Excerpted from eLumen: A Brief Introduction by David Shupe, July 2007

24. Library of Degree-Level SLOs Excerpted from eLumen: A Brief Introduction by David Shupe, July 2007

25. And Rubrics Link to SLOs Excerpted from eLumen: A Brief Introduction by David Shupe, July 2007

26. from the Science committee from the Biology Department from the faculty committee on critical thinking from the faculty committee on communication skills Science and Gen Ed SLOs/Rubrics Excerpted from eLumen: A Brief Introduction by David Shupe, July 2007

27. Scorecard for All Students Excerpted from eLumen: A Brief Introduction by David Shupe, July 2007

28. Class Scores by Student Excerpted from eLumen: A Brief Introduction by David Shupe, July 2007

29. Aggregated Data for Course Excerpted from eLumen: A Brief Introduction by David Shupe, July 2007

30. Course Aggregates by Program Excerpted from eLumen: A Brief Introduction by David Shupe, July 2007

31. Calibrated Peer Review • Web-based program that enables frequent writing assignments with minimal impact on instructor time • Uses peer review • Promotes deeper learning http://cpr.molsci.ucla.edu/

32. Calibrated Peer Review in Psychology 101

33. Critical Thinking in Introductory Psych Course SLO on Pseudoscience skepticism: Students will correctly identify non-scientific explanations of human behavior and explain why those explanations are not based upon science and do not provide reliable or valid explanations of behavior or predictions of future behavior.

34. The Pseudoscience Belief Test Adapted from…Walker, Hoekstra, & Vogl, (2002). Science education is no guarantee of skepticism, Skeptic, vol 9, no 3.

35. Critical Thinking Experiment Using a SLO as the Dependent Variable Calibrated Peer Review Lesson on Graphology Randomly Assigned 90 Students Pseudoscience Belief Post-test Pseudoscience Belief Pre-test Calibrated Peer Review Lesson on Different Topic

36. CPR Procedure • Students read assignment • Students read resource materials • Students wrote a short essay in response to the materials: Why or why I believe graphology is a reliable, valid way to measure and predict personality. • Students are “calibrated” – prepared to score essays written by their peers. • Students receive a detailed grade report for the assignment.

37. Graphology Belief ScoresStatistical Summary

38. Mean Pre and Post-Test Scoreson Graphology Belief Question

39. Example EssayThe Detection of a Pseudoscience: Graphology Elaine Quigley’s posting on the website www.businessballs.com is littered with “red flags” that expose graphology as the pseudoscience/pseudopsychology that it is. While an attempt to promote graphology, Quigley’s posting fails to measure up to several of Cotton and Scalise’s guidelines for “baloney detection.” This paper will examine four areas in which graphology fails to live up to its claim of being “science.”In an attempt to display graphology’s validity, Quigley cites the notion that it is “a very old and respected science.” The fact that it has existed for approximately 3,000 years is used to justify Quigley’s notion that graphology is a science. However, one educated in the definition of science knows that the age of a theory is not a factor used to determine its validity. In fact, there are many beliefs that have been around for thousands of years that cannot be tested and therefore cannot be deemed as scientifically reliable. Graphology is just one of many ideas that cannot be justified despite their age. Quigley also fails to tell how the “science” of graphology has been tested and proven. Instead, she simply states that graphology is a “reliable indicator of personality and behavior” and expects her readers to accept this statement as fact. She also mentions that “the science is still being researched and expanded.” This is the extent to which she approaches the issues related to the research of graphology. Without explaining the testing that was done to prove the methods reliability, how is one to know that graphology is indeed reliable? Indeed, the answer is simple. It is impossible to be sure of the reliability of a measure of personality if the measure itself cannot be tested. In addition to not presenting methods for testing the claims of graphology, Quigley also fails to present evidence in support of its validity. Instead, she simply states that “it is not easy to explain how and why graphology works, nevertheless it continues to be used, respected and appreciated by many.” Could it be that the only “evidence” for the reliability of graphology is the satisfaction that its users experience? Unfortunately, being “used” and “accepted” characteristics required of a science. Finally, the vast majority of information provided by Quigley is anecdotal and leads up to a sales pitch for her services. She provides vague stories about how graphology has been used to produce more successful hiring processes and personal relationships. The information is presented more as an advertisement than a scientific work. Quigley goes into more detail on her experience as a graphologist than she does on the aspects of graphology that would qualify it as a science. In conclusion, it is quite clear that based on the evidence presented in this paper, graphology qualifies as a pseudoscience rather than a science. The claims of graphologist Elaine Quigley fail to show that graphology is indeed a science. Instead, she relies on the age of graphology and anecdotal evidence in support of graphology while ignoring issues related to methods for testing graphology’s claims and the results that have resulted in tests of its validity. Looking critically at “discoveries” is no doubt a useful tool that extends beyond the subject of graphology. The methods for recognizing pseudosciences compiled by Cotton and Scalise are certainly tools that would empower all people and prevent them from being fooled by pseudoscientific claims.

40. Questions and Answers for CPR Peer Reviewers 1. Did the essay begin with a topic sentence? 2. Was the essay free of spelling and grammatical errors? 3. Did the essay present at least four (4) different reasons for supporting or denying the validity of graphology (or handwriting analysis) as a method of assessing personality and/or predicting behavior? 4. Did the essay have balance? Although this may seem subjective, do you feel that it provided a balance among each of the points made? For example, was each point was explained in the same amount of detail. 5. Did the author's arguments seem convincing to you? 6. Did the author conclude with any reflection about whether this assignment was or was not helpful to his or her learning? In other words, did the author indicate that this assignment might help him or her judge the validity explanations of behavior encountered in the popular media (newspaper, radio, TV, magazines, etc.)? 7. How would you rate this text? (Scale of 1 – 10)

41. Student’s Screen: Detailed Results

42. Instructor Screen: Student Progress

43. Instructor Screen: One Student’s Results

44. Instructor’s Screen: Student Results SLO Data

45. Classroom Responders • Engage students • Monitor student understanding • Quickly and easily collect and store assessment data • Use publisher item banks or create your own

46. Renaissance Classroom Response System PBS Demo

47. Renaissance Learningfor clicker training resources http://www.renlearn.com

48. Scanning Technology • Embedding Questions in Multiple Sections and Classes • Basing Multiple-Choice Questions on Case Studies and Complex Scenarios http://www.scantron.com and http://www.renlearn.com

49. Surveys and Tests • Online or Scannable • Surveys • Pre and post surveys of student self evaluation of progress • Faculty and business community or advisory groups related to expected learning outcomes • Student satisfaction (indirect outcomes) • Quizzes/Tests • Practice and graded Cross Match

50. Concept Maps http://cmap.ihmc.us/