Phil Dunwoody, Gerald Kruse Juniata College Huntingdon, PA

1. Assessment: A micro view, with Student Learning Outcomes (SLOs), and a macro view with the Psychology Department’s Assessment Plan Phil Dunwoody, Gerald Kruse Juniata College Huntingdon, PA

2. Learning Outcomes • In new course proposals, faculty must provide the “objectives, skills and knowledge students are expected to gain” and answer the question, “how will you evaluate the students in the course?” • A Student Learning Outcome (SLO) for a course is a brief statement of what a student will know and be able to do at the end of a course. For more info: Assessment Quickies #3, Levels of Student Learning http://www.youtube.com/watch?v=9XTpyloIuxA&feature=context-cha

3. Syllabi Upload • All faculty are required to upload electronic copies of their syllabi http://services.juniata.edu/faculty/syllabi/index.html • Random sample of 50 courses, during the 2010-11 and 2012-13 academic years (pre and post Middle States) • In 2010-11, 15 courses (30%) and in 2012-13, 21 courses (42%) did not have a syllabus on file* *the next section or closest number course was substituted to maintain sample size of 50, and in the meantime CNS has implemented a “reminder” button

4. Rubric, existence only • 2 - Explicitly listed as Student Learning Outcomes, Student Goals, Goals, etc. • 1 - NOT Explicitly listed BUTCAN be inferred from the course syllabus • 0 - NOT listedAND can NOT be inferred from the course syllabus

5. Results of 50 Syllabi Reviewed *source: Gettysburg Committee on Learning Assessment (COLA) newsletter

6. SLO – a “micro” view of one • MA 116, Discrete Structures – first semester IT/CS • Foundational math for study in IT and CS:logic, proof, set theory, combinatorics, relations, functions, graphs, and trees • SLO in syllabusPrevious:... develop an understanding of some of the mathematical structures fundamental to computer science including ... induction ...New:apply mathematical induction to prove conjectures about sequences

7. SLO – it’s not scary • Direct measures of student learningQuizzes, tests, portfolios, papers, standardized tests, capstone projects, … • Take advantage of existing measures!Inductive Proof on Quiz immediately after ModuleInductive Proof on exam • Two additionsPre-assessment proofLink to Kahn Academy video on Induction

8. SLO – Closing the loop • Fall 2012, N = 14 students • No one successfully completed the pre-assessment proof • 5 students reported watching the Kahn Academy video (watching it wasn’t incentivized) • “Mechanical Proof”Prove for all integers n > 0 that 1 + 2 + ... + n = n*(n+1)/2 • “Deeper Proof”As a person arrives at a meeting, they shake hands with all the other people present. If n people are at the meeting, then prove n*(n-1)/2 handshakes occurred.

9. SLO – Closing the loop • “Mechanical” Proofs after Module and after Exam • “Deeper” Proof on Exam

10. Psychology Department Goals • In 2007 the APA published guidelines and 10 goals for undergraduate education. • Voluntary • No APA accreditation for undergraduate programs • Our department adopted these goals • Differentiated between direct goals and indirect goals • Established SLO for all direct goals

11. Our Indirect Goals… • Goal 4: Application of Psychology. Students will understand and apply psychological principles to personal, social, and organizational issues. • Goal 5: Values in Psychology. Students will be able to weigh evidence, tolerate ambiguity, act ethically, and reflect other values that are the underpinnings of psychology as a discipline. • Goal 6: Information and Technological Literacy. Students will demonstrate information competence and the ability to use computers and other technology for many purposes. • Goal 8: Sociocultural and International Awareness. Students will recognize, understand, and respect the complexity of sociocultural and international diversity. • Goal 9: Personal Development. Students will develop insight into their own and others’ behavior and mental processes and apply effective strategies for self-management and self-improvement. • Goal 10: Career Planning and Development. Students will emerge from the major with realistic ideas about how to implement their psychological knowledge, skills, and values in occupational pursuits in a variety of settings.

12. Indirect Goal Example • Goal 8: Sociocultural and International Awareness. Students will recognize, understand, and respect the complexity of sociocultural and international diversity. • Goal 8 is an indirect goal for our psychology department. While it is addressed in some of our courses it is not directly assessed.

13. Our Direct Goals… • Goal 1: Knowledge Base of Psychology. Students will demonstrate familiarity with the major concepts, theoretical perspectives, empirical findings, and historical trends in psychology. • Goal 2: Research Methods in Psychology. Students will understand and apply basic research methods in psychology, including research design, data analysis, and interpretation. • Goal 3: Critical Thinking Skills in Psychology. Students will respect and use critical and creative thinking, skeptical inquiry, and, when possible, the scientific approach to solve problems related to behavior and mental processes. • Goal 7: Communication Skills. Students will be able to communicate effectively in a variety of formats.

14. Direct Goal Example • Goal 1: Knowledge Base of Psychology. Students will demonstrate familiarity with the major concepts, theoretical perspectives, empirical findings, and historical trends in psychology. • SLO: Students will be able to answer content questions about the major content areas within the field of psychology (Abnormal Psychology, Learning & Conditioning, Development, Cognition, Biopsychology, and Social Psychology). • Teaching Methods: All psychology POEs are required to take at least four courses out of the five major content areas (Abnormal Psychology, Learning & Conditioning, Development, Cognition, Biopsychology, and Social Psychology). These courses will include readings, lectures, projects, and assessments designed to develop and test student mastery of the relevant content material. • Assessment: Each senior Psychology POE will take the Psychology Area Concentration Achievement Test (PACAT). This is a nationally normed test and we receive percentile scores to compare our students with similar students nationally. The PACAT provides separate sections and scores for each of the 5 major areas listed above. All senior psychology students are required to take this assessment in a senior capstone. We expect all students to have a minimum overall score at or above the 25th percentile. We expect our departmental average for all students to be at or above the 50th percentile. This assessment is given by the instructor of Senior Capstone.

15. Direct Goal Example Goal 2: Research Methods in Psychology. Students will understand and apply basic research methods in psychology, including research design, data analysis, and interpretation. SLO1: Students will be able to answer conceptual and applied questions about research methods and statistics. SLO2: Students will be able to accurately identify and interpret key aspects of research methodology and statistical analysis while reading an empirical psychology article. SLO3: Students will be able to accurately create and interpret a research study with hypothetical results to test a hypothesis. Teaching Methods: All psychology POEs are required to take Statistics for the Social Sciences (NDSS214) and Research Methods (PY309). These courses will include readings, lectures, projects, and assessments designed to develop and test student mastery of research methods and statistics. Assessment SLO1: All students will take the Psychology Area Concentration Achievement Test (PACAT) in the senior capstone. Scores on the PACAT subsections of Experimental Design and Statistics will be used to assess SLO1. This assessment is administered by the instructor of Senior Capstone. Assessment SLO2 & SLO3: All students in Research Methods (PY309) will complete a graded assessment at the conclusion of PY309 designed to evaluate SLO1 and SLO2. All sections of PY309 will utilize the same assessment tool in any given year. This assessment is given by the instructors of Research Methods.

16. Common Exam Scenario • Professor John is interested in the impact of color on attention. He thinks that brightly colored shop signs will have a greater effect during the day than at night. To test this hypothesis he uses an animal model, presenting rats with dishes of different colors under different lighting conditions. Each dish is filled with sand, but in each trial, only one dish has a Cheerio hidden in the sand—that dish is either dull blue or bright red, with the rest gray. The rats’ search times to find the Cheerio are graphed below in ms:

17. Common Exam Scenario 6) What type of statistical test must Professor John do to test his hypothesis? What must be significant to support his hypothesis? 7) Interpret Describe any main effects and/or interaction effects shown in the data graphed above. (Assume that any difference of more than 1ms is statistically significant.) 8) Is there support for Professor John’s hypothesis? Explain your answer. 9) What group at Professor John’s school had to approve this study? What is one thing the professor had to prove to them to obtain approval? 10) Professor John writes a clear and well-sourced article about his findings. In his discussion, he makes predictions about human behavior when seeking restaurants and shops during the day versus late at night, based on these results. What is one validity concern an editor might raise about these predictions?

18. Yearly Assessment Report

19. Suggestions for writing SLOs • Be specific • Don’t use passive voice, use active verbs • Analyze • Classify • Design • Evaluate • Perform • Synthesize • Teach Assessment Quickies #2, Writing Student Learning Outcomes http://www.youtube.com/watch?v=KA0DeysBX7M&feature=plcp

20. Levels of Student Learning • BasicDefine, Label, Recognize, … • Mid-rangeApply, Interpret, Solve • AdvancedCompose, Formulate, Plan, Judge, Defend Assessment Quickies #3, Levels of Student Learning http://www.youtube.com/watch?v=9XTpyloIuxA&feature=context-cha