Department of Psychology & Human Ecology School of Education and Behavioral Sciences

1. Department of Psychology & Human EcologySchool of Education and Behavioral Sciences Masters of Science in Behavioral Sciences 421601 610 Program Quality Improvement Report 2009-2010

2. MSBS Program Objectives & Student-Learning Objectives To foster the ability to synthesize and critically analyze information. 1. Students will synthesize and analyze information critically. To promote research skills in the behavioral sciences. 2. Students will demonstrate basic research skills in the behavioral sciences. To promote basic knowledge of theories, methods, and findings in the behavioral sciences. 3. Students will demonstrate knowledge of theories, methods, and findings in the behavioral sciences. To promote competence in the application of basic knowledge within the field of the behavioral sciences. 4. Students will apply basic knowledge within the field of the behavioral sciences. Program Quality Improvement Report 2009-2010

3. Holistic Alignment of Program Objectives to University, School, Departmental, & Program Mission The MSBS Program is dedicated to providing a quality education for a diverse and dynamic student body. As a consequence, our program is designed to “serve that segment of the population employed in or interested in a broad spectrum of professions in the behavioral sciences.” Our students are educated to be … and have proven to be a dynamic force in the behavioral sciences in our community. Program Quality Improvement Report 2009-2010

4. Holistic Alignment of Program Objectives to University, School, Departmental, & Program Mission The MSBS Program is committed to fostering a student-centered academic environment that combines innovative classroom teaching with experiential learning. We have listened to our students and have restructured our program to meet their career and educational goals. We have provided them an education anchored by dynamic professionals actively working in the field of the behavioral sciences and have collaborated with the behavioral sciences community to ensure an experiential learning environment. Program Quality Improvement Report 2009-2010

5. Holistic Alignment of Program Objectives to University, School, Departmental, & Program Mission The MSBS Program is dedicated to preparing students for professional success, responsible citizenship, life-long learning, and meaningful contributions to a rapidly changing world. We have structured our program to prepare students for successful professional licensure in the behavioral sciences with a strong ethical underpinning which will ensure significant and dynamic contributions to a rapidly changing community. Program Quality Improvement Report 2009-2010

6. Holistic Alignment of Program Objectives to University, School, Departmental, & Program Mission With a strong emphasis on the study of psychological cultural diversity and a zeal for a strong work ethic, the MSBS Program is a driving force in the cultural life and economic development of our community. Focused on promoting theoretical & methodological cultural competencies, our program challenges our students to be a driving force for compassionate care in our community. Program Quality Improvement Report 2009-2010

7. Measures of Learning Outcomes • Direct Measures: • Locally developed tests • Within course tests • End of program comprehensive examination • Literature reviews scored by rubrics • Indirect Measures: • Graduates’ exit interviews • Employer/supervisor evaluation of practicum experience. Program Quality Improvement Report 2009-2010

8. Measures of Learning and Service Outcomes 3. Strategies for addressing shortfalls in student learning prior to graduation: a. A Student Data Base has been created: i. Demographic Information ii. Undergraduate performance iii. Educational/Career goals iv. Recruitment & Retention Information v. Advisement Information vi. Academic Tracking Information Program Quality Improvement Report 2009-2010

9. Measures of Learning and Service Outcomes b. Using the Student Data Base to facilitate identification of shortfalls, online learning modules will be developed in order to help student remediation. i. Synthesis & Analysis ii. Research Skills iii. Theories, methods & findings iv. Applications Program Quality Improvement Report 2009-2010

10. ACTION PLAN 09-10 PQIRObjective 1: Students will synthesize and analyze information critically Create embedded questions in Research Methods related to synthesis and critical analysis. Develop validity and reliability for embedded questions. Action Taken: ABANDONED Create rubric to assess synthesis and critical analysis in Research Methods final/applied project. Develop validity and reliability for the rubric. Action Taken: MODIFICATION: Create a rubric to assess synthesis and critical analysis in within the first 18 credit hours and then again in the last 18 credit hours. Program Quality Improvement Report 2009-2010 10

11. ACTION PLAN 09-10 PQIRObjective 1: Students will synthesize and analyze information critically • Assess synthesis and critical analysis in first course post 21 credit hours. A. Action Taken: 2.B. • Continue to indirectly assess synthesis and critical analysis in the exit interview process. A. Action Taken: Exit interview data collection on synthesis and critical analysis continues. Program Quality Improvement Report 2009-2010

12. ACTION PLAN 09-10 PQIRObjective 1: Students will synthesize and analyze information critically Program Quality Improvement Report 2009-2010

13. ACTION PLAN 09-10 PQIRObjective 1: Students will synthesize and analyze information critically Program Quality Improvement Report 2009-2010

14. ACTION PLAN 09-10 PQIRObj. 2: Students will demonstrate basic research skills in the behavioral sciences. • Operationally Define Research Skills. A. Action Taken: Operational Definition of Research Skills. 1. Students will demonstrate knowledge in research ethics. 2. Students will analyze and synthesize information in the existing professional literature. 3. Students will demonstrate the ability to form a logical and testable hypothesis. Program Quality Improvement Report 2009-2010

15. ACTION PLAN 09-10 PQIRObj. 2: Students will demonstrate basic research skills in the behavioral sciences. • Operationally Define Research Skills. • Action Taken: Operational Definition of Research Skills • Students will demonstrate the ability to develop research designs appropriate to test the hypothesis. • Students will demonstrate knowledge of statistics. • Students will demonstrate the ability to select the appropriate statistics for data analysis. • Students will demonstrate the ability to analyze data and interpret results. • Students will demonstrate the ability to present findings. Program Quality Improvement Report 2009-2010

16. ACTION PLAN 09-10 PQIRObj. 2: Students will demonstrate basic research skills in the behavioral sciences. • Develop embedded questions in Research Methods, Ethics and Statistics in order to directly measure the students knowledge of research skills. • Develop reliability and validity for embedded questions. 2. Action Taken: Embedded questions within three main courses were developed in Spring 2010 and data collection will begin Fall 2010. 1. Action Taken: Question were disseminated to topic matter experts to determine Content Validity. Experts were asked to sort questions by subject content. Crombach Alpha coefficient were performed and the results indicate a r = .92 Program Quality Improvement Report 2009-2010

17. ACTION PLAN 09-10 PQIRObj. 2: Students will demonstrate basic research skills in the behavioral sciences. • Develop a rubric to evaluate the students ability to apply Research Skills in an end of course project. A. Develop reliability and validity for applied Research Skills rubric. • Action Taken: Initial development of the rubric began Spring 2010. Developmental work continues. A. Action Taken: Pending finalization of rubric. Program Quality Improvement Report 2009-2010

18. ACTION PLAN 09-10 PQIRObj. 2: Students will demonstrate basic research skills in the behavioral sciences. • Assess research skills in the behavioral sciences in first course post 21 credit hours. • Action Take: AMENDENED Assess research skills in the behavioral sciences within the first 18 credit hours. New programmatic changes have been submitted for Regent consideration. • Continue to indirectly assess synthesis and critical analysis in the exit interview process. A. Action Taken: Continues to be assessed. Program Quality Improvement Report 2009-2010

19. Learning Objective 1Outcome Data Students will synthesis and analyze information critically. Rubric Measure - Direct Measure Exit Interview Questions pertaining to synthesis and analysis – Indirect Measure. Program Quality Improvement Report 2009-2010 19

20. Learning Objective 1: Students will synthesize and analyze information critically Program Quality Improvement Report 2009-2010 20

21. Assignment • With the exception of Graduate Statistics, every graduate course has literary assignment meant to test the students’ ability to synthesize and analyze information. These assignments all include a review of the professional literature with an emphases on the students’ ability to pull information from different sources to create a cohesive analysis of the information. Program Quality Improvement Report 2009-2010

22. Learning Obj. 1 – Direct MeasureRubric Outcome CATEGORIES: • Introduction & Summary • Thesis • Organization • Supporting Information • Evidence & Examples • Paragraphs • Transitions • Conclusion • Mechanics Program Quality Improvement Report 2009-2010

23. Learning Obj. 1 – Direct MeasureRubric Outcome1 = Below Standard2 = Approaching Standard3 = Meets Standard4 = Above Standardhas not met programmatic expectations Program Quality Improvement Report 2009-2010

24. Learning Obj. 1 – Direct MeasureRubric Outcome • Trend Analysis: First semester to use this measure. No trend analysis. • Action Plan: • Continue to collect Interrater Analysis data. • Timeline: Spring 2011 Program Quality Improvement Report 2009-2010

25. Learning Obj. 1 – Direct MeasureRubric Outcome • Action Plan: • Intervention to improve performance: • Determine the source of the problem. Is it a poor measure, poor instructor preparation, or poor student performance? • Give students the rubric with syllabus. i. Timeline: Each semester • Develop an online module to aid in understanding and usage of the dimensions of Synthesis & Critical Analysis. i. Timeline: Start development Spg 2011; Finished by Spg 2012 Program Quality Improvement Report 2009-2010

26. Learning Objective 1: Students will synthesize and analyze information critically Program Quality Improvement Report 2009-2010 26

27. Learning Objective 1: Indirect MeasureExit Interview Results 2009-2010 Please rate how well the program prepared you to synthesize information on a scale of 1 (low) to 7 (high)? n = 38 Mean = 5.72 Standard Deviation = 0.92 Please rat how well the program prepared you to critically analyze information on a scale from 1 (low) to 7 (high)? N = 38 Mean = 5.78 Standard Deviation = 1.01 Program Quality Improvement Report 2009-2010

28. Learning Obj. 1: Indirect MeasureExit Interview Program Quality Improvement Report 2009-2010

29. Learning Obj. 1: Indirect MeasureExit Interview Action Plan: • Continue to collect data on the Exit Interview. • It is expected that the corrective plan for improving performance on the Rubric measure will improve on Exit Interview reporting. Program Quality Improvement Report 2009-2010

30. Learning Objective 2Outcome Data Students will demonstrate basic research skills in the behavioral sciences. 1. Direct Measure: Research Proposal graded by Rubric. 2. Indirect Measure: Exit Interview Questions pertaining to attainment of research skills in the behavioral sciences. Program Quality Improvement Report 2009-2010

31. Learning Objective 2: Students will demonstrate research skills in the behavioral sciences.Rubric Outcome – Direct Measure Program Quality Improvement Report 2009-2010 31

32. Learning Obj. 2 – Direct MeasureRubric Measure • Students will demonstrate knowledge in research ethics. • Students will analyze and synthesize information in the existing professional literature. • Students will demonstrate the ability to form a logical and testable hypothesis. Program Quality Improvement Report 2009-2010

33. Learning Obj. 2 – Direct MeasureRubric Measure • Students will demonstrate the ability to develop research designs appropriate to test the hypothesis. • Students will demonstrate knowledge of statistics. • Students will demonstrate the ability to select the appropriate statistics for data analysis. • Students will demonstrate the ability to analyze data and interpret results. • Students will demonstrate the ability to present findings. Program Quality Improvement Report 2009-2010

34. Learning Obj. 2 – Direct MeasureRubric Outcome1 = Below Standard3 = Approaching Standard4 = Meets Standard5 = Above StandardDoes not meet standard Program Quality Improvement Report 2009-2010

35. Learning Obj. 2 – Direct MeasureRubric Outcome • Trend Analysis: First semester to use this measure. No trend analysis. • Action Plan: • Continue Development of Rubric: • At this point the rubric is just a likert scale without descriptions. • Timeline: Sprg 2011 • Needs to create a statistics & ethics rubric. • Needs to develop specific description of each standard. • Collect Interrater Reliability data. a. Timeline: Each semester for graduating students. Program Quality Improvement Report 2009-2010

36. Learning Obj. 2 – Direct MeasureRubric Outcome • Action Plan: • Shortfalls in student performance. • Determine the source of the problem. • Poor rubric? • Poor rater training? • Poor student performance? Program Quality Improvement Report 2009-2010

37. Learning Objective 2: Students will demonstrate research skills in the behavioral sciences.Exit Interview – Indirect Measure Program Quality Improvement Report 2009-2010 37

38. Learning Obj. 2 – Indirect MeasureExit Interview Outcome Rate how well the program promoted competence in research skills on a scale from 1 (low) to 7 (high) ? n = 38 Mean = 5.89 Standard Deviation = 1.02 Program Quality Improvement Report 2009-2010

39. Learning Obj. 2: Indirect MeasureExit Interview Outcome Program Quality Improvement Report 2009-2010

40. Learning Obj. 2: Indirect MeasureExit Interview Outcome Action Plan: • Continue to collect data on the Exit Interview. • It is expected that the corrective plan for improving performance on the Rubric measure will improve rating on Exit Interview reporting. Program Quality Improvement Report 2009-2010

41. Learning Objective 4 Students will apply basic knowledge within the field of the behavioral sciences. 1. Rubric Outcome – Direct Measure 2. Exit Interview Question pertaining to the application of knowledge within the field of the behavioral sciences– Indirect Measure Program Quality Improvement Report 2009-2010

42. Learning Objective 4: Students will apply basic knowledge within the field of the behavioral sciences.Rubric Outcome – Direct Measure Program Quality Improvement Report 2009-2010 42

43. Learning Obj. 4 – Direct MeasureRubric Measure Areas rated: • Individual counseling • Group counseling • Marital, Family, Couples counseling • Child counseling • Assessment/diagnosis • Treatment planning • Makes appropriate referrals Program Quality Improvement Report 2009-2010

44. Learning Obj. 4 – Direct MeasureRubric Measure Continued: • Consults with other professionals • Conducts research • Knows licensing law and rules • Conforms to Rules of Professional Conduct • Uses disclosure statement • Is prompt/current on paperwork and records • Cares for own mental health Program Quality Improvement Report 2009-2010

45. Learning Obj. 4 – Direct MeasureRubric Measure Continued: • Utilizes supervision sessions effectively • Maintains professional boundaries • Stays within limits of competence level • Keeps current with professional literature Program Quality Improvement Report 2009-2010

46. Learning Obj. 4 – Direct MeasureRubric Measure Levels of appraisal: 0 = No observation 1 = Needs Improvement 2 = Acceptable 3 = Above average skill Note: “No observations” were not used in calculation of proficiency of application Program Quality Improvement Report 2009-2010

47. Learning Obj. 4 – Direct MeasureRubric Measure Program Quality Improvement Report 2009-2010

48. Learning Obj. 4 – Direct MeasureRubric Measure Program Quality Improvement Report 2009-2010

49. Learning Obj. 4 – Direct MeasureRubric Measure Action Plan: • Continue to collect data this upcoming year. • Attempt to collect state-wide data to use as a comparison group. • Make the rubric more detailed in order to help practicum supervisors further understand the standard. • Use supervisor rating to make curriculum changes. Program Quality Improvement Report 2009-2010

50. Learning Objective 4: Students will apply basic knowledge within the field of the behavioral sciences. Exit Interview – Indirect Measure Program Quality Improvement Report 2009-2010 50