Key General Education Outcomes through the Lenses of Four Instruments

1. Key General Education Outcomes through the Lenses of Four Instruments Youssouf Diallo, Ph.D, Research Associate Office of University Assessment Uche Ohia, Ph.D, Director of University Assessment Valencia E. Matthews, Ph.D, Assistant Dean, College of Arts & Sciences Florida A&M University Contacts email: assessment@famu.edu Telephone: (850) 412 5265 IARC 2008_June 13, Scottsdale, AZ

2. FAMU Profile • Florida Agricultural and Mechanical University is a public land-grant , comprehensive historically black institution. • Student population is approx. 12,000. • The 3rd oldest institution in the State University System (SUS). • Offers undergraduate, graduate, research and professional programs.

3. Context Under the guidance of the Office of University Assessment, the assessment process is participatory and involves the following stakeholders: Students, Faculty, staff, and administrators; College/School/Departmental/Administrative committees; General Education Assessment Committee; Institutional Level Assessment Committee;

4. INTRODUCTION (1) This study presents a comparative analysis of results from four separate instruments measuring General Education outcomes.

5. INTRODUCTION (2) The study looked at the consistency of the results across four instruments: NSSE, MAPP, FAMU Exit and Alumni Surveys; Special attention was placed on potential significant differences of performance in critical thinking and communication skills between students from this institution and those from peer institutions. Areas of strengths and weaknesses were highlighted for improvement;

6. The Four Instruments National Survey of Student Engagement (NSSE) is designed to address the following: Information about what students actually do in college, and what they are learning; Information that can be used to improve the educational experience of students; An answer to the fundamental question; are educational practices and curricula producing desirable learning outcomes?

7. The Four Instruments (2) 2.Measure of Academic Proficiency and Progress (MAPP) Test Measures college-level reading, mathematics, writing, and critical thinking in the context of the humanities, social sciences, and natural sciences; Designed for colleges and universities to assess their general education programs;

8. The Four Instruments (3) 3.The FAMU Exit Survey Is designed to provide information about graduating students on both undergraduate and graduate levels by soliciting feedback regarding their experiences at FAMU and their future plans. 4.FAMU Alumni Survey The FAMU Alumni Survey is designed to collect information on Alumni employment status, continued education, and satisfaction with the University.

9. Goals of the Study To Discuss and share assessment findings on students’ communication and critical thinking skills; To Compare strengths and weaknesses observed from the results of NSSE, MAPP, Exit Survey, and Alumni Survey; To increase credibility of findings by a check of general consistency of the results for internal and external reporting; To provide solid foundation for decision making;

10. CONCEPTUAL FOUNDATION (1) Increasing need of credible and realistic assessment of student learning outcome; Cartwright et al. (2007), “outcomes assessment is the process of collecting information that will tell an organization whether the services, activities, or experiences it offers are having the desired impact on those who partake in them” (p. 1).

11. CONCEPTUAL FOUNDATION (2) Anderson (2004) argues that “as expectations for public expenditures rise, so do public expectations for accountability of those expenditures” (p.17); Obtaining accountability through internal and external assessment tools; Combination of qualitative and quantitative methods.

12. RESEARCH QUESTIONS Are the results on students’ communication and critical thinking skills consistent across the four instruments (NSSE, MAPP, Exit Survey, and Alumni Survey)? How does students’ performance in communication and critical thinking skills on the NSSE test compare to that of peer institutions? How does students’ performance in communication and critical thinking skills on the MAPP test compare to the national average? What improvements are warranted by the findings?

13. Formulate statements of outcomes or objectives aligned to the institutional mission/goals; Ascertain criteria for success; Measure student/service performance using direct and indirect methods; Observe and analyzing results for congruence between expected and actual outcomes; Use the results to effect improvement of instructional programs and administrative and educational support services; and Strengthen programs and services by continuously evaluating, planning, allocating resources and implementing new approaches to ensure congruence between expected and actual outcomes. FAMOUS ASSESSMENT APPROACH

14. METHODOLOGY(1) Target Population and Instruments Data collected from two groups of participants: First year students Seniors or graduates Participants per Instrument: NSSE: First year and Seniors MAPP: First year (freshmen) Exit Survey: graduating students (all students who applied for graduation) Alumni Survey: Graduates Where applicable, results from each instrument were compared to the others;

15. METHODOLOGY (2) Data Analysis NSSE survey Data Results of FAMU first year (FY) and senior (SR) students were compared to selected Peer Institutions; Examined how results compared to the MAPP results; MAPP test Data Mean difference tests between FAMU students’ results of 2006 and 2007; Mean difference test between FAMU student results and the national average; Examined how results compared to the NSSE results;

16. METHODOLOGY (3) Data Analysis Exit Survey Data Data on communication and critical thinking skills were summarized for each semester and year for the period of 2005 to 2007 by combining percentages of those who agree and strongly agree to have acquired these skills; Examined how results compared to the NSSE and Alumni Survey results; Alumni Survey Data The data were summarized to present percentages of those who agree and strongly agree to have acquired desired skills in communication and critical thinking as a result of their experience at the University; Examined how results compared to the NSSE and Exit Surveys results;

17. RESULTS (1)

18. RESULTS (2)

19. QUESTION 1Are the results on students’ communication and critical thinking skills consistent across the four instruments (NSSE, MAPP, Exit Survey, and Alumni Survey)? Communication First year students: NSSE results (37%) indicate higher ratings on communication skills of FAMU students’ than the MAPP results (25%), slightly different; Senior/graduates: NSSE results (44%) are not consistent with both those of the Exit (92%) and Alumni Survey (91%); Results of Exit (92%) and Alumni Survey (91%) are consistent;

20. QUESTION 1Are the results on students’ communication and critical thinking skills consistent across the four instruments (NSSE, MAPP, Exit Survey, and Alumni Survey)? Critical Thinking First year students: NSSE results (61%) indicate much more higher ratings on critical thinking skills of FAMU students’ than the MAPP results (6%), slightly different; Senior/graduates: NSSE results (74%) are lower than both those of the Exit (94%) and Alumni Survey (96%); Results of Exit (94%) and Alumni Survey (96%) are consistent;

21. QUESTION 2How does students’ performance in communication and critical thinking skills on the NSSE test compare to that of peer institutions? Communication Skills First year students: Percent of student performing medium or high: FAMU (37%), Peer Institutions (38%) - similar; FAMU students performed significantly higher than peers on about 17% of items measuring communication skills; Senior/graduates: Percent of student performing medium or high: FAMU (44%), Peer Institutions (42%) - similar; FAMU students performed significantly higher than peers on about 67% of items measuring communication skills;

22. QUESTION 2How does students’ performance in communication and critical thinking skills on the NSSE test compare to that of peer institutions? Critical Thinking Skills First year students: Percent of student performing medium or high: FAMU (61%), Peer Institutions (59%) - similar; FAMU students did not performed significantly higher than peers on any items measuring critical thinking skills; Senior/graduates: Percent of student performing medium or high: FAMU (44%), Peer Institutions (42%) - similar; FAMU students performed significantly higher than peers on about 67% of items measuring critical thinking skills;

23. QUESTION 3How does students’ performance in communication and critical thinking skills on the MAPP test compare to the national average? In Table 3 the mean scores in reading and writing are significantly lower (at α =0.05) than the national average. Table 3: National Average Compared to FAMU Freshmen MAPP Results - 2007 Sample Size: FAMU = 773, National = 29,592, The star (*) sign indicate that the difference between National and FAMU score is significant at α = 0.025 (0.05/2) error rate level since the reported p-values are for a tow-tailed test

24. QUESTION 3How does students’ performance in communication and critical thinking skills on the MAPP test compare to the national average? FAMU Freshmen MAPP Test Results for 2006 and 2007 Compared to the National MAPP Mean Score

25. QUESTION 4What improvements are warranted by the findings? Institution’s Use of Results Based on the unacceptable results of non proficiency of our students in critical thinking skills, the University decided to make “Enhancing Performance in Critical Thinking Skills” the topic of its Quality Enhancement Plan (QEP). Recently adopted an policy requiring students’ participation in tests and surveys used by the institution to assess student learning outcomes;

26. QUESTION 4What improvements are warranted by the findings? Impact Emergence of a dynamic process that involves key assessment players; Awareness about improving students’ critical thinking skills and communication skills. All programs have identified and implemented critical thinking and communication learning outcomes which is a requirement of the Florida Board of Governors under its Academic Learning Compacts (ALC).

27. SUMMARY OF RESULTS (1) The findings indicate low performance of FAMU students in critical thinking skills; MAPP results from 2005-2006 and 2006-2007, about 93 % percent of freshmen are not proficient in critical thinking; The NSSE, Exit, and Alumni surveys results on critical thinking and communication skills indicate similarities in the findings;

28. SUMMARY OF RESULTS (2) The results also indicate a sizable difference between the three indirect measures and the direct measure, the MAPP test; Results from the direct instrument, the MAPP test, did not show any difference in the critical thinking skills between freshmen of 2006 and those of 2007;

29. SUMMARY OF RESULTS (3) Results from the indirect instruments show an overall high performance of both first year and senior students in critical thinking skills while the results from the direct instrument indicate that about 93 % percent of freshmen are not proficient in critical thinking;

30. Success Factors Participatory approach of the assessment process; All degree programs implementing assessment plans, document them, and use assessment results to improve outcomes; Direct involvement of students, faculty, and the administration; Communications strategies about the issues adopted by the OUA and the Institutional Level assessment committees, ILAC and GEAC, through assessment roundtables, specific assessment training sessions, and during other forums and conferences;

31. Recommendations COMMUNICATION SKILLS Require students to complete communication courses and sequence; Require students to seek tutoring; Include more in-class comprehension specific reading assignments; Provide more instruction and practice in writing critical analyses; Set and maintain maximum enrollment in communication courses to 22; Establish university funded resource lab; Require lab work as partial fulfillment of composition classes; Re-establish a university funded reading lab;

32. Recommendations CRITICAL THINKING SKILLS Develop and use a separate internal instrument that directly measures critical thinking skills. Require students to do more critical reading exercises; Create an institutional climate that promotes student inquiry Provide more opportunities for debate and discussion in class Establish a multidisciplinary laboratory

33. QUESTIONS? THANK YOU