1. Pilot Study of the CAAP Critical Thinking Test April 27, 2005 Lanette Raymond Research Associate, Suffolk County Community College

2. CAAP Test Description • 32 item multiple choice test • Designed for use with college students • Administered within a single class meeting, • Appeared relatively credible in an in-class administration protocol • Provided documentation of reliability and validity across community college populations

3. CAAP Subscore Customization • CAAP Critical Thinking test contents closely match the SUNY CT learning objectives • Variance in the way the results are reported • ACT developed a customized report for these sub-scores showing normative comparisons against ACT national community college data. • ACT provided the student data files to SCCC, for further analysis of this data

4. Administration and Sample • Fall, 2004 • Administered in-class to 154 SCCC students in 7 general education courses • Predominately white (77%) • Traditional age (60% 20 years old or younger, 25% between 21 and 25 years old) • 50% male, 50% female • Mostly sophomore status (46%) • Fulltime enrollment (85%)

5. Student Motivation • No motivational tactics were employed • CAAP-CT instrument included an item that addressed students' self-reported motivation levels • One-third of students (n = 52) did not respond to the motivation item • 5 students indicated that they “gave no effort” (n = 1) or “gave little effort” (n = 4) to the assessment test.

6. Student Motivation • Lower motivation results in less optimal performance • Less motivated students’ scoresare less reliable and less valid • Reporting sample is based on data from the 97 students who reported moderate to best effort • The reliability coefficient (calculated with the data from the original 154 tests) for objective 1 (26 items) is within acceptable range (alpha = .80) • Due to the small number of items (6 items) contributing to objective 2, its reliability coefficient is much lower (alpha = .49).

9. Results • Confirmatory factor analysis substantiates the utility of the CAAP-CT test as a measure of 2 separate but related sets of critical thinking skills based on the 2 GEAR learning objectives

10. 2-factor model of Critical Thinking based on the GEAR Objectives

11. Results • All of the items loaded well onto their respective factors, with item 1 being only slightly below 1.96 (at 1.81). • The model shows an excellent fit to the data (χ2(463) = 466, p = .46, CFI = .94), providing additional context validity to the assessment.

12. ** Data for item 20 should be listed under objective 2

13. Standards Does not meet standard 59% or less Approaches standard 60% - 69% Meets standard 70% - 79% Exceeds standard 80% or more

14. Standards