Administrator Beware: Checking the Validity of Published Subscales

1. Administrator Beware: Checking the Validity of Published Subscales Nancy D. Floyd University of South Carolina South Carolina Association for Institutional Research Columbia, SC March 5, 2012

2. Holland’s theory of vocational personality: six types • Realistic • Investigative • Artistic • Social • Enterprising • Conventional

3. Academic DisciplinesSmart, Feldman & Ethington (2000) • College freshmen completed CIRP Freshman Survey in 1986 • Followed up with Senior Survey in 1990 • Holland type estimated using subscales comprised of items on CIRP Freshman Survey • Consistency of typing with both freshman major and graduating major studied, as well as changes during four years

4. CIRP sections used in Holland type estimation (from 1996 Freshman Survey) Rate yourself . . compared to the average person your age (1=Lowest 10%; 5=Highest 10%): • Academic ability • Artistic ability • Competitiveness • Cooperativeness • Creativity • Drive to achieve • Emotional health • Leadership ability • Mathematical ability • Physical health • Popularity • Public speaking ability • Self-confidence (intellectual) • Self-confidence (social) • Self-understanding • Spirituality • Understanding of others • Writing ability Please indicate the importance to you personally of . . : (1=Not Important; 4=Essential) • Becoming accomplished in one of the performing arts (acting, dancing, etc.) • Becoming an authority in my field • Obtaining recognition from my colleagues for contributions to my special field • Influencing the political structure • Influencing social values • Raising a family • Having administrative responsibility for the work of others • Being very well off financially • Helping others who are in difficulty • Making a theoretical contribution to science • Writing original works (poems, novels, short stories, etc.) • Creating artistic work (painting, sculpture, decorating, etc.) • Becoming successful in a business of my own • Becoming involved in programs to clean up the environment • Developing a meaningful philosophy of life • Participating in a community action program • Helping to promote racial understanding • Keeping up to date with political affairs • Becoming a community leader

5. Smart & Associates’ (2000) Holland estimation • Item scores converted to T(50,10) scores • Items comprising a subscale are averaged • Student’s highest average is their primary Holland type (Smart, Feldman & Ethington 2000) • Subscales not constructed for Realistic and Conventional subscales; Smart et al had removed these from analysis

6. Sample Comparison Smart, Feldman & Ethington (2000) USC 1996 incoming freshmen N=2438 57.9 percent female 26.4 percent nonwhite 40.7 percent undeclared majors • N=2309 • 52.7 percent female

7. Models Tested

8. Proposed Investigative Subscale

9. Proposed Artistic Subscale

10. Proposed Social Subscale

11. Proposed Enterprising Subscale

12. Implications • If factor structure didn’t hold for one institution that should have been similar to development sample for Smart, Feldman & Ethington, have to question validity of original subscales. • If original subscales are invalid, one of two things: our sample is not representative of research sample, or the factors were not produced correctly or tested enough. • If the sample is not representative, why am I buying CIRP and NSSE like it is? • If the factors were not produced correctly or tested enough, can I have faith in other factor subscales? • If I can’t get original response data back from a survey developer, I can’t check factor structure to see if it holds.

13. Questions? nfloyd@mailbox.sc.edu