Using National Studies of Student Engagement to Support Institutional Change Nathan Marti, CCSSE

1. Using National Studies of Student Engagement to Support Institutional Change Nathan Marti, CCSSE Todd Chamberlain, NSSE FAIR Conference June 23, 2004

2. What Really Matters in College:Student Engagement “The research is unequivocal: students who are actively involved in both academic and out-of-class activities gain more from the college experience than those who are not so involved.” Ernest T. Pascarella & Patrick T. Terenzini, How College Affects Students

3. Lessons from the Research • What matters most to desired outcomes is what students do, not who they are • A key factor for student learning is the quality of effort students devote to educationally purposeful activities

4. National Survey of Student Engagement(pronounced “nessie”)Community College Survey of Student Engagement(pronounced “sessie”) College student surveys that assess the extent to which students engage in educational practices associated with high levels of learning and development

5. Penetrating NSSE/CCSSE Findings • Overview of Reports • Start Broad, then Dig Deeper • Identify Significant Items • Focus on What is Important to YOUR Institution’s Priorities

6. Customized Institutional Report • Respondent Characteristics • National benchmarks • Institutional data • Means and frequencies • Subpopulations • Comparisons by Carnegie/size, national

7. Respondent Characteristics

8. Respondent Characteristics:Does This Represent Your Campus? • Response Rate • Sampling Error • Student Characteristics • Gender • Race/Ethnicity • Class Level • Enrollment Status • Comparisons by Carnegie/size, national and consortium when relevant

9. Benchmarks: High Level Views of the Data NSSE/CCSSE developed five indicators, or benchmarks, to represent the multi-dimensional nature of student engagement at the institutional, sector and national levels Supportive Campus Environment/ Support for Learners Enriching Educational Experiences/ Student Effort Level of Academic Challenge Active & Collaborative Learning Student- Faculty Interaction

10. Benchmark Report

11. Mean Summary ReportFrequencyDistributionReport

12. Means & Frequency Reports • Look at the Items that Make Up Each Benchmark • Which Items Have Significantly Higher/Lower Responses than Comparison Groups? • Practical Significance: Identify Standardized Effect Sizes greater than .2

13. Activity: Highlight Key Findings • Review Benchmark and Means Summary Report • Small groups by benchmark • Examine a benchmark • Which items differ?

14. Digging Deeper: Using the Data • Identify important subpopulations • Determine the outcome that matters • Determine the factors that influence the outcome

15. Principles for Data-Driven Learning-Centered Change

16. 1. Get the ideas right • Focus on a real problem (e.g., persistence, raising expectations, success in major field courses) • Concentrate on effective educational practices

17. Organizational culture valuing High expectations Respect for diverse talents Emphasis on early years of study Characteristics of Educationally Effective Colleges

18. Curriculum Coherence in learning Synthesizing experiences Integrating education and experience Ongoing practice of learned skills Characteristics of Educationally Effective Colleges

19. Instruction Active learning Assessment and feedback Collaboration Adequate time on task Out-of-class contact with faculty Characteristics of Educationally Effective Colleges

20. 2. Get grass roots buy-in • Leaders endorse, but don’t dictate • Structures not (nearly) as important as relationships • Validate pockets of quality • The 10% rule

21. 2. Get grass roots buy-in Examples • Ask deans about their concerns • Focus groups • Get students “engaged” in the improvement effort (Illinois State, Oregon State) • Faculty version of NSSE survey

22. 3. Keep the stakes and volume low • Avoid “winners” & “losers” • Suspend disbelief • Denial management • Go public later than sooner

23. 4. It’s the culture (stupid) • Culture is (almost) always (at least) part of the problem • Focus on “reculturing” and “revisioning” • Use familiar (or at least understandable) language

24. 5. Think and act systemically • Link innovations and change efforts from different parts of the campus (e.g., Greater Expectations, Gen Ed reform, SOTL, NSSE, service learning, diversity)

25. Presenting NSSE/CCSSE Findings • Potential Audiences? • Consider Audiences Before Administration—It May Influence Decisions • Internal Use & External Reporting • Be Strategic

26. Internal Audiences Source: NSSE 2003 Report Card

27. External Audiences Source: NSSE 2003 Report Card

28. Ways to Communicate Results • Customize Report to Reach Campus Audiences • Organize NSSE data by colleges, depts, programs • Focus on items of interest to stakeholder and communicate understandable percentages • Use data to start discussions • Use data to correct misunderstandings • Assemble and distribute detailed information on method and credibility of survey

29. Activity: Connecting Findings with Relevant Audiences • What was the most interesting finding? • Who needs to know about this? • How can you let them know?

30. National Context Putting Your Results in Context: • How good is good enough? • What would it mean to have 80% of your students indicate that they are satisfied with the college?

31. Sampling Methods Sampling Consistency Across Colleges: • CCSSE takes a stratified random sample of courses • NSSE takes random samples of first- and fourth-year students • Results are representative of the populations from which they are drawn • Over-sampling for targeted populations

32. Comparing Your Results Sampling Consistency Across Colleges: • Because results are comparable across institutions, institutions can situate their results • Anything on the survey can potentially be benchmarked

33. Benchmarking Two Approaches: • Normative - compares your students’ responses to those of students at other colleges and universities. • Criterion - compares your school’s performance against a predetermined value or level appropriate for your students, given your institutional mission, size, curricular offerings, funding, and so forth.

34. Criterion Referenced Approach • Most valued activities Ask faculty & staff what is most valued in institution, department, then present relevant data • Eliminate “Nevers” Work on reducing or eliminating reports by students of never doing specific engagement activities

35. Assessment Purposes • Accountability • Improvement

36. Activity: Identify Appropriate Benchmarks • Internal Differences: How do part- and full-time students differ? • External Examples: How does your institution differ from similar colleges?

37. Principles for Good Practice in Undergraduate Education(Chickering & Gamson, 1987) • Student-faculty contact • Active learning • Prompt feedback • Time on task • High expectations • Respect for diverse learning styles • Cooperation among students

38. Lessons from the Research • What matters most is what students do, not who they are • A key factor is the quality of effort students expend • Educationally effective institutions channel student energy toward the right activities

39. Principles For Promoting Student Engagement • Intentionality - Not leaving serendipity to chance • Alignment (mission, curriculum, student experiences) • Collaboration (pull many levers) • Assessment and feedback to guide/document improvement

40. How is Engagement Measured? • Switch benchmarks • How would you measure this construct?

41. Psychometric Findings: Reliability • Both instruments have been shown to be reliable and valid • Good construct reliability: benchmarks items measure the same construct • Structural equation models have been used to demonstrate that there is strong consistency across subpopulations (gender & part- v. full-time) and across years

42. Psychometric Findings: Validity

43. Psychographic Data • NSSE data used to form clusters of students • http://mypage.iu.edu/~chuzhao/air03dpi.pdf • Psychographic data consistently was more effective at predicting outcomes than demographic data

44. The Florida Opportunity • How does engagement relate to outcome? • Persistence • Goal Attainment

45. Activity: Expectations and Desires in Student Engagement • How many students responded often or very often? • What would you like to see? • How did they respond? • How can you increase engagement?

46. Questions and Suggestions • Questions for NSSE and CCSSE • Suggestions for NSSE and CCSSE