Division of student development Developing Surveys (Magis@Work) Divisional Assessment Committee in partnership with the Professional Development Committee February 1st, 2018
Goals for the Workshop • Discuss general tips for survey design • Consider the assessment of student learning and student success • Contribute to the development of a bank of questions to use within DSD • Receive practical feedback on survey instruments
WHY: Why Administer a Survey? • What you are assessing should be determined by your goals • and learning outcomes for the program / service. • Think ahead: once you get the data, what will you do with it? • Seek out information on which you can act. • What do you hope to be able to communicate with the data? • Don’t ask questions if you already have the information. • Determine whether a survey is the best format for the data you • are gathering.
WHO: Asking the Right People • Consider the difference between population and sample. • Consider different sampling strategies (such as random, stratified, or convenience sampling). • Consider how representative the sample may be. • Beyond thinking about the people who will receive your survey, consider: • whether there are people with whom you may want to collaborate when administering the survey. • who can provide assistance (the Campus Labs • team can provide feedback on content and • structure)
WHAT TO ASK, and HOW TO ASK IT: Steps in Survey Design • Outline topics and draft items • Choose response formats • Write and edit items • Determine the order • Plan the administration • Pilot and revise as needed • Use and share the findings
Steps in Survey Design: Choosing Response Formats • Quantitative or Qualitative • Broad picture of what is happening vs. why / how it is happening • Open-ended or Closed • Single Response or Multiple Choice (“Select All that Apply”… or “Select Top 3 Options”) • Ranking • Likert Scale • Define the options • Neutral point or forced choice? • Number of levels
Steps in Survey Design: Write & Edit Items • Use clear language, and ask one thing at a time. • Bloom’s Taxonomy • Avoid loaded or leading questions (no assumptions). • Provide options for all (Never, N/A). • Ask for a useful level of detail: • Categories versus specifics (example: majors, or numbers versus a range) • Yes / No & Please Explain • Consider the number of questions • Attrition increases at 22 questions or 13 minutes.
Steps in Survey Design: Determine the Order • Objective before subjective (Have you participated in PROGRAM? How would you rate PROGRAM)? • Begin with less threatening questions. • Important questions near the start. • Save open-ended questions for the end. • Place questions about demographics at the end.
Steps in Survey Design: Plan and Pilot • Outline topics and draft items • Choose response formats • Write and edit items • Determine the order • Plan the administration • Consider time of year, • communication, incentives • Pilot and reviseas needed • Use and share the findings
Steps in Survey Design: Interpret, Use, & Share Findings • Qualitative: • Themes (coding) • Nominal / named categories (in-state or out-of-state). Present frequencies or percentages (tables, charts, graphs). • Quantitative: • Ordinal: order/ranking, but not meaningful distance between points (class year). Present frequencies or percentages. • Interval: order with equal fixed distances between data points, but no absolute zero (SAT or ACT scores). • Ratio/scale: meaningful zero point (family contribution in $)
Interpret, Use, & Share Findings • Presenting qualitative findings • Simple descriptive statistics • Central tendency (most common, middle, average value) • Dispersion (variation or spread in relation to central tendency) • Measures of relationship (correlation, regression) • Measures of comparison (t-tests, ANOVA, MANOVA) • Share findings with staff when making decisions, and with students so they know their voice matters.
Plan 2020 • Building a more just, humane, and sustainable world • Plan 2020 charges us to analyze and support these student success outcomes: • Retention • Timely graduation • Progression • Career readiness • Post-graduation success
Career Readiness & Post-Graduation Success • Employers look for candidates who excel in areas such as: teamwork, problem-solving, communication, work ethic, and leadership (NACE, 2016). • The likelihood that alumni will be engaged at work is: (a) 2.2x higher if mentors encouraged them to pursue their goals, and (b) 1.8x higher if they were active in co-curricular activities while in university (Gallup, 2014). • The likelihood that alumni are emotionally attached to their alma mater is: (a) 4.1x higher if mentors encouraged them to pursue their goals, and (b) 2.7x higher if they were active in co-curricular activities while in university (Gallup, 2014).
Student Success is Associated with: • Building formal and informal relationships with staff and faculty • High-impact educational opportunities that foster global learning, exploration of social identities, and social-perspective-taking (Kuh, 2008)* • Academic self-efficacy (belief in their academic abilities) • Campus engagement (this may be the most important condition for retention) • On-campus employment / Federal Work Study positions (financial assistance, resources, and relationships built) • A sense of belonging on campus (campus climate affects this) • *All other points are from Tinto, 2012
Small Group Activity • Get into groups of 2 to 3 people and introduce • yourselves. • In small groups, please exchange copies of • your surveys so each person is reviewing a • different survey and providing feedback. You • may want to use the handout to help. • with main points from this presentation. • Then, please share feedback about each • survey with small group members.
Resources and Support • Creation of bank of questions for DSD assessment • Resources • Links on DSD Assessment Website • Campus Labs • Divisional Assessment Committee • Free online course on assessment in student affairs • Day-long Assessment Conference offered by Campus Labs at LSC the week of May 21st (more details to come soon)
References Assessment Certificate Program Campus Labs: https://baselinesupport.campuslabs.com/hc/en-us/articles/204840519-Assessment- 101-The-ABCs-of-Survey-Design Gallup (2014). Great jobs, great lives: The 2014 Gallup-Purdue Index Report. Kuh, George D. (2008). High-impact educational practices: What they are, who has access to them, and why they matter. Washington, D.C.: AAC&U. NACE: http://www.naceweb.org/talent-acquisition/candidate-selection/the-attributes-employers-seek- on-a-candidates-resume/ Schuh, J. H. & Associates (2009). Assessment methods for student affairs. San Francisco: Jossey- Bass. Tinto, V. (2012). Completing college: Rethinking institutional Action. Chicago: The University of Chicago Press.