Every thing you want to know about surveys…

1. Every thing you want to know about surveys… And aren't afraid to ask! Carl Berger January 30, 2001 All sources are available, look on the last slide.

2. But it’s really not about surveys… (Gratuitous comments and/or insights…on the bottom of some slides) • It’s about helping Presidents, Provosts, CIO's, Designers and Creators to make decisions! But… you've got to know the territory!

3. Why? • What do they know about …? • What do the need to do with ….? • What are they bringing to ….? • ?

4. But how do you know? • Guess • Traditional Sources • Conventional Wisdom • Anecdotes • Or… • You could ask them!

5. Even more important… • Helping decision makers • Those who have little time to develop deep knowledge • Present in ways that decisions seem to pop up • Or don’t! Lots of time its realizing that this data snipit can't help a decision!

6. Topics • Asking the right question • Asking the question right! • Getting a great response (What is a great response?) • Decision Graphics • Hidden meanings • An example and some surprising? results

7. Advantages to asking (survey) • Real data, almost always dumps myths • Understand all groups, early adopters, early majority, late majority, luddites • Faculty, Students, Administrators… The real pay-off is that they understand that they are part of the process helping to make a better product!

8. Disadvantages of asking • Expensive • Time consuming • Will they understand it when were done? • And worse yet… • you may find out that which you don’t want to know! About two years ago we invested heavily in NetG. (an on-line training program) Good idea?… stay tuned…

9. Take the plunge, needs, wants and reality • Enough to give credence Representative • Need to get luddites • Keep cost down • Understandable • And… • Getting the right questions but even more… • Getting the questions right and.. • Getting the right analysis and presentation!

10. On the shoulders of … Flashlight http://www.tltgroup.org/programs/flashlight.html UCLA http://www.uncwil.edu/oir/faculty_folder/ucla_survey_99/survey_results.htm Michigan http://sitemaker.med.umich.edu/cberger/reports Berkeley, Cal State System and Others on the way, Stanford, Penn State, Minnesota but best… YOU! Beg, borrow or steal good items and question styles. But give credit!

11. The 12 Step Program to Success • 1. Select sponsors and participants • 2. Develop areas and questions with them • 3. Initial survey development with them • 4. Test with small group, revise • 5. Contact and give survey • 6. Follow-up on non respondents • 7. Data entry • 8. Analysis • 9. Presentation • 10. Distribution • 11. Feedback • 12. They want next version

12. Asking the right question • Don't jump too quickly to a survey: • The Ehrmann technique (Open focus groups and listen, listen, listen) • A majority of three (If you hear the same from three folks, unsolicited, then it's a survey item) • Now build a survey • Small sample trial (Try out some survey types) • Single page (A few 1-pagers with several groups) • Check, check and check again for interpretation and errors • A big survey infrequently to prevent "surveyed to death” • Several small surveys for formative evaluation

13. Asking the question right • Don’t ask “Do you use…?” Ask “How often do you use…” • Don’t ask “Do you use either a or b?” Ask “Check all that apply” • Don’t ask “Would you like…” • Offer ‘Use it now’, ‘Would like very much’, ‘Would like’, ‘Would use if provided’, ‘Don’t want to use’, ‘Don’t know what your talking about’ Make questions do multiple duty. "How often" also tells yes, no, + lots more

14. Underlying constructs • Think of an underlying scale • Try to avoid yes or no • Use common or natural intervals • Not…Often…Frequently…sometimes…once…never • But…1/day…1/week…1/month…1/term…1/yr… never With a little creativity you can create a log scale from the last one!

15. Non respondents • 1. Make a non respondent list. • 2. Compare against a representative list. We used our LDAP. • 3. Look for over or under represented groups. • 4. Over sample them in relation to the population. (make comparable) • 5. Then follow up as above. Send e-mail, campus mail, and finally call. • 6. Don't use phone or personal calls to get promises of response. • 7. Either carry out the survey over the phone or go to their office. • 8. Stop when you have a 'good feeling.’ • 9. Try long enough to get a significant group or collapse trying. • 10. Finally, include, successful non respondents to check out some unusual claims.

16. Data entry • Enter for analysis and presentation • Use different codes for missing, not applicable, or filled out incorrectly • Look out for multiple missing data codes. • Try out a sample set analysis before it is too late. • Do a small amount (25%) of double entry to get reliability measures. Make all missing data codes 9999. Data entry persons like 9, 99, 999 But you then can't do a simple search and replace for missing data.

17. Data analysis • Look out if you average. Medians may be better. • Report stats sparingly. • But look for variation. • Look for out of range. (stats can do that) • Don’t be afraid of complex analysis (factor analysis, MDS, etc) Outlying data can really move averages. Also don't be afraid of advanced stats. New computer programs can help you visualize complex data. Try StatView for some real eye openers.

18. Presentation • Minimize tables for comparison • Use graphs. http://www.uncwil.edu/oir/faculty_folder/ucla_survey_99/survey_results.htm You could spend hours looking for conclusions…

19. Presentation Same data as a graph…(unmodified Microsoft graph…blehh…We'll fix this later.) Just converting it to a chart isn't more helpful…

20. Chart junk… False 3D Terrible background <-- Lousy Colors Poor Layout Worse… No way to see real differences These errors are caused by Accepting Excel defaults. With a little work the results--> are clean and clear Tufte would love this one!

21. Decision graphics • Making data clear with graphics and.. • Using graphics to help decision makers • Combines complex chart data • Uses visual design theory • Uses perception theory Decision Graphics started in the 80's to help parents understand Individualize Learning Programs for special education students.

22. From tables to a decision graphic (7 steps) Step one:… Start with a table of data Question 26 Use Tech for… What a table to try to figure out! But at least it is sorted by the total of Already use, would like very much, would like somewhat.

23. Question 26 Use Tech for… Step 2… Convert table to an Excel Graph I'll never know why MS builds starts with such terrible charts!

24. Step 3: Convert to a bar chart (set legend below the chart) Question 26 Use Tech for…

25. Step 4: Change to a stacked bar chart… Question 26 Use Tech for… Aha…now that sorting makes sense as you look along the dark blue bars.

26. Step 5: Expand bars (100% wide and less space between)… Question 26 Use Tech for…

27. Step 6:Change colors to flow from cool to warm Question 26 Use Tech for…

28. Step 7: What the heck.. Shade those colors… Question 26 Use Tech for… Bottom Line… Our faculty want course web pages but no distance teaching!

29. Remember the UNC data?(unmodified Microsoft graph)

30. With color metrics… Conclusion… Not much difference!

31. Ranking versus top three… • Ranking works with few choices. • Selecting top three will take care of ranking 4 choices • Selecting top three is easier to take • Results easier to analyze and display

32. Easier to take

33. But the results are revealing… What method do you like to use to learn technology? And guess which University just spent big bucks for on-line computer classes?

34. A little factor analysis Yep, we've got leading and second wave faculty plus the good old AV types

35. The U of Michigan case study • The 1999 Faculty Survey • Distributed in February 1999 • 1500 faculty, stratified random sample • 19 Schools and Colleges • 743 responses • Results to CIO in August 1999 • Released in to the public in March 2000

36. Survey Categories • Use • Resources • Support

37. Q29 Base Academic Unit Had to use a log scale. Med School rules! (We had one school with more responses than faculty, what a story!)

38. Questions 29-35Demographics…

39. Question 27Would you use the web for… Very little use now but… pent up demand for next wave use.

40. Q10 How often do you use… Surveyed just as our CourseTools was coming on line. Next survey???

41. Question 15 Concerns… Bottom Line… It's time, reliability and support!

42. Adding students <-Faculty1999 UM Faculty Survey Students -> 2000 UM Student Survey Not too different, second wave for both?

43. CIO and SACUA CIO Staff ITD Staff OIT Staff ISR 743 faculty members! Special thanks to: Kati Bauer Steven Burdick Jose Marie Griffiths Gary Gatien Karen Kost Nicole Kirgis Kathleen McClatchey Eric Rabkin Credits

44. Flashlight, part of the TLT group • Online database • In depth rather than broad • Excellent source • Part of a broad program • http://www.tltgroup.org/programs/flashlight.html

45. The Full MontyThe Michigan 1999 Faculty Survey and the 2000 Student Survey Form • Original blank survey (.doc) [Faculty and Student] • and…Initial results (.pdf) [Faculty] • but wait…there’s more…This presentation (.ppt) • are available at www.carat.umich.edu • Follow links to projects and scroll to faculty survey… • Or…if you want a special question answered carl.berger@umich.edu We can help decisions and improve learning and teaching!