Download
1 / 23
240 likes | 378 Vues
Data Collection & Technology. EDUC 894 Week 6. Plan for Today. Next Week No class, consultations available E-mail me by next Monday to set up a time Reading handouts Instrument Focus: WebSurveys What NOT to do: An Example How many problems can you find? What To Do : Principles
E N D
Data Collection & Technology EDUC 894 Week 6
Plan for Today • Next Week • No class, consultations available • E-mail me by next Monday to set up a time • Reading handouts • Instrument Focus: WebSurveys • What NOT to do: An Example • How many problems can you find? • What To Do: Principles • Designing a Valid & Reliable Survey • The Wario Survey • Working with WebSurvey data • Instrument Focus: Eye-tracking & Brainwaves • Field trip to the Engrammetron -----------------------------Break----------------------------- • Group Work • Honing down your instruments
Survey Design • (Most) surveys are conducted from an objectivist perspective • Writing a survey is all about making decisions: • Recognizing when a choice is being made • Making choices that will create a reliableinstrument that will help you make validinferencesto answer your research questions • It’s (usually) not about making the “best” decision in some abstract sense, but making a choice that will be most useful for your purposes
Limitations of Surveys • Self-report data • Notoriously unreliable • Indicates only what we think, not reality • Data at one point in time • Limited Sample • Retrospective questions rely on memory • The Devil is in the Details • Question form, wording, and answer choices can inadvertently dictate responses • Sampling • Who responds isn’t random Image Source: http://www.nynpa.com/images/NYNPA0081.jpg
Survey Decisions • Topics Covered • Number & Focus of Items per Topic • Item Sequence & Presentation • Item Type / Format • Item Wording • Always think about what you will do with the answers you get – i.e. how will you analyze the data?
What NOT to do • Take the survey posted here (link sent via email) https://my.sfu.ca/cgi-bin/WebObjects/WebSurvey.woa/wa/survey?17437110 • How many problems in each category can you find? • Topics Covered • Number & Focus of Items per Topic • Item Sequence & Presentation • Item Type / Format • Item Wording
Survey Decisions Topics Covered • Remember the idea of content-related validity? • Goal:Determine the most popular animals • Assumed “like” indicates popularity • What pet you had as a child may or may not relate • Are the plants in you garden a better indicator? • Feelings about gardening don’t give relevant info • Caveat: Questions to lay a foundation • Define your terms • Are you concerned with popularity of animals (in general) or only pets? • Can ask a similar question about plants / garden plants • What about ability self-esteem (in general) or self-esteem as related to academic work? • How would this affect your survey design?
Survey Decisions Topics Covered Survey Academic Self Esteem Social Athletic
Survey Decisions Number & Focus of Items per Topic • Number: Do you have enough questions on each topic to give a reliable measure? • Three questions is usually the absolute minimum given for a scale • Focus: Do questions give you info you can actually triangulate? • How will you compare the names of plants reported to be in their garden with Likert scales given for specific plants? • What will you do with the question about liking “trees without flowers”?
Survey DecisionsItem Sequence & Presentation • Question sets and introductions • Broad to narrow progression • How satisfied are you with service in the food court? • Do servers in the food court smile when taking your order? • Will you group or distribute similar content items? • Putting similar items together will help learners think about what you want them to • But, if you have multiple items try and get convergence on a construct, putting them near each other will lead to a false reliability • Pagination • No one likes to scroll forever • No one likes unnecessary clicking • The “required item” dilemma
Survey DecisionsItem Type / Format • Open versus Closed Ended Questions • It all depends on what you are looking for, but from an objectivist perspective: “Use close ended questions where at all possible, if you can limit the possible answers it can really speed up the data crunching when you are finished” • You may need open-ended questions when you don’t know the range of possible answers • But then you will have to go through and code these answers for themes • Thus, don’t ask an open-question with a limited number of possible answers or where answers range predictably • “Do you like to garden?” • “How much do you like dogs?”
Survey DecisionsOpen Ended Question Guidelines • Be specific • What did you think of class? BAD • What 3 things did you like best about class? GOOD • What 3 things did you like best about the class content? BETTER • Leave people enough space to write • A philosophy of life, love and happiness is not 64 characters or less! • Don’t ask too many open-ended questions • Your participants will tire and it will show in their responses • G. W. Bush example
Survey DecisionsClosed Ended Question Guidelines • Question Types 1 • Yes/No • Lists • Single choice (“best answer”, radio button) • Multi (“all that apply”, checkbox) • Can do a series of these in a grid, but dangers of confusion and auto-pilot
Survey DecisionsClosed Ended Question Guidelines • Question Types 2 • Rating Scales • Ordinal • E.g. Rank the following from best to worst • Likert • E.g. Levels of agreement • Likert-style • E.g. Levels of approval, satisfaction etc. • Frequency • E.g. Often, sometimes, almost never • Bipolar • E.g. Outgoing to introverted
Survey DecisionsClosed Ended Question Guidelines • Focus on Rating Scales • Graphic vs. Finite Choice • Think about the following: • Does your scale goes to the extremes? • Does your scale have enough choices? • Participants are less likely to choose the anchors (n-2) • Does your scale imply interval spacing? • Will your scale have a “neutral” option? Mark an X on the line to show your level of agreement Disagree Agree Indicate your level of agreement by circling your choice Strongly Disagree Disagree Neutral Agree Strongly Agree
Survey DecisionsItem Wording 1 • Clear & Concise • What do you do when you get a headache?UNCLEAR • What headache medicine do you use? CLEARER • Concrete • Did you enjoy the book? ABSTRACT • Did you recommend the book to a friend? CONCRETE • One idea per question • Should BC continue its urban density and needle exchange programs? CONFOUNDED • Be careful when asking personal questions • Use categories to avoid emotional responses
Survey DecisionsItem Wording 2 • Avoid leading questions • Do you think Barak Obama will continue to be a successful candidate? • Do you think Barak Obama will continue to be plagued by challenges about his level of experience? • The challenge of reversed questions • Can be used to break up question sequence or to get at something in a different way, but big reliability issues • I do not like venus fly-traps EXPLICIT • Self-Esteem Survey: Sometimes I doubt myself IMPLICIT
Survey DecisionsThe Big Picture • Every time you have to make a decision, always think about 3 things • How will your decision affect participant responses? • How will your decision affect your analysis? • What does your decision imply about the underlying construct?
