Survey Construction, Validity, Reliability

1. Survey Construction, Validity, Reliability What to think about when creating a survey instrument.

2. Topics • What is a survey? What are the steps in a survey study? • How do I construct questions? • Validity • Reliability

3. What is a survey? What are the steps in a survey study? • “A survey is a system for collecting information from or about people to describe, compare, or explain their knowledge, attitudes, and behavior.” (Fink, 2003) • Steps: • 1. Define the objectives/goals* • 2. Design the study: population*, sample*, sample size*, timeline*, instrument construction • 3. Integrate validity and reliability into survey instrument development (including pilot testing) • A. Review, test, revise • B. Repeat as necessary • 4. After IRB approval: Administer the survey (internet, mail, mixed mode)* • 5. Data cleaning & management* • 6. Data analysis* • 7. Reporting results* *Not today 

4. Advantages and Disadvantages of Surveys Advantages Disadvantages Wording can bias response Impersonal Doesn’t get full story Low response rates (Consider the Tailored Design Method by Don Dillman) Self-selection bias Not generalizable • Anonymous • Inexpensive • Easy to compare and Analyze • Lots of people, lots of data • Use pre-existing instruments

5. Social Exchange • Establish Trust • Token of appreciation • Sponsorship of legitimate authority • Make task appear important • Invoke other exchange relationships • Increase Rewards • Reduce Social Costs • Avoid: subordinate language, embarrassment, inconvenience • Minimize requests for personal informaiton

6. Writing Questions • Match survey questions to research objectives, goals, research questions, or hypotheses!!! • Straight forward questions yield straight forward responses. • Avoid questions about which you are “just curious” • Include necessary demographics whenever possible (describe sample) • If you can use a previously developed survey instrument, do it! • Validity and Reliability already done • Get author permission, some cost money • Make sure it is accepted in the literature • Do NOT change it! (without the original author’s permission, but then you lose the previous validity and reliability)

7. Question Design • Physical format/appearance • Qualtrics is nice • Visual layout is clean • Order of questions • Questions about announced subject first! • Order should be logical • Group items that are similar • Personal and demographic questions at the end

8. Cautions • You get what you measure: Choose your questions carefully • Keep it simple: Be aware of the literacy level of your respondents • Shorter is better: Concise, easier to understand, easier to answer • Ranking is hard: And often misunderstood leading to invalid data • People don’t often read the instructions • Each concept gets its own question! (Beware of AND and OR) • Questions should be concrete and behavioral, not conceptual • Binary questions (yes/no) contain less information than ordinal questions (Hierarchy of information content) • Define terms before asking questions • Avoid jargon and “loaded” words

9. Question Types • Open vs. Closed • Levels of Measurement (Information) • Nominal (Sex, race/ethnicity) • Ordinal (Likert, Likert-type, unipolar ordinal) • Interval • Ratio

10. Ordinal questions • Ordinal • Unipolar • e.g. never, rarely, sometimes, often, always • 4 to 5 points • Bipolar • Likert and Likert-type • strongly disagree, disagree, somewhat disagree, neither agree nor disagree, somewhat agree, agree, strongly agree • Even vs. odd number of responses • Left-side bias • Continuous as ordinal • Down-coding age and income

11. Validity • Face Validity • Content Validity • Criterion Related Validity • Concurrent (correlated with a “gold standard”) • Predictive (ability to forecast) • Construct Validity • Convergent • Divergent

12. Face Validity • Generally untrained judges • Does the instrument appear to measure the construct? • Very weak evidence

13. Content Validity • Definition: “The extent to which an instrument adequately samples the research domain of interest when attempting to measure phenomena (Carmines and Zeller, 1979) • Begin with literature review to identify the entire domain of content • Develop items associated with the identified domain of content • Content Validity Index (Proportion of experts who agree item is relevant), Cohen’s Kappa • In mixed mode surveys, triangulate quantitative responses with opnen-ended responses

14. Criterion Related Validity • Compares new instrument against another instrument or predictor • Concurrent validity • “Gold Standard” • Known/published psychometric properties • Published and used by researchers in the field • Could choose test that measures “opposite” • Correlation coefficient • Predictive validity • Useful in predicting future behavior/events/attitudes/outcomes • Correlation coefficient

15. Construct Validity • Theoretical relationship of new instrument with other constructs or behaviors • New instrument correlates with other similar (not exact) measures (convergent validity) and does not correlate with others (divergent validity • New instrument discriminates one group from another • Not always quantifiable • Requires theory related to constructs

16. Reliability

17. Cronbach’s Alpha • Internal Consistency • Only appropriate when scale scores are developed • Only use items that are thought to measure a single construct • Prefer scales to have Cronbach’s alphas greater than .7 • If Cronbach’s alpha is greater than .9, are some items redundant? • Not generally appropriate for knowledge tests (unless they measure a single construct) • Not appropriate when scale scores are not calculated

18. Scaling and Scoring • Hypothesize which items form a scale • Perform a factor analysis • Eliminate items that crossload, fail to discriminate among respondents, strongly correlate with other items, are frequently misunderstood or left blank • Calculate Cronbach’s alpha, and alpha if item deleted • Interpretability is important! • What does a big number mean, what does a small number mean? • When you read the items do they make sense for a single construct

19. Other Considerations • Sample size requirements increase with: • Decreasing levels of information • Smaller effect size detection • Software for sample size and power analysis • Pilot testing is critical • Helps identify errors in the survey • Identifies potential design/redesign issues • Predicts potential problems you might encounter

20. End Notes • Consider applying for CTR-IN pilot grants! • http://www.isu.edu/healthsciences/ichr/ • Teri Peterson • Best way to contact is through email: peteteri@isu.edu • Phone is less reliable (X5333 or X4861)