Research Methods & Design in Psychology

1. Research Methods & Design in Psychology Lecture 2 Survey Design 2 Lecturer: James Neill

2. Overview • Survey construction - nuts & nolts • Sampling • Ethics • Levels of measurement • Measurement error

3. What is a Survey? • A standardised stimulus • A measuring instrument • A way of converting fuzzy psychological stuff into hard data for analysis

4. Survey Construction – Nuts & Bolts • Constructing questions • Modes of response • Response formats <-> LOM • Measurement error • Survey formatting

5. Constructing questions • Define target constructs • Check related research & questionnaires • Draft items (aim to have multiple indicators) • Pre-test & revise

6. When drafting questions aim to: • Focus directly on topic/issue • Be clear • Be brief • Avoid big words • Use simple and correct grammar

7. Bias in questions • Inapplicable • Over-demanding • Ambiguous • Double negatives • Double-barrelled • Leading • Loaded

8. Bias in responding • Social desirability • Acquiescence or Yea- and Nay-saying • Self-serving bias • Order effects

9. Modes of Survey Administration Interview • high demand characteristics • can elicit more information Questionnaire • lower demand characteristics • information may be less rich

10. Objective vs. subjective Objective: How times during 2000 did you visit a G.P.? Subjective: Think about the visits you made to a G.P. during 2000. How well did you understand the medical advice you received?perfectly very well reasonably poorly not at all

11. Open-ended vs. close-ended Open-ended • rich information can be gathered • useful for descriptive, exploratory work • difficult and subjective to analyse, • time consuming Close-ended • important information may be lost forever • useful for hypothesis testing • easy and objective to analyse • time efficient

12. Open-ended questions - Examples What are the main issues you are currently facing in your life? How many hours did you spend studying this week? _________

13. Close-ended questions – Example 1 What are the main issues you are currently facing in your life? (please all that apply)  financial  physical/health  academic  employment/unemployment  intimate relations  social relations  other (please specify) ________________________________

14. Close-ended questions – Example 2 How many hours did you spend studying this week?  less than 5 hours  > 5 to 10 hours  > 10 to 20 hours  more than 20 hours

15. Close-ended rating scales • Likert scale • Graphic rating scale • Semantic differential scale • Non-verbal scale • Frequency scale

16. Likert Scale Pick a number from the scale to show how much you agree or disagree with each statement: 1 2 3 4 5 strongly disagree neutral agree strongly disagree agree 1 2 3 4 5 strongly agree neutral disagree strongly agree disagree

17. Graphic Rating Scale How would you rate your enjoyment of the movie you just saw? Mark with a cross (X) not enjoyable very enjoyable

18. Semantic Differential Scale What is your view of smoking? Tick to show your opinion. Bad ___:___:___:___:___:___:___ Good Strong ___:___:___:___:___:___:___ Weak Masculine ___:___:___:___:___:___:___ Feminine Unattractive ___:___:___:___:___:___:___ Attractive Passive ___:___:___:___:___:___:___ Active

19. Non-verbal Scale Point to the face that shows how you feel about what happened to the toy.

20. Verbal Frequency Scale Over the past month, how often have you argued with your intimate partner? 1. All the time 2. Fairly often 3. Occasionally 4. Never 5. Doesn’t apply to me at the moment

21. Sensitivity & Reliability • Scale should be sensitive yet reliable. • Watch out for too few or too many options

22. Scale of measurement guidelines General aim: Maximise sensitivity (i.e. more options) Maximise reliability (i.e. less options) How many measurement options? • Minimum = 2 • Average = 3 to 7 • Maximum = 10?

23. FEELING ABOUT SOMETHING EXTREMELY POSITIVE EXTREMELY NEGATIVE 2-Categories GOOD NOT GOOD 3-Categories GOOD FAIR POOR 4-Categories VERY GOOD GOOD FAIR POOR 5-Categories EXCELLENT VERY GOOD GOOD FAIR POOR

24. Watch out for too many or too few responses “Capital punishment should be reintroduced for serious crimes” 1 = Agree 2 = Disagree 1 = Very, Very Strongly Agree 7 = Slightly Disagree 2 = Very Strongly Agree 8 = Disagree 3 = Strongly Agree 9 = Strongly Disagree 4 = Agree 10 = V. Strongly Disagree 5 = Slightly Agree 11 = V, V Strongly Disagree 6 = Neutral

25. Sampling • Sampling Terminology • What is Sampling? • Sampling Techniques • Example: Shere Hite’s Sex Survey • Summary of Sampling Strategy

26. Sampling Terminology • Population • Sampling Frame • Sample • Representativeness

27. What is sampling? “Sampling is the process of selecting units (e.g., people, organizations) from a population of interest so that by studying the sample we may fairly generalize our results back to the population from which they were chosen.” - Trochim, 2002

28. Sampling Techniques • Probability sampling • Random • Systematic • Cluster • Multi-Stage Cluster • Non-probability sampling • Quota • Convenience • Snowball

29. Representativeness of sample depends on: • adequacy of sampling frame • selection strategy • adequacy of sample size • response rate – both the % & representativeness of people in sample who actually complete survey • Note: It is better to have a small, good sample than a large, poor sample.

30. Sampling Example:Shere Hite‘American Sexology’

31. Male-Female Relations • Shere Hite ‘doyenne of sex polls’ • Media furors & worldwide attention • 127-item questionnaire about marriage & relations between sexes • 4500 USA women, 14 to 85 years • Society and men need to change to improve lives of women

32. Some of Hite’s findings.... • 70% married for 5 years having affairs... (usually more for ‘emotional closeness’ than sex) • 76% did not feel guilty • 87% had a closer female friend than husband • 98% wanted “basic changes” to love relationships • only 13% married for 2+years were still in love • 84% were emotionally unsatisfied • 95% reported emotional & psychological harassment from their men

33. Some of the critical comments.... • “She goes in with prejudice & comes out with a statistic.” • “The survey often seems merely to provide an occasion for the author’s own male-bashing diatribes.” • “Hite uses statistics to bolster her opinion that American women are justifiably fed up with American men.”

34. Response rate & Selection bias - 1 100,000 questionnaires Sent to a variety of women’s groups - feminist organisations, church groups, garden clubs, etc. 4,500 replied(4.5% return rate)

35. Response rate & Selection bias - 2 “We get pretty nervous if respondents in our survey go under 70%. Respondents to surveys differ from nonrespondents in one important way: they go to the trouble of filling out what in this case was a very long, complicated, and personal questionnaire.”- Regina Herzog, University of Michigan Institute for Social Research

36. Summary of sampling strategy • Identify target population and sampling frame • Selection sampling method • Calculate power and required sample size • Maximise return rate

37. Survey Format Checklist • Introduction/covering letter or verbal introducation • e.g. Who are you? Are you bona fide? Purpose of survey? Ethical approval? How results will be used? Confidentiality? Further info? Complaints? • Instructions • Sets the “mind frame”, but be aware few people will read it without good prompting and being easy-to-read • Group like questions together • Consider order effects, habituation, fatigue, switching between response formats

38. Survey Format • Font type / size, number of pages, margins, double vs. single-siding, colour, etc. • Demographics - single section, usually at beginning or end of questionnaire, only use relevant questions • Space for comments? • Ending the questionnaire – say thanks! • Pre-test the questionnaire & revise/refine

39. Pre-test & Revise • Pre-test items and ask for feedback • Revise: • items which don’t apply to everybody • redundancy • skewed response items • misinterpreted items • non-completed items • Reconsider ordering & layout

40. Ethical issues: How to treat respondents • Minimise risk/harm to respondents • Informed consent • Confidentiality / anonymity • No coercion • Minimal deceit • Fully debrief

41. Other ethical issues • Honour promises to provide respondents with research reports • Be aware of potential sources of bias/ conflicts of interest • Represent research literature fairly • Don’t search data for pleasing findings • Acknowledge all sources • Don’t fake (or unfairly manipulate) data • Honestly report research findings

42. Levels of Measurement=Type of Data Levels of measurement = type of data

43. 4 levels of measurement • Nominal/Category • Ordinal • Interval • Ratio

44. Levels of measurement – discrete vs. continuous • Categorical / Nominal (Discrete) • Ordinal / Rank (Discrete) • Interval (Discrete?) • Ratio (Continuous)

45. Each level has the properties of the preceeding levels, plus something more!

46. Categorical / Nomimal • Arbitrary assignment of #s to categories e.g. male = 1, female = 2 • No useful information, except as labels

47. Ordinal /Ranked Scales • #s convey order, but not distance e.g. in a race, 1st, 2nd, 3rd, etc. • Often must be treated as categorical

48. Interval Scales • #s convey order & distance, 0 is arbitrary e.g. temperature (degrees C) • Usually treat as continuous for >5 intervals

49. Ratio Scales • #s convey order & distance, meaningful 0 e.g. height, age • ratios - e.g. 2 x old, 3 x high

50. Why do levels of measurement matter? different analytical proceduresare used for different levels of data More powerful statistics can be applied to higher levels