Evaluating Survey Evidence: A Tale of Two Surveys

1. Evaluating Survey Evidence:A Tale of Two Surveys By: Brian Blake, Ph.D. Jennifer Vedder Erin Sonenstein Consumer-Industrial Research Program Cleveland State University Vanguard Conference National College of Natural Medicine Portland, Oregon January 11, 2011

2. “A survey usually raises more questions than it answers!” “Be careful that a survey doesn’t lead you in a wrong direction!” Too often we hear…

3. Too little do we listen… “A goal without a plan is just a wish.” - A. Saint-Exupery, 1900-1944 “You designed this research backwards!!!”

4. Five Key Questions • Are we listening to the right people? • Are we listening to enough of the right people? • Can we trust what they say? • Do we understand what they tell us? • Does the information provide guidance for action?

5. Two Study Designs “Gen-Pop” ND Survey

6. “Gen-Pop” • Descriptive and Prescriptive Online survey of Oregon residents to: • Identify the public’sfamiliarity with NDs • Gauge the public’s image of NDs relative to other health care providers (HCPs), broadly defined • Unearth the distinct segments that compose the Oregon population, the segments being sectors that differ in regard to their orientations toward medical care • Estimate the demand for naturopathic medicine (NM) and how it changes from one segment to another • Calculate the “targetability” of each segment, i.e., the relative contribution of each segment to the spread of NM in Oregon • Devise a marketing/communication/service distribution strategy based on the above to increase usage of NDs and NM care in Oregon.

7. “Gen-Pop” • Discover parameters that can be effectively brought to the attention of Oregon legislators and health insurance carriers that can provide a rough gauge of: • Unmet health care needs in Oregon • Public satisfaction and particular difficulties experienced with health care coverage • Readiness to turn to NDs as a Primary Care Physician

8. ND Survey Mainly Descriptive Online survey of the nation’s ND community to: • Profile the ND community in America in regard to a variety of parameters: • Engagement in particular professional activities • Professional goals and interests • Provision of alternative treatment modalities • Form of reimbursement • Involvement in a primary care role • Reliance upon various forms of medications in prescriptions • Income from NM and other income streams • Demographic characteristics • Licensures and certifications beyond the ND

9. ND Survey • Chart the diversity of NDs in the above regards and identify sectors or “types” of NDs that compose the contemporary ND community in America. • Assess the needs and priorities of each ND type/sector in order to learn how to serve their distinct interests. • Identify graduates of ND programs who are not presently involved in NM. Who are they? Why are they no longer active? • In conjunction with information from the Oregon General Population survey explore the viability of mutually beneficial referral networks with non-ND healthcare providers.

10. Five Key Questions • Are we listening to the right people? • Are we listening to enough of the right people? • Can we trust what they say? • Do we understand what they tell us? • Does the information provide guidance for action?

11. Q1: Are we listening to the right people? • “Right people” = Representative sample of population • So, how do we obtain a representative sample? • Six steps to a representative sample…

12. For a representative sample… • Specify population clearly. Who is in? Who is out? • Gen-Pop… adults (+18 years old) in Oregon who have received care from a health care provider in the previous 2 years (from January 1, 2009) • ND Survey… persons who graduated in the past ten years (since 2001) with an ND degree from an accredited US college and/or who are current members of the American Association of Naturopathic Physicians (AANP) or a state level ND organization (e.g., OANP). Does not include current students working toward an ND degree. Includes ND degree holders not presently active in NM

13. For a representative sample… • Devise the “sampling frame,” a population listing with reasonable coverage current and is error free • Gen-Pop - members of an online panel residing in Oregon previously screened for age and medical care based on vendor records and confirmed by questions in survey. Like e-Rewards • Evaluation - practical, not perfect - coverage OK, but suspect - current and clean • ND Survey - listing of alumni records of five major ND colleges in US, membership list of AANP, membership listing of all state level ND organizations, compiled-cleaned by HRI

14. For a representative sample… 3. Draw probability sample Here a “stratified random sample” • Gen-Pop - sample proportionate to population of Oregon counties (most recent US Census estimate) • ND Survey - sample drawn proportionate to the ND population in each of 50 states (% calculated from sampling frame)

15. For a representative sample… 4. Design to obtain a high response rate Dillman review Finding in ancient crypt High rate essentials

16. For a representative sample… 5. Design so that subsample weighting of obtained data is possible • Gen-Pop - double weighting • by county to make data set proportional to county’s population (Census based) • by those few demographics (e.g., age) empirically shown in study to be predictive of readiness to see an ND (Census based) • ND Survey - single weighting • by state (based on percent of sample frame)

17. Five Key Questions • Are we listening to the right people? • Are we listening to enough of the right people? • Can we trust what they say? • Do we understand what they tell us? • Does the information provide guidance for action?

18. Q2: Are we listening to enough of the right people? • Standard formulae based on: • desired precision, e.g., ±3%, ±2% band • desired level of confidence that population characteristic is within band, often 95% • variability in characteristic of interest • more variability, larger sample needed • typically assume maximum • population vs. sector • number/size of subpopulations to be analyzed “stand alone,” males vs. females • projected response rate, e.g., if project 50% response, send out double needed sample size

19. B. Gen-Pop • Given: population =3.8 million (adjusted from 3,421,399) qualified (had care) = 70%, response rate = 20% variability = pop. max (50-50), seg mod (70-30) For OR whole need 600 precision = +/- 4% confidence = 95% send out = 3,000 • For typical (of 5) segment with average 150 per segment precision = =/- 7% confidence = 95% send out = 3750

20. C. ND Survey • Given: • For population (US) as a whole = 6,000, response= 50%, 90% qualify • Variability = pop max (50-50), seg mod (70-30) precision = ±3% confidence = 95% • For typical (of 5) segment precision = ±6% confidence = 90%

21. D. Recommend • Gen-Pop need 750 (assuming 5 segments) send out 3750 (assuming 20% response) possible “booster” sample if familiarity with NDs low • ND Survey needed - 750 send out - 1,500

22. Five Key Questions • Are we listening to the right people? • Are we listening to enough of the right people? • Can we trust what they say? • Do we understand what they tell us? • Does the information provide guidance for action?

23. Q3: Can we trust what they say? • Eliminate “bad respondents,” for example • overstatements • suspicious patterns of answers • Eliminate bad items for “illogical” answers based on relationships (correlations) among items • For Gen-Pop • health care provider (HCP) familiarity with licensed phentologist • What they say versus what they do. Stated Impact vs. Predicted Impact for HCP characteristics

24. Q3: Can we trust what they say? • Evaluation • Minimal checks for ND • Practical approach for Gen-Pop

25. Five Key Questions • Are we listening to the right people? • Are we listening to enough of the right people? • Can we trust what they say? • Do we understand what they tell us? • Does the information provide guidance for action?

26. Q4: Do we understand what they tell us? • How to interpret a given question • e.g., suppose numerous NDs say they place high priority upon time to educating the public about health care issues? • Why do so? Good for business? Desire to advance NM care? Enjoy being in limelight? • Use of mathematically derived dimensions that “look underneath” overt answers in both Gen-Pop and ND Surveys • Example of dimensions in image of NDs and other HCPs in Gen-Pop • First, the HCPs considered • Second, the characteristics • Then, the dimensions

27. HCPs Included • A conventional medical doctor (MD degree) • Naturopathic doctor (ND degree) • Licensed physicians assistant (PA) for primary care • Licensed practical nurse/licensed nurse practitioner/registered nurse for primary care • Licensed midwife/Dula • Psychiatrist (MD) • Psychologist (PhD/PsyD) or other licensed mental health professional • Chiropractor (DC degree) • Physical therapist (PT degree) • Certified massage therapist • Licensed acupuncturist • Registered dietician • Yoga, other mediation/controlled movement trainer

28. How well are particular HCPs described by each characteristic? (Scale 1-5)