1 / 31

From Start to Pilot: Comprehensive Redesign of a Questionnaire Measuring Foreign Direct Investment

From Start to Pilot: Comprehensive Redesign of a Questionnaire Measuring Foreign Direct Investment. Alfred D. Tuttle Rebecca L. Morrison United States Census Bureau David H. Galler United States Bureau of Economic Analysis. Overview – Redesign of BEA Form BE-605 .

ady
Télécharger la présentation

From Start to Pilot: Comprehensive Redesign of a Questionnaire Measuring Foreign Direct Investment

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. From Start to Pilot: Comprehensive Redesign of a Questionnaire Measuring Foreign Direct Investment Alfred D. Tuttle Rebecca L. Morrison United States Census Bureau David H. Galler United States Bureau of Economic Analysis

  2. Overview – Redesign of BEA Form BE-605 • Reimbursable project with BEA • Multiple, intensive stages of research, design, testing, revision • Qualitative and quantitative evaluations • New BE-605 form mailed out in March 2007

  3. Outline • Background – BEA Form BE-605 • Summary of stages of research • Findings • Next steps

  4. Background • BE-605 (quarterly) and BE-15 (annual) • Foreign Direct Investment (FDI) – >10% of voting rights in an incorporated entity (or equivalent interest in unincorporated entity) • Sample – foreign-owned businesses in US

  5. Research stages and methods • Background investigations • Expert review of form • Focus groups with survey analysts • Observations of callbacks • Respondent debriefings

  6. Research stages and methods • Cognitive testing of mockups

  7. Research stages and methods • Pilot test • Respondent debriefings • Evaluation questions

  8. Findings from background research • Data collection and design issues: • Dense formatting • Economic accounting concepts differ from US GAAP • Separate instructions

  9. Recommendations • Formatting recommendations • Letter size pages (not legal) • More “open” visual design • Incorporate instructions into form: • within questions • adjacent to questions on facing page • key instructions become questions • Diagrams to augment questions

  10. Original form

  11. Recommendation: More “open” design

  12. Recommendation: Embedded instructions

  13. Recommendation: Adjacent instructions

  14. Recommendations: Questions from instructions, and diagrams

  15. Cognitive testing • Problematic sections mocked-up and tested (not entire forms) • 5 rounds of cognitive interviews • Iterative process: • mock-ups revised between rounds • design features modified & re-tested

  16. Findings from cognitive testing • New design features were effective: • Majority of respondents liked the more open format • Embedded instructions would likely improve data quality • Some “InstructionsQuestions” and diagrams required extensive modification, though most did not

  17. Developing 605 pilot • Redesigned entire BE-605 form • Close collaboration with BEA survey staff

  18. Pilot test • Mailed 2nd quarter 2006 • Sent to sub-sample (n = 653; survey sample ~4,000) • Completing pilot form optional • Response rate: 53% (350 returned forms) during initial 8-week processing period

  19. Pilot form evaluation • Respondent debriefings • Evaluation questions in form

  20. Findings – R debriefings • Favorable impressions of new form: • More pleasing to the eye • White reporting spaces easier to see against colored background • Larger font, bold text, more space • Letter size pages easier to handle • Diagrams were helpful • Embedded instructions helpful

  21. Findings – R debriefings • Drawbacks: • More pages • Booklet format makes it harder to fax • Took longer to complete than original form (disruption of reporting routine)

  22. Limitations • Stymied by experienced Rs, reporting routines… • …though even experienced Rs were helped by new formatting and instructions.

  23. Findings – Evaluation questions • Four questions appended to survey • Each item received between 279 and 290 responses

  24. Findings – Evaluation questions • Instruction placement? • 79.3% - Near questions • 18.3% - In separate booklet • 2.4% - Not applicable – I haven’t completed the usual BE-605

  25. Findings – Evaluation questions • Open space and number of pages? • 48.7% - less open space, fewer pages • 47% - more open space, more pages • 4.3% - Not applicable – I haven’t completed the usual BE-605

  26. Findings – Evaluation questions • Use of diagrams? • 65.4% - very helpful or somewhat helpful • 11.7% - only a little helpful • 18% - did not need them • 3.9% - confusing

  27. Findings – Evaluation questions • Easy/difficult compared to usual form? • 50.5% - harder • 42.5% - easier • 7% - Not applicable – I haven’t completed the usual BE-605

  28. Findings – Evaluation questions • Instructions preferred near questions • Organization charts helpful • Easier or more difficult? • Preference for more/fewer pages?

  29. Summary • Redesigned BE-605 went into production in March 2007 • Recommended analysis of edit checks during form processing

  30. Summary • Multi-method approach to comprehensive survey redevelopment

  31. Thanks! • Alfred D. Tuttle • alfred.d.tuttle@census.gov • Rebecca L. Morrison • rebecca.l.morrison@census.gov • David Galler • david.galler@bea.gov

More Related