From Start to Pilot: Comprehensive Redesign of a Questionnaire Measuring Foreign Direct Investment

1. From Start to Pilot: Comprehensive Redesign of a Questionnaire Measuring Foreign Direct Investment Alfred D. Tuttle Rebecca L. Morrison United States Census Bureau David H. Galler United States Bureau of Economic Analysis

2. Overview – Redesign of BEA Form BE-605 • Reimbursable project with BEA • Multiple, intensive stages of research, design, testing, revision • Qualitative and quantitative evaluations • New BE-605 form mailed out in March 2007

3. Outline • Background – BEA Form BE-605 • Summary of stages of research • Findings • Next steps

4. Background • BE-605 (quarterly) and BE-15 (annual) • Foreign Direct Investment (FDI) – >10% of voting rights in an incorporated entity (or equivalent interest in unincorporated entity) • Sample – foreign-owned businesses in US

5. Research stages and methods • Background investigations • Expert review of form • Focus groups with survey analysts • Observations of callbacks • Respondent debriefings

6. Research stages and methods • Cognitive testing of mockups

7. Research stages and methods • Pilot test • Respondent debriefings • Evaluation questions

8. Findings from background research • Data collection and design issues: • Dense formatting • Economic accounting concepts differ from US GAAP • Separate instructions

9. Recommendations • Formatting recommendations • Letter size pages (not legal) • More “open” visual design • Incorporate instructions into form: • within questions • adjacent to questions on facing page • key instructions become questions • Diagrams to augment questions

10. Original form

11. Recommendation: More “open” design

12. Recommendation: Embedded instructions

13. Recommendation: Adjacent instructions

14. Recommendations: Questions from instructions, and diagrams

15. Cognitive testing • Problematic sections mocked-up and tested (not entire forms) • 5 rounds of cognitive interviews • Iterative process: • mock-ups revised between rounds • design features modified & re-tested

16. Findings from cognitive testing • New design features were effective: • Majority of respondents liked the more open format • Embedded instructions would likely improve data quality • Some “InstructionsQuestions” and diagrams required extensive modification, though most did not

17. Developing 605 pilot • Redesigned entire BE-605 form • Close collaboration with BEA survey staff

18. Pilot test • Mailed 2nd quarter 2006 • Sent to sub-sample (n = 653; survey sample ~4,000) • Completing pilot form optional • Response rate: 53% (350 returned forms) during initial 8-week processing period

19. Pilot form evaluation • Respondent debriefings • Evaluation questions in form

20. Findings – R debriefings • Favorable impressions of new form: • More pleasing to the eye • White reporting spaces easier to see against colored background • Larger font, bold text, more space • Letter size pages easier to handle • Diagrams were helpful • Embedded instructions helpful

21. Findings – R debriefings • Drawbacks: • More pages • Booklet format makes it harder to fax • Took longer to complete than original form (disruption of reporting routine)

22. Limitations • Stymied by experienced Rs, reporting routines… • …though even experienced Rs were helped by new formatting and instructions.

23. Findings – Evaluation questions • Four questions appended to survey • Each item received between 279 and 290 responses

24. Findings – Evaluation questions • Instruction placement? • 79.3% - Near questions • 18.3% - In separate booklet • 2.4% - Not applicable – I haven’t completed the usual BE-605

25. Findings – Evaluation questions • Open space and number of pages? • 48.7% - less open space, fewer pages • 47% - more open space, more pages • 4.3% - Not applicable – I haven’t completed the usual BE-605

26. Findings – Evaluation questions • Use of diagrams? • 65.4% - very helpful or somewhat helpful • 11.7% - only a little helpful • 18% - did not need them • 3.9% - confusing

27. Findings – Evaluation questions • Easy/difficult compared to usual form? • 50.5% - harder • 42.5% - easier • 7% - Not applicable – I haven’t completed the usual BE-605

28. Findings – Evaluation questions • Instructions preferred near questions • Organization charts helpful • Easier or more difficult? • Preference for more/fewer pages?

29. Summary • Redesigned BE-605 went into production in March 2007 • Recommended analysis of edit checks during form processing

30. Summary • Multi-method approach to comprehensive survey redevelopment

31. Thanks! • Alfred D. Tuttle • alfred.d.tuttle@census.gov • Rebecca L. Morrison • rebecca.l.morrison@census.gov • David Galler • david.galler@bea.gov