Greg Griffiths for: Rob Burnside Emma Farrell

1. A Coherent Approach to Questionnaire Design Standards, Trends, and Implementation at the ABS Greg Griffiths for: Rob Burnside Emma Farrell ICES 3 Montreal June 2007

2. Overview • How ABS enterprise questionnaire design standards and guidelines are: • implemented organizationally • created and maintained; • scope and content of the standards • trends - changes over time: • role of form design methodologists • new modes and questionnaire structures; and • the Total Respondent experience

3. Big Picture Mandatory use of agreed standards for in-scope instruments detailed in Corporate policy manual. Common conventions, look and feel. Bringing about use of best practice. Organization arrangements give responsibility to areas for different aspects of final forms Form development process is supported by: • Standards manual containing guidelines and examples available on the ABS intranet; • Organizational structures incl survey-specific meetings, an annual review of core questions; and • A forms image database with workflow/ sign-off process; Holds images and linked discussion for interim, current and historical forms back to 1998

4. Responsibilities of areas involved Methodology Division Forms Consultancy Group (FCG): • maintain and extend standards and guidelines; • understand mode effects and try to minimize; • R&D into form design (all modes) Operational sections - manage survey process and produce output Economic Statistics Data Centre - includes a specialised area constructing forms to standards and coordinating testing Conceptual sections - establish user needs Economic Standards Section - consistency of core economic questions and classifications National Accounts Branch - to ensure inputs for Australian National Accounts

5. Collection Management Unit (CMU) Economic Standards and Classifications Section (ESCS) Business Statistics Centres (BSCs) Provider Contact Unit (PCU) FCG Administrative Data Acquisition Unit (ADAU) Despatch and Collection Unit (DCU) Household Statistics Centres (HSCs) Population Survey Operations (PSO)

6. Scope and content Form standards cover: • format, structure and question wording principles, common components; • all modes - paper mail out, fax, CATI, Excel, TDE/IVR • detailed specifications and examples for different types of questions and forms; • contact letters and emails Development and evaluation standards cover: observational studies, usability testing, cognitive testing ... suitability of study design for answering development questions.

7. Creation and maintenance of standards Relies on maintaining knowledge of ABS and international theoretical and applied research and experience; Maintenance and planned updating of standards based on: • Advice on development methods and approaches • Emerging issues • Involvement in field testing • Organizing experiments to examine aspects of design form length effects, scanning design, effectiveness etc. • Urgency of development need

8. Trends: 1 - Evolving role of ABS form design methodologists (FCG) ctd Pre 2000 Policing layout standards 2000 Methodology standards established. Need to examine broader measurement error recognized. 2001 More modes and more development methods 2002/03 Total Approach Management and a wider view- every interaction with the respondent is significant - appropriate, consistent and effective contact 2007 Multimodal Data Collection project funded 2007 Move of household based collections into same branch as FCG. New structure under consideration.

9. Trends - 2: more modes and question types More modes: • structure, presentation and wording changes to keep things the same conceptually • implies more, different testing and development techniques; • increase testing and creation cost • form design increasingly intertwined with IT capability and applications More question structures: • Complex • Subjective (i.e. non-economic) • More like household questionnaire questions

10. Trends: 3 - More development techniques More development techniques • right technique to explore the right thing • more use of cognitive and exploratory methods • Post Enumeration Studies, behavior coding, card sorting, interviewer debriefing • Expert review • Classic usability testing for eform and CATI • Classic experimental design where appropriate

11. Trends: 4 - Total Approach Management Every interaction with a data provider is important. The form is more than the form. • pre approach and covering letters, • brochures and explanatory material, • Telephone, fax and e-mail contacts The response experience is more than filling in the form: • Response enhancing techniques - social influence theories • mode of choice, incentives ….

12. Summary Strong senior management recognition of the importance of form design which is reflected in organizational structure. Formal form development process with defined roles and sign-off. Corporate systems ensuring communication of standards. Visibility of deliberations & decisions.

13. Summary A specialized group of methodologists exists outside operational areas • Responsible for standards and development methods • Involved in day-to-day form development as well as R&D into respondent related measurement error • Centralization of methodology for all modes and methods ensuring consistency • Maintains critical mass of expertise • Maintains a strong commitment to research and the communication of results / new developments to operational areas