Economic Directorate’s Quality Audit Program

1. Economic Directorate’sQuality Audit Program Steven S. Klement Office of Statistical Methods and Research for Economic Programs

2. Agenda • Origins of the Quality Audit Program (QAP) • Scope • Current Program • Some Results • Discussion

3. Origins of the QAP

4. Origins of the QAP • What form should the Quality Management Program take? • Quality Profile vs. Quality Audit • Developing the self-assessment tool • Administration of the audit process

5. Scope • There are about 60 programs in the Economic Directorate that are subject to QAP • Each program will undergo an audit every five years • QAP must average twelve audits per year to meet this goal

6. Scope (cont.) • The Quality Audit Program (QAP) looks for compliance with the OMB Standards and Guidelines for Statistical Surveys (Sep 2006) • Does the program’s documentation show compliance? • Is there a plan to produce quality estimates?

7. Scope (cont.) • Auditors do not look in depth to see if execution matches the plan • If the plan calls to compute a variance, the auditor looks to see that there is a variance in the final data product • QAP does not look at the actual computer program which calculates the number

8. Conduct of the QAP • Currently run by the Lead Auditor for Economic Programs • New Assistant Lead Auditor position just created • Overseen by the Research & Methodology (R&M) Assistant Division Chiefs (ADC) • Census Bureau’s Quality Improvement Staff assists in training • Volunteer auditors from the Divisions within the Economic Directorate

9. Timeline

10. Scheduling • A five-year program is put together by the Lead Auditor and reviewed by the Divisions • Includes all programs not already subject to an outside audit • There are limited exceptions • The Division’s comment on schedule and request deferments • The Lead Auditor schedules the actual dates • 3-4 months out plans dates for next year’s audits • 2-3 months out auditors are selected

11. Auditor Selection • The Divisions within the Econ Directorate submit managers & supervisors names to be auditors • The Lead Auditor: • Coordinates schedules • Assigns auditors to each audit.

12. Conduct of the Audit • Engagement Notification sent nine weeks out • Training conducted 7-9 weeks out • Checklist is due 4.5 weeks out • Program area presentation is four weeks out

13. Checklist Example

14. Conduct of the Audit (cont.) • Document review is 1-3 weeks prior • It is the first part of the actual audit • Done by auditors alone • On-site week is when auditors meet with program area • Draft findings are discussed and then finalized • Issues are resolved in groups or one-on-one

15. Final Report • The final report is authored by the Lead Auditor • Reviewed by the auditors • Program Area is given chance to respond • After receiving input and completing revisions, report is sent to management

16. Program Action Plan (PAP) • Conducting an audit is of little value in of itself • The value comes in when improvements are made • Therefore, the QAP directs that the program area develop a PAP • The PAP is the plan for corrective action • Due two months after the audit • The auditors comment on the feasibility of the PAP to address audit concerns

17. Follow Up • Finally, progress with the PAP is followed by the Lead Auditor • Once the PAP is approved Lead Auditor: • Checks for progress after six months • Continues to check yearly until PAP is completed

18. Some Results • Nonresponse Bias Study: • It was found that there was a general lack of NR bias studies in the Directorate • Study required if response rate is below 80% • Planning for study required if response rate is near 80% OR falling towards 80% • Directorate sponsored a pilot NR study • The Directorate created a new team to help programs conduct these studies

19. Some Results (cont.) • Document storage: • Most programs are well-documented • However, the documentation is scattered • Multiple document storage systems in use • Many programs did not use any system • For a number of programs, the audit was the first time all documentation was in one location • The Directorate will roll out a new centralized easy to use system in summer 2010.

20. Some Results (cont.) • Holds people accountable for their documentation • It must be accessible and understandable • It must outline processes actually used • Both help in knowledge transfer over time • Compliance with standards is brought to the forefront (instead of an afterthought)

21. Key Finding • Almost everything one program is deficient or needs improvement on… …another program has a best practice for! • The question: ‘How much is transferable?’ • Auditors take good ideas back to their own programs… …and they share their good ideas with the audited program

22. QAP Going Forward • Look at ways to spread Best Practices • Identify areas that require corporate solutions • Help identify areas requiring additional or concentrated resources

23. Questions? • Thank you for your time Steven S. Klement Lead Auditor for Economic Programs steven.s.klement@census.gov (301)763-6598

24. URLs • OMB Standards & Guidelines for Statistical Surveys (Sep 2006) http://www.whitehouse.gov/omb/inforeg/statpolicy/standards_stat_surveys.pdf • Guidance on Agency Survey and Statistical Information Collections http://www.whitehouse.gov/omb/inforeg/pmc_survey_guidance_2006.pdf