Improving Accountability by Using External Audit and Verification of Data

1. Improving Accountability by Using External Audit and Verification of Data Kathleen Mooney SERVE Assistant Evaluation Specialist Greensboro, NC Kmooney@serve.org Presentation to the Canadian Evaluation Society/Société Canadienne D’ Évaluation Vancouver, B.C. June 4, 2003 Session 302

2. What is the strategy? External Audit and Verification of Data

3. Primary Purpose and Use • Definitions: • Formal review of financial situation or accounts • A methodological examination and review • A third-party evaluation or external evaluation, …an auditor may be a meta-evaluator, typically serving in a formative and summative role Sources: Merriam-Webster’s Collegiate Dictionary (1998) Scriven, Micheal. Evaluation Thesaurus (1991)

4. Primary Purpose and Use • Ensure Accuracy • Ensure Reliability • Ensure Utility • Ensure Credibility

5. How Was This Strategy Used in REL Performance Indicators Work? • Strategy Encouraged by Funding Source • Ensured “Buy-in” from all REL Directors • Establish Cost-Sharing from all Labs • RFP/External Bidding Process • Selected External Contactor • Government Auditing Standards • Standards for Evaluating the Quality of Program Performance Data

6. Methodology and Scope • Sample • All 10 labs participated in the audit • Field Visits • 2 labs • Randomly selected • Aggregating lab/Non-aggregating • Telephone Interview • 9 labs

7. Methodology and Scope • Data Analysis • Flowcharting • Review and Crosschecking • Audit Trail Analysis • Content Analysis • Statistical Analysis • Database Inspection and Accuracy Tests

8. How Was This Strategy Used in REL Performance Indicators Work? Audit Questions: • How valid (relevant) are the performance indicator data in regard to definitions of the indicators? • How reliable (accurate) are the reported performance indicator data counts and statistics?

9. How Was This Strategy Used in REL Performance Indicators Work? Audit Questions (cont): • Do the Laboratories have adequate procedures in place to receive, review, verify, process, store, analyze, report and update the performance indicator data? • Are quality assurance controls for these procedures utilized by the Laboratories?

10. Audit Results “Overall, given that adequate procedures and quality assurance controls where in place, the REL performance indicator data for 2002 were found to be reliable, valid and accurately reported to OERI/IES.”

11. Benefits • Ensures the transparency of the system and data collection methods/results • Improves documentation of processes/results • Provides a seal of approval on data quality • Validated by: “Experienced Public/Private Entity” • Provides recommendations to increase rigor of the data collection system • Survey uniformity • Under-reporting • Intended to increase credibility of the system

12. Limitations • Does not always increase credibility • Data collection process will change next year • Consequences may outweigh rewards • Did not sufficiently address the issue of data utility • How can data results be used more effectively at all levels of the system?