Tracy Dyke Redmond Senior Associate June 23, 2011

1. Measuring Impact of Compliance Assistance on Auto Body Shops using an Experimental & Quasi-Experimental Evaluation Design Tracy Dyke Redmond Senior Associate June 23, 2011

2. Typology of Evaluation Designs* • Pre-test + post-test comparison with statistical matching • Regression discontinuity • Pre-test + post-test comparison with judgmental matching • Pipeline control group • Pre-test + post-test comparison with delayed baseline • Pre-test + post-test treatment group with post-test only comparison group • Post-test only for treatment and comparison group Increasing Statistical Strength* *Not necessarily overall evaluation strength * Not all possible evaluation designs shown Adapted from Bamberger, Rugh, and Margy: RealWorld Evaluation,Sage Publications 2006. Experimental Design Quasi-Experimental Design Non-Experimental Design

3. “Statistically Valid” Pilot Project for Auto Body Shops • Treatment and comparison areas selected judgmentally, with subjects randomly selected from within these areas • Pro: • Flexible • Reasonably good estimate of project impact when there are good matching criteria • Con: • Assumes comparison group similar to treatment group and equally willing to participate • Does not assess project implementation • Pre-test + post-test comparison with statistical matching • Regression discontinuity • Pre-test + post-test comparison with judgmental matching • Pipeline control group • Pre-test + post-test comparison with delayed baseline • Pre-test + post-test treatment group with post-test only comparison group • Post-test only for treatment and comparison group Adapted from Bamberger, Rugh, and Margy: RealWorld Evaluation, Sage Publications 2006.

4. “Statistically Valid” Pilot Project for Auto Body Shops • Project implemented in phases: subjects in phase 2 serve as control group for subjects in phase 1 • Pro: • Does not require external control group; design relatively inexpensive and easy to use • Con: • Assumes phase 1 and 2 groups are similar (which may not be true) • Requires that phase 2 group does not have access to phase 1 treatment (spillover problem) • Pre-test + post-test comparison with statistical matching • Regression discontinuity • Pre-test + post-test comparison with judgmental matching • Pipeline control group • Pre-test + post-test comparison with delayed baseline • Pre-test + post-test treatment group with post-test only comparison group • Post-test only for treatment and comparison group Adapted from Bamberger, Rugh, and Margy: RealWorld Evaluation, Sage Publications 2006.

5. “Statistically Valid” Pilot Project for Auto Body Shops • Goal: test impact of EPA compliance assistance (e.g., workshops, webinars, materials) on auto body shop compliance with air and hazardous waste regulations • Context: existing hazardous waste regulations, new air regulations coming into effect 2011 (Surface Coating Rule) • Requirements: • Conduct representative measurement of all regulated entities, not just voluntary participants • Do not prevent shops from receiving compliance assistance (or not for very long) • Also test phone survey validity (that methodology not covered in this presentation)

6. Control and Comparison Groups • Population = auto body shops located in areas with elevated air toxics risks and subject to the Surface Coating Rule • Massachusetts selected as study area because EPA Region 1 planned compliance assistance campaign • Treatment and control group in Massachusetts: randomly assigned, considered equivalent • Includes auto body shops in eastern MA with elevated risk • Excludes communities with pre-existing aggressive assistance/enforcement campaigns • Comparison group selected in Tidewater and Piedmont regions of Virginia on the basis of: • No expected compliance assistance from EPA or state • State regulations related to RCRA and air emissions • Number of shops located in areas of elevated-risk

7. “Statistically Valid” Pilot Project Design March – Early July 2010 Summer 2010- January 2011 October 2009 – January 2010 March – Early July 2010 • CA: • Mailings • Webinars • Compliance • Assistance (CA): • Mailings • Workshops/ • Webinars On-site surveys Followed by on-site CA On-site surveys Followed by on-site CA MA - A On-site surveys Followed by on-site CA • CA: • Mailings • Webinars On-site surveys Followed by on-site CA MA - B On-site surveys Followed by on-site CA VA On-site surveys

8. “Statistically Valid” Pilot Project Design March – Early July 2010 Summer 2010- January 2011 October 2009 – January 2010 March – Early July 2010 • CA: • Mailings • Webinars • Compliance • Assistance (CA): • Mailings • Workshops/ • Webinars On-site surveys Followed by on-site CA On-site surveys Followed by on-site CA MA - A Compare randomly assigned treatment vs. control group in MA On-site surveys Followed by on-site CA • CA: • Mailings • Webinars On-site surveys Followed by on-site CA MA - B On-site surveys Followed by on-site CA VA On-site surveys

9. “Statistically Valid” Pilot Project Design March – Early July 2010 Summer 2010- January 2011 October 2009 – January 2010 March – Early July 2010 • CA: • Mailings • Webinars • Compliance • Assistance (CA): • Mailings • Workshops/ • Webinars On-site surveys Followed by on-site CA On-site surveys Followed by on-site CA MA - A On-site surveys Followed by on-site CA • CA: • Mailings • Webinars On-site surveys Followed by on-site CA MA - B Compare “difference-in-differences”: Pre-test to Post-test in MA, compared to Pre-test to Post-test in VA On-site surveys Followed by on-site CA VA On-site surveys

10. Methodological Notes and Next Steps Among valid shops, response rates between 80 – 85% for both states and both years However, list problems (e.g., shops going out of business) led to need for many “backup shops;” increased cost and effort of project Evidence of shops not on the list, operating “under the radar” in VA Some evidence of spillover problems (e.g., some VA shops may have accessed EPA Region 1 webinars) Currently working to analyze 2011 data and develop comparisons; report expected in late 2011

11. Thanks to: EPA HQ for developing the vision and supporting the project EPA Region 1 for identify the list of shops, implementing the assistance, and helping conduct site visits EPA Region 3, Virginia, and Massachusetts for participating ERG and Ski Fabyanic for conducting hundreds of site visits Chris Leggett and Michael Crow for helping develop the methodology and analyzing the data The auto body shops, for letting us in the door!

12. IEc INDUSTRIAL ECONOMICS, INCORPORATED 617.354.0074