Quantitative Findings from On-Site Evaluation of Energy Efficiency Program Service Delivery

1. Quantitative Findings from On-Site Evaluation of Energy Efficiency Program Service Delivery Jackie Berger APPRISE

2. Presentation Overview • Process Field Study design • Sample of findings • Audit observation findings • Measure installation observation findings • Final inspection observation findings • Program recommendations • Training • Equipment • Management

3. Process field study design

4. Process Field Study Focus • Audit approach and implementation • Energy education delivery • Measure installation/operation of crews • Post-weatherization quality assurance inspection approach and implementation • Client interaction • Interactions among agency staff • Implications for management, procedures, and training

5. Process Field Study Approach • Go beyond anecdotes • Quantify findings across all visits • Develop check lists and rating scales • Train weatherization experts to implement consistently • Enrich data with descriptive information • Make recommendations for program improvement based on significant issues

6. Check List ExampleAudit Air Leakage andInsulation Diagnostics

7. Rating Design

8. Rating ExampleAttic Insulation Quality

9. Field StudyFlow Chart

10. Agency Selection • WAP Agencies • Almost 1,000 deliver services across the country • 400 comprised agency billing data sample • Field Process Study – 20 agencies in 20 different states • Selection based on: • Number of states per region based on WAP funding • State selection based on part of region and program size • Agencies selected to represent:

11. Observation Team • Social Scientists • One week observation at agency • Weatherization manager and staff interviews • Observations – client intake, audit, measure installation, final inspection • Written agency summary report • Weatherization Experts • 3-Week Observations at Agency • Observations – audit, measure installation, final inspection • Post Observation Data Entry • Training • Monthly update calls • Final debriefing

12. Process Field StudyObservations

13. Process field study audit observation findings

14. Audit Observations Check Lists • Audit preparation • Audit introduction • Audit home walk through • Heating assessment • Air conditioning assessment • Ventilation assessment • Water heater assessment • Air leakage and insulation diagnostics • Diagnostic testing* • Combustion safety testing* • Refrigerator assessment • Lighting assessment • Exit interview *Should have been performed, was performed, performed correctly **Described on write-up, recorded thoroughly and accurately Ratings • Audit introduction • Auditor attempt to engage client • Client engagement in audit • Auditor communication skills • Quantity of education • Quality of education • Exit interview • Completeness of inspection • Completeness of testing • Quality of testing • Auditor efficiency • Auditor professionalism • Auditor respect of client’s home • Auditor safety practices • Audit comprehensiveness* • Planned work comprehensiveness* • Audit write up *Excellent, good, poor

15. Audit Introduction Out of 104 observations completed.

16. Audit Introduction Rating Design

17. Audit Introduction Rating

18. Audit Diagnostic Tests

19. Audit Write-UpRating Design

20. Quality of Audit Write-Up Rating

21. Communication of Audit Findings • Audit write-ups do not convey educational opportunities • Work order not always clear or specific enough for installer to understand the desired outcome • Tools that improved information transfer • Pre-work walk through with installer • Attaching photos of critical areas to the work order • In process check-ups with the installers • Air sealing is a particular area for improved communication • Need for flexibility to change work order when problems are found during installations

22. Audit Strengths • Inspection – 99% inspected every accessible room • Heating system – 94% inspected • Filters – 72% inspected • Ventilation • 72% inspected the kitchen • 78% inspected the bathroom ventilation. • Water heater – 93% inspected • Air conditioning – 82% inspected outside unit. • Insulation – 89% measured all accessible attics. • Testing – 97% conducted blower door test (84% correctly)

23. Audit Opportunities for Improvement • Bathroom ventilation flow – 24% assessed • Hot water – 39% checked temperature at the faucet • Shower flow – none of the auditors measured flow • Insulation – 49% measured in exterior walls. • Blower door – 67% used while inspecting for leaks • Zonal pressure diagnostics – 42% done (87% correctly) • IR camera – 49% used (70% correctly)

24. Audit Findings Opportunities • Utilize energy bills • Increase client engagement • Identify client-specific opportunities for reducing energy use • Auditor understanding of pressure boundaries • Auditor understanding of testing purpose and procedures • Conduct worst case draft testing Strengths • Respect and concern for clients • Explanation of WAP program and process • Use of data collection forms • Auditors meet with contractors to explain audit findings and work scope

25. Process field study Measure Installation Observation findings

26. Measure Installation Observations Check Lists • Measure installation preparation • Client interaction • Introduction • Air sealing procedures • Attic insulation • Thermostat installation • CFLs Ratings • Windows – sealed, interior storm, exterior storm, sash kit, repair • Doors – weather stripping, treated, replacement, installation • Air sealing (8 different locations rated) • Overall air sealing quality, cleanliness • Attic insulation quality, cleanliness • Attic hatch work quality • Walk-up attic work quality, cleanliness • Wall insulation quality, cleanliness, siding preservation/interior wall finishing • Basement insulation quality, cleanliness • Crawl space - vapor barrier insulation, perimeter wall insulation, vents sealing, ceiling insulation and air barrier installation • Ventilation – existing fans ducted to outside, whole house, attic, kitchen, bathroom

27. Measure Installation Observations Ratings (continued) • Heating system – tune-up, repair, replacement, installation, vent correction • Thermostat installation • Air conditioning – sealing around, tune-up, replacement • Evaporative cooler – repair, replacement • Duct sealing, duct insulation, duct improvement cleanliness • Hot water temperature adjustment, heater wrapped, pipes insulated, heater repaired, replaced, heater venting work • Subcontractor treatment of refrigerator and home • Dryer venting • Low cost – AC cooling coils cleaned, HVAC filters cleaned or replaced, faucet aerators, showerheads, CO detector, smoke detector, fire extinguisher • Crew efficiency, professionalism, cleanliness, safety practices • Communication, attempt to engage client, client engagement, education quantity, education quality

28. Crews and Contractors Observed

29. Measure Installation Preparation Out of 114 observations.

30. Air Sealing Rating Design

31. Measure Installation Air Sealing Ratings

32. Measure Installation Strengths • Air sealing prioritized the right areas – 77 percent prioritized sealing at the top and bottom of the envelope. • Attic insulation followed best practices – 77 percent completed attic floor sealing prior to insulation, 88 percent had no gaps or voids, and 95 percent used appropriate materials.

33. Measure InstallationMid-High Ratings

34. Measure Installation Opportunities for Improvement • Air sealing diagnostics – 22 percent used the blower door to guide air sealing and 11 percent used the zonal pressure test to affirm appropriate pressure boundaries. • Air sealing – 57 percent sealed all major opportunities. Low Ratings

35. Measure Installation Findings Opportunities • Increased use of blower door when air sealing • Respect for clients’ homes (booties, covering furniture) • Crew member safety • Increased assessment of HVAC contractors • Explain CFLs when installing • Client education Strengths • Crew flexibility and dedication (above and beyond work scope) • Crew problem solving • Some very high quality work • Experienced HVAC contractors

36. Process field study final inspection Observation findings

37. Final Inspection Observations Check Lists • Home walkthrough • Diagnostic tests* • Combustion safety tests* • Draft tests* • Occupant interaction *Should have been performed, was performed, performed correctly Ratings • Inspection completeness • Testing completeness • Testing quality • Inspector attempt to engage client • Client engagement • Inspector communication skills • Education quantity • Education quality • Inspector efficiency • Inspector professionalism • Inspector cleanliness • Inspector safety practices

38. Final InspectionOccupant Interaction

39. Final InspectionCO Combustion Safety Tests

40. Final Inspection Testing Quality Rating Design

41. Final Inspection Strengths • The inspectors discussed the work that was performed – 75 percent discussed the work that was performed with the client. • Inspections were complete – 90 percent examined all accessible rooms. • Blower door testing was usually conducted – 85 percent conducted this test (85 percent of those performed correctly.)

42. Final Inspection Opportunities for Improvement • Zonal pressure test – 33 percent performed the test (87 percent done correctly.) • IR camera – 44 percent used the camera (95 percent done correctly.) • Combustion safety – 31 percent performed this test on the gas stove (100 percent done correctly.) • Worst case draft test – 60 percent performed this test on the heating system (92 percent correctly.)

43. Final Inspection Findings Opportunities • Increased client education • Explain measures installed • Reinforce client action plan • Improved testing quality • Increased assessment of installation quality • Reduced use of auditor to conduct final inspection – additional perspective Strengths • Good job when following work order closely • Time saved when contractor attends • Information to clients – who to call if they have problems • Referrals to additional assistance programs

44. Summary

45. Average RatingsBy Agency and Overall

46. Training Needs • Energy bill utilization – to assess needs and educate clients • Building science fundamentals • Critical thinking – assessing unique situations • Combustion safety testing • Zonal pressure testing • IR camera use • Ventilation assessment • Safe work practices – worker safety and lead safe work • Interviewing skills – how to understand the client’s needs • Client education

47. Equipment Needs • GPS • IR camera • Boroscope and fiber optic scope and video • Personal safety equipment

48. Management Opportunities • Standards and procedures • Policy manuals • Forms and checklists