Presentation Transcript

1. Workforce Education and Training Public Workshop 2010-2012 Whole House Process Evaluation Training Assessment Findings and Recommendations May 23, 2014
2. Meeting Agenda Introduction & Opening Remarks Safety & Workshop Overview CPUC Decision, why are we here? Review key findings from Process Evaluations IOU Description of efforts to address findings Additional discussion and public comments Workshop wrap-up
3. 2013-2014 Energy Efficiency Decision Direct the IOUs to work with the Commission Staff, the California Energy Commission and others to convene a workshop to review Energy Upgrade California workforce training needs upon completion of IOU-administered Energy Upgrade California process evaluations in 2012. This workshop shall review Energy Upgrade California evaluation findings relevant to Energy Upgrade California training programs, and seek stakeholder feedback on priority training improvements needed, and identify a timeline to put such improvements into place via both IOU and any local government administered Energy Upgrade California or related training programs. Decision 12-05-015, DECISION PROVIDING GUIDANCE ON 2013-2014 ENERGY EFFICIENCY PORTFOLIOS AND 2012 MARKETING, EDUCATION, AND OUTREACH 5/18/2012, Pg. 175.
4. Home Upgrade Sub-Program Theory The Office of the Vice President (U.S) report identifies three market barriers to comprehensive residential retrofits: Lack of customer and contractor awareness and access to information; Lack of access to financing; and Lack of access to skilled workers. To assist in the effort to overcome these problems and market barriers, EUC (Home Upgrade) for single family residences will: Offer a statewide entry level approach (Enhanced Basic/Modified Flex Path) and a comprehensive and flexible performance based approach (Advanced Path) whole house incentives to help build the home performance contracting industry and offer customers and building owners and managers an easy entry point on the path to home performance (barrier 1); Educate customers on the house-as-a-system concept and to encourage behavior changes that increase residential energy efficiency (barrier 1); Educate contractors on the benefits of learning how to properly sell and install whole house measures as part of coordinated WE&T efforts (barrier 1& 3); Offer incentives that influence customers to undertake comprehensive residential retrofits (barrier 1); and Coordinate with relevant utility financing programs and external funding and financing mechanisms at the county, state and federal levels. (barrier 2).
5. Home Upgrade WE&T Implementation Plan Training will be coordinated with the statewide IOU WE&T program, local government residential retrofit and contractor training programs that are tied directly to workforce education and training efforts on a state and federal level. In addition, IOU programs will continue to offer training on: Advanced house-as-a-system concepts and issues; Combustion and other safety training updates; Green building techniques applicable to the program; Blower Door Based Air Sealing; Codes and standards (Title-24) implications; Advanced lighting, HVAC technologies and problem solving; and Business training (including the enhancement of sales, marketing, training, and accounting skills).

6. Whole House Process EvaluationTraining Assessment Findings and Recommendations

   Review of key findings from Process Evaluation for WE&T Training evaluation performed by: ASWB Engineering, Inc. in partnership with: SBW Consulting, Inc. Opinion Dynamics Corporation
7. Presentation Outline Description of Study Methodologies Adult Learning Principles Findings & Recommendations for SCE/SoCalGas Contractor Training Assessment PG&E Contractor Training Assessment SDG&E Assessment Common Contractor Training Findings and Recommendations
8. Training Assessment Objectives Effectiveness and appropriateness Do training objectives address EUC tasks?
9. Training Assessment Activities Evaluated: Program docs & training materials Online offerings / webinars Contractor surveys Field performance results – QA data
10. Training Assessment Activities Interviewed key personnel Utility managers Program managers (3rd party) Course instructors QA/QC evaluators (SCE only)
11. Methods: Training Assessment Criteria How well the training and materials: Supported behavior change Included motivation to action Applied adult learning principles
12. Methods: Adult Learning Principles Learner orientation, buy in, engagement Learner success engineering Practice, application, and interactivity Learning objectives
13. Effect of Teaching Styles and Average Retention Rates
14. Delivery Mode and Average Retention Rates Practice and application are crucial to training effectiveness
15. Relating Learning Levels to Performance Outcomes Bloom’s taxonomy
16. Key Findings and Recommendations SCE/SoCalGas Contractor Training Assessment SCE/SCG Contractor Training Assessment
17. Basic Package (HU) Training (SCE/SoCalGas) Findings: Basic Package now Home Upgrade (HU) Workbook = slides + reference Good match-up between content and measures Includes lecture and hands-on Lacks: Formal objectives Adult learning principles
18. Basic Package (HU) Training (SCE/SoCalGas) Recommendations: Improve training by adding Worksheets More hands-on practice Ways to demonstrate proficiency (e.g., asbestos identification and insulation measurement) Formal learning objectives Adult learning principles
19. Advanced Package (AHU) Training (SCE/SoCalGas) Findings: Advanced Package now Advanced Home Upgrade (AHU) EUC requires BPI Building Analyst certification Certification can be obtained with no formal training Certification does not guarantee EUC skill set Dozens of third-party trainings offered
20. Advanced Package (AHU) Training (SCE/SoCalGas) Findings: CBPCA 12-day course provides: Solid foundation for Advanced (AHU) work Strong combustion safety curriculum Some hands-on practice Lacks: Formal training objectives Adult learning principles Descriptive text for photos
21. Advanced Package (AHU) Training (SCE/SoCalGas) Recommendations: For CBPCA 12-day course: Add training objectives Apply adult learning principles Add descriptive text to photos Supplement BPI training with: More hands-on practice with field testing procedures, protocols Practice of program-specific desk work (forms, energy modeling) Remedial training for specific performance gaps (asbestos, air sealing, etc.)
22. Concerning HVAC Quality Installation and EUC (SCE/SoCalGas) Findings: For minor HVAC repair work, BPI training is adequate For major HVAC repair or replacement, Permit required Title 24 quality standards apply HVAC QI training addresses Title 24 code Participation in the HVAC QI Program would benefit EUC contractors
23. Online Learning Center (SCE/SoCalGas) Findings: New resource at time of assessment On-demand training on: Job Reporting Template (no longer applicable) Energy Modeling Job Processing Combustion Safety Testing Mandatory participation; must pass online exam
24. Online Learning Center (SCE/SoCalGas) Findings: Well designed training Addresses some performance gap issues Incorporates adult learning features Useful for BPI certified contractors who did not attend CBPCA 12-day course
25. Online Learning Center (SCE/SoCalGas) Recommendations: Record exam scores for multiple employees Expand topics: BPI and program standards Field inspection / testing Asbestos identification Insulation and weatherization
26. Mandatory Mentoring (SCE/SoCalGas) Findings: Required of one person from each contractor for both program levels Three topics: Field Work Job Processing Energy Pro May 23, 2014
27. Mandatory Mentoring (SCE/SoCalGas) Findings: Not formal training per se Mentoring includes: Observation of performance Evaluation and scoring Feedback Contractors want more mentoring Appointment requests don’t reflect
28. Mandatory Mentoring (SCE/SoCalGas) Recommendations: Standardize mentoring protocols Seek input from QC firm (RHA) Remedial training based on QA/QC findings Make mentoring more convenient One option: Remote desktop mentoring Energy Pro Job Processing
29. Proposed Inspector-Trainer Role (SCE/SoCalGas) Recommendation: Establish an internal inspector-trainer within each contractor’s staff—will be responsible for: Attending all required training Providing in-house training Ensuring quality of work Suggestions: Collaborative effort between QA/QC (RHA) and mentors (ICF) for training inspector-trainers Develop and provide train-the-trainer materials for inspector-trainers
30. Proposed Marketing and Sales Training (SCE/SoCalGas) Finding: Little marketing or sales content in training materials Recommendations: Suggest formal marketing and sales training Classroom opportunity to practice skills Opportunity to apply adult learning principles
31. Key Findings and Recommendations PG&E Contractor Training Assessment PG&E Contractor Training Assessment
32. Basic Package (HU) Training (PG&E) Findings: Workbook = Slides and reference material Slides show examples in the field Photos good, but lack descriptive text Program reqs taught at end, not detailed Valuable for Basic and Advanced Jobs Little overlap with PG&E Advanced Training Title doesn’t reflect value
33. Basic Package (HU) Training (PG&E) Findings: Adult Learning Principles Some aspects incorporated Offers contractor perspective Incorporates participant experience Connects training to real tasks Includes hands-on practice
34. Basic Package (HU) Training (PG&E) Findings: Searched training for all program standards/ best practices Some not covered: CAS Appliance Testing CO Monitor Installation Air and Duct Sealing Pipe and Attic Insulation Showerheads
35. Basic Package (HU) Training (PG&E) Recommendations: Improve slides and workbooks Rename class to appeal to all Begin day two with program-specific content Include detailed program requirements More combustion safety
36. Basic Package (HU) Training (PG&E) Recommendations: Adhere to Adult Learning Principles Add interactive activities, quizzes Formal training objectives Opportunities for section reviews
37. Advanced Package (AHU) Training (PG&E) Findings: Slides and lecture only, classroom Detailed program requirements Very practical, relates 100% to tasks Clarifies differences - BPI and EUC reqs 90% felt prepared after attending
38. Advanced Package (AHU) Training (PG&E) Findings: Training Materials Workbook = slides and reference docs Shows how to fill required forms Differences from BPI clearly highlighted Lots of helpful reference material Many slides too dense with text
39. Advanced Package (AHU) Training (PG&E) Findings: Adult Learning Principles Some aspects incorporated Offers contractor perspective Connects training to tasks Connects to participant experience
40. Advanced Package (AHU) Training (PG&E) Findings: Certain program standards / best practices not covered: CAS Appliance Testing CO Monitor Installation and Requirements Air and Duct Sealing Cool Roofs Attic Insulation Radiant Barriers Full details in report
41. Advanced Package (AHU) Training (PG&E) Recommendations: Improve the slides and workbooks Extract content from text-heavy slides Can use “notes” pages in PPT Incorporate “missing” program reqs Unless all contractors take Basic, Add a second day to the class Add hands-on practice
42. Advanced Package (AHU) Training (PG&E) Recommendations: Adhere to Adult Learning Principles Add interactive activities Formal training objectives Inquire and adjust to student expectations Opportunities for section reviews
43. Duct Leakage to Outside Hands-On Training (PG&E) Findings: Optional training, only EUC contractors Teaches duct testing procedures Brief presentation, lots of hands-on Recommendations: Improve slides and workbooks Inquire and adjust to participants’ expectations
44. Combustion Safety and Depressurization Testing Hands-on Training (PG&E) Findings: Optional, not EUC-specific Provides much-needed practice Partial use of adult learning principles Some slides lack descriptive text Contractors desire even more practice Some appliances not covered
45. Combustion Safety and Depressurization Testing Hands-on Training (PG&E) Recommendations: Improve slides and workbooks Inquire and adjust to participant expectations Add a second day Cover more appliances Provide EUC-specific version
46. Contractor Training Needs (PG&E) Findings: Many contractors request: More practice One-day or shorter trainings More total training Deeper training in specific areas
47. Contractor Backgrounds (PG&E) Findings: Diverse home-performance backgrounds: Three main groups: Home performance pros Experienced in certain aspects No experience (new entrants, BPI trained) “One size fits all” approach is not optimal
48. Redesign Training into Modules (PG&E) One option: break training into modules Example required trainings: Combustion Safety – (BPI-BA, non-BPI) EUC Standards and QA Example optional trainings: Building Science Sealing & Insulating (attic, ducts, shell) Leakage Testing Adv. Combustion Safety Online Modules Self-guided learning for specific topics
49. Utilization of Existing Training Resources (PG&E) Findings: Resource - Courses at Energy Education Centers Program team aware of opportunity No single place to see all useful trainings Many offered rarely, easily missed No roadmap showing recommended order
50. Utilization of Existing Training Resources (PG&E) Recommendations: Utilize relevant Energy Training Center Courses: Air Sealing and Insulating Existing Homes Auditing Electricity Use in Homes Balanced Ventilation for High Performing Homes Windows Selection for New and Existing Homes Optimizing Residential HVAC System Performance
51. Utilization of Existing Training Resources (PG&E) Recommendations: Training Roadmap Show all home performance trainings Recommended order to take Cross Advertising Share roadmap at home performance trainings Home Performance Training Calendar Online calendar with registration links
52. Utilization of Existing Training Resources (PG&E) Recommendations: Self-Assessment and Personal Training Plan Online survey generates recommended trainings Select desired trainings Personal Training Alerts Registration alerts when class is scheduled
53. Training-relevant Findings and Recommendations SDG&E Whole House Process Evaluation SDG&E Whole House Process Evaluation
54. Whole House Process Evaluation(SDG&E) Findings: Process evaluation conducted by Evergreen Economics Training provided by CCSE (California Center for Sustainable Energy) Basic – 3 day course (unless BPI) Advanced – BPI BA exam prep Optional Mentoring Sales and marketing webinars
55. Whole House Programs – Data Collection (SDG&E) Data collection included: Phone survey with 30 of 78 EUC participants In-depth interviews with 14 participating contractors In-depth interviews with: Program staff QA/QC Contractor recruitment implementers
56. Whole House Programs – Findings (SDG&E) Evergreen Findings: Contractor interviews 57% reported receiving outside to participate in EUC 75% reported undergoing BPI training. Top contractor (# of jobs) trained outside prog National chain franchisee, trained internally Most contractors found the trainings valuable In particular: assessments, calcs, house-as-a-system 50% reported in-field issues not covered in training Due to diversity of homes
57. Whole House Programs – Findings(SDG&E)
58. Whole House – Comparison to Best Practices Program promotes whole house upgrades through availability of incentives; may need to increase promotion support as ARRA efforts discontinue Leverage interest in non-energy benefits; EUC includes HP sales training Balancing the need for verification and QC with streamlined processes remains a challenge
59. Common Recommendations for all IOU’s Common Recommendations for all
60. Different Contractor Needs Recommendation: “Onramps” for different levels Whole House Pros  just program rules Specialized Contractors  targeted training Inexperienced  comprehensive training
61. Must-cover Training Topics for All Methods: Analyzed field QA score sheets Identified most frequent issues in the field Identified typical scenarios Detailed descriptions in full report
62. Must-cover Training Topics for All Findings: Many in common between SCE/SoCalGas and PG&E Likely similar for SDG&E Applies to Basic (HU) & Advanced (AHU)
63. Must-cover Training Topics for All Safety Detecting natural gas leaks Combustion spillage tests Combustion ventilation air Recognizing asbestos Contractor’s responsibility Safety awareness can’t hurt Affects timeline and savings
64. Must-cover Training Topics for All Measure installation, assessment Attic insulation, even coverage Attic hatches, buried ducts Finding all large air leaks Insulating ductwork Blower door testing Duct leakage testing Duct assembly, sealing
65. Provide Ongoing Safety Training Findings: Safety issues are noted frequently and may be missed by contractors
66. Provide Ongoing Safety Training Recommendations: Provide ongoing safety training to contractors Input from inspectors and QC Issues to emphasize: Low house air leakage Possible asbestos in ducts CAZ testing Gas leaks
67. IOU Description of efforts to address Process Evaluation findings
68. Context 90% of contractors reported their training was sufficient to allow them to participate in the program.1 94% of PG&E customers state that they would recommend their contractor to others.2 87% of PG&E jobs passed field QC first time in 2012. 1Recent stats show 93% pass rate. The IOUs have 276 participating contractors and 24 participating raters who have completed over 12,000 jobs to date.3 Immediate use of learning leads to 90% retention rate.1 12010-2012 EUC Process Evaluation. 2PG&E post upgrade homeowner surveys. 3PG&E April 2014 monthly report to CPUC
69. Common Recommendations for all
70. Different Contractor Needs (SCE/SCG & PG&E) Recommendation: “Onramps” for different levels Whole House Pros  just program rules Specialized Contractors  targeted training Inexperienced  comprehensive training
71. On Ramps and Roadmaps This is a good recommendation. IOUs have gone from relying on classroom training to: Individualized Training Plan Mentoring (in-field) Joint / Witness QC Linking inspection results to contractor training Online Training Center accessible 24/7 One page cheat sheets (measure specific)
72. Findings: Must-cover Training Topics Findings: Many in common between SCE/SoCalGas and PG&E Likely similar for SDG&E Applies to Basic (HU) & Advanced (AHU)
73. Recommendation: Safety Detecting natural gas leaks Combustion spillage tests Combustion ventilation air Recognizing asbestos Contractor’s responsibility Safety awareness can’t hurt Affects timeline and savings
74. CAZ Testing Training Improvements 2013-2014 PG&E and SEMPRA harmonized their augmented CAZ Safety testing protocols Combines most stringent of NGAT and BPI protocols and action levels IOUs held 18one-day CAZ training classes and trained 441contractors on CAS in 2013-2014. Extensive field mentoring on CAS (63 for SCE/SoCalGas) PG&E has 93% CAS pass rate (97.5% at appliance level) SCE/SoCalGas has 93% CAS pass rate SDG&E 100% CAS pass rate
75. Recommendation: Measure installation, assessment Attic insulation, even coverage Attic hatches, buried ducts Finding all large air leaks Insulating ductwork Blower door testing Duct leakage testing Duct assembly, sealing
76. Measures Installation Training Improvements 2013-2014 *Total persons trained on all program elements listed here. **SDG&E conducts one orientation for both programs ***Insulation, air sealing, mechanical ventilation and HVAC are combined in number above
77. In addition to classes, all IOUs offer… In-field mentoring. Feedback during oversight QC inspections. Development of Corrective Action plan for struggling contractors. Probation leading to enhanced oversight, etc. Customized training plan development for new or existing contractors. In-field Installation Guide
78. Provide Ongoing Safety Training Findings: Safety issues are noted frequently and may be missed by contractors
79. Provide Ongoing Safety Training Recommendations: Provide ongoing safety training to contractors Input from inspectors and QC Issues to emphasize: Low house air leakage Possible asbestos in ducts CAZ testing Gas leaks
80. On-Going Safety Training All IOUs offer: Asbestos in Ducts as part of Home Upgrade and Advanced Home Upgrade program training CAZ Failures as part of CAS training and program training Mechanical Ventilation as part of HVAC QI ACCA training All IOUs incorporate field QC findings results into contractor training because the same implementers who do training also perform field QC We will look for opportunities to more quickly give feedback on field failures/learnings to contractors
81. SCE and SoCalGas® Energy Upgrade California Home Upgrade Specific
82. Outline Program Response: Participation Workshop Contractor Training for Home Upgrade Contractor Training for Advanced Home Upgrade HVAC QI On-line Resource Center Contractor Mentoring Safety Appendix
83. Participation Workshop Program Response Large Group Format was Not Achieving Objectives Difficult to Convey Details: Overall lower understanding of program requirements Contractor Confusion and Frustration Reduced impact on market transformation Many contractors enrolled but were not active As of May 2013 participation workshops and the recruiting program were revised
84. Participation Workshop Program Response Revised Contractor Recruitment Process Step 1 – Identification of potential contractors Step 2 – Screening of potential contractors Step 3 – Initial Meeting between Acct Mgr and potential contractor (at contractor place of business) Step 4 – Contractor onboarding One-on-one training (energy pro, processing) The Learning Center (Combustion Safety) Document (insurance, CSLB, etc.)submittal.
85. Participation Workshop Program Response Numerous handouts have been developed to facilitate understanding of program requirements, processes, and resources.
86. Contractor Training for Home Upgrade Program Response Numerous worksheets and materials have been developed.
87. Contractor Training for Home Upgrade Program Response The Basic Path training was revised An Online Learning Center replaced the 3 day in classroom training. Provides new contractors an overview of home performance and diagnostic testing procedures. ICF Account Managers assess proficiency through direct observation and one-on-one training. Ongoing as-needed support is provided by the Account Management Team (Account Manager & Project Coordinator)
88. Advanced Home Upgrade Training Program Response Advanced Path Training renamed BPI Building Analyst Training BPI BA Training is now administered by Energy Conservation Institute (ECI) The course consists of: 2 days classroom training 4 days in field training (one more day than prior training) 16 hours of an Online Learning Center training done at their own pace prior to attending the training class. Cost to contractors is $1,500 reimbursed by Program when 5 jobs are completed.
89. Advanced Home Upgrade Training Program Response Additional training opportunities for contractors This includes oversight of the first 3 project submittals (pre- and post-retrofit) and energy modeling. Training to help the contractors develop the Scope of Work is being evaluated for implementation in Q4 of 2014. Currently variable speed pool pump installation training is being offered to cross train/promote the PLA program.
90. Advanced Home Upgrade Training Program Response Additional training opportunities for contractors – cont. Targeted training implemented for performance gaps (addresses top appeal issues from QA/QC inspection process) Combustion Safety Project Processing Energy Modeling Customer Service Marketing Data Collection Inspection Guidelines
91. HVAC QI & EUC-AHU Program Response In January 2014, Title 24 training was conducted, 18 Energy Upgrade Participating Contractors attended. Quality Installation (QI) is currently being integrated into Energy Upgrade California. 47% of the QI contractors are also Energy Upgrade contractors
92. Online Learning Center Program Response BPI BA policies and procedures are addressed in the ECI-led BPI BA training. Two modules (JRT and Job Processing) have been removed from the learning center. The JRT is no longer in use (August 2012) and has been replaced by an online intake tool. Job Processing is now included in the one-on-one contractor recruitment/training. Expansion of training topics is considered as the need arises.
93. Contractor Mentoring Program Response Pre-retrofit field mentoring has been expanded to include HU (previously included AHU only) Mentoring is required on first three jobs Post-retrofit field mentoring (Witness QC) is offered on all projects (HU and AHU). Completed 63 sessions in 2013 Both pre- and post- mentoring processes follow the BPI BA Technical Standards and Inspection Guidelines (adopted from third party quality control – RHA.)
94. Contractor Mentoring Program Response To increase effectiveness, Energy Pro and Job Processing mentoring was shifted to the one-on-one training format in the contractors office. Quarterly contractor reviews are conducted to make sure they are sampled accordingly. Additional training and support is offered to those contractors who show, through the QA/QC process, that they are in need of additional training.
95. Contractor Mentoring Program Response Contractors receive a monthly dashboard report to give them ongoing feedback from the program team.
96. Ongoing Safety Training Program Response Greatly expanded the safety training within the program. It is handled on multiple fronts: Technical Trainings (Half Day - Monthly in 2013, Quarterly in 2014) One-on-one trainings (2 – 4 hours long) Field Mentoring (3 - 8 hours long) Witness QC Mentoring (3 - 6 hours long) Topics include: Potentially Asbestos Containing Material (PACM) identification Gas Leaks Insulation identification
97. SCE/SoCalGas Program Contractors Since mid-2013, the program carefully screen and recruit program contractors to ensure quality performance * Inactive contractors were proactively removed from the program
98. Completed In-Person Contractor Training * These sessions are now in person versus in the Online Learning Center
99. Completed In-Person Contractor Training Topics included: Combustion Safety Project Processing Energy Modeling Customer Service Marketing Data Collection Inspection Guidelines
100. Continued - Online Learning Center (Required for Program Contractors) * The JRT is no longer in use (August 2012) and has been replaced by an online intake tool. Job Processing is now included in the one-on-one contractor recruitment/training by the Account Management Teams.
101. BPI Certification Results In 2013, BPI training was primarily offered to contractors identified through the prescreening process. The Energy Upgrade procedures were separated from the BPI BA training *For 2013, no training for Q1/Q2; training cancelled for Q3, due to lack of participation)
102. Additional discussion and public comment