Affordable Comfort Indianapolis: Assessing Energy Education Effectiveness

Affordable Comfort Indianapolis:Assessing Energy Education Effectiveness Author, “Poverty and the Public Utility” Involved in notable assistance programs Consultant - Service delivery infrastructure Program design and evaluation Utility billing systems Presented by: Kevin Monte de Ramos KMONTE@kmdr.net May 18, 2005

A Think Tank on Poverty and the Public Utility Public Policy Analysis Program Evaluation Process Assessment Infrastructure Development Collaborative Strategic Planning Performance Management

Session Objectives • Outline an approach to gauge energy education effectiveness • Do we really need to evaluate energy education? • Just what types of programs are we talking about? • Are current approaches working? • Are there other models which can be used? • Establish strategies for program managers to evaluate educators • Not everyone makes a great teacher/educator! • We know a good teacher when we see one. Right? • In what settings do our educators excel? • How can we replicate our successes? • Get contributions from session attendees and other presenters • Room is full of individuals that earn a living talking! • Both educators and managers must be able to communicate. • Besides, I just too old to enjoy listening to myself talk.

Where to start?The beginning I suppose … -- Sumerian Law Clerk, 1758 BC The first office jerk to suffer the dire consequence of poor judgment and bad humor

Why Evaluate Energy Ed? “We already know energy education is effective.”Q. Really, how do you know? A. It has been studied before! “We prefer to invest our money in program services.” Q. Sounds smart. Which services will you invest in? A. Energy education. We want to develop an in-home video. Q. Nice. Are your customers asking for the video? A. No, but they need help with programmable thermostats, filters, etc. Q. Really, how did you choose the content for your video? A. We identified these problems in field and through follow-up surveys. Q. So you verify customer behavior and track program outcomes? A. Sure, evaluations are mandated by our state commission. “Sorry, we just don’t have the funds available for evaluation.” Q. Would it help if we could provide some program statistics for your fund raising activities.A. Sure, are you able to do that for us!!!

What is energy education? Energy Education Learning/teaching activities, often interdisciplinary in nature, that focus on such topics as energy, resources, conversions, conservation forms, uses and issues – including both general and technical educational programs. -- Thesaurus of Environmental Education Terms Environmental Education and Training Partnership • Objectives of educational and information programs • Educate energy consumers regarding ways to increase the energy efficiency of their buildings and activities in order to convince them to take actions that help them manage their consumption or adopt more energy efficient practices. • Inform energy consumers and/or other market actors about program participation opportunities in order to increase enrollment in these programs. • Inform energy consumers and/or other market actors about energy issues, behaviors or products in an effort to transform the normal operations of the market, so that the targeted market actors make more energy efficient operational, purchase or behavior-oriented decisions without direct program interventions or incentives. • -- The California Evaluation Framework, 2004

Just what types of programs are we talking about? Encourage measure installation, energy conservation behavior, increase awareness Increase energy awareness, conduct home energy survey, influence head of household Influence purchasing decisions, lower energy demand and use, transform market. Increase energy awareness, promote energy efficiency programs, increase brand value Encourage measure installation, energy conservation behavior, increase awareness Payment compliance, timeliness, total dollars Lower home energy use, encourage energy efficient behavior, heating system maintenance Influence organizational decisions, impact energy demand and use Legislative action, public support •  Energy Audit Programs • School-based Energy Education Programs • Appliance Rebate Programs • Energy Efficiency Advertising Campaigns •  Weatherization Training Programs •  Budget Counseling Programs • Integrated in-home education sessions  Technology Adoption Programs Energy Efficiency Lobby Efforts

What do these programs have in common? WHERE YOUR FIND THESE PROGRAM COMPONENTS Utility-sponsored low-income assistance State Weatherization Assistance Programs Often associated with private hardship funds Often linked to LIHEAP and other state cash grant assistance Frequently used by REACh applicants/award recipients UNDERLYING COMMONALITIES All impact utility operations All encourage participant behavioral modification Services offered with an expected outcome Frequently require third party assessment/verification WHY WE ARE TALKING ABOUT THEM Performance metrics still under development Legislatively capped at 5% of LIHEAP expenditures Cost effectiveness has been challenged Education components difficult to study

Early findings from PACE bibliography “An Annotated Bibliography of Research-verified Energy Education Programs” Version 2 – July 1994 from the Professional Assn of Consumer Energy Education SELECTED FINDINGS 1984, Timothy Dunsworth attributed 4.3% savings to lost-cost weatherization training Provided by Minneapolis Energy Office via the Neighborhood Energy Workshop program 1987, Tom Lent estimated the 7% incremental effect resulting from an in-home energy Education visit by the Energy Coordinating Agency of Philadelphia via PA WAP 1989, Patti Witti and Martin Kushler found a similar 7% impact via Michigan's Low-Income Weatherization Energy Education and Incentives Program 1991, Marilee Harrigan of the Alliance to Save Energy found an 8% incremental effectwhen 3 in-home education visits were added to PECO’s load management program MORE …

Early findings from PACE bibliography SELECTED FINDINGS (Con’t) 1993, Marialena Selvaggio found a small but significant difference between of 'high intensity‘education services over the 'low intensity' and 'medium intensity' offerings. 1992, Judy Gregory studied recipients of Ohio's Home Weatherization Assistance Programwho participated in the Client Education Pilot Program (CEPP). She estimated an incremental effect of energy education to be 6.7%. While participants of the 1989 HWAP program realized just over a 3% incremental effect for energy education. Taking both reports into consideration, Judy Gregory indicated that energy education without a follow-up visit may yield lower savings than does programs including a follow-up education visits. 1994, Financial Energy Management modest, statistically insignificant, decreases in energybut significant increases in energy efficient resulting from tenant education in multi-unit, HUD-managed housing in Colorado.

Early findings from PACE bibliography SUMMARY OF APPROACHES USED General approach was to look at energy use one year prior to and following participation in home weatherization programs. Difference between groups participating in various levels of energy educational activities. Educational effects ranged between 4% - 9% of pre-program consumption. More recent studies suggest findings up to 12% of pre-program consumption levels. Beginning to track energy efficient behaviors of customers. Educational activities often associated with use of installed measures; such as timers, programmable thermostats, weatherization kits.

Early findings from PACE bibliography PROBLEMS WITH THE APPROACHES USED Educational effects inseparable from other program offerings - Only small samples available to study - Groups often interrelated - Individual methods/messages cannot be studied Measure effects confound educational effects - Thermostat effects (default schedule, manual setback, customer specified) - Timers (educational effect or just an untouched measure) - Wx Kits (savings from measures installed measures or other client behaviors) Participation effects confound educational effects - Lighting (change in use patterns or simply more efficient bulbs) - Interactive Effects (discussions with technicians vs. educational session) - Raised Awareness (motivated to see savings following weatherization)

Similar concerns expressed by GAO-01 • REACh PROGRAM OVERVIEW • OCS has awarded $30 million to 54 projects by year 2000 • Objective to address the home heating and cooling needs of low-income households • USE OF FUNDS • Energy education, home energy audits, home weatherization, and budget counseling • Consumer cooperatives for energy purchasing and using solar and/or wind power • Job skill or employment development services • Other financial assistance, examples include overdue rent or mortgage payments • GENERAL CONCERNS • The HHS has not developed performance goals and measurable indicators • Performance plans do not address how the REACH relates to LIHEAP • “Performance goals and indicators can be beneficial to any federal programs … [as they] provide a clearer basis for selecting projects to fund …”

Findings of the General Accounting Office “…Evaluation reports have substantial design and implementation shortcomings that compromise the validity of [their] findings.” “… other shortcomings … preclude an overall assessment of the projects’ effectiveness.” “Every state project plan should be [able] to measure whether its activities are more cost- effective in the long term than energy assistance payments alone, none of the project evaluation reports provided such an analysis. “… most of the evaluation reports did not report lessons learned or best practices” NOT WHAT WE WANT TO HEAR WHEN CONGRESS IS SCRUTINIZING ITS BUDGET EXPENDITURES AND 20% CUTS ACROSS THE BOARD ARE UNDER CONSIDERATION!!!

Its time to reconsider how we assess energy education CHALLENGE NUMBER 1 If we wish to influence customer behavior, shouldn’t we first understand the process of behavioral change? CHALLENGE NUMBER 2 Research suggests that information/education does NOT leadto behavioral changes! Are we to believe these findings? CHALLENGE NUMBER 3 Positive change comes with a cost! Are they (our clients) ready for change? Better yet, are we ready for change? CHALLENGE NUMBER 4 Inertia is overcome only by an external force acting on an object alreadyin motion! Where will this force come from? Evaluation firms …unlikely to abandon proven industry approaches. Will program managers consider approaches from other industries? Or must pressures first come from senior management, regulators, or government offices?

A transtheoretical approach:The six stages change -- Drs. James Prochaska, John Norcross, and Carlo Diclemente,

Why consider their approach? Recognized as one of the ‘most important developments’ in smoking cessation and health behavior change. Widely adopted by the Centers of Disease Control for programs addressing AIDS and HIV. Used by the National Health Service of Great Britain in their promotional campaigns against smoking. Better than action-oriented programs at achieving long-term behavioral modification. Proven to raise program participation rates and lower risk profiles of participants.

Is it appropriate for utility programs? Environmental policy and energy efficiency programs also use promotional and informational campaigns across broad market segments We struggle with program participation rates and long-term behavioral compliance. Will be considered by Dr. Lori Megdal at IEPEC (August 2005) for use to assess market transformation programs KMDR Research will apply this approach within our industry to assess a broad range of educational programs, both processes and impacts.

The Six Stages of Change Precontemplation: The stage before problem behaviors are acknowledged; therefore, no desire to change. Instead, the individual is frustrated and wants others to change. Contemplation: The stage where the problem is first acknowledged and the individual begins to embrace the idea of a new self without the destructive behavior. Preparation: The stage at which individuals begin planning specific actions, environmental adjustments, and modifications to their normal routines. Action: A time when the first steps are being taken. Others begin to take notice, either assisting or resisting these newly adopted behaviors. Maintenance: The stage whereby individuals fight relapsing to old behaviors by modifying their plans and developing new action items. Termination: A stage where individuals are no longer tempted or have desires to return to problem behaviors.

Key terms for our discussions Theories: In this case, the scientific grounding upon which behavioral change is sought. Five major theories are relevant to change: psychoanalytic (Freud/Jung), humanistic/existentialism (Rogers/May), gestalt/experiential (Perls/Janov), cognitive (Ellis/Beck), behavioral (Skinner/Wolpe). Processes: The strategies available to encourage and support behavioral modification. Broadly speaking, there are nine important processes. Techniques: The specific tactics used within a given process. There are a limitless number of techniques, that add to our confusion. Stages: Identifiable mental states of individuals that adhere to an established model and follow well known patterns. MUST WE ALL BECOME BEHAVIORAL PSYCHOLOGISTS. NO, but we should be aware of the processes and when they are effectively used.

Relationship between Stages and Processes Precontemplation Contemplation Preparation Action Maintenance Termination Consciousness-Raising Social Liberation Emotional Arousal Self-Reevaluation Commitment Environmental Control Countering Reward Helping Relationships • Based on the ground-breaking transtheoretical approach developed by Drs. James Prochaska, John Norcross, and Carl Declementeand adapted for use within the utility industry by Kevin Monte de Ramos of KMDR Research.

Consciousness Raising • Acknowledgement: To properly define and own problem behaviors • Recognition of Defense Mechanisms: The realization that we take many steps to justify our own actions, beliefs, and values … even when they are destructive to ourselves, our families, and our own principles. • Ability to Counter Defense Mechanisms: The transformation of maladaptive defenses into positive behavior. • HOW IT APPLIES TO OUR INDUSTRY: • The foundation for our information programs; such as, school programs, electric safety, environmental awareness, etc. • Used by both sides of low-income advocacy; personal responsibility vs. social ill. • Educators must be able identify and counter defense mechanisms • Directors must be able to clearly define the problems.

Helping Relationships • Breaking Down the Defenses: Challenging those we wish to serve. • Interpersonal Strategies: Must be able to relate to others. • Going Public: Getting others to openly seek resolve to their problems. • Support Groups: Mentoring others through their behavioral problem. • Dynamic Relationships: Support must evolve with our stage of progress. • HOW IT APPLIES TO OUR INDUSTRY: • Are we helping those through poverty or just making excuses for them? • What support network do we offer program participants? • If participants are non-compliant, maybe we should look at the relationship? • Could utilities do more to sound an alarm? Do we monitor payment practices and acknowledge key milestones?

Social Liberation • Societal Pressures: Public support for encouraging constructive behavior and discouraging destructive behaviors. • Raising Awareness: Making the general public aware of problem behaviors. • Identifying Alternatives: Prescriptive choices for those wishing to change. • Responsible Freedom: Allowing others is society to constrain individual choice and suppressing desired activities because it is in the best interest of all. • HOW IT APPLIES TO OUR INDUSTRY: • Regulators and advocates are agents of social liberation in our industry. • Legislative action and agency directives can impact our efforts. • Can be used to build additional support for cash assistance like HeatShare. • Universal Service Funds are great examples of this process.

Emotional Arousal • Impetus: Emotions provide the energy necessary to fuel change. • Resolve: A firmness in purpose supported by our emotional state. • Stimulus: Using events and other environmental factors to elicit and accelerate physiological or psychological activity. • HOW IT APPLIES TO OUR INDUSTRY: • Fears and stigma can inhibit persons from seeking assistance. • Used by trainers and educators, often without intent. • This ability helps distinguish good leaders/educators. • Emotional events can have long-lasting psychological impacts, used correctly this process can yield lasting returns with little added investment. • Do our programs target emotional responses of participants?

Self-Reevaluation • Emotional Appraisal: Consciously and systematically measuring emotions with respect to problem and desired behaviors. • Cognitive Appraisal: Monitoring predefined metrics of progress. • Hanging in the Balance: Behavioral modification has both positive and negative consequence. Since the scales tip in both directions, we must carefully maintain proper balance. • HOW IT APPLIES TO OUR INDUSTRY: • Participants cycle from one payment plan to the next. Upon continued failure, these participants lose utility service. If customers were aware of this pattern, they may be able to break the pattern themselves? • Tools could be used trigger emotional and cognitive appraisals. Also, help us to understand the social and psychological factors driving behaviors. • The appraisals also serve as a critical metric of success; ie. Pros vs. Cons ratios are indicative success.

Commitment • Willingness: Acting or ready to act gladly; eagerly compliant. One must be willing to acknowledge failings before seeking a new reality. • Confidence: A feeling of assurance. One must be confident that change will generate enough reward to offset the costs of behavioral modification. • Faith: A belief that does not rest on logical proof or material evidence. • Anxiety. A state of uneasiness and apprehension. It is how one copes with anxiety that predicts action. • HOW IT APPLIES TO OUR INDUSTRY: • Commitment wanes as anxiety overwhelms. This may be the underlying factor in poor payment behavior. Other household events and rising anxiety make the utility bill less urgent. • What techniques could be added to strengthen participant commitment to program goals and objectives?

Countering • Active Diversion: Staying busy and choosing healthy alternatives over problem behaviors. It is hard to eat while exercising or meditating. • Counterthinking: Changing one’s response to a given situation. Freeing oneself from rigid behavioral patterns. • Assertiveness: Freely expressing thoughts, feelings, wishes, and intentions can alleviate many emotional triggers of problem behaviors. • HOW IT APPLIES TO OUR INDUSTRY: • Budget counseling may suggest paying one bill before spending on non-essentials. Instead of satisfying one’s short-term desire, participants are taught instead to focus on family stability that results in a warm comfortable home. • Can you identify thoughts and actions that should be substituted with another? When cold, we could suggest wearing a sweater or exercising instead of turning up the heat. • Teach customers to be assertive about the energy bills.

Environmental Control • Avoidance: Staying away from people, places, and objects that trigger undesirable behavior or removing temptations from our home/work. • Controlled Exposure: Trial runs used to practice scripted responses and refine countering techniques. Going to happy hour despite a desire to curb eating, stop smoking, and avoiding alcohol. • Reminders: To do lists and calendars serve as valuable reminders. • HOW IT APPLIES TO OUR INDUSTRY: • Programmable thermostats are used to control home temperature settings and timers used to schedule lighting. Refrigerator magnets serve as reminders. • Pre-payment meters could serve to limit over consumption. This technology, in credit mode, could serve as reminders by providing immediate feedback regarding use of energy appliances. • Would participants agree to utility control of home thermostats in exchange for a flat monthly bill?

Reward/Punishment • Covert Management: Self-administered ‘pats on the back’ A simple ‘It feels good to be in control” can make all the difference. • Contracting: Formal and informal contracts with either yourself or others that reward both avoiding disruptive behaviors and substituting preferred alternatives. “I bet you $10, she will go out with me this weekend!” • Escalating Expectations: Raising individual expectations as behaviors improve helps to keep us moving forward. This may be a natural result of self-reevaluation and commitment. • HOW IT APPLIES TO OUR INDUSTRY: • We ask participants to sign forms indicating actions that will be taken a throughout. Similarly, arrearage forgiveness reward participants for payment compliance. • What can we do to help participants manage their own progress or raise their expectations in life? • Are we challenging participants to ration energy consumption?

Addressing Session ObjectivesApplying this approach to our efforts

Evaluating our Educators Understanding the processes of change helps us understand whether or not the techniques employed by our educators will work. Matching educational efforts with participants at varying stages of change can help us achieve greater cost-effectiveness. Educators employing techniques at appropriate times will be more effective than others that utilize techniques in an untimely manner. Can your educators identify the problem behaviors? Are educators pointing out potential pitfalls and providing strategies to get around them? Just how engaged are participants in the conversation? Are the emotional involved? Have they abandoned their defense mechanisms. In short, passive observation or ‘secret shoppers’ can be used to gauge the ability of the educator to effect behavioral modification.

Questions and Answers How can you determine whether or not energy education is an effective component within your program? Simply treating education as another energy saving measure will not resolve the question of cost-effectiveness. This is especially true when education involves the use/installation of other measures. Before we can assign causative effects of energy education, we must first track the actions taken by our participants. New methods will be needed for this; sub-metering, follow-up surveys, and customer diaries. By identifying the processes used within our programs, we can establish objective metrics to track program performance. Eventually, we can assign dollars to these metrics and quantify educational cost-effectiveness.

Questions and Answers How do I know if I am connecting with my clients? How should we deal with varied learning styles, in the classroom and in the field? Educators can monitor dialog and emotional states of participants to determine their level of influence. Dialog that moves from defense mechanisms to acknowledgement will serve as a positive indication. Similarly, emotional arousal puts customers into a receptive state. Mirroring of body language also suggests a participant receptiveness. Trainers have a more difficult task before them. Reading a group is like reading an individual. Acknowledging nods, uniform actions, and probative dialog are also demonstrative of participant receptiveness. We do not need to be practicing behavioral psychologists to employ these methods. Many individuals can read persons instinctively. For the rest of us, there are many books available in the self-help/psychology sections of your favorite bookstore.

Questions and Answers What are realistic savings expectations from client education? There is no pat answer to this question. As indicated earlier, most studies conducted thus far do not adequately isolate the educational effects nor do they link achieved savings with specific behavioral changes. So, we have nothing to build upon. We hear estimates ranging from 4% to 12% percent of pre-program consumption levels for the energy effect. Also, we have seen increased payment frequency for participants in arrearage forgiveness programs, resulting in larger annual contributions toward home heating. One cause for concern is that only 20% of individuals seeking help with problems are ready to take action. If that statistic holds true for energy education, action-oriented participants would have to achieve savings 5 times that of stated program impacts. Is this level of savings realistic? We can calculate technical potential to validate the assumptions.

Affordable Comfort Indianapolis: Assessing Energy Education Effectiveness