John H. Reed Innovologie, LLC jreed@innovologie

1. A Generic Theory BasedLogic Model for Creating Scientifically Based Program Logic ModelsAn Expert Lecture Presented to theAmerican Evaluation AssociationThe Research, Technology, and Development Evaluation TIGPortland, OregonNovember 4, 2006 John H. Reed Innovologie, LLC jreed@innovologie.com Work presented here was completed for the U.S. DOE Office of Energy Efficiency and Renewable Energy by Sandia National Laboratories, Albuquerque, New Mexico, USA under Contract DE-AC04-94AL8500. Sandia is operated by Sandia Corporation, a subsidiary of Lockheed Martin Corporation. Opinions expressed are solely those of the authors. This work is preliminary. This work may be cited or used only with permission.

2. Acknowledgements • Co-authors • Gretchen Jordan (Sandia National Laboratories) • Ed Vine (Lawrence Berkeley National Laboratory) • US Department of Energy Sponsor • Jeff Dowd

3. Overview • Next generation logic models • The problem • US DOE’s Office of Energy Efficiency and Renewal Energy • Diffusion of innovations - a theory of change • Placement of a theory of change in the logic model • A generic theory of change and more domain focused versions • Researchable issues deriving from the model • Summary and conclusions

4. This Generation of Logic Models • The logic model framework is a heuristic based on the performance spectrum (resources, activities, outputs, outcomes) • Users apply the heuristic to their understanding of a program • The resulting logic model with its description of how target actors and partners respond is dependent on the knowledge and skills of the model developer and may or may not closely resemble reality and may or may not be grounded in accepted social science theory.

5. A Theory Based Logic Model • Focuses more attention on outcomes • Increases attention and understanding to what target actors and partners do in response to program outputs • Grounds this in widely accepted social theory imbuing the logic model with a theory of change rather than asking the user to create one. • Makes it easier for program managers and evaluators to: • understand effects their programs are having • develop more realistic models • design and conduct better evaluations

6. The Problem • A governmental organization with multiple offices and programs conducting energy research and delivering a broad range of products and services to induce the adoption of efficient technologies and practices • The organization regularly measures and tracks large numbers of outputs • There are numerous partners and target audiences forming a complex market mosaic • The need for accountability requires that outputs of the programs be causally linked to the actions of partners and target audiences which in turn produce effects that result in the desired impacts (outcomes) • It is important to predict and measure the effects and impacts and attribute them to the program • More consistency is needed in the metrics and measures across programs allowing better defense of the program and roll-up of effects

7. The Office of Energy Efficiency and Renewable Energy of the US Department of Energy Undertakes Activities to Build Infrastructure Fund and Promote Adoption Analyze and Plan Develop Technical Information Assist Public Entities Assist Businesses Outreach and Partner Assist and Fund Purchases Provide Tools and Technical Assistance Reviewing and Reporting To target Technical and other personnel in laboratories, government, firms, colleges, universities Federal, state, local agencies, and public organizations Manufacturers distributors retailers End-user organizations, firms and individuals So that there will be Actions by End-users Actions by Knowledge Workers Actions by Public Entities Actions by Manufac-turers and Business Thereby Reducing energy use and emissions, enhancing productivity, and global security

8. Examples of Actorsin Each of Four Domains

9. Examples of Actionsin Each of Four Domains

10. The Office of Energy Efficiency and Renewable Energy of the US Department of Energy Undertakes Activities to Build Infrastructure Fund and Promote Adoption Analyze and Plan Develop Technical Information Assist Public Entities Assist Businesses Outreach and Partner Assist and Fund Purchases Provide Tools and Technical Assistance Reviewing and Reporting To target Technical and other personnel in laboratories, government, firms, colleges, universities Federal, state, and local agencies Manufacturers distributors retailers End-user organizations, firms and individuals So that these target audiences will Create, advance, and package market and technical knowledge to make energy efficiency more accessible and implementable Change the policies, structure and operation of public entities to smooth the advance energy efficiency Create and enhance products, create and align market channels, enhance marketing, and develop installation and support infrastructures Adopt, replicate, institutionalize, and enculturate energy efficient technologies and practices Thereby Reducing energy use and emissions, enhancing productivity, and global security

11. The Office of Energy Efficiency and Renewable Energy of the US Department of Energy Undertakes Activities to Build Infrastructure Fund and Promote Adoption Analyze and Plan Develop technical information Assist Public Entities Assist businesses Outreach and Partner Assist and fund purchases Provide tools and technical assistance Reviewing and Reporting To Target Knowing these activities and their corresponding outputs • We need to describe and measure the expected response of: • Knowledge workers • Public Entities • Manufacturing and Business • End-users Technical and other personnel in laboratories, government, firms, colleges, universities Federal, state, and local agencies Manufacturers distributors retailers End-user organizations, firms and individuals So that these target audiences will Create, advance, and package market and technical knowledge to make energy efficiency more accessible and implementable Change the policies, structure and operation of public entities to smooth the advance energy efficiency Create and enhance products, create and align market channels, enhance marketing, and develop installation and support infrastructures Adopt, replicate, institutionalize, and enculturate energy efficient technologies and practices Thereby Reducing energy use and emissions, enhancing productivity and global security, creating sustainable habitats

12. The Solution • A theory based logic model that encompasses the range of program delivery activities from which program managers can choose relevant elements to develop a logic model customized to their own program • The inclusion of a widely accepted theory of social change in the outcome space of the logic model that will aid program managers to systematically describe how their program outputs will lead to their desired long term goals

13. Diffusion of innovations model Socio-cultural/market environment • Market structure • Market segments • Prior practice • Culture and norms • Innovativeness Communication field • Broadcast • Contagion Feedback Awareness Implementation Confirmation Persuasion Decision Continued adoption Later adoption Adoption Characteristics of the decision-making unit Product Characteristics Discontinuance Continued rejection Rejection • Adopter type • Personality type • Communication behavior • Socio-economic status • Relative advantage • Compatibility • Complexity • Trialability • Observability Source: Everett Rogers 2003 as modified by Innovologie, LLC. 2005

14. Broadcast Contagion • Changed practices directly due to program • Finance / contract • Design/ specification • Implement • Operation / maintenance Innovators Early Adopters Early and late majority Awareness of resources, EERE opportunities, technologies and practices Value confirmed • Institutional change • • Incorporate into standard operating procedures • • Increased knowledge, promotion & advocacy • Government policy, codes, programs support adoption • Product, sales, service are available, high quality, profitable • Production and transaction costs decline Participants replicate changes Seek information / become persuaded • Product value accepted • Relative advantage • Compatibility • Complexity • Trialability • Observability External households/ facilities/firms observe and emulate changes Widespread acceptance Decide to partici-pate in a program Decide to imple-ment practices / measures Energy and non-energy impacts A Version of Diffusion of Innovation to Link Outputs and Impacts

15. The Office of Energy Efficiency and Renewable Energy of the US Department of Energy Undertakes Activities to Build Infrastructure Fund and Promote Adoption Analyze and Plan Develop Technical Information Assist Public Entities Assist Businesses Outreach and Partner Assist and Fund Purchases Provide Tools and Technical Assistance Reviewing and Reporting To Target Appropriate versions of the model can be developed for knowledge workers, public entities, and businesses as well Technical and other personnel in laboratories, government, firms, colleges, universities Federal, state, and local agencies Manufacturers distributors retailers End-user organizations, firms and individuals So that there will be actions ByManufac-turers and Businesses ByKnowledge Workers ByPublic Entities ByEnd-users Thereby Reducing energy use and emissions, enhancing productivity, and global security

16. A Generic Version of the Diffusion Model Appropriate to Manufacturers and Businesses Broadcast Contagion • Implement production/distribution of products • Finance • • Capital set aside • • Subsidies • -Design / plan • • Production facility • • Distribution channels • • Showrooms • • Training • Implement • • Production facility built • • Product network established • • Products produced • • Train personnel / distributors • • Products distributed • • Products stocked • • Products sold • • Products serviced • Operation / maintenance • •Service network established • •Service / maintenance workers trained Innovators Early Adopters Early and late majority • Awareness of: • information resources • Technology potential • - Program opportunities • Information seeking/ persuasion • Product and market research • Persons trained • KSA’s learned • Advice received from peers • Participated in codes and standards setting • Participate in demonstrations • Supply production value accepted • Relative advantage • Profit • Goodwill • Market leadership • Compatibility • Manufacturing facility • Existing business lines • Complements business environment • Complexity • Low complexity • Ease of manufacture • Trialability • Demonstration • Pilot Test • Observability • Value confirmed • Production goals met • Financial goals met • Demand stable / rising • Profit goals met • Product leadership recognized External firms observe changes learn through peer interaction External firms emulate changes • Sustained institutional change • Production meets demand • Economies of scale occur • Product becomes permanent part of product line • Product captures market share • Product development path identified • Increased promotion • Peer firms • Advocacy networks • One-to-one interactions Decide to participate in a program Decide to implement practices / measures Long term impacts

17. A Generic Version of the Diffusion Model Appropriate to End-users Broadcast Contagion External households /facilities/firms observe changes learn through peer interaction • Value confirmed • Energy benefits observed • Non-energy benefits supporting value propositions observed • Changed technologies/ practices directly due to the program • Finance • • Incentives • • Buy down • • Life cycle costing used • • Budgeting / capital set aside • • Performance contracts • • Loans • Design/redesign/specification • • System/whole system approaches • • Purchasing specifications • Implement system redesigns and upgrades • • Install efficiency measures (insulation, control systems, etc) • • One-to-one replacement with efficient equipment at failure • • Early one-to-one replacement • Operation / maintenance Innovators Early Adopters Early and late majority • Awareness of: • information resources • potential/ efficiency opportunities • - program opportunities (program offerings/ labels, etc) • EERE opportunities • Program participants emulates changes within household/facility/firm • Changes at the same site • Changes at another site External households /facilities/firms emulate changes • Information seeking/ persuasion • Literature/web search • Audit • Peer recommendation evaluation • Obtain and try tools • Training • Attendance at Showcase / demonstration attendance • Hands-on experience • Sustained change • Households • • Develop energy efficiency orientation • • Adopts new energy efficiency habits • • Take advocacy stance • - Firms • • Adopt corporate energy policy • • Appoint corporate energy manager • • Incorporate energy efficiency as a criteria for personal performance • • Adopt energy efficiency as part of competitive strategy • • Incorporate energy efficiency into branding • • Take advocacy stance • Widespread acceptance of product among target group • Recognition • Market tipped • Increased market penetration • Product costs decline Decide to participate in a program Decide to implement practices / measures • Increased promotion • Advocacy groups • Peer networks • One-to-one interactions • Product value accepted • Relative advantage • Compatibility • Complexity • Trialability • Observability Long term impacts

18. A Generic Version of the Diffusion Model Appropriate to Public Entities Broadcast Contagion Innovators Early Adopters Early and late majority Other government entities observe and seek information • Awareness of: • information resources • Policy / implementation needs • Information gaps • Training opportunities • Opportunities for coordination • Opportunities to provide citizen services / meet needs • Potential energy efficiency opportunities Decide to implement practices / measures Decide to participate in a program Sustained institutional change • Make program a permanent offering • Adopt energy efficiency as governmental goal • Appoint governmental energy manager • Incorporate energy efficiency as a criteria for personal performance • Incorporate energy efficiency into branding • Incorporate energy efficiency purchasing and building into government related purchases, new construction, remodeling • Take advocacy stance Government entity promotes at other sites or promotes other types of activities • Implementation • Finance • • Incentives • • Grants • • Performance contracts • • Loans • • Buy downs • Design/redesign/specification • • Codes and Standards • • Internal policies • • Internal efficiency programs • - Coordination • • Coordinated federal, regional, state, local policies • • Coordinated codes and standards • Appliances • Buildings • Net metering • Trained government infrastructure • More efficient public policies • Energy efficiency programs • Demonstrations • Pilot programs • Audits • Performance contracts • Information • Purchasing • Operation / maintenance • Information seeking/ persuasion • Literature/web search • Technical analyses completed • Policies analyzed • Peer contacts made • Peer recommendations received • Market assessments completed • Training completed • Evaluations completed Other government entities observe and emulate • Increased promotion • Advocacy groups • Peer networks • One-to-one interactions • Product value accepted • Relative advantage • • Protects commons • • Social and economic benefits for large numbers of citizens • • Low cost • • Low political cost • Compatibility • • Fits existing policy / legal / administrative frameworks • • Culturally acceptable • • Fills market gap or need • • Constituency or easily developed constituency • Complexity • • Easily understood • • Simple administration • Trialability • • Can be piloted • • Demonstrated • Observability • • Effects easily measured • • Effects measured in a timely way • Widespread acceptance of product among target group • Recognition • Market tipped • Increased market penetration Energy and nonenergy impacts • Value confirmed • Measure energy savings • Measure and observe non-energy benefits

19. A Generic Version of the Diffusion Model Appropriate to Knowledge Entities Broadcast Contagion • Replication: Emulate changes within Organization/household/ facility/firm • Changes at the same site • Changes at another site • Implemented: Behavior/ Practices Changed • Capital • - Larger, stronger knowledge community/ critical mass, core competencies • Invest in more research in this area • Design/Plan • Changed knowledge seeking behavior (e.g. research approach, how calculate savings, use different procedures, consider labeling criteria) • Considered limitations, benefits/ opportunities (e.g. in setting standards) • Implement • Changed attitude, perception of value used in negotiations, text books, technical specifications • Used tool to measure, choose option, shift loads • Maintain • - keep data base current, correct, consistent over time (e.g. climate data, wind resources data) Confirm Value • Knowledge valued in community • Knowledge cited by others • Knowledge used by others Innovators Early Adopters Early and late majority Access Information and are Aware of: - Knowledge gaps - information resources (curriculum, tech reports, etc.) - Program opportunities (program offerings/ labels, etc) - EERE potential/ opportunities Sustained Knowledge R&D community, Policy makers, Households, Firms use and share the knowledge routinely in textbooks, technical manuals, regulations, etc. Others observe changes and learn through peer interaction • Persuasion / information search • - More knowledge sought in this area • New research ideas developed • Obtained and tried knowledge tools (e.g., software, curriculum) • Demonstrated/ saw demonstration • Hands-on experience • Product filter: Knowledge/tool has value • Relative advantage • • More credible information • • Information confers • - economic benefit • - efficiency benefit • - production benefit • - quality benefit • Compatible • • With other knowledge/tools • • Production requirements • • Organizational goals/systems • • Culture • Complexity • • Easy to understand • • Simplifies existing knowledge • Can be tried by target audience • Benefits of use can be Observed • Advocacy Spreads word to others Awareness of confirmed value spreads through • Advocacy groups • Peer networks • One-to-one interactions Replication by those not directly involved with the program -Widespread acceptance of product among target group - Tip the market - Market penetration increases

20. A Comparison of three Product Characteristics Business Public Knowledge • Supply production value accepted • Relative advantage • • Profit • • Goodwill • • Market leadership • Compatibility • • Manufacturing facility • • Existing business lines • • Complements business environment • Complexity • • Low complexity • • Ease of manufacture • Product value accepted • Relative advantage • • Protects commons • • Social and economic benefits for large numbers of citizens • • Low cost • • Low political cost • Compatibility • • Fits existing policy / legal / administrative frameworks • • Culturally acceptable • • Fills market gap or need • • Constituency or easily developed constituency • Complexity • • Easily understood • • Simple administration • Knowledge/tool has value • Relative advantage • • More credible information • • Information confers • - economic benefit • - efficiency benefit • - production benefit • - quality benefit • Compatible • • With other knowledge/tools • • Production requirements • • Organizational goals/systems • • Culture • Complexity • • Easy to understand • • Simplifies existing knowledge

21. Generic Researchable Issues and Questions Matrix for All Concepts

22. An Example of Tracking Projects Through Stages

23. EERE’s Impact Evaluation Framework

24. What we have done • Developed a generic model that incorporates a theory of change • Adapted that tool to a specific institutional setting • Laid it all out in an impact framework that users can use to model their programs and develop an impact evaluation

25. What the tool does - I • Provides a common framework for implementation and evaluation across programs in a complex multi-program environment • Moves us from simplistic, ad-hoc, and post-hoc explanations of behavior to a scientifically grounded understanding of programs • It moves logic modeling from a descriptive analytic posture to a program defining posture • Managers in a multi-program environment will share a common understanding of how programs work

26. What the tool does - II • Program managers will be better able to understand their programs and define and manage evaluations • Common language and approach should help to increase communication about programs and evaluation among all players • The model should help to detect details missing from our understanding of programs • It should focus attention on the important details rather than all the details • In multi-program environments should move us in the direction of: • Common researchable questions • Common metrics • Shared evaluations and evaluation resources

27. Contact information • John H. Reed, Principal and Owner, Innovologie, LLC., 305 Summer Garden Way, Rockville, MD 20850-2887, 301 340-8701, jreed@innovologie.com, www.innovologie.com