The DEA assessment procedure guideline

1. The DEA assessment procedure guideline Assessment of development impacts from energy interventions Example: Wind power system development ≠ the ”physical” wind power system development = improvements among energyusers from access to the services enabled by the wind power system improvement = e.g. swifter/cheaper production or consumption, higher incomes, reduced drudgery, higher livingstandards, better health

2. development ”impact” formalized (for now) Agents (region, village, households, individuals) Behaviors and practices over a period of time Outcome for the agent(s) or other agent(s) The difference between outcomes is the impact we seek. Impact Intervention/ program/project Agents (other/later) Modified behaviors and practices over a period of time Modified outcome for the agent(s) or other agent(s)

3. Impact from hypothetical wind powered irrigation project Agent: Rural villagers Practice: Rainfed agriculture Outcome: Harvest size/quality/ diversification Impact The difference in harvest size/quality/-diversification is an impact Wind powered irrigation project Agent: Rural villagers (other/later) Modified practice: Irrigated agriculture Modified outcome: Greater/improved/more diversified harvests

4. Mediating circumstances E.g. location- or time-specific rainfall, temperatures, ínfrastructure functionality, customs/traditions, prices Mediating circumstances Agent: Rural village Practice: Rainfed agriculture Outcome: Harvest size/quality/ diversification Harvest sizes/quality/diversification affected by mediating circumstances? How assess the impact from the intervention? Attribution problem Impact Agent: Rural village (other/later Modified practice: Irrigated agriculture Modified outcome: Greater/improved/more diversified harvests Mediating circumstances Wind powered irrigation project

5. General ambition Empirically assess a causal linkage from the technological input of a given energy intervention to observable development results via the utlization of the (improved) energy services provided by the intervention while keeping an eye on mediating processes

6. Causality challenges (1) • Energy does not in itself e.g. quench, feed, house, or clothe people Energy services may be a vital input into meeting those needs => long and complex chain of causality • Energy services often bring about improvements in many aspects of life – difficult to capture or conceive of all such improvements • Impacts of energy accessmay take many years to become manifest

7. Causality challenges (2) The type/range of energy services made available varies across energy interventions (different services from wind driven water pumps, solar homes, improved stoves, solar geezers, MFPs) Households, enterprises, or institutions differ in • abilities and willingness to invest in new technologies • energy – related preferences, traditions and behaviors • Existing demand patterns vary geographically across regions, districts, settlements within districts, and users within settlements • Impacts that are observed in one context do not necessarily appear in another

8. Key requirements to assessment methodology One –size–fits–all approach not possible Need for • flexible approach/methodology – adaptable to different types of interventions and contexts • knowledge of ”realities on the ground” – energy services needed and used among the specific end-users affected by the intervention • systemize interventions into inputs brought in and observable results – causes, effects, and circumstances that intervene • Draw on methodology used for interventions in other sectors, that has been adapted to energy by a group of experts assembled by GVEP.

9. Assessment procedure Design stage Focus, conceptualization, and modeling • Identify the stakeholders of your project and their IA needs. • Define/delimit the intervention and your focus on the intervention. • Illustrate and model the project with results and chains of causality. Bridge model to the real world situation • Choose observable indicators for all elements in the results chains. • Specify the appropriate data collection methods for each link. • Construct a research plan and discuss it with your stakeholders. Implementation stage • Collect data. • Analyze the data. • Draw conclusions and write report. • Present the results to your stakeholders.

10. The implementation stage 7. choice of data collection methods – contingent on study-specific circumstances in the design stage 8. data analysis methods and types of conclusions depend on data collection methods. 9.–10. means of presentation to stakeholders - also case specific

11. Where can we provide universally applicable guidance? Design stage Focus, conceptualization and modeling • Identify the stakeholders of your project and their IA needs. • Define/delimit the intervention and your focus on the intervention. • Illustrate and model the project with results and chains of causality. Bridge model to reality • Choose indicators for all elements in the causality chains. • Specify the appropriate data collection methods for each link. • Construct a research plan and discuss it with your stakeholders. Implementation stage • Collect data. • Analyze the data. • Draw conclusions and write report. • Present the results to your stakeholders.

12. Conceivable stakeholders domestic public authorities share holders, financial partners donors donor country public authorities beneficiaries/users/clients project management team women’s groups community based organisations researchers in academia the impact assessor him-/herself Factors that affect the assessment design: what kind of information different stakeholders require the intended use of that information how stakeholders would like the information communicated to themselves and other stakeholders Stakeholders

13. General format guidance possible Design stage Focus, conceptualization and modeling • √ Identify the stakeholders of your project and their IA needs. • Define/delimit the intervention and your focus on the intervention. • Illustrate and model the project with results and chains of causality. Bridge model to reality • Choose observable indicators for all elements in the causality chains. • Specify the appropriate data collection methods for each link. • Construct a research plan and discuss it with your stakeholders.

14. Intervention – define, delimit and focus Does the intervention: • provide/improve energy utilization for one or several purposes? • involve one or several types of technology? Single technology, single purpose - wind driven water pump for irrigation Single technology, multi-purposes - rural solar PV charge stations offering battery charging services for several types of equipment Multi-technology, single purpose – Solar PV and diesel powered water pumps for rural irrigation Multi-technology, multi-purposes - grid electrification: lighting, communication water access, refrigeration and entertainment for homes, agricultural sector, health sector and schools. Which are your stakeholders’ main areas of interest?

15. General format guidance possible Design stage Focus, conceptualization and modeling • √ Identify the stakeholders of your project and their IA needs. • √Define/delimit the intervention and your focus on the intervention. • Illustrate and model the project with results and chains of causality. . Bridge model to reality • Choose observable indicators for all elements in the causality chains. • Specify the appropriate data collection methods for each link. • Construct a research plan and discuss it with your stakeholders.

16. Illustration and modeling – the “results chain” terminology • Inputs: the material, financial, human, and social resources used • Activities: actions taken in order to produce specific outputs • Outputs: resultant products, capital goods and services (domestic, collective or institutional) • Outcome: short- and medium-term effects of an intervention’s outputs • effect - a “change, intended or unintended, due - directly or indirectly - to an intervention” • Impacts: positive and/or negative long-term effects, subject to greater influence of mediating circumstances than outcomes (inverse “log–frame”)

17. Revised terminology Outcome: Agents practicing irrigated agriculture Output: Irrigation services Impact:: Higher agricultural income (”further downstream”) Input: Wind power equipment Mediating circumstances

18. Alternative application of results chain : solar home systems (focus: income generation) Input Solar home system components + + Sales, distribution, and installation of SHS components Activity Output Installed solar home systems Use of output Lighting Income generating activties in the evening Outcome Impact Income poverty reduction

19. General format guidance possible Design stage Focus, conceptualization and modeling • √ Identify the stakeholders of your project and their IA needs. • √Define/delimit the intervention and your focus on the intervention. • √Illustrate and model the project with results and chains of causality. Bridge model to reality • Choose observable indicators for all elements in the causality chains. • Specify the appropriate data collection methods for each link. • Construct a research plan and discuss it with your stakeholders.

20. Indicators A measurable key characteristic of the agents, their behaviour, or their circumstances, that can be related to the intervention’s desired outcome. Operational objective of the assessment: Capture differences in indicator values between agents that have experienced the intervention and those that have not.

21. Choosing indicators – some considerations • Pertinence to your project • Pertinence to national or regional (aggregate) development objectives • Interest for project stakeholders • Ease and cost of measurement or data collection • Possibilities for triangulation between sources of information Information format - not necessarily numeric. Other examples; • pictures • videos • voice recordings • change in behavior or attitudes

22. Solar home systems with indicators: focus on income generation Solar home system components - no of solar panels - no of inverters Input + + Sales, distribution, and installation of SHS components - no of SHS sold - no of SHS installed Activity Installed solar home systems - no of SHS installed - no of SHS operational Output - hours of lighting - no of light bulbs used Use of output Lighting Income generating activties in the evening - evening working hours - goods/services production Outcome Impact Income poverty reduction - change in household income

23. General format guidance possible Design stage Focus, conceptualization and modeling • √ Identify the stakeholders of your project and their IA needs. • √Define/delimit the intervention and your focus on the intervention. • √Illustrate and model the project with results and chains of causality. Bridge model to reality • √Choose indicators for all elements in the causality chains. • Specify the appropriate data collection methods for each link. • Construct a research plan and discuss it with your stakeholders.

24. Data collection methods Assessment depends crucially on data quality! Each indicator is ideally measured using more than on data collection methods, e.g.: • Physical measurement (satellite data on forest cover) • Data extraction from public statistics (school attendance or agricultural production) • Interviews • Extraction of accounting or administrative data of a public or private organization (ESCO customer records) • Focus groups and other participative methods • Household or firm surveys Often combinations of methods!

25. General format guidance possible Design stage Focus, conceptualization and modeling • √ Identify the stakeholders of your project and their IA needs. • √Define/delimit the intervention and your focus on the intervention. • √Illustrate and model the project with results and chains of causality. Bridge model to reality • √Choose indicators for all elements in the causality chains. • √Specify the appropriate data collection methods for each link. • Construct a research plan and discuss it with your stakeholders.

26. General format guidance possible Design stage Focus, conceptualization and modeling • √ Identify the stakeholders of your project and their IA needs. • √Define/delimit the intervention and your focus on the intervention. • √Illustrate and model the project with results and chains of causality. Bridge model to reality • √ Choose indicators for all elements in the causality chains. • √ Specify the appropriate data collection methods for each link. • √ Construct a research plan and discuss it with your stakeholders. Implementation stage • Collect data. • Analyze the data. • Draw conclusions and write report. • Present the results to your stakeholders.

27. Concluding remarks Impact assessments of energy interventions are possible – but challenging!! • Huge variety in types of energy interventions • Multitude of resultant energy services • Complex demand side – difficult to trace development impacts. • Need for systemizing intervention and consequences; modelling and design requires creativity, analytical skills, and knowledge of the reality on the ground. • Collection of high quality data and dissemination – possibly the most difficult components!