Measuring the impact of Egovernment Deepak T. Bhatia Sectoral and E-government Applications Practice Lead Global ICT dep

1. Measuring the impact of EgovernmentDeepak T. BhatiaSectoral and E-government Applications Practice LeadGlobal ICT department (GICT)The World Bank Deepak T. Bhatia

2. It is no longer acceptable for technology to be utilized simply for technology’s sake. Individual e-government projects need to demonstrate their contribution to overall government objectives – with the focus being on results and impact. Deepak T. Bhatia

3. Why e-Government? • Battling • bureaucracy • Improving • Efficiency • Better service • delivery • Strengthening governance and accountability Deepak T. Bhatia

4. In the World Bank the Number of Projects with ICT is growing About 150 contain ICT components (out of 250 new WB projects each year). 57% of active projects have ICT components (this is 930 projects) ICT components’ combined volume is $6.2 billion Deepak T. Bhatia

5. Analysis of e-Government Impact • World Bank commissioned study to explore impact of e-government on citizens, in collaboration with the Indian Institute of Management, Ahmedabad, India and the London School of Economics • Effort to go beyond anecdotal assessments commonly available • Sample of 5 mature, wide scope/scale projects selected for analysis from India • Chosen to represent different stages of overall IT readiness and evolution Deepak T. Bhatia

6. Learning from Past Assessments • Variety of approaches have been used for assessments -client satisfaction surveys, expert opinion • Often studies have been done by agencies that may be seen as being interested in showing positive outcome • Lack of credibility of results-different studies of the same project show very different outcomes • Lack of rigor in sampling-results can not be easily generalized • Lack of rigor in controlling for external influence-need for counterfactuals ignored. • Lack of a standard methodology-making it difficult to compare projects Deepak T. Bhatia

7. Dimensions of measurement - Stakeholders • Impact on Clients: • All measurements on the basis of a sample of clients for each major service availed by the client. Measurements would be done for electronic delivery of services as well as for the earlier mode of delivery of the same service. In cases where alternate (non electronic modes) are currently being used by the same set of users in similar contexts elsewhere, measurement would be recorded for such usage • Impact on Agencies (Including Partners in Implementation): • All measurements on the basis of data collected from agency records, a sample of employees or a group of managers. Measurements would be done for electronic delivery of services as well as for the earlier mode of delivery of the same service • Impact on Society: • Some of the questions to be framed as degree of improvements resulting from the implementation of a specific project Deepak T. Bhatia

8. Measurement Framework - Dimensions Deepak T. Bhatia

9. Proposed Framework • Focuses on retrospective assessment of e-delivery systems(B2C and B2B) • Balanced approach between case study and quantitative analysis • Recognizes that some part of the value to different stakeholders can not be monetized • Understand how inputs lead to outputs and outcomes in different project contexts • A practical methodology that can be used for designing bench mark surveys, M&E systems and prospective evaluation of projects in countries with various delivery models and paucity of data Deepak T. Bhatia

10. Methodology for Assessment • Select mature implementations , of e-delivery of services. • Collect data through structured survey from clients, employees, supervisors using counterfactuals ( for old non computerized delivery and new e-delivery system) • Customize survey instrument to each project, adapt in local language • Data can be collected through Internet survey, face to face interviews and focus groups • Determine sample frame and size so that results can be extrapolated to the entire population (often 300 clients may be sufficient). • Collect data on investments, operating costs, activity levels, revenues, employee strength from agencies Deepak T. Bhatia

11. Projects assessed • Bhoomi project. 2001. delivering two online services: issuance of records of rights, tenancy, and crop inspection register (RTC) and filing of requests for mutation (changes in land ownership) for affecting changes in land records. • Karnataka Valuation and E-Registration (KAVERI) project. 2003 delivering three key services: registration of property sale and purchase deeds, issuance of nonencumbrance certificates, and issuance of copies of previously registered deeds. • Computer-aided Administration of Registration Department (CARD). 1998. same three online services provided by KAVERI. • eSeva. 2002, These one-stop service centers now deliver 135 services from central, state, and local governments and public utilities, and are used by 3.1 million citizens a month • Ahmedabad Municipal Corporation (AMC) Civic Centers project. 2002, deliver three important services: annual payment of property taxes, issuance of birth and death certificates, and issuance of shop licenses. Deepak T. Bhatia

12. Caveat • largely relies on the survey of stakeholder perceptions of public services before and after these projects. • Thus not an impact evaluation of investments in narrowly defined ICT, which typically includes hardware, software, and communication equipment. • Rather, a review of the outcomes of packaged efforts involved in the transition from manual services to those using ICT as a key component. Deepak T. Bhatia

13. Impact: Computerized service delivery is preferable to manual Deepak T. Bhatia

14. 120 100 80 60 40 20 0 -20 -40 -60 -80 -100 Bhoomi Bhoomi KAVERI CARD eSeva AMC RTC Mutation User reports of changes in travel costs, by project (number of trips * distance travelled) Rupees per service • Bhoomi costs went up due to change in location of service centers from villages to “taluka” headquarters Deepak T. Bhatia

15. 100 90 80 70 60 50 40 30 20 10 0 Bhoomi Bhoomi KAVERI CARD eSeva AMC RTC Mutation User reports of reductions in waiting times, by projectMinutes • decreased in all five projects. • With manual systems, users often had to wait in long lines to receive services. Automation has increasedthe efficiency of document processing. Deepak T. Bhatia

16. User perception of improvement in service quality • Rating on elements such as service delivery location, accessibility, convenience, cost, transparency, and service orientation Deepak T. Bhatia

17. Users’ top four desired features of services, by project Deepak T. Bhatia

18. Proposed conceptual framework Proposed conceptual assessment framework

19. Additional conclusions • Out of the three projects where significant corruption was reported in the manual system, one project was able to eliminate corruption through computerization • Incremental operating costs for agencies per transaction are less than the reduction in the direct cost of accessing the service reported by the clients. • Negative aspect is the observed variability of impacts across different delivery centers in a project • Agencies: ability to cope with growth in transactions was enhanced • Citizens strongly supported the idea that more agencies need to be computerized, but not necessarily through government investments Deepak T. Bhatia

20. Conclusions • Undertake ex-post evaluation of successful ongoing projects. • Can be used for for ex-ante evaluation, with following assumptions • extent and adoption rate of user demand need to be estimated • Project designs and investment levels must be commensurate with anticipated benefits to users. • quality of the enabling environment should be assessed to gauge the expected impacts of planned projects • Detailed surveys at the initial stages of project planning, would help establish a baseline. Deepak T. Bhatia

21. Lessons Learned • IT cannot be a goal in itself - projects must have a direct link to process reform • E-Government is a long term endeavor: not a silver bullet, not for immediate gains • ‘E’ stands for ‘Everything-but-IT’: institutional change, political support, capacity development.. technology is the easy part • Ultimately can generate significant cost and efficiency gains But requires… Deepak T. Bhatia

22. Focusing on results Deepak T. Bhatia

23. Impact Study team Subhash Bhatnagar , IIM, Ahmedabad, India (Primary Author) Jiro Tominaga, WB Deepak Bhatia, WB Shirin Madon, London School of Economics Presentation research Asheeta Bhavnani, WB Mikhail Bunchuk, WB Reviewers David L. Cieslikowski Christine Zhen-Wei Qiang Credits Deepak T. Bhatia

24. Annex Deepak T. Bhatia

25. How have governments been measuring results of e-government investments? • France – Mareva - A Method of Analysis and Value Enhancement • Germany -WiBe - Economic Efficiency Assessment Methodology • USA – Performance Reference Model Framework • Australia: Demand and Value Assessment Methodology • All these government methodologies broadly focus on the same dimensions. • They offer two levels of impact assessment: • first, in terms of how projects provide a business case justification for expenditure and whether they meet the targets set for them, • and second, in terms of how projects meet the goals of the agencies concerned and, in turn, how that helps achieve wider government strategies. Deepak T. Bhatia

26. World Bank and Impact measurement experience • What is the WBs experience with measuring impact? • Independent evaluation group - IEG measures impact of Bank programs and projects, and draws lessons from experience. The Bank also has a large program of impact evaluation although there may not be any that have centered around the ICT sector. • Whether WB prescribes ONE methodology to all countries? How much tailoring is needed? • IEG generally uses objective based methodology to measure impact. But if this question refers to tailoring strategies then the key finding from IEG work is that strategies including ICT must be tailored to country circumstances. Deepak T. Bhatia