Seminar on Performance Budgeting and Fiscal Transparency, Tangier, Morocco, April 21-23, 2009

1. Seminar on Performance Budgeting and Fiscal Transparency, Tangier, Morocco, April 21-23, 2009 Session 8. Monitoring and Evaluation: Challenges and Issues Nowook Park (npark@kipf.re.kr) Center for Performance Evaluation and Management Korea Institute of Public Finance

2. Contents Impactof performance budgeting in Korea 1 Roles of the MOF, line ministries, parliament, and audit office 2 3 Performance Information : Indicators and Evaluations 4 How to motivate performance? How to engage politicians? 5

3. 1. Impact of Performance Budgeting in Korea

4. Observations from Program Review Process • Evaluation results • Quality of performance information has not improved much • Programs are showing better results • Link between evaluation results and budget • Evaluation results are utilized at every stage of budget process • Moving away from incremental budgeting • Evaluated programs are subject to bigger budget change compared to other programs

5. Evaluation Results byTotal Score

6. Evaluation Results by Ratings

7. Evaluation Results by Section

8. Utilization of Evaluation Results • MOSF encouraged ministries/agencies to use the results in reshuffling budget allocation • MOSF announced at least 10% budget-cut would be done to “ineffective” programs, in principle • MOSF submitted evaluation results to the National Assembly upon their request • Evaluation results are open to public since 2006

9. Use of Performance Information by Agencies (2005)

10. Use of Performance Information by MOSF (2005)

11. Use of Performance Information by Legislature (2005)

12. Link between Evaluation and Budgeting (2005) ( Unit : 100,000won, % )

13. Link between Evaluation and Budgeting (2006) ( Unit : 100,000won, % )

14. Link between Evaluation and Budgeting (2007) ( Unit : 100,000won, % )

15. Moving Away from Incremental Budgeting Programs have been subject to larger budget change after evaluation Coefficient of Variation in Funding Change (Excluding Programs of which funding change is greater than 200%)

16. Cultural Changes among Line Ministries • Budget allocation based on performance is spreading among line ministries • Utilization of SABP results in budget requests • Changing practices of program management • Evaluation activities become active among line ministries • Need for program evaluations are recognized and accepted among line ministries

17. 2. Roles of the MOF, line ministries, parliament, and audit office

18. Ministry of Strategy and Finance (MOSF) • Evaluate line ministries’ program • Issuing guidelines on performance-based budgeting to line ministries/agencies • Evaluating their program’s performance • Use performance information in budget formulation • Encouraging line ministries/agencies to use performance evaluation results in preparing their budget requests • Incorporating the performance information into its decisions during budget formulation

19. Line ministries/agencies • Producing performance information in compliance with central budget authority’s initiative • Submitting strategic plans, annual performance plans and performance reports • Evaluate their programs based on SABP checklist • Conduct program evaluation • Use performance information in budget request

20. The National Assembly • Before the enactment of National Finance Act, there had not been official role of the National Assembly • They had requested evaluation results for budget deliberation on an ad hoc basis • The National Assembly receives annual performance plan (from 2008) and report (from 2009) with budget documents • The National Assembly Budget Office intends to analyze budget in connection with performance information

21. The National Audit Office • Before the enactment of National Finance Act, there had not been official role of the National Audit Office • As an independent audit office within the Administration, it has not played any official role • However, they examined the operation of performance budgeting system and produced report • The National Audit Office is assigned with the role of verifying annual performance report from 2009

22. 3. Performance Information : Indicators and Evaluations

23. Performance Indicators Status • Slight increase in outcome measures but there are big room for further improvement • Outcome performance measures increase by 3.2 percentage points, while output performance measures decrease by 4.1 percentage points • More than 40% of programs do not have relevant performance indicators in SABP

24. Performance Indicators Examples Recommendation: Using outcome measure ex) Enrollment rates: The postsecondary enrollment gap between low- and high-income high school graduates will decrease each year. Recommendation : Using ‘actual value of attainment’ instead of ‘percentage of achievement’. (FY 2008 manual of KPART)

25. Performance Indicators Problems • Difficulties to develop outcome measures in some programs • Program producing results after long period • Using milestone indicators may help • However, using different evaluation cycle needs to be considered • Successful management of program may not change outcome measures • Reduce scope of outcome measures • Particular program may not worth evaluating • Data do not exist • Investment in data is necessary • Compare cost and benefit of producing new data

26. Performance Indicators Lessons • It is desirable to develop outcome measures, but it is not possible for every program • Be aware of some exceptions and apply it flexibly according to program type

27. Program Evaluations Status • Self-Assessment of Budgetary Program • Review based on the checklist • Among the checklist, there is a question whether program evaluation is conducted or not • It encourage line ministries to conduct program evaluation at least once in three years • In-depth Program Evaluation • In-depth evaluation for selected programs by the central budget authority

28. Program Evaluations Status - KPART Q. 3-1. Did you conduct comprehensive program evaluation objectively? • Has your program been evaluated in depth using verifiable data ? • Has your program evaluation been conducted by independent organization ? ex) external institution, audit office, internal evaluation expert, etc. • Does evaluation cover important issues of program ? Note) ‘yes’ in case that evaluation is in progress or conducted within 3 years latest YES !

29. Program Evaluations Problems • Little experience with program evaluation among program managers • Lack of data to prohibits meaningful program evaluation • Lack of fund to conduct program evaluation • Difficulties in maintaining independence of evaluators • Trade off between independence and expertise in the society where social ties are strong and dense

30. Program Evaluations Example : Bad Promotion of the intellectual property (IP) Result : Couldn’t find the evidence for the regional impact of the IP center ! Obstacle • Insufficient Data - absence of data which is needed to conduct the evaluation - using customer satisfaction (or number of counseling) instead of the number of intellectual property in each localities • Lack of understanding on program evaluation among program managers

31. Program Evaluations Example : Good Rural development program Result of Effectiveness Find the evidence for the regional impact! - There was positive relation between the program’s spending and the infrastructure - In addition, the infrastructure created value-added business Result of Management Find the problems of the management system! - The management system is not developed and the each manager’s autonomy is limited

32. Program Evaluations Lessons • Plan ahead for data collection • Data should go hand in hand with program management • Data management is a part of program management • Be creative in securing fund for evaluation • Evaluation should be part of program management • In budget negotiation, evaluation plan should be presented

33. 4. How to motivate performance? How to engage politicians?

34. Institutionalized Incentives for line ministries How performance information will be used in budget preparation is stipulated in guidelines for budget preparation In principle, 10% budget cut for ineffective program is encouraged to line ministries It draws attention from program managers but not from high ranking decision makers Results from SABP is used as a part of evaluation information for the whole department Now high ranking decision makers pay attention But their attention is focused more on getting good scores than on improving programs’ performance

35. Forms of Information does Matter Performance information (performance indicator) at monitoring level has not been able to draw attention from decision makers They are internally useful information, but not utilized by central budget authority Remains to be seen how they will be utilized by involved parties Performance indicators are good starting point for communication rather than decision making Performance information (program ratings) from program reviews are actively used by central budget authority It summarizes various information into simple ratings Power of Simplicity!

36. Politician’s Engagement Although there is no formal mechanism for politician’s engagement, politicians show interest in performance information They requests performance information on an ad hoc basis There has not been much suspicion on the evaluation results from the Administration so far It remains to be seen whether politicians will use performance information more systematically with the submission of annual performance plan and report to the National Assembly, Publicizing performance information may help to force politicians to pay attention to PI

37. Thank You !