1 / 17

Gabriela Pérez Yarahuán Universidad Iberoamericana

Exploring Politics, Accountability and Evaluation Use in the Mexican Federal Government Education Programs. Gabriela Pérez Yarahuán Universidad Iberoamericana. Motivation. Late 1990’s – 2000’s political changes leading to democratization in Mexico.

ellison
Télécharger la présentation

Gabriela Pérez Yarahuán Universidad Iberoamericana

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Exploring Politics, Accountability and Evaluation Use in the Mexican Federal Government Education Programs Gabriela Pérez Yarahuán Universidad Iberoamericana

  2. Motivation • Late 1990’s – 2000’s political changes leading to democratization in Mexico. • Late 1990’s, changes in social policy. National large scale poverty program (PROGRESA) was implemented and evaluated. • Since the late 90’s several institutions for accountability, transparency, evaluation are created. • Since the early 2000’s social programs evaluated once a year. • ….so? • How can we study the effects of these institutions on policy/programs?

  3. Accountability Performance Evaluation

  4. Evaluation use Types of use Determinants of use Evaluationprocess Quality and credibility of evaluation studies Communication and publication practice Applicability and relevance of findings Timeliness Unit responsible for evaluation Externalfactors Legal framework Political climate Availability of other sources of information Personal characteristics of policy decision makers Commitment or receptiveness to evaluation. • Symbolic • Conceptual • Instrumental (Weiss 2008; Chelimsky 2006; Rossi and Freeman 2004; Pitman, Feinstein, and Ingram 2005; Cousins and Leithwood 1986; Patton 1997; Weiss 1999; Marra 2000; Balthasar's 2008)

  5. ProgramEvaluationSystem in MexicoTwoperiods 2001-2006 2007-2009 Aspects such as the communication and publication practice of evaluation results are present, as well as the intention to make evaluations have more applicability and relevance of findings. LGDS and LFPRH First Annual Evaluation Program was issued in 2007. “Consistency and Results Evaluation” . Mechanism to follow up on recommendations. Specific Performance Evaluation implemented. • Official norms, guidelines and laws were vague, not explicit and at most ambiguous regarding evaluations objective, content and use. • Except for the references of their use in the Budget bill discussions, there is an absence, during this period, of specific mechanisms to incorporate evaluation in the policy process regarding the improvement, change, modification of social programs and policies.

  6. Methods • In order to investigate on the instrumental use of evaluations: • 9 programs were selected with variability of budget, agency, year of creation. • Four categories of changes were established. • General Objective • Target Population • Selection criteria • Type of benefits provided • Operation Rules were analyzed to count changes in each of the categories for each year of program operation. • Evaluation were analyzed to count reccomendations on each of the categories established.

  7. Methods II Survey • In order to understand how the evaluation studies are used by public officials in charge of program operation or in evaluation areas in federal agencies, this research also included a survey of evaluation use applied to public officials. • The survey was designed to measure perception on types of evaluation use and the variables that may affect evaluation use. • The survey was voluntarily and anonymous and was sent to 293 public officials in 19 agencies of the federal government.

  8. Results

  9. The total count of changes registered were 111, with the highest percentage (37%) of those being in the Selection Criteria of beneficiaries. The total count for recommendations was 80. The highest percentage of recommendations was in the Target Population category (36%). The spearman correlation coefficient between changes and recommendations is high for the Target Population category (0.812), and for Benefits (0.667); moderate for General Objective (0.432) and rather low for Selection Criteria (0.098). • The three programs with the greatest amount of recommendations and change, as can be seen in the figure are FONART, OPORTUNIDADES and LICONSA, all three from the Department of Social Development. • Changes in ORs seem to have a moderate to low negative correlation with size of budget (-.319), while the opposite happens with recommendations with a moderate positive correlation (0.439).

  10. Survey • The effective rate of response was 19% (56) with respondents from 16 of the 19 agencies. • 54% of respondents belonged to operation units of programs, and 46% to evaluation units. • Their demographics showed that 54% of respondents were men, and the average age was between 46 and 55 years. • The agency with the greatest group of respondents was Social Development with 21% of the sample. • The sample had a wide spectrum of administrative level officials, being the greatest group a middle level manager (Director de Area). • The average level of years of experience in their current position was between 3 and 5 years.

  11. The great majority of respondents (88%) thought that evaluations had some type of use for programs or agencies. • A majority of respondents (79%) mentioned impact evaluation as the most useful type of evaluation. When asked to rank (on a scale from 1 to10) most common uses, the results were as follows: • To provide information about a program´s design, process and impact: 8.2 • To modify Operating Rules: 8.2 • For accountability: 7.2 • To position the program to public opinion: 6.4 • To negotiate budget: 5.2 • When asked to rank users of evaluations, the three best ranked actors were: CONEVAL (8.0), Evaluation Units within agencies (7.8), Directors of Programs (7.2); the three worst ranked users were Congress (5.2), Beneficiaries (4.6) and Citizens (4.0).

  12. In a scale from 1 to 10, respondents ranked the following elements for their importance in the use of evaluations. • Qualifications of evaluator: 8.9 • Quality and adequate communication of findings: 8.7 • Receptiveness to evaluation: 8.6 • Clarity in evaluation design: 8.5 • Involvement in evaluation design by program directors: 8.1 • Legal framework for evaluation use 7.8 • Organizational climate 6.6 • Political external factors: 6.2

  13. Final Comments • The relationship between accountability institutions and government performance must be explored in detail. Evaluation and its use play an important role. • Mexico shows a clear development towards a more structured and coherent set of rules regulating evaluation making and its use. • This research makes use of small sample of programs and a survey of perception on evaluation use. With this data we observe a rinsing pattern on instrumental evaluation use. • The results of the survey confirm the idea that evaluation appears to be highly ranked as an instrument for government change. • More research is necessary to asses the effectiveness of the system to produce the necessary changes to increase government performance.

More Related