1 / 22

A DESIGN OF THE METAEVALUATION MODEL 16. May. 2000 Keun-bok Kang, Professor

A DESIGN OF THE METAEVALUATION MODEL 16. May. 2000 Keun-bok Kang, Professor Chungnam National University (CNU), Taejon, Korea Tel & Fax : 82-42-821-5845 ; E-mail : kbkang@cnu.ac.kr Chan-goo Yi, Principal Researcher Electronics and Telecommunications Research Institute (ETRI), Taejon, Korea

jshackelford
Télécharger la présentation

A DESIGN OF THE METAEVALUATION MODEL 16. May. 2000 Keun-bok Kang, Professor

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. A DESIGN OF THE METAEVALUATION MODEL 16. May. 2000 Keun-bok Kang, Professor Chungnam National University (CNU), Taejon, Korea Tel & Fax : 82-42-821-5845 ; E-mail : kbkang@cnu.ac.kr Chan-goo Yi, Principal Researcher Electronics and Telecommunications Research Institute (ETRI), Taejon, Korea Tel : 82-42-860-6924 ; Fax : 82-42-861-6880 ; E-mail : yicg@etri.re.kr Keun-bok Kang (CNU) & Chan-goo Yi (ETRI) ; Taejon, Korea

  2. CONTENTS 1. Introduction 2. Definition of Metaevaluation 3. Previous Researches on the Metaevaluation 4. Building the Metaevaluation Model 4-1. Component Factors of Metaevaluation 4-2. Design of the Metaevaluation Model Keun-bok Kang (CNU) & Chan-goo Yi (ETRI) ; Taejon, Korea

  3. 1. Introduction ▣ Purpose of this study - designing of metaevaluation model that can be used in practice - in order to achieve this : • review the previous approaches to metaevaluation (especially the component factors of metaevaluation) • identify the component factors of metaevaluation • suggest the metaevaluation model Keun-bok Kang (CNU) & Chan-goo Yi (ETRI) ; Taejon, Korea - 1 -

  4. 2. Definition of Metaevaluation ▣ usually metaevaluation is understood as evaluation of evaluation ▣ according to how they understand the nature and the purpose of policy evaluation, they define the metaevaluation differently ▣ policy evaluation : - (purpose of policy evaluation) produce valid and useful policy relevant information to stakeholders, to decide in making the policy or changing the existing policy, to improve the policy implementation, to identify the responsibility of policy process participants Keun-bok Kang (CNU) & Chan-goo Yi (ETRI) ; Taejon, Korea - 2 -

  5. ▣ Definition of metaevaluation : - evaluation on the evaluation and evaluation system, during or after the evaluation, in terms of evaluation paradigm, evaluation resources, evaluation process, evaluation performance and evaluation utilization ▣ Purpose of metaevaluation : - to value the evaluation(system) : summative evaluation - to improve the evaluation(system) : formative evaluation - to promote the utilization of evaluation : summative / formative evaluation Keun-bok Kang (CNU) & Chan-goo Yi (ETRI) ; Taejon, Korea - 3 -

  6. 3. Previous Researches on the Metaevaluation (expecially the components of metaevaluation) 3-1. metaevaluation is an evaluation which examines the value and merits of policy evaluation itself (Stufflebeam, 1981) - when policy evaluation is understood as examining the value and merits of policy activities - metaevaluation is a process of classifying, acquiring and utilizing descriptive and perceptive information related effectiveness, actuality, ethicality and technical appropriateness of certain evaluation activities - the purpose of metaevaluation is to be a guide for evaluation activities, and openly report upon the merits and demerits of evaluation itself Keun-bok Kang (CNU) & Chan-goo Yi (ETRI) ; Taejon, Korea - 4 -

  7. 3-2. metaevaluation is examining the validity of the evaluation design, data collection and analysis - when the purpose of evaluation is understood as producing valid information for policy results - metaevaluation is focused on the technical or structural adequacy of evaluation (for example, Cook & Gruder, 1978 ; Chelimsky, 1987 ; Smith & Hauer, 1990 ; Greene, 1992) Keun-bok Kang (CNU) & Chan-goo Yi (ETRI) ; Taejon, Korea - 5 -

  8. 3-3. metaevaluation is assessing the usefulness of the evaluation - when the purpose of evaluation is understood as providing useful information to stakeholders (policymakers, decision makers, program managers, clients, etc) - metaevaluation is focused on assessing the evaluation utilization (for example, Mackay, 1992) Keun-bok Kang (CNU) & Chan-goo Yi (ETRI) ; Taejon, Korea - 6 -

  9. 3-4. metaevaluation is checking the reliability of the evaluative judgements(evaluation results), and the evaluation process that ensure the reduction of the bias - when evaluation is understood as making unbiased evaluative judgements of merits or worth of the policy results - for example, Scriven suggests his ‘Metaevaluation Checklist’ (not identified, quoted from Rogers, 1995) Keun-bok Kang (CNU) & Chan-goo Yi (ETRI) ; Taejon, Korea - 7 -

  10. 3-5. Larson & Berliner's Standards ▣ Larson & Berliner(1983) classified the three component factors of metaevaluation, and further divided each factor into detailed evaluation items - factors of evaluation input • resources and techniques used in evaluation, environment - factors of evaluation process • actual evaluation activities carried out in accordance with evaluation plan - factors of evaluation outcome • number of decisions affected by the evaluation Keun-bok Kang (CNU) & Chan-goo Yi (ETRI) ; Taejon, Korea - 8 -

  11. 3-5. Larson & Berliner's Standards (continued) - this research was the first systematic approach to metaevaluation - but, they did not concern the evaluation purposes and types - they suggested 23 evaluation items for metaevaluation, but some items were overlapped and the classification of items is somewhat inappropriate - they had no little concern on the rational and due process Keun-bok Kang (CNU) & Chan-goo Yi (ETRI) ; Taejon, Korea - 9 -

  12. 3-6. The Joint Committee on Standards for Educational Evaluation (1994) - developed 30 principles(standards) for metaevaluation. • they were grouped as such : Utility, Feasibility, Propriety, Accuracy - these standards are a useful checklist for metaevlauation, but the committee does not suggest a satisfactory framework for metaevaluation (little guidance on how to plan evaluation in order to meets these standards) - it is needed that these standards to be checked for applicability in other countries and other types of program (Rogers, 1995) Keun-bok Kang (CNU) & Chan-goo Yi (ETRI) ; Taejon, Korea - 10 -

  13. 3-7. Patricia Rogers' Metaevaluation Framework (1995) ▣a framework for evaluating evaluation in terms of their intended and actual impact on the implementation of existing programs ▣her metaevaluation framework consists of : - 5 evaluation criteria for intermediate outcomes of program evaluation • producing valid information • providing useful information to decision-makers • producing unbiased judgements of merit or worth • involvement & illumination of relevant stakeholders • empowerment of the intended program clients Keun-bok Kang (CNU) & Chan-goo Yi (ETRI) ; Taejon, Korea - 11 -

  14. 3-7. Patricia Rogers' Metaevaluation Framework (1995) (continued) - evaluation's impact on processes of implementing existing programs or replacing existing programs with alternatives - evaluation's contribution to the development of programs which meets need ▣ this framework is very systematic, but the evaluation object is limited to the existing programs which is implementing Keun-bok Kang (CNU) & Chan-goo Yi (ETRI) ; Taejon, Korea - 12 -

  15. 4. Building the Metaevaluation Model 4-1. Component Factors of Metaevaluation -evaluation paradigm - evaluation resources - evaluation process - evaluation performance - evaluation utilization Keun-bok Kang (CNU) & Chan-goo Yi (ETRI) ; Taejon, Korea - 13 -

  16. ① Evaluation Paradigm - evaluation purpose : rationality - evaluation type : appropriateness - evaluation object : suitability • different level and range of evaluation object may lead different evaluation results Keun-bok Kang (CNU) & Chan-goo Yi (ETRI) ; Taejon, Korea - 14 -

  17. ② Evaluation Resources - evaluation manpower • quality and quantity of evaluators • user involvement - evaluation organization • structural and functional appropriateness of evaluation organization - evaluation budget and information • appropriateness of evaluation budget • quality(adequacy, reliability, etc) and quantity of information Keun-bok Kang (CNU) & Chan-goo Yi (ETRI) ; Taejon, Korea - 15 -

  18. ③ Evaluation Process - evaluation procedure : objectivity and fairness - timing of evaluation : fitness to the evaluation type etc. - evaluation methodology : accuracy and validity - evaluation criteria : appropriateness - evaluation indicators : rationality Keun-bok Kang (CNU) & Chan-goo Yi (ETRI) ; Taejon, Korea - 16 -

  19. ④ Evaluation Performance - evaluation outcome : validity - evaluation information : usefulness Keun-bok Kang (CNU) & Chan-goo Yi (ETRI) ; Taejon, Korea - 17 -

  20. ⑤ Evaluation Utilization ▣preparation and dissemination of evaluation report - clarity and impartiality of report - timeliness and dissemination of report ▣utilization of evaluation results - instrumental utilization • contribution to improving the implementing existing program or to changing the existing program, to developing the new program - conceptual utilization • enlightenment of policy Keun-bok Kang (CNU) & Chan-goo Yi (ETRI) ; Taejon, Korea - 18 -

  21. 4-2. Design of the Metaevaluation Model component factors of metaevaluation items of metaevaluation & criteria rationality purpose evaluation paradigm type appropriateness object suitability (level and range) quality / quantity manpower user involvement evaluation resource structural and functional appropriateness organization budget appropriateness quantity and quality (adequacy, reliability, etc) information Keun-bok Kang (CNU) & Chan-goo Yi (ETRI) ; Taejon, Korea - 19 -

  22. (continued) component factors of metaevaluation items of metaevaluation & criteria objectivity and fairness procedure evaluation process timing fitness to evaluation type etc. accuracy and validity methodology appropriateness criteria indicator rationality evaluation performance validity outcome usefulness information clarity and impartiality report timeliness and dissemination evaluation utilization Improving & changing the existing program Instrumental utilization developing the new program conceptual utilization enlightment of policy Keun-bok Kang (CNU) & Chan-goo Yi (ETRI) ; Taejon, Korea - 20 -

More Related