Good Morning
Programme Evaluation in Health Care
Contents • Introduction • Focus of evaluation • Types of evaluation • Elements of evaluation • Design of evaluation • Constraints on using results of evaluation • Evaluation of dental health programme • Economic evaluation • Summary and conclusion • References
Introduction • Evaluation has been defined as a systematic, rigorous, and meticulous application of scientific methods to assess the design, implementation, improvement or outcomes of a program. • It is the comparison of actual impacts against strategic plans.
Evaluation is mostly concerned with the final outcome and with factors associated with it. • While monitoring is confined to day to day or on-going operations.
The purpose of evaluation Evaluation allows us to • Measure the progress of each activity • Measure the effectiveness of each activity • Identify problems in carrying out the activities • Plan revision and modification and • Justify the costs of administering the programme
In the words of WHO expert committee on National Health planning in developing countries, evaluation measures; • The degree to which objectives and targets are fulfilled and the quality of the results obtained. • The productivity of available resources in achieving clearly defined objectives. • How much output or cost effectiveness is achieved.
Focus of evaluation • Evaluation studies ultimately focus on the goals, objectives, or intent of the program or activity being studied.
Donabedianrightly sees structure, process, and outcomes as inextricably linked.
Types of Evaluation Scriven ; two types. • Formative Evaluationand • Summative Evaluation
Formative evaluation • Refers to the internal evaluation of a program. • It is an examination of the processes or activities of a program as they are taking place. • It is usually carried out to aid in the development of a program in its early phases.
Example; a fluoride rinse program • Para-professionals are trained to administer three types of fluoride rinses under a strict sequence of procedures. • After 3 days of operation observed to determine the extent of adherence to a strict sequence of procedures. If the sequence is incorrect, formative evaluation allows the program to make remedial changes at that point.
Summative evaluation • Summative evaluation, by contrast, judges the merit or worth of a program after it has been in operation. • It is an attempt to determine whether a fully operational program is meeting the goals for which it was developed.
Elements of Evaluation • Relevance • Adequacy • Accessibility • Acceptability • Effectiveness • Efficiency • Impact
1. Relevance • Relevance or requisiteness relates to the appropriateness of the service, whether it is needed at all. • If there is no need, the service can hardly be of any value. • For example, vaccination against smallpox is now irrelevant because the disease no longer exists.
2. Adequacy: • It implies that sufficient attention has been paid to certain previously determined courses of action. • For example, the staff allocated to a certain programme may be described as inadequate if sufficient attention was not paid to the quantum of work-load and targets to be achieved.
3. Accessibility • It is the proportion of the given population that can be expected to use a specified facility, service, etc. • The barriers to accessibility may be physical (e.g., distance, travel, time); economic (e.g., travel cost, fee charged); or social and cultural (e.g., caste or language barrier).
4. Acceptability: • The service provided may be accessible, but not acceptable to all, e.g., male sterilization, screening for rectal cancer.
5. Effectiveness: • Effectiveness has been defined as ‘the ratio between the achievement of the programme activity and the desired level of achievement proposed by planners. • It is the extent to which the underlying problem is prevented or alleviated.
Resources are evaluated by dividing the actual expenditure of resources by the planned expenditure. • Activities are evaluated by dividing actual performance by planned performance. • Objectives are evaluated by dividing actual attainment by planned attainment.
6. Efficiency • Efficiency has been defined as the ratio between the result that might be achieved through expenditure of a specific amount of resources and the result that might be achieved through a minimum of expenditure (WHO 1974). • It is a measure of how well resources (money, men, material and time) are utilized to achieve a given effectiveness.
7. Impact • It is an expression of the overall effect of a programme, services or institution on health status and socio-economic development. • For example, as a result of malaria control in India, not only the incidence of malaria dropped down, but all aspects of life-agricultural, industrial and social-showed an improvement.
If the target of 100 per cent immunization has been reached, it must also lead to reduction in the incidence or elimination of vaccine preventable diseases. • If the target of village water supply has been reached, it must also lead to a reduction in the incidence of diarrhoeal diseases.
General steps of Evaluation
1. Determine what is to be evaluated Generally speaking there are 3 types of evaluation based on what is to be evaluated a. Evaluation of structure- Whether facilities, equipment, manpower and organisation meet a standard accepted by experts as good.
b. Evaluation of process • Including problems of recognition, diagnostic procedures, treatment and clinical management, care and prevention. • The way in which the various activities of the programme are carried out is evaluated by comparing with a predetermined standard.
c. Evaluation of outcome • Concerned with end results that is whether person using health services experience measurable benefits such as improved survival or reduced disability. • The traditional outcome components are the 5 Ds of ill-health viz. disease, discomfort, dissatisfaction, disability and death.
2. Establish standards and criteria • Standards and criteria must be developed in accordance with the focus of evaluation.
Structural criteria : • Process criteria: • Outcome criteria:
3. Plan the methodology to be applied • A format in keeping with the purpose of evaluation must be prepared for gathering information desired. • Standards and criteria must be included at the planning stage.
4. Gather information • Evaluation requires collection of data or information. • The type of information required may include political, cultural, economic, environmental and administrative factors influencing the health situation
Data collection methods • Self administered survey • Telephone surveys • Face to face structured • Archival trend data • Observation • Record review • Focus groups • Unstructured interview/narratives • Open ended questions on a written survey • Participant observation • Archival research
5. Analyse the results • The analysis and interpretation of data and feedback to all individuals concerned should take place within the shortest time feasible, once information has been gathered. • In addition, opportunities should be provided for discussing the evaluation results.
6. Take action • For evaluation to be truly productive, emphasis should be placed on actions - actions designed to support, strengthen or otherwise modify the services involved.
7. Re-evaluate • Evaluation is an on-going process aimed mainly at rendering health activities more relevant, more efficient and more effective.
References • Community oral health. Cynthia M. Pine. 1st edition. • Preventive and Social Medicine. K. Park. 20th edition • Jong’s Community Dental Health. George Gluck, Warren Mrganstein. 5th edition. • Dental public health. Geoffrey Slack, Brian Burt. 2nd edition • Concepts in dental public health. Jill Mason. • Framework for Program Evaluation in Public Health – CDC. Weekly REPORT. September 17, 1999 / Vol. 48 / No. RR-11
Poul Erik Petersen and Stella Kwan. Evaluation of community-based oral health promotion and oral disease prevention - WHO recommendations for improved evidence in public health practice. Community Dental Health (2004) 21 (Supplement), 319-329 • Health programme evaluation. Guiding principles for its application in the managerial process for national health development. World Health Organisation, Geneva 1981
Thank You
