Evaluation of “Context-based Learning” Patrick Martiny (ITM), Bangkok, 14 July 2009

1. Evaluation of “Context-based Learning” Patrick Martiny (ITM), Bangkok, 14 July 2009

2. Background and CBL • Experience in PC from Ayutthaya Project . • Policies for expansion of primary care, universal coverage, “hospital downsizing “. • Increase of demand for primary care services. • Need for better staffing of PC units, but also to improve skills of existing personnel at PCU / CMU. • Curriculums in Community and Family Medicine. • 4 month Nurse Practitioner training • “CBL” • CBL concept = in service capacity building & along with reinforcement of the partnership between primary care services and referral hospital (community or provincial).

3. Method of HCRP / CBL intervention • Starting from a concept (CBL) and the definition of a package of activities (3+3). • Identification of good practices related to CBL. • Recruitment of the promoters as national consultants. • Support to national consultants in order to: • reinforce their own local initiative, • exchange about their experience within the national consultants’ team, • develop further a package of CBL related activities in their own district, • document their practice and advocate / support for scaling up. • Wrapping up: evaluation, refinement of the concept and communication for scaling up.

4. Initial package of activities for CBL (3+3) • Introduction of standard guidelines. • In service training program (training-for-purpose). • Support to health centre staff through supervision. • Complementary learning activities like: • communication by telephone between health centre and hospital staff, • case conferences, • follow up of patients admitted to hospital.

5. Evaluation of “CBL” • Evaluation of CBL is not an administrative but a realistic, scientific evaluation: not a control as such, but stress laid on knowledge production: • Did CBL work? What is feasible? What is effective? • In which context does it work? What is necessary to make it work? • Administrative evaluation may be useful for sanction and reward. • Realistic evaluation may be useful for communication beyond HCRP, CBL advocacy and scaling up.

6. Nitayarumphong, Mercenier (1992)

7. Scientific evaluation Confrontation of the situation analysis with a reference conceptual model, pre-existing knowledge and past experience allows making an empirical decision in order to improve on the situation (the decision is empirical because based on a dynamic hypothesis which should be verified). The empirical decision is translated in action, or a set of actions. Evaluation takes place at two levels: operational evaluation which verifies if decision has been properly implemented, how and under which conditions; analytical evaluation which verifies if decision was good and has led to expected results in the concerned context. During implementation, the dynamic hypothesis can be challenged and amended if decision cannot be implemented (assessed through operational evaluation), or if implementation does not lead to expected results (assessed through analytical evaluation) – feed back -. It is expected that HSR modifies the situation, enriches the reference conceptual model as well as the experience of the actors. Adapted from Mercenier P, Grodos D. (2000). Health Systems Research: a clearer methodology for more effective action. Antwerp, Studies in Health Services Organisation & Policy, 15, 49-55.

8. Realistic evaluationPawson & Tilley (1997) • Does it work? • In which context does it work? What is necessary to make it work? A mechanism M in a context C produce un output O

9. Realistic evaluation Blaise and Kegels, 2004. For illustration, see Bart Criel et al. (2004) Community health insurance in sub-Saharan Africa: researching the context. Tropical Medicine and International Health: 9 (10), 1041–1043.

10. Method used for evaluation • Continuous follow up and technical support • Observation, interviews, meetings and working session, reading • Final field visits (24/6 - 4/7/2009) • All five CBL sites (Khorat, Phachi, Ayuttaya, Rasisalai, Thepa) • 10 days • Individual and group interviews 106 persons: district hospital directors (or provincial), district health officer and staff (DHO), trainers ( medical doctors, nurses, …), trainees (health staff from PCU), heads of PCU/CMU, community representatives (users), provincial chief medical officer and staff (PHO), directors of local branch of NHSO • Workshop to share and discuss the results with some key stakeholders (14/7/2009) • contribution of participants to be part of the evaluation.

11. Diversified introduction of CBL activities

12. A variety of contexts • Community Hospital (Thepha, Rasisalai, Phachi) or Provincial Hospital (Ayutthaya, Khorat) • Support team practising at the referral hospital (Rasisalai, Thepha, Phachi). • Availability of specialists (Khorat, Ayutthaya) • Availability of several family doctors with experience (Khorat, Thepha). • Partnership with DHO, or competition. • Number of facilities to support: from 31 (Khorat) to 7 (Phachi). • Large (286,353 in Khorat) or small population (30,000 in Phachi) to be covered. • Rural (Phachi, Thepha, Rasisalai) or urban environment (Ayutthaya, Khorat).

13. Findings from implementation of package of CBL activities (3+3) ... ... and other CBL related activities (workshop on F&C medicine, medical consultations at CMU and coached home visits.

14. Guidelines • May be just a tool to be introduced for better care delivery, or also a “technical contract” to refer to for in-service training and supervision (e.g. Prof Surakiat’s flowcharts). • Standards procedures jointly agreed on by PCU/CMU and Hospital staff may increase confidence of patients in care at PCU – same care, same drugs - (Rasisalai, Khorat). • Their development may be an opportunity to reinforce relationship between specialists and family doctors / nurses (Khorat). • Development of guidelines starting from PCU may increase self confidence, professionalism and self esteem of the PCU staff (Phachi).

15. In-service training • Practice at the community hospital (Rasisalai, ...) - versus at regional hospital - leads to: • skills more adapted to needs • confident, respectful and sustainable relationship between PCU and hospital staff • for learning together • for better working together (integrated health system) • Learning besides at PCU /CMU reinforces skills for integration of care and activities in the community (Rasisalai). • Some self learning teaches to learn continuously. • Training in a small group (multidisciplinary) improve on further team work and continuous learning within the team. • Need for qualified, committed and PC oriented trainers. • Guidelines for PCU /CMU should be taken into account.

16. Supervision • Audit (control) is much more common - and easy - than supportive supervision (brotherhood). • There is however some agreement on the concept of supportive supervision recalled in the 2006 World Health Report (WHO). • May be felt awkward to be observed in ones’ practice, but it is however considered as useful by PCU/CMU staff. • Supportive supervision needs human resources, commited and qualified in PC, Family and Community Medicine, and in Integrated Health System. • Miscellaneous activities may be used as opportunities for supervision (medical consultation, home visits, community activities…) if basic principles of supportive supervision are respected.

17. Case conferences • Various schemes: • Academic presentations by residents (Khorat). • Informal lunch discussion (or more formal presentation) at CMU level, based on local findings (Thepa, Khorat, Ayutthaya). • Meeting of PCU and Hospital staff at Referral hospital (Rasisalai, Ayutthaya, Thepha, Khorat). • Various objectives: • More knowledge for better care. • Further development of the relationship between PCU and hospital. • Self esteem of participants, professionalism.

18. Telephone calls to/from hospitalFollow up of admitted patients • Reasons for calling: • Organisation of patient’s admission or discharge, • Seeking advice for diagnosis and decision, use of drug, management. • Conditions to be fulfilled: • Communication means (mobile phone, Internet), • Availability and willingness of hospital staff (nurses, dentists, pharmacists, doctors). • May be a natural consequence of: • The relationship developed between PCU and hospital staff, • Acquired professionalism.

19. Coaching home visits(Phachi, Ayutthaya, Thepha) • Target elderly, chronic and disable patients, psychosocial cases: needs for multidisciplinary team, polyvalence and communication (which may lead to learning). • Hospital staff may above all support the PCU staff or on the contrary, lead the activity (risk of disempowering or substituting to PCU staff). • An opportunity for teaching techniques to PCU staff and for supervision. • All staff learns from the patients’ environment, which may influence their provision of care.

20. Medical consultation at CMU(Khorat, Ayuttaya, Thepa) • Medical Doctor may above all take care of his own consultation (specialists in Ayutthaya), or by contrast, also support consultation by CMU staff (family doctors in Ayutthaya = care managers). • May be an opportunity for supportive supervision, review of patient’ charts, case conference, home visits, activities in the community, along with CMU staff (Thepha, Ayutthaya). • Workload (chronic patients) may hamper coaching : time is necessary for coaching (Ayuttaya, Thepa). • Better if coach is trained in family / community medicine, and is ... committed.

21. Week-end Workshops(Khorat, Thepha, Ayutthaya) • Complementary to CBL activities: introduction of family and community medicine concepts (theoretical). • Possibility to test the concepts in one’s practice between the workshop weekends, and report (practical). • Trainers and trainees may work in the same health system (Khorat, Thepha); workshops may then reinforce relationship between PCU/MCU staff and trainers (context based). • Workshop may however be attended by staff from other districts (Rasisalai to Khorat; Phachi to Ayutthaya ) • Specific sessions may be scheduled for coaches. • Follow up of workshop may be ensured during coached home visits.

22. Impact of CBL approach Diffusion of good practicesRecommendations

23. Impact of CBL approach • Relationship between PCU/CMU and community hospital staff • PCU/CMU dares to approach hospital staff (feels at home) • Hospital staff respects more PCU/CMU staff (attitude, counter referrals; some specialists feel concerned by PC). • Confidence of PCU/CMU staff in their capacity for care provision (technical and emotional) and for continuous improvement (learning, Internet). • Skills, commitment, professionalism of PCU/CMU staff (seeking advice, prompt referral, follow up of admissions). • Reinforcement of teamwork at PCU for care and learning. • Satisfaction of users (they appreciate standards of care, links between hospital and PCU/CMU, contribution of doctors). • Better partnership between community hospital and DHO (Thepha, Ayutthaya, Rasisalai...) – is a condition and a result?

24. Some transfers of good practices within the CBL team • WE workshops on Family and Community Medicine from Khorat to Thepa • Participation of Rasisalai staff in Khorat workshop. • Rasisalai « in service training » procedures introduced in Thepa and about to be in Ayutthaya. • Coaching modalities from Thepha to Ayutthaya. • Home visit modalities from Phachi to Ayutthaya. • Assessment of learning needs from Rasisalai to Khorat. • ...

25. Scaling up • Diffusion of CBL good practices in nearby districts is limited. • Contexts are different, conditions must be fulfilled, but stakeholders’ willingness is above all necessary. • Reluctance for appropriation since it is better for one’s ego to be a pioneer than a follower. • Opportunities do exist for advocacy, and are already being used (Rasisalai, ...): forums set up by PHO and NHSO, funding for investment in human resource development (NHSO), for Tambon upgraded hospital program (MOPH). • Further introduction of CBL may need some technical support from national consultants.

26. Main recommendations • “Pioneers” to maintain improving on CBL implementation in their own district. • CBL to be further advocated (PHO - NHSO forums, field visits, large distribution of the toolkit). • Better to advocate CBL principles and present success stories as examples only, with their limits (not as guidelines to be followed). Let free to adapt and customize. • Success to be demonstrated, concrete results to be showed; field visits must be supported by a convincing discourse. • “Pioneers” having been involved in the CBL process to remain available for technical support for CBL introduction in other districts. • Use to be made of possible funding for development (NHSO, MOPH).

27. Context Based Learning: not just a “3+3”package of activities, but a concept • The concept has been tested on the field. It is valid. • CBL is the development of a sustainable learning and functional relationship: • between community hospital and HC/PCU/CMU staff • based on shared concepts (primary care, family and community medicine, integrated health system, total quality management and continuous learning). • for learning and better working together within an integrated health system) • for better quality primary care. • The concept may be valid even in a non “delegation of tasks” context, even when doctors are full time appointed in CMU.