Evaluation Planning and Management John Wooten ~ Lynn Keeys

1. WELCOME Evaluation Planning and Management John Wooten ~ Lynn Keeys June 2012

2. Session 5: Specifying the Evaluation Evaluation Road Map: Stage 2 Objective: • Understand and prioritize key actions to finalize an evaluation plan

3. Evaluation Need Triggered! Ok, the need for an evaluation of your project has been… …TRIGGERED! Where do you start?! What would you do!?

4. Get Organized…. …and get moving! Organize/activate evaluation planning and management team * Finalize evaluation design and plan * Organize/prepare reference/background materials Finalize/formalize/follow-up logistics and administrative arrangements

5. Organize/activate Evaluation Planning and Management Team (EPM) Who as EPM Chair? EPM Confirm/assign team roles and responsibilities Identify evaluation triggers and priority pending decisions (Do not assume!) Finalize/activate participatory approach plan Advertise intent to evaluate (new requirement) Set timeframe and review availability of key parties Explore acquisition and assistance approaches (A&A) Confirm evaluation readiness *

6. Organize/activate Evaluation Planning and Management Team (EPM) EPM • Confirm evaluation readiness… • Clarity of design theory • Stakeholder understanding • Good files management • Good stakeholder communications * • Know your project and program context ** (Finger-on-pulse management)

7. Organize/activate Evaluation Planning and Management Team (EPM) EPM • Know your project and program context • Socio –economic –political climate • Fit with project’s implementation and USAID’s program cycle (conflicts/interruptions)  • “Evaluation burden” * • Availability of project personnel • Access to relevant sites • Physical access to relevant sites • Baseline data status * • Adequacy/availability of funds for evaluation

8. Finalize Evaluation Design and Plan …Sequenced and tightlyi-n-t-e-g-r-a-t-e-d

9. Evaluation Purpose/Use? Clear, focused, not overly stacked… WARNING: Possible surprises as interpretations may vary significantly Keyed to: • Evaluation triggers • Pending priority decisions (USAID etal) • Participatory inputs (consensus)

10. Key Evaluation Questions? Good evaluation questions are… • Reasonable/answerable with available resources, based on empirical evidence • Align with the project’s objectives/design • Specific rather than wordy and fuzzy • Broadly representative • Helpful in setting the “standard for success” • Instructive to the evaluation team in making priority and timely decisions

11. Key Evaluation Questions? General guidelines… • AVOID “yes/no” questions * • Minimize the number of questions • Test project development frameworks (RF, Logframe) • Inform data processes • Key to pending management decisions • Activate participatory approach • Descriptive stratification (what, which direction, why, on track?) • Allow room/time to P.R.O.B.E.

12. Key Evaluation Questions? Some typical pitfalls… • No correlation to pending decisions (USAID et al) • Primarily reflect a USAID perspective • Alternate questions elsewhere in SOW • Questions that conflict with/tangential to purpose Obvious items naturally in evaluation report Excessive number of questions Too tightly focused, no room to probe Wordy, multi-faceted questions that say little

13. Key Evaluation Questions? A flexible guide…

14. Key Evaluation Questions? Evaluation design matrix… An indicative evaluation work plan

15. Statement of Work Team needs; selection process… • Competency requirements • Expatriate/local consultants • and assignments • Team augmentation plans • Team size and gender mix • Availability requirements • Acquisition and assistance • Potential real/perceived conflicts of interest • Reference requirements (!) • Alternate candidates • Team leadership assignment • Affordability concerns • Unique requirements

16. Statement of Work Team selection pitfalls…

17. Statement of Work By now, EPM is armed with… Key evaluation questions Consensus on evaluation purpose Team requirements Indicative budget Confirmation of funds availability Notification of intent to evaluate Participatory inputs on all the above Confirmed readiness to evaluate • Agreed evaluation timeframe

18. Statement of Work Detailed draft evaluation SOW… • Purpose/use of evaluation • Project summary data sheet • Context/background • Key evaluation questions • Illustrative evaluation design and methodologies • Evaluation schedule/ logistics plan • Team skills/experience requirements • Proposal /team criteria • Communications plan • Deliverables and approvals* • Reporting relationships • Attachments*

19. Statement of Work Deliverables and approvals… • Oral, PowerPoint-assisted debriefings (specify number and audiences, including interim debriefings) • Organized collected quantitative data (hard/soft copies; new requirement) Evaluation work plan (detailed evaluation design, data collection methodologies, instruments, disaggregation requirements, and data analysis plan) Draft/final reports with attachments (number of hard/soft copies; due dates)

20. Statement of Work Attachments to SOW… • Results and logical frameworks (Not available?) • Illustrative report outline/required attachments • Project maps and illustrative site visits (if pre-determined) • Key persons list • List/sources of background materials * • USG budget estimate and level of effort (budget for internaluseonly; finalize after peer review of SOW)

21. Statement of Work Final checks… Check for internalconsistencyand adequate reflection of otherrelevantperspectives Organize peer review of SOW (new requirement; include USAID/Washington, stakeholders… Prepare final LOE and budget *

22. Detailed Budget Consult ‘new best friends!’ Budget for internal use only… • Mission financial management specialists • IP budget specialists • Program budget specialists • A&A office staff • Executive/management officers • IP home office staff • Stakeholders budget officers • Mission M&E officers

23. Budget Budget categories… Sample… Labor/technical assistance (expatriate, local, regional, home office support; loaded/unloaded daily rates) Travel and transportation (air/ground travel, visas, excess baggage, medical costs…) Per diem (international, local, weekends) Other direct costs (catch-all item per recent experience) Overhead, general and administrative, profit (see your ‘new best friend’) Mission and stakeholder reps (budget separately for team augmentation plans)

24. Organize Background Materials… From where/whom? Which materials? How to get them to the evaluation team? Where to put them? How to motivate stakeholders to contribute timely? Does it really matter? What about items not accessible post-project? Are you serious?! How much is enough?

25. Organize Background Materials… Excerpts of mission strategy and sector assessments Project design documents and major modifications Mission acquisition and assistance documents and major modifications Original and current approved performance management plan(s) Baseline and mid-line survey(s) The most recent and sample project implementation reports (performance management plan, quarterly, annual, special and financial reports, the latter reflecting budget and expenditure data by project component) Summary detail of planned and actual partner and host country contributions

26. Organize Background Materials… Summary of project’s major accomplishments, problems and pending plans Project and beneficiary mapping (locations, group sizes, timing of interventions, etc.) Key relevant mission documents (portfolio review summary, trip reports, decision memoranda, success stories, mission-IP home office correspondence) Relevant USAID internal assessments, prior evaluations, management/portfolio reviews, audits, major meeting/conference notes (internal/external) Relevant other donor materials Relevant host government materials (laws, policies, audits, reviews, etc.

27. Organize Background Materials… • Project chronology of major events • Performance data quality assessment insights • Key persons list • Relevant products from your own continuous learningabout recent developments within the technical area of expertise • Useful checklists and ‘how to’ guides: (See Tab 6) • EPM assessment of evaluation experience/report • “Constructing and Evaluation Report”, USAID’s M&E TIPS #17 • “Constructing and Evaluation Report”, un-# TIPS, Management Systems International, April 2006 • “Checklist for Assessing USAID Evaluation Reports”

28. Finalize, formalize, follow-up logistics and administrative arrangements… • Yes, I’m serious~ • Determine which part is the EPM’s responsibility and what is left for the evaluation team

29. Finalize, formalize, follow-up logistics and administrative arrangements … • Typical pitfalls… • Considering logistics and administrative arrangements only or primarily for the evaluation team without examining the broader logistical and administrative environment within which the team will function • Planning but failing to formalize and follow-up a good logistics plan • EPM taking a ‘hands off’ approach to team logistics, • thinking this is the full responsibility of the evaluation team

30. Finalize, formalize, follow-up logistics and administrative arrangements … • Team procurement arrangements (l-o-n-g lead times) • Proposal review and team candidate selection • Pre-arrival orientation preps and assignments • Travel/transportation requirements/arrangements (international, in-country, regional for evaluation team and staff of USAID, IP and stakeholders) • Site visit selections and preparations (absolutely critical. Consider assigning site visit managers to coordinate details. …DO IT WELL IN ADVANCE) • Meeting venues per site visit (materials, equipment, refreshments and special needs requirements)

31. Finalize, formalize, follow-up logistics and administrative arrangements … • Stakeholder/beneficiaries briefings and preparations • Language interpretation and translation services • Files reviews (USAID, IP, others; noted well in advance) • Final team augmentation plans (USAID, IP, stakeholder, beneficiary reps – examine pros and cons • Preliminary/flexible interview schedules for high level officials • Housing and meals requirements/arrangements (expatriates, locals) • Security requirements/arrangements

32. Finalize, formalize, follow-up logistics and administrative arrangements … • Materials, equipment and supplies requirements/arrangements (cameras, recording devices, public address systems, notepads, pens, computers, printers, internet access, phones, faxes, chairs, flip charts, projection equipment) • Social events (sponsored by evaluation team, stakeholders and/or beneficiaries) • Formal notifications (briefing/debriefing meetings participants, known interviewees, site visit managers, beneficiary reps logistics suppliers and control group reps (unique challenge) • Preliminary evaluation follow-up plan (report distribution, summaries, excerpts, vignettes, etc.)

33. Aaahhh… EPM is NowWell Organized…. The EPM is now ready to select the evaluation team and start organizing initial team-building sessions EPM is activated and well organized Evaluation design and plan are well prepared and cleared, with good participatory inputs Reference and background materials are being assembled and organized Evaluation logistics are good draft and are being followed-up

34. Evaluation Planning and Management John Wooten ~ Lynn Keeys  • Thank you~ (jwootenjr@yahoo.com) (lynnkeeys50@yahoo.com)