Using the Guiding Principles for Evaluators to Improve Your Practice

1. Using the Guiding Principles for Evaluators to Improve Your Practice Meredith Stocking, Atlanta-Area Evaluation Association Note: This presentation has been adapted in part from the AEA’s Guiding Principles Training Packet

2. Objectives • Increase knowledge of the AEA Guiding Principles for Evaluators (GP) • Analyze the Guiding Principles in a program evaluation context • Consider how the Guiding Principles can be used to inform your evaluation practice

3. What are ethical problems in evaluation anyway? • Issues of moral accountability that involve doing the “right” or “wrong” thing • Often involve the welfare of others • Choices between unfavorable alternatives

4. AEA’s Development of the Guiding Principles for Evaluators • 1986: Founding of American Evaluation Association • 1993-1994: Original five Guiding Principlesfor Evaluators developed and ratified • 2002-2004: GP reviewed and updated • 2004: Revised GP endorsed through referendum of AEA membership

5. Format and Purpose of the Guiding Principles for Evaluators • Format of the Guiding Principles (long version): • 10 assumptions • 5 principles with supporting sub-principles • 2 pages of background • http://www.eval.org/publications/guidingprinciples.asp • Purpose of the Guiding Principles: • Promote ethical evaluation practice • Foster continuing professional development • Stimulate discussion within and outside evaluation

6. Assumptions Behind the Guiding Principles for Evaluators The Guiding Principles: • Proactively guide everyday practice • Cover all kinds of evaluation • Do not apply in every situation • Are not independent, but overlap • Sometimes conflict • Were developed in the context of Western cultures

7. Principle A: Systematic Inquiry • Evaluators conduct systematic, data-based inquiries: • Adhere to highest technical standards • Explore strengths and shortcomings of evaluation questions and approaches • Communicate approaches, methods and limitations accurately

9. Principle B: Competence • Evaluators provide competent performance to stakeholders: • Possess appropriate skills and experience • Demonstrate cultural competence • Practice within limits of competence • Continually improve competencies

10. Principle C: Integrity/Honesty • Negotiate honestly with clients and stakeholders • Disclose values, interests and conflicts of interest • Represent accurately methods, data and findings • Disclose source of request and financial support for evaluation Evaluators display honesty and integrity and attempt to ensure them throughout the entire evaluation process:

11. Principle D: Respect for People Evaluators respect security, dignity and self-worth of all stakeholders: • Understand evaluation context • Get informed consent and protect confidentiality • Maximize benefits and minimize harm • Foster social equity • Respect differences among stakeholders

13. Principle E: Responsibilities for General and Public Welfare Evaluators take into account general and public interests: • Include a full range of stakeholders • Examine assumptions and potential side effects • Balance client and stakeholder needs • Allow all relevant stakeholders access to findings in understandable forms

14. Case Study: Evaluating the Health Care Collaborative Background: The Health Care Collaborative (HCC), a signature program of a local nonprofit organization, is designed to increase the delivery of primary health care services to residents in a low-income, underserved neighborhood. The HCC uses trained neighborhood residents as outreach health workers to raise health-issues awareness among residents and to give them options for accessing health care. Health care providers who are collaboration partners deliver a range of services to program participants. HCC’s principal funder recently approached the nonprofit’s board or directors to request an external evaluation of the program. Margaret, the nonprofit’s director, is meeting with Jane, a former board member and professor at the local University, to ask her to consider taking on the evaluation. Margaret: Jane, it’s been such a long time! It’s great to see you! How have things been going? Jane: Everything’s great! I am really busy leading a large multi-site evaluation with one of my colleagues, and still teaching a few evaluation classes. What can I do for you today? Margaret: Well, our main funder just asked us for an external evaluation of our Health Care Collaborative Program. Jane: Yes, that’s a fabulous initiative. What do they want to know? Continued on the next slide.

15. Margaret: They are saying that they need more information than the program’s reporting system alone can provide- principally about how the neighborhood residents and program participants view HCC, and how the program is meeting or not meeting identified service needs. Jane: That sounds pretty straight forward. Have you issued an RFP? Margaret: Well, you know that we always have such a positive response to the work we do, but the program has been subject to recent criticism from some new neighborhood residents. We are really relying on the funder to renew our grant next year, and we don’t want the results of the evaluation to be skewed by the opinion of a vocal few. We’d really like to work with someone we trust who is familiar with our track record. Jane: Honestly Margaret, you are a great friend, but I am really slammed right now. I don’t think I can take on another project. Margaret: What if you designed the evaluation and provided some oversight, but recruited one or two graduate students to do the data collection and analysis? Jane: I suppose that may be an option. Can you give me an idea of the current composition of the community and the stakeholders that would be part of the evaluation team? I need to think about whether I have any students that would be a good fit. It sounds like you’re going to need to get some fairly broad community involvement given the information needs of the funder. Margaret: Well, we only have about eight months to complete the evaluation and our budget is fairly tight, so we may need to limit stakeholder participation on the evaluation team. The make-up of the community has been in flux lately. The African American and Latino populations are still big, but the neighborhood has seen a lot of East African refugees arrive as well as some immigrants from Eastern European nations. It’s becoming more and more diverse. Continued on the next slide.

16. A few days later: Jane is talking with one of her best graduate students, Alisa, about possibly working on the HCC evaluation. Alisa did some data analysis for Jane last semester and she knows he does good work. Alisa is also has the advantage of being fluent in Spanish. Alisa: The project sounds really interesting. And I am actually looking for a subject to write my Master’s thesis about. This could be a perfect topic! What kind of design are you thinking of? Jane: Probably a mixed-methods approach. We could do some surveys of participants, program staff, and health care providers. Also, one of the things they want to know is how beneficiaries and other folks in the neighborhood feel about the program, so a few focus groups would be good. Do you have any experience conducting focus groups? Alisa: I could probably wing it. I took a workshop on qualitative methods one time. I mean, its not like we are talking about rigorous experimental methods or anything. How difficult can it be? Jane: I suppose it’s a nice opportunity for you to get some practice. It’s just a small nonprofit program, and I don’t imagine the stakes of the evaluation are too high. Besides, I don’t have any other students with your language skills. When are you available to get started? Alisa: I could start as soon as you’d like. But I would have to do all the data collection during the day. I have all night classes this semester. Continued on the next slide.

17. After data collection: Alisa and Margaret meet to discuss the initial findings and plans for data dissemination. Alisa will report back to Jane, who is unable to make the meeting due to travel for her other evaluation project. Margaret: So Alisa, how did data collection go? What are you finding? Alisa: I ended up conducting three focus groups- one for senior citizens, another for adult non-senior males, and the third for adult, non-senior females. I am still in the process of analyzing the survey and focus group data, but all in all participants are overwhelmingly positive and satisfied with your services. And this is probably no news to you, but the program serves a disproportionate number of Hispanic adults compared to the neighborhood’s composition. I think that it may be important to consider why other racial and ethnic groups aren’t taking advantage of your services in greater numbers. But since we didn’t involve non-participants in the study, we really can’t say. Margaret: The board and the funder will be very happy to hear about the positive results. I am planning on giving the main evaluation briefing at next month’s meeting of the HCC board to which the funder will be invited. We can discuss the issue of how to better engage other racial and ethnic groups with the program staff at a later date. It’s really a programming issue and not something the funder needs to worry about. Alisa: Really? Margaret: Yeah. But don’t worry about it. We will definitely use the information. Alisa: Ok. Hey, there was one more thing I wanted to ask you about. Jane and I wanted to do a presentation on this evaluation at our national evaluation conference next month. Would that be OK with you? Of course, we will be sure to protect the confidentiality of all of the participants. Margaret: That’s a great idea! I am really happy that you are getting so much out of this opportunity. And I heard from Jane that your thesis is really looking nice. I would love to get a copy of it when you are done. Alisa: Sure thing! Discuss!

18. Food for Thought • What elements of competence are at stake? How do omissions in competence affect the ability to meet the principle of systematic inquiry? What about respect for people? • In what ways does the evaluation address potential weaknesses or criticisms of convenience in gathering data as opposed to systematic methods? • How would you assess the honesty and integrity of those involved in the evaluation? If you were part of this evaluation, what would you do to resolve or handle issues related to integrity? • Is there anything problematic about the way that stakeholders were included in the evaluation activities? How does the level and kind of stakeholder involvement relate to the principles of respect for people and public interest? • Is the evaluation reporting adequate? What about the use of findings? • To what extent does this evaluation attempt to foster social equity?

19. Discussion • What ethical dilemmas have you encountered in your own evaluation practice? How did you handle the situation? • How can you use the Guiding Principles as you design and conduct your own evaluations?

20. “Not knowing what constitutes best practice is incompetence. Knowing what best practice is, but not knowing how to achieve it, may be inexperience. Knowingly not following best practice, when one knows how to achieve it, is unethical.” Smith, N. (2002). An analysis of ethical challenges in evaluation. American Journal of Evaluation, 23, 199-206. THANK YOU!