Program Design & Evaluation Ethics and Values

1. Program Design & EvaluationEthics and Values

2. Every profession has distinct ethical obligations to the public. These obligations include professional competency, integrity, honesty, confidentiality, objectivity, public safety, and fairness.

3. All of these obligations are intended to preserve and safeguard public confidence

4. Stem cell field rocked by scam of star scientist The star scientist at the heart of that deception – Hwang Woo Suk -- resigned Friday from his university post, his meteoric rise to fame coming to a wrenching end. 12/24/05 Unfortunately, all too often we hear reports in the media of moral dilemmas and unethical behavior by professionals. These reports naturally receive considerable attention by the public, whose confidence in the profession is undermined with each report.

5. Educators are not immune to unethical practices Houston, Texas Testing Scandal in Texas Schools NPR News March 21, 2005 by Claudia Sanchez

6. Existing Ethical Standards • Concerns about ethical practices in assessment are not new. • As early as 1972, the National Council on Measurement in Education (NCME), the Association for Measurement and Evaluation in Guidance (AMEG), and the American Counseling Association (ACA) developed a position paper on the responsible use of tests that was intended to ensure that tests are given, and examinees are treated, fairly and wisely (AMEG, 1972).

7. Testing & Evaluation Standards • Later in the 1970s, ACA developed a statement on the responsibilities of the users of standardized tests and evaluation • That document was revised as recently as 1989 (ACA, 1989).

8. Assessment Code of Ethics • Both of these early documents recognized the need to positively influence the practices of those who use tests in ways that promote responsible use. • These statements have been followed by the development of ethical standards by a number of other organizations having an interest, or directly involved, in testing assessment, and evaluation.

9. What About Evaluation Ethics? Principles & Standards for Program Evaluators

10. American National Standards InstituteANSI The American National Standards Institute (ANSI) is a private, non-profit organization (501(c)3) that administers and coordinates the U.S. voluntary standardization and conformity assessment system. The Institute's mission is to enhance both the global competitiveness of U.S. business and the U.S. quality of life by promoting and facilitating voluntary consensus standards and conformity assessment systems, and safeguarding their integrity. Founded in 1918

11. Evaluation Standards JOINT COMMITTEE ON STANDARDS FOR EDUCATIONAL EVALUATION Created in 1975, the Joint Committee is a coalition of major professional associations concerned with the quality of evaluation. The Joint Committee is housed at The Evaluation Center, Western Michigan University. The Joint Committee has published three sets of standards for evaluations: 1. The Personnel Evaluation Standards published in 1988 2. The Program Evaluation Standards published in 1994 3. The Student Evaluations Standards was published in 2003 The Joint Committee is accredited by the American National Standards Institute (ANSI). Standards approved by ANSI become American National Standards.

12. Four Program Evaluation Standards“…as it is written, so it shall be done! • Utility Standards • Feasibility Standards • Propriety Standards • Accuracy Standards

13. Utility Standards(7) To ensure that the evaluation will serve the information needs of intended users • U1 Stakeholder Identification Persons involved in or affected by the evaluation should be identified, so that their needs can be addressed. • U2 Evaluator Credibility The persons conducting the evaluation should be both trustworthy and competent to perform the evaluation, so that the evaluation findings achieve maximum credibility and acceptance.

14. Utility Standards • U3 Information Scope and Selection Information collected should be broadly selected to address pertinent questions about the program and be responsive to the needs and interests of clients and other specified stakeholders • U4 Values Identification The perspectives, procedures, and rationale used to interpret the findings should be carefully described, so that the bases for value judgments are clear.

15. Utility Standards • U5 Report Clarity Evaluation reports should clearly describe the program being evaluated, including its context, and the purposes, procedures, and findings of the evaluation, so that essential information is provided and easily understood • U6 Report Timeliness and Dissemination Significant interim findings and evaluation reports should be disseminated in a timely manner.

16. Utility Standards • U7 Evaluation Impact Evaluations should be planned, conducted, and reported in ways that encourage follow-through by stakeholders, so that the likelihood that the evaluation will be used is increased, and to ensure that the evaluation will serve the information needs of intended users

17. Feasibility Standards (3) To ensure that an evaluation will be realistic, prudent, diplomatic, and frugal F1 Practical Procedures The evaluation procedures should be practical, to keep disruption to a minimum while needed information is obtained.

18. Feasibility Standards • F2 Political Viability The evaluation should be planned and conducted with anticipation of the different positions of various interest groups, so that their cooperation may be obtained, and so that possible attempts by any of these groups to curtail evaluation operations or to bias or misapply the results can be averted or counteracted. • F3 Cost Effectiveness The evaluation should be efficient and produce information of sufficient value, so that the resources expended can be justified

19. Propriety Standards (8) Intended to ensure that an evaluation will be conducted legally, ethically, and with due regard for the welfare of those involved in the evaluation, as well as those affected by its results. P1 Service Orientation Evaluations should be designed to assist organizations to address and effectively serve the needs of the full range of targeted participants. P2 Formal Agreements Obligations of the formal parties to an evaluation (what is to be done, how, by whom, when) should be agreed to in writing, so that these parties are obligated to adhere to all conditions of the agreement or formally to renegotiate it.

20. Propriety Standards • P3 Rights of Human Subjects Evaluations should be designed and conducted to respect and protect the rights and welfare of human subjects. • P4 Human Interactions Evaluators should respect human dignity and worth in their interactions with other persons associated with an evaluation, so that participants are not threatened or harmed. • P5 Complete and Fair Assessment The evaluation should be complete and fair in its examination and recording of strengths and weaknesses of the program being evaluated, so that strengths can be built upon and problem areas addressed.

21. Propriety Standards • P6 Disclosure of Findings The formal parties to an evaluation should ensure that the full set of evaluation findings along with pertinent limitations are made accessible to the persons affected by the evaluation and any others with expressed legal rights to receive the results. • P7 Conflict of Interest Conflict of interest should be dealt with openly and honestly, so that it does not compromise the evaluation processes and results. • P8 Fiscal Responsibility The evaluator's allocation and expenditure of resources should reflect sound accountability procedures and otherwise be prudent and ethically responsible, so that expenditures are accounted for and appropriate

22. Accuracy Standards Intended to ensure that an evaluation will reveal and convey technically adequate information about the features that determine worth or merit of the program being evaluated. A1 Program Documentation The program being evaluated should be described and documented clearly and accurately, so that the program is clearly identified. A2 Context Analysis The context in which the program exists should be examined in enough detail, so that its likely influences on the program can be identified. A3 Described Purposes and Procedures The purposes and procedures of the evaluation should be monitored and described in enough detail, so that they can be identified and assessed

23. Accuracy Standards A4 Defensible Information Sources The sources of information used in a program evaluation should be described in enough detail, so that the adequacy of the information can be assessed. A5 Valid Information The information-gathering procedures should be chosen or developed and then implemented so that they will assure that the interpretation arrived at is valid for the intended use. A6 Reliable Information The information-gathering procedures should be chosen or developed and then implemented so that they will assure that the information obtained is sufficiently reliable for the intended use.

24. Accuracy Standards A7 Systematic Information The information collected, processed, and reported in an evaluation should be systematically reviewed, and any errors found should be corrected. A8 Analysis of Quantitative Information Quantitative information in an evaluation should be appropriately and systematically analyzed so that evaluation questions are effectively answered. A9 Analysis of Qualitative Information Qualitative information in an evaluation should be appropriately and systematically analyzed so that evaluation questions are effectively answered.

25. Accuracy Standards A10 Justified Conclusions The conclusions reached in an evaluation should be explicitly justified, so that stakeholders can assess them. A11 Impartial Reporting procedures should guard against distortion caused by personal feelings and biases of any party to the evaluation, so that evaluation reports fairly reflect the evaluation findings. A12 Meta-evaluation The evaluation itself should be formatively and summatively evaluated against these and other pertinent standards, so that its conduct is appropriately guided and, on completion, stakeholders can closely examine its strengths and weaknesses.

26. What is Meta-Evaluation? Simply stated, meta-evaluation is the evaluation of an evaluation, evaluation system or evaluation device to enhance the validity of the evaluation results. Operationally, metaevaluation is also defined as the process of delineating, obtaining, and applying descriptive information and judgmental information - about the utility, feasibility, propriety, and accuracy of an evaluation and its systematic nature, competent conduct, integrity/honesty, respectfulness, and social responsibility - to guide the evaluation and/or report its strengths and weaknesses.

27. Four Program Evaluation Standards • Utility Standards – serve the information needs of the intended users • Feasibility Standards – realistic, prudent, diplomatic, and frugal • Propriety Standards – legal, ethical, and due regard for welfare of participants • Accuracy Standards –reveal and convey information to determine the worth, value, and merit of the program evaluated

28. American Evaluation AssociationFive Guiding Principles for Evaluators • Ratified by the AEA membership, July 2004 • Supports Four Program Evaluation Standards

29. Principle 1 Systematic InquiryEvaluators conduct systematic, data-based inquiries 1. To ensure the accuracy and credibility of the evaluative information they produce, evaluators should adhere to the highest technical standards appropriate to the methods they use.  2. Evaluators should explore with the client the shortcomings and strengths both of the various evaluation questions and the various approaches that might be used for answering those questions. 3. Evaluators should communicate their methods and approaches accurately and in sufficient detail to allow others to understand, interpret and critique their work..

30. Principle 2CompetenceEvaluators provide competent performance to stakeholders • 1. Evaluators should possess the education, abilities, skills and experience appropriate to undertake the tasks proposed in the evaluation. • 2.  To ensure recognition, accurate interpretation and respect for diversity, evaluators should ensure that the members of the evaluation team collectively demonstrate cultural competence. 3.  Evaluators should practice within the limits of their professional training and competence, and should decline to conduct evaluations that fall substantially outside those limits. • 4.  Evaluators should continually seek to maintain and improve their competencies, in order to provide the highest level of performance in their evaluations.

31. Principle 3 Integrity/Honesty:  Evaluators must display honesty and integrity in their own behavior and in the evaluation process. • 1.  Evaluators should negotiate honestly with clients and relevant stakeholders concerning the costs, tasks to be undertaken, limitations of methodology, scope of results likely to be obtained, and uses of data resulting from a specific evaluation. • 2.  Before accepting an evaluation assignment, evaluators should disclose any roles or relationships they have that might pose a conflict of interest with their role as an evaluator. • 3.  Evaluators should record all changes made in the originally negotiated project plans, and the reasons why the changes were made. • 4.  Evaluators should be explicit about their own, their clients', and other stakeholders' interests and values concerning the conduct and outcomes of an evaluation.

32. Principle 3Integrity/Honesty (continued) • 5. Evaluators should not misrepresent their procedures, data or findings. • 6.  If evaluators determine that certain procedures or activities are likely to produce misleading evaluative information or conclusions, they have the responsibility to communicate their concerns and the reasons for them. If discussions with the client do not resolve these concerns, the evaluator should decline to conduct the evaluation. • 7.  Evaluators should disclose all sources of financial support for an evaluation, and the source of the request for the evaluation.

33. Principle 4 Respect for People:  Evaluators respect the security, dignity and self-worth of respondents, program participants, clients, and other evaluation stakeholders. • 1.  Evaluators should seek a comprehensive understanding of the important contextual elements of the evaluation. • 2.  Evaluators should abide by current professional ethics, standards, and regulations regarding risks, harms, and burdens that might befall those participating in the evaluation; regarding informed consent for participation in evaluation; and regarding informing participants and clients about the scope and limits of confidentiality. • 3.  Because justified negative or critical conclusions from an evaluation must be explicitly stated, evaluations sometimes produce results that harm client or stakeholder interests. Under this circumstance, evaluators should seek to maximize the benefits and reduce any unnecessary harms that might occur, provided this will not compromise the integrity of the evaluation findings.

34. Principle 4 Respect for People (continued) • 4.  Knowing that evaluations may negatively affect the interests of some stakeholders, evaluators should conduct the evaluation and communicate its results in a way that clearly respects the stakeholders' dignity and self-worth. • 5.  Where feasible, evaluators should attempt to foster social equity in evaluation, so that those who give to the evaluation may benefit in return. For example, evaluators should seek to ensure that those who bear the burdens of contributing data and incurring any risks do so willingly. Program participants should be informed that their eligibility to receive services does not hinge on their participation in the evaluation. • 6.  Evaluators have the responsibility to understand and respect differences among participants, such as differences in their culture, religion, gender, disability, age, sexual orientation and ethnicity, and to account for potential implications of these differences when planning, conducting, analyzing, and reporting evaluations.

35. Principle 5Responsibilities for General and Public Welfare Evaluators articulate and take into account the diversity of general and public interests and values that may be related to the evaluation. • 1.  When planning and reporting evaluations, evaluators should include relevant perspectives and interests of the full range of stakeholders.   • 2.  Evaluators should consider not only the immediate operations and outcomes of whatever is being evaluated, but also its broad assumptions, implications and potential side effects

36. Principle 5 Responsibilities for General and Public Welfare(continued) • 3.  Freedom of information is essential in a democracy. Evaluators should allow all relevant stakeholders access to evaluative information in forms that respect people and honor promises of confidentiality. In all cases, evaluators should strive to present results clearly and simply so that clients and other stakeholders can easily understand the evaluation process and results. • 4.  Evaluators should maintain a balance between client needs and other needs. Evaluators necessarily have a special relationship with the client who funds or requests the evaluation. By virtue of that relationship, evaluators must strive to meet legitimate client needs whenever it is feasible and appropriate to do so. However, that relationship can also place evaluators in difficult dilemmas when client interests conflict with other interests, or when client interests conflict with the obligation of evaluators for systematic inquiry, competence, integrity, and respect for people.

37. Principle 5 Responsibilities for General and Public Welfare(continued) • 5.  Evaluators have obligations that encompass the public interest and good. These obligations are especially important when evaluators are supported by publicly-generated funds; but clear threats to the public good should never be ignored in any evaluation. Because the public interest and good are rarely the same as the interests of any particular group (including those of the client or funder), evaluators will usually have to go beyond analysis of particular stakeholder interests and consider the welfare of society as a whole.

38. Summary of 5 Guiding Evaluation Principles • 1. Systematic Inquiry: Evaluators conduct systematic, data-based inquiries about whatever is being evaluated. • 2. Competence: Evaluators provide competent performance to stakeholders. • 3. Integrity/Honesty: Evaluators ensure the honesty and integrity of the entire evaluation process. • 4. Respect for People: Evaluators respect the security, dignity and self-worth of the respondents, program participants, clients, and other stakeholders with whom they interact. • 5. Responsibilities for General and Public Welfare: Evaluators articulate and take into account the diversity of interests and values that may be related to the general and public welfare.

39. Program Design & EvaluationStandards and Principles • Four Program Evaluation Standards Utility, Feasibility, Propriety, & Accuracy • Five Guiding Principles for Evaluators Systematic Inquiry, Competence, Integrity/Honesty, Respect for People, Responsibilities for General & Public Welfare

40. Program Design & Evaluation StandardsPrinciples Five Guiding Principles for Evaluators SCIRR Systematic Inquiry, Competence, Integrity/Honesty, Respect for People, Responsibilities for General & Public Welfare Four Program Evaluation Standards UFPA Utility, Feasibility, Propriety, Accuracy

41. PROGRAM DESIGN & EVALUATION PROCESS Define Purpose Engage Stakeholders Describe Program Evaluation 5 Guiding Principles Report Evaluation Ensure Use & Share Lessons 4 Standards 5 Principles 6 Steps Focus Design, Develop Methodology, Data Collection Plan Analyze Data Justify Conclusions 5 Guiding Principles Collect the Data & Gather Evidence

42. When conducting program evaluations, we must always remember the words of wisdom coined by a famous scientific researcher and evaluator: “Not everything that can be counted counts…and not everything that counts can be counted.” Einstein