The Use of Decision Analysis in Program Evaluation

1. The Use of Decision Analysis in Program Evaluation Farrokh Alemi, Ph.D.

2. Program Evaluation Is Pervasive Requested and funded by virtually every department of health and social services, not to mention many legislatures, governors, and city administrations.

3. Process of Program Evaluation

4. Various Approaches to Program Evaluation • Experimental Design • Quasi-experimental • Case study

5. Program Evaluations Are Ignored • Not relevant, unrelated to decisions • Poor design • “Evaluations, we believe, exist (or perhaps only should exist) to facilitate intelligent decision making ... an evaluation research program will often satisfy curiosity. But if it does no more, if it does not improve the basis for decisions about the program and its competitors, then it loses its distinctive character as evaluation research and becomes simply research” (p. 140).

6. Program evaluation should be tied to the decision-making process • Identify the primary users of the evaluation • Identify the decision-making needs and provide information to meet their needs • Suggest options • Identify and reduce the most important uncertainties • Explain how results were arrived at • Examine the sensitivity of the evaluation findings • Present results more quickly and when they can be most useful.

7. Example of a Program • Nursing home costs in the United States • Surveys of nursing home quality are too frequent, are too intensive, and have little relation to the health status and functional ability of nursing home residents. • Evaluate the process of surveying nursing home quality

8. Step 1. Identify the decision makers • Examine the potential users • Invite them to help devise the plan of action • In the example, three groups might be expected to use the evaluation results: • The state government, to decide what program to implement; • The federal government, to decide whether to support the state’s decision and whether to transfer aspects of the project to other states • Several lobbying groups (nursing home associations), to choose their positions on the topic

9. Step 2. Examine concerns and assumptions • Determine concerns • In the example • Paperwork burden for quality assurance • Cost of the quality assurance process diverts money from resident care • Helped not only to identify problems but also to facilitate solutions • Attend not just to clients’ medical needs but also to their psychological and social ones • Identify and clarify each decision maker’s assumptions • In the example • The state must play a policing role • The state should adopt the role of change agent

10. Step 3. Add your observations • Examine reports of problems to see that they are, indeed, real • In the example • Literally follow the quality assurance team through a nursing home and draw a flowchart of the process   • Observational studies can add substantial explanatory power and credibility to an evaluation • Stories have powerful explanatory value, often more than the statistical conclusions of the evaluation

11. Step 4. Conduct a mock evaluation • Keeps the decision maker informed and involved • Similar to a real one except that experts’ opinions replace much of the data • Critics: findings that may be proven wrong • Supporters: ethics of withholding information • Speculate how findings might affect actions • In the nursing home study • Asked the group to consider what they would do differently • The discussion revealed that evaluation will find existing efforts were inefficient and ineffective • Developed an alternative method of nursing home quality assurance

12. Step 5. Pick a focus • Decision makers usually expand the scope of the upcoming evaluation • Fixed resources force us to choose which decision makers’ uncertainties to address • For example, a sequence of decisions that affected whether evaluation findings would lead to action. • Some factors in the decision-making process may be beyond the expertise of the evaluation team. • For example, the evaluators might not be qualified to assess the mood of Congress. • Limited budgets means that specificity in some areas must be sacrificed to gain greater detail elsewhere

13. Step 6. Identify criteria • Set the evaluation criteria based on program objectives and proposed strengths and weakness of the program • The nursing home evaluation, we used these criteria to evaluate the process: • Relation to regulatory action • Ease of use • Reliability • Validity • Impact • Cost • Divided the state into three regions • A panel of experts rated numerically the severity of different violations

14. Step 7. Set expectations • Decision makers express what they expect to find • It identifies the decision makers’ biases • It gives a basis for comparing evaluation findings to the decision maker’s expectations • In the nursing home example, decision makers expected that the alternative method would be slightly better than the current method, but they were surprised at how much better it performed. • Substantial cost savings • Improvements in effectiveness

15. Step 8. Compare actual and expected performance • Collect data to compare actual and expected performance. • Replace “actual” and “expected” performance comparisons with comparison of the control (or currently operating) and experimental (new) methods.

16. Step 9. Examine sensitivity of actions to findings • Find threshold above which the actions would change • In the nursing home example • One method of quality assessment 5 percent more expensive than another. • Savings of 20 percent would induce decision makers to change • Sensitivity analysis allows decision makers to modify their confidence in the evaluation findings.

17. Take Home Lesson Program Evaluation can be Improved through orientation to Policy Maker’s Decisions