Selecting an Appropriate Design for the Evaluation

1. Selecting an Appropriate Design for the Evaluation

2. What do we mean by a design for the evaluation? • The design of the evaluation is the arrangement that will make sure it reliably tells you what you need to know • An appropriate design will show you whether you actually got results, and whether those results were likely due to your actions or the circumstances you created, or to other factors

3. Why should you choose a design for your evaluation? • So your evaluation will be reliable • So you can pinpoint areas you need to work on, as well as those that are successful • So your results are credible • So you can identify factors unrelated to what you’re doing that have an effect – positive or negative – on your results and on the lives of participants • So you can identify unintended consequences (both positive and negative) and correct for them • So you’ll have a coherent plan and organizing structure for your evaluation

4. When should you choose a design for your evaluation? • In the ideal, when you’re planning the program and evaluation, before you start any implementation • In reality, for many organizations, at the beginning of a program cycle, or as a new group of participants enters

5. Who should be involved in choosing a design for your evaluation? • It’s extremely helpful to include someone with research experience to help you decide on an appropriate design, and to help you understand what each possible design entails

6. How do you select an appropriate design for your evaluation? • Take into consideration the necessary research considerations • Understand threats to internal validity ( whether the intervention produced the change) • History • Maturation • The effects of testing or observation on participants • Changes in measurement • Regression toward the mean • The selection of participants • The loss of data or participants • The nature of change • A combination of the effects of two or more of these • (continued)

7. How do you select an appropriate design for your evaluation? • Understand threats to external validity (generalizability of your findings) • Interaction of testing or data collection and the program or intervention • Interaction of selection procedures and the program or intervention • The effects of the research arrangements • The interference of multiple programs or interventions • (continued)

8. How do you select an appropriate design for your evaluation? • Understand common research designs • Pre- and post- single group design • Interrupted time series design with a single group (simple time series) • Interrupted time series with multiple groups (multiple time series) • Control or comparison group • Choose a design • Consider your evaluation questions • Consider the nature of your program • Consider what your participants and staff will consent to • Consider your time constraints • Consider your resources