200 likes | 315 Vues
How does your program work?. Using program theories to guide your evaluation Mary S Marczak Research and Evaluation Specialist, University of Minnesota Extension. Theory of Life: Calvin as the Evaluator. Do you have a program theory?.
E N D
How does your program work? Using program theories to guide your evaluation Mary S Marczak Research and Evaluation Specialist, University of Minnesota Extension
Do you have a program theory? All programs whose objectives are to cause changes in participants are inherently theory based. Wilder Research Poll #1
To what extent do you agree/disagree with the following statement? “The term theory as used by program developers and implementers typically is used to mean a vague notion or hunch not usually based on social science information.” • Strongly Disagree • Tend to Disagree • Tend to Agree • Strongly Agree
Program theory and evaluation • Building relationships and coming to a shared understanding • Test the program theory (Camp K, MN 4-H) • Formative evaluation (performance indicators, activities-theory fit, implementation quality) • Understand why programs do or do not work (theory failure? Or implementation failure?) • Attributing outcomes to the program • What else?
Program theory: what is it? A plausible and sensible model of how a program is supposed to work L. Bickman, 1987
A systematic delineation of cause and effect relationships M. Scheirer, 1987
Identification of resources, activities, and outcomes of a program and the causal assumptions that connects these J. Wholey, 1987
Scientific explanations of how social change is possible C. McClintock, 1987
How should theory be constructed? Two camps • Program theory should be constructed by key program stakeholders (Wholey, Patton) • Program theory should stress social science theory and knowledge (Chen, Rossi) Poll #2
In which “camp” do you put yourself as an evaluator? A) Theory should be constructed by program stakeholders B) Theory should be constructed based on social science theory and knowledge C) Some of both but more heavily influenced by stakeholders D) Some of both but more heavily influenced by social science theory E) They are equally important
Pulling it together: Huey-tsyh Chen Two categories of theories that should be explicit in any program to be evaluated. Each have a role in terms of what we might evaluate. • Descriptive • Prescriptive
Descriptive theories Assumptions about the nature of the problem and the population affected, limiting conditions, and what aspect of the problem should be solved.
Descriptive theory example: http://eci.ext.msstate.edu/info/theory.pdf
Prescriptive theories How to go about solving the problem or bringing about change; how should activities be constructed? How should activities/intervention be implemented? What is the causal connection to social change?
Prescriptive theory example: J Baur and S Son, 2009 Powerpoint
Pulling it all together example: Danes, 2009 “Staying Resilient in Times of Change” http://cehd.umn.edu/fsos/Centers/RuralMNLife/
Defining the constructs- what are the key components, theory behind the construct, measures. Word Documents
References • L. Bickman (ed.) Using program theory in evalulation. New Directions for Program Evaluation, no. 33. San Francisco: Jossey-Bass, Spring, 1987. (multiple authors) • Chen, H-T. Issues in constructing program theory. New Directions for Program Evaluation, no. 47, Fall 1990. • Chen, H-T and Rossi, P. H. The theory driven approach to validity. Evaluation and program Planning, 1987, 10, 95-103. • Patton, M. Q. A context and boundaries for theory-driven approach to validity. Evaluation and Program Planning, 1989, 12, 375-377. • Wilder Research Center. Program theories and logic models. St. Paul, MN: Wilder Research Center. http://chicano.umn.edu/pdf/resourcesProgram%20Theories%20Example%20Logic%20Model.pdf