Participant-Oriented Evaluation

1. Participant-Oriented Evaluation Prepared by: Daniel Wagner Jahmih Aglahmie Kathleen Samulski Joshua Rychlicki

2. Participant-Oriented Evaluation • Evaluation approach focusing on the participants • Participants have interest in the results • Developed in response to the needs and interests of those associated with the evaluation

3. Characteristics of a Participant-Oriented Approach • The evaluation is dependent on inductive reasoning • The evaluation utilizes data from multiple sources • The evaluation does not follow a standard plan • The evaluation records multiple point-of-views or perspectives

4. An Evolved Approach Other evaluation approaches have been adapted into participant-oriented approaches.

5. Three Theorists Involved in Approach • Robert Stake • Egon Guba & Yvonne Lincoln • Michael Q. Patton

6. Robert Stake • First evaluation theorist to introduce Participant Oriented Evaluation into the field of education. • Introduced the theory of Responsive Evaluation which focuses on “re-directing data gathering and interpretative efforts around emerging issues of importance to program practitioners in the evaluation setting” (Abma, 2005) • Created the Countenance Framework in 1967. The model refers to the two faces of evaluation: description & judgment. • Description includes the evaluator’s observation and list of benchmarks for the activities being evaluated. Judgment is the evaluator’s overall rating of merit.

7. Guba & Lincoln • Built on Stake’s Responsive Evaluation theory and came up with the method of fourth generation evaluation. This method was used when listing the generations of evaluation. • Generations include: - 1st generation: evaluators measure participants - 2nd generation: evaluators describe participant - 3rd generation: evaluators judge participants - 4th generation: evaluators respond to participants. • The evaluation includes 12 steps. • Contacting • Organizing • Identifying Stakeholders • Developing within-group Joint Constructions • Enlarging joint stakeholder constrictions • Sorting out Resolved Claims, Concerns, and Issues • Prioritizing Unresolved Items • Collecting Information • Preparing for Negotiation • Carrying Out Negotiation • Reporting • Recycling

8. Michael Patton • Identified more than 100 evaluations • Known for his words: “Research is aimed at truth. Evaluation is aimed at action” • He focuses on ‘identifying and working with key participants in organizations who can develop a vision for the value of gathering information and using it to improve the functions of the organization in ways responsive to each situation.’ (Williams, 2000)

9. Key Considerations • (Fitzpatrick. Sanders, and Worthen, 2004), report that most proponents of participant-oriented evaluation view the participants in the object of evaluation as key to the evaluation. • As opposed to evaluating in a top-down manner, participant-oriented evaluation is conducted with a bottom-up approach • Qualitative methods are used much more than quantitative methods. • Jacobs (1985) describes that qualitative methods, “seek to describe not the frequency but the behaviors, beliefs, and feelings of the respondents in a setting” (p. 5). • Jacobs continues to describe qualitative data as “direct quotations, observations, personal impressions, analyses of records and documents, summaries of unobtrusive measures, case studies, and the like” (pp. 5-6). • Jacobs maintains the naturalistic design process in participant-oriented evaluation is emergent and the setting is changeable and natural. • As Fitzpatrick, et al. report, “the evaluator is in the role of the learner, and the stakeholders serve as teachers” (p. 149).

10. Advantages of This Approach • Gain new insights and usable theories • Educational, social or corporate programs • Flexibility • Addresses user training as a feature of the assessment • Consideration to related variables • Numerous data collection techniques • Empowers stakeholders • Reaches even those who are at times left out of the evaluation process • Add a political constituent • Cultivates activism (Fitzpatrick, Sanders, and Worthen, 2004)

11. Disadvantages of This Approach • Subjective • Relies heavily on human observation and personal perspective • Expensive • Labor-intensive • Time-consuming and extensive • Evaluator loses objectivity • may become caught up in advocacy trends (Fitzpatrick, Sanders, and Worthen, 2004)

12. Implementation of Participant-Oriented Evaluation When to use • When cost is not an issue • When time allows • When responding to stakeholders • When evaluator can take on the role of the learner When not to use • When detailed quantitative data is essential • When there is “controlled, laboratory or contrived setting” (Jacobs, 1985).

13. Scenarios for Use of Participant-Oriented Evaluation • Social reform • Evaluating programs developed to improve the environment • Education • Evaluating new and existing instructional programs on all levels • Evaluating instructional materials • Evaluating professional development • Corporations • Evaluating new and existing training programs • Government • Evaluating programs • Whenever qualitative formative evaluation is conducted

14. References Abma, T.A.(2005). Responsive Evaluation. Evaluation and Program Planning: Online version. Retrieved September 28, 2007 from World Wide Webhttp://64.233.167.104/search?q=cache:DyCD4R_ToE0J:mailer.fsu.edu/~sullivan/SEA_Newsletter/Responsive_Evaluation.pdf+what+is+responsive+evaluation&hl=en&ct=clnk&cd=6&gl=us Fitzpatrick, J., Sanders, J., & Worthen, B. (2004). Program Evaluation: Alternative Approaches and Practical Guidelines, (3rd Ed.), Boston: Allyn & Bacon. Jacobs, R. L. (1985). Naturalistic Inquiry and Qualitative Methods: Implications for Training and Development. Columbus, OH: Ohio State University. (ERIC Document Reproduction Service No. ED305518). Williams, D. D. (2000). Evaluation of learning objects and instruction using learning objects. In D. A. Wiley (Ed.), The Instructional Use of Learning Objects: Online Version. Retrieved September 27, 2007, from the World Wide Web: http://reusability.org/read/chapters/williams.doc