Mixing the Boundaries of Research: Examining Mixed Methods from a Participatory/Use Perspective

1. Mixing the Boundaries of Research: Examining Mixed Methods from a Participatory/Use Perspective Robert Shumer University of Minnesota Kim Sabo Flores Consultant

2. Fourth Generation Evaluation: Guba and Lincoln (1989) • The first generation refers to studies that were concerned with measurement, with a focus on testing and mathematical representations of data. This was referred to as the “measurement generation” Evolution of Evaluation and Research

3. The second generation focused more on description. • Citing the 8 Year Study conducted by Ralph Tyler at Ohio State, the authors discuss how the efforts to measure change led to the development of formative evaluation and research, where there was a focus on the process as well as the outcomes 2nd Generation

4. The third generation added the elements of judgment • Evaluators and researchers had to judge how the standards and outcomes expected (theories included) were to be assessed and how explanations about a study’s value and worth were to be presented. 3rd Generation

5. Believed knowledge is a constructed phenomenon • The evaluator assumes a managerial role; involves stakeholders and others with an interest in the study in both the design and the measurement of the investigation. • The focus is on interactive, negotiated studies that have participants and others engaged in the investigative process. 4th Generation

6. Fourth Generation Evaluation is based upon: • Responsive focusing ---determining what questions are to be asked and what information is to be collected on basis of stakeholder inputs • Constructivist methodology: carrying out the inquiry process within the ontological and epistemological presuppositions of the constructivist paradigm Essential Elements

7. Based on constructivist theories of learning • Piaget • Vygotsky • Bruner • Dewey • Learning is a function of experience, reflected upon, placed in learner’s schema, and then enlarged/modified by subsequent experiences Educational Foundation

8. Confidentiality • Bias • Violation of trust Research Issues Associated with 4th Generation Evaluation/Research

9. Meaning is derived from use and application • Vygotsky – learning is a social endeavor…thus interaction is key • Expansion of understanding is based on dialogic process --- efforts to compare different forms of reality and meaning Purpose of Constructivist Learning/Evaluation

10. Based on negotiated planning with stakeholders • Based on process of agreeing upon measures and action • Based upon continuous interaction of various appropriate approaches: can be quantitative, qualitative, or mixed • Mixed appears to be most complete because it allows for most points of view/perspective Rigorous Research and Evaluation

11. Differences between Quantitative, Qualitative, and Mixed Methods • Greatest challenge is mixing methods….planning from the beginning how data will be recorded and for what purpose • Best mixed approaches plan for integration of data at the beginning; failure to do so usually doesn’t allow for good use of multiple sources Patton Framework

12. Kim Sabo Flores : youth participatory evaluation • Youth participate because they best understand their culture….can construct meaning through their experience • Evaluation and understanding are inseparable – yet assumed young people can’t think, plan, act, and reflect • Use methods and approaches that are part of youth culture – talking, acting, showing, modeling Examples

13. From Beyond Resistance • UCLA: Institute for Democracy, Education, and Access Ernest Morrell • An example of critical research: critical research is collaborative; requires proximity/intimacy with people • Trustworthiness is better criterion than objectivity • Youth carry out research projects in the community • Develop surveys, interviews, focus groups • Develop Educational Bill of Rights • Youth run/win presidency/v.presidency of school…work for change in community A Story of Youth Participation

14. A model of a mixed approach…crossing the boundaries of university and community • Minnesota AmeriCorps evaluation/research: 5 year project • Engaged graduate students in course evaluating community –based programs • Conducted interviews, developed surveys, monitored documents, conducted cost-benefit studies, individual profiles, case studies of Corps programs and of individuals (special needs woman…42 page report • All data designed to tell complete story: % of Corps members who dropped out (Native Am/Af. American), benefits of program; policy implications (affect on SSI for Handicapped),……..cost effectiveness of money spent, etc….. • Focus of evaluation on use….by programs, policy people, agencies Shumer

15. Research in Service-Learning…Raffles School….elite secondary school…top 2% of students • Conduct year long project in service-learning • Required to develop an evaluation/research plan that tracks their work, measures their progress, and provide evidence of learning and of impact • Service-learningpartnership.org/research/IFA Youth Model in Singapore

16. On my honour, I promise that I will do my best, To do my duty to God and to the Republic of Singapore, to help other people and To keep the Scout Law. • The Scout Promise is indeed a powerful driving force. The commitment to self-awareness, discipline, and compassion are timeless qualities that exemplify the scouting movement. Developing the Idea

17. While many scouts in Singapore are actively pursuing progress badges and various awards, the Extension Scouting scene in Singapore is not as vibrant. This is attributed to various factors, such as a lack of public awareness and lack of support from the Singapore Scouts Association (SSA). In fact, most mainstream Scouts are unaware of Extension Scout activities or even of the fact that Extension Scout troops exist. • The Extension Scouts in Singapore form a diversified group. There are three (3) main categories: physical disabilities, mental disabilities and social disabilities. Developing the Idea

18. Researching Stage: finding out about topic, programs Planning Stage: deciding what to do in program Activities: defining the various activities they would develop over the course of the year Outcomes: determining what impact they had on the Extension Scouts and themselves Future Plans: what they would do to continue development of Extension Scouts Stages of Project

19. These are 16 year olds conducting research/evaluation! • Learn to use evaluation as an active method to get feedback before, during, and after their activities • Reflective components all along the way…learning about Special Scouts…and about themselves • Produces engaged citizens who make change Comments on Extension Scout Program

20. Take 5 minutes to develop plan to evaluate this conference: determine the important questions to ask. • Discuss answers…what were your perceptions and did they differ from others? • What recommendations would you make to leadership to change things for the better? Let’s Do It, Too

21. What ideas/activities will you implement in your home setting after the conference that will focus on 4th Generation Evaluation? • How will you make it participatory and channeled toward use? By whom? Going Home and….

22. Questions and comments And….