About this presentation– For the Reviewer

1. About this presentation– For the Reviewer • I am a member NAE4-HA Research Committee • In 2006 we were invited to submit a proposal for two evaluation workshops (basic/intermediate & advanced intermediate) for the 2007 conference. • I was the leader of the team that planned and presented a proposal for the basic/intermediate workshop • It was accepted and a team of five individuals—with me as the facilitator and a co-presenter—presented a workshop at the 2007 NAE4-HA Conference in Atlanta. • What follows is my section of the workshop on qualitative and questioning techniques for evaluation

2. Questions Can Be the Answer! Furthering understanding and abilities in the magical land of [qualitative] evaluation 2007 NAE4-HA Conference – Atlanta, Georgia G. F. Varrella, Ph.D., WSU Extension-Spokane County

3. Our Discussion • The value and place of qualitative work • Two methods • Basic considerations for generating questions • A bit of background on the “art” of asking questions…and still be tuned into your goals, content, and context!

4. Some days it may feel like all we are every asked is… “Higher”

5. (… and here comes the [sales] pitch) Qualitative evaluation & research is flexible You can free yourself from recipes…which is especially appealing if you love the topic…draw on your expertise and enthusiasm! With a little freedom comes the option to start at a level you that you choose…that fits your audience (making you more nimble)

6. As the process of inquiry continues, you have the right to revise and rewrite your rules and reexamine your suppositions • As the inquiry continues, you will discover which methodologies work best (i.e., qual., quant., mixed) (Freeman, et al., 2007)

7. Higher! • You are asking open-ended questions • Seeking the facts from traditional and “emotional/personal” perspectives • …and you are more able to align your evaluative inquiries and questions to your stated outcomes

8. Interviews • At the most basic level, are conversations. • That, • attempt to understand points of view, • unfold the meaning of experiences, • uncover changes in individuals, groups and communities. From Hrabik, Leahy, Minter, & Westover

9. Guides for Interviews • Be personal and unobtrusive • Obtain direct feedback • Seek understanding “dig deeper” • Observe behaviors and reactions • Obtain rich, detailed data • To be flexible From Hrabik, Leahy, Minter, & Westover

10. Types of Interviews • Informal Conversation • Guided Interview • Structured Interview Adapted from M. Patton, 1990 by Hrabik, Leahy, Minter, & Westover

11. A Nifty Form of Qualitative Work—Appreciative Inquiry(Hammond, 1996) • The key principle of AI (not artificial insemination, you aggies) is that it is a generative process • Data are not always “discovered” and “collected” • The data simply “are” … it depends on your purpose and therefore may be “generated” or “constructed” by various means that fit your purpose

12. AI • We are good at talking about what did not work • Looking for what works [worked] • What do we do well • What has made us feel successful, effective, i.e., good about our work and roles

13. Assumptions of AI • Every group does something well • Our focus becomes our reality • Asking questions influences the group • Linking past successes to future events is powerful • …elements of social justice…

14. AI – Key Things – GV’s Take • Lots of listening • Throw out assumptions and [try to] check your ego at the door • You are not neutral, so be sure to be a good recorder (i.e. listen and record w/o too much translation) • You eventually will have the opportunity to dialog (“discourse” is the $50 term) • It takes more time

15. PS – A Problem to Solve ID problem Analysis of causes Analysis of possible solutions Action plan Action AI – A mystery to be embraced Looking for the good Envision what might be Discourse on “What might be” Innovate: “What will be” AI is not Problem Solving Or is it?

16. The Power and the Jumble • The power is when a group becomes engaged. Remember that listening part and the basic rule that people like to talk about themselves. • At first, the data generated—i.e., from your “field and interview notes” may feel like nothing but a jumble of unorganized “stuff”…

17. A case example: “I thought I smelled something!

18. A Case Example • A needs assessment, a program planning process, and an evaluation (sometimes it does not matter) • What do you value most about 4-H as a parent? • What are the benefits of 4-H for your kids? • What would you hope to see Spokane County 4-H do in the future?

19. Well Suited Documenting and explaining real events Understanding how participants make sense and give meaning to their life Identifying unanticipated or taken-for-granted influences and phenomena Not As Well Suited Testing relationships or establishing cause and effect Addressing the generality or wider prevalence of the phenomena being studied or evaluated Comparing groups or a relationship among variables Predicting outcomes (Schram, 2003) Qualitative Questions

20. Good Questions & Conducting Interviews/Focus G.’s • Keep your goals in mind. • Specific the rationale for the discussion you have with the stakeholders or group • Keep those questions open-ended • Field test your questions • Keep them direct and brief • Consider the age group with whom you will be talking (field testing helps)

21. That you can dare to be different and still be valid and reliable

22. The art of questioning

23. Wait Time – M. Bud Rowe • Waiting for what? • Wait time I • Wait time II • Problems with “I don’t know” (with kids) • Will not occur as often • Can also “Pass” • Can also “Ask another for help” • The posture and attitude of the facilitator is critical

24. Questioning Habits – Penick, et al. • H istory • R elationships • A pplication • S peculation • E xplanation • ??Evaluation??

25. Things to Remember! • Practice your wait time • How can you do this? • How will the student react • Practice your listening skills - all the time!

26. Things to Remember! • Ask one question at a time • Avoid “why” questions whenever possible • Only ask a question when you expect an answer (avoid rhetorical questions)

27. “Well hey, these things just snap right off!”

28. Love the “so what” Short term, intermediate term, and long term outcomes are the Extension Holy Grail If you can’t suspend or curb your opinions & bias…ask for a fresh look from a respected colleague Formative is “”kuhl;” summative goes on a shelf Look for the surprises, it will make it more interesting If you think you havethe definitive answer – think again When you define your questions, attend only to what matters. Using a mixed methodology might be your answer You are not a purely objective observer (so get over it ) Plan your evaluative inquiry as you plan your project and don’t forget to share that so what…in writing! Gary’s Top 10 [at this moment]words from the not-too-wise…