1 / 35

Developing Consensus on Common Models and Standard Set of Indicators: FCS and 4-H Case Studies

Developing Consensus on Common Models and Standard Set of Indicators: FCS and 4-H Case Studies. 2007 CSREES ADMINISTRATIVE Officers Conference Seattle, WA. Mary McPhail Gray Deputy Administrator Families, 4-H and Nutrition 800 9th Street, SW Washington, DC 20024 202-720-2908

marin
Télécharger la présentation

Developing Consensus on Common Models and Standard Set of Indicators: FCS and 4-H Case Studies

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Developing Consensus on Common Models and Standard Set of Indicators: FCS and 4-H Case Studies 2007 CSREES ADMINISTRATIVE Officers Conference Seattle, WA Mary McPhail Gray Deputy Administrator Families, 4-H and Nutrition 800 9th Street, SW Washington, DC 20024 202-720-2908 mgray@csrees.usda.gov Cooperative State Research, Education, and Extension Service http://www.csrees.usda.gov

  2. CES Human Resources are Extensive In 2005 U.S. CES staffing totaled 14,650 (FTE) Of which: • 8,925 county agents/advisors/educators (B. Sc. or M. Sc.; in 3,038 counties) • 4,050 specialists (Ph.D. faculty; many with joint appointments) • 715 directors/supervisors • 960 administrative support Cooperative State Research, Education, and Extension Service http://www.csrees.usda.gov

  3. Federal Partner Interests (Goals) • Impact Reporting to Administration (White House) and Congress • Clarity of Program Marketing/Justification • Identification of New Public and Private Funding Sources Cooperative State Research, Education, and Extension Service http://www.csrees.usda.gov

  4. New Plan of Work Reporting • Bart Hewitt OPA summarized the wealth of information being reported in new electronic Plan of Work system • 85 2007 Plans of Work received, 50 were combined submissions and 35 were single entity • 1018 Planned Programs, with from 1 to 76 programs reported per plan (with median of 9 planned programs) Cooperative State Research, Education, and Extension Service http://www.csrees.usda.gov

  5. Knowledge Areas Reported by Planned Programs • For 2007 Plans of Work, each program included up to 10 knowledge areas (KAs) and outcomes were not linked to specific KAs • For 2008 Plans of Work, each program may include up to 20 knowledge areas • Beginning with the 2008 Plans of Work, outcomes are linked to specific KAs, making it easier to drill down and locate specific outcomes Cooperative State Research, Education, and Extension Service http://www.csrees.usda.gov

  6. Top Ten Program Themes by Frequency in 2007 POW • 1. Youth Development and 4-H • 2. Rural and community Development • 3. Natural Resources and Environment • 4. Health • 5. Plant Production • 6. Animal Systems • 7. Nutrition • 8. Agricultural Systems General • 9. Sustainable Agriculture • 10. Food Safety Cooperative State Research, Education, and Extension Service http://www.csrees.usda.gov

  7. Many programs, many outcomes • Wealth of data -- Plans of Work and Annual Reports, Planned Programs, KAs • Wealth of outcomes associated with plans, programs, KAs • Can outcomes be associated with meaningful and measurable indicators? • How many different indicators will we have? • Who will combine them, how, and when? Cooperative State Research, Education, and Extension Service http://www.csrees.usda.gov

  8. 2007 Top KAs in 4-H or FCS By Full-Time Equivalent Staff • 802, Human Development and Family Well-Being, 4.2% • 806, Youth Development, 10.5% • 703, Nutrition Education and Behavior, 3.4% • 801, Individual and Family Resource Management, 2.7% Cooperative State Research, Education, and Extension Service http://www.csrees.usda.gov

  9. Benefits of Consensus Indicators (from Silliman, 2007 review) • Monitoring programs consistently • Mobilizing public opinion • Tracking goal achievement • Focusing accountability • Informing program improvements • Standardizing program evaluation Cooperative State Research, Education, and Extension Service http://www.csrees.usda.gov

  10. 4-H and FCS Program Impact Reporting Challenges • Breadth of program activity • Job stress of State and County Faculty • Identification with constituents for rewards • Lack of evaluation resources (training, time, budget, commitment) • Persuasion to use the Logic Model Cooperative State Research, Education, and Extension Service http://www.csrees.usda.gov

  11. Program Development Planning-Implementation-Evaluation Program Action- Logic Model Inputs Outputs Activities Participation Outcomes-Impact Short term Medium Term Long Term What we invest Staff Volunteers Time Money Research base Materials Equipment Technology Partners Conduct workshops, meetings Deliver services Develop products, curriculum, resources Train Provide counseling Assess Facilitate Partner Work with Media What we do Who we reach Participants Clients Agencies Decision-makers Customers Satisfaction What the short term results are Learning Awareness Knowledge Attitudes Skills Opinions Aspirations Motivations What the medium term results are Action Behavior Practice Decision-making Policies Social Action What the ultimate impact(s) is Conditions Social Economic Civic Environmental Priorities: Consider: Mission Vision Values Mandates Resources Local Dynamics Collaborators Competitors Intended Outcomes Situation Needs and Assets Symptoms versus problems Stakeholder engagement Assumptions External Factors Evaluation Focus - Collect Data – Analyze and Interpret - Report University of Wisconsin Logic Model http://www.uwex.edu/ces/pdande/evaluation/evallogicmodel.html

  12. Federal Impact Reporting Response • Persuasion and Motivation regarding goals • Deductive (national) identification of desired program outcomes • Inductive identification of state program outcome statements Cooperative State Research, Education, and Extension Service http://www.csrees.usda.gov

  13. Federal/LGU System Process of Building Consensus • An FCS case • A 4-H case Cooperative State Research, Education, and Extension Service http://www.csrees.usda.gov

  14. Indicators for 4-H Youth Development Program • State 4-H Program Leaders identified need for quality and accountability indicators at 2006 annual meeting • National Program Leader and representative from National 4-H Council charged with forming a committee of State 4-H Program Leaders to develop indicators • National 4-H Council committed private funds to support the committee’s efforts • Review of the literature was commissioned and completed by Dr. Ben Silliman, NC State Cooperative State Research, Education, and Extension Service http://www.csrees.usda.gov

  15. 4-H National Mission Mandates • Science, Engineering, and Technology (SET): agriculture and life sciences, family resource management, human development and family • Youth as Citizens (YAC): civic involvement, community service, leadership in youth and community organizations • Healthy Living (HL): nutrition and physical activity, safety, mental health Cooperative State Research, Education, and Extension Service http://www.csrees.usda.gov

  16. Most Frequent 4-H Programs by Theme • Youth Development (General) • Science, Engineering, Technology (SET) • Youth Citizenship (YC) • Healthy Living (HL) • Other (OT) Ranked in order of frequency, except ‘Other’ Cooperative State Research, Education, and Extension Service http://www.csrees.usda.gov

  17. Examples of 4-H Outcomes, Youth Development • Youth become caring and contributing members of society through positive experiences in a diverse 4-H Youth Development program • Youth demonstrate mastery and competencies needed to become engaged citizens by -- assuming leadership positions in communities -- developing and implementing action plans to address community needs -- becoming productive members of the work force Cooperative State Research, Education, and Extension Service http://www.csrees.usda.gov

  18. Examples of 4-H Outcomes, SET Program • Knowledge: Young people will understand role of science, engineering, technology in the contemporary world • Behavior: … will develop skills and abilities for success in education and careers that rely on application of science and technology • Condition: 4-H program is contributing to the long-term economic and social stability of a technologically advanced workforce Cooperative State Research, Education, and Extension Service http://www.csrees.usda.gov

  19. Examples of 4-H Outcomes, Youth Citizenship • More youth are involved in leadership roles in their communities • Youth governmental board developed to influence legislations affecting youth • Youth are involved in community organizations as partners Cooperative State Research, Education, and Extension Service http://www.csrees.usda.gov

  20. Examples of 4-H Outcomes, Healthy Living • Improve the lives of citizens through positive human development, healthy lifestyles and nutrition, and lifelong wellness. • Increase the number of youth who can maintain a healthy lifestyle – learning how to select a balanced, nutritious diet • … developing skills to prepare food themselves • … developing skills and positive attitude to make lifelong physical activity a lifelong habit Cooperative State Research, Education, and Extension Service http://www.csrees.usda.gov

  21. Science, Engineering, & Technology Research (Silliman, 2007) • National Science Standards: Thinking, not just memorizing facts • 4-H SET Abilities needed for academic, career success: • Science Anchors (content: biology, chemistry, geology, physics, etc.) • Science Abilities (processes: observation, hypothesizing, experimentation, analysis, etc.) Cooperative State Research, Education, and Extension Service http://www.csrees.usda.gov

  22. Youth-as-Citizens Research (Silliman, 2007) • Civics: Impact/engagement depends on knowledge of how government and organizations work • Service: Positive attitudes, experiences in service increase adult service; increased academic, life skills; reduced risk behaviors • Leadership: Engagement increases leadership and service activities (more as duration and responsibility increase) Cooperative State Research, Education, and Extension Service http://www.csrees.usda.gov

  23. Healthy Living Research (Silliman, 2007) • Nutrition/Dietary Habits: Unhealthy dietary habits and limited activity placing youth at risk into the future • Safety: Unintended injuries declining but practical activities require training • Mental Health: Stress management critical to varied physical and emotional health outcomes Cooperative State Research, Education, and Extension Service http://www.csrees.usda.gov

  24. Example: 4-H SET Outcomes • Short-Term Outcome Indicators • Number increasing knowledge/skills • Number improving attitude/aspirations • Number increasing science process skills (observation, comparison, hypothesis, analysis, problem solving) • Long-Term Outcome Indicators • Number demonstrating changed behavior in science learning (e.g., school success, pursue additional learning) • Number applying science process skills (advanced project, mentoring others, young adult education and career choices) Cooperative State Research, Education, and Extension Service http://www.csrees.usda.gov

  25. Example: 4-H SET Program Quality Indicators • Essential Elements of Inquiry: 1) asking questions; 2) framing hypotheses; 3) gathering and interpreting evidence; 4) revising explanations to fit data • Effective environment: Learner-centered… Knowledge-centered…Assessment-centered…Community-centered • Science-by-Inquiry: guided content and process explanations • Science-as-Inquiry: shared discovery and reflection Cooperative State Research, Education, and Extension Service http://www.csrees.usda.gov

  26. Next Steps for 4-H Indicators Work • Evaluating for Impact Committee co-chaired by Dr. Suzanne Le Menestrel and Dr. Mary Arnold, Oregon State U., will use commissioned paper to recommend indicators in each of the three mission mandate areas • Evaluating for Impact Committee will develop recommendations for measuring indicators • Recommendations will be reviewed by original committee of State 4-H Program Leaders Cooperative State Research, Education, and Extension Service http://www.csrees.usda.gov

  27. FCS Impact Reporting Strategic Planning • 1999 Annual Meeting of FCS State Leaders identified national program impacts as priority • Committee of State Leaders identified six major program themes supported by 1/3 or more of states by reviewing 2000-2004 POW’s • Six committees of state specialists led by Program Leaders built consensus for program outcomes known from research literature Cooperative State Research, Education, and Extension Service http://www.csrees.usda.gov

  28. FCS Foundational work to build upon • Southern Regional work on Outcome Evaluation • National Extension Cares childcare initiative • National Extension Parenting Education Model • Healthy People-Healthy Communities national initiative • Financial Security Specialists’ decade of work • CSREES Reporting System, eXtension and One Solution Cooperative State Research, Education, and Extension Service http://www.csrees.usda.gov

  29. FCS Named Program Themes • Childcare and Afterschool Programs • Community Nutrition and Food Safety • Financial Security • Health • Housing and Environment • Parenting Cooperative State Research, Education, and Extension Service http://www.csrees.usda.gov

  30. FCS Program Outcome Examples: Childcare • Childcare personnel increase their knowledge of developmentally appropriate guidance • Childcare personnel show an improvement in practice after CE education • Parents demonstrate greater knowledge in selecting appropriate child care for their family Cooperative State Research, Education, and Extension Service http://www.csrees.usda.gov

  31. FCS Program Outcome Indicators: Financial Security • Program participants increase their ability to achieve financial self-sufficiency and stability through increasing savings, reducing debt, setting goals, etc. • Program participants increase their ability to achieve financial security through regular savings, retirement planning, establishing record systems, managing health care costs, etc. Cooperative State Research, Education, and Extension Service http://www.csrees.usda.gov

  32. FCS Program Outcome Indicators: Community Nutrition, Food Safety and Security • Program attendees acquire the knowledge to change their diet toward the recommendations of the Dietary Guidelines. • Program attendees learn to avoid cross contamination in food preparation • Program attendees make a shopping list, compare prices, use sales fliers, etc. Cooperative State Research, Education, and Extension Service http://www.csrees.usda.gov

  33. Next Steps in Building Consensus • Leadership from 4-H Evaluating for Impact Committee • Leadership from FCS State Leaders • Leadership from F,4-H & N and Planning and Accountability Units at CSREES Cooperative State Research, Education, and Extension Service http://www.csrees.usda.gov

  34. How Can you Help? • What is the public value of our work in youth and in family? • How can you evaluate: the increased access to meaningful information, the increased equity or justice achieved, and the short or long term savings accrued? • Everyone needs an “elevator speech!” Cooperative State Research, Education, and Extension Service http://www.csrees.usda.gov

  35. Thank You for Your Interest and Commitment! • Mary McPhail Gray, Ph.D. • mgray@csrees.usda.gov • 202-720-2908 Cooperative State Research, Education, and Extension Service http://www.csrees.usda.gov

More Related