Measuring Program Success: The Importance of Program Evaluation

1. Measuring Program Success: The Importance of Program Evaluation OCAN 4th Annual Conference September 9, 2003 Ramona S. Thomas, KnowledgeWorks Foundation David B. Swedlow, Ohio College Access Network

2. Objectives • Evaluation Overview • Working Definition • Types of Evaluation • Evaluation in Practice in College Access • Why do evaluation? • Evaluation Framework • Principles and Suggestions for Evaluation • OCAN Evaluation Overview • Additional Resources

3. Evaluation Overview - Definition • Program evaluation is not… • A method of quality assurance for published or public documents • An account of expenditures or use of resources • A status report on the development or implementation of a particular program

4. Evaluation Overview - Definition • Program evaluation is the process through which research techniques are used to assess design, implementation, success, and impact • The primary objective is to assess and determine whether goals, objectives, and desired outcomes were reached and, to the extent possible, explain why and how

5. Evaluation Overview - Types • Based on length (or when conducted) • Formative evaluations • assess programs (or program activities) during their early stages or while still in progress • focus on process • Summative evaluations • assess the effectiveness of a program at the end of the program activities (or over an extended time frame) • focus on outcomes

6. Evaluation in PracticeWhy do evaluation? • Program Evaluation • Provides an opportunity to learn more about what you do (processes and activities) and the effects of your work (outcomes) • Can help improve program goals, objectives, and service delivery • Meets private and public accountability standards and requirements

7. Evaluation in Practice - Framework • Common approach to program evaluation • Identify problem • Develop program • Design/conduct evaluation • Generate evaluation report • Disseminate evaluation report to funders (or select group)

8. Evaluation in Practice - Framework

9. Evaluation in PracticePrinciples and Suggestions • Align with program goals and objectives • Clarify the purpose, goals, objectives, and desired results of your program • Identify short-term, intermediate, and long-term outcomes • Logic models are useful planning tools • Objectives should encompass activities that reflect the experiences of those you serve • Ensure that results focus on outcomes of your activities

10. Evaluation in PracticePrinciples and Suggestions • Have clearly defined questions, outcomes, expectations, and uses • Be clear about what you really want to know about your program and its outcomes • Try to answer what works, how it works, and why it works • Identify clear and concise evaluation objectives (they should match your program’s services and/or activities) • Carefully prioritize multiple purposes

11. Evaluation in PracticePrinciples and Suggestions • Follow multiple approaches • Collect and analyze data from multiple sources and perspectives • Use various data collection methods (e.g., surveys, interviews, focus groups, documents, other available data) • Establish a baseline and collect baseline data • Measure progress toward long-term goals • Pay attention to rigor, flexibility, and cost-effectiveness

12. Evaluation in PracticePrinciples and Suggestions • Include time for reflection and sharing • Are you surprised by the findings or are they what you expected? In what ways? • What new information about your program and/or its outcomes has surfaced? • How can you use evaluation to improve your program and service delivery? • Are there others who could benefit from what you have learned? What is the best way to share it with them?

13. OCAN Evaluation Overview • Goals of the OCAN Evaluation • Examine the effectiveness and influence of OCAN as a statewide, college access organization • Assess sample of OCAN member programs • Examine the partnership among KnowledgeWorks Foundation, OCAN, the Ohio Board of Regents (OBR), and others

14. OCAN Evaluation Overview • Component on OCAN Member Programs • Contextual and demographic information • Activities, assistance received from OCAN, perceptions about the quality and value of OCAN • Program administration and effectiveness of services provided to participants • Member programs’ evaluation/assessment activities • Relationship with OCAN

15. Resources • Evaluation Toolkit. Project STAR for Corporation for National Service (http://www.projectstar.org/star/Library/ toolkit.html) • Evaluation: A systemic approach. P. Rossi & H. Freeman. (1993). Sage Publications: Newbury Park, CA. • Evaluation Handbook. W.K. Kellogg Foundation (http://www.wkkf.org/Pubs/ Tools/Evaluation/Pub770.pdf)

16. Objectives • Evaluation Overview • Definition • Types of Evaluation • Evaluation in Practice in College Access • Why do evaluation? • Evaluation Framework • Principles and Suggestions for Evaluation • OCAN Evaluation Overview • Additional Resources

17. Data for Evaluation – Objectives • How do we prepare for evaluation? • What goals do we wish to achieve? • What are some outcomes for college access? • What are some outcomes for college success? • How can the web-enabled student tracking database support evaluation?

18. How do we prepare for evaluation? • What are our program goals & objectives? • Are student needs tied to the objectives? • Are activities designed to meet objectives? • Do outcomes relate to the activities? • Will the achievement of the outcomes advance the goals of the program?

19. What are some goals of our programs? • To increase the college-going rate of students • To increase the number of community college students that transition to 4-yr colleges • To increase the number of low-income students that can afford the rising costs of tuition & fees • To increase retention and graduation rates of underserved, low-income and first-generation college students • To increase the number of adults in the population with bachelors degrees

20. What are some outcomes for access? • Attending a college fair or visit • Meeting with advisor on college applications • Submitting a completed FAFSA on time • Performing an electronic scholarship search • Taking the SAT or ACT • Shadowing with a mentor at a worksite • Applying for a last-dollar scholarship • Completing requirements for high school graduation and college admission

21. What are some outcomes for success? • Admitted to postsecondary education • Received financial aid package from college • Awarded last-dollar scholarship • Enrolled in postsecondary institution • Made it! – first term, successfully • Retained! – completed first year successfully • Submitted transcript for scholarship renewal • Selected major • Graduated! - 2-yr or 4-yr degree

22. What can we collect on the student tracking system? • Demographic data • Parent/guardian/contact information • Financial data & college cost data • Secondary school academic progress • Participation in college access activities • College entrance exam scores (SAT/ACT) • Scholarships and grants awarded • Postsecondary education progress (enrollment, retention, grade point averages, majors, transfers, graduation)

23. What do we need to do to prepare to implement the student tracking system? • Gain commitment from board and staff • Communicate with school districts • Secure confidentiality signoffs • Collect social security numbers • Create individual accounts for advisors, staff • Identify participation activities and events • Work with OCAN to establish baselines