1. TAH Project Evaluation Great - Yet Realistic - Expectations Jeff Sun, Sun Associates

2. Our Background • Initiative-wide Qualitative Evaluation • District-level initiatives • State-level initiatives • Strategic Planning • STEM, Science, Instructional Technology, Social Studies, New Media • Evaluation of Teacher Professional Development • 10 TAH projects (7 of these are current)

3. Qualitative is the Emphasis • Makes the connection between what projects say they want to do and what participants say that projects have done. • This data is invariably qualitative and accounts for a wide range of effects • Quantitative data helps paint the picture in broad outlines... • ...but qualitative data helps you understand what the quantitative data means and to use the evaluation strategically

4. This Presentation • We want to talk about how to • Create a good evaluation • Use evaluation strategically • Work effectively with evaluators

5. Four Functions for Evaluation • Evaluation performs four meaningful functions in a project’s life... • As a part of the proposal • To clarifying a project’s work • To provide formative assessment of Progress and Implementation • Summatively, to account for the project’s success (or lack thereof)

6. First Function - The Proposal • As expressed in the logic map, evaluation helps explain: • Need for the project • The project’s actions in response to those needs • Who the project will serve • What the project expects to produce • All in one tidy package in the proposal text!

7. Issues Related to Function 1 • We need time to develop the evaluation plan • Ideally, we come in when the basic work is set, but the details are not fully fixed • We need space to fully describe the evaluation process • It’s hard to create this plan in 2 paragraphs or a half page • We need connect with those who will be actually implementing the project if funded • Proposal-writers will help you get funded...but it’s often no coincidence that they’re nowhere to be found once you get the grant.

8. Second Function - Clarification • Once you’re funded, the real work begins • Use the evaluation plan as an organizer for talking with partners about • what the project will be doing • expectations for partners and participants • This discussion clarifies goals and develops a shared vision for project success • Out of this discussion come the project’s performance indicators...which we organize into rubrics • It would be nice to have all of your indicators before funding, as part of the proposal • But realistically, there’s never time for that • Projects need their own indicators • ED’s indicators are not the same and cannot replace your own indicators

9. Issues Related to Function 2 • We can’t help if we aren’t at the table • Often, we have to really bug projects to bring us on board early • Evaluation is too often seen as the thing that happens after the work is done...not as the thing that can get work done. • We read the proposal and base the actual evaluation on what is written • This can be a problem when the proposal was written by the now-absent proposal-writer • The clarification discussion - assuming it happens early enough - is where this problem is resolved

10. Third Function - Formative • Collecting the data and sharing it as the project progresses • Measuring what the project is doing against what it said that it would do • Using indicator rubrics to account for data that is highly qualitative

11. Issues Related to Function 3 • Our challenge is to identify data collection points that get at the essence of the project and its impact. • This can be one of the most important parts of our work. • Questions continue the process of clarifying the project work through participant feedback on impact • Next most difficult is connecting directly with participants

12. Fourth Function - Summative • Annual comparison of progress to goals • Sometimes more reporting than “annual” is important • More reporting might be useful in the beginning, less reporting might be necessary as the project matures • Summative data should play a role in the development of new proposals

13. Issues Related to Function 4 • Qualitative data is valuable for projects, but does not seem to be something that ED is interested in. • This is a problem because projects cannot really produce meaningful data that ED is interested in. • So, do we spend extra time (money) producing two sets of data? • Or do we sacrifice the meaningful data to concentrate on producing what we’re told to create? • This has been the on-going debate among TAH evaluators on a national level

14. Questions? Comments? • What challenges have you faced when considering project evaluation? • How have you used evaluation in different stages of your project? (from planning to reporting) • jsun@sun-associates.com or 978-251-1600 x 204