Proposal Preparation

1. Proposal Preparation I3 Project Evaluation Formative and Summative Costello L. Brown April 3, 2009

2. Evaluation Plan* Evaluation plan that addresses effectiveness of strategies, including any institutional policies, practices, or mechanisms developed and implemented; and, as appropriate, provide for evaluation of any products produced, and collection and analyses of relevant student or faculty data. * From I3 Program Announcement

3. EVALUATION?* • Evaluation is the process of determining significance or worth, usually by careful appraisal and study. • Evaluation is the analysis and comparison of actual progress vs. prior plans, oriented toward improving plans for future implementation. • Evaluation is the process of determining the worth or value of something. This involves assigning values to the thing or person being evaluated. *http://www.sil.org/linguaLinks/literacy/ReferenceMaterials/GlossaryOfLiteracyTerms/WhatIsEvaluation.htm

4. EVALUATION? Answers the following questions: "How well did we do?" (Qualitative Evaluation) "How much did we do? (Quantitative Evaluation) “How do you know your project or program was successful? What evidence do you have to demonstrate this success?”.

5. FORMATIVE EVALUATION?* • Formative evaluation is a method of judging the worth of a program while the program activities are forming or happening. • Formative evaluation focuses on the process (Bhola,1990) Example: Collecting continuous feedback from participants in a program in order to improve the program while it is still in progress. *http://www.sil.org/linguaLinks/literacy/ReferenceMaterials/GlossaryOfLiteracyTerms/WhatIsEvaluation.htm

6. SUMMATIVE EVALUATION?* • Summative Evaluation is a method of judging the worth of a program at the end of the program activities. • Summative evaluation focuses on the outcome (Bhola,1990) Examples: 1) Final grade in a course; 2) Collecting data on the increase in the graduation rate at the end of a program that focused on this issue. *http://www.sil.org/linguaLinks/literacy/ReferenceMaterials/GlossaryOfLiteracyTerms/WhatIsEvaluation.htm

7. FORMATIVE VS.SUMMATIVE EVALUATION?* • Formative focuses on Improvement: Summative focus on Judgment (midterms vs. final grade) *http://www.sil.org/linguaLinks/literacy/ReferenceMaterials/GlossaryOfLiteracyTerms/WhatIsEvaluation.htm

8. Evaluation* Evaluation has varied definitions…. Accepted definition for NSF Projects: “Systematic investigation of the worth or merit of an object…” -Joint committee on Standards for Education Evaluation *Dennis W. Sunal, Susan E. Thomas,http://astlc.ua.edu/ College of Education, University of Alabama

9. Goals of I3 • Increase synergy & collaboration among NSF/EHR-funded projects and within/ between institutions • Enhance sustainability • Promote integration of STEM research and education • Broaden participation, attend to critical educational junctures, and/or provide for more globally engaged workforce • Encourage research on intra- or inter-institutional integration & broader impacts

10. I3 Proposals • Proposed projects can have a focus that is: • Intra-institutional • Inter-institutional • Proposed projects may be: • Implementation only • Research only • A blend of the two

11. Planning Evaluation Addresses the Following…* • Why was the project developed? • Who are the stakeholders? • What do the stakeholders want to know? • What are the activities and strategies that will address the problem which was identified? *Dennis W. Sunal, Susan E. Thomas,http://astlc.ua.edu/ College of Education, University of Alabama

12. Planning Evaluation Addresses the Following…* • Where will the program be located? • How long will the program operate? • How much does it cost in relation to outcomes? • What are the measurable outcomes to be achieved? • How will data be collected? *Dennis W. Sunal, Susan E. Thomas,http://astlc.ua.edu/ College of Education, University of Alabama

13. Progress Evaluation* • Are participants moving toward goals, improving understanding of the research process • Are numbers of students reached increasing? • Is progress sufficient to reach goals? • Assess progress in meeting goals • Collect information to learn if benchmarks of progress were met, what impact activities have had, and to determine unexpected developments • Useful throughout life of project *Dennis W. Sunal, Susan E. Thomas,http://astlc.ua.edu/ College of Education, University of Alabama

14. Summative Evaluation Assesses project’s success and answers… Purpose: Assess the quality and impact of a fully mature project • Was the project successful? • To what extent did the project meet the overall goals? • What components were the most effective? • Were the results worth the projects cost? • Is the project replicable? *Dennis W. Sunal, Susan E. Thomas,http://astlc.ua.edu/ College of Education, University of Alabama

15. References 1. Dennis W. Sunal, Susan E. Thomas, Alabama Science Teaching and Learning Center,College of Education, University of Alabama http://astlc.ua.edu/ 2. NSF 02-057, The 2002 User-Friendly Handbook for Project Evaluation: Joy Frechtling, Westat 3. Kirkhart, K. E. (2005). Through a cultural lens: Reflections on validity and theory in evaluation. In S. Hood, R. Hopson, & H. Frierson (Eds.), The role of culture and cultural context; A mandate for inclusion, the discovery of truth, and understanding in evaluative theory and practice (pp. 21-38). Greenwich, CT: Information Age.

16. Some Discussion Questions 1. How do I find a good external evaluator? 2. Do I need both an external and an internal evaluator. 3. How much should I budget for evaluation? 3. What types of data need to be collected and who will collect it?