Writing More Effective Proposals V

1. Writing More Effective Proposals V Project Evaluation Sustainability Dissemination 1

2. Evaluation

3. Session Outcomes After the session, participants should be able to: Discuss the importance of goals, outcomes, and questions in the evaluation process Cognitive and affective outcomes Be aware of several types of evaluation tools Advantages, limitations, and appropriateness Discuss data interpretation issues Variability, alternate explanations Develop an evaluation plan with an evaluator Outline a first draft of an evaluation plan

4. Evaluation and Assessment • Session addresses: Project Evaluation • May involve evaluating individual and group performance – but in the context of the project • Project evaluation • Formative – monitoring progress to improve approach • Summative – characterizing final accomplishments

5. Evaluation and Project Goals/Outcomes • Effective evaluation starts with carefully defined project goals and expected outcomes • Goals and expected outcomes related to: • Project management • Initiating or completing an activity • Finishing a “product” • Student behavior • Modifying a learning outcome • Modifying an attitude or a perception

6. Approaches to Data Collection • Quantitative methods • What happened? • Qualitative methods • Why? • Mixed methods

7. Examples of Tools for Evaluating and Assessing Learning Outcomes • Surveys • Concept Invento’’ries • Standardized tests • Measuring critical thinking, cognitive ability, attitudes • Rubrics for analyzing student products • Observations • Interviews • Focus groups

8. ActivityConsidering an Existing Tool • Individually identify several questions • Share these with a neighbor or two • Report to the group Suppose you were considering an existing tool (e.g., a concept inventory) for use in your project’s evaluation What questions would you consider in deciding if the tool is appropriate?

9. PD’s ResponseEvaluating a Existing Tool • Nature of the tool • Is the tool relevant to what was taught? • Is the tool competency based? • Is the tool conceptual or procedural? • Prior validation of the tool • Has the tool been tested? Is it sensitive? • Is there information on reliability and validity? • Has it been compared to other tools? • Does it discriminate between a novice and an expert? • Experience of others with the tool • Has the tool been used by others besides the developer? At other sites? With other populations? • Is there normative data?

10. Topics to be Addressed in an Evaluation Plan • Name & qualifications of the evaluation expert • Get the evaluator involved early in the proposal development phase • Explicit relationship between intervention and outcomes • Goals, outcomes, and evaluation questions • Instruments for evaluating each outcome • Protocols defining when and how data will be collected • Analysis & interpretation procedures • Confounding factors & approaches to minimize impact • Formative evaluation techniques for monitoring and improving the project as it evolves • Summative evaluation techniques for characterizing the accomplishments of the completed project

11. Other Sources • NSF’s User Friendly Handbook for Project Evaluation • http://www.nsf.gov/pubs/2002/nsf02057/start.htm • Eval-ate.org • Online Evaluation Resource Library (OERL) • http://oerl.sri.com/ • Field-Tested Learning Assessment Guide (FLAG) • http://www.wcer.wisc.edu/archive/cl1/flag/default.asp • Student Assessment of Their Learning Gains (SALG) • http://www.salgsite.org/ • Evaluating & Improving Undergraduate Teaching in STEM • National Research Council (2003)

12. How to Find an Evaluator Institution’s Center for Teaching and Learning Institution’s Center for Evaluation (in STEM) Colleges and Departments (Education, Ed Psych, Engineering, Science, Public Admin) Colleagues Professional Societies American Evaluation Association Website

13. What to Expect from Your Evaluator • Knowledge about evaluation design and methodology(evaluation competencies) • Experience in evaluating your type of project • A quality evaluation plan • A contract or an agreement with deliverables and timeline • Knowledge of evaluation principles, standards, ethics • [Increasing] knowledge about your project • Frequent, respectful communication

14. What Your Evaluator Expects from You • A clear definition of your project goals, objectives, activities, and expected outcomes • All available information about your project • Frequent communication • Questions • Access to data/participants • Help with problem-solving • Resources and payment

15. Helpful Hints • Identify an evaluator in advance. • Match evaluation with goals, objectives, and activities. • Ensure design provides evidence about what is and isn’t working for improvement. • Remember to include not only accountability information, but also impact and effectiveness. • Develop indicators to measure progress and include time lines.

16. 10 Helpful Hints (con’t) • Develop the plan jointly with an evaluator. • Assign responsibilities for various components. • Use the evaluation literature. • Keep in mind your evaluation stakeholders. • Use descriptions, tables, visuals

17. Fatal Flaws • The evaluation section • Is missing. • States that after receiving funding, the evaluation will be developed. • Only evaluates easy things. • Has unreasonable or unrealistic budget. • Does not align with priorities of funding program.

18. Fatal Flaws (con’t) The evaluation section • States PIs will do all the evaluation. • Is lacking in details. • Was cut and pasted from another proposal. • Uses too much jargon. • Does not explain methods and instruments to be used.

19. Questions and Answers

20. Dissemination

21. Dissemination: The Standard Three • Conference presentation(s) • 2. Web site • Publication(s) • These are necessary but not sufficient, i.e., everyone lists these • Question: how many from “target” audience sees/hears this?

22. Program Officers’ Suggestions:Dissemination • Be proactivein promoting website & materials “If you build it they will NOT come” • Target and involve a specific sub-population • Those who teach similar course(s) at other locations • Ask them to reviewvarious products, data, and approaches • Make use of local/regional resources

23. Program Officers’ Suggestions:Dissemination • Work with others to organize • Email exchanges and listserves • Informal meetings at a conference or on-campus • Faculty development workshops (on-campus and at conferences) • Explore beta test sites

24. Program Officers’ Suggestions (2) • Be specific in publication efforts • Indicate the specific conferences and journals • Include conference travel and journal page charges in budget • Explore other venues • Digital libraries, PKAL (http://www.pkal.org), State Academy of Science meetings • Science news publication and lay press • Professional society and specialty listserves

25. Program Officers’ Suggestions (3) • Integrate dissemination with community building and evaluation • Be specific about how the project will serve as a “pilot” for subsequent expansion to other institutions. • How does the dissemination plan, and its evaluation, contribute to getting “buy-in” from others for future adoption? • Explore commercialization • Discuss contacts with software and textbook publishers

26. Sustainability

27. From Solicitation Sustainability: The purpose of the TUES program is to bring about lasting improvement in undergraduate STEM education. Proposals should address sustainability and should demonstrate that there is a reasonable expectation of persistent effects of the grant funded work consistent with the aims of the project.

28. Internal Support To sustain/grow project activities after the grant period always requires ongoing institutional support This support typically requires financial resources – supplies, repairs, stipends, space, equipment, release time, etc. A demonstrably successful project and a good relationship with administration are keys for obtaining this support

29. External Support A successful TUES project can be basis of Type 2 TUES grant Grants from local businesses, industries, or foundations Fund raising from alums/others