Proposal Review Process Mock Review Webinar

1. Proposal Review ProcessMock Review Webinar Ning Fang Don Millard Division of Undergraduate Education National Science Foundation November 10, 2010

2. Webinar Agenda • Introduction • Overview of the review process • Instructions on producing a panel review • Organization of participants into teams and designation of panel chairs • Local teams discuss strengths and weaknesses • Teams locally report/discuss results • Reporting to virtual group with ratings – note: facilitators will be asked to select an individual to report BREAK (15min) • Individuals consider ways to improve the proposal • Local teams discuss suggestions for improvement • Teams locally report/discuss results  (5min) • Report back to virtual group - note: facilitator picks an individual to report • PD commentary on responses • Think  – What have I learned today that I will use in preparing my next proposal? • Share your thoughts with local participants • Facilitator reports results back to virtual group Wrap-up Q&A

3. Webinar Goals - Expected Outcomes • Help participants to: • Become more familiar with the proposal review process • Better understand the TUES-specific criteria • Better understand the use of intellectual merit/broader impact criteria in reviewing proposals • Develop more competitive proposals that effectively meet the expectations of the TUES program

4. TUES Program

5. TUES vs. CCLI • Title changed to emphasize the special interest in projects that have the potential to transform undergraduate STEM education  • Review criteria was modified to emphasize the desire for projects that: • Propose materials, processes, or models that have the potential to • Enhance student learning • Be adapted easily by other sites • Involve a significant effort to facilitate adaptation at other sites • Institutionalize the approach at the investigator's college or university as appropriate (e.g., for the Type) • Have the potential to contribute to a paradigm shift in undergraduate STEM education

6. TUES Program • Vision:Excellent STEM education for all undergraduate students • Reflects national concerns about producing: • Skilled STEM professionals (including K-12 teachers) • Citizens knowledgeable about STEM and how it relates to their lives • Seeks to build a community of faculty committed to improving undergraduate STEM education • Encourages projects with potential to transform undergraduate STEM education

7. TUES Project Components • Creating Learning Materials and Strategies • Guided by research on teaching and learning • Incorporate and be inspired by advances within the discipline • Implementing New Instructional Strategies • Contribute to understanding on how existing strategies: • Can be widely adopted • Are transferred to diverse settings • Impact student learning in diverse settings • Developing Faculty Expertise • Enable faculty to acquire new knowledge and skills in order to revise their curricula and teaching practices • Involve a diverse group of faculty

8. TUES Project Components (cont) • Assessing and Evaluating Student Achievement: • Develop and disseminate valid and reliable tests of STEM knowledge • Collect, synthesize, and interpret information about student understanding, reasoning, practical skills, interests, attitudes or other valued outcomes • Conducting Research on Undergraduate STEM Education: • Explore how: • Effective teaching strategies and curricula enhance learning and attitudes • Widespread practices have diffused through the community • Faculty and programs implement changes in their curriculum

9. Instructional Materials and Methods Projects • Projects developing instructional materials and methods should: • Be based on how students learn • Consider transferability and dissemination throughout the project's lifetime • Involve efforts to facilitate adaptation at other sites in more advanced projects

10. Type 1 Projects • Expect to award approximately 10% • Total budget: up to $200,000 for 2 to 3 years • $250,000 when 4-year and 2-year schools collaborate • Typically involve a single institution & one programcomponent – but there are exceptions • Contribute to the understanding of undergraduate STEM education • Informative evaluation effort based on the project's specific expected outcomes • Institutionalized at the participating colleges and universities • Deadlines: • May 26, 2011 (A-M) • May 27, 2011 (N-W)

11. Type 2, 3, and CRP Projects • Type 2 Projects • 20 to 25 awards expected • Total budget: up to $600K for 2 to 4 years • Type 3 Projects • 3 to 5 awards expected • Budget negotiable, but not to exceed $5M over 5 years • Tues Central Resource Projects • 1 to 3 awards expected • Budget negotiable, depending on the scope and scale of the activity • Small focused workshop projects -- 1 to 2 years & up to $100K • Large scale projects -- 3 to 5 years & $300K to $3M Deadline: January 14, 2011

12. Review Process

13. NSF Peer Review Process • Reviewers are solicited by program directors • 2010 TUES Type 1 Proposals Review: • 4 panels/program director (60-70 proposals/PD) • 6-8 reviewers/panel • 17 proposals/panel – not all read by every panelist • Approximately 130 engineering reviewers • Reviewers assign individual ratings and develop a review 2-3 weeks prior to coming to the panel meeting • Panel meeting is held in DC area – over a 2 day period

14. Panel Review Meeting • Panel Chair (picked by program director ahead of time) establishes order of proposal review process • Proposals are discussed individually • A “scribe” is designated to capture all of the points brought up in discussion and produce a summary review – called the “Panel Summary” • Panel returns on day 2 to review all the proposals’ panel summaries

15. Audience for Reviews • NSF program directors • Informs recommendations relative to funding • Guides pre-award negotiations • Applicants • If proposal is funded: • Provides suggestions for improving project • If proposal is not funded: • Provides information to guide a revision of the proposal

16. NSF Review Criteria • All proposals are evaluated using the NSB-approved review criterion • Intellectual merit • Broader impacts • The TUES Solicitation provides two sets of suggested questions to help define these criteria • Standard NSF set • TUES specific set

17. Caution Regarding Suggested Questions • Suggested questions are only a guide for considering intellectual merit and broader impacts • Suggested questions are NOT: • A complete list of “requirements” • Applicable to every proposal • An official checklist

18. NSF Suggested Questions for Intellectual Merit • Will the project: • Include activities important in advancing knowledge? • Involve qualified proposer(s)? • Contain creative and original concepts? • Have a well conceived and organized plan? • Include sufficient access to resources?

19. NSF Suggested Questions for Broader Impacts • Will the project: • Advance discovery - promote teaching & learning? • Broaden participation of underrepresented groups? • Enhance the infrastructure? • Include broad dissemination? • Benefit society?

20. TUES Suggested Questions for Intellectual Merit Will the project: • Produce one or more of the following: • Exemplary materials, processes, or models that enhance student learning and can be adopted by other sites • Important findings related to student learning? • Build on existing knowledge about STEM education? • Have explicit and appropriate expected measurable outcomes integrated into an evaluation plan? • Include an evaluation effort that is likely to produce useful information? • Institutionalize the approach at the investigator's college or university (as appropriate for the Type)

21. TUES Suggested Questions for Broader Impacts Will the project: • Involve a significant effort to facilitate adaptation at other sites? • Contribute to the understanding of STEM education? • Help build and diversify the STEM education community? • Have a broad impact on STEM education in an area of recognized need or opportunity? • Have the potential to contribute to a paradigm shift in undergraduate STEM education?

22. Writing the Review

23. Review Material • The Entire Proposal is Used to Inform Reviewers • Project Summary • Project Description • Biographical Sketches • Budget • Supplementary Documentation

24. Overview • A review should indicate anopinion on the merit of the project • The rating should indicate an overall evaluation of the proposal’s merit • Excellent, Very Good, Good, Fair, and Poor • Text boxes are provided to: • Describe positive aspects • Not just list them -- Provide details • Identify concerns (or weaknesses) • Not just list them -- Provide details • Offer suggestions for improvement Rating and text should be consistent

25. Text Boxes for Reviewer Comments • Describe the strengths and weaknesses (or concerns) relative to the review criteria • Include all of the positive aspects, concerns, and issues • Use a comfortable style and format • Use good grammar, style and complete sentences • A very brief review is not very helpful

26. Summary Text Box • Provide any additional information that you want to transmit • One-sentence summary of the proposal • Summary of the critique • Statement indicating why the proposed project is important or not • Suggestions for improvements • Other comments

27. Characteristics of Informative Reviews • Uses appropriate style • Contains adequate details • Contains understandable, specific, and complete statements • Relates strengths and weakness to review criteria • Indicates why an item is a strength or weakness • Justifies the proposal rating in the written critique A reader should be able to guess the rating from the written text

28. Specific and Complete Comments • Identify a strength or weaknesses • “The evaluation plan is a strength.” • Identify a strength or weaknesses and indicate why it is one • “The evaluation plan includes a competent, independent evaluator...” • “The background discussion is well referenced, shows a good understanding of the prior work, supports the proposed work...”

29. Specific and Complete Comments (cont.) • Identify a strength or weaknesses and indicate why it is one and why it is important • “The evaluation plan includes a competent, independent evaluator, … and it will guide the investigators as the project evolves and provide a measure of its effectiveness at the end.” • Identify a strength, indicate why it is one, why it is important, and how it could be improved • “The evaluation plan is a strength because it includes a competent, independent evaluator, … and this will guide the investigators as the project evolves and provide a measure of its effectiveness at the end. It could be improved by adding …”

30. Other Important Ideas • Be constructive in your written comments • Provide suggestions to help applicants improve their proposals • Do not be overly critical in your ratings • Most meritorious proposals (fundable proposals) have some weaknesses • Some are correctable through negotiations

31. Check Your Reviews • How many strengths did you note? • How many weaknesses did you note? • Are these numbers consistent with your rating? • Do your statements indicate why an item is a strength or weakness? • Do they indicate why a strength or weakness is important?

32. Strengths & Weakness

33. Team Activity Process • Local teams discuss the strengths and weaknesses identified in the individual reviews (20min) • Team “scribe” takes notes during discussion • Teams locally report/discuss results with facilitator’s guidance (10min) • Facilitators guide discussion and selection of an individual to report to the full virtual group • Participating organizations will be called upon by webinar moderator – PLEASE BE READY TO REPORT BACK • Program Directors will offer comments on reports (15min)

34. Reporting • Redundancy is OK – indicative of common perception • Responses will be solicited as follows: • Intellectual Merit • Strengths • Weaknesses • Broader Impacts • Strengths • Weaknesses • Overall Perception • Is this a proposal worthy of funding?

35. ONE MINUTEUntil Reporting Begins

36. Reporting • Redundancy is OK – indicative of common perception • Responses will be solicited as follows: • Intellectual Merit • Strengths • Weaknesses • Broader Impacts • Strengths • Weaknesses • Overall Perception • Is this a proposal worthy of funding?

37. Questions?

38. BREAK(15 min)

39. ONE MINUTEUntil Next Session

40. Suggestions for Improvement

41. Improvements Activity Process • Individuals consider ways that the proposal could be improved – create a list (5min) • Local teams discuss the suggestions for improvement (10min) • Team “scribe” takes notes during discussion • Teams locally report/discuss results with facilitator’s guidance (5min) • Facilitators guide discussion and selection of an individual to report to the full virtual group • Participating organizations will be called upon by webinar moderator – PLEASE BE READY TO REPORT BACK • Program Directors will offer comments on reports (15min)

42. Reporting • Try to minimize redundant responses • Identify approaches for: • Building on strengths • Overcoming weaknesses • Responses should include potential improvements to: • Idea • Project infrastructure • Project implementation plan • Evaluation plan • Dissemination plan • Proposal participation/involvement

43. ONE MINUTEUntil Reporting Begins

44. Reporting • Try to minimize redundant responses • Identify approaches for: • Building on strengths • Overcoming weaknesses • Responses should include potential improvements to: • Idea • Project infrastructure • Project implementation plan • Evaluation plan • Dissemination plan • Proposal participation/involvement

45. Sample Proposal Review Panel Summary and Reviewer Comments

46. Panel Summary -- Intellectual Merit (1) • The proposal is the evolution of the successful studio pedagogy developed at RPI to a mobile studio pedagogy • In this way the proposal builds on a instructional methodology accepted in the STEM literature. • The primary innovation is the development of a low cost virtual instrument board that, when coupled with a computer/laptop, provides students with a portable experimental platform. • The proposal is student focused and grounded in the STEM knowledge base.

47. Panel Summary -- Intellectual Merit (2) • This experimental platform serves as a low cost replacement for the laboratory equipment found in the original RPI studio classrooms and provides the students with an "anywhere, anytime" experimental platform. The PIs are well qualified to carry out the work. • In addition, the proposal was responsive to the cyclic model of knowledge creation contained in the CCLI request for proposals. • The panel felt the proposal would be strengthened by a more detailed evaluation plan with clearly stated, measurable outcomes.

48. Panel Summary -- Broader Impacts • The broader impacts of the proposed work include the potential to significantly impact a large number of under served students by providing a low cost, portable experimental platform. • In addition, the inclusion of an historically black college or university and a community college should magnify the impact of the proposed pedagogy. • This partnership was made concrete through an instructor exchange program between the institutions.

49. Individual Reviews -- Intellectual Merit Weaknesses (1) • While the evaluation team appears to have worked together for several years, some basic information about the qualifications of those individuals should have been included in the proposal. • It is not clear that it will positively impact student learning -- just because a student has the means to perform an experiment 24/7 does not mean that they will. • Proposal could benefit from clarifying or identifying the role of laboratory experiences where actual industrial-quality laboratory equipment is utilized, as too much reliance on the proposed pedagogy might also leave the student with limited experience in the use of the real industrial grade and scale laboratory equipment. • The proposal could be strengthened with more definition of the role of each partner.

50. Individual Reviews -- Intellectual Merit Weaknesses (2) • Moving from the original studio to the mobile studio essentially means that the new work is now based on an untested model. The proposal could be strengthened by addressing this issue. • It has not been demonstrated that the mobile studio is as effective as the original studio with its significant faculty involvement and structure. • This proposal would have been much stronger as a Phase I or if some preliminary data on efficacy were given. • The proposal mentions a test that was done at Howard University with this work but no details are given. • It fails to explicitly address how it will focus on components of the cyclic model for innovation in STEM education.