Gavin Fulmer, Janice Earle, Kusum Singh, and Celeste Pea

1. Developing a Competitive Educational Research Proposal for the National Science Foundation’s Division of Research on Learning Gavin Fulmer, Janice Earle, Kusum Singh, and Celeste Pea Division of Research on Learning in Formal and Informal Settings, National Science Foundation AERA Conference Workshop, April 9, 2011

2. Workshop Goals Goals • Familiarize with NSF educational research programs • Consider common issues of competitive NSF proposals • Discuss your proposal ideas • Prepare you to write a competitive proposal

3. Workshop Agenda

4. Introduction to NSF and EHR Educational Research Programs

5. NSF Strategic Goals • Transform the Frontiers • Invest in challenging potentially transformative research • Sharpen the merit review to better identify such research; • Emphasize interdisciplinary and system-oriented approaches that often lead to transformational concepts. • Innovate for Society • Make investments that lead to results and resources that are useful to society • Build the capacity of the nation’s citizenry for addressing societal challenges through science and engineering • Support the development of innovative learning systems • Perform as a model organization • Achieve management excellence through leadership, accountability, and personal responsibility • Infuse learning as an essential element of the NSF culture with emphasis on professional development and personal growth

6. NSF Organizational Structure

7. EHR’s Mission To enable excellence in U.S. STEM education at all levels and in all settings in order to support the development of a diverse and well-prepared workforce of scientists, technicians, engineers, mathematicians, and educators.

8. EHR’s Structure • DGE—Division of Graduate Education • DUE—Division of Undergraduate Education • HRD—Human Resources Development • DRL—Division of Research on Learning in Formal and Informal Settings

9. EHR Organization * Not all programs are listed.

10. DRL Cycle of Innovation evaluate and generalize synthesize and theorize implement, study, and improve explore, hypothesize, and clarify design, develop, and test

11. Standards of Evidence • Explore: Triangulation, consideration/elimination of rival explanations • Design: Systematic monitoring and data collection of subjects’ responses to an innovation • Implement: Tests the intervention under conditions consistent with theory of action in several sites

12. Standards of Evidence, cont. • Scale-up: Designs include experimental, quasi-experimental (and possibly non-experimental) approaches that allow causal inference and document impact at progressively larger scale and more varied contexts • Synthesize: meta-analytic techniques that include criteria for “what’s in” and “what’s out”

13. QUESTIONS

14. Description of DRL Educational Research programs

15. DRL’s Core Programs • Research and Evaluation on Education in Science and Engineering (REESE) • Discovery Research K-12 (DRK-12) • Faculty Early Career Development Program (CAREER) • Information Technology Experiences for Students and Teachers (ITEST) • Informal Science Education (ISE)

16. Research and Evaluation in Education in Science and Engineering (REESE) Program Solicitation 10-586 • Supports research and evaluation that enhances STEM education and learning • Seven research strands • Four award types

17. REESE Research Strands • National STEM education policies • Research on implementation • STEM learning in formal and informal settings • Cyberlearning and learning technologies • Methods, models, and measurement for research and evaluation • Cognitive underpinnings of STEM learning • Neural bases of STEM learning

18. REESE Award Information Note. These are the expected FY 2011 numbers.

19. New REESE Strand: FIRE • FIRE (Fostering Interdisciplinary Research in Education) • Program Solicitation 11-526 • Proposal due date: April 29, 2011 • Purpose: Provide opportunity (up to 2 years and for $400,000) for scholars to cross disciplinary boundaries to acquire the skills and knowledge needed to conduct rigorous research on STEM learning and education. • (1) development of innovative theoretical, methodological and analytic approaches to understanding complex STEM education issues

20. FIRE • (2) to broaden and deepen the pool of those engaged in STEM education research • Investigators must identify a mentor in a to-be-learned field of interest • Proposals must contain both a research and an education component • Examples: A cognitive scientist who wants to deepen their understanding of some STEM content; an economist who wants to learn more about education research; a biologist who wants to understand how to use HLM

21. Discovery Research K-12 (DRK-12) Program Solicitation 10-610 • Proposals were due January, 2011 • Enables advances in PK-12 student and teacher learning of the STEM disciplines through the development, implementation, and study of resources, models, and technologies • Five program challenges • Four award types

22. DR K-12 Program Information • Challenges • How can improved assessment of student knowledge and skills advance preK-12 STEM teaching and learning? • How can all students be assured the opportunity to learn significant STEM content? • How can we enhance the ability of teachers to provide high quality STEM education for all students? • How are effective innovations successfully implemented, scaled, and sustained in schools and districts in a cost effective manner? • How can next-generation, cyber-enabled learning materials radically transform students’ STEM learning experiences and enhance their abilities and interests in STEM fields? • Must be at the preK-12 level

23. DR K-12 Award Information

24. CAREER Program Solicitation 11-690 • Proposals due July 25, 2011 • Minimum of $400,000 over 5 years • Describes both • Research project • Educational activities • Provide letters of support from Chair and any collaborators

25. CAREER For DRL CAREER proposals • Align your project to one of the other programs (e.g., REESE or DR K-12). • Educational activities may be part of the proposed research project • Research in K-12 schools or on students’ thinking • Development of courses and curriculum modules, etc.

26. PRIME • Program solicitation 10-615 • Development, demonstration and validation of innovative new methodologies and approaches to the evaluation of STEM education programs • New ways of determining the usefulness and impact of evaluation of STEM education projects • New theoretical foundations that expand perspectives on evaluating STEM • Developing the capacity and infrastructure that increases the number of researchers and evaluators that produce sound and appropriate evaluations of STEM projects

27. PRIME 9 APR 2011 27

28. QUESTIONS

29. Merit Review Process and Criteria

30. Proposal Review Process and Timeline DGA Organization submits via FastLane Program Officers Ad hoc NSF Program Division Director Concur Award Advise Recommend Panel Decline Organization DGA Award Proposal Receipt at NSF DD Concur 6 Months 30 Days

31. Who reviews? • DRL Panels and Reviewers • Experts in the fields of STEM education, STEM content, methodology, cognitive science, psychology, sociology, anthropology; school-based experts; etc. • Panelists read up to 11 proposals at time • Ad hoc reviewers

32. NATIONAL SCIENCE FOUNDATION 4201 Wilson Boulevard Arlington, Virginia 22230 Dear Dr. Doe, The National Science Foundation hereby awards a grant of... NATIONAL SCIENCE FOUNDATION 4201 Wilson Boulevard Arlington, Virginia 22230 Dear Dr. Doe, I regret to inform you that the National Science Foundation is unable to support your proposal referenced above... $ The Proposer Receives… & Context statement & Award/Declination letter Reviews Panel Summary (if applicable)

33. NSF Merit Review Criteria • Intellectual merit • Important to advancing knowledge and understanding • Proposers’ qualifications • Sufficient access to resources • Proposed activity well-conceived and organized • Creative, original, and/or potentially transformative • Data management plan • Post-doc mentoring plan, if applicable • Evaluation

34. NSF Merit Review Criteria Broader impacts • Promote teaching, training, and learning? • Broaden the participation of underrepresented groups? • Enhance the infrastructure for research and education? • Disseminate results broadly? • Benefit society?

35. QUESTIONS

36. 10-Minute Break

37. Discussing Excerpts from Funded Proposals

38. Small Group Agenda • Listen to brief description of each proposal • Break into groups • Read excerpts • Discuss excerpts following prompts • Elect a scribe and speaker to (1) note of your group’s comments and (2) share with whole group, respectively

39. Descriptions of Each Proposal

40. Achievement Gaps, 9mos-8th grade [Grissmer, REESE] • Study of possible causes of observed achievement gaps in mathematics • Uses ECLS-K to study individual, family, and social predictors with • Longitudinal growth modeling, • econometric modeling, and • twin “quasi-causal” studies.

41. College Calculus[Bressoud, REESE] • Explores what works in college calculus • Looks at • Demographics of enrollees; • Impact of various instructional characteristics; • Case studies of exemplary programs; • Develop theoretical basis for studying calculus student success • Disseminate through math professional societies to influence field across USA and abroad

42. College Calculus[Bressoud, REESE] • National investigation of Calculus 1 • Identify factors that contribute to success; • See how these factors are leveraged within highly successful programs. • Methods are large-scale survey (both instructors and students) with explanatory case studies

43. NAEP Mathematics[Kloosterman, REESE] • PIs have secure access to Main and long-term trend NAEP items and responses • Clustering items by content to explore trends in Main NAEP performance over the past 20 years • Comparing performance with Common Core of Data to look for differences based on school curriculum selection and other variables

44. Embodied Physics Cognition[Beilock, FIRE] • Testing whether different regions of brain are activated under varied instruction • Using a wheel to study angular momentum • Watching another student • Reading a passage • Long presumed, but not demonstrated • Students complete questions about the concept while in fMRI machine • Also tests confounding of language vs. visuo-motor regions during these sessions

45. Evolution in Grades 2-3[Metz, REESE] • Applies the learning progression perspective to the conceptual understanding of evolution • Studies curriculum enactment and student learning in (1) a project-run summer enrichment program where research team has control, and (2) inner-city public school classrooms • Includes instructionally embedded assessments, assessments to measure students’ conceptual understanding and embedded case studies

46. Elementary Science synthesis[Slavin, DR K-12] • Conducting a synthesis of curriculum interventions in elementary science • Includes quantitative meta-analysis and narrative review of the literature • Purpose is to understand best practices within or across curriculum materials

47. Teachers & Educative Curriculum[Davis, REESE] • Educative curriculum materials support student and teacher learning • Explores teachers’ learning & practices and students’ learning when using such materials • Includes design-research framework • Interviews about needs for educative support, • Developing and piloting materials, • Efficacy study using random assignment.

48. Questions for Discussion • How do they demonstrate importance of the project? • How do they communicate the methods? • What strengthsand weaknesses do you notice about the proposed project? • Are there commonalities or differences? • What are lessons learned from reading these excerpts?

49. Wrap-Up • Discuss main ideas & insights from this small group discussion • Prepare 2-3 major points summarizing “lessons learned”

50. 10-Minute Break