Webinar for Activities Review Team On the Cutting Edge pre-GSA REVIEW CAMP

1. Webinar for Activities Review Team On the Cutting Edge pre-GSA REVIEW CAMP September 25, 2013 Pleasemute your phone by pressing*6 Technical problems? Contact Krista at kherbstr@carleton.edu Webinar begins at: 1pm Eastern | 12 pm Central | 11 am Mountain | 10 am Pacific Review team instructions page at: http://serc.carleton.edu/NAGTWorkshops/review.html

2. THANK YOU! • The Cutting Edge Review Process is: • A tremendous community service; • An opportunity for you to see a lot of great activities; • An opportunity to see what goes into a quality activity to inform your own work; • The realization of a decade-long goal of developing an online, peer-reviewed collection of activities

3. The collections • SERC: Science Education Resource Center at Carleton College • SERC office and staff helps develop and manage web resources for many projects through collaboration • Many different collections of activities, submitted for different projects • On the Cutting Edge: the first of the projects hosted by SERC • Most of the activities in the Cutting Edge collections were submitted in connection with workshops.

4. Cutting Edge Reviewed Collection • On the Cutting Edge is conducting a review of activities in the Cutting Edgecollections • Each activity reviewed once and ranked: • Exemplary (Part of Reviewed Collection) • Pass (Part of Reviewed Collection) • Keep (Not part of Reviewed Collection) • De-accession

5. Cutting Edge Reviewed Collection • Activities ranked as “Exemplary” • Come up first in searches • Are designated on individual the ActivitySheet as being part of Exemplary Teaching Activities collection

6. Cutting Edge Reviewed Collection • Activities ranked as “Pass” • Come up second in searches • Are designated on individual ActivitySheet as being part of the Peer Reviewed Teaching Activities collection

7. Cutting Edge Reviewed Collection • Activities ranked as “Keep” • Come up last in searches • Have no designation on the ActivitySheet • Worth keeping as catalyst idea • Might be too local or items might be missing or has other problematic aspects

8. Review process • Activities that have been collected across the Cutting Edge collections • Each activity receives 2 reviews • Authors of “Exemplary” and “Pass” activities receive letters • Explains review process • Indicates activity rank • Indicates that reviewer comments are available on request if the author wishes to revise • If author does revise, the activity will be reviewed again

9. Plan for webinar • Explain the web interface • Clarify review criteria • Answer questions • As we go along, please post questions in chat

10. Your list of items • When you click on Review Toolon Review Team Instructions page, your login will take you to a page that lists only your items to review

11. Your list of items • If you have not yet completed the review, you will see: • Click the URL to go to the ActivitySheet and download the actual activity and any supporting materials.

12. The review tool • Click Review It to bring up the Review Tool

13. The review tool • You will evaluate the activity in five categories • Scientific accuracy • Alignment of goals, activity, and assessment • Pedagogic effectiveness • Robustness • Activity description

14. The review tool • For each category, questions plus rubric provide guidance for what to consider

15. The review tool • Summary score will tabulate automatically • Exemplary = 4; Very good = 3; Adequate = 2; Problematic = 1 • Comments help the editors understand your ranking – please don’t leave these boxes blank!!

16. The review tool • At the end of the form, you will add your view about what it would take to raise the activity to Exemplary status if it fell short in your review • The editors will use your comments to respond to authors on request. Please phrase your comments in a collegial fashion.

17. Your list of items • Once you have submitted a review, your review list indicates completion for that item

18. Can you revise a review? • Yes – click on the Review It link and then the link to what you submitted previously • Your original rankings will come up, and you can change them and add to/change your comments. • Click submit when done.

19. Pause for questions • Anyone??

20. Your review • Review the activity in the context for which it was designed • Not just whether it’s good for a particular upper level course – many will be for other courses (e.g., intro geo) • Not everything has to be a full lab or major assignment (e.g., a back-of-the-envelope calculation could be Exemplary) • Not every activity needs to be usable by all instructors (e.g., a lab requiring specific software/math/expertise background)

21. Your review • Make a summary list of the activities that you reviewed • List both the total numerical score and your overall assessment • Exemplary • “Exemplary minus” • Pass • Keep • De-accession • Bring with you to review team meeting

22. Your review • Exemplary • Must have good science, good pedagogy, and all materials so that someone else can adapt/adopt, nothing “broken” • Can be “local” if it is also a good template or model for others to emulate • Does not need to have answer key or to provide an instructor with background • We have never required these so cannot ding someone for not including them • Scoring • Exemplary or very good in all categories • Exemplary in at least three of the five. • 18 or higher.

23. Your review • Exemplary minus • Could be made Exemplary with only a small amount of work, such as: • fixing a URL • uploading the latest version of the assignment or adding instructor tips • fleshing out the ActivitySheet • This is not a formal category, but it would help us a lot to have your list of “Exemplary Minus” activities • These will be ranked as Pass, but knowing that they are “Exemplary minus” will help the editors craft feedback if authors request it.

24. Your review • Pass – these become part of the Reviewed Collection • Those that aren’t Exemplary but still have value to others • Must bemore than just the germ of an idea • Must haveall of the components • These must have no scientific errors. • If you think there are errors, confirm this with someone else on the review team. • Those with scientific errors should go into the Keep or De-accession category, depending on the severity of the problem.

25. Your review • Keep – no designation on ActivitySheet, will come up last in a search • Nucleus of a good idea • Insufficient info for someone to adapt or adopt or hasscientific errors • Author does not receive a letter

26. Your review • De-accession • Not an activity or very fragmentary • Has truly egregious problems

27. Summary • Review each activity using rubric • Score the activity in each of 5 categories • Write a summary evaluation for each • Remember that these were submitted voluntarily to a community collection • Be kind but clear • Make a summary list • Exemplary • Exemplary minus (technically a Pass) • Pass • Keep • De-accession

28. Your assignment • Each team member has ~20activities to review • Reviews must be completed and submitted using review tool before the Review Camp • At Review Camp we will • Discuss issues • Resolve discrepant reviews • Arrive at final rankings • Hopefully do a few more reviews

29. Final Thoughts • Please write your reviews in the same spirit you would like to receive them • We all teach “out of field”—so don’t be dismayed if you have an activity not related to your immediate interests • Just use your best professional judgment.

30. Final Thoughts • The goal is to have a high quality, coherent, and comprehensive collection for use by all. • Acknowldege the great work that is available; credit and kudos to authors • Help authors improve their activities • Weed out those that are not ready for prime time

31. Thank you! • Email Dave or Krista with any questions • mogk@montana.edu • kherbstr@carleton.edu • Have fun, stay in touch, post messages or questions to the listserv, and we’ll see you in Denver!