Big Data. Big Conversation. Drexel University’s Journey to Meaningful Assessment

1. Big Data. Big Conversation.Drexel University’s Journey to MeaningfulAssessment Hosted by The Chronicle of Higher Education Sponsored by Waypoint Outcomes

2. overview • Introductions • Housekeeping • Challenges of deep, engaged assessment practices:Three interconnected examples: • Decentralized, faculty-led assessmentFirst-Year Humanities • Normed, large-scale assessmentFirst-Year Writing • Situational assessment and external assessorsAACSB / Business School • Q & A • Resources: • http://waypointoutcomes.com/resources/bigdata • Sample rubrics, PowerPoint, link to the Webinar recording (once available)

3. panel • Dr. Rebecca Ingalls • Director of the Freshman Writing Program • Dr. Karen Nulton • Director of Writing Assessment • Dr. Scott Warnock • Director of the Drexel Writing Center and Writing Across the Curriculum • 800+ registrants in 23 countries • Moderator: • Andrew McCannCEO and FounderWaypoint Outcomes: software for 21st century education

4. Housekeeping • Ask a question via the box in your console and we’ll address as many as possible during Q&A • You can download a copy of the slides via a button on your console • Today’s presentation is being recorded; you’ll receive a follow-up email with the link to share with colleagues

5. Assessment capability stage • Chronicle / Waypoint Survey of 672 registrants

7. format • Three interconnected examples: • Decentralized, faculty-led assessment: First-Year Humanities • Normed, large-scale assessment: First-Year Writing • Situational assessment: AACSB / Biz School • For each: • Scenario • What were the learning outcomes, the goals, and the context? • Mechanics • How did the team approach the situation? • How was assessment organized? • Data • What data was collected? • How was it analyzed? • Action • What wisdom came from analyzing the data? • What actions were taken?

8. About drexel university • Drexel University • 23,000 students (12,000 undergraduates) • Comprehensive research institution • Cooperative Education: Art, Science, Technology • Drexel Advantages • Robust Freshman Writing Program • Writing program administrators invested in assessment • In-house writing assessment expert • Supportive administration • Resources • Pedagogical buy-in • Waypoint Outcomes already licensed and in place • Many good campus partners • Middle States on the horizon (regional accreditor)

9. assessment • A long-distance race with many challenges: • Clearly articulated learning outcomes • Money • Institutional partnerships • A workable assessment plan • A workable assessment system • A technology to empower that system • Personnel requirements • “Stickiness”

10. assessment • A long-distance race with many challenges • Clearly articulated learning outcomes • Money • Institutional partnerships • A workable assessment plan • A workable assessment system • A technology to empower that system • Personnel requirements • “Stickiness” • Bottom line • You’re going to be challenged to get to the finish line.

11. First-year humanities: Scenario • Three-course writing sequence (trimesters) • Engineering Design & Humanities; Composition • Focus on teaching, collaboration, consistency, and feedback to students • Responses from “real” teachers to “real” students about “real” assignments

12. First-year humanities: mechanics • Backbone of “template” rubrics created and shared via Waypoint • Some required usage of rubrics, some ad hoc • Retrospectively gathered data • Undergraduate research assistant mined data

13. First-year humanities: data • Data from actual course activity: 2004 – 2010 • 160 unique assignments/rubrics; 222 classes spanning from ENGL 101 to ENGL 103 • 30 instructors; 4,596 students • 948 elements matched to 17 competencies

14. First-year humanities: action • Curriculum Changes • Moved research and evidence into English 101 • Faculty development • Partnering with library • Politics and institutional culture • How do you get what your faculty and program deserve? • Data-gathering exploration reduced cap sizes, created course releases, funded faculty training

15. First-year writing: Scenario • Plan necessitated • University-wide Communication Student Learning Outcomes (SLOs) • Assessment of FWP base • Outcomes of “meta-analysis” for FWP • Outcomes of Communication Outcomes for the rest of the university • Large support, large assessment

16. First-year writing: mechanics • Pre- and post-assessment to examine value-add of FWP • Collected samples on Day 1 of English 101 and on last day of English 103 • Prompts asked students to articulate how they would approach a writing task (meta-discourse) • Samples and training • Worked with internal and external experts to create rubric (based on SLOs) • Recruited 35 outside experts, who were trained online • Assessors paid $150 for 1.5-hour training and scoring 60 papers (5 pre-scored) • Data collected on prompt by type and whether guided or unguided versions • Scoring Process • Samples loaded into Waypoint • External assessors score asynchronously; data aggregated automatically

17. First-year writing: data • Aggregate: Fall vs. Spring

18. First-year writing: data • Guided: Fall vs. Spring

19. First-year humanities: action • Presentation at FWP workshop • Conversation outside of the FWP • FWP is only the beginning of writing at Drexel • Teaching tool • Visual literacy • Mechanics • Portfolios and reflective analysis • Publication • Leadership in Composition & Rhetoric

20. Lebow college of business: Scenario • AACSB accreditation requirements • “Writing” identified as one of four key goals • LeBow partnered with English to develop a methodology • Situated assessment: “real” documents, readers, responses • Making the subjectivity of readers’ natural responses to writing an asset rather than a problem • Piloted assessment in 2006; full-scale assessment 2007-11 (we have analyzed the data until 2010)

21. Lebow college of business: mechanics • 600 student writing projects • 30 assessors: 15 from English, 15 from industry • Each assesses 20 projects • $100 stipends: $3,000 assessor budget • One-hour, online training • No norming • Electronic distribution of documents • LeBow writing rubric in Waypoint

22. Lebow college of business: data • AY 2007 - 2010 • 1,151 documents • 109 assessors (51 Business & 59 English) • Total of 2,161assessments • Sampled documents from six different majors • Control group • Evidence lowest element in all the years but one • Average Evidence score by year stays in 0.62-0.69 range • Ethics & Gross Mistakes close to 0.85 mark

23. Lebow college of business: action • How is the student writing in your program? • A conversation driven largely by anecdote • Statistical profile of LeBow writers • The quality question is one for the LeBow faculty to decide • Data-gathering requirement for accreditation • Evidence, again • Working with faculty from two courses specifically on evidence-based writing • Fascinating glimpse of two assessor populations

24. Campus conversation • Sum of three models: sustainable culture of assessment • Formal and grassroots assessment, empowered via Waypoint’s deep integration with the LMS • Writing as key way of linking campus initiatives • University Committee on Assessment • Executive Assessment Council • Portfolio Task Force • Writing Task Force • Library Advisory and Search Committees • Data used on campus to build consensus and dialogue • Arts & Sciences, Business, Library, Provost’s Office • Data shows links about student writing behaviors: Evidence

25. Q&A http://waypointoutcomes.com/resources/bigdata Sample rubrics, PowerPoint, link to the webinar recording (once available)

26. software for 21st century education