The Institutional Artifact Portfolio Process:

2. The Institutional Artifact Portfolio Process: An Effective and Nonintrusive Method for General Education Assessment Renée M. Tobin, Derek J. Herrmann, & Kelly L. Whalen Illinois State University

3. Illinois State University: Who We Are • DRU Fall 2010 census day figures • Total enrollment: 20,762 • Undergraduate enrollment: 18,254 • 64.4% of all students are female • 13.1% of all students are minority

4. Illinois State University: Who We Are • 96.8% of undergrads are from Illinois • 93.1% of undergrads are < 24 years old • 94.6% of undergrads are full-time students

5. Illinois State University: Who We Are Fact Book (Planning and Institutional Research) – Fall 2010 • First-year students ACT mean composite scores: • Illinois State 24.1 • State of Illinois 20.7 • National 21.0

6. Illinois State University: Who We Are • Transfer Students • 1,838 new transfer students in Fall 2010 • ~50% of bachelor’s degrees granted annually • 85.0% retention from first-year to sophomore

7. University Assessment ServicesIllinois State University • UAS Office of the Provost • IR Finance and Planning The two offices work closely; however, UAS is responsible for the management of all program-level assessment.

8. History of ISU’s Gen Ed Program • Senate approval in 1992 • Full implementation in 1998 • Assessment task force in 2005 • Full implementation in 2008 • New task force in 2011-2012 • Assessment subcommittee!

9. Current Gen Ed Program • 12 Goals with 40 distinct skills/abilities • 42 credit hours • 190 potential courses • Approximately 13,410 enrollments per semester • 3 Cores: • Inner – (5 courses) • Middle – (5 courses) • Outer – (4 courses)

10. Current Gen Ed Assessment • Comprehensive, yet manageable • Purpose: To provide the Council for General Education with sound evidence to base decisions regarding the program.

11. Important ContributorsAKA Campus Buy-In • Director of General Education • Council for General Education • General Education Assessment Task Force • Assessment Advisory Council

12. Research • Reviewed past methodologies employed at ISU • Reviewed methodologies employed at other institutions

13. Initial Challenge • Goals of General Education • Numerous (12 with 40 abilities) • Difficult for faculty/staff/students to remember and reference • In some cases, challenging to measure

14. Solution • Four Shared Learning Outcomes • Common and integrated elements of the established goals of Gen Ed • Also aimed at eliminating some of the division that is present between Gen Ed and the Major

16. Selecting an Assessment Method:The Institutional Portfolio (Seybert) • Principles • Gen Ed is responsibility of entire campus • Be minimally intrusive (both faculty and students) • Use existing student work

17. Selecting an Assessment Method:The Institutional Portfolio (Seybert) • Process • “Artifacts” are collected • Faculty teams review using rubrics • Results are compiled and reported to committee • Committee makes decisions based on results

18. Selecting an Assessment Method:The Institutional Portfolio (Seybert) • Characteristics • Faculty review teams are multidisciplinary • Review is invisible to students and not intrusive to faculty • Process is labor intensive and requires resources • Process is dynamic and “messy”

19. Method – Requirements

20. Method – Requirements • Non-intrusive • Cover the 12 goals of general education • Comprehensive, Manageable • Institution-focused

22. General Overview for Assessment • Phase One: Obtaining the artifacts • Phase Two: Sampling the artifacts • Phase Three: Reviewing the artifacts

23. Phase One: Obtaining the Artifacts Instructor Participation: • Read the invitation letter from Provost • Identify artifact(s) that address at least 1 Primary Trait • Complete the online participation form • Allow UAS access to the artifacts for up to 24 hours

25. Phase Two: Sampling the Artifacts Goal – Obtain a random sample of 100 artifacts per core (300 total artifacts per Shared Learning Outcome) Behind the Scenes at UAS: • Calculate the total number of enrollments, based on class size • Remove all identifying information and oversample • End of the term, recalculate and use the appropriate proportions

26. Sampling Example

27. Sampling Example

28. Sampling Example

29. Phase Three: Reviewing the Artifacts • Instructors solicited to apply to be IAP reviewers • Interdisciplinary two-person teams (3 per Shared Learning Outcome and an alternate) are trained in use of established rubrics • Review teams complete consensus analysis using developed rubrics for the 4 Shared Learning Outcomes

30. Review Process Review Week: • Day One: • Reviewers interpret the rubrics as a group • Calibration training using practice artifacts • Days Two through Four: • Reviewers split into teams and complete their binders • Final rubrics collected throughout

31. Phase Three: Reviewing the Artifacts

33. Excerpt from the Public Opportunity Rubric:

34. Phase Three: Reviewing the Artifacts • Instructors solicited to apply to be IAP reviewers • Interdisciplinary two-person teams (3 per Shared Learning Outcome and an alternate) are trained in use of established rubrics • Review teams complete consensus analysis using developed rubrics for the 4 Shared Learning Outcomes • The final data are reported to Council of General Education to formulate commendations/recommendations

35. Results

36. Results • Reliability • Inter-rater reliability (calibration artifacts) • Percent agreement • Public Opportunity: 42% - 71% • Critical Inquiry and Problem Solving: 61% - 69% • Intraclass correlation coefficient • Diverse and Global Perspectives: .70 - .91 • Life-Long Learning: .26 - .63 • Test-retest reliability (repeat calibration artifacts) • Exploratory factor analysis

37. Results • Focus on trends for each Shared Learning Outcome • No consistent differences across freshmen, sophomores, juniors, and seniors • No clear differences emerged when comparing native and transfer students

38. Results • Maximum review value consistent with instructions

39. Discussion • Limitations of the results • Work in progress • Expand from 3 Cores to many course categories • Data based on courses from one semester only

40. Discussion • Future directions • More time calibrating during review process • Reports with data from two different semesters • General Education Task Force (and its Assessment subcommittee)

41. Conclusions • Advantages of the IAP • Nonintrusive to instructional faculty and staff and students • Allows for program-level assessment • Maintains the institution as the focus

42. Conclusions • Challenges of the IAP • Faculty participation • Locus of generalizability • Takes time

43. Conclusions • Lessons learned • Patience and public relations • Streamline the process • Oversample • Reviewer calibration is key • Alternate reviewer • Closing the loop

44. QUESTIONS?

45. For More Information • http://gened.illinoisstate.edu/ • http://assessment.illinoisstate.edu/generaleducation/ • http://assessment.illinoisstate.edu/about/newsletter.shtml

46. Illinois State University University Assessment Services Normal, IL 309.438.2135 assessment@ilstu.edu Renée M. Tobin, Ph.D. Derek J. Herrmann Kelly L. Whalen Acting Director Coordinator Graduate Assistant rmtobin@ilstu.edudjherrm@ilstu.eduklwhal2@ilstu.edu