Assessing Institutional Effectiveness

1. Assessing Institutional Effectiveness Michael F. Middaugh Assistant Vice President for Institutional Research and Planning University of Delaware Commissioner and Vice Chair Middle States Commission on Higher Education

2. Context

3. Robert Zemsky and William Massy - 1990 �[The academic ratchet] is a term to describe the steady, irreversible shift of faculty allegiance away from the goals of a given institution, toward those of an academic specialty. The ratchet denotes the advance of an entrepreneurial spirit among faculty nationwide, leading to increased emphasis on research and publication, and on teaching one�s specialty in favor of general introduction courses, often at the expense of coherence in an academic curriculum. Institutions seeking to enhance their own prestige may contribute to the ratchet by reducing faculty teaching and advising responsibilities across the board, enabling faculty to pursue their individual research and publication with fewer distractions. The academic ratchet raises an institution�s costs, and it results in undergraduates paying more to attend institutions in which they receive less attention than in previous decades.� (Zemsky and Massy, 1990, p. 22)

4. Boyer Commission on Educating Undergraduates - 1998 �To an overwhelming degree, they [research universities] have furnished the cultural, intellectual, economic, and political leadership of the nation. Nevertheless, the research universities have too often failed, and continue to fail, their undergraduate populations�Again and again, universities are guilty of advertising practices they would condemn in the commercial world. Recruitment materials display proudly the world-famous professors, the splendid facilities and ground breaking research that goes on within them, but thousands of students graduate without ever seeing the world-famous professors or tasting genuine research. Some of their instructors are likely to be badly trained or untrained teaching assistants who are groping their way toward a teaching technique; some others may be tenured drones who deliver set lectures from yellowed notes, making no effort to engage the bored minds of the students in front of them.� (Boyer Commission, pp. 5-6)

5. U.S. News �America�s Best Colleges� - 1996 �The trouble is that higher education remains a labor-intensive service industry made up of thousands of stubbornly independent and mutually jealous units that support expensive and vastly underused facilities. It is a more than $200 billion-a-year economic enterprise � many of whose leaders oddly disdain economic enterprise, and often regard efficiency, productivity, and commercial opportunity with the same hauteur with which Victorian aristocrats viewed those �in trade�� The net result is a hideously inefficient system that, for all its tax advantages and public and private subsidies, still extracts a larger share of family income than almost anywhere else on the planet�� (America�s Best Colleges, p. 91)

6. National Commission on the Cost of Higher Education - 1998 ��because academic institutions do not account differently for time spent directly in the classroom and time spent on other teaching and research activities, it is almost impossible to explain to the public how individuals employed in higher education use their time. Consequently, the public and public officials find it hard to be confident that academic leaders allocate resources effectively and well. Questions about costs and their allocation to research, service, and teaching are hard to discuss in simple, straightforward ways and the connection between these activities and student learning is difficult to draw. In responding to this growing concern, academic leaders have been hampered by poor information and sometimes inclined to take issue with those who asked for better data. Academic institutions need much better definitions and measures of how faculty members, administrators, and students use their time.� (National Commission on Cost of Higher Education, p. 20)

7. Spellings Commission on the Future of Higher Education 2006 �We believe that improved accountability is vital to ensuring the success of all of the other reforms we propose. Colleges and universities must become more transparent about cost, price, and student success outcomes, and must willingly share this information with students and families. Student achievement, which is inextricably connected to institutional success, must be measured by institutions on a �value-added� basis that takes into account students� academic baseline when assessing their results. This information should be available to students, and reported publicly in aggregate form to provide consumers and policymakers an accessible, understandable way to measure the relative effectiveness of different colleges and universities.� (Spellings Commission, p.4)

8. Middle States Accreditation StandardsExpectations: Assessment & Planning It is the Commission�s intent, through the self-study process, to prompt institutions to reflect on those assessment activities currently in place (both for institutional effectiveness and student learning), to consider how these assessment activities inform institutional planning, and to determine how to improve the effectiveness and integration of planning and assessment.

9. MSCHE Linked Accreditation Standards:Standard 14: Student Learning Outcomes Assessment of student learning demonstrates that, at graduation, or other appropriate points, the institution�s students have knowledge, skills, and competencies consistent with institutional and appropriate higher education goals.

10. MSCHE Linked Accreditation Standards:Standard 7: Institutional Assessment The institution has developed and implemented an assessment process that evaluates its overall effectiveness in achieving its mission and goals and its compliance with accreditation standards.

11. MSCHE Linked Accreditation Standards:Standard 2: Planning, Resource Allocationand Institutional Renewal An institution conducts ongoing planning and resource allocation based on its mission and goals, develops objectives to achieve them, and utilizes the results of its assessment activities for institutional renewal. Implementation and subsequent evaluation of the success of the strategic plan and resource allocation support the development and change necessary to improve and to maintain quality.

12. I Wouldn�t Want You To Think I had Access Only To Depressing Quotes �The nicest thing about not planning is that failure comes as a complete surprise and is not preceded by a period of worry and depression.� John Preston, Boston College �A common mistake that people make when trying to design something completely foolproof is to underestimate the ingenuity of complete fools.� Douglas Adams, Hitchhikers Guide to the Galaxy

13. Variables While we will discuss several variables today that contribute to assessment of institutional effectiveness, keep in mind that you don�t have to measure everything. PRIORITIZE within the context of your institution�s culture and needs.

14. Students Admitted Entering Continuing Non-Returning Graduating Alumni

15. Environmental Issues Student and Faculty Engagement Student and Staff Satisfaction Employee Productivity Compensation - Market - Equity Campus Climate Economic Impact

16. STUDENTS

17. Admitted Students What can we learn from monitoring admissions cycles? What additional drill down is needed to fully understand student admissions behavior?

18. Drilling Down Why do some students to whom we extend an offer of admission choose to attend our institution? Why do other students to whom we extend an offer of admission choose to attend a different school? How is our institution perceived by prospective students within the admissions marketplace? What sources of information do students draw upon in shaping those perceptions? What is the role of financial aid in shaping the college selection decision?

19. Survey Research is Useful in Addressing These Questions �Home-Grown� College Student Selection Survey Commercially Prepared - College Board Admitted Student Questionnaire - College Board Admitted Student Questionnaire-Plus Commercially prepared allows for benchmarking

20. Survey respondents are then asked to rate the focal institution on the 16 dimensions, compared with other institutions to which they applied and were accepted.NOTE: Student perceptions don�t have to be accurate to be real. It is the reality of student perceptions that must be addressed.

21. Important Perceptions about University of Delaware

23. Entering Students ACT College Student Needs Assessment Survey: Ask respondents to identify skills areas � academic and social � where they feel they will need assistance in the coming year. College Student Expectations Questionnaire: Asks respondents to assess their level of expectations with respect to intellectual, social, and cultural engagement with faculty and other students in the coming year.

24. Continuing/Returning Students Student Satisfaction Research - ACT Survey of Student Opinions - Noel-Levitz Student Satisfaction Inventory ACT Survey of Student Opinions Student use of, and satisfaction with 21 programs and services typically found at a college or university (e.g. academic advising, library, computing, residence life, food services, etc.) Student satisfaction with 43 dimensions of campus environment (e.g., out-of-classroom availability of faculty, availability of required courses, quality of advisement information, facilities, admissions and registration procedures, etc.) Self-estimated intellectual, personal, and social growth; Overall impressions of the college experience NOTE: Survey is available in four-year and two-year college versions.

25. What About Non-Returning Student Research?Drilling Deeper�.. Commercial instruments exist, but response rates tend to be low, and reported reasons for leaving politically correct � personal or financial reasons. For the last several years, we have administered the Survey of Student Opinions during the Spring term to a robust sample of students across freshman, sophomore, junior, and senior classes. The following Fall, the respondent pool is disaggregated into those who took the Survey and returned in the Fall, and those who took the Survey, did not return in the Fall, and did not graduate. Test for statistically significant differences in response patterns between the two groups.

26. Campus Pulse Surveys Based upon information gleaned from Survey of Student Opinions, we annually develop five or six short, web-based focused Campus Pulse Surveys directed at specific issues that surfaced. Among recent Campus Pulse Surveys: Registration Procedures Within a PeopleSoft Environment Quality of Academic Advising at the University Personal Security on Campus Issues Related to Diversity within the Undergraduate Student Body

27. Student Engagement

28. Benchmarks of Effective Educational Practice(NSSE) Level of academic challenge Course prep, quantity of readings and papers, course emphasis, campus environment emphasis Student interactions with faculty members Discuss assignments/grades, career plans & readings outside of class, prompt feedback, student-faculty research Supportive campus environment Resources to succeed academically, cope w/ non-academic issues, social aspect, foster relationships w/ students, faculty, staff Active and collaborative learning Ask questions & contribute in class, class presentations, group work, tutor peers, community-based projects, discuss course-related ideas outside class Enriching educational experiences Interact w/ students of a different race or ethnicity, w/ different religious & political beliefs, different opinions & values, campus environment encourages contact among students of different economic, social, & racial or ethnic backgrounds, use of technology, participate in wide-range of activities (internships, community service, study abroad, independent study, senior capstone, co-curricular activities, learning communities)

30. Assessing Student Learning Outcomes I�ll provide only a brief overview, as there are others (Linda Suskie, Trudy Banta, Jeff Seybert) who are far better versed than I am. That said, understand that assessment of student learning is at the core of demonstrating overall institutional effectiveness. Assessment of student learning is a direct response to the inadequacy of student grades for describing general student learning outcomes.

31. According to Paul Dressel of Michigan State University (1983), Grades Are: � An inadequate report of an inaccurate judgment by a biased and variable judge of the extent to which a student has attained an undefined level of mastery of an unknown proportion of an indefinite material. �

32. There is no �one size fits all� approach to assessment of learning across the disciplines None of these should be applied to evaluation of individual student performance for purposes of grading and completion/graduation status. 1. Standardized Tests General Education or Discipline Specific State, Regional, or National Licensure Exams 2. Locally Produced Tests/Items �Stand Alone� or Imbedded 3. Portfolios/Student Artifacts Collections of Students� Work Can Be Time Consuming, Labor Intensive, and Expensive 4. Final Projects Demonstrate Mastery of Discipline and/or General Education 5. Capstone Experiences/Courses Entire Course, Portion of a Course, or a Related Experience (Internship, Work Placement, etc.)

33. Non-Student Measures of Institutional Effectiveness:Teaching Productivity, Instructional Costs,and Externally Funded Scholarship

34. Delaware Study of Instructional Costs and Productivity Over the past decade, the Delaware Study of Instructional Costs and Productivity has emerged as the tool of choice for benchmarking data on faculty teaching loads, direct instructional costs, and externally funded faculty scholarship, all at the academic discipline level of analysis. The emphasis on disciplinary analysis is non-trivial. Over 80 percent of the variance in instructional expenditures across four-year postsecondary institutions is accounted for by the disciplines that comprise a college�s or university�s curriculum.

35. Using Delaware Study Data

36. We provide the Provost with data from multiple years of the Delaware Study, looking at the University indicators as a percentage of the national benchmark for research universities. The Provost receives a single sheet for each academic department, with graphs reflecting the following indicators: Undergraduate Fall SCRH/FTE Faculty, Total Fall SCRH/FTE Faculty; Total AY SCRH/FTE Faculty (All); Fall Class Sections/FTE Faculty; Direct Cost/SCRH; External Funds/FTE faculty

37. Science Department

38. The Delaware Study � Faculty Activity Study In Fall of 2001, the University of Delaware was awarded a second three-year FIPSE grant. This grant underwrote the cost of developing data collection instruments and protocols for assessing out-of-classroom facets of faculty activity. Once again, the grant supported an Advisory Committee charged with responsibility for refining and enhancing data collection instrumentation, data definitions, and study methodology. Faculty Activity Study collects data on 43 discrete variables related to instruction, scholarship, service to the institution, service to the profession, and public service.

39. What About Administrative Productivity???

40. Excellent Resource Taylor, B.E. and Massy, W.F. Strategic Indicators for Higher Education: Vital Benchmarks and Information to Help You Evaluate and Improve Your Institution�s Performance. Princeton, New Jersey: Peterson�s, 1996.

41. 10 Key Strategic Indicators forInstitutional Health Revenue Structure Expenditure Structure Excess (Deficit) of Current Fund Revenues Over Current Fund Expenditures Percentage of Freshman Applicants Accepted and Percentage of Applicants Who Enroll Ratio of Full Time Students to Full Time Faculty

42. 10 Key Strategic Indicators forInstitutional Health Institutional Scholarship and Fellowship Expenditures as a Percent of Total Tuition and Fees Income Tenure Status of Full Time Faculty Percent of Total Full Time Equivalent Employees Who Are Faculty Estimated Maintenance Backlog as a Percentage of the Replacement Value of Plant Percent of Living Alumni Who Have Given At Any Time During The Past Five Years

43. Useful Resource Strategic Financial Analysis for Higher Education. Praeger, Sealy and Company LLC; KPMG LLP, Bearing Point, Inc., 2006 Available from National Association of College and University Business Officers: www.nacubo.org.

44. Other Issues Related to Institutional Effectiveness In order to attract and retain the most capable faculty and staff, you have to compensate them. AAUP Academe (March/April Issue) annually publishes average faculty salary, by rank, for most institutions in the country, 2-year and 4-year CUPA-HR and Oklahoma Salary Study annually publish average faculty salary, by rank, by discipline, by Carnegie institution type. Also publish average salary for newly hired Assistant Professor. Salary equity and salary compression studies

45. Other Issues Related to Institutional Effectiveness In order to attract and retain the most capable faculty and staff, you have to provide a hospitable workplace. Employee Satisfaction Studies Campus Climate Studies

46. From the Department ofShameless Self-Promotion Middaugh, M.F. Planning and Assessment in Higher Education: Demonstrating Institutional Effectiveness. San Francisco: Jossey-Bass Publishers, 2009.

47. That�s All, Folks! What have I missed that you would like covered? Other questions???? middaugh@udel.edu