INSTITUTIONAL EFFECTIVENESS REQUIREMENTS

1. INSTITUTIONAL EFFECTIVENESS REQUIREMENTS Alexei G. Matveev, Ph.D. NORFOLK STATE UNIVERSITY Southern University at New Orleans // Workshop on SACS Reaffirmation Preparations New Orleans, LA // October 16-17, 2008

2. Two boys are walking down the street. The first boy says, “I’ve been really busy this summer. I’ve been teaching my dog to talk.” His friend responds, “Wow! I can’t wait to have a conversation with your dog.” The first boy shakes his head. “I said I’ve been teaching him. I didn’t say he learned anything.” from Mary J. Allen (2004). Assessing Academic Programs in Higher Education Anker Publishing, Inc.: Bolton, MA.

3. Institutional Effectiveness Questions • Does the University achieve its stated goals? • How do we know that the University achieves its stated goals? • What do we do to ensure that the University continues to achieve its stated goals with excellence?

4. Agenda • Importance of Demonstrating Institutional Effectiveness • Higher education context • SACS Principles of Accreditation • Components of Institutional Effectiveness • Characteristics • Evidence

5. Institutional Effectiveness (IE): Definition • Systematic, explicit, and documented process of measuring performance against the mission in all aspects of an institution (SACS, 2005, p. 9) • Intentional process of ascertaining the level of achievement of an institution’s mission and its educational goals and objectives (SACS, 2005, p. 8)

6. IE: Goal • Continuous Quality Enhancement • “The concept of quality enhancement is at the heart of the Commission’s philosophy of accreditation. . . • Although evaluation of an institution’s educational quality and its effectiveness in achieving its mission is a difficult task. . . , an institution is expected to document quality and effectiveness in all its major aspects”(SACS, 2001/2004, p. 5) • Purpose of IE Requirements • Establish an approach to ensure and facilitate continuous quality enhancement • Provide evidence of continuous quality enhancement

7. Importance of Institutional Effectiveness

8. Importance of Demonstrating IE • Changing Student Population • Students as consumers/customers • Education as investment • “Too many decisions about higher education—from those made by policymakers to those made by students and families—rely heavily on reputation and rankings derived to a large extent from inputs such as financial resources rather than outcomes.” (The Secretary of Education Commission on the Future of Higher Education, 2006)

9. Importance of Demonstrating IE • Academic Quality • The higher education community accepts student achievement of expected learning outcomes as the key indicator of quality (AAC&U, 2004, p.1; Ewell/CHEA, 2002; Massy, 2003; Ratcliff, 1997; SACS, 2001/2004; Tagg, 2003) • Whereas “quality” was once defined in terms of inputs and resources – what the institution has—it’s now defined in terms of processes and outcomes—what the institution does with what it has (Wergin, 2005, p. 36)

10. Importance of Demonstrating IE • Accountability • “Postsecondary education institutions should measure and report meaningful student learning outcomes.” • “The federal government should provide incentives for … institutions to develop interoperable outcomes-focused accountability systems designed to be accessible and useful for students, policymakers, and the public, as well as for internal management and institutional improvement.” (The Secretary of Education Commission on the Future of Higher Education, 2006)

11. IE and SACS Accreditation Requirements and Standards • Core Requirement 2.5: Institution-Wide Approach to Continuous Quality Enhancement • The institution engages in ongoing, integrated, and institution-wide research-based planning and evaluation processes that (1) incorporate a systematic review of institutional mission, goals, and outcomes; (2) result in continuing improvement in institutional quality; and (3) demonstrate the institution is effectively accomplishing its mission.

12. IE and SACS Accreditation Requirements and Standards (Cont’d) • Comprehensive Standard 3.3.1: Program-LevelEvidence of Continuous Quality Enhancement / Outcomes Assessment. The institution 1. identifies expected outcomes for • Educational programs (including student learning outcomes for educational program) • Administrative support services • Educational support services • Research • Public/community service 2. assesses whether it achieves these outcomes; and 3. provides evidence of improvement based on those results.

13. CR 2.5 and CS 3.3.1 • CR 2.5 requires that an institution have an effective process for producing improvement and accomplishing its mission. • CS 3.3.1 requires that an institution identify outcomes (resulting from the process required in CR 2.5), evaluate achievement of those outcomes, and demonstrate improvement based on the results of that evaluation.

14. IE and SACS Principles of Accreditation “Even though the concept of institutional effectiveness may not be explicitly referenced in all of the comprehensive standards, the accreditation process assumes that all programs and services wherever offered within the context of the institution’s activity are reviewed as part of the institutional effectiveness process.” (SACS, 2005, p. 9)

15. IE and SACS Selected Accreditation Requirements and Standards (Cont’d) • CR 2.9 Learning Resources and Services • CR 2.10 Student Support Services • CR 2.11.1 Financial Resources • CR 2.12 Quality Enhancement Plan • CS 3.2.8 Qualified Administrative Officers • CS 3.4.10 Responsibility for Curriculum • CS 3.8.3 Qualified Staff: Learning Resources • CS 3.9.3 Qualified Staff: Student Affairs • CS 3.11.3 Physical Facilities • FR 4.1 Student Achievement

16. Example: IE and CR 2.10 “Support documentation and assessment data is lacking to indicate that student support programs, services, and activities support the institution's mission to promote student learning and development. Throughout the narrative reference is made to using assessment of services utilizing the CAS Professional Standards for Higher Education and Assessment Practices in Student Affairs but no examples of assessment instruments or results are given” (SACS reviewer’s comment)

17. Example: IE and FR 4.1 “Sufficient evidence is lacking regarding job placement data, graduate school acceptance, certification/licensures, etc., for those programs not represented by an external accrediting body, as well as some programs with external accreditation” (SACS reviewer’s comment)

18. CR 2.5 Institutional Effectiveness The institution engages in ongoing, integrated, and institution-wide research-based planning and evaluation processes that (1) incorporate a systematic review of institutional mission, goals, and outcomes; (2) result in continuing improvement in institutional quality; and (3) demonstrate the institution is effectively accomplishing its mission.

19. IE: What does SACS staff say? • Donna Wilkinson, VP SACS-COC • Conditions for Effective Planning and Evaluation Processes • Characteristics of Sound Planning and Evaluation Processes

20. IE: What has been learned from commonly cited issues? • Mission-Driven IE Process • “The review process by the Council for Assessment and Planning covers the number of reports completed and submitted, but it does not serve to assess the value of the reports with regard to assessment of institutional mission. The process lacks a link to the institutional mission, stated or implied, at any level -college/department or upper-level administration.” (SACS reviewer’s comment)

21. IE: What has been learned from commonly cited issues? (Cont’d) • Systematically Reviewed IE Process • “While the recent approval of the Mission and Goals by the Board provides evidence that the college periodically reviews its mission and goals, it does not provide evidence of a systematic review of the mission and goals.” (SACS reviewer’s comment)

22. IE: What has been learned from commonly cited issues? (Cont’d) • Aligned/Coherent IE Process “While there are institution-wide, integrated, data-based planning processes built on a well-defined strategic plan with appropriate goals and objectives that are adhered to by all the units on campus, there is no immediately obvious linkage between a particular goal, an assessment method, specific criteria for success, and actions taken to improve.” (SACS reviewer’s comment)

23. IE: What has been learned from commonly cited issues? (Cont’d) • Systemic and Documented IE Process “An institutional effectiveness process began in 2005. Not all units have completed the process or if they have, evidence is not shown.” (SACS reviewer’s comment)

24. IE: What has been learned from commonly cited issues? (Cont’d) • IE Process Explicitly Linked to Analysis, Decision-Making, and Budgeting “The extensive assessment reports from all the units on campus contain a wealth of specific indicators of success and a lot of raw data, but there appears to be no attempt to interpret the meaning of the data and link them to actions in a way that closes the assessment loop.  Neither does the narrative address how funding decisions are linked to the assessments.” (SACS reviewer’s comment)

25. IE: What has been learned from commonly cited issues? (Cont’d) • Improvement-Focused IE Process • “The [name of the institution’s] plan, however, is an accountability plan, requiring annual reports of enrollment, retention, graduation rates, etc. with no clear evaluation process and no description of how the results of the reports will be used.” (SACS reviewer’s comment) • “The report did not provide explicit description of use of results for continual improvement.  A “model program review system” was in place, but again, use of results was not specified.”  (SACS reviewer’s comment)

26. IE/CR 2.5 Important Indicators: What has been learned from commonly cited issues? (Cont’d) • Mature IE Process • “The institution’s processes have been strengthened greatly over the past four years and the institution documents a commitment to continuing improvement of its processes and the use of their results.  However, the institution is still developing those processes and has not yet compiled results that may be used to document continuing improvements.  The institution needs to document that it has attained results from its institutional effectiveness efforts and used the results for continuous improvements.” (SACS reviewer’s comment)

27. Components of Institutional Effectiveness Process

28. IE Process: A Model

29. Components of IE Process • Planning • Budgeting • Accountability and Assessment • Feedback Mechanisms • Coordination • IE Program Support

30. Components of IE Process: Analysis • Structural Element • Characteristics • Evidence

31. Components of IE Process: 1. Planning • 1.1 Environmental scans • Characteristics: • Periodic • External and Internal • Triangulated • Mission-based interpretation • Evidence: • Participants and data collection methods • Findings and analyses • Communication/distribution

32. Components of IE Process: 1. Planning (Cont’d) • 1.2. Strategic Plan development and implementation • Characteristics: • Explicitly mission-driven (mission statements are typically reviewed during the strategic planning processes) • Environmental scan data-based • Enforced, integrated in annual planning and reporting • Evidence: • Strategic Plan • Strategic Plan flowchart • Strategic Plan Committee meeting minutes • Annual Implementation Reports

33. Components of IE Process: 1. Planning (Cont’d) • 1.3 Alignment between internal plans and external mandates • Characteristics: • Alignment, not hierarchy • Evidence: • Crosswalks between plans • Common indicators

34. Components of IE Process: 2. Budgeting • 2. Budgeting • Characteristics: • Mission-driven and explicitly aligned with Strategic Plan • Assessment and evaluation data-based • Open and participatory process • Evidence: • Budget • Budget development process flowchart • Budget request forms and instructions • Budget Committee meeting minutes

35. Components of IE Process: 3. Accountability and Assessment • 3.1.1 Annual Reporting • Characteristics: • Standardized Annual Report template based on the Strategic Plan goals • Activities and outputs-based • Unit/provider-focused • Evidence: • Annual Reports (Units, Divisions, University) • Annual Report templates and instructions

36. Components of IE Process: 3. Accountability and Assessment • 3.1.2 Personnel Evaluation • Characteristics: • Based on the Strategic Plan objectives • Transparent process • Evidence: • Personnel Handbooks • Job descriptions • Personnel evaluation process flowchart • Evaluation forms and samples

37. Components of IE Process: 3. Accountability and Assessment • 3.2.1 Unit/Program Assessment • Characteristics: • Formative • Mission-driven and Outcomes-based • Open • Student/customer/client-focused • Evidence: • Matrix of submitted and accepted reports and sample reports • Assessment report templates and instructions • Departmental/Committee meeting minutes

38. Components of IE Process: 3. Accountability and Assessment • 3.2.2 Core Competency/Gen Ed Assessment • Characteristics: • Based on coherent Gen Ed rationale and well-articulated outcomes aligned with the University Mission • Assigned to designated individual or group (e.g., Gen Ed Council) but Participatory • Comprehensive • Representative • Evidence: • General education assessment process flowchart • General Education Program assessment report

39. Components of IE Process: 3. Accountability and Assessment • 3.2.3 University Assessment • Characteristics: • Representative • Focused on campus climate/environment • Based on perceptions, satisfaction and self-reported growth • Evidence: • Matrix of survey administration • Reports analyzing and interpreting survey data • Summary reports, displays, and presentations

40. Components of IE Process: 3. Accountability and Assessment • 3. 3 Program Review • Characteristics: • Comprehensive and summative • Periodic • Based on perceptions, satisfaction and self-reported growth • Evidence: • Schedule of program reviews and specialized accreditation visits • Program review criteria, process flowchart, and instructions • Program review reports and self-studies

41. Components of IE Process: 4. Feedback Mechanisms • Strategic Plan • University Annual Theme • President and VP’s annual objectives • Annual Reports • Unit Plan for upcoming year • Personnel Evaluations • Improvements Needed section • Assessment Reports • Improvement Plans and Improvements Made sections

42. Components of IE Process: 5. Coordination President’s Cabinet / Strategic Planning Council • Institutional Research • Enrollment data analysis and reporting • Planning and Budget • Financial planning and reporting • Institutional Effectiveness and Assessment • Outcomes assessment and quality assurance

43. Components of IE Process: 6. Support • Senior Leadership support and guidance • Designated individuals/offices to coordinate IE process • Policies • Procedures, process flowcharts, forms and templates • Handbooks and Manuals • Faculty and Staff Workshops • Software, technology • Websites

44. Sample Outline for CR 2.5 Narrative • Introduction • Planning • Strategic planning process • Budgeting • Feedback Mechanisms • Evaluation • Annual reports • Personnel evaluation • Assessment • Unit/program • Core learning areas • University • State accountability program reviews and specialized accreditations 5. Coordination of IE process 4. Continuous Quality Enhancement: Mission-Critical Indicators 1.Student success 2. Management structures 3. Funding 4. Improvements in IE process 6. Conclusion

45. Review: How Well Did the Narrative Address the IE Requirement? (from Green Book) • What are the institution’s processes for systematic, ongoing, integrated, research-based reviews that result in continuous improvement? • How does the institution demonstrate a sustained, documented history of planning evaluation cycles, including the use of results for improvement, to accomplish the institution’s mission? • Is there appropriate institutional research and budgetary support for assessment programs throughout the institution?

46. Review: How Well Did the Narrative Address the IE Requirement? (from Green Book) • What is the evidence that data from various sources concerning the effectiveness of programs and services are being used to make decisions for improvement? • How is the institutional effectiveness process related to the budget? • Are appropriate internal and external constituents and stakeholders involved in the planning and assessment process?

47. Questions and Discussion

48. INSTITUTIONAL EFFECTIVENESS REQUIREMENTS Alexei G. Matveev, Ph.D. NORFOLK STATE UNIVERSITY Southern University at New Orleans // Workshop on SACS Reaffirmation Preparations New Orleans, LA // October 16-17, 2008