A&F Program Reviews

1. A&F Program Reviews • Objectives of this presentation: • As a result of viewing this presentation you will be able to: • Describe the purpose of an A&F Program Review • Describe the difference between the Full and Alternate Cycle Program Reviews • Describe the three phases and associated steps of each process • Describe the data gathering methods recommended for each process • List who is responsible for each step in the processes

2. To comply with SACSCOC accreditation requirements • To maintain a formal unit review process • To conduct two formal reviews every 12-years • To focus on unit objectives, assessment, and planning • To assist units to incrementally improve • To ensure setting and completion of unit objectives • To evaluate the quality of work performed • To continuously improve the quality of customer service Purpose of A&F Program Reviews

3. SACS Focus In support of clearly stated mission…

4. Full Program Review Steps

5. Full Program ReviewPHASE 1: Self-study

6. Phase 1: Self-study • Conduct research/writing that leads to the development of a self-study “binder”

7. Any statements that the unit uses to define its central purpose to include mission, vision and values where possible Committee should review unit’s mission, vision and values and how they relate to the divisional and university missions, visions and values Phase 1 Data Gathering:Mission, Vision and Values

8. Objectives on the planning database Internal objectives Major future initiative for change/improvement Phase 1 Data Gathering: Goals and Objectives

9. Every unit must have an up to date policies and procedures manual, either in print, electronically, or both • Policies: Important unit policies, rules, standards, etc. • Procedures: Step by step procedures for each primary function Phase 1 Data Gathering: Updated Policies & Procedures

10. A process by which members of the unit collaboratively discuss and determine: • Strengths: The unit’s strengths in terms of productivity, quality, customer service, worker satisfaction, etc. • Weaknesses: The opposite of strengths • Opportunities: Current and potential future events that will provide the unit with an opportunity to improve or add to services and/or performance • Threats: Current and potential future obstacles Phase 1 Data Gathering: S.W.O.T. Analysis

11. The names and contact information for key individual customers The names of departments, units, teams, that are considered core customer groups. Phase 1 Data Gathering: Key Customers & Customer Groups

12. Names, titles, and contact information for persons at other universities/institutions that perform the same or similar work. • If the external reviewer has not been determined, this information can help • If a peer study is to be done, the unit can best determine who at other institutions is able to provide the most helpful analysis Phase 1 Data Gathering: Peers

13. What specifically does the unit do well? • Verified customer service success • “High scores on recent customer survey” • High employee morale, teamwork • Low turnover, warm culture, effective conflict management, etc. • Overcoming obstacles, adapting to change • “We successfully navigate changes, differing customer demands, etc.” • Quality performance • “We very rarely receive complaints from customers or those to whom we report” • Etc. Phase 1 Data Gathering: SWOT - Strengths

14. What specifically does the unit struggle with? • Verified and consistent customer complaints • Low employee morale, teamwork • High turnover, poor conflict management, etc. • Reticence to change • Frequent mistakes in a particular area, errors • Poor customer access (geography, technology, etc.) • Understaffed or resourced • Etc Phase 1 Data Gathering: SWOT - Weaknesses

15. What are potential chances for improvement/success? • New position(s) • Change in legislation or policy that will allow more freedom of operation • New unit responsibilities • New core customer groups • Changes in organization/structure • New technology becoming available • Competition • Etc. Phase 1 Data Gathering: SWOT - Opportunities

16. Potential obstacle to mission, vision, objectives • Reduced funding • Loss of key personnel • Changes in legislation or policy that will create obstacles • Competition • Exclusive knowledge-holders • Potential ethics issues • Changes in organization/structure • Etc Phase 1 Data Gathering: SWOT - Threats

17. Full Program ReviewPHASE 2: Committee Work

18. The goal of the committee: Conduct research that results in an accurate list of findings concerning unit performance and a list of recommendations for unit improvement. • Finding: A statement of fact or discovery as a result of research • Recommendation: A specific suggestion that, if carried out, would likely result in improved unit quality/performance Phase 2:Committee Phase Goal

19. Phase 2: Serving on the CommitteeExpectation of Service on Committees Here’s the current Program Review Schedule:

20. Assist the unit to improve performance by making specific recommendations based on the committee’s research-based knowledge Phase 2: Serving on the CommitteeThe Committee’s Role

21. Read the self-study binder Attend all committee meetings Serve as a sub-committee member or chair Participate in one or more research projects related to the review Write and/or collaborate on writing Provide advice and support to committee chairs Contribute to committee discussions Contribute to the drafting of the report Phase 2: Serving on the CommitteeCommittee Member Duties

22. Focus group • Survey • Internal study • Employee interviews • Analysis of processes and procedures • Peer review • Report writing • Etc. Phase 2: Serving on the Committee Potential Subcommittees

23. Phase 2: Committee Research

24. Phase 2: Data GatheringPersonal Interviews • Unit head, unit members, AVP, Senior VP, key individual customers, etc. • Determine persons to be interviewed • Write interview questions • Schedule one-on-one interviews • Sample Questions:

25. ER should determine: How the unit compares to industry standards in terms of services and operation What best practices the unit might consider adopting Any gaps in the unit’s legal or policy compliance Changes the external reviewer would recommend Phase 2: Data GatheringExternal Reviewer (ER)

26. Work with the Unit Head to brainstorm possible ERs • Committee chairs contact the ER • Arrange schedule • Write to confirm schedule and expectations for study and report • Co-chairs are responsible for liaisoning with the ER • Related expenses come from the unit budget & normally include travel and lodging Phase 2: Data GatheringExternal Reviewer (ER) -The person should be a trusted & knowledgeable professional from another institution/university who conducts an on-campus visit and review • External Reviewer Basic Steps: • Contact/Schedule • Written expectations • Visit • Report • Expense reimbursement

27. A gathering of 8-12 unit constituents assembled to discuss unit performance through moderated discussion • A purposeful, yet informal discussion of issues • Questions are prepared in advance • Usually a moderator/facilitator and at least one note taker • Normally, more than one focus group is conducted for a given review • Much more a discussion or dialogue than a “survey” – not all questions must be answered by all participants. Phase 2: Data GatheringFocus Groups

28. Who: A focus group is a gathering of 8-12 unit constituents/customers assembled to discuss unit performance • Why: • Get somewhat informal, discussion-based feedback on unit performance • Learn the words/terms customers use in relation to the unit (for possible use in other research) • Brainstorm ideas for unit improvement • Learn the “hot button” areas of unit performance Phase 2: Data GatheringFocus Groups

29. Encourage each person to speak his/her mind Develop an understanding of the “language” of unit constituents Determine perceptions and misconceptions Explore terms and concepts thoroughly Ask “why?” Hear the unit’s “story” from the group’s perspective Generate ideas for unit improvement Phase 2: Data GatheringFocus Groups

30. What a Focus Group IS NOT: A statistically reliable survey Research of a representative sample A gripe session A lecture A source of statistical conclusions Phase 2: Data GatheringFocus Groups

31. FOCUS GROUP STEPS Establish moderator and note-taker(s) Arrange time, date, and location Determine 18-21 participants who share a common connection with the unit (three groups of 6-7) Send out initial invitation letters 4 weeks in advance Arrange for light refreshments Make telephone follow-up call Determine participants and send reminder e-mail Complete work on questions and strategy Send final reminder e-mail one week prior Phase 2: Data GatheringFocus Groups

32. MODERATOR STRATEGIES • Use an inverse pyramid approach to questions • Start with broad, “soft” questions • What do you think are the one or two most important ways in which computer technology has changed your work life? • What is the most fulfilling aspect about your work at JMU? • Work toward more specific questions and questions that may be more emotionally charged • Let’s talk about your last interaction with the Help Desk. How did it go? Was your problem solved quickly? Were you made to feel important? • What mistakes do you think the university is making in its approach to diversity? Phase 2: Data GatheringFocus Groups

33. MODERATOR STRATEGIES • Develop ways for the people to relax • Refreshments, opening ice breaker, introductions, etc. • Promise to start and end on time (90 minutes tops) • Assure individual confidentiality/anonymity • Be alert – look for opportunities to ask follow-up questions or pursue topics initiated by the group members • Spread the conversation around, don’t be afraid to call on the quiet ones and gently interrupt the dominators • Remember the goal is to harvest perceptions, ideas, concepts, etc. • Assure the members of the group that their opinions matter. (In fact, their opinions are all that matter in a focus group such as this.) Phase 2: Data GatheringFocus Groups

34. Sample questions • Let’s talk about how you interact with the unit. What does that usually look like? • As a group, let’s see if we can list 5-7 ways in which the unit could be more effective. • What are stories you can share that illustrate the unit’s strengths and weaknesses? • What are areas in which the unit does not seem to understand your particular needs? • If you were put in charge of the unit, what’s the first change you would implement? Phase 2: Data GatheringFocus Groups

35. Review the documents in the Self-study, previous program review reports, websites, etc. • Read the documents to: • Learn about key unit processes and procedures • Understand how the unit communicates itself to its customers – to understand how it truly perceives its purpose and make suggestions for improvement • Learn about the unit’s strengths and weaknesses related to communication and organization Phase 2: Data GatheringDocument Review

36. If the self-study binder is included, a document review should be conducted • Otherwise: Previous program review reports, the unit’s website, publication material, etc. • Gather information concerning: • Key unit processes and procedures • How the unit communicates its purpose, services, and objectives internally and externally • Learn about the unit’s strengths and weaknesses related to communication and organization Phase 2: Data GatheringDocument Review

37. Committee members bring to the committee an existing knowledge and experience base. A committee that works well together can effectively harness this information to make very helpful recommendations • Committee members are chosen, in part, for their already-existing knowledge of and connection to the unit • This collaboration includes interaction and discussion designed to determine findings and recommendations that will benefit the unit and its customers Phase 2: Data Gathering Committee Collaboration

38. Used when the unit has a broad spectrum of key customers throughout the university (Public Safety, Parking, Payroll, etc.) • Quickly gather quantitative unit performance data • Measure customer satisfaction in key areas • Learn key strengths and areas of needed improvement Phase 2: Data GatheringSurveys

39. Pyramid approach – general to specific • Be as specific as possible with each question • Do not combine two or more questions into one • Questions should be objective and not leading • Think about making it easy to respond to: • Keep it short – five to ten minutes or less • Assure confidentiality and anonymity • Provide an initial invitation and follow-up reminder • ALL SURVEYS MUST BE REVIEWED BY THE PROGRAM REVIEW COORDINATOR BEFORE DISTRIBUTION Phase 2: Data GatheringSurvey Basics

40. Phase 2: Data GatheringSurvey Questions

41. Rick Larson (larsonrd@jmu.edu) 8-4248 Phase 2: Data GatheringSurvey Resource Person

42. Includes contacting knowledgeable industry professionals (5-10) at other universities, institutions This method is used when the unit has a particularly technical function or requires a level of specialized expertise, certification, etc. This method is particularly well suited when evaluating processes in units such as those in IT, Finance, and Public Safety Phase 2: Data GatheringPeer Studies/Best Practices

43. Potential Questions for Peer Study Phone Interviews: • Briefly describe your unit’s key function(s) and / or service(s) • How many staff? Full-time? Part-time? • Annual budget? • Who are your primary customers? • What do you expect will be the most important factors affecting your work in the next 24 months? • Of what service elements are you most proud? • What changes have you made in the past year to improve your customer service? • Which of your services do you consider a “best practice”. Please describe what you do for each. Phase 2: Data GatheringPeer Studies/Best Practices

44. This includes a detailed review of the steps associated with one or more processes such that the committee may list findings and recommendations. Examples include: • Paperwork – Is it handled accurately and efficiently? • Data – Is it entered efficiently, always secure, and managed in accordance with relevant laws and policy? • Does information flow lead to effective customer service? Phase 2: Data GatheringInternal Process Studies

45. Used when the unit’s customer service performance is highly dependent on internal processes, data flows, document management, etc. or if concerns have been voiced about quality or timeliness • Payroll, Accounts Payable, HR, etc. • Why: • Learn where customer service performance is hindered by internal flow issues • Learn obstacles related to organization and/or process Phase 2: Data GatheringInternal Process Studies

46. Subcommittees send reports & data to committee chairs Chairs lead committee in first draft of report – executive summary, findings, recommendations Unit Head reviews draft and has an opportunity to respond – this step allows ownership by Unit Head Co-chairs lead committee in finalizing report Co-chairs meet with unit head, AVP, & Senior VP Accepted recommendations become objectives Phase 2:Conclusion

47. Full Program ReviewPHASE 3: Progress Review

48. Senior Vice President meets with the AVP to discuss progress. Phase 3:

49. Alternate Cycle Program Review

50. Alternate Cycle Program Review Steps