Program Evaluator (PEV) Face-to-Face Training

1. Program Evaluator (PEV) Face-to-Face Training

2. Welcome! “Our duty is to believe that for which we have sufficient evidence, and to suspend our judgment when we have not.” ~ John Lubbock Facilitator Introductions List of Participants – on each table Introduce yourself – first time you speak

3. ABET Welcome ABET Headquarters Representative

4. Why Are We Here?Course Purpose • Provide you with the knowledge, skills, and abilities to conduct an ABET accreditation evaluation. • Learn how to provide guidance to programs for improvement. • Our goal: To improve consistency of program evaluation.

5. Course Purpose(cont.) This Face-to-Face Training is an experiential workshop that simulates a real visit as closely as possible. It works in conjunction with the on-line Pre-Work modules, designed for your independent and convenient learning. Outputs from Pre-Work Outputs from Face to Face Face to Face Training Pre-Work Process

6. Why Are We Here?Learning Outcomes At completion of the training, you will be able to: • Plan an effective, cordial, efficient campus visit. • Interact with your team during a visit. • Given a specific situation or evidence (e.g., transcripts, student work): • Determine which criteria are relevant to determining the accreditation action. • Determine if the program is in compliance with a criterion. If not, determine the level of compliance and the accreditation action.

7. Learning Outcomes (cont.) At completion of the training, you will be able to: • Conduct interviews of faculty, staff, and students. • Write an exit statement to the institution. • Conduct an informal debrief with the program head and exit meeting with leaders of the institution. • Complete Program Evaluator forms. • Demonstrate “in practice” effective use of PEV competencies in the PEV Competency Model.

8. Agenda - Day One Morning

9. Agenda - Day One Afternoon

10. Agenda – Day Two

11. Agreements Speak with intention. Listen with attention. Know that the wisdom is in the room. Ensure representation from all pertinent voices. Model the PEV competencies in all facets of the training. Turn off/silence electronic devices (cell phones, laptops, etc.). Start/stop on time.

12. Training Materials • Notebook • Slide Presentation • Reference Notebook • General Accreditation Materials • Commission Specific Materials • Exercise Handouts • Table Supplies • Issue Bins

13. Module 2: Connection to Pre-Work

14. Pre-Work Modules Overview The Accreditation Process The Role of the Program Evaluator Continuous Improvement of Student Learning Applying the Criteria The PEV Appraisal Process

15. Completed Pre-Work Outputs from Pre-Work Pre-Work Process Face to Face Training Outputs ` Face to Face Pre-Requisite Work Modules: • Overview • The Accreditation Process • The Role of the Program Evaluator • CI of Student Learning • Applying the Criteria • The PEV Appraisal Process • Completed: • Proficiency Assessments (3) • Upper State University Self-Study and Transcript Analysis (PEV Report and PEV Worksheet as appropriate for Commission)

16. Pre-Work Refresher Cards Exercise In your table groups: • Use the refresher cards to test your knowledge of Pre-Work topics: • Each take a turn asking, answering, and recording the result. • If you get an incorrect response, another group member may ‘help’ out. • Your Support Facilitator has the answers and is available for assistance. • Record the answers your team gets correctly the first time, second time, and third time AND the number of cards you have gone through. • YOU HAVE 50 MINUTES. Handout: Refresher Cards

17. Upper State University Pre-Work Analysis • Pre-work analysis provides starting point for evaluation of Upper State University. • Face-to-face training simulates a campus visit to Upper State University. • “On-site” activities reveal additional insight and information about the program. Therefore: • Keep an open mind. • Stay in information-gathering mode. • Do not look through the lens of “reinforcing your pre-drawn conclusions.”

18. Upper State University Site Visit At the end of the training you’ll leave with: • Table team decision whether the program meets the criteria. • Team consensus. • My interpretation and your interpretation of like circumstances must be the same.

19. ABET Code of Conduct(Rules of Procedure Section EIGHT A & B) • Accept responsibility for decisions you make ... • Serve only in areas of competence. • Act as agent of ABET and avoid conflicts of interest. • Keep decisions confidential. • Be objective and truthful. • Behave honorably, ethically and lawfully. • Report concerns with respect to internal financial matters. • Treat all persons fairly. • Assist colleagues … in following the Code of Conduct. • Support prompt and fair adjudication of alleged violations...

20. Conflict of Interest(Rules of Procedure Section SEVEN A & B and APPM II.C) • Current or past employment as faculty, staff, or consultant by the institution or program; • Current or past discussion or negotiation of employment with the institution or program; • Attendance as student at the institution; • Receipt of an honorary degree from the institution; • An institution or program where a close family relative is, or was, a student or employee within the past 10 years; or, • An unpaid official relationship within the past 10 years with an institution.

21. Dress Code for Visits • Day 0: Business Casual (Usually a Sunday afternoon campus visit. Time used to review course materials, assessment information, and tour laboratories.)

22. Dress Code for Visits • Day 1: Business Professional (Meetings with Faculty, Students, Support Areas) • Day 2: Business Professional (Exit Meeting with CEO of Institution)

23. JIT Evaluation

24. Module 3Display Materials and Facilities

25. Interview withLaboratory Manager

26. What to Look For: Facilities • Labs/Design Studios • Sufficient number and size of labs. • Appropriate coverage across the breadth of specializations within the program. • Appropriate equipment, in good repair. • Appropriate student access (including evening and weekend access). • Appropriate technician support and instructional support in lab.

27. What to Look For: Facilities • Labs/Design Studios, cont. • Safe physical arrangement and appropriate safety practices followed. • Comments from students about the lab experience. • Is there some type of “Laboratory Plan” for maintaining and upgrading the instructional laboratories? This is not required by ABET, but, in general, a formal or informal plan of some sort should exist.

28. What to Look For: Facilities • Classrooms • Appropriate physical arrangement, support for computer projection, etc. • Not overcrowded. • Support facilities • Sufficient computer access, with appropriate off-hours access. • Appropriate spaces for students to gather (not an explicit criterion but relates to several criteria). • Appropriate shop with parts, repair facilities, etc. • Faculty offices • Sufficient size, privacy.

29. Panel Discussion

30. Display Materials

31. Display Materials

32. Display Materials (APPM II.G.6.b) • Course materials including course syllabi, textbooks, example assignments and exams, and examples of student work, typically ranging from excellent through poor. • Evidence program educational objectives are based on the needs of the stated program constituencies. • Evidence of the assessment, evaluation, and attainment of student outcomes. • Evidence of actions taken to improve the program.

33. Small Group Sessions with USU Display Materials • Descriptions of materials for: • Curriculum/Course Work • Senior Design Projects, Capstone or Integrating Experience (if required) • Demonstration of Student Outcomes • Assessment Materials Handouts 3-1 & 3-2

34. What to Look For: Course Materials • Courses appear appropriate to accomplish the Program Educational Objectives and Student Outcomes. • Student work indicates active engagement, demonstration of learning, and reasonable grading standards • Evidence with respect to specific Student Outcomes (including those in the criteria) as appropriate to the assessment plan. • A few missing items, or even entire missing courses, are not necessarily systemic problems. Pursue any major gap to see if it represents a serious problem.

35. What to Look For: Laboratory Reports • Evidence (lab reports) of appropriate student learning (not just cookbook). • Evidence to support Student Outcomes.

36. What to Look For: Major Design or Integrating Experience • Evidence that all students complete a major design, capstone or integrating experience (if criteria require) that draws on previous courses and incorporates standards and realistic constraints. Student work should demonstrate this via a complete project report or some other mechanism.

37. What to Look For: Assessment Materials • Examples of instruments used and actual data collected. • Summaries of the data with results reported in a usable form. • Recommendations for program improvement based on the data. • Implementation and results.

38. Documentation (APPM II.A.6) • Institutional catalogs and similar publications must clearly indicate the programs accredited by the commissions of ABET as separate and distinct from any other programs or kinds of accreditation. • Each accredited program must be specifically identified as “accredited by the _________ Accreditation Commission of ABET, http://www.abet.org.” • Each ABET-accredited program must publicly state the program’s educational objectives and student outcomes. • Each ABET-accredited program must publicly post annual student enrollment and graduation data per program.

39. JIT Evaluation

40. Module 4:Sunday Team Meeting

41. Sunday Team Meeting Agenda Agenda Review Visit Schedule Program Evaluator Pre-Visit Briefings Program Strengths Potential Program Shortcomings Recommended Action Display Material and Facility Findings Agree on Common Findings and Plans to Investigate Debrief Handout 4-1

42. Actual Visit Sunday Meeting • Usually involves review of multiple programs with one PEV per program. • Discussions will focus on common shortcomings across programs and what the team needs to do to ensure consistent evaluation and recommended action. • Usually two meetings: • Before review of display materials and facility tours. • After review of display materials and facility tours.

43. JIT Evaluation

44. Module 5:Interviews

45. Why Interview? Direct interface with the members of the program: • Facilitates understanding of the program. • Provides an opportunity to clarify items in the Self-Study and to probe for further details. • Effective way to identify problems not addressed in the Self-Study.

46. Panel Discussion

47. Faculty Interview

48. Managing Problem InterviewsExercise In your table groups: • Draw card from scenario card deck. • Share ideas for dealing with the situation: • What would you say in this situation? • Facilitator: Capture ideas on sticky-notes. • Post ideas/responses for each scenario on appropriate flipchart in room. Handout: Interview Cards

49. The Art of Asking Questions • All interview questions should be DESIGNED to elicit a detailed response. • Determine and clarify objectives BEFORE the interview. • Prepare questions.

50. The Art of Asking Questions • Begin with “why”, “how”, “what”, “when”, and “where”. • Avoid leading questions that assume an answer. • Prompt for specific information: “Show me.”