Enhancing Student Assessment through the Delphi Method: A Case Study in Construction Management

1. Using the Delphi method as a form of assessment

2. Delphi Procedure • Complete a survey in which respondents answer a number of open-ended questions • From these responses, create a new survey that lists all the open-ended responses, and the number of respondents that wrote each response • Respondents rate their agreement with each statement on a five-point likert scale, (1) meaning "strongly disagree" and (5) meaning "strongly agree." • Statements with means of 4.0 or higher and standard deviations below 1.0 are designated as areas of agreement • With this information, one can compare like responses over time (pre and post test), conduct factor analyses (given sufficient sample size), and other statistical tests

3. Why Delphi method? • Focus groups can be difficult to manage • Assumption of anonymity could be violated • “Reticent” versus “generous” respondents • Power dynamics

4. Why Delphi method? • Rating consensus based on open-ended responses might be helpful/meaningful • Allows for a natural means of conversation about how results can be used because participants can also be the decision makers • When combined with other findings, can add specific information (e.g., adds meaning to state licensure exams, etc.) • When systematically managed, might include faculty’s review of student work samples, making it more of a “direct” observation of student performance • Might also be a way of getting employers, intern hosts, alumni, and others to participate • Can be managed electronically, without requirement that all participants sit in one room at the same time

5. Case Study: Construction Management Assessment Study • Its program accreditor, the American Council for Construction Education (ACCE), required a submission of a self study; in the self study, faculty had to assess eighteen student learning outcomes • The faculty had only six months in which to conduct alumni, employer, internship studies, in addition to some direct assessment of student work • There was no capstone course or other place in the program where students could be assessed on all accreditation student learning standards • The ACCE was contacted, and they responded to proposal to use a Delphi method affirmatively (“it is not our policy to comment on the assessment method used by programs…”

6. Task One: Selected representative work samples • Faculty were asked to identify which samples of student work they would use to evaluate students on each standard • Used as a source document

7. Accreditation standards listed Sources of evidence are listed

8. Task Two: Identify Strengths and Weaknesses in Student Performance • Faculty able to comment on each standard were then asked to look at the samples of student work they identified and make a list of strengths and weaknesses • Lists sent to analyst • There was fairly good compliance

9. Task Three: Analyze comments • Statements of strengths and weaknesses listed according to ACCE standard • Some statement combined if sufficiently the same • According to each standard, list is created. Those with like statements (coming from separate faculty) were identified as such; for example, after statement, “Students need to understand how to use accounting to submit bids better” (3 of you made this statement)

10. Task Four: Create survey • Each item on the list was assigned a five-point Liker-scale (5=strongly agree, 1=strongly disagree): Example Program Goal 1: The graduate will communicate effectively1. Graduating students need improvement in grammar. (3 of you gave this answer) 2. Graduating students need improvement in spelling. (3 of you gave this answer) …

11. Task Five: Collect and Analyze Information • Information inputted on SPSS and analyzed • Items with higher than 4.0, and standard deviation of less than 1.0 are declared “items of consensus”

12. Task Six: Report So Faculty Can Discuss and Make Recommendations • Results reported to faculty in a way that they can quickly use information • Graphical representations are often the best, where standard is listed next to assessment finding

13. Strengths and weaknesses listed next to standard

14. Task Seven: List With Findings of Other Assessments to Discover Themes (Triangulate) • Perhaps more work than usual, but process can be very helpful, if not meaningful and potentially impactful

15. Results of Delphi assessment listed along with findings of other assessments

16. Final Thoughts • The “jury is still out” in respect to the use of the Delphi method as a direct assessment of student learning • Use of this method is a good method of adding specificity to portfolio assessments that identify student performance as “exceeds expectations;” we do not know what that means until we ask faculty evaluators to tell us what that means, or what issues derive from that judgment • Can be a method of inviting faculty to play an active part in both (1) assessment itself; and (2) discussing what recommendations should be made to enhance student performance