PROPOSED CHANGES TO RATING METHODS

1. PROPOSED CHANGES TO RATING METHODS Compatibility, Workload, Past Performance, and Category Weighting February 11, 2016

2. Purpose • Provide consultant community and DOTD practitioners with an update on the progress of the DOTD/ACEC initiative • Obtain feedback concerning the specific recommendations for implementation

3. Background • In June 2015, DOTD Chief Engineer and ACEC BOD signed document approving recommendations for implementation • Approved 35 recommendations for implementation from 57 in 6 categories (see next chart) • Several (7) were pending, not ready • Sub committee A (procurement) continued to work • Many additional meetings (Jun –Dec 2015) • Presented to steering committee on 28 Jan 2016 and approved to move forward with this briefing

4. DOTD/ACEC Initiative Status

5. DOTD-ACEC/L Round TableGroup A: Procurement • Committee of DOTD and Consultants • Focus on consultant selection process • Actively working for over 3 years • Committee members: DOTDACEC/L Ed Wedge Simone Ardoin Bob Boagni Charles Nickel Ray Mumphrey Bob Basinger Debbie Guest Masood Rasoulian Gordon Nelson Paul Vaught Lawrence Hamm Buddy Porta Heather Huval Connie Porter Nick Ferlito Jesse Rauser Suzanne McCain Hunter Lancaster

6. Goals Evaluate the existing consultant selection process and look for opportunities for improvement by: • Ensuring that the process appropriately balances the desire to provide as many firms as possible with the opportunity to perform the work with the need to ensure that the most technically experienced firms are selected • Increasing transparency by simplifying the process and making it less subjective and more consistent

7. Key Issues • Compatibility and Workload scoring methodologies are unnecessarily complex • Determine alternate approach to evaluate those categories • Create and define new input variables • Project Magnitude • Firm Size Designation • Better address how past performance scores for new firms are assigned

8. Key Issues • Rating categories can be divided into “experience” and “non-experience” ratings • “Experience” Categories: • Staff Experience, Firm Experience, Past Performance • “Non-Experience” Categories • Compatibility, Workload, Location • Greater variation between consultants in the “non-experience” categories leads to their over-weighting in the selection process

9. PROJECT MAGNITUDE

10. Project Magnitude Project Magnitude is proposed to be determined by: 1. Complexity 2. Contract Time or Schedule (Current Solicitation) 3. Amount of the Contract (Projected for all Phases/Stages) 4. Route Classification.

11. Project Magnitude

12. Project Magnitude

13. FIRM SIZE

14. Firm Size Firm Size designation is determined using the table below:

15. COMPATIBILITY RATING

16. Compatibility Rating Once the project magnitude and firm size have been determined, the proposed compatibility rating* can be obtained by the following table. *When a team is considered, the compatibility rating will be based on the prime consultant only.

17. Compatibility Rating

18. WORKLOAD RATING

19. Workload Rating Once the firm size and remaining DOTD work amount have been determined, the proposed workload rating can be obtained by the following table.

20. Workload Rating

21. PAST PERFORMANCERATING

22. Past Performance • Existing: Firms who have not received a rating for a work category will be assigned a rating equal to the lowest of the following: • The average rating of the firms submitting • The statewide average rating for that category as of the date the advertisement was posted.

23. Past Performance • Proposed: Firms who have not received a rating for a work category will be assigned a rating equal to the lowest of the following: • The average rating of the firms submitting • The statewide average rating for that category as of the date the advertisement was posted • The satisfactory rating of 3.

24. CATEGORY WEIGHTING

25. Category Weighting The score is based on six categories: • Firm Experience • Staff Experience • Past Performance • Compatibility • Workload • Location

26. Category Weighting Three of the six categories can be thought of as “experience” based and are generally associated with technical ability to perform the work. • Firm Experience • Staff Experience • Past Performance

27. Category Weighting The other three categories can be thought of as “non-experience” based and are generally associated with the distribution of opportunity for work. • Compatibility • Workload • Location

28. Category Weighting The “experience” ratings are currently weighted as follows: • Firm Experience 3 • Staff Experience 4 • Past Performance 6 Total: 13

29. Category Weighting The “non-experience” ratings are currently weighted as follows: • Compatibility 3 • Workload 5 • Location 4 Total: 12

30. Category Weighting Although the “experience” scores are higher, the lower “non-experience” scores dominate the ranking, simply due to their larger variations.

31. Category Weighting This is a known consequence of the current system. In some cases, such as complex projects, where technical experience based selection is of critical importance, current practice is to neutralize some of the “non-experience” scores to help alleviate this issue.

32. Category Weighting Based on a sample set of contracts, the “experience” scores need to be weighted more to fully compensate for the larger variations typically observed in the “non-experience” scores.

33. Category Weighting A Sensitivity Analysis was performed using a sample set of contracts, which were scored using the proposed category weightings. This resulted in: • 50% of the contract selections were governed by the “experience” factors. • 75% of the time, non-experience factors were not of sufficient influence to move a firm ranking by more than one-third (when experience only scores were compared with the overall final scores).

34. Category Weighting The category weightings are as follows: Proposed Proposed Existing Typical Specialty Firm Experience 3 3 4 Staff Experience 4 4 5 Past Performance 6 6 7 Compatibility 3 2 1 Workload 5 3 2 Location 4 2 1 Total 25 20 20

35. Category Weighting It has been noted that the “experience” scores may still need to be weighted even more in order to achieve our goal. However, since this is based on a relatively small sample of contracts, it was felt that the proposed weightings should be used first and after one year, if it has been determined that additional weighting is required, then at that time they can be adjusted further.

36. One Year Measurables • Percentage of number 1 ranked firm controlled by “Experience” categories (at least 50%) • Firms final rankings not influenced by more than one third by the non-experience score (at least 75%)

37. Action Plan • Receive feedback from meeting today by email to Consultant Contracts, heather.huval@la.gov by February 19, 2016 • Group A evaluate feedback and make final recommendations • Submit to Chief Engineer and Steering Committee for approval • Goal is to approve process by 2016 Louisiana Transportation Conference and implementation by July 1, 2016

38. Consultant Contract ServicesContact Information Masood Rasoulian 225-379-1433 masood.rasoulian@la.gov Contracts Administration Kathy Ward 225-379-1893 kathy.ward@la.gov Contracts Manager Wanda Crawford 225-379-1406 wanda.crawford@la.gov Agreements/Invoicing Manager Heather Huval 225-379-1733 heather.huval@la.gov Advertisement Specialist Larry Hamm 225-379-1457 lawrence.hamm@la.gov Engineer 5 -DCL

39. QUESTIONS ? Latest information to be posted on the Consultant Contracts Services website under Latest News