Presentation Transcript

1. MAJOR SUPPORT SERVICE CONTRACTAcquisition Audit Steps

   FAA’s Air Traffic Controller Optimum Training Solution Contract Terrence Letko Program Director, Office of Acquisition and Procurement U.S. Department of Transportation Office of Inspector General
2. Audit background
3. Audit Background The Federal Aviation Administration (FAA) plans to hire over 11,700 air traffic controllers through fiscal year 2021 to replace the large number of air traffic controllers that were hired after the 1981 strike and are now eligible to retire. In September 2008, FAA awarded the $859-million Air Traffic Control Optimum Training Solution (ATCOTS) contract intended to provide up to 10 years of controller training support and to assist in modernizing the Agency’s controller training program.
4. Audit Background Key ATCOTS contract goals include: reducing total training costs reducing training time, and developing training innovations that can be adapted to new technologies—particularly those related to the Next Generation Air Transportation System (NextGen). In September 2010, we reported on FAA’s weak acquisition practices and lack of effective contract oversight for the ATCOTS contract. In its first 2 years, the ATCOTS contract exceeded negotiated contract values by $46 million, and the program did not achieve desired training goals.
5. Audit Team Audit was conducted by the Office of Acquisition and Procurement Audits. None of the audit team members worked on the prior audit issued in September 2010, except for the Program Director. None of the team members had experience auditing air traffic control programs or operations. We matrixed this audit with our Office of the AIG for Aviation Audits to gain an understanding of air traffic operations.
6. Why We Did This Audit The Chairwoman of the Subcommittee on Financial and Contracting Oversight, Senate Committee on Homeland Security and Governmental Affairs, requested that we conduct a follow-up review of FAA’s ATCOTS contract to determine the Agency’s progress in addressing our prior recommendations as well as report on new issues. Senator Clair McCaskill.
7. Audit Objectives Consistent with the Chairman’s request, our audit objectives were to determine whether FAA: (1) has implemented changes to improve program and contract oversight, (2) can achieve ATCOTS training goals under the current contract, and (3) has established effective performance measures to support ATCOTS training goals.
8. Audit steps and findings
9. Questions What should you do to meet the audit objectives? Where do you start reviewing on a contract or acquisition management audit? What do you do if this was the initial audit? Does the follow-on audit involve retracing some steps from the initial audit? Reviewing some of the same documents? What is a good scope and methodology?
10. Where to Start Follow-up on Prior Recommendations Contract File – Terms and Conditions and Modifications Request for Proposal or Solicitation Information Request Review Information to Contractors (Historical Data on existing contract) and Questions and Answers Provided Review Statement of Work (Compare to Bid & Contract Documents) Review Source Selection Official’s Document, Source Selection Board Report, Technical Evaluation Report (Includes Risk Assessment), Cost Evaluation Report for Competitive Procurement Review the Justification for Award for Sole Source Award Goals of the Contract or Program Use SIR/RFP/SOW
11. Where to Start Bid Proposal (Successful Contractor) Review the Government’s Independent Cost Estimate and Price Analysis Project Status Reports – Contractor’s Earned Value Management Status Reports-if used Performance Evaluation Plans and Results for Cost Incentives and Award Fees Determine whether this is a Performance-Based Service Contract PBSC contracting emphasizes that all aspects of an acquisition be structured around the purpose of the work to be performed as opposed to the manner in which the work is to be performed. The contractor has freedom to determine how to meet the Government’s performance objectives.
12. What We Found – Initial Review The cost overruns increased to $89 million through the first 4 years of contract performance. From $43 million after Year 2. FAA did not address our September 2010 recommendation to assess its training needs before extending the ATCOTS contract. Recommendation included rebaselining – more funds or recompeting the contract. To compensate for annual funding shortages, FAA reallocated funding from future contract years to meet the current year’s rising costs. As a result, FAA prematurely exhausted the contract’s 5-year base funding in only 4 years (see figure 1) without conducting the analysis we recommended in our prior report.
13. What We Found – Initial Review With Follow-on Planned base period spending $437,068,359 Total spent in 4 years $437,010,720 By end of year 4, almost no funding was left for year 5. By end of year 2, FAA had already spent half the base period funding.
14. What We Found – Initial Steps Performance-based contract was bid, but not implemented. The SOW called for a performance-based contract, which would allow the contractor to use its own ideas and methods to deliver training. However, after contract award the contractor was required to “mirror” the services and methods of the incumbent contractors, including adherence to all existing processes. The contract required mirroring for 2 years. The contract allowed for changes to methods and modernization of training through submissions of proposals and change orders to the contract.
15. What We Found – Initial Steps Section C.1.5 of the Performance Work Statement of the contract incorporates the provisions of the Statement of Work in the RFP. “This performance –based statement (PWS) establishes the scope of services to be provided by a single contractor under a performance-based contract.” (underlines & italics added)
16. What We Found – Initial Steps Section C.1.5 added conflicting requirement to “Mirror” existing procedures and practices. “Upon the start of ATCOTS, the duties performed by the contractor will mirror those performed under existing contracts.…The contractor will adhere to existing processes and training infrastructure (e.g. the current curriculum and simulators).” (Underlines added)
17. What We Found – Initial Steps Contract’s PWS conflicts with Contract Goals. Improve quality and consistency Reduce total costs to FAA of training Leverage best practices and innovation to provide comprehensive training Develop flexible training that can be adapted to met changing requirements, e.g. new technologies Develop flexible training around candidate competencies
18. Proposed Performance Measures Were Inadequate For example, one revised performance measure allowed Raytheon to earn a financial award if as few as half of all developmental controllers at terminal facilities completed training on time.However, the percentage of terminal developmental controllers who complete training on time was historically higher than 50 percent. As a result, this measure requires the contractor to achieve a passing rate that is actually lower than the historical average and is not an incentive for accomplishing exceptional performance.
19. Proposed Performance Measures – Were Inadequate FAA’s program officials were unfamiliar with implementing a performance-based contract and stated that while the contractor should have significant leeway when implementing its methodology, they intended to revise the performance measures for future performance evaluation periods. Allowing the contractor to simply use its own measures conflicts with FAA contracting guidance stating that the Government should evaluate the performance measures before awarding the contract to ensure they are measurable and likely to improve the outcomes of the program.
20. Recommended Steps – What We Found Raytheon’s bid proposal (negotiated costs) called for reducing instructor staffing levels by 30 percent during the first 3 years of the contract. FAA’s technical evaluation report identified this reduction as a high (red) risk, indicating that there was a 60- to 80-percent likelihood that training needs would not be achieved due to the limited staffing hours proposed. Risk Likelihood – Likely; Consequence – Critical. The Source Selection Board determined that specific weaknesses, such as the spoke and hub services delivery structure, could be overcome. Further, the Board concluded that the contractor’s inability to demonstrate knowledge of how En Route Automation Modernization (ERAM) could impact training would also be overcome when FAA issued training requirements via a detailed Work Plan at contract award. Found by reviewing the Technical Evaluation Report and Source Section Board’s Report.
21. What We Found Risk Assessment – Initial Review
22. What We Found Risk Assessment – Initial Review
23. What We Found – Initial Review Requirements in SIR/RFP Not Adequately Defined. The contract solicitation stated that bidders were expected to train approximately 4,000 developmental controllers annually and allowed for cost adjustments. However, FAA underestimated its initial training requirements, including the number and types of controllers that Raytheon and other bidders of the contract were expected to train. During the first year of the contract alone, about 5,620 controllers needed training—41 percent more than FAA originally estimated. How was this found. There was a gap of over one year between the SIR/RFP and contract award. The historical information was not updated. We had an advantage in that we obtained the contractor’s change order proposal (found through the contract file review).
24. Recommended Steps – What We Found IGCE Difference Was Not Sufficiently Addressed. The contractor’s cost proposal was nearly $358 million, or 29 percent lower than FAA’s own Independent Government cost estimate (IGCE). According to FAA guidance, a deviation of more than 15 percent from the Government’s estimate requires FAA to address any estimated cost discrepancies. An effective remedial action would have been to make contractors aware of any potential errors or underestimates and ask for a revised proposals. After the contract was awarded, the Office of Financial Controls—an independent review team reporting to FAA’s Chief Financial Officer—recommended that the ATCOTS program office address the 29-percent cost difference between the cost proposal and IGCE. However, the program office never responded to the review team’s recommendations.
25. Recommended Steps – What We Found No Specific RFP Training Requirements for ERAM or Other New Systems. The RFP and support mentioned stated that the contractor was responsible for providing training for ERAM. However, FAA did not provide an estimate for the amount of training that this would require. The Technical Evaluation Team reported a significant weakness because not enough hours were bid for ERAM, but did not require that the contractor(s) adjust its bid. The contractor bid about10,000 hours for ERAM training, but delivered over 76,000 hours of ERAM training for Year 1 alone, resulting in an $8 million change order request. How was this identified? We reviewed the SOW and bid, Tech Evaluation and Cost Evaluation Reports, and Source Selection Reports. We also reviewed the contractor’s proposal for a contract change order proposal. (In later steps we verified these issues with the contractor.)
26. Recommended Steps – What We Found New Simulators Were Introduced After Contract Award. FAA deployed 115 TSS (simulators) and RNAV (ground radar) without including training requirements in the RFP. Requirement is unclear. FAA Increased the Training Facilities Supported At Award. Amendment J-7 to the RFP directed the bidders to staff 159 field facilities (towers/terminals). Upon contract award it was determined that the incumbent contractor already supported 195 facilities. At award the contractor was provided with an Annual Work Plan (AWP) that required that the contractor transition to the 195 facilities in 60 days. Bid Problem. The bid was understated because the build-up to 195 facilities was bid gradually. AWP (Monthly Work Plan) Requirements Weaknesses. Monthly amendments to the AWP required staffing support at 252 facilities by the end of Year 1.
27. Recommended Steps – What We Found No Requirements in the RFP or SOW for Training of New Hire Transfers. The contractors change order request proposal indicated that 267 new hire (developmental) controllers transferred facilities, typically within the first two years. These students had to be retrained at the transfer-in facilities. Many new hires were washing-out at higher-level facilities. The RFP and Contract did not specify the number of Certified Professional Controllers In Transit (CPC-ITs). The change order proposal indicated that the number increased from 400 in 2008 to over 700 in 2011. This was not in the bid baseline. The RFP did not indicate that the contractor was responsible for training Certified Professional Controllers (CPCs) (non-developmental instruction). Approximately 10,000 CPCs. Involves proficiency training.
28. Remaining Scope and Methodology Interview the Program Office (Project Manager, Contracting Officer’s Representative, and Staff). Interview Contracting Officer and Team. Visit the Air Traffic Controller Training Academy in Oklahoma City. Meet with the Contractor (Project Manager and Staff). Review correspondence within the contract files. Review Operations Managers responsible for Annual Work Plans, operating statics (including new hires, etc.). Meet with the prior project manager (now Head of Educational Services). Conduct visits to a sample of En Route, TRACON, and tower facilities to interview training officials, union representatives, and contractor training managers. Review operations and training progress. (12 visits). Review resources and training delivery.
29. Remaining Scope and Methodology Evaluate support for contractor oversight (quality of training). Review performance plan changes and performance awards. Although not planned for all officials, eventually meet with VP Safety & Training, Senior Acquisition Executive, Chief Executive Officer, and FAA Administrator. Brief Senate Committee staffers. Participate on Contractor Quality Review. Continue review contract files and meeting with contract and project management staff. Hold status meetings with the contractor. Meet with FAA budget office (unplanned step).
30. What We Found – Remaining Steps At the end of Year 1 the contractor laid of 30 percent of its instructors to help control costs. Contract started with about 1,500 instructors. Cost overruns continued. The contract had almost completely ran off of funds by the middle of Year 4. This had great risks for disrupting services. IG notifies Hill and requests a meeting with the FAA Administrator to address concerns. Senator McCaskill issues letter of concern to the FAA Administrator.
31. What We Found – Remaining Steps FAA notifies the contractor that funding for the rest of Year 4 will be reduced.The contractor is forced to reduce instructors by another 30 percent. This brings reduction in instructors to 44 percent. From over 1,500 to about 835. FAA uses its own highly-paid CPCs to conduct classroom and simulator instructionor delayed training particularly were CPC’s were unavailable to train. Could not continuously pull controllers off the floor.
32. What We Found – Remaining Steps FAA did not review its training needs/requirements as requested in September 2010 to determine whether to rebaseline; exercise the first 3-Year Option Period; or recompete a contract that is better structured with pre-determined requirements. FAA was forced to exercise the Option. FAA begins to administer the contract on a requirements-basis, similar to a time and materials contract.
33. What We Found – Remaining Steps CPC’s and training managers indicate that they were not recording ATCOTS instruction hours in the Labor Distribution System (Cost Accounting System). Some are paid overtime. FAA does not know its total training costs. FAA’s facilities are forced to shut down training simulators because CPCs are unable to write scripts or are unable to operate the simulators, in order to replace laid-off instructors.
34. What We Found – Remaining Steps FAA has not identified its total training requirements, including training that the contractor will conduct and portion that will be provided internally. FAA has not identified facilities with available staff to train. FAA’s AWP is incomplete. Does not include training for new systems or proficiency training, but training for CPC-ITs is now identified. The AWP was terminated after Year 1, but reinstated after our initial review.
35. What We Found – Remaining Steps The incentive fees were not effective at controlling costs because FAA did not establish accurate cost targets. FAA’s AMS states that an incentive fee should be used when a reasonable and attainable cost target can be established that is likely to motivate the contractor to manage effectively. However, because FAA was unable to clearly define its training requirements, it could not establish accurate cost targets for the incentive fee. Additionally, FAA did not establish or revise cost targets until near the end of each year—too late to prevent cost overruns for the year in question.
36. What We Found – Remaining Steps FAA paid the contractor a total of $19 million in cost incentives—which include about $14 million in incentive fees and $5 million in cost-related award fees—despite $89 million in cost overruns. Award fees were not tied to key contract goals. During the base period, FAA paid the contractor 93 percent ($17.3 million) of the available award fees for meeting performance measures that did not motivate the contractor to achieve FAA’s training goals.
37. What We Found – Remaining Steps For example, FAA paid the contractor a portion of the award fee for meeting a performance measure related to staffing efficiency, which called for the contractor to stay within a set range of staffing hours. However, the contractor stated that it was not motivated to optimize staffing or lower staffing costs because any efforts to reduce staffing below the set range of hours would have lowered its award fee in this category. There were no performance measures for decreasing training time (after Year 1) or for innovating training.
38. What We Found – Remaining Steps FAA decides not to award any award fees for Period 7 (second half of Year 4). FAA states that the contractor did not deliver a sufficient amount of training at its facilities. However, the contractor argued that funds ran out and that it was required to lay off almost 300 instructors. Further, the FAA did not sufficiently backfill for the laid off instructors with its own work force. FAA rejects 11 Contractor white papers to innovate the training program, but accepts one. FAA terminated the pilot program for innovating training after Year 1. Contractor argues that simply innovation, such as conducting some initial training over the web could have saved funds.
39. What We Found – Remaining Steps FAA was unable to accomplish all of its training goals under ATCOTS.We focused our review on three of the ATCOTS goals: reducing training costs, reducing training time, and developing innovation to provide comprehensive training. We found that FAA has not been able to achieve the goals to reduce training time or leverage innovation to provide comprehensive training, and it has not assessed whether it has reduced training costs.
40. Key Goals Not Met A primary goal of the ATCOTS contract is to reduce the time it takes for a developmental to become a CPC. Between fiscal years 2009 and 2012, the time to certify controllers increased by an overall average of 41 percent—taking 9 months longer on average to certify each controller. Time to certify increased the most at terminal facilities with an average increase of 57 percent, or almost 11 months longer on average to certify controllers.
41. Key Goals Not Met Time to Certify In Years En Route Overall Terminal
42. Key Goals Not Met FAA facility managers we spoke to at 9 of 13 facilities warned that contractor staffing reductions will result in increased internal training taught by CPCs, increased overtime, and training delays. Contractor reductions have prompted some facilities, such as Las Vegas Tower, to plan to reduce simulator training, which facility training managers stated may further increase controller training times. Simulators help train not only developmental controllers but also the entire controller workforce—especially on new major systems or new procedures. However, simulators rely heavily on contractors to build simulation scenarios, and eight of the facility managers we spoke to reported that they did not have enough FAA staff to support simulator training on their own.
43. Contract Management Problems Also Occurred In September 2010, we reported that FAA did not have qualified acquisition personnel to administer the ATCOTS contract. In our current review, we determined that FAA has completely re-staffed its program office and contract management office with new employees, including the program manager, contracting officer (CO), and contracting officer representatives (COR). CO that handled day-to-day contract administration but who did not have the certification required by AMS to administer the $859-million contract. The CO performing day-to-day administration of the contract was in the process of completing certification training. According to FAA’s Acquisition Management System, a CO must apply for, acquire, and maintain certification at the appropriate level for the work performed.
44. Key Goals Not Met In 2013, only 6 months after the CO obtained the required certification, FAA assigned a new CO to the ATCOTS contract—the sixth CO since the contract was awarded. However, that CO left after a few months. FAA has also experienced high turnover with other critical program staff; for example, the ATCOTS program has had four program managers and eight CORs over the life of the contract. FAA has not adequately maintained its contract files, which exist in two separate locations—one physical file and one virtual file—and do not contain a complete history of all contract actions required by FAA’s Acquisition Management System (AMS). For example, the CO had to request official contract documentation directly from the contractor because FAA’s contracting staff did not maintain those documents.
45. Contract Management Problems Also Occurred Incomplete contract files prevent newer staff from readily accessing and understanding the contract’s complete history.
46. Testimony and Follow-up Update. FAA does not plan to award the second Option and is the process of recompeting a new contract.
47. Summary of Key Steps
48. Key Steps Review the contract file and modifications. Review the Source Selection Board and Decision Documents. Were issued sufficiently addressed to reduce risks of cost overruns, delays, or performance issues. Review the Technical Evaluation Report and Technical Evaluation Plan. Review the Cost Evaluation Plan and report. Be aware whenever the bid costs differ significantly from the Independent Government Cost Estimate. Determine whether technical weaknesses and high risks were addressed. Meet with the contractor and review the contractor’s status reports and areas areas of concern, such as cost overruns. Determine whether the award fee performance measures address each of the key goals of the contract. Did the agency change the cost target when overruns occurred? Was the target date reasonable-not too long? Were the requirements clear and did the agency provide reasonable quantitative estimates of the requirements? Did requirements change after award? Were there any change orders and what were the reasons? Were the statement of work and Performance work statements in agreement? Was the project management office performing reasonable risk assessments? Was the contractor participating in plans to mitigate the risk?
49. Key Steps Review the contract file and modifications. Review the Source Selection Board and Decision Documents. Were issued sufficiently addressed to reduce risks of cost overruns, delays, or performance issues. Review the Technical Evaluation Report and Technical Evaluation Plan. Review the Cost Evaluation Plan and report. Be aware whenever the bid costs differ significantly from the Independent Government Cost Estimate. Determine whether technical weaknesses and high risks were addressed. Meet with the contractor and review the contractor’s status reports and areas areas of concern, such as cost overruns. Determine whether the award fee performance measures address each of the key goals of the contract. Did the agency change the cost target when overruns occurred? Was the target date reasonable-not too long? Were the requirements clear and did the agency provide reasonable quantitative estimates of the requirements? Did requirements change after award? Were there any change orders and what were the reasons? Were the statement of work and Performance work statements in agreement? Was the project management office performing reasonable risk assessments? Was the contractor participating in plans to mitigate the risk?