Auditing Large and Complex IT Projects Auditor General State Of Florida
Overview • Basic IT project risks. • Example IT projects in Florida. • Issues and challenges to auditing large IT projects. • Checklist of suggested audit questions. • Suggestions for further reading.
Basic IT Project Risks • Planning. • Procurement. • Contract management. • Development and Implementation. • Post-implementation.
IT Project Risks - Planning • Lack of clearly defined goals, objectives, and requirements. • Lack of identification of project risks. • Poorly defined project management structure. • Unrealistic budget – time and dollars. • Lack of stakeholder & user buy-in. • Extent and impact of business process reengineering not sufficiently addressed. • Inadequate basis for development of procurement criteria. • No established baseline for measuring cost savings
IT Project Risks - Procurement • Poorly defined statement of work. • Lack of basis for competitive solicitation. • Flawed evaluation of proposals. • Litigation after award.
IT Project Risks – Contract Management • Contract terms that don’t adequately protect the state’s interest. • Poorly defined deliverables. • Lack of agreed upon performance measures. • Payment without suitable performance. • Lack of recourse for non-performance or poor performance by contractors. • Vague ownership provisions – data, software, hardware. • Vague or missing provisions regarding data security. • Poor or lacking termination & transition clauses to protect state’s interest should either party cancel.
IT Project Risks - Development and Implementation • Implementation of software that does not function properly & include good controls. • Poor data security. • Inadequate training and knowledge transfer, resulting in lack of sufficient knowledge to operate, maintain, and use the new system • Cost overruns. • Failure to meet implementation deadlines • Over-customization of vendor software making future upgrades costly to obtain and implement. • Loss of data integrity during conversion/cutover.
IT Project Risks – Post Implementation • Security vulnerabilities as a result of security not being hardened after implementation. • Integrity of system compromised through poor maintenance and change control. • On-going reliance on contractors. • In-house knowledge not sufficient to maintain system or infrastructure. • Unable to turn off old systems that were to be replaced. • Unable to sustain user buy-in.
Example Florida Projects • My Florida Marketplace. • People First. • MyFlorida Alliance. • Aspire. • Various educational entities.
MyFlorida Marketplace • A Web-based e-procurement system for state agencies. • Developed by customizing Ariba software. • Application Service Provider – Accenture. • Responsible State agency: Department of Management Services. • Total contract cost : $93.9 million. • Contract term: October 2002 through November 2010. • Funded through 1% transaction fee on purchases.
MyFlorida Marketplace – Planning and Procurement Issues • Cost-benefit and risk analysis not conducted prior to decision to outsource and issuance of ITN. • Insufficient involvement of key end-users and stakeholders in development of ITN. • Lack of significant baseline data for planning and analysis. • No mechanism to capture and track statewide costs associated with MFMP.
MyFlorida Marketplace – Contract and Project Management Issues • System not formally accepted prior to implementation. • Heavy reliance on ASP without sufficient monitoring of performance. • Limited monitoring by DMS. • Third-party monitor’s duties strayed from monitoring ASP.
MyFlorida Marketplace – IT Functionality and Control Issues • Deficient change control process. • System performance and capacity management needed improvement; performance issues existed. • Disaster recovery plan not timely approved and lacked important provisions. • Insufficient back-up provisions. • Deficiencies in security controls.
MyFlorida Marketplace – IT Functionality and Control Issues • Data integrity issues. • Problems with accounting system interface. • Problems with attached scanned documents. • Declining use of system by State agencies. • System functionality issues. • System performance issues. • Workflow inefficiencies.
People First • An HR outsourcing initiative. • Includes a Web-based enterprise-wide ERP system supporting: • HR administration. • Benefits administration. • Payroll administration. • Staffing administration.
People First • Service provider – Convergys. • 9-year contract. • $349.9 million. • People First application built using SAP software. • System phased in between May 2003 and January 2005. • Convergys provides Florida with a SAS 70 report on its service center.
People First – Planning and Procurement Issues • Cost-benefit and risk analyses not performed prior to release of ITN. • Inaccuracies in cost estimates within Business Plan. • No system to track statewide project cost. • Deficiencies in evaluation and negotiation processes.
People First – Contractual Issues • Legal records retention requirements not included in contract. • No provision for State to approve new or changes in subcontractors. • No provision for subcontractors to obtain background checks. • Many deliverables not timely provided. • Additional amounts paid to third-party monitor for performing services already required.
People First – Operational Problems • System functionality problems and errors. • Lack of written security guidelines. • Off-shoring of State employee personnel data. • Planned system components not implemented, requiring workarounds by the agencies.
MyFlorida Alliance • An effort of the State Technology Office to reengineer its IT functions and governance structure through outsourcing many of its primary functions. • STO was responsible for centralized management of IT for the executive branch of Florida government.
MyFlorida Alliance – Functions to be Outsourced • Enterprise e-communications. • Enterprise technology services desk. • Enterprise applications management. • Enterprise data center operations and consolidation. • Enterprise portal, enterprise security, various others planned.
MyFlorida Alliance • Two prime contractors – Bearing Point and Accenture. • Contracts signed August 13, 2003. • 7 year term. • $324.7 million. • Procurement method: ITN. • All agreements since terminated by STO. • STO abolished in law effective 7/1/07.
MyFlorida Alliance • Inadequate planning & documentation to support decisions to outsource or use ITN method. • Deficiencies in proposal evaluation and negotiation, limiting fairness and competition. • Contracts lacked provisions to protect the State and did not pinpoint total cost to State. • Cost savings analyses questionable or not available.
Aspire • Replacement for the State’s general ledger accounting and cash management systems. • Project beginning date - 9/8/2003. • Original planned rollout in three waves of agencies going live, from July 2005 through December 2005. • Rollout schedule amended five times. • Project suspended 5/17/2007, with $89 million spent to date.
IT Projects at Florida’s Educational Entities • University ERP implementations – for example, University of Florida. • Insufficient system testing • Insufficient staff training • Functional system problems • System governance issues • IT security and control issues • Various community colleges and school districts.
Issues and Challenges to Auditing Large IT Projects • When to audit. • During the project? • Postaudit? • Impact of outsourcing. • Authority to audit contractor & subcontractors • Contractor responsiveness to audit requests • Applicability of SAS 70 audits, if available, to non-financial audit objectives • Lack of sufficient entity knowledge of system. • Difficulty in extracting data needed for audit • Availability of entity staff to timely respond to audit requests during implementation or post-implementation.
Suggested Audit Checklist • Planning. • Procurement. • Contract Provisions and Management. • Project Management.
Suggested Further Reading • MyFlorida Marketplace • 2007-076 (IT Audit) • 2006-015 (IT Audit) • 2005-116 (Operational Audit) • People First • 2007-087 (Operational Audit) • 2005-047 (Operational Audit)
Suggested Further Reading • MyFlorida Alliance • 2005-008 • University of Florida • 2006-040 (Operational Audit) • 2006-145 (IT Audit) • Chapter 2007-115, Laws of Florida • Abolishes STO and creates AEIT
Questions?
