Natural Gas Pipeline

1. DAMPIER TO BUNBURY Natural Gas Pipeline

2. Maximo Data Loss Aug 2011 Summary of incident On Wednesday morning 3rd Aug 2011 at approximately 7:36am, the Maximo server had one or more pieces of code executed on it that were leftover code from the Maximo 4 to Maximo 6 upgrade. This code purged data all the data in many tables and refreshed the database with legacy data. At 7:36am on the morning our servers had 179,000 workorders and at 7:37 the system only had 3,000 workorders. All the workorders and associated info were purged from the MAXIMO production database. We managed to restore from last nights backup and were back up and running by 10:00am with a loss of only 1hrs transactions Although it has been difficult to identify we have been able to replicate the scenario of the event that occurred during the incident. We have been able to collect sufficient actions to safeguard the event from re occurring via the ICAM actions. The procedures themselves were not named such that they could reflect a loss of data as a result but were named “Fetch_Data” which resulted in data loss. The code was not executed by a process from within the Maximo Application, but was executed via a process that connected outside of the application which has been identified as “SPID 53” or process ID 53. This has been identified as “Crystal Reports” which executed the stored procedures when browsing the database with no warning or prompt that stored procedures would be executed. The code fired as part of Crystal reports simply listing what procedures were available to display. Crystal reports was believed to be a read only tool used to query databases, and the fault was that legacy code that instructed data deletion was left on the server. It has become evident that any ODBC software including MS Access or Excel etc could have had the same behaviour and outcomes due to the stored procedures inadvertently being executed. Immediate action has been taken to remove the stored procedures from the database to prevent any reoccurrence.

3. Maximo Data Loss Aug 2011 • The Stored Procedures visible in the database • Figure 2 illustrates the “Insert_Record_Proc” stored procedure that was executed and caused 10 of the Maximo tables to be truncated and subsequently populated with data from the Maximo 4. The Full procedure is available in the supporting Zip files. • The Truncate Table statement “Removes all rows from a table without logging the individual row deletions. TRUNCATE TABLE is functionally the same as the DELETE statement with no WHERE clause; however, TRUNCATE TABLE is faster and uses fewer system and transaction log resources.” • The procedure then inserts all records from the a “Dummy” table (Maximo4) into the workorder table • more than one stored procedure was executed, and they were executed in an order by SPID 53. • The “Objects” listed in the transaction Log are in the order of Invcost, InvCost_Temp, maxsequence and po. This correlates to the commands in the stored procedures “Fetch_InvCost”, “Fetch_InvCostTemp” and “Fetch_PO”. Note: The Fetch_PO stored procedure updates the maxsequence table. Furthermore, all of these transactions took place within 12 seconds.

4. Maximo Data Loss Aug 2011 Legacy Stored Procedures The following stored procedures reference Maximo4, Dummy tables or look like code from the migration and can be assumed as legacy code and should be removed from the server. Their naming convention in no way identifies them as dangerous procedures that can result in the loss of data. These procedures were removed first from Test Environment and then from production after full end to end testing.

5. Maximo Data Loss Aug 2011 • Crystal Reports • Crystal reports is a reporting tool widely used to connect to databases. As its name infers, it is a “Reporting”tool that insinuates read only activity. Crystal reports was the tool that executed all the stored procedures by clicking the “>>” button as illustrated below in figure 4. Expected results from browsing the data available in the database was not to execute any stored procedures, but to list available database tables. The cause of the data loss could be considered a “Bug” • The potential for the same thing to occur may happen when using other tools such as Microsft Access or Microsoft Excel (subject to further testing). • The data loss was an accident waiting to happen . • Figure 4.The button that triggered the execution of the legacy stored procedures

6. Maximo Data Loss Aug 2011 • Questions?