Exhausting Your Means or Leaving No Stone Unturned

1. Exhausting Your Means or Leaving No Stone Unturned • Doug Tapscott • Systems Analyst VI • Siemens Medical Solutions USA, Inc. • Malvern, PA

2. Exhausting Your Means, Leaving No Stone Unturned • Agenda • What’s that mean? • Rules of Thumb • Matrix • Examples

3. Exhausting Your Means, Leaving No Stone Unturned • What’s that mean? • Exhausting your means equals leaving no turn unstoned. • It means looking in all the areas, not just the performance counters. • It means identifying the Key Performance Indicators (KPI’s). • It means understanding the applications performance profile/behavior with respect to the hosting hardware and O/S (performance trending). • It means understanding the application architecture/environment (what’s running where and why).

4. Exhausting Your Means, Leaving No Stone Unturned • Rules of Thumb • Charts have more impact than raw numbers, and raw numbers have more impact than opinion. • Let the charts and/or the data tell the story. • Work in specific time periods where possible. • Know the hardware and the O/S. • Know the architecture/environment. • Know the Key Performance Indicators (KPI’s) and the performance profile. • Know what changed most recently. • Know the difference between “Alert Monitoring” and “Performance Trending”. • Understand the tools at your disposal. • Be willing to learn.

5. Exhausting Your Means, Leaving No Stone Unturned • Matrix We’ve all seen the server trouble shooting and performance management flow chart. With “Yes/No” decision points and “Action” items, it addresses the major areas of: • CPU • Memory • Disk • I/O • Network The focus is trouble shooting. It is difficult to see beyond this very simple decision ladder to the way the Operating system and the application are behaving.

6. Exhausting Your Means, Leaving No Stone Unturned • Matrix The hardware is the bottom layer. The next is the Operating System. The top is the Application. Hardware Operating System Hardware Application Operating System Hardware

7. Exhausting Your Means, Leaving No Stone Unturned • Matrix • Combine the ladder with the layered view to get the matrix. • Each cell of the matrix may be addressed with: • Performance counters • Various 3rd party tools (network, SAN Storage, O/S, scripts, etc) • Various native O/S tools

8. Exhausting Your Means, Leaving No Stone Unturned • Matrix

9. Exhausting Your Means, Leaving No Stone Unturned • Matrix

10. Exhausting Your Means, Leaving No Stone Unturned • Examples – Scenario 1 An application server needed updates to handle added application features. This included application updates, O/S patches, firmware updates and memory (2GB to 4GB). The server appeared to come up okay. All services started as expected. No alerts were generated. No issues were recorded. When users began to access the application, a response time issue was immediately noted. The first proposal was to back off the patches and the application updates and reboot. The second proposal was that the SAN was at fault. The third proposal was that the network was to blame. Items noted initially: Increased Paging Extended Disk Read Times

11. Exhausting Your Means, Leaving No Stone Unturned Examples – Scenario 1

12. Exhausting Your Means, Leaving No Stone Unturned Examples – Scenario 1

13. Exhausting Your Means, Leaving No Stone Unturned Examples – Scenario 1

14. Exhausting Your Means, Leaving No Stone Unturned Examples – Scenario 1 So far we don’t have the root cause.

15. Exhausting Your Means, Leaving No Stone Unturned Examples – Scenario 1 What about Memory? Tremendous decrease in Available Memory.

16. Exhausting Your Means, Leaving No Stone Unturned Examples – Scenario 1 Did the memory needs increase? No, Committed Bytes haven’t changed.

17. Exhausting Your Means, Leaving No Stone Unturned Examples – Scenario 1 Was something suddenly using more memory? No, in fact, every thing is suffering.

18. Exhausting Your Means, Leaving No Stone Unturned Examples – Scenario 1 Could the application changes have caused this? No, the key app processes are suffering.

19. Exhausting Your Means, Leaving No Stone Unturned Examples – Scenario 1 What about the paging? Notice the Hard Page Faults that started with the reboot.

20. Exhausting Your Means, Leaving No Stone Unturned Examples – Scenario 1 What type of paging was happening? Heavy read activity was pushing Paging.

21. Exhausting Your Means, Leaving No Stone Unturned Examples – Scenario 1 This makes things more interesting.

22. Exhausting Your Means, Leaving No Stone Unturned • Examples – Scenario 1 We looked at the server. The original 2GB RAM was upgraded to 4GB of RAM at the outage. What was supposed to be a 4GB RAM server was now a 512MB RAM server. Fully 7/8 of the RAM chips in the server had failed at the replacement and power up reboot. Having lost 83% of RAM, the server was still able to start all the processes and run. When the load increased, the committed bytes need was fulfilled with paging space and continual reads from the SAN disk where the application was installed. The server was memory thrashing. Taking the server down and replacing the memory chips returned the server to it’s targeted 4GB RAM status. This also restored the response time for the end users of the application, quieting the paging and the disk access. How’d the application performance look after the resolution?

23. Exhausting Your Means, Leaving No Stone Unturned Examples – Scenario 1 Overall, with 4GB RAM, Memory usage appeared better.

24. Exhausting Your Means, Leaving No Stone Unturned Examples – Scenario 1 Committed Bytes didn’t change throughout the incident. So the application updates had no negative affect on the way memory was used.

25. Exhausting Your Means, Leaving No Stone Unturned Examples – Scenario 1 Page file activity was lower with 4GB RAM than it had been with 2GB RAM. This was a predicted outcome of the upgrade to 4GB.

26. Exhausting Your Means, Leaving No Stone Unturned Examples – Scenario 1 Memory Page Faults were reduced with 4GB RAM vs the previous 2GB RAM.

27. Exhausting Your Means, Leaving No Stone Unturned Examples – Scenario 1 An interesting effect (increase) was seen regarding System code Resident Bytes with 4GB RAM.

28. Exhausting Your Means, Leaving No Stone Unturned Examples – Scenario 1 A very interesting effect (increase) was seen regarding System Driver Resident Bytes with 4GB RAM. The server doesn’t appear to be paging out/in Driver Resident bytes.

29. Exhausting Your Means, Leaving No Stone Unturned Examples – Scenario 1 The SAN drive M: performance returned to it’s previous level.

30. Exhausting Your Means, Leaving No Stone Unturned Examples – Scenario 1 There was a slight improvement of SAN drive M: Read Time performance with 4GB RAM. This was accompanied by reduced Avg Disk Queue Length.

31. Exhausting Your Means, Leaving No Stone Unturned Examples – Scenario 1 There was a decrease in the size of the process working set overall and for the 2 key processes, “CVi” and “cfi”. This was as expected with the application update.

32. Exhausting Your Means, Leaving No Stone Unturned Examples – Scenario 1 There was also a decrease in the process page file bytes overall and for the 2 key processes, “CVi” and “cfi”. This was as expected with the application update.

33. Exhausting Your Means, Leaving No Stone Unturned Examples – Scenario 1

34. Exhausting Your Means, Leaving No Stone Unturned • Examples – Scenario 2 A customer has an ERP application that is having troubles with the heavy update schedule. It’s taking longer than 24 hours to apply updates to the application. When they couple this with their O/S updates and firmware updates, they are losing almost 2 days of time at every monthly update. If they put the updates off for a month, they lose almost 3 days of time. This is an *nix server as a single image (no lpar). They were running a separate *nix based database server. The application is deployed as a 2 tier architecture. The app server had 8GB of RAM, 4 internal disks (mirrored), 4 cpu’s and SAN attached storage for the application. All O/S mount points were on the internal disks.

35. Exhausting Your Means, Leaving No Stone Unturned • Examples – Scenario 2 The server data looked like this:

36. Exhausting Your Means, Leaving No Stone Unturned Examples – Scenario 2

37. Exhausting Your Means, Leaving No Stone Unturned • Examples – Scenario 2 We mapped the /tmp space to hdisk1 & hdisk2. We reviewed the ERP application to confirm its default use of /tmp space during the update process. We used an O/S environment variable to point the application temp space usage to a file structure on the SAN.

38. Exhausting Your Means, Leaving No Stone Unturned Examples – Scenario 2

39. Exhausting Your Means, Leaving No Stone Unturned Examples – Scenario 2

40. Exhausting Your Means, Leaving No Stone Unturned Examples – Scenario 2

41. Exhausting Your Means, Leaving No Stone Unturned Examples – Scenario 2 Most important: Customer saw a 74% reduction in the elapsed update time, from 2 day’s to 12 hours (fits within a single day window). The End…of this cycle. Thank You.