Where's Waldo Uncovering Hard-to-Find Application Killers

1. Where's Waldo Uncovering Hard-to-Find Application Killers Claire Cates SAS Institute, Inc Claire.Cates@sas.com

3. Always something different • Just like the Waldo pictures, performance is always different and not easy to find. • Performance based on the test that was run • Performance based on the data that was used • Performance based on the hardware used • Performance based on the number of users

4. What is a Performance Problem • When jobs cannot execute within a required time period. • Something that makes the system run slower than it should • When a system cannot sustain expected or unexpected growth (scalability) • More users • More data • More processors

5. Causes of Performance Problems • Hardware • Software • Integration of software systems • Network • Web

6. How Do You Find Performance Problems • Vendor profilers • Looking at system variables (cache hits, IO timings,…) • Profilers that analyze the code • Code reviews • Network sniffers • Profiles that profile the entire machine • Code instrumentation

7. Common Software Performance Problems Antipatterns • Excessive Memory Allocation • The Ramp • Unnecessary Processing • One Lane Bridge • How Many Times Do I have to tell you. • Unbalanced Processing • More is Less

8. Excessive Memory Allocation • Memory Allocations/Frees are expensive • OS calls • If your system is multi-threaded, locking may need to occur • Often times memory is allocated temporarily within a function • Often times structures are allocated and freed and not reused • Memory allocations in loops. • Possible Fixes • Use a stack variable instead of temporary memory allocations • Maintain a free chain for structures that are used often • Make one large allocation outside loop – split it within the loop

9. The Ramp • When processing slows down over time • Can be caused by long lists that grow and need to be searched • Can be caused by too many resources being used • Another kind of Ramp happens when the system doesn’t scale well. • Possible fixes • Use a different algorithm when you know large amounts of data will be required • Make sure all resources (especially memory) are returned.

10. Unnecessary Processing • Loops that contain statements that do not need to execute on each iteration of the loop • Macros that are inefficient because they are too general • Zeroing of allocated memory that does not need to be zeroed. • Inefficient algorithms • Possible fixes • Rework algorithm • Don’t use macros

11. One Lane Bridge • Resource contention – every thread waiting on the same resource • Eventually causes a traffic jam • Possible fixes • Change the algorithm to use less resources that are shared

12. How Many Times Do I Have To Tell You • Calling a routine too often with the same parameters • Sometimes a routine (foo) is the hot spot yet, (foo) is fine, it is just that foo is called to often. • Most common occurrence of this that I see occurs when (afoo) (bfoo) (cfoo) (foo) So cfoo is called twice and foo is called 4 times all because afoo does not know what bfoo and cfoo actually call

13. Unbalanced Processing • Occurs in a multi-threaded environment running on a multi-processor machine. • When one or two threads do all the processing while other threads are idle • Possible fixes • Rebalance the work in the threads so that all threads process a similar amount of work

14. More is Less or Thundering Herd • Adding threads does not make a system run faster • The system spends its time thrashing • Thrashing can occur because of • Excessive thread switching • Destroying the system memory or I/O caches when each thread runs • Possible fixes • Don’t create more threads than there is work to be done

15. Common Profilers in Use at SAS • InTune – z/OS • VTune – Intel – PCs • Caliper – HP • Rational Quantify – Windows, Solaris, HP, Linux • Jprobe • OS tools – not really profilers but gives information on • System calls • IO bottlenecks • Caching problems

16. SAS’s use of Quantify • Other Profilers we use • Event / timer based • Based more on the function level, not the source level • Network traffic & other processes on the machine distorted the results • Great results from all profilers • Tips I’m using with Quantify do apply to these other tools

17. Quantify differences… • Quantify • Counts instruction cycles for the user code • Times the system calls (can be turned off) • Does not have to run on a clean machine (a machine with no other activity) – developer machine • Provides performance information down to the source line level • Performance can be broken down by threads

18. A Little about Quantify • Quantify integrates itself with your system. As your system executes, Quantify will accumulate a mass of data. • Function counts for every function used • The amount of time spent in every function • The amount of time spent in every function plus the amount of time spent in all its descendent routines. • Full Call tree information

19. Quantify Quick Overview

26. Excessive Memory Allocation

36. How Many Times Do I Have to Tell You?

43. More Is Less