Performance Analysis

1. Performance Analysis Chapter 24

2. Understand the basic terminology of performance monitoring and analysis. • Understand proper methods of monitoring a system’s performance. • Knowledge of the tools that allow you to monitor system performance. • Understand how to analyze the data provided by the monitoring tools. • Understand how to apply the data to improve system performance. • Understand what to tune, and why to tune it. Chapter Goals

3. Right-size the system to start with. • You do not want to start with an overtaxed system with the intention of providing a turbo-charged service. UNIX is very demanding on hardware. UNIX generally provides each process with (the illusion of) unlimited resources. This often leads to problems when system resources are taxed. Windows operating systems and applications often understate system requirements. The OS and/or applications will operate in a sparse environment, but the performance is often abysmal. General Performance Tuning Rules

4. Determine the hardware requirements of specific types of servers. • Generally, e-mail and web servers require high-throughput network links, and medium to large memory capacity. Mail servers typically require significantly more disk space than web servers. Database servers typically require large amounts of memory, high capacity, high-speed disk systems, and significant processing elements. Timeshare systems require significant processing elements, and large amounts of memory. General Performance Tuning Rules

5. Monitor critical systems from day one in order to get a baseline of what “normal” job mixes and performance levels are for each system. • Before making changes to a system configuration, make sure user jobs are not causing problems. • Check for rabbit jobs, users running too many jobs, or jobs of an inappropriate size on the system. • A performance problem may be temporary, so you need to think through any changes before you implement them. • You might also want to discuss proposed changes with other system administrators as a sanity check. General Performance Tuning Rules

6. Once you are ready to make changes, take a scientific approach to implementing them. • You want to ensure that the impact of each change is independently measurable. • You also want to make sure you have a goal in mind, at which point you stop tuning and move on to other projects. • Before you begin making changes to the system, consider the following. • Always know exactly what you are trying to achieve. • Measure the current system performance before making any changes. • Make one change at a time. General Performance Tuning Rules

7. Once you do make a change, make sure to monitor the altered system for a long enough period to know how the system performs under various conditions (light load, heavy load, I/O load, swapping). • Do not be afraid to back out of a change if it appears to be causing problems. • When you back a change out, go back to the system configuration immediately previous to the “bad” configuration. Do not try to back out one change and insert another change at the same time. • Take copious notes. • These are often handy when you upgrade the OS, and have to start the entire process over. Change Rules

8. Install as much memory as you can afford. • Disk systems can also have a substantial impact on system performance. • Network adapters are well-known bottlenecks. • Eliminate unused drivers, daemons, and processes on the system. • Know and understand the resources required by the applications you are running. Resource Rules

9. Bandwidth: • The amount of a resource available. If a highway contains four lanes (two in each direction), each car holds four people, and the maximum speed limit allows 6 cars per second to pass over a line across the road, the “bandwidth” of the road is 24 people per second. Increasing the number of lanes will increase the bandwidth. • Throughput: • Percentage of the bandwidth you are actually getting. Continuing with the road example, if the cars only hold one person, the protocol is inefficient (not making use of the available capacity). If traffic is backed up due to an accident and only one or two cars per second can pass the line, the system is congested, and the throughput is impacted. Likewise, if there is a toll booth on the road, the system experiences delays (latency) related to the operation of the toll booth. Terminology

10. Utilization: • How much of the resource was used. It is possible to use 100% of the resource, and yet have 0% throughput (consider a traffic jam at rush hour). • Latency: • How long it takes for something to happen. In the case of the road example, how long does it take to pay the toll? • Response time: • How long the user thinks it takes for something to occur. • Knee: • Point at which throughput starts to drop off as load increases. Terminology

11. Benchmark: • Set of statistics that (hopefully) shows the true bandwidth and/or throughput of a system. • Baseline: • Set of statistics that shows the performance of a system over a long period of time. • Instantaneous data about the system’s performance is rarely useful for tuning the system. But long-term data is not very useful either, as peaks and valleys in the performance graph tend to disappear over time. • You need to know the long-term performance characteristics, as well as the “spikes” caused by short-lived processes. A good way to obtain long-term (and short-term) information is to run the vmstat command every five seconds for a 24-hour period. Collect the data points, reduce/graph these data points, and study the results. Terminology

12. Task Manager • The Cygwin package allows the administrator to build and install several UNIX tools to monitor system performance. • For sites that do not use the Cygwin toolkit, there are several third-party native Windows tools that might be useful when you need to monitor system performance. Among these tools are: • vtunehttp://developer.intel.com/software/ products/vtune/ • SysInternals http://www.sysinternals.com/ Windows Monitoring

13. ps • top • vmstat • iostat • nfsstat • netstat • mpstat • accounting UNIX Monitoring

14. Most versions of UNIX ship with an accounting package that can monitor the system performance, and record information about commands used. • Many sites run the detailed system accounting package in order to bill departments/users for the use of the computing resources they consume. • The accounting packages can also be very useful tools for tracking system performance. • Although the accounting information is generally most useful as a post-mortem tool (after the processes has completed), it is sometimes possible to gather semi real-time information from the system accounting utilities. • System auditing packages can give a lot of information about the use of the system, but these packages also add considerable load to the system. • Process accounting utilities will generally add 5% of overhead to the system load, and auditing utilities can add (up to) 20% of overhead load to the system. Unix Monitoring

15. Why run accounting? • Bill for resources used. • CPU time used • Memory used • Disk space used • Printer page accounting • Detailed job flow accounting (Banks/Insurance/Stock trading) • Keep track of every keystroke • Keep track of every transaction • Security • track every network connection • track every local login • Track every keystroke Accounting

16. Two types of accounting • Process accounting • Track what commands are used • Track what system calls are issued • Track what libraries are used • Good for security (audit trail) • Good when multiple users have access to system • Good way to track what utilities and applications are being used, and who is using them. Accounting

17. Detailed accounting • Track every I/O operation • Disk • Tape • tty • Network • Video • Audio • Primarily used for billing Accounting

18. Charging for computer use • Almost unheard of in academia (today). • Some Universities charge research groups for CPU time. • Some Universities charge for printer supplies. • Some Universities charge for disk space and backups. • Most companies that run accounting have a central computing facility. • Subsidiaries buy computing time from the central group. • Accounting is used to pay for support, supplies, … Accounting

19. Why avoid accounting? • Log files are huge • Must have disk space for them. • 15 minutes of detailed accounting on a system with one user generated a 20 MB log file! • 15 minutes of process accounting on a system with one user generated a 10 MB log file! • Must have (and bill) cpu time for accounting. • Accounting can require a lot of CPU/disk resources • Who will pay for the CPU/disk resources used by accounting • Must decide what information to keep, and what to pitch. Accounting

20. What can accounting track? • Some of the common things to track: • CPU time • Memory usage • Disk usage • I/O usage • Connect time • Dial-up/Dial-out usage • Printer accounting Accounting

21. Solaris • Auditing • Perform audit trail accounting • Relies on the Basic Security Module (BSM). • Can monitor TONS of stuff. • Processes • Function/subroutine calls • System calls • Ioctls • Libraries loaded • File operations (open close read write create remove) • File system operations (stat, chmod chown, …) • Can configure to monitor successful/unsuccessful operations • Can monitor on a per user basis Accounting

22. Solaris • Audit binaries • auditconfig • auditd – the audit daemon • praudit – print audit information • auditon – turn on auditing Accounting

23. Solaris • Audit files • Control Files in /etc/security • audit_class • audit_control • audit_data • audit_event • audit_startup • audit_user • audit_warn • device_allocate • device_maps Accounting

24. Solaris • Audit Files • Data Files in /var/audit • YYYYMMDDHHMMSS.YYYYMMDDHHMMSS.hostname • YYYYMMDDHHMMSS.not_terminated.hostname Accounting

25. Solaris • Accounting • Daily Accounting • Connect Accounting • Process Accounting • Disk Accounting • Calculating User Fees Accounting

26. Solaris • Accounting • /usr/lib/acct/acctdisk • /usr/lib/acct/acctdusg • /usr/lib/acct/accton • /usr/lib/acct/acctwtmp • /usr/lib/acct/closewtmp • /usr/lib/acct/utmp2wtmp Accounting

27. Solaris • Accounting binaries • acctcom – search/print accounting files • acctcms – generate command accounting from logs • acctcon – turn accounting on/off • acctmerg – merge multiple account forms into a report • Acctprc – programs to generate process accounting logs • fwtmp – manipulate connect accounting records • runacct – run daily accounting summary Accounting

28. Solaris • Accounting • Data Files • /var/adm/pacct • /var/adm/acct/fiscal • /var/adm/acct/nite • /var/adm/acct/sum Accounting

29. Unix User Interface researchers report that an average user perceives a system to be slow when response times are longer than 0.7 seconds! Performance Analysis

30. CPU time – • How long does the user’s job take to complete? • Is the job time critical? • What other jobs are running? • Context switches are costly. • Must share cpu cycles with other processes • What is the system load average? • Memory speed – • Does the job need to be loaded into memory? • How quickly can memory be filled with pertinent information? • Is the job swapped out? • Swapping brings disk system into picture. • Swapping invalidates cache for this job. • Swapping is easy to eliminate/minimize! • Does the job fit into cache? Performance Analysis

31. Disk I/O bandwidth – • Bus Speed • Controller width/speed • How fast can information be pulled off of disk? • SCSI vs IDE vs RAID • Rotational latency • Caching in controller/drive • Disk system speed will have an effect on memory speed (swapping). • Network I/O bandwidth – • Are files stored on a network file system? • Does network file system do any caching? • Shared/switched media? • Full/half duplex? Performance Analysis

32. CPU bound jobs are difficult to measure. • Use ps and top to see what is running. • Use uptime to determine load averages • 1 minute average is good for “spiky” load problems • 5 minute average is good metric to monitor for “normal” activity • 15 minute average is good indicator of overload conditions • Use sar to determine the system cpu states. • System accounting can track amount of time each CPU spends working on idle/system/user jobs. • Use mpstat to determine what multi-processor systems are doing. • One busy processor and one idle processor is probably “normal” operation. • Use vmstat and iostat to determine percentage of time system is running user/kernel processes. • Less detail than sar, but good general information. Performance Analysis

33. How can you improve CPU performance? • More cpu(s) • Faster cpu(s) • Lock jobs to specific cpu(s) • Lock cpu(s) to specific tasks Performance Analysis

34. Before you can diagnose performance problems, you must have a good idea of what is reasonable for your system. • Monitor system and develop a fingerprint of typical job mixes, load average, memory use, disk use, network throughput, number of users, swapping, job size. • If something happens to the performance use these metrics to determine what has changed. • Did jobs get larger? • More disk or network I/O? • Less free memory? • More swapping? • More users? • More jobs? Performance Analysis

35. In general, the output of the top, vmstat, w, and other utilities that show processor-state statistics can tell you a lot about the performance of the CPU subsystem. • If the CPU is in user mode more than 90% of the time, with little or no idle time, it is executing application code. • This is probably what you want it to do, but too many user jobs running concurrently may be detrimental to any one job getting any work done. • If the CPU is in system mode more than 30% of the time, it is executing system code (probably I/O, or other system calls). • Context switches are a symptom of high I/O activity (if the interrupt rate is also high). • If seen in conjunction with high system call activity, it is a sign of poor code (nonlocal data, open, read, close, or loop). • If the CPU is idle more than 10% of the time, the system is waiting on I/O (disk/network). • This could be a symptom of poor application code (no internal buffering) or overloaded disk/network subsystems. CPU Performance

36. If the system exhibits a high rate of context switches, the system is displaying symptoms of a number of possible problems. • Context switches occur when one job yields the processor to another job. • This may occur because the scheduler time slice expired for the running job, because the running job required input/output or because a system interrupt occurred. • If the number of context switches is high, and the interrupt rate is high, the system is probably performing I/O. • If the number of context switches is high, and the system call rate is high, the problem is likely the result of bad application coding practices. • Such practices include a program loop that repeatedly performs the sequence “open a file, read from the file, close the file.” CPU Performance

37. If the system exhibits a high trap rate, and few system calls, the system is probably experiencing page faults, experiencing memory errors, or attempting to execute unimplemented instructions. • Some chips do not contain instructions to perform certain mathematical operations. • Such systems require that the system generate a trap that causes the system to use software routines to perform the operation. • An example of this situation occurs when you attempt to run a SPARC V8 binary on a SPARC V7 system. • The SPARC V7 system contains no integer multiply/divide hardware. SPARC V8 systems contain hardware multiply/divide instructions, so compiling a program on the V8 architecture imbeds these instructions in the program. • When this same program is run on a V7 system, the OS has to trap the instructions, call a software routine to perform the calculation, and then return to the running program with the answer. CPU Performance

38. Memory is a critical system resource. • Unix is very good at finding/hoarding memory for disk/network buffers. • Unix buffering scheme • At boot time, size memory. Kernel takes all memory and hoards it As jobs start, kernel begrudgingly gives some memory back to them. In some versions of UNIX: Disk buffers are allocated on file system (disk partition) basis Network buffers are allocated on a per-interface basis. Performance Analysis

39. Memory is a critical system resource. • Before upgrading the cluster systems OIT looked at the memory question: • With 64 Meg jobs took X minutes to run. • With 128 Meg of memory, the same jobs took X/3 minutes to run. • With 256 Meg of memory, the same job did not run any faster, but you could run multiple instances of same job with no degradation in performance. • Memory is cheap. Buy lots! Performance Analysis

40. Monitoring memory use. • Use pstat -s to look at swap information on BSD systems. • Use swap -l to look at swap on System V systems. • Use sar -r to look at swap information • Use vmstat to look at memory statistics. • Use top to monitor job sizes and swap information. • If there is any sign of swapping • Memory is cheap! Buy Lots! • Can adjust reclaim rate, and other memory system parameters, but it is usually more profitable to add memory. Performance Analysis

41. Unlike CPU tuning, memory tuning is a bit more objective. Quantifying CPU performance can be somewhat elusive, but quantifying memory usage is usually pretty straightforward. • Job Size • An easy diagnostic for memory problems is to add up the size of all jobs running on the system, and compare this to the size of the system’s physical memory. • If the size of the jobs is grossly out of proportion to the size of the system memory, you need to do something to change this situation. • You could use a scheduler that uses job size as one of the criteria for allowing a job to run, remove some processes from the system (for example migrate some applications to another server), or add memory to lessen the disparity in the requested versus available memory. Memory Performance

42. Swapping/Paging • Under BSD operating systems, the amount of virtual memory is equal to the swap space allocated on the system disks plus the size of the shared text segments in memory. • The BSD VM system required that you allocate swap space equal to or greater than the size of memory. Many BSD environments recommended that you allocate swap space equal to 4x the size of real memory. • Under System V UNIX kernels, the total amount of virtual memory is equal to the size of the swap space plus the size of memory, minus a small amount of “overhead” space. • The system does not begin to swap until the job memory requirements exceed the size of the system memory. Memory Performance

43. You can estimate the system’s virtual memory requirements on BSD systems by looking at the output of the top and/or ps commands. • If you add up the values in the RSS columns (resident set size), you can get an idea of the real memory usage on the system. • Adding up the values in the SZ column gives you an estimation of the VM requirements for the system. • If the total of all SZ values increases over time (with the same jobs running), one or more applications probably have memory leaks. • The system will eventually run out of swap space, and hang or crash. • Some kernels allow you to modify the page scan/reclaim process. • This allows you to alter how long a page stays in real memory before it is swapped or paged out. • Such modifications are tricky, and should only be performed if you know what you are doing. Memory Performance

44. If you see that the scan rate (sr column in vmstat output) value is roughly equal to the free rate (fr column in vmstat output), the system is releasing pages as quickly as they are scanned. • If you tune the memory scan parameters to increase the period between when the page is scanned and when it is paged out (allow pages to stay in memory for a longer period), the VM system performance may improve. • On the other hand, if the sr value is greater than the fr value, decreasing the period between scan and paging time may improve VM system performance. Memory Performance

45. VM Symptoms • The following indicators may be useful when tuning the VM system. • Paging activity may be an indicator of file system activity. • Swapping activity is usually an indicator of large memory processes thrashing. • Attaches and reclaim activity is often a symptom of a program in a loop performing a “file open, read, and close” operation. • If the output of netstat –s shows a high error rate, the system may be kernel memory starved. This often leads to dropped packets, and memory allocation (malloc) failures. Memory Performance

46. Shared Memory • Large database applications often want to use shared memory for communications among the many modules that make up the database package. • By sharing the memory, the application can avoid copying chunks of data from one routine to another, therefore improving system performance and maximizing the utilization of system resources. • This generally works fine, until the application requests more shared memory than the system has available. • When this situation occurs, system performance will often nosedive. • Under Solaris, the /usr/bin/ipcs command may be used to monitor the status of the shared memory, and semaphore system. Memory Performance

47. mmap • If an application is running from a local file system, you might want to look into using the mmap utility to map open files into the system address space. • The use of mmap replaces the open, malloc, and read cycles with much more efficient operation for read-only data. • When the application is using network file systems, this might actually cause a degradation of system performance. • Using the cachefs file system with NFS will improve this situation, as this allows the system to page to a local disk instead of through the network to an NFS disk. Memory Performance

48. How can you improve the memory system? • Add memory • It’s cheap • Use limits • they’re ugly • payoff is not (usually) very good. Performance Analysis

49. Disk I/O is one of the most critical factors in system performance. • Most file access goes through the disk I/O system. • Multiple “hot” file systems on one disk will be a problem. • Slow disks will be a problem • Narrow controllers will be a problem • Partitioning of disks will have an effect on buffering • Disk geometry of disk will have an effect on buffering • Swapping/paging goes through the disk I/O system. • Split swap space over multiple spindles to increase interleave • If swapping: Buy More Memory (It’s cheap) • Use iostat to look at the disk I/O system. Performance Analysis

50. Swapping • In general, if a system is swapping this is a symptom that it does not have enough physical memory. • Add memory to the system to minimize the swapping/paging activity before continuing. • You might also consider migrating some of the load to other systems in order to minimize contention for existing resources. • If the system contains the maximum memory, and the system is still swapping, there are some things you can do to improve the performance of the swapping/paging subsystem. • First, try to split the swap partitions across several disk drives and (if possible) disk controllers. • Most current operating systems can interleave swap writes across several disks to improve performance. • Adding controllers to the swap system can increase the bandwidth of the swap subsystem immensely. Disk Performance