Operating System Performance Optimization

1. Operating System Performanceprepared byJason MeyerPresented byGroup 4

2. Agenda • Mindcraft Benchmarks • First Benchmark • Open Benchmark • Linux Optimizations • General Tips • Hardware Tuning • Network Tuning • File Serving • Web Serving • Mail Serving • References

3. Operating System Performance • Mindcraft’s first Benchmark • April 13, 1999 • Performed a File and Web Server comparison between Windows NT and RedHat Linux 5.2 (kernel 2.2.2) • Used Dell PowerEdge 6300m with 4x400MHz Pentium II Xeon processors, 4 GB RAM, and a RAID disk.

4. Operating System Performance • Mindcraft’s first Benchmark • Linux: • Samba 2.0.3 as the SMB file server • Apache 1.3.4 as the Web server. • Windows NT: • embedded SMB file server • Internet Information Server 4.0 Web server.

5. Operating System Performance • Mindcraft’s first Benchmark • File server performance tested with Ziff-Davis Benchmark Operation NetBench 5.01 • Measures throughput in Bytes/second • A number of test systems read and write files on a server • NetBench test suite made up of a number of mixes. • A mix is a particular configuration of NetBench parameters. • Each mix increases the number of test systems • Mindcraft used 10 mixes, ranging from 1 to 144 test systems (clients).

6. Operating System Performance • Mindcraft’s first Benchmark • Web server performance tested with Ziff-Davis Benchmark Operation WebBench 2.0 • A number of test systems request URLs. • Measures 2 things: • The number of HTTP GET requests per second. • The number of bytes per second that a Web server sends to all test systems

7. Operating System Performance • Mindcraft’s first Benchmark Results: • Windows NT Server 4.0 was 2.5 times faster than Linux as a File Server and 3.7 times faster as a Web Server

8. Operating System Performance • Outcry from Linux Community • Mindcraft did not reveal they were funded by Microsoft, worked in a Microsoft lab, and had help from Microsoft to tune NT • Mindcraft claimed to have attempted to tune Linux but couldn’t get any help • Many people in the Linux community believed they purposefully tried not to tune Linux, or at the least made no effort to tune Linux.

9. Operating System Performance • Mindcraft’s Response • Re-ran the benchmark (but never released results). • Mindcraft offered to do another “Open Benchmark”. • Leaders from the Linux community were invited to offer tips for improving performance and were allowed to be present for the actual test. • Took place June 30, 1999

10. Operating System Performance • Open Benchmark results: • Overall, results were similar to the first benchmark. • Windows NT was about 2 times faster as a File Server and a Web Server, depending on the setup used.

11. Operating System Performance Open Benchmark results:

12. Operating System Performance • Linux Community response: • Some downplayed these results for various reasons • Claimed benchmarks not reflective of real world • Claimed Microsoft had unfair advantage in tuning performance. • At the least, this showed that Linux was not miles ahead of NT or at least had room for improvement. • The following website was developed to aid in improving performance on Linux: http://linuxperf.nl.linux.org/

13. Operating System Performance • General Tips: • The following tips can be relevant no matter what the primary purpose of a machine is. • If the system seems to be running slow, the first step is to track down why it is slow, and then find a fix for the problem. • Even if no problems are noticeable, the following tips could help improve performance.

14. Operating System Performance • Common Problems that hinder performance: • Swapping: Problem if process needs to be continuously mapped back into main memory. • Can check with “top” and seeing the process ‘kswapd’ takes a large percentage of the CPU for several seconds. • Can also check with vmstat and iostat.

15. Operating System Performance • Common Problems that hinder performance: • Example of top where swapping is not a problem: 2:22pm up 21 days, 5:39, 2 users, load average: 0.34, 0.08, 0.02 54 processes: 53 sleeping, 1 running, 0 zombie, 0 stopped CPU0 states: 0.0% user, 6.1% system, 0.0% nice, 93.3% idle CPU1 states: 6.2% user, 2.1% system, 0.0% nice, 91.0% idle Mem: 1545264K av, 1188708K used, 356556K free, 0K shrd, 28944K buff Swap: 1542232K av, 0K used, 1542232K free 967872K cached PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME COMMAND 4997 fcsl 17 0 31256 30M 9896 D 8.7 2.0 0:04 otbsaf 764 root 9 0 0 0 0 SW 0.3 0.0 0:00 rpciod 4933 root 9 0 644 644 540 S 0.1 0.0 0:00 in.telnetd 4962 fcsl 10 0 1064 1064 856 R 0.1 0.0 0:00 top 1 root 8 0 520 520 452 S 0.0 0.0 0:15 init 2 root 8 0 0 0 0 SW 0.0 0.0 0:00 keventd 3 root 19 19 0 0 0 SWN 0.0 0.0 0:00 ksoftirqd_CPU0 4 root 18 19 0 0 0 SWN 0.0 0.0 0:00 ksoftirqd_CPU1 5 root 9 0 0 0 0 SW 0.0 0.0 0:00 kswapd

16. Operating System Performance • Common Problems that hinder performance: • Example of vmstat where swapping is not a problem [fcsl@seneca ~]$ vmstat procs memory swap io system cpu r b w swpd free buff cache si so bi bo in cs us sy id 0 0 0 0 309944 28944 984756 0 0 0 0 6 2 0 0 6 where swap: si: Amount of memory swapped in from disk (kB/s). so: Amount of memory swapped to disk (kB/s).

17. Operating System Performance • Common Problems cont’d.: • Accessing the disk too hard • Can check by running `ps axlw` and the STAT field will contain a “D” when the process is waiting on the disk. • Not much to help, perhaps use a faster disk or RAID.

18. Operating System Performance • Common Problems cont’d.: • Example of ps alwx without anything waiting on disk access: [fcsl@ute OTBSAF]$ ps alwx F UID PID PPID PRI NI VSZ RSS WCHAN STAT TTY TIME COMMAND 100 0 1 0 8 0 1412 468 do_sel S ? 0:18 init [5] 040 0 2 1 9 0 0 0 contex SW ? 0:00 [keventd] 040 0 3 0 19 19 0 0 ksofti SWN ? 0:09 [ksoftirqd_CPU0] 040 0 4 0 19 19 0 0 ksofti SWN ? 0:01 [ksoftirqd_CPU1] 040 0 5 0 9 0 0 0 kswapd SW ? 1:21 [kswapd] 040 0 6 0 9 0 0 0 bdflus SW ? 0:00 [bdflush] 040 0 7 0 9 0 0 0 kupdat SW ? 0:10 [kupdated] 040 0 10 1 9 0 0 0 down_i SW ? 0:00 [scsi_eh_0] 040 0 11 1 9 0 0 0 down_i SW ? 0:00 [scsi_eh_1] . . . NOTE, for the STAT field, S = sleeping W = has no resident pages N = low-priority task

19. Operating System Performance • Common Problems cont’d.: • All time spent in User processes • A user-level program chews up all the CPU • Use top to find the process that is the culprit • Optimize this process or move it to another machine.

20. Operating System Performance • Common Problems cont’d.: • Example of top with the process “guimain” chewing up all the CPU 2:57pm up 21 days, 6:12, 11 users, load average: 0.15, 0.03, 0.02 92 processes: 85 sleeping, 7 running, 0 zombie, 0 stopped CPU0 states: 0.1% user, 0.2% system, 0.0% nice, 99.7% idle CPU1 states: 99.9% user, 0.1% system, 0.0% nice, 0.0% idle Mem: 1545264K av, 1491272K used, 53992K free, 0K shrd, 23536K buff Swap: 1542232K av, 26592K used, 1515640K free 1222088K cached PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME COMMAND 32021 fcsl 17 020292 18M 3932 R 99.9 1.2 0:10 guimain 6997 fcsl 9 0 3072 3068 1928 S 0.1 0.1 0:00 xterm . . .

21. Operating System Performance • Common Problems cont’d.: • All time spent in Kernel processes • If the machine spends a large amount of time in system mode, then a process is probably wasting system calls. • Use `ps axlw` to see what function call a process is running at any given time • May be able to reduce the impact of the system calls by adding memory or upgrading the CPU.

22. Operating System Performance • Common Problems cont’d.: • Network Problems • Improve Service Speed (time to respond to a query) • Fewer concurrent connections, which reduces the server load. • DNS Lookup Time • Trying to find the hostname associated with an IP address can be slow. • Process is usually idle while this lookup occurs. • As more requests arrive, more process will be waiting (sleeping) • Can check with `ps axlw` for processes with an “S” or with `strace –p`.

23. Operating System Performance • Common Problems cont’d.: • Kernel Problems • System may spend all of its time in system mode for no reason. • Can try to use kernel profiling to find the slowest kernel function. • Change lilo.conf so “profile=2” is passed to the kernel so that the kernel will keep statistics.

24. Operating System Performance • Common Problems cont’d.: For example:    image=/boot/2.2.7-sard             label=linux             append="profile=2“ • Make sure/etc/System.map matches your kernel, run lilo, and reboot. • Use `readprofile` to view statistics on each of the kernel's functions to find the slowest.

25. Operating System Performance • Hardware Tuning • Use hdparm to optimize IDE hard drives. • Software raid may improve performance. • Tune the disk I/O elevators • Useful if system continuously does a large amount of disk I/O • Changes how long the I/O scheduler will let a request sit in the queue before it has to be handled. • I/O scheduler collapses some requests together, which can increase throughput.

26. Operating System Performance • Network Tuning • Try to use 100BTX instead on 10BTX • If building a router, check the "Optimize as router" box in the kernel configuration.  • ip_frag_low_thresh and ip_frag_high_thresh should be increased in some situations, especially NFS and Samba servers.  • Server takes far more connections and puts them together in memory before having to ditch their fragments.  • Can safely increase these to several megabytes without any problems.

27. Operating System Performance • Tuning for particular applications: • The following slides provide more detailed tips for particular applications • File Server • Web Server • Mail Server

28. Operating System Performance • Tuning a File Server: • Increase default packet sizes used by NFS clients • With larger packet sizes, fewer requests needed to transfer the same amount of data. For example: mount -o rsize=8192,wsize=8192 -t nfs host:/dir /mnt/point Or on a Samba server in smb.conf : socket options = SO_SNDBUF=4096 SO_RCVBUF=4096

29. Operating System Performance • Tuning a File Server: • Use larger block sizes • Helpful for file systems dedicated to fairly large files • Increase the block size from 1024 bytes to 4096 bytes. • Leads to less fragmentation, faster fsck’s, faster deletes, and faster raw read speed, due to the reduced number of seeks

30. Operating System Performance • Tuning a File Server: • Unfortunately, must re-format the drive: mke2fs -b 4096 -m 1 /dev/whatever where the –m 1 adds an additional improvement by only reserving 1% of the drive for root instead of 5%. • Must be careful, though, if there are many small files – this can lead to much wasted disk space.

31. Operating System Performance • Tuning a File Server: • Don’t use the atime attribute • This records the last access time (separate from the last modified time) and is not very useful. • This leads to significant improvements when running `find` as well as accessing frequently changing files like the contents of /var/spool/news

32. Operating System Performance • Tuning a File Server: • To turn off the atime attribute: chattr -R +A /var/spool (for a whole directory tree) Or mount an entire partition with no access time updating, by editing /etc/fstab: # device mount point type options dump frequency fsck pass number /dev/sdb1 /var/spool/news ext2 defaults,noatime 1 2

33. Operating System Performance • Tuning a Web Server: • Apache is a web server used by Linux. • MaxRequestsPerChild should be set high (10000). • This sets the number of requests a server child process will handle before it dies. • If this value is too low, apache will be constantly restarting httpds. • Increase the number of available httpd processes • Default is 256, which can be exceeded • add -DHARD_SERVER_LIMIT=4000 to CFLAGS in Configuration • Or modify the apache source code • This may require modifying the kernel to support more processes.

34. Operating System Performance • Tuning a Web Server: • Use Efficient Server-side scripting • If you do a lot of CGI in perl, consider using mod_perl. • The Apache::Registry module can in most cases speed up your CGI requests with no actual changes to your code. • Similar apache modules for most popular scripting languages (python, Ruby, Php, tcl). • Work around the process model • Scalability problem with apache: one process per connection. • This can require a large amount of resources: RAM, schedular slots, ability to grab locks, database connections, file descriptors.

35. Operating System Performance • Tuning a Web Server: • Two methods to help: • Use a dedicated server for static files, if there are lots of static files. • Can use a very light apache setup, or something like thttpd, boa, khttpd, or TUX. • Use a reverse-proxy • In a reverse proxy, clients make requests for content in the name-space of the reverse proxy. The reverse proxy then decides where to send those requests and returns the content as if it was itself the origin. • Advantages include content caching, load balancing, and the prospect of moving slow connections to lighter weight servers.   

36. Operating System Performance • Email Server Tuning: • Mail servers comprised of two parts: • Mail transfer agent, MTA (also called an SMTP server) • Mail retrieval agent, MRA • Limiting factor seems to be disk I/O • MTAs queue messages to a queue directory (sendmail uses /var/spool/mqueue) • MRAs continually access the mail store (sendmail uses /var/spool/mail) • Put these directories on different filesystems that are on different disks to minimize the MTA and MRA from fighting for bandwidth for the same set of disks.

37. Operating System Performance • Email Server Tuning • MTA Tuning: • Don’t allow mail server to be used as an open relay • Only allow outgoing messages if from correct LAN/WAN • Otherwise, spammers will use the ail server to deliver their messages. • MRU Tuning: • IMAP vs. POP3 • POP3 is older and faster • IMAP is newer and more complex (more features), so slower. • 3 ways to store mail: single file, directory hierarchy, and database • Performance depends on how the particular system is used.

38. Operating System Performance • Repeat of History • In April 2003, Microsoft had VeriTest run another benchmark, comparing Windows Server 2003 and Linux • Results claimed Windows 2003 was about twice as fast as a File Server. • Linux community again up in arms about validity of results • Again, much support for Microsoft to fine tune Windows Server 2003, with little to no support to tune Linux.

39. Operating System Performance • Repeat of History Results of Veritest benchmarks:

40. Operating System Performance • Caveat • Most of these performace improvements taken from http://linuxperf.nl.linux.org • There are no dates on this website, so cannot tell how recent or up-to-date these recommendations are (though it says it was created as a direct result of the Mindcraft benchmarks in 1999). • In newer versions of Linux, some of these recommendations may have already been implemented by default.

41. Operating System Performance • Conclusions of these benchmarks • Highly tuned Microsoft Windows products (NT, Windows Server 2003) may or may not be faster than Linux essentially straight out of the box. • Microsoft products only measured faster in benchmarks that don’t have much applicability to the real world (though there aren’t many better alternatives for benchmarking). • Need to be aware of why Operating Systems perform well or poorly on benchmarks, to be able to find ways to improve their performance.

42. References • http://www.mindcraft.com/whitepapers/nts4rhlinux.html (links to results from original benchmark and the open benchmark). • http://www.kegel.com/mindcraft_redux.html (lists several issues responsible for the poor Linux performance in the benchmarks). • http://linuxperf.nl.linux.org/ (performance tips for Linux) • “Windows 2000 Performance Tuning”, available at http://www.microsoft.com/windows2000/server/evaluation/performance/reports/perftune.asp

43. References • “Microsoft Windows Server 2003 vs. Linux Competitive File Server Performance Comparison,” available at: http://www.veritest.com/clients/reports/microsoft/default.asp • http://www.linuxworld.com/story/32673.htm(response to above comparison). • http://perl.apache.org/ (information on mod_perl) • http://people.redhat.com/alikins/system_tuning.html (system tuning Linux)