An Empirical Study of the Reliability in UNIX Utilities

1. An Empirical Study of the Reliability in UNIX Utilities Barton Miller Lars Fredriksen Brysn So Presented by Liping Cai

2. Overview • Introduction • Tools • Tests • Results and Analysis • Conclusions

3. Introduction • Why the study is important • Contributes to the testing community a large list of real bugs; • Finds the bugs that might indicate future security holes; • Finds that crashes might be caused by carelessly typing; • Would like some meaningful and predictable response when inadvertently feeding programs noisy input; • Major utilities should not crash because of noisy phone lines; • Would like to see the interactions between random testing and more traditional industrial software testing;

4. Introduction • The project proceeded in four steps: • Construct programs to generate random characters and to help test interactive utilities; • Use these programs to test a large number of utilities; • Identify the strings that crash the programs; • Identify the cause of the program crashes and categorize the common mistakes;

5. The Tools • Fuzz: generating random Input Strings fuzz 100000 –o outfile | deqn • -- capable of producing both printable and control characters, only printable characters, or either of these groups along with the NULL character. • -- can specify a delay between each character. • -- allows you to specify the seed for the random number generators, to provide for repeatable test. • -- can record its output stream in a file, in addition to printing to its standard output. • -- there are options to randomly insert newline characters in the output stream, and to limit the length of the output stream.

6. The Tools • Ptyjig: Testing Interactive Utilities -- allocates a pseudo-terminal files -- starts the specified utility program. -- ptyjig passes characters that are sent to its input through the pseudo-terminal to be read by the utility -- example: fuzz 100000 –o outfile | ptyjig vi

7. The Tools • The scripts: automating the tests • Executes all the utilities for a given set of input characteristics • Checks for the existence of a “core” file after each utility terminates; indicating the crash of that utility • The core file and offending input data file are saved for later analysis

8. The tests • 7 versions of UNIX • 88 utility programs such as the mail program, screen editors, compilers and document formatting packages • Each utility program was subjected to several different types of input streams. • The major variations in test data were including control characters, NULL, and maximum length of the input stream. • The input streams for interactive utilities have slightly different characteristics. • To avoid overflowing the input buffers on the terminal device, the input was split into random length lines with a mean length of 128 characters.

10. The results and analysis • Were able to crash or hang a significant number of utility programs on each system(from 24-33%); • Detected 2 types of error results, crashing and hanging. • Considered crashed if the program terminated producing a core file • Considered hung if the program continued executing producing no output while having available input or continued to produce output after its input had stopped. • A utility might crash on one system but not on another; • Grouped the causes of the crashes into different categories;

11. The results and analysis(Crash causes) • Pointer/Arrays • Not checking for exceeding the range of the array. • References through a null pointer. • Not Checking Return code • Sign of careless programming • Input Functions • gets(), scanf() – no means of bounds checking are possible. • Sub-Process • Access to sub-process should be carefully controlled or you should insure that the program input to the sub-process is first checked. Alternatively, the utility should be programmed to tolerate with the failure of a sub-process

12. The results and analysis(Crash causes) • Interaction Effects • !o%8f causes the VAX version of csh to crash and !o%888888888f hung most versions of csh • Bad Error Handler • The error recovery routine only increments a count of the number of errors detected, did not exit the code. • Signed Characters • Can be considered sub-case of pointer/array error • Race Conditions • Undetermined Errors

13. Conclusions(Comments on the Results) • All Arrays reference should be checked for valid bounds • All input fields should be bounded • Check all system call return values • Pointer values should often be checked before being used. • Judiciously extend your trust to others • If you redefine something to look too much like something else, you may eventually forget about the redefinition • Error handlers should handle errors • Goto statements are generally a bad idea

14. Conclusions(Comments on Lurking Bugs) • Bugs often be ignored • It is often difficult to isolate exactly what activity caused the program to crash. • It quicker to try a slightly different method to get the current job done than it is to find and report a bug. • The number of bugs in UNIX might also be explained by its evolution. • UNIX should not be singled out as a buggy operating system

15. FUZZ-Revisited • First, the continued prevalence of bugs in the basic UNIX utilities seems a bit disturbing. • Second, the reliability of network services and X-Window servers is good news. • Third, X-Window applications are no less prone to failure (and seem to be more so) than the basic utilities. • Fourth, the reliability of the freely-distributed GNU and Linux software was surprisingly good, and noticeably better than the commercially produced software.

16. References • ftp://grilled.cs.wisc.edu/fuzz • ftp://ftp.cs.wisc.edu/paradyn/technical_papers/fuzz-revisited.pdf • ftp://ftp.cs.wisc.edu/paradyn/technical_papers/fuzz-nt.pdf