Software Tools Design COEN 276

Software Tools DesignCOEN 276 Instructor: Chris Vail Fall 1998

Introduction • Grading policy: • Homework 40% • Midterm 30% • Final 30% • Programming assignments • Individual effort! • Design • efficiency, maintainability • Correctness • Documentation • comments, man page

The Objectives • Use tools to build software, build tools to construct software • practice writing software (lots of practice) • learn software design from basic to complex • Platforms • Unix system calls, Utilities, IPC • Shell programming • Perl programming • X11/Motif Window programming • World-Wide Web, HTML, CGI, JavaScript • Java

History of Unix • 1970-1974, “Simple is beautiful” • early stage of AT&T Bell Lab, on PDP-11 machine • Unix is not an acronym, but a weak pun on MULTICS • 1976, first licensed release, Version 6 • 1978, first portable version, Version 7 • 1979, Berkeley 3BSD • 1983, System V as industry standard • 1984, Microsoft releases Xenix • 1986, BSD 4.3, AT&T Version 9 • 1987, SVR4, Mach, … • 1993, Linux

Unix System in the 90’s • SunOS (BSD) • Solaris • Novell Unix • Linux • Free BSD • HPUX • AIX • Mach (Both BSD and System V emulator on Mach microkernel) • SCO Unix

Unix Philosophy • Easy to program by combining small building blocks • Not (naive) user friendly • A clean minimal design (nothing extra, nothing unnecessary) • Open access: • Provide tools and mechanism to combining the tools • Minimal restrictions to the ways of doing things • A user can be very creative (and frustrated). • Major Unix OS features: • Kernel • Shell • File System

Unix Operating System Structure • OS mediates between the user and the computer User Application Programs Shell Kernel Hardware

Kernel • Manages memory and allocates it to each process • Schedules work done by the CPU • Organizes transfer of data from one part of machine to another • Accepts instructions from shell and carries them out • Enforces access permission on the file system

Shell • Command interpreter • Create customized environments • Write shell scripts • Define command aliases • Manipulate command history • File and command completion • Edit the command line

File System • Logical method for organizing and storing large amounts of information. • Easy to manage. • File: basic storage unit. • Types: • ordinary file (stores info) • directory (holds other files and directories) • special file (represents physical devices like printers, terminals, etc) • pipe (temporary file for command linkage)

UNIX: Multi-user Multi-tasking • More than one user can run at the same time and more than one task can run at the same time • Unix is multiuser multitasking, Window NT is, Windows is not. • In Unix, each program is started as a process. • A process is a program in execution. • Usually only one copy of a program, but there may be many processes running the same program. • To each interactive user (window): • only one process in foreground • may have several processes in background

Processes Process 0: Kernel bootstrap. Start process 1. kernel kernal mode user mode Process 1: create processes to allow login. /etc/init fork exec fork exec inetd lpd condition terminal for login /etc/getty /etc/getty httpd exec exec check password /bin/login /bin/login exec exec command interpreter shell shell

Unix Process • Init process • last step in booting procedure • create other processes to allow the users to login • Getty process • conditions for terminal connection • wait for user-id • display login on the screen • Login process • check password with the uid • execute .profile or .login (depends on default shell) • display shell prompt • Shell process (command line interpreter) • Shell prompt ($, %)

UNIX Process • Process environment • Process id, parent-process-id, process-group-id • Opened files • Working directory • File creation mask • User ID, Group ID • Resource limits • Environment variables • Code • A child process inherits parent’s environment.

Processes use ps to see the processes that you are running. $ ps PID TTY TIME CMD 221 pts/4 4:15 netscape 201 pts/4 0:05 bash 215 pts/4 1:15 emacs-19 • use “&” to execute a task in background • Example: $ sort infile > outfile & • use ctrl-Z to suspend the foreground task, and then use bg. • use fg to move a background task to foreground.

Shell: Command Interpreter • Bourne Shell: the default shell (sh) • original unix shell • does not have interactive features of newer shells • widely used for writing shell scripts • standard on Unix System V • C Shell (csh): available on BSD Unix and most other systems • with syntax similar to the C language • with many enhancement over the Bourne shell. • Korn Shell (ksh): AT&T’s answer to C Shell • combine features of Bourne and C shells • very efficient • Other shells: tcsh, bash

Day-to-Day Use Command Function Meaning cat Display a file conCATenate cp Copies a file CoPy mv Renames a file or moves it MoVe rm Delete files ReMove lpr Sends a file to a printer Line Printer lp (Sys V) ls Lists the contents of a directory LiSt chmod Changes the “mode” of permissions Change MODe pwd Shows the current directory Print WorkingDir cd Change current directory Change Dir mkdir Create a directory MaKe DIR rmdir Delete a directory ReMove DIR ps Shows the processes on the system Process Status man Shows info. about commands Manual df Shows file system status Disk File du Shows the space allocation of files Disk Utilization grep Search for patterns in files

Standard Command Format • command [options] [arguments] wc [-c | -m | -C] [-lw] [file …] • stuff in brackets is optional • boldface words are literals (must be typed as is) • italicized (or <> enclosed) words are args (replace appropriately) • Options modify how the command works • command [options] [--] [<file> …] options ::= -option white-space options* option ::= noargoption* argoption | noargoption+ noargoption ::= letter argoption ::= letter whitespace string $cc -po zap zap.c

Some Examples • ls [-alRF…] file-list • a for listing all files including the dot files • l for long format (file type, permissions, #links, owner, group, etc) • R for recursive, list subdirectories. • F for listing directories with a trailing / • ps [<options>] • List the information about running processes • Example: %ps -el # print the info about all processes (e) in the long format (l)

On-line Documentation • For shell command, system programs, and library functions. • %man [n] <command> e.g. %man wait %man 1 wait %man man %man 1 man %man -k <keywords> • Man(ual) page format Name Synopsis Description (options, defaults, detail deisc., examples) Files See Also Bugs

I/O Redirection • Redirection and Pipe “>“ redirects standard output (screen) to a file E.g. ls > dirlist “<“ redirects standard input (keyboard) from a file E.g. sort < infile > outfile “|” pipe output of program 1 to input of program 2 E.g. who | wc Or getit < in | check_it | process_it | format_id > out “>>“ appends output to a file E.g. ls -l >> oldfile Exercise: find out the definition of “tee”.

Sequential vs. Concurrent Process • Sequential: %date %ps -ef OR %date; ps -ef; who %who • Concurrent: %pgm1 & prgm2 > file1 & pgm3 %make > log & %sort +1 pdir; ((pr dir | lpr) & sort +1 local))

File Name Expansion • Each shell program recognizes a set of special characters called meta characters. • The metacharacters are used to create patterns to match filenames and command names. • Bourne and Korn shell meta/wildcard characters • * matches any string (including null) • ? matches any one character • [a-dA-D] matches any one character in the range • [!AB123] matches a char not in this range • \ escape • ~<username> : (not bourne shell) the home dir of the user.

File Name Expansion • Assume we have the following files under the current directory: 120, 235, 2#5, a.out, c22, c*2, Doc.1, Doc.2, Doc.3, one.c,two.c, three.c ls *.c ls c*2 ls [a-z]*[!0-9] a.* *.* ls ??? cd ~foo ls *

Filters • Most UNIX utilities are filters • A filter is a program which • reads input (text) from standard input only • writes output to standard output only • writes error to standard error only • may use temporary files for intermediate results • Filters can be combined to work together using pipes • Pipe: takes stdout of one command and uses it as stdin of another command ls | wc

Command Alias • Assign your own name for a command • Syntax is shell dependent alias ll ‘ls -l’ C shell alias ll=‘ls -l’ Korn, Bourne shell • Displaying the value of an alias alias ll (displays value)

Unix File Systems • File: a collection of data • Just a sequence of bytes • no record or block structure required • Directory • A file that contains information for files • distinction between a directory and a file • system can alter the contents of a directory • rooted tree file structure (inverted tree) • directories can contain both files and other directories info.

Unix File System Road Map • Special files: /dev/* represents I/O devices. / /dev /etc /var /bin /tmp /usr /mnt /home passwd hosts …... spool adm include lib sue john etc bin 5bin printer mail(in) mail(out) uucp messages wtmp …

File Systems and the I-nodes • Each disk drive contains one or more file systems • Each file system occupies a number of cylinder groups. • Each file system has a superblock, an i-node table and files • The superblock is at a fixed location relative to the beginning of the file system. It contains a description of the file system. • One can find the location of the I-node table thru superblock. • Each entry of the I-node table is an I-node, which uniquely represents a file in the file system. • An I-node contains uid, gid, time of creation, modification, and access, permissions, location of the file on the disk. superblock I-node table file1 file2 free file3 free

Mounting File Systems • A file system must be mounted before it can be used • Root file system is mounted at boot time. • A file system can be mounted to a dir. of another mounted fs. • Mounting is done in memory / /dev /etc /var /bin /tmp /usr /mnt /home a file system

Permissions • Every file has a set of permissions associated with it • three types of permissions: read ( r), write (w), and execute (x) • three sets of permission: user, group, world. • In Unix system, users are identified by numbers:uid, gid ls -l -rwxr-xr-x 1 root 3743 Jan 4 1970 test user group others #links owner size (time of last mod) (file name) • Problem: how do you write a program which allows different users to access a set of files? E.g. the program “passwd”

Permissions (cont.) • Solution: use the set-uid bit • When a user execute a program with the set-uid bit set, the user assume the identity of the owner of the program. • For example ls -l /bin/passwd -rwsr-xr-x 1 root 8454 Jan 4 1994 /bin/passwd set-uid • Set-uid bit may break the security wall. (users can run the /bin/passwd and act like root) • Only special programs can be set-uid program, particularly if the owner is root.

Chmod • Change the access permissions of a file • chmod <permissions> <filename> • permissions can be specified as 3 octal digits, <user,group,others>, the three bits of an octal means r,w,x • Example: chmod 755 test • permissions can be specified as +x, or u+x, or g+r, … • “chmod +s test” sets the set-uid bit for file test. • If a directory has “x” in its permision, the dir is searchable, ie., one can do “ls” on the directory.

Pathnames • Every file and directory in the file system can be identified by a “full path name” (route from root to file) /home/sue/email/f1 • Relative path name • location relative to current directory . Current directory .. Parent directory • if cwd is /home/sue: ls email ls ./email cd .. ls ../fred / home sue fred email docs f2 f1

Guidelines for writing Unix Tools • Standard command format • Recognize meta-characters (handle multiple files) • Standard I/O (stdin,stdout, stderr. If file arg is absent use std) • Keep messages and prompts to a minimum. • Provide verbose options • Input/output data should be text whenever possible. • Use dot files and environment variables for frequently used info. • Use standard library and tools to save coding effort.

Some Details • cp [-ir…] file1 file2 cp [-ir…] file-list directory • i for interactive. prompt use whenever a file will be overwritten • r for recursive. copy a directory tree • ls [-alRF…] file-list • a for listing all files including the dot files • l for long format • R for recrusive. list the all subdirectories. • F for listing directories with a trailing / • date [+format] • %date ‘+%h %d, 19%y’ Oct 1, 1996

Some Details (cont.) • wc file-list • display the number of lines, words and charaters • more file-list • Browse through text files on page at a time. • head [-n …] file-list • Display the first n lines of the files (default=10) • tail [+n|-n| -f| …] • Display the last few lines in the files (default = 10) • Example: %tail +5 foo # display the last parf of foo starting from line 5 %tail -5 foo # display the last five lines of foo %tail +30 foo | head -15 | more #display line 30-45 of foo %tail -f foo # wait and display the new lines appended to foo

Some Details (cont.) • cut -c list file • cut -f list [-dChar] file • Cut out selected charaters or fields from each line of a file • Examples: %cut -c 1-5,15-20 foo # extract chars 1-5 and 5-20 from each line of foo. %cut -f 1,3 -d” “ moo # extract field 1 and 3 from each line of moo. • paste file1 file2 • Concatenate corresponding lines of the given input files • Example (reverse two fields of the file abc) %cut -f1 abc > abc1 %cut -f2 abc > abc2 %paste abc2 abc1 > xyz

Some Details (cont.) • grep, egrep, fgrep grep [-nv...] pattern [file-list] • Search the input text files for lines matcing the pattern %grep Unix doc.1 # Display the lines in doc.1 that contains “Unix” %grep -n Unix doc.* # Display the lines with line numbers %grep -v Unix doc.1 # Display the lines which do not contain “Unix” • sort [-tC…] [-o outfile] [field-list] [file-list] • sort the files %sort +1 list1 # sort list 1 starting from field 2 to the end of the line %sort +2-3 list2 # sort list2 on the third field %sort -n -o list4 list3 sort list3 numerically and place the output in list4

diff • diff file1 file 2 • Display different lines that are found when comparing two files • It prints a message that users ed-lide notation (a - append, c - change, d -delete) to describe how a group of lines has changed. • It also describes what changes need to be made to the first file to make it the same as the second file. • Example file1 file2 file3 apples apples oranges oranges oranges bananas bananas kumquats kiwis peaches

diff (cont.) %diff file1 file2 3c3 <bananas ----------------- >kumquats %diff file1 file3 1d0 <apples 3a3,4 >kiwis >peaches

comm file1 file2 • Takes two sorted text fiels and print common lines and lines which appear exclusively in one file on separate colmns. • column1: lines only in file1, column 2: lines only in file2; col 3: comm • Example file1 file2 %comm file1 file2 apple apple apple banana cantaloup banana grape grade cantaloup orange kiwi grape lemon kiwi %comm -12 file1 file2 apple grape

tr [-csd…] pattern1 pattern2 • Translate input character to output character based on the input and output patterns • Example %tr ‘[A-Z]’ ‘[a-z]’ <in >out # xlate all letters to lower case. %tr -s ‘\012\011\040’ ‘\012\012\012’ < in > out # xlate blank, tab and newline chars to newline chars and squeeze (-s) consecutive newline char into one %tr -cs ‘[a-z][A-Z]’ ‘[\012*]’ < in > out # change all non-aplphbetic (-c) chars to newline chars and squeeze consecutive newlne char into one. %tr -d ‘\040’ < in > out # delete all blanks.

uniq [-cdu…] file-list • Display a fiel, removing all successive repeated lines • Example: file1: %uniq file1 apple apple banana banana banana apple apple banana %sort fruit | uniq -c apple 2 banana 3 %tr -cs ‘[a-z][A-Z]’ ‘[\012*]’ < fileA | sort | uniq # show a list of distinct words in fileA.

find <dir-name> <exp> • Recursively search the directory tree rooted at <pathname> and find all files whose names satisfy <exp> • There are many details in the expression. • Examples: %find . -name \*.doc -print # list all files names ending with .doc %find /etc/source -atime 2 -print # print the names of the files under /etc/source whose lst access time was 2 days ago. %find . -name “[a-z]*” -exec rm {} \; # remove the files under the curent directory whose names begin with a lower case letter. %find / $-name a.out -o -name “*.o” $ -atime +7 -exec {} \; # remove the object and binary executable files under the root direory which have not be accessed more than 7 days.

cpio -i[cdv] -o[cBv] • System V file archive and backup progam • Example %find proj -print | cpio -ocBv > /dev/rmt8 # cpio get file names from stdin. “-o” create archive which is redirected to the tape device. %find proj -print | cpio -ocBv > proj.cpi # get file name from stdin and “-o” createsarchive which is redirected to proj.cpio %cpio -icdv “*.c” </dev/rmt8 # -i read from archive file from the tape device. -d creates directories as needed.

tar [options] [file-list] • key := c (create) | t (table of content) |r (append the file) | u (update the file) • options := v (verbose) | b (block) | f (file name follows) | m (use extraction time as the mod file) … • Create/extracting archive files for backup and transporting files %tar cvf proj.tar proj # create archive file proj.tar from file or dir proj %tar xvf proj.tar # extract files in proj.tar % tar tf proj.tar # list of the filenames in proj.tar without extracting data. %tar cf - proj | (cd /newproj/; tar xvpf -) # copy proj to the directory /newproj/. “p” to keep all the protection mode. • cp -r copies a dir tree but all the time info is gone. Tar preserve the time info. %tar cbf 20 proj.tar /usr/local/proj # avoid using full path names. When you extract the file, tar will insist to put fiels to /usr/local/proj.

uuencode & uudecode • Generate an ASCII encoded version of the give files • Example: %uuencode file.bin newfile.bin > file.bin.uu # encode file.bin and put the result in file.bin.uu %uudecode file.bin.uu # decode the file file.bin.uu and generate a new file newfile.bin • Sending a dir tree via email %tar cvf proj.tar proj %compress proj.tar # compress proj.tar to proj.tar.Z %uuencode proj.tar.Z proj.tar.Z | mail qli … at the receiving end, extract the mail and save it in xx %uudecode xx %zcat proj.tar.Z | %tar xvf -

sed [-n] | [-e] sed_cmd file_lists • A stream editor. It copies the lines from the file_list or stdin to stdout, editing the lines in the process. • Examples: %sed -n ‘/hello/p’ < input > output # copy the lines contains “hello”. -n suppress stdout so only the lines that matches are copied. %sed ‘5,7d’ file1 # delete lines 5 to 7 from file1. File1 is unchanged. %sed ‘s/Unix/UNIX/’ doc2 #replace the first occurrence of Unix in each line by UNIX. %sed ‘s/Unix/UNIX/g’ doc2 # replace all Unix by UNIX

awk [-f progfile] [-Fc] [prog] [files] • Pattern matching and stream editor. • Example: • Program awkexample: BEGIN { linetype=0} # initialization NR == 1 { print $1 “ “ $NF} # if it is the first line, print the last field /^$/ { print “This is an empty line” } /^Unix/ { printf(“Line starts with Unix\n %s\n”, $0); linetype=1; next;} /NonUnix$/ { printf(“End with NonUnix\n”); linetype=0; next;} linetype == 1 { print $0} END {printf(“%d lines processed\n”, NR);} # finishing it up

Software Tools Design COEN 276