Chapter 3 UNIX Utilities II

1. Chapter 3 UNIX Utilities II By C. Shing ITEC Dept Radford University

2. Objectives • Understand how to filter files • Understand how to sort files • Understand how to search for files • Understand how to compare files • Understand how to archive, transform and compress/uncompress files • Understand how to schedule processes • Understand hard and soft links • Understand how to attach different file systems • Understand how to find every character in file

3. Filters (Find file Content) • grep (Global or Get Regular Expression and Print), egrep (Extended grep), fgrep (Fast grep) • uniq • awk (Aho, Weinberger, Kernighan): C control structures • sed (Stream Editor)

4. Filters • grep, egrep, fgrep: filter out lines that do/don’t have specified pattern in the file content • grep • Syntax: grep -option pattern filename • Option: • i: ignore case • l: list files containing the pattern • n: display line # • v: displays lines that don't match the pattern • w: match to whole word only • Pattern: regular expression

5. Filters- grep

6. Filters- grep

7. Filters- grep • Examples: • grep smith /etc/passwd show all lines that have smith pattern • grep -wn lower /usr/include/* use word: lower and show line # • grep -ni lower /usr/include/* ignore case and show line number • ls -l | grep ^d show directory • ls -l | grep ‘^..w’ show files that owner can write • cut -d: -f5 /etc/passwd | grep -i ' smith$' | more show under field 5, any line has word end with smith case insensitive

8. Filters- egrep • egrep • Syntax: egrep -option pattern filename or grep –e -option pattern filename • Option: • i: ignore case • l: list files containing the pattern • n: display line # • v: displays lines that don't match the pattern • w: match to whole word only • Pattern: extended regular expression

9. Filters- egrep

10. Filters- egrep • Examples: • egrep sm.?ith: /etc/passwd all lines contain 0 or 1 of any character after sm • egrep -wn stdio \| STDIO /usr/include/stdio.h any line containing stdio or STDIO • egrep -ni to+ /usr/include/* | grep too all lines contain too • ls -l /devices| egrep –v ^d show all non-directory files (device has major #: routine #, minor #: index in the routine) • cut -d: -f5 /etc/passwd | egrep -i ' smith.+$' | more any line containing smith followed by 1 or more characters

11. Filters- fgrep • fgrep • Syntax: fgrep -option string filename • Option: • i: ignore case • l: list files containing the pattern • n: display line # • v: displays lines that don't match the pattern • w: match to whole word only

12. Filters- fgrep • Examples: • fgrep smith /etc/passwd • fgrep –wn lower /usr/include/*

13. Filters- uniq • uniq: filter out duplicated adjacent lines • Syntax: uniq -option [infile [outfile]] • Option: • number: # of fields to be skipped (delimiter: space) • c: precedes # of duplications

14. Filters- uniq • Examples: • cut –d: -f4 /etc/passwd | uniq -c list # of same adjacent GID lines

15. Compare - cmp • cmp: find 1st byte that is different • Syntax: cmp -option file1 file2 • Option: • l: list offset and line number of 1st mismatched byte • S: all output inhibited

16. Compare - diff • diff, sdiff: display differences so file1 can be converted to file2 (sdiff shows actual lines involved) • Syntax: diff -option file1 file2 or sdiff -option file1 file2 • Option: • i: ignore case

17. Compare - diff • Output form: • File1Line# a File2Line#Start, File2Line#Stop > • File1Line#Start, File1Line#Stop d lineCount < • File1Line#Start, File1Line#Stop c File2Line#Start, File2Line#Stop < --- >

18. Compare - diff • Example: • diff /etc/auto_net_rucs /etc/auto_net_rucs2 • diff /etc/auto_net_rucs /etc/auto_net_ruacad

19. Sort • Sort: in ascending order • Syntax: sort –option file • Option: • t: c-delimiter default - space or tab • +n1: n1 is the index number for the starting sort field • -n2: n2 is the stopping sort field index number (non-inclusive) • b: ignore leading blanks • f: ignore case for sort fields • n: sort fields in numerical order • M: sort fields in month order • r: sort in reverse (descending) order • +n3, -n4: more comparison when equal in using +n1,-n2

20. Sort • Examples: • sort +0 -1 -b /usr/include/stdio.hstarts from and to the 1st field (index=0), ignore leading spaces and sort /usr/include/stdio.h in ascending order (delimiter is space or tab). • sort -t: +4 -5 -b +0r /etc/passwd | moreUse : as the delimiter of fields, sort in ascending order according the Real Name (index=4) in /etc/passwd, ignore leading spaces; if sorted fields are equal, then sort by the username in descending order • sort -t: +2 -3n /etc/passwd | moresort by uid (numerically)

21. Search (Find file/directory name/attribute) • Find: search the expression from the path • Syntax: find path expression

22. Search

23. Search

24. Search

25. Search • Examples: • find ~ -name '*.c' -print | moreIt recursively goes down from your home directory to find any c files and print on screen. • find . -mtime -14 –print | more It displays all files that has been modified within the past 2 weeks. • find ~ -perm 4701 –ctime -7 superuser needs check any executable set to suid within last 7 days(maybe hacker)

26. Search • Examples: • (find / -name handler.py > /dev/tty) > & /dev/null This finds the python file and sends errors to /dev/null, and results are displayed on the terminal, so you don’t see error in screen • find / -name handler.py >& errorfile The error will goes to the file errorfile

27. Search (Cont.) • Examples: (Cont.) • find . -name a.out -ls -exec rm {} \;It recursively goes down from the current directory to delete a.out files and print those names on screen. (This rm command may execute 100 times if there are 100 a.out files found) Note: a more efficient way is to use xargs command as below: (Linux only) find . -name a.out -ls | xargs rm (This will put all files found in the parameters of the command rm and execute once) If the command line parameters of rm has limited to 10, then we can use find . -name a.out -ls | xargs –n10 rm (This will execute the command 10 times, each time it removes 10 files.)

28. Search (Cont.) • Examples: (Cont.) • find . \( -name '*.c' -o -name '*.java' \) -print | moreIt recursively goes down from the current directory to find any c files or java files and print on screen.

29. Filters - awk • awk: scan every line that matches certain criteria (Programmable Text Processing ) or no condition, perform some actions to them Use C syntax and C library function • Syntax: each line in form - [condition][{action}] • awkPgm (use –f): line in file • awk -option -f awkPgm file • Command: line (in quotes) • awk -option '[condition][{action}]' file • Option: • Fc: c – Fields delimiter tab or space - default delimiter

30. Filters - awk • Condition

31. Filters - awk • Action

32. Filters - awk • Action

33. Filters - awk • Examples:(all necessary Unix format files(Do not use DOS format) are in the folder) • awk -F: '{print NF, $0}' /etc/passwd|morewhere NF: # of fields (predefined variable), $0: entire line • awk -F: -f awk1 /etc/passwd|moreIt prints 1st, 3rd fields and number of fields (no spaces in between)

34. Filters - awk • awk -F: -f awk2 /etc/passwd | moreIt prints out 1st, 3rd fields and number of fields (with spaces in between) for line 2 and 3(NR: record number) • awk -F: -f awk3 /etc/passwd | moreIt prints out line # along with each line and count total # of line s and fields. • awk -F: -f awk4 /etc/passwd | moreIt prints out from the last field back to the 1st field for each line.

35. Filters - awk • awk -F: -f awk5 /etc/passwd |moreIt prints out the line that have the pattern (t followed by one or more characters and end with e). • awk -F: -f awk6 /etc/passwd |moreIt prints out the line that contains "smith". • awk -F: -f awk7 /etc/passwd |moreIt prints out the line that contains "Smith:".

36. Filters - awk • awk -F: -f awk8 /etc/passwd |moreIt prints out all lines in between (inclusively) the beginning line that contains "root" to the ending line that contains "local“ • awk -f awk9 awktestlibIt shows awk library functions.

37. Filters - awk 11. Example of using if .. else

38. Filters - sed • sed: scan every line and perform simple editing action on lines that match certain criteria • Syntax: sed -option scriptfile filename • Option: • f: scriptfile is file of command lines no option: use command line in quotes • n: suppress print duplication caused by p • e: multiple instructions in command line

39. Filters - sed

40. Filters - sed

41. Filters - sed • Examples:(all necessary Unix format files(Do not use DOS format) are in the folder) • sed 's/^/\$ /' awk1 > awk1.indentThis gives a $ at the beginning of each line in the file awk1 and creates output file awk1.indent. • sed 's/^ *//' awk3This removes leading spaces in each line of the file awk3. • sed -f sed1 awk3This removes all lines that contains $ in the file awk3. • sed '2,4d' awk9This removes line 2 to line 4 in the file awk9.

42. Filters - sed • sed '$d' awk9This removes last line in the file awk9. • sed -n '2,4p' awk9This prints from line 2 to line 4 in the file awk9. • sed 5q awk9This prints from line 1 to line 5 in the file awk9. • sed –f sedinsert awk9 This inserts 1st and last line. • sed –f sedchange awk9 This changes the last line. • sed –f sedreplace awk9 This replace all “printf” by “fprintf” • sed ‘1,$s/the/THE/g’ awk9 replace the word ‘the’ by ‘THE’ in the entire file awk9 • sed –n –e ‘1,2p’ –e ‘$p’ awk9 prints line 1,2 and the last line of the file awk9 • sed –n –e ‘/<TABLE>/,/<\/TABLE>/w table.html’ –e ‘/<FRAME>/,/<\/FRAME>/w frame.html’ entire.html extract table and frame parts of the entire.html and save them into individual files

43. Archive • cpio (not in MS-DOS): small files • tar (also available in MS-DOS) : backup on tape - single volume • dump (or ufsdump): incremental backup can store in many volumes • restore (or ufsstore): recover

44. Archive - cpio • cpio • Syntax: cpio –option [file][directory] • Option: • o: takes standard input and creates output cpio format files with those filenames • v: displays those filenames • i: takes a cpio format file from standard input and creates those filenames • d: creates directory • t: creates table of contents • p: copies files into the directory • l: creates links (instead of copying) in the directory

45. Archive - cpio • Examples: • mkdir tmpcp *.c tmpcd tmpls *.c | cpio -ov > backupIt creates a cpio format output file conatining all c files • ls -l backuprm -f *.clscpio -i < backupIt recreates all original files from the cpio file.ls -lcd ..rm -rf tmp

46. Archive - tar • tar • Syntax: tar –option [tarfilename] filelist • Option: • c: creates tar format file • f: tarfile • r: unconditionally append to the tarfile • t: creates table of contents • v: verbose output • x: extract from tarfile • Z: compress file

47. Archive - tar • Examples: • tar -cvf tarfile *.ctar all c files to the current firectory • tar -tvf tarfileread current contents of tarfile • tar -rvf tarfile *.txtunconditionally adds all txt files to the end of the tarfile • tar -tvf tarfile | moremkdir tmpmv tarfile tmpcd tmp • tar -xvf tarfile `tar -tf tarfile | grep filecp.c `It only untar the file filecp.c from the tarfile. • tar czf tarcompressfile achive and then compress file

48. Archive – dump/restore • dump • Syntax: dump level f option dumpfile filesystemwhere dumpfile is defaulted to /etc/rmt0 • Option: • v: verify each volumn • w: displays filesystems that need to be dumped

49. Archive – dump/restore • Examples: • dump 0 fv /dev/rmt0 /dev/da0

50. Archive – dump/restore • restore • Syntax: restore -option f dumpfile files where dumpfile is defaulted to /etc/rmt0 • Option: • t: show table of contents • x: restore only files specified • i: interactive mode