Mastering Unix Bash Commands for Efficient Scripting

1. Introduction to Unix – CS 21 Lecture 12

2. Lecture Overview • A few more bash programming tricks • The here document • Trapping signals in bash • cut and tr • sed • awk

3. The Here Document • Redirects stdin to a specific set of text located inside the same file • << COMMAND << MARKER Data MARKER

4. Example of Here

5. Why Is This Useful? • Allows you to keep all relevant information in one file • Example: Database you want to search • Don’t need to clutter up your directory with unnecessary temporary files

6. Trapping Signals • Catching a signal is also called trapping a signal • You can tell bash programs what to do when they receive different signals • Analogy: When a postcard arrives, what do I do?

7. The ‘trap’ Command • Usage: trap ‘COMMAND’ Signals • Example: • trap ‘cat errorMsg’ 4 6 • In order to prevent you from running a program forever, signal number 9 cannot be trapped

8. Example Of trap

9. Two Helper Filters • cut • Break individual lines apart • tr • Change characters into different characters

10. The ‘cut’ Command • More precise control over individual lines • Will cut out certain words from each individual line so they can be processed • Usage: cut [FLAGS] FILE

11. Flags • -d • Delimiter • -f • Field number • Example • cut –d: -f3 myFile

12. Example Of Cut

13. The ‘tr’ Command • Translate • Change on a one to one basis characters from one thing to another • Usage: tr ‘Set1’ ‘Set2’ • Example: tr ‘abc’ ‘ABC’ < myFile

14. Example Of tr

15. Two More Powerful Tools • sed • Stream Editor • awk • Alfred Aho, Peter Weinberger, and Brian Kernighan

16. The ‘sed’ Command/Language • Filter • Like grep, sort, or uniq, it takes input and performs some operation on it to filter the output • Usage: sed ‘Address Command’ • Address specifies where to make changes • Command specifies what change to make • Example: • sed ‘4d’ textFile

17. Address Specification • Addresses could be line numbers or regular expressions • No address – each line • One address – only that line • Two comma separated addresses – All lines in between • ! – All other lines

18. Commands Available To sed • a\ • Append text • c\ • Replace text • i\ • Insert text before • d • Delete lines • s • Make substitutions

19. Examples

20. More Examples

21. Substitution Example • Same syntax as vi

22. When Would You Want To Use sed? • sed works on streams, so it is perfect to be placed in the middle of a pipe in order to change the output from one format to another • Example: • If a program always prints out 4 lines of junk for every good line, sed can be used to weed out the junk

23. Example

24. awk • Answers the question: • What do I do if I want to search for a pattern and actually use it? • Combination of grep and commands • Searches for some pattern or condition and performs some command on it • Complete programming language • Looks a lot like C syntactically • Variables are declared bash style

25. Pattern And Command • awk in its most basic form simply executes a command on all lines that match (or adhere to) a certain pattern • Usage: awk ‘Pattern { Command }’ FILE • Just like sed, if there is no pattern, then every line will be matched

26. Example

27. Different Ways To Run Awk • awk ‘Pattern { Command }’ • awk –f awkFile inputFile • Since awk itself can be a complex language, you can store all the commands in a file and run it with the –f flag

28. Important awk Concepts • Record • Every line of an input file is a record • The current record can be referenced with $0 • Field • Every word in a record is called a field • Each field is numbered and can be referred to • $1 is the first record, $2 is the second, etc.

29. Special Predefined awk Variables • RS • The character that acts as a record separator • Default is the end of a line • FS • The character that acts as a field separator • Default is whitespace (space, tab, etc) • Can be redefined with the –F flag

30. Example

31. Other awk Variables • NF = number of fields in the current record • NR = Total number of records seen so far • OFS = Output field separator • ORS = Output record separator

32. BEGIN And END Blocks • Two special patterns that can be matched • BEGIN • Commands are executed before any records are looked at • END • Commands are executed after all records are processed

33. Example

34. Awk Patterns • /regular expression/ -> same as egrep • Relational expression • >, <, >=, <=, == • Pattern && pattern • Pattern || pattern • Pattern1 ? Pattern2 : pattern3 • If Pattern1 is True, then Pattern2, else pattern 3 • (pattern) • ! Pattern • Pattern1, pattern2

35. Example Patterns

36. Awk Actions • Enclosed in { } • () Grouping • $ Field reference • ++ -- Increment, decrement • ^ Exponentiation • + - ! Plus, minus, not • * / % Multiplication, division, and modulus

37. Control Flow Statements • Inside of commands, you can have control flow • if • while • for

38. If Syntax if (condition) { Statements } else { Statements }

39. While Syntax while (Condition) { Statements }

40. For Syntax for (Declaration ; Condition ; Increment ) { Statements } for ( j=0; j < 5; j++) { print “hello world” }

41. When Would You Want To Use awk? • Whenever you want to search for some pattern and perform some action • Example: I want to go through and calculate the average score on the Midterm

42. Example

43. Another Example • Adding 12 points to everyone’s midterm score

44. Putting Them Together – awk and sed

45. awk Versus bash $ arguments • Always enclose everything to awk in single quotes so they don’t get interpreted • $1 to awk means something completely different than $1 to bash • $1 in awk means first field • $1 in bash means first command line argument

46. Next Time • Looking at some of the string and mathematical awk functionality • find • Putting everything together • The complete bash programming necessities • Quiz 2 – Next Tuesday