Shell Programming

1. Shell Programming Guntis Barzdins Girts Folkmanis

2. Lecture outline • Shell features • Helper utilities, introduction • Connecting utilities with shell scripting • Helper utilities in detail • Piping, advanced examples • Shell scripts as files, Internal shell commands

3. Shell features • We will talk about bash, there might be differences for other shells. • bash - GNU Bourne-Again Shell • Authors: Brian Fox and Chet Ramey • Free Software Foundation • Popular in different distributions • Tip: To find your current shell, type following command $ echo $SHELL /bin/bash

4. 'Bash' bug ("Shellshock" bug) Wade Mealing 2014-09-14 22:24:57 EDT A flaw was found in the bash functionality that evaluates specially formatted environment variables passed to it from another environment. An attacker could use this feature to override or bypass restrictions to the environment to execute shell commands before restrictions have been applied. Certain services and applications allow remote unauthenticated attackers to provide environment variables, allowing them to exploit this issue. Acknowledgements: Red Hat would like to thank StephaneChazelas for reporting this issue. env x='() { :;}; echo vulnerable' bash -c 'echo this is a test'

5. Shell Features • The shell itself is defined in SUS (Single UNIX Specification) as regardscalling conventions and switches. The language interpreted by the shell is alsopart of the standard • The shell standard derives from the POSIX.2standard, which is not freely available (thecurrent standard, SUS, stands as IEEE Std1003.1 2001and is identical to POSIX.2)

6. Shell features • Two types of usage: • Command line - interactive • Shell script, usually non-interactive • Shell script defined as: • "Shell Script is series of commands written in plain text file. Shell script is just like batch file is MS-DOS but have more power than the MS-DOS batch file."

7. Shell features • Two types of commands: • Internal commands – built in the shell interpreter • External commands – calling other executable files • Almost everything applies to both command line usage and shell scripts

8. External commands • Execution of external programs – most common task External program: /bin/ls girtsf@linux tmp $ ls -l /lib total 4035 -rwxr-xr-x 1 root root 7488 Oct 6 12:33 cpp drwxr-xr-x 13 root root 1024 Oct 25 15:57 dev-state drwxr-xr-x 2 root root 1024 Jun 28 09:53 evms drwxr-xr-x 2 root root 2048 Aug 23 15:25 iptables -rwxr-xr-x 1 root root 92716 Oct 14 13:10 ld-2.3.4.so -rwxr-xr-x 1 root root 22800 Oct 14 13:17 ld-linux.so.1 ...

9. External commands • Environment variable $PATH determines where to search for external programs. • girtsf@linux tmp $ echo $PATH /bin:/usr/bin:/usr/local/bin:/opt/bin • “:” as separator • Current directory “.” is usually not in PATH for security reasons.

10. External commands • girtsf@linux tmp $ echo $PATH /bin:/usr/bin:/usr/local/bin:/opt/bin • With /bin in path, typing “ls” suffices to run /bin/ls. • Example of unsetting path: • girtsf@linux tmp $ unset PATH girtsf@linux tmp $ ls bash: ls: No such file or directory girtsf@linux tmp $

11. Internal commands • A large list of built in commands, that are handled internally without running an external command • Most commonly used internal command is cd, used to change the current working directory: • girtsf@linux girtsf $ cd /tmp/ girtsf@linux tmp $

12. Aliasing • Aliasing is the process of assigning a command to a shorter “alias” • This allows you to type the shorter command instead of the longer one. • Aliasing is useful for changes that you want all of the time. • alias rm “rm –i” • Aliasing is similar to shell function definitions • dos2unix() { cat $1 | perl -pe 's/\r\n$/\n/g'; } • unix2dos() { cat $1 | perl -pe 's/\n$/\r\n/g'; }

13. Internal commands girtsf@linux tmp $ help GNU bash, version 2.05b.0(1)-release (i686-pc-linux-gnu) These shell commands are defined internally. Type `help' to see this list. Type `help name' to find out more about the function `name'. Use `info bash' to find out more about the shell in general. Use `man -k' or `info' to find out more about commands not in this list. A star (*) next to a name means that the command is disabled. %[DIGITS | WORD] [&] (( expression )) . filename : [ arg... ] [[ expression ]] alias [-p] [name[=value] ... ] bg [job_spec] bind [-lpvsPVS] [-m keymap] [-f fi break [n] builtin [shell-builtin [arg ...]] case WORD in [PATTERN [| PATTERN]. cd [-L|-P] [dir] command [-pVv] command [arg ...] ...

14. SUS: shell grammar complete_command : list separator | list ; list : list separator_op and_or | and_or ; and_or : pipeline | and_or AND_IF linebreak pipeline | and_or OR_IF linebreak pipeline ; pipeline : pipe_sequence | Bang pipe_sequence ; pipe_sequence : command | pipe_sequence '|' linebreak command ; command : simple_command | compound_command | compound_command redirect_list | function_definition ; compound_command : brace_group | subshell | for_clause | case_clause | if_clause | while_clause | until_clause ; subshell : '(' compound_list ')' ; compound_list : term | newline_list term | term separator | newline_list term separator ; term : term separator and_or | and_or ; for_clause : For name linebreak do_group | For name linebreak in sequential_sep do_group | For name linebreak in wordlist sequential_sep do_group ; name : NAME /* Apply rule 5 */ ; in : In /* Apply rule 6 */ ; wordlist : wordlist WORD | WORD ; case_clause : Case WORD linebreak in linebreak case_list Esac | Case WORD linebreak in linebreak case_list_ns Esac | Case WORD linebreak in linebreak Esac ; case_list_ns : case_list case_item_ns | case_item_ns ; case_list : case_list case_item | case_item ; case_item_ns : pattern ')' linebreak | pattern ')' compound_list linebreak | '(' pattern ')' linebreak | '(' pattern ')' compound_list linebreak ; case_item : pattern ')' linebreak DSEMI linebreak | pattern ')' compound_list DSEMI linebreak | '(' pattern ')' linebreak DSEMI linebreak | '(' pattern ')' compound_list DSEMI linebreak ; pattern : WORD /* Apply rule 4 */ | pattern '|' WORD /* Do not apply rule 4 */ ; if_clause : If compound_list Then compound_list else_part Fi | If compound_list Then compound_list Fi ; else_part : Elif compound_list Then else_part | Else compound_list ; while_clause : While compound_list do_group ; until_clause : Until compound_list do_group ; function_definition : fname '(' ')' linebreak function_body ; function_body : compound_command /* Apply rule 9 */ | compound_command redirect_list /* Apply rule 9 */ ; fname : NAME /* Apply rule 8 */ ; brace_group : Lbrace compound_list Rbrace ; do_group : Do compound_list Done /* Apply rule 6 */ ; simple_command : cmd_prefix cmd_word cmd_suffix | cmd_prefix cmd_word | cmd_prefix | cmd_name cmd_suffix | cmd_name ; cmd_name : WORD /* Apply rule 7a */ ; cmd_word : WORD /* Apply rule 7b */ ; cmd_prefix : io_redirect | cmd_prefix io_redirect | ASSIGNMENT_WORD | cmd_prefix ASSIGNMENT_WORD ; cmd_suffix : io_redirect | cmd_suffix io_redirect | WORD | cmd_suffix WORD ; redirect_list : io_redirect | redirect_list io_redirect ; io_redirect : io_file | IO_NUMBER io_file | io_here | IO_NUMBER io_here ; io_file : '<' filename | LESSAND filename | '>' filename | GREATAND filename | DGREAT filename | LESSGREAT filename | CLOBBER filename ; filename : WORD /* Apply rule 2 */ ; io_here : DLESS here_end | DLESSDASH here_end ; here_end : WORD /* Apply rule 3 */ ; newline_list : NEWLINE | newline_list NEWLINE ; linebreak : newline_list | /* empty */ ; separator_op : '&' | ';' ; separator : separator_op linebreak | newline_list ; sequential_sep : ';' linebreak | newline_list ; %token WORD %token ASSIGNMENT_WORD %token NAME %token NEWLINE %token IO_NUMBER %token AND_IF OR_IF DSEMI /* '&&' '||' ';;' */ %token DLESS DGREAT LESSAND GREATAND LESSGREAT DLESSDASH /* '<<' '>>' '<&' '>&' '<>' '<<-' */ %token CLOBBER /* '>|' */ /* The following are the reserved words. */ %token If Then Else Elif Fi Do Done /* 'if' 'then' 'else' 'elif' 'fi' 'do' 'done' */ %token Case Esac While Until For /* 'case' 'esac' 'while' 'until' 'for' */ /* These are reserved words, not operator tokens, and are recognized when reserved words are recognized. */ %token Lbrace Rbrace Bang /* '{' '}' '!' */ %token In /* 'in' */

15. Helper utilities • Helper utilities – various small external programs that are helpful when working with shell scripts or command line • Called from shell (scripts or command line) • Somehow transforms input into output, based on the parameters

16. Helper utilities • cat - concatenate files and print on the standard output • Syntax: cat [file1] [file2] … [fileN] girtsf@linux etc $ cat gentoo-release shells Gentoo Base System version 1.4.16 # /etc/shells: valid login shells # $Header: /home/cvsroot/gentoo-src/rc-scripts/etc/shells,v 1.5 2003/07/15 20:36:32 azarah Exp $ /bin/sh /bin/bash /bin/tcsh /bin/csh /bin/esh /bin/ksh /bin/zsh /bin/sash

17. Helper utilities • echo – displays a line of text • Besides a program /bin/echo, also usually built in the shell (takes precedence) • Syntax: echo [STRING] ... girtsf@linux girtsf $ echo quick brown fox quick brown fox Can be used to display environment variables girtsf@linux girtsf $ echo $HOME /home/girtsf

18. Helper utilities • wc - print the number of newlines, words, and bytes in files • wc [options] [file1] [file2] … [fileN] • By default, newlines, words and byte counts are displayed • Options • -c : print only byte count • -w : print only word count • -l : print only line count

19. Helper utilities • Example use of wc: girtsf@linux etc $ wc /etc/passwd 50 76 2257 /etc/passwd girtsf@linux etc $ wc -l /etc/passwd 50 /etc/passwd lines words bytes lines only

20. Helper utilities • grep - print lines matching a pattern • grep PATTERN [file1] [file2] … [fileN] • The lines that contain PATTERN are printed to standard output. • If no files are specified, input is taken from standard input (more later). • Advanced versions of grep allow using regular expressions in PATTERN.

21. Helper utilities • File “testfile” contains the following lines girtsf@linux girtsf $ cat testfile the quick brown fox jumped over the lazy dog • We search for “the”: girtsf@linux girtsf $ grep the testfile the quick brown the lazy dog • Only lines containing the substring “the” are printed.

22. Helper utilities • Some useful parameters for grep: • -i : ignore case (“the” finds “the”, “The”, “THE”,…) • -l : output only filenames that match, not the contents • -B <n> : output also n lines before the matching line • -A <n>: output also n lines after the matching line • See the man page (“man grep”) for all parameters

23. Helper utilities • tee - read from standard input and write to standard output and files • Syntax: tee [File1] [File2] .. [FileN] • Example of tee taking user’s input from terminal and writing to 3 files: girtsf@linux tmp $ tee a b c some string^D some string girtsf@linux tmp $ cat a some string girtsf@linux tmp $ cat b some string girtsf@linux tmp $ cat c some string Inred – my input, ending with Control-D, which is the EOF (End of File) character. This input is read as standard input by tee.

24. Helper utilities • Any program can be used as a helper program • More examples later

25. Connecting utilities with shell scripting • Standard I/O • I/O redirection to/from file • I/O redirection using a pipe • Backticks

26. Standard I/O • Every process, when run, has 3 already open data streams (file descriptors): • Standard input • Standard output • Standard error

27. Standard I/O • When run interactively (from command line), these streams are attached to the terminal they are running from • Standard input is attached to user’s keyboard input • Standard output is attached to user’s terminal output • Standard error, similarly to output, is attached to user’s terminal output • Usually referred to as stdin, stdout, stderr.

28. Standard output & error • “ls” command does not use input, but uses stdout, stderr. • The second line is the stdout from “ls” command: girtsf@linux etc $ ls -l /etc/passwd -rw-r--r-- 1 root root 2257 Oct 22 13:35 /etc/passwd • The second line is from stderr from “ls” command: girtsf@linux etc $ ls -l /etc/asdfasdf ls: /etc/asdfasdf: No such file or directory • Both stdout and stderr simultaneously: girtsf@linux tmp $ ls -l /etc/passwd /etc/asdfasdf ls: /etc/asdfasdf: No such file or directory -rw-r--r-- 1 root root 2257 Oct 22 13:35 /etc/passwd

29. I/O Redirection to/from file • By default, the 3 streams are attached to terminal • This can be overridden when executing the command and is called “redirection” • “>” specifies that stdout is redirected to file • “<“ specifies that stdin is taken from file • “2>” specifies that stderr is redirected to file

30. I/O Redirection to/from file • Syntax: • <cmd> [ > <file1>] [ < <file2> ] [ 2> <file3> ] • For those redirections that are specified, the respective stream will be attached to the specified file • None, one, two or all three types can be specified • If output file exists: > - replace file; >> - append to file

31. I/O Redirection to file • Example of stdout redirection to file girtsf@linux tmp $ ls -l /lib/ > direktorijas_saraksts girtsf@linux tmp $ cat direktorijas_saraksts total 4035 -rwxr-xr-x 1 root root 7488 Oct 6 12:33 cpp drwxr-xr-x 13 root root 1024 Oct 25 15:57 dev-state drwxr-xr-x 2 root root 1024 Jun 28 09:53 evms drwxr-xr-x 2 root root 2048 Aug 23 15:25 iptables ...

32. I/O Redirection to file • Example of stdout redirection to file girtsf@linux tmp $ ls -l /asdf > direktorijas_saraksts ls: /asdf: No such file or directory girtsf@linux tmp $ cat direktorijas_saraksts • The file is empty, as no output was sent to stdout, as error message was send to stderr, which still was attached to user’s terminal

33. I/O Redirection to file • Example of stderr redirection to file girtsf@linux tmp $ ls -l /asdfasdf 2> errlog girtsf@linux tmp $ cat errlog ls: /asdfasdf: No such file or directory • Now stderr was redirected to file and file contained the error message.

34. I/O Redirection to file • Example of stdout, stderr redirection to file girtsf@linux tmp $ ls -l /asdfasdf /lib 2>errlog >sar girtsf@linux tmp $ cat errlog ls: /asdfasdf: No such file or directory girtsf@linux tmp $ cat sar /lib: total 4035 -rwxr-xr-x 1 root root 7488 Oct 6 12:33 cpp drwxr-xr-x 13 root root 1024 Oct 25 15:57 dev-state drwxr-xr-x 2 root root 1024 Jun 28 09:53 evms drwxr-xr-x 2 root root 2048 Aug 23 15:25 iptables ...

35. I/O Redirection from file • Example of stdin redirection • First, we create file “a” with the following content the quick brown fox jumped over a quick brown fox • Use wc (word count) by not supplying the file name, but redirecting standard input girtsf@linux tmp $ wc < a 2 10 50

36. I/O Redirection with pipes • Task: given a file, output the total number of words in those lines, that contain substring “the”. • Example input: girtsf@linux girtsf $ cat testfile the quick brown fox jumped over the lazy dog • Lines 1 and 3 match, total number of words = 6.

37. I/O Redirection with pipes • Solution with redirection to files: • First find the lines, save them into temp file • Then use wc (word count) utility to count the number of words girtsf@linux girtsf $ grep the testfile > tmpfile girtsf@linux girtsf $ wc –w < tmpfile 6

38. I/O Redirection with pipes • Temporary file was used to redirect the standard output of grep to file • The standard input to wc was taken from temporary file • Easier way – connect the standard output of one program to standard input of another one directly

39. I/O Redirection with pipes • Syntax: program1 | program2 (| - pipe symbol) • Called “piping output from one program to another” • This example: girtsf@linux girtsf $ grep the testfile | wc -w 6 • No temporary files. Elegant!

40. Backticks • Backticks – reverse apostrophes “`” (usually the same key as tilde ~ character) • Using backticks sub-commands are executed, their result written in the place they are defined • Example: girtsf@linux tmp $ cd `echo /home` girtsf@linux home $ Substituted with “/home”

41. Helper utilities • We will examine the following utilities: • cut • sort • uniq • awk • sed

42. cut • cut - remove sections from each line of files • Syntax: cut [OPTION]... [FILE]... • Options: • -d DELIM : use DELIM instead of TAB character • -f LIST : output only these fields (delimited by DELIM) • -c LIST : output only these characters • See man page for more options

43. cut • Example – output second word on each line: • Delimiter: space “ “ • Fields: 2 girtsf@linux tmp $ cat a the quick brown fox jumped over a quick brown fox girtsf@linux tmp $ cut -f 2 -d ' ' a quick over

44. cut • Example – output characters 1-3, 5, 7-end • Use –c to choose the needed characters girtsf@linux tmp $ cat a thequick brown fox jumped over a quick brown fox girtsf@linux tmp $ cut -c 1-3,5,7- a theqick brown fox jume over a quick brown fox

45. sort • sort - sort lines of text files • sort [OPTION]... [FILE]... • Writes sorted concatenation of all FILE(s) to standard output. • Interesting options: • -r : reverse • -n : compare according to string numerical value • See man page for more options

46. sort • sort - sort text file reversed girtsf@linux tmp $ cat a fish dog animal bird girtsf@linux tmp $ sort -r a fish dog bird animal

47. sort • Sort numeric file (as text) girtsf@linux tmp $ cat a 5412 this line should go last 998 this line should go second 50 this line should go first 999 this line should go third girtsf@linux tmp $ sort a 50 this line should go first 5412 this line should go last 998 this line should go second 999 this line should go third

48. sort • Sort numeric file as numbers girtsf@linux tmp $ cat a 5412 this line should go last 998 this line should go second 50 this line should go first 999 this line should go third girtsf@linux tmp $ sort -n a 50 this line should go first 998 this line should go second 999 this line should go third 5412 this line should go last

49. uniq • uniq - remove duplicate lines from a sorted file • uniq [OPTION]... [INPUT [OUTPUT]] • Discards all but one of successive identical lines from INPUT (or standard input), writing to OUTPUT (or standard output). • Can be used together with sort, to get file without duplicate lines.

50. Just sorted: $ cat a | sort bird bird dog dog fish fish fly sort | uniq: $ cat a | sort | uniq bird dog fish fly uniq