Bioinformatics Programming

1. Bioinformatics Programming EE, NCKU Tien-Hao Chang (Darby Chang)

2. In the last slide • Terminology • Unix, Linux and UNIX • Linux vs. Window • Unix-like system • commands, permissions, shell • cd, ls, du, ln, sort, find, tar, wc, … • Ubuntu • Shell script • gcc

3. http://brownblog.info/wp-content/uploads/2008/02/ignore.JPG Other features worth to mention

4. More about Shells

5. ShellsShell Startup • The file .profile (sh) or .login (csh) is used at login to: • set path • define functions • set terminal parameters (stty) • set terminal type • Other files such as .bash_profile and .bashrc. You can man shell ($ man bash), since shell is also a program, for more information.

6. ShellsSample .profile File • PATH=/usr/bin:/usr/ucb:/usr/local/bin:. • export PATH • PS1="{ ‘hostname‘ ‘whoami‘ }" • ls() { /bin/ls -sbF "$@"; } • ll() { ls -al "$@"; } • stty erase ˆH • eval ‘tset -Q -s -m ’:?xterm’‘ • umask 077

7. Shells.login and .cshrc • .login runs only at login time • tell whether you have mail • tell who else is online • configure terminal settings • .cshrc runs whenever the shell starts • set environment and shell variables • set aliases • Other advanced shells inherit similar concepts. For example, the corresponding files in bash are .bash_profile and .bashrc, respectively.

8. ShellsJob Control • Putting a job into the background by appending & to the command line • ˆZ to stop while job is running • bg continue stopped job in background • fg return the job to the foreground • jobs list background jobs • kill kill a background job

9. ShellsHistory • C Shell, Korn shell and others retain information about former commands executed within the shell • Use history and savehist variables to set number of commands retained: • in .cshrc: • set history=100 savehist=50 • saved in ˜/.history between logins • Examples • $ history nn # prints last nn commands • $ !! # repeats the last command • $ !nn # repeats the command numbered nn • $ !string repeats latest command starting with string

10. ShellsChanging your Shell • $ chsh /bin/sh • The new shell must be the full path name • Frequently standard shells: • Bourne /bin/sh • Korn /bin/ksh • C /bin/csh • Alternate shells should be listed in /etc/shells • tcsh (/bin/tcsh) and bash (/bin/bash) are the most common alternatives • To try some other shell, type it at the system prompt • useful when you want to check some compatibility • type exit to return to normal

11. Special Unix Features

12. Special Unix FeaturesI/O Redirection and Piping • Output redirection to a file • Input redirection from a file • Piping — output of one command becomes the input of a subsequent command • Standard File Descriptors • stdin standard input to the program • stdout standard output from the program • stderr standard error output

13. > redirect standard output to file • $ command > outfile • $ ls > foo • >> append standard output to file • $ command >> outfile • $ echo ‘foo’ >> foo • < input redirection from file • $ command < infile • $ sort < foo • less useful since most commands accept filenames as arguments • | pipe output to another command • $ command1 | command2 • $ ls | sort

14. >& redirect stdout and stderr to file • >>& append stdout and stderr to file • |& pipe stdout and stderr to command • 2> redirect stderr to file • >file 2>&1 redirect both stdout and stderr to file • >>file 2>&1 append both stdout and stderr to file • 2>&1|command pipe stdout and stderr to command • To redirect stdout and stderr to two separate files: • $ (command > outfile) >& errfile • $ command > outfile 2> errfile • To discard stderr: • $ command 2 > /dev/null • /dev/null is a “black hole” for bits

15. Special Unix FeaturesOther Symbols • ; command separator • & run the command in the background • && run the following command only if previous command completes successfully • || run the following command only if previous command did not complete successfully • () grouping, commands within parentheses are executed in a subshell

16. Special Unix FeaturesQuoting • \ escape the following character (take it literally) • $ echo \”\” • ‘’ don’t allow any special meaning to characters within single quotes (except ! in csh) • $ echo ‘shell is $SHELL’ • “” allow variable and command substitution inside double quotes (does not disable $ and \ within the string) • $ echo “shell is $SHELL” • `command‘backquotes take the output of command and substitute it into the command line (works inside double-quotes) • $ echo `ls`

17. Special Unix FeaturesWild Cards • ? match any single character • * match any string of zero or more characters • [abc] match any one of the enclosed characters • [a-z] match any character in the range a through z • [!def] (sh) match any characters not one of the • [ˆdef] (csh) enclosed characters • {ab,bc,cd} match any set of characters separated by comma • ˜ user’s own home directory • ˜user home directory of specified user

18. Remember These symbols may vary from shell to shell — see the man pages

20. Screen

21. http://www.mergersandinquisitions.com/wp-content/uploads/2008/05/alttab-key.jpghttp://www.mergersandinquisitions.com/wp-content/uploads/2008/05/alttab-key.jpg No Alt-Tab to switch between programs

22. Screen • Screen is best described as a terminal multiplexer • Prevent multiple terminal emulators • More important, to live with sessions rather than terminals • To start • $ screen

23. ScreenCommands • In screen, all commands begin with ^a (Ctrl+a) • ? help • c create a new window, each created window is assigned with a number • w list current windows • [0-9] switch window by number • - switch to an empty window (boss coming) • n, [space] switch to the next window • p, [backspace] switch to the previous window • a, ^a switch to the last window (recall button) • A name the current window • ', " switch window by name • x, ^x lock the current window

24. ScreenSessions • Each screen (and all the associated window) is a session • When you type ‘screen’, you start a session. Then you use ‘^a c’ to create some windows. The status (screen, connections, …) of all these windows are logged by the session. • Commands • ^a d detach the current session • ^a DD detach the current session and logout • ^a z, ^a ^z make the current session background, of course you can use ‘fg’ to restore it

25. Options • $ screen -ls list sessions • $ screen -d [pid] detach a remote session (which is attached, but not to the current terminal) • $ screen -D [pid] detach and logout a remote session • $ screen –r [pid] reattach a session • $ screen –R reattach the youngest session, create a new one if necessary • $ screen –x attach to a not detached screen session (multi display mode) • $ screen -d -r reattach a session and if necessary detach it first • $ screen -D -R attach here and now • In detail this means: If a session is running, then reattach. If necessary detach and logout remotely first. If it was not running create it and notify the user. This is the author's favorite. • $ screen -wipe clean dead sessions

27. Relation among Session, terminal and window in screen. One to many, many to one, or many to many? A drawing would be necessary.

28. Text Processing

29. Text ProcessingEditors • vi • Visual Editor • No alternative (except you choose emacs), it is the best way to force you guys to learn vi • Pronounce both letters: V-I, never “Vy” • Three modes • Command mode (“beep mode”) • Insert mode (“no beep mode”) • Command line mode (“colon mode”) • Commands are generally case sensitive

30. viCursor Movement • h, j, k, l alternates for arrows • [n]h left [n]character(s) • [n]j down [n] character(s) • [n]k up [n] line(s) • [n]l down [n] line(s) • ˆF forward one screen • ˆB back one screen • ˆD down half screen • ˆU up half screen • G go to last line of file • $ end of current line • ˆ beginning of text on current line • 0 beginning of current line • [n]w forward [n] word(s) • [n]b back [n] word(s)

31. viInserting/Deleting Text • i insert text before the cursor • a append text after the cursor • I insert text at beginning of line • A append text at end of line • o open new line after current line • O open new line before current line • dd delete current line • [n]dd delete [n] line(s) • [n]dw delete [n] word(s) • D delete from cursor to end of line • x delete current character • [n]x delete [n] characters • X delete previous character (like backspace) • Confused? Remember range command unit philosophy

32. viChange Commands • cw change word • [n]cw change next [n] word(s) • c$ change from cursor to end of line • ˜ change case of character • J joins current line and next line • u undo the last command just done • . repeat last change • [n]yy yank [n] line(s) to buffer • [n]yw yank [n] word(s) to buffer • p puts yanked or deleted text after cursor • P puts yanked or deleted text before cursor

33. viFile Manipulation • :w write changes to file • :wq write changes and quit • :w! force overwrite of file • :q quit if no changes made • :q! quit without saving changes • :! shell escape • :r! insert result of shell command at cursor position

35. How to Change the next 10 lines?

36. What is dG

37. Text Processing Commands

38. Text Processing Commandsgrep • grep — search the argument for all occurrences of the search string • grep [option]... regexpfile • search for the number 15 • $ grep '15' file • count the number of lines matching the search criterion • $ grep-c '15' file • search for lines not matching the search criterion • $ grep-v '15' file • search for 11, 12 or 15 • $ grep '1[125]' file • search for all lines that begin with a space • $ grep '^ ' file • search for lines begin with the characters 1 through 9 • grep '^[1-9]' file

39. Text Processing CommandsAdvanced grep Examples • $ wget -q -O- http://url.to.web/ | grep 'a href' | head • list the first 10 links of a given web page • wget • power of pipe • $ grepMemTotal /proc/meminfo • $ grep 'model name' /proc/cpuinfo • show RAM and CPU info • everything is a file • $ set | grep $USER • set • using environment variables

40. Text Processing Commandssed • sed — stream editor for editing files from script or command line • sed [option]... edit_command file • changes all incidents of a comma into a comma followed by a space • $ sed 's/,/, /g' file • filter for lines containing ‘Date: ’ and ‘From: ’ and replace these without the colon (perform multiple operations) • $ sed -e 's/Date: /Date /' -e 's/From: /From /‘ • print only those lines of the file from the one beginning with "Date:" up to, and including, the one beginning with "Name:" • $ sed -n '/^Date:/,/^Name:/p‘

41. Text ProcessingCommands • awk — scan for patterns in a file and process the results • awk program file • $ cat /etc/passwd | tr a-z A-Z | awk -F: '{printf ("user %-16s %c %5d %5d\n",$1,$2,$3,$3)}‘ • $ awk 'BEGIN { x=0 } /^$/ { x=x+1 } END { print "I found " x " blank lines. :)" }' file • awk is good at column processing • awk is almost a script language

43. How Much Do you remember?

44. 2 Things You Should Learn The way to interpret these magic commands Remember that there are such facilities

45. What to Do First When you see this $ sed -n 1,10p

46. http://farm1.static.flickr.com/34/108805307_c43af20f59.jpg Google? Sure But you should try man (or --help) first

47. $ sed -n 1,10p • It is similar to head • Then, why not use head? • Suppose that if you want the ninth and tenth lines • $ sed -n 9,10p

48. $ sed -n 1,10p It is similar to head. Then, why not use head? Suppose that if you want the ninth and tenth lines. $ sed -n 9,10p

49. UNIXThe Last Reminding • How to remember these? • Just use them! Force yourself to use them! • Briefly speaking • want to search some lines? try grep • want to edit some lines? try sed • want to handle columns? try awk • something awk script cannot do (I doubt), try perl one-liner

50. Our UNIX tutorial is finished