UNX122 – File Manipulation

1. UNX122 – File Manipulation • File and Directory utilities: ls, ls -a, ls -F, ls -l, ls -ld, cat, less, more, rm -r, rm -rf, cp, mv,cd, mkdir, rmdir, mkdir -p, df, du • Useful commands: echo, date, who, w • File manipulation: head, tail, sort, grep, uniq, diff, tr • Command aids: find, file, which, whereis, locate • Communication: write, talk, mesg, mail, finger, ftp • Printing: lpr, lpq, lprm • Computation: bc

2. File Name Expansion: *, ?, [], [!], • Special characters for File Name Expansion: *, ?, [ ], [^ ] • * = Match zero or more of any character. • ? = Match one character only. • [] = Match any one character enclosed. • [a-z] = Match any lower case character from a to z. • [!] = NOT match any character enclosed. • [!A-Z] = NOT match an upper case character from A to Z.

3. Demo Directory structure ~/unx122/outline/ -- marks -- class -- policy /course/Day1/ -- presn1.ppt -- lab1.doc -- Lab2.doc /course/day2/ -- basics.ppt -- commands.doc /course/Day3/ -- 1answer -- 2answer -- 3answer /sales/ -- .proposal.txt -- .secrets.doc 1. mkdir –p unx122 2. mkdir –p unx122/outline 3. mkdir –p unx122/course/Day1 4. mkdir –p unx122/course/day2 5. mkdir –p unx122/course/Day3 6. mkdir –p unx122/sales Create Directories 1. cd unx122/outline 2. touch marks class policy 3. cd ~/unx122/course/Day1 4. touch presn1.ppt lab1.doc Lab2.doc 5. cd ~/ unx122/course/day2 6. touch basics.ppt commands.doc 7. cd ~/ unx122/course/Day3 8. touch 1ans 2ans 3ans 9. cd ~/ unx122/sales 10. touch .proposal.txt .secrets.doc Create Files

4. cat, more • cat <filename> = view contents of a file • Options: • -n = show line numbers • cat > <filename> = send keyboard entries to filename • more <filename> = display contents of a file, allows scrolling one screen at at time. • [spacebar] = to see next screen • f = forward one screen • b = backward one screen • q = quit more display

5. cp <sourcefile> <targetfile> = copy one file to another file changing the filename or location. • Options: • -r = copy all subdirs also • -i = interactive – prompt overwrites • mv <path/source> <path/target> • Options: • -i = interactive – prompt overwrites • -f = non-interactive – no prompting • ln –s <actual filename> <linkname> = create a soft link • ln <actual filename> <linkname> = create a hard link • inode – info about a file, owner, type, permissions & address cp, mv, ln

6. Hard Links vs. Soft Links • Hard Links • Points to storage Inode number of a file • Link count shows number of hard links to the file • Hard link count reduced when link is deleted • Cannot span File System boundaries • Listed with “-” in long listing • Soft Links • Points to pathname of a file • Link is NOT deleted when original file is deleted • Link count is 1 • Can span File System boundaries • Identified by “->” in long listing, pointing to linked file • Listed with “l” in long listing

7. cd, mkdir, rmdir , rm • cd <path> = change current directory to directory at path • mkdir <dirname> = create a directory • options: • -p = create all intermediate directories. • rmdir <dirname> = remove an empty directory • rm <path/dirname> = remove a directory and subdirectories • options: • -r = recursive ** use with CAUTION • -i = interactive – prompts overwrite • -f = non-interactive – no prompting

8. df, du • df = shows total space available in a file system • options: • -k = display space in kilobytes • du = shows space used up by each directory and all its subdirectories. • options: • -a = disk usage for each file • -k = disk usage in kilobytes • -s = total disk usage for all files specified

9. Quoting and echo • Quoting: • ‘ = single quotes - remove special meaning • “ = double quotes - strings and retain special meaning • ` = back quotes - execute commands included • echo = display text to the terminal (STDOUT) • Eg. echo Hello there how are you • Options: • \n = new line character • \t = print tab • \c = continue next output in the same line

10. Useful commands: date • date = display current date and time • Eg. date ‘+%A %d-%b-%Y’ = Saturday 01-Feb-2003 • Options: • %A = Day of week • %d = Day number • %b = Short month name • %B = Full month name • %m = Month number • %Y = Full 4 digit year

11. Useful commands: who, finger • who = show all users currently logged into the UNIX system. Shows username, terminal type, login date & time, remote machine. • Options: • -q = show usernames and remote machine name • -H = show column headers • finger = show username, Full Name, terminal ID and Idle time.

12. Standard Input, Standard Output, Standard Error and Filters • STDIN = input to any program or command • STDOUT = output from any program or command • STDERR = error messages • Filter = a program which accepts input from STDIN and writes output to STDOUT. • Filters = grep, cut, tee, tr • eg. cat /etc/profile | grep “apache$” • eg. head -500 /etc/passwd | tail -5 | cut -d”:” -f1

13. Redirection: • Redirection symbols (<, >, >>) work with files only. • File redirection • eg. > = overwrite output • >> = append output • < = input • 2> = overwrite error output • 2>> = append error output

14. Pipes and tee • Pipe: | = pipe symbol separating two commands where the output of the first command is automatically redirected to the input of the next command. • Pipes | = piping symbol • eg. ls -al | grep “^d” • tee –a <filename> = appends a copy of output to <filename> as well as display to STDOUT (terminal). • Note: This command is usually used together with piped input from another command.

15. File manipulation: grep • grep <options> <string> = search for a string pattern whether contained in files or from other forms of input (eg. keyboard). • Eg.: grep “root” /etc/passwd = show all lines containing the string “root”. • Options: • -c displays a count of # of matching lines • -i ignore case • -v displays lines that do not match the pattern • -w = match entire words only

16. File manipulation: sort • sort <filename> = arrange lines of <filename> in ascending or descending order. • Eg. sort /etc/passwd = rearrange the order of lines in the file “/etc/passwd” and send output to screen (STDOUT). • Options: • -r = sort in reverse (descending order) • -n = sort numerically • +0 = sort on first field (field #’s start from 0) • -t “:” = set field separator to “:” (colon) • -u = sorts and only displays unique lines • -o <filename> = sort and send output to <filename>

17. File manipulation : cut • cut <filename> = extract one or more columns of data from STDIN or filename. • Eg.: cut –d”:” –f5 /etc/passwd = to extract only the 5th. column from the file /etc/passwd. • options: • -cx-y = cut characters from position x to position y of each line of input. • -d”:” = use “:” as field delimiters, default is a tab • -f<n> = cut the nth. field from each line of input (field #’s start from 1).

18. File manipulation : head, tail, tr • head -n <filename> = display first “n” lines of a file, default = 10 • Eg.: head –5 /etc/group = show first 5 lines of file /etc/group. • tail -n <filename> = display last “n” lines of a file, default = 10 • tr <string1> <string2> = change characters in string1 into characters from string2, works with Standard Input (STDIN) and Standard Output (STDOUT) • Eg.: cat /etc/passwd | tr : X | more = view the contents of /etc/passwd, change all “:” to “X”, then display one screen of text at at time.

19. File manipulation : uniq, diff • uniq <filename> = display unique lines from a <filename>, used after a sort. Duplicates are not shown. • Note: You must use the sort command before using uniq. • diff <file1> <file2> = shows the differences between each line of two files. Shows no output if files are identical. • Output shows “c” = change, “a” = add, “d” = delete.

20. find <path> -name <filename pattern> = print names of all filenames matching < filename pattern>, starting search from <path>. • Note: < filename pattern> must be in quotes (“”). • Eg.: find . –name “a*” = print all file names starting with the letter “a” starting to search from the current directory. • options: • -type <f or d> = f = files, d = directories • -user <username> = find files of a particular user name • -exec <command string> {} \; = execute a command on all files found. Note: you must have “{} \;” after the command string. • -perm <octal value> = find files with listed permissions. Command info: find

21. Command info: find - Examples • find . -name "pho*" -exec cp {} ~/fol/ \; - find all file names starting with “pho” and copy to the folder named “~/fol”. • find . -name “unx*" -exec more {} \; - find all file names starting with “unx” and view each file with “more”. • find . -name “unx*" -user “hlee” -exec mv {} . \; - find all file names starting with “unx” which belong to user “hlee” and move them to the current directory. • find / -name “phon*" -type d -exec mv {} . \; - starting from root, find all folder names starting with “phon” and move them to the current directory.

22. Command info: file, which • file <filename> = show type of file <filename>, could be a command (executable) or contain ASCII text only. • which <command> = searches the entire path for a command, showing the file location. • Eg.: which cp = where is the command cp located.

23. Communication: write, talk, mesg • write <username> = write a message to a user who is currently logged on to your system. • options: • Ctrl+d = end message transmission • talk <username> = starts a full screen chat session. • Ctrl+z = exit from chat session. • mesg y = turn messages on, so you could receive messages • mesg n = turn messages off, you do not receive messages

24. Communication: mail • mail <mail address> <filename> = to send e-mail to <mail address> containing text from <message file >. • Eg.: mail –s “Hello anyone” anybody@hotmail.com < hello.txt = to send mail to anybody@hotmail.com with subject “Hello anyone” and message body coming from file hello.txt • Options: • -s “Enter subject here” = place this string in subject line. • -f <filename> = get message text stored in <filename>.

25. Communication: ftp • ftp <ftp server> = file transfer to/from your machine to a remote < ftp server >. You should have a username and password to access the FTP server. • Eg.: ftp people.senecac.on.ca = open an FTP session with an FTP server named people.senecac.on.ca to U/L or D/L files. • Options: • bin = start binary mode transfer (no file conversion). • ls = dir listing showing file names only. • dir = dir listing showing all file details. • get <filename> = download <filename> to your PC • put <filename> = upload <filename> from your PC • bye = exit FTP session

26. Printing: lpr, lpq, lprm • lpr <filename> = print <filename> to the default print queue. • lpq = show all outstanding print jobs and their status. • lprm <job number> = remove jobs from the printer spooling queue

27. wc <filename> = count the number of characters, words and lines in <filename>. • Eg.: wc /etc/group = show the number of characters, words and lines in file /etc/group. • Options: • -c = count characters only • -w = count words only • -l = count lines only • bc = start a Unix command line calculator • Options: • scale=2 = means set the number of decimals to 2. • Ctrl+d = to end the calculator session. Computation: wc, bc

28. File Permission: umask • Files permissions are “umasked” with 666 • Directory permissions are “umasked” with 777 • umask <octal> = set default file and directory permissions when created. • Eg.: umask 023 = set the default file permissions as 643 or rwx, r-x, r--

29. chmod = change file permissions at any time. • Eg.: chmod 764 <filename> = set file permissions to 764 or rwx, rw-, r-- • Options: • -R = recursive through all subdirs. • -v = verbose, show actions. chmod u=rwx <filename> = set user to rwx • chmod u=rwx <filename> = set user to rwx • chmod g=rwx <filename> = set group to rwx • chmod o=+x <filename> = add x to other do not change r, w • chmod a=+r <filename> = add r to all (user, group, other) • chmod o=-w <filename> = remove w to other only File Permission: chmod

30. Utilities: ps, kill • ps = get a list of status of current processes. • Eg.: ps -ef = show environment and full listing of all running processes • Options: • -e = show environment • -f = full listing • kill <process id> = end a process. • Eg.: kill -9 <process id> = terminate <process id>. • Options: • -9 = forced kill.

31. Utilities: &, bg, fg, Ctrl+z, jobs • <command> & = execute <command> in the background. • Eg.: find / –name “sales*” & = start from root to find all files starting with “sales” and execute in the background. • Ctrl + z = suspend the current command execution. • bg = send the current job to the background • fg = bring most recent background job to the foreground • jobs = list all job numbers being executed. • kill %<job number> = terminate <job number>