Efficient File and Program Location in Unix

1. Introduction to Unix – CS 21 Lecture 13

2. Lecture Overview • Finding files and programs • which • whereis • find • xargs • Putting it all together for some complex programs

3. What Do You Do When You Want To Locate A File? • Option 1: ls • Can be very time consuming • Option 2: Other programs • which • Used to see what you are running • whereis • Used to see where programs are located • find • A general tool to find all sorts of files in the directory

4. which • Usage: which PROGRAMNAME • Tells you what executable is being run when you run a program • Useful to tell when more than one executable exists (i.e. one in /bin and one in /sbin) • Only searches your PATH

5. Example Of which

6. whereis • Usage: whereis PROGRAMNAME • Locates the source, executable, and man page for a command • Searches set paths

7. Example Of whereis

8. find • Locate files based on any number of factors • More than just the name • When it was modified • What permissions it has • What size it is

9. Running find • Usage: find [path] [expression] • A little different from all other programs you’ve run • Instead of a letter, options are the complete word • Example: • Instead of –n, you use -name

10. Expression Options To find • -name “pattern” • Matches the name of a file • -iname • Matches the name of a file but is case-insensitive • -empty • Matches empty files • -type • d • f • l

11. Examples

12. More Expression Options • -atime N • File was last accessed N days ago • -mtime N • File was last changed N days ago • +N, -N • +N = More than N days ago • -N = Less than N days ago

13. Examples

14. Combining Expression Options • -o • Or • -a • And • \( \) • Grouping

15. Action Commands • -print • Simply prints out the name of the file that was found • Most common action • -exec • Executes a command • -ok • Executes a command, but prompts the user first

16. The exec Option – A Powerful But Dangerous Option • -exec COMMAND \; • {} gets replaced with the name of the file that was found • Example: • find . –name “*.txt” –exec echo {} \; • Same as find . –name “*.txt” -print

17. Examples

18. When Would You Ever Do This? • The –exec option allows you to perform some command on all files that match a certain criteria • Example: • Someone cracked into your system and you want to check all recently modified files for any changes against a backup • find –mtime –2 –exec diff {} .snapshot/{}

19. Problem • The exec command will run the command once for every file that matches • A lot of overhead • May take a lot longer than you’d like • Solution: • xargs

20. The xargs Command • Take values from standard in and convert them into command line parameters • Usage: xargs COMMAND • Example: • xargs grep pattern

21. What? File1.txt File2.txt … Program File1.txt File2.txt … xargs stdin Program

22. Why? • When a program outputs a list that will be used as input to another program • Used mainly in pipes with find • Example: • find . –name *.txt | xargs grep ‘Jason’ • find . –name *.txt –exec grep ‘Jason’ {} \; • Calls grep many, many times less than the second call and takes less time

23. Another xargs example • Problem: You want to kill all of your processes • First, list all your processes • ps aux | grep login • Next, select all the PIDs • cut –d’ ‘ –f2 • Now pass all those PIDs to kill • ps aux | grep login | cut –d’ ‘ –f2 | xargs kill -9

24. A Simple Cleanup Script #!/bin/bash ps aux | grep login | cut –d’ ‘ –f2 | xargs kill -9 exit 0 >./exitScript.sh &

25. When Would You Use Bash Versus Awk? • What is bash good at? • Finding and manipulating files and directories • What is bash bad at? • Manipulating data inside files • What is awk good at? • Manipulating data inside files • What is awk bad at? • Finding and manipulating files and directories

26. Building A Complex Program • Start small • Always try to get something working before you get something working well • Build up • Add functionality one part at a time and test as you go

27. Building A Complex Program In Bash • Starting small – #!/bin/bash exit 0 • Slowly add functionality #!/bin/bash read input echo $input exit 0

28. Building Up A Complex Program – In awk • Start small { print $0 } • Slowly add functionality /Pattern/ { print $0 }

29. Hints • Break the problem down into smaller components • Try printing first – always print as you go • Insert temporary files or variables if needed • Example: • If you are required to process any number of command line parameters one at a time, start by handling just one • After you can handle one, try two • Generalize after you have gotten the first part down

30. Example

31. Computing Problem • Find all files we haven’t changed in 2 days and move them to a folder called oldStuff • Should we use bash or awk? • Bash shell programming would be better

32. Break It Down • Find all files we haven’t changed in two days • find ~ –mtime +2 -print • Move each file to the oldStuff folder • mv file ~/oldStuff/

33. Put It Together • We need some way to connect the first part with the second part • How about a variable? • Store the list of file names and then go through them one by one • var=`command`

34. Script Solution #!/bin/bash filesToMove = `find ~ -mtime +2 –print` for f in $filesToMove do mv $f ~/oldStuff/ done exit 0

35. Another Computing Problem • Remove all files that have the phrase “Remove Me” located on the first line • Which should we use, bash or awk? • We are looking inside files, but we aren’t manipulating the data, so we should use bash.

36. Break It Down • Find all files • find . –name “*” -print • Get the first line • head -1 • Look for “Remove Me” • grep “Remove Me” • Remove the file if “Remove Me” was there • rm file

37. Put It Together • Variables to pass information along • List of all files • Temporary files or variables to store results and test • Results from grep

38. One Script Solution #!/bin/bash allFiles=`find . –name “*” –print` for f in $allFiles do head –1 $f | grep “Remove Me” > .tmpFile if [ -s .tmpFile ] ; then rm –i $f fi done rm –f .tmpFile exit 0

39. Another Computing Problem • Tax calculator • Given a receipt, calculate the tax • Receipt is a file containing the name of the product and price of the product on one line • Pork Tenderloin:10.85 • Should we use bash or awk? • awk would be better in this case

40. Break It Down • Calculate the tax on an individual item • Get the item • $2 • Multiply it by the tax • $2 * .0775 • Sum up all the taxes

41. Put It Together • To keep a sum, we need to store the running total in some location • How about a variable? • totalTax = totalTax + newTax • We need to initialize the tax in the beginning and print out the tax at the end

42. Awk Script NF == 2 { totalTax = totalTax + $2 * .0775 } BEGIN { totalTax=0 } END { print totalTax }

43. Problem 4 • Given a file that contains inventory, print out all items that are out of stock • File format: ITEM/PRICE/QUANTITY • Example: Needlenose pliers/2.50/4 • Should we use bash or awk? • Awk is better this time since we are checking the data

44. Script Solution { if ($3 == 0) { print $1 } }

45. Next Time • Quiz #2 • Bash and awk programs • sed • Perl Programming Introduction