The Linux Audit Framework

1. The Linux Audit Framework Gary Smith, EMSL, Pacific Northwest National Laboratory

2. A Little Context • Cyber Security is all about managing risk. • How do you think about managing risk? • The Five Golden Principles of Security • Know your system • Principle of Least Privilege • Defense in Depth • Protection is key but detection is a must. • Know your enemy. 2

3. Introduction • Linux audit helps make your system more secure by providing you with a means to analyze what is happening on your system in great detail. • It does not, however, provide additional security itself—it does not protect your system from code malfunctions or any kind of exploits. • Instead, Audit is useful for tracking these issues and helps you take additional security measures, like SELinux, to prevent them. • Audit consists of several components, each contributing crucial functionality to the overall framework. • The audit kernel module intercepts the system calls and records the relevant events. 3

4. Introduction (cont.) • The auditd daemon writes the audit reports to disk. • Various command line utilities take care of displaying, querying, and archiving the audit trail. 4

5. Linux Audit Framework Capabilities • Audit enables you to do the following: • Associate Users with Processes • Audit maps processes to the user ID that started them. • This makes it possible for the administrator or security officer to exactly trace which user owns which process and is potentially doing malicious operations on the system. • Review the Audit Trail • Linux audit provides tools that write the audit reports to disk and translate them into human readable format. • Review Particular Audit Events • Audit provides a utility that allows you to filter the audit reports for certain events of interest. 5

6. Linux Audit Framework Capabilities (1) • You can filter for: • User • Group • Audit ID • Remote Hostname • Remote Host Address • System Call • System Call Arguments • File • File Operations • Session • Success or Failure 6

7. Linux Audit Framework Capabilities (2) • Apply a Selective Audit • Audit provides the means to filter the audit reports for events of interest and also to tune audit to record only selected events. • You can create your own set of rules and have the audit daemon record only those of interest to you. • Prevent Audit Data Loss • Audit provides several mechanisms to prevent the loss of audit data in the event of a loss of system resources. 7

8. The Components of Linux Audit 8

9. Configuring The Linux Audit Framework • Before you can actually start generating audit logs and processing them, you must configure the audit framework. • Julius Caesar said, “Gallia est omnis divisa in tres partes”, and just like Gaul, the configuring the audit framework is divided into three parts: • The Audit Daemon Configuration • The Audit Rules • The Audispd Daemon Configuration 9

10. /etc/audit/auditd.conf • The /etc/audit/auditd.conf configuration file determines how the audit system functions once the daemon has been started. • For most use cases, the default settings shipped with the package should suffice. • Let’s take a look at a sample auditd configuration file. 10

11. A Sample auditd.conf log_file = /var/log/audit/audit.log log_format = RAW log_group = root priority_boost = 4 flush = INCREMENTAL freq = 20 num_logs = 5 dispatcher = /sbin/audispd disp_qos = lossy name_format = NONE ##name = mydomain max_log_file = 6 max_log_file_action = ROTATE space_left = 75 space_left_action = SYSLOG action_mail_acct = root admin_space_left = 50 admin_space_left_action = SUSPEND disk_full_action = SUSPEND 11

12. Setting Up Audit Rules • We’ve given auditd its marching orders; now we have to define what we are interested in auditing. • Audit rules are used to specify which components of your system are audited. • There are three basic types of audit rules: • Basic audit system parameters • File and directory watches • System call audits • Before creating an audit rule set and before rolling it out to your system, carefully determine which components to audit. • Extensive auditing can cause a substantial logging load. • Remember: First match wins! 12

13. Setting Up Audit Rules (1) • Make sure that your system provides enough disk space to store large audit logs and test your audit rule set extensively before rolling it out to production. • Audit rules can either be passed to the audit system by the command line using auditctl or bundled into a rules file located under /etc/audit/audit.rules that is read during the start of the audit daemon. 13

14. A Sample audit.rules # basic audit system parameters -D -b 8192 -f 1 -e 1 # some file and directory watches -w /etc/audit/auditd.conf -p rxwa -w /etc/audit/audit.rules -p rxwa -w /var/log/audit/ -w /etc/passwd -p rwxa -w /sbin/auditctl –p x # an example system call rule -a entry,always -S umask 14

15. File Watches Caveats • Directory watches produce less verbose logs than exact file watches. • When in need of detailed file-related records, enable separate file watches for all files of interest. • Pathname globbing of any kind is not supported by audit. Always use the exact pathnames. • Auditing can only be performed on existing files. • Any files added while the audit daemon is already running are ignored until the audit rule set is updated to watch the new files. 15

16. Assigning Keys to Rules • Assigning keys to your audit rules helps you to identify any records related to this rule in the logs. An example rule plus key: -w /etc/selinux -k MAC-Policy • You may use the same key on different rules in order to be able to group rules when searching for them. • It is also possible to apply multiple keys to a rule -w /sbin/auditctl -p x -k privileged -k ids-exec-info • Using the ausearch log analyzer, you can easily filter for any events related to this particular rule. 16

17. Auditing the Execution of Setuid/Setgid Binaries • Let’s say that as a matter of compliance, you have to audit the execution of setuid/setgid binaries on your system. • How do you do set that up? • First, run a script like this at boot time from /etc/rc.localsending the output to a temp file, /tmp/snorf, for example. 17

18. Auditing the Execution of Setuid/Setgid Binaries (1) #!/bin/bash # Find all the file systems that are locally mounted for i in `/bin/egrep '(ext4|ext3|ext2)' /etc/fstab | /bin/awk '{print $2}'` do # Find all the files on the file system found above and print out # and audit rule for it /usr/bin/find $i -xdev -type f \( -perm -4000 -o -perm 2000 \) -print | \ /bin/sort | /bin/awk '{ print "-a always,exit -F path=" $1 " -F perm=x \ -F auid>500 -F auid!=-1 -k privileged -k ids-exec-high" }' done 18

19. Auditing the Execution of Setuid/Setgid Binaries (2) • And you get something like this (YMMV depending on what’s installed). -a always,exit -F path=/bin/fusermount -F perm=x -F auid>500 -F auid!=-1 -k privileged -k ids-exec-high -a always,exit -F path=/bin/ping -F perm=x -F auid>500 -F auid!=-1 -k privileged -k ids-exec-high -a always,exit -F path=/bin/ping6 -F perm=x -F auid>500 -F auid!=-1 -k privileged -k ids-exec-high -a always,exit -F path=/bin/su -F perm=x -F auid>500 -F auid!=-1 -k privileged -k ids-exec-high -a always,exit -F path=/usr/bin/chage -F perm=x -F auid>500 -F auid!=-1 -k privileged -k ids-exec-high -a always,exit -F path=/usr/bin/chfn -F perm=x -F auid>500 -F auid!=-1 -k privileged -k ids-exec-high -a always,exit -F path=/usr/bin/chsh -F perm=x -F auid>500 -F auid!=-1 -k privileged -k ids-exec-high -a always,exit -F path=/usr/bin/crontab -F perm=x -F auid>500 -F auid!=-1 -k privileged -k ids-exec-high 19

20. Auditing the Execution of Setuid/Setgid Binaries (3) • Then, point auditctl at the temp file to add the newly created audit rules. • The auditctl program is used to control the behavior, get status, and add or delete rules into the kernel’s audit system. /sbin/auditctl –R /tmp/snorf • A couple of things about auditctl: • auditctl is not a filter, so output cannot be piped into it. • Rules files for auditctl must be owned by root. 20

21. System Call Auditing • System call auditing lets you track your system's behavior on a level even below the application level. • The audit subsystem supports an ample collection of events, to include the tracing of arbitrary system calls identified by system call name, or by system call number. • The audit subsystem can also filter by PID, UID, system call success, system call argument, and many other possibilities. • When designing these rules, consider that auditing a great many system calls may increase your system load and cause you to run out of disk space. • Remember: First match wins! 21

22. System Call Audit Rules Examples • -a always,exit –S settimeofday • -a always,exit -F arch=b64 -S chmod • -a always,exit -F arch=b64 -S chown -F exit=-EACCES • -a always,exit -F arch=b64 -S creat -F exit=-EACCES –F uid>=500 • -a always,exit -F arch=b64 -S unlink -F auid>=500 -F auid!=-1 • -a always,exit -F arch=b64 -S init_module -S delete_module –k modules –k ids-sys-info • -a always,exit -S execve –F auid>500 -F uid=0 –k root-exe • -a never,exit -F path=/somefile.dat -S unlink -S unlinkat -S rename -S renameat • For details on setting up audit rules, see auditctl(8). 22

23. Creating Audit Reports • The audit records are stored in /var/log/audit/audit.log. • grep is your friend and you can pull stuff out of the audit log and get stuff like this: • type=SYSCALL msg=audit(1365719016.212:333043): arch=c000003e syscall=171 success=yes exit=0 a0=7fff86310c37 a1=6 a2=d a3=7fff8630f3b0 items=0 ppid=22300 pid=22311 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=37491 comm="domainname" exe="/bin/hostname" subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 key=73797374656D2D6C6F63616C65016964732D7379732D6C6F77 • Maybe grep isn’t your friend, after all. • The raw audit data auditd stores in the /var/log/audit directory is quite complex. • To find what you want, you might have to sift through bazillions of other events before you locate the one that you want. 23

24. Creating Audit Reports (1) • Use aureport to create concise, human-readable reports. • Some of the useful options are: • --summary • --failed • --start and --end (aureport understands today, yesterday, now, recent, this-week, this-month, and this-year) • --auth, --avc, --login, --user, --executable, --syscall • To get started, do aureport –summary and you get something like this: 24

25. Creating Audit Reports (2) Summary Report ====================== Range of time in logs: 03/01/2013 08:17:01.765 - 03/11/2013 16:07:49.252 Selected time for report: 03/01/2013 08:17:01 - 03/11/2013 16:07:49.252 Number of changes in configuration: 669 Number of changes to accounts, groups, or roles: 10 Number of logins: 11 Number of failed logins: 2941 Number of authentications: 38 Number of failed authentications: 5 Number of users: 3 Number of terminals: 10 Number of host names: 6 Number of executables: 20 Number of files: 597 Number of AVC's: 347 Number of MAC events: 11 Number of failed syscalls: 76 Number of anomaly events: 0 Number of responses to anomaly events: 0 Number of crypto events: 11882 Number of keys: 20 Number of process IDs: 17444 Number of events: 120152 25

26. Creating Audit Reports (3) • Lets look at some of the failed logins with aureport –l –failed: Login Report ============================================ # date time auid host term exe success event ============================================ 1. 03/07/2013 15:00:37 root 192.168.1.101 ssh /usr/sbin/sshd no 178116 2. 03/07/2013 15:01:52 root 192.168.1.101 ssh /usr/sbin/sshd no 178135 3. 03/07/2013 15:01:54 (unknown user) 192.168.1.101 ssh /usr/sbin/sshd no 178146 4. 03/07/2013 15:01:54 (unknown user) 192.168.1.101 ssh /usr/sbin/sshd no 178147 5. 03/07/2013 15:02:01 root 192.168.1.101 ssh /usr/sbin/sshd no 178169 6. 03/07/2013 15:02:02 (invalid user) 192.168.1.101 ssh /usr/sbin/sshd no 178184 7. 03/07/2013 15:02:02 (invalid user) 192.168.1.101 ssh /usr/sbin/sshd no 178187 8. 03/07/2013 15:02:02 (unknown user) 192.168.1.101 ssh /usr/sbin/sshd no 178199 9. 03/07/2013 15:02:04 (unknown user) 192.168.1.101 ssh /usr/sbin/sshd no 178203 10. 03/07/2013 15:02:04 (unknown user) 192.168.1.101 ssh /usr/sbin/sshd no 178205 11. 03/07/2013 15:02:04 (invalid user) 192.168.1.101 ssh /usr/sbin/sshd no 178210 12. 03/07/2013 15:02:06 (unknown user) 192.168.1.101 ssh /usr/sbin/sshd no 178220 13. 03/07/2013 15:02:06 (invalid user) 192.168.1.101 ssh /usr/sbin/sshd no 178230 14. 03/07/2013 15:02:33 ftp 192.168.1.101 ssh /usr/sbin/sshd no 178332 26

27. Drilling Deeper with ausearch • Using aureport lets you to create overall summaries of what is happening on the system, but if you want to drill deeper into the details of a particular event, ausearch is the tool to use. • ausearchallows you to search the audit logs using special keys and search phrases that relate to most of the flags that appear in event messages in /var/log/audit/audit.log • A methodology to use is find an event class of interest with aureport and then drill down into the nitty-gritty with ausearch. • For instance, you use aureport –syscall –failed to see the failed system calls. Use ausearch and one of the event ids to get more information. 27

28. Drilling Deeper with ausearch (1) • From aureport –syscall –fail we get: Syscall Report ======================================= # date time syscall pid comm auid event ======================================= 1. 04/09/2013 16:19:15 87 4006 semodule 25016 53039 • From ausearch –i –a 53039 we get: type=PATH msg=audit(04/09/2013 16:19:15.564:53039) : item=0 name=/etc/selinux/targeted/modules/tmp/disable_dontaudit inode=394090 dev=fd:00 mode=dir,700 ouid=root ogid=root rdev=00:00 obj=unconfined_u:object_r:semanage_store_t:s0 type=CWD msg=audit(04/09/2013 16:19:15.564:53039) : cwd=/root type=SYSCALL msg=audit(04/09/2013 16:19:15.564:53039) : arch=x86_64 syscall=unlink success=no exit=-2(No such file or directory) a0=0x7f7e0a429c60 a1=0x0 a2=0x2 a3=0x7fff08fa5450 items=1 ppid=20218 pid=4006 auid=blotto uid=root gid=root euid=root suid=root fsuid=root egid=root sgid=root fsgid=root tty=pts0 ses=6353 comm=semodule exe=/usr/sbin/semodule subj=unconfined_u:unconfined_r:semanage_t:s0-s0:c0.c1023 key=MAC-policy key=ids-sys-low key=ids-file-info 28

29. Drilling Deeper with ausearch (2) • A useful feature, if you tagged your audit rules with keys, is to search for events based on those keys. • For instance, ausearch -i -k identity –start recent and we get: type=PATH msg=audit(03/10/2013 17:28:14.821:326340) : item=0 name=/etc/shadow inode=786095 dev=fd:00 mode=file,000 ouid=root ogid=root rdev=00:00 type=CWD msg=audit(04/11/2013 17:28:14.821:326340) : cwd=/home/gorgo type=SYSCALL msg=audit(04/11/2013 17:28:14.821:326340) : arch=x86_64 syscall=open success=no exit=-13(Permission denied) a0=0x7fffe185972e a1=O_WRONLY|O_TRUNC a2=0x0 a3=0x7fffe18571a0 items=1 ppid=22814 pid=25045 auid=gorgo uid=gorgo gid=users euid=gorgo suid=gorgo fsuid=gorgo egid=users sgid=users fsgid=users tty=pts0 ses=46582 comm=cp exe=/bin/cp key=identity key=ids-file-info 29

30. Visualizing Audit Data • Neither the data trail in /var/log/audit/audit.log nor the different report types generated by aureport, provide an intuitive reading experience to the user. • The aureport output is formatted in columns and thus easily available to any sed, perl, or awk scripts that users might connect to the audit framework to visualize the audit data. • A solution: mkbar and mkgraph were created by Steve Grubb at Red Hat. They are available from http://people.redhat.com/sgrubb/audit/visualize/. • Note: These scripts need gnuplot and graphviz to create their visualizations. 30

31. Visualizing Audit Data (1) • Create a plot of events with aureport -e -i --summary | mkbar events 31

32. Visualizing Audit Data (2) • Create a summary of syscall events with aureport -s -i --summary | mkbar syscall 32

33. Visualizing Audit Data (3) • To create a summary chart of successful or failed events of any of the above event types, just add the --success or --failed option to the respective aureport command. • To cover a certain period of time only, use the -ts and -te options on aureport. • Any of these commands can be tweaked further by narrowing down its scope using grep or egrep and regular expressions. 33

34. Visualizing Audit Data Relationships • To illustrate the relationship between different kinds of audit objects, such as users and system calls, use the script mkgraph. • Graphs can also be combined to illustrate complex relationships. • See the comments in the mkgraph script for further information and an example. 34

35. Visualizing Audit Data Relationships (1) • To graph the syscalls to programs, do aureport -s -i | awk '/^[0-9]/ { printf "%s %s\n", $6, $4 }' | sort | uniq | mkgraph syscall-vs-program 35

36. Visualizing Audit Data Relationships (2) • To graph to successful programs to files, do LC_ALL=C aureport -f -i --success | awk '/^[0-9]/ { print $7" "$4 }' | sort | uniq | mkgraph program-vs-file 36

37. Resources • The Audit Manual Pages • There are several man pages installed along with the audit tools that provide valuable and very detailed information: • auditd(8) The Linux Audit daemon • auditd.conf(5) The Linux Audit daemon configuration file • auditctl(8) A utility to assist controlling the kernel's audit system • autrace(8) A program similar to strace • ausearch(8) A tool to query audit daemon logs • aureport(8) A tool that produces summary reports of audit daemon logs • audispd.conf(5) The audit event dispatcher configuration file • audispd(8) The audit event dispatcher daemon talking to plugin programs. 37

38. Resources (1) • http://people.redhat.com/sgrubb/audit/index.html The home page of the Linux audit project. This site contains several specifications relating to different aspects of Linux audit, as well as a short FAQ. • /usr/share/doc/audit The audit package itself contains a README with basic design information and sample .rules files for different scenarios: • capp.rules: Controlled Access Protection Profile (CAPP) • lspp.rules: Labeled Security Protection Profile (LSPP) • nispom.rules: National Industrial Security Program Operating Manual Chapter 8(NISPOM) • stig.rules: Secure Technical Implementation Guide (STIG) 38

39. T-t-t-t-that’s all, folks! Gary Smith Information System Security Officer, Molecular Science Computing, EMSL, Pacific Northwest National Laboratory Richland, WA gary.smith@pnnl.gov 39