Light-weight Auditing Framework

This note describes a patch against 2.6.4-rc1-bk2 that provides a low-overhead system-call auditing framework for Linux that is usable by LSM components (e.g., SELinux). Comments will be appreciated.

The main goals were to provide system call auditing with 1) as low overhead as possible, and 2) without duplicating functionality that is already provided by SELinux (and/or other security infrastructures). This framework will work "stand-alone", but is not designed to provide, e.g., CAPP functionality without another security component in place.

There are two main parts, one that is always on (generic logging in audit.c) and one that you can disable at boot- or run-time (per-system-call auditing in auditsc.c). The patch includes changes to security/selinux/avc.c as an example of how system-call auditing can be integrated with other code that identifies auditable events.

Logging:
1) Uses a netlink socket for communication with user-space. All messages are logged via the netlink socket if a user-space daemon is listening. If not, the messages are logged via printk to the syslog daemon (by default).
2) Messages can be dropped (optionally) based on message rate or memory use (this isn't fully integrated into the selinux/avc.c part of the patch: the avc.c code that currently does this can be eliminated).
3) When some part of the kernel generates part of an audit record, the partial record is sent immediately to user-space, AND the system call "auditable" flag is automatically set for that call -- thereby producing extra information at syscall exit (if syscall auditing is enabled).

System-call auditing:
1) At task-creation time, an audit context is allocated and linked off the task structure.
2) At syscall entry time, if the audit context exists, information is filled in (syscall number, timestamp; but not arguments).
3) During the system call, calls to getname() and path_lookup() are intercepted. These routines are called when the kernel is actually looking up information that will be used to make the decision about whether the syscall will succeed or fail. An effort has been made to avoid copying the information that getname generates, since getname is already making a kernel-private copy of the information. [Note that storing copies of all syscall arguments requires complexity and overhead that arguably isn't needed. With this patch, for example, if chroot("foo") fails because you are not root, "foo" will not appear in the audit record because the kernel determined the syscall cannot proceed before it ever needed to look up "foo". This approach avoids storing user-supplied information that could be misleading or unreliable (e.g., due to a cooperative shared-memory attack) in favor of reporting information actually used by the kernel.]
4) At syscall exit time, if the "auditable" flag has been set (e.g., because SELinux generated an avc record; or some other part of the kernel detected an auditable event), the syscall-part of the audit record is generated, including file names and inode numbers (if available). Some of this information is currently complementary to the information that selinux/avc.c generates (e.g., file names and some inode numbers), but some is less complete (e.g., getname doesn't return a fully-qualified path, and this patch does not add the overhead of determining one). [Note that the complete audit record comes to userspace in pieces, which eliminates the need to store messages for arbitrarily long periods inside the kernel.]
5) At task-exit time, the audit context is destroyed.

At steps 1, 2, and 4, simple filtering can be done (e.g., a database role uid might have syscall auditing disabled for performance reasons). The filtering is simple and could be made more complex. However, I tried to implement as much filtering as possible without adding significant overhead (e.g., d_path()). In general, the audit framework should rely on some other kernel component (e.g., SELinux) to make the majority of the decisions about what is and is not auditable.

A few (obvious) things are not yet implemented:
1) Filtering on syscall personality (i.e., i386 calls on x86_64).
2) Multi-platform syscall support (currently syscall auditing has only been implemented on i386, x86_64, and ppc64).
3) Documentation.

I have also written very crude user-space tools that can be used to test the features provided (auditd-0.3.tar.gz). This demo tarball, a readme.txt (see for how to compile), and the patch (and more recent patches?) are available from: http://people.redhat.com/faith/audit/

 arch/i386/kernel/entry.S         |    6 
 arch/i386/kernel/ptrace.c        |    8 
 arch/ppc64/kernel/entry.S        |   15 
 arch/ppc64/kernel/ptrace.c       |   27 -
 arch/x86_64/ia32/ia32entry.S     |   12 
 arch/x86_64/kernel/entry.S       |   21 
 arch/x86_64/kernel/ptrace.c      |   23 -
 fs/namei.c                       |   15 
 include/asm-i386/thread_info.h   |    5 
 include/asm-ppc64/thread_info.h  |    3 
 include/asm-x86_64/thread_info.h |    5 
 include/linux/audit.h            |  210 +++++++++
 include/linux/fs.h               |   10 
 include/linux/netlink.h          |    1 
 include/linux/sched.h            |    5 
 init/Kconfig                     |   20 
 kernel/Makefile                  |    2 
 kernel/audit.c                   |  771 ++++++++++++++++++++++++++++++++++
 kernel/auditsc.c                 |  872 +++++++++++++++++++++++++++++++++++++++
 kernel/fork.c                    |    9 
 security/selinux/avc.c           |  146 ++----
 security/selinux/include/avc.h   |    7 
 security/selinux/ss/services.c   |    2 
 23 files changed, 2065 insertions(+), 130 deletions(-)

Due to size, the patch is available from;

    http://people.redhat.com/faith/audit/audit-20040226.1411.patch

I haven't actually looked at the code, but why don't you use Olaf Kirch's auditing framework that's used in production and already has gotten the wizzbang certification you seem to be aiming at.

Whether we want syscall auditing in mainline is a completely different question..

Different goals. My goals are to provide a generic very-low-overhead auditing framework that can be used as a service by more complex systems (e.g., SELinux). In contrast to Olaf's work, for example, my patch does not have intimate knowledge of system call parameters and semantics. This decreases the invasiveness of the patch and the work required for long-term maintainability.

The price for this simplicity is that the "language" for describing which system calls to audit is also very simple (and is, therefore, not independently sufficient for certifications). However, I assume that a system undergoing certification will be using SELinux or another security infrastructure that will make auditing _and_ other decisions -- adding sophistication to the auditing infrastructure only duplicates the work that the security module will provide.

I believe we need a light-weight, maintainable framework that is versatile enough to be used for non-security purposes (e.g., debugging). In general, my patch meets these requirements since it provides very little that helps specifically with security (failure modes, loginuid, and small helper functions).

Hi Rik,

I've looked at your patch. The idea to intercept path name arguments etc inside getname and path_lookup is interesting. I had to jump through a lot of hoops to prevent the caller from messing with the syscall arguments inbetween the syscall and the audit intercept, and that made the code a lot uglier than I had wanted.

This type of race condition is the major reason why I now believe intercepting syscall entry/exit is not a good choice for auditing. For debugging yes, but not for auditing.

However, I wonder your approach is sufficient to obtain certification. The first thing that jumped at me was your comment that "netlink may lose packets". This is an absolute show-stopper for CAPP compliance. In general, during testing we noticed that passing audit from kernel to a user land daemon can become a serious bottleneck. A better approach may be to have the kernel write audit records directly to the audit trail; this solves both the record loss issue and improves performance.

Another issue is that you have system calls such as link or rename that consume two path name arguments, and your system call record needs to identify which is which.

In general, I wonder how your frame work will audit other, non-pathname system call arguments. Is that entirely a decision of the security module? If so, then why create a generic audit framework at all?

In general, there is a question of whether the security framework could be sufficient for auditing purposes. As it stands now, I can see two shortcomings. The obvious one is that most security operations are performed _before_ the operation, while audit records need to include the outcome of the system call, so they are performed _after_ the operation. That is something that could be solved the way your patch seems to be doing it, i.e. flag the ongoing system call "interesting" and intercept the syscall return.

The second issue with security operations though is that it is far from clear if we cover all "relevant" system calls. CAPP is not very clear regarding what is "relevant". For instance it's far from obivous to me whether a system that fails with EFAULT or EINVAL needs to be audited or not.

I'd be happy if you could prove me wrong, but I believe that debugging and (certifiable) system call auditing don't mix very well. They might be able to share a minimal framework (e.g. for delivering an audit trail to user land and/or a file).

The motivation behind these two objectives is quite different: audit wants to capture security relevant operations, and deliver the events reliably. Debugging doesn't care about reliability, but OTOH wants to capture as wide a selection of system call events as possible.

Olaf

Olaf Kirch     |  Stop wasting entropy - start using predictable
[email protected]   |  tempfile names today!
---------------+ 

Hi Olaf. Thanks for your comments.

I didn't mean to imply that netlink was unreliable -- the drops aren't
arbitrary: you can detect them and you can decide if and when to drop. Any logging system will have the problem of consuming kernel memory for messages that are on their way to user-space. For those not interested in CAPP compliance, the drops are tunable based on rate and size ("backlog" in the code). For those who are interested in CAPP, these limits would not be in place, and total system memory would be the limiting factor.

This seems like it would solve several problems. I'll have to think about it more. (You still have to worry about what happens when the drive is too slow for the message rate, how to detect this, and, perhaps, how to limit its impact on systems where reliable delivery of messages is not absolutely critical. With netlink, these worries are easy to deal with.)

In order to keep the kernel-space piece tiny, I think the user-space daemon should do as much of this work as possible. For example, sys_rename always calls getname(oldname) before getname(newname), so it is possible to tell which is which in user space.

Yes, that is a decision of the security module. The security module is already looking at the arguments it believes to be important, so adding additional overhead for auditing those arguments seems redundant. The generic audit framework is providing more general services, such as the netlink transport and the syscall entry/exit hooks services that can be used to audit the syscall that was running when an auditable event occurred. I.e., we probably don't want every security module using a netlink number and patching entry.S.

Yes, the patch handles this. The sequence is:
1) syscall entry: build audit context
2) security operation detects auditable event and sets flag
3) syscall exit: if flag is set, syscall-specific record emitted

The audit framework doesn't know enough to do #2 -- that's the role of the security module.

I don't have an interpretation of CAPP either, but I will explore ways to make these two points easier to deal with.

In addition to delivering the audit trail to user-space, both objectives may also require adding hooks to entry.S. If delivery and hooking is useful for both, we should provide it only once inside the kernel. I've tried to provide a generic framework that can be used as a service for different objectives, with tuning that can favor one objective over the other. It can grow to have more features, but I'm not sure if much can be taken away (auditsc.c can be eliminated, but then you lose the entry.S hooks).

OK, I've thought about this more. I can appreciate the argument that writing directly to disk avoids the performance impact of having a user-space daemon write to disk (e.g., scheduling and, possibly, extra copies). However, I do not believe this addresses the fundamental problem and, because of the nature of the problem, I do not believe the performance improvement will be observed in all cases. Let me explain.

CPUs can generate audit records magnitudes of order faster than they can be written to disk. Further, the production is non-uniform. Because this problem involves a bursty producer and a slow consumer, I view it as a queuing problem. Writing directly to disk makes the consumer marginally faster, but it does not change the fundamental relationships in the problem -- the consumer is still magnitudes of order slower than the producer. This means that, even though the direct-write consumer is faster, the speed of the whole queue hasn't changed much, if at all.

Further, I am concerned that using direct writes instead of netlink moves hard problems from one (simple, flexibly) layer to another (more complex, less flexible) layer. For example, the file system may have already made some decisions for you that you could have made differently if you had access to information about the netlink-managed queue.

I'd like to second the sentiment - for syscalltrack, we would love to have a framework that can be used for debugging system call events.

Cheers,
Muli

Muli Ben-Yehuda
http://www.mulix.org | http://mulix.livejournal.com/

I should clarify to avoid any confusion. The audit framework I am proposing does _not_ allow for generic hooking of system calls.

However, the ability to audit the call is generic in the sense that another part of the kernel can request that an audit record be generated for the current in-progress system call (or filtering can be used to request that audit records be generated). Although this should be sufficient for SELinux and other security infrastructures, as well as many non-security uses, it would only support a subset of what syscalltrack can do currently (i.e., audit can provide information about the syscall, but can't be used to change the behavior of the syscall).

Do you have an perf numbers for simply enabling the auditint framework? Seems to be a lot of use of likely/unlikely. Are these actually useful optimizations? Doesn't seem like CONFIG_AUDIT=n disables all the code. Not sure if the rest of the patch lot is mainlineable or not, but here are some simple nits/questions.

Some basic CodingStyle bits:

+static int audit_initialized = 0;

no need to zero initialized static data.

+ if (in_irq()) BUG();
+ if (!context) BUG();

convert to BUG_ON(in_irq());
           BUG_ON(!context);

+#ifdef CONFIG_AUDIT +/***********************************************************************/
+/** audit.c **/
+/***********************************************************************/

Noisy comments.

+ audit_log_format(ab, " dev=%02x:%02x",
+ MAJOR(context->names[i].rdev),
+ MINOR(context->names[i].rdev));

format_dev_t() ?

+static int __init audit_enable(char *str)
+{
+ if (!strncmp(str, "on", 2) || !strncmp(str, "yes", 3))
+ audit_default = 1;
+ else if (!strncmp(str, "off", 3) || !strncmp(str, "no", 2))
+ audit_default = 0;
+ else
+ audit_default = simple_strtol(str, NULL, 0);

just pick one method, strings or int (int is so simple), and document the method for enable/disable.

+void audit_log_vformat(struct audit_buffer *ab, const char *fmt, va_list args)

as of now, this could be static, and not exported, no?

+ ab = kmalloc(sizeof(*ab), GFP_ATOMIC);

need atomic allocations here? also, i wasn't clear on what makes sure the audit_buffers can't be overflown, the bits in vformat talking about 1024 bytes?

+ audit_log_format(ab, "<toolong>");

is it useful to at least get the final d_name?

thanks,
-chris

Linux Security Modules     http://lsm.immunix.org     http://lsm.bkbits.net

Comparing no patch with the patch applied and booted with audit=1 (so the audit context is allocated for each task and filled in for every syscall, but not sent to user-space), using lmbench 2.0.4:

Processor, Processes - times in microseconds - smaller is better
----------------------------------------------------------------
Host OS Mhz null null open selct sig sig fork exec sh
call I/O stat clos TCP inst hndl proc proc proc
--------- ------------- ---- ---- ---- ---- ---- ----- ---- ---- ---- ---- ---- 1bk8stock Linux 2.6.3-b 2004 0.08 0.26 14.7 16.4 5.797 0.23 2.00 143. 443. 1804 4aon-nos Linux 2.6.3-b 2004 0.21 0.40 14.5 16.7 12.0 0.40 2.19 136. 441. 1717

At a higher level, I compiled the kernel three times with:
export CCACHE_DISABLE=1
make mrproper >& /dev/null
cp arch/x86_64/defconfig .config
yes '' | make oldconfig >& /dev/null
time make > /dev/null

Situation Best time (of 3) Factor

No output to user-space:

audit=0 (no audit context)       3:23.04u 34.560s 4:00.76 98%    1.000
audit=1 (audit context built)    3:23.47u 33.810s 4:01.02 98%    0.999
audit=1 (10 rules, no records)   3:28.39u 35.670s 4:07.48 98%    0.97

Output to user-space for _every_ syscall:

audit=1 (* all to /dev/null)     3:34.96u 1:16.98s 5:06.40 95%   0.79
audit=1 (** all to file)         4:22.82u 1:45.67s 7:11.51 85%   0.56

* Every syscall for the whole compile was audited and logged to user-space. The user-space daemon sent all messages to /dev/null.
** Every syscall for the whole compile was audited and logged to
user-space. The user-space daemon sent all messages to a file (which grew to 1.4GB on an IDE drive). This is the worst case scenario.

The goal is for the non-auditing case to be as fast as possible, so all of the likely/unlikely notations favor that case.

The bit tests in entry.S are still there, but those are the same tests that are used for ptrace, and there is nothing that sets the bits. So, aside from that test, all of the code should be disabled.

Except where noted below, I have either incorporated all your suggestions or made notes in the code to do so later. The new patch is
at: http://people.redhat.com/faith/audit/audit-20040302.1632.patch

These functions can be called in any context, so sleeping is not an option.

Yes, and the vsnprintf in audit_log_vformat.

I think, e.g. the code that calls audit_get/putname is still there.

Oops, I wasn't clear re: the static initialized data...I just meant to give a couple examples, there were more:

+static int         audit_default = 0;
+static int         audit_pid = 0;
+static int         audit_rate_limit    = 0;
+static int         audit_freelist_count = 0;

etc...

thanks,
-chris

Linux Security Modules     http://lsm.immunix.org     http://lsm.bkbits.net

When syscall auditing is disabled, the body of the if will become a nop because of a #define, so the compiler will remove the whole if. I don't want to move the if into a macro, since this would make it look like the function was called all the time. I don't want the if in the function because I'm trying not to call the function except when necessary.

I could put #ifdef CONFIG_AUDITSYSCALL around these statements, but I find that often makes code harder to read. However, in this case, that might avoid some confusion.

Yes, sorry, please see:

    http://people.redhat.com/faith/audit/audit-20040303.0544.patch