audit collector startup help

DJ (or anyone) -

Is there a HOWTO for activating the 1.7.5 aggregating feature? My apologies if I missed this earlier.

I believe that the collector needs to uncomment the lines in /etc/auditd/auditd.conf and the senders/clients need to set active=yes, remote=<IP-address> in the audisp-remote.conf file.

However, my collector auditd fails on start; it might be that I do not have it configured correctly.

I have : audit-1.7.5-1.fc9.i386

Thx,
LCB.

LC (Lenny) Bruzenak
[email protected]

Just the man pages.

The collector needs the listener configured in /etc/audit/auditd.conf:

tcp_listen_port = 1237

The clients need the audisp-remote module enabled and configured:

/etc/audisp/plugins.d/au-remote.conf:
active = yes

/etc/audisp/audisp-remote.conf:
remote_server = 192.16.1.12 (your server's IP, not mine ;)
port = 1237 (or use some other port, up to you)
transport = tcp

Additional options:
format = managed
network_retry_time = 1
max_tries_per_record = 10
max_time_per_record = 7

You'll have to enable the connection through tcp_wrappers as well, if you have that option enabled, as well as whatever firewall rules may apply.

Messages?

Thanks for the above.
I am only looking at the server/collector startup right now.

Not real helpful so far (/var/log/messages - any other place?):

Sep  9 13:41:15 fryspc auditd[3786]: Init complete, auditd 1.7.5
listening for events (startup state enable)
Sep  9 13:41:15 fryspc auditd[3786]: Cannot bind tcp listener socket to
port 1237
Sep  9 13:41:15 fryspc auditd[3786]: The audit daemon is exiting.

Thx!
LCB.

LC (Lenny) Bruzenak
[email protected]

Have you tried other ports? There might be something on that port already.

I looked (with lsof) and there was nothing. I tried port 12037; same thing.

It is not a firewall issue:
[root@fryspc etc]# iptables -L
Chain INPUT (policy ACCEPT)
target prot opt source destination

Chain FORWARD (policy ACCEPT)
target prot opt source destination

Chain OUTPUT (policy ACCEPT)
target prot opt source destination

I do not any tcp_wrappers issues.
I also accept prelude events on this machine.

This has to be something easy I've overlooked...I do appreciate the suggestions.

Thx,
LCB.

I have other things listening away:
rpcbind 1845 rpc 8u IPv4 6916 TCP
*:sunrpc (LISTEN)
rpc.statd 1864 rpcuser 9u IPv4 6988 TCP
*:58971 (LISTEN)
sshd 2152 root 3u IPv4 7975 TCP
*:ssh (LISTEN)
sshd 2152 root 4u IPv6 7977 TCP
*:ssh (LISTEN)
mysqld 2246 mysql 10u IPv4 8088 TCP
localhost.localdomain:mysql (LISTEN)
prelude-m 2277 root 0u IPv4 8201 TCP
localhost.localdomain:prelude (LISTEN)
prelude-m 2277 root 2u IPv4 8204 TCP
fryspc:prelude (LISTEN)
sendmail 2301 root 4u IPv4 8325 TCP
localhost.localdomain:smtp (LISTEN)
httpd 2320 root 4u IPv6 8381 TCP
*:http (LISTEN)
cupsd 2375 root 4u IPv4 8606 TCP
localhost.localdomain:ipp (LISTEN)
httpd 6341 apache 4u IPv6 8381 TCP
*:http (LISTEN)
httpd 6342 apache 4u IPv6 8381 TCP
*:http (LISTEN)
httpd 6343 apache 4u IPv6 8381 TCP
*:http (LISTEN)
httpd 6344 apache 4u IPv6 8381 TCP
*:http (LISTEN)
httpd 6345 apache 4u IPv6 8381 TCP
*:http (LISTEN)
httpd 6346 apache 4u IPv6 8381 TCP
*:http (LISTEN)
httpd 6347 apache 4u IPv6 8381 TCP
*:http (LISTEN)
httpd 6348 apache 4u IPv6 8381 TCP
*:http (LISTEN)

LC (Lenny) Bruzenak
[email protected]

Hmmm.... well, one option is to put a perror() after the bind() call in src/auditd-listen.c and run auditd in the foreground (./auditd -f) to see *why* it's failing.

Good suggestion...I did that and now the one I built doesn't fail. Then I stopped the daemon, replaced the original one, restarted the daemon. It works also!

Only thing I did in between was load about 100 packages needed for the rebuild. Is there any chance that one of these had some necessary magic I was missing?

Thx,
LCB.

LC (Lenny) Bruzenak
[email protected]

I guess the only way to know is remove all the packages and retry...

LCB.

LC (Lenny) Bruzenak
[email protected]

More likely, something was holding the socket in CLOSE_WAIT or something and happened to time out while you were updating everything.

Actually I believe one of the packages must installed my policy as enforcing.

Thanks(!) to an excellent setroubleshoot pop-up I believe that was my problem:

Source Context:  unconfined_u:system_r:auditd_t:s0
Target Context:  system_u:object_r:anon_inodefs_t:s0
Target Objects:  anon_inode [ file ]
Source:  auditdSource 
Path:  /sbin/auditd
Port:  <Unknown>
Host:  fryspc
Source RPM Packages:  audit-1.7.5-1.fc9
Target RPM Packages:  
Policy RPM:  selinux-policy-3.3.1-87.fc9
Selinux Enabled:  True
Policy Type:  targeted
MLS Enabled:  True
Enforcing Mode:  Enforcing
Plugin Name:  catchall_file
Host Name:  fryspc
Platform:  Linux fryspc 2.6.26.3-29.fc9.i686 #1 SMP Wed Sep 3 03:42:27 EDT 2008 i686 athlon
Alert Count:  1
First Seen:  Thu 11 Sep 2008 10:08:57 AM CDT
Last Seen:  Thu 11 Sep 2008 10:08:57 AM CDT
Local ID:  8b4ff486-ae1c-4448-bf38-9b56658ebc01

Line Numbers:
Raw Audit Messages :
host=fryspc type=AVC msg=audit(1221145737.208:55): avc: denied { write } for pid=3280 comm="auditd" path="anon_inode:[eventfd]" dev=anon_inodefs ino=18 scontext=unconfined_u:system_r:auditd_t:s0 tcontext=system_u:object_r:anon_inodefs_t:s0 tclass=file
host=fryspc type=SYSCALL msg=audit(1221145737.208:55): arch=40000003 syscall=4 success=no exit=-13 a0=8 a1=bfb98880 a2=8 a3=b7f6aab8 items=0 ppid=1 pid=3280 auid=500 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=1 comm="auditd" exe="/sbin/auditd" subj=unconfined_u:system_r:auditd_t:s0 key=(null)

### note: I do not have an MLS policy on this machine (although the setroubleshoot summary says I do) - and I didn't change any policy defaults.

[lenny@fryspc ~]$ rpm -qa | grep policy
checkpolicy-2.0.16-3.fc9.i386
policycoreutils-2.0.52-8.fc9.i386
selinux-policy-targeted-3.3.1-87.fc9.noarch
selinux-policy-devel-3.3.1-87.fc9.noarch
policycoreutils-gui-2.0.52-8.fc9.i386
selinux-policy-3.3.1-87.fc9.noarch

Thx,
LCB.

LC (Lenny) Bruzenak
[email protected]

My sender fails to connect to my collector.

Is there any reason a MLS-policy F9 audisp-remote should be unable to connect to a targeted-policy F9 auditd? I have no ipsec or anything else involved...

I am looking for some hint as to why the connection is failing but I see only this on the sender:

- lsof says I'm stuck on SYN_SENT
TCP comms:38827->192.168.30.120:tsdos390 (SYN_SENT)

- audit search on sender
ausearch -ts today -i -c audisp-remote:
...
----
type=SYSCALL msg=audit(09/11/2008 16:14:45.102:19013) : arch=x86_64 syscall=connect success=no exit=-110(Connection timed out) a0=3 a1=7f99ab0f20e0 a2=10 a3=7fffb289cf50 items=0 ppid=25435 pid=25436 auid=root uid=root gid=root euid=root suid=root fsuid=root egid=root sgid=root fsgid=root tty=(none) ses=61 comm=audisp-remote exe=/sbin/audisp-remote subj=system_u:system_r:audisp_remote_t:s15:c0.c1023 key=(null)

Same audit versions on each (1.7.5-1).

On the sender, I can do a "newrole -l SystemHigh" and connect via "telnet <collector> 1237", so I don't think it is the level giving me any grief - sender is in permissive mode so there are AVCs but it should work.

Eventually on the sender I get this:

Sep 11 16:57:12 comms audisp-remote: Error connecting to 192.168.30.120: Connection timed out - exiting
Sep 11 16:57:14 comms audispd: plugin /sbin/audisp-remote terminated unexpectedly

On the collector machine I see the listen socket open but I see no denials in the messages log or the audit log.

Any suggestions?

Thx,
LCB.

LC (Lenny) Bruzenak
[email protected]

Have you tried killing the client auditd and manually restarting it (i.e. /etc/rc.d/init.d/auditd restart) ?

I wonder if there's some catch-22 at system startup, relative to network startup.

I have, but actually I use :
run_init service auditd restart

otherwise my audit logs get created at SYSLO which is bad on an MLS system.

I have had to restart often because while I am looking for a hint the timeout happens and the audisp-remote process dies.

Thx,
LCB.

LC (Lenny) Bruzenak
[email protected]

DJ,

Thanks for the above. The network_retry_time (et. al.) must be in the later version.

I have: audispd-plugins-1.7.5-1.fc9.x86_64 ; there is no mention of that one in the man page and I get this message on startup:

Sep 12 11:43:48 comms audisp-remote: Unknown keyword "network_retry_time" in line 14 of /etc/audisp/audisp-remote.conf
Sep 12 11:43:48 comms auditd[4411]: Init complete, auditd 1.7.5 listening for events (startup state enable)
Sep 12 11:43:48 comms audispd: plugin /sbin/audisp-remote terminated unexpectedly

So I Removed the timing parameters.

Now I get this:
...

Sep 12 11:46:20 comms audisp-remote: lost/losing sync, bad magic number
Sep 12 11:46:20 comms audisp-remote: lost/losing sync, bad magic number

...

I do not see any errors in the message log on the collector.

Any ideas?

Thx again!
LCB.

LC (Lenny) Bruzenak
[email protected]

You need 1.7.6 then, or SVN. Coming soon to a theater near you ;-)

So do you think that the "lost/losing sync, bad magic number" issue is fixed in 1.7.6 or just the timing parameters?

LCB.

LC (Lenny) Bruzenak
[email protected]

What 1.7.6 adds is the ability to close the connection and start again, without losing any audit records, if/when it detects any communications problems.

So any clue as to why I get a "bad magic number" on this version?

Thx,
LCB.

LC (Lenny) Bruzenak
[email protected]

Nope. It means the server sent back a header that was corrupt, or sent back something other than a header.

The header is defined in lib/private.h

After looking at this I had a hunch - the collector machine is 32-bit, the sender 64-bit.

I reverse the sender/collector and I don't get this error anymore. The machine architecture is not representative of the intended final deployment, just the available machines I was using to test.

So now the bad news is that I still do not see events passing from sender to collector. This may be to an incorrect assumption on my part. I assume that all events on the sender make it to the collector. Is this true always? I send in a forced event (on the sender):

[root@fryspc audisp]# auditctl -m TEST
[root@fryspc audisp]# ausearch -ts recent -i | grep TEST
type=USER msg=audit(09/12/2008 18:34:34.930:126) : user pid=4866 uid=root auid=root ses=11 subj=unconfined_u:unconfined_r:auditctl_t:s0-s0:c0.c1023 msg='TEST: exe=/sbin/auditctl (hostname=?, addr=?, terminal=pts/2 res=success)'

But I cannot see this event on the collector.

The good news is that I see the connection on both ends (using port 1237=tsdos, lsof results):
collector:
auditd 9422 root 11u IPv4 55889 TCP comms:tsdos390->fryspc:55303 (ESTABLISHED)
collector:
sender:
audisp-re 4846 root 3u IPv4 26741 TCP fryspc:55303->192.168.31.142:tsdos390 (ESTABLISHED)

Should I see this event, and if so, do you have any idea as to why I do not? I also have the same thing from another 64-bit sender.

Thx,
LCB.

LC (Lenny) Bruzenak
[email protected]

And the magic number has the high bit set. I wonder if there's a sign extension in there somewhere?

Can you try between two 32 bit hosts?

I didn't add any filters - anything that makes it to audisp-remote eventually gets queued in the server's event queue.

All remote messages will have "node=" in them somewhere. Can you grep for that manually in your server's audit logs? I wonder if ausearch is skipping them.