peculiar disappearance of most audit rules

Hi, I have started using 'auditd', mostly to monitor various directories where packages get installed to check for changes in their contents, with rules like:

  -w /bin                         -p wa -k pkg-s
  -w /boot                        -p wa -k pkg-s
  -w /etc                         -p wa -k pkg-s
  -w /lib                         -p wa -k pkg-s
  -w /lib32                       -p wa -k pkg-s
  -w /lib64                       -p wa -k pkg-s
  -w /opt                         -p wa -k pkg-s
  -w /usr                         -p wa -k pkg-s

  -w /fs/sozan/loc                -p wa -k pkg-l
  -w /fs/sozan/loc32-el5          -p wa -k pkg-l
  -w /fs/sozan/loc64-u12          -p wa -k pkg-l
  -w /fs/sozan/com                -p wa -k pkg-l
  -w /fs/sozan/com32-el5          -p wa -k pkg-l
  -w /fs/sozan/com64-u12          -p wa -k pkg-l

After setting them, I can verify that for example creating, updating and deleting a file in '/boot' or '/opt' gets reported.

Wheat then happens is that even if I set 'auditctl -e 2' some of the rules disappear, usually at around the same time as 'cron.daily' scripts run, and some more disappear later. This usually seems to relate to times where there some significant IO activity ('mlocate' scan, backup), but this is a guess.

For example:

time->Thu Apr 17 07:58:44 2014
type=CONFIG_CHANGE msg=audit(1397717924.255:37148): op="remove rule"
dir="/boot" key="pkg-s" list=4 res=1 time->Thu Apr 17 07:59:04 2014
type=CONFIG_CHANGE msg=audit(1397717944.762:37151): op="remove rule"
dir="/opt" key="pkg-s" list=4 res=1 time->Thu Apr 17 10:01:02 2014
type=CONFIG_CHANGE msg=audit(1397725262.301:37157): op="remove rule"
dir="/fs/sozan/loc64-u12" key="pkg-l" list=4 res=1 time->Thu Apr 17 10:01:02 2014
type=CONFIG_CHANGE msg=audit(1397725262.301:37156): op="remove rule"
dir="/fs/sozan/loc32-el5" key="pkg-l" list=4 res=1

There is no equivalent line in 'dmesg'.

I understand that the 'audit' kernel modules may remove rules if they refer to invalid paths, but all the relevant directories do exist, as for example '/boot' and '/opt' are the standard usual directories in the "root" tree itself:

  $  ls -ldn /boot /opt /fs/sozan/loc64-u12 /fs/sozan/loc32-el5
  drwxr-xr-x 3 0 0 4096 Apr 21 07:22 /boot
  drwxrwsr-x 7 1 1   61 Jul 30  2011 /fs/sozan/loc32-el5
  drwxrwsr-x 5 1 1   39 Oct  4  2011 /fs/sozan/loc64-u12
  drwxr-xr-x 7 0 0 4096 Apr 20 14:52 /opt

  $  df /boot/. /opt/. /fs/sozan/loc64-u12/. /fs/sozan/loc32-el5/.
  Filesystem     1M-blocks  Used Available Use% Mounted on
  /dev/sda3          24815 16853      4106  81% /
  /dev/sda3          24815 16853      4106  81% /
  /dev/sda6          90048 82355      7694  92% /fs/sozan
  /dev/sda6          90048 82355      7694  92% /fs/sozan

This is happening on two similarly configured Ubuntu 12.04 systems with both 3.2 and 3.11 Ubuntu "official" kernels. I also have an AppArmor configuration which seem to trigger bugs in AppArmor, but all the relative profiles are essentially unchanged.

Eventually around almost all of the rules I have set "disappear". For example of all these rules:

LIST_RULES: exit,always dir=/fs/sozan/search (0x10) perm=r key=pkg-r
LIST_RULES: exit,always dir=/fs/sozan/mlocate (0x11) perm=r key=pkg-r
....
LIST_RULES: exit,always dir=/bin (0x4) perm=wa key=pkg-s
LIST_RULES: exit,always dir=/boot (0x5) perm=wa key=pkg-s
LIST_RULES: exit,always dir=/etc (0x4) perm=wa key=pkg-s
LIST_RULES: exit,always dir=/lib (0x4) perm=wa key=pkg-s
LIST_RULES: exit,always dir=/lib32 (0x6) perm=wa key=pkg-s
LIST_RULES: exit,always dir=/lib64 (0x6) perm=wa key=pkg-s
LIST_RULES: exit,always dir=/opt (0x4) perm=wa key=pkg-s
LIST_RULES: exit,always dir=/usr (0x4) perm=wa key=pkg-s
LIST_RULES: exit,always dir=/fs/sozan/loc (0xd) perm=wa key=pkg-l
LIST_RULES: exit,always dir=/fs/sozan/loc32-el5 (0x13) perm=wa key=pkg-l
LIST_RULES: exit,always dir=/fs/sozan/loc64-u12 (0x13) perm=wa key=pkg-l
LIST_RULES: exit,always dir=/fs/sozan/com (0xd) perm=wa key=pkg-l
LIST_RULES: exit,always dir=/fs/sozan/com32-el5 (0x13) perm=wa key=pkg-l
LIST_RULES: exit,always dir=/fs/sozan/com64-u12 (0x13) perm=wa key=pkg-l

Only the first two have not "disappeared" on one of the systems.

This is rather peculiar, please let me know if it is a configuration error, an issue, and any fixes or workaround if available (other than running 'auditctl -R /etc/audit/audit.rules' every few minutes via CRON).

--
Linux-audit mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/linux-audit

What happens is that the text path that you put in a watch is a human convenience. The kernel doesn't understand strings, it understands numbers. It changes the path into device and inode information.

Quoted 5 lines · Steve Grubb

What happens is that the text path that you put in a watch is a human
convenience. The kernel doesn't understand strings, it understands numbers. It
changes the path into device and inode information.

Cool. So I am guessing the rule works even if someone creates a hard link to the same watched path and access files through that other path?

Quoted 11 lines

For example:

time->Thu Apr 17 07:58:44 2014
type=CONFIG_CHANGE msg=audit(1397717924.255:37148): op="remove rule" dir="/boot" key="pkg-s" list=4 res=1 time->Thu Apr 17 07:59:04 2014
type=CONFIG_CHANGE msg=audit(1397717944.762:37151): op="remove rule" dir="/opt" key="pkg-s" list=4 res=1 time->Thu Apr 17 10:01:02 2014

type=CONFIG_CHANGE msg=audit(1397725262.301:37157): op="remove rule" dir="/fs/sozan/loc64-u12" key="pkg-l" list=4 res=1 time->Thu Apr 17 10:01:02 2014
type=CONFIG_CHANGE msg=audit(1397725262.301:37156): op="remove rule" dir="/fs/sozan/loc32-el5" key="pkg-l" list=4 res=1

[ ... ] device and inode information. This is, technically, what your watch is on. If the inode disappears, then the rule is ejected. Rules can survive across renames but not deletions.

I don't know what is managing your system, but its probably deleting paths.

Quoted 2 lines

I don't know what is managing your system, but its probably deleting paths.

I am the sole user (as far as I know...) of both systems, [ ... ] None of the "disappeared" paths seems to have been modified in any way. [ ... ] Anyhow, I have now recorded the inos of the watched directories, and I shall also run 'inotifywait -m /' to catch if possible any changes in '/opt' and '/boot'.

Quoted 3 lines

[ ... ] device and inode information. This is, technically, what your watch is on. If the inode disappears, then the rule is ejected. Rules can survive across renames but not deletions.

Quoted 9 lines

Quoted 2 lines

I don't know what is managing your system, but its probably deleting paths.

I am the sole user (as far as I know...) of both systems, [ ... ] None of the "disappeared" paths seems to have been modified in any way. [ ... ] Anyhow, I have now recorded the inos of the watched directories, and I shall also run 'inotifywait -m /' to catch if possible any changes in '/opt' and '/boot'.

I have done this and this morning during 'mlocate' treewalking some of the usual paths disappeared; I verified the inos and the 'inotify' output and no inos changed nor any of the watched directories changed.

Since the list of directories that *do not* disappear is usually:

LIST_RULES: exit,always dir=/bin (0x4) perm=wa key=pkg-s
LIST_RULES: exit,always dir=/etc (0x4) perm=wa key=pkg-s
LIST_RULES: exit,always dir=/lib (0x4) perm=wa key=pkg-s
LIST_RULES: exit,always dir=/usr (0x4) perm=wa key=pkg-s
LIST_RULES: exit,always dir=/fs/sozan/loc (0xd) perm=wa key=pkg-l
LIST_RULES: exit,always dir=/fs/sozan/com (0xd) perm=wa key=pkg-l

and those that disappear tend to be far less frequently used

directories like '/boot', '/opt', '/lib32'. Rereading this:

Quoted 3 lines

Quoted 3 lines

[ ... ] device and inode information. This is, technically, what your watch is on. If the inode disappears, then the rule is ejected. Rules can survive across renames but not deletions.

it appears that I misread earlier: this says "inode", not "inum". Also it says "inode disappears", which is not necessarily always because the on-disk inode is deleted.

Thus I have come up with a potential explanation:

* The 'audit' module does not identify the watched file and directory by (device,ino) but by a pointer to an inode table entry, a bit like a filesystem module would.
* During treewalks a lot of inodes get cached in the in-memory inode table.
* This creates pressure on the inode tables and thus the least used (in some sense) inodes get evicted, and this includes those for the "disappearing directories".
* When these least used inodes are evicted the 'audit' module sees it as if it was a removal of the inode.

If the above is the right explanation it is a pretty big deal,

because it means that a way to disable many/most 'audit' watches on files is to create and access a lot of inodes, which is pretty easy to do.

Thus I have come up with a potential explanation:

* The 'audit' module does not identify the watched file and directory by (device,ino) but by a pointer to an inode table entry, a bit like a filesystem module would.

* During treewalks a lot of inodes get cached in the in-memory inode table.
* This creates pressure on the inode tables and thus the least used (in some sense) inodes get evicted, and this includes those for the "disappearing directories".
* When these least used inodes are evicted the 'audit' module sees it as if it was a removal of the inode.

* inode: Pinned so long as the inode is associated with a watch, from
* inotify_add_watch() to the final put_inotify_watch().

[ ... ]

Quoted 5 lines

Thus I have come up with a potential explanation:

* The 'audit' module does not identify the watched file and directory by (device,ino) but by a pointer to an inode table entry, a bit like a filesystem module would.

I had a look at the code and it seems it relies on 'inotify' and the code does get pointers at the relevant in-memory inode descriptors.

Quoted 7 lines

* During treewalks a lot of inodes get cached in the in-memory inode table.
* This creates pressure on the inode tables and thus the least used (in some sense) inodes get evicted, and this includes those for the "disappearing directories".
* When these least used inodes are evicted the 'audit' module sees it as if it was a removal of the inode.

To corroborate this I have been running:

  while true
  do
    for D in $(< audit-names.txt)
    do (cd "$D" && exec sleep 3001)&
    done
    sleep 3001
  done

Which has the effect of marking the relevant directories as the active current directories of each 'sleep' process, and none of those directories were "disappeared" from the 'audit' active rules list.

The 'inotify' code has a comment that claims:

Quoted 2 lines

* inode: Pinned so long as the inode is associated with a watch, from
* inotify_add_watch() to the final put_inotify_watch().

They use 'igrab'/'iput', and 'audit_tree.c' and 'audit_watch.c' uses them, so I wonder what is missing.

--
Linux-audit mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/linux-audit

What's the kernel in question?

audit hasn't used "inotify" in a long time. We now use "fsnotify".

but in either case, the inodes aren't supposed to be able to be kicked out of core...

From the source I have looked at, the *notify code seems to

Quoted 2 lines

but in either case, the inodes aren't supposed to be able to be kicked out of core...

But on 3 different system I have they really seem to be evicted, and with regularity, and this does not happen if the inodes are kept open.

From the source I have looked at, the *notify code seems to attempt to hold on to the inodes that are watched, but perhaps it has some hidden assumptions that the 'audit' module does not satisfy.