Filesystem access statistics

Hi,
I subscribed to the list after checking with Steve that it was not an outlandish choice of places where to ask my questions.

I need to look at a portion of the filesystem namespace and maintain aggregate statistics on access patterns. In other words, I have a large filesystem and would like to find out which are the hot spots. I don't need to keep track of every single file access: since the file count is in the order of millions, that would swamp the actual I/O, the analysis and the people looking at the final data. It would make sense to just group accesses by looking at the top N levels (anything accessed at levels N+1, N+2, etc. would be coalesced into the parent directory at level N).

I think that I can't be the only one with such a need. In my case, the information is going to be used to change the way the tree is going to be laid out in the future, as well as determining when parts of it can be made read-only (after an inactivity period). I can also see the information being useful for selective incremental backups - just look at the hot spots - or for smarter ordering during a disaster recovery restore (if you're recovering from random access storage, not tape). Maybe even locate/slocate/rlocate/mlocate could take advantage of it.

What would be the best approach to this? Inotify doesn't seem to cut it, because it can't handle recursive watches. I can't afford placing watches all over the place. Given the sheer number of operations being tracked, it looks like I'd need some custom code that audits all file/directory operations, determines if there's a match (I'm only interested in a specific tree, not everything under /), increments internal counters and throws the event away. Is there code I could look at for ideas?

Thanks in advance for any help.

Rudi

I would think that you could write a program to do this via the audit dispatcher interface. In auditd.conf,

dispatcher = /usr/bin/your-program
log_format = nolog

This will tell the audit daemon you don't want the records written to disk and that it should pass events to the dispatcher. You can get example code in skeleton.c. (rpm -ql audit | grep skel ) The skeleton.c program around line 123 is where you would add your code to examine the PATH records. You would likely want to do

if (hdr.type == AUDIT_PATH) {
process the path
}

so that you only look at the right kind of records. If you have too many records coming out of the audit system, you might be able to suppress some in the kernel by adding rules like this:

-a always,exclude -F 'msgtype<PATH' -F 'msgtype>PATH'

You can then set the audit rules for whatever you want to measure, if all you want to measure is the opens,

-a always,exit -S open

You can use devmajor and devminor fields to limit the audit system to reporting opens on an exact partition. This is highly recommended. On my system, I would do something like:

-a always,exit -S open -F devmajor=3 -F devminor=6

to watch the /tmp directory.

Bootchart.org used the audit system to examine the boot order

http://bootchart.org/misc/filemon/filemond

Sounds interesting.

I think we've laid out an approach above.

1) set custom rules to watch just the syscalls you want on the exact partitions you want.
2) put the analysis in a program hanging off of audit event dispatcher and turn off audit logging to disk.

You need audit-1.1 or higher. audit-1.1 has some ABI changes that make it incompatible with audit-1.0 systems. Let me know if that is a problem. Also, you will need 2.6.17 kernel if you use the '<' or '>' rule operators.

-Steve

Will that preempt any other audit users that might be looking for events downstream? Sounds a bit too drastic, although I guess I am not the typical case, so an application as "intrusive" as mine won't be needed on the average system.

libaudit.h from audit-libs-devel 1.1.5-1 only has AUDIT_FS_INODE. Is this new in 1.2 or a typo? I saw mention of a new filesystem API in the audit RPM changelog. Is that part of it?

That's a very good question by itself. Anything that peeks into a directory should do, I guess. That would mean not just opens, but also directory traversals, unlink calls, etc. Are there aliases of any kind? The kernel just gained a bunch of new *at() syscalls. If I had written this a month or two ago, I would have most likely missed them. Is there a way to look for present and future syscalls dealing with files/inodes?

That's a good idea where applicable. Thanks.

Rudi

Yes it will, but the audit event dispatcher is not ready yet. So, if you want something today, you can just take over the interface. Then re-write it as a plugin later when the event dispatcher is finished.

libaudit.h includes linux/audit.h which defines the AUDIT_PATH message type.

Its been around for at least a year.

No.

No.

Not at the moment.

-Steve