File change notification

Hi everybody.

This is my first post to lkml, so please be patient with me

I have been wondering for some time why there is no decent file change notification mechanism in linux. Is there some deep philosophical reason for this, or is it just that nobody has found the time to implement it? If it is the latter, I am willing to implement it as long there is a chance to get this accepted into the mainstream kernel.

Is there already somebody working on this problem? I know a few efforts that try to do something similar, but they all work by intercepting every single system call that has to do with files, and they are thus not very fast. See for example
<http://www.bangstate.com/software.html#changedfiles>
<http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/openxdsm/openxdsm/eventmodule/>

The dnotify mechanism is very limited since it requires a file handle, it is not working for whole directory trees and it does not report much useful information. For example to watch for changes in the /home tree you would have to open every single directory in the tree, which would probably not even work since it would require more than the maximum number of file handles. If you have a directory with many files in it, the only thing dnotify tells you is that something has changed in the directory, so you have to rescan the whole directory to find out which file has changed. Kind of defeats the purpose of change notification...

What I would like to have would be some way to watch for certain changes anywhere in a whole tree or even the whole file system. This new mechanism should have no measurable performance impact and should log all information that is readily available. The amount of new code in kernel space should be as small as possible. So complicated stuff like pattern matching would have to happen in user space.

I wrote some experimental mechanism yesterday. Whenever a file is accessed or changed, I write all easily available information to a ring buffer which is presented to user space as a device. The information that is easily available is the inode number of the file or directory that has changed, the inode number of the directory in which the change took place, and in most cases the name of the dentry of the file that has changed.

Here is the struct I use for logging:

typedef struct
{
unsigned long event; /* the type of change. As of now, I use the same
constants as dnotify */
unsigned long file_ino; /* the inode of the file that has changed (if
available) */
unsigned long src_ino; /* the inode of the directory in which the
change took place (if available) */
unsigned long dst_ino; /* the inode of the destination directory.
This is only used when logging moves */
unsigned char name[DNAME_LEN]; /* the name of the dentry of the
changed file. This is simply truncated if it is too long. */ } inotify_info;

Information that is not easily available and therefore not logged includes the full path(s) of the changed file/directory. If you want or need that, you will have to look it up in user space.

Since I only log information that can be gathered at practically no cost, the logging does not take any noticeable time in kernel space.

I attached a patch containing the code I wrote yesterday. This is very preliminary and probably not bug-free. I would like to have feedback on the general approach before I spend more time refining this.

best regards,

Rüdiger Klaehn

diff -urN -X dontdiff vanilla/fs/dnotify.c develop/fs/dnotify.c
@@ -153,8 +153,9 @@
 void dnotify_parent(struct dentry *dentry, unsigned long event)
 {
 	struct dentry *parent;
-
 	spin_lock(&dentry->d_lock);
+	/* call inotify for this dentry */
+	inotify_dentrychange(dentry,event);
 	parent = dentry->d_parent;
 	if (parent->d_inode->i_dnotify_mask & event) {
 		dget(parent);

diff -urN -X dontdiff vanilla/fs/Makefile develop/fs/Makefile
@@ -10,7 +10,7 @@
 		namei.o fcntl.o ioctl.o readdir.o select.o fifo.o locks.o \
 		dcache.o inode.o attr.o bad_inode.o file.o dnotify.o \
 		filesystems.o namespace.o seq_file.o xattr.o libfs.o \
-		fs-writeback.o mpage.o direct-io.o aio.o
+		fs-writeback.o mpage.o direct-io.o aio.o inotify.o
 
 obj-$(CONFIG_EPOLL)		+= eventpoll.o
 obj-$(CONFIG_COMPAT)		+= compat.o

diff -urN -X dontdiff vanilla/include/linux/dnotify.h develop/include/linux/dnotify.h
@@ -22,6 +22,8 @@
 
 static inline void inode_dir_notify(struct inode *inode, unsigned long event)
 {
+	/* call inotify for this inode */
+	inotify_inodechange(inode,event);
 	if ((inode)->i_dnotify_mask & (event))
 		__inode_dir_notify(inode, event);
 }

diff -urN -X dontdiff vanilla/include/linux/inotify.h develop/include/linux/inotify.h
@@ -0,0 +1,17 @@
+/*
+ * Inode notification for Linux
+ *
+ * Copyright (C) 2003,2004 R�diger Klaehn
+ */
+
+#include <linux/fs.h>
+#define IN_ACCESS	0x00000001	/* Node accessed */
+#define IN_MODIFY	0x00000002	/* Node modified */
+#define IN_CREATE	0x00000004	/* Node created */
+#define IN_DELETE	0x00000008	/* Node removed */
+#define IN_RENAME	0x00000010	/* Node renamed */
+#define IN_ATTRIB	0x00000020	/* Node changed attibutes */
+
+extern void inotify_inodechange(struct inode *inode, unsigned long event);
+extern void inotify_dentrychange(struct dentry *dentry,unsigned long event);
+extern void inotify_dentrymove(struct dentry *dentry,struct dentry *dsrc,unsigned long event);

Well, there's fam. But AFAIK that's all done in user space, and your approach would be significantly more efficient (as a matter of fact, fam could be modified to use your change device as a first level of notification.)

I'll be interested in testing this, or (if you wish) help get it done. I'm a kernel hacking newbie at the moment, but I have tinkered around enough with the VFS to be able to work on this. Up to you.

Javier Fernandez-Ivern

Fam is a user space library that has some nice features such as network transparent change notification. It currently uses the dnotify mechanism if the underlying kernel supports it, but as I mentioned the dnotify mechanism requires an open file handle and works only for single directories. If the underlying os does not support dnotify, fam resorts to polling for file changes (yuk!).

When I have the basics worked out for the new mechanism, one thing I would like to do is to modify the fam library to use it. That would be a good way to test it, and it would also immediately benefit the big desktop environments which use fam.

Any help is appreciated. At the moment I am quite happy about what is being logged, but I am not sure how to best expose this to the interested user space processes. A device was the easiest way, and so that is what I used. Other possibilities would be a file in /proc or something completely different. I got the impression that dbus might be useful for this, but I have no idea how to use it.

If you want to help me, just apply the patch and see wether it works for you. You could also take a look at the code and see if you see something completely broken.

best regards and a happy new year,

Rüdiger

p.s. I attached a small test program. It just reads the contents of the /dev/inotify device and prints them in (somewhat) human readable form. For example to watch for changes in .txt files I would use inotify_test | grep txt. This would print out a line whenever a txt file anywhere on a system has been created or changed.

/*
* Inode notification for Linux
*
* Copyright (C) 2003,2004 R�diger Klaehn
*/

#define IN_ACCESS 0x00000001 /* Node accessed */
#define IN_MODIFY 0x00000002 /* Node modified */
#define IN_CREATE 0x00000004 /* Node created */
#define IN_DELETE 0x00000008 /* Node removed */
#define IN_RENAME 0x00000010 /* Node renamed */
#define IN_ATTRIB 0x00000020 /* Node changed attibutes */
#define DNAME_LEN 32

typedef struct

{
	unsigned long event;
	unsigned long file_ino;
	unsigned long src_ino;
	unsigned long dst_ino;
	unsigned char name[DNAME_LEN];
} in_info;

#include <stdio.h>
#include "inotify.h"

void geteventname(unsigned long event,char *buffer,int n)

{
	unsigned long events[6]=
	{
		IN_ACCESS,
		IN_MODIFY,
		IN_CREATE,
		IN_DELETE,
		IN_RENAME,
		IN_ATTRIB
	};
	const char *names[6]=
	{
		"IN_ACCESS",
		"IN_MODIFY",
		"IN_CREATE",
		"IN_DELETE",
		"IN_RENAME",
		"IN_ATTRIB"
	};
	int i;
	strncpy(buffer,"",n);
	for(i=0;i<6;i++)
	{
		if(event&events[i])
		{
			if(strlen(buffer)>0)
				strncat(buffer,"|",n);
			strncat(buffer,names[i],n);
		}
	}
}

int main()

{
	FILE *file;
	in_info info;
	int len;
	char eventname[256];
	file=fopen("/dev/inotify","rb");
	if(!file)
	{
		printf("Error: could not open /dev/inotify!\n");
		return -1;
	}

	while(1)
	{
		len=fread(&info,1,sizeof(info),file);
		if(len==sizeof(info)) {
			info.name[DNAME_LEN-1]=0;
			geteventname(info.event,eventname,255);	
			printf("%s %lu %lu %lu %s\n",
				eventname,
				info.file_ino,
				info.src_ino,
				info.dst_ino,
				info.name);
		}

		usleep(200);
	}
	fclose(file);
}

Rüdiger Klaehn wrote:

Rudiger, I've been reading your code to try and understand it, and I found one think I'm not so sure about:

...

Javier Fernandez-Ivern

Have you had a look at dazuko. It provides a consistent file access notification mechanism (and also intervention for denying access) across linux and freebsd. It is currently being used by various on-access virus scanners. It is under active development and supports 2.6 (and 2.4)

http://www.dazuko.org/about.shtml

Seems like a good idea. I've always thought it would be nice to use something like this to maintain a dynamic locatedb (among many other potential uses).

~mc

Candidate for "Wackiest sys_call_table patching 2004". In a word "ick". Code not to be read on a full stomach.

Dave

 Dave Jones     http://www.codemonkey.org.uk

Oh well, hadn't read the kernel code yet. Although certainly the *goal* is a good one - a cross platform interface for file change notification.

Yes I see what you mean after having a look. Would make sense to convert this over to use only LSM hooks (which it appears to use already) if that was possible - or maybe could be done using a VFS proxy filesystem that monitors/relays calls to an underlying fs.

The userspace interface appears relatively sane with its chardev interface although due to it's ability to intervene on file access, it creates a single point of failure that can livelock the machine if the userspace monitoring proccess dies. This could be worked around if the poilicy was to go unmonitored in the case that the userspace process dies.

~mc

I don't think using LSM hooks for this is a good idea. While it makes the changes less invasive, it forces you to enable LSM just to get file
change modifications.

Javier Fernandez-Ivern

Hi Rudi,

Am Mittwoch, 31. Dezember 2003 17:42 schrieb Rüdiger Klaehn:

I'm also interested in receiving file change notifications, especially as I would like to get this working for Samba in a sane way.

However I don't think your approach would help me much. I simply don't want to get every file being changed on the whole machine getting reported to me.
I don't want to look up the inode every time, just to know if it belongs to a directory I'm interested in.

Actually I *like* dnotify being local to a given directory and having a fd so I know where the signal I receive belongs to.

So my selfish reasoning makes me want either
- dnotify being able to pass some more information if requested (I actually tried this and it basically works, it is just too crappy to post here)
or
- make poll()/epoll() work for file/directory access

So much for what I want :-)

...Juergen

hmm...

#ln tree1/sub/dir/file tree2/sub/dir/file
#watch_tree tree1 &
#do_something_to tree2/sub/dir/file

A dnotify can potentially know about open, chown, chmod, utimes and possibly link of the files by watching the paths and cwd; meaning it won't know about alternate paths. How is it to know about read, write, fchown, fchmod and truncate?

Perhaps someone else has a more fertile imagination but short of looking up all the file inode numbers of the tree in question and watching that whole list this sounds futile.

jw schultz wrote:

[snip]

Take a look at fs/read_write.c. There are calls to dnotify_parent in all file operation functions. There is a comment in fs/dnotify.c which says that dnotify_parent is "hopelessly wrong, but unfixable without API changes". Another good reason for a new file change notification api...

The only thing I am not so sure about is mmap. I think a mmapped file will not create change notifications.

Whats wrong with that? You would just have to know the inode numbers of all directories in the subtree you are interested in. Then you can do a really fast inode->name translation using a hashtable or something. At least it is much more lightweight than having to open all directories :-)

I think that it is much cleaner and faster to report the inode numbers of the changed files since inode numbers are unique per filesystem and they are immediately available. The complicated mapping of inodes to path names should happen in user space only for the files the userspace process is interested in.

This seems to be a descendant of this one: <http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/openxdsm/openxdsm/eventmodule/>

It works, but it is a slow and ugly hack to just hook into every syscall that does something with files. And it is quite slow and ambiguous to log paths instead of inodes since file paths are not unique (hard links etc).

But I will take a look at that code to see how they get the events to user space.

File change notification would be immensely useful. I think there is a consensus about that.

Inode number are not necessarily unique per filesystem. Any filesystem that uses iget4 can have several objects that have the same inode number. For instance, Coda uses 128-bit file-identifiers and the i_ino number is a simple hash that is 'typically' unique. There are also filesystems that invent inode numbers whenever inodes are brought into the cache, but which have no persistency when the inode_cache is pruned. So the next time you see the same object, it could have a different (unique) inode number.

Returning paths is probably also not quite that easy when you only have the inode. Because of hardlinks we can have several paths (aliases) that lead to the same object, but the kernel does not necessarily have all of those paths available in the dcache.

Jan