PROBLEM: 2.6.35.7 Inotify events missing

Hi,

I write a tool that keep in sync one source (master) to multiple targets (slaves) to provide always up to date mirrors in order to distribute static content at different locations.

That was working fine on 2.6.26.5 kernel, except bugs that are my fault and which are fixed the way the time goes.

A day I upgraded to 2.6.33.5, then 2.6.33.7, finally to 2.6.35.7, and I always end up with the same ending, it seems inotify can miss some VFS events from time to time.

I wrote a simple tool which is a cleaned code of the first explained master process that only display received inotify events and with the recursion stuff that keep all directories watched.

First, this is not due to queue limit depth, limits are way over requirements. I don't received IN_Q_OVERFLOW event either.

$ cat /proc/sys/fs/inotify/max_{queued_events,user_instances,user_watches}
524288
128
1000000

That may help, files and FS tree are modified over NFS.

Here is what I got:

# stat ./xnlegacies/statique/wiki/cache/d/db303d42803e58bb5f73467291c9a863.xhtml
  File: `./xnlegacies/statique/wiki/cache/d/db303d42803e58bb5f73467291c9a863.xhtml'
  Size: 11696           Blocks: 24         IO Block: 4096   regular file
Device: 90ah/2314d      Inode: 144835214   Links: 1
Access: (0644/-rw-r--r--)  Uid: (20981/ UNKNOWN)   Gid: (26734/ UNKNOWN)
Access: 2010-03-18 14:04:59.000000000 +0000
Modify: 2010-10-14 06:44:27.000000000 +0000
Change: 2010-10-14 06:44:27.000000000 +0000

# stat ./xnlegacies/statique/wiki/cache/d/db303d42803e58bb5f73467291c9a863.i
  File: `./xnlegacies/statique/wiki/cache/d/db303d42803e58bb5f73467291c9a863.i'
  Size: 30575           Blocks: 64         IO Block: 4096   regular file
Device: 90ah/2314d      Inode: 144835393   Links: 1
Access: (0644/-rw-r--r--)  Uid: (20981/ UNKNOWN)   Gid: (26734/ UNKNOWN)
Access: 2010-03-18 14:04:59.000000000 +0000
Modify: 2010-10-14 06:44:26.000000000 +0000
Change: 2010-10-14 06:44:26.000000000 +0000

So, here is two files that were modified at about 2010-10-14 06:44:26.

And now is what I got in the event log of my inotify logger:

# grep db303d42803e58bb5f73467291c9a863 debugevents 
2010/10/12 20:06:48 1286914008:604497: wd=16623 mask=2 cookie=0 len=48 name=db303d42803e58bb5f73467291c9a863.i pathname=./xnlegacies/statique/wiki/cache/d/db303d42803e58bb5f73467291c9a863.i
2010/10/12 20:06:48 1286914008:605215: wd=16623 mask=2 cookie=0 len=48 name=db303d42803e58bb5f73467291c9a863.i pathname=./xnlegacies/statique/wiki/cache/d/db303d42803e58bb5f73467291c9a863.i
2010/10/12 20:06:48 1286914008:686793: wd=16623 mask=2 cookie=0 len=48 name=db303d42803e58bb5f73467291c9a863.xhtml pathname=./xnlegacies/statique/wiki/cache/d/db303d42803e58bb5f73467291c9a863.xhtml
2010/10/12 20:06:48 1286914008:687372: wd=16623 mask=2 cookie=0 len=48 name=db303d42803e58bb5f73467291c9a863.xhtml pathname=./xnlegacies/statique/wiki/cache/d/db303d42803e58bb5f73467291c9a863.xhtml

I did not receive events about the modification of those files at 2010-10-14 06:44, but we received previous events 2 days ago

Here is another example:

# stat ./synfig/forums/files/thumb_5_ec61d61306649e73f2549775c2ee3e1c   
  File: `./synfig/forums/files/thumb_5_ec61d61306649e73f2549775c2ee3e1c'
  Size: 23886           Blocks: 48         IO Block: 4096   regular file
Device: 90ah/2314d      Inode: 178390723   Links: 1
Access: (0664/-rw-rw-r--)  Uid: (15653/ UNKNOWN)   Gid: (27452/ UNKNOWN)
Access: 2010-10-15 05:28:35.000000000 +0000
Modify: 2010-10-15 05:28:35.000000000 +0000
Change: 2010-10-15 05:28:35.000000000 +0000

Here is a file that was modified at 05:28:35.

# grep thumb_5_ec61d61306649e73f2549775c2ee3e1c debugevents 
2010/10/15 05:28:35 1287120515:058819: wd=16079 mask=100 cookie=0 len=48 name=thumb_5_ec61d61306649e73f2549775c2ee3e1c pathname=./synfig/forums/files/thumb_5_ec61d61306649e73f2549775c2ee3e1c

I received an event that the file was created, at the time of the event the file was 0 byte length, the file was then filled, but we did not received events about the fact that the file was modified.

If I try to do the same, I get IN_CREATE, then IN_MODIFY,

# echo "test" > hFq0gUrK1EGwoT2pnpCO

# grep hFq0gUrK1EGwoT2pnpCO debugevents 
2010/10/18 19:40:59 1287430859:712722: wd=1 mask=100 cookie=0 len=32 name=hFq0gUrK1EGwoT2pnpCO pathname=./hFq0gUrK1EGwoT2pnpCO
2010/10/18 19:40:59 1287430859:713543: wd=1 mask=2 cookie=0 len=32 name=hFq0gUrK1EGwoT2pnpCO pathname=./hFq0gUrK1EGwoT2pnpCO

I attached my quick-n-dirty inotify logger. Source code of the full replication software is available here[1], if it helps.

I can provide as many information as possible and I am willing to do so ;)

Regards,
Sylvain

[1] svn://svn.tuxfamily.org/svnroot/vhffs4/vhffs/trunk/vhffs-fssync/

/*
* VHFFSFSSYNC: Scalable file system replication over TCP *
* Copyright 2008 Sylvain Rochet <[email protected]> *

 *  This program is free software; you can redistribute it and/or modify
 *  it under the terms of the GNU General Public License as published by
 *  the Free Software Foundation; either version 3 of the License, or
 *  (at your option) any later version.
 *
 *  This program is distributed in the hope that it will be useful,
 *  but WITHOUT ANY WARRANTY; without even the implied warranty of
 *  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 *  GNU General Public License for more details.

*
* You should have received a copy of the GNU General Public License
* along with this program. If not, see http://www.gnu.org/licenses/.
*/

#ifndef __linux__
#error This software is only running on Linux-based OS, bye! #endif

#define _FILE_OFFSET_BITS 64

#define DEBUG_NET 1
#define DEBUG_INOTIFY 1
#define DEBUG_EVENTS 1
//#define NDEBUG

#include <assert.h>
#include <unistd.h>
#include <stdio.h>
#include <stdlib.h>
#include <time.h>
#include <sys/time.h>
#include <fcntl.h>
#include <dirent.h>
#include <string.h>
#include <errno.h>
#include <sys/inotify.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <glib.h>
#include <sys/select.h>
#include <signal.h>
#include <sys/socket.h>
#include <netdb.h>
#include <arpa/inet.h>
#include <netinet/in.h>
#include <netinet/tcp.h>
#include <sys/sendfile.h>
#include <getopt.h>

char vhffsfssync_debugtsstr[256];

char *vhffsfssync_debugts() {
	time_t t;
	struct tm *tmp;
	struct timeval tv;

	t = time(NULL);
	tmp = localtime(&t);
	gettimeofday(&tv, NULL);

	strftime(vhffsfssync_debugtsstr, 200, "%Y/%m/%d %H:%M:%S ", tmp);
	sprintf(vhffsfssync_debugtsstr+strlen(vhffsfssync_debugtsstr), "%lld:%.6ld" , (long long)tv.tv_sec, (long)tv.tv_usec );

	return vhffsfssync_debugtsstr;
}

/* -- inotify stuff -- */

#define VHFFSFSSYNC_BUF_LEN 4096
//#define VHFFSFSSYNC_WATCH_MASK IN_ATTRIB|IN_CREATE|IN_DELETE|IN_CLOSE_WRITE|IN_MODIFY|IN_MOVED_FROM|IN_MOVED_TO|IN_DONT_FOLLOW|IN_ONLYDIR #define VHFFSFSSYNC_WATCH_MASK IN_ATTRIB|IN_CREATE|IN_DELETE|IN_MODIFY|IN_MOVED_FROM|IN_MOVED_TO|IN_DONT_FOLLOW|IN_ONLYDIR // Not used yet: IN_DELETE_SELF, IN_MOVE_SELF
// Will never be used: IN_ACCESS, IN_OPEN, IN_CLOSE_NOWRITE

// each monitor entry is associated with a path, we need to keep it to compute the path //char **vhffsfssync_wd_to_watch = NULL;
//int vhffsfssync_wd_to_watch_len = 0; // number of allocated paths GHashTable *vhffsfssync_wd_to_watch;

typedef struct vhffsfssync_watch_ {
	int wd;
	char *dirname;
	struct vhffsfssync_watch_ *parent;
} vhffsfssync_watch;

// return a timestamp in ms (it loops for 100000 sec)

/*inline int vhffsfssync_timestamp()  {
	struct timeval tv;
	gettimeofday(&tv, NULL);

return (tv.tv_sec%100000)*1000+tv.tv_usec/1000;
}*/

struct vhffsfssync_cookie {
	uint32_t id;
	vhffsfssync_watch *watch;
	char *filename;
	gboolean isdir;
};

static struct vhffsfssync_cookie vhffsfssync_cookie;

// protos
char *vhffsfssync_pathname(vhffsfssync_watch *watch, const char *filename);
vhffsfssync_watch *vhffsfssync_add_watch(int inotifyfd, vhffsfssync_watch *parent, const char *dirname, uint32_t mask); int vhffsfssync_del_watch(int inotifyfd, vhffsfssync_watch *watch);
vhffsfssync_watch *vhffsfssync_add_watch_recursively(int inotifyfd, vhffsfssync_watch *parent, const char *dirname, uint32_t mask); int vhffsfssync_manage_event_remove(int inotifyfd, vhffsfssync_watch *watch, char *filename);
int vhffsfssync_manage_event_create(int inotifyfd, vhffsfssync_watch *watch, char *filename);
int vhffsfssync_manage_event(int inotifyfd, struct inotify_event *event);
int vhffsfssync_fake_events_recursively(int inotifyfd, vhffsfssync_watch *watch);

/* -- inotify stuff -- */

char *vhffsfssync_pathname(vhffsfssync_watch *watch, const char *filename) {

	GString *pathname = g_string_sized_new(256);
	char **dirnames, **curnames, **endnames;
	uint32_t a;

	a = 16;
	dirnames = malloc( a * sizeof(char*) );
	curnames = dirnames;
	endnames = dirnames+a;

	if(filename) {
		*(curnames++) = (char*)filename;
	}

	while(watch) {
		*(curnames++) = watch->dirname;
		watch = watch->parent;
		if(curnames == endnames) {
			a += 16;
			dirnames = realloc( dirnames, a * sizeof(char*) );
			curnames = dirnames+a-16;
			endnames = dirnames+a;
		}
	}

	curnames--;
	g_string_append(pathname, *(curnames--) );
	while( curnames >= dirnames ) {
		g_string_append_c(pathname, '/');
		g_string_append(pathname, *(curnames--)  );
	}
	free(dirnames);

	return g_string_free(pathname, FALSE);
}

vhffsfssync_watch *vhffsfssync_add_watch(int inotifyfd, vhffsfssync_watch *parent, const char *dirname, uint32_t mask) {

	int wd;
	char *pathname;
	vhffsfssync_watch *watch;

	pathname = vhffsfssync_pathname(parent, dirname);
#if DEBUG_INOTIFY
	printf("%s: t+ %s\n", vhffsfssync_debugts(), pathname);

#endif

	wd = inotify_add_watch(inotifyfd, pathname, mask);
	if(wd < 0) {
		if(errno == ENOSPC)  {
			fprintf(stdout, "%s: Maximum number of watches reached, consider adding more...\n", vhffsfssync_debugts() );
		}
		free(pathname);
		return NULL;
	}

if( (watch = g_hash_table_lookup(vhffsfssync_wd_to_watch, &wd)) ) {

		// this was already watched, update name and reattach to the new parent
		free(watch->dirname);
		watch->dirname = g_strdup(dirname);
		watch->parent = parent;

#if DEBUG_INOTIFY
		printf("%s: u+ %d %s\n", vhffsfssync_debugts(), wd, pathname);
#endif
		free(pathname);
		return watch;
	}

	watch = malloc(sizeof(vhffsfssync_watch));
	watch->wd = wd;
	watch->dirname = g_strdup(dirname);
	watch->parent = parent;

// _wd = g_new(int, 1);

//	*_wd = wd;
	g_hash_table_insert(vhffsfssync_wd_to_watch, &watch->wd, watch);

// if(wd >= vhffsfssync_wd_to_watch_len) {
// vhffsfssync_wd_to_watch_len = ( (wd >>10) +1) <<10;
// vhffsfssync_wd_to_watch = realloc( vhffsfssync_wd_to_watch, vhffsfssync_wd_to_watch_len * sizeof(void*) );
// }
// vhffsfssync_wd_to_watch[wd] = strdup(pathname);

#if DEBUG_INOTIFY
	printf("%s: a+ %d %s\n", vhffsfssync_debugts(), wd, pathname);
#endif
	free(pathname);
	return watch;
}

int vhffsfssync_del_watch(int inotifyfd, vhffsfssync_watch *watch) {

#if DEBUG_INOTIFY
	char *pathname = vhffsfssync_pathname(watch, NULL);
	printf("%s: - %d %s\n", vhffsfssync_debugts(), watch->wd, pathname);
	free(pathname);
#endif
	g_hash_table_remove(vhffsfssync_wd_to_watch, &watch->wd);
	inotify_rm_watch(inotifyfd, watch->wd);
	free(watch->dirname);
	free(watch);
	return 0;
}

vhffsfssync_watch *vhffsfssync_add_watch_recursively(int inotifyfd, vhffsfssync_watch *parent, const char *dirname, uint32_t mask) {

	vhffsfssync_watch *watch;
	char *pathname;
	DIR *d;

	watch = vhffsfssync_add_watch(inotifyfd, parent, dirname, mask);
	if(!watch) return NULL;

	pathname = vhffsfssync_pathname(parent, dirname);
	d = opendir(pathname);
	free(pathname);
	if(d) {
		struct dirent *dir;
		while( (dir = readdir(d)) )  {
			if(dir->d_type == DT_DIR && strcmp(dir->d_name, ".") && strcmp(dir->d_name, "..") )  {
				vhffsfssync_add_watch_recursively(inotifyfd, watch, dir->d_name, mask);
			}
		}
		closedir(d);
	}

	return watch;
}

int vhffsfssync_manage_event_remove(int inotifyfd, vhffsfssync_watch *watch, char *filename)  {
	char *pathname;

	pathname = vhffsfssync_pathname(watch, filename);

#if DEBUG_INOTIFY
	printf("%s: ==> REMOVE %s\n", vhffsfssync_debugts(), pathname);

#endif

	free(pathname);
	return 0;
}

int vhffsfssync_manage_event_create(int inotifyfd, vhffsfssync_watch *watch, char *filename)  {
	struct stat st;
	char *pathname;

	pathname = vhffsfssync_pathname(watch, filename);

	if(! lstat(pathname, &st) )  {

		if( S_ISREG(st.st_mode) )  {
#if DEBUG_INOTIFY
			printf("%s: ==> CREATE %s\n", vhffsfssync_debugts(), pathname);
#endif
		}

		else if( S_ISDIR(st.st_mode) )  {
			vhffsfssync_watch *newwatch;
#if DEBUG_INOTIFY
			printf("%s: ==> MKDIR %s\n", vhffsfssync_debugts(), pathname);
#endif
			newwatch = vhffsfssync_add_watch(inotifyfd, watch, filename, VHFFSFSSYNC_WATCH_MASK);
			/* there is a short delay between the mkdir() and the add_watch(),
			   we need to send events about the data which have already been written */
			vhffsfssync_fake_events_recursively( inotifyfd, newwatch );
		}

		else if( S_ISLNK(st.st_mode) )  {
			char *linkto;
			int ret;
			linkto = malloc(st.st_size +1);
			ret = readlink(pathname, linkto, st.st_size);
			if( ret >= 0 )  {
				linkto[st.st_size] = '\0';
#if DEBUG_INOTIFY
				printf("%s: ==> SYMLINK %s -> %s\n", vhffsfssync_debugts(), pathname, linkto);
#endif
			}
			free(linkto);
			if(ret < 0) {
				if(errno == ENOENT) {
					// file already disappeared (common for temporary files)
				} else {
					fprintf(stdout, "%s: cannot readlink() '%s': %s\n", vhffsfssync_debugts(), pathname, strerror(errno));
					free(pathname);
					return -1;
				}
			}

		}
		/* we don't need other file types (chr, block, fifo, socket, ...) */
	}
	else {
		if(errno == ENOENT) {
			// file already disappeared (common for temporary files)
		} else {
			fprintf(stdout, "%s: cannot lstat() '%s': %s\n", vhffsfssync_debugts(), pathname, strerror(errno));
			free(pathname);
			return -1;
		}
	}

	free(pathname);
	return 0;
}

int vhffsfssync_manage_event(int inotifyfd, struct inotify_event *event) {

	vhffsfssync_watch *watch;
	char *pathname;
#if DEBUG_INOTIFY
	printf("%s: wd=%d mask=%x cookie=%d len=%d", vhffsfssync_debugts(), event->wd, event->mask, event->cookie, event->len);
	if(event->len > 0) printf(" name=%s", event->name);

#endif

	if(event->wd < 0) {
		fprintf(stdout, "\n%s: Maximum number of events reached, some events are lost\n", vhffsfssync_debugts() );
		return -1;
	}

	watch = g_hash_table_lookup(vhffsfssync_wd_to_watch, &event->wd);
	assert( watch != NULL );

	if(event->len > 0)  {
		pathname = vhffsfssync_pathname(watch, event->name);
	} else {
		pathname = vhffsfssync_pathname(watch, NULL);
	}

	printf(" pathname=%s\n", pathname);

	// this event is not waiting for a cookie, delete file if necessary (IN_MOVED_FROM not followed with IN_MOVED_TO)
	if( !(event->mask & IN_MOVED_TO) && vhffsfssync_cookie.id )  {

		vhffsfssync_manage_event_remove(inotifyfd, vhffsfssync_cookie.watch, vhffsfssync_cookie.filename);
		vhffsfssync_cookie.id = 0;
		free(vhffsfssync_cookie.filename);
	}

	// new mtime, mode, owner, group (and also other stuff like atime, but we are not using them)
	if( event->mask & IN_ATTRIB )  {
		struct stat st;
#if DEBUG_INOTIFY
		printf("%s: IN_ATTRIB\n", vhffsfssync_debugts() );
#endif
		if(! lstat(pathname, &st) )  {
		
		}
		else {
			if(errno == ENOENT) {
				// file already disappeared (common for temporary files)
			} else {
				fprintf(stdout, "%s: cannot lstat() '%s': %s\n", vhffsfssync_debugts(), pathname, strerror(errno));
			}
		}

	// new file, directory, or symlink
	} else if( event->mask & IN_CREATE )  {
#if DEBUG_INOTIFY
		printf("%s: IN_CREATE\n", vhffsfssync_debugts() );
#endif
		vhffsfssync_manage_event_create(inotifyfd, watch, event->name);

	// deleted file, directory or symlink
	} else if( event->mask & IN_DELETE )  {
#if DEBUG_INOTIFY
		printf("%s: IN_DELETE\n", vhffsfssync_debugts() );
#endif
		vhffsfssync_manage_event_remove(inotifyfd, watch, event->name);

	// watch deleted, not used
	} else if( event->mask & IN_DELETE_SELF )  {
#if DEBUG_INOTIFY
		printf("%s: IN_DELETE_SELF\n", vhffsfssync_debugts());
#endif
		// We don't send REMOVE here because the dir can be deleted before the
		// event was added, in this case the add_watch failed to monitor this dir
		// and we'll not receive a IN_DELETE_SELF for it
		//
		// Anyway, a IN_IGNORE event will be sent, IN_DELETE_SELF is only
		// useful for monitored files, that is not used here.

	// file modified
	} else if( event->mask & IN_MODIFY )  {
#if DEBUG_INOTIFY
		printf("%s: IN_MODIFY\n", vhffsfssync_debugts() );
		/* we can send the data here */
		printf("%s: ==> SEND %s\n", vhffsfssync_debugts(), pathname);

#endif

	// file modified and closed
	} else if( event->mask & IN_CLOSE_WRITE )  {
#if DEBUG_INOTIFY
		printf("%s: IN_CLOSE_WRITE\n", vhffsfssync_debugts() );
		/* we must send the data here */
		printf("%s: ==> SEND %s\n", vhffsfssync_debugts(), pathname);

#endif

	// watch moved, not used
	} else if( event->mask & IN_MOVE_SELF )  {
#if DEBUG_INOTIFY
		printf("%s: IN_MOVE_SELF\n", vhffsfssync_debugts() );
#endif
		// not needed (we can rely on IN_MOVED_FROM and IN_MOVED_TO)

	// file/symlink/directory moved
	//
	// only from: delete the file/symlink/directory
	// only to: create the file/symlink/directory
	// both: mv the file/symlink/directory
	//
	} else if( event->mask & IN_MOVED_FROM )  {
#if DEBUG_INOTIFY
		printf("%s: IN_MOVED_FROM\n", vhffsfssync_debugts() );
#endif
		// set the cookie
		vhffsfssync_cookie.id = event->cookie;
		vhffsfssync_cookie.watch = watch;
		vhffsfssync_cookie.filename = strdup(event->name);
		vhffsfssync_cookie.isdir = !!( event->mask & IN_ISDIR );

	} else if( event->mask & IN_MOVED_TO )  {
#if DEBUG_INOTIFY
		printf("%s: IN_MOVED_TO\n", vhffsfssync_debugts() );
#endif
		// mv
		if(vhffsfssync_cookie.id == event->cookie)  {
#if DEBUG_INOTIFY
			char *tmp = vhffsfssync_pathname(vhffsfssync_cookie.watch, vhffsfssync_cookie.filename);
			printf("%s: ==> MOVE (%d -> %d) %s -> %s  (used cookie %d)\n", vhffsfssync_debugts(), vhffsfssync_cookie.watch->wd, watch->wd, tmp, pathname, vhffsfssync_cookie.id);
			free(tmp);
#endif
			if( vhffsfssync_cookie.isdir )  {

				char *frompathname = vhffsfssync_pathname(vhffsfssync_cookie.watch, vhffsfssync_cookie.filename);
				vhffsfssync_add_watch(inotifyfd, watch, event->name, VHFFSFSSYNC_WATCH_MASK);
				free(frompathname);
			}
			else {
				vhffsfssync_manage_event_remove(inotifyfd, vhffsfssync_cookie.watch, vhffsfssync_cookie.filename);
				vhffsfssync_manage_event_create(inotifyfd, watch, event->name);
			}

			vhffsfssync_cookie.id = 0;
			free(vhffsfssync_cookie.filename);
		}
		// create
		else  {
			vhffsfssync_manage_event_create(inotifyfd, watch, event->name);
		}

	// watch deleted, clean it
	} else if( event->mask & IN_IGNORED )  {
#if DEBUG_INOTIFY
		printf("%s: IN_IGNORED\n", vhffsfssync_debugts() );
#endif
		vhffsfssync_del_watch(inotifyfd, watch);

	// this event is not handled, this should not happen
	} else {
#if DEBUG_INOTIFY
		printf("%s: OOOOOOOPPPSS!!!!!\n", vhffsfssync_debugts()  );
#endif
	}

	free(pathname);
	return 0;
}

int vhffsfssync_fake_events_recursively(int inotifyfd, vhffsfssync_watch *watch)  {
	DIR *d;
	char *pathname;

	pathname = vhffsfssync_pathname(watch, NULL);
	d = opendir(pathname);
	free(pathname);

	if(d) {
		struct dirent *dir;
		while( (dir = readdir(d)) ) {
			if( strcmp(dir->d_name, ".") && strcmp(dir->d_name, "..") ) {
				// recursivity is done through vhffsfssync_manage_event_create()
				// which calls this function
				vhffsfssync_manage_event_create(inotifyfd, watch, dir->d_name);
			}
		}
		closedir(d);
	}

	return 0;
}

static void usage_exit(int ret_code, char *progname)  {
	printf ("Usage: %s [OPTION]... DIRECTORY\n"
		"Remote synchronous file-copying tool, this is the server (the master)\n\n"
		"  -f, --foreground\tDon't daemonise the server, display errors on the console\n"
		"  -b, --bind=IP\t\tListen to the specified IP address\n"
		"  -p, --port=PORT\tListen to this port\n"
		"      --pidfile=PATH\tWrite the pid to that file\n"
		"  -h, --help\t\tDisplay this help and exit\n"
		"  -v, --version\t\tOutput version information and exit\n",
		progname);
	exit(ret_code);
}

int main(int argc, char *argv[]) {

	int inotifyfd, flags;
	vhffsfssync_watch *watch;

	char *root = NULL;

	struct option long_options[] = {
		{ "help", no_argument, NULL, 'h' },
		{ "version", no_argument, NULL, 'v' },
		{ 0, 0, 0, 0 }
	};

	while(1) {
		int option_index = 0, c;
		c = getopt_long(argc, argv, "hv", long_options, &option_index);
		if(c == -1)
			break;

		switch(c) {
			case 'h':
				usage_exit(0, argv[0]);

			case 'v':
#ifdef VERSION
				fputs("vhffsfssync_master " VERSION "\n", stdout);
#else
				fputs("vhffsfssync_master\n", stdout);
#endif
				exit(0);

			case '?':
				/* `getopt_long' already printed an error message. */
				fprintf(stdout,"Try `%s --help' for more information.\n", argv[0]);
				exit(1);

			default:
				abort();
		}
	}

	if(optind != argc-1)
		usage_exit(1, argv[0]);

	root = argv[optind++];

	/* chdir() to the filesystem to monitor */
	if(!root) return -1;
	if( root[strlen(root)-1] == '/' ) root[strlen(root)-1] = '\0';
#if DEBUG_INOTIFY
	printf("%s: Monitoring %s\n", vhffsfssync_debugts(), root);
#endif
	if( chdir(root) < 0 ) {
		fprintf(stdout, "%s: cannot chdir() to %s: %s\n", vhffsfssync_debugts(), root, strerror(errno));
		return -1;
	}
	if( chroot(".") < 0 ) {
		fprintf(stdout, "%s: cannot chroot() to %s: %s\n", vhffsfssync_debugts(), root, strerror(errno));
		//return -1;
	}
	root = ".";

	/* -- inotify stuff -- */

	vhffsfssync_wd_to_watch = g_hash_table_new_full(g_int_hash, g_int_equal, NULL, NULL);
	vhffsfssync_cookie.watch = NULL;
	vhffsfssync_cookie.id = 0;
	vhffsfssync_cookie.filename = NULL;

	inotifyfd = inotify_init();

	/* set inotifyfd to non-blocking */
	flags = fcntl(inotifyfd, F_GETFL);
	if(flags >= 0) {
		flags |= O_NONBLOCK;
		fcntl(inotifyfd, F_SETFL, flags);
	}

	watch = vhffsfssync_add_watch_recursively(inotifyfd, NULL, root, VHFFSFSSYNC_WATCH_MASK);
	if(!watch)  {
		fprintf(stdout, "%s: Maximum number of watches probably reached, consider adding more or fixing what is being wrong before running me again (strace is your friend)... byebye!\n", vhffsfssync_debugts() );
		return -1;
	}

	printf("%s: Ready\n", vhffsfssync_debugts() );

	/* -- main loop -- */
	while(1)  {
		int max_fd = 0;
		fd_set readfs;
		int ret;

		FD_ZERO(&readfs);

		/* inotify events */
		FD_SET(inotifyfd, &readfs);
		if(inotifyfd > max_fd) max_fd = inotifyfd;

		ret = select(max_fd + 1, &readfs, NULL, NULL, NULL);
		if(ret < 0)  {
			switch(errno)  {
				case EAGAIN:
				case EINTR:
					break;
				default:
					fprintf(stdout, "%s: select() failed: %s\n", vhffsfssync_debugts(), strerror(errno));
			}
		}
		if(ret > 0)  {
			/* inotify events */
			if( FD_ISSET(inotifyfd, &readfs) )  {
				char buf[VHFFSFSSYNC_BUF_LEN];
				ssize_t len;

				len = read(inotifyfd, buf, VHFFSFSSYNC_BUF_LEN);
				if(len < 0)  {
					switch(errno)  {
						case EAGAIN:
						case EINTR:
							break;
						default:
							fprintf(stdout, "%s: read() failed on inotify fd(%d): %s\n", vhffsfssync_debugts(), inotifyfd, strerror(errno));
					}
				}
				else {
					char *cur = buf;
					int n=0;
					while(len > 0)  {
						int register next;
						struct inotify_event *ie;

						ie = (struct inotify_event*)cur;
						vhffsfssync_manage_event( inotifyfd, ie );
						next = sizeof(struct inotify_event);
						next += ie->len;
						len -= next;
						cur += next;
						n++;
					}
					//printf("COIN: %d events read\n", n);;
				}
			}

		}

		fflush(stdout);
//		sleep(10);
	}

	return 0;
}

Hi,

I finally find out why.

The NFS server does not always know the name of the modified file, if the modified inode was cleared from the VFS cache fsnotify does not know as well the filename then inotify child events on directories are silently tossed.

Easy way to reproduce:

Add a few printk debug (here it only works if /data is the NFS export):

@@ -975,6 +975,8 @@
        inode = dentry->d_inode;
        exp   = fhp->fh_export;
 
+       printk("nfsd write inode=%ld name=%s\n", inode->i_ino, dentry->d_name.name);
+
        /*
         * Request sync writes if
         *  -   the sync export option has been set, or

diff -Nru begin//include/linux/fsnotify.h linux-3.0/include/linux/fsnotify.h
@@ -216,8 +232,15 @@
                mask |= FS_ISDIR;
 
        if (!(file->f_mode & FMODE_NONOTIFY)) {
+               if( !strcmp(path->mnt->mnt_mountpoint->d_name.name, "data") )
+                       printk("fsnotify modify inode=%ld name=%s\n", inode->i_ino, file->f_dentry->d_name.name);
                fsnotify_parent(path, NULL, mask);
                fsnotify(inode, mask, path, FSNOTIFY_EVENT_PATH, NULL, 0);
+       } else {
+               if( !strcmp(path->mnt->mnt_mountpoint->d_name.name, "data") )
+                       printk("fsnotify modify-nonotify inode=%ld name=%s\n", inode->i_ino, file->f_dentry->d_name.name);
        }
 }
 

On the NFS client, open a fd and send some data:

# exec 1> test
# ls -la
# 

On the NFS server, check the kern log:

Aug 20 00:57:44 inotifydebug kernel: nfsd write inode=13 name=test
Aug 20 00:57:44 inotifydebug kernel: fsnotify modify inode=13 name=test

Everything goes well.

Now, clear the VFS cache on the NFS server:

# echo 3 > /proc/sys/vm/drop_caches 

On the NFS client, send some data to the fd:

# ls -la
# 

On the NFS server, check the kern log:

Aug 20 00:58:56 inotifydebug kernel: nfsd write inode=13 name=
Aug 20 00:58:56 inotifydebug kernel: fsnotify modify inode=13 name=

The filename is lost, fsnotify does not know the filename anymore, therefore inotify cannot send event about a modified file in a watched directory.

End of the story.

I guess this is almost impossible to fix this fsnotify bug, this is due by the fact that NFS use inode as file identifiers, so in some case this is impossible to know the modified filepath, and therefore impossible to match the file event to the directory watch.

Kind regards,
Sylvain

Oh dear, that's a security hole, if something is using inotify/dnotify to watch and assumes that file contents (on the same machine, i.e. server in this case) do not change if there's no event received.

It also breaks cache applications which make the same assumption. Is a solution to open inotify watches on every file individually? If so that seems quite severe.

I do quite like the idea of using it to break past fanotify security restrictions though ;-)

Can it also be bypassed with sys_open_by_handle_at?

Possible solution:

One way to look at this as like NFS having a secret hard link to the file, which does not show up in st_nlink.

Hard links are already a bit tricky with fsnotify and directory watches. You can monitor a directory, but a file in it can change contents through another path.

However, you can track changes of hard-linked files accurately by either putting a watch directly on all files whose st_nlink >= 2, and/or making sure you have watches on enough distinct directories that they contain st_nlink entries for the same file between them, because at least one of those directories will get an event. This is quite practical: You watch the files directly, until such time as you have found all its links (if you ever do), then you can remove the direct file watches.

That gives me an idea to help with the NFS no-name watching:

It looks like when a file is referenced by inode without a path, the problem is there's no path, so no directory inode to receive the event?

Then this can be solved, in principle (if there's no better way), by watching a "virtual directory" that gets all events for when the access doesn't have a parent directory. There needs to be some way to watch it, and some way to get the appropriate file from the event (as there is no real directory. Or maybe there could be a virtual filesystem (like /proc, /sys etc.) containing a magic directory that receives these inode-only events, such that lookups in that directory yield the affected file. Exactly as if the directory contains a hard link to every file, perhaps a text encoding of the handles passed through sys_open_by_handle_at.

-- Jamie
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

Hi Jamie,

It also probably means that fanotify misses some events when a filesystem is modified over NFS. If fanotify is used the way it is designed, i.e. with an antiviruse software, this may be an interesting way to skip the antiviruse check.

Here we go:

NFS server, run the fanotify example tool:

~/fanotify-example# ./fanotify -m /data/

NFS client, open a fd then do some I/O:

# exec 1> test
# ls -la
# 

NFS server log:

/data/test: pid=1235 modify close(writable)

NFS server, cache clearing:

# echo 3 > /proc/sys/vm/drop_caches 

NFS client, more I/O:

# ls -la

NFS server log:

/data: pid=1234 modify close(writable)

We receive an event... which is obviously wrong. This is even worse than no event at all, we receive an event about the wrong inode, the parent inode of the modified file actually.

This is what I am going to do, at least temporarily, I only need to watch about a million file (and slowly counting).

The startup time to watch an entire filesystem using inotify already require a full filesystem walk, watching all files and directories instead of directories only will not change much because most of the time is spent waiting I/O operations. This may however require a lot more memory both on kernel side and userland side.

No clue, this should be checked but there is no evident reason that it cannot be bypassed this way as well.

Yup, I agree.

No filepath and no filename at all actually. There is no way to find the (or "a" if the file is linked to more than one directory) filename other than walking among all the directory tree to find where the inode is linked. We need a directory entry (dirent) to send an event about a modified file inside a watched directory.

By doing that, we'll only get the inode nb as we cannot fetch the filename.

Sylvain

That sounds like a proper bug, maybe it can be fixed at least?

Watching an entire filesystem entails reading all the directories, but you don't have to fetch the inodes of files. But still, it's very slow (takes about 15 minutes on my /home from cold cache, just to read the million or so directories).

There was some work on propagating events upwards so that efficient recursive watches could be established, in the context of fanotify but it would make sense to be available to all fsnotify users. I wonder how that went.

Yes... That's ok if it's one we are tracking inode->multiple-paths in userspace anyway (for hard links). But it's quite demanding if we hoped to avoid fetching and storing that in userspace for st_nlink == 1 files.

In that case it is still better to get a notification "something unknown on this FS has changed", rather than no notification. Userspace would react by flushing all of its cached knowledge of things under directory watches that don't have direct watches. But at least that's reliable and correct behaviour, and if it happens often, userspace heuristics can react by watching priority inodes more directly.

If that's the common case, then these nameless, pathless events could just trigger a simple event with catch-all IN_NO_PATH flag set, referring to the filesytem but no more detail than that. inotify would accept that flag when adding a watch, ignore the inode given but remember the filesystem, and send all events with no path to the watch(es) created with that flag on that filesystem. It's a flag because the event type is still useful.

All the best,
-- Jamie

There is a better way - stop using idiotify... It has always been a mistake, driven down our throats by filemangler and desktop crowd. Broken in many, _many_ respects... Deprecate that crap, remove it completely in a couple of revisions, let these clowns cope. Yeah, yeah, I know... Not going to happen ;-/ One can dream, though...

What do you suggest as a functioning alternative to inotify?

---
~Randy

Depends on what particular (ab)use do you have in mind...

Well you still have your sense of humour...

I've never understood why you think it's about the file manager / desktop, or why you so strongly dislike the feature. It originated there historically, but that is not it's primary use.

The implementation, sure, but you seem to dislike the very *principle* of subscribing to changes.

Every interesting use of inotify that I've seen is for some kind of cache support, to eliminate the majority of stat() calls, to remove disk I/O (no stat means no inode), to ensure correctness (st_mtime is coarse and unreliable), and to avoid having to modify every application which might affect any file from which cached items are derived to explicitly notify all the applications which might use any of those files.

You like high performance, reliable and correct behaviour, and high scalability. So I have never understood why you dislike the change-subscription principle so strongly, because it is a natural ally to those properties.

All the best,
-- Jamie
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

It seems rather fragile as an mtime replacement unless it's also got some sort of logging built in at a pretty low level so that you don't lose events while you're not listening.

And of course events have to be defined very carefully to avoid problems such as this one.

I don't think we've seen a design that does all of that yet.

--b.
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

It mainly serves as an accelerator for existing stat/mtime checks, though it does improve change detection in the last second or so since a previous change, which with mtime you have to make pessimistic or sometimes-incorrect assumptions.

Quite a few programs use inotify now because it saves a little power, and is a bit more responsive than, say, polling config files with stat().

For reliable filesystem tracking across times when not listening, especially if you don't trust the clock to have no backward steps (and you should not), a lazy change count file attribute would do. It's been discussed but never implemented.

This thread has revealed quite a big hole, I agree. Apps cannot even use their normal filesystem-type whitelisting to catch this. This is bad!

It is not the first hole that was found in inotify/dnotify, but it's the first one I'm aware of that wasn't pointed out long ago and then quietly ignored :-/

Designs get discussed from time to time, over the decades.

I think one of the reasons it doesn't go further is Al's well-known objection -- why put the effort in if you know it will be rejected. And a widespread view that it's just unimportant GUI file manager fluff. The latter also means dependability issues have tended not to be taken seriously.

-- Jamie
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

I know you weren't asking for design suggestions, but somehow I just couldn't help myself :-)

The (or "a") problem with {d,i,fa}notify is that it makes a core assumption that is flawed. i.e. that a file is in some directory. It might be nice if that were a reliable fact but thanks to our founding fathers, it is not. If a file only ever had one name - never more nor less - and could not have that name changed while it were open, then quite a lot of things would be a lot easier. And probably a lot of things would be a lot harder. But we don't live in that world (others do - I think you know where it is).

So we must drop this assumption.

Getting notification on an fd when the opened file changes makes perfect sense. Some /proc and /sys files already provide this functionality and we can expect that more will. Adding that to regular filesystems may not be out of the question. This would be useful, but of limited use. You could find out when a given file changed - either an mtime-like change or a ctime-like change. By monitoring a directory you could find out when a name was added or removed. But to find out when "any file in a directory changes" you would need to open and monitor every file, which is expensive.

The other ("another") problem is the lack of recursion. You can find out when a file in a directory changes, but not a file in a directory tree. This
significantly reduces the value. We really want to know about directory
trees. However a "directory tree" - much like "all the files in a directory" isn't really a very well defined concept - at least from the perspective of providing notifications. You cannot easily answer "is this file in that tree?" or "which tree(s) is this file in?".

However there are well defined sets of files such that we could reliably generate notifications if any file in the set were changed, or if a file were added-to or remove-from the set. We should be looking for these sorts of sets and seeing which are useful.

e.g.
- all files in a given filesystem. Generating notification for any change
in a given filesystem is a well defined task. It might generate too much noise, but it would still have a place.

- all files with a given uid (or gid).

- all directories. or all regular files

- all setuid, setgid, or world-writable files

Each of these are strongly defined and we can map from file to set quite easily. We could obviously intersect the sets to, so I could get events when any directory owned by me on a particular filesystem was changed. It would even be reasonable for the events to contain a newly opened fd from which I can extract dev/inode info and possibly extract a path name.

However this still might not be fine-grained enough. While a "directory tree" is not really a well defined concept, it is in my mind. e.g. it seems reasonable to want to find out about all changes in $HOME/.config

I can see two approaches to this - though there might be others. All of them must in some way create a strong concept of a directory tree.

One is to use bind mounts. i.e. I effectively do

    mount --bind $HOME/.config $HOME/.config

and ask for events from the newly created vfsmnt.
This will not catch changes made through file descriptors that were opened before I did the mount, or through hard links from some other directory tree. But for a particular use-case that might not be a problem.

The other requires support from the filesystem and so cannot be provided universally. It could possibly be imposed generically for filesystems that support extended attributes .... but I feel dirty even suggesting that (Dobby must now go and iron his hands!)

The filesystem could support the concept of a 'directory tree' much like BTRFS allows subvolumes which a like independent filesystems within the one big filesystem. However for this purpose the 'directory tree' would be a very light weight concept (it wouldn't need its own inode number space).

For example, each inode could store an extra number which is the inode number of the root of its directory tree. This would be inherited from parent during create. Renaming or linking a file would fail if the target had a different directory tree number. (renaming a file with only one link might
succeed and change the directory tree number). An empty directory could be
told to become a root somehow.

Then you would have a strong concept of a directory tree that could be used for notifications.
Obviously this approach could not be used to solve any immediate problems. But if new filesystems started supporting light-weight-directory-trees as a well defined set of files, then in 5-10 years we might have a nice working solution.

[of course then you need to layer any design you come up with on NFS ... but that can probably be done in user-space with libraries and daemons].

NeilBrown

I'm missing what the extra vfsmount gets you here. The problems seem just the same as if you don't have one.

Oh, wait, I see, it's that the file descriptors are associated with vfsmounts, not just dentries. Hm.

--b.

"Hm" might be right. writes have a 'struct file', but mkdir and rename etc just get an inode.
So to be able to notify based on vfsmnt, mkdirat - for examine - would need to pass the path to vfs_mkdir rather than path.dentry->d_inode, and vfs_mkdir would have to pass that path to fsnotify_mkdir. So more intrusive that I imagined, but still quite do-able.... if it were thought to be useful.

NeilBrown

--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

Hi,

Just checked that and... as planned it can be bypassed with name_to_handle_at/open_by_handle_at as well.

(I added my quick'n'dirty test code, maybe someone will be interested by a working fhandle example on Linux 3.0.x).

Sylvain

#include <stdio.h>
#include <stdlib.h>

#include <fcntl.h>
#include <unistd.h>
#include <errno.h>
#include <string.h>

struct file_handle {
	int handle_size;
	int handle_type;
	void *f_handle;
};

int main(int argc, char *argv[])

{
	int ret;
	int fd;
	int mntfd;
	struct file_handle fh;
	fh.handle_type = 0;
	fh.f_handle = malloc(100);
	fh.handle_size = 100/sizeof(int);
	errno  = 0;

	printf("%s\n", argv[1]);

	int mntid = 20;

	ret = syscall(303, 0, argv[1], &fh, &mntid, 0);
	if (ret) {
		perror("303 Error:");
		exit(1);
	}
	printf("%d - %d\n", mntid, fh.handle_size);

	printf("sleep\n");
	sleep(60);


	mntfd = open("/data", O_RDONLY);
	printf("dir = %d\n", mntfd );

	fd = syscall(304, mntfd, &fh, O_WRONLY);
	if (fd < 0 ) {
		perror("304 Error:");
		exit(1);
	}
	printf("fd = %d\n", fd);


	write(fd, "toto", 4);


	if( close(fd) ) {
		printf("close failed\n");
	}

	return 0;
}

This may not be helpful but if you export the filesystem with the "subtree_check" option then the NFS server will always know a name for any file that it modifies.

This is not without cost though, which is why it is no longer the default.

If the client opens a file, the server moves it to a new directory and then the server cache is flushed, then the client will get an ESTALE error. This is rarely a problem in practice but when it is it can be quite annoying...

NeilBrown