Skip to contentSkip to conversation

Filessytem Events Reporter V3

PATCH In review Yi Yang 2 participants 6 messages Apr 10, 2006 – Apr 11, 2006

Subsystems (3)
lore
Share & tools
lore

Yi Yang

author
#0 lore raw

PATCH 1/1 Filessytem Events Reporter V3

Compared with Filesystem Events Reporter v2, the following changes are done:
- Can be built as module.
- Fix some bugs pointed out by Evgeniy
- Substitute spinlock with mutex
- Complete exit cleanup

This patch implements a filsystem events report facitily, Filesystem Events Reporter, the user can monitor filesystem activities via it, currently, it
can monitor access, attribute change, open, create, modify, delete, move and close of any file or directory as well as mount/umount.

Every filesystem event will include tgid, uid and gid of the process
which triggered this event, process name, file or directory name
operated by it.

Filesystem Events Reporter is never a duplicate of inotify, inotify
just concerns change on file or directory, Beagle uses it to watch file changes in order to regenerate index for it, inotify can't tell us who did that change and what is its process name, but filesystem events reporter can do these, moreover inotify's overhead is greater than filesystem events reporter, inotify needs compare inode with
watched file or directories list to decide whether it should generate an
inotify_event, some locks also increase overhead, filesystem event
connector hasn't these overhead, it just generates a fsevent and send.

To be important, filesystem event reporter doesn't add any new system call, the user space application can make use of it by netlink socket, but inotify added several system calls, many events mechanism in kernel
have used netlink as communication way with user space, for example,
KOBJECT_UEVENT, PROC_EVENTS, to use netlink will make it more possible
to unify events interface to netlink, the user space application can use
it very easy.

Every application using fsevent can set its own fsevent filter list without effect in other applications using fsevent, there are three filter lists, they are pid filter list, uid filetr list and gid filter list, respectively, moreover, there is a fsevent mask used to control those fsevents which fail to match three filter lists, an application using fsevent can listen those fsevents it want to monitor and ignore those fsevents it doesn't interest in by set series of filters, there is a fsevent mask used to take effects on all the applications using fsevent, it can be set by sysctl and proc interface.

 fs/Kconfig               |   10 
 fs/Makefile              |    3 
 fs/fsevent.c             |  613 +++++++++++++++++++++++++++++++++++++++++++++++
 fs/fsevent_hook.c        |    5 
 fs/namespace.c           |   12 
 include/linux/fsevent.h  |  167 ++++++++++++
 include/linux/fsnotify.h |   37 ++
 include/linux/netlink.h  |    1 
 8 files changed, 848 insertions(+)
include/linux/netlink.h renamed from include/linux/netlink.h.orig +1 -0
@@ -21,6 +21,7 @@
 #define NETLINK_DNRTMSG		14	/* DECnet routing messages */
 #define NETLINK_KOBJECT_UEVENT	15	/* Kernel messages to userspace */
 #define NETLINK_GENERIC		16
+#define NETLINK_FSEVENT		17	/* File system events to userspace */
 
 #define MAX_LINKS 32		
 
include/linux/fsnotify.h renamed from include/linux/fsnotify.h.orig +37 -0
@@ -15,6 +15,7 @@
 
 #include <linux/dnotify.h>
 #include <linux/inotify.h>
+#include <linux/fsevent.h>
 
 /*
  * fsnotify_move - file old_name at old_dir was moved to new_name at new_dir
@@ -45,6 +46,8 @@ static inline void fsnotify_move(struct 
 	if (source) {
 		inotify_inode_queue_event(source, IN_MOVE_SELF, 0, NULL);
 	}
+	raise_fsevent_move(old_dir, old_name, new_dir, new_name,
+			   FSEVENT_MOVE | (isdir?FSEVENT_ISDIR:0));
 }
 
 /*
@@ -56,6 +59,8 @@ static inline void fsnotify_nameremove(s
 		isdir = IN_ISDIR;
 	dnotify_parent(dentry, DN_DELETE);
 	inotify_dentry_parent_queue_event(dentry, IN_DELETE|isdir, 0, dentry->d_name.name);
+	raise_fsevent(dentry,
+		      FSEVENT_DELETE | (isdir?FSEVENT_ISDIR:0));
 }
 
 /*
@@ -74,6 +79,7 @@ static inline void fsnotify_create(struc
 {
 	inode_dir_notify(inode, DN_CREATE);
 	inotify_inode_queue_event(inode, IN_CREATE, 0, name);
+	raise_fsevent_create(inode, name, FSEVENT_CREATE);
 }
 
 /*
@@ -83,6 +89,8 @@ static inline void fsnotify_mkdir(struct
 {
 	inode_dir_notify(inode, DN_CREATE);
 	inotify_inode_queue_event(inode, IN_CREATE | IN_ISDIR, 0, name);
+	raise_fsevent_create(inode, name,
+			     FSEVENT_CREATE | FSEVENT_ISDIR);
 }
 
 /*
@@ -99,6 +107,8 @@ static inline void fsnotify_access(struc
 	dnotify_parent(dentry, DN_ACCESS);
 	inotify_dentry_parent_queue_event(dentry, mask, 0, dentry->d_name.name);
 	inotify_inode_queue_event(inode, mask, 0, NULL);
+	raise_fsevent(dentry, FSEVENT_ACCESS |
+				((S_ISDIR(inode->i_mode))?FSEVENT_ISDIR:0));
 }
 
 /*
@@ -115,6 +125,8 @@ static inline void fsnotify_modify(struc
 	dnotify_parent(dentry, DN_MODIFY);
 	inotify_dentry_parent_queue_event(dentry, mask, 0, dentry->d_name.name);
 	inotify_inode_queue_event(inode, mask, 0, NULL);
+	raise_fsevent(dentry, FSEVENT_MODIFY |
+				((S_ISDIR(inode->i_mode))?FSEVENT_ISDIR:0));
 }
 
 /*
@@ -130,6 +142,9 @@ static inline void fsnotify_open(struct 
 
 	inotify_dentry_parent_queue_event(dentry, mask, 0, dentry->d_name.name);
 	inotify_inode_queue_event(inode, mask, 0, NULL);	
+	raise_fsevent(dentry, FSEVENT_OPEN |
+				((S_ISDIR(inode->i_mode))?FSEVENT_ISDIR:0));
+
 }
 
 /*
@@ -148,6 +163,8 @@ static inline void fsnotify_close(struct
 
 	inotify_dentry_parent_queue_event(dentry, mask, 0, name);
 	inotify_inode_queue_event(inode, mask, 0, NULL);
+	raise_fsevent(dentry, FSEVENT_CLOSE |
+				((S_ISDIR(inode->i_mode))?FSEVENT_ISDIR:0));
 }
 
 /*
@@ -163,6 +180,8 @@ static inline void fsnotify_xattr(struct
 
 	inotify_dentry_parent_queue_event(dentry, mask, 0, dentry->d_name.name);
 	inotify_inode_queue_event(inode, mask, 0, NULL);
+	raise_fsevent(dentry, FSEVENT_MODIFY_ATTRIB |
+				((S_ISDIR(inode->i_mode))?FSEVENT_ISDIR:0));
 }
 
 /*
@@ -213,6 +232,24 @@ static inline void fsnotify_change(struc
 		inotify_dentry_parent_queue_event(dentry, in_mask, 0,
 						  dentry->d_name.name);
 	}
+
+#ifdef CONFIG_FS_EVENTS
+	{
+	u32 fsevent_mask = 0;
+	if (ia_valid & (ATTR_UID | ATTR_GID | ATTR_MODE))
+		fsevent_mask |= FSEVENT_MODIFY_ATTRIB;
+	if ((ia_valid & ATTR_ATIME) && (ia_valid & ATTR_MTIME))
+		fsevent_mask |= FSEVENT_MODIFY_ATTRIB;
+	else if (ia_valid & ATTR_ATIME)
+		fsevent_mask |= FSEVENT_ACCESS;
+	else if (ia_valid & ATTR_MTIME)
+		fsevent_mask |= FSEVENT_MODIFY;
+	if (ia_valid & ATTR_SIZE)
+		fsevent_mask |= FSEVENT_MODIFY;
+	if (fsevent_mask)
+		raise_fsevent(dentry, fsevent_mask);
+	}
+#endif /* CONFIG_FS_EVENTS */
 }
 
 #ifdef CONFIG_INOTIFY	/* inotify helpers */
fs/namespace.c renamed from fs/namespace.c.orig +12 -0
@@ -25,6 +25,7 @@
 #include <linux/mount.h>
 #include <asm/uaccess.h>
 #include <asm/unistd.h>
+#include <linux/fsevent.h>
 #include "pnode.h"
 
 extern int __init init_rootfs(void);
@@ -612,6 +613,13 @@ asmlinkage long sys_umount(char __user *
 		goto dput_and_out;
 
 	retval = do_umount(nd.mnt, flags);
+
+	if (retval == 0) {
+		char * tmp = getname(name);
+		raise_fsevent_umount(tmp);
+		putname(tmp);
+	}
+
 dput_and_out:
 	path_release_on_umount(&nd);
 out:
@@ -1459,6 +1467,10 @@ asmlinkage long sys_mount(char __user * 
 	retval = do_mount((char *)dev_page, dir_page, (char *)type_page,
 			  flags, (void *)data_page);
 	unlock_kernel();
+
+	if (retval == 0)
+		raise_fsevent_mount((char *)dev_page, dir_page);
+
 	free_page(data_page);
 
 out3:
fs/Kconfig renamed from fs/Kconfig.orig +10 -0
@@ -405,6 +405,16 @@ config INOTIFY
 
 	  If unsure, say Y.
 
+config FS_EVENTS
+	tristate "Report filesystem events to userspace"
+	---help---
+	  Provide a facility that reports filesystem events to userspace. The
+	  reported event include access, write, utime, chmod, chown, chgrp,
+	  close, open, create, rename, unlink, mkdir, rmdir, mount, umount.
+
+	  The user can set filesystem events filter to filter its events, so
+	  that he just get those events he concerns.
+
 config QUOTA
 	bool "Quota support"
 	help
fs/Makefile renamed from fs/Makefile.orig +3 -0
@@ -13,6 +13,9 @@ obj-y :=	open.o read_write.o file_table.
 		ioprio.o pnode.o drop_caches.o
 
 obj-$(CONFIG_INOTIFY)		+= inotify.o
+obj-$(CONFIG_FS_EVENTS)		+= fsevent.o
+fseventbase-$(CONFIG_FS_EVENTS)	:= fsevent_hook.o
+obj-y				+= $(fseventbase-y) $(fseventbase-m)
 obj-$(CONFIG_EPOLL)		+= eventpoll.o
 obj-$(CONFIG_COMPAT)		+= compat.o compat_ioctl.o
 
include/linux/fsevent.h new file +167 -0

167 lines not on this page · raw message

fs/fsevent.c new file +613 -0

613 lines not on this page · raw message

fs/fsevent_hook.c new file +5 -0
@@ -0,0 +1,5 @@
+#include <linux/fsevent.h>
+
+int (* _raise_fsevent)
+        (const char * oldname, const char * newname, u32 mask) = 0;
+EXPORT_SYMBOL(_raise_fsevent);
#1 lore raw
Quoted 5 lines · Yi Yang

Compared with Filesystem Events Reporter v2, the following changes are done:
- Can be built as module.
- Fix some bugs pointed out by Evgeniy
- Substitute spinlock with mutex
- Complete exit cleanup

...

include/linux/fsevent.h line 2

Quoted 5 lines
include/linux/fsevent.h new file +2 -0
@@ -0,0 +1,167 @@
+/*
+ * fsevent.h - filesystem events connector

I think you want to change this sentence to "... reporter", don't you :)

...

include/linux/fsevent.h line 9

Quoted 7 lines
include/linux/fsevent.h new file +7 -0
+static inline int filter_fsevent(u32 filter_mask, u32 event_mask)
+{
+	event_mask &= FSEVENT_MASK;
+	event_mask &= filter_mask;
+	if (event_mask == 0) {
+		return -1;
+	}

Coding style...
...

include/linux/fsevent.h line 13

Quoted 4 lines
include/linux/fsevent.h new file +4 -0
+	(*mask) &= fsevents_mask;
+	if ((*mask) == 0) {
+		ret = -1;
+	}

Ditto.

...

include/linux/fsevent.h line 26

Quoted 13 lines
include/linux/fsevent.h new file +13 -0
+static void fsevent_recv(struct sock *sk, int len)
+{
+	struct sk_buff *skb = NULL;
+	struct nlmsghdr *nlhdr = NULL;
+	struct fsevent_filter * filter = NULL;
+	pid_t pid;
+	int inc_flag = 0;
+
+	if (exit_flag == 1)
+		return;
+
+	while ((skb = skb_dequeue(&sk->sk_receive_queue)) != NULL) {
+		skb_get(skb);

Really suspicious, only you own skb at this place, no need to grab a reference, which, btw, will never be released.

include/linux/fsevent.h line 27

Quoted 1 line
include/linux/fsevent.h new file +1 -0
+		if (skb->len >= FSEVENT_FILTER_MSGSIZE) {

I'm not sure about your size checks.
I think it should be compared with nlhdr->nlmsg_len?

include/linux/fsevent.h line 32

Quoted 5 lines
include/linux/fsevent.h new file +5 -0
+			nlhdr = (struct nlmsghdr *)skb->data;
+			filter = NLMSG_DATA(nlhdr);
+			pid = NETLINK_CREDS(skb)->pid;
+			if (find_fsevent_listener(pid) == NULL)
+				inc_flag = 1;

This logic is broken if several skbs are processed at once, since inc_flag is not cleared if find_fsevent_listener() fails for second skb.

include/linux/fsevent.h line 36

Quoted 4 lines
include/linux/fsevent.h new file +4 -0
+			if (set_fsevent_filter(filter, pid) == 0) {
+				if (inc_flag == 1)
+					atomic_inc(&fsevent_listener_num);
+			}

Why not
if ((find_fsevent_listener(pid) == NULL) &&

	(set_fsevent_filter(filter, pid) == 0))
		atomic_inc(&fsevent_listener_num);

include/linux/fsevent.h line 38

Quoted 2 lines
include/linux/fsevent.h new file +2 -0
+		}
+		kfree_skb(skb);

Here you only release a reference to skb, but do not free skb itself.

Quoted 10 lines
include/linux/fsevent.h new file +10 -0
+	}
+}
+
+#define DEFINE_FILTER_MATCH_FUNC(filtertype, key) 			\
+	static int match_##filtertype(listener * p,			\
+				struct fsevent * event,			\
+				struct sk_buff * skb)			\
+	{								\
+		int ret = 0;						\
+		filtertype * xfilter = NULL;				\

include/linux/fsevent.h line 58

Quoted 10 lines
include/linux/fsevent.h new file +10 -0
+		struct sk_buff * skb2 = NULL;				\
+		struct list_head *  head = &(p->key##_filter_list_head);  \
+		list_for_each_entry(xfilter, head, list) {		\
+			if (xfilter->key != event->key)			\
+				continue;				\
+			ret = filter_fsevent(xfilter->mask, event->type); \
+			if ( ret != 0)					\
+				return -1;				\
+			skb2 = skb_clone(skb, GFP_KERNEL);		\
+       			if (skb2 == NULL)				\

Coding style.

Quoted 5 lines
include/linux/fsevent.h new file +5 -0
+				return -1;				\
+			NETLINK_CB(skb2).dst_group = 0;			\
+			NETLINK_CB(skb2).dst_pid = p->pid;		\
+			NETLINK_CB(skb2).pid = 0;			\
+			return (netlink_unicast(fsevent_sock, skb2,	\

include/linux/fsevent.h line 73

Quoted 10 lines
include/linux/fsevent.h new file +10 -0
+					p->pid, MSG_DONTWAIT));		\
+		}							\
+		return -1;						\
+	}								\
+
+DEFINE_FILTER_MATCH_FUNC(pid_filter, pid)
+
+DEFINE_FILTER_MATCH_FUNC(uid_filter, uid)
+
+DEFINE_FILTER_MATCH_FUNC(gid_filter, gid)

You send the same data for each type of filters, maybe it is design approach, but why don't you want to send that data in one skb?

include/linux/fsevent.h line 78

Quoted 5 lines
include/linux/fsevent.h new file +5 -0
+#define MATCH_XID(key, listenerp, event, skb) 			\
+	ret = match_##key##_filter(listenerp, event, skb); 	\
+	if (ret == 0) {					 	\
+		kfree_skb(skb);				 	\
+	        continue;				 	\

Your match funtions can not return 0.

Quoted 6 lines
include/linux/fsevent.h new file +6 -0
+	}						 	\
+	do {} while (0)					 	\
+
+static int fsevent_send_to_process(struct sk_buff * skb)
+{
+	listener * p  = NULL, * q = NULL;

include/linux/fsevent.h line 94

Quoted 10 lines
include/linux/fsevent.h new file +10 -0
+	struct fsevent * event = NULL;
+	struct sk_buff * skb2 = NULL;
+	int ret = 0;
+
+	event = (struct fsevent *)(skb->data + sizeof(struct nlmsghdr));
+	mutex_lock(&listener_list_mutex);
+	list_for_each_entry_safe(p, q, &listener_list_head, list) {
+		MATCH_XID(pid, p, event, skb);
+		MATCH_XID(uid, p, event, skb);
+		MATCH_XID(gid, p, event, skb);

Khm, you free the same skb three times here if each filter returns 0, is it right? I do not see where appropriate reference is grabbed.

Well, since your match functions can not return 0, so it is ok, but in this case you do not need a check in MATCH_XID() check.

Quoted 97 lines
include/linux/fsevent.h new file +97 -0
+		if (filter_fsevent(p->mask, event->type) == 0) {
+			 skb2 = skb_clone(skb, GFP_KERNEL);
+	                 if (skb2 == NULL)
+	                 	return -1;
+	                 NETLINK_CB(skb2).dst_group = 0;
+	                 NETLINK_CB(skb2).dst_pid = p->pid;
+	                 NETLINK_CB(skb2).pid = 0;
+	                 ret = netlink_unicast(fsevent_sock, skb2,
+	                                p->pid, 0);
+			if (ret == -ECONNREFUSED) {
+				atomic_dec(&fsevent_listener_num);
+				cleanup_dead_listener(p);
+			}
+		}
+	}
+	mutex_unlock(&listener_list_mutex);
+	return ret;
+}
+
+static void fsevent_commit(void * unused)
+{
+	struct sk_buff * skb = NULL;
+		
+	while((skb = skb_dequeue(&get_cpu_var(fsevent_send_queue)))
+		!= NULL) {
+		fsevent_send_to_process(skb);
+		kfree_skb(skb);
+		put_cpu_var(fsevent_send_queue);
+	}
+}
+
+static struct ctl_table fsevent_mask_sysctl[] = {
+	{
+		.ctl_name = FSEVENT_MASK_CTL_NAME,
+		.procname = "fsevent_mask",
+		.data = &fsevents_mask,
+		.maxlen = sizeof(u32),
+		.mode = 0644,
+		.proc_handler = &proc_dointvec,
+	},
+	{ .ctl_name = 0 }
+};
+
+static struct ctl_table fs_root_sysctl[] = {
+	{
+		.ctl_name = CTL_FS,
+		.procname = "fs",
+		.mode = 0555,
+		.child = fsevent_mask_sysctl,
+	},
+	{ .ctl_name = 0 }
+};
+
+static int __init fsevent_init(void)
+{
+	int cpu;
+	struct sk_buff_head * listptr;
+	struct work_struct * workptr;
+
+	fsevent_sock = netlink_kernel_create(NETLINK_FSEVENT, 0,
+					 fsevent_recv, THIS_MODULE);
+	if (!fsevent_sock)
+		return -EIO;
+	for_each_cpu(cpu) {
+		listptr = &per_cpu(fsevent_send_queue, cpu);
+		skb_queue_head_init(listptr);
+		workptr = &per_cpu(fsevent_work, cpu);
+		INIT_WORK(workptr, fsevent_commit, NULL);
+	}
+
+	if (register_sysctl_table(fs_root_sysctl, 0) == NULL)
+                return -ENOMEM;
+
+	_raise_fsevent = __raise_fsevent;
+
+	return 0;
+}
+
+static void __exit fsevent_exit(void)
+{
+	listener * p = NULL, * q = NULL;
+	int cpu;
+	int wait_flag = 0;
+	struct sk_buff_head * skb_head = NULL;
+
+	fsevents_mask = 0;
+	_raise_fsevent = 0;
+	exit_flag = 1;
+
+	for_each_cpu(cpu)
+		schedule_work(&per_cpu(fsevent_work, cpu));
+
+	while (1) {
+		wait_flag = 0;
+		for_each_cpu(cpu) {
+			skb_head = &per_cpu(fsevent_send_queue, cpu);
+			if (skb_head->qlen != 0) {

include/linux/fsevent.h

Quoted 10 lines
include/linux/fsevent.h new file +10 -0
+				wait_flag = 1;
+				break;
+			}
+		}
+		if (wait_flag == 1) {
+			set_current_state(TASK_INTERRUPTIBLE);
+			schedule_timeout(HZ/10);
+		} else
+			break;
+	}

This is still broken.
You race with schedule_work() in this loop. It requires flush_scheduled_work().

And I still have soume doubts about __raise_fsevent(). What if you set fsevents_mask to zero after __raise_fsevent() is started, but not yet queued an skb, and above loop and scheduled work are completed?
You need some type of completion of the last worker...

include/linux/fsevent.h

Quoted 2 lines
include/linux/fsevent.h new file +2 -0
+	atomic_set(&fsevent_sock->sk_rmem_alloc, 0);
+	atomic_set(&fsevent_sock->sk_wmem_alloc, 0);

This is really wrong, since it hides skb processing errors like double freeing or leaks.

Quoted 17 lines
include/linux/fsevent.h new file +14 -0
+	sock_release(fsevent_sock->sk_socket);
+	mutex_lock(&listener_list_mutex);
+	list_for_each_entry_safe(p, q, &listener_list_head, list) {
+		cleanup_dead_listener(p);
+	}
+	mutex_unlock(&listener_list_mutex);
+}
+
+module_init(fsevent_init);
+module_exit(fsevent_exit);
+
+MODULE_LICENSE("GPL");
+MODULE_AUTHOR("Yi Yang <[email protected]>");
+MODULE_DESCRIPTION("File System Events Reporter");

fs/fsevent_hook.c line 5

Quoted 5 lines
fs/fsevent_hook.c new file +5 -0
@@ -0,0 +1,5 @@
+#include <linux/fsevent.h>
+
+int (* _raise_fsevent)
+        (const char * oldname, const char * newname, u32 mask) = 0;
+EXPORT_SYMBOL(_raise_fsevent);

Well, this is a hack :)

Btw, it would be nice to have some kind of microbenchmark, like http://permalink.gmane.org/gmane.linux.kernel/292755
just to see how things go...

Signature · 1 line
	Evgeniy Polyakov
author
#2 lore raw
Quoted 66 lines · Evgeniy Polyakov
Quoted 6 lines · Yi Yang

Compared with Filesystem Events Reporter v2, the following changes are done:
- Can be built as module.
- Fix some bugs pointed out by Evgeniy
- Substitute spinlock with mutex
- Complete exit cleanup

...

include/linux/fsevent.h line 2

Quoted 6 lines
include/linux/fsevent.h new file +2 -0
@@ -0,0 +1,167 @@
+/*
+ * fsevent.h - filesystem events connector

I think you want to change this sentence to "... reporter", don't you :)

...

include/linux/fsevent.h line 9

Quoted 8 lines
include/linux/fsevent.h new file +7 -0
+static inline int filter_fsevent(u32 filter_mask, u32 event_mask)
+{
+	event_mask &= FSEVENT_MASK;
+	event_mask &= filter_mask;
+	if (event_mask == 0) {
+		return -1;
+	}

Coding style...
...

include/linux/fsevent.h line 13

Quoted 5 lines
include/linux/fsevent.h new file +4 -0
+	(*mask) &= fsevents_mask;
+	if ((*mask) == 0) {
+		ret = -1;
+	}

Ditto.

...

Quoted 4 lines
include/linux/fsevent.h new file +3 -0
+static void fsevent_recv(struct sock *sk, int len)
+{
+	struct sk_buff *skb = NULL;

include/linux/fsevent.h line 26

Quoted 10 lines
include/linux/fsevent.h new file +10 -0
+	struct nlmsghdr *nlhdr = NULL;
+	struct fsevent_filter * filter = NULL;
+	pid_t pid;
+	int inc_flag = 0;
+
+	if (exit_flag == 1)
+		return;
+
+	while ((skb = skb_dequeue(&sk->sk_receive_queue)) != NULL) {
+		skb_get(skb);

Really suspicious, only you own skb at this place, no need to grab a reference, which, btw, will never be released.

Quoted 6 lines

include/linux/fsevent.h line 27

Quoted 2 lines
include/linux/fsevent.h new file +1 -0
+		if (skb->len >= FSEVENT_FILTER_MSGSIZE) {

I'm not sure about your size checks.
I think it should be compared with nlhdr->nlmsg_len?

At this point, skb->len should be the same as nlhdr->nlmsg_len.

Quoted 19 lines

include/linux/fsevent.h line 32

Quoted 7 lines
include/linux/fsevent.h new file +5 -0
+			nlhdr = (struct nlmsghdr *)skb->data;
+			filter = NLMSG_DATA(nlhdr);
+			pid = NETLINK_CREDS(skb)->pid;
+			if (find_fsevent_listener(pid) == NULL)
+				inc_flag = 1;

This logic is broken if several skbs are processed at once, since inc_flag is not cleared if find_fsevent_listener() fails for second skb.

include/linux/fsevent.h line 36

Quoted 5 lines
include/linux/fsevent.h new file +4 -0
+			if (set_fsevent_filter(filter, pid) == 0) {
+				if (inc_flag == 1)
+					atomic_inc(&fsevent_listener_num);
+			}

Why not if ((find_fsevent_listener(pid) == NULL) &&

Quoted 4 lines
	(set_fsevent_filter(filter, pid) == 0))
		atomic_inc(&fsevent_listener_num);

Your method is right.

Quoted 43 lines

include/linux/fsevent.h line 38

Quoted 3 lines
include/linux/fsevent.h new file +2 -0
+		}
+		kfree_skb(skb);

Here you only release a reference to skb, but do not free skb itself.

Quoted 11 lines
include/linux/fsevent.h new file +10 -0
+	}
+}
+
+#define DEFINE_FILTER_MATCH_FUNC(filtertype, key) 			\
+	static int match_##filtertype(listener * p,			\
+				struct fsevent * event,			\
+				struct sk_buff * skb)			\
+	{								\
+		int ret = 0;						\
+		filtertype * xfilter = NULL;				\

include/linux/fsevent.h line 58

Quoted 10 lines
include/linux/fsevent.h new file +10 -0
+		struct sk_buff * skb2 = NULL;				\
+		struct list_head *  head = &(p->key##_filter_list_head);  \
+		list_for_each_entry(xfilter, head, list) {		\
+			if (xfilter->key != event->key)			\
+				continue;				\
+			ret = filter_fsevent(xfilter->mask, event->type); \
+			if ( ret != 0)					\
+				return -1;				\
+			skb2 = skb_clone(skb, GFP_KERNEL);		\
+       			if (skb2 == NULL)				\

Coding style.

Quoted 11 lines
include/linux/fsevent.h new file +11 -0
+				return -1;				\
+			NETLINK_CB(skb2).dst_group = 0;			\
+			NETLINK_CB(skb2).dst_pid = p->pid;		\
+			NETLINK_CB(skb2).pid = 0;			\
+			return (netlink_unicast(fsevent_sock, skb2,	\
+					p->pid, MSG_DONTWAIT));		\
+		}							\
+		return -1;						\
+	}								\
+
+DEFINE_FILTER_MATCH_FUNC(pid_filter, pid)
Quoted 9 lines

include/linux/fsevent.h line 73

Quoted 5 lines
include/linux/fsevent.h new file +4 -0
+
+DEFINE_FILTER_MATCH_FUNC(uid_filter, uid)
+
+DEFINE_FILTER_MATCH_FUNC(gid_filter, gid)

You send the same data for each type of filters, maybe it is design approach, but why don't you want to send that data in one skb?

netlink control block is not the same, netlink_broadcast is a typical case.

Quoted 10 lines

include/linux/fsevent.h line 78

Quoted 7 lines
include/linux/fsevent.h new file +5 -0
+#define MATCH_XID(key, listenerp, event, skb) 			\
+	ret = match_##key##_filter(listenerp, event, skb); 	\
+	if (ret == 0) {					 	\
+		kfree_skb(skb);				 	\
+	        continue;				 	\

Your match funtions can not return 0.

It can, if sending is successfull, netlink_unicast will return 0.

Quoted 22 lines
Quoted 8 lines
include/linux/fsevent.h new file +6 -0
+	}						 	\
+	do {} while (0)					 	\
+
+static int fsevent_send_to_process(struct sk_buff * skb)
+{
+	listener * p  = NULL, * q = NULL;

include/linux/fsevent.h line 94

Quoted 10 lines
include/linux/fsevent.h new file +10 -0
+	struct fsevent * event = NULL;
+	struct sk_buff * skb2 = NULL;
+	int ret = 0;
+
+	event = (struct fsevent *)(skb->data + sizeof(struct nlmsghdr));
+	mutex_lock(&listener_list_mutex);
+	list_for_each_entry_safe(p, q, &listener_list_head, list) {
+		MATCH_XID(pid, p, event, skb);
+		MATCH_XID(uid, p, event, skb);
+		MATCH_XID(gid, p, event, skb);

Khm, you free the same skb three times here if each filter returns 0, is it right? I do not see where appropriate reference is grabbed.

Quoted 3 lines

Well, since your match functions can not return 0, so it is ok, but in this case you do not need a check in MATCH_XID() check.

There are really some issues here, I'll fix it.

Quoted 114 lines
Quoted 99 lines
include/linux/fsevent.h new file +97 -0
+		if (filter_fsevent(p->mask, event->type) == 0) {
+			 skb2 = skb_clone(skb, GFP_KERNEL);
+	                 if (skb2 == NULL)
+	                 	return -1;
+	                 NETLINK_CB(skb2).dst_group = 0;
+	                 NETLINK_CB(skb2).dst_pid = p->pid;
+	                 NETLINK_CB(skb2).pid = 0;
+	                 ret = netlink_unicast(fsevent_sock, skb2,
+	                                p->pid, 0);
+			if (ret == -ECONNREFUSED) {
+				atomic_dec(&fsevent_listener_num);
+				cleanup_dead_listener(p);
+			}
+		}
+	}
+	mutex_unlock(&listener_list_mutex);
+	return ret;
+}
+
+static void fsevent_commit(void * unused)
+{
+	struct sk_buff * skb = NULL;
+		
+	while((skb = skb_dequeue(&get_cpu_var(fsevent_send_queue)))
+		!= NULL) {
+		fsevent_send_to_process(skb);
+		kfree_skb(skb);
+		put_cpu_var(fsevent_send_queue);
+	}
+}
+
+static struct ctl_table fsevent_mask_sysctl[] = {
+	{
+		.ctl_name = FSEVENT_MASK_CTL_NAME,
+		.procname = "fsevent_mask",
+		.data = &fsevents_mask,
+		.maxlen = sizeof(u32),
+		.mode = 0644,
+		.proc_handler = &proc_dointvec,
+	},
+	{ .ctl_name = 0 }
+};
+
+static struct ctl_table fs_root_sysctl[] = {
+	{
+		.ctl_name = CTL_FS,
+		.procname = "fs",
+		.mode = 0555,
+		.child = fsevent_mask_sysctl,
+	},
+	{ .ctl_name = 0 }
+};
+
+static int __init fsevent_init(void)
+{
+	int cpu;
+	struct sk_buff_head * listptr;
+	struct work_struct * workptr;
+
+	fsevent_sock = netlink_kernel_create(NETLINK_FSEVENT, 0,
+					 fsevent_recv, THIS_MODULE);
+	if (!fsevent_sock)
+		return -EIO;
+	for_each_cpu(cpu) {
+		listptr = &per_cpu(fsevent_send_queue, cpu);
+		skb_queue_head_init(listptr);
+		workptr = &per_cpu(fsevent_work, cpu);
+		INIT_WORK(workptr, fsevent_commit, NULL);
+	}
+
+	if (register_sysctl_table(fs_root_sysctl, 0) == NULL)
+                return -ENOMEM;
+
+	_raise_fsevent = __raise_fsevent;
+
+	return 0;
+}
+
+static void __exit fsevent_exit(void)
+{
+	listener * p = NULL, * q = NULL;
+	int cpu;
+	int wait_flag = 0;
+	struct sk_buff_head * skb_head = NULL;
+
+	fsevents_mask = 0;
+	_raise_fsevent = 0;
+	exit_flag = 1;
+
+	for_each_cpu(cpu)
+		schedule_work(&per_cpu(fsevent_work, cpu));
+
+	while (1) {
+		wait_flag = 0;
+		for_each_cpu(cpu) {
+			skb_head = &per_cpu(fsevent_send_queue, cpu);
+			if (skb_head->qlen != 0) {

include/linux/fsevent.h

Quoted 10 lines
include/linux/fsevent.h new file +10 -0
+				wait_flag = 1;
+				break;
+			}
+		}
+		if (wait_flag == 1) {
+			set_current_state(TASK_INTERRUPTIBLE);
+			schedule_timeout(HZ/10);
+		} else
+			break;
+	}

This is still broken.
You race with schedule_work() in this loop. It requires flush_scheduled_work().

Quoted 5 lines

And I still have soume doubts about __raise_fsevent(). What if you set fsevents_mask to zero after __raise_fsevent() is started, but not yet queued an skb, and above loop and scheduled work are completed?

I think it is OK, schedule_timeout will release cpu to work queues, work queues should have enough time to finish their works, I don't know what is your reason.

Quoted 10 lines

You need some type of completion of the last worker...

include/linux/fsevent.h

Quoted 3 lines
include/linux/fsevent.h new file +2 -0
+	atomic_set(&fsevent_sock->sk_rmem_alloc, 0);
+	atomic_set(&fsevent_sock->sk_wmem_alloc, 0);

This is really wrong, since it hides skb processing errors like double freeing or leaks.

If userspace application terminated exceptionally, there are some skbs not to be consumed on socket, so if you rmmod it, sock_release will report some failure information, the above two statements will remove this error.

Quoted 28 lines
Quoted 19 lines
include/linux/fsevent.h new file +14 -0
+	sock_release(fsevent_sock->sk_socket);
+	mutex_lock(&listener_list_mutex);
+	list_for_each_entry_safe(p, q, &listener_list_head, list) {
+		cleanup_dead_listener(p);
+	}
+	mutex_unlock(&listener_list_mutex);
+}
+
+module_init(fsevent_init);
+module_exit(fsevent_exit);
+
+MODULE_LICENSE("GPL");
+MODULE_AUTHOR("Yi Yang <[email protected]>");
+MODULE_DESCRIPTION("File System Events Reporter");

fs/fsevent_hook.c line 5

Quoted 5 lines
fs/fsevent_hook.c new file +5 -0
@@ -0,0 +1,5 @@
+#include <linux/fsevent.h>
+
+int (* _raise_fsevent)
+        (const char * oldname, const char * newname, u32 mask) = 0;
+EXPORT_SYMBOL(_raise_fsevent);

Well, this is a hack :)

Quoted 4 lines

Btw, it would be nice to have some kind of microbenchmark, like http://permalink.gmane.org/gmane.linux.kernel/292755
just to see how things go...

I have a userspace application to test fsevent, I'll release it to community in order to find more issues on fsevent.

#3 lore raw
Quoted 7 lines · Yi Yang
Quoted 6 lines

fs/fsevent.c line 419

Quoted 2 lines
fs/fsevent.c new file +1 -0
+		if (skb->len >= FSEVENT_FILTER_MSGSIZE) {

I'm not sure about your size checks.
I think it should be compared with nlhdr->nlmsg_len?

At this point, skb->len should be the same as nlhdr->nlmsg_len.

Hmm, skb->len includes size of netlink header, but nlhdr->nlmsg_len does not.

Quoted 42 lines
Quoted 33 lines
Quoted 8 lines
fs/fsevent.c new file +7 -0
+#define DEFINE_FILTER_MATCH_FUNC(filtertype, key) 			\
+	static int match_##filtertype(listener * p,			\
+				struct fsevent * event,			\
+				struct sk_buff * skb)			\
+	{								\
+		int ret = 0;						\
+		filtertype * xfilter = NULL;				\

fs/fsevent.c line 451

Quoted 10 lines
fs/fsevent.c new file +10 -0
+		struct sk_buff * skb2 = NULL;				\
+		struct list_head *  head = &(p->key##_filter_list_head);  \
+		list_for_each_entry(xfilter, head, list) {		\
+			if (xfilter->key != event->key)			\
+				continue;				\
+			ret = filter_fsevent(xfilter->mask, event->type); \
+			if ( ret != 0)					\
+				return -1;				\
+			skb2 = skb_clone(skb, GFP_KERNEL);		\
+       			if (skb2 == NULL)			 \

Coding style.

Quoted 11 lines
fs/fsevent.c new file +11 -0
+				return -1;				\
+			NETLINK_CB(skb2).dst_group = 0;			\
+			NETLINK_CB(skb2).dst_pid = p->pid;		\
+			NETLINK_CB(skb2).pid = 0;			\
+			return (netlink_unicast(fsevent_sock, skb2,	\
+					p->pid, MSG_DONTWAIT));		\
+		}							\
+		return -1;						\
+	}								\
+
+DEFINE_FILTER_MATCH_FUNC(pid_filter, pid)
Quoted 9 lines

fs/fsevent.c line 466

Quoted 5 lines
fs/fsevent.c new file +4 -0
+
+DEFINE_FILTER_MATCH_FUNC(uid_filter, uid)
+
+DEFINE_FILTER_MATCH_FUNC(gid_filter, gid)

You send the same data for each type of filters, maybe it is design approach, but why don't you want to send that data in one skb?

Quoted 1 line

netlink control block is not the same, netlink_broadcast is a typical case.

Yes, I see, pid is changed.

Quoted 10 lines
Quoted 9 lines

fs/fsevent.c line 472

Quoted 6 lines
fs/fsevent.c new file +5 -0
+#define MATCH_XID(key, listenerp, event, skb) 			\
+	ret = match_##key##_filter(listenerp, event, skb); 	\
+	if (ret == 0) {					 	\
+		kfree_skb(skb);				 	\
+	        continue;				 	\

Your match funtions can not return 0.

It can, if sending is successfull, netlink_unicast will return 0.

No, it returns skb->len on success.
netlink_broadcast() returns 0 on success.

Quoted 35 lines
Quoted 35 lines
Quoted 20 lines
fs/fsevent.c new file +19 -0
+static void __exit fsevent_exit(void)
+{
+	listener * p = NULL, * q = NULL;
+	int cpu;
+	int wait_flag = 0;
+	struct sk_buff_head * skb_head = NULL;
+
+	fsevents_mask = 0;
+	_raise_fsevent = 0;
+	exit_flag = 1;
+
+	for_each_cpu(cpu)
+		schedule_work(&per_cpu(fsevent_work, cpu));
+
+	while (1) {
+		wait_flag = 0;
+		for_each_cpu(cpu) {
+			skb_head = &per_cpu(fsevent_send_queue, cpu);
+			if (skb_head->qlen != 0) {

fs/fsevent.c line 596

Quoted 10 lines
fs/fsevent.c new file +10 -0
+				wait_flag = 1;
+				break;
+			}
+		}
+		if (wait_flag == 1) {
+			set_current_state(TASK_INTERRUPTIBLE);
+			schedule_timeout(HZ/10);
+		} else
+			break;
+	}

This is still broken.
You race with schedule_work() in this loop. It requires flush_scheduled_work().

Quoted 8 lines
Quoted 5 lines

And I still have soume doubts about __raise_fsevent(). What if you set fsevents_mask to zero after __raise_fsevent() is started, but not yet queued an skb, and above loop and scheduled work are completed?

I think it is OK, schedule_timeout will release cpu to work queues, work queues should have enough time
to finish their works, I don't know what is your reason.

It is not guaranteed that scheduled work will be processed until flush_scheduled_work() completion, no matter how many times processor has idle cycles.

Second issue is that both above loop and work can be finished, but some __raise_fsevent() will be still in progress.

Quoted 10 lines
Quoted 10 lines

You need some type of completion of the last worker...

fs/fsevent.c line 599

Quoted 3 lines
fs/fsevent.c new file +2 -0
+	atomic_set(&fsevent_sock->sk_rmem_alloc, 0);
+	atomic_set(&fsevent_sock->sk_wmem_alloc, 0);

This is really wrong, since it hides skb processing errors like double freeing or leaks.

Quoted 5 lines

If userspace application terminated exceptionally, there are some skbs not to be consumed on socket, so
if you rmmod it, sock_release will report some failure information, the above two statements will remove this
error.

All queues will be flushed, when socket is freed, and if sock_release() shows that assertion is failed, this definitely means you broke socket accounting, for example freed skb two times.

Quoted 1 line
Quoted 1 line
Quoted 1 line
fs/fsevent.c new file +1 -0
+	sock_release(fsevent_sock->sk_socket);

...

Quoted 7 lines
Quoted 4 lines

Btw, it would be nice to have some kind of microbenchmark, like http://permalink.gmane.org/gmane.linux.kernel/292755
just to see how things go...

I have a userspace application to test fsevent, I'll release it to community in order to find more issues on
fsevent.

And please publish some numbers so people could make some prognosis of system behaviour.

Signature · 1 line
	Evgeniy Polyakov
author
#4 lore raw
Quoted 10 lines · Evgeniy Polyakov
Quoted 7 lines · Yi Yang
Quoted 7 lines

fs/fsevent.c line 419

Quoted 3 lines
fs/fsevent.c new file +1 -0
+		if (skb->len >= FSEVENT_FILTER_MSGSIZE) {

I'm not sure about your size checks.
I think it should be compared with nlhdr->nlmsg_len?

Quoted 6 lines
Quoted 2 lines

At this point, skb->len should be the same as nlhdr->nlmsg_len.

Hmm, skb->len includes size of netlink header, but nlhdr->nlmsg_len does not.

No, it included length of netlink header, please see it. struct nlmsghdr
{
__u32 nlmsg_len; /* Length of message including
header */

       __u16           nlmsg_type;     /* Message content */
       __u16           nlmsg_flags;    /* Additional flags */
       __u32           nlmsg_seq;      /* Sequence number */
       __u32           nlmsg_pid;      /* Sending process PID */

};

Quoted 113 lines
Quoted 45 lines
Quoted 35 lines
Quoted 10 lines
fs/fsevent.c new file +7 -0
+#define DEFINE_FILTER_MATCH_FUNC(filtertype, key) 			\
+	static int match_##filtertype(listener * p,			\
+				struct fsevent * event,			\
+				struct sk_buff * skb)			\
+	{								\
+		int ret = 0;						\
+		filtertype * xfilter = NULL;				\

fs/fsevent.c line 451

Quoted 10 lines
fs/fsevent.c new file +10 -0
+		struct sk_buff * skb2 = NULL;				\
+		struct list_head *  head = &(p->key##_filter_list_head);  \
+		list_for_each_entry(xfilter, head, list) {		\
+			if (xfilter->key != event->key)			\
+				continue;				\
+			ret = filter_fsevent(xfilter->mask, event->type); \
+			if ( ret != 0)					\
+				return -1;				\
+			skb2 = skb_clone(skb, GFP_KERNEL);		\
+       			if (skb2 == NULL)			 \

Coding style.

Quoted 11 lines
fs/fsevent.c new file +11 -0
+				return -1;				\
+			NETLINK_CB(skb2).dst_group = 0;			\
+			NETLINK_CB(skb2).dst_pid = p->pid;		\
+			NETLINK_CB(skb2).pid = 0;			\
+			return (netlink_unicast(fsevent_sock, skb2,	\
+					p->pid, MSG_DONTWAIT));		\
+		}							\
+		return -1;						\
+	}								\
+
+DEFINE_FILTER_MATCH_FUNC(pid_filter, pid)
Quoted 10 lines

fs/fsevent.c line 466

Quoted 6 lines
fs/fsevent.c new file +4 -0
+
+DEFINE_FILTER_MATCH_FUNC(uid_filter, uid)
+
+DEFINE_FILTER_MATCH_FUNC(gid_filter, gid)

You send the same data for each type of filters, maybe it is design approach, but why don't you want to send that data in one skb?

Quoted 2 lines

netlink control block is not the same, netlink_broadcast is a typical case.

Yes, I see, pid is changed.

Quoted 10 lines
Quoted 10 lines

fs/fsevent.c line 472

Quoted 7 lines
fs/fsevent.c new file +5 -0
+#define MATCH_XID(key, listenerp, event, skb) 			\
+	ret = match_##key##_filter(listenerp, event, skb); 	\
+	if (ret == 0) {					 	\
+		kfree_skb(skb);				 	\
+	        continue;				 	\

Your match funtions can not return 0.

Quoted 2 lines

It can, if sending is successfull, netlink_unicast will return 0.

No, it returns skb->len on success.
netlink_broadcast() returns 0 on success.

Quoted 35 lines
Quoted 35 lines
Quoted 21 lines
fs/fsevent.c new file +19 -0
+static void __exit fsevent_exit(void)
+{
+	listener * p = NULL, * q = NULL;
+	int cpu;
+	int wait_flag = 0;
+	struct sk_buff_head * skb_head = NULL;
+
+	fsevents_mask = 0;
+	_raise_fsevent = 0;
+	exit_flag = 1;
+
+	for_each_cpu(cpu)
+		schedule_work(&per_cpu(fsevent_work, cpu));
+
+	while (1) {
+		wait_flag = 0;
+		for_each_cpu(cpu) {
+			skb_head = &per_cpu(fsevent_send_queue, cpu);
+			if (skb_head->qlen != 0) {

fs/fsevent.c line 596

Quoted 10 lines
fs/fsevent.c new file +10 -0
+				wait_flag = 1;
+				break;
+			}
+		}
+		if (wait_flag == 1) {
+			set_current_state(TASK_INTERRUPTIBLE);
+			schedule_timeout(HZ/10);
+		} else
+			break;
+	}

This is still broken.
You race with schedule_work() in this loop. It requires flush_scheduled_work().

Quoted 9 lines
Quoted 6 lines

And I still have soume doubts about __raise_fsevent(). What if you set fsevents_mask to zero after __raise_fsevent() is started, but not yet queued an skb, and above loop and scheduled work are completed?

I think it is OK, schedule_timeout will release cpu to work queues, work queues should have enough time to finish their works, I don't know what is your reason.

Quoted 7 lines

It is not guaranteed that scheduled work will be processed until flush_scheduled_work() completion, no matter how many times processor has idle cycles.

Second issue is that both above loop and work can be finished, but some __raise_fsevent() will be still in progress.

I knew your meaning, if you have a better way, tell me.

Quoted 13 lines
Quoted 13 lines
Quoted 13 lines

You need some type of completion of the last worker...

fs/fsevent.c line 599

Quoted 4 lines
fs/fsevent.c new file +2 -0
+	atomic_set(&fsevent_sock->sk_rmem_alloc, 0);
+	atomic_set(&fsevent_sock->sk_wmem_alloc, 0);

This is really wrong, since it hides skb processing errors like double freeing or leaks.

Quoted 8 lines
Quoted 4 lines

If userspace application terminated exceptionally, there are some skbs not to be consumed on socket, so if you rmmod it, sock_release will report some failure information, the above two statements will remove this error.

All queues will be flushed, when socket is freed, and if sock_release() shows
that assertion is failed, this definitely means you broke socket accounting, for example freed skb two times.

I searched those code, it didn't decrease sk_rmem_alloc, do you mean skb_queue_purge will decrease it?

Quoted 12 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
fs/fsevent.c new file +1 -0
+	sock_release(fsevent_sock->sk_socket);

...

Quoted 5 lines
Quoted 5 lines

Btw, it would be nice to have some kind of microbenchmark, like http://permalink.gmane.org/gmane.linux.kernel/292755
just to see how things go...

Quoted 7 lines
Quoted 3 lines

I have a userspace application to test fsevent, I'll release it to community in order to find more issues on fsevent.

And please publish some numbers so people could make some prognosis of system behaviour.

Do you mean perfornance indice?

#5 lore raw
Quoted 23 lines · Yi Yang
Quoted 11 lines · Evgeniy Polyakov
Quoted 7 lines · Yi Yang
Quoted 7 lines

fs/fsevent.c line 419

Quoted 3 lines
fs/fsevent.c new file +1 -0
+		if (skb->len >= FSEVENT_FILTER_MSGSIZE) {

I'm not sure about your size checks.
I think it should be compared with nlhdr->nlmsg_len?

Quoted 6 lines
Quoted 2 lines

At this point, skb->len should be the same as nlhdr->nlmsg_len.

Hmm, skb->len includes size of netlink header, but nlhdr->nlmsg_len does not.

No, it included length of netlink header, please see it. struct nlmsghdr
{
__u32 nlmsg_len; /* Length of message including
header */

Quoted 5 lines
       __u16           nlmsg_type;     /* Message content */
       __u16           nlmsg_flags;    /* Additional flags */
       __u32           nlmsg_seq;      /* Sequence number */
       __u32           nlmsg_pid;      /* Sending process PID */

};

You are right, you do not use netlink header size explicitly in skb size calculation during allocation, so this is ok.

Quoted 46 lines
Quoted 46 lines
Quoted 45 lines
Quoted 35 lines
Quoted 21 lines
fs/fsevent.c new file +19 -0
+static void __exit fsevent_exit(void)
+{
+	listener * p = NULL, * q = NULL;
+	int cpu;
+	int wait_flag = 0;
+	struct sk_buff_head * skb_head = NULL;
+
+	fsevents_mask = 0;
+	_raise_fsevent = 0;
+	exit_flag = 1;
+
+	for_each_cpu(cpu)
+		schedule_work(&per_cpu(fsevent_work, cpu));
+
+	while (1) {
+		wait_flag = 0;
+		for_each_cpu(cpu) {
+			skb_head = &per_cpu(fsevent_send_queue, cpu);
+			if (skb_head->qlen != 0) {

fs/fsevent.c line 596

Quoted 10 lines
fs/fsevent.c new file +10 -0
+				wait_flag = 1;
+				break;
+			}
+		}
+		if (wait_flag == 1) {
+			set_current_state(TASK_INTERRUPTIBLE);
+			schedule_timeout(HZ/10);
+		} else
+			break;
+	}

This is still broken.
You race with schedule_work() in this loop. It requires flush_scheduled_work().

Quoted 6 lines

And I still have soume doubts about __raise_fsevent(). What if you set fsevents_mask to zero after __raise_fsevent() is started, but not yet queued an skb, and above loop and scheduled work are completed?

I think it is OK, schedule_timeout will release cpu to work queues, work queues should have enough time
to finish their works, I don't know what is your reason.

Quoted 8 lines
Quoted 7 lines

It is not guaranteed that scheduled work will be processed until flush_scheduled_work() completion, no matter how many times processor has idle cycles.

Second issue is that both above loop and work can be finished, but some __raise_fsevent() will be still in progress.

I knew your meaning, if you have a better way, tell me.

Some kind of flag to specify that work is in progress, or reference counters for subsystem, or even use mutuxes in sending path... There are a lot of variants, from complex and fast to slow and simple.

Quoted 19 lines
Quoted 19 lines
Quoted 18 lines
Quoted 4 lines

You need some type of completion of the last worker...

Quoted 8 lines

fs/fsevent.c line 599

Quoted 4 lines
fs/fsevent.c new file +2 -0
+	atomic_set(&fsevent_sock->sk_rmem_alloc, 0);
+	atomic_set(&fsevent_sock->sk_wmem_alloc, 0);

This is really wrong, since it hides skb processing errors like double freeing or leaks.

If userspace application terminated exceptionally, there are some skbs not to be consumed on socket, so
if you rmmod it, sock_release will report some failure information, the above two statements will remove this
error.

Quoted 7 lines
Quoted 5 lines

All queues will be flushed, when socket is freed, and if sock_release() shows
that assertion is failed, this definitely means you broke socket accounting, for example freed skb two times.

I searched those code, it didn't decrease sk_rmem_alloc, do you mean skb_queue_purge will decrease it?

Yes.

__kfree_skb() calls skb->destructor() callback, which can be sock_rfree() or sock_wfree() for example, depending on socket type, which decreases appropriate queue accounting variable.

Quoted 17 lines
Quoted 17 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
fs/fsevent.c new file +1 -0
+	sock_release(fsevent_sock->sk_socket);

...

Quoted 9 lines
Quoted 5 lines

Btw, it would be nice to have some kind of microbenchmark, like http://permalink.gmane.org/gmane.linux.kernel/292755
just to see how things go...

I have a userspace application to test fsevent, I'll release it to community in order to find more issues on
fsevent.

Quoted 4 lines
Quoted 3 lines

And please publish some numbers so people could make some prognosis of system behaviour.

Do you mean perfornance indice?

Probably.

It is quite hard to tell how well system behaves only looking into the code, although it tells a lot.

Create for example a lot of files, or measure some other parameters which are more close to real life behaviour, and check the time with and without your code turned on, how many messages were lost if any, CPU usage and so on...

Signature · 1 line
	Evgeniy Polyakov