Skip to contentSkip to conversation

Filessytem Events Reporter V2

PATCH In review Yi Yang 3 participants 11 messages Apr 5, 2006 – Apr 7, 2006

Subsystems (3)
lore
Share & tools
lore

Yi Yang

author
#0 lore raw

PATCH 1/1 Filessytem Events Reporter V2

Compared with Filesystem Events Reporter v1, the following changes are done:
- Use workqueue/keventd instead of kfseventd.
- fsevent_mask can be set by sysctl and proc
interface.
- Add missed spinlock
- Ensure fsevent sequence number is uniqe

This patch implements a filsystem events report facitily, Filesystem Events Reporter, the user can monitor filesystem activities via it, currently, it
can monitor access, attribute change, open, create, modify, delete, move and close of any file or directory as well as mount/umount.

Every filesystem event will include tgid, uid and gid of the process
which triggered this event, process name, file or directory name
operated by it.

Filesystem Events Reporter is never a duplicate of inotify, inotify
just concerns change on file or directory, Beagle uses it to watch file changes in order to regenerate index for it, inotify can't tell us who did that change and what is its process name, but filesystem events reporter can do these, moreover inotify's overhead is greater than filesystem events reporter, inotify needs compare inode with
watched file or directories list to decide whether it should generate an
inotify_event, some locks also increase overhead, filesystem event
connector hasn't these overhead, it just generates a fsevent and send.

To be important, filesystem event reporter doesn't add any new system call, the user space application can make use of it by netlink socket, but inotify added several system calls, many events mechanism in kernel
have used netlink as communication way with user space, for example,
KOBJECT_UEVENT, PROC_EVENTS, to use netlink will make it more possible
to unify events interface to netlink, the user space application can use
it very easy.

Every application using fsevent can set its own fsevent filter list without effect in other applications using fsevent, there are three filter lists, they are pid filter list, uid filetr list and gid filter list, respectively, moreover, there is a fsevent mask used to control those fsevents which fail to match three filter lists, an application using fsevent can listen those fsevents it want to monitor and ignore those fsevents it doesn't interest in by set series of filters, there is a fsevent mask used to take effects on all the applications using fsevent, it can be set by sysctl and proc interface.

 fs/Kconfig               |   10 
 fs/Makefile              |    1 
 fs/fsevent.c             |  598 +++++++++++++++++++++++++++++++++++++++++++++++
 fs/namespace.c           |   12 
 include/linux/fsevent.h  |  131 ++++++++++
 include/linux/fsnotify.h |   37 ++
 include/linux/netlink.h  |    1 
 7 files changed, 790 insertions(+)
include/linux/netlink.h renamed from include/linux/netlink.h.orig +1 -0
@@ -21,6 +21,7 @@
 #define NETLINK_DNRTMSG		14	/* DECnet routing messages */
 #define NETLINK_KOBJECT_UEVENT	15	/* Kernel messages to userspace */
 #define NETLINK_GENERIC		16
+#define NETLINK_FSEVENT		17	/* File system events to userspace */
 
 #define MAX_LINKS 32		
 
include/linux/fsnotify.h renamed from include/linux/fsnotify.h.orig +37 -0
@@ -15,6 +15,7 @@
 
 #include <linux/dnotify.h>
 #include <linux/inotify.h>
+#include <linux/fsevent.h>
 
 /*
  * fsnotify_move - file old_name at old_dir was moved to new_name at new_dir
@@ -45,6 +46,8 @@ static inline void fsnotify_move(struct 
 	if (source) {
 		inotify_inode_queue_event(source, IN_MOVE_SELF, 0, NULL);
 	}
+	raise_fsevent_move(old_dir, old_name, new_dir, new_name,
+			   FSEVENT_MOVE | (isdir?FSEVENT_ISDIR:0));
 }
 
 /*
@@ -56,6 +59,8 @@ static inline void fsnotify_nameremove(s
 		isdir = IN_ISDIR;
 	dnotify_parent(dentry, DN_DELETE);
 	inotify_dentry_parent_queue_event(dentry, IN_DELETE|isdir, 0, dentry->d_name.name);
+	raise_fsevent(dentry,
+		      FSEVENT_DELETE | (isdir?FSEVENT_ISDIR:0));
 }
 
 /*
@@ -74,6 +79,7 @@ static inline void fsnotify_create(struc
 {
 	inode_dir_notify(inode, DN_CREATE);
 	inotify_inode_queue_event(inode, IN_CREATE, 0, name);
+	raise_fsevent_create(inode, name, FSEVENT_CREATE);
 }
 
 /*
@@ -83,6 +89,8 @@ static inline void fsnotify_mkdir(struct
 {
 	inode_dir_notify(inode, DN_CREATE);
 	inotify_inode_queue_event(inode, IN_CREATE | IN_ISDIR, 0, name);
+	raise_fsevent_create(inode, name,
+			     FSEVENT_CREATE | FSEVENT_ISDIR);
 }
 
 /*
@@ -99,6 +107,8 @@ static inline void fsnotify_access(struc
 	dnotify_parent(dentry, DN_ACCESS);
 	inotify_dentry_parent_queue_event(dentry, mask, 0, dentry->d_name.name);
 	inotify_inode_queue_event(inode, mask, 0, NULL);
+	raise_fsevent(dentry, FSEVENT_ACCESS |
+				((S_ISDIR(inode->i_mode))?FSEVENT_ISDIR:0));
 }
 
 /*
@@ -115,6 +125,8 @@ static inline void fsnotify_modify(struc
 	dnotify_parent(dentry, DN_MODIFY);
 	inotify_dentry_parent_queue_event(dentry, mask, 0, dentry->d_name.name);
 	inotify_inode_queue_event(inode, mask, 0, NULL);
+	raise_fsevent(dentry, FSEVENT_MODIFY |
+				((S_ISDIR(inode->i_mode))?FSEVENT_ISDIR:0));
 }
 
 /*
@@ -130,6 +142,9 @@ static inline void fsnotify_open(struct 
 
 	inotify_dentry_parent_queue_event(dentry, mask, 0, dentry->d_name.name);
 	inotify_inode_queue_event(inode, mask, 0, NULL);	
+	raise_fsevent(dentry, FSEVENT_OPEN |
+				((S_ISDIR(inode->i_mode))?FSEVENT_ISDIR:0));
+
 }
 
 /*
@@ -148,6 +163,8 @@ static inline void fsnotify_close(struct
 
 	inotify_dentry_parent_queue_event(dentry, mask, 0, name);
 	inotify_inode_queue_event(inode, mask, 0, NULL);
+	raise_fsevent(dentry, FSEVENT_CLOSE |
+				((S_ISDIR(inode->i_mode))?FSEVENT_ISDIR:0));
 }
 
 /*
@@ -163,6 +180,8 @@ static inline void fsnotify_xattr(struct
 
 	inotify_dentry_parent_queue_event(dentry, mask, 0, dentry->d_name.name);
 	inotify_inode_queue_event(inode, mask, 0, NULL);
+	raise_fsevent(dentry, FSEVENT_MODIFY_ATTRIB |
+				((S_ISDIR(inode->i_mode))?FSEVENT_ISDIR:0));
 }
 
 /*
@@ -213,6 +232,24 @@ static inline void fsnotify_change(struc
 		inotify_dentry_parent_queue_event(dentry, in_mask, 0,
 						  dentry->d_name.name);
 	}
+
+#ifdef CONFIG_FS_EVENTS
+	{
+	u32 fsevent_mask = 0;
+	if (ia_valid & (ATTR_UID | ATTR_GID | ATTR_MODE))
+		fsevent_mask |= FSEVENT_MODIFY_ATTRIB;
+	if ((ia_valid & ATTR_ATIME) && (ia_valid & ATTR_MTIME))
+		fsevent_mask |= FSEVENT_MODIFY_ATTRIB;
+	else if (ia_valid & ATTR_ATIME)
+		fsevent_mask |= FSEVENT_ACCESS;
+	else if (ia_valid & ATTR_MTIME)
+		fsevent_mask |= FSEVENT_MODIFY;
+	if (ia_valid & ATTR_SIZE)
+		fsevent_mask |= FSEVENT_MODIFY;
+	if (fsevent_mask)
+		raise_fsevent(dentry, fsevent_mask);
+	}
+#endif /* CONFIG_FS_EVENTS */
 }
 
 #ifdef CONFIG_INOTIFY	/* inotify helpers */
fs/namespace.c renamed from fs/namespace.c.orig +12 -0
@@ -25,6 +25,7 @@
 #include <linux/mount.h>
 #include <asm/uaccess.h>
 #include <asm/unistd.h>
+#include <linux/fsevent.h>
 #include "pnode.h"
 
 extern int __init init_rootfs(void);
@@ -612,6 +613,13 @@ asmlinkage long sys_umount(char __user *
 		goto dput_and_out;
 
 	retval = do_umount(nd.mnt, flags);
+
+	if (retval == 0) {
+		char * tmp = getname(name);
+		raise_fsevent_umount(tmp);
+		putname(tmp);
+	}
+
 dput_and_out:
 	path_release_on_umount(&nd);
 out:
@@ -1459,6 +1467,10 @@ asmlinkage long sys_mount(char __user * 
 	retval = do_mount((char *)dev_page, dir_page, (char *)type_page,
 			  flags, (void *)data_page);
 	unlock_kernel();
+
+	if (retval == 0)
+		raise_fsevent_mount((char *)dev_page, dir_page);
+
 	free_page(data_page);
 
 out3:
fs/Kconfig renamed from fs/Kconfig.orig +10 -0
@@ -405,6 +405,16 @@ config INOTIFY
 
 	  If unsure, say Y.
 
+config FS_EVENTS
+	tristate "Report filesystem events to userspace"
+	---help---
+	  Provide a facility that reports filesystem events to userspace. The
+	  reported event include access, write, utime, chmod, chown, chgrp,
+	  close, open, create, rename, unlink, mkdir, rmdir, mount, umount.
+
+	  The user can set filesystem events filter to filter its events, so
+	  that he just get those events he concerns.
+
 config QUOTA
 	bool "Quota support"
 	help
fs/Makefile renamed from fs/Makefile.orig +1 -0
@@ -13,6 +13,7 @@ obj-y :=	open.o read_write.o file_table.
 		ioprio.o pnode.o drop_caches.o
 
 obj-$(CONFIG_INOTIFY)		+= inotify.o
+obj-$(CONFIG_FS_EVENTS)		+= fsevent.o
 obj-$(CONFIG_EPOLL)		+= eventpoll.o
 obj-$(CONFIG_COMPAT)		+= compat.o compat_ioctl.o
 
include/linux/fsevent.h new file +131 -0

131 lines not on this page · raw message

fs/fsevent.c new file +598 -0

598 lines not on this page · raw message

#1 lore raw

fs/fsevent.c line 409

Quoted 10 lines · Yi Yang

<snip>

fs/fsevent.c new file +8 -0
+static void cleanup_dead_listener(listener * x)
+{
+	pid_filter * p = NULL, * pq = NULL;
+	uid_filter * u = NULL, * uq = NULL;
+	gid_filter * g = NULL, * gq = NULL;
+
+	if (p == NULL)
+		return;
<snip>

I think you ment "if (x == NULL)" here. Otherwise, the condition will always be true. btw, I'm not reviewing your code, just stumbled across this while reading it.

author
#2 lore raw
Quoted 17 lines · Carlos Silva

fs/fsevent.c line 409

Quoted 11 lines · Yi Yang

<snip>

fs/fsevent.c new file +8 -0
+static void cleanup_dead_listener(listener * x)
+{
+	pid_filter * p = NULL, * pq = NULL;
+	uid_filter * u = NULL, * uq = NULL;
+	gid_filter * g = NULL, * gq = NULL;
+
+	if (p == NULL)
+		return;
<snip>

I think you ment "if (x == NULL)" here. Otherwise, the condition will always be true. btw, I'm not reviewing your code, just stumbled across this while reading it.

Yes, it is my mistake, thank you.

#3 lore raw
In reply to Yi Yang
Quoted 11 lines · Yi Yang

Compared with Filesystem Events Reporter v1, the following changes are done:
- Use workqueue/keventd instead of kfseventd.
- fsevent_mask can be set by sysctl and proc
interface.
- Add missed spinlock
- Ensure fsevent sequence number is uniqe

This patch implements a filsystem events report facitily, Filesystem Events Reporter, the user can monitor filesystem activities via it, currently, it
can monitor access, attribute change, open, create, modify, delete, move and close of any file or directory as well as mount/umount.

Comments below.

...

Quoted 24 lines
fs/fsevent.c new file +24 -0
+
+static int fsevent_ack(enum fsevent_type type, pid_t pid, u32 seq)
+{
+	struct fsevent *event;
+	unsigned int size;
+	struct sk_buff * skb = NULL;
+	struct nlmsghdr * nlhdr = NULL;
+
+	size = NLMSG_SPACE(sizeof(struct fsevent));
+	                                                                                                                                       
+	skb = alloc_skb(size, GFP_KERNEL);
+	if (!skb)
+	        return -ENOMEM;
+	                                                                                                                                       
+	nlhdr = NLMSG_PUT(skb, 0, seq, NLMSG_DONE, size - sizeof(*nlhdr));
+	event = NLMSG_DATA(nlhdr);
+
+	ktime_get_ts(&event->timestamp);
+	event->cpu = -1;
+	event->type = type; 
+	event->pid = 0;
+	event->uid = 0;
+	event->gid = 0;
+	event->len = 0;

fs/fsevent.c line 263

Quoted 10 lines
fs/fsevent.c new file +10 -0
+	event->pname_len = 0;
+	event->fname_len = 0;
+	event->new_fname_len = 0;
+	event->err = 0;
+	                                                                                                                                       
+	NETLINK_CB(skb).dst_group = 0;
+	NETLINK_CB(skb).dst_pid = pid;
+	NETLINK_CB(skb).pid = 0;
+
+	return (netlink_unicast(fsevent_sock, skb, pid, MSG_DONTWAIT));

netlink_unicast() uses boolean value but ont MSG_* flags for nonblocking, so this should be netlink_unicast(fsevent_sock, skb, pid, 0);

Quoted 4 lines
fs/fsevent.c new file +4 -0
+nlmsg_failure:
+	kfree_skb(skb);
+	return -1;
+}

...

Quoted 6 lines
fs/fsevent.c new file +6 -0
+static void fsevent_recv(struct sock *sk, int len)
+{
+	struct sk_buff *skb = NULL;
+	struct nlmsghdr *nlhdr = NULL;
+	struct fsevent_filter * filter = NULL;
+	pid_t pid;

fs/fsevent.c line 446

Quoted 10 lines
fs/fsevent.c new file +10 -0
+
+	while ((skb = skb_dequeue(&sk->sk_receive_queue)) != NULL) {
+		skb_get(skb);
+		if (skb->len >= FSEVENT_FILTER_MSGSIZE) {
+			nlhdr = (struct nlmsghdr *)skb->data;
+			filter = NLMSG_DATA(nlhdr);
+			pid = NETLINK_CREDS(skb)->pid;
+			if (find_fsevent_listener(pid) == NULL)
+				atomic_inc(&fsevent_listener_num);
+			set_fsevent_filter(filter, pid);

What is the logic behind this steps?
If there are no listeners you increment it's number no matter if it will or not be added in set_fsevent_filter().

Quoted 18 lines
fs/fsevent.c new file +18 -0
+		}
+		kfree_skb(skb);
+	}
+}
+
+#define DEFINE_FILTER_MATCH_FUNC(filtertype, key) 			\
+	static int match_##filtertype(listener * p,			\
+				struct fsevent * event,			\
+				struct sk_buff * skb)			\
+	{								\
+		int ret = 0;						\
+		filtertype * xfilter = NULL;				\
+		struct sk_buff * skb2 = NULL;				\
+		struct list_head *  head = &(p->key##_filter_list_head);  \
+		list_for_each_entry(xfilter, head, list) {		\
+			if (xfilter->key != event->key)			\
+				continue;				\
+			ret = filter_fsevent(xfilter->mask, event->type); \

fs/fsevent.c line 474

Quoted 10 lines
fs/fsevent.c new file +10 -0
+			if ( ret != 0)					\
+				return -1;				\
+			skb2 = skb_clone(skb, GFP_KERNEL);		\
+       			if (skb2 == NULL)				\
+				return -ENOMEM;				\
+			NETLINK_CB(skb2).dst_group = 0;			\
+			NETLINK_CB(skb2).dst_pid = p->pid;		\
+			NETLINK_CB(skb2).pid = 0;			\
+			return (netlink_unicast(fsevent_sock, skb2,	\
+					p->pid, MSG_DONTWAIT));		\

The same issue about nonblocking sending.

Quoted 51 lines
fs/fsevent.c new file +51 -0
+		}							\
+		return -ENODEV;						\
+	}								\
+
+DEFINE_FILTER_MATCH_FUNC(pid_filter, pid)
+
+DEFINE_FILTER_MATCH_FUNC(uid_filter, uid)
+
+DEFINE_FILTER_MATCH_FUNC(gid_filter, gid)
+
+#define MATCH_XID(key, listenerp, event, skb) 			\
+	ret = match_##key##_filter(listenerp, event, skb); 	\
+	if (ret == 0) {					 	\
+		kfree_skb(skb);				 	\
+	        continue;				 	\
+	}						 	\
+	do {} while (0)					 	\
+
+static int fsevent_send_to_process(struct sk_buff * skb)
+{
+	listener * p  = NULL, * q = NULL;
+	struct fsevent * event = NULL;
+	struct sk_buff * skb2 = NULL;
+	int ret = 0;
+
+	event = (struct fsevent *)(skb->data + sizeof(struct nlmsghdr));
+	spin_lock(&listener_list_lock);
+	list_for_each_entry_safe(p, q, &listener_list_head, list) {
+		MATCH_XID(pid, p, event, skb);
+		MATCH_XID(uid, p, event, skb);
+		MATCH_XID(gid, p, event, skb);
+
+		if (filter_fsevent(p->mask, event->type) == 0) {
+			 skb2 = skb_clone(skb, GFP_KERNEL);
+	                 if (skb2 == NULL)
+	                 	return -ENOMEM;
+	                 NETLINK_CB(skb2).dst_group = 0;
+	                 NETLINK_CB(skb2).dst_pid = p->pid;
+	                 NETLINK_CB(skb2).pid = 0;
+	                 ret = netlink_unicast(fsevent_sock, skb2,
+	                                p->pid, MSG_DONTWAIT);
+			if (ret == -ECONNREFUSED) {
+				atomic_dec(&fsevent_listener_num);
+				cleanup_dead_listener(p);
+			}
+		}
+	}
+	spin_unlock(&listener_list_lock);
+	return ret;
+}
+

fs/fsevent.c line 535

Quoted 10 lines
fs/fsevent.c new file +10 -0
+static void fsevent_commit(void * unused)
+{
+	struct sk_buff * skb = NULL;
+		
+	while((skb = skb_dequeue(&get_cpu_var(fsevent_send_queue)))
+		!= NULL) {
+		fsevent_send_to_process(skb);
+		put_cpu_var(fsevent_send_queue);
+	}
+}

Really strange mix of per-cpu variables for optimized performance and global spin locking.
Consider using RCU for list of listeners.

You use unicast delivery for netlink messages.
According to my investigation [1], it's performance is better only when there is only one listener (or maybe two in some cases), but then it is noticebly slower than broadcasting.

1. http://marc.theaimsgroup.com/?l=linux-netdev&m=114424884216006&w=2

Quoted 28 lines
fs/fsevent.c new file +28 -0
+static struct ctl_table fsevent_mask_sysctl[] = {
+	{
+		.ctl_name = FSEVENT_MASK_CTL_NAME,
+		.procname = "fsevent_mask",
+		.data = &fsevents_mask,
+		.maxlen = sizeof(u32),
+		.mode = 0644,
+		.proc_handler = &proc_dointvec,
+	},
+	{ .ctl_name = 0 }
+};
+
+static struct ctl_table fs_root_sysctl[] = {
+	{
+		.ctl_name = CTL_FS,
+		.procname = "fs",
+		.mode = 0555,
+		.child = fsevent_mask_sysctl,
+	},
+	{ .ctl_name = 0 }
+};
+
+static int __init fsevent_init(void)
+{
+	int cpu;
+	struct sk_buff_head * listptr;
+	struct work_struct * workptr;
+

fs/fsevent.c line 574

Quoted 10 lines
fs/fsevent.c new file +10 -0
+	fsevent_sock = netlink_kernel_create(NETLINK_FSEVENT, 0,
+					 fsevent_recv, THIS_MODULE);
+	if (!fsevent_sock)
+		return -EIO;
+	for_each_cpu(cpu) {
+		listptr = &per_cpu(fsevent_send_queue, cpu);
+		skb_queue_head_init(listptr);
+		workptr = &per_cpu(fsevent_work, cpu);
+		INIT_WORK(workptr, fsevent_commit, NULL);
+	}

Btw, you need some rebalancing of the per-cpu queues, probably in keventd, since CPUs can go offline and your messages will stuck foreve there.

Quoted 5 lines
fs/fsevent.c new file +5 -0
+	if (register_sysctl_table(fs_root_sysctl, 0) == NULL)
+                return -ENOMEM;
+
+	return 0;
+}

fs/fsevent.c line 590

Quoted 10 lines
fs/fsevent.c new file +10 -0
+
+static void __exit fsevent_exit(void)
+{
+	listener * p = NULL, * q = NULL;
+	sock_release(fsevent_sock->sk_socket);
+	spin_lock(&listener_list_lock);
+	list_for_each_entry_safe(p, q, &listener_list_head, list) {
+		cleanup_dead_listener(p);
+	}
+	spin_unlock(&listener_list_lock);

Broken. Your work can be pending on this stage, but you already removed and freed resources.

Quoted 9 lines
fs/fsevent.c new file +8 -0
+}
+
+module_init(fsevent_init);
+module_exit(fsevent_exit);
+
+MODULE_LICENSE("GPL");
+MODULE_AUTHOR("Yi Yang <[email protected]>");
+MODULE_DESCRIPTION("File System Events Reporter");
Signature · 1 line
	Evgeniy Polyakov
author
#4 lore raw
Quoted 48 lines · Evgeniy Polyakov
Quoted 7 lines · Yi Yang

Compared with Filesystem Events Reporter v1, the following changes are done:
- Use workqueue/keventd instead of kfseventd.
- fsevent_mask can be set by sysctl and proc
interface.
- Add missed spinlock
- Ensure fsevent sequence number is uniqe

Quoted 5 lines

This patch implements a filsystem events report facitily, Filesystem Events Reporter, the user can monitor filesystem activities via it, currently, it
can monitor access, attribute change, open, create, modify, delete, move and close of any file or directory as well as mount/umount.

Comments below.

...

Quoted 29 lines
fs/fsevent.c new file +29 -0
+
+static int fsevent_ack(enum fsevent_type type, pid_t pid, u32 seq)
+{
+	struct fsevent *event;
+	unsigned int size;
+	struct sk_buff * skb = NULL;
+	struct nlmsghdr * nlhdr = NULL;
+
+	size = NLMSG_SPACE(sizeof(struct fsevent));
+	                                                                                                                                       
+	skb = alloc_skb(size, GFP_KERNEL);
+	if (!skb)
+	        return -ENOMEM;
+	                                                                                                                                       
+	nlhdr = NLMSG_PUT(skb, 0, seq, NLMSG_DONE, size - sizeof(*nlhdr));
+	event = NLMSG_DATA(nlhdr);
+
+	ktime_get_ts(&event->timestamp);
+	event->cpu = -1;
+	event->type = type; 
+	event->pid = 0;
+	event->uid = 0;
+	event->gid = 0;
+	event->len = 0;
+	event->pname_len = 0;
+	event->fname_len = 0;
+	event->new_fname_len = 0;
+	event->err = 0;
+	                                                                                                                                       
Quoted 9 lines

fs/fsevent.c line 263

Quoted 6 lines
fs/fsevent.c new file +5 -0
+	NETLINK_CB(skb).dst_group = 0;
+	NETLINK_CB(skb).dst_pid = pid;
+	NETLINK_CB(skb).pid = 0;
+
+	return (netlink_unicast(fsevent_sock, skb, pid, MSG_DONTWAIT));

netlink_unicast() uses boolean value but ont MSG_* flags for nonblocking, so this should be netlink_unicast(fsevent_sock, skb, pid, 0);

a example invocation in file net/netlink/af_netlink.c:
netlink_unicast(in_skb->sk, skb, NETLINK_CB(in_skb).pid, MSG_DONTWAIT); so, it hasn't any problem.

Quoted 23 lines
Quoted 6 lines
fs/fsevent.c new file +4 -0
+nlmsg_failure:
+	kfree_skb(skb);
+	return -1;
+}

...

Quoted 13 lines
fs/fsevent.c new file +13 -0
+static void fsevent_recv(struct sock *sk, int len)
+{
+	struct sk_buff *skb = NULL;
+	struct nlmsghdr *nlhdr = NULL;
+	struct fsevent_filter * filter = NULL;
+	pid_t pid;
+
+	while ((skb = skb_dequeue(&sk->sk_receive_queue)) != NULL) {
+		skb_get(skb);
+		if (skb->len >= FSEVENT_FILTER_MSGSIZE) {
+			nlhdr = (struct nlmsghdr *)skb->data;
+			filter = NLMSG_DATA(nlhdr);
+			pid = NETLINK_CREDS(skb)->pid;
Quoted 9 lines

fs/fsevent.c line 446

Quoted 4 lines
fs/fsevent.c new file +3 -0
+			if (find_fsevent_listener(pid) == NULL)
+				atomic_inc(&fsevent_listener_num);
+			set_fsevent_filter(filter, pid);

What is the logic behind this steps?
If there are no listeners you increment it's number no matter if it will or not be added in set_fsevent_filter().

fsevent_recv is used to receive listener's commands, a listener must send commands in order to get fsevents it interests, so this is the best point to increment number of listeners. set_fsevent_filter will add listener to listener list, so it is OK.

Quoted 92 lines
Quoted 20 lines
fs/fsevent.c new file +18 -0
+		}
+		kfree_skb(skb);
+	}
+}
+
+#define DEFINE_FILTER_MATCH_FUNC(filtertype, key) 			\
+	static int match_##filtertype(listener * p,			\
+				struct fsevent * event,			\
+				struct sk_buff * skb)			\
+	{								\
+		int ret = 0;						\
+		filtertype * xfilter = NULL;				\
+		struct sk_buff * skb2 = NULL;				\
+		struct list_head *  head = &(p->key##_filter_list_head);  \
+		list_for_each_entry(xfilter, head, list) {		\
+			if (xfilter->key != event->key)			\
+				continue;				\
+			ret = filter_fsevent(xfilter->mask, event->type); \

fs/fsevent.c line 474

Quoted 10 lines
fs/fsevent.c new file +10 -0
+			if ( ret != 0)					\
+				return -1;				\
+			skb2 = skb_clone(skb, GFP_KERNEL);		\
+       			if (skb2 == NULL)				\
+				return -ENOMEM;				\
+			NETLINK_CB(skb2).dst_group = 0;			\
+			NETLINK_CB(skb2).dst_pid = p->pid;		\
+			NETLINK_CB(skb2).pid = 0;			\
+			return (netlink_unicast(fsevent_sock, skb2,	\
+					p->pid, MSG_DONTWAIT));		\

The same issue about nonblocking sending.

Quoted 58 lines
fs/fsevent.c new file +58 -0
+		}							\
+		return -ENODEV;						\
+	}								\
+
+DEFINE_FILTER_MATCH_FUNC(pid_filter, pid)
+
+DEFINE_FILTER_MATCH_FUNC(uid_filter, uid)
+
+DEFINE_FILTER_MATCH_FUNC(gid_filter, gid)
+
+#define MATCH_XID(key, listenerp, event, skb) 			\
+	ret = match_##key##_filter(listenerp, event, skb); 	\
+	if (ret == 0) {					 	\
+		kfree_skb(skb);				 	\
+	        continue;				 	\
+	}						 	\
+	do {} while (0)					 	\
+
+static int fsevent_send_to_process(struct sk_buff * skb)
+{
+	listener * p  = NULL, * q = NULL;
+	struct fsevent * event = NULL;
+	struct sk_buff * skb2 = NULL;
+	int ret = 0;
+
+	event = (struct fsevent *)(skb->data + sizeof(struct nlmsghdr));
+	spin_lock(&listener_list_lock);
+	list_for_each_entry_safe(p, q, &listener_list_head, list) {
+		MATCH_XID(pid, p, event, skb);
+		MATCH_XID(uid, p, event, skb);
+		MATCH_XID(gid, p, event, skb);
+
+		if (filter_fsevent(p->mask, event->type) == 0) {
+			 skb2 = skb_clone(skb, GFP_KERNEL);
+	                 if (skb2 == NULL)
+	                 	return -ENOMEM;
+	                 NETLINK_CB(skb2).dst_group = 0;
+	                 NETLINK_CB(skb2).dst_pid = p->pid;
+	                 NETLINK_CB(skb2).pid = 0;
+	                 ret = netlink_unicast(fsevent_sock, skb2,
+	                                p->pid, MSG_DONTWAIT);
+			if (ret == -ECONNREFUSED) {
+				atomic_dec(&fsevent_listener_num);
+				cleanup_dead_listener(p);
+			}
+		}
+	}
+	spin_unlock(&listener_list_lock);
+	return ret;
+}
+
+static void fsevent_commit(void * unused)
+{
+	struct sk_buff * skb = NULL;
+		
+	while((skb = skb_dequeue(&get_cpu_var(fsevent_send_queue)))
+		!= NULL) {
+		fsevent_send_to_process(skb);
Quoted 9 lines

fs/fsevent.c line 159

Quoted 4 lines
fs/fsevent.c new file +3 -0
+		put_cpu_var(fsevent_send_queue);
+	}
+}

Really strange mix of per-cpu variables for optimized performance and global spin locking.
Consider using RCU for list of listeners.

per cpu queue is used to avoid raise_fsevent to contend spinlock, but listener_list_lock just is used to synchronize the operations of userspace applications(listener) on listener list, it just protect listener list.

Of course, your advice is good, RCU will be better, I'm considering substitute spinlock with RCU, maybe list*_rcu functions can help me.

Quoted 6 lines

You use unicast delivery for netlink messages. According to my investigation [1], it's performance is better only when there is only one listener (or maybe two in some cases), but then it is
noticebly slower than broadcasting.

1. http://marc.theaimsgroup.com/?l=linux-netdev&m=114424884216006&w=2

Because fsevent has to deliver different events to different listeners, so I must use netlink_unicast, in fact, netlink_broadcast also must send skb to every member of the group, so in my opinion, they haven't big difference.

Can you explain why there is such a big difference between netlink_unicast and netlink_broadcast?

Quoted 36 lines
Quoted 36 lines
fs/fsevent.c new file +35 -0
+static struct ctl_table fsevent_mask_sysctl[] = {
+	{
+		.ctl_name = FSEVENT_MASK_CTL_NAME,
+		.procname = "fsevent_mask",
+		.data = &fsevents_mask,
+		.maxlen = sizeof(u32),
+		.mode = 0644,
+		.proc_handler = &proc_dointvec,
+	},
+	{ .ctl_name = 0 }
+};
+
+static struct ctl_table fs_root_sysctl[] = {
+	{
+		.ctl_name = CTL_FS,
+		.procname = "fs",
+		.mode = 0555,
+		.child = fsevent_mask_sysctl,
+	},
+	{ .ctl_name = 0 }
+};
+
+static int __init fsevent_init(void)
+{
+	int cpu;
+	struct sk_buff_head * listptr;
+	struct work_struct * workptr;
+
+	fsevent_sock = netlink_kernel_create(NETLINK_FSEVENT, 0,
+					 fsevent_recv, THIS_MODULE);
+	if (!fsevent_sock)
+		return -EIO;
+	for_each_cpu(cpu) {
+		listptr = &per_cpu(fsevent_send_queue, cpu);
+		skb_queue_head_init(listptr);
Quoted 9 lines

fs/fsevent.c line 574

Quoted 4 lines
fs/fsevent.c new file +3 -0
+		workptr = &per_cpu(fsevent_work, cpu);
+		INIT_WORK(workptr, fsevent_commit, NULL);
+	}

Btw, you need some rebalancing of the per-cpu queues, probably in keventd, since CPUs can go offline and your messages will stuck foreve there.

Does keventd not do it? if so, keventd should be modified.

Quoted 12 lines
Quoted 12 lines
fs/fsevent.c new file +11 -0
+	if (register_sysctl_table(fs_root_sysctl, 0) == NULL)
+                return -ENOMEM;
+
+	return 0;
+}
+
+static void __exit fsevent_exit(void)
+{
+	listener * p = NULL, * q = NULL;
+	sock_release(fsevent_sock->sk_socket);
+	spin_lock(&listener_list_lock);
Quoted 9 lines

fs/fsevent.c line 522

Quoted 5 lines
fs/fsevent.c new file +4 -0
+	list_for_each_entry_safe(p, q, &listener_list_head, list) {
+		cleanup_dead_listener(p);
+	}
+	spin_unlock(&listener_list_lock);

Broken. Your work can be pending on this stage, but you already removed and freed resources.

Sorry, I don't consider it, I will add it, thank you very much.

Quoted 13 lines
Quoted 11 lines
fs/fsevent.c new file +8 -0
+}
+
+module_init(fsevent_init);
+module_exit(fsevent_exit);
+
+MODULE_LICENSE("GPL");
+MODULE_AUTHOR("Yi Yang <[email protected]>");
+MODULE_DESCRIPTION("File System Events Reporter");
#5 lore raw
Quoted 10 lines · Yi Yang
Quoted 7 lines

fs/fsevent.c line 263

Quoted 3 lines
fs/fsevent.c new file +2 -0
+
+	return (netlink_unicast(fsevent_sock, skb, pid, MSG_DONTWAIT));

netlink_unicast() uses boolean value but ont MSG_* flags for nonblocking, so this should be netlink_unicast(fsevent_sock, skb, pid, 0);

a example invocation in file net/netlink/af_netlink.c:
netlink_unicast(in_skb->sk, skb, NETLINK_CB(in_skb).pid, MSG_DONTWAIT); so, it hasn't any problem.

Well...

static inline long sock_sndtimeo(const struct sock *sk, int noblock)

{
	return noblock ? 0 : sk->sk_sndtimeo;
}

int netlink_unicast(struct sock *ssk, struct sk_buff *skb, u32 pid, int nonblock)

{
	struct sock *sk;
	int err;
	long timeo;

	skb = netlink_trim(skb, gfp_any());

	timeo = sock_sndtimeo(ssk, nonblock);

I mean that it is boolean value, MSG_PEEK will produce the same result. But it is a matter of coding style probably.

Quoted 31 lines
Quoted 22 lines
Quoted 5 lines
fs/fsevent.c new file +4 -0
+nlmsg_failure:
+	kfree_skb(skb);
+	return -1;
+}

...

Quoted 13 lines
fs/fsevent.c new file +13 -0
+static void fsevent_recv(struct sock *sk, int len)
+{
+	struct sk_buff *skb = NULL;
+	struct nlmsghdr *nlhdr = NULL;
+	struct fsevent_filter * filter = NULL;
+	pid_t pid;
+
+	while ((skb = skb_dequeue(&sk->sk_receive_queue)) != NULL) {
+		skb_get(skb);
+		if (skb->len >= FSEVENT_FILTER_MSGSIZE) {
+			nlhdr = (struct nlmsghdr *)skb->data;
+			filter = NLMSG_DATA(nlhdr);
+			pid = NETLINK_CREDS(skb)->pid;
Quoted 9 lines

fs/fsevent.c line 446

Quoted 4 lines
fs/fsevent.c new file +3 -0
+			if (find_fsevent_listener(pid) == NULL)
+				atomic_inc(&fsevent_listener_num);
+			set_fsevent_filter(filter, pid);

What is the logic behind this steps?
If there are no listeners you increment it's number no matter if it will or not be added in set_fsevent_filter().

Quoted 5 lines

fsevent_recv is used to receive listener's commands, a listener must send commands in order to get fsevents it
interests, so this is the best point to increment number of listeners. set_fsevent_filter will add listener to listener
list, so it is OK.

And what if set_fsevent_filter() fails?

Quoted 106 lines
Quoted 91 lines
Quoted 19 lines
fs/fsevent.c new file +18 -0
+		}
+		kfree_skb(skb);
+	}
+}
+
+#define DEFINE_FILTER_MATCH_FUNC(filtertype, key) 			\
+	static int match_##filtertype(listener * p,			\
+				struct fsevent * event,			\
+				struct sk_buff * skb)			\
+	{								\
+		int ret = 0;						\
+		filtertype * xfilter = NULL;				\
+		struct sk_buff * skb2 = NULL;				\
+		struct list_head *  head = &(p->key##_filter_list_head);  \
+		list_for_each_entry(xfilter, head, list) {		\
+			if (xfilter->key != event->key)			\
+				continue;				\
+			ret = filter_fsevent(xfilter->mask, event->type); \

fs/fsevent.c line 474

Quoted 10 lines
fs/fsevent.c new file +10 -0
+			if ( ret != 0)					\
+				return -1;				\
+			skb2 = skb_clone(skb, GFP_KERNEL);		\
+       			if (skb2 == NULL)			 \
+				return -ENOMEM;				\
+			NETLINK_CB(skb2).dst_group = 0;			\
+			NETLINK_CB(skb2).dst_pid = p->pid;		\
+			NETLINK_CB(skb2).pid = 0;			\
+			return (netlink_unicast(fsevent_sock, skb2,	\
+					p->pid, MSG_DONTWAIT));		\

The same issue about nonblocking sending.

Quoted 58 lines
fs/fsevent.c new file +58 -0
+		}							\
+		return -ENODEV;						\
+	}								\
+
+DEFINE_FILTER_MATCH_FUNC(pid_filter, pid)
+
+DEFINE_FILTER_MATCH_FUNC(uid_filter, uid)
+
+DEFINE_FILTER_MATCH_FUNC(gid_filter, gid)
+
+#define MATCH_XID(key, listenerp, event, skb) 			\
+	ret = match_##key##_filter(listenerp, event, skb); 	\
+	if (ret == 0) {					 	\
+		kfree_skb(skb);				 	\
+	        continue;				 	\
+	}						 	\
+	do {} while (0)					 	\
+
+static int fsevent_send_to_process(struct sk_buff * skb)
+{
+	listener * p  = NULL, * q = NULL;
+	struct fsevent * event = NULL;
+	struct sk_buff * skb2 = NULL;
+	int ret = 0;
+
+	event = (struct fsevent *)(skb->data + sizeof(struct nlmsghdr));
+	spin_lock(&listener_list_lock);
+	list_for_each_entry_safe(p, q, &listener_list_head, list) {
+		MATCH_XID(pid, p, event, skb);
+		MATCH_XID(uid, p, event, skb);
+		MATCH_XID(gid, p, event, skb);
+
+		if (filter_fsevent(p->mask, event->type) == 0) {
+			 skb2 = skb_clone(skb, GFP_KERNEL);
+	                 if (skb2 == NULL)
+	                 	return -ENOMEM;
+	                 NETLINK_CB(skb2).dst_group = 0;
+	                 NETLINK_CB(skb2).dst_pid = p->pid;
+	                 NETLINK_CB(skb2).pid = 0;
+	                 ret = netlink_unicast(fsevent_sock, skb2,
+	                                p->pid, MSG_DONTWAIT);
+			if (ret == -ECONNREFUSED) {
+				atomic_dec(&fsevent_listener_num);
+				cleanup_dead_listener(p);
+			}
+		}
+	}
+	spin_unlock(&listener_list_lock);
+	return ret;
+}
+
+static void fsevent_commit(void * unused)
+{
+	struct sk_buff * skb = NULL;
+		
+	while((skb = skb_dequeue(&get_cpu_var(fsevent_send_queue)))
+		!= NULL) {
+		fsevent_send_to_process(skb);
Quoted 9 lines

fs/fsevent.c line 159

Quoted 4 lines
fs/fsevent.c new file +3 -0
+		put_cpu_var(fsevent_send_queue);
+	}
+}

Really strange mix of per-cpu variables for optimized performance and global spin locking.
Consider using RCU for list of listeners.

per cpu queue is used to avoid raise_fsevent to contend spinlock, but listener_list_lock just is used
to synchronize the operations of userspace applications(listener) on listener list, it just protect listener
list.

Quoted 3 lines

Of course, your advice is good, RCU will be better, I'm considering substitute spinlock with RCU,
maybe list*_rcu functions can help me.

You get global lock in each processor when traverse the list &listener_list_lock.

And you call GFP_KERNEL allocation under that lock, which is wrong.

If all your code is called from process context (it looks so), you could mutexes.

Quoted 5 lines
Quoted 5 lines

You use unicast delivery for netlink messages.
According to my investigation [1], it's performance is better only when there is only one listener (or maybe two in some cases), but then it is noticebly slower than broadcasting.

Quoted 7 lines
Quoted 2 lines

1. http://marc.theaimsgroup.com/?l=linux-netdev&m=114424884216006&w=2

Because fsevent has to deliver different events to different listeners, so I must use netlink_unicast,
in fact, netlink_broadcast also must send skb to every member of the group, so in my opinion,
they haven't big difference.

And what if there are several listeners for the same type of events?

Quoted 2 lines

Can you explain why there is such a big difference between netlink_unicast and netlink_broadcast?

Netlink broadcast clones skbs, while unicasting requires the whole new one.

Quoted 5 lines
Quoted 4 lines

Btw, you need some rebalancing of the per-cpu queues, probably in keventd, since CPUs can go offline and your messages will stuck foreve there.

Does keventd not do it? if so, keventd should be modified.

How does keventd know about your own structures?
You have an per-cpu object, but your keventd function gets object from running cpu, not from any other cpus.

Signature · 1 line
	Evgeniy Polyakov
author
#6 lore raw
Quoted 68 lines · Evgeniy Polyakov
Quoted 11 lines · Yi Yang
Quoted 7 lines

fs/fsevent.c line 263

Quoted 4 lines
fs/fsevent.c new file +2 -0
+
+	return (netlink_unicast(fsevent_sock, skb, pid, MSG_DONTWAIT));

netlink_unicast() uses boolean value but ont MSG_* flags for nonblocking, so this should be netlink_unicast(fsevent_sock, skb, pid, 0);

a example invocation in file net/netlink/af_netlink.c:
netlink_unicast(in_skb->sk, skb, NETLINK_CB(in_skb).pid, MSG_DONTWAIT); so, it hasn't any problem.

Well...

static inline long sock_sndtimeo(const struct sock *sk, int noblock)

{
	return noblock ? 0 : sk->sk_sndtimeo;
}

int netlink_unicast(struct sock *ssk, struct sk_buff *skb, u32 pid, int nonblock)

{
	struct sock *sk;
	int err;
	long timeo;

	skb = netlink_trim(skb, gfp_any());

	timeo = sock_sndtimeo(ssk, nonblock);

I mean that it is boolean value, MSG_PEEK will produce the same result. But it is a matter of coding style probably.

Quoted 33 lines
Quoted 33 lines
Quoted 6 lines
fs/fsevent.c new file +4 -0
+nlmsg_failure:
+	kfree_skb(skb);
+	return -1;
+}

...

Quoted 8 lines
fs/fsevent.c new file +6 -0
+static void fsevent_recv(struct sock *sk, int len)
+{
+	struct sk_buff *skb = NULL;
+	struct nlmsghdr *nlhdr = NULL;
+	struct fsevent_filter * filter = NULL;
+	pid_t pid;

fs/fsevent.c line 446

Quoted 10 lines
fs/fsevent.c new file +10 -0
+
+	while ((skb = skb_dequeue(&sk->sk_receive_queue)) != NULL) {
+		skb_get(skb);
+		if (skb->len >= FSEVENT_FILTER_MSGSIZE) {
+			nlhdr = (struct nlmsghdr *)skb->data;
+			filter = NLMSG_DATA(nlhdr);
+			pid = NETLINK_CREDS(skb)->pid;
+			if (find_fsevent_listener(pid) == NULL)
+				atomic_inc(&fsevent_listener_num);
+			set_fsevent_filter(filter, pid);

What is the logic behind this steps?
If there are no listeners you increment it's number no matter if it will or not be added in set_fsevent_filter().

Quoted 7 lines
Quoted 4 lines

fsevent_recv is used to receive listener's commands, a listener must send commands in order to get fsevents it interests, so this is the best point to increment number of listeners. set_fsevent_filter will add listener to listener list, so it is OK.

And what if set_fsevent_filter() fails?

I didn't consider this case, thanks, I will do with it.

Quoted 111 lines
Quoted 103 lines
Quoted 93 lines
Quoted 21 lines
fs/fsevent.c new file +18 -0
+		}
+		kfree_skb(skb);
+	}
+}
+
+#define DEFINE_FILTER_MATCH_FUNC(filtertype, key) 			\
+	static int match_##filtertype(listener * p,			\
+				struct fsevent * event,			\
+				struct sk_buff * skb)			\
+	{								\
+		int ret = 0;						\
+		filtertype * xfilter = NULL;				\
+		struct sk_buff * skb2 = NULL;				\
+		struct list_head *  head = &(p->key##_filter_list_head);  \
+		list_for_each_entry(xfilter, head, list) {		\
+			if (xfilter->key != event->key)			\
+				continue;				\
+			ret = filter_fsevent(xfilter->mask, event->type); \

fs/fsevent.c line 474

Quoted 10 lines
fs/fsevent.c new file +10 -0
+			if ( ret != 0)					\
+				return -1;				\
+			skb2 = skb_clone(skb, GFP_KERNEL);		\
+       			if (skb2 == NULL)			 \
+				return -ENOMEM;				\
+			NETLINK_CB(skb2).dst_group = 0;			\
+			NETLINK_CB(skb2).dst_pid = p->pid;		\
+			NETLINK_CB(skb2).pid = 0;			\
+			return (netlink_unicast(fsevent_sock, skb2,	\
+					p->pid, MSG_DONTWAIT));		\

The same issue about nonblocking sending.

Quoted 58 lines
fs/fsevent.c new file +58 -0
+		}							\
+		return -ENODEV;						\
+	}								\
+
+DEFINE_FILTER_MATCH_FUNC(pid_filter, pid)
+
+DEFINE_FILTER_MATCH_FUNC(uid_filter, uid)
+
+DEFINE_FILTER_MATCH_FUNC(gid_filter, gid)
+
+#define MATCH_XID(key, listenerp, event, skb) 			\
+	ret = match_##key##_filter(listenerp, event, skb); 	\
+	if (ret == 0) {					 	\
+		kfree_skb(skb);				 	\
+	        continue;				 	\
+	}						 	\
+	do {} while (0)					 	\
+
+static int fsevent_send_to_process(struct sk_buff * skb)
+{
+	listener * p  = NULL, * q = NULL;
+	struct fsevent * event = NULL;
+	struct sk_buff * skb2 = NULL;
+	int ret = 0;
+
+	event = (struct fsevent *)(skb->data + sizeof(struct nlmsghdr));
+	spin_lock(&listener_list_lock);
+	list_for_each_entry_safe(p, q, &listener_list_head, list) {
+		MATCH_XID(pid, p, event, skb);
+		MATCH_XID(uid, p, event, skb);
+		MATCH_XID(gid, p, event, skb);
+
+		if (filter_fsevent(p->mask, event->type) == 0) {
+			 skb2 = skb_clone(skb, GFP_KERNEL);
+	                 if (skb2 == NULL)
+	                 	return -ENOMEM;
+	                 NETLINK_CB(skb2).dst_group = 0;
+	                 NETLINK_CB(skb2).dst_pid = p->pid;
+	                 NETLINK_CB(skb2).pid = 0;
+	                 ret = netlink_unicast(fsevent_sock, skb2,
+	                                p->pid, MSG_DONTWAIT);
+			if (ret == -ECONNREFUSED) {
+				atomic_dec(&fsevent_listener_num);
+				cleanup_dead_listener(p);
+			}
+		}
+	}
+	spin_unlock(&listener_list_lock);
+	return ret;
+}
+
+static void fsevent_commit(void * unused)
+{
+	struct sk_buff * skb = NULL;
+		
+	while((skb = skb_dequeue(&get_cpu_var(fsevent_send_queue)))
+		!= NULL) {
+		fsevent_send_to_process(skb);
Quoted 10 lines

fs/fsevent.c line 159

Quoted 5 lines
fs/fsevent.c new file +3 -0
+		put_cpu_var(fsevent_send_queue);
+	}
+}

Really strange mix of per-cpu variables for optimized performance and global spin locking.
Consider using RCU for list of listeners.

Quoted 7 lines

per cpu queue is used to avoid raise_fsevent to contend spinlock, but listener_list_lock just is used to synchronize the operations of userspace applications(listener) on listener list, it just protect listener list.

Of course, your advice is good, RCU will be better, I'm considering substitute spinlock with RCU, maybe list*_rcu functions can help me.

Quoted 8 lines

You get global lock in each processor when traverse the list &listener_list_lock.

And you call GFP_KERNEL allocation under that lock, which is wrong.

If all your code is called from process context (it looks so), you could mutexes.

Yes, mutex should be the best choice.

Quoted 13 lines
Quoted 5 lines
Quoted 5 lines

You use unicast delivery for netlink messages. According to my investigation [1], it's performance is better only when there is only one listener (or maybe two in some cases), but then it is
noticebly slower than broadcasting.

Quoted 7 lines
Quoted 3 lines

1. http://marc.theaimsgroup.com/?l=linux-netdev&m=114424884216006&w=2

Because fsevent has to deliver different events to different listeners, so I must use netlink_unicast, in fact, netlink_broadcast also must send skb to every member of the group, so in my opinion, they haven't big difference.

Quoted 9 lines

And what if there are several listeners for the same type of events?

Quoted 2 lines

Can you explain why there is such a big difference between netlink_unicast and netlink_broadcast?

Netlink broadcast clones skbs, while unicasting requires the whole new one.

No, I also use clone to send skb, so they should have the same overhead.

Quoted 13 lines
Quoted 8 lines
Quoted 6 lines

Btw, you need some rebalancing of the per-cpu queues, probably in keventd, since CPUs can go offline and your messages will stuck foreve there.

Does keventd not do it? if so, keventd should be modified.

How does keventd know about your own structures?
You have an per-cpu object, but your keventd function gets object from running cpu, not from any other cpus.

#7 lore raw
Quoted 8 lines · Yi Yang
Quoted 7 lines
Quoted 3 lines

Can you explain why there is such a big difference between netlink_unicast and netlink_broadcast?

Netlink broadcast clones skbs, while unicasting requires the whole new one.

No, I also use clone to send skb, so they should have the same overhead.

I missed that.
After rereading fsevent_send_to_process() I do not see how original skb is freed though.

Quoted 11 lines
Quoted 11 lines
Quoted 7 lines
Quoted 5 lines

Btw, you need some rebalancing of the per-cpu queues, probably in keventd, since CPUs can go offline and your messages will stuck foreve there.

Does keventd not do it? if so, keventd should be modified.

How does keventd know about your own structures?
You have an per-cpu object, but your keventd function gets object from running cpu, not from any other cpus.

Signature · 1 line
	Evgeniy Polyakov
author
#8 lore raw
Quoted 9 lines · Evgeniy Polyakov
Quoted 7 lines · Yi Yang
Quoted 7 lines
Quoted 3 lines

Can you explain why there is such a big difference between netlink_unicast and netlink_broadcast?

Netlink broadcast clones skbs, while unicasting requires the whole new one.

Quoted 7 lines
Quoted 2 lines

No, I also use clone to send skb, so they should have the same overhead.

I missed that.
After rereading fsevent_send_to_process() I do not see how original skb is freed though.

I'm considering how to free it, because cloned skbs share data with original skb, so this case is special, I try to clarify the logic of kfree_skb.

Quoted 17 lines
Quoted 15 lines
Quoted 14 lines
Quoted 10 lines
Quoted 7 lines

Btw, you need some rebalancing of the per-cpu queues, probably in keventd, since CPUs can go offline and your messages will stuck foreve there.

Does keventd not do it? if so, keventd should be modified.

How does keventd know about your own structures?
You have an per-cpu object, but your keventd function gets object

from running cpu, not from any other cpus.

#9 lore raw
Quoted 15 lines · Yi Yang

Evgeniy Polyakov ??????:

Quoted 14 lines
Quoted 10 lines · Yi Yang
Quoted 8 lines
Quoted 4 lines

Can you explain why there is such a big difference between netlink_unicast and netlink_broadcast?

Netlink broadcast clones skbs, while unicasting requires the whole new one.

No, I also use clone to send skb, so they should have the same overhead.

Quoted 7 lines
Quoted 4 lines

I missed that.
After rereading fsevent_send_to_process() I do not see how original skb is freed though.

I'm considering how to free it, because cloned skbs share data with original skb, so this case is special,
I try to clarify the logic of kfree_skb.

Just call kfree_skb() after fsevent_send_to_process() or at the very end of this function. If unicast delivering fails you also need to free cloned skb.

Quoted 17 lines
Quoted 17 lines
Quoted 15 lines
Quoted 14 lines
Quoted 10 lines
Quoted 7 lines

Btw, you need some rebalancing of the per-cpu queues, probably in keventd, since CPUs can go offline and your messages will stuck foreve there.

Does keventd not do it? if so, keventd should be modified.

How does keventd know about your own structures?
You have an per-cpu object, but your keventd function gets object

from running cpu, not from any other cpus.

Signature · 1 line
	Evgeniy Polyakov
#10 lore raw
Quoted 20 lines · Evgeniy Polyakov
Quoted 19 lines · Yi Yang

Evgeniy Polyakov ??????:

Quoted 18 lines
Quoted 10 lines · Yi Yang
Quoted 8 lines
Quoted 4 lines

Can you explain why there is such a big difference between netlink_unicast and netlink_broadcast?

Netlink broadcast clones skbs, while unicasting requires the whole new one.

No, I also use clone to send skb, so they should have the same overhead.

I missed that.
After rereading fsevent_send_to_process() I do not see how original skb is freed though.

Quoted 6 lines
Quoted 3 lines

I'm considering how to free it, because cloned skbs share data with original skb, so this case is special,
I try to clarify the logic of kfree_skb.

Just call kfree_skb() after fsevent_send_to_process() or at the very end of this function. If unicast delivering fails you also need to free cloned skb.

For clarification: I mean if any error happens before netlink_unicast() call, you need to free skb. If netlink_unicast() is called, it will take care of skb in any case.

Quoted 20 lines
Quoted 17 lines
Quoted 17 lines
Quoted 15 lines
Quoted 14 lines
Quoted 10 lines
Quoted 7 lines

Btw, you need some rebalancing of the per-cpu queues, probably in keventd, since CPUs can go offline and your messages will stuck foreve there.

Does keventd not do it? if so, keventd should be modified.

How does keventd know about your own structures?
You have an per-cpu object, but your keventd function gets object

from running cpu, not from any other cpus.

Signature · 1 line
	Evgeniy Polyakov
Signature · 1 line
	Evgeniy Polyakov