: Filesystem Event Reporter V4
Subsystems (3)
PATCH 1/1 : Filesystem Event Reporter V4
I always have no time to update this patch in the past several months,
eventually I got some time to do it, compared to v2, new changes is:
- Use reference counter to synchronize rmmod, also consider process migration from a CPU to another during of calling to raise_fsevent*
- Free all the skbs queued on fsevent_sock before releasing fsevent_sock
- Rework some inline functions in include/linux/fsevent.h
This patch implements a new feature -- Filesystem Event Reporter, the user
can monitor filesystem activities via it, currently, it can monitor access,
write, utime, chmod, chown, chgrp, close, open, create, rename, unlink, mkdir,
rmdir, mount, umount.
Every filesystem event includes tgid, uid, gid and name of the process which triggered this event, the operated file or directory name. Any application with
root privelege can set its fsevent filter list in order to just get those events
who concerns but won't take effects in other applications to get any fsevents they are interested in, a fsevent filter may be based on event mask, pid, uid
and gid. There are three kinds of filter lists, they are pid filter list, uid filetr list and gid filter list, respectively, moreover, there is a fsevent mask used to
control those fsevents which fail to match three filter lists, an application
using fsevent can listen those fsevents it want to monitor and ignore those fsevents it hasn't interest in by set series of filters, there is a fsevent mask used to take effects on all the applications using fsevent, it can be set by sysctl and proc interface.
On every cpu, there is a fsevent queue, filesystem code path just puts the correspoding fsevent on current cpu, then it can continue to execute
the followed code without any lock, so there is a low overhead, a work per cpu is responsible for processing all the fsevents on this cpu and send them to listener applications.
Filesystem Events Reporter is never a duplicate of inotify, inotify
just concerns change on file or directory, Beagle uses it to watch file changes in order to regenerate index for it, inotify can't tell us who did that change and what is its process name, but filesystem events reporter can do these, moreover inotify's overhead is greater than filesystem events reporter, inotify needs compare inode with
watched file or directories list to decide whether it should generate an
inotify_event, some locks also increase overhead, filesystem event
connector hasn't these overhead, it just generates a fsevent and send.
To be important, filesystem event reporter doesn't add any new system call, the user space application can make use of it by netlink socket, but inotify added several system calls, many events mechanism in kernel
have used netlink as communication way with user space, for example,KOBJECT_UEVENT, PROC_EVENTS, to use netlink will make it more possible to unify events interface to netlink, the user space application can use it very easily.
fs/Kconfig | 10 fs/Makefile | 3 fs/fsevent.c | 653 +++++++++++++++++++++++++++++++++++++++++++++++ fs/fsevent_hook.c | 47 +++ fs/namespace.c | 12 include/linux/fsevent.h | 190 +++++++++++++ include/linux/fsnotify.h | 36 ++ include/linux/netlink.h | 1 8 files changed, 952 insertions(+)
fs/Kconfig renamed from fs/Kconfig.orig +10 -0
@@ -427,6 +427,16 @@ config INOTIFY_USER If unsure, say Y. +config FS_EVENTS + tristate "Report filesystem events to userspace" + ---help--- + Provide a facility that reports filesystem events to userspace. The + reported event include access, write, utime, chmod, chown, chgrp, + close, open, create, rename, unlink, mkdir, rmdir, mount, umount. + + The user can set filesystem events filter to filter its events, so + that he just get those events he concerns. + config QUOTA bool "Quota support" help
fs/Makefile renamed from fs/Makefile.orig +3 -0
@@ -14,6 +14,9 @@ obj-y := open.o read_write.o file_table. obj-$(CONFIG_INOTIFY) += inotify.o obj-$(CONFIG_INOTIFY_USER) += inotify_user.o +obj-$(CONFIG_FS_EVENTS) += fsevent.o +fseventbase-$(CONFIG_FS_EVENTS) := fsevent_hook.o +obj-y += $(fseventbase-y) $(fseventbase-m) obj-$(CONFIG_EPOLL) += eventpoll.o obj-$(CONFIG_COMPAT) += compat.o compat_ioctl.o
fs/fsevent.c new file +653 -0
653 lines not on this page · raw message
fs/fsevent_hook.c new file +47 -0
47 lines not on this page · raw message
fs/namespace.c renamed from fs/namespace.c.orig +12 -0
@@ -24,6 +24,7 @@#include <linux/mount.h> #include <asm/uaccess.h> #include <asm/unistd.h> +#include <linux/fsevent.h> #include "pnode.h" extern int __init init_rootfs (void );
@@ -656,6 +657,13 @@ asmlinkage long sys_umount(char __user * goto dput_and_out; retval =do_umount (nd.mnt, flags); + +if (retval ==0 ) { +char * tmp =getname (name); +raise_fsevent_umount (tmp); +putname (tmp); + } + dput_and_out:path_release_on_umount (&nd); out:
@@ -1577,6 +1585,10 @@ asmlinkage long sys_mount(char __user * retval =do_mount ((char *)dev_page, dir_page, (char *)type_page, flags, (void *)data_page);unlock_kernel (); + +if (retval ==0 ) +raise_fsevent_mount ((char *)dev_page, dir_page); +free_page (data_page); out3:
include/linux/fsevent.h new file +190 -0
190 lines not on this page · raw message
include/linux/fsnotify.h renamed from include/linux/fsnotify.h.orig +36 -0
@@ -16,6 +16,7 @@#include <linux/dnotify.h> #include <linux/inotify.h> #include <linux/audit.h> +#include <linux/fsevent.h> /* * fsnotify_d_instantiate - instantiate a dentry for inode
@@ -67,6 +68,8 @@ static inline void fsnotify_move(structif (source) {inotify_inode_queue_event (source,IN_MOVE_SELF ,0 ,NULL ,NULL ); } +raise_fsevent_move (old_dir, old_name, new_dir, new_name, +FSEVENT_MOVE | (isdir?FSEVENT_ISDIR :0 ));audit_inode_child (new_name, source, new_dir); }
@@ -79,6 +82,8 @@ static inline void fsnotify_nameremove(s isdir =IN_ISDIR ;dnotify_parent (dentry,DN_DELETE );inotify_dentry_parent_queue_event (dentry,IN_DELETE |isdir,0 , dentry->d_name.name); +raise_fsevent (dentry, +FSEVENT_DELETE | (isdir?FSEVENT_ISDIR :0 )); }/*
@@ -98,6 +103,7 @@ static inline void fsnotify_create(strucinode_dir_notify (inode,DN_CREATE );inotify_inode_queue_event (inode,IN_CREATE ,0 , dentry->d_name.name, dentry->d_inode); +raise_fsevent_create (inode, dentry->d_name.name,FSEVENT_CREATE );audit_inode_child (dentry->d_name.name, dentry->d_inode, inode); }
@@ -109,6 +115,8 @@ static inline void fsnotify_mkdir(structinode_dir_notify (inode,DN_CREATE );inotify_inode_queue_event (inode,IN_CREATE |IN_ISDIR ,0 , dentry->d_name.name, dentry->d_inode); +raise_fsevent_create (inode, dentry->d_name.name, +FSEVENT_CREATE |FSEVENT_ISDIR );audit_inode_child (dentry->d_name.name, dentry->d_inode, inode); }
@@ -126,6 +134,8 @@ static inline void fsnotify_access(strucdnotify_parent (dentry,DN_ACCESS );inotify_dentry_parent_queue_event (dentry, mask,0 , dentry->d_name.name);inotify_inode_queue_event (inode, mask,0 ,NULL ,NULL ); +raise_fsevent (dentry,FSEVENT_ACCESS | + ((S_ISDIR (inode->i_mode))?FSEVENT_ISDIR :0 )); }/*
@@ -142,6 +152,8 @@ static inline void fsnotify_modify(strucdnotify_parent (dentry,DN_MODIFY );inotify_dentry_parent_queue_event (dentry, mask,0 , dentry->d_name.name);inotify_inode_queue_event (inode, mask,0 ,NULL ,NULL ); +raise_fsevent (dentry,FSEVENT_MODIFY | + ((S_ISDIR (inode->i_mode))?FSEVENT_ISDIR :0 )); }/*
@@ -157,6 +169,8 @@ static inline void fsnotify_open(structinotify_dentry_parent_queue_event (dentry, mask,0 , dentry->d_name.name);inotify_inode_queue_event (inode, mask,0 ,NULL ,NULL ); +raise_fsevent (dentry,FSEVENT_OPEN | + ((S_ISDIR (inode->i_mode))?FSEVENT_ISDIR :0 )); }/*
@@ -175,6 +189,8 @@ static inline void fsnotify_close(structinotify_dentry_parent_queue_event (dentry, mask,0 , name);inotify_inode_queue_event (inode, mask,0 ,NULL ,NULL ); +raise_fsevent (dentry,FSEVENT_CLOSE | + ((S_ISDIR (inode->i_mode))?FSEVENT_ISDIR :0 )); }/*
@@ -190,6 +206,8 @@ static inline void fsnotify_xattr(structinotify_dentry_parent_queue_event (dentry, mask,0 , dentry->d_name.name);inotify_inode_queue_event (inode, mask,0 ,NULL ,NULL ); +raise_fsevent (dentry,FSEVENT_MODIFY_ATTRIB | + ((S_ISDIR (inode->i_mode))?FSEVENT_ISDIR :0 )); }/*
@@ -240,6 +258,24 @@ static inline void fsnotify_change(strucinotify_dentry_parent_queue_event (dentry, in_mask,0 , dentry->d_name.name); } + +#ifdef CONFIG_FS_EVENTS + { +u32 fsevent_mask =0 ; +if (ia_valid & (ATTR_UID |ATTR_GID |ATTR_MODE )) + fsevent_mask |=FSEVENT_MODIFY_ATTRIB ; +if ((ia_valid &ATTR_ATIME ) && (ia_valid &ATTR_MTIME )) + fsevent_mask |=FSEVENT_MODIFY_ATTRIB ; +else if (ia_valid &ATTR_ATIME ) + fsevent_mask |=FSEVENT_ACCESS ; +else if (ia_valid &ATTR_MTIME ) + fsevent_mask |=FSEVENT_MODIFY ; +if (ia_valid &ATTR_SIZE ) + fsevent_mask |=FSEVENT_MODIFY ; +if (fsevent_mask) +raise_fsevent (dentry, fsevent_mask); + } +#endif /* CONFIG_FS_EVENTS */ }#ifdef CONFIG_INOTIFY /* inotify helpers */
include/linux/netlink.h renamed from include/linux/netlink.h.orig +1 -0
@@ -21,6 +21,7 @@#define NETLINK_DNRTMSG 14/* DECnet routing messages */ #define NETLINK_KOBJECT_UEVENT 15/* Kernel messages to userspace */ #define NETLINK_GENERIC 16 +#define NETLINK_FSEVENT 17/* File system events to userspace */ #define MAX_LINKS 32
Hello.
Quoted 4 lines · Yi Yang
This patch implements a new feature -- Filesystem Event Reporter, the user
can monitor filesystem activities via it, currently, it can monitor access,
write, utime, chmod, chown, chgrp, close, open, create, rename, unlink, mkdir,
rmdir, mount, umount.
With each new version you add more features, but it seems that code is racy in some places. I've some comments after quite look.
Quoted 129 lines
fs/Kconfig | 10 fs/Makefile | 3 fs/fsevent.c | 653 +++++++++++++++++++++++++++++++++++++++++++++++ fs/fsevent_hook.c | 47 +++ fs/namespace.c | 12 include/linux/fsevent.h | 190 +++++++++++++ include/linux/fsnotify.h | 36 ++ include/linux/netlink.h | 1 8 files changed, 952 insertions(+)Signed-off-by Yi Yang <[email protected]>fs/Kconfig renamed from fs/Kconfig.orig +10
-0@@ -427,6 +427,16 @@ config INOTIFY_USER If unsure, say Y. +config FS_EVENTS + tristate "Report filesystem events to userspace" + ---help--- + Provide a facility that reports filesystem events to userspace. The + reported event include access, write, utime, chmod, chown, chgrp, + close, open, create, rename, unlink, mkdir, rmdir, mount, umount. + + The user can set filesystem events filter to filter its events, so + that he just get those events he concerns. + config QUOTA bool "Quota support" helpfs/Makefile renamed from fs/Makefile.orig +3
-0@@ -14,6 +14,9 @@ obj-y := open.o read_write.o file_table. obj-$(CONFIG_INOTIFY) += inotify.o obj-$(CONFIG_INOTIFY_USER) += inotify_user.o +obj-$(CONFIG_FS_EVENTS) += fsevent.o +fseventbase-$(CONFIG_FS_EVENTS) := fsevent_hook.o +obj-y += $(fseventbase-y) $(fseventbase-m) obj-$(CONFIG_EPOLL) += eventpoll.o obj-$(CONFIG_COMPAT) += compat.o compat_ioctl.ofs/fsevent.c new file +83
-0@@ -0,0 +1,653 @@ +/* +* fsevent.c +* +* 2006 Copyright (c) Yi Yang <[email protected]> +* All rights reserved. +* +* This program is free software; you can redistribute it and/or modify +* it under the terms of the GNU General Public License as published by +* the Free Software Foundation; either version 2 of the License, or +* (at your option) any later version. +* +* This program is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with this program; if not, write to the Free Software +* Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA +*/ + +#include <linux/kernel.h> +#include <linux/module.h> +#include <linux/skbuff.h> +#include <linux/netlink.h> +#include <linux/moduleparam.h> +#include <linux/fsevent.h> +#include <linux/skbuff.h> +#include <net/sock.h> +#include <linux/list.h> +#include <linux/percpu.h> +#include <linux/cpu.h> +#include <linux/kthread.h> +#include <linux/notifier.h> +#include <linux/compiler.h> +#include <linux/workqueue.h> +#include <linux/sysctl.h> +#include <linux/mutex.h> + +#define FSEVENT_MASK_CTL_NAME -2 + +static DEFINE_PER_CPU(struct sk_buff_head, fsevent_send_queue); +static DEFINE_PER_CPU(struct work_struct, fsevent_work); + +typedef struct pid_filter { +pid_t pid; +u32 mask; +struct list_head list; +}pid_filter ; + +typedef struct uid_filter { +uid_t uid; +u32 mask; +struct list_head list; +}uid_filter ; + +typedef struct gid_filter { +gid_t gid; +u32 mask; +struct list_head list; +}gid_filter ; + +typedef struct fsevent_listener { +pid_t pid; +struct list_head pid_filter_list_head; +struct list_head uid_filter_list_head; +struct list_head gid_filter_list_head; +u32 mask; +struct list_head list; +}listener ; + + +/* The netlink socket. */ +static struct sock * fsevent_sock =NULL ; +static LIST_HEAD (listener_list_head); +static DEFINE_MUTEX (listener_list_mutex); + +static atomic_t fsevent_count =ATOMIC_INIT (0 ); +static int fsevent_burst_limit =100000 ; +static unsigned long fsevent_ratelimit =5 *HZ ; +static unsigned long last =0 ; +static int fsevent_sum =0 ; +static u32 fsevents_mask =FSEVENT_MASK ;
fs/fsevent.c line 93
Quoted 10 lines
fs/fsevent.c new file +10
-0+static atomic_t fsevent_listener_num =ATOMIC_INIT (0 ); +static int exit_flag =0 ; +static struct ctl_table_header * fsevent_mask_ctlhdr =NULL ; + +extern int *get_fsevent_refcnt (void ); +extern void put_fsevent_refcnt (void ); +extern int per_cpu_fsevent_refcnt (int cpu); +extern void init_fsevent_refcnt (int cpu); +extern void init_missed_fsevent_refcnt (int cpu); +extern atomic_t *per_cpu_missed_refcnt (int cpu);
This protects against nothing, since there are now correct reference counters.
fs/fsevent.c line 99
Quoted 5 lines
fs/fsevent.c new file +5
-0+static inline void get_seq (__u32 *ts,int *cpu) +{ + *ts =atomic_inc_return (&fsevent_count); + *cpu =smp_processor_id (); +}
Only correct if you call it with disabled preemption.
Quoted 28 lines
fs/fsevent.c new file +28
-0+ +static void append_string (char **dest,const char *src,size_t len) +{ +strncpy (*dest, src, len); + (*dest)[len] ='\0' ; + *dest += len +1 ; +} + +static inline int filter_fsevent (u32 filter_mask,u32 event_mask) +{ + event_mask &=FSEVENT_MASK ; + event_mask &= filter_mask; +if (event_mask ==0 ) { +return -1 ; + } +return 0 ; +} + +static int filter_fsevent_all (u32 * mask) +{ +int ret =0 ; + + (*mask) &=FSEVENT_MASK ; + +if ((((*mask) &FSEVENT_ISDIR ) ==FSEVENT_ISDIR ) + && ((fsevents_mask &FSEVENT_ISDIR ) ==0 )) { + ret =-1 ; + goto out;
fs/fsevent.c line 137
Quoted 10 lines
fs/fsevent.c new file +10
-0+ } + + (*mask) &= fsevents_mask; +if ((*mask) ==0 ) { + ret =-1 ; + } + +out: +return ret; +}
This function is called in the context which allows to change @fsevents_mask, so it is racy.
Quoted 38 lines
fs/fsevent.c new file +38
-0+static void fsevent_send (struct sk_buff * skb) +{ +struct sk_buff_head * head = &get_cpu_var (fsevent_send_queue); +skb_queue_tail (head, skb); +schedule_work (&per_cpu (fsevent_work,smp_processor_id ())); +put_cpu_var (fsevent_send_queue); +} + +int ___raise_fsevent (const char * oldname,const char * newname,u32 mask) +{ +struct fsevent *event; +int namelen =0 ; +char * nameptr =NULL ; +unsigned int size; +struct nlmsghdr * nlhdr; +struct sk_buff * skb =NULL ; + +if (filter_fsevent_all (&mask) !=0 ) +return -1 ; + +if (atomic_read (&fsevent_listener_num) <=0 ) +return -1 ; + +if (time_after (jiffies, last)) { + last = jiffies + fsevent_ratelimit; + fsevent_sum =0 ; + } +else { +if (fsevent_sum > fsevent_burst_limit) +return -1 ; + fsevent_sum++; + } + + namelen =strlen (current->comm) +strlen (oldname) +2 ; +if (newname) + namelen +=strlen (newname) +1 ; + + size =NLMSG_SPACE (sizeof (struct fsevent ) + namelen);
fs/fsevent.c line 186
Quoted 10 lines
fs/fsevent.c new file +10
-0+ + skb =alloc_skb (size,GFP_KERNEL ); +if (!skb) +return -1 ; + + nlhdr =NLMSG_PUT (skb,0 ,0 ,NLMSG_DONE , size -sizeof (*nlhdr)); + event =NLMSG_DATA (nlhdr); + +get_seq (&(nlhdr->nlmsg_seq), &event->cpu); +ktime_get_ts (&event->timestamp);
This timestamp has different size on 32 and 64 bit platforms, so you can not use it for example on x86_64.
Quoted 8 lines
fs/fsevent.c new file +8
-0+ event->type = mask; + event->pid = current->tgid; + event->uid = current->uid; + event->gid = current->gid; + nameptr = event->name; + event->pname_len =strlen (current->comm); +append_string (&nameptr, current->comm, event->pname_len); + event->fname_len =strlen (oldname);
fs/fsevent.c line 204
Quoted 10 lines
fs/fsevent.c new file +10
-0+append_string (&nameptr, oldname, event->fname_len); + event->len = event->pname_len + event->fname_len +2 ; + event->new_fname_len =0 ; +if (newname) { + event->new_fname_len =strlen (newname); +append_string (&nameptr, newname, event->new_fname_len); + event->len += event->new_fname_len +1 ; + } +fsevent_send (skb); +return 0 ;
A lot of strlen() slows code down noticebly.
Quoted 28 lines
fs/fsevent.c new file +28
-0+nlmsg_failure: +kfree_skb (skb); +return -1 ; +} + +static int fsevent_ack (enum fsevent_type type,pid_t pid,u32 seq) +{ +struct fsevent *event; +unsigned int size; +struct sk_buff * skb =NULL ; +struct nlmsghdr * nlhdr =NULL ; + + size =NLMSG_SPACE (sizeof (struct fsevent )); + + skb =alloc_skb (size,GFP_KERNEL ); +if (!skb) +return -ENOMEM ; + + nlhdr =NLMSG_PUT (skb,0 , seq,NLMSG_DONE , size -sizeof (*nlhdr)); + event =NLMSG_DATA (nlhdr); + +ktime_get_ts (&event->timestamp); + event->cpu =-1 ; + event->type = type; + event->pid =0 ; + event->uid =0 ; + event->gid =0 ; + event->len =0 ;
fs/fsevent.c line 243
Quoted 10 lines
fs/fsevent.c new file +10
-0+ event->pname_len =0 ; + event->fname_len =0 ; + event->new_fname_len =0 ; + event->err =0 ; + +NETLINK_CB (skb).dst_group =0 ; +NETLINK_CB (skb).dst_pid = pid; +NETLINK_CB (skb).pid =0 ; + +return (netlink_unicast (fsevent_sock, skb, pid,MSG_DONTWAIT ));
You can lose acks. If you do care about them you need some kind of ack sequence number.
Quoted 186 lines
fs/fsevent.c new file +186
-0+nlmsg_failure: +kfree_skb (skb); +return -1 ; +} + +static void set_fsevent_mask (u32 * to_mask,u32 from_mask,int mode) +{ +if (mode ==FSEVENT_FILTER_IGNORE ) + (*to_mask) &= ~(from_mask); +else if (mode ==FSEVENT_FILTER_LISTEN ) + (*to_mask) |= from_mask; +} + +#define DEFINE_FILTER_FIND_FUNC (type, key) \ + type * find_##type(struct list_head * head, key##_t id) \ + { \ + int alloc_flag = 1; \ + type * entry = NULL; \ + \ + list_for_each_entry(entry, head, list) { \ + if (entry->key == id) { \ + alloc_flag = 0; \ + break; \ + } \ + } \ + \ + if (alloc_flag == 1) { \ + entry = (type *)kmalloc(sizeof(type), GFP_KERNEL); \ + if (entry == NULL) \ + return NULL; \ + memset(entry, 0, sizeof(type)); \ + entry->key = id; \ + list_add_tail(&(entry->list), head); \ + } \ + return entry; \ + } \ + +DEFINE_FILTER_FIND_FUNC(pid_filter, pid) + +DEFINE_FILTER_FIND_FUNC(uid_filter, uid) + +DEFINE_FILTER_FIND_FUNC(gid_filter, gid) + +DEFINE_FILTER_FIND_FUNC(listener, pid) + +static int set_fsevent_filter (struct fsevent_filter * filter,pid_t pid) +{ +enum fsevent_type type; +u32 mask =0 ; +int control =0 ; +listener * listenerp =NULL ; +pid_filter * pfilter =NULL ; +uid_filter * ufilter =NULL ; +gid_filter * gfilter =NULL ; +int ret =0 ; + + mask = filter->mask; + control = filter->control; + type = filter->type; + mask &=FSEVENT_MASK ; +if (mask ==0 ) { + ret =-1 ; + goto out; + } + +mutex_lock (&listener_list_mutex); + listenerp =find_listener (&listener_list_head, pid); +if (unlikely (listenerp ==NULL )) { +mutex_unlock (&listener_list_mutex); +return -1 ; + } + +if (!(listenerp->pid_filter_list_head.next)) { +INIT_LIST_HEAD (&(listenerp->pid_filter_list_head)); +INIT_LIST_HEAD (&(listenerp->uid_filter_list_head)); +INIT_LIST_HEAD (&(listenerp->gid_filter_list_head)); + } + +if ((type &FSEVENT_FILTER_ALL ) ==FSEVENT_FILTER_ALL ) { +if (control ==FSEVENT_FILTER_REMOVE ) { +atomic_dec (&fsevent_listener_num); +list_del (&(listenerp->list)); +kfree (listenerp); + }else +set_fsevent_mask (&(listenerp->mask), mask, control); + }else if ((type &FSEVENT_FILTER_PID ) ==FSEVENT_FILTER_PID ) { + pfilter =find_pid_filter (&(listenerp->pid_filter_list_head), + filter->id.pid); +if (unlikely (pfilter ==NULL )) +return -1 ; + +if (control ==FSEVENT_FILTER_REMOVE ) { +list_del (&(pfilter->list)); +kfree (pfilter); + }else +set_fsevent_mask (&(pfilter->mask), mask, control); + }else if ((type &FSEVENT_FILTER_UID ) ==FSEVENT_FILTER_UID ) { + ufilter =find_uid_filter (&(listenerp->uid_filter_list_head), + filter->id.uid); +if (unlikely (ufilter ==NULL )) +return -1 ; + +if (control ==FSEVENT_FILTER_REMOVE ) { +list_del (&(ufilter->list)); +kfree (ufilter); + }else +set_fsevent_mask (&(ufilter->mask), mask, control); + }else if ((type &FSEVENT_FILTER_GID ) ==FSEVENT_FILTER_GID ) { + gfilter =find_gid_filter (&(listenerp->gid_filter_list_head), + filter->id.gid); +if (unlikely (gfilter ==NULL )) +return -1 ; + +if (control ==FSEVENT_FILTER_REMOVE ) { +list_del (&(gfilter->list)); +kfree (gfilter); + }else +set_fsevent_mask (&(gfilter->mask), mask, control); + } +mutex_unlock (&listener_list_mutex); + ret =0 ; + +out: +fsevent_ack (type, pid,0 ); +return (ret); +} + +static listener *find_fsevent_listener (pid_t pid) +{ +listener * listenerp =NULL ; +mutex_lock (&listener_list_mutex); +list_for_each_entry (listenerp, &listener_list_head, list) { +if (listenerp->pid == pid) { +mutex_unlock (&listener_list_mutex); +return listenerp; + } + } +mutex_unlock (&listener_list_mutex); +return NULL ; +} + +static void cleanup_dead_listener (listener * x) +{ +pid_filter * p =NULL , * pq =NULL ; +uid_filter * u =NULL , * uq =NULL ; +gid_filter * g =NULL , * gq =NULL ; + +if (x ==NULL ) +return ; + +list_del (&(x->list)); + +list_for_each_entry_safe (p, pq, &(x->pid_filter_list_head), list) { +list_del (&(p->list)); +kfree (p); + } + +list_for_each_entry_safe (u, uq, &(x->uid_filter_list_head), list) { +list_del (&(u->list)); +kfree (u); + } + +list_for_each_entry_safe (g, gq, &(x->gid_filter_list_head), list) { +list_del (&(g->list)); +kfree (g); + } + +kfree (x); +} + +static void fsevent_recv (struct sock *sk,int len) +{ +struct sk_buff *skb =NULL ; +struct nlmsghdr *nlhdr =NULL ; +struct fsevent_filter * filter =NULL ; +pid_t pid; +int inc_flag =0 ; + +if (exit_flag ==1 ) +return ; + +while ((skb =skb_dequeue (&sk->sk_receive_queue)) !=NULL ) { +skb_get (skb); +if (skb->len >=FSEVENT_FILTER_MSGSIZE ) { + nlhdr = (struct nlmsghdr *)skb->data; + filter =NLMSG_DATA (nlhdr);
fs/fsevent.c line 440
Quoted 10 lines
fs/fsevent.c new file +10
-0+ pid =NETLINK_CREDS (skb)->pid; +if (find_fsevent_listener (pid) ==NULL ) + inc_flag =1 ; +if (set_fsevent_filter (filter, pid) ==0 ) { +if (inc_flag ==1 ) +atomic_inc (&fsevent_listener_num); + } + + } +kfree_skb (skb);
You leak skb here, no need to grab it's reference counter above.
Quoted 16 lines
fs/fsevent.c new file +16
-0+ } +} + +#define DEFINE_FILTER_MATCH_FUNC (filtertype, key) \ + static int match_##filtertype(listener * p, \ + struct fsevent * event, \ + struct sk_buff * skb) \ + { \ + int ret = 0; \ + filtertype * xfilter = NULL; \ + struct sk_buff * skb2 = NULL; \ + struct list_head * head = &(p->key##_filter_list_head); \ + list_for_each_entry(xfilter, head, list) { \ + if (xfilter->key != event->key) \ + continue; \ + ret = filter_fsevent(xfilter->mask, event->type); \
fs/fsevent.c line 466
Quoted 10 lines
fs/fsevent.c new file +10
-0+if ( ret !=0 ) \ +return -1 ; \ + skb2 =skb_clone (skb,GFP_KERNEL ); \ +if (skb2 ==NULL ) \ +return -1 ; \ +NETLINK_CB (skb2).dst_group =0 ; \ +NETLINK_CB (skb2).dst_pid = p->pid; \ +NETLINK_CB (skb2).pid =0 ; \ +return (netlink_unicast (fsevent_sock, skb2, \ + p->pid,MSG_DONTWAIT )); \
You reinvent netlink_broadcast() in some way...
Quoted 26 lines
fs/fsevent.c new file +26
-0+ } \ +return -1 ; \ + } \ + +DEFINE_FILTER_MATCH_FUNC(pid_filter, pid) + +DEFINE_FILTER_MATCH_FUNC(uid_filter, uid) + +DEFINE_FILTER_MATCH_FUNC(gid_filter, gid) + +#define MATCH_XID (key, listenerp, event, skb) \ + ret = match_##key##_filter(listenerp, event, skb); \ + if (ret == 0) { \ + kfree_skb(skb); \ + continue; \ + } \ + do {} while (0) \ + +static int fsevent_send_to_process (struct sk_buff * skb) +{ +listener * p =NULL , * q =NULL ; +struct fsevent * event =NULL ; +struct sk_buff * skb2 =NULL ; +int ret =0 ; + + event = (struct fsevent *)(skb->data +sizeof (struct nlmsghdr ));
fs/fsevent.c line 502
Quoted 10 lines
fs/fsevent.c new file +10
-0+mutex_lock (&listener_list_mutex); +list_for_each_entry_safe (p, q, &listener_list_head, list) { +MATCH_XID (pid, p, event, skb); +MATCH_XID (uid, p, event, skb); +MATCH_XID (gid, p, event, skb); + +if (filter_fsevent (p->mask, event->type) ==0 ) { + skb2 =skb_clone (skb,GFP_KERNEL ); +if (skb2 ==NULL ) +return -1 ;
Mutex is not unlocked.
fs/fsevent.c line 507
Quoted 5 lines
fs/fsevent.c new file +5
-0+NETLINK_CB (skb2).dst_group =0 ; +NETLINK_CB (skb2).dst_pid = p->pid; +NETLINK_CB (skb2).pid =0 ; + ret =netlink_unicast (fsevent_sock, skb2, + p->pid,0 );
Btw, pid in netlink messages should not be considered as 'process id' of the sending process. Processes can be changed, but pid will be the same.
Quoted 16 lines
fs/fsevent.c new file +16
-0+if (ret == -ECONNREFUSED ) { +atomic_dec (&fsevent_listener_num); +cleanup_dead_listener (p); + } + } + } +mutex_unlock (&listener_list_mutex); +return ret; +} + +static void fsevent_commit (void * unused) +{ +struct sk_buff * skb =NULL ; +int * refcnt, missed_refcnt; +int cpuid; +
fs/fsevent.c line 533
Quoted 10 lines
fs/fsevent.c new file +10
-0+ +while ((skb =skb_dequeue (&get_cpu_var (fsevent_send_queue))) + !=NULL ) { +fsevent_send_to_process (skb); +kfree_skb (skb); +put_cpu_var (fsevent_send_queue); + } + +if (exit_flag ==1 ) { + refcnt =get_fsevent_refcnt ();
What prevents from adding another skb into the queue between above loop and check for flag?
Quoted 8 lines
fs/fsevent.c new file +8
-0+if (*refcnt <=0 ) { + *refcnt =-1 ; +put_fsevent_refcnt (); +return ; + } + cpuid =smp_processor_id (); + + missed_refcnt =atomic_read (per_cpu_missed_refcnt (cpuid));
fs/fsevent.c line 551
Quoted 10 lines
fs/fsevent.c new file +10
-0+if (missed_refcnt == *refcnt) + { + *refcnt =-1 ; +printk ("cpu#%d: missed refcnt = %d\n" , cpuid, missed_refcnt); + } +else { +if (missed_refcnt !=0 ) { +printk ("cpu#%d: refcnt = %d, missed refcnt = %d\n" , cpuid, *refcnt, missed_refcnt); + } + }
Above operation seems racy, what prevents from changing missed_refcnt after it was read?
Quoted 58 lines
fs/fsevent.c new file +58
-0+put_fsevent_refcnt (); + } +} + +static struct ctl_table fsevent_mask_sysctl[] = { + { + .ctl_name =FSEVENT_MASK_CTL_NAME , + .procname ="fsevent_mask" , + .data = &fsevents_mask, + .maxlen =sizeof (u32), + .mode =0644 , + .proc_handler = &proc_dointvec, + }, + { .ctl_name =0 } +}; + +static struct ctl_table fs_root_sysctl[] = { + { + .ctl_name =CTL_FS , + .procname ="fs" , + .mode =0555 , + .child = fsevent_mask_sysctl, + }, + { .ctl_name =0 } +}; + +static int __init fsevent_init (void ) +{ +int cpu; +struct sk_buff_head * listptr; +struct work_struct * workptr; + + fsevent_sock =netlink_kernel_create (NETLINK_FSEVENT ,0 , + fsevent_recv,THIS_MODULE ); +if (!fsevent_sock) +return -EIO ; +for_each_possible_cpu (cpu) { +init_fsevent_refcnt (cpu); +init_missed_fsevent_refcnt (cpu); + listptr = &per_cpu (fsevent_send_queue, cpu); +skb_queue_head_init (listptr); + workptr = &per_cpu (fsevent_work, cpu); +INIT_WORK (workptr, fsevent_commit,NULL ); + } + + fsevent_mask_ctlhdr =register_sysctl_table (fs_root_sysctl,0 ); +if (fsevent_mask_ctlhdr ==NULL ) +return -ENOMEM ; + + __raise_fsevent = ___raise_fsevent; + +return 0 ; +} + +static void __exit fsevent_exit(void) +{ +listener * p =NULL , * q =NULL ; +int cpu;
fs/fsevent.c line 619
Quoted 10 lines
fs/fsevent.c new file +10
-0+int wait_flag =1 ; +struct sk_buff * skb =NULL ; + + fsevents_mask =0 ; + exit_flag =1 ; + +while (wait_flag ==1 ) { + wait_flag =0 ; +for_each_possible_cpu (cpu) { +if (per_cpu_fsevent_refcnt (cpu) >=0 ) {
This check is racy against above checks (marked as racy in comments).
Quoted 4 lines
fs/fsevent.c new file +4
-0+ wait_flag =1 ; +schedule_work (&per_cpu (fsevent_work, cpu)); + } + }
fs/fsevent.c line 633
Quoted 10 lines
fs/fsevent.c new file +10
-0+flush_scheduled_work (); + } + __raise_fsevent =0 ; + +while ((skb =skb_dequeue (&fsevent_sock->sk_receive_queue)) !=NULL ) { +kfree_skb (skb); + } +while ((skb =skb_dequeue (&fsevent_sock->sk_write_queue)) !=NULL ) { +kfree_skb (skb); + }
Why are you doing this? It looks wrong, since socket's queue is cleaned automatically.
fs/fsevent.c line 638
Quoted 5 lines
fs/fsevent.c new file +5
-0+printk ("sk_rmem_alloc=%d, sk_wmem_alloc=%d\n" , +atomic_read (&fsevent_sock->sk_rmem_alloc), +atomic_read (&fsevent_sock->sk_wmem_alloc)); +atomic_set (&fsevent_sock->sk_rmem_alloc,0 ); +atomic_set (&fsevent_sock->sk_wmem_alloc,0 );
100 % broken - you should not look into that variables, if kernel prints assertions about sizes, that means either leaked skbs or double freeings.
Quoted 41 lines
fs/fsevent.c new file +15
-0+sock_release (fsevent_sock->sk_socket); +mutex_lock (&listener_list_mutex); +list_for_each_entry_safe (p, q, &listener_list_head, list) { +cleanup_dead_listener (p); + } +mutex_unlock (&listener_list_mutex); +unregister_sysctl_table (fsevent_mask_ctlhdr); +} + +module_init (fsevent_init); +module_exit (fsevent_exit); + +MODULE_LICENSE ("GPL" ); +MODULE_AUTHOR ("Yi Yang <[email protected]>" ); +MODULE_DESCRIPTION ("File System Events Reporter" );fs/fsevent_hook.c new file +23
-0@@ -0,0 +1,47 @@ +#include <linux/fsevent.h> + +int (* __raise_fsevent) + (const char * oldname,const char * newname,u32 mask) =0 ; +EXPORT_SYMBOL (__raise_fsevent); + +static DEFINE_PER_CPU(int, raise_fsevent_refcnt); +static DEFINE_PER_CPU(atomic_t, missed_fsevent_refcnt); + +int *get_fsevent_refcnt (void ) +{ +return &get_cpu_var (raise_fsevent_refcnt); +} +EXPORT_SYMBOL (get_fsevent_refcnt); + +void put_fsevent_refcnt (void ) +{ +put_cpu_var (raise_fsevent_refcnt); +} +EXPORT_SYMBOL (put_fsevent_refcnt); + +int per_cpu_fsevent_refcnt (int cpu) +{
fs/fsevent_hook.c line 33
Quoted 10 lines
fs/fsevent_hook.c new file +10
-0+return per_cpu (raise_fsevent_refcnt, cpu); +} +EXPORT_SYMBOL (per_cpu_fsevent_refcnt); + +void init_fsevent_refcnt (int cpu) +{ +int * refcnt = &per_cpu (raise_fsevent_refcnt, cpu); + *refcnt =0 ; +} +EXPORT_SYMBOL (init_fsevent_refcnt);
This is racy.
fs/fsevent_hook.c line 46
Quoted 12 lines
fs/fsevent_hook.c new file +12
-0+void init_missed_fsevent_refcnt (int cpu) +{ +atomic_t * missed_refcnt = &per_cpu (missed_fsevent_refcnt, cpu); + +atomic_set (missed_refcnt,0 ); +} +EXPORT_SYMBOL (init_missed_fsevent_refcnt); + +atomic_t *per_cpu_missed_refcnt (int cpu) +{ +return &per_cpu (missed_fsevent_refcnt, cpu); +}
Racy.
Quoted 104 lines
fs/fsevent_hook.c new file +1
-0+EXPORT_SYMBOL (per_cpu_missed_refcnt);fs/namespace.c renamed from fs/namespace.c.orig +12
-0@@ -24,6 +24,7 @@#include <linux/mount.h> #include <asm/uaccess.h> #include <asm/unistd.h> +#include <linux/fsevent.h> #include "pnode.h" extern int __init init_rootfs (void );@@ -656,6 +657,13 @@ asmlinkage long sys_umount(char __user * goto dput_and_out; retval =do_umount (nd.mnt, flags); + +if (retval ==0 ) { +char * tmp =getname (name); +raise_fsevent_umount (tmp); +putname (tmp); + } + dput_and_out:path_release_on_umount (&nd); out:@@ -1577,6 +1585,10 @@ asmlinkage long sys_mount(char __user * retval =do_mount ((char *)dev_page, dir_page, (char *)type_page, flags, (void *)data_page);unlock_kernel (); + +if (retval ==0 ) +raise_fsevent_mount ((char *)dev_page, dir_page); +free_page (data_page); out3:include/linux/fsevent.h new file +65
-0@@ -0,0 +1,190 @@ +/* +* fsevent.h - filesystem events reporter +* +* Copyright (C) 2006 Yi Yang <[email protected]> +* Based on cn_proc.h by Matt Helsley, IBM Corp +* +* This program is free software; you can redistribute it and/or modify +* it under the terms of the GNU General Public License as published by +* the Free Software Foundation; either version 2 of the License, or +* (at your option) any later version. +* +* This program is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with this program; if not, write to the Free Software +* Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA +*/ + +#ifndef LINUX_FSEVENT_H +#define LINUX_FSEVENT_H + +#include <linux/types.h> +#include <linux/time.h> +#include <linux/netlink.h> + +enum fsevent_type { +FSEVENT_ACCESS =0x00000001 ,/* File was accessed */ +FSEVENT_MODIFY =0x00000002 ,/* File was modified */ +FSEVENT_MODIFY_ATTRIB =0x00000004 ,/* Metadata changed */ +FSEVENT_CLOSE =0x00000008 ,/* File was closed */ +FSEVENT_OPEN =0x00000010 ,/* File was opened */ +FSEVENT_MOVE =0x00000020 ,/* File was moved */ +FSEVENT_CREATE =0x00000040 ,/* File was created */ +FSEVENT_DELETE =0x00000080 ,/* File was deleted */ +FSEVENT_MOUNT =0x00000100 ,/* File system is mounted */ +FSEVENT_UMOUNT =0x00000200 ,/* File system is umounted */ + +/* The following definitions are command types for fsevent filter +* or acknowlege types of the corresponding commands +*/ +FSEVENT_FILTER_ALL =0x08000000 ,/* For all events */ +FSEVENT_FILTER_PID =0x10000000 ,/* For some process ID */ +FSEVENT_FILTER_UID =0x20000000 ,/* For some user ID */ +FSEVENT_FILTER_GID =0x40000000 ,/* For some group ID */ + +FSEVENT_ISDIR =0x80000000 /* It is set for a dir */ +}; + +#define FSEVENT_MASK 0x800003ff + +typedef unsigned long fsevent_mask_t ; + +enum filter_control { +FSEVENT_FILTER_LISTEN =1 ,/* Listen fsevents mask defines*/ +FSEVENT_FILTER_IGNORE ,/* Ignore fsevents mask defines*/ +FSEVENT_FILTER_REMOVE ,/* Remove a given filter */ +}; + +struct fsevent_filter { +/* filter type, it just is one of them +* FSEVENT_FILTER_ALL +* FSEVENT_FILTER_PID
include/linux/fsevent.h line 75
Quoted 10 lines
include/linux/fsevent.h new file +10
-0+* FSEVENT_FILTER_UID +* FSEVENT_FILTER_GID +*/ +enum fsevent_type type;/* filter type */ + +/* mask of file system events the user listen or ignore +* if the user need to ignore all the events of some pid +* , gid or uid, he(she) must set mask to FSEVENT_MASK. +*/ +fsevent_mask_t mask;
This is wrong, since it has different size on 64 and 32 platforms.
include/linux/fsevent.h line 88
Quoted 13 lines
include/linux/fsevent.h new file +13
-0+union { +pid_t pid; +uid_t uid; +gid_t gid; + } id; + +enum filter_control control; +}; + +struct fsevent { +__u32 type; +__u32 cpu; +struct timespec timestamp;
Wrong size.
Quoted 28 lines
include/linux/fsevent.h new file +28
-0+pid_t pid; +uid_t uid; +gid_t gid; +int err; +__u32 len; +__u32 pname_len; +__u32 fname_len; +__u32 new_fname_len; +char name[0 ]; +}; + +#define FSEVENT_FILTER_MSGSIZE \ + (sizeof(struct fsevent_filter) + sizeof(struct nlmsghdr)) + +#ifdef __KERNEL__ +#if defined(CONFIG_FS_EVENTS) || defined(CONFIG_FS_EVENTS_MODULE) +extern int (* __raise_fsevent) + (const char * oldname, const char * newname, u32 mask); + +extern int * get_fsevent_refcnt(void); +extern void put_fsevent_refcnt(void); +extern atomic_t * per_cpu_missed_refcnt(int cpu); + +static inline int _raise_fsevent + (const char * oldname, const char * newname, u32 mask) +{ + int ret; + int * refcnt = NULL;
include/linux/fsevent.h line 126
Quoted 10 lines
include/linux/fsevent.h new file +10
-0+int start_cpuid, end_cpuid; + + refcnt =get_fsevent_refcnt (); + start_cpuid =smp_processor_id (); +if ((*refcnt ==-1 ) || (__raise_fsevent ==0 )) { +put_fsevent_refcnt (); +return -1 ; + } + (*refcnt)++; +put_fsevent_refcnt ();
This looks really racy.
What prevents from rescheduling here?
include/linux/fsevent.h line 139
Quoted 13 lines
include/linux/fsevent.h new file +13
-0+ ret =__raise_fsevent (oldname, newname, mask); + refcnt =get_fsevent_refcnt (); + end_cpuid =smp_processor_id (); +if (start_cpuid != end_cpuid) { +printk ("fsevent: process '%s' migration: cpu#%d -> cpu#%d." , current->comm, start_cpuid, end_cpuid); +atomic_inc (per_cpu_missed_refcnt (start_cpuid)); +put_fsevent_refcnt (); +return -1 ; + } + (*refcnt)--; +put_fsevent_refcnt (); +return ret; +}
What prevents change for __raise_fsevent in that function?
Quoted 176 lines
include/linux/fsevent.h new file +50
-0+static inline void raise_fsevent (struct dentry * dentryp,u32 mask) +{ +if (dentryp->d_inode && (MAJOR (dentryp->d_inode->i_rdev) ==4 )) +return ; +_raise_fsevent (dentryp->d_name.name,NULL , mask); +} + +static inline void raise_fsevent_move (struct inode * olddir, +const char * oldname,struct inode * newdir, +const char * newname,u32 mask) +{ +_raise_fsevent (oldname, newname, mask); +} + +static inline void raise_fsevent_create (struct inode * inode, +const char * name,u32 mask) +{ +_raise_fsevent (name,NULL , mask); +} + +static inline void raise_fsevent_mount (const char * devname, +const char * mountpoint) +{ +_raise_fsevent (devname, mountpoint,FSEVENT_MOUNT ); +} + +static inline void raise_fsevent_umount (const char * mountpoint) +{ +_raise_fsevent (mountpoint,NULL ,FSEVENT_UMOUNT ); +} +#else +static void raise_fsevent (struct dentry * dentryp,u32 mask) +{} + +static void raise_fsevent_move (struct inode * olddir,const char * oldname, +struct inode * newdir,const char * newname,u32 mask) +{} + +static void raise_fsevent_create (struct inode * inode, +const char * name,u32 mask) +{} + +static void raise_fsevent_mount (const char * devname,const char * mountpoint) +{} + +static void raise_fsevent_umount (const char * mountpoint) +{} +#endif /* CONFIG_FS_EVENTS || CONFIG_FS_EVENTS_MODULE */ +#endif /* __KERNEL__ */ +#endif /* LINUX_FSEVENT_H */ include/linux/fsnotify.h renamed from include/linux/fsnotify.h.orig +36
-0@@ -16,6 +16,7 @@#include <linux/dnotify.h> #include <linux/inotify.h> #include <linux/audit.h> +#include <linux/fsevent.h> /* * fsnotify_d_instantiate - instantiate a dentry for inode @@ -67,6 +68,8 @@ static inline void fsnotify_move(structif (source) {inotify_inode_queue_event (source,IN_MOVE_SELF ,0 ,NULL ,NULL ); } +raise_fsevent_move (old_dir, old_name, new_dir, new_name, +FSEVENT_MOVE | (isdir?FSEVENT_ISDIR :0 ));audit_inode_child (new_name, source, new_dir); }@@ -79,6 +82,8 @@ static inline void fsnotify_nameremove(s isdir =IN_ISDIR ;dnotify_parent (dentry,DN_DELETE );inotify_dentry_parent_queue_event (dentry,IN_DELETE |isdir,0 , dentry->d_name.name); +raise_fsevent (dentry, +FSEVENT_DELETE | (isdir?FSEVENT_ISDIR :0 )); }/* @@ -98,6 +103,7 @@ static inline void fsnotify_create(strucinode_dir_notify (inode,DN_CREATE );inotify_inode_queue_event (inode,IN_CREATE ,0 , dentry->d_name.name, dentry->d_inode); +raise_fsevent_create (inode, dentry->d_name.name,FSEVENT_CREATE );audit_inode_child (dentry->d_name.name, dentry->d_inode, inode); }@@ -109,6 +115,8 @@ static inline void fsnotify_mkdir(structinode_dir_notify (inode,DN_CREATE );inotify_inode_queue_event (inode,IN_CREATE |IN_ISDIR ,0 , dentry->d_name.name, dentry->d_inode); +raise_fsevent_create (inode, dentry->d_name.name, +FSEVENT_CREATE |FSEVENT_ISDIR );audit_inode_child (dentry->d_name.name, dentry->d_inode, inode); }@@ -126,6 +134,8 @@ static inline void fsnotify_access(strucdnotify_parent (dentry,DN_ACCESS );inotify_dentry_parent_queue_event (dentry, mask,0 , dentry->d_name.name);inotify_inode_queue_event (inode, mask,0 ,NULL ,NULL ); +raise_fsevent (dentry,FSEVENT_ACCESS | + ((S_ISDIR (inode->i_mode))?FSEVENT_ISDIR :0 )); }/* @@ -142,6 +152,8 @@ static inline void fsnotify_modify(strucdnotify_parent (dentry,DN_MODIFY );inotify_dentry_parent_queue_event (dentry, mask,0 , dentry->d_name.name);inotify_inode_queue_event (inode, mask,0 ,NULL ,NULL ); +raise_fsevent (dentry,FSEVENT_MODIFY | + ((S_ISDIR (inode->i_mode))?FSEVENT_ISDIR :0 )); }/* @@ -157,6 +169,8 @@ static inline void fsnotify_open(structinotify_dentry_parent_queue_event (dentry, mask,0 , dentry->d_name.name);inotify_inode_queue_event (inode, mask,0 ,NULL ,NULL ); +raise_fsevent (dentry,FSEVENT_OPEN | + ((S_ISDIR (inode->i_mode))?FSEVENT_ISDIR :0 )); }/* @@ -175,6 +189,8 @@ static inline void fsnotify_close(structinotify_dentry_parent_queue_event (dentry, mask,0 , name);inotify_inode_queue_event (inode, mask,0 ,NULL ,NULL ); +raise_fsevent (dentry,FSEVENT_CLOSE | + ((S_ISDIR (inode->i_mode))?FSEVENT_ISDIR :0 )); }/* @@ -190,6 +206,8 @@ static inline void fsnotify_xattr(structinotify_dentry_parent_queue_event (dentry, mask,0 , dentry->d_name.name);inotify_inode_queue_event (inode, mask,0 ,NULL ,NULL ); +raise_fsevent (dentry,FSEVENT_MODIFY_ATTRIB | + ((S_ISDIR (inode->i_mode))?FSEVENT_ISDIR :0 )); }/* @@ -240,6 +258,24 @@ static inline void fsnotify_change(strucinotify_dentry_parent_queue_event (dentry, in_mask,0 , dentry->d_name.name); } + +#ifdef CONFIG_FS_EVENTS + { +u32 fsevent_mask =0 ; +if (ia_valid & (ATTR_UID |ATTR_GID |ATTR_MODE )) + fsevent_mask |=FSEVENT_MODIFY_ATTRIB ; +if ((ia_valid &ATTR_ATIME ) && (ia_valid &ATTR_MTIME )) + fsevent_mask |=FSEVENT_MODIFY_ATTRIB ; +else if (ia_valid &ATTR_ATIME ) + fsevent_mask |=FSEVENT_ACCESS ; +else if (ia_valid &ATTR_MTIME ) + fsevent_mask |=FSEVENT_MODIFY ; +if (ia_valid &ATTR_SIZE ) + fsevent_mask |=FSEVENT_MODIFY ; +if (fsevent_mask) +raise_fsevent (dentry, fsevent_mask); + } +#endif /* CONFIG_FS_EVENTS */ }#ifdef CONFIG_INOTIFY /* inotify helpers */ include/linux/netlink.h renamed from include/linux/netlink.h.orig +1
-0@@ -21,6 +21,7 @@#define NETLINK_DNRTMSG 14/* DECnet routing messages */ #define NETLINK_KOBJECT_UEVENT 15/* Kernel messages to userspace */ #define NETLINK_GENERIC 16 +#define NETLINK_FSEVENT 17/* File system events to userspace */ #define MAX_LINKS 32
Signature · 1 line
Evgeniy Polyakov
Quoted 4 lines · Evgeniy Polyakov
Hello.
On Sat, Sep 30, 2006 at 11:24:55PM +0800, Yi Yang ([email protected]) wrote:
Quoted 9 lines
Quoted 5 lines
This patch implements a new feature -- Filesystem Event Reporter, the user
can monitor filesystem activities via it, currently, it can monitor access,
write, utime, chmod, chown, chgrp, close, open, create, rename, unlink, mkdir,
rmdir, mount, umount.With each new version you add more features, but it seems that code is racy in some places. I've some comments after quite look.
Thank for your careful review, I'll check them and send a new version.
Quoted 11 lines
fs/fsevent.c line 93
Quoted 7 lines
fs/fsevent.c new file +6
-0+extern int *get_fsevent_refcnt (void ); +extern void put_fsevent_refcnt (void ); +extern int per_cpu_fsevent_refcnt (int cpu); +extern void init_fsevent_refcnt (int cpu); +extern void init_missed_fsevent_refcnt (int cpu); +extern atomic_t *per_cpu_missed_refcnt (int cpu);This protects against nothing, since there are now correct reference counters.
Maybe you misunderstand them, new-added reference will synchronize rmmod, you know there are possibly some processes using raise_fsevent hook when a user rmmods fsevent, if just set raise_fsevent to NULL, the kernel will panic because fsevent is released but some processes need to refer to it. Andrew thinks any lock operation shouldn't be involved in filesytem code path, so reference count per cpu is a good choice, the principle behind it is any process will increase reference count of calling raise_fsevent before it is ready for calling raise_fsevent and decrease that reference count after finishing the call, the process scheduler possibly migrate a raise_fsevent caller process from current cpu to another cpu because of workload balance, so for the process migration case, reference count won't work well, I added another reference count -- missed_refcnt which records this kind of migration.
when a user rmmods fsevent, the work kthread per cpu will check that reference count, if it is zero, the work kthread can ensure there is not any process on this cpu using raise_fsevent, so fsevent can be rmmoded on this cpu, if current reference count is equal to missed_refcnt , fsevent can also decide it can exit safely, because that also means there is not any process on this cpu using raise_fsevent, otherwise, the work kthread must yield the cpu and let the process using raise_fsevent progress until any one of the above two cases happens.
In fact, the new code in this version just implement the rmmod synchronization, I spend much time working out this solution, it is really hard a bit not to use any lock to implement it. :)
Quoted 10 lines
fs/fsevent.c line 99
Quoted 7 lines
fs/fsevent.c new file +5
-0+static inline void get_seq (__u32 *ts,int *cpu) +{ + *ts =atomic_inc_return (&fsevent_count); + *cpu =smp_processor_id (); +}Only correct if you call it with disabled preemption.
Yes, you're right, I'll change it.
Quoted 16 lines
Quoted 16 lines
fs/fsevent.c new file +16
-0+static int filter_fsevent_all (u32 * mask) +{ +int ret =0 ; + + (*mask) &=FSEVENT_MASK ; + +if ((((*mask) &FSEVENT_ISDIR ) ==FSEVENT_ISDIR ) + && ((fsevents_mask &FSEVENT_ISDIR ) ==0 )) { + ret =-1 ; + goto out; + } + + (*mask) &= fsevents_mask; +if ((*mask) ==0 ) { + ret =-1 ; + }
Quoted 9 lines
fs/fsevent.c line 137
Quoted 5 lines
fs/fsevent.c new file +4
-0+ +out: +return ret; +}This function is called in the context which allows to change @fsevents_mask, so it is racy.
Yes, to avoid atomic or lock operation is not easy. :)
Quoted 8 lines
fs/fsevent.c line 186
Quoted 4 lines
fs/fsevent.c new file +3
-0+ +get_seq (&(nlhdr->nlmsg_seq), &event->cpu); +ktime_get_ts (&event->timestamp);This timestamp has different size on 32 and 64 bit platforms, so you can not use it for example on
x86_64.
Thank you, I'll change it.
Quoted 14 lines
Quoted 14 lines
fs/fsevent.c new file +13
-0+ event->type = mask; + event->pid = current->tgid; + event->uid = current->uid; + event->gid = current->gid; + nameptr = event->name; + event->pname_len =strlen (current->comm); +append_string (&nameptr, current->comm, event->pname_len); + event->fname_len =strlen (oldname); +append_string (&nameptr, oldname, event->fname_len); + event->len = event->pname_len + event->fname_len +2 ; + event->new_fname_len =0 ; +if (newname) { + event->new_fname_len =strlen (newname);
Quoted 9 lines
fs/fsevent.c line 204
Quoted 6 lines
fs/fsevent.c new file +5
-0+append_string (&nameptr, newname, event->new_fname_len); + event->len += event->new_fname_len +1 ; + } +fsevent_send (skb); +return 0 ;A lot of
strlen()slows code down noticebly.
Yes, I should use it once and saves it to a variable.
Quoted 5 lines
Quoted 5 lines
fs/fsevent.c new file +5
-0+ event->fname_len =0 ; + event->new_fname_len =0 ; + event->err =0 ; + +NETLINK_CB (skb).dst_group =0 ;
Quoted 9 lines
fs/fsevent.c line 243
Quoted 5 lines
fs/fsevent.c new file +4
-0+NETLINK_CB (skb).dst_pid = pid; +NETLINK_CB (skb).pid =0 ; + +return (netlink_unicast (fsevent_sock, skb, pid,MSG_DONTWAIT ));You can lose acks. If you do care about them you need some kind of ack sequence number.
Good, I'll consider it.
Quoted 11 lines
Quoted 11 lines
fs/fsevent.c new file +11
-0+ +while ((skb =skb_dequeue (&sk->sk_receive_queue)) !=NULL ) { +skb_get (skb); +if (skb->len >=FSEVENT_FILTER_MSGSIZE ) { + nlhdr = (struct nlmsghdr *)skb->data; + filter =NLMSG_DATA (nlhdr); + pid =NETLINK_CREDS (skb)->pid; +if (find_fsevent_listener (pid) ==NULL ) + inc_flag =1 ; +if (set_fsevent_filter (filter, pid) ==0 ) { +if (inc_flag ==1 )
Quoted 9 lines
fs/fsevent.c line 440
Quoted 6 lines
fs/fsevent.c new file +5
-0+atomic_inc (&fsevent_listener_num); + } + + } +kfree_skb (skb);You leak skb here, no need to grab it's reference counter above.
You're right, I should remove skb_get();
Quoted 22 lines
Quoted 22 lines
fs/fsevent.c new file +21
-0+ } +} + +#define DEFINE_FILTER_MATCH_FUNC (filtertype, key) \ + static int match_##filtertype(listener * p, \ + struct fsevent * event, \ + struct sk_buff * skb) \ + { \ + int ret = 0; \ + filtertype * xfilter = NULL; \ + struct sk_buff * skb2 = NULL; \ + struct list_head * head = &(p->key##_filter_list_head); \ + list_for_each_entry(xfilter, head, list) { \ + if (xfilter->key != event->key) \ + continue; \ + ret = filter_fsevent(xfilter->mask, event->type); \ + if ( ret != 0) \ + return -1; \ + skb2 = skb_clone(skb, GFP_KERNEL); \ + if (skb2 == NULL) \ + return -1; \
Quoted 9 lines
fs/fsevent.c line 466
Quoted 6 lines
fs/fsevent.c new file +5
-0+NETLINK_CB (skb2).dst_group =0 ; \ +NETLINK_CB (skb2).dst_pid = p->pid; \ +NETLINK_CB (skb2).pid =0 ; \ +return (netlink_unicast (fsevent_sock, skb2, \ + p->pid,MSG_DONTWAIT )); \You reinvent
netlink_broadcast()in some way...
Because pids of the destination processes are unknow before tranversing the fsevent filter list, and every fsevent just is sent to those process which are intereted in it, any other process shouldn't receive a fsevent it aren't intereted in.
Quoted 10 lines
Quoted 10 lines
fs/fsevent.c new file +10
-0+struct fsevent * event =NULL ; +struct sk_buff * skb2 =NULL ; +int ret =0 ; + + event = (struct fsevent *)(skb->data +sizeof (struct nlmsghdr )); +mutex_lock (&listener_list_mutex); +list_for_each_entry_safe (p, q, &listener_list_head, list) { +MATCH_XID (pid, p, event, skb); +MATCH_XID (uid, p, event, skb); +MATCH_XID (gid, p, event, skb);
Quoted 9 lines
fs/fsevent.c line 502
Quoted 6 lines
fs/fsevent.c new file +5
-0+ +if (filter_fsevent (p->mask, event->type) ==0 ) { + skb2 =skb_clone (skb,GFP_KERNEL ); +if (skb2 ==NULL ) +return -1 ;Mutex is not unlocked.
Yes, I missed it before "return -1", I'll change it.
Quoted 11 lines
fs/fsevent.c line 507
Quoted 7 lines
fs/fsevent.c new file +5
-0+NETLINK_CB (skb2).dst_group =0 ; +NETLINK_CB (skb2).dst_pid = p->pid; +NETLINK_CB (skb2).pid =0 ; + ret =netlink_unicast (fsevent_sock, skb2, + p->pid,0 );Btw, pid in netlink messages should not be considered as 'process id' of the sending process. Processes can be changed, but pid will be the same.
I don't understand your comment, do you mean the other id can identify the sender of fsevent correctly, that a pid is used again will take a very long time, I think pid can identify a fsevent sender process, can you explain your comment more detailded?
Quoted 12 lines
Quoted 12 lines
fs/fsevent.c new file +12
-0+static void fsevent_commit (void * unused) +{ +struct sk_buff * skb =NULL ; +int * refcnt, missed_refcnt; +int cpuid; + + +while ((skb =skb_dequeue (&get_cpu_var (fsevent_send_queue))) + !=NULL ) { +fsevent_send_to_process (skb); +kfree_skb (skb); +put_cpu_var (fsevent_send_queue);
Quoted 9 lines
fs/fsevent.c line 533
Quoted 5 lines
fs/fsevent.c new file +4
-0+ } + +if (exit_flag ==1 ) { + refcnt =get_fsevent_refcnt ();What prevents from adding another skb into the queue between above loop and check for flag?
before adding a fsevent to the queue, a process will check exit_flag, if it is set to 1, that process won't queue the fsevent and return immediately.
Quoted 15 lines
Quoted 15 lines
fs/fsevent.c new file +14
-0+if (*refcnt <=0 ) { + *refcnt =-1 ; +put_fsevent_refcnt (); +return ; + } + cpuid =smp_processor_id (); + + missed_refcnt =atomic_read (per_cpu_missed_refcnt (cpuid)); +if (missed_refcnt == *refcnt) + { + *refcnt =-1 ; +printk ("cpu#%d: missed refcnt = %d\n" , cpuid, missed_refcnt); + } +else {
Quoted 9 lines
fs/fsevent.c line 551
Quoted 5 lines
fs/fsevent.c new file +4
-0+if (missed_refcnt !=0 ) { +printk ("cpu#%d: refcnt = %d, missed refcnt = %d\n" , cpuid, *refcnt, missed_refcnt); + } + }Above operation seems racy, what prevents from changing
missed_refcntafter it was read?
if the case you said is hit, missed_refcnt must be not equal to missed_refcnt, because they are for the same cpu, so no problem, it will be checked in the next work schedule.
Quoted 9 lines
Quoted 9 lines
fs/fsevent.c new file +9
-0+static void __exit fsevent_exit(void) +{ +listener * p =NULL , * q =NULL ; +int cpu; +int wait_flag =1 ; +struct sk_buff * skb =NULL ; + + fsevents_mask =0 ; + exit_flag =1 ;
Quoted 9 lines
fs/fsevent.c line 619
Quoted 6 lines
fs/fsevent.c new file +5
-0+ +while (wait_flag ==1 ) { + wait_flag =0 ; +for_each_possible_cpu (cpu) { +if (per_cpu_fsevent_refcnt (cpu) >=0 ) {This check is racy against above checks (marked as racy in comments).
No problem, the real processing is within the work kthread, that will avoid this.
Quoted 11 lines
Quoted 11 lines
fs/fsevent.c new file +10
-0+ wait_flag =1 ; +schedule_work (&per_cpu (fsevent_work, cpu)); + } + } +flush_scheduled_work (); + } + __raise_fsevent =0 ; + +while ((skb =skb_dequeue (&fsevent_sock->sk_receive_queue)) !=NULL ) { +kfree_skb (skb);
Quoted 9 lines
fs/fsevent.c line 633
Quoted 5 lines
fs/fsevent.c new file +4
-0+ } +while ((skb =skb_dequeue (&fsevent_sock->sk_write_queue)) !=NULL ) { +kfree_skb (skb); + }Why are you doing this? It looks wrong, since socket's queue is cleaned automatically.
When I release fsevent_sock, the kernel always printk a message which says "sk_rmem_alloc isn't zero", I don't know why, I doubt there are some packets in recieve and write queue, so try to free them. but sk_rmem_alloc is always non-zero, so I must set it to 0, the kernel doesn't printk.
Quoted 12 lines
fs/fsevent.c line 638
Quoted 7 lines
fs/fsevent.c new file +5
-0+printk ("sk_rmem_alloc=%d, sk_wmem_alloc=%d\n" , +atomic_read (&fsevent_sock->sk_rmem_alloc), +atomic_read (&fsevent_sock->sk_wmem_alloc)); +atomic_set (&fsevent_sock->sk_rmem_alloc,0 ); +atomic_set (&fsevent_sock->sk_wmem_alloc,0 );100 % broken - you should not look into that variables, if kernel prints assertions about sizes, that means either leaked skbs or double freeings.
Can you tell me when sk_rmem_alloc will increase and when it will decrease? How to check whether the skb leakage or double free happened? When I test this, sk_rmem_alloc is always 888, I don't understand it.
Quoted 11 lines
fs/fsevent_hook.c line 33
Quoted 8 lines
fs/fsevent_hook.c new file +7
-0+ +void init_fsevent_refcnt (int cpu) +{ +int * refcnt = &per_cpu (raise_fsevent_refcnt, cpu); + *refcnt =0 ; +} +EXPORT_SYMBOL (init_fsevent_refcnt);This is racy.
This doesn't take effect in the normal processing, the work kthread will do the real work which will ensure no racy.
Quoted 17 lines
Quoted 4 lines
fs/fsevent_hook.c new file +2
-0+void init_missed_fsevent_refcnt (int cpu) +{fs/fsevent_hook.c line 46
Quoted 10 lines
fs/fsevent_hook.c new file +10
-0+atomic_t * missed_refcnt = &per_cpu (missed_fsevent_refcnt, cpu); + +atomic_set (missed_refcnt,0 ); +} +EXPORT_SYMBOL (init_missed_fsevent_refcnt); + +atomic_t *per_cpu_missed_refcnt (int cpu) +{ +return &per_cpu (missed_fsevent_refcnt, cpu); +}Racy.
See the above comments
Quoted 10 lines
Quoted 10 lines
include/linux/fsevent.h new file +10
-0+ +struct fsevent_filter { +/* filter type, it just is one of them +* FSEVENT_FILTER_ALL +* FSEVENT_FILTER_PID +* FSEVENT_FILTER_UID +* FSEVENT_FILTER_GID +*/ +enum fsevent_type type;/* filter type */ +
Quoted 9 lines
include/linux/fsevent.h line 75
Quoted 6 lines
include/linux/fsevent.h new file +5
-0+/* mask of file system events the user listen or ignore +* if the user need to ignore all the events of some pid +* , gid or uid, he(she) must set mask to FSEVENT_MASK. +*/ +fsevent_mask_t mask;This is wrong, since it has different size on 64 and 32 platforms.
thanks, I'll check it.
Quoted 18 lines
Quoted 5 lines
include/linux/fsevent.h new file +3
-0+union { +pid_t pid; +uid_t uid;include/linux/fsevent.h line 88
Quoted 10 lines
include/linux/fsevent.h new file +10
-0+gid_t gid; + } id; + +enum filter_control control; +}; + +struct fsevent { +__u32 type; +__u32 cpu; +struct timespec timestamp;Wrong size.
I don't understand it, can you explain it further?
Quoted 12 lines
Quoted 12 lines
include/linux/fsevent.h new file +12
-0+ +static inline int _raise_fsevent + (const char * oldname,const char * newname,u32 mask) +{ +int ret; +int * refcnt =NULL ; +int start_cpuid, end_cpuid; + + refcnt =get_fsevent_refcnt (); + start_cpuid =smp_processor_id (); +if ((*refcnt ==-1 ) || (__raise_fsevent ==0 )) { +put_fsevent_refcnt ();
Quoted 9 lines
include/linux/fsevent.h line 126
Quoted 5 lines
include/linux/fsevent.h new file +4
-0+return -1 ; + } + (*refcnt)++; +put_fsevent_refcnt ();This looks really racy.
What prevents from rescheduling here?
This has disabled the preemption, so it is impossible to reshcedule.
Quoted 18 lines
Quoted 5 lines
include/linux/fsevent.h new file +3
-0+ ret =__raise_fsevent (oldname, newname, mask); + refcnt =get_fsevent_refcnt (); + end_cpuid =smp_processor_id ();include/linux/fsevent.h line 139
Quoted 10 lines
include/linux/fsevent.h new file +10
-0+if (start_cpuid != end_cpuid) { +printk ("fsevent: process '%s' migration: cpu#%d -> cpu#%d." , current->comm, start_cpuid, end_cpuid); +atomic_inc (per_cpu_missed_refcnt (start_cpuid)); +put_fsevent_refcnt (); +return -1 ; + } + (*refcnt)--; +put_fsevent_refcnt (); +return ret; +}What prevents change for
__raise_fseventin that function?
If reference count is not -1, rmmod won't change __raise_fsevent. the key is two new-added refrence counters.
Quoted 23 lines · Yi Yang
Quoted 11 lines
fs/fsevent.c line 93
Quoted 7 lines
fs/fsevent.c new file +6
-0+extern int *get_fsevent_refcnt (void ); +extern void put_fsevent_refcnt (void ); +extern int per_cpu_fsevent_refcnt (int cpu); +extern void init_fsevent_refcnt (int cpu); +extern void init_missed_fsevent_refcnt (int cpu); +extern atomic_t *per_cpu_missed_refcnt (int cpu);This protects against nothing, since there are now correct reference counters.
Maybe you misunderstand them, new-added reference will synchronize rmmod, you know there are possibly some processes using
raise_fseventhook when a user rmmods fsevent, if just setraise_fseventto NULL, the kernel will panic
because fsevent is released but some processes need to refer to it. Andrew thinks
any lock operation shouldn't be involved in filesytem code path, so reference count
per cpu is a good choice, the principle behind it is any process will increase reference
count of callingraise_fseventbefore it is ready for callingraise_fseventand decrease
Quoted 8 lines
that reference count after finishing the call, the process scheduler possibly
migrate araise_fseventcaller process from current cpu to another cpu because of
workload balance, so for the process migration case, reference count won't work well,
I added another reference count --missed_refcntwhich records this kind of migration.
I do not say that they are broken, but you in some places you access per-cpu variuables without turning preemption off. I think some locking or preemption tweaks should be done there to explicitly mark critical regions.
Quoted 10 lines
Quoted 10 lines
fs/fsevent.c line 507
Quoted 6 lines
fs/fsevent.c new file +5
-0+NETLINK_CB (skb2).dst_group =0 ; +NETLINK_CB (skb2).dst_pid = p->pid; +NETLINK_CB (skb2).pid =0 ; + ret =netlink_unicast (fsevent_sock, skb2, + p->pid,0 );Btw, pid in netlink messages should not be considered as 'process id' of the sending process. Processes can be changed, but pid will be the same.
Quoted 5 lines
I don't understand your comment, do you mean the other id can identify the sender of
fsevent correctly, that a pid is used again will take a very long time, I think pid can identify
a fsevent sender process, can you explain your comment more detailded?
PID in the netlink message is just identity number, it is not and should not be considered as 'process id'. You do not know in advance how long it will take to wrap PIDs, so do not make any assumptions on netlink pid field at all.
Quoted 21 lines
Quoted 12 lines
Quoted 12 lines
fs/fsevent.c new file +12
-0+static void fsevent_commit (void * unused) +{ +struct sk_buff * skb =NULL ; +int * refcnt, missed_refcnt; +int cpuid; + + +while ((skb =skb_dequeue (&get_cpu_var (fsevent_send_queue))) + !=NULL ) { +fsevent_send_to_process (skb); +kfree_skb (skb); +put_cpu_var (fsevent_send_queue);Quoted 9 lines
fs/fsevent.c line 533
Quoted 5 lines
fs/fsevent.c new file +4
-0+ } + +if (exit_flag ==1 ) { + refcnt =get_fsevent_refcnt ();What prevents from adding another skb into the queue between above loop and check for flag?
Quoted 3 lines
before adding a fsevent to the queue, a process will check
exit_flag, if it is set to 1, that
process won't queue the fsevent and return immediately.
But you check for exit_flag in fsevent_commit() without any locks.
Quoted 25 lines
Quoted 16 lines
Quoted 16 lines
fs/fsevent.c new file +12
-0+if (*refcnt <=0 ) { + *refcnt =-1 ; +put_fsevent_refcnt (); +return ; + } + cpuid =smp_processor_id (); + + missed_refcnt =atomic_read (per_cpu_missed_refcnt (cpuid)); +if (missed_refcnt == *refcnt) + { + *refcnt =-1 ; + printk("cpu#%d: missed refcnt = %d\n", cpuid,
missed_refcnt);fs/fsevent.c new file +3
-0+ } +else { +if (missed_refcnt !=0 ) {Quoted 9 lines
Review
Quoted 5 lines
(quoted diff) +1
-0+ printk("cpu#%d: refcnt = %d, missed refcnt =%d\n", cpuid, *refcnt,
missed_refcnt);(quoted diff) +2
-0+ } + }Above operation seems racy, what prevents from changing
missed_refcntafter it was read?
Quoted 4 lines
if the case you said is hit,
missed_refcntmust be not equal tomissed_refcnt, because they are for the same cpu, so no problem, it will be checked
in the next work schedule.
Since it is called with disabled preemption it is ok, but in that case you do not need missed_refcnt to be atomic.
Quoted 41 lines
Quoted 9 lines
Quoted 9 lines
fs/fsevent.c new file +9
-0+static void __exit fsevent_exit(void) +{ +listener * p =NULL , * q =NULL ; +int cpu; +int wait_flag =1 ; +struct sk_buff * skb =NULL ; + + fsevents_mask =0 ; + exit_flag =1 ;Quoted 9 lines
fs/fsevent.c line 619
Quoted 6 lines
fs/fsevent.c new file +5
-0+ +while (wait_flag ==1 ) { + wait_flag =0 ; +for_each_possible_cpu (cpu) { +if (per_cpu_fsevent_refcnt (cpu) >=0 ) {This check is racy against above checks (marked as racy in comments).
No problem, the real processing is within the work kthread, that will avoid this.
Quoted 12 lines
Quoted 12 lines
fs/fsevent.c new file +9
-0+ wait_flag =1 ; +schedule_work (&per_cpu (fsevent_work, cpu)); + } + } +flush_scheduled_work (); + } + __raise_fsevent =0 ; + +while ((skb =skb_dequeue (&fsevent_sock->sk_receive_queue)) !=NULL ){
(quoted diff) +1
-0+ kfree_skb(skb);Quoted 9 lines
fs/fsevent.c line 633
Quoted 5 lines
fs/fsevent.c new file +4
-0+ } +while ((skb =skb_dequeue (&fsevent_sock->sk_write_queue)) !=NULL ) { +kfree_skb (skb); + }Why are you doing this? It looks wrong, since socket's queue is cleaned automatically.
Quoted 6 lines
When I release
fsevent_sock, the kernel always printk a message which says "sk_rmem_allocisn't zero",
I don't know why, I doubt there are some packets in recieve and write queue, so try to free them.
butsk_rmem_allocis always non-zero, so I must set it to 0, the kernel doesn't printk.
That means that you broke socket accounting in some way. sock_release() should do all cleanup for you.
Quoted 12 lines
Quoted 12 lines
fs/fsevent.c line 638
Quoted 7 lines
fs/fsevent.c new file +5
-0+printk ("sk_rmem_alloc=%d, sk_wmem_alloc=%d\n" , +atomic_read (&fsevent_sock->sk_rmem_alloc), +atomic_read (&fsevent_sock->sk_wmem_alloc)); +atomic_set (&fsevent_sock->sk_rmem_alloc,0 ); +atomic_set (&fsevent_sock->sk_wmem_alloc,0 );100 % broken - you should not look into that variables, if kernel prints assertions about sizes, that means either leaked skbs or double freeings.
Quoted 5 lines
Can you tell me when
sk_rmem_allocwill increase and when it will decrease? How to
check whether the skb leakage or double free happened? When I test this,sk_rmem_alloc
is always 888, I don't understand it.
Each time you add skb into socket queue appropriate socket is charged for value equal to sizeof(skb)+sizeof(skb_shared_info)+aligned size of the data. That number is added to the one of the sk_r/wmem_alloc, depending on the direction of the skb way, skb's destructor is set to the function which will remove appropiate amount of from above variables.
When you call sock_release() all skbs are removed and freed, so socket accounting is corrected in kfree_skb(), which (if there are no users) calls destructor and frees skb and data.
If you see asserions that above variables are not zero, that means that you either removed skb from the queue and forgot to free it, or freed it several times (although it will be likely a crash in this case), or you overwrote that variables after some memory corruption.
Quoted 11 lines
Quoted 11 lines
fs/fsevent_hook.c line 33
Quoted 8 lines
fs/fsevent_hook.c new file +7
-0+ +void init_fsevent_refcnt (int cpu) +{ +int * refcnt = &per_cpu (raise_fsevent_refcnt, cpu); + *refcnt =0 ; +} +EXPORT_SYMBOL (init_fsevent_refcnt);This is racy.
Quoted 3 lines
This doesn't take effect in the normal processing, the work kthread will do the real
work which will ensure no racy.
Then just remove it, and actually the whole modularity does not seems a good idea, although it is of course your decision to make design static or not. I would implement such things with dynamic registration of the clients and just make fsevent statically built into the kernel.
Quoted 37 lines
Quoted 9 lines
Quoted 9 lines
include/linux/fsevent.h new file +9
-0+struct fsevent_filter { +/* filter type, it just is one of them +* FSEVENT_FILTER_ALL +* FSEVENT_FILTER_PID +* FSEVENT_FILTER_UID +* FSEVENT_FILTER_GID +*/ +enum fsevent_type type;/* filter type */ +Quoted 9 lines
include/linux/fsevent.h line 75
Quoted 6 lines
include/linux/fsevent.h new file +5
-0+/* mask of file system events the user listen or ignore +* if the user need to ignore all the events of some pid +* , gid or uid, he(she) must set mask to FSEVENT_MASK. +*/ +fsevent_mask_t mask;This is wrong, since it has different size on 64 and 32 platforms.
thanks, I'll check it.
Quoted 18 lines
Quoted 5 lines
include/linux/fsevent.h new file +3
-0+union { +pid_t pid; +uid_t uid;include/linux/fsevent.h line 88
Quoted 10 lines
include/linux/fsevent.h new file +10
-0+gid_t gid; + } id; + +enum filter_control control; +}; + +struct fsevent { +__u32 type; +__u32 cpu; +struct timespec timestamp;Wrong size.
Quoted 1 line
I don't understand it, can you explain it further?
It has different size on 32 and 64 bit platforms, since it uses longs. Your application will not work if kernel and userspace has different size of the long, like can happen with for example x86_64.
Quoted 20 lines
Quoted 11 lines
Quoted 11 lines
include/linux/fsevent.h new file +11
-0+static inline int _raise_fsevent + (const char * oldname,const char * newname,u32 mask) +{ +int ret; +int * refcnt =NULL ; +int start_cpuid, end_cpuid; + + refcnt =get_fsevent_refcnt (); + start_cpuid =smp_processor_id (); +if ((*refcnt ==-1 ) || (__raise_fsevent ==0 )) { +put_fsevent_refcnt ();Quoted 9 lines
include/linux/fsevent.h line 126
Quoted 5 lines
include/linux/fsevent.h new file +4
-0+return -1 ; + } + (*refcnt)++; +put_fsevent_refcnt ();This looks really racy.
What prevents from rescheduling here?
Quoted 1 line
This has disabled the preemption, so it is impossible to reshcedule.
No, put_fsevent_refcnt() andbles it again.
Or is it disabled on higher layer?
Quoted 18 lines
Quoted 9 lines
Quoted 9 lines
include/linux/fsevent.h new file +5
-0+ ret =__raise_fsevent (oldname, newname, mask); + refcnt =get_fsevent_refcnt (); + end_cpuid =smp_processor_id (); +if (start_cpuid != end_cpuid) { + printk("fsevent: process '%s' migration: cpu#%d -> cpu#%d.",
current->comm,start_cpuid,end_cpuid);include/linux/fsevent.h new file +3
-0+atomic_inc (per_cpu_missed_refcnt (start_cpuid)); +put_fsevent_refcnt (); +return -1 ;Quoted 9 lines
include/linux/fsevent.h line 139
Quoted 6 lines
include/linux/fsevent.h new file +5
-0+ } + (*refcnt)--; +put_fsevent_refcnt (); +return ret; +}What prevents change for
__raise_fseventin that function?
Quoted 3 lines
If reference count is not -1, rmmod won't change
__raise_fsevent. the key is two new-added
refrence counters.
You do it without preemption disabled and any other locks...
Signature · 1 line
Evgeniy Polyakov
Quoted 7 lines
I do not say that they are broken, but you in some places you access per-cpu
variuables without turning preemption off. I think some locking or preemption tweaks should be done there to explicitly mark critical regions.
You're right, some places have such issues, I just considered how to avoid the lock or atomic operation, Andrew ever mentioned the lock is unacceptible in file system code path, so I always avoid the lock or atomic operation.
Quoted 11 lines
Quoted 1 line
Quoted 1 line
What prevents from adding another skb into the queue between
above loop
> >and check for flag?
> >
> before adding a fsevent to the queue, a process will checkexit_flag, if
> it is set to 1, that
> process won't queue the fsevent and return immediately.But you check for
exit_flaginfsevent_commit()without any locks.
Only rmmod will set exit_flag, other users are readers, so I think the lock is unnecessary, only one issue is that I should clear fsevent_queue in the last section of fsevent_exit.
Quoted 12 lines
Quoted 3 lines
Quoted 3 lines
Above operation seems racy, what prevents from changing
missed_refcnt
> >after it was read?
> >
> if the case you said is hit,missed_refcntmust be not equal to >missed_refcnt, because they are for the same cpu, so no problem, it will
> be checked
> in the next work schedule.
Quoted 4 lines
Since it is called with disabled preemption it is ok, but in that case
you do not needmissed_refcntto be atomic.
in include/linux/fsevent.h, it is possibly accessed from diffrent cpus, so atomic is necessary.
Quoted 29 lines
Quoted 2 lines
Quoted 2 lines
Why are you doing this? It looks wrong, since socket's queue is
cleaned
> >automatically.
> >
> When I releasefsevent_sock, the kernel always printk a message which
> says "sk_rmem_allocisn't zero",
> I don't know why, I doubt there are some packets in recieve and write
> queue, so try to free them.
> butsk_rmem_allocis always non-zero, so I must set it to 0, the kernel
> doesn't printk.That means that you broke socket accounting in some way.
sock_release()should do all cleanup for you.Each time you add skb into socket queue appropriate socket is charged for
value equal to sizeof(skb)+sizeof(skb_shared_info)+aligned size of the data.
That number is added to the one of the sk_r/wmem_alloc, depending on the
direction of the skb way, skb's destructor is set to the function which
will remove appropiate amount of from above variables. When you callsock_release()all skbs are removed and freed, so socket accounting is corrected inkfree_skb(), which (if there are no users)
Quoted 9 lines
calls destructor and frees skb and data.
If you see asserions that above variables are not zero, that means that
you either removed skb from the queue and forgot to free it, or freed it
several times (although it will be likely a crash in this case), or you
overwrote that variables after some memory corruption.
maybe that surplus skb_get is the root cause.
Quoted 8 lines
Quoted 4 lines
Quoted 3 lines
This is racy.
This doesn't take effect in the normal processing, the work
kthread will
> do the real
> work which will ensure no racy.
Quoted 8 lines
Then just remove it, and actually the whole modularity does not seems a
good idea, although it is of course your decision to make design static
or not. I would implement such things with dynamic registration of the
clients and just make fsevent statically built into the kernel.
It is hard a bit for the subsystem using the hook mechanism to be implemented as a module. In fact, all the newly-added code in this patch is for modularity. :)
Really that is a way to build as a static infrastructure, the filesystem init code calls a fsevent register API to enable it, but unregister is not a trivial, the syncronization issue still exists. Nevertheless, this is really is a way I can try.
Quoted 9 lines
Quoted 5 lines
Quoted 4 lines
This looks really racy.
What prevents from rescheduling here?This has disabled the preemption, so it is impossible to reshcedule.
No,
put_fsevent_refcnt()andbles it again.
Or is it disabled on higher layer?
I think your "reschedule" means process migration, those code just considers this issue, missed_refcnt is just for this, start_cpuid is used to identify the cpu before migration, end_cpuid is used to identify the cpu after migration, if start_cpuid is equal to end_cpuid, we can think there is no migration happened, otherwise, missed_refcnt[start_cpuid] will increase, because there are possibly several prcoesses on different cpus to modify this value, so it is defined as atomic.
Quoted 11 lines
Quoted 5 lines
Quoted 4 lines
What prevents change for
__raise_fseventin that function?If reference count is not -1, rmmod won't change
__raise_fsevent. the
> key is two new-added
> refrence counters.You do it without preemption disabled and any other locks...
Only rmmod will change __raise_fsevent and it will set it to 0 just after all the filesystem code paths nerver call it, if reference count on anuy cpu is not -1, rmmod will wait for it until this cpu doesn't call raise_fsevent any more, rmmod will set it to 0 just after all the reference count on all the cpu are -1, so only one user -- rmmod -- is accessing it in that time, this is safe.