fsnotify: Avoid SRCU stalls with fanotify permission events
Subsystems (6)
Jan Kara
maintainerv4 cover fsnotify: Avoid SRCU stalls with fanotify permission events
Hello,
This is the fourth revision of my patches to avoid SRCU stalls when fanotify waits for response to permission events from userspace processes. Thanks to Amir, Paul, and Miklos for review! It also passes a new LTP test that tries to provoke hangs in fanotify subsystem when there are unanswered fanotify permission events. If nobody has more objections, I'll push the changes to my tree.
Amir, I have dropped your Reviewed-by from the 4 patches that were originally one patch. I have made sure though, that the resulting diff is identical. So just let me know whether you are OK with me adding your Reviewed-by back.
Miklos, is the split like this better for you?
Paul, can you please test audit with these patches applied?
Changes since v3:
* added Reviewed-by tags
* split adding of fsnotify_mark_connector into 4 smaller parts as Miklos asked
* simplified API of fsnotify_prepare/finish_user_wait()
Changes since v2:
* added Reviewed-by tags
* dropped fsnotify_put_list() abstraction
* use rcu_assign_pointer() where appropriate
Changes since v1:
* renamed fsnotify_mark_list to fsnotify_mark_connector and couple other things
* updated some comments and changelogs to better explain what is going on
* made audit use inode pointer as a key again
* added Reviewed-by tags
* dropped two audit fixes that got already merged
* added cleanup of mark destruction functions
Patch set overview
------------------
Currently, fanotify waits for response to a permission even from userspace process while holding fsnotify_mark_srcu lock. That has a consequence that when userspace process takes long to respond or does not respond at all, fsnotify_mark_srcu period cannot ever complete blocking reclaim of any notification marks and also blocking any process that did synchronize_srcu() on fsnotify_mark_srcu. Effectively, this eventually blocks anybody interacting with the notification subsystem. Miklos has some real world reports of this happening. Although this in principle a problem of broken userspace application (which futhermore has to have CAP_SYS_ADMIN in init_user_ns, so it is not a security problem), it is still nasty that a simple error can block the kernel like this.
This patch set solves this problem. The basic idea of the solution is that when fanotify needs to wait for response from userspace process, it grabs reference to the mark which generated the event and drops fsnotify_mark_srcu lock. When userspace responds, we grab fsnotify_mark_srcu again, drop the mark reference, and continue iterating the list of marks attached to the inode / vfsmount delivering the event to other notification groups. What complicates this simple approach is that the mark for which we wait for response has to stay pinned in the list of marks attached to the inode / vfsmount so that we can resume iteration of the list when userspace responds but on the other hand when the inode gets unlinked while we wait for userspace reponse, we need to destroy the mark (or at least detach it from the inode).
The first 3 patches contain some initial fixes and cleanups. Patches 4-9 implement attaching of marks to inode / vfsmount via a dedicated structure which allows us to detach list of marks from the object without having to destroy the list itself. Patches 10-12 implement removal of mark from the list of marks attached to an object when last mark reference is dropped. Patches 13-16 then implement dropping of SRCU lock when waiting on response from userspace. Patches 17-25 are mostly trivial cleanups that get rid of trivial wrappers and one pointer in the mark structure.
Patches have survived testing with inotify/fanotify tests in LTP.
Finally, to ease experimenting with the patches I've pushed them out togit://git.kernel.org/pub/scm/linux/kernel/git/jack/linux-fs.git for_testing
Honza
Jan Kara
maintainerPATCH v4 1/25 fsnotify: Remove unnecessary tests when showing fdinfo
show_fdinfo() iterates group's list of marks. All marks found there are guaranteed to be alive and they stay so until we release group->mark_mutex. So remove uncecessary tests whether mark is alive.
--- fs/notify/fdinfo.c | 6 +----- 1 file changed, 1 insertion(+), 5 deletions(-)
fs/notify/fdinfo.c +1 -5
@@ -76,8 +76,7 @@ static void inotify_fdinfo(struct seq_file *m, struct fsnotify_mark *mark)struct inotify_inode_mark *inode_mark;struct inode *inode; -if (!(mark->flags &FSNOTIFY_MARK_FLAG_ALIVE ) || - !(mark->flags &FSNOTIFY_MARK_FLAG_INODE )) +if (!(mark->flags &FSNOTIFY_MARK_FLAG_INODE ))return ; inode_mark = container_of(mark, struct inotify_inode_mark, fsn_mark);
@@ -113,9 +112,6 @@ static void fanotify_fdinfo(struct seq_file *m, struct fsnotify_mark *mark)unsigned int mflags =0 ;struct inode *inode; -if (!(mark->flags &FSNOTIFY_MARK_FLAG_ALIVE )) -return ; -if (mark->flags &FSNOTIFY_MARK_FLAG_IGNORED_SURV_MODIFY ) mflags |=FAN_MARK_IGNORED_SURV_MODIFY ;
Signature · 1 line
2.10.2
Jan Kara
maintainerPATCH v4 2/25 inotify: Remove inode pointers from debug messages
In reply to Jan KaraPrinting inode pointers in warnings has dubious value and with future changes we won't be able to easily get them without either locking or chances we oops along the way. So just remove inode pointers from the warning messages.
--- fs/notify/inotify/inotify_fsnotify.c | 4 ++-- fs/notify/inotify/inotify_user.c | 25 ++++++++++--------------- 2 files changed, 12 insertions(+), 17 deletions(-)
fs/notify/inotify/inotify_fsnotify.c +2 -2
@@ -155,8 +155,8 @@ static int idr_callback(int id, void *p, void *data)* BUG() that was here. */ if (fsn_mark) -printk (KERN_WARNING "fsn_mark->group=%p inode=%p wd=%d\n" , - fsn_mark->group, fsn_mark->inode, i_mark->wd); +printk (KERN_WARNING "fsn_mark->group=%p wd=%d\n" , + fsn_mark->group, i_mark->wd);return 0 ; }
fs/notify/inotify/inotify_user.c +10 -15
@@ -431,18 +431,16 @@ static void inotify_remove_from_idr(struct fsnotify_group *group,* if it wasn't.... */ if (wd ==-1 ) { -WARN_ONCE (1 ,"%s: i_mark=%p i_mark->wd=%d i_mark->group=%p" -" i_mark->inode=%p\n" , __func__, i_mark, i_mark->wd, - i_mark->fsn_mark.group, i_mark->fsn_mark.inode); +WARN_ONCE (1 ,"%s: i_mark=%p i_mark->wd=%d i_mark->group=%p\n" , + __func__, i_mark, i_mark->wd, i_mark->fsn_mark.group); goto out; }/* Lets look in the idr to see if we find it */ found_i_mark =inotify_idr_find_locked (group, wd);if (unlikely (!found_i_mark)) { -WARN_ONCE (1 ,"%s: i_mark=%p i_mark->wd=%d i_mark->group=%p" -" i_mark->inode=%p\n" , __func__, i_mark, i_mark->wd, - i_mark->fsn_mark.group, i_mark->fsn_mark.inode); +WARN_ONCE (1 ,"%s: i_mark=%p i_mark->wd=%d i_mark->group=%p\n" , + __func__, i_mark, i_mark->wd, i_mark->fsn_mark.group); goto out; }
@@ -453,12 +451,10 @@ static void inotify_remove_from_idr(struct fsnotify_group *group,*/ if (unlikely (found_i_mark != i_mark)) {WARN_ONCE (1 ,"%s: i_mark=%p i_mark->wd=%d i_mark->group=%p " -"mark->inode=%p found_i_mark=%p found_i_mark->wd=%d " -"found_i_mark->group=%p found_i_mark->inode=%p\n" , - __func__, i_mark, i_mark->wd, i_mark->fsn_mark.group, - i_mark->fsn_mark.inode, found_i_mark, found_i_mark->wd, - found_i_mark->fsn_mark.group, - found_i_mark->fsn_mark.inode); +"found_i_mark=%p found_i_mark->wd=%d " +"found_i_mark->group=%p\n" , __func__, i_mark, + i_mark->wd, i_mark->fsn_mark.group, found_i_mark, + found_i_mark->wd, found_i_mark->fsn_mark.group); goto out; }
@@ -468,9 +464,8 @@ static void inotify_remove_from_idr(struct fsnotify_group *group,* one ref grabbed by inotify_idr_find */ if (unlikely (atomic_read (&i_mark->fsn_mark.refcnt) <3 )) { -printk (KERN_ERR "%s: i_mark=%p i_mark->wd=%d i_mark->group=%p" -" i_mark->inode=%p\n" , __func__, i_mark, i_mark->wd, - i_mark->fsn_mark.group, i_mark->fsn_mark.inode); +printk (KERN_ERR "%s: i_mark=%p i_mark->wd=%d i_mark->group=%p\n" , + __func__, i_mark, i_mark->wd, i_mark->fsn_mark.group);/* we can't really recover with bad ref cnting.. */ BUG (); }
Signature · 1 line
2.10.2
Jan Kara
maintainerPATCH v4 3/25 fanotify: Move recalculation of inode / vfsmount mask under mark_mutex
In reply to Jan KaraMove recalculation of inode / vfsmount notification mask under group->mark_mutex of the mark which was modified. These are the only places where mask recalculation happens without mark being protected from detaching from inode / vfsmount which will cause issues with the following patches.
--- fs/notify/fanotify/fanotify_user.c | 15 ++++++--------- 1 file changed, 6 insertions(+), 9 deletions(-)
fs/notify/fanotify/fanotify_user.c +6 -9
@@ -541,6 +541,8 @@ static int fanotify_remove_vfsmount_mark(struct fsnotify_group *group, removed =fanotify_mark_remove_from_mask (fsn_mark, mask, flags, &destroy_mark); +if (removed &real_mount (mnt)->mnt_fsnotify_mask) +fsnotify_recalc_vfsmount_mask (mnt);if (destroy_mark)fsnotify_detach_mark (fsn_mark);mutex_unlock (&group->mark_mutex);
@@ -548,9 +550,6 @@ static int fanotify_remove_vfsmount_mark(struct fsnotify_group *group,fsnotify_free_mark (fsn_mark);fsnotify_put_mark (fsn_mark); -if (removed &real_mount (mnt)->mnt_fsnotify_mask) -fsnotify_recalc_vfsmount_mask (mnt); -return 0 ; }
@@ -571,6 +570,8 @@ static int fanotify_remove_inode_mark(struct fsnotify_group *group, removed =fanotify_mark_remove_from_mask (fsn_mark, mask, flags, &destroy_mark); +if (removed & inode->i_fsnotify_mask) +fsnotify_recalc_inode_mask (inode);if (destroy_mark)fsnotify_detach_mark (fsn_mark);mutex_unlock (&group->mark_mutex);
@@ -579,8 +580,6 @@ static int fanotify_remove_inode_mark(struct fsnotify_group *group,/* matches the fsnotify_find_inode_mark() */ fsnotify_put_mark (fsn_mark); -if (removed & inode->i_fsnotify_mask) -fsnotify_recalc_inode_mask (inode);return 0 ; }
@@ -656,10 +655,9 @@ static int fanotify_add_vfsmount_mark(struct fsnotify_group *group, } } added =fanotify_mark_add_to_mask (fsn_mark, mask, flags); -mutex_unlock (&group->mark_mutex); -if (added & ~real_mount (mnt)->mnt_fsnotify_mask)fsnotify_recalc_vfsmount_mask (mnt); +mutex_unlock (&group->mark_mutex);fsnotify_put_mark (fsn_mark);return 0 ;
@@ -694,10 +692,9 @@ static int fanotify_add_inode_mark(struct fsnotify_group *group, } } added =fanotify_mark_add_to_mask (fsn_mark, mask, flags); -mutex_unlock (&group->mark_mutex); -if (added & ~inode->i_fsnotify_mask)fsnotify_recalc_inode_mask (inode); +mutex_unlock (&group->mark_mutex);fsnotify_put_mark (fsn_mark);return 0 ;
Signature · 1 line
2.10.2
Jan Kara
maintainerPATCH v4 4/25 audit: Abstract hash key handling
In reply to Jan KaraAudit tree currently uses inode pointer as a key into the hash table. Getting that from notification mark will be somewhat more difficult with coming fsnotify changes. So abstract getting of hash key from the audit chunk and inode so that we can change the method to obtain a key easily.
--- kernel/audit_tree.c | 39 ++++++++++++++++++++++++++++----------- 1 file changed, 28 insertions(+), 11 deletions(-)
kernel/audit_tree.c +28 -11
@@ -163,33 +163,48 @@ enum {HASH_SIZE = 128};static struct list_head chunk_hash_heads[HASH_SIZE ];static __cacheline_aligned_in_smp DEFINE_SPINLOCK (hash_lock ); -static inline struct list_head *chunk_hash (const struct inode *inode) +/* Function to return search key in our hash from inode. */ +static unsigned long inode_to_key (const struct inode *inode) { -unsigned long n = (unsigned long )inode /L1_CACHE_BYTES ; +return (unsigned long )inode; +} + +/* +* Function to return search key in our hash from chunk. Key 0 is special and +* should never be present in the hash. +*/ +static unsigned long chunk_to_key (struct audit_chunk *chunk) +{ +return (unsigned long )chunk->mark.inode; +} + +static inline struct list_head *chunk_hash (unsigned long key) +{ +unsigned long n = key /L1_CACHE_BYTES ;return chunk_hash_heads + n %HASH_SIZE ; }/* hash_lock & entry->lock is held by caller */ static void insert_hash (struct audit_chunk *chunk) { -struct fsnotify_mark *entry = &chunk->mark; +unsigned long key =chunk_to_key (chunk);struct list_head *list; -if (!entry->inode) +if (!key)return ; - list =chunk_hash (entry->inode); + list =chunk_hash (key);list_add_rcu (&chunk->hash, list); }/* called under rcu_read_lock */ struct audit_chunk *audit_tree_lookup (const struct inode *inode) { -struct list_head *list =chunk_hash (inode); +unsigned long key =inode_to_key (inode); +struct list_head *list =chunk_hash (key);struct audit_chunk *p;list_for_each_entry_rcu (p, list, hash) { -/* mark.inode may have gone NULL, but who cares? */ -if (p->mark.inode == inode) { +if (chunk_to_key (p) == key) {atomic_long_inc (&p->refs);return p; }
@@ -588,7 +603,8 @@ int audit_remove_tree_rule(struct audit_krule *rule)static int compare_root (struct vfsmount *mnt,void *arg) { -return d_backing_inode (mnt->mnt_root) == arg; +return inode_to_key (d_backing_inode (mnt->mnt_root)) == + (unsigned long )arg; }void audit_trim_trees (void )
@@ -623,9 +639,10 @@ void audit_trim_trees(void)list_for_each_entry (node, &tree->chunks, list) {struct audit_chunk *chunk =find_chunk (node);/* this could be NULL if the watch is dying else where... */ -struct inode *inode = chunk->mark.inode; node->index |=1U <<31 ; -if (iterate_mounts (compare_root, inode, root_mnt)) +if (iterate_mounts (compare_root, + (void *)chunk_to_key (chunk), + root_mnt)) node->index &= ~(1U <<31 ); }spin_unlock (&hash_lock);
Signature · 1 line
2.10.2
Jan Kara
maintainerPATCH v4 5/25 fsnotify: Update comments
In reply to Jan KaraAdd a comment that lifetime of a notification mark is protected by SRCU and remove a comment about clearing of marks attached to the inode. It is stale and more uptodate version is at fsnotify_destroy_marks() which is the function handling this case.
--- fs/notify/mark.c | 13 +------------ 1 file changed, 1 insertion(+), 12 deletions(-)
fs/notify/mark.c +1 -12
@@ -51,7 +51,7 @@* * LIFETIME: * Inode marks survive between when they are added to an inode and when their -* refcnt==0. +* refcnt==0. Marks are also protected by fsnotify_mark_srcu. * * The inode mark can be cleared for a number of different reasons including: * - The inode is unlinked for the last time. (fsnotify_inode_remove)
@@ -61,17 +61,6 @@* - The fsnotify_group associated with the mark is going away and all such marks * need to be cleaned up. (fsnotify_clear_marks_by_group) * -* Worst case we are given an inode and need to clean up all the marks on that -* inode. We take i_lock and walk the i_fsnotify_marks safely. For each -* mark on the list we take a reference (so the mark can't disappear under us). -* We remove that mark form the inode's list of marks and we add this mark to a -* private list anchored on the stack using i_free_list; we walk i_free_list -* and before we destroy the mark we make sure that we dont race with a -* concurrent destroy_group by getting a ref to the marks group and taking the -* groups mutex. - -* Very similarly for freeing by group, except we use free_g_list. -* * This has the very interesting property of being able to run concurrently with * any (or all) other directions. */
Signature · 1 line
2.10.2
Jan Kara
maintainerPATCH v4 6/25 fsnotify: Attach marks to object via dedicated head structure
In reply to Jan KaraCurrently notification marks are attached to object (inode or vfsmnt) by a hlist_head in the object. The list is also protected by a spinlock in the object. So while there is any mark attached to the list of marks, the object must be pinned in memory (and thus e.g. last iput() deleting inode cannot happen). Also for list iteration in fsnotify() to work, we must hold fsnotify_mark_srcu lock so that mark itself and mark->obj_list.next cannot get freed. Thus we are required to wait for response to fanotify events from userspace process with fsnotify_mark_srcu lock held. That causes issues when userspace process is buggy and does not reply to some event - basically the whole notification subsystem gets eventually stuck.
So to be able to drop fsnotify_mark_srcu lock while waiting for response, we have to pin the mark in memory and make sure it stays in the object list (as removing the mark waiting for response could lead to lost notification events for groups later in the list). However we don't want inode reclaim to block on such mark as that would lead to system just locking up elsewhere.
This commit tries to pave a way towards solving these conflicting lifetime needs. Instead of anchoring the list of marks directly in the object, we anchor it in a dedicated structure (fsnotify_mark_connector) and just point to that structure from the object. In the following patch we will also protect the list by a spinlock contained in that structure. With this, we will be able to detach notification marks from object without having to modify the list itself.
--- fs/inode.c | 3 - fs/mount.h | 2 +- fs/namespace.c | 3 - fs/notify/dnotify/dnotify.c | 4 +- fs/notify/fdinfo.c | 12 +-- fs/notify/fsnotify.c | 32 +++++--- fs/notify/fsnotify.h | 17 +++-- fs/notify/inode_mark.c | 74 ++++-------------- fs/notify/mark.c | 157 +++++++++++++++++++++++++++++---------- fs/notify/vfsmount_mark.c | 41 +++------- include/linux/fs.h | 4 +- include/linux/fsnotify_backend.h | 38 +++++++--- kernel/audit_tree.c | 30 ++++++-- kernel/auditsc.c | 4 +- 14 files changed, 236 insertions(+), 185 deletions(-)
fs/inode.c +0 -3
@@ -371,9 +371,6 @@ void inode_init_once(struct inode *inode)INIT_LIST_HEAD (&inode->i_lru);address_space_init_once (&inode->i_data);i_size_ordered_init (inode); -#ifdef CONFIG_FSNOTIFY -INIT_HLIST_HEAD (&inode->i_fsnotify_marks); -#endif }EXPORT_SYMBOL (inode_init_once);
fs/mount.h +1 -1
@@ -59,7 +59,7 @@ struct mount {struct mountpoint *mnt_mp;/* where is it mounted */ struct hlist_node mnt_mp_list;/* list mounts with the same mountpoint */ #ifdef CONFIG_FSNOTIFY -struct hlist_head mnt_fsnotify_marks; +struct fsnotify_mark_connector *mnt_fsnotify_marks;__u32 mnt_fsnotify_mask;#endif int mnt_id;/* mount identifier */
fs/namespace.c +0 -3
@@ -233,9 +233,6 @@ static struct mount *alloc_vfsmnt(const char *name)INIT_LIST_HEAD (&mnt->mnt_slave_list);INIT_LIST_HEAD (&mnt->mnt_slave);INIT_HLIST_NODE (&mnt->mnt_mp_list); -#ifdef CONFIG_FSNOTIFY -INIT_HLIST_HEAD (&mnt->mnt_fsnotify_marks); -#endif init_fs_pin (&mnt->mnt_umount, drop_mountpoint); }return mnt;
fs/notify/dnotify/dnotify.c +2 -2
@@ -69,8 +69,8 @@ static void dnotify_recalc_inode_mask(struct fsnotify_mark *fsn_mark)if (old_mask == new_mask)return ; -if (fsn_mark->inode) -fsnotify_recalc_inode_mask (fsn_mark->inode); +if (fsn_mark->connector->inode) +fsnotify_recalc_inode_mask (fsn_mark->connector->inode); }/*
fs/notify/fdinfo.c +6 -6
@@ -76,11 +76,11 @@ static void inotify_fdinfo(struct seq_file *m, struct fsnotify_mark *mark)struct inotify_inode_mark *inode_mark;struct inode *inode; -if (!(mark->flags &FSNOTIFY_MARK_FLAG_INODE )) +if (!(mark->connector->flags &FSNOTIFY_OBJ_TYPE_INODE ))return ; inode_mark = container_of(mark, struct inotify_inode_mark, fsn_mark); - inode =igrab (mark->inode); + inode =igrab (mark->connector->inode);if (inode) {/* * IN_ALL_EVENTS represents all of the mask bits
@@ -115,8 +115,8 @@ static void fanotify_fdinfo(struct seq_file *m, struct fsnotify_mark *mark)if (mark->flags &FSNOTIFY_MARK_FLAG_IGNORED_SURV_MODIFY ) mflags |=FAN_MARK_IGNORED_SURV_MODIFY ; -if (mark->flags &FSNOTIFY_MARK_FLAG_INODE ) { - inode =igrab (mark->inode); +if (mark->connector->flags &FSNOTIFY_OBJ_TYPE_INODE ) { + inode =igrab (mark->connector->inode);if (!inode)return ; seq_printf(m, "fanotify ino:%lx sdev:%x mflags:%x mask:%x ignored_mask:%x ",
@@ -125,8 +125,8 @@ static void fanotify_fdinfo(struct seq_file *m, struct fsnotify_mark *mark)show_mark_fhandle (m, inode);seq_putc (m,'\n' );iput (inode); - }else if (mark->flags &FSNOTIFY_MARK_FLAG_VFSMOUNT ) { -struct mount *mnt =real_mount (mark->mnt); + }else if (mark->connector->flags &FSNOTIFY_OBJ_TYPE_VFSMOUNT ) { +struct mount *mnt =real_mount (mark->connector->mnt);seq_printf (m,"fanotify mnt_id:%x mflags:%x mask:%x ignored_mask:%x\n" , mnt->mnt_id, mflags, mark->mask, mark->ignored_mask);
fs/notify/fsnotify.c +23 -9
@@ -193,6 +193,7 @@ int fsnotify(struct inode *to_tell, __u32 mask, const void *data, int data_is,struct hlist_node *inode_node =NULL , *vfsmount_node =NULL ;struct fsnotify_mark *inode_mark =NULL , *vfsmount_mark =NULL ;struct fsnotify_group *inode_group, *vfsmount_group; +struct fsnotify_mark_connector *inode_conn, *vfsmount_conn;struct mount *mnt;int idx, ret =0 ;/* global tests shouldn't care about events on child only the specific event */
@@ -210,8 +211,8 @@ int fsnotify(struct inode *to_tell, __u32 mask, const void *data, int data_is,* SRCU because we have no references to any objects and do not * need SRCU to keep them "alive". */ -if (hlist_empty (&to_tell->i_fsnotify_marks) && - (!mnt ||hlist_empty (&mnt->mnt_fsnotify_marks))) +if (!to_tell->i_fsnotify_marks && + (!mnt || !mnt->mnt_fsnotify_marks))return 0 ;/* * if this is a modify event we may need to clear the ignored masks
@@ -226,16 +227,24 @@ int fsnotify(struct inode *to_tell, __u32 mask, const void *data, int data_is, idx =srcu_read_lock (&fsnotify_mark_srcu);if ((mask &FS_MODIFY ) || - (test_mask & to_tell->i_fsnotify_mask)) - inode_node =srcu_dereference (to_tell->i_fsnotify_marks.first, - &fsnotify_mark_srcu); + (test_mask & to_tell->i_fsnotify_mask)) { + inode_conn = to_tell->i_fsnotify_marks; +if (inode_conn) + inode_node =srcu_dereference (inode_conn->list.first, + &fsnotify_mark_srcu); + }if (mnt && ((mask &FS_MODIFY ) || (test_mask & mnt->mnt_fsnotify_mask))) { - vfsmount_node =srcu_dereference (mnt->mnt_fsnotify_marks.first, - &fsnotify_mark_srcu); - inode_node =srcu_dereference (to_tell->i_fsnotify_marks.first, - &fsnotify_mark_srcu); + inode_conn = to_tell->i_fsnotify_marks; +if (inode_conn) + inode_node =srcu_dereference (inode_conn->list.first, + &fsnotify_mark_srcu); + vfsmount_conn = mnt->mnt_fsnotify_marks; +if (vfsmount_conn) + vfsmount_node =srcu_dereference ( + vfsmount_conn->list.first, + &fsnotify_mark_srcu); }/*
@@ -293,6 +302,8 @@ int fsnotify(struct inode *to_tell, __u32 mask, const void *data, int data_is, }EXPORT_SYMBOL_GPL (fsnotify); +extern struct kmem_cache *fsnotify_mark_connector_cachep; +static __init int fsnotify_init (void ) {int ret;
@@ -303,6 +314,9 @@ static __init int fsnotify_init(void)if (ret)panic ("initializing fsnotify_mark_srcu" ); + fsnotify_mark_connector_cachep =KMEM_CACHE (fsnotify_mark_connector, +SLAB_PANIC ); +return 0 ; }core_initcall (fsnotify_init);
fs/notify/fsnotify.h +10 -7
@@ -8,6 +8,8 @@#include "../mount.h" +struct fsnotify_mark_connector ; +/* destroy all events sitting in this groups notification queue */ extern void fsnotify_flush_notify (struct fsnotify_group *group);
@@ -15,7 +17,7 @@ extern void fsnotify_flush_notify(struct fsnotify_group *group);extern struct srcu_struct fsnotify_mark_srcu;/* Calculate mask of events for a list of marks */ -extern u32 fsnotify_recalc_mask (struct hlist_head *head); +extern u32 fsnotify_recalc_mask (struct fsnotify_mark_connector *conn);/* compare two groups for sorting of marks lists */ extern int fsnotify_compare_groups(struct fsnotify_group *a,
@@ -23,10 +25,6 @@ extern int fsnotify_compare_groups(struct fsnotify_group *a,extern void fsnotify_set_inode_mark_mask_locked (struct fsnotify_mark *fsn_mark,__u32 mask); -/* Add mark to a proper place in mark list */ -extern int fsnotify_add_mark_list (struct hlist_head *head, -struct fsnotify_mark *mark, -int allow_dups);/* add a mark to an inode */ extern int fsnotify_add_inode_mark(struct fsnotify_mark *mark, struct fsnotify_group *group, struct inode *inode,
@@ -39,20 +37,25 @@ extern int fsnotify_add_vfsmount_mark(struct fsnotify_mark *mark,/* vfsmount specific destruction of a mark */ extern void fsnotify_destroy_vfsmount_mark (struct fsnotify_mark *mark);/* inode specific destruction of a mark */ -extern void fsnotify_destroy_inode_mark (struct fsnotify_mark *mark); +extern struct inode *fsnotify_destroy_inode_mark (struct fsnotify_mark *mark);/* Find mark belonging to given group in the list of marks */ extern struct fsnotify_mark *fsnotify_find_mark (struct hlist_head *head,struct fsnotify_group *group);/* Destroy all marks in the given list protected by 'lock' */ -extern void fsnotify_destroy_marks (struct hlist_head *head,spinlock_t *lock); +extern void fsnotify_destroy_marks (struct fsnotify_mark_connector **connp, +spinlock_t *lock);/* run the list of all marks associated with inode and destroy them */ static inline void fsnotify_clear_marks_by_inode (struct inode *inode) { +if (!inode->i_fsnotify_marks) +return ;fsnotify_destroy_marks (&inode->i_fsnotify_marks, &inode->i_lock); }/* run the list of all marks associated with vfsmount and destroy them */ static inline void fsnotify_clear_marks_by_mount (struct vfsmount *mnt) { +if (!real_mount (mnt)->mnt_fsnotify_marks) +return ;fsnotify_destroy_marks (&real_mount (mnt)->mnt_fsnotify_marks, &mnt->mnt_root->d_lock); }
fs/notify/inode_mark.c +14 -60
115 lines not on this page · raw message
fs/notify/mark.c +116 -41
253 lines not on this page · raw message
fs/notify/vfsmount_mark.c +9 -32
73 lines not on this page · raw message
include/linux/fs.h +3 -1
@@ -545,6 +545,8 @@ is_uncached_acl(struct posix_acl *acl)#define IOP_XATTR 0x0008#define IOP_DEFAULT_READLINK 0x0010 +struct fsnotify_mark_connector ; +/* * Keep mostly read-only and often accessed (especially for * the RCU path lookup and 'stat' data) fields at the beginning
@@ -644,7 +646,7 @@ struct inode {#ifdef CONFIG_FSNOTIFY __u32 i_fsnotify_mask;/* all events this inode cares about */ -struct hlist_head i_fsnotify_marks; +struct fsnotify_mark_connector *i_fsnotify_marks;#endif #if IS_ENABLED (CONFIG_FS_ENCRYPTION )
include/linux/fsnotify_backend.h +27 -11
@@ -194,6 +194,27 @@ struct fsnotify_group {#define FSNOTIFY_EVENT_INODE 2/* +* Inode / vfsmount point to this structure which tracks all marks attached to +* the inode / vfsmount. The reference to inode / vfsmount is held by this +* structure. We destroy this structure when there are no more marks attached +* to it. The structure is protected by fsnotify_mark_srcu. +*/ +struct fsnotify_mark_connector { +#define FSNOTIFY_OBJ_TYPE_INODE 0x01 +#define FSNOTIFY_OBJ_TYPE_VFSMOUNT 0x02 +unsigned int flags;/* Type of object [lock] */ +union {/* Object pointer [lock] */ +struct inode *inode; +struct vfsmount *mnt; + }; +union { +struct hlist_head list; +/* Used listing heads to free after srcu period expires */ +struct fsnotify_mark_connector *destroy_next; + }; +}; + +/* * A mark is simply an object attached to an in core inode which allows an * fsnotify listener to indicate they are either no longer interested in events * of a type matching mask or only interested in those events.
@@ -224,18 +245,13 @@ struct fsnotify_mark {spinlock_t lock;/* List of marks for inode / vfsmount [obj_lock] */ struct hlist_node obj_list; -union {/* Object pointer [mark->lock, group->mark_mutex] */ -struct inode *inode;/* inode this mark is associated with */ -struct vfsmount *mnt;/* vfsmount this mark is associated with */ - }; +/* Head of list of marks for an object [mark->lock, group->mark_mutex] */ +struct fsnotify_mark_connector *connector;/* Events types to ignore [mark->lock, group->mark_mutex] */ __u32 ignored_mask; -#define FSNOTIFY_MARK_FLAG_INODE 0x01 -#define FSNOTIFY_MARK_FLAG_VFSMOUNT 0x02 -#define FSNOTIFY_MARK_FLAG_OBJECT_PINNED 0x04 -#define FSNOTIFY_MARK_FLAG_IGNORED_SURV_MODIFY 0x08 -#define FSNOTIFY_MARK_FLAG_ALIVE 0x10 -#define FSNOTIFY_MARK_FLAG_ATTACHED 0x20 +#define FSNOTIFY_MARK_FLAG_IGNORED_SURV_MODIFY 0x01 +#define FSNOTIFY_MARK_FLAG_ALIVE 0x02 +#define FSNOTIFY_MARK_FLAG_ATTACHED 0x04unsigned int flags;/* flags [mark->lock] */ void (*free_mark)(struct fsnotify_mark *mark);/* called on final put+free */ };
@@ -343,7 +359,7 @@ extern void fsnotify_free_mark(struct fsnotify_mark *mark);extern void fsnotify_clear_vfsmount_marks_by_group (struct fsnotify_group *group);/* run all the marks in a group, and clear all of the inode marks */ extern void fsnotify_clear_inode_marks_by_group (struct fsnotify_group *group); -/* run all the marks in a group, and clear all of the marks where mark->flags & flags is true*/ +/* run all the marks in a group, and clear all of the marks attached to given object type */ extern void fsnotify_clear_marks_by_group_flags (struct fsnotify_group *group,unsigned int flags);extern void fsnotify_get_mark (struct fsnotify_mark *mark);extern void fsnotify_put_mark (struct fsnotify_mark *mark);
kernel/audit_tree.c +23 -7
@@ -172,10 +172,25 @@ static unsigned long inode_to_key(const struct inode *inode)/* * Function to return search key in our hash from chunk. Key 0 is special and * should never be present in the hash. +* +* Must be called with chunk->mark.lock held to protect from connector +* becoming NULL. */ +static unsigned long __chunk_to_key (struct audit_chunk *chunk) +{ +if (!chunk->mark.connector) +return 0 ; +return (unsigned long )chunk->mark.connector->inode; +} +static unsigned long chunk_to_key (struct audit_chunk *chunk) { -return (unsigned long )chunk->mark.inode; +unsigned long key; + +spin_lock (&chunk->mark.lock); + key =__chunk_to_key (chunk); +spin_unlock (&chunk->mark.lock); +return key; }static inline struct list_head *chunk_hash (unsigned long key)
@@ -187,7 +202,7 @@ static inline struct list_head *chunk_hash(unsigned long key)/* hash_lock & entry->lock is held by caller */ static void insert_hash (struct audit_chunk *chunk) { -unsigned long key =chunk_to_key (chunk); +unsigned long key =__chunk_to_key (chunk);struct list_head *list;if (!key)
@@ -248,7 +263,8 @@ static void untag_chunk(struct node *p)mutex_lock (&entry->group->mark_mutex);spin_lock (&entry->lock); -if (chunk->dead || !entry->inode) { +if (chunk->dead || !entry->connector || + !entry->connector->inode) {spin_unlock (&entry->lock);mutex_unlock (&entry->group->mark_mutex);if (new)
@@ -276,8 +292,8 @@ static void untag_chunk(struct node *p)if (!new) goto Fallback; -if (fsnotify_add_mark_locked (&new->mark, entry->group, entry->inode, -NULL ,1 )) { +if (fsnotify_add_mark_locked (&new->mark, entry->group, + entry->connector->inode,NULL ,1 )) {fsnotify_put_mark (&new->mark); goto Fallback; }
@@ -408,7 +424,7 @@ static int tag_chunk(struct inode *inode, struct audit_tree *tree)mutex_lock (&old_entry->group->mark_mutex);spin_lock (&old_entry->lock); -if (!old_entry->inode) { +if (!old_entry->connector || !old_entry->connector->inode) {/* old_entry is being shot, lets just lie */ spin_unlock (&old_entry->lock);mutex_unlock (&old_entry->group->mark_mutex);
@@ -418,7 +434,7 @@ static int tag_chunk(struct inode *inode, struct audit_tree *tree) }if (fsnotify_add_mark_locked (chunk_entry, old_entry->group, - old_entry->inode,NULL ,1 )) { + old_entry->connector->inode,NULL ,1 )) {spin_unlock (&old_entry->lock);mutex_unlock (&old_entry->group->mark_mutex);fsnotify_put_mark (chunk_entry);
kernel/auditsc.c +2 -2
@@ -1591,7 +1591,7 @@ static inline void handle_one(const struct inode *inode)struct audit_tree_refs *p;struct audit_chunk *chunk;int count; -if (likely (hlist_empty (&inode->i_fsnotify_marks))) +if (likely (!inode->i_fsnotify_marks))return ; context = current->audit_context; p = context->trees;
@@ -1634,7 +1634,7 @@ static void handle_path(const struct dentry *dentry) seq =read_seqbegin (&rename_lock);for (;;) {struct inode *inode =d_backing_inode (d); -if (inode &&unlikely (!hlist_empty (&inode->i_fsnotify_marks))) { +if (inode &&unlikely (inode->i_fsnotify_marks)) {struct audit_chunk *chunk; chunk =audit_tree_lookup (inode);if (chunk) {
Signature · 1 line
2.10.2
Re: [PATCH 06/25] fsnotify: Attach marks to object via dedicated head structure
Quoted 27 lines · Jan Kara
Currently notification marks are attached to object (inode or vfsmnt) by a
hlist_headin the object. The list is also protected by a spinlock in the object. So while there is any mark attached to the list of marks, the object must be pinned in memory (and thus e.g. lastiput()deleting inode cannot happen). Also for list iteration infsnotify()to work, we must holdfsnotify_mark_srculock so that mark itself andmark->obj_list.nextcannot get freed. Thus we are required to wait for response to fanotify events from userspace process withfsnotify_mark_srculock held. That causes issues when userspace process is buggy and does not reply to some event - basically the whole notification subsystem gets eventually stuck.So to be able to drop
fsnotify_mark_srculock while waiting for response, we have to pin the mark in memory and make sure it stays in the object list (as removing the mark waiting for response could lead to lost notification events for groups later in the list). However we don't want inode reclaim to block on such mark as that would lead to system just locking up elsewhere.This commit tries to pave a way towards solving these conflicting lifetime needs. Instead of anchoring the list of marks directly in the object, we anchor it in a dedicated structure (
fsnotify_mark_connector) and just point to that structure from the object. In the following patch we will also protect the list by a spinlock contained in that structure. With this, we will be able to detach notification marks from object without having to modify the list itself.
Quoted 2 lines
Signed-off-by Jan Kara <[email protected]>---
The patch split was useful for review on one hand, but it generated some weird looking functions like fsnotify_destroy_marks() and grab_connector(), which whose name does not accurately reflect what they do in the context of this patch.
However, since those are totally reshuffled in next patches, you may re-add
Re: [PATCH 06/25] fsnotify: Attach marks to object via dedicated head structure
In reply to Jan KaraQuoted 19 lines · Jan Kara
Currently notification marks are attached to object (inode or vfsmnt) by a
hlist_headin the object. The list is also protected by a spinlock in the object. So while there is any mark attached to the list of marks, the object must be pinned in memory (and thus e.g. lastiput()deleting inode cannot happen). Also for list iteration infsnotify()to work, we must holdfsnotify_mark_srculock so that mark itself andmark->obj_list.nextcannot get freed. Thus we are required to wait for response to fanotify events from userspace process withfsnotify_mark_srculock held. That causes issues when userspace process is buggy and does not reply to some event - basically the whole notification subsystem gets eventually stuck.So to be able to drop
fsnotify_mark_srculock while waiting for response, we have to pin the mark in memory and make sure it stays in the object list (as removing the mark waiting for response could lead to lost notification events for groups later in the list). However we don't want inode reclaim to block on such mark as that would lead to system just locking up elsewhere.
Quoted 7 lines
This commit tries to pave a way towards solving these conflicting lifetime needs. Instead of anchoring the list of marks directly in the object, we anchor it in a dedicated structure (
fsnotify_mark_connector) and just point to that structure from the object. In the following patch we will also protect the list by a spinlock contained in that structure. With this, we will be able to detach notification marks from object without having to modify the list itself.
This is still a bit big to review easily. I'd suggest further splitup into:
- move hlist_head from inode/mnt into connector
- move inode/mnt ptr from mark into connector (no refcounting changes)
- inode refcounting cleanup
More comments inline.
Quoted 65 lines
Signed-off-by Jan Kara <[email protected]>--- fs/inode.c | 3 - fs/mount.h | 2 +- fs/namespace.c | 3 - fs/notify/dnotify/dnotify.c | 4 +- fs/notify/fdinfo.c | 12 +-- fs/notify/fsnotify.c | 32 +++++--- fs/notify/fsnotify.h | 17 +++-- fs/notify/inode_mark.c | 74 ++++-------------- fs/notify/mark.c | 157 +++++++++++++++++++++++++++++---------- fs/notify/vfsmount_mark.c | 41 +++------- include/linux/fs.h | 4 +- include/linux/fsnotify_backend.h | 38 +++++++--- kernel/audit_tree.c | 30 ++++++-- kernel/auditsc.c | 4 +- 14 files changed, 236 insertions(+), 185 deletions(-)fs/inode.c +0
-3@@ -371,9 +371,6 @@ void inode_init_once(struct inode *inode)INIT_LIST_HEAD (&inode->i_lru);address_space_init_once (&inode->i_data);i_size_ordered_init (inode); -#ifdef CONFIG_FSNOTIFY -INIT_HLIST_HEAD (&inode->i_fsnotify_marks); -#endif }EXPORT_SYMBOL (inode_init_once);fs/mount.h +1
-1@@ -59,7 +59,7 @@ struct mount {struct mountpoint *mnt_mp;/* where is it mounted */ struct hlist_node mnt_mp_list;/* list mounts with the same mountpoint */ #ifdef CONFIG_FSNOTIFY -struct hlist_head mnt_fsnotify_marks; +struct fsnotify_mark_connector *mnt_fsnotify_marks;__u32 mnt_fsnotify_mask;#endif int mnt_id;/* mount identifier */ fs/namespace.c +0
-3@@ -233,9 +233,6 @@ static struct mount *alloc_vfsmnt(const char *name)INIT_LIST_HEAD (&mnt->mnt_slave_list);INIT_LIST_HEAD (&mnt->mnt_slave);INIT_HLIST_NODE (&mnt->mnt_mp_list); -#ifdef CONFIG_FSNOTIFY -INIT_HLIST_HEAD (&mnt->mnt_fsnotify_marks); -#endif init_fs_pin (&mnt->mnt_umount, drop_mountpoint); }return mnt;fs/notify/dnotify/dnotify.c +0
-0
fs/notify/dnotify/dnotify.c static void dnotify_recalc_inode_mask(struct fsnotify_mark *fsn_mark) · line 72
Quoted 6 lines
fs/notify/dnotify/dnotify.c +1
-2@@ -69,8 +69,8 @@ static void dnotify_recalc_inode_mask(struct fsnotify_mark *fsn_mark)if (old_mask == new_mask)return ; -if (fsn_mark->inode) -fsnotify_recalc_inode_mask (fsn_mark->inode); + if (fsn_mark->connector->inode
This doesn't make sense in the context of this patch. Now mark->connector is set to NULL where previously mark->inode was set to
null. So the right condition would be "if (fsn_mark->connector)
...".
)
Quoted 57 lines
fs/notify/dnotify/dnotify.c +1
-0+fsnotify_recalc_inode_mask (fsn_mark->connector->inode); }/* fs/notify/fdinfo.c +6
-6@@ -76,11 +76,11 @@ static void inotify_fdinfo(struct seq_file *m, struct fsnotify_mark *mark)struct inotify_inode_mark *inode_mark;struct inode *inode; -if (!(mark->flags &FSNOTIFY_MARK_FLAG_INODE )) +if (!(mark->connector->flags &FSNOTIFY_OBJ_TYPE_INODE ))return ; inode_mark = container_of(mark, struct inotify_inode_mark, fsn_mark); - inode =igrab (mark->inode); + inode =igrab (mark->connector->inode);if (inode) {/* * IN_ALL_EVENTS represents all of the mask bits @@ -115,8 +115,8 @@ static void fanotify_fdinfo(struct seq_file *m, struct fsnotify_mark *mark)if (mark->flags &FSNOTIFY_MARK_FLAG_IGNORED_SURV_MODIFY ) mflags |=FAN_MARK_IGNORED_SURV_MODIFY ; -if (mark->flags &FSNOTIFY_MARK_FLAG_INODE ) { - inode =igrab (mark->inode); +if (mark->connector->flags &FSNOTIFY_OBJ_TYPE_INODE ) { + inode =igrab (mark->connector->inode);if (!inode)return ; seq_printf(m, "fanotify ino:%lx sdev:%x mflags:%x mask:%x ignored_mask:%x ",@@ -125,8 +125,8 @@ static void fanotify_fdinfo(struct seq_file *m, struct fsnotify_mark *mark)show_mark_fhandle (m, inode);seq_putc (m,'\n' );iput (inode); - }else if (mark->flags &FSNOTIFY_MARK_FLAG_VFSMOUNT ) { -struct mount *mnt =real_mount (mark->mnt); + }else if (mark->connector->flags &FSNOTIFY_OBJ_TYPE_VFSMOUNT ) { +struct mount *mnt =real_mount (mark->connector->mnt);seq_printf (m,"fanotify mnt_id:%x mflags:%x mask:%x ignored_mask:%x\n" , mnt->mnt_id, mflags, mark->mask, mark->ignored_mask);fs/notify/fsnotify.c +1
-0@@ -193,6 +193,7 @@ int fsnotify(struct inode *to_tell, __u32 mask, const void *data, int data_is,struct hlist_node *inode_node =NULL , *vfsmount_node =NULL ;struct fsnotify_mark *inode_mark =NULL , *vfsmount_mark =NULL ;struct fsnotify_group *inode_group, *vfsmount_group; +struct fsnotify_mark_connector *inode_conn, *vfsmount_conn;struct mount *mnt;int idx, ret =0 ;/* global tests shouldn't care about events on child only the specific event */
fs/notify/fsnotify.c int fsnotify(struct inode *to_tell, __u32 mask, const void *data, int data_is, · line 215
Quoted 7 lines
fs/notify/fsnotify.c +2
-2@@ -210,8 +211,8 @@ int fsnotify(struct inode *to_tell, __u32 mask, const void *data, int data_is,* SRCU because we have no references to any objects and do not * need SRCU to keep them "alive". */ -if (hlist_empty (&to_tell->i_fsnotify_marks) && - (!mnt ||hlist_empty (&mnt->mnt_fsnotify_marks))) +if (!to_tell->i_fsnotify_marks && + (!mnt || !mnt->mnt_fsnotify_marks))
The checks against hlist_empty() could still be useful to optimize away the memory barrier imposed by srcu_read_lock() in case there were marks on the object but not anymore (i.e. the connector is non-NULL, but the hlist is empty).
fs/notify/fsnotify.c int fsnotify(struct inode *to_tell, __u32 mask, const void *data, int data_is, · line 231
Quoted 12 lines
fs/notify/fsnotify.c +2
-3return 0 ;/* * if this is a modify event we may need to clear the ignored masks @@ -226,16 +227,24 @@ int fsnotify(struct inode *to_tell, __u32 mask, const void *data, int data_is, idx =srcu_read_lock (&fsnotify_mark_srcu);if ((mask &FS_MODIFY ) || - (test_mask & to_tell->i_fsnotify_mask)) - inode_node =srcu_dereference (to_tell->i_fsnotify_marks.first, - &fsnotify_mark_srcu); + (test_mask & to_tell->i_fsnotify_mask)) { + inode_conn = to_tell->i_fsnotify_marks;
Needs lockless_dereference().
Quoted 6 lines
fs/notify/fsnotify.c +4
-0+if (inode_conn) + inode_node =srcu_dereference (inode_conn->list.first, + &fsnotify_mark_srcu); + } if (mnt && ((mask & FS_MODIFY) ||
fs/notify/fsnotify.c int fsnotify(struct inode *to_tell, __u32 mask, const void *data, int data_is, · line 243
Quoted 10 lines
fs/notify/fsnotify.c +5
-4(test_mask & mnt->mnt_fsnotify_mask))) { - vfsmount_node =srcu_dereference (mnt->mnt_fsnotify_marks.first, - &fsnotify_mark_srcu); - inode_node =srcu_dereference (to_tell->i_fsnotify_marks.first, - &fsnotify_mark_srcu); + inode_conn = to_tell->i_fsnotify_marks; +if (inode_conn) + inode_node =srcu_dereference (inode_conn->list.first, + &fsnotify_mark_srcu); + vfsmount_conn = mnt->mnt_fsnotify_marks;
And this one too.
Quoted 67 lines
fs/notify/fsnotify.c +9
-0+if (vfsmount_conn) + vfsmount_node =srcu_dereference ( + vfsmount_conn->list.first, + &fsnotify_mark_srcu); }/* @@ -293,6 +302,8 @@ int fsnotify(struct inode *to_tell, __u32 mask, const void *data, int data_is, }EXPORT_SYMBOL_GPL (fsnotify); +extern struct kmem_cache *fsnotify_mark_connector_cachep; +static __init int fsnotify_init (void ) {int ret;@@ -303,6 +314,9 @@ static __init int fsnotify_init(void)if (ret)panic ("initializing fsnotify_mark_srcu" ); + fsnotify_mark_connector_cachep =KMEM_CACHE (fsnotify_mark_connector, +SLAB_PANIC ); +return 0 ; }core_initcall (fsnotify_init);fs/notify/fsnotify.h +4
-6@@ -8,6 +8,8 @@#include "../mount.h" +struct fsnotify_mark_connector ; +/* destroy all events sitting in this groups notification queue */ extern void fsnotify_flush_notify (struct fsnotify_group *group);@@ -15,7 +17,7 @@ extern void fsnotify_flush_notify(struct fsnotify_group *group);extern struct srcu_struct fsnotify_mark_srcu;/* Calculate mask of events for a list of marks */ -extern u32 fsnotify_recalc_mask (struct hlist_head *head); +extern u32 fsnotify_recalc_mask (struct fsnotify_mark_connector *conn);/* compare two groups for sorting of marks lists */ extern int fsnotify_compare_groups(struct fsnotify_group *a,@@ -23,10 +25,6 @@ extern int fsnotify_compare_groups(struct fsnotify_group *a,extern void fsnotify_set_inode_mark_mask_locked (struct fsnotify_mark *fsn_mark,__u32 mask); -/* Add mark to a proper place in mark list */ -extern int fsnotify_add_mark_list (struct hlist_head *head, -struct fsnotify_mark *mark, -int allow_dups);/* add a mark to an inode */ extern int fsnotify_add_inode_mark(struct fsnotify_mark *mark, struct fsnotify_group *group, struct inode *inode,@@ -39,20 +37,25 @@ extern int fsnotify_add_vfsmount_mark(struct fsnotify_mark *mark,/* vfsmount specific destruction of a mark */ extern void fsnotify_destroy_vfsmount_mark (struct fsnotify_mark *mark);/* inode specific destruction of a mark */ -extern void fsnotify_destroy_inode_mark (struct fsnotify_mark *mark); +extern struct inode *fsnotify_destroy_inode_mark (struct fsnotify_mark *mark);/* Find mark belonging to given group in the list of marks */ extern struct fsnotify_mark *fsnotify_find_mark(struct hlist_head *head,
fs/notify/fsnotify.h extern int fsnotify_add_vfsmount_mark(struct fsnotify_mark *mark, · line 51
Quoted 10 lines
fs/notify/fsnotify.h +4
-1struct fsnotify_group *group);/* Destroy all marks in the given list protected by 'lock' */ -extern void fsnotify_destroy_marks (struct hlist_head *head,spinlock_t *lock); +extern void fsnotify_destroy_marks (struct fsnotify_mark_connector **connp, +spinlock_t *lock);/* run the list of all marks associated with inode and destroy them */ static inline void fsnotify_clear_marks_by_inode (struct inode *inode) { +if (!inode->i_fsnotify_marks) +return ;
Should move the above check inside fsnotify_destroy_marks(). Locking rules also dictate that the check be done under inode lock, although that may not matter in practice.
fs/notify/fsnotify.h extern int fsnotify_add_vfsmount_mark(struct fsnotify_mark *mark, · line 58
Quoted 7 lines
fs/notify/fsnotify.h +2
-0fsnotify_destroy_marks (&inode->i_fsnotify_marks, &inode->i_lock); }/* run the list of all marks associated with vfsmount and destroy them */ static inline void fsnotify_clear_marks_by_mount (struct vfsmount *mnt) { +if (!real_mount (mnt)->mnt_fsnotify_marks) +return ;
And the same here.
Quoted 82 lines
fs/notify/fsnotify.h +0
-0fsnotify_destroy_marks (&real_mount (mnt)->mnt_fsnotify_marks, &mnt->mnt_root->d_lock); }fs/notify/inode_mark.c +10
-27@@ -37,15 +37,16 @@void fsnotify_recalc_inode_mask (struct inode *inode) {spin_lock (&inode->i_lock); - inode->i_fsnotify_mask =fsnotify_recalc_mask (&inode->i_fsnotify_marks); + inode->i_fsnotify_mask =fsnotify_recalc_mask (inode->i_fsnotify_marks);spin_unlock (&inode->i_lock);__fsnotify_update_child_dentry_flags (inode); } -void fsnotify_destroy_inode_mark (struct fsnotify_mark *mark) +struct inode *fsnotify_destroy_inode_mark (struct fsnotify_mark *mark) { -struct inode *inode = mark->inode; +struct inode *inode = mark->connector->inode; +bool empty;BUG_ON (!mutex_is_locked (&mark->group->mark_mutex));assert_spin_locked (&mark->lock);@@ -53,15 +54,18 @@ void fsnotify_destroy_inode_mark(struct fsnotify_mark *mark)spin_lock (&inode->i_lock);hlist_del_init_rcu (&mark->obj_list); - mark->inode =NULL ; + empty =hlist_empty (&mark->connector->list); + mark->connector =NULL ;/* * this mark is now off the inode->i_fsnotify_marks list and we * hold the inode->i_lock, so this is the perfect time to update the * inode->i_fsnotify_mask */ - inode->i_fsnotify_mask =fsnotify_recalc_mask (&inode->i_fsnotify_marks); + inode->i_fsnotify_mask =fsnotify_recalc_mask (inode->i_fsnotify_marks);spin_unlock (&inode->i_lock); + +return empty ? inode :NULL ; }/* @@ -69,7 +73,7 @@ void fsnotify_destroy_inode_mark(struct fsnotify_mark *mark)*/ void fsnotify_clear_inode_marks_by_group (struct fsnotify_group *group) { -fsnotify_clear_marks_by_group_flags (group,FSNOTIFY_MARK_FLAG_INODE ); +fsnotify_clear_marks_by_group_flags (group,FSNOTIFY_OBJ_TYPE_INODE ); }/* @@ -81,64 +85,14 @@ struct fsnotify_mark *fsnotify_find_inode_mark(struct fsnotify_group *group, {struct fsnotify_mark *mark; -spin_lock (&inode->i_lock); - mark =fsnotify_find_mark (&inode->i_fsnotify_marks, group); -spin_unlock (&inode->i_lock); - -return mark; -} - -/* -* If we are setting a mark mask on an inode mark we should pin the inode -* in memory. -*/ -void fsnotify_set_inode_mark_mask_locked (struct fsnotify_mark *mark, -__u32 mask) -{ -struct inode *inode; - -assert_spin_locked (&mark->lock); - -if (mask && - mark->inode && - !(mark->flags &FSNOTIFY_MARK_FLAG_OBJECT_PINNED )) {
fs/notify/inode_mark.c struct fsnotify_mark *fsnotify_find_inode_mark(struct fsnotify_group *group, · line 114
Quoted 10 lines
fs/notify/inode_mark.c +0
-10- mark->flags |=FSNOTIFY_MARK_FLAG_OBJECT_PINNED ; - inode =igrab (mark->inode); -/* -* we shouldn't be able to get here if the inode wasn't -* already safely held in memory. But bug in case it -* ever is wrong. -*/ -BUG_ON (!inode); - } -}
I think the above warrants a separate patch. This patch subtly changes the rules of grabbing the inode reference and it's not
explained why this is okay. Removing whole functions also confuses
the patch itself.
Quoted 10 lines
fs/notify/inode_mark.c +0
-10- -/* -* Attach an initialized mark to a given inode. -* These marks may be used for the fsnotify backend to determine which -* event types should be delivered to which group and for which inodes. These -* marks are ordered according to priority, highest number first, and then by -* the group's location in memory. -*/ -int fsnotify_add_inode_mark(struct fsnotify_mark *mark, - struct fsnotify_group *group, struct inode *inode,
fs/notify/inode_mark.c struct fsnotify_mark *fsnotify_find_inode_mark(struct fsnotify_group *group, · line 89
Quoted 10 lines
fs/notify/inode_mark.c +2
-8- int allow_dups) -{ -int ret; - - mark->flags |=FSNOTIFY_MARK_FLAG_INODE ; - -BUG_ON (!mutex_is_locked (&group->mark_mutex)); -assert_spin_locked (&mark->lock); +if (!inode->i_fsnotify_marks) +return NULL ;
Check in fsnotify_find_mark()?
Quoted 57 lines
fs/notify/inode_mark.c +2
-5spin_lock (&inode->i_lock); - mark->inode = inode; - ret =fsnotify_add_mark_list (&inode->i_fsnotify_marks, mark, - allow_dups); - inode->i_fsnotify_mask =fsnotify_recalc_mask (&inode->i_fsnotify_marks); + mark =fsnotify_find_mark (&inode->i_fsnotify_marks->list, group);spin_unlock (&inode->i_lock); -return ret; +return mark; }/** fs/notify/mark.c +7
-6@@ -83,6 +83,8 @@#define FSNOTIFY_REAPER_DELAY (1)/* 1 jiffy */ struct srcu_struct fsnotify_mark_srcu; +struct kmem_cache *fsnotify_mark_connector_cachep; +static DEFINE_SPINLOCK (destroy_lock);static LIST_HEAD (destroy_list);@@ -104,12 +106,12 @@ void fsnotify_put_mark(struct fsnotify_mark *mark) }/* Calculate mask of events for a list of marks */ -u32 fsnotify_recalc_mask (struct hlist_head *head) +u32 fsnotify_recalc_mask (struct fsnotify_mark_connector *conn) {u32 new_mask =0 ;struct fsnotify_mark *mark; -hlist_for_each_entry (mark, head, obj_list) +hlist_for_each_entry (mark, &conn->list, obj_list) new_mask |= mark->mask;return new_mask; }@@ -137,10 +139,9 @@ void fsnotify_detach_mark(struct fsnotify_mark *mark) mark->flags &= ~FSNOTIFY_MARK_FLAG_ATTACHED ; -if (mark->flags &FSNOTIFY_MARK_FLAG_INODE ) { - inode = mark->inode; -fsnotify_destroy_inode_mark (mark); - }else if (mark->flags &FSNOTIFY_MARK_FLAG_VFSMOUNT ) +if (mark->connector->flags &FSNOTIFY_OBJ_TYPE_INODE ) + inode =fsnotify_destroy_inode_mark (mark); +else if (mark->connector->flags &FSNOTIFY_OBJ_TYPE_VFSMOUNT )fsnotify_destroy_vfsmount_mark (mark);else BUG ();
fs/notify/mark.c void fsnotify_detach_mark(struct fsnotify_mark *mark) · line 159
Quoted 5 lines
fs/notify/mark.c +1
-1@@ -155,7 +156,7 @@ void fsnotify_detach_mark(struct fsnotify_mark *mark)spin_unlock (&mark->lock); -if (inode && (mark->flags &FSNOTIFY_MARK_FLAG_OBJECT_PINNED )) +if (inode)
And this should also be part of the inode refcounting change.
Quoted 30 lines
fs/notify/mark.c +5
-3iput (inode);atomic_dec (&group->num_marks);@@ -220,7 +221,8 @@ void fsnotify_destroy_mark(struct fsnotify_mark *mark,fsnotify_free_mark (mark); } -void fsnotify_destroy_marks (struct hlist_head *head,spinlock_t *lock) +void fsnotify_destroy_marks (struct fsnotify_mark_connector **connp, +spinlock_t *lock) {struct fsnotify_mark *mark;@@ -233,11 +235,12 @@ void fsnotify_destroy_marks(struct hlist_head *head, spinlock_t *lock)* calling fsnotify_destroy_mark() more than once is fine. */ spin_lock (lock); -if (hlist_empty (head)) { +if (hlist_empty (&(*connp)->list)) {spin_unlock (lock);break ; } - mark = hlist_entry(head->first, struct fsnotify_mark, obj_list); + mark = hlist_entry((*connp)->list.first, struct fsnotify_mark, + obj_list);/* * We don't update i_fsnotify_mask / mnt_fsnotify_mask here * since inode / mount is going away anyway. So just remove @@ -249,6 +252,16 @@ void fsnotify_destroy_marks(struct hlist_head *head, spinlock_t *lock)fsnotify_destroy_mark (mark, mark->group);
fs/notify/mark.c void fsnotify_destroy_marks(struct hlist_head *head, spinlock_t *lock) · line 262
Quoted 10 lines
fs/notify/mark.c +8
-0fsnotify_put_mark (mark); } +/* +* Skip freeing of connector when inode is just unlinked. We leave that +* until destroy_inode() is called at which moment nobody will be +* looking at the inode anymore. +*/ +if ((*connp)->flags &FSNOTIFY_OBJ_TYPE_INODE && + !((*connp)->inode->i_state &I_CLEAR )) +return ;
This is ugly. Why not rather add a fsnotify_connector_free() and
call that explicitly on inode/vfsmount destruction?
fs/notify/mark.c void fsnotify_set_mark_mask_locked(struct fsnotify_mark *mark, __u32 mask) · line 261
Quoted 12 lines
fs/notify/mark.c +2
-3+kmem_cache_free (fsnotify_mark_connector_cachep, *connp); + *connp =NULL ; }void fsnotify_set_mark_mask_locked (struct fsnotify_mark *mark,__u32 mask)@@ -256,9 +269,6 @@ void fsnotify_set_mark_mask_locked(struct fsnotify_mark *mark, __u32 mask)assert_spin_locked (&mark->lock); mark->mask = mask; - -if (mark->flags &FSNOTIFY_MARK_FLAG_INODE ) -fsnotify_set_inode_mark_mask_locked (mark, mask);
Also related to inode refcounting?
Quoted 17 lines
fs/notify/mark.c +7
-3}void fsnotify_set_mark_ignored_mask_locked (struct fsnotify_mark *mark,__u32 mask)@@ -304,37 +314,111 @@ int fsnotify_compare_groups(struct fsnotify_group *a, struct fsnotify_group *b)return -1 ; } -/* Add mark into proper place in given list of marks */ -int fsnotify_add_mark_list (struct hlist_head *head,struct fsnotify_mark *mark, -int allow_dups) +/* +* Get mark connector, make sure it is alive and return with its lock held. +* This is for users that get connector pointer from inode or mount. Users that +* hold reference to a mark on the list may directly lock connector->lock as +* they are sure list cannot go away under them. +*/ +static struct fsnotify_mark_connector *fsnotify_grab_connector(
fs/notify/mark.c int fsnotify_compare_groups(struct fsnotify_group *a, struct fsnotify_group *b) · line 333
Quoted 10 lines
fs/notify/mark.c +10
-0+ struct fsnotify_mark_connector *conn) +{ +if (!conn) +return NULL ; +if (conn->flags &FSNOTIFY_OBJ_TYPE_INODE ) +spin_lock (&conn->inode->i_lock); +else if (conn->flags &FSNOTIFY_OBJ_TYPE_VFSMOUNT ) +spin_lock (&conn->mnt->mnt_root->d_lock); +else +return NULL ;
Can this happen? If not, then add a WARNING().
Quoted 18 lines
fs/notify/mark.c +18
-0+return conn; +} + +static void fsnotify_put_connector (struct fsnotify_mark_connector *conn) +{ +if (conn->flags &FSNOTIFY_OBJ_TYPE_INODE ) +spin_unlock (&conn->inode->i_lock); +else if (conn->flags &FSNOTIFY_OBJ_TYPE_VFSMOUNT ) +spin_unlock (&conn->mnt->mnt_root->d_lock); +} + +/* +* Add mark into proper place in given list of marks. These marks may be used +* for the fsnotify backend to determine which event types should be delivered +* to which group and for which inodes. These marks are ordered according to +* priority, highest number first, and then by the group's location in memory. +*/ +static int fsnotify_add_mark_list(struct fsnotify_mark *mark,
fs/notify/mark.c int fsnotify_compare_groups(struct fsnotify_group *a, struct fsnotify_group *b) · line 361
Quoted 10 lines
fs/notify/mark.c +7
-0+ struct inode *inode, struct vfsmount *mnt, + int allow_dups) { struct fsnotify_mark *lmark, *last = NULL; +struct fsnotify_mark_connector *conn; +struct fsnotify_mark_connector **connp;int cmp; +int err =0 ; + +BUG_ON (!inode && !mnt);
Use WARN_ON() instead.
fs/notify/mark.c int fsnotify_compare_groups(struct fsnotify_group *a, struct fsnotify_group *b) · line 368
Quoted 7 lines
fs/notify/mark.c +7
-0+if (inode) + connp = &inode->i_fsnotify_marks; +else + connp = &real_mount (mnt)->mnt_fsnotify_marks; +restart: +spin_lock (&mark->lock); + conn =fsnotify_grab_connector (*connp);
This looks a bit too subtle. We have the inode/vfsmount. We have the matching lock. Could store the address of the lock in a local variable and use that for locking.
Quoted 8 lines
fs/notify/mark.c +8
-0+if (!conn) { +spin_unlock (&mark->lock); + conn =kmem_cache_alloc (fsnotify_mark_connector_cachep, +GFP_KERNEL ); +if (!conn) +return -ENOMEM ; +INIT_HLIST_HEAD (&conn->list); +if (inode) {
fs/notify/mark.c int fsnotify_compare_groups(struct fsnotify_group *a, struct fsnotify_group *b) · line 386
Quoted 10 lines
fs/notify/mark.c +10
-0+ conn->flags =FSNOTIFY_OBJ_TYPE_INODE ; + conn->inode = inode; + }else { + conn->flags =FSNOTIFY_OBJ_TYPE_VFSMOUNT ; + conn->mnt = mnt; + } +if (cmpxchg (connp,NULL , conn)) { +/* Someone else created list structure for us */ +kmem_cache_free (fsnotify_mark_connector_cachep, conn); + }
Setting *connp should be done with inode/vfsmount lock already regained (if I understand the locking rules correctly) and then no need to have cmpxchg magic in there.
fs/notify/mark.c int fsnotify_compare_groups(struct fsnotify_group *a, struct fsnotify_group *b) · line 393
Quoted 10 lines
fs/notify/mark.c +5
-3+ goto restart; + }/* is mark the first mark? */ -if (hlist_empty (head)) { -hlist_add_head_rcu (&mark->obj_list, head); -return 0 ; +if (hlist_empty (&conn->list)) { +hlist_add_head_rcu (&mark->obj_list, &conn->list); +igrab (inode);
And this is again an inode grabbing change
Quoted 23 lines
fs/notify/mark.c +7
-4+ goto added; }/* should mark be in the middle of the current list? */ -hlist_for_each_entry (lmark, head, obj_list) { +hlist_for_each_entry (lmark, &conn->list, obj_list) { last = lmark; -if ((lmark->group == mark->group) && !allow_dups) -return -EEXIST ; +if ((lmark->group == mark->group) && !allow_dups) { + err = -EEXIST ; + goto out_err; + } cmp =fsnotify_compare_groups (lmark->group, mark->group);if (cmp >=0 ) {hlist_add_before_rcu (&mark->obj_list, &lmark->obj_list); -return 0 ; + goto added; } }
fs/notify/mark.c int fsnotify_compare_groups(struct fsnotify_group *a, struct fsnotify_group *b) · line 421
Quoted 10 lines
fs/notify/mark.c +6
-1BUG_ON (last ==NULL );/* mark should be the last entry. last is the current last entry */ hlist_add_behind_rcu (&mark->obj_list, &last->obj_list); -return 0 ; +added: + mark->connector = conn; +out_err: +fsnotify_put_connector (conn); +spin_unlock (&mark->lock); +return err;
And AFAICS, this is also due to the inode grabbing change.
Quoted 105 lines
fs/notify/mark.c +8
-17}/* @@ -366,25 +450,16 @@ int fsnotify_add_mark_locked(struct fsnotify_mark *mark,list_add (&mark->g_list, &group->marks_list);atomic_inc (&group->num_marks);fsnotify_get_mark (mark);/* for i_list and g_list */ - -if (inode) { - ret =fsnotify_add_inode_mark (mark, group, inode, allow_dups); -if (ret) - goto err; - }else if (mnt) { - ret =fsnotify_add_vfsmount_mark (mark, group, mnt, allow_dups); -if (ret) - goto err; - }else { -BUG (); - } - -/* this will pin the object if appropriate */ -fsnotify_set_mark_mask_locked (mark, mark->mask);spin_unlock (&mark->lock); + ret =fsnotify_add_mark_list (mark, inode, mnt, allow_dups); +if (ret) + goto err; +if (inode) -__fsnotify_update_child_dentry_flags (inode); +fsnotify_recalc_inode_mask (inode); +else +fsnotify_recalc_vfsmount_mask (mnt);return ret; err:@@ -453,7 +528,7 @@ void fsnotify_clear_marks_by_group_flags(struct fsnotify_group *group,*/ mutex_lock_nested (&group->mark_mutex,SINGLE_DEPTH_NESTING );list_for_each_entry_safe (mark, lmark, &group->marks_list, g_list) { -if (mark->flags & flags) +if (mark->connector->flags & flags)list_move (&mark->g_list, &to_free); }mutex_unlock (&group->mark_mutex);fs/notify/vfsmount_mark.c +5
-20@@ -31,7 +31,7 @@void fsnotify_clear_vfsmount_marks_by_group (struct fsnotify_group *group) { -fsnotify_clear_marks_by_group_flags (group,FSNOTIFY_MARK_FLAG_VFSMOUNT ); +fsnotify_clear_marks_by_group_flags (group,FSNOTIFY_OBJ_TYPE_VFSMOUNT ); }/* @@ -43,13 +43,13 @@ void fsnotify_recalc_vfsmount_mask(struct vfsmount *mnt)struct mount *m =real_mount (mnt);spin_lock (&mnt->mnt_root->d_lock); - m->mnt_fsnotify_mask =fsnotify_recalc_mask (&m->mnt_fsnotify_marks); + m->mnt_fsnotify_mask =fsnotify_recalc_mask (m->mnt_fsnotify_marks);spin_unlock (&mnt->mnt_root->d_lock); }void fsnotify_destroy_vfsmount_mark (struct fsnotify_mark *mark) { -struct vfsmount *mnt = mark->mnt; +struct vfsmount *mnt = mark->connector->mnt;struct mount *m =real_mount (mnt);BUG_ON (!mutex_is_locked (&mark->group->mark_mutex));@@ -58,9 +58,9 @@ void fsnotify_destroy_vfsmount_mark(struct fsnotify_mark *mark)spin_lock (&mnt->mnt_root->d_lock);hlist_del_init_rcu (&mark->obj_list); - mark->mnt =NULL ; + mark->connector =NULL ; - m->mnt_fsnotify_mask =fsnotify_recalc_mask (&m->mnt_fsnotify_marks); + m->mnt_fsnotify_mask =fsnotify_recalc_mask (m->mnt_fsnotify_marks);spin_unlock (&mnt->mnt_root->d_lock); }@@ -74,35 +74,12 @@ struct fsnotify_mark *fsnotify_find_vfsmount_mark(struct fsnotify_group *group,struct mount *m =real_mount (mnt);struct fsnotify_mark *mark; -spin_lock (&mnt->mnt_root->d_lock); - mark =fsnotify_find_mark (&m->mnt_fsnotify_marks, group); -spin_unlock (&mnt->mnt_root->d_lock); - -return mark; -} - -/* -* Attach an initialized mark to a given group and vfsmount. -* These marks may be used for the fsnotify backend to determine which -* event types should be delivered to which groups. -*/ -int fsnotify_add_vfsmount_mark (struct fsnotify_mark *mark, -struct fsnotify_group *group,struct vfsmount *mnt, -int allow_dups)
fs/notify/vfsmount_mark.c struct fsnotify_mark *fsnotify_find_vfsmount_mark(struct fsnotify_group *group, · line 78
Quoted 10 lines
fs/notify/vfsmount_mark.c +2
-8-{ -struct mount *m =real_mount (mnt); -int ret; - - mark->flags |=FSNOTIFY_MARK_FLAG_VFSMOUNT ; - -BUG_ON (!mutex_is_locked (&group->mark_mutex)); -assert_spin_locked (&mark->lock); +if (!m->mnt_fsnotify_marks) +return NULL ;
Move check to fsnotify_find_mark()?
Quoted 48 lines
fs/notify/vfsmount_mark.c +2
-4spin_lock (&mnt->mnt_root->d_lock); - mark->mnt = mnt; - ret =fsnotify_add_mark_list (&m->mnt_fsnotify_marks, mark, allow_dups); - m->mnt_fsnotify_mask =fsnotify_recalc_mask (&m->mnt_fsnotify_marks); + mark =fsnotify_find_mark (&m->mnt_fsnotify_marks->list, group);spin_unlock (&mnt->mnt_root->d_lock); -return ret; +return mark; }include/linux/fs.h +3
-1@@ -545,6 +545,8 @@ is_uncached_acl(struct posix_acl *acl)#define IOP_XATTR 0x0008#define IOP_DEFAULT_READLINK 0x0010 +struct fsnotify_mark_connector ; +/* * Keep mostly read-only and often accessed (especially for * the RCU path lookup and 'stat' data) fields at the beginning @@ -644,7 +646,7 @@ struct inode {#ifdef CONFIG_FSNOTIFY __u32 i_fsnotify_mask;/* all events this inode cares about */ -struct hlist_head i_fsnotify_marks; +struct fsnotify_mark_connector *i_fsnotify_marks;#endif #if IS_ENABLED (CONFIG_FS_ENCRYPTION )include/linux/fsnotify_backend.h +7
-0@@ -194,6 +194,27 @@ struct fsnotify_group {#define FSNOTIFY_EVENT_INODE 2/* +* Inode / vfsmount point to this structure which tracks all marks attached to +* the inode / vfsmount. The reference to inode / vfsmount is held by this +* structure. We destroy this structure when there are no more marks attached +* to it. The structure is protected by fsnotify_mark_srcu. +*/ +struct fsnotify_mark_connector { +#define FSNOTIFY_OBJ_TYPE_INODE 0x01
include/linux/fsnotify_backend.h struct fsnotify_group { · line 213
Quoted 10 lines
include/linux/fsnotify_backend.h +10
-0+#define FSNOTIFY_OBJ_TYPE_VFSMOUNT 0x02 +unsigned int flags;/* Type of object [lock] */ +union {/* Object pointer [lock] */ +struct inode *inode; +struct vfsmount *mnt; + }; +union { +struct hlist_head list; +/* Used listing heads to free after srcu period expires */ +struct fsnotify_mark_connector *destroy_next;
This field is not used in this patch at all.
Quoted 81 lines
include/linux/fsnotify_backend.h +10
-11+ }; +}; + +/* * A mark is simply an object attached to an in core inode which allows an * fsnotify listener to indicate they are either no longer interested in events * of a type matching mask or only interested in those events. @@ -224,18 +245,13 @@ struct fsnotify_mark {spinlock_t lock;/* List of marks for inode / vfsmount [obj_lock] */ struct hlist_node obj_list; -union {/* Object pointer [mark->lock, group->mark_mutex] */ -struct inode *inode;/* inode this mark is associated with */ -struct vfsmount *mnt;/* vfsmount this mark is associated with */ - }; +/* Head of list of marks for an object [mark->lock, group->mark_mutex] */ +struct fsnotify_mark_connector *connector;/* Events types to ignore [mark->lock, group->mark_mutex] */ __u32 ignored_mask; -#define FSNOTIFY_MARK_FLAG_INODE 0x01 -#define FSNOTIFY_MARK_FLAG_VFSMOUNT 0x02 -#define FSNOTIFY_MARK_FLAG_OBJECT_PINNED 0x04 -#define FSNOTIFY_MARK_FLAG_IGNORED_SURV_MODIFY 0x08 -#define FSNOTIFY_MARK_FLAG_ALIVE 0x10 -#define FSNOTIFY_MARK_FLAG_ATTACHED 0x20 +#define FSNOTIFY_MARK_FLAG_IGNORED_SURV_MODIFY 0x01 +#define FSNOTIFY_MARK_FLAG_ALIVE 0x02 +#define FSNOTIFY_MARK_FLAG_ATTACHED 0x04unsigned int flags;/* flags [mark->lock] */ void (*free_mark)(struct fsnotify_mark *mark);/* called on final put+free */ };@@ -343,7 +359,7 @@ extern void fsnotify_free_mark(struct fsnotify_mark *mark);extern void fsnotify_clear_vfsmount_marks_by_group (struct fsnotify_group *group);/* run all the marks in a group, and clear all of the inode marks */ extern void fsnotify_clear_inode_marks_by_group (struct fsnotify_group *group); -/* run all the marks in a group, and clear all of the marks where mark->flags & flags is true*/ +/* run all the marks in a group, and clear all of the marks attached to given object type */ extern void fsnotify_clear_marks_by_group_flags (struct fsnotify_group *group,unsigned int flags);extern void fsnotify_get_mark (struct fsnotify_mark *mark);extern void fsnotify_put_mark (struct fsnotify_mark *mark);kernel/audit_tree.c +17
-2@@ -172,10 +172,25 @@ static unsigned long inode_to_key(const struct inode *inode)/* * Function to return search key in our hash from chunk. Key 0 is special and * should never be present in the hash. +* +* Must be called with chunk->mark.lock held to protect from connector +* becoming NULL. */ +static unsigned long __chunk_to_key (struct audit_chunk *chunk) +{ +if (!chunk->mark.connector) +return 0 ; +return (unsigned long )chunk->mark.connector->inode; +} +static unsigned long chunk_to_key (struct audit_chunk *chunk) { -return (unsigned long )chunk->mark.inode; +unsigned long key; + +spin_lock (&chunk->mark.lock); + key =__chunk_to_key (chunk); +spin_unlock (&chunk->mark.lock); +return key; }static inline struct list_head *chunk_hash (unsigned long key)@@ -187,7 +202,7 @@ static inline struct list_head *chunk_hash(unsigned long key)/* hash_lock & entry->lock is held by caller */ static void insert_hash (struct audit_chunk *chunk) { -unsigned long key =chunk_to_key (chunk); +unsigned long key =__chunk_to_key (chunk);struct list_head *list;if (!key)
kernel/audit_tree.c static void untag_chunk(struct node *p) · line 267
Quoted 6 lines
kernel/audit_tree.c +2
-1@@ -248,7 +263,8 @@ static void untag_chunk(struct node *p)mutex_lock (&entry->group->mark_mutex);spin_lock (&entry->lock); -if (chunk->dead || !entry->inode) { +if (chunk->dead || !entry->connector || + !entry->connector->inode) {
entry->connector->inode will always be positive, at least in this patch.
Quoted 15 lines
kernel/audit_tree.c +2
-2spin_unlock (&entry->lock);mutex_unlock (&entry->group->mark_mutex);if (new)@@ -276,8 +292,8 @@ static void untag_chunk(struct node *p)if (!new) goto Fallback; -if (fsnotify_add_mark_locked (&new->mark, entry->group, entry->inode, -NULL ,1 )) { +if (fsnotify_add_mark_locked (&new->mark, entry->group, + entry->connector->inode,NULL ,1 )) {fsnotify_put_mark (&new->mark); goto Fallback; }
kernel/audit_tree.c static int tag_chunk(struct inode *inode, struct audit_tree *tree) · line 427
Quoted 5 lines
kernel/audit_tree.c +1
-1@@ -408,7 +424,7 @@ static int tag_chunk(struct inode *inode, struct audit_tree *tree)mutex_lock (&old_entry->group->mark_mutex);spin_lock (&old_entry->lock); -if (!old_entry->inode) { +if (!old_entry->connector || !old_entry->connector->inode) {
Same here, connector->inode will never be NULL.
Quoted 17 lines
kernel/audit_tree.c +1
-1/* old_entry is being shot, lets just lie */ spin_unlock (&old_entry->lock);mutex_unlock (&old_entry->group->mark_mutex);@@ -418,7 +434,7 @@ static int tag_chunk(struct inode *inode, struct audit_tree *tree) }if (fsnotify_add_mark_locked (chunk_entry, old_entry->group, - old_entry->inode,NULL ,1 )) { + old_entry->connector->inode,NULL ,1 )) {spin_unlock (&old_entry->lock);mutex_unlock (&old_entry->group->mark_mutex);fsnotify_put_mark (chunk_entry);kernel/auditsc.c +0
-0
kernel/auditsc.c static inline void handle_one(const struct inode *inode) · line 1594
Quoted 5 lines
kernel/auditsc.c +1
-1@@ -1591,7 +1591,7 @@ static inline void handle_one(const struct inode *inode)struct audit_tree_refs *p;struct audit_chunk *chunk;int count; -if (likely (hlist_empty (&inode->i_fsnotify_marks))) +if (likely (!inode->i_fsnotify_marks))
Need to check hlist_empty() as well? Perhaps add an inline helper to fsnotify_backend.h: fsnotify_inode_marks_empty().
kernel/auditsc.c static void handle_path(const struct dentry *dentry) · line 1637
Quoted 9 lines
kernel/auditsc.c +1
-1return ; context = current->audit_context; p = context->trees;@@ -1634,7 +1634,7 @@ static void handle_path(const struct dentry *dentry) seq =read_seqbegin (&rename_lock);for (;;) {struct inode *inode =d_backing_inode (d); -if (inode &&unlikely (!hlist_empty (&inode->i_fsnotify_marks))) { +if (inode &&unlikely (inode->i_fsnotify_marks)) {
Here as well.
Quoted 6 lines
kernel/auditsc.c +0
-1struct audit_chunk *chunk; chunk =audit_tree_lookup (inode);if (chunk) { --2.10.2
Amir Goldstein
reviewerRe: [PATCH 06/25] fsnotify: Attach marks to object via dedicated head structure
Quoted 36 lines · Miklos Szeredi
Quoted 19 lines · Jan Kara
Currently notification marks are attached to object (inode or vfsmnt) by a
hlist_headin the object. The list is also protected by a spinlock in the object. So while there is any mark attached to the list of marks, the object must be pinned in memory (and thus e.g. lastiput()deleting inode cannot happen). Also for list iteration infsnotify()to work, we must holdfsnotify_mark_srculock so that mark itself andmark->obj_list.nextcannot get freed. Thus we are required to wait for response to fanotify events from userspace process withfsnotify_mark_srculock held. That causes issues when userspace process is buggy and does not reply to some event - basically the whole notification subsystem gets eventually stuck.So to be able to drop
fsnotify_mark_srculock while waiting for response, we have to pin the mark in memory and make sure it stays in the object list (as removing the mark waiting for response could lead to lost notification events for groups later in the list). However we don't want inode reclaim to block on such mark as that would lead to system just locking up elsewhere.Quoted 7 lines
This commit tries to pave a way towards solving these conflicting lifetime needs. Instead of anchoring the list of marks directly in the object, we anchor it in a dedicated structure (
fsnotify_mark_connector) and just point to that structure from the object. In the following patch we will also protect the list by a spinlock contained in that structure. With this, we will be able to detach notification marks from object without having to modify the list itself.This is still a bit big to review easily. I'd suggest further splitup into:
- move
hlist_headfrom inode/mnt into connector
- move inode/mnt ptr from mark into connector (no refcounting changes)
- inode refcounting cleanupMore comments inline.
...
Quoted 17 lines
Quoted 17 lines
fs/notify/fsnotify.c +1
-1@@ -193,6 +193,7 @@ int fsnotify(struct inode *to_tell, __u32 mask, const void *data, int data_is,struct hlist_node *inode_node =NULL , *vfsmount_node =NULL ;struct fsnotify_mark *inode_mark =NULL , *vfsmount_mark =NULL ;struct fsnotify_group *inode_group, *vfsmount_group; +struct fsnotify_mark_connector *inode_conn, *vfsmount_conn;struct mount *mnt;int idx, ret =0 ;/* global tests shouldn't care about events on child only the specific event */ @@ -210,8 +211,8 @@ int fsnotify(struct inode *to_tell, __u32 mask, const void *data, int data_is,* SRCU because we have no references to any objects and do not * need SRCU to keep them "alive". */ - if (hlist_empty(&to_tell->i_fsnotify_marks) &&
Quoted 9 lines
fs/notify/fsnotify.c int fsnotify(struct inode *to_tell, __u32 mask, const void *data, int data_is, · line 215
Quoted 3 lines
fs/notify/fsnotify.c +2
-1- (!mnt || hlist_empty(&mnt->mnt_fsnotify_marks))) +if (!to_tell->i_fsnotify_marks && + (!mnt || !mnt->mnt_fsnotify_marks))The checks against
hlist_empty()could still be useful to optimize away the memory barrier imposed bysrcu_read_lock()in case there were marks on the object but not anymore (i.e. the connector is non-NULL, but the hlist is empty).
Is this really a case worth optimizing?
Can an empty connector stay alive more then a few grace periods?
Not that it is hard to check hlist_empty(). I'm just wondering, if this would be code with no visible gain.
Jan Kara
maintainerRe: [PATCH 06/25] fsnotify: Attach marks to object via dedicated head structure
In reply to Miklos SzerediQuoted 28 lines · Miklos Szeredi
Quoted 19 lines · Jan Kara
Currently notification marks are attached to object (inode or vfsmnt) by a
hlist_headin the object. The list is also protected by a spinlock in the object. So while there is any mark attached to the list of marks, the object must be pinned in memory (and thus e.g. lastiput()deleting inode cannot happen). Also for list iteration infsnotify()to work, we must holdfsnotify_mark_srculock so that mark itself andmark->obj_list.nextcannot get freed. Thus we are required to wait for response to fanotify events from userspace process withfsnotify_mark_srculock held. That causes issues when userspace process is buggy and does not reply to some event - basically the whole notification subsystem gets eventually stuck.So to be able to drop
fsnotify_mark_srculock while waiting for response, we have to pin the mark in memory and make sure it stays in the object list (as removing the mark waiting for response could lead to lost notification events for groups later in the list). However we don't want inode reclaim to block on such mark as that would lead to system just locking up elsewhere.Quoted 7 lines
This commit tries to pave a way towards solving these conflicting lifetime needs. Instead of anchoring the list of marks directly in the object, we anchor it in a dedicated structure (
fsnotify_mark_connector) and just point to that structure from the object. In the following patch we will also protect the list by a spinlock contained in that structure. With this, we will be able to detach notification marks from object without having to modify the list itself.
Quoted 5 lines
This is still a bit big to review easily. I'd suggest further splitup into:
- move
hlist_headfrom inode/mnt into connector
- move inode/mnt ptr from mark into connector (no refcounting changes)
- inode refcounting cleanup
OK, I got back to this after sorting out some more urgent stuff with BDIs... I did the split and the result really looks more manageable. Thanks for suggestion!
Quoted 8 lines
Quoted 8 lines
fs/notify/dnotify/dnotify.c +0
-0@@ -69,8 +69,8 @@ static void dnotify_recalc_inode_mask(struct fsnotify_mark *fsn_mark)if (old_mask == new_mask)return ;
Quoted 8 lines
fs/notify/dnotify/dnotify.c static void dnotify_recalc_inode_mask(struct fsnotify_mark *fsn_mark)
Quoted 3 lines
fs/notify/dnotify/dnotify.c +1
-2-if (fsn_mark->inode) -fsnotify_recalc_inode_mask (fsn_mark->inode); + if (fsn_mark->connector->inodeThis doesn't make sense in the context of this patch. Now
mark->connectoris set to NULL where previouslymark->inodewas set to
null. So the right condition would be "if (fsn_mark->connector)
...".
Right, I'll fix that.
Quoted 17 lines
Quoted 17 lines
fs/notify/fsnotify.c +1
-1@@ -193,6 +193,7 @@ int fsnotify(struct inode *to_tell, __u32 mask, const void *data, int data_is,struct hlist_node *inode_node =NULL , *vfsmount_node =NULL ;struct fsnotify_mark *inode_mark =NULL , *vfsmount_mark =NULL ;struct fsnotify_group *inode_group, *vfsmount_group; +struct fsnotify_mark_connector *inode_conn, *vfsmount_conn;struct mount *mnt;int idx, ret =0 ;/* global tests shouldn't care about events on child only the specific event */ @@ -210,8 +211,8 @@ int fsnotify(struct inode *to_tell, __u32 mask, const void *data, int data_is,* SRCU because we have no references to any objects and do not * need SRCU to keep them "alive". */ - if (hlist_empty(&to_tell->i_fsnotify_marks) &&
Quoted 8 lines
fs/notify/fsnotify.c int fsnotify(struct inode *to_tell, __u32 mask, const void *data, int data_is, · line 215
Quoted 3 lines
fs/notify/fsnotify.c +2
-1- (!mnt || hlist_empty(&mnt->mnt_fsnotify_marks))) +if (!to_tell->i_fsnotify_marks && + (!mnt || !mnt->mnt_fsnotify_marks))The checks against
hlist_empty()could still be useful to optimize away the memory barrier imposed bysrcu_read_lock()in case there were marks on the object but not anymore (i.e. the connector is non-NULL, but the hlist is empty).
Yes, at this point in the series it would be a worthwhile optimization. But since following patches change the rules so that empty connector structures get discarded, I don't think making the optimization in this patch is worth it.
Quoted 6 lines
Quoted 6 lines
fs/notify/fsnotify.c +0
-0return 0 ;/* * if this is a modify event we may need to clear the ignored masks @@ -226,16 +227,24 @@ int fsnotify(struct inode *to_tell, __u32 mask, const void *data, int data_is, idx =srcu_read_lock (&fsnotify_mark_srcu);
Quoted 8 lines
fs/notify/fsnotify.c int fsnotify(struct inode *to_tell, __u32 mask, const void *data, int data_is, · line 231
Quoted 6 lines
fs/notify/fsnotify.c +2
-3if ((mask &FS_MODIFY ) || - (test_mask & to_tell->i_fsnotify_mask)) - inode_node =srcu_dereference (to_tell->i_fsnotify_marks.first, - &fsnotify_mark_srcu); + (test_mask & to_tell->i_fsnotify_mask)) { + inode_conn = to_tell->i_fsnotify_marks;Needs
lockless_dereference().
Right. Will fix.
Quoted 4 lines
Quoted 4 lines
fs/notify/fsnotify.h +0
-0
...
Quoted 9 lines
fs/notify/fsnotify.h
Quoted 5 lines
/* run the list of all marks associated with inode and destroy them */ static inline void
fsnotify_clear_marks_by_inode(struct inode *inode){fs/notify/fsnotify.h +2
-0+if (!inode->i_fsnotify_marks) +return ;Should move the above check inside
fsnotify_destroy_marks(). Locking rules also dictate that the check be done under inode lock, although that may not matter in practice.
Yeah, it does not matter in practice but it's just nicer. Moved (and the other similar occurences as well).
Quoted 4 lines
Quoted 4 lines
fs/notify/mark.c +0
-0
...
Quoted 7 lines
Quoted 7 lines
fs/notify/mark.c +3
-0@@ -249,6 +252,16 @@ void fsnotify_destroy_marks(struct hlist_head *head, spinlock_t *lock)fsnotify_destroy_mark (mark, mark->group);fsnotify_put_mark (mark); } +/* +* Skip freeing of connector when inode is just unlinked. We leave that +* until destroy_inode() is called at which moment nobody will be
Quoted 8 lines
fs/notify/mark.c void fsnotify_destroy_marks(struct hlist_head *head, spinlock_t *lock) · line 262
Quoted 5 lines
fs/notify/mark.c +5
-0+* looking at the inode anymore. +*/ +if ((*connp)->flags &FSNOTIFY_OBJ_TYPE_INODE && + !((*connp)->inode->i_state &I_CLEAR )) +return ;This is ugly. Why not rather add a
fsnotify_connector_free()and
call that explicitly on inode/vfsmount destruction?
Good idea. Changed.
Quoted 5 lines
Quoted 5 lines
fs/notify/mark.c +3
-0+static int fsnotify_add_mark_list (struct fsnotify_mark *mark, +struct inode *inode,struct vfsmount *mnt, +int allow_dups) {struct fsnotify_mark *lmark, *last =NULL ;
Quoted 8 lines
fs/notify/mark.c int fsnotify_compare_groups(struct fsnotify_group *a, struct fsnotify_group *b) · line 361
Quoted 6 lines
fs/notify/mark.c +5
-0+struct fsnotify_mark_connector *conn; +struct fsnotify_mark_connector **connp;int cmp; +int err =0 ; + +BUG_ON (!inode && !mnt);Use
WARN_ON()instead.
OK.
Quoted 11 lines
fs/notify/mark.c int fsnotify_compare_groups(struct fsnotify_group *a, struct fsnotify_group *b) · line 368
Quoted 7 lines
fs/notify/mark.c +7
-0+if (inode) + connp = &inode->i_fsnotify_marks; +else + connp = &real_mount (mnt)->mnt_fsnotify_marks; +restart: +spin_lock (&mark->lock); + conn =fsnotify_grab_connector (*connp);This looks a bit too subtle. We have the inode/vfsmount. We have the matching lock. Could store the address of the lock in a local variable and use that for locking.
Done.
Quoted 19 lines
Quoted 8 lines
fs/notify/mark.c +8
-0+if (!conn) { +spin_unlock (&mark->lock); + conn =kmem_cache_alloc (fsnotify_mark_connector_cachep, +GFP_KERNEL ); +if (!conn) +return -ENOMEM ; +INIT_HLIST_HEAD (&conn->list); +if (inode) {fs/notify/mark.c int fsnotify_compare_groups(struct fsnotify_group *a, struct fsnotify_group *b) · line 386
Quoted 10 lines
fs/notify/mark.c +10
-0+ conn->flags =FSNOTIFY_OBJ_TYPE_INODE ; + conn->inode = inode; + }else { + conn->flags =FSNOTIFY_OBJ_TYPE_VFSMOUNT ; + conn->mnt = mnt; + } +if (cmpxchg (connp,NULL , conn)) { +/* Someone else created list structure for us */ +kmem_cache_free (fsnotify_mark_connector_cachep, conn); + }
Quoted 3 lines
Setting *connp should be done with inode/vfsmount lock already regained (if I understand the locking rules correctly) and then no need to have cmpxchg magic in there.
Yes, it will have to come in following patches but I agree it is not needed here. Changed.
Quoted 19 lines
Quoted 19 lines
include/linux/fsnotify_backend.h +11
-0@@ -194,6 +194,27 @@ struct fsnotify_group {#define FSNOTIFY_EVENT_INODE 2/* +* Inode / vfsmount point to this structure which tracks all marks attached to +* the inode / vfsmount. The reference to inode / vfsmount is held by this +* structure. We destroy this structure when there are no more marks attached +* to it. The structure is protected by fsnotify_mark_srcu. +*/ +struct fsnotify_mark_connector { +#define FSNOTIFY_OBJ_TYPE_INODE 0x01 +#define FSNOTIFY_OBJ_TYPE_VFSMOUNT 0x02 +unsigned int flags;/* Type of object [lock] */ +union {/* Object pointer [lock] */ +struct inode *inode;
Quoted 8 lines
include/linux/fsnotify_backend.h struct fsnotify_group { · line 213
Quoted 6 lines
include/linux/fsnotify_backend.h +6
-0+struct vfsmount *mnt; + }; +union { +struct hlist_head list; +/* Used listing heads to free after srcu period expires */ +struct fsnotify_mark_connector *destroy_next;This field is not used in this patch at all.
Yes, will move to later patches.
Quoted 6 lines
Quoted 6 lines
kernel/audit_tree.c +0
-0@@ -248,7 +263,8 @@ static void untag_chunk(struct node *p)
Quoted 7 lines
kernel/audit_tree.c static void untag_chunk(struct node *p) · line 267
Quoted 5 lines
kernel/audit_tree.c +2
-1mutex_lock (&entry->group->mark_mutex);spin_lock (&entry->lock); -if (chunk->dead || !entry->inode) { +if (chunk->dead || !entry->connector || + !entry->connector->inode) {
entry->connector->inodewill always be positive, at least in this patch.
Correct. I will move it to a later patch.
Quoted 5 lines
Quoted 5 lines
kernel/auditsc.c +0
-0@@ -1591,7 +1591,7 @@ static inline void handle_one(const struct inode *inode)
Quoted 8 lines
kernel/auditsc.c static inline void handle_one(const struct inode *inode) · line 1594
Quoted 5 lines
kernel/auditsc.c +1
-1struct audit_tree_refs *p;struct audit_chunk *chunk;int count; -if (likely (hlist_empty (&inode->i_fsnotify_marks))) +if (likely (!inode->i_fsnotify_marks))Need to check
hlist_empty()as well? Perhaps add an inline helper tofsnotify_backend.h:fsnotify_inode_marks_empty().
I've added the hlist_empty() check. I don't think fsnotify_inode_marks_empty() helper is needed as it will be used only in these two places.
Thanks for review!
Honza
Signature · 2 lines
Jan Kara <[email protected]> SUSE Labs, CR
Jan Kara
maintainerPATCH v4 7/25 fsnotify: Move fsnotify_destroy_marks()
In reply to Jan KaraMove fsnotify_destroy_marks() to be later in the fs/notify/mark.c. It will need some functions that are declared after its current declaration. No functional change.
--- fs/notify/mark.c | 86 ++++++++++++++++++++++++++++---------------------------- 1 file changed, 43 insertions(+), 43 deletions(-)
fs/notify/mark.c +43 -43
@@ -221,49 +221,6 @@ void fsnotify_destroy_mark(struct fsnotify_mark *mark,fsnotify_free_mark (mark); } -void fsnotify_destroy_marks (struct fsnotify_mark_connector **connp, -spinlock_t *lock) -{ -struct fsnotify_mark *mark; - -while (1 ) { -/* -* We have to be careful since we can race with e.g. -* fsnotify_clear_marks_by_group() and once we drop 'lock', -* mark can get removed from the obj_list and destroyed. But -* we are holding mark reference so mark cannot be freed and -* calling fsnotify_destroy_mark() more than once is fine. -*/ -spin_lock (lock); -if (hlist_empty (&(*connp)->list)) { -spin_unlock (lock); -break ; - } - mark = hlist_entry((*connp)->list.first, struct fsnotify_mark, - obj_list); -/* -* We don't update i_fsnotify_mask / mnt_fsnotify_mask here -* since inode / mount is going away anyway. So just remove -* mark from the list. -*/ -hlist_del_init_rcu (&mark->obj_list); -fsnotify_get_mark (mark); -spin_unlock (lock); -fsnotify_destroy_mark (mark, mark->group); -fsnotify_put_mark (mark); - } -/* -* Skip freeing of connector when inode is just unlinked. We leave that -* until destroy_inode() is called at which moment nobody will be -* looking at the inode anymore. -*/ -if ((*connp)->flags &FSNOTIFY_OBJ_TYPE_INODE && - !((*connp)->inode->i_state &I_CLEAR )) -return ; -kmem_cache_free (fsnotify_mark_connector_cachep, *connp); - *connp =NULL ; -} -void fsnotify_set_mark_mask_locked (struct fsnotify_mark *mark,__u32 mask) {assert_spin_locked (&mark->lock);
@@ -574,6 +531,49 @@ void fsnotify_detach_group_marks(struct fsnotify_group *group) } } +void fsnotify_destroy_marks (struct fsnotify_mark_connector **connp, +spinlock_t *lock) +{ +struct fsnotify_mark *mark; + +while (1 ) { +/* +* We have to be careful since we can race with e.g. +* fsnotify_clear_marks_by_group() and once we drop 'lock', +* mark can get removed from the obj_list and destroyed. But +* we are holding mark reference so mark cannot be freed and +* calling fsnotify_destroy_mark() more than once is fine. +*/ +spin_lock (lock); +if (hlist_empty (&(*connp)->list)) { +spin_unlock (lock); +break ; + } + mark = hlist_entry((*connp)->list.first, struct fsnotify_mark, + obj_list); +/* +* We don't update i_fsnotify_mask / mnt_fsnotify_mask here +* since inode / mount is going away anyway. So just remove +* mark from the list. +*/ +hlist_del_init_rcu (&mark->obj_list); +fsnotify_get_mark (mark); +spin_unlock (lock); +fsnotify_destroy_mark (mark, mark->group); +fsnotify_put_mark (mark); + } +/* +* Skip freeing of connector when inode is just unlinked. We leave that +* until destroy_inode() is called at which moment nobody will be +* looking at the inode anymore. +*/ +if ((*connp)->flags &FSNOTIFY_OBJ_TYPE_INODE && + !((*connp)->inode->i_state &I_CLEAR )) +return ; +kmem_cache_free (fsnotify_mark_connector_cachep, *connp); + *connp =NULL ; +} +/* * Nothing fancy, just initialize lists and locks and counters. */
Signature · 1 line
2.10.2
Re: [PATCH 07/25] fsnotify: Move fsnotify_destroy_marks()
Quoted 6 lines · Jan Kara
Move
fsnotify_destroy_marks()to be later in thefs/notify/mark.c. It will need some functions that are declared after its current declaration. No functional change.Signed-off-by Jan Kara <[email protected]>---
You may re-add
Quoted 110 lines
fs/notify/mark.c | 86 ++++++++++++++++++++++++++++---------------------------- 1 file changed, 43 insertions(+), 43 deletions(-)fs/notify/mark.c +43
-44@@ -221,49 +221,6 @@ void fsnotify_destroy_mark(struct fsnotify_mark *mark,fsnotify_free_mark (mark); } -void fsnotify_destroy_marks (struct fsnotify_mark_connector **connp, -spinlock_t *lock) -{ -struct fsnotify_mark *mark; - -while (1 ) { -/* -* We have to be careful since we can race with e.g. -* fsnotify_clear_marks_by_group() and once we drop 'lock', -* mark can get removed from the obj_list and destroyed. But -* we are holding mark reference so mark cannot be freed and -* calling fsnotify_destroy_mark() more than once is fine. -*/ -spin_lock (lock); -if (hlist_empty (&(*connp)->list)) { -spin_unlock (lock); -break ; - } - mark = hlist_entry((*connp)->list.first, struct fsnotify_mark, - obj_list); -/* -* We don't update i_fsnotify_mask / mnt_fsnotify_mask here -* since inode / mount is going away anyway. So just remove -* mark from the list. -*/ -hlist_del_init_rcu (&mark->obj_list); -fsnotify_get_mark (mark); -spin_unlock (lock); -fsnotify_destroy_mark (mark, mark->group); -fsnotify_put_mark (mark); - } -/* -* Skip freeing of connector when inode is just unlinked. We leave that -* until destroy_inode() is called at which moment nobody will be -* looking at the inode anymore. -*/ -if ((*connp)->flags &FSNOTIFY_OBJ_TYPE_INODE && - !((*connp)->inode->i_state &I_CLEAR )) -return ; -kmem_cache_free (fsnotify_mark_connector_cachep, *connp); - *connp =NULL ; -} -void fsnotify_set_mark_mask_locked (struct fsnotify_mark *mark,__u32 mask) {assert_spin_locked (&mark->lock);@@ -574,6 +531,49 @@ void fsnotify_detach_group_marks(struct fsnotify_group *group) } } +void fsnotify_destroy_marks (struct fsnotify_mark_connector **connp, +spinlock_t *lock) +{ +struct fsnotify_mark *mark; + +while (1 ) { +/* +* We have to be careful since we can race with e.g. +* fsnotify_clear_marks_by_group() and once we drop 'lock', +* mark can get removed from the obj_list and destroyed. But +* we are holding mark reference so mark cannot be freed and +* calling fsnotify_destroy_mark() more than once is fine. +*/ +spin_lock (lock); +if (hlist_empty (&(*connp)->list)) { +spin_unlock (lock); +break ; + } + mark = hlist_entry((*connp)->list.first, struct fsnotify_mark, + obj_list); +/* +* We don't update i_fsnotify_mask / mnt_fsnotify_mask here +* since inode / mount is going away anyway. So just remove +* mark from the list. +*/ +hlist_del_init_rcu (&mark->obj_list); +fsnotify_get_mark (mark); +spin_unlock (lock); +fsnotify_destroy_mark (mark, mark->group); +fsnotify_put_mark (mark); + } +/* +* Skip freeing of connector when inode is just unlinked. We leave that +* until destroy_inode() is called at which moment nobody will be +* looking at the inode anymore. +*/ +if ((*connp)->flags &FSNOTIFY_OBJ_TYPE_INODE && + !((*connp)->inode->i_state &I_CLEAR )) +return ; +kmem_cache_free (fsnotify_mark_connector_cachep, *connp); + *connp =NULL ; +} +/* * Nothing fancy, just initialize lists and locks and counters. */ --2.10.2
Jan Kara
maintainerPATCH v4 8/25 fsnotify: Lock object list with connector lock
In reply to Jan KaraSo far list of marks attached to an object (inode / vfsmount) was protected by i_lock or mnt_root->d_lock. This dictates that the list must be empty before the object can be destroyed although the list is now anchored in the fsnotify_mark_connector structure. Protect the list by a spinlock in the fsnotify_mark_connector structure to decouple lifetime of a list of marks from a lifetime of the object. This also simplifies the code quite a bit since we don't have to differentiate between inode and vfsmount lists in quite a few places anymore.
--- fs/notify/dnotify/dnotify.c | 3 +- fs/notify/fsnotify.h | 37 +++-------- fs/notify/inode_mark.c | 46 +------------- fs/notify/mark.c | 129 +++++++++++++++++++++++---------------- fs/notify/vfsmount_mark.c | 41 ++----------- include/linux/fsnotify_backend.h | 5 +- 6 files changed, 97 insertions(+), 164 deletions(-)
fs/notify/dnotify/dnotify.c +1 -2
@@ -69,8 +69,7 @@ static void dnotify_recalc_inode_mask(struct fsnotify_mark *fsn_mark)if (old_mask == new_mask)return ; -if (fsn_mark->connector->inode) -fsnotify_recalc_inode_mask (fsn_mark->connector->inode); +fsnotify_recalc_mask (fsn_mark->connector); }/*
fs/notify/fsnotify.h +8 -29
@@ -8,56 +8,35 @@#include "../mount.h" -struct fsnotify_mark_connector ; -/* destroy all events sitting in this groups notification queue */ extern void fsnotify_flush_notify (struct fsnotify_group *group);/* protects reads of inode and vfsmount marks list */ extern struct srcu_struct fsnotify_mark_srcu; -/* Calculate mask of events for a list of marks */ -extern u32 fsnotify_recalc_mask (struct fsnotify_mark_connector *conn); -/* compare two groups for sorting of marks lists */ extern int fsnotify_compare_groups (struct fsnotify_group *a,struct fsnotify_group *b); -extern void fsnotify_set_inode_mark_mask_locked (struct fsnotify_mark *fsn_mark, -__u32 mask); -/* add a mark to an inode */ -extern int fsnotify_add_inode_mark (struct fsnotify_mark *mark, -struct fsnotify_group *group,struct inode *inode, -int allow_dups); -/* add a mark to a vfsmount */ -extern int fsnotify_add_vfsmount_mark (struct fsnotify_mark *mark, -struct fsnotify_group *group,struct vfsmount *mnt, -int allow_dups); - -/* vfsmount specific destruction of a mark */ -extern void fsnotify_destroy_vfsmount_mark (struct fsnotify_mark *mark); -/* inode specific destruction of a mark */ -extern struct inode *fsnotify_destroy_inode_mark (struct fsnotify_mark *mark); -/* Find mark belonging to given group in the list of marks */ -extern struct fsnotify_mark *fsnotify_find_mark (struct hlist_head *head, -struct fsnotify_group *group); -/* Destroy all marks in the given list protected by 'lock' */ -extern void fsnotify_destroy_marks (struct fsnotify_mark_connector **connp, -spinlock_t *lock); +/* Find mark belonging to given group attached to given connector */ +extern struct fsnotify_mark *fsnotify_find_mark ( +struct fsnotify_mark_connector **connp, +struct fsnotify_group *group); +/* Destroy all marks connected via given connector */ +extern void fsnotify_destroy_marks (struct fsnotify_mark_connector **connp);/* run the list of all marks associated with inode and destroy them */ static inline void fsnotify_clear_marks_by_inode (struct inode *inode) {if (!inode->i_fsnotify_marks)return ; -fsnotify_destroy_marks (&inode->i_fsnotify_marks, &inode->i_lock); +fsnotify_destroy_marks (&inode->i_fsnotify_marks); }/* run the list of all marks associated with vfsmount and destroy them */ static inline void fsnotify_clear_marks_by_mount (struct vfsmount *mnt) {if (!real_mount (mnt)->mnt_fsnotify_marks)return ; -fsnotify_destroy_marks (&real_mount (mnt)->mnt_fsnotify_marks, - &mnt->mnt_root->d_lock); +fsnotify_destroy_marks (&real_mount (mnt)->mnt_fsnotify_marks); }/* prepare for freeing all marks associated with given group */ extern void fsnotify_detach_group_marks (struct fsnotify_group *group);
fs/notify/inode_mark.c +2 -44
60 lines not on this page · raw message
fs/notify/mark.c +78 -51
225 lines not on this page · raw message
fs/notify/vfsmount_mark.c +4 -37
57 lines not on this page · raw message
include/linux/fsnotify_backend.h +4 -1
@@ -200,6 +200,7 @@ struct fsnotify_group {* to it. The structure is protected by fsnotify_mark_srcu. */ struct fsnotify_mark_connector { +spinlock_t lock;#define FSNOTIFY_OBJ_TYPE_INODE 0x01#define FSNOTIFY_OBJ_TYPE_VFSMOUNT 0x02unsigned int flags;/* Type of object [lock] */
@@ -243,7 +244,7 @@ struct fsnotify_mark {struct list_head g_list;/* Protects inode / mnt pointers, flags, masks */ spinlock_t lock; -/* List of marks for inode / vfsmount [obj_lock] */ +/* List of marks for inode / vfsmount [connector->lock] */ struct hlist_node obj_list;/* Head of list of marks for an object [mark->lock, group->mark_mutex] */ struct fsnotify_mark_connector *connector;
@@ -330,6 +331,8 @@ extern struct fsnotify_event *fsnotify_remove_first_event(struct fsnotify_group/* functions used to manipulate the marks attached to inodes */ +/* Calculate mask of events for a list of marks */ +extern void fsnotify_recalc_mask (struct fsnotify_mark_connector *conn);/* run all marks associated with a vfsmount and update mnt->mnt_fsnotify_mask */ extern void fsnotify_recalc_vfsmount_mask (struct vfsmount *mnt);/* run all marks associated with an inode and update inode->i_fsnotify_mask */
Signature · 1 line
2.10.2
Re: [PATCH 08/25] fsnotify: Lock object list with connector lock
Quoted 10 lines · Jan Kara
So far list of marks attached to an object (inode / vfsmount) was protected by
i_lockormnt_root->d_lock. This dictates that the list must be empty before the object can be destroyed although the list is now anchored in thefsnotify_mark_connectorstructure. Protect the list by a spinlock in thefsnotify_mark_connectorstructure to decouple lifetime of a list of marks from a lifetime of the object. This also simplifies the code quite a bit since we don't have to differentiate between inode and vfsmount lists in quite a few places anymore.Signed-off-by Jan Kara <[email protected]>
Apart from one nit below, you may re-add
...
Quoted 4 lines
fs/notify/mark.c +4
-0+ +static struct inode *fsnotify_detach_from_object (struct fsnotify_mark *mark) +{ +struct fsnotify_mark_connector *conn;
fs/notify/mark.c void fsnotify_put_mark(struct fsnotify_mark *mark) · line 158
Quoted 10 lines
fs/notify/mark.c +10
-0+struct inode *inode =NULL ; + + conn = mark->connector; +spin_lock (&conn->lock); +hlist_del_init_rcu (&mark->obj_list); +if (hlist_empty (&conn->list)) { +if (conn->flags &FSNOTIFY_OBJ_TYPE_INODE ) + inode = conn->inode; + }else +__fsnotify_recalc_mask (conn);
I was told that "if {} else ;" is a coding practice that asks for trouble and I tend to agree,
so please add {} to else case.
Quoted 6 lines
fs/notify/mark.c +4
-0+ mark->connector =NULL ; +spin_unlock (&conn->lock); + +return inode; }
Jan Kara
maintainerRe: [PATCH 08/25] fsnotify: Lock object list with connector lock
Quoted 9 lines · Amir Goldstein
fs/notify/mark.c void fsnotify_put_mark(struct fsnotify_mark *mark) · line 158
Quoted 8 lines
fs/notify/mark.c +8
-0+ conn = mark->connector; +spin_lock (&conn->lock); +hlist_del_init_rcu (&mark->obj_list); +if (hlist_empty (&conn->list)) { +if (conn->flags &FSNOTIFY_OBJ_TYPE_INODE ) + inode = conn->inode; + }else +__fsnotify_recalc_mask (conn);
Quoted 3 lines
I was told that "if {} else ;" is a coding practice that asks for trouble and I tend to agree,
so please add {} to else case.
Right. Fixed. Thanks for review.
Honza
Signature · 2 lines
Jan Kara <[email protected]> SUSE Labs, CR
Jan Kara
maintainerPATCH v4 9/25 fsnotify: Free fsnotify_mark_connector when there is no mark attached
In reply to Jan KaraCurrently we free fsnotify_mark_connector structure only when inode / vfsmount is getting freed. This can however impose noticeable memory overhead when marks get attached to inodes only temporarily. So free the connector structure once the last mark is detached from the object. Since notification infrastructure can be working with the connector under the protection of fsnotify_mark_srcu, we have to be careful and free the fsnotify_mark_connector only after SRCU period passes.
--- fs/mount.h | 2 +- fs/notify/fsnotify.c | 9 +++-- fs/notify/fsnotify.h | 4 --- fs/notify/mark.c | 97 ++++++++++++++++++++++++++++++++++++++-------------- include/linux/fs.h | 2 +- 5 files changed, 79 insertions(+), 35 deletions(-)
fs/mount.h +1 -1
@@ -59,7 +59,7 @@ struct mount {struct mountpoint *mnt_mp;/* where is it mounted */ struct hlist_node mnt_mp_list;/* list mounts with the same mountpoint */ #ifdef CONFIG_FSNOTIFY -struct fsnotify_mark_connector *mnt_fsnotify_marks; +struct fsnotify_mark_connector __rcu *mnt_fsnotify_marks;__u32 mnt_fsnotify_mask;#endif int mnt_id;/* mount identifier */
fs/notify/fsnotify.c +6 -3
@@ -228,7 +228,8 @@ int fsnotify(struct inode *to_tell, __u32 mask, const void *data, int data_is,if ((mask &FS_MODIFY ) || (test_mask & to_tell->i_fsnotify_mask)) { - inode_conn = to_tell->i_fsnotify_marks; + inode_conn =srcu_dereference (to_tell->i_fsnotify_marks, + &fsnotify_mark_srcu);if (inode_conn) inode_node =srcu_dereference (inode_conn->list.first, &fsnotify_mark_srcu);
@@ -236,11 +237,13 @@ int fsnotify(struct inode *to_tell, __u32 mask, const void *data, int data_is,if (mnt && ((mask &FS_MODIFY ) || (test_mask & mnt->mnt_fsnotify_mask))) { - inode_conn = to_tell->i_fsnotify_marks; + inode_conn =srcu_dereference (to_tell->i_fsnotify_marks, + &fsnotify_mark_srcu);if (inode_conn) inode_node =srcu_dereference (inode_conn->list.first, &fsnotify_mark_srcu); - vfsmount_conn = mnt->mnt_fsnotify_marks; + vfsmount_conn =srcu_dereference (mnt->mnt_fsnotify_marks, + &fsnotify_mark_srcu);if (vfsmount_conn) vfsmount_node = srcu_dereference( vfsmount_conn->list.first,
fs/notify/fsnotify.h +0 -4
@@ -27,15 +27,11 @@ extern void fsnotify_destroy_marks(struct fsnotify_mark_connector **connp);/* run the list of all marks associated with inode and destroy them */ static inline void fsnotify_clear_marks_by_inode (struct inode *inode) { -if (!inode->i_fsnotify_marks) -return ;fsnotify_destroy_marks (&inode->i_fsnotify_marks); }/* run the list of all marks associated with vfsmount and destroy them */ static inline void fsnotify_clear_marks_by_mount (struct vfsmount *mnt) { -if (!real_mount (mnt)->mnt_fsnotify_marks) -return ;fsnotify_destroy_marks (&real_mount (mnt)->mnt_fsnotify_marks); }/* prepare for freeing all marks associated with given group */
fs/notify/mark.c +71 -26
187 lines not on this page · raw message
include/linux/fs.h +1 -1
@@ -646,7 +646,7 @@ struct inode {#ifdef CONFIG_FSNOTIFY __u32 i_fsnotify_mask;/* all events this inode cares about */ -struct fsnotify_mark_connector *i_fsnotify_marks; +struct fsnotify_mark_connector __rcu *i_fsnotify_marks;#endif #if IS_ENABLED (CONFIG_FS_ENCRYPTION )
Signature · 1 line
2.10.2
Re: [PATCH 09/25] fsnotify: Free fsnotify_mark_connector when there is no mark attached
Quoted 9 lines · Jan Kara
Currently we free
fsnotify_mark_connectorstructure only when inode / vfsmount is getting freed. This can however impose noticeable memory overhead when marks get attached to inodes only temporarily. So free the connector structure once the last mark is detached from the object. Since notification infrastructure can be working with the connector under the protection offsnotify_mark_srcu, we have to be careful and free thefsnotify_mark_connectoronly after SRCU period passes.Signed-off-by Jan Kara <[email protected]>
you may re-add
Re: [PATCH 09/25] fsnotify: Free fsnotify_mark_connector when there is no mark attached
In reply to Jan KaraQuoted 7 lines · Jan Kara
Currently we free
fsnotify_mark_connectorstructure only when inode / vfsmount is getting freed. This can however impose noticeable memory overhead when marks get attached to inodes only temporarily. So free the connector structure once the last mark is detached from the object. Since notification infrastructure can be working with the connector under the protection offsnotify_mark_srcu, we have to be careful and free thefsnotify_mark_connectoronly after SRCU period passes.
This adds a bunch of complexity. How about instead:
- protect setting *_fsnotify_marks (forward pointer) with lock in
containing object
- keep count of marks not freed for connector
- when object (inode or vfsmount) is being destroyed and*_fsnotify_marks is non-NULL:
o lock object; re-check forward pointer; lock connector; set
back-pointer to NULL; unlock connector; unlock object
- when mark counter goes to zero:
o lock connector; check back pointer, if nonzero:
+ trylock object, if failed: unlock connector, relax & retry
+ clear forward pointer; unlock object
o unlock connector
This ensures freeing only after srcu period (since the mark is already srcu protected). It also ensures that multiple connectors for the same object won't exist at the same time (the connector will be reused if a new mark is added before the srcu period has passed).
While the locking looks somewhat challenging, I think it's still going to be simpler than the current one.
Thanks,
Miklos
Quoted 289 lines
Signed-off-by Jan Kara <[email protected]>--- fs/mount.h | 2 +- fs/notify/fsnotify.c | 9 +++-- fs/notify/fsnotify.h | 4 --- fs/notify/mark.c | 97 ++++++++++++++++++++++++++++++++++++++-------------- include/linux/fs.h | 2 +- 5 files changed, 79 insertions(+), 35 deletions(-)fs/mount.h +1
-1@@ -59,7 +59,7 @@ struct mount {struct mountpoint *mnt_mp;/* where is it mounted */ struct hlist_node mnt_mp_list;/* list mounts with the same mountpoint */ #ifdef CONFIG_FSNOTIFY -struct fsnotify_mark_connector *mnt_fsnotify_marks; +struct fsnotify_mark_connector __rcu *mnt_fsnotify_marks;__u32 mnt_fsnotify_mask;#endif int mnt_id;/* mount identifier */ fs/notify/fsnotify.c +6
-3@@ -228,7 +228,8 @@ int fsnotify(struct inode *to_tell, __u32 mask, const void *data, int data_is,if ((mask &FS_MODIFY ) || (test_mask & to_tell->i_fsnotify_mask)) { - inode_conn = to_tell->i_fsnotify_marks; + inode_conn =srcu_dereference (to_tell->i_fsnotify_marks, + &fsnotify_mark_srcu);if (inode_conn) inode_node =srcu_dereference (inode_conn->list.first, &fsnotify_mark_srcu);@@ -236,11 +237,13 @@ int fsnotify(struct inode *to_tell, __u32 mask, const void *data, int data_is,if (mnt && ((mask &FS_MODIFY ) || (test_mask & mnt->mnt_fsnotify_mask))) { - inode_conn = to_tell->i_fsnotify_marks; + inode_conn =srcu_dereference (to_tell->i_fsnotify_marks, + &fsnotify_mark_srcu);if (inode_conn) inode_node =srcu_dereference (inode_conn->list.first, &fsnotify_mark_srcu); - vfsmount_conn = mnt->mnt_fsnotify_marks; + vfsmount_conn =srcu_dereference (mnt->mnt_fsnotify_marks, + &fsnotify_mark_srcu);if (vfsmount_conn) vfsmount_node = srcu_dereference( vfsmount_conn->list.first,fs/notify/fsnotify.h +0
-4@@ -27,15 +27,11 @@ extern void fsnotify_destroy_marks(struct fsnotify_mark_connector **connp);/* run the list of all marks associated with inode and destroy them */ static inline void fsnotify_clear_marks_by_inode (struct inode *inode) { -if (!inode->i_fsnotify_marks) -return ;fsnotify_destroy_marks (&inode->i_fsnotify_marks); }/* run the list of all marks associated with vfsmount and destroy them */ static inline void fsnotify_clear_marks_by_mount (struct vfsmount *mnt) { -if (!real_mount (mnt)->mnt_fsnotify_marks) -return ;fsnotify_destroy_marks (&real_mount (mnt)->mnt_fsnotify_marks); }/* prepare for freeing all marks associated with given group */ fs/notify/mark.c +71
-26@@ -89,10 +89,14 @@ struct kmem_cache *fsnotify_mark_connector_cachep;static DEFINE_SPINLOCK (destroy_lock);static LIST_HEAD (destroy_list); +static struct fsnotify_mark_connector *connector_destroy_list;static void fsnotify_mark_destroy_workfn (struct work_struct *work); static DECLARE_DELAYED_WORK(reaper_work, fsnotify_mark_destroy_workfn); +static void fsnotify_connector_destroy_workfn (struct work_struct *work); +static DECLARE_WORK(connector_reaper_work, fsnotify_connector_destroy_workfn); +void fsnotify_get_mark (struct fsnotify_mark *mark) {atomic_inc (&mark->refcnt);@@ -143,22 +147,62 @@ void fsnotify_recalc_mask(struct fsnotify_mark_connector *conn) } } +/* Free all connectors queued for freeing once SRCU period ends */ +static void fsnotify_connector_destroy_workfn (struct work_struct *work) +{ +struct fsnotify_mark_connector *conn, *free; + +spin_lock (&destroy_lock); + conn = connector_destroy_list; + connector_destroy_list =NULL ; +spin_unlock (&destroy_lock); + +synchronize_srcu (&fsnotify_mark_srcu); +while (conn) { + free = conn; + conn = conn->destroy_next; +kmem_cache_free (fsnotify_mark_connector_cachep, free); + } +} +static struct inode *fsnotify_detach_from_object (struct fsnotify_mark *mark) {struct fsnotify_mark_connector *conn;struct inode *inode =NULL ; +bool free_conn = false; conn = mark->connector;spin_lock (&conn->lock);hlist_del_init_rcu (&mark->obj_list);if (hlist_empty (&conn->list)) { -if (conn->flags &FSNOTIFY_OBJ_TYPE_INODE ) +if (conn->flags &FSNOTIFY_OBJ_TYPE_INODE ) { inode = conn->inode; +rcu_assign_pointer (inode->i_fsnotify_marks,NULL ); + inode->i_fsnotify_mask =0 ; + conn->inode =NULL ; + conn->flags &= ~FSNOTIFY_OBJ_TYPE_INODE ; + }else if (conn->flags &FSNOTIFY_OBJ_TYPE_VFSMOUNT ) { +rcu_assign_pointer ( +real_mount (conn->mnt)->mnt_fsnotify_marks, +NULL ); +real_mount (conn->mnt)->mnt_fsnotify_mask =0 ; + conn->mnt =NULL ; + conn->flags &= ~FSNOTIFY_OBJ_TYPE_VFSMOUNT ; + } + free_conn = true; }else __fsnotify_recalc_mask (conn); mark->connector =NULL ;spin_unlock (&conn->lock); +if (free_conn) { +spin_lock (&destroy_lock); + conn->destroy_next = connector_destroy_list; + connector_destroy_list = conn; +spin_unlock (&destroy_lock); +queue_work (system_unbound_wq, &connector_reaper_work); + } +return inode; }@@ -320,11 +364,24 @@ int fsnotify_compare_groups(struct fsnotify_group *a, struct fsnotify_group *b)* they are sure list cannot go away under them. */ static struct fsnotify_mark_connector *fsnotify_grab_connector ( -struct fsnotify_mark_connector *conn) +struct fsnotify_mark_connector *__rcu *connp) { +struct fsnotify_mark_connector *conn; +int idx; + + idx =srcu_read_lock (&fsnotify_mark_srcu); + conn =srcu_dereference (*connp, &fsnotify_mark_srcu);if (!conn) -return NULL ; + goto out;spin_lock (&conn->lock); +if (!(conn->flags & (FSNOTIFY_OBJ_TYPE_INODE | +FSNOTIFY_OBJ_TYPE_VFSMOUNT ))) { +spin_unlock (&conn->lock); +srcu_read_unlock (&fsnotify_mark_srcu, idx); +return NULL ; + } +out: +srcu_read_unlock (&fsnotify_mark_srcu, idx);return conn; }@@ -351,7 +408,7 @@ static int fsnotify_add_mark_list(struct fsnotify_mark *mark, connp = &real_mount (mnt)->mnt_fsnotify_marks; restart:spin_lock (&mark->lock); - conn =fsnotify_grab_connector (*connp); + conn =fsnotify_grab_connector (connp);if (!conn) {spin_unlock (&mark->lock); conn = kmem_cache_alloc(fsnotify_mark_connector_cachep,@@ -362,13 +419,15 @@ static int fsnotify_add_mark_list(struct fsnotify_mark *mark,INIT_HLIST_HEAD (&conn->list);if (inode) { conn->flags =FSNOTIFY_OBJ_TYPE_INODE ; - conn->inode = inode; + conn->inode =igrab (inode); }else { conn->flags =FSNOTIFY_OBJ_TYPE_VFSMOUNT ; conn->mnt = mnt; }if (cmpxchg (connp,NULL , conn)) {/* Someone else created list structure for us */ +if (inode) +iput (inode);kmem_cache_free (fsnotify_mark_connector_cachep, conn); } goto restart;@@ -377,7 +436,6 @@ static int fsnotify_add_mark_list(struct fsnotify_mark *mark,/* is mark the first mark? */ if (hlist_empty (&conn->list)) {hlist_add_head_rcu (&mark->obj_list, &conn->list); -igrab (inode); goto added; }@@ -485,7 +543,7 @@ struct fsnotify_mark *fsnotify_find_mark(struct fsnotify_mark_connector **connp,struct fsnotify_mark *mark;struct fsnotify_mark_connector *conn; - conn =fsnotify_grab_connector (*connp); + conn =fsnotify_grab_connector (connp);if (!conn)return NULL ;hlist_for_each_entry (mark, &conn->list, obj_list) {@@ -565,23 +623,20 @@ void fsnotify_detach_group_marks(struct fsnotify_group *group) } } +/* Destroy all marks attached to inode / vfsmount */ void fsnotify_destroy_marks (struct fsnotify_mark_connector **connp) {struct fsnotify_mark_connector *conn;struct fsnotify_mark *mark; -while ((conn =fsnotify_grab_connector (*connp))) { +while ((conn =fsnotify_grab_connector (connp))) {/* * We have to be careful since we can race with e.g. -* fsnotify_clear_marks_by_group() and once we drop 'lock', -* mark can get removed from the obj_list and destroyed. But -* we are holding mark reference so mark cannot be freed and -* calling fsnotify_destroy_mark() more than once is fine. +* fsnotify_clear_marks_by_group() and once we drop the list +* lock, mark can get removed from the obj_list and destroyed. +* But we are holding mark reference so mark cannot be freed +* and calling fsnotify_destroy_mark() more than once is fine. */ -if (hlist_empty (&conn->list)) { -spin_unlock (&conn->lock); -break ; - } mark = hlist_entry(conn->list.first, struct fsnotify_mark, obj_list);fsnotify_get_mark (mark);@@ -589,16 +644,6 @@ void fsnotify_destroy_marks(struct fsnotify_mark_connector **connp)fsnotify_destroy_mark (mark, mark->group);fsnotify_put_mark (mark); } -/* -* Skip freeing of connector when inode is just unlinked. We leave that -* until destroy_inode() is called at which moment nobody will be -* looking at the inode anymore. -*/ -if ((*connp)->flags &FSNOTIFY_OBJ_TYPE_INODE && - !((*connp)->inode->i_state &I_CLEAR )) -return ; -kmem_cache_free (fsnotify_mark_connector_cachep, *connp); - *connp =NULL ; }/* include/linux/fs.h +1
-2@@ -646,7 +646,7 @@ struct inode {#ifdef CONFIG_FSNOTIFY __u32 i_fsnotify_mask;/* all events this inode cares about */ -struct fsnotify_mark_connector *i_fsnotify_marks; +struct fsnotify_mark_connector __rcu *i_fsnotify_marks;#endif #if IS_ENABLED (CONFIG_FS_ENCRYPTION ) --2.10.2
Jan Kara
maintainerRe: [PATCH 09/25] fsnotify: Free fsnotify_mark_connector when there is no mark attached
Quoted 24 lines · Miklos Szeredi
Quoted 7 lines · Jan Kara
Currently we free
fsnotify_mark_connectorstructure only when inode / vfsmount is getting freed. This can however impose noticeable memory overhead when marks get attached to inodes only temporarily. So free the connector structure once the last mark is detached from the object. Since notification infrastructure can be working with the connector under the protection offsnotify_mark_srcu, we have to be careful and free thefsnotify_mark_connectoronly after SRCU period passes.This adds a bunch of complexity. How about instead:
- protect setting
*_fsnotify_marks(forward pointer) with lock in
containing object
- keep count of marks not freed for connector
- when object (inode or vfsmount) is being destroyed and*_fsnotify_marksis non-NULL:
o lock object; re-check forward pointer; lock connector; set
back-pointer to NULL; unlock connector; unlock object
- when mark counter goes to zero:
o lock connector; check back pointer, if nonzero:
+ trylock object, if failed: unlock connector, relax & retry
+ clear forward pointer; unlock object
o unlock connector
Quoted 2 lines
This ensures freeing only after srcu period (since the mark is already srcu protected).
I don't think this works. We have unlocked readers of *_fsnotify_marks protected only by SRCU. So you have to wait for SRCU period to end after you have cleared the *_fsnotify_marks pointer (so that no new reader of your structure can come in) and before freeing the connector structure so that you are sure there are no more readers dereferencing the old value of the pointer.
So the only complexity you'd save with your scheme is the one contained in fsnotify_grab_connector() where we want to grab connector->lock without having connector pinned. And that does not look like that big difference compared to your locking...
Honza
Quoted 299 lines
It also ensures that multiple connectors for the
same object won't exist at the same time (the connector will be reused if a new mark is added before the srcu period has passed).While the locking looks somewhat challenging, I think it's still going to be simpler than the current one.
Thanks,
MiklosQuoted 289 lines
Signed-off-by Jan Kara <[email protected]>--- fs/mount.h | 2 +- fs/notify/fsnotify.c | 9 +++-- fs/notify/fsnotify.h | 4 --- fs/notify/mark.c | 97 ++++++++++++++++++++++++++++++++++++++-------------- include/linux/fs.h | 2 +- 5 files changed, 79 insertions(+), 35 deletions(-)fs/mount.h +1
-1@@ -59,7 +59,7 @@ struct mount {struct mountpoint *mnt_mp;/* where is it mounted */ struct hlist_node mnt_mp_list;/* list mounts with the same mountpoint */ #ifdef CONFIG_FSNOTIFY -struct fsnotify_mark_connector *mnt_fsnotify_marks; +struct fsnotify_mark_connector __rcu *mnt_fsnotify_marks;__u32 mnt_fsnotify_mask;#endif int mnt_id;/* mount identifier */ fs/notify/fsnotify.c +6
-3@@ -228,7 +228,8 @@ int fsnotify(struct inode *to_tell, __u32 mask, const void *data, int data_is,if ((mask &FS_MODIFY ) || (test_mask & to_tell->i_fsnotify_mask)) { - inode_conn = to_tell->i_fsnotify_marks; + inode_conn =srcu_dereference (to_tell->i_fsnotify_marks, + &fsnotify_mark_srcu);if (inode_conn) inode_node =srcu_dereference (inode_conn->list.first, &fsnotify_mark_srcu);@@ -236,11 +237,13 @@ int fsnotify(struct inode *to_tell, __u32 mask, const void *data, int data_is,if (mnt && ((mask &FS_MODIFY ) || (test_mask & mnt->mnt_fsnotify_mask))) { - inode_conn = to_tell->i_fsnotify_marks; + inode_conn =srcu_dereference (to_tell->i_fsnotify_marks, + &fsnotify_mark_srcu);if (inode_conn) inode_node =srcu_dereference (inode_conn->list.first, &fsnotify_mark_srcu); - vfsmount_conn = mnt->mnt_fsnotify_marks; + vfsmount_conn =srcu_dereference (mnt->mnt_fsnotify_marks, + &fsnotify_mark_srcu);if (vfsmount_conn) vfsmount_node = srcu_dereference( vfsmount_conn->list.first,fs/notify/fsnotify.h +0
-4@@ -27,15 +27,11 @@ extern void fsnotify_destroy_marks(struct fsnotify_mark_connector **connp);/* run the list of all marks associated with inode and destroy them */ static inline void fsnotify_clear_marks_by_inode (struct inode *inode) { -if (!inode->i_fsnotify_marks) -return ;fsnotify_destroy_marks (&inode->i_fsnotify_marks); }/* run the list of all marks associated with vfsmount and destroy them */ static inline void fsnotify_clear_marks_by_mount (struct vfsmount *mnt) { -if (!real_mount (mnt)->mnt_fsnotify_marks) -return ;fsnotify_destroy_marks (&real_mount (mnt)->mnt_fsnotify_marks); }/* prepare for freeing all marks associated with given group */ fs/notify/mark.c +71
-26@@ -89,10 +89,14 @@ struct kmem_cache *fsnotify_mark_connector_cachep;static DEFINE_SPINLOCK (destroy_lock);static LIST_HEAD (destroy_list); +static struct fsnotify_mark_connector *connector_destroy_list;static void fsnotify_mark_destroy_workfn (struct work_struct *work); static DECLARE_DELAYED_WORK(reaper_work, fsnotify_mark_destroy_workfn); +static void fsnotify_connector_destroy_workfn (struct work_struct *work); +static DECLARE_WORK(connector_reaper_work, fsnotify_connector_destroy_workfn); +void fsnotify_get_mark (struct fsnotify_mark *mark) {atomic_inc (&mark->refcnt);@@ -143,22 +147,62 @@ void fsnotify_recalc_mask(struct fsnotify_mark_connector *conn) } } +/* Free all connectors queued for freeing once SRCU period ends */ +static void fsnotify_connector_destroy_workfn (struct work_struct *work) +{ +struct fsnotify_mark_connector *conn, *free; + +spin_lock (&destroy_lock); + conn = connector_destroy_list; + connector_destroy_list =NULL ; +spin_unlock (&destroy_lock); + +synchronize_srcu (&fsnotify_mark_srcu); +while (conn) { + free = conn; + conn = conn->destroy_next; +kmem_cache_free (fsnotify_mark_connector_cachep, free); + } +} +static struct inode *fsnotify_detach_from_object (struct fsnotify_mark *mark) {struct fsnotify_mark_connector *conn;struct inode *inode =NULL ; +bool free_conn = false; conn = mark->connector;spin_lock (&conn->lock);hlist_del_init_rcu (&mark->obj_list);if (hlist_empty (&conn->list)) { -if (conn->flags &FSNOTIFY_OBJ_TYPE_INODE ) +if (conn->flags &FSNOTIFY_OBJ_TYPE_INODE ) { inode = conn->inode; +rcu_assign_pointer (inode->i_fsnotify_marks,NULL ); + inode->i_fsnotify_mask =0 ; + conn->inode =NULL ; + conn->flags &= ~FSNOTIFY_OBJ_TYPE_INODE ; + }else if (conn->flags &FSNOTIFY_OBJ_TYPE_VFSMOUNT ) { +rcu_assign_pointer ( +real_mount (conn->mnt)->mnt_fsnotify_marks, +NULL ); +real_mount (conn->mnt)->mnt_fsnotify_mask =0 ; + conn->mnt =NULL ; + conn->flags &= ~FSNOTIFY_OBJ_TYPE_VFSMOUNT ; + } + free_conn = true; }else __fsnotify_recalc_mask (conn); mark->connector =NULL ;spin_unlock (&conn->lock); +if (free_conn) { +spin_lock (&destroy_lock); + conn->destroy_next = connector_destroy_list; + connector_destroy_list = conn; +spin_unlock (&destroy_lock); +queue_work (system_unbound_wq, &connector_reaper_work); + } +return inode; }@@ -320,11 +364,24 @@ int fsnotify_compare_groups(struct fsnotify_group *a, struct fsnotify_group *b)* they are sure list cannot go away under them. */ static struct fsnotify_mark_connector *fsnotify_grab_connector ( -struct fsnotify_mark_connector *conn) +struct fsnotify_mark_connector *__rcu *connp) { +struct fsnotify_mark_connector *conn; +int idx; + + idx =srcu_read_lock (&fsnotify_mark_srcu); + conn =srcu_dereference (*connp, &fsnotify_mark_srcu);if (!conn) -return NULL ; + goto out;spin_lock (&conn->lock); +if (!(conn->flags & (FSNOTIFY_OBJ_TYPE_INODE | +FSNOTIFY_OBJ_TYPE_VFSMOUNT ))) { +spin_unlock (&conn->lock); +srcu_read_unlock (&fsnotify_mark_srcu, idx); +return NULL ; + } +out: +srcu_read_unlock (&fsnotify_mark_srcu, idx);return conn; }@@ -351,7 +408,7 @@ static int fsnotify_add_mark_list(struct fsnotify_mark *mark, connp = &real_mount (mnt)->mnt_fsnotify_marks; restart:spin_lock (&mark->lock); - conn =fsnotify_grab_connector (*connp); + conn =fsnotify_grab_connector (connp);if (!conn) {spin_unlock (&mark->lock); conn = kmem_cache_alloc(fsnotify_mark_connector_cachep,@@ -362,13 +419,15 @@ static int fsnotify_add_mark_list(struct fsnotify_mark *mark,INIT_HLIST_HEAD (&conn->list);if (inode) { conn->flags =FSNOTIFY_OBJ_TYPE_INODE ; - conn->inode = inode; + conn->inode =igrab (inode); }else { conn->flags =FSNOTIFY_OBJ_TYPE_VFSMOUNT ; conn->mnt = mnt; }if (cmpxchg (connp,NULL , conn)) {/* Someone else created list structure for us */ +if (inode) +iput (inode);kmem_cache_free (fsnotify_mark_connector_cachep, conn); } goto restart;@@ -377,7 +436,6 @@ static int fsnotify_add_mark_list(struct fsnotify_mark *mark,/* is mark the first mark? */ if (hlist_empty (&conn->list)) {hlist_add_head_rcu (&mark->obj_list, &conn->list); -igrab (inode); goto added; }@@ -485,7 +543,7 @@ struct fsnotify_mark *fsnotify_find_mark(struct fsnotify_mark_connector **connp,struct fsnotify_mark *mark;struct fsnotify_mark_connector *conn; - conn =fsnotify_grab_connector (*connp); + conn =fsnotify_grab_connector (connp);if (!conn)return NULL ;hlist_for_each_entry (mark, &conn->list, obj_list) {@@ -565,23 +623,20 @@ void fsnotify_detach_group_marks(struct fsnotify_group *group) } } +/* Destroy all marks attached to inode / vfsmount */ void fsnotify_destroy_marks (struct fsnotify_mark_connector **connp) {struct fsnotify_mark_connector *conn;struct fsnotify_mark *mark; -while ((conn =fsnotify_grab_connector (*connp))) { +while ((conn =fsnotify_grab_connector (connp))) {/* * We have to be careful since we can race with e.g. -* fsnotify_clear_marks_by_group() and once we drop 'lock', -* mark can get removed from the obj_list and destroyed. But -* we are holding mark reference so mark cannot be freed and -* calling fsnotify_destroy_mark() more than once is fine. +* fsnotify_clear_marks_by_group() and once we drop the list +* lock, mark can get removed from the obj_list and destroyed. +* But we are holding mark reference so mark cannot be freed +* and calling fsnotify_destroy_mark() more than once is fine. */ -if (hlist_empty (&conn->list)) { -spin_unlock (&conn->lock); -break ; - } mark = hlist_entry(conn->list.first, struct fsnotify_mark, obj_list);fsnotify_get_mark (mark);@@ -589,16 +644,6 @@ void fsnotify_destroy_marks(struct fsnotify_mark_connector **connp)fsnotify_destroy_mark (mark, mark->group);fsnotify_put_mark (mark); } -/* -* Skip freeing of connector when inode is just unlinked. We leave that -* until destroy_inode() is called at which moment nobody will be -* looking at the inode anymore. -*/ -if ((*connp)->flags &FSNOTIFY_OBJ_TYPE_INODE && - !((*connp)->inode->i_state &I_CLEAR )) -return ; -kmem_cache_free (fsnotify_mark_connector_cachep, *connp); - *connp =NULL ; }/* include/linux/fs.h +1
-2@@ -646,7 +646,7 @@ struct inode {#ifdef CONFIG_FSNOTIFY __u32 i_fsnotify_mask;/* all events this inode cares about */ -struct fsnotify_mark_connector *i_fsnotify_marks; +struct fsnotify_mark_connector __rcu *i_fsnotify_marks;#endif #if IS_ENABLED (CONFIG_FS_ENCRYPTION ) --2.10.2
Signature · 2 lines
Jan Kara <[email protected]> SUSE Labs, CR
Jan Kara
maintainerPATCH v4 10/25 inotify: Do not drop mark reference under idr_lock
In reply to Jan KaraDropping mark reference can result in mark being freed. Although it should not happen in inotify_remove_from_idr() since caller should hold another reference, just don't risk lock up just after WARN_ON unnecessarily. Also fold do_inotify_remove_from_idr() into the single callsite as that function really is just two lines of real code.
--- fs/notify/inotify/inotify_user.c | 24 ++++++------------------ 1 file changed, 6 insertions(+), 18 deletions(-)
fs/notify/inotify/inotify_user.c +6 -18
@@ -397,21 +397,6 @@ static struct inotify_inode_mark *inotify_idr_find(struct fsnotify_group *group,return i_mark; } -static void do_inotify_remove_from_idr (struct fsnotify_group *group, -struct inotify_inode_mark *i_mark) -{ -struct idr *idr = &group->inotify_data.idr; -spinlock_t *idr_lock = &group->inotify_data.idr_lock; -int wd = i_mark->wd; - -assert_spin_locked (idr_lock); - -idr_remove (idr, wd); - -/* removed from the idr, drop that ref */ -fsnotify_put_mark (&i_mark->fsn_mark); -} -/* * Remove the mark from the idr (if present) and drop the reference * on the mark because it was in the idr.
@@ -419,6 +404,7 @@ static void do_inotify_remove_from_idr(struct fsnotify_group *group,static void inotify_remove_from_idr (struct fsnotify_group *group,struct inotify_inode_mark *i_mark) { +struct idr *idr = &group->inotify_data.idr;spinlock_t *idr_lock = &group->inotify_data.idr_lock;struct inotify_inode_mark *found_i_mark =NULL ;int wd;
@@ -470,13 +456,15 @@ static void inotify_remove_from_idr(struct fsnotify_group *group,BUG (); } -do_inotify_remove_from_idr (group, i_mark); +idr_remove (idr, wd); +/* Removed from the idr, drop that ref. */ +fsnotify_put_mark (&i_mark->fsn_mark); out: + i_mark->wd =-1 ; +spin_unlock (idr_lock);/* match the ref taken by inotify_idr_find_locked() */ if (found_i_mark)fsnotify_put_mark (&found_i_mark->fsn_mark); - i_mark->wd =-1 ; -spin_unlock (idr_lock); }/*
Signature · 1 line
2.10.2
Jan Kara
maintainerPATCH v4 11/25 fsnotify: Move queueing of mark for destruction into fsnotify_put_mark()
In reply to Jan KaraCurrently we queue mark into a list of marks for destruction in __fsnotify_free_mark() and keep the last mark reference dangling. After the worker waits for SRCU period, it drops the last reference to the mark which frees it. This scheme has the disadvantage that if we hold reference to a mark and drop and reacquire SRCU lock, the mark can get freed immediately which is slightly inconvenient and we will need to avoid this in the future.
Move to a scheme where queueing of mark into a list of marks for destruction happens when the last reference to the mark is dropped. Also drop reference to the mark held by group list already when mark is removed from that list instead of dropping it only from the destruction worker.
--- fs/notify/mark.c | 73 ++++++++++++++++++++++---------------------------------- 1 file changed, 29 insertions(+), 44 deletions(-)
fs/notify/mark.c +29 -44
@@ -99,15 +99,18 @@ static DECLARE_WORK(connector_reaper_work, fsnotify_connector_destroy_workfn);void fsnotify_get_mark (struct fsnotify_mark *mark) { +WARN_ON_ONCE (!atomic_read (&mark->refcnt));atomic_inc (&mark->refcnt); }void fsnotify_put_mark (struct fsnotify_mark *mark) {if (atomic_dec_and_test (&mark->refcnt)) { -if (mark->group) -fsnotify_put_group (mark->group); - mark->free_mark (mark); +spin_lock (&destroy_lock); +list_add (&mark->g_list, &destroy_list); +spin_unlock (&destroy_lock); +queue_delayed_work (system_unbound_wq, &reaper_work, +FSNOTIFY_REAPER_DELAY ); } }
@@ -210,14 +213,18 @@ static struct inode *fsnotify_detach_from_object(struct fsnotify_mark *mark)* Remove mark from inode / vfsmount list, group list, drop inode reference * if we got one. * -* Must be called with group->mark_mutex held. +* Must be called with group->mark_mutex held. The caller must either hold +* reference to the mark or be protected by fsnotify_mark_srcu. */ void fsnotify_detach_mark (struct fsnotify_mark *mark) {struct inode *inode =NULL ;struct fsnotify_group *group = mark->group; -BUG_ON (!mutex_is_locked (&group->mark_mutex)); +WARN_ON_ONCE (!mutex_is_locked (&group->mark_mutex)); +WARN_ON_ONCE (!srcu_read_lock_held (&fsnotify_mark_srcu) && +atomic_read (&mark->refcnt) <1 + + !!(mark->flags &FSNOTIFY_MARK_FLAG_ATTACHED ));spin_lock (&mark->lock);
@@ -246,18 +253,20 @@ void fsnotify_detach_mark(struct fsnotify_mark *mark)iput (inode);atomic_dec (&group->num_marks); + +/* Drop mark reference acquired in fsnotify_add_mark_locked() */ +fsnotify_put_mark (mark); }/* -* Prepare mark for freeing and add it to the list of marks prepared for -* freeing. The actual freeing must happen after SRCU period ends and the -* caller is responsible for this. +* Free fsnotify mark. The mark is actually only marked as being freed. The +* freeing is actually happening only once last reference to the mark is +* dropped from a workqueue which first waits for srcu period end. * -* The function returns true if the mark was added to the list of marks for -* freeing. The function returns false if someone else has already called -* __fsnotify_free_mark() for the mark. +* Caller must have a reference to the mark or be protected by +* fsnotify_mark_srcu. */ -static bool __fsnotify_free_mark (struct fsnotify_mark *mark) +void fsnotify_free_mark (struct fsnotify_mark *mark) {struct fsnotify_group *group = mark->group;
@@ -265,7 +274,7 @@ static bool __fsnotify_free_mark(struct fsnotify_mark *mark)/* something else already called this function on this mark */ if (!(mark->flags &FSNOTIFY_MARK_FLAG_ALIVE )) {spin_unlock (&mark->lock); -return false; +return ; } mark->flags &= ~FSNOTIFY_MARK_FLAG_ALIVE ;spin_unlock (&mark->lock);
@@ -277,25 +286,6 @@ static bool __fsnotify_free_mark(struct fsnotify_mark *mark)*/ if (group->ops->freeing_mark) group->ops->freeing_mark (mark, group); - -spin_lock (&destroy_lock); -list_add (&mark->g_list, &destroy_list); -spin_unlock (&destroy_lock); - -return true; -} - -/* -* Free fsnotify mark. The freeing is actually happening from a workqueue which -* first waits for srcu period end. Caller must have a reference to the mark -* or be protected by fsnotify_mark_srcu. -*/ -void fsnotify_free_mark (struct fsnotify_mark *mark) -{ -if (__fsnotify_free_mark (mark)) { -queue_delayed_work (system_unbound_wq, &reaper_work, -FSNOTIFY_REAPER_DELAY ); - } } void fsnotify_destroy_mark(struct fsnotify_mark *mark,
@@ -506,20 +496,13 @@ int fsnotify_add_mark_locked(struct fsnotify_mark *mark,return ret; err: - mark->flags &= ~FSNOTIFY_MARK_FLAG_ALIVE ; + mark->flags &= ~(FSNOTIFY_MARK_FLAG_ALIVE | +FSNOTIFY_MARK_FLAG_ATTACHED );list_del_init (&mark->g_list); -fsnotify_put_group (group); - mark->group =NULL ;atomic_dec (&group->num_marks); -spin_unlock (&mark->lock); -spin_lock (&destroy_lock); -list_add (&mark->g_list, &destroy_list); -spin_unlock (&destroy_lock); -queue_delayed_work (system_unbound_wq, &reaper_work, -FSNOTIFY_REAPER_DELAY ); - +fsnotify_put_mark (mark);return ret; }
@@ -618,7 +601,7 @@ void fsnotify_detach_group_marks(struct fsnotify_group *group)fsnotify_get_mark (mark);fsnotify_detach_mark (mark);mutex_unlock (&group->mark_mutex); -__fsnotify_free_mark (mark); +fsnotify_free_mark (mark);fsnotify_put_mark (mark); } }
@@ -676,7 +659,9 @@ void fsnotify_mark_destroy_list(void)list_for_each_entry_safe (mark, next, &private_destroy_list, g_list) {list_del_init (&mark->g_list); -fsnotify_put_mark (mark); +if (mark->group) +fsnotify_put_group (mark->group); + mark->free_mark (mark); } }
Signature · 1 line
2.10.2
Jan Kara
maintainerPATCH v4 12/25 fsnotify: Detach mark from object list when last reference is dropped
In reply to Jan KaraInstead of removing mark from object list from fsnotify_detach_mark(), remove the mark when last reference to the mark is dropped. This will allow fanotify to wait for userspace response to event without having to hold onto fsnotify_mark_srcu.
--- fs/notify/mark.c | 172 ++++++++++++++++++++++++--------------- include/linux/fsnotify_backend.h | 4 +- kernel/audit_tree.c | 20 +---- 3 files changed, 111 insertions(+), 85 deletions(-)
fs/notify/mark.c +105 -67
@@ -49,7 +49,13 @@* * A list of notification marks relating to inode / mnt is contained in * fsnotify_mark_connector. That structure is alive as long as there are any marks -* in the list and is also protected by fsnotify_mark_srcu. +* in the list and is also protected by fsnotify_mark_srcu. A mark gets +* detached from fsnotify_mark_connector when last reference to the mark is dropped. +* Thus having mark reference is enough to protect mark->connector pointer +* and to make sure fsnotify_mark_connector cannot disappear. Also because we remove +* mark from g_list before dropping mark reference associated with that, any +* mark found through g_list is guaranteed to have mark->connector set +* until we drop group->mark_mutex. * * LIFETIME: * Inode marks survive between when they are added to an inode and when their
@@ -103,17 +109,6 @@ void fsnotify_get_mark(struct fsnotify_mark *mark)atomic_inc (&mark->refcnt); } -void fsnotify_put_mark (struct fsnotify_mark *mark) -{ -if (atomic_dec_and_test (&mark->refcnt)) { -spin_lock (&destroy_lock); -list_add (&mark->g_list, &destroy_list); -spin_unlock (&destroy_lock); -queue_delayed_work (system_unbound_wq, &reaper_work, -FSNOTIFY_REAPER_DELAY ); - } -} -static void __fsnotify_recalc_mask (struct fsnotify_mark_connector *conn) {u32 new_mask =0 ;
@@ -168,36 +163,67 @@ static void fsnotify_connector_destroy_workfn(struct work_struct *work) } } -static struct inode *fsnotify_detach_from_object (struct fsnotify_mark *mark) +static struct inode *fsnotify_detach_connector_from_object ( +struct fsnotify_mark_connector *conn) +{ +struct inode *inode =NULL ; + +if (conn->flags &FSNOTIFY_OBJ_TYPE_INODE ) { + inode = conn->inode; +rcu_assign_pointer (inode->i_fsnotify_marks,NULL ); + inode->i_fsnotify_mask =0 ; + conn->inode =NULL ; + conn->flags &= ~FSNOTIFY_OBJ_TYPE_INODE ; + }else if (conn->flags &FSNOTIFY_OBJ_TYPE_VFSMOUNT ) { +rcu_assign_pointer (real_mount (conn->mnt)->mnt_fsnotify_marks, +NULL ); +real_mount (conn->mnt)->mnt_fsnotify_mask =0 ; + conn->mnt =NULL ; + conn->flags &= ~FSNOTIFY_OBJ_TYPE_VFSMOUNT ; + } + +return inode; +} + +static void fsnotify_final_mark_destroy (struct fsnotify_mark *mark) +{ +if (mark->group) +fsnotify_put_group (mark->group); + mark->free_mark (mark); +} + +void fsnotify_put_mark (struct fsnotify_mark *mark) {struct fsnotify_mark_connector *conn;struct inode *inode =NULL ;bool free_conn = false; +/* Catch marks that were actually never attached to object */ +if (!mark->connector &&atomic_dec_and_test (&mark->refcnt)) { +fsnotify_final_mark_destroy (mark); +return ; + } + +/* +* We have to be careful so that traversals of obj_list under lock can +* safely grab mark reference. +*/ +if (!atomic_dec_and_lock (&mark->refcnt, &mark->connector->lock)) +return ; + conn = mark->connector; -spin_lock (&conn->lock);hlist_del_init_rcu (&mark->obj_list);if (hlist_empty (&conn->list)) { -if (conn->flags &FSNOTIFY_OBJ_TYPE_INODE ) { - inode = conn->inode; -rcu_assign_pointer (inode->i_fsnotify_marks,NULL ); - inode->i_fsnotify_mask =0 ; - conn->inode =NULL ; - conn->flags &= ~FSNOTIFY_OBJ_TYPE_INODE ; - }else if (conn->flags &FSNOTIFY_OBJ_TYPE_VFSMOUNT ) { -rcu_assign_pointer ( -real_mount (conn->mnt)->mnt_fsnotify_marks, -NULL ); -real_mount (conn->mnt)->mnt_fsnotify_mask =0 ; - conn->mnt =NULL ; - conn->flags &= ~FSNOTIFY_OBJ_TYPE_VFSMOUNT ; - } + inode =fsnotify_detach_connector_from_object (conn); free_conn = true; }else __fsnotify_recalc_mask (conn); mark->connector =NULL ;spin_unlock (&conn->lock); +if (inode) +iput (inode); +if (free_conn) {spin_lock (&destroy_lock); conn->destroy_next = connector_destroy_list;
@@ -205,20 +231,31 @@ static struct inode *fsnotify_detach_from_object(struct fsnotify_mark *mark)spin_unlock (&destroy_lock);queue_work (system_unbound_wq, &connector_reaper_work); } - -return inode; +/* +* Note that we didn't update flags telling whether inode cares about +* what's happening with children. We update these flags from +* __fsnotify_parent() lazily when next event happens on one of our +* children. +*/ +spin_lock (&destroy_lock); +list_add (&mark->g_list, &destroy_list); +spin_unlock (&destroy_lock); +queue_delayed_work (system_unbound_wq, &reaper_work, +FSNOTIFY_REAPER_DELAY ); }/* -* Remove mark from inode / vfsmount list, group list, drop inode reference -* if we got one. +* Mark mark as detached, remove it from group list. Mark still stays in object +* list until its last reference is dropped. Note that we rely on mark being +* removed from group list before corresponding reference to it is dropped. In +* particular we rely on mark->connector being valid while we hold +* group->mark_mutex if we found the mark through g_list. * * Must be called with group->mark_mutex held. The caller must either hold * reference to the mark or be protected by fsnotify_mark_srcu. */ void fsnotify_detach_mark (struct fsnotify_mark *mark) { -struct inode *inode =NULL ;struct fsnotify_group *group = mark->group;WARN_ON_ONCE (!mutex_is_locked (&group->mark_mutex));
@@ -227,31 +264,15 @@ void fsnotify_detach_mark(struct fsnotify_mark *mark) !!(mark->flags & FSNOTIFY_MARK_FLAG_ATTACHED));spin_lock (&mark->lock); -/* something else already called this function on this mark */ if (!(mark->flags &FSNOTIFY_MARK_FLAG_ATTACHED )) {spin_unlock (&mark->lock);return ; } - mark->flags &= ~FSNOTIFY_MARK_FLAG_ATTACHED ; - - inode =fsnotify_detach_from_object (mark); - -/* -* Note that we didn't update flags telling whether inode cares about -* what's happening with children. We update these flags from -* __fsnotify_parent() lazily when next event happens on one of our -* children. -*/ -list_del_init (&mark->g_list); -spin_unlock (&mark->lock); -if (inode) -iput (inode); -atomic_dec (&group->num_marks);/* Drop mark reference acquired in fsnotify_add_mark_locked() */
@@ -433,7 +454,9 @@ static int fsnotify_add_mark_list(struct fsnotify_mark *mark,hlist_for_each_entry (lmark, &conn->list, obj_list) { last = lmark; -if ((lmark->group == mark->group) && !allow_dups) { +if ((lmark->group == mark->group) && + (lmark->flags &FSNOTIFY_MARK_FLAG_ATTACHED ) && + !allow_dups) { err = -EEXIST ; goto out_err; }
@@ -484,7 +507,7 @@ int fsnotify_add_mark_locked(struct fsnotify_mark *mark, mark->group = group;list_add (&mark->g_list, &group->marks_list);atomic_inc (&group->num_marks); -fsnotify_get_mark (mark);/* for i_list and g_list */ +fsnotify_get_mark (mark);/* for g_list */ spin_unlock (&mark->lock); ret =fsnotify_add_mark_list (mark, inode, mnt, allow_dups);
@@ -530,7 +553,8 @@ struct fsnotify_mark *fsnotify_find_mark(struct fsnotify_mark_connector **connp,if (!conn)return NULL ;hlist_for_each_entry (mark, &conn->list, obj_list) { -if (mark->group == group) { +if (mark->group == group && + (mark->flags &FSNOTIFY_MARK_FLAG_ATTACHED )) {fsnotify_get_mark (mark);spin_unlock (&conn->lock);return mark;
@@ -610,23 +634,39 @@ void fsnotify_detach_group_marks(struct fsnotify_group *group)void fsnotify_destroy_marks (struct fsnotify_mark_connector **connp) {struct fsnotify_mark_connector *conn; -struct fsnotify_mark *mark; +struct fsnotify_mark *mark, *old_mark =NULL ; +struct inode *inode; -while ((conn =fsnotify_grab_connector (connp))) { -/* -* We have to be careful since we can race with e.g. -* fsnotify_clear_marks_by_group() and once we drop the list -* lock, mark can get removed from the obj_list and destroyed. -* But we are holding mark reference so mark cannot be freed -* and calling fsnotify_destroy_mark() more than once is fine. -*/ - mark = hlist_entry(conn->list.first, struct fsnotify_mark, - obj_list); + conn =fsnotify_grab_connector (connp); +if (!conn) +return ; +/* +* We have to be careful since we can race with e.g. +* fsnotify_clear_marks_by_group() and once we drop the conn->lock, the +* list can get modified. However we are holding mark reference and +* thus our mark cannot be removed from obj_list so we can continue +* iteration after regaining conn->lock. +*/ +hlist_for_each_entry (mark, &conn->list, obj_list) {fsnotify_get_mark (mark);spin_unlock (&conn->lock); +if (old_mark) +fsnotify_put_mark (old_mark); + old_mark = mark;fsnotify_destroy_mark (mark, mark->group); -fsnotify_put_mark (mark); +spin_lock (&conn->lock); } +/* +* Detach list from object now so that we don't pin inode until all +* mark references get dropped. It would lead to strange results such +* as delaying inode deletion or blocking unmount. +*/ + inode =fsnotify_detach_connector_from_object (conn); +spin_unlock (&conn->lock); +if (old_mark) +fsnotify_put_mark (old_mark); +if (inode) +iput (inode); }/*
@@ -659,9 +699,7 @@ void fsnotify_mark_destroy_list(void)list_for_each_entry_safe (mark, next, &private_destroy_list, g_list) {list_del_init (&mark->g_list); -if (mark->group) -fsnotify_put_group (mark->group); - mark->free_mark (mark); +fsnotify_final_mark_destroy (mark); } }
include/linux/fsnotify_backend.h +2 -2
@@ -244,9 +244,9 @@ struct fsnotify_mark {struct list_head g_list;/* Protects inode / mnt pointers, flags, masks */ spinlock_t lock; -/* List of marks for inode / vfsmount [connector->lock] */ +/* List of marks for inode / vfsmount [connector->lock, mark ref] */ struct hlist_node obj_list; -/* Head of list of marks for an object [mark->lock, group->mark_mutex] */ +/* Head of list of marks for an object [mark ref] */ struct fsnotify_mark_connector *connector;/* Events types to ignore [mark->lock, group->mark_mutex] */ __u32 ignored_mask;
kernel/audit_tree.c +4 -16
@@ -172,27 +172,15 @@ static unsigned long inode_to_key(const struct inode *inode)/* * Function to return search key in our hash from chunk. Key 0 is special and * should never be present in the hash. -* -* Must be called with chunk->mark.lock held to protect from connector -* becoming NULL. */ -static unsigned long __chunk_to_key (struct audit_chunk *chunk) +static unsigned long chunk_to_key (struct audit_chunk *chunk) { -if (!chunk->mark.connector) +/* We have a reference to the mark so it should be attached. */ +if (WARN_ON_ONCE (!chunk->mark.connector))return 0 ;return (unsigned long )chunk->mark.connector->inode; } -static unsigned long chunk_to_key (struct audit_chunk *chunk) -{ -unsigned long key; - -spin_lock (&chunk->mark.lock); - key =__chunk_to_key (chunk); -spin_unlock (&chunk->mark.lock); -return key; -} -static inline struct list_head *chunk_hash (unsigned long key) {unsigned long n = key /L1_CACHE_BYTES ;
@@ -202,7 +190,7 @@ static inline struct list_head *chunk_hash(unsigned long key)/* hash_lock & entry->lock is held by caller */ static void insert_hash (struct audit_chunk *chunk) { -unsigned long key =__chunk_to_key (chunk); +unsigned long key =chunk_to_key (chunk);struct list_head *list;if (!key)
Signature · 1 line
2.10.2
Jan Kara
maintainerPATCH v4 13/25 fsnotify: Remove special handling of mark destruction on group shutdown
In reply to Jan KaraCurrently we queue all marks for destruction on group shutdown and then destroy them from fsnotify_destroy_group() instead from a worker thread which is the usual path. However worker can already be processing some list of marks to destroy so this does not make 100% all marks are really destroyed by the time group is shut down. This isn't a big problem as each mark holds group reference and thus group stays partially alive until all marks are really freed but there's no point in complicating our lives - just wait for the delayed work to be finished instead.
--- fs/notify/fsnotify.h | 6 ++---- fs/notify/group.c | 10 ++++++---- fs/notify/mark.c | 7 ++++--- 3 files changed, 12 insertions(+), 11 deletions(-)
fs/notify/fsnotify.h +2 -4
@@ -36,10 +36,8 @@ static inline void fsnotify_clear_marks_by_mount(struct vfsmount *mnt) }/* prepare for freeing all marks associated with given group */ extern void fsnotify_detach_group_marks (struct fsnotify_group *group); -/* -* wait for fsnotify_mark_srcu period to end and free all marks in destroy_list -*/ -extern void fsnotify_mark_destroy_list (void ); +/* Wait until all marks queued for destruction are destroyed */ +extern void fsnotify_wait_marks_destroyed (void );/* * update the dentry->d_flags of all of inode's children to indicate if inode cares
fs/notify/group.c +6 -4
@@ -66,14 +66,16 @@ void fsnotify_destroy_group(struct fsnotify_group *group)*/ fsnotify_group_stop_queueing (group); -/* clear all inode marks for this group, attach them to destroy_list */ +/* Clear all marks for this group and queue them for destruction */ fsnotify_detach_group_marks (group);/* -* Wait for fsnotify_mark_srcu period to end and free all marks in -* destroy_list +* Wait until all marks get really destroyed. We could actually destroy +* them ourselves instead of waiting for worker to do it, however that +* would be racy as worker can already be processing some marks before +* we even entered fsnotify_destroy_group(). */ -fsnotify_mark_destroy_list (); +fsnotify_wait_marks_destroyed ();/* * Since we have waited for fsnotify_mark_srcu in
fs/notify/mark.c +4 -3
@@ -685,7 +685,7 @@ void fsnotify_init_mark(struct fsnotify_mark *mark,* Destroy all marks in destroy_list, waits for SRCU period to finish before * actually freeing marks. */ -void fsnotify_mark_destroy_list (void ) +static void fsnotify_mark_destroy_workfn (struct work_struct *work) {struct fsnotify_mark *mark, *next;struct list_head private_destroy_list;
@@ -703,7 +703,8 @@ void fsnotify_mark_destroy_list(void) } } -static void fsnotify_mark_destroy_workfn (struct work_struct *work) +/* Wait for all marks queued for destruction to be actually destroyed */ +void fsnotify_wait_marks_destroyed (void ) { -fsnotify_mark_destroy_list (); +flush_delayed_work (&reaper_work); }
Signature · 1 line
2.10.2
Jan Kara
maintainerPATCH v4 14/25 fsnotify: Provide framework for dropping SRCU lock in ->handle_event
In reply to Jan Karafanotify wants to drop fsnotify_mark_srcu lock when waiting for response from userspace so that the whole notification subsystem is not blocked during that time. This patch provides a framework for safely getting mark reference for a mark found in the object list which pins the mark in that list. We can then drop fsnotify_mark_srcu, wait for userspace response and then safely continue iteration of the object list once we reaquire fsnotify_mark_srcu.
--- fs/notify/fsnotify.h | 6 +++ fs/notify/group.c | 1 + fs/notify/mark.c | 82 ++++++++++++++++++++++++++++++++++++++++ include/linux/fsnotify_backend.h | 5 +++ 4 files changed, 94 insertions(+)
fs/notify/fsnotify.h +6 -0
@@ -8,6 +8,12 @@#include "../mount.h" +struct fsnotify_iter_info { +struct fsnotify_mark *inode_mark; +struct fsnotify_mark *vfsmount_mark; +int srcu_idx; +}; +/* destroy all events sitting in this groups notification queue */ extern void fsnotify_flush_notify (struct fsnotify_group *group);
fs/notify/group.c +1 -0
@@ -126,6 +126,7 @@ struct fsnotify_group *fsnotify_alloc_group(const struct fsnotify_ops *ops)/* set to 0 when there a no external references to this group */ atomic_set (&group->refcnt,1 );atomic_set (&group->num_marks,0 ); +atomic_set (&group->user_waits,0 );spin_lock_init (&group->notification_lock);INIT_LIST_HEAD (&group->notification_list);
fs/notify/mark.c +82 -0
100 lines not on this page · raw message
include/linux/fsnotify_backend.h +5 -0
@@ -79,6 +79,7 @@ struct fsnotify_event;struct fsnotify_mark ;struct fsnotify_event_private_data ;struct fsnotify_fname ; +struct fsnotify_iter_info ;/* * Each group much define these ops. The fsnotify infrastructure will call
@@ -162,6 +163,8 @@ struct fsnotify_group {struct fsnotify_event *overflow_event;/* Event we queue when the * notification list is too * full */ +atomic_t user_waits;/* Number of tasks waiting for user +* response */ /* groups can define private fields here or use the void *private */ union {
@@ -367,6 +370,8 @@ extern void fsnotify_clear_marks_by_group_flags(struct fsnotify_group *group, unextern void fsnotify_get_mark (struct fsnotify_mark *mark);extern void fsnotify_put_mark (struct fsnotify_mark *mark);extern void fsnotify_unmount_inodes (struct super_block *sb); +extern void fsnotify_finish_user_wait (struct fsnotify_iter_info *iter_info); +extern bool fsnotify_prepare_user_wait (struct fsnotify_iter_info *iter_info);/* put here because inotify does some weird stuff when destroying watches */ extern void fsnotify_init_event(struct fsnotify_event *event,
Signature · 1 line
2.10.2
Jan Kara
maintainerPATCH v4 15/25 fsnotify: Pass fsnotify_iter_info into handle_event handler
In reply to Jan KaraPass fsnotify_iter_info into ->handle_event() handler so that it can release and reacquire SRCU lock via fsnotify_prepare_user_wait() and fsnotify_finish_user_wait() functions. These functions also make sure current marks are appropriately pinned so that iteration protected by srcu in fsnotify() stays safe.
--- fs/notify/dnotify/dnotify.c | 3 ++- fs/notify/fanotify/fanotify.c | 3 ++- fs/notify/fsnotify.c | 19 +++++++++++++------ fs/notify/inotify/inotify.h | 3 ++- fs/notify/inotify/inotify_fsnotify.c | 3 ++- fs/notify/inotify/inotify_user.c | 2 +- include/linux/fsnotify_backend.h | 3 ++- kernel/audit_fsnotify.c | 3 ++- kernel/audit_tree.c | 3 ++- kernel/audit_watch.c | 3 ++- 10 files changed, 30 insertions(+), 15 deletions(-)
fs/notify/dnotify/dnotify.c +2 -1
@@ -85,7 +85,8 @@ static int dnotify_handle_event(struct fsnotify_group *group,struct fsnotify_mark *inode_mark, struct fsnotify_mark *vfsmount_mark, u32 mask, const void *data, int data_type, - const unsigned char *file_name, u32 cookie) + const unsigned char *file_name, u32 cookie, + struct fsnotify_iter_info *iter_info) { struct dnotify_mark *dn_mark;struct dnotify_struct *dn;
fs/notify/fanotify/fanotify.c +2 -1
@@ -178,7 +178,8 @@ static int fanotify_handle_event(struct fsnotify_group *group,struct fsnotify_mark *inode_mark, struct fsnotify_mark *fanotify_mark, u32 mask, const void *data, int data_type, - const unsigned char *file_name, u32 cookie) + const unsigned char *file_name, u32 cookie, + struct fsnotify_iter_info *iter_info) { int ret = 0;struct fanotify_event_info *event;
fs/notify/fsnotify.c +13 -6
@@ -127,7 +127,8 @@ static int send_to_group(struct inode *to_tell,struct fsnotify_mark *vfsmount_mark, __u32 mask, const void *data, int data_is, u32 cookie, - const unsigned char *file_name) + const unsigned char *file_name, + struct fsnotify_iter_info *iter_info) { struct fsnotify_group *group = NULL;__u32 inode_test_mask =0 ;
@@ -178,7 +179,7 @@ static int send_to_group(struct inode *to_tell,return group->ops->handle_event (group, to_tell, inode_mark, vfsmount_mark, mask, data, data_is, - file_name, cookie); + file_name, cookie, iter_info); }/*
@@ -194,8 +195,9 @@ int fsnotify(struct inode *to_tell, __u32 mask, const void *data, int data_is,struct fsnotify_mark *inode_mark =NULL , *vfsmount_mark =NULL ;struct fsnotify_group *inode_group, *vfsmount_group;struct fsnotify_mark_connector *inode_conn, *vfsmount_conn; +struct fsnotify_iter_info iter_info;struct mount *mnt; -int idx, ret =0 ; +int ret =0 ;/* global tests shouldn't care about events on child only the specific event */ __u32 test_mask = (mask & ~FS_EVENT_ON_CHILD );
@@ -224,7 +226,7 @@ int fsnotify(struct inode *to_tell, __u32 mask, const void *data, int data_is, !(mnt && test_mask & mnt->mnt_fsnotify_mask))return 0 ; - idx =srcu_read_lock (&fsnotify_mark_srcu); + iter_info.srcu_idx =srcu_read_lock (&fsnotify_mark_srcu);if ((mask &FS_MODIFY ) || (test_mask & to_tell->i_fsnotify_mask)) {
@@ -284,8 +286,13 @@ int fsnotify(struct inode *to_tell, __u32 mask, const void *data, int data_is, vfsmount_mark =NULL ; } } + + iter_info.inode_mark = inode_mark; + iter_info.vfsmount_mark = vfsmount_mark; + ret =send_to_group (to_tell, inode_mark, vfsmount_mark, mask, - data, data_is, cookie, file_name); + data, data_is, cookie, file_name, + &iter_info);if (ret && (mask &ALL_FSNOTIFY_PERM_EVENTS )) goto out;
@@ -299,7 +306,7 @@ int fsnotify(struct inode *to_tell, __u32 mask, const void *data, int data_is, } ret =0 ; out: -srcu_read_unlock (&fsnotify_mark_srcu, idx); +srcu_read_unlock (&fsnotify_mark_srcu, iter_info.srcu_idx);return ret; }
fs/notify/inotify/inotify.h +2 -1
@@ -27,6 +27,7 @@ extern int inotify_handle_event(struct fsnotify_group *group,struct fsnotify_mark *inode_mark, struct fsnotify_mark *vfsmount_mark, u32 mask, const void *data, int data_type, - const unsigned char *file_name, u32 cookie); + const unsigned char *file_name, u32 cookie, + struct fsnotify_iter_info *iter_info);extern const struct fsnotify_ops inotify_fsnotify_ops;
fs/notify/inotify/inotify_fsnotify.c +2 -1
@@ -67,7 +67,8 @@ int inotify_handle_event(struct fsnotify_group *group,struct fsnotify_mark *inode_mark, struct fsnotify_mark *vfsmount_mark, u32 mask, const void *data, int data_type, - const unsigned char *file_name, u32 cookie) + const unsigned char *file_name, u32 cookie, + struct fsnotify_iter_info *iter_info) { struct inotify_inode_mark *i_mark;struct inotify_event_info *event;
fs/notify/inotify/inotify_user.c +1 -1
@@ -477,7 +477,7 @@ void inotify_ignored_and_remove_idr(struct fsnotify_mark *fsn_mark,/* Queue ignore event for the watch */ inotify_handle_event (group,NULL , fsn_mark,NULL ,FS_IN_IGNORED , -NULL ,FSNOTIFY_EVENT_NONE ,NULL ,0 ); +NULL ,FSNOTIFY_EVENT_NONE ,NULL ,0 ,NULL ); i_mark = container_of(fsn_mark, struct inotify_inode_mark, fsn_mark);/* remove this mark from the idr */
include/linux/fsnotify_backend.h +2 -1
@@ -98,7 +98,8 @@ struct fsnotify_ops {struct fsnotify_mark *inode_mark, struct fsnotify_mark *vfsmount_mark, u32 mask, const void *data, int data_type, - const unsigned char *file_name, u32 cookie); + const unsigned char *file_name, u32 cookie, + struct fsnotify_iter_info *iter_info);void (*free_group_priv)(struct fsnotify_group *group);void (*freeing_mark)(struct fsnotify_mark *mark,struct fsnotify_group *group);void (*free_event)(struct fsnotify_event *event);
kernel/audit_fsnotify.c +2 -1
@@ -168,7 +168,8 @@ static int audit_mark_handle_event(struct fsnotify_group *group,struct fsnotify_mark *inode_mark, struct fsnotify_mark *vfsmount_mark, u32 mask, const void *data, int data_type, - const unsigned char *dname, u32 cookie) + const unsigned char *dname, u32 cookie, + struct fsnotify_iter_info *iter_info) { struct audit_fsnotify_mark *audit_mark;const struct inode *inode =NULL ;
kernel/audit_tree.c +2 -1
@@ -979,7 +979,8 @@ static int audit_tree_handle_event(struct fsnotify_group *group,struct fsnotify_mark *inode_mark, struct fsnotify_mark *vfsmount_mark, u32 mask, const void *data, int data_type, - const unsigned char *file_name, u32 cookie) + const unsigned char *file_name, u32 cookie, + struct fsnotify_iter_info *iter_info) { return 0; }
kernel/audit_watch.c +2 -1
@@ -472,7 +472,8 @@ static int audit_watch_handle_event(struct fsnotify_group *group,struct fsnotify_mark *inode_mark, struct fsnotify_mark *vfsmount_mark, u32 mask, const void *data, int data_type, - const unsigned char *dname, u32 cookie) + const unsigned char *dname, u32 cookie, + struct fsnotify_iter_info *iter_info) { const struct inode *inode;struct audit_parent *parent;
Signature · 1 line
2.10.2
Jan Kara
maintainerPATCH v4 16/25 fanotify: Release SRCU lock when waiting for userspace response
In reply to Jan KaraWhen userspace task processing fanotify permission events screws up and does not respond, fsnotify_mark_srcu SRCU is held indefinitely which causes further hangs in the whole notification subsystem. Although we cannot easily solve the problem of operations blocked waiting for response from userspace, we can at least somewhat localize the damage by dropping SRCU lock before waiting for userspace response and reacquiring it when userspace responds.
--- fs/notify/fanotify/fanotify.c | 17 +++++++++++++++-- 1 file changed, 15 insertions(+), 2 deletions(-)
fs/notify/fanotify/fanotify.c +15 -2
@@ -61,14 +61,26 @@ static int fanotify_merge(struct list_head *list, struct fsnotify_event *event)#ifdef CONFIG_FANOTIFY_ACCESS_PERMISSIONS static int fanotify_get_response (struct fsnotify_group *group, -struct fanotify_perm_event_info *event) +struct fanotify_perm_event_info *event, +struct fsnotify_iter_info *iter_info) {int ret;pr_debug ("%s: group=%p event=%p\n" , __func__, group, event); +/* +* fsnotify_prepare_user_wait() fails if we race with mark deletion. +* Just let the operation pass in that case. +*/ +if (!fsnotify_prepare_user_wait (iter_info)) { + event->response =FAN_ALLOW ; + goto out; + } +wait_event (group->fanotify_data.access_waitq, event->response); +fsnotify_finish_user_wait (iter_info); +out:/* userspace responded, convert to something usable */ switch (event->response) {case FAN_ALLOW :
@@ -220,7 +232,8 @@ static int fanotify_handle_event(struct fsnotify_group *group,#ifdef CONFIG_FANOTIFY_ACCESS_PERMISSIONS if (mask &FAN_ALL_PERM_EVENTS ) { - ret =fanotify_get_response (group,FANOTIFY_PE (fsn_event)); + ret =fanotify_get_response (group,FANOTIFY_PE (fsn_event), + iter_info);fsnotify_destroy_event (group, fsn_event); }#endif
Signature · 1 line
2.10.2
Jan Kara
maintainerPATCH v4 17/25 fsnotify: Remove fsnotify_set_mark_{,ignored_}mask_locked()
In reply to Jan KaraThese helpers are now only a simple assignment and just obfuscate what is going on. Remove them.
--- fs/notify/dnotify/dnotify.c | 9 +++------ fs/notify/fanotify/fanotify_user.c | 9 ++++----- fs/notify/inotify/inotify_user.c | 6 ++---- fs/notify/mark.c | 14 -------------- include/linux/fsnotify_backend.h | 4 ---- 5 files changed, 9 insertions(+), 33 deletions(-)
fs/notify/dnotify/dnotify.c +3 -6
@@ -52,7 +52,7 @@ struct dnotify_mark {*/ static void dnotify_recalc_inode_mask (struct fsnotify_mark *fsn_mark) { -__u32 new_mask, old_mask; +__u32 new_mask =0 ;struct dnotify_struct *dn; struct dnotify_mark *dn_mark = container_of(fsn_mark, struct dnotify_mark,
@@ -60,14 +60,11 @@ static void dnotify_recalc_inode_mask(struct fsnotify_mark *fsn_mark)assert_spin_locked (&fsn_mark->lock); - old_mask = fsn_mark->mask; - new_mask =0 ;for (dn = dn_mark->dn; dn !=NULL ; dn = dn->dn_next) new_mask |= (dn->dn_mask & ~FS_DN_MULTISHOT ); -fsnotify_set_mark_mask_locked (fsn_mark, new_mask); - -if (old_mask == new_mask) +if (fsn_mark->mask == new_mask)return ; + fsn_mark->mask = new_mask;fsnotify_recalc_mask (fsn_mark->connector); }
fs/notify/fanotify/fanotify_user.c +4 -5
@@ -510,13 +510,12 @@ static __u32 fanotify_mark_remove_from_mask(struct fsnotify_mark *fsn_mark, tmask &= ~FAN_ONDIR ; oldmask = fsn_mark->mask; -fsnotify_set_mark_mask_locked (fsn_mark, tmask); + fsn_mark->mask = tmask; }else {__u32 tmask = fsn_mark->ignored_mask & ~mask;if (flags &FAN_MARK_ONDIR ) tmask &= ~FAN_ONDIR ; - -fsnotify_set_mark_ignored_mask_locked (fsn_mark, tmask); + fsn_mark->ignored_mask = tmask; } *destroy = !(fsn_mark->mask | fsn_mark->ignored_mask);spin_unlock (&fsn_mark->lock);
@@ -598,13 +597,13 @@ static __u32 fanotify_mark_add_to_mask(struct fsnotify_mark *fsn_mark, tmask |=FAN_ONDIR ; oldmask = fsn_mark->mask; -fsnotify_set_mark_mask_locked (fsn_mark, tmask); + fsn_mark->mask = tmask; }else {__u32 tmask = fsn_mark->ignored_mask | mask;if (flags &FAN_MARK_ONDIR ) tmask |=FAN_ONDIR ; -fsnotify_set_mark_ignored_mask_locked (fsn_mark, tmask); + fsn_mark->ignored_mask = tmask;if (flags &FAN_MARK_IGNORED_SURV_MODIFY ) fsn_mark->flags |=FSNOTIFY_MARK_FLAG_IGNORED_SURV_MODIFY ; }
fs/notify/inotify/inotify_user.c +2 -4
@@ -516,14 +516,12 @@ static int inotify_update_existing_watch(struct fsnotify_group *group, i_mark = container_of(fsn_mark, struct inotify_inode_mark, fsn_mark);spin_lock (&fsn_mark->lock); - old_mask = fsn_mark->mask;if (add) -fsnotify_set_mark_mask_locked (fsn_mark, (fsn_mark->mask | mask)); + fsn_mark->mask |= mask;else -fsnotify_set_mark_mask_locked (fsn_mark, mask); + fsn_mark->mask = mask; new_mask = fsn_mark->mask; -spin_unlock (&fsn_mark->lock);if (old_mask != new_mask) {
fs/notify/mark.c +0 -14
@@ -394,20 +394,6 @@ void fsnotify_destroy_mark(struct fsnotify_mark *mark,fsnotify_free_mark (mark); } -void fsnotify_set_mark_mask_locked (struct fsnotify_mark *mark,__u32 mask) -{ -assert_spin_locked (&mark->lock); - - mark->mask = mask; -} - -void fsnotify_set_mark_ignored_mask_locked (struct fsnotify_mark *mark,__u32 mask) -{ -assert_spin_locked (&mark->lock); - - mark->ignored_mask = mask; -} -/* * Sorting function for lists of fsnotify marks. *
include/linux/fsnotify_backend.h +0 -4
@@ -346,10 +346,6 @@ extern void fsnotify_init_mark(struct fsnotify_mark *mark, void (*free_mark)(strextern struct fsnotify_mark *fsnotify_find_inode_mark (struct fsnotify_group *group,struct inode *inode);/* find (and take a reference) to a mark associated with group and vfsmount */ extern struct fsnotify_mark *fsnotify_find_vfsmount_mark (struct fsnotify_group *group,struct vfsmount *mnt); -/* set the ignored_mask of a mark */ -extern void fsnotify_set_mark_ignored_mask_locked (struct fsnotify_mark *mark,__u32 mask); -/* set the mask of a mark (might pin the object into memory */ -extern void fsnotify_set_mark_mask_locked (struct fsnotify_mark *mark,__u32 mask);/* attach the mark to both the group and the inode */ extern int fsnotify_add_mark (struct fsnotify_mark *mark,struct fsnotify_group *group,struct inode *inode,struct vfsmount *mnt,int allow_dups);
Signature · 1 line
2.10.2
Jan Kara
maintainerPATCH v4 18/25 fsnotify: Remove fsnotify_recalc_{inode|vfsmount}_mask()
In reply to Jan KaraThese helpers are just very thin wrappers now. Remove them.
--- fs/notify/fanotify/fanotify_user.c | 8 ++++---- fs/notify/inode_mark.c | 5 ----- fs/notify/inotify/inotify_user.c | 2 +- fs/notify/vfsmount_mark.c | 5 ----- include/linux/fsnotify_backend.h | 4 ---- 5 files changed, 5 insertions(+), 19 deletions(-)
fs/notify/fanotify/fanotify_user.c +4 -4
@@ -541,7 +541,7 @@ static int fanotify_remove_vfsmount_mark(struct fsnotify_group *group, removed =fanotify_mark_remove_from_mask (fsn_mark, mask, flags, &destroy_mark);if (removed &real_mount (mnt)->mnt_fsnotify_mask) -fsnotify_recalc_vfsmount_mask (mnt); +fsnotify_recalc_mask (real_mount (mnt)->mnt_fsnotify_marks);if (destroy_mark)fsnotify_detach_mark (fsn_mark);mutex_unlock (&group->mark_mutex);
@@ -570,7 +570,7 @@ static int fanotify_remove_inode_mark(struct fsnotify_group *group, removed =fanotify_mark_remove_from_mask (fsn_mark, mask, flags, &destroy_mark);if (removed & inode->i_fsnotify_mask) -fsnotify_recalc_inode_mask (inode); +fsnotify_recalc_mask (inode->i_fsnotify_marks);if (destroy_mark)fsnotify_detach_mark (fsn_mark);mutex_unlock (&group->mark_mutex);
@@ -655,7 +655,7 @@ static int fanotify_add_vfsmount_mark(struct fsnotify_group *group, } added =fanotify_mark_add_to_mask (fsn_mark, mask, flags);if (added & ~real_mount (mnt)->mnt_fsnotify_mask) -fsnotify_recalc_vfsmount_mask (mnt); +fsnotify_recalc_mask (real_mount (mnt)->mnt_fsnotify_marks);mutex_unlock (&group->mark_mutex);fsnotify_put_mark (fsn_mark);
@@ -692,7 +692,7 @@ static int fanotify_add_inode_mark(struct fsnotify_group *group, } added =fanotify_mark_add_to_mask (fsn_mark, mask, flags);if (added & ~inode->i_fsnotify_mask) -fsnotify_recalc_inode_mask (inode); +fsnotify_recalc_mask (inode->i_fsnotify_marks);mutex_unlock (&group->mark_mutex);fsnotify_put_mark (fsn_mark);
fs/notify/inode_mark.c +0 -5
@@ -30,11 +30,6 @@#include "../internal.h" -void fsnotify_recalc_inode_mask (struct inode *inode) -{ -fsnotify_recalc_mask (inode->i_fsnotify_marks); -} -/* * Given a group clear all of the inode marks associated with that group. */
fs/notify/inotify/inotify_user.c +1 -1
@@ -532,7 +532,7 @@ static int inotify_update_existing_watch(struct fsnotify_group *group,/* update the inode with this new fsn_mark */ if (dropped || do_inode) -fsnotify_recalc_inode_mask (inode); +fsnotify_recalc_mask (inode->i_fsnotify_marks); }
fs/notify/vfsmount_mark.c +0 -5
@@ -29,11 +29,6 @@#include <linux/fsnotify_backend.h> #include "fsnotify.h" -void fsnotify_recalc_vfsmount_mask (struct vfsmount *mnt) -{ -fsnotify_recalc_mask (real_mount (mnt)->mnt_fsnotify_marks); -} -void fsnotify_clear_vfsmount_marks_by_group (struct fsnotify_group *group) {fsnotify_clear_marks_by_group_flags (group,FSNOTIFY_OBJ_TYPE_VFSMOUNT );
include/linux/fsnotify_backend.h +0 -4
@@ -337,10 +337,6 @@ extern struct fsnotify_event *fsnotify_remove_first_event(struct fsnotify_group/* Calculate mask of events for a list of marks */ extern void fsnotify_recalc_mask (struct fsnotify_mark_connector *conn); -/* run all marks associated with a vfsmount and update mnt->mnt_fsnotify_mask */ -extern void fsnotify_recalc_vfsmount_mask (struct vfsmount *mnt); -/* run all marks associated with an inode and update inode->i_fsnotify_mask */ -extern void fsnotify_recalc_inode_mask (struct inode *inode);extern void fsnotify_init_mark (struct fsnotify_mark *mark,void (*free_mark)(struct fsnotify_mark *mark));/* find (and take a reference) to a mark associated with group and inode */ extern struct fsnotify_mark *fsnotify_find_inode_mark (struct fsnotify_group *group,struct inode *inode);
Signature · 1 line
2.10.2
Jan Kara
maintainerPATCH v4 19/25 fsnotify: Inline fsnotify_clear_{inode|vfsmount}_mark_group()
In reply to Jan KaraInline these helpers as they are very thin. We still keep them as we don't want to expose details about how list type is determined.
--- fs/notify/inode_mark.c | 8 -------- fs/notify/vfsmount_mark.c | 5 ----- include/linux/fsnotify_backend.h | 14 ++++++++++---- 3 files changed, 10 insertions(+), 17 deletions(-)
fs/notify/inode_mark.c +0 -8
@@ -31,14 +31,6 @@#include "../internal.h" /* -* Given a group clear all of the inode marks associated with that group. -*/ -void fsnotify_clear_inode_marks_by_group (struct fsnotify_group *group) -{ -fsnotify_clear_marks_by_group_flags (group,FSNOTIFY_OBJ_TYPE_INODE ); -} - -/* * given a group and inode, find the mark associated with that combination. * if found take a reference to that mark and return it, else return NULL */
fs/notify/vfsmount_mark.c +0 -5
@@ -29,11 +29,6 @@#include <linux/fsnotify_backend.h> #include "fsnotify.h" -void fsnotify_clear_vfsmount_marks_by_group (struct fsnotify_group *group) -{ -fsnotify_clear_marks_by_group_flags (group,FSNOTIFY_OBJ_TYPE_VFSMOUNT ); -} -/* * given a group and vfsmount, find the mark associated with that combination. * if found take a reference to that mark and return it, else return NULL
include/linux/fsnotify_backend.h +10 -4
@@ -354,12 +354,18 @@ extern void fsnotify_destroy_mark(struct fsnotify_mark *mark,extern void fsnotify_detach_mark (struct fsnotify_mark *mark);/* free mark */ extern void fsnotify_free_mark (struct fsnotify_mark *mark); -/* run all the marks in a group, and clear all of the vfsmount marks */ -extern void fsnotify_clear_vfsmount_marks_by_group (struct fsnotify_group *group); -/* run all the marks in a group, and clear all of the inode marks */ -extern void fsnotify_clear_inode_marks_by_group (struct fsnotify_group *group);/* run all the marks in a group, and clear all of the marks attached to given object type */ extern void fsnotify_clear_marks_by_group_flags (struct fsnotify_group *group,unsigned int flags); +/* run all the marks in a group, and clear all of the vfsmount marks */ +static inline void fsnotify_clear_vfsmount_marks_by_group (struct fsnotify_group *group) +{ +fsnotify_clear_marks_by_group_flags (group,FSNOTIFY_OBJ_TYPE_VFSMOUNT ); +} +/* run all the marks in a group, and clear all of the inode marks */ +static inline void fsnotify_clear_inode_marks_by_group (struct fsnotify_group *group) +{ +fsnotify_clear_marks_by_group_flags (group,FSNOTIFY_OBJ_TYPE_INODE ); +}extern void fsnotify_get_mark (struct fsnotify_mark *mark);extern void fsnotify_put_mark (struct fsnotify_mark *mark);extern void fsnotify_unmount_inodes (struct super_block *sb);
Signature · 1 line
2.10.2
Jan Kara
maintainerPATCH v4 20/25 fsnotify: Rename fsnotify_clear_marks_by_group_flags()
In reply to Jan KaraThe _flags() suffix in the function name was more confusing than explaining so just remove it. Also rename the argument from 'flags' to 'type' to better explain what the function expects.
--- fs/notify/mark.c | 12 +++++------- include/linux/fsnotify_backend.h | 6 +++--- 2 files changed, 8 insertions(+), 10 deletions(-)
fs/notify/mark.c +5 -7
@@ -67,7 +67,7 @@* - The fs the inode is on is unmounted. (fsnotify_inode_delete/fsnotify_unmount_inodes) * - Something explicitly requests that it be removed. (fsnotify_destroy_mark) * - The fsnotify_group associated with the mark is going away and all such marks -* need to be cleaned up. (fsnotify_clear_marks_by_group) +* need to be cleaned up. (fsnotify_detach_group_marks) * * This has the very interesting property of being able to run concurrently with * any (or all) other directions.
@@ -626,11 +626,9 @@ struct fsnotify_mark *fsnotify_find_mark(struct fsnotify_mark_connector **connp,return NULL ; } -/* -* clear any marks in a group in which mark->flags & flags is true -*/ -void fsnotify_clear_marks_by_group_flags (struct fsnotify_group *group, -unsigned int flags) +/* Clear any marks in a group with given type */ +void fsnotify_clear_marks_by_group (struct fsnotify_group *group, +unsigned int type) {struct fsnotify_mark *lmark, *mark;LIST_HEAD (to_free);
@@ -646,7 +644,7 @@ void fsnotify_clear_marks_by_group_flags(struct fsnotify_group *group,*/ mutex_lock_nested (&group->mark_mutex,SINGLE_DEPTH_NESTING );list_for_each_entry_safe (mark, lmark, &group->marks_list, g_list) { -if (mark->connector->flags & flags) +if (mark->connector->flags & type)list_move (&mark->g_list, &to_free); }mutex_unlock (&group->mark_mutex);
include/linux/fsnotify_backend.h +3 -3
@@ -355,16 +355,16 @@ extern void fsnotify_detach_mark(struct fsnotify_mark *mark);/* free mark */ extern void fsnotify_free_mark (struct fsnotify_mark *mark);/* run all the marks in a group, and clear all of the marks attached to given object type */ -extern void fsnotify_clear_marks_by_group_flags (struct fsnotify_group *group,unsigned int flags); +extern void fsnotify_clear_marks_by_group (struct fsnotify_group *group,unsigned int type);/* run all the marks in a group, and clear all of the vfsmount marks */ static inline void fsnotify_clear_vfsmount_marks_by_group (struct fsnotify_group *group) { -fsnotify_clear_marks_by_group_flags (group,FSNOTIFY_OBJ_TYPE_VFSMOUNT ); +fsnotify_clear_marks_by_group (group,FSNOTIFY_OBJ_TYPE_VFSMOUNT ); }/* run all the marks in a group, and clear all of the inode marks */ static inline void fsnotify_clear_inode_marks_by_group (struct fsnotify_group *group) { -fsnotify_clear_marks_by_group_flags (group,FSNOTIFY_OBJ_TYPE_INODE ); +fsnotify_clear_marks_by_group (group,FSNOTIFY_OBJ_TYPE_INODE ); }extern void fsnotify_get_mark (struct fsnotify_mark *mark);extern void fsnotify_put_mark (struct fsnotify_mark *mark);
Signature · 1 line
2.10.2
Jan Kara
maintainerPATCH v4 21/25 fsnotify: Remove fsnotify_detach_group_marks()
In reply to Jan KaraThe function is already mostly contained in what fsnotify_clear_marks_by_group() does. Just update that function to not select marks when all of them should be destroyed and remove fsnotify_detach_group_marks().
--- fs/notify/fsnotify.h | 2 -- fs/notify/group.c | 9 +++++++- fs/notify/mark.c | 45 +++++++++------------------------------- include/linux/fsnotify_backend.h | 2 ++ 4 files changed, 20 insertions(+), 38 deletions(-)
fs/notify/fsnotify.h +0 -2
@@ -40,8 +40,6 @@ static inline void fsnotify_clear_marks_by_mount(struct vfsmount *mnt) {fsnotify_destroy_marks (&real_mount (mnt)->mnt_fsnotify_marks); } -/* prepare for freeing all marks associated with given group */ -extern void fsnotify_detach_group_marks (struct fsnotify_group *group);/* Wait until all marks queued for destruction are destroyed */ extern void fsnotify_wait_marks_destroyed (void );
fs/notify/group.c +8 -1
@@ -67,7 +67,14 @@ void fsnotify_destroy_group(struct fsnotify_group *group)fsnotify_group_stop_queueing (group);/* Clear all marks for this group and queue them for destruction */ -fsnotify_detach_group_marks (group); +fsnotify_clear_marks_by_group (group,FSNOTIFY_OBJ_ALL_TYPES ); + +/* +* Some marks can still be pinned when waiting for response from +* userspace. Wait for those now. fsnotify_prepare_user_wait() will +* not succeed now so this wait is race-free. +*/ +wait_event (group->notification_waitq, !atomic_read (&group->user_waits));/* * Wait until all marks get really destroyed. We could actually destroy
fs/notify/mark.c +10 -35
75 lines not on this page · raw message
include/linux/fsnotify_backend.h +2 -0
@@ -207,6 +207,8 @@ struct fsnotify_mark_connector {spinlock_t lock;#define FSNOTIFY_OBJ_TYPE_INODE 0x01#define FSNOTIFY_OBJ_TYPE_VFSMOUNT 0x02 +#define FSNOTIFY_OBJ_ALL_TYPES (FSNOTIFY_OBJ_TYPE_INODE | \ + FSNOTIFY_OBJ_TYPE_VFSMOUNT)unsigned int flags;/* Type of object [lock] */ union {/* Object pointer [lock] */ struct inode *inode;
Signature · 1 line
2.10.2
Jan Kara
maintainerPATCH v4 22/25 fsnotify: Remove fsnotify_find_{inode|vfsmount}_mark()
In reply to Jan KaraThese are very thin wrappers, just remove them. Drop fs/notify/vfsmount_mark.c as it is empty now.
--- fs/notify/Makefile | 2 +- fs/notify/dnotify/dnotify.c | 4 ++-- fs/notify/fanotify/fanotify_user.c | 12 ++++++----- fs/notify/fsnotify.h | 4 ---- fs/notify/inode_mark.c | 10 --------- fs/notify/inotify/inotify_user.c | 2 +- fs/notify/vfsmount_mark.c | 42 -------------------------------------- include/linux/fsnotify_backend.h | 8 ++++---- kernel/audit_tree.c | 3 ++- kernel/audit_watch.c | 2 +- 10 files changed, 18 insertions(+), 71 deletions(-) delete mode 100644 fs/notify/vfsmount_mark.c
fs/notify/Makefile +1 -1
@@ -1,5 +1,5 @@ obj-$(CONFIG_FSNOTIFY) += fsnotify.o notification.o group.o inode_mark.o \ - mark.o vfsmount_mark.o fdinfo.o + mark.o fdinfo.o obj-y += dnotify/ obj-y += inotify/
fs/notify/dnotify/dnotify.c +2 -2
@@ -157,7 +157,7 @@ void dnotify_flush(struct file *filp, fl_owner_t id)if (!S_ISDIR (inode->i_mode))return ; - fsn_mark =fsnotify_find_inode_mark (dnotify_group, inode); + fsn_mark =fsnotify_find_mark (&inode->i_fsnotify_marks, dnotify_group);if (!fsn_mark)return ; dn_mark = container_of(fsn_mark, struct dnotify_mark, fsn_mark);
@@ -313,7 +313,7 @@ int fcntl_dirnotify(int fd, struct file *filp, unsigned long arg)mutex_lock (&dnotify_group->mark_mutex);/* add the new_fsn_mark or find an old one. */ - fsn_mark =fsnotify_find_inode_mark (dnotify_group, inode); + fsn_mark =fsnotify_find_mark (&inode->i_fsnotify_marks, dnotify_group);if (fsn_mark) { dn_mark = container_of(fsn_mark, struct dnotify_mark, fsn_mark);spin_lock (&fsn_mark->lock);
fs/notify/fanotify/fanotify_user.c +7 -5
@@ -532,7 +532,8 @@ static int fanotify_remove_vfsmount_mark(struct fsnotify_group *group,int destroy_mark;mutex_lock (&group->mark_mutex); - fsn_mark =fsnotify_find_vfsmount_mark (group, mnt); + fsn_mark =fsnotify_find_mark (&real_mount (mnt)->mnt_fsnotify_marks, + group);if (!fsn_mark) {mutex_unlock (&group->mark_mutex);return -ENOENT ;
@@ -561,7 +562,7 @@ static int fanotify_remove_inode_mark(struct fsnotify_group *group,int destroy_mark;mutex_lock (&group->mark_mutex); - fsn_mark =fsnotify_find_inode_mark (group, inode); + fsn_mark =fsnotify_find_mark (&inode->i_fsnotify_marks, group);if (!fsn_mark) {mutex_unlock (&group->mark_mutex);return -ENOENT ;
@@ -577,7 +578,7 @@ static int fanotify_remove_inode_mark(struct fsnotify_group *group,if (destroy_mark)fsnotify_free_mark (fsn_mark); -/* matches the fsnotify_find_inode_mark() */ +/* matches the fsnotify_find_mark() */ fsnotify_put_mark (fsn_mark);return 0 ;
@@ -645,7 +646,8 @@ static int fanotify_add_vfsmount_mark(struct fsnotify_group *group,__u32 added;mutex_lock (&group->mark_mutex); - fsn_mark =fsnotify_find_vfsmount_mark (group, mnt); + fsn_mark =fsnotify_find_mark (&real_mount (mnt)->mnt_fsnotify_marks, + group);if (!fsn_mark) { fsn_mark =fanotify_add_new_mark (group,NULL , mnt);if (IS_ERR (fsn_mark)) {
@@ -682,7 +684,7 @@ static int fanotify_add_inode_mark(struct fsnotify_group *group,return 0 ;mutex_lock (&group->mark_mutex); - fsn_mark =fsnotify_find_inode_mark (group, inode); + fsn_mark =fsnotify_find_mark (&inode->i_fsnotify_marks, group);if (!fsn_mark) { fsn_mark =fanotify_add_new_mark (group, inode,NULL );if (IS_ERR (fsn_mark)) {
fs/notify/fsnotify.h +0 -4
@@ -24,10 +24,6 @@ extern struct srcu_struct fsnotify_mark_srcu;extern int fsnotify_compare_groups (struct fsnotify_group *a,struct fsnotify_group *b); -/* Find mark belonging to given group attached to given connector */ -extern struct fsnotify_mark *fsnotify_find_mark ( -struct fsnotify_mark_connector **connp, -struct fsnotify_group *group);/* Destroy all marks connected via given connector */ extern void fsnotify_destroy_marks (struct fsnotify_mark_connector **connp);/* run the list of all marks associated with inode and destroy them */
fs/notify/inode_mark.c +0 -10
@@ -30,16 +30,6 @@#include "../internal.h" -/* -* given a group and inode, find the mark associated with that combination. -* if found take a reference to that mark and return it, else return NULL -*/ -struct fsnotify_mark *fsnotify_find_inode_mark (struct fsnotify_group *group, -struct inode *inode) -{ -return fsnotify_find_mark (&inode->i_fsnotify_marks, group); -} -/** * fsnotify_unmount_inodes - an sb is unmounting. handle any watched inodes. * @sb: superblock being unmounted.
fs/notify/inotify/inotify_user.c +1 -1
@@ -509,7 +509,7 @@ static int inotify_update_existing_watch(struct fsnotify_group *group, mask =inotify_arg_to_mask (arg); - fsn_mark =fsnotify_find_inode_mark (group, inode); + fsn_mark =fsnotify_find_mark (&inode->i_fsnotify_marks, group);if (!fsn_mark)return -ENOENT ;
fs/notify/vfsmount_mark.c deleted +0 -42
42 lines not on this page · raw message
include/linux/fsnotify_backend.h +4 -4
@@ -340,10 +340,10 @@ extern struct fsnotify_event *fsnotify_remove_first_event(struct fsnotify_group/* Calculate mask of events for a list of marks */ extern void fsnotify_recalc_mask (struct fsnotify_mark_connector *conn);extern void fsnotify_init_mark (struct fsnotify_mark *mark,void (*free_mark)(struct fsnotify_mark *mark)); -/* find (and take a reference) to a mark associated with group and inode */ -extern struct fsnotify_mark *fsnotify_find_inode_mark (struct fsnotify_group *group,struct inode *inode); -/* find (and take a reference) to a mark associated with group and vfsmount */ -extern struct fsnotify_mark *fsnotify_find_vfsmount_mark (struct fsnotify_group *group,struct vfsmount *mnt); +/* Find mark belonging to given group in the list of marks */ +extern struct fsnotify_mark *fsnotify_find_mark ( +struct fsnotify_mark_connector **connp, +struct fsnotify_group *group);/* attach the mark to both the group and the inode */ extern int fsnotify_add_mark (struct fsnotify_mark *mark,struct fsnotify_group *group,struct inode *inode,struct vfsmount *mnt,int allow_dups);
kernel/audit_tree.c +2 -1
@@ -385,7 +385,8 @@ static int tag_chunk(struct inode *inode, struct audit_tree *tree)struct node *p;int n; - old_entry =fsnotify_find_inode_mark (audit_tree_group, inode); + old_entry =fsnotify_find_mark (&inode->i_fsnotify_marks, + audit_tree_group);if (!old_entry)return create_chunk (inode, tree);
kernel/audit_watch.c +1 -1
@@ -102,7 +102,7 @@ static inline struct audit_parent *audit_find_parent(struct inode *inode)struct audit_parent *parent =NULL ;struct fsnotify_mark *entry; - entry =fsnotify_find_inode_mark (audit_watch_group, inode); + entry =fsnotify_find_mark (&inode->i_fsnotify_marks, audit_watch_group);if (entry) parent = container_of(entry, struct audit_parent, mark);
Signature · 1 line
2.10.2
Jan Kara
maintainerPATCH v4 23/25 fsnotify: Drop inode_mark.c
In reply to Jan Karainode_mark.c now contains only a single function. Move it to fs/notify/fsnotify.c and remove inode_mark.c.
--- fs/notify/Makefile | 4 +-- fs/notify/fsnotify.c | 57 ++++++++++++++++++++++++++++++++ fs/notify/inode_mark.c | 88 -------------------------------------------------- 3 files changed, 59 insertions(+), 90 deletions(-) delete mode 100644 fs/notify/inode_mark.c
fs/notify/Makefile +2 -2
@@ -1,5 +1,5 @@ -obj-$(CONFIG_FSNOTIFY) += fsnotify.o notification.o group.o inode_mark.o \ - mark.o fdinfo.o +obj-$(CONFIG_FSNOTIFY) += fsnotify.o notification.o group.o mark.o \ + fdinfo.o obj-y += dnotify/ obj-y += inotify/
fs/notify/fsnotify.c +57 -0
63 lines not on this page · raw message
fs/notify/inode_mark.c deleted +0 -88
88 lines not on this page · raw message
Signature · 1 line
2.10.2
Jan Kara
maintainerPATCH v4 24/25 fsnotify: Add group pointer in fsnotify_init_mark()
In reply to Jan KaraCurrently we initialize mark->group only in fsnotify_add_mark_lock(). However we will need to access fsnotify_ops of corresponding group from fsnotify_put_mark() so we need mark->group initialized earlier. Do that in fsnotify_init_mark() which has a consequence that once fsnotify_init_mark() is called on a mark, the mark has to be destroyed by fsnotify_put_mark().
--- fs/notify/dnotify/dnotify.c | 5 ++--- fs/notify/fanotify/fanotify_user.c | 4 ++-- fs/notify/inotify/inotify_user.c | 5 ++--- fs/notify/mark.c | 17 ++++++++++------- include/linux/fsnotify_backend.h | 12 +++++++----- kernel/audit_fsnotify.c | 7 ++++--- kernel/audit_tree.c | 15 ++++++++------- kernel/audit_watch.c | 5 +++-- 8 files changed, 38 insertions(+), 32 deletions(-)
fs/notify/dnotify/dnotify.c +2 -3
@@ -305,7 +305,7 @@ int fcntl_dirnotify(int fd, struct file *filp, unsigned long arg)/* set up the new_fsn_mark and new_dn_mark */ new_fsn_mark = &new_dn_mark->fsn_mark; -fsnotify_init_mark (new_fsn_mark, dnotify_free_mark); +fsnotify_init_mark (new_fsn_mark, dnotify_group, dnotify_free_mark); new_fsn_mark->mask = mask; new_dn_mark->dn =NULL ;
@@ -318,8 +318,7 @@ int fcntl_dirnotify(int fd, struct file *filp, unsigned long arg) dn_mark = container_of(fsn_mark, struct dnotify_mark, fsn_mark);spin_lock (&fsn_mark->lock); }else { -fsnotify_add_mark_locked (new_fsn_mark, dnotify_group, inode, -NULL ,0 ); +fsnotify_add_mark_locked (new_fsn_mark, inode,NULL ,0 );spin_lock (&new_fsn_mark->lock); fsn_mark = new_fsn_mark; dn_mark = new_dn_mark;
fs/notify/fanotify/fanotify_user.c +2 -2
@@ -627,8 +627,8 @@ static struct fsnotify_mark *fanotify_add_new_mark(struct fsnotify_group *group,if (!mark)return ERR_PTR (-ENOMEM ); -fsnotify_init_mark (mark, fanotify_free_mark); - ret =fsnotify_add_mark_locked (mark, group, inode, mnt,0 ); +fsnotify_init_mark (mark, group, fanotify_free_mark); + ret =fsnotify_add_mark_locked (mark, inode, mnt,0 );if (ret) {fsnotify_put_mark (mark);return ERR_PTR (ret);
fs/notify/inotify/inotify_user.c +2 -3
@@ -561,7 +561,7 @@ static int inotify_new_watch(struct fsnotify_group *group,if (unlikely (!tmp_i_mark))return -ENOMEM ; -fsnotify_init_mark (&tmp_i_mark->fsn_mark, inotify_free_mark); +fsnotify_init_mark (&tmp_i_mark->fsn_mark, group, inotify_free_mark); tmp_i_mark->fsn_mark.mask = mask; tmp_i_mark->wd =-1 ;
@@ -574,8 +574,7 @@ static int inotify_new_watch(struct fsnotify_group *group, goto out_err;/* we are on the idr, now get on the inode */ - ret =fsnotify_add_mark_locked (&tmp_i_mark->fsn_mark, group, inode, -NULL ,0 ); + ret =fsnotify_add_mark_locked (&tmp_i_mark->fsn_mark, inode,NULL ,0 );if (ret) {/* we failed to get on the inode, get off the idr */ inotify_remove_from_idr (group, tmp_i_mark);
fs/notify/mark.c +10 -7
@@ -546,10 +546,10 @@ static int fsnotify_add_mark_list(struct fsnotify_mark *mark,* These marks may be used for the fsnotify backend to determine which * event types should be delivered to which group. */ -int fsnotify_add_mark_locked (struct fsnotify_mark *mark, -struct fsnotify_group *group,struct inode *inode, +int fsnotify_add_mark_locked (struct fsnotify_mark *mark,struct inode *inode,struct vfsmount *mnt,int allow_dups) { +struct fsnotify_group *group = mark->group;int ret =0 ;BUG_ON (inode && mnt);
@@ -565,8 +565,6 @@ int fsnotify_add_mark_locked(struct fsnotify_mark *mark,spin_lock (&mark->lock); mark->flags |=FSNOTIFY_MARK_FLAG_ALIVE |FSNOTIFY_MARK_FLAG_ATTACHED ; -fsnotify_get_group (group); - mark->group = group;list_add (&mark->g_list, &group->marks_list);atomic_inc (&group->num_marks);fsnotify_get_mark (mark);/* for g_list */
@@ -591,12 +589,14 @@ int fsnotify_add_mark_locked(struct fsnotify_mark *mark,return ret; } -int fsnotify_add_mark (struct fsnotify_mark *mark,struct fsnotify_group *group, -struct inode *inode,struct vfsmount *mnt,int allow_dups) +int fsnotify_add_mark (struct fsnotify_mark *mark,struct inode *inode, +struct vfsmount *mnt,int allow_dups) {int ret; +struct fsnotify_group *group = mark->group; +mutex_lock (&group->mark_mutex); - ret =fsnotify_add_mark_locked (mark, group, inode, mnt, allow_dups); + ret =fsnotify_add_mark_locked (mark, inode, mnt, allow_dups);mutex_unlock (&group->mark_mutex);return ret; }
@@ -714,12 +714,15 @@ void fsnotify_destroy_marks(struct fsnotify_mark_connector **connp)* Nothing fancy, just initialize lists and locks and counters. */ void fsnotify_init_mark (struct fsnotify_mark *mark, +struct fsnotify_group *group,void (*free_mark)(struct fsnotify_mark *mark)) {memset (mark,0 ,sizeof (*mark));spin_lock_init (&mark->lock);atomic_set (&mark->refcnt,1 ); mark->free_mark = free_mark; +fsnotify_get_group (group); + mark->group = group; }/*
include/linux/fsnotify_backend.h +7 -5
@@ -339,15 +339,17 @@ extern struct fsnotify_event *fsnotify_remove_first_event(struct fsnotify_group/* Calculate mask of events for a list of marks */ extern void fsnotify_recalc_mask (struct fsnotify_mark_connector *conn); -extern void fsnotify_init_mark (struct fsnotify_mark *mark,void (*free_mark)(struct fsnotify_mark *mark)); +extern void fsnotify_init_mark (struct fsnotify_mark *mark, +struct fsnotify_group *group, +void (*free_mark)(struct fsnotify_mark *mark));/* Find mark belonging to given group in the list of marks */ extern struct fsnotify_mark *fsnotify_find_mark (struct fsnotify_mark_connector **connp,struct fsnotify_group *group); -/* attach the mark to both the group and the inode */ -extern int fsnotify_add_mark (struct fsnotify_mark *mark,struct fsnotify_group *group, -struct inode *inode,struct vfsmount *mnt,int allow_dups); -extern int fsnotify_add_mark_locked (struct fsnotify_mark *mark,struct fsnotify_group *group, +/* attach the mark to the inode or vfsmount */ +extern int fsnotify_add_mark (struct fsnotify_mark *mark,struct inode *inode, +struct vfsmount *mnt,int allow_dups); +extern int fsnotify_add_mark_locked (struct fsnotify_mark *mark,struct inode *inode,struct vfsmount *mnt,int allow_dups);/* given a group and a mark, flag mark to be freed when all references are dropped */ extern void fsnotify_destroy_mark(struct fsnotify_mark *mark,
kernel/audit_fsnotify.c +4 -3
@@ -103,15 +103,16 @@ struct audit_fsnotify_mark *audit_alloc_mark(struct audit_krule *krule, char *pa goto out; } -fsnotify_init_mark (&audit_mark->mark, audit_fsnotify_free_mark); +fsnotify_init_mark (&audit_mark->mark, audit_fsnotify_group, + audit_fsnotify_free_mark); audit_mark->mark.mask =AUDIT_FS_EVENTS ; audit_mark->path = pathname;audit_update_mark (audit_mark, dentry->d_inode); audit_mark->rule = krule; - ret =fsnotify_add_mark (&audit_mark->mark, audit_fsnotify_group, inode,NULL , true); + ret =fsnotify_add_mark (&audit_mark->mark, inode,NULL , true);if (ret <0 ) { -audit_fsnotify_mark_free (audit_mark); +fsnotify_put_mark (&audit_mark->mark); audit_mark =ERR_PTR (ret); } out:
kernel/audit_tree.c +8 -7
@@ -154,7 +154,8 @@ static struct audit_chunk *alloc_chunk(int count)INIT_LIST_HEAD (&chunk->owners[i].list); chunk->owners[i].index = i; } -fsnotify_init_mark (&chunk->mark, audit_tree_destroy_watch); +fsnotify_init_mark (&chunk->mark, audit_tree_group, + audit_tree_destroy_watch); chunk->mark.mask =FS_IN_IGNORED ;return chunk; }
@@ -256,7 +257,7 @@ static void untag_chunk(struct node *p)spin_unlock (&entry->lock);mutex_unlock (&entry->group->mark_mutex);if (new) -free_chunk (new); +fsnotify_put_mark (&new->mark); goto out; }
@@ -280,8 +281,8 @@ static void untag_chunk(struct node *p)if (!new) goto Fallback; -if (fsnotify_add_mark_locked (&new->mark, entry->group, - entry->connector->inode,NULL ,1 )) { +if (fsnotify_add_mark_locked (&new->mark, entry->connector->inode, +NULL ,1 )) {fsnotify_put_mark (&new->mark); goto Fallback; }
@@ -346,7 +347,7 @@ static int create_chunk(struct inode *inode, struct audit_tree *tree)return -ENOMEM ; entry = &chunk->mark; -if (fsnotify_add_mark (entry, audit_tree_group, inode,NULL ,0 )) { +if (fsnotify_add_mark (entry, inode,NULL ,0 )) {fsnotify_put_mark (entry);return -ENOSPC ; }
@@ -418,11 +419,11 @@ static int tag_chunk(struct inode *inode, struct audit_tree *tree)spin_unlock (&old_entry->lock);mutex_unlock (&old_entry->group->mark_mutex);fsnotify_put_mark (old_entry); -free_chunk (chunk); +fsnotify_put_mark (&chunk->mark);return -ENOENT ; } -if (fsnotify_add_mark_locked (chunk_entry, old_entry->group, +if (fsnotify_add_mark_locked (chunk_entry, old_entry->connector->inode,NULL ,1 )) {spin_unlock (&old_entry->lock);mutex_unlock (&old_entry->group->mark_mutex);
kernel/audit_watch.c +3 -2
@@ -157,9 +157,10 @@ static struct audit_parent *audit_init_parent(struct path *path)INIT_LIST_HEAD (&parent->watches); -fsnotify_init_mark (&parent->mark, audit_watch_free_mark); +fsnotify_init_mark (&parent->mark, audit_watch_group, + audit_watch_free_mark); parent->mark.mask =AUDIT_FS_WATCH ; - ret =fsnotify_add_mark (&parent->mark, audit_watch_group, inode,NULL ,0 ); + ret =fsnotify_add_mark (&parent->mark, inode,NULL ,0 );if (ret <0 ) {audit_free_parent (parent);return ERR_PTR (ret);
Signature · 1 line
2.10.2
Re: [PATCH 24/25] fsnotify: Add group pointer in fsnotify_init_mark()
Quoted 10 lines · Jan Kara
Currently we initialize
mark->grouponly infsnotify_add_mark_lock(). However we will need to accessfsnotify_opsof corresponding group fromfsnotify_put_mark()so we needmark->groupinitialized earlier. Do that infsnotify_init_mark()which has a consequence that oncefsnotify_init_mark()is called on a mark, the mark has to be destroyed byfsnotify_put_mark().Signed-off-by Jan Kara <[email protected]>From a quick glance at the audit code, this look ok to me.
Quoted 260 lines
--- fs/notify/dnotify/dnotify.c | 5 ++--- fs/notify/fanotify/fanotify_user.c | 4 ++-- fs/notify/inotify/inotify_user.c | 5 ++--- fs/notify/mark.c | 17 ++++++++++------- include/linux/fsnotify_backend.h | 12 +++++++----- kernel/audit_fsnotify.c | 7 ++++--- kernel/audit_tree.c | 15 ++++++++------- kernel/audit_watch.c | 5 +++-- 8 files changed, 38 insertions(+), 32 deletions(-)fs/notify/dnotify/dnotify.c +2
-3@@ -305,7 +305,7 @@ int fcntl_dirnotify(int fd, struct file *filp, unsigned long arg)/* set up the new_fsn_mark and new_dn_mark */ new_fsn_mark = &new_dn_mark->fsn_mark; -fsnotify_init_mark (new_fsn_mark, dnotify_free_mark); +fsnotify_init_mark (new_fsn_mark, dnotify_group, dnotify_free_mark); new_fsn_mark->mask = mask; new_dn_mark->dn =NULL ;@@ -318,8 +318,7 @@ int fcntl_dirnotify(int fd, struct file *filp, unsigned long arg) dn_mark = container_of(fsn_mark, struct dnotify_mark, fsn_mark);spin_lock (&fsn_mark->lock); }else { -fsnotify_add_mark_locked (new_fsn_mark, dnotify_group, inode, -NULL ,0 ); +fsnotify_add_mark_locked (new_fsn_mark, inode,NULL ,0 );spin_lock (&new_fsn_mark->lock); fsn_mark = new_fsn_mark; dn_mark = new_dn_mark;fs/notify/fanotify/fanotify_user.c +2
-2@@ -627,8 +627,8 @@ static struct fsnotify_mark *fanotify_add_new_mark(struct fsnotify_group *group,if (!mark)return ERR_PTR (-ENOMEM ); -fsnotify_init_mark (mark, fanotify_free_mark); - ret =fsnotify_add_mark_locked (mark, group, inode, mnt,0 ); +fsnotify_init_mark (mark, group, fanotify_free_mark); + ret =fsnotify_add_mark_locked (mark, inode, mnt,0 );if (ret) {fsnotify_put_mark (mark);return ERR_PTR (ret);fs/notify/inotify/inotify_user.c +2
-3@@ -561,7 +561,7 @@ static int inotify_new_watch(struct fsnotify_group *group,if (unlikely (!tmp_i_mark))return -ENOMEM ; -fsnotify_init_mark (&tmp_i_mark->fsn_mark, inotify_free_mark); +fsnotify_init_mark (&tmp_i_mark->fsn_mark, group, inotify_free_mark); tmp_i_mark->fsn_mark.mask = mask; tmp_i_mark->wd =-1 ;@@ -574,8 +574,7 @@ static int inotify_new_watch(struct fsnotify_group *group, goto out_err;/* we are on the idr, now get on the inode */ - ret =fsnotify_add_mark_locked (&tmp_i_mark->fsn_mark, group, inode, -NULL ,0 ); + ret =fsnotify_add_mark_locked (&tmp_i_mark->fsn_mark, inode,NULL ,0 );if (ret) {/* we failed to get on the inode, get off the idr */ inotify_remove_from_idr (group, tmp_i_mark);fs/notify/mark.c +10
-7@@ -546,10 +546,10 @@ static int fsnotify_add_mark_list(struct fsnotify_mark *mark,* These marks may be used for the fsnotify backend to determine which * event types should be delivered to which group. */ -int fsnotify_add_mark_locked (struct fsnotify_mark *mark, -struct fsnotify_group *group,struct inode *inode, +int fsnotify_add_mark_locked (struct fsnotify_mark *mark,struct inode *inode,struct vfsmount *mnt,int allow_dups) { +struct fsnotify_group *group = mark->group;int ret =0 ;BUG_ON (inode && mnt);@@ -565,8 +565,6 @@ int fsnotify_add_mark_locked(struct fsnotify_mark *mark,spin_lock (&mark->lock); mark->flags |=FSNOTIFY_MARK_FLAG_ALIVE |FSNOTIFY_MARK_FLAG_ATTACHED ; -fsnotify_get_group (group); - mark->group = group;list_add (&mark->g_list, &group->marks_list);atomic_inc (&group->num_marks);fsnotify_get_mark (mark);/* for g_list */ @@ -591,12 +589,14 @@ int fsnotify_add_mark_locked(struct fsnotify_mark *mark,return ret; } -int fsnotify_add_mark (struct fsnotify_mark *mark,struct fsnotify_group *group, -struct inode *inode,struct vfsmount *mnt,int allow_dups) +int fsnotify_add_mark (struct fsnotify_mark *mark,struct inode *inode, +struct vfsmount *mnt,int allow_dups) {int ret; +struct fsnotify_group *group = mark->group; +mutex_lock (&group->mark_mutex); - ret =fsnotify_add_mark_locked (mark, group, inode, mnt, allow_dups); + ret =fsnotify_add_mark_locked (mark, inode, mnt, allow_dups);mutex_unlock (&group->mark_mutex);return ret; }@@ -714,12 +714,15 @@ void fsnotify_destroy_marks(struct fsnotify_mark_connector **connp)* Nothing fancy, just initialize lists and locks and counters. */ void fsnotify_init_mark (struct fsnotify_mark *mark, +struct fsnotify_group *group,void (*free_mark)(struct fsnotify_mark *mark)) {memset (mark,0 ,sizeof (*mark));spin_lock_init (&mark->lock);atomic_set (&mark->refcnt,1 ); mark->free_mark = free_mark; +fsnotify_get_group (group); + mark->group = group; }/* include/linux/fsnotify_backend.h +7
-5@@ -339,15 +339,17 @@ extern struct fsnotify_event *fsnotify_remove_first_event(struct fsnotify_group/* Calculate mask of events for a list of marks */ extern void fsnotify_recalc_mask (struct fsnotify_mark_connector *conn); -extern void fsnotify_init_mark (struct fsnotify_mark *mark,void (*free_mark)(struct fsnotify_mark *mark)); +extern void fsnotify_init_mark (struct fsnotify_mark *mark, +struct fsnotify_group *group, +void (*free_mark)(struct fsnotify_mark *mark));/* Find mark belonging to given group in the list of marks */ extern struct fsnotify_mark *fsnotify_find_mark (struct fsnotify_mark_connector **connp,struct fsnotify_group *group); -/* attach the mark to both the group and the inode */ -extern int fsnotify_add_mark (struct fsnotify_mark *mark,struct fsnotify_group *group, -struct inode *inode,struct vfsmount *mnt,int allow_dups); -extern int fsnotify_add_mark_locked (struct fsnotify_mark *mark,struct fsnotify_group *group, +/* attach the mark to the inode or vfsmount */ +extern int fsnotify_add_mark (struct fsnotify_mark *mark,struct inode *inode, +struct vfsmount *mnt,int allow_dups); +extern int fsnotify_add_mark_locked (struct fsnotify_mark *mark,struct inode *inode,struct vfsmount *mnt,int allow_dups);/* given a group and a mark, flag mark to be freed when all references are dropped */ extern void fsnotify_destroy_mark(struct fsnotify_mark *mark,kernel/audit_fsnotify.c +4
-3@@ -103,15 +103,16 @@ struct audit_fsnotify_mark *audit_alloc_mark(struct audit_krule *krule, char *pa goto out; } -fsnotify_init_mark (&audit_mark->mark, audit_fsnotify_free_mark); +fsnotify_init_mark (&audit_mark->mark, audit_fsnotify_group, + audit_fsnotify_free_mark); audit_mark->mark.mask =AUDIT_FS_EVENTS ; audit_mark->path = pathname;audit_update_mark (audit_mark, dentry->d_inode); audit_mark->rule = krule; - ret =fsnotify_add_mark (&audit_mark->mark, audit_fsnotify_group, inode,NULL , true); + ret =fsnotify_add_mark (&audit_mark->mark, inode,NULL , true);if (ret <0 ) { -audit_fsnotify_mark_free (audit_mark); +fsnotify_put_mark (&audit_mark->mark); audit_mark =ERR_PTR (ret); } out:kernel/audit_tree.c +8
-7@@ -154,7 +154,8 @@ static struct audit_chunk *alloc_chunk(int count)INIT_LIST_HEAD (&chunk->owners[i].list); chunk->owners[i].index = i; } -fsnotify_init_mark (&chunk->mark, audit_tree_destroy_watch); +fsnotify_init_mark (&chunk->mark, audit_tree_group, + audit_tree_destroy_watch); chunk->mark.mask =FS_IN_IGNORED ;return chunk; }@@ -256,7 +257,7 @@ static void untag_chunk(struct node *p)spin_unlock (&entry->lock);mutex_unlock (&entry->group->mark_mutex);if (new) -free_chunk (new); +fsnotify_put_mark (&new->mark); goto out; }@@ -280,8 +281,8 @@ static void untag_chunk(struct node *p)if (!new) goto Fallback; -if (fsnotify_add_mark_locked (&new->mark, entry->group, - entry->connector->inode,NULL ,1 )) { +if (fsnotify_add_mark_locked (&new->mark, entry->connector->inode, +NULL ,1 )) {fsnotify_put_mark (&new->mark); goto Fallback; }@@ -346,7 +347,7 @@ static int create_chunk(struct inode *inode, struct audit_tree *tree)return -ENOMEM ; entry = &chunk->mark; -if (fsnotify_add_mark (entry, audit_tree_group, inode,NULL ,0 )) { +if (fsnotify_add_mark (entry, inode,NULL ,0 )) {fsnotify_put_mark (entry);return -ENOSPC ; }@@ -418,11 +419,11 @@ static int tag_chunk(struct inode *inode, struct audit_tree *tree)spin_unlock (&old_entry->lock);mutex_unlock (&old_entry->group->mark_mutex);fsnotify_put_mark (old_entry); -free_chunk (chunk); +fsnotify_put_mark (&chunk->mark);return -ENOENT ; } -if (fsnotify_add_mark_locked (chunk_entry, old_entry->group, +if (fsnotify_add_mark_locked (chunk_entry, old_entry->connector->inode,NULL ,1 )) {spin_unlock (&old_entry->lock);mutex_unlock (&old_entry->group->mark_mutex);kernel/audit_watch.c +3
-3@@ -157,9 +157,10 @@ static struct audit_parent *audit_init_parent(struct path *path)INIT_LIST_HEAD (&parent->watches); -fsnotify_init_mark (&parent->mark, audit_watch_free_mark); +fsnotify_init_mark (&parent->mark, audit_watch_group, + audit_watch_free_mark); parent->mark.mask =AUDIT_FS_WATCH ; - ret =fsnotify_add_mark (&parent->mark, audit_watch_group, inode,NULL ,0 ); + ret =fsnotify_add_mark (&parent->mark, inode,NULL ,0 );if (ret <0 ) {audit_free_parent (parent);return ERR_PTR (ret); --2.10.2
Jan Kara
maintainerPATCH v4 25/25 fsnotify: Move ->free_mark callback to fsnotify_ops
In reply to Jan KaraPointer to ->free_mark callback unnecessarily occupies one long in each fsnotify_mark although they are the same for all marks from one notification group. Move the callback pointer to fsnotify_ops.
--- fs/notify/dnotify/dnotify.c | 3 ++- fs/notify/fanotify/fanotify.c | 6 ++++++ fs/notify/fanotify/fanotify.h | 1 + fs/notify/fanotify/fanotify_user.c | 9 ++------- fs/notify/inotify/inotify.h | 2 ++ fs/notify/inotify/inotify_fsnotify.c | 11 +++++++++++ fs/notify/inotify/inotify_user.c | 14 ++------------ fs/notify/mark.c | 13 +++++++------ include/linux/fsnotify_backend.h | 6 +++--- kernel/audit_fsnotify.c | 4 ++-- kernel/audit_tree.c | 4 ++-- kernel/audit_watch.c | 4 ++-- 12 files changed, 42 insertions(+), 35 deletions(-)
fs/notify/dnotify/dnotify.c +2 -1
@@ -135,6 +135,7 @@ static void dnotify_free_mark(struct fsnotify_mark *fsn_mark)static struct fsnotify_ops dnotify_fsnotify_ops = { .handle_event = dnotify_handle_event, + .free_mark = dnotify_free_mark, };/*
@@ -305,7 +306,7 @@ int fcntl_dirnotify(int fd, struct file *filp, unsigned long arg)/* set up the new_fsn_mark and new_dn_mark */ new_fsn_mark = &new_dn_mark->fsn_mark; -fsnotify_init_mark (new_fsn_mark, dnotify_group, dnotify_free_mark); +fsnotify_init_mark (new_fsn_mark, dnotify_group); new_fsn_mark->mask = mask; new_dn_mark->dn =NULL ;
fs/notify/fanotify/fanotify.c +6 -0
@@ -266,8 +266,14 @@ static void fanotify_free_event(struct fsnotify_event *fsn_event)kmem_cache_free (fanotify_event_cachep, event); } +static void fanotify_free_mark (struct fsnotify_mark *fsn_mark) +{ +kmem_cache_free (fanotify_mark_cache, fsn_mark); +} +const struct fsnotify_ops fanotify_fsnotify_ops = { .handle_event = fanotify_handle_event, .free_group_priv = fanotify_free_group_priv, .free_event = fanotify_free_event, + .free_mark = fanotify_free_mark, };
fs/notify/fanotify/fanotify.h +1 -0
@@ -2,6 +2,7 @@#include <linux/path.h> #include <linux/slab.h> +extern struct kmem_cache *fanotify_mark_cache;extern struct kmem_cache *fanotify_event_cachep;extern struct kmem_cache *fanotify_perm_event_cachep;
fs/notify/fanotify/fanotify_user.c +2 -7
@@ -40,7 +40,7 @@extern const struct fsnotify_ops fanotify_fsnotify_ops; -static struct kmem_cache *fanotify_mark_cache__read_mostly ; +struct kmem_cache *fanotify_mark_cache__read_mostly ;struct kmem_cache *fanotify_event_cachep__read_mostly ;struct kmem_cache *fanotify_perm_event_cachep__read_mostly ;
@@ -444,11 +444,6 @@ static const struct file_operations fanotify_fops = { .llseek = noop_llseek, }; -static void fanotify_free_mark (struct fsnotify_mark *fsn_mark) -{ -kmem_cache_free (fanotify_mark_cache, fsn_mark); -} -static int fanotify_find_path (int dfd,const char __user *filename,struct path *path,unsigned int flags) {
@@ -627,7 +622,7 @@ static struct fsnotify_mark *fanotify_add_new_mark(struct fsnotify_group *group,if (!mark)return ERR_PTR (-ENOMEM ); -fsnotify_init_mark (mark, group, fanotify_free_mark); +fsnotify_init_mark (mark, group); ret =fsnotify_add_mark_locked (mark, inode, mnt,0 );if (ret) {fsnotify_put_mark (mark);
fs/notify/inotify/inotify.h +2 -0
@@ -31,3 +31,5 @@ extern int inotify_handle_event(struct fsnotify_group *group, struct fsnotify_iter_info *iter_info);extern const struct fsnotify_ops inotify_fsnotify_ops; + +extern struct kmem_cache *inotify_inode_mark_cachep;
fs/notify/inotify/inotify_fsnotify.c +11 -0
@@ -177,9 +177,20 @@ static void inotify_free_event(struct fsnotify_event *fsn_event)kfree (INOTIFY_E (fsn_event)); } +/* ding dong the mark is dead */ +static void inotify_free_mark (struct fsnotify_mark *fsn_mark) +{ +struct inotify_inode_mark *i_mark; + + i_mark = container_of(fsn_mark, struct inotify_inode_mark, fsn_mark); + +kmem_cache_free (inotify_inode_mark_cachep, i_mark); +} +const struct fsnotify_ops inotify_fsnotify_ops = { .handle_event = inotify_handle_event, .free_group_priv = inotify_free_group_priv, .free_event = inotify_free_event, .freeing_mark = inotify_freeing_mark, + .free_mark = inotify_free_mark, };
fs/notify/inotify/inotify_user.c +2 -12
@@ -49,7 +49,7 @@ static int inotify_max_user_instances __read_mostly;static int inotify_max_queued_events__read_mostly ;static int inotify_max_user_watches__read_mostly ; -static struct kmem_cache *inotify_inode_mark_cachep__read_mostly ; +struct kmem_cache *inotify_inode_mark_cachep__read_mostly ;#ifdef CONFIG_SYSCTL
@@ -486,16 +486,6 @@ void inotify_ignored_and_remove_idr(struct fsnotify_mark *fsn_mark,atomic_dec (&group->inotify_data.user->inotify_watches); } -/* ding dong the mark is dead */ -static void inotify_free_mark (struct fsnotify_mark *fsn_mark) -{ -struct inotify_inode_mark *i_mark; - - i_mark = container_of(fsn_mark, struct inotify_inode_mark, fsn_mark); - -kmem_cache_free (inotify_inode_mark_cachep, i_mark); -} -static int inotify_update_existing_watch (struct fsnotify_group *group,struct inode *inode,u32 arg)
@@ -561,7 +551,7 @@ static int inotify_new_watch(struct fsnotify_group *group,if (unlikely (!tmp_i_mark))return -ENOMEM ; -fsnotify_init_mark (&tmp_i_mark->fsn_mark, group, inotify_free_mark); +fsnotify_init_mark (&tmp_i_mark->fsn_mark, group); tmp_i_mark->fsn_mark.mask = mask; tmp_i_mark->wd =-1 ;
fs/notify/mark.c +7 -6
@@ -197,9 +197,12 @@ static struct inode *fsnotify_detach_connector_from_object(static void fsnotify_final_mark_destroy (struct fsnotify_mark *mark) { -if (mark->group) -fsnotify_put_group (mark->group); - mark->free_mark (mark); +struct fsnotify_group *group = mark->group; + +if (WARN_ON_ONCE (!group)) +return ; + group->ops->free_mark (mark); +fsnotify_put_group (group); }void fsnotify_put_mark (struct fsnotify_mark *mark)
@@ -714,13 +717,11 @@ void fsnotify_destroy_marks(struct fsnotify_mark_connector **connp)* Nothing fancy, just initialize lists and locks and counters. */ void fsnotify_init_mark (struct fsnotify_mark *mark, -struct fsnotify_group *group, -void (*free_mark)(struct fsnotify_mark *mark)) +struct fsnotify_group *group) {memset (mark,0 ,sizeof (*mark));spin_lock_init (&mark->lock);atomic_set (&mark->refcnt,1 ); - mark->free_mark = free_mark;fsnotify_get_group (group); mark->group = group; }
include/linux/fsnotify_backend.h +3 -3
@@ -103,6 +103,8 @@ struct fsnotify_ops {void (*free_group_priv)(struct fsnotify_group *group);void (*freeing_mark)(struct fsnotify_mark *mark,struct fsnotify_group *group);void (*free_event)(struct fsnotify_event *event); +/* called on final put+free to free memory */ +void (*free_mark)(struct fsnotify_mark *mark); };/*
@@ -260,7 +262,6 @@ struct fsnotify_mark {#define FSNOTIFY_MARK_FLAG_ALIVE 0x02#define FSNOTIFY_MARK_FLAG_ATTACHED 0x04unsigned int flags;/* flags [mark->lock] */ -void (*free_mark)(struct fsnotify_mark *mark);/* called on final put+free */ };#ifdef CONFIG_FSNOTIFY
@@ -340,8 +341,7 @@ extern struct fsnotify_event *fsnotify_remove_first_event(struct fsnotify_group/* Calculate mask of events for a list of marks */ extern void fsnotify_recalc_mask (struct fsnotify_mark_connector *conn);extern void fsnotify_init_mark (struct fsnotify_mark *mark, -struct fsnotify_group *group, -void (*free_mark)(struct fsnotify_mark *mark)); +struct fsnotify_group *group);/* Find mark belonging to given group in the list of marks */ extern struct fsnotify_mark *fsnotify_find_mark( struct fsnotify_mark_connector **connp,
kernel/audit_fsnotify.c +2 -2
@@ -103,8 +103,7 @@ struct audit_fsnotify_mark *audit_alloc_mark(struct audit_krule *krule, char *pa goto out; } -fsnotify_init_mark (&audit_mark->mark, audit_fsnotify_group, - audit_fsnotify_free_mark); +fsnotify_init_mark (&audit_mark->mark, audit_fsnotify_group); audit_mark->mark.mask =AUDIT_FS_EVENTS ; audit_mark->path = pathname;audit_update_mark (audit_mark, dentry->d_inode);
@@ -203,6 +202,7 @@ static int audit_mark_handle_event(struct fsnotify_group *group,static const struct fsnotify_ops audit_mark_fsnotify_ops = { .handle_event = audit_mark_handle_event, + .free_mark = audit_fsnotify_free_mark, };static int __init audit_fsnotify_init (void )
kernel/audit_tree.c +2 -2
@@ -154,8 +154,7 @@ static struct audit_chunk *alloc_chunk(int count)INIT_LIST_HEAD (&chunk->owners[i].list); chunk->owners[i].index = i; } -fsnotify_init_mark (&chunk->mark, audit_tree_group, - audit_tree_destroy_watch); +fsnotify_init_mark (&chunk->mark, audit_tree_group); chunk->mark.mask =FS_IN_IGNORED ;return chunk; }
@@ -1003,6 +1002,7 @@ static void audit_tree_freeing_mark(struct fsnotify_mark *entry, struct fsnotifystatic const struct fsnotify_ops audit_tree_ops = { .handle_event = audit_tree_handle_event, .freeing_mark = audit_tree_freeing_mark, + .free_mark = audit_tree_destroy_watch, };static int __init audit_tree_init (void )
kernel/audit_watch.c +2 -2
@@ -157,8 +157,7 @@ static struct audit_parent *audit_init_parent(struct path *path)INIT_LIST_HEAD (&parent->watches); -fsnotify_init_mark (&parent->mark, audit_watch_group, - audit_watch_free_mark); +fsnotify_init_mark (&parent->mark, audit_watch_group); parent->mark.mask =AUDIT_FS_WATCH ; ret =fsnotify_add_mark (&parent->mark, inode,NULL ,0 );if (ret <0 ) {
@@ -508,6 +507,7 @@ static int audit_watch_handle_event(struct fsnotify_group *group,static const struct fsnotify_ops audit_watch_fsnotify_ops = { .handle_event = audit_watch_handle_event, + .free_mark = audit_watch_free_mark, };static int __init audit_watch_init (void )
Signature · 1 line
2.10.2
Quoted 9 lines · Jan Kara
Hello,
This is the fourth revision of my patches to avoid SRCU stalls when fanotify waits for response to permission events from userspace processes. Thanks to Amir, Paul, and Miklos for review! It also passes a new LTP test that tries to provoke hangs in fanotify subsystem when there are unanswered fanotify permission events. If nobody has more objections, I'll push the changes to my tree.
Quoted 5 lines
Amir, I have dropped your Reviewed-by from the 4 patches that were originally one patch. I have made sure though, that the resulting diff is identical. So just let me know whether you are OK with me adding your Reviewed-by back.
Miklos, is the split like this better for you?
Will try to review the whole series tomorrow.
Thanks,
Miklos