fsnotify: optimize the case of no parent watcher
Subsystems (2)
Amir Goldstein
reviewerPATCH v3 1/1 fsnotify: optimize the case of no parent watcher
If parent inode is not watching, check for the event in masks of sb/mount/inode masks early to optimize out most of the code in __fsnotify_parent() and avoid calling fsnotify().
Jens has reported that this optimization improves BW and IOPS in an io_uring benchmark by more than 10% and reduces perf reported CPU usage.
before:
+ 4.51% io_uring [kernel.vmlinux] [k] fsnotify + 3.67% io_uring [kernel.vmlinux] [k] __fsnotify_parent
after:
+ 2.37% io_uring [kernel.vmlinux] [k] __fsnotify_parent
---
Jan,
Considering that this change looks like a clear win and it actually the change that you suggested, I cleaned it up a bit and posting for your consideration.
I've kept the wrappers fsnotify_path() and fsnotify_sb_has_watchers() although they are not directly related to the optimization, because they makes the code a bit nicer IMO.
Jens,
I took the liberty to add Reported-and-tested-by to attribute for your help even though the patch that you tested was slightly different than this cleaned up version, because logically it should be identical.
Thanks,
Amir.
Changes since v2:
- Add helper fsnotify_object_watched()
- Move the optimization from inlined fsnotify_path() to extern __fsnotify_parent()
- Optimize for any event - not only content events
- Drop the SB_I_CONTENT_WATCHED flag
fs/notify/fsnotify.c | 28 +++++++++++++++++----------- include/linux/fsnotify.h | 22 +++++++++++++++------- 2 files changed, 32 insertions(+), 18 deletions(-)
fs/notify/fsnotify.c +17 -11
@@ -141,7 +141,7 @@ void __fsnotify_update_child_dentry_flags(struct inode *inode) }/* Are inode/sb/mount interested in parent and name info with this event? */ -static bool fsnotify_event_needs_parent (struct inode *inode,struct mount *mnt, +static bool fsnotify_event_needs_parent (struct inode *inode,__u32 mnt_mask,__u32 mask) {__u32 marks_mask =0 ;
@@ -160,13 +160,22 @@ static bool fsnotify_event_needs_parent(struct inode *inode, struct mount *mnt,/* Did either inode/sb/mount subscribe for events with parent/name? */ marks_mask |=fsnotify_parent_needed_mask (inode->i_fsnotify_mask); marks_mask |=fsnotify_parent_needed_mask (inode->i_sb->s_fsnotify_mask); -if (mnt) - marks_mask |=fsnotify_parent_needed_mask (mnt->mnt_fsnotify_mask); + marks_mask |=fsnotify_parent_needed_mask (mnt_mask);/* Did they subscribe for this event with parent/name info? */ return mask & marks_mask; } +/* Are there any inode/mount/sb objects that are interested in this event? */ +static inline bool fsnotify_object_watched (struct inode *inode,__u32 mnt_mask, +__u32 mask) +{ +__u32 marks_mask = inode->i_fsnotify_mask | mnt_mask | + inode->i_sb->s_fsnotify_mask; + +return mask & marks_mask &ALL_FSNOTIFY_EVENTS ; +} +/* * Notify this dentry's parent about a child's events with child name info * if parent is watching or if inode/sb/mount are interested in events with
@@ -179,7 +188,7 @@ int __fsnotify_parent(struct dentry *dentry, __u32 mask, const void *data, int data_type) {const struct path *path =fsnotify_data_path (data, data_type); -struct mount *mnt = path ?real_mount (path->mnt) :NULL ; +__u32 mnt_mask = path ?real_mount (path->mnt)->mnt_fsnotify_mask :0 ;struct inode *inode =d_inode (dentry);struct dentry *parent;bool parent_watched = dentry->d_flags &DCACHE_FSNOTIFY_PARENT_WATCHED ;
@@ -190,16 +199,13 @@ int __fsnotify_parent(struct dentry *dentry, __u32 mask, const void *data,struct qstr *file_name =NULL ;int ret =0 ; -/* -* Do inode/sb/mount care about parent and name info on non-dir? -* Do they care about any event at all? -*/ -if (!inode->i_fsnotify_marks && !inode->i_sb->s_fsnotify_marks && - (!mnt || !mnt->mnt_fsnotify_marks) && !parent_watched) +/* Optimize the likely case of nobody watching this path */ +if (likely (!parent_watched) && +likely (!fsnotify_object_watched (inode, mnt_mask, mask)))return 0 ; parent =NULL ; - parent_needed =fsnotify_event_needs_parent (inode, mnt, mask); + parent_needed =fsnotify_event_needs_parent (inode, mnt_mask, mask);if (!parent_watched && !parent_needed) goto notify;
include/linux/fsnotify.h +15 -7
@@ -17,6 +17,12 @@#include <linux/slab.h> #include <linux/bug.h> +/* Are there any inode/mount/sb objects that are being watched at all? */ +static inline bool fsnotify_sb_has_watchers (struct super_block *sb) +{ +return atomic_long_read (&sb->s_fsnotify_connectors); +} +/* * Notify this @dir inode about a change in a child directory entry. * The directory entry may have turned positive or negative or its inode may
@@ -30,7 +36,7 @@ static inline int fsnotify_name(__u32 mask, const void *data, int data_type, struct inode *dir, const struct qstr *name, u32 cookie) { - if (atomic_long_read(&dir->i_sb->s_fsnotify_connectors) == 0) + if (!fsnotify_sb_has_watchers(dir->i_sb)) return 0; return fsnotify(mask, data, data_type, dir, name, NULL, cookie);
@@ -44,7 +50,7 @@ static inline void fsnotify_dirent(struct inode *dir, struct dentry *dentry,static inline void fsnotify_inode (struct inode *inode,__u32 mask) { -if (atomic_long_read (&inode->i_sb->s_fsnotify_connectors) ==0 ) +if (!fsnotify_sb_has_watchers (inode->i_sb))return ;if (S_ISDIR (inode->i_mode))
@@ -59,7 +65,7 @@ static inline int fsnotify_parent(struct dentry *dentry, __u32 mask, {struct inode *inode =d_inode (dentry); -if (atomic_long_read (&inode->i_sb->s_fsnotify_connectors) ==0 ) +if (!fsnotify_sb_has_watchers (inode->i_sb))return 0 ;if (S_ISDIR (inode->i_mode)) {
@@ -89,15 +95,17 @@ static inline void fsnotify_dentry(struct dentry *dentry, __u32 mask)fsnotify_parent (dentry, mask, dentry,FSNOTIFY_EVENT_DENTRY ); } -static inline int fsnotify_file (struct file *file,__u32 mask) +static inline int fsnotify_path (const struct path *path,__u32 mask) { -const struct path *path; +return fsnotify_parent (path->dentry, mask, path,FSNOTIFY_EVENT_PATH ); +} +static inline int fsnotify_file (struct file *file,__u32 mask) +{if (file->f_mode &FMODE_NONOTIFY )return 0 ; - path = &file->f_path; -return fsnotify_parent (path->dentry, mask, path,FSNOTIFY_EVENT_PATH ); +return fsnotify_path (&file->f_path, mask); }#ifdef CONFIG_FANOTIFY_ACCESS_PERMISSIONS
Signature · 1 line
2.34.1
Jan Kara
maintainerQuoted 19 lines · Amir Goldstein
If parent inode is not watching, check for the event in masks of sb/mount/inode masks early to optimize out most of the code in
__fsnotify_parent()and avoid callingfsnotify().Jens has reported that this optimization improves BW and IOPS in an
io_uringbenchmark by more than 10% and reduces perf reported CPU usage.before:
(quoted diff) +2
-0+ 4.51% io_uring [kernel.vmlinux] [k] fsnotify + 3.67% io_uring [kernel.vmlinux] [k] __fsnotify_parentafter:
+ 2.37%
io_uring[kernel.vmlinux] [k]__fsnotify_parentReported-and-tested-by Jens Axboe <[email protected]>Signed-off-by Amir Goldstein <[email protected]>
Quoted 7 lines
---
Jan,
Considering that this change looks like a clear win and it actually the change that you suggested, I cleaned it up a bit and posting for your consideration.
Agreed, I like this. What did you generate this patch against? It does not apply on top of current Linus' tree (maybe it needs the change sitting in VFS tree - which is fine I can wait until that gets merged)?
Quoted 3 lines
I've kept the wrappers
fsnotify_path()andfsnotify_sb_has_watchers()although they are not directly related to the optimization, because they makes the code a bit nicer IMO.
Yeah, these cleanups are fine. I would have prefered them as a separate patch (some people might want the performance improvement to be backported and this makes it unnecessarily more complex) but don't resend just because of that.
Honza
Signature · 2 lines
Jan Kara <[email protected]> SUSE Labs, CR
Amir Goldstein
reviewerQuoted 25 lines · Jan Kara
Quoted 23 lines · Amir Goldstein
If parent inode is not watching, check for the event in masks of sb/mount/inode masks early to optimize out most of the code in
__fsnotify_parent()and avoid callingfsnotify().Jens has reported that this optimization improves BW and IOPS in an
io_uringbenchmark by more than 10% and reduces perf reported CPU usage.before:
(quoted diff) +2
-0+ 4.51% io_uring [kernel.vmlinux] [k] fsnotify + 3.67% io_uring [kernel.vmlinux] [k] __fsnotify_parentafter:
+ 2.37%
io_uring[kernel.vmlinux] [k]__fsnotify_parentReported-and-tested-by Jens Axboe <[email protected]>Signed-off-by Amir Goldstein <[email protected]>---
Jan,
Quoted 8 lines
Quoted 3 lines
Considering that this change looks like a clear win and it actually the change that you suggested, I cleaned it up a bit and posting for your consideration.
Agreed, I like this. What did you generate this patch against? It does not apply on top of current Linus' tree (maybe it needs the change sitting in VFS tree - which is fine I can wait until that gets merged)?
Yes, it is on top of Christian's vfs-fixes branch.
Quoted 9 lines
Quoted 3 lines
I've kept the wrappers
fsnotify_path()andfsnotify_sb_has_watchers()although they are not directly related to the optimization, because they makes the code a bit nicer IMO.Yeah, these cleanups are fine. I would have prefered them as a separate patch (some people might want the performance improvement to be backported and this makes it unnecessarily more complex) but don't resend just because of that.
Makes sense.
Thanks,
Amir.
Jan Kara
maintainerQuoted 30 lines · Amir Goldstein
Quoted 29 lines · Jan Kara
Quoted 26 lines · Amir Goldstein
If parent inode is not watching, check for the event in masks of sb/mount/inode masks early to optimize out most of the code in
__fsnotify_parent()and avoid callingfsnotify().Jens has reported that this optimization improves BW and IOPS in an
io_uringbenchmark by more than 10% and reduces perf reported CPU usage.before:
(quoted diff) +2
-0+ 4.51% io_uring [kernel.vmlinux] [k] fsnotify + 3.67% io_uring [kernel.vmlinux] [k] __fsnotify_parentafter:
+ 2.37%
io_uring[kernel.vmlinux] [k]__fsnotify_parentReported-and-tested-by Jens Axboe <[email protected]>Signed-off-by Amir Goldstein <[email protected]>---
Jan,
Considering that this change looks like a clear win and it actually the change that you suggested, I cleaned it up a bit and posting for your consideration.
Quoted 6 lines
Quoted 4 lines
Agreed, I like this. What did you generate this patch against? It does not apply on top of current Linus' tree (maybe it needs the change sitting in VFS tree - which is fine I can wait until that gets merged)?
Yes, it is on top of Christian's vfs-fixes branch.
Merged your improvement now (and I've split off the cleanup into a separate change and dropped the creation of fsnotify_path() which seemed a bit pointless with a single caller). All pushed out.
Honza
Signature · 2 lines
Jan Kara <[email protected]> SUSE Labs, CR
Amir Goldstein
reviewerQuoted 37 lines · Jan Kara
Quoted 35 lines · Amir Goldstein
Quoted 33 lines · Jan Kara
Quoted 19 lines · Amir Goldstein
If parent inode is not watching, check for the event in masks of sb/mount/inode masks early to optimize out most of the code in
__fsnotify_parent()and avoid callingfsnotify().Jens has reported that this optimization improves BW and IOPS in an
io_uringbenchmark by more than 10% and reduces perf reported CPU usage.before:
(quoted diff) +2
-0+ 4.51% io_uring [kernel.vmlinux] [k] fsnotify + 3.67% io_uring [kernel.vmlinux] [k] __fsnotify_parentafter:
+ 2.37%
io_uring[kernel.vmlinux] [k]__fsnotify_parentReported-and-tested-by Jens Axboe <[email protected]>Signed-off-by Amir Goldstein <[email protected]>Quoted 7 lines
---
Jan,
Considering that this change looks like a clear win and it actually the change that you suggested, I cleaned it up a bit and posting for your consideration.
Agreed, I like this. What did you generate this patch against? It does not apply on top of current Linus' tree (maybe it needs the change sitting in VFS tree - which is fine I can wait until that gets merged)?
Quoted 6 lines
Quoted 1 line
Yes, it is on top of Christian's vfs-fixes branch.
Merged your improvement now (and I've split off the cleanup into a separate change and dropped the creation of
fsnotify_path()which seemed a bit pointless with a single caller). All pushed out.
Thanks!
Amir.
Amir Goldstein
reviewerQuoted 39 lines · Amir Goldstein
Quoted 37 lines · Jan Kara
Quoted 35 lines · Amir Goldstein
Quoted 33 lines · Jan Kara
Quoted 19 lines · Amir Goldstein
If parent inode is not watching, check for the event in masks of sb/mount/inode masks early to optimize out most of the code in
__fsnotify_parent()and avoid callingfsnotify().Jens has reported that this optimization improves BW and IOPS in an
io_uringbenchmark by more than 10% and reduces perf reported CPU usage.before:
(quoted diff) +2
-0+ 4.51% io_uring [kernel.vmlinux] [k] fsnotify + 3.67% io_uring [kernel.vmlinux] [k] __fsnotify_parentafter:
+ 2.37%
io_uring[kernel.vmlinux] [k]__fsnotify_parentReported-and-tested-by Jens Axboe <[email protected]>Signed-off-by Amir Goldstein <[email protected]>Quoted 7 lines
---
Jan,
Considering that this change looks like a clear win and it actually the change that you suggested, I cleaned it up a bit and posting for your consideration.
Agreed, I like this. What did you generate this patch against? It does not apply on top of current Linus' tree (maybe it needs the change sitting in VFS tree - which is fine I can wait until that gets merged)?
Quoted 7 lines
Quoted 6 lines
Quoted 1 line
Yes, it is on top of Christian's vfs-fixes branch.
Merged your improvement now (and I've split off the cleanup into a separate change and dropped the creation of
fsnotify_path()which seemed a bit pointless with a single caller). All pushed out.
Jan & Jens,
Although Jan has already queued this v3 patch with sufficient performance improvement for Jens' workloads, I got a performance regression report from kernel robot on will-it-scale microbenchmark (buffered write loop) on my fan_pre_content patches, so I tried to improve on the existing solution.
I tried something similar to v1/v2 patches, where the sb keeps accounting of the number of watchers for specific sub-classes of events.
I've made two major changes:
1. moved to counters into a per-sb state object fsnotify_sb_connector as Christian requested
2. The counters are by fanotify classes, not by specific events, so they can be used to answer the questions:
a) Are there any fsnotify watchers on this sb?
b) Are there any fanotify permission class listeners on this sb?
c) Are there any fanotify pre-content (a.k.a HSM) class listeners on this sb?
I think that those questions are very relevant in the real world, because a positive answer to (b) and (c) is quite rare in the real world, so the overhead on the permission hooks could be completely eliminated in the common case.
If needed, we can further bisect the class counters per specific painful events (e.g. FAN_ACCESS*), but there is no need to do that before we see concrete benchmark results.
Jens,
If you feel like it, you can see if this branch further improves your workloads:
https://github.com/amir73il/linux/commits/fsnotify-perf/
Jan,
Whenever you have the time, feel free to see if this is a valid direction, if not for the perf optimization then we are going to need the fsnotify_sb_connector container for other features as well.
Thanks!
Amir.
Jan Kara
maintainerQuoted 65 lines · Amir Goldstein
Quoted 37 lines · Amir Goldstein
Quoted 36 lines · Jan Kara
Quoted 35 lines · Amir Goldstein
Quoted 33 lines · Jan Kara
Quoted 19 lines · Amir Goldstein
If parent inode is not watching, check for the event in masks of sb/mount/inode masks early to optimize out most of the code in
__fsnotify_parent()and avoid callingfsnotify().Jens has reported that this optimization improves BW and IOPS in an
io_uringbenchmark by more than 10% and reduces perf reported CPU usage.before:
(quoted diff) +2
-0+ 4.51% io_uring [kernel.vmlinux] [k] fsnotify + 3.67% io_uring [kernel.vmlinux] [k] __fsnotify_parentafter:
+ 2.37%
io_uring[kernel.vmlinux] [k]__fsnotify_parentReported-and-tested-by Jens Axboe <[email protected]>Signed-off-by Amir Goldstein <[email protected]>Quoted 7 lines
---
Jan,
Considering that this change looks like a clear win and it actually the change that you suggested, I cleaned it up a bit and posting for your consideration.
Agreed, I like this. What did you generate this patch against? It does not apply on top of current Linus' tree (maybe it needs the change sitting in VFS tree - which is fine I can wait until that gets merged)?
Quoted 7 lines
Quoted 6 lines
Quoted 1 line
Yes, it is on top of Christian's vfs-fixes branch.
Merged your improvement now (and I've split off the cleanup into a separate change and dropped the creation of
fsnotify_path()which seemed a bit pointless with a single caller). All pushed out.Jan & Jens,
Although Jan has already queued this v3 patch with sufficient performance improvement for Jens' workloads, I got a performance regression report from kernel robot on will-it-scale microbenchmark (buffered write loop) on my
fan_pre_contentpatches, so I tried to improve on the existing solution.I tried something similar to v1/v2 patches, where the sb keeps accounting of the number of watchers for specific sub-classes of events.
I've made two major changes:
1. moved to counters into a per-sb state objectfsnotify_sb_connectoras Christian requested
2. The counters are by fanotify classes, not by specific events, so they can be used to answer the questions:
a) Are there any fsnotify watchers on this sb?
b) Are there any fanotify permission class listeners on this sb?
c) Are there any fanotify pre-content (a.k.a HSM) class listeners on this sb?
Quoted 8 lines
I think that those questions are very relevant in the real world, because a positive answer to (b) and (c) is quite rare in the real world, so the overhead on the permission hooks could be completely eliminated in the common case.
If needed, we can further bisect the class counters per specific painful events (e.g. FAN_ACCESS*), but there is no need to do that before we see concrete benchmark results.
OK, I think this idea is sound, I'd just be interested whether the 0-day bot (or somebody else) is able to see improvement with this. Otherwise why bother :)
Quoted 5 lines
Jan,
Whenever you have the time, feel free to see if this is a valid direction, if not for the perf optimization then we are going to need the
fsnotify_sb_connectorcontainer for other features as well.
So firstly the name fsnotify_sb_connector really confuses me. I'd save "connector" names to fsnotify_mark_connector. Maybe fsnotify_sb_info?
Then I dislike how we have to specialcase superblock in quite a few places and add these wrappers and what not. This seems to be mostly caused by the fact that you directly embed fsnotify_mark_connector into fsnotify_sb_info. What if we just put fsnotify_connp_t there? I understand that this will mean one more pointer fetch if there are actually marks attached to the superblock and the event mask matches s_fsnotify_mask. But in that case we are likely to generate the event anyway so the cost of that compared to event generation is negligible?
And I'd allocate fsnotify_sb_info on demand from fsnotify_add_mark_locked() which means that we need to pass object pointer (in the form of void *) instead of fsnotify_connp_t to various mark adding functions (and transform it to fsnotify_connp_t only in fsnotify_add_mark_locked() after possibly setting up fsnotify_sb_info). Passing void * around is not great but it should be fairly limited (and actually reduces the knowledge of fsnotify internals outside of the fsnotify core).
Honza
Signature · 2 lines
Jan Kara <[email protected]> SUSE Labs, CR
Amir Goldstein
reviewerQuoted 30 lines
Quoted 30 lines
Quoted 5 lines
Quoted 4 lines
Merged your improvement now (and I've split off the cleanup into a separate change and dropped the creation of
fsnotify_path()which seemed a bit pointless with a single caller). All pushed out.Jan & Jens,
Although Jan has already queued this v3 patch with sufficient performance improvement for Jens' workloads, I got a performance regression report from kernel robot on will-it-scale microbenchmark (buffered write loop) on my
fan_pre_contentpatches, so I tried to improve on the existing solution.I tried something similar to v1/v2 patches, where the sb keeps accounting of the number of watchers for specific sub-classes of events.
I've made two major changes:
1. moved to counters into a per-sb state objectfsnotify_sb_connectoras Christian requested
2. The counters are by fanotify classes, not by specific events, so they can be used to answer the questions:
a) Are there any fsnotify watchers on this sb?
b) Are there any fanotify permission class listeners on this sb?
c) Are there any fanotify pre-content (a.k.a HSM) class listeners on this sb?I think that those questions are very relevant in the real world, because a positive answer to (b) and (c) is quite rare in the real world, so the overhead on the permission hooks could be completely eliminated in the common case.
Quoted 8 lines
Quoted 3 lines
If needed, we can further bisect the class counters per specific painful events (e.g. FAN_ACCESS*), but there is no need to do that before we see concrete benchmark results.
OK, I think this idea is sound, I'd just be interested whether the 0-day bot (or somebody else) is able to see improvement with this. Otherwise why bother :)
Exactly.
Quoted 9 lines
Quoted 5 lines
Jan,
Whenever you have the time, feel free to see if this is a valid direction, if not for the perf optimization then we are going to need the
fsnotify_sb_connectorcontainer for other features as well.So firstly the name
fsnotify_sb_connectorreally confuses me. I'd save "connector" names tofsnotify_mark_connector. Maybefsnotify_sb_info?
Sure.
Quoted 9 lines
Then I dislike how we have to specialcase superblock in quite a few places and add these wrappers and what not. This seems to be mostly caused by the fact that you directly embed
fsnotify_mark_connectorintofsnotify_sb_info. What if we just putfsnotify_connp_tthere? I understand that this will mean one more pointer fetch if there are actually marks attached to the superblock and the event mask matchess_fsnotify_mask. But in that case we are likely to generate the event anyway so the cost of that compared to event generation is negligible?
I guess that can work.
I can try it and see if there are any other complications.
Quoted 7 lines
And I'd allocate
fsnotify_sb_infoon demand fromfsnotify_add_mark_locked()which means that we need to pass object pointer (in the form of void *) instead offsnotify_connp_tto various mark adding functions (and transform it tofsnotify_connp_tonly infsnotify_add_mark_locked()after possibly setting upfsnotify_sb_info). Passing void * around is not great but it should be fairly limited (and actually reduces the knowledge of fsnotify internals outside of the fsnotify core).
Unless I am missing something, I think we only need to pass an extra sb arg to fsnotify_add_mark_locked()? and it does not sound like a big deal. For adding an sb mark, connp arg could be NULL, and then we get connp from sb->fsnotify_sb_info after making sure that it is allocated.
I will get to look at it in ~2 weeks.
Thanks for the quick feedback.
Amir.
Jan Kara
maintainerQuoted 33 lines · Amir Goldstein
Quoted 33 lines
Quoted 33 lines
Quoted 5 lines
Quoted 4 lines
Merged your improvement now (and I've split off the cleanup into a separate change and dropped the creation of
fsnotify_path()which seemed a bit pointless with a single caller). All pushed out.Jan & Jens,
Although Jan has already queued this v3 patch with sufficient performance improvement for Jens' workloads, I got a performance regression report from kernel robot on will-it-scale microbenchmark (buffered write loop) on my
fan_pre_contentpatches, so I tried to improve on the existing solution.I tried something similar to v1/v2 patches, where the sb keeps accounting of the number of watchers for specific sub-classes of events.
I've made two major changes:
1. moved to counters into a per-sb state objectfsnotify_sb_connectoras Christian requested
2. The counters are by fanotify classes, not by specific events, so they can be used to answer the questions:
a) Are there any fsnotify watchers on this sb?
b) Are there any fanotify permission class listeners on this sb?
c) Are there any fanotify pre-content (a.k.a HSM) class listeners on this sb?I think that those questions are very relevant in the real world, because a positive answer to (b) and (c) is quite rare in the real world, so the overhead on the permission hooks could be completely eliminated in the common case.
If needed, we can further bisect the class counters per specific painful events (e.g. FAN_ACCESS*), but there is no need to do that before we see concrete benchmark results.
...
Quoted 21 lines
Quoted 9 lines
Then I dislike how we have to specialcase superblock in quite a few places and add these wrappers and what not. This seems to be mostly caused by the fact that you directly embed
fsnotify_mark_connectorintofsnotify_sb_info. What if we just putfsnotify_connp_tthere? I understand that this will mean one more pointer fetch if there are actually marks attached to the superblock and the event mask matchess_fsnotify_mask. But in that case we are likely to generate the event anyway so the cost of that compared to event generation is negligible?I guess that can work.
I can try it and see if there are any other complications.Quoted 7 lines
And I'd allocate
fsnotify_sb_infoon demand fromfsnotify_add_mark_locked()which means that we need to pass object pointer (in the form of void *) instead offsnotify_connp_tto various mark adding functions (and transform it tofsnotify_connp_tonly infsnotify_add_mark_locked()after possibly setting upfsnotify_sb_info). Passing void * around is not great but it should be fairly limited (and actually reduces the knowledge of fsnotify internals outside of the fsnotify core).
Quoted 4 lines
Unless I am missing something, I think we only need to pass an extra sb arg to
fsnotify_add_mark_locked()? and it does not sound like a big deal. For adding an sb mark, connp arg could be NULL, and then we get connp fromsb->fsnotify_sb_infoafter making sure that it is allocated.
Yes that would be another possibility but frankly I like passing the 'object' pointer instead of connp pointer a bit more. But we can see how the code looks like.
Honza
Signature · 2 lines
Jan Kara <[email protected]> SUSE Labs, CR
Amir Goldstein
reviewerQuoted 58 lines · Jan Kara
Quoted 33 lines · Amir Goldstein
Quoted 33 lines
Quoted 33 lines
Quoted 5 lines
Quoted 4 lines
Merged your improvement now (and I've split off the cleanup into a separate change and dropped the creation of
fsnotify_path()which seemed a bit pointless with a single caller). All pushed out.Jan & Jens,
Although Jan has already queued this v3 patch with sufficient performance improvement for Jens' workloads, I got a performance regression report from kernel robot on will-it-scale microbenchmark (buffered write loop) on my
fan_pre_contentpatches, so I tried to improve on the existing solution.I tried something similar to v1/v2 patches, where the sb keeps accounting of the number of watchers for specific sub-classes of events.
I've made two major changes:
1. moved to counters into a per-sb state objectfsnotify_sb_connectoras Christian requested
2. The counters are by fanotify classes, not by specific events, so they can be used to answer the questions:
a) Are there any fsnotify watchers on this sb?
b) Are there any fanotify permission class listeners on this sb?
c) Are there any fanotify pre-content (a.k.a HSM) class listeners on this sb?I think that those questions are very relevant in the real world, because a positive answer to (b) and (c) is quite rare in the real world, so the overhead on the permission hooks could be completely eliminated in the common case.
If needed, we can further bisect the class counters per specific painful events (e.g. FAN_ACCESS*), but there is no need to do that before we see concrete benchmark results.
...
Quoted 21 lines
Quoted 9 lines
Then I dislike how we have to specialcase superblock in quite a few places and add these wrappers and what not. This seems to be mostly caused by the fact that you directly embed
fsnotify_mark_connectorintofsnotify_sb_info. What if we just putfsnotify_connp_tthere? I understand that this will mean one more pointer fetch if there are actually marks attached to the superblock and the event mask matchess_fsnotify_mask. But in that case we are likely to generate the event anyway so the cost of that compared to event generation is negligible?I guess that can work.
I can try it and see if there are any other complications.Quoted 7 lines
And I'd allocate
fsnotify_sb_infoon demand fromfsnotify_add_mark_locked()which means that we need to pass object pointer (in the form of void *) instead offsnotify_connp_tto various mark adding functions (and transform it tofsnotify_connp_tonly infsnotify_add_mark_locked()after possibly setting upfsnotify_sb_info). Passing void * around is not great but it should be fairly limited (and actually reduces the knowledge of fsnotify internals outside of the fsnotify core).
Quoted 8 lines
Quoted 4 lines
Unless I am missing something, I think we only need to pass an extra sb arg to
fsnotify_add_mark_locked()? and it does not sound like a big deal. For adding an sb mark, connp arg could be NULL, and then we get connp fromsb->fsnotify_sb_infoafter making sure that it is allocated.Yes that would be another possibility but frankly I like passing the 'object' pointer instead of connp pointer a bit more. But we can see how the code looks like.
Ok, here it is:
https://github.com/amir73il/linux/commits/fsnotify-sbinfo/
I agree that the interface does end up looking better this way.
I've requested to re-test performance on fsnotify-sbinfo.
You can use this rebased branch to look at the diff from the the previous patches that were tested by 0day:
https://github.com/amir73il/linux/commits/fsnotify-sbconn/
If you have the bandwidth to consider those patches as candidates for (the second half of?) 6.9 merge window, I can post them for review.
The main benefit of getting them merged in 6.9 is that I could work on pre-content events for 6.10 without having to worry about perf regressions.
Thanks,
Amir.
Jan Kara
maintainerQuoted 67 lines · Amir Goldstein
Quoted 57 lines · Jan Kara
Quoted 33 lines · Amir Goldstein
Quoted 33 lines
Quoted 33 lines
Quoted 5 lines
Quoted 4 lines
Merged your improvement now (and I've split off the cleanup into a separate change and dropped the creation of
fsnotify_path()which seemed a bit pointless with a single caller). All pushed out.Jan & Jens,
Although Jan has already queued this v3 patch with sufficient performance improvement for Jens' workloads, I got a performance regression report from kernel robot on will-it-scale microbenchmark (buffered write loop) on my
fan_pre_contentpatches, so I tried to improve on the existing solution.I tried something similar to v1/v2 patches, where the sb keeps accounting of the number of watchers for specific sub-classes of events.
I've made two major changes:
1. moved to counters into a per-sb state objectfsnotify_sb_connectoras Christian requested
2. The counters are by fanotify classes, not by specific events, so they can be used to answer the questions:
a) Are there any fsnotify watchers on this sb?
b) Are there any fanotify permission class listeners on this sb?
c) Are there any fanotify pre-content (a.k.a HSM) class listeners on this sb?I think that those questions are very relevant in the real world, because a positive answer to (b) and (c) is quite rare in the real world, so the overhead on the permission hooks could be completely eliminated in the common case.
If needed, we can further bisect the class counters per specific painful events (e.g. FAN_ACCESS*), but there is no need to do that before we see concrete benchmark results.
...
Quoted 21 lines
Quoted 9 lines
Then I dislike how we have to specialcase superblock in quite a few places and add these wrappers and what not. This seems to be mostly caused by the fact that you directly embed
fsnotify_mark_connectorintofsnotify_sb_info. What if we just putfsnotify_connp_tthere? I understand that this will mean one more pointer fetch if there are actually marks attached to the superblock and the event mask matchess_fsnotify_mask. But in that case we are likely to generate the event anyway so the cost of that compared to event generation is negligible?I guess that can work.
I can try it and see if there are any other complications.Quoted 7 lines
And I'd allocate
fsnotify_sb_infoon demand fromfsnotify_add_mark_locked()which means that we need to pass object pointer (in the form of void *) instead offsnotify_connp_tto various mark adding functions (and transform it tofsnotify_connp_tonly infsnotify_add_mark_locked()after possibly setting upfsnotify_sb_info). Passing void * around is not great but it should be fairly limited (and actually reduces the knowledge of fsnotify internals outside of the fsnotify core).Quoted 8 lines
Quoted 4 lines
Unless I am missing something, I think we only need to pass an extra sb arg to
fsnotify_add_mark_locked()? and it does not sound like a big deal. For adding an sb mark, connp arg could be NULL, and then we get connp fromsb->fsnotify_sb_infoafter making sure that it is allocated.Yes that would be another possibility but frankly I like passing the 'object' pointer instead of connp pointer a bit more. But we can see how the code looks like.
Quoted 5 lines
Ok, here it is:
https://github.com/amir73il/linux/commits/fsnotify-sbinfo/
I agree that the interface does end up looking better this way.
Yep, the interface looks fine. I have left some comments on github regarding typos and some suspicious things.
Quoted 9 lines
I've requested to re-test performance on fsnotify-sbinfo.
You can use this rebased branch to look at the diff from the the previous patches that were tested by 0day:
https://github.com/amir73il/linux/commits/fsnotify-sbconn/
If you have the bandwidth to consider those patches as candidates for (the second half of?) 6.9 merge window, I can post them for review.
Well, unless Linus does rc8, I don't think we should queue these for the merge window as it is too late by now. But please post them for review, I'll have a look. I can then push them to my tree early into a stable branch and you can base your patches on my branch. If the patches then need to go through VFS tree, Christian is fine with pulling my tree...
Honza
Signature · 2 lines
Jan Kara <[email protected]> SUSE Labs, CR
Christian Brauner
maintainerQuoted 84 lines · Jan Kara
Quoted 67 lines · Amir Goldstein
Quoted 57 lines · Jan Kara
Quoted 33 lines · Amir Goldstein
Quoted 33 lines
Quoted 33 lines
Quoted 5 lines
Quoted 4 lines
Merged your improvement now (and I've split off the cleanup into a separate change and dropped the creation of
fsnotify_path()which seemed a bit pointless with a single caller). All pushed out.Jan & Jens,
Although Jan has already queued this v3 patch with sufficient performance improvement for Jens' workloads, I got a performance regression report from kernel robot on will-it-scale microbenchmark (buffered write loop) on my
fan_pre_contentpatches, so I tried to improve on the existing solution.I tried something similar to v1/v2 patches, where the sb keeps accounting of the number of watchers for specific sub-classes of events.
I've made two major changes:
1. moved to counters into a per-sb state objectfsnotify_sb_connectoras Christian requested
2. The counters are by fanotify classes, not by specific events, so they can be used to answer the questions:
a) Are there any fsnotify watchers on this sb?
b) Are there any fanotify permission class listeners on this sb?
c) Are there any fanotify pre-content (a.k.a HSM) class listeners on this sb?I think that those questions are very relevant in the real world, because a positive answer to (b) and (c) is quite rare in the real world, so the overhead on the permission hooks could be completely eliminated in the common case.
If needed, we can further bisect the class counters per specific painful events (e.g. FAN_ACCESS*), but there is no need to do that before we see concrete benchmark results.
...
Quoted 21 lines
Quoted 9 lines
Then I dislike how we have to specialcase superblock in quite a few places and add these wrappers and what not. This seems to be mostly caused by the fact that you directly embed
fsnotify_mark_connectorintofsnotify_sb_info. What if we just putfsnotify_connp_tthere? I understand that this will mean one more pointer fetch if there are actually marks attached to the superblock and the event mask matchess_fsnotify_mask. But in that case we are likely to generate the event anyway so the cost of that compared to event generation is negligible?I guess that can work.
I can try it and see if there are any other complications.Quoted 7 lines
And I'd allocate
fsnotify_sb_infoon demand fromfsnotify_add_mark_locked()which means that we need to pass object pointer (in the form of void *) instead offsnotify_connp_tto various mark adding functions (and transform it tofsnotify_connp_tonly infsnotify_add_mark_locked()after possibly setting upfsnotify_sb_info). Passing void * around is not great but it should be fairly limited (and actually reduces the knowledge of fsnotify internals outside of the fsnotify core).Quoted 8 lines
Quoted 4 lines
Unless I am missing something, I think we only need to pass an extra sb arg to
fsnotify_add_mark_locked()? and it does not sound like a big deal. For adding an sb mark, connp arg could be NULL, and then we get connp fromsb->fsnotify_sb_infoafter making sure that it is allocated.Yes that would be another possibility but frankly I like passing the 'object' pointer instead of connp pointer a bit more. But we can see how the code looks like.
Quoted 5 lines
Ok, here it is:
https://github.com/amir73il/linux/commits/fsnotify-sbinfo/
I agree that the interface does end up looking better this way.
Yep, the interface looks fine. I have left some comments on github regarding typos and some suspicious things.
Quoted 7 lines
I've requested to re-test performance on fsnotify-sbinfo.
You can use this rebased branch to look at the diff from the the previous patches that were tested by 0day:
Quoted 8 lines
Quoted 2 lines
If you have the bandwidth to consider those patches as candidates for (the second half of?) 6.9 merge window, I can post them for review.
Well, unless Linus does rc8, I don't think we should queue these for the merge window as it is too late by now. But please post them for review, I'll have a look. I can then push them to my tree early into a stable branch and you can base your patches on my branch. If the patches then need to go through VFS tree, Christian is fine with pulling my tree...
I'm absolutely opposed to touching anything that you do. I'm joking of course, I'm very happy to pull from you!
Quoted 6 lines · Amir Goldstein
Jens,
If you feel like it, you can see if this branch further improves your workloads:
Baseline is current -git with changes I have for 6.9, so not with the previous patch. 4 optanes, usually random reads, and being run on a dual socket Intel(R) Xeon(R) Platinum 8458P CPU @ 2.7GHz.
IOPS=16.20M, BW=7.91GiB/s, IOS/call=32/31
IOPS=16.20M, BW=7.91GiB/s, IOS/call=32/32
IOPS=16.20M, BW=7.91GiB/s, IOS/call=32/31
and in perf profile, we see:
+ 3.16% io_uring [kernel.kallsyms] [k] fsnotify + 2.04% io_uring [kernel.kallsyms] [k] __fsnotify_parent
with this branch pulled in:
IOPS=17.44M, BW=8.52GiB/s, IOS/call=32/31
IOPS=17.45M, BW=8.52GiB/s, IOS/call=32/31
IOPS=17.45M, BW=8.52GiB/s, IOS/call=32/31
and perf diff shows:
2.04% -1.15% [kernel.kallsyms] [k] __fsnotify_parent
3.16% [kernel.kallsyms] [k] fsnotifywith a big reduction for __fsnotify_parent() and fsnotify() being totally gone. In absolute terms, this is all we see in perf profile with the patch:
+ 0.89% io_uring [kernel.kallsyms] [k] __fsnotify_parent
iow, we went from over 5% of added overhead to less than 1%.
Signature · 1 line
Jens Axboe