kernfs: allocate inodes and dentry outside kernfs_rwsem
Subsystems (3)
PATCH v4 1/1 kernfs: allocate inodes and dentry outside kernfs_rwsem
In the Meta fleet, on 6.16 with the kernfs changes from 7.4 backported, we still see long cgroupfs stalls:
- oomd and below waited 72 seconds in kernfs_fop_readdir() while an out of memory container created cgroups. They resumed when the kernel OOM killer killed the container's workload.
- oomd waited 15 minutes in kernfs_iop_lookup(). PID 1 waited for kernfs_rwsem in cgroup_rmdir() while holding cgroup_mutex.
- A lock profiler sampling part of the fleet saw kernfs_iop_lookup() hold kernfs_rwsem across memcg reclaim on 26 hosts in 29 days, for up to 625 ms.
The profiler shows where these stalls come from. kernfs_iop_lookup() and the export ops allocate a new inode while holding kernfs_rwsem for read. The inode is charged to the caller's memcg. If that memcg is at memory.max, the allocation can stay in reclaim for a long time, sometimes until an OOM kill. A writer then queues on the lock and every new reader queues behind it. On cgroup2 this blocks every cgroupfs user on the machine.
Such lookups are common in a memcg that is out of memory. Its reclaim drops its kernfs dentries fast, and the files of a new cgroup have no inodes yet.
Allocate the inode before taking kernfs_rwsem, as kernfs_iop_mkdir() already does, and attach it under the lock with kernfs_get_inode_locked(). That helper adds a new inode only for an active node, so __kernfs_remove() still finds every inode it must clear i_nlink on, as commit dc1a0cc95737 ("kernfs: don't lose IN_DELETE_SELF when decoding a file handle") needs.
The inode goes unused when the name does not exist, the node is inactive, or it already has an inode. In a VM, a lookup of a missing name takes about 1 us longer, 4.5 us instead of 3.5 us on cgroupfs. Cold lookups of existing sysfs and cgroupfs files take 2 to 3% longer.
kernfs_get_parent_dentry() also held the lock across d_obtain_alias(), which allocates a dentry. Hold it only around the inode lookup, as __kernfs_fh_to_dentry() already does. This also gives ->get_parent the active test that the IN_DELETE_SELF fix left out: a parent that is being removed now gets -ESTALE. Before, it got an inode anyway, possibly a new one whose i_nlink __kernfs_remove() never cleared.
kernfs_fill_super() preallocates the root inode the same way, so no inode is allocated under kernfs_rwsem any more.
Tested on a 316-CPU machine. A task in a full 256 MB memcg with slow swap looks up the files of new cgroups. Worst waits in 60 seconds:
before after
oomd, for kernfs_rwsem 236 ms 0.1 ms below, for kernfs_rwsem 356 ms 0.8 ms systemd, for kernfs_rwsem 160 ms 37 us readdir of another directory 530 ms 3.9 ms
The lookup itself is still slow. The VFS still holds its directory's lock, so other work in that one directory still waits.
---
Christian, this depends on Jori's series in vfs-7.5.directory. Could you take it on top of it? Greg, you asked to wait for -rc1 and for testing in -next. Are you OK with this going through the vfs tree?
Changes since v3:
- Rebased on vfs-7.5.directory. Use the kernfs_get_inode_locked() that Jori added there for kernfs_iop_mkdir().
- ->get_parent: a parent that is being removed now gets -ESTALE even if it already has an inode, because the helper tests the node first.
- New cost numbers. v3 costs the same as v4 in it.
- Kept Tejun's ack, as the approach he suggested is unchanged.
v3: https://lore.kernel.org/[email protected]
Changes since v2:
- Allocate the inode before taking kernfs_rwsem and free it if unused, instead of dropping and retaking the lock. kernfs_iget() becomes kernfs_get_inode_locked() (Tejun).
- Do the same in kernfs_fill_super() (Tejun).
- Describe the ->get_parent active test (Tejun).
- Retested on another machine, numbers updated.
v2: https://lore.kernel.org/[email protected]
Changes since v1:
- dentry allocation outside kernfs_rwsem (Sashiko).
- Couple of comments updated.
v1: https://lore.kernel.org/[email protected]
fs/kernfs/dir.c | 12 ++++++++++-- fs/kernfs/inode.c | 3 +++ fs/kernfs/mount.c | 49 +++++++++++++++++++++++++++++------------------ 3 files changed, 43 insertions(+), 21 deletions(-)
fs/kernfs/dir.c +10 -2
@@ -1251,9 +1251,12 @@ static struct dentry *kernfs_iop_lookup(struct inode *dir,struct kernfs_node *parent = dir->i_private;struct kernfs_node *kn;struct kernfs_root *root; -struct inode *inode =NULL ; +struct inode *inode =NULL , *new;const struct ns_common *ns =NULL ; +/* allocate outside kernfs_rwsem, see kernfs_get_inode_locked() */ + new =new_inode (dir->i_sb); + root =kernfs_root (parent);down_read (&root->kernfs_rwsem);if (kernfs_ns_enabled (parent))
@@ -1267,9 +1270,13 @@ static struct dentry *kernfs_iop_lookup(struct inode *dir,*/ if (!kernfs_active (kn)) {up_read (&root->kernfs_rwsem); +iput (new);return NULL ; } - inode =kernfs_get_inode (dir->i_sb, kn); +if (new) { + inode =kernfs_get_inode_locked (new, kn); + new =NULL ; + }if (!inode) inode =ERR_PTR (-ENOMEM ); }
@@ -1282,6 +1289,7 @@ static struct dentry *kernfs_iop_lookup(struct inode *dir,if (!IS_ERR (inode))kernfs_set_rev (parent, dentry);up_read (&root->kernfs_rwsem); +iput (new);/* instantiate and hash (possibly negative) dentry */ return d_splice_alias (inode, dentry);
fs/kernfs/inode.c +3 -0
@@ -269,6 +269,9 @@ static int kernfs_inode_test(struct inode *inode, void *data)* while the inode is inserted and __kernfs_remove() always finds it to * clear its i_nlink. @inode is consumed. * +* The caller allocates @inode before taking kernfs_rwsem, because the +* allocation can block in reclaim and hold up other users of the lock. +* * Returns the inode now attached to @kn, which is @inode unless one was * already hashed for it (e.g. by a file handle decode), or NULL if @kn * is no longer active.
fs/kernfs/mount.c +30 -19
94 lines not on this page · raw message
base-commit: ecdfc51a6a78aaf1c4ac1d8b6ea199cc2ed020ac
Signature · 1 line
2.53.0-Meta
Quoted 263 lines
Op 11-11-2026 16:50 EST schreef Shakeel Butt <[email protected]>:
In the Meta fleet, on 6.16 with the kernfs changes from 7.4 backported, we still see long cgroupfs stalls:
- oomd and below waited 72 seconds in
kernfs_fop_readdir()while an out of memory container created cgroups. They resumed when the kernel OOM killer killed the container's workload.- oomd waited 15 minutes in
kernfs_iop_lookup(). PID 1 waited forkernfs_rwsemincgroup_rmdir()while holdingcgroup_mutex.- A lock profiler sampling part of the fleet saw
kernfs_iop_lookup()holdkernfs_rwsemacross memcg reclaim on 26 hosts in 29 days, for up to 625 ms.The profiler shows where these stalls come from.
kernfs_iop_lookup()and the export ops allocate a new inode while holdingkernfs_rwsemfor read. The inode is charged to the caller's memcg. If that memcg is at memory.max, the allocation can stay in reclaim for a long time, sometimes until an OOM kill. A writer then queues on the lock and every new reader queues behind it. On cgroup2 this blocks every cgroupfs user on the machine.Such lookups are common in a memcg that is out of memory. Its reclaim drops its kernfs dentries fast, and the files of a new cgroup have no inodes yet.
Allocate the inode before taking
kernfs_rwsem, askernfs_iop_mkdir()already does, and attach it under the lock withkernfs_get_inode_locked(). That helper adds a new inode only for an active node, so__kernfs_remove()still finds every inode it must cleari_nlinkon, as commitdc1a0cc95737("kernfs: don't loseIN_DELETE_SELFwhen decoding a file handle") needs.The inode goes unused when the name does not exist, the node is inactive, or it already has an inode. In a VM, a lookup of a missing name takes about 1 us longer, 4.5 us instead of 3.5 us on cgroupfs. Cold lookups of existing sysfs and cgroupfs files take 2 to 3% longer.
kernfs_get_parent_dentry()also held the lock acrossd_obtain_alias(), which allocates a dentry. Hold it only around the inode lookup, as__kernfs_fh_to_dentry()already does. This also gives ->get_parent the active test that theIN_DELETE_SELFfix left out: a parent that is being removed now gets-ESTALE. Before, it got an inode anyway, possibly a new one whosei_nlink__kernfs_remove()never cleared.
kernfs_fill_super()preallocates the root inode the same way, so no inode is allocated underkernfs_rwsemany more.Tested on a 316-CPU machine. A task in a full 256 MB memcg with slow swap looks up the files of new cgroups. Worst waits in 60 seconds:
before after
oomd, for kernfs_rwsem 236 ms 0.1 ms below, for kernfs_rwsem 356 ms 0.8 ms systemd, for kernfs_rwsem 160 ms 37 us readdir of another directory 530 ms 3.9 msThe lookup itself is still slow. The VFS still holds its directory's lock, so other work in that one directory still waits.
Suggested-by Tejun Heo <[email protected]>Acked-by Tejun Heo <[email protected]>Signed-off-by Shakeel Butt <[email protected]>---
Christian, this depends on Jori's series in vfs-7.5.directory. Could you take it on top of it? Greg, you asked to wait for -rc1 and for testing in -next. Are you OK with this going through the vfs tree?
Changes since v3:
- Rebased on vfs-7.5.directory. Use thekernfs_get_inode_locked()that Jori added there forkernfs_iop_mkdir().
- ->get_parent: a parent that is being removed now gets-ESTALEeven if it already has an inode, because the helper tests the node first.
- New cost numbers. v3 costs the same as v4 in it.
- Kept Tejun's ack, as the approach he suggested is unchanged.v3: https://lore.kernel.org/[email protected]
Changes since v2:
- Allocate the inode before takingkernfs_rwsemand free it if unused, instead of dropping and retaking the lock.kernfs_iget()becomeskernfs_get_inode_locked()(Tejun).
- Do the same inkernfs_fill_super()(Tejun).
- Describe the ->get_parent active test (Tejun).
- Retested on another machine, numbers updated.v2: https://lore.kernel.org/[email protected]
Changes since v1:
- dentry allocation outsidekernfs_rwsem(Sashiko).
- Couple of comments updated.v1: https://lore.kernel.org/[email protected]
fs/kernfs/dir.c | 12 ++++++++++-- fs/kernfs/inode.c | 3 +++ fs/kernfs/mount.c | 49 +++++++++++++++++++++++++++++------------------ 3 files changed, 43 insertions(+), 21 deletions(-)fs/kernfs/dir.c +10
-2@@ -1251,9 +1251,12 @@ static struct dentry *kernfs_iop_lookup(struct inode *dir,struct kernfs_node *parent = dir->i_private;struct kernfs_node *kn;struct kernfs_root *root; -struct inode *inode =NULL ; +struct inode *inode =NULL , *new;const struct ns_common *ns =NULL ; +/* allocate outside kernfs_rwsem, see kernfs_get_inode_locked() */ + new =new_inode (dir->i_sb); + root =kernfs_root (parent);down_read (&root->kernfs_rwsem);if (kernfs_ns_enabled (parent))@@ -1267,9 +1270,13 @@ static struct dentry *kernfs_iop_lookup(struct inode *dir,*/ if (!kernfs_active (kn)) {up_read (&root->kernfs_rwsem); +iput (new);return NULL ; } - inode =kernfs_get_inode (dir->i_sb, kn); +if (new) { + inode =kernfs_get_inode_locked (new, kn); + new =NULL ; + }if (!inode) inode =ERR_PTR (-ENOMEM ); }@@ -1282,6 +1289,7 @@ static struct dentry *kernfs_iop_lookup(struct inode *dir,if (!IS_ERR (inode))kernfs_set_rev (parent, dentry);up_read (&root->kernfs_rwsem); +iput (new);/* instantiate and hash (possibly negative) dentry */ return d_splice_alias (inode, dentry);fs/kernfs/inode.c +3
-0@@ -269,6 +269,9 @@ static int kernfs_inode_test(struct inode *inode, void *data)* while the inode is inserted and __kernfs_remove() always finds it to * clear its i_nlink. @inode is consumed. * +* The caller allocates @inode before taking kernfs_rwsem, because the +* allocation can block in reclaim and hold up other users of the lock. +* * Returns the inode now attached to @kn, which is @inode unless one was * already hashed for it (e.g. by a file handle decode), or NULL if @kn * is no longer active. fs/kernfs/mount.c +30
-19@@ -99,8 +99,8 @@ static struct dentry *__kernfs_fh_to_dentry(struct super_block *sb, int fh_type, bool get_parent) {struct kernfs_super_info *info =kernfs_info (sb); +struct inode *inode =ERR_PTR (-ESTALE ), *new;struct kernfs_node *kn; -struct inode *inode;u64 id;if (fh_len <2 )@@ -124,32 +124,32 @@ static struct dentry *__kernfs_fh_to_dentry(struct super_block *sb,return NULL ; } + new =new_inode (sb); +if (!new) +return ERR_PTR (-ENOMEM ); +/* -* Hold kernfs_rwsem across the lookup as well as kernfs_get_inode(). -* __kernfs_remove() deactivates the subtree and clears i_nlink on its -* inodes under the write lock, so under the read lock either -* kernfs_find_and_get_node_by_id() refuses the node, or the inode is -* in the inode hash before the ilookup() pass goes looking for it. +* Hold kernfs_rwsem from the lookup through kernfs_get_inode_locked(). +* Then __kernfs_remove() either finds the inode to clear i_nlink, or +* the node is already inactive and no inode is created. */ scoped_guard(rwsem_read, &info->root->kernfs_rwsem) { kn =kernfs_find_and_get_node_by_id (info->root, id); -if (!kn) -return ERR_PTR (-ESTALE ); - -if (get_parent) { +if (kn && get_parent) {struct kernfs_node *parent; parent =kernfs_get_parent (kn);kernfs_put (kn); kn = parent; -if (!kn) -return ERR_PTR (-ESTALE ); } - - inode =kernfs_get_inode (sb, kn); -kernfs_put (kn); +if (kn) { + inode =kernfs_get_inode_locked (new, kn); + new =NULL ; +kernfs_put (kn); + } } +iput (new);return d_obtain_alias (inode); }@@ -171,9 +171,17 @@ static struct dentry *kernfs_get_parent_dentry(struct dentry *child) {struct kernfs_node *kn =kernfs_dentry_node (child);struct kernfs_root *root =kernfs_root (kn); +struct inode *inode, *new; -guard (rwsem_read)(&root->kernfs_rwsem); -return d_obtain_alias (kernfs_get_inode (child->d_sb,kernfs_parent (kn))); + new =new_inode (child->d_sb); +if (!new) +return ERR_PTR (-ENOMEM ); + + scoped_guard(rwsem_read, &root->kernfs_rwsem) { + inode =kernfs_get_inode_locked (new,kernfs_parent (kn)); + } + +return d_obtain_alias (inode); }static const struct export_operations kernfs_export_ops = {@@ -294,7 +302,7 @@ static int kernfs_fill_super(struct super_block *sb, struct kernfs_fs_context *k {struct kernfs_super_info *info =kernfs_info (sb);struct kernfs_root *kf_root = kfc->root; -struct inode *inode; +struct inode *inode, *new;struct dentry *root; info->sb = sb;@@ -314,8 +322,11 @@ static int kernfs_fill_super(struct super_block *sb, struct kernfs_fs_context *k sb->s_shrink->seeks =0 ;/* get root inode, initialize and unlock it */ + new =new_inode (sb); +if (!new) +return -ENOMEM ;down_read (&kf_root->kernfs_rwsem); - inode =kernfs_get_inode (sb, info->root->kn); + inode =kernfs_get_inode_locked (new, info->root->kn);up_read (&kf_root->kernfs_rwsem);if (!inode) {pr_debug ("kernfs: could not get root inode\n" );
Quoted 3 lines
base-commit:
ecdfc51a6a78aaf1c4ac1d8b6ea199cc2ed020acSignature · 1 line
2.53.0-Meta
Does this compile? I added a bool parameter to kernfs_get_inode_locked(), for this patch to be able to do a lookup first, but you haven't added that. I think you want to set lookup=true everywhere, right?
Thanks,
Jori.
Quoted 4 lines · Jori Koolstra
Quoted 4 lines
Op 11-11-2026 16:50 EST schreef Shakeel Butt <[email protected]>:
[...]
Quoted 10 lines
Quoted 10 lines
fs/kernfs/mount.c +4
-1+ new =new_inode (sb); +if (!new) +return -ENOMEM ;down_read (&kf_root->kernfs_rwsem); - inode =kernfs_get_inode (sb, info->root->kn); + inode =kernfs_get_inode_locked (new, info->root->kn);up_read (&kf_root->kernfs_rwsem);if (!inode) {pr_debug ("kernfs: could not get root inode\n" );
Quoted 8 lines
Quoted 3 lines
base-commit:
ecdfc51a6a78aaf1c4ac1d8b6ea199cc2ed020acSignature · 1 line
2.53.0-MetaDoes this compile? I added a bool parameter to
kernfs_get_inode_locked(), for this patch to be able to do a lookup first, but you haven't added that. I think you want to set lookup=true everywhere, right?
Yes, it builds on vfs-7.5.directory (base-commit ecdfc51a6a78). Christian's fixup there dropped @lookup, so kernfs_get_inode_locked() now takes just (inode, kn).
I don't think this patch needs @lookup. inode_insert5() already returns an inode that is already hashed, so an ilookup() first only changes things for an inactive node. Lookup and the fh_to_* paths only get there with an active node, under the same kernfs_rwsem hold, and fill_super works on a new superblock. In ->get_parent, an inactive parent means the directory being decoded is being removed. reconnect_one() then returns -ESTALE either way, because readdir skips the removed child.
Jori Koolstra
Reviewed-byQuoted 29 lines
Op 12-11-2026 08:29 EST schreef Shakeel Butt <[email protected]>:
Quoted 4 lines · Jori Koolstra
Quoted 4 lines
Op 11-11-2026 16:50 EST schreef Shakeel Butt <[email protected]>:
[...]
Quoted 10 lines
Quoted 10 lines
fs/kernfs/mount.c +4
-1+ new =new_inode (sb); +if (!new) +return -ENOMEM ;down_read (&kf_root->kernfs_rwsem); - inode =kernfs_get_inode (sb, info->root->kn); + inode =kernfs_get_inode_locked (new, info->root->kn);up_read (&kf_root->kernfs_rwsem);if (!inode) {pr_debug ("kernfs: could not get root inode\n" );Quoted 8 lines
Quoted 3 lines
base-commit:
ecdfc51a6a78aaf1c4ac1d8b6ea199cc2ed020acSignature · 1 line
2.53.0-MetaDoes this compile? I added a bool parameter to
kernfs_get_inode_locked(), for this patch to be able to do a lookup first, but you haven't added that. I think you want to set lookup=true everywhere, right?
Quoted 3 lines
Yes, it builds on vfs-7.5.directory (base-commit
ecdfc51a6a78). Christian's fixup there dropped @lookup, sokernfs_get_inode_locked()now takes just (inode, kn).
Ah, I didn't notice that.
Quoted 4 lines
I don't think this patch needs @lookup.
inode_insert5()already returns an inode that is already hashed, so anilookup()first only changes things for an inactive node.
Fair enough, I was thinking you added it on purpose becasue ilookup() does not get the inode_hash_lock lock. But if you see no regression by always taking the lock, it's cleaner to get rid of the lookup flag.
At any rate, feel free to add:
Hi Shakeel,
fs/kernfs/mount.c static struct dentry *__kernfs_fh_to_dentry(struct super_block *sb, · line 129
Quoted 3 lines · Shakeel Butt
fs/kernfs/mount.c +3
-0+ new =new_inode (sb); +if (!new) +return ERR_PTR (-ENOMEM );
Not a big deal, but one thing changed from v3 that the changelog doesn't mention. Without an ilookup() first, every caller now needs the preallocated inode even when one is already hashed. Before this patch iget_locked() returned a hashed inode without allocating. So when new_inode() fails, decoding a handle for a live node whose inode is cached now fails with -ENOMEM, and a stale handle gets -ENOMEM instead of -ESTALE. kernfs_iop_lookup() does the same for a node that already has an inode.
From a quick check I did with failslab limited to inode_cache and an fd held open on a cgroup2 directory, open_by_handle_at() on its handle succeeds without this patch and fails with ENOMEM with it.
Now, new_inode() only fails there for an OOM victim, or in a memcg whose OOM killer is disabled or has nothing left to kill, and those tasks are failing most other allocations too, so I don't think it's worth complicating kernfs_get_inode_locked() for. But it is a second case besides an inactive node where an ilookup() first changes the result.
Not a problem, just an observation, so feel free to add:
Thanks!
Chris
Quoted 8 lines · Shakeel Butt
In the Meta fleet, on 6.16 with the kernfs changes from 7.4 backported, we still see long cgroupfs stalls:
- oomd and below waited 72 seconds in
kernfs_fop_readdir()while an out of memory container created cgroups. They resumed when the kernel OOM killer killed the container's workload.[...]
Applied to the vfs-7.5.directory branch of the vfs/vfs.git tree. Patches in the vfs-7.5.directory branch should appear in linux-next soon.
Please report any outstanding bugs that were missed during review in a new review to the original patch series allowing us to drop it.
It's encouraged to provide Acked-bys and Reviewed-bys even though the patch has now been applied. If possible patch trailers will be updated.
Note that commit hashes shown below are subject to change due to rebase, trailer updates or similar. If in doubt, please check the listed branch.
tree: https://git.kernel.org/pub/scm/linux/kernel/git/vfs/vfs.git branch: vfs-7.5.directory
[1/1] kernfs: allocate inodes and dentry outside kernfs_rwsem
https://git.kernel.org/vfs/vfs/c/5c1ca763748b