kernfs: remove kernfs_rwsem from dentry revalidation
Subsystems (3)
cover kernfs: remove kernfs_rwsem from dentry revalidation
At Meta, we are seeing important system daemons that poll cgroupfs and sysfs geth stuck in kernfs_dop_revalidate() for minutes. The two that hurt most are the ones we can least afford to lose: oomd, which decides what to kill when a machine runs out of memory, and below[1], which records the telemetry used to understand what happened afterwards.
kernfs_dop_revalidate() takes kernfs_rwsem for read once per path component of every walk into a kernfs mount. Linux rwsems do not permit reader lock stealing once a writer is queued, so one writer -- a cgroup created or destroyed, a device renamed -- parks the entire incoming reader stream in uninterruptible sleep. Daemons polling cgroup files in a loop are exactly the workload that turns this into a convoy, and cgroup churn is exactly what a busy machine does.
Nothing the callback reads needs the semaphore. kn->active is an atomic_t already tested lock-free elsewhere, kn->__parent and kn->name are RCU pointers, kn->ns can be compared rather than dereferenced, and parent->dir.rev is a plain counter.
1/4 uses the parent inode and name the VFS already passes to
->d_revalidate() rather than recovering them from mutable dentry fields, comparing the name by explicit length
2/4 annotates the directory revision counter for lockless access 3/4 compares namespace tags by pointer
4/4 removes kernfs_rwsem from the callback
LOOKUP_RCU still returns -ECHILD. kernfs_iop_permission() forces every walk out of RCU-walk before children are revalidated, so lifting it here would have no effect until that path is fixed; left to a separate series.
Readers walking cgroupfs and sysfs while another thread churns cgroups, renames netdevs and adds/removes devices, on an 8-CPU VM:
kernfs_rwsem read contentions revalidate among
acquisitions top call sites
before 48,593,360 1,744,517 #1 and #2
after 1,280,280 429,846 absentReader path-walk throughput improved 35-53% over the same workload.
Tested against an unpatched control of the same tree, built and booted with KASAN, KCSAN (default and STRICT), PROVE_LOCKING, PROVE_RCU, DEBUG_ATOMIC_SLEEP and LOCK_STAT. The deactivated, renamed and namespace-moved reject paths and negative-dentry invalidation all behave as before. KCSAN_STRICT over 180s reports no data race involving kernfs_dop_revalidate() or any field it reads, and there are no KASAN, lockdep or might-sleep reports across millions of concurrent path walks. The only kernfs KCSAN reports are in kernfs_refresh_inode(), present identically on the control and addressed separately.
Shakeel Butt (4):
kernfs: Use VFS lookup context in d_revalidate()
kernfs: Prepare directory revisions for lockless reads
kernfs: Avoid namespace dereference in d_revalidate()
kernfs: Remove kernfs_rwsem from dentry revalidation
fs/kernfs/dir.c | 76 +++++++++++++++---------------------- fs/kernfs/kernfs-internal.h | 9 ++--- 2 files changed, 35 insertions(+), 50 deletions(-)
base-commit: 7079a12d7506b07fb53b54a664bfad5fa9b16d70
Signature · 1 line
2.53.0-Meta
PATCH 1/4 kernfs: Use VFS lookup context in d_revalidate()
The VFS supplies a stable parent inode and expected name to the revalidation callback. Use them instead of recovering the same information from mutable dentry fields.
Compare the name using its explicit length because it may point into the pathname and need not be terminated at name->len. This also prepares the callback for lockless operation.
--- fs/kernfs/dir.c | 39 +++++++++++++++++---------------------- 1 file changed, 17 insertions(+), 22 deletions(-)
fs/kernfs/dir.c +17 -22
@@ -1171,23 +1171,19 @@ struct kernfs_node *kernfs_create_empty_dir(struct kernfs_node *parent,static int kernfs_dop_revalidate (struct inode *dir,const struct qstr *name,struct dentry *dentry,unsigned int flags) { -struct kernfs_node *kn, *parent; +struct kernfs_node *kn, *kn_parent; +struct kernfs_node *parent = dir->i_private;struct kernfs_root *root; +const char *kn_name;if (flags &LOOKUP_RCU )return -ECHILD ;/* Negative hashed dentry? */ if (d_really_is_negative (dentry)) { -/* If the kernfs parent node has changed discard and -* proceed to ->lookup. -* -* There's nothing special needed here when getting the -* dentry parent, even if a concurrent rename is in -* progress. That's because the dentry is negative so -* it can only be the target of the rename and it will -* be doing a d_move() not a replace. Consequently the -* dentry d_parent won't change over the d_move(). +/* +* If the kernfs parent node has changed discard and proceed to +* ->lookup. * * Also kernfs negative dentries transitioning from * negative to positive during revalidate won't happen
@@ -1195,14 +1191,11 @@ static int kernfs_dop_revalidate(struct inode *dir, const struct qstr *name,* changes and the lookup re-done so that a new positive * dentry can be properly created. */ - root =kernfs_root_from_sb (dentry->d_sb); + root =kernfs_root (parent);down_read (&root->kernfs_rwsem); - parent =kernfs_dentry_node (dentry->d_parent); -if (parent) { -if (kernfs_dir_changed (parent, dentry)) { -up_read (&root->kernfs_rwsem); -return 0 ; - } +if (kernfs_dir_changed (parent, dentry)) { +up_read (&root->kernfs_rwsem); +return 0 ; }up_read (&root->kernfs_rwsem);
@@ -1220,18 +1213,20 @@ static int kernfs_dop_revalidate(struct inode *dir, const struct qstr *name,if (!kernfs_active (kn)) goto out_bad; - parent =kernfs_parent (kn); + kn_parent =kernfs_parent (kn);/* The kernfs node has been moved? */ -if (kernfs_dentry_node (dentry->d_parent) != parent) +if (parent != kn_parent) goto out_bad;/* The kernfs node has been renamed */ -if (strcmp (dentry->d_name.name,kernfs_rcu_name (kn)) !=0 ) + kn_name =kernfs_rcu_name (kn); +if (name->len !=strlen (kn_name) || +memcmp (name->name, kn_name, name->len)) goto out_bad;/* The kernfs node has been moved to a different namespace */ -if (parent &&kernfs_ns_enabled (parent) && -kernfs_ns_id (kernfs_info (dentry->d_sb)->ns) !=kernfs_ns_id (kn->ns)) +if (kn_parent &&kernfs_ns_enabled (kn_parent) && +kernfs_ns_id (kernfs_info (dir->i_sb)->ns) !=kernfs_ns_id (kn->ns)) goto out_bad;up_read (&root->kernfs_rwsem);
Signature · 1 line
2.53.0-Meta
PATCH 2/4 kernfs: Prepare directory revisions for lockless reads
In reply to Shakeel ButtNegative dentry revalidation only needs to sample the parent directory generation and compare it with the value recorded at lookup time. Annotate those accesses with READ_ONCE() and WRITE_ONCE() so the comparison can safely move outside kernfs_rwsem.
Revision updates remain serialized by kernfs_rwsem. Assert that writer contract in kernfs_inc_rev(); the read half of the increment stays plain because the semaphore excludes other writers.
--- fs/kernfs/kernfs-internal.h | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-)
fs/kernfs/kernfs-internal.h +4 -5
@@ -147,20 +147,19 @@ static inline struct kernfs_node *kernfs_dentry_node(struct dentry *dentry)static inline void kernfs_set_rev (struct kernfs_node *parent,struct dentry *dentry) { - dentry->d_time = parent->dir.rev; +WRITE_ONCE (dentry->d_time,READ_ONCE (parent->dir.rev)); }static inline void kernfs_inc_rev (struct kernfs_node *parent) { - parent->dir.rev++; +lockdep_assert_held_write (&parent->dir.root->kernfs_rwsem); +WRITE_ONCE (parent->dir.rev, parent->dir.rev +1 ); }static inline bool kernfs_dir_changed (struct kernfs_node *parent,struct dentry *dentry) { -if (parent->dir.rev != dentry->d_time) -return true; -return false; +return READ_ONCE (parent->dir.rev) !=READ_ONCE (dentry->d_time); }extern const struct super_operations kernfs_sops;
Signature · 1 line
2.53.0-Meta
PATCH 3/4 kernfs: Avoid namespace dereference in d_revalidate()
In reply to Shakeel ButtCommit 1fe989e1c42a ("kernfs: use namespace id instead of pointer for hashing and comparison") changed dentry revalidation to compare namespace IDs along with the comparisons that determine visible directory ordering.
Dereferencing a namespace tag that kernfs_rename_ns() can replace is not suitable once dentry revalidation stops taking kernfs_rwsem. Use pointer equality for this non-user-visible equality check instead. Namespace IDs uniquely identify namespace objects, so pointer and ID equality cannot disagree for valid tags. Hashing and directory ordering continue to use IDs.
kn->ns becomes a lockless read in the next commit, so mark both sides of it now. The read is in kernfs_dop_revalidate(); the stores that can run while the node is visible are the two in kernfs_rename_ns(). The remaining stores, in kernfs_create_dir_ns(), kernfs_create_empty_dir() and kernfs_create_link(), all precede kernfs_add_one() and need no marking.
--- fs/kernfs/dir.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
fs/kernfs/dir.c +3 -3
@@ -1226,7 +1226,7 @@ static int kernfs_dop_revalidate(struct inode *dir, const struct qstr *name,/* The kernfs node has been moved to a different namespace */ if (kn_parent &&kernfs_ns_enabled (kn_parent) && -kernfs_ns_id (kernfs_info (dir->i_sb)->ns) !=kernfs_ns_id (kn->ns)) +kernfs_info (dir->i_sb)->ns !=READ_ONCE (kn->ns)) goto out_bad;up_read (&root->kernfs_rwsem);
@@ -1873,7 +1873,7 @@ int kernfs_rename_ns(struct kernfs_node *kn, struct kernfs_node *new_parent,rcu_assign_pointer (kn->__parent, new_parent); - kn->ns = new_ns; +WRITE_ONCE (kn->ns, new_ns);if (new_name)rcu_assign_pointer (kn->name, new_name);
@@ -1881,7 +1881,7 @@ int kernfs_rename_ns(struct kernfs_node *kn, struct kernfs_node *new_parent,kernfs_put (old_parent); }else {/* name assignment is RCU protected, parent is the same */ - kn->ns = new_ns; +WRITE_ONCE (kn->ns, new_ns);if (new_name)rcu_assign_pointer (kn->name, new_name); }
Signature · 1 line
2.53.0-Meta
PATCH 4/4 kernfs: Remove kernfs_rwsem from dentry revalidation
In reply to Shakeel Buttkernfs_dop_revalidate() takes kernfs_rwsem for read once per path component of every walk into a kernfs mount. Linux rwsems do not permit reader lock stealing once a writer is queued, so a single writer parks the whole incoming reader stream in uninterruptible sleep, stalling cgroup-polling daemons for minutes.
Nothing the callback reads requires the semaphore. kn->active is an atomic_t that kernfs_find_and_get_node_by_id() already tests through __kernfs_active(); kn->__parent and kn->name are RCU pointers whose old values are freed only after a grace period; kn->ns is now compared rather than dereferenced; parent->dir.rev was annotated earlier in this series.
What the semaphore does provide is a coherent snapshot, and that is not needed. ->d_revalidate() answers a question about a single instant, and the answer is already stale when it returns: a rename landing just after up_read() gives the same outcome as one observed mid-read. A lockless reader can only return "valid" for the (parent, name, namespace) triple identifying the dentry it was handed, and that triple was true when the dentry was instantiated, so it reports a genuine past state exactly as the locked version did. Removal is backstopped by kernfs_get_active() failing in the subsequent open().
Take an RCU read lock instead. kernfs_parent() and kernfs_rcu_name() work
unchanged: the condition in their rcu_dereference_check() is an alternative to holding the RCU read lock, not an extra requirement. The negative dentry path needs nothing, as @dir pins the parent. The namespace check can use @parent directly once the preceding check establishes it equals kernfs_parent(kn), so the kn_parent local and its NULL test go away.
kernfs_ns_enabled() reads @parent->flags, which KERNFS_ACTIVATED and KERNFS_REMOVING update as a plain read-modify-write under kernfs_rwsem. Those bits are not read here and KERNFS_NS cannot change once the directory has children, so mark the read data_race() rather than READ_ONCE(), which would not silence KCSAN against the unmarked writers anyway.
kernfs_iop_permission() still forces every walk out of RCU-walk before children are revalidated, so lifting the LOOKUP_RCU bail here would have no observable effect; it is left to the series fixing that path.
Readers walking cgroupfs and sysfs against concurrent cgroup and netdev
churn: kernfs_rwsem read acquisitions drop from 48,593,360 to 1,280,280, and kernfs_dop_revalidate() no longer appears among its contention sites.
--- fs/kernfs/dir.c | 49 ++++++++++++++++++++----------------------------- 1 file changed, 20 insertions(+), 29 deletions(-)
fs/kernfs/dir.c +20 -29
@@ -1171,9 +1171,8 @@ struct kernfs_node *kernfs_create_empty_dir(struct kernfs_node *parent,static int kernfs_dop_revalidate (struct inode *dir,const struct qstr *name,struct dentry *dentry,unsigned int flags) { -struct kernfs_node *kn, *kn_parent;struct kernfs_node *parent = dir->i_private; -struct kernfs_root *root; +struct kernfs_node *kn;const char *kn_name;if (flags &LOOKUP_RCU )
@@ -1191,49 +1190,41 @@ static int kernfs_dop_revalidate(struct inode *dir, const struct qstr *name,* changes and the lookup re-done so that a new positive * dentry can be properly created. */ - root =kernfs_root (parent); -down_read (&root->kernfs_rwsem); -if (kernfs_dir_changed (parent, dentry)) { -up_read (&root->kernfs_rwsem); -return 0 ; - } -up_read (&root->kernfs_rwsem); - -/* The kernfs parent node hasn't changed, leave the -* dentry negative and return success. -*/ -return 1 ; +return !kernfs_dir_changed (parent, dentry); } kn =kernfs_dentry_node (dentry); - root =kernfs_root (kn); -down_read (&root->kernfs_rwsem); + +guard (rcu)();/* The kernfs node has been deactivated */ -if (!kernfs_active (kn)) - goto out_bad; +if (!__kernfs_active (kn)) +return 0 ; - kn_parent =kernfs_parent (kn);/* The kernfs node has been moved? */ -if (parent != kn_parent) - goto out_bad; +if (kernfs_parent (kn) != parent) +return 0 ;/* The kernfs node has been renamed */ kn_name =kernfs_rcu_name (kn);if (name->len !=strlen (kn_name) ||memcmp (name->name, kn_name, name->len)) - goto out_bad; +return 0 ; -/* The kernfs node has been moved to a different namespace */ -if (kn_parent &&kernfs_ns_enabled (kn_parent) && +/* +* The kernfs node has been moved to a different namespace. +* +* KERNFS_NS is set by kernfs_enable_ns() while @parent still has no +* children, so it cannot change while a child of @parent is being +* revalidated. The other bits in that word, KERNFS_ACTIVATED and +* KERNFS_REMOVING, are updated under kernfs_rwsem and are not read +* here, so racing with them is intentional and harmless. +*/ +if (data_race (kernfs_ns_enabled (parent)) &&kernfs_info (dir->i_sb)->ns !=READ_ONCE (kn->ns)) - goto out_bad; +return 0 ; -up_read (&root->kernfs_rwsem);return 1 ; -out_bad: -up_read (&root->kernfs_rwsem); -return 0 ; }const struct dentry_operations kernfs_dops = {
Signature · 1 line
2.53.0-Meta
Re: [PATCH 4/4] kernfs: Remove kernfs_rwsem from dentry revalidation
Hi Shakeel and Christian,
Quoted 85 lines · Shakeel Butt
kernfs_dop_revalidate()takeskernfs_rwsemfor read once per path component of every walk into a kernfs mount. Linux rwsems do not permit reader lock stealing once a writer is queued, so a single writer parks the whole incoming reader stream in uninterruptible sleep, stalling cgroup-polling daemons for minutes.Nothing the callback reads requires the semaphore.
kn->activeis anatomic_tthatkernfs_find_and_get_node_by_id()already tests through__kernfs_active();kn->__parentandkn->nameare RCU pointers whose old values are freed only after a grace period;kn->nsis now compared rather than dereferenced;parent->dir.revwas annotated earlier in this series.What the semaphore does provide is a coherent snapshot, and that is not needed. ->d_revalidate() answers a question about a single instant, and the answer is already stale when it returns: a rename landing just after
up_read()gives the same outcome as one observed mid-read. A lockless reader can only return "valid" for the (parent, name, namespace) triple identifying the dentry it was handed, and that triple was true when the dentry was instantiated, so it reports a genuine past state exactly as the locked version did. Removal is backstopped bykernfs_get_active()failing in the subsequentopen().Take an RCU read lock instead.
kernfs_parent()andkernfs_rcu_name()work
unchanged: the condition in theirrcu_dereference_check()is an alternative to holding the RCU read lock, not an extra requirement. The negative dentry path needs nothing, as @dir pins the parent. The namespace check can use @parent directly once the preceding check establishes it equalskernfs_parent(kn), so thekn_parentlocal and its NULL test go away.
kernfs_ns_enabled()reads @parent->flags, whichKERNFS_ACTIVATEDandKERNFS_REMOVINGupdate as a plain read-modify-write underkernfs_rwsem. Those bits are not read here andKERNFS_NScannot change once the directory has children, so mark the readdata_race()rather thanREAD_ONCE(), which would not silence KCSAN against the unmarked writers anyway.
kernfs_iop_permission()still forces every walk out of RCU-walk before children are revalidated, so lifting theLOOKUP_RCUbail here would have no observable effect; it is left to the series fixing that path.Readers walking cgroupfs and sysfs against concurrent cgroup and netdev
churn:kernfs_rwsemread acquisitions drop from 48,593,360 to 1,280,280, andkernfs_dop_revalidate()no longer appears among its contention sites.Signed-off-by Shakeel Butt <[email protected]>--- fs/kernfs/dir.c | 49 ++++++++++++++++++++----------------------------- 1 file changed, 20 insertions(+), 29 deletions(-)fs/kernfs/dir.c +2
-15@@ -1171,9 +1171,8 @@ struct kernfs_node *kernfs_create_empty_dir(struct kernfs_node *parent,static int kernfs_dop_revalidate (struct inode *dir,const struct qstr *name,struct dentry *dentry,unsigned int flags) { -struct kernfs_node *kn, *kn_parent;struct kernfs_node *parent = dir->i_private; -struct kernfs_root *root; +struct kernfs_node *kn;const char *kn_name;if (flags &LOOKUP_RCU )@@ -1191,49 +1190,41 @@ static int kernfs_dop_revalidate(struct inode *dir, const struct qstr *name,* changes and the lookup re-done so that a new positive * dentry can be properly created. */ - root =kernfs_root (parent); -down_read (&root->kernfs_rwsem); -if (kernfs_dir_changed (parent, dentry)) { -up_read (&root->kernfs_rwsem); -return 0 ; - } -up_read (&root->kernfs_rwsem); - -/* The kernfs parent node hasn't changed, leave the -* dentry negative and return success. -*/ -return 1 ; +return !kernfs_dir_changed (parent, dentry); } kn =kernfs_dentry_node (dentry); - root =kernfs_root (kn);
fs/kernfs/dir.c static int kernfs_dop_revalidate(struct inode *dir, const struct qstr *name, · line 1203
Quoted 10 lines
fs/kernfs/dir.c +4
-3-down_read (&root->kernfs_rwsem); + +guard (rcu)();/* The kernfs node has been deactivated */ -if (!kernfs_active (kn)) - goto out_bad; +if (!__kernfs_active (kn)) +return 0 ;
I have couple of reports on android kernel based on 6.12 with callstack
[ 41.137176][ T5373] pc : kernfs_dop_revalidate+0x28/0x108 [ 41.137180][ T5373] lr : path_openat+0x7ec/0x1b04 [ 41.137183][ T5373] sp : ffffffc0b02fbba0 [ 41.137184][ T5373] x29: ffffffc0b02fbba0 x28: ffffffc0b02fbda4 x27: ffffff8917f90000 [ 41.137187][ T5373] x26: 0000000000000030 x25: ffffff8918985860 x24: 0000000000028000 [ 41.137189][ T5373] x23: ffffff8918985040 x22: 0000000000000040 x21: 0000000000000000 [ 41.137192][ T5373] x20: ffffff8918985040 x19: ffffff894321d000 x18: ffffffdc3adec8c0 [ 41.137194][ T5373] x17: 000000004495e485 x16: 000000004495e485 x15: ffffffffff000000 [ 41.137197][ T5373] x14: 7d6770f3f4cee991 x13: 0000000000737365 x12: 0000000000737365 [ 41.137199][ T5373] x11: ffffff8918985088 x10: 0000000000000018 x9 : 0000000100000000 [ 41.137202][ T5373] x8 : ffffff89dc6c3d30 x7 : 705f636f6c6c6100 x6 : 0080808080808080 [ 41.137204][ T5373] x5 : ffffffc0b02fbad0 x4 : 0000000000000000 x3 : 0000000000000000 [ 41.137207][ T5373] x2 : 0000000000000000 x1 : 0000000000000100 x0 : ffffff8918985040 [ 41.137210][ T5373] Call trace: [ 41.137211][ T5373] kernfs_dop_revalidate+0x28/0x108 [ 41.137213][ T5373] path_openat+0x7ec/0x1b04 [ 41.137215][ T5373] do_filp_open+0xac/0x130 [ 41.137216][ T5373] do_sys_openat2+0x140/0x21c [ 41.137219][ T5373] __arm64_sys_openat+0x70/0x9c [ 41.137222][ T5373] invoke_syscall+0x58/0xf0 [ 41.137224][ T5373] do_el0_svc+0x64/0xe0 [ 41.137226][ T5373] el0_svc+0x4c/0xdc [ 41.137228][ T5373] el0t_64_sync_handler+0x20/0xf4 [ 41.137230][ T5373] el0t_64_sync+0x1bc/0x1c0
After investigation it seems there is race where kernfs_dop_revalidate() crashed dereferencing kn->__parent in kernfs_root(kn) at offset 0x8).
Although your patch eliminates kernfs_root(kn), the underlying race where kn == NULL still exists in this patch and will now crash in __kernfs_active(kn) at offset 0x4.
Scenario seems to be during lockless VFS path lookup (lookup_fast()), cached dentries are revalidated without holding dentry locks:
1. lookup_fast() finds a cached positive dentry where dentry->d_inode != NULL.
2. In kernfs_dop_revalidate(), d_really_is_negative(dentry)
evaluates to false, so it skips the negative dentry branch.
3. Concurrently, slab memory reclaim under memory pressure evicts
unused dentries:
shrink_node() -> shrink_slab() -> super_cache_scan()
-> prune_dcache_sb() -> shrink_dentry_list()
-> dentry_unlink_inode()
-> __d_clear_type_and_inode(dentry)
-> dentry->d_inode = NULL; 4. kernfs_dop_revalidate() now runs:
kn = kernfs_dentry_node(dentry);
kernfs_dentry_node() tests if (d_really_is_negative(dentry)) return NULL;. Because dentry->d_inode was cleared to NULL bydentry_unlink_inode(), kernfs_dentry_node(dentry) returns NULL
While crafting fix for 6.12, I found this series reworking kernfs_dop_revalidate().
Just to be clear, your series is not introducing the problem.
But perhaps we can improve this with a check like below?
- /* The kernfs node has been deactivated */
- if (!__kernfs_active(kn))
+ /* The kernfs node is unlinked or deactivated */
+ if (!kn || !__kernfs_active(kn))
return 0;Let me know your thoughts or if you want a separate patch
Thanks,
Sandeep.
Re: [PATCH 4/4] kernfs: Remove kernfs_rwsem from dentry revalidation
Quoted 157 lines · Sandeep Dhavale
Hi Shakeel and Christian,
Quoted 85 lines · Shakeel Butt
kernfs_dop_revalidate()takeskernfs_rwsemfor read once per path component of every walk into a kernfs mount. Linux rwsems do not permit reader lock stealing once a writer is queued, so a single writer parks the whole incoming reader stream in uninterruptible sleep, stalling cgroup-polling daemons for minutes.Nothing the callback reads requires the semaphore.
kn->activeis anatomic_tthatkernfs_find_and_get_node_by_id()already tests through__kernfs_active();kn->__parentandkn->nameare RCU pointers whose old values are freed only after a grace period;kn->nsis now compared rather than dereferenced;parent->dir.revwas annotated earlier in this series.What the semaphore does provide is a coherent snapshot, and that is not needed. ->d_revalidate() answers a question about a single instant, and the answer is already stale when it returns: a rename landing just after
up_read()gives the same outcome as one observed mid-read. A lockless reader can only return "valid" for the (parent, name, namespace) triple identifying the dentry it was handed, and that triple was true when the dentry was instantiated, so it reports a genuine past state exactly as the locked version did. Removal is backstopped bykernfs_get_active()failing in the subsequentopen().Take an RCU read lock instead.
kernfs_parent()andkernfs_rcu_name()work
unchanged: the condition in theirrcu_dereference_check()is an alternative to holding the RCU read lock, not an extra requirement. The negative dentry path needs nothing, as @dir pins the parent. The namespace check can use @parent directly once the preceding check establishes it equalskernfs_parent(kn), so thekn_parentlocal and its NULL test go away.
kernfs_ns_enabled()reads @parent->flags, whichKERNFS_ACTIVATEDandKERNFS_REMOVINGupdate as a plain read-modify-write underkernfs_rwsem. Those bits are not read here andKERNFS_NScannot change once the directory has children, so mark the readdata_race()rather thanREAD_ONCE(), which would not silence KCSAN against the unmarked writers anyway.
kernfs_iop_permission()still forces every walk out of RCU-walk before children are revalidated, so lifting theLOOKUP_RCUbail here would have no observable effect; it is left to the series fixing that path.Readers walking cgroupfs and sysfs against concurrent cgroup and netdev
churn:kernfs_rwsemread acquisitions drop from 48,593,360 to 1,280,280, andkernfs_dop_revalidate()no longer appears among its contention sites.Signed-off-by Shakeel Butt <[email protected]>--- fs/kernfs/dir.c | 49 ++++++++++++++++++++----------------------------- 1 file changed, 20 insertions(+), 29 deletions(-)fs/kernfs/dir.c +2
-15@@ -1171,9 +1171,8 @@ struct kernfs_node *kernfs_create_empty_dir(struct kernfs_node *parent,static int kernfs_dop_revalidate (struct inode *dir,const struct qstr *name,struct dentry *dentry,unsigned int flags) { -struct kernfs_node *kn, *kn_parent;struct kernfs_node *parent = dir->i_private; -struct kernfs_root *root; +struct kernfs_node *kn;const char *kn_name;if (flags &LOOKUP_RCU )@@ -1191,49 +1190,41 @@ static int kernfs_dop_revalidate(struct inode *dir, const struct qstr *name,* changes and the lookup re-done so that a new positive * dentry can be properly created. */ - root =kernfs_root (parent); -down_read (&root->kernfs_rwsem); -if (kernfs_dir_changed (parent, dentry)) { -up_read (&root->kernfs_rwsem); -return 0 ; - } -up_read (&root->kernfs_rwsem); - -/* The kernfs parent node hasn't changed, leave the -* dentry negative and return success. -*/ -return 1 ; +return !kernfs_dir_changed (parent, dentry); } kn =kernfs_dentry_node (dentry); - root =kernfs_root (kn);fs/kernfs/dir.c static int kernfs_dop_revalidate(struct inode *dir, const struct qstr *name, · line 1203
Quoted 10 lines
fs/kernfs/dir.c +4
-3-down_read (&root->kernfs_rwsem); + +guard (rcu)();/* The kernfs node has been deactivated */ -if (!kernfs_active (kn)) - goto out_bad; +if (!__kernfs_active (kn)) +return 0 ;I have couple of reports on android kernel based on 6.12 with callstack
[ 41.137176][ T5373] pc : kernfs_dop_revalidate+0x28/0x108 [ 41.137180][ T5373] lr : path_openat+0x7ec/0x1b04 [ 41.137183][ T5373] sp : ffffffc0b02fbba0 [ 41.137184][ T5373] x29: ffffffc0b02fbba0 x28: ffffffc0b02fbda4 x27: ffffff8917f90000 [ 41.137187][ T5373] x26: 0000000000000030 x25: ffffff8918985860 x24: 0000000000028000 [ 41.137189][ T5373] x23: ffffff8918985040 x22: 0000000000000040 x21: 0000000000000000 [ 41.137192][ T5373] x20: ffffff8918985040 x19: ffffff894321d000 x18: ffffffdc3adec8c0 [ 41.137194][ T5373] x17: 000000004495e485 x16: 000000004495e485 x15: ffffffffff000000 [ 41.137197][ T5373] x14: 7d6770f3f4cee991 x13: 0000000000737365 x12: 0000000000737365 [ 41.137199][ T5373] x11: ffffff8918985088 x10: 0000000000000018 x9 : 0000000100000000 [ 41.137202][ T5373] x8 : ffffff89dc6c3d30 x7 : 705f636f6c6c6100 x6 : 0080808080808080 [ 41.137204][ T5373] x5 : ffffffc0b02fbad0 x4 : 0000000000000000 x3 : 0000000000000000 [ 41.137207][ T5373] x2 : 0000000000000000 x1 : 0000000000000100 x0 : ffffff8918985040 [ 41.137210][ T5373] Call trace: [ 41.137211][ T5373] kernfs_dop_revalidate+0x28/0x108 [ 41.137213][ T5373] path_openat+0x7ec/0x1b04 [ 41.137215][ T5373] do_filp_open+0xac/0x130 [ 41.137216][ T5373] do_sys_openat2+0x140/0x21c [ 41.137219][ T5373] __arm64_sys_openat+0x70/0x9c [ 41.137222][ T5373] invoke_syscall+0x58/0xf0 [ 41.137224][ T5373] do_el0_svc+0x64/0xe0 [ 41.137226][ T5373] el0_svc+0x4c/0xdc [ 41.137228][ T5373] el0t_64_sync_handler+0x20/0xf4 [ 41.137230][ T5373] el0t_64_sync+0x1bc/0x1c0After investigation it seems there is race where
kernfs_dop_revalidate()crashed dereferencingkn->__parentinkernfs_root(kn)at offset0x8).
Although your patch eliminateskernfs_root(kn), the underlying race where kn == NULL still exists in this patch and will now crash in__kernfs_active(kn)at offset0x4.Scenario seems to be during lockless VFS path lookup (
lookup_fast()), cached dentries are revalidated without holding dentry locks:1.
lookup_fast()finds a cached positive dentry wheredentry->d_inode!= NULL.
2. Inkernfs_dop_revalidate(),d_really_is_negative(dentry)
evaluates to false, so it skips the negative dentry branch.
3. Concurrently, slab memory reclaim under memory pressure evicts
unused dentries:shrink_node() -> shrink_slab() -> super_cache_scan() -> prune_dcache_sb() -> shrink_dentry_list() -> dentry_unlink_inode() -> __d_clear_type_and_inode(dentry) -> dentry->d_inode = NULL;
Quoted 5 lines
4.
kernfs_dop_revalidate()now runs:kn = kernfs_dentry_node(dentry);
kernfs_dentry_node()tests if (d_really_is_negative(dentry)) return NULL;. Becausedentry->d_inodewas cleared to NULL bydentry_unlink_inode(),kernfs_dentry_node(dentry)returns NULL
Hi Sandeep, thanks for the report. Do you have a reproducer? If yes, can you reproduce with this series?
Quoted 7 lines
While crafting fix for 6.12, I found this series reworking
kernfs_dop_revalidate().
Just to be clear, your series is not introducing the problem.But perhaps we can improve this with a check like below?
Quoted 7 lines
- /* The kernfs node has been deactivated */ - if (!__kernfs_active(kn)) + /* The kernfs node is unlinked or deactivated */ + if (!kn || !__kernfs_active(kn)) return 0;Let me know your thoughts or if you want a separate patch
I think first we need a real reproducer before considering anything. I would not add a check unless we have proof that it is really needed.
Re: [PATCH 4/4] kernfs: Remove kernfs_rwsem from dentry revalidation
Quoted 3 lines
Hi Sandeep, thanks for the report. Do you have a reproducer? If yes, can you reproduce with this series?
Hi Shakeel,
No luck with the reproducer yet, and reports have been sporadic.
Quoted 2 lines
I think first we need a real reproducer before considering anything. I would not add a check unless we have proof that it is really needed.
Ok, I understand. Do you have any opinion on the suspected call chain which makes dentry->d_inode = NULL; ?
Partner report said it happens under memory pressure, and userspace component was reading the CMA stats through
sysfs [1]. So I naturally looked at the shrinker and suspected that call chain, which made sense for my 6.12 kernel call stack.
Thanks,
Sandeep.
[1] https://cs.android.com/android/platform/superproject/+/android-latest-release:system/memory/libmeminfo/sysmeminfo.cpp;l=545?q=sysme&ss=android%2Fplatform%2Fsuperproject
Re: [PATCH 4/4] kernfs: Remove kernfs_rwsem from dentry revalidation
Quoted 11 lines · Sandeep Dhavale
Quoted 3 lines
Hi Sandeep, thanks for the report. Do you have a reproducer? If yes, can you reproduce with this series?
Hi Shakeel,
No luck with the reproducer yet, and reports have been sporadic.Quoted 2 lines
I think first we need a real reproducer before considering anything. I would not add a check unless we have proof that it is really needed.
Ok, I understand. Do you have any opinion on the suspected call chain which makes
dentry->d_inode= NULL; ?
I don't see how reclaim can set dentry->d_inode = NULL for a dentry with elevated dentry->d_lockref.count because kernfs_dop_revalidate gets the dentry whose dentry->d_lockref.count is elevated. Reclaim specifically skips such dentries.
Re: [PATCH 4/4] kernfs: Remove kernfs_rwsem from dentry revalidation
Hi Shakeel,
Quoted 4 lines
I don't see how reclaim can set
dentry->d_inode= NULL for a dentry with elevateddentry->d_lockref.countbecausekernfs_dop_revalidategets the dentry whosedentry->d_lockref.countis elevated. Reclaim specifically skips such dentries.
Thank you for your input! That may explain why I am unable to reproduce led by memory stress narrative. Let me attempt to reproduce this first and I will provide an update.
Thanks again,
Sandeep.
Re: [PATCH 4/4] kernfs: Remove kernfs_rwsem from dentry revalidation
Quoted 8 lines · Sandeep Dhavale
Hi Shakeel,
Quoted 4 lines
I don't see how reclaim can set
dentry->d_inode= NULL for a dentry with elevateddentry->d_lockref.countbecausekernfs_dop_revalidategets the dentry whosedentry->d_lockref.countis elevated. Reclaim specifically skips such dentries.Thank you for your input! That may explain why I am unable to reproduce led by memory stress narrative. Let me attempt to reproduce this first and I will provide an update.
It looks possible the dentry isn't negative, instead it might be possible
it's a use after free of the inode. There are functions that are called
directly by file systems that use kernfs.
I always thought that was the reason for the lock, not so much a need to
synchronise with the VFS (eg. in ref-walk mode it's likely the inode read
lock is held).
I wonder if there's a concurrent remove going on when this happens?
You could enable debug logging and see if you can see any node removal log
messages. The logging isn't very chatty so it might be difficult to get
further context from it but it could provide a lead to follow up on.
Ian
Christian Brauner
AppliedQuoted 9 lines · Shakeel Butt
kernfs: remove
kernfs_rwsemfrom dentry revalidationAt Meta, we are seeing important system daemons that poll cgroupfs and sysfs geth stuck in
kernfs_dop_revalidate()for minutes. The two that hurt most are the ones we can least afford to lose: oomd, which decides what to kill when a machine runs out of memory, and below[1], which records the telemetry used to understand what happened afterwards.[...]
Applied to the vfs-7.4.kernfs branch of the vfs/vfs.git tree. Patches in the vfs-7.4.kernfs branch should appear in linux-next soon.
Please report any outstanding bugs that were missed during review in a new review to the original patch series allowing us to drop it.
It's encouraged to provide Acked-bys and Reviewed-bys even though the patch has now been applied. If possible patch trailers will be updated.
Note that commit hashes shown below are subject to change due to rebase, trailer updates or similar. If in doubt, please check the listed branch.
tree: https://git.kernel.org/pub/scm/linux/kernel/git/vfs/vfs.git branch: vfs-7.4.kernfs
[1/4] kernfs: Use VFS lookup context in d_revalidate()
https://git.kernel.org/vfs/vfs/c/b3c97f5cb084 [2/4] kernfs: Prepare directory revisions for lockless reads https://git.kernel.org/vfs/vfs/c/272e0997df44 [3/4] kernfs: Avoid namespace dereference in d_revalidate() https://git.kernel.org/vfs/vfs/c/e1c629a8dbc7 [4/4] kernfs: Remove kernfs_rwsem from dentry revalidation https://git.kernel.org/vfs/vfs/c/8c6ad578e2f0
Ian Kent
Reviewed-byQuoted 10 lines · Christian Brauner
Quoted 9 lines · Shakeel Butt
kernfs: remove
kernfs_rwsemfrom dentry revalidationAt Meta, we are seeing important system daemons that poll cgroupfs and sysfs geth stuck in
kernfs_dop_revalidate()for minutes. The two that hurt most are the ones we can least afford to lose: oomd, which decides what to kill when a machine runs out of memory, and below[1], which records the telemetry used to understand what happened afterwards.[...]
Quoted 8 lines
Applied to the vfs-7.4.kernfs branch of the vfs/vfs.git tree. Patches in the vfs-7.4.kernfs branch should appear in linux-next soon.
Please report any outstanding bugs that were missed during review in a new review to the original patch series allowing us to drop it.
It's encouraged to provide Acked-bys and Reviewed-bys even though the patch has now been applied. If possible patch trailers will be updated.
I have a report against kernfs where I cannot find a reason for significant
contention, primarily on ->revaliate(), in either of two vmcores, exactly
the case described in this series.
I reviewed the series and it looks good to me.
The descriptions justifying the changes also look sound.
While I think renames and removals should be ok based on the descriptions
justifying the change (my biggest concern with removing the rwsem) I'll
continue to ponder its implications for a while.
Quoted 17 lines
Note that commit hashes shown below are subject to change due to rebase, trailer updates or similar. If in doubt, please check the listed branch.
tree: https://git.kernel.org/pub/scm/linux/kernel/git/vfs/vfs.git branch: vfs-7.4.kernfs[1/4] kernfs: Use VFS lookup context in
d_revalidate()https://git.kernel.org/vfs/vfs/c/b3c97f5cb084 [2/4] kernfs: Prepare directory revisions for lockless reads https://git.kernel.org/vfs/vfs/c/272e0997df44 [3/4] kernfs: Avoid namespace dereference in d_revalidate() https://git.kernel.org/vfs/vfs/c/e1c629a8dbc7 [4/4] kernfs: Remove kernfs_rwsem from dentry revalidation https://git.kernel.org/vfs/vfs/c/8c6ad578e2f0