kernfs: remove kernfs_rwsem from dentry revalidation
Status from this message: Applied to the vfs-7.4.kernfs branch of the vfs/vfs.git tree.
Get series with b4
b4 am [email protected]Versions newest first
Patches in v1 Applied
| # | Subject and files | Size | Replies | Applied |
|---|---|---|---|---|
| 1/4 | kernfs: Use VFS lookup context in d_revalidate()fs/ | +17 | 0 replies | |
| 2/4 | kernfs: Prepare directory revisions for lockless readsfs/ | +4 | 0 replies | |
| 3/4 | kernfs: Avoid namespace dereference in d_revalidate()fs/ | +3 | 0 replies | |
| 4/4 | kernfs: Remove kernfs_rwsem from dentry revalidationfs/ | +20 | 6 replies |
Reviews trailers per patch
None of the 4 patches has a Reviewed-by or Acked-by of their own; the series has one for all of them.
| Patch | Ian Kent |
|---|---|
| Series | R-b |
| 1/4 | |
| 2/4 | |
| 3/4 | |
| 4/4 |
Trailers given in replies are filled; those carried in the patches from earlier versions are outlined with dashes.
Cover letter [PATCH 0/4] kernfs: remove kernfs_rwsem from dentry revalidation
At Meta, we are seeing important system daemons that poll cgroupfs and sysfs geth stuck in kernfs_dop_revalidate() for minutes. The two that hurt most are the ones we can least afford to lose: oomd, which decides what to kill when a machine runs out of memory, and below[1], which records the telemetry used to understand what happened afterwards.
kernfs_dop_revalidate() takes kernfs_rwsem for read once per path component of every walk into a kernfs mount. Linux rwsems do not permit reader lock stealing once a writer is queued, so one writer -- a cgroup created or destroyed, a device renamed -- parks the entire incoming reader stream in uninterruptible sleep. Daemons polling cgroup files in a loop are exactly the workload that turns this into a convoy, and cgroup churn is exactly what a busy machine does.
Nothing the callback reads needs the semaphore. kn->active is an atomic_t already tested lock-free elsewhere, kn->__parent and kn->name are RCU pointers, kn->ns can be compared rather than dereferenced, and parent->dir.rev is a plain counter.
1/4 uses the parent inode and name the VFS already passes to
->d_revalidate() rather than recovering them from mutable dentry fields, comparing the name by explicit length
2/4 annotates the directory revision counter for lockless access 3/4 compares namespace tags by pointer
4/4 removes kernfs_rwsem from the callback
LOOKUP_RCU still returns -ECHILD. kernfs_iop_permission() forces every walk out of RCU-walk before children are revalidated, so lifting it here would have no effect until that path is fixed; left to a separate series.
Readers walking cgroupfs and sysfs while another thread churns cgroups, renames netdevs and adds/removes devices, on an 8-CPU VM:
kernfs_rwsem read contentions revalidate among
acquisitions top call sites
before 48,593,360 1,744,517 #1 and #2
after 1,280,280 429,846 absentReader path-walk throughput improved 35-53% over the same workload.
Tested against an unpatched control of the same tree, built and booted with KASAN, KCSAN (default and STRICT), PROVE_LOCKING, PROVE_RCU, DEBUG_ATOMIC_SLEEP and LOCK_STAT. The deactivated, renamed and namespace-moved reject paths and negative-dentry invalidation all behave as before. KCSAN_STRICT over 180s reports no data race involving kernfs_dop_revalidate() or any field it reads, and there are no KASAN, lockdep or might-sleep reports across millions of concurrent path walks. The only kernfs KCSAN reports are in kernfs_refresh_inode(), present identically on the control and addressed separately.
Shakeel Butt (4):
kernfs: Use VFS lookup context in d_revalidate()
kernfs: Prepare directory revisions for lockless reads
kernfs: Avoid namespace dereference in d_revalidate()
kernfs: Remove kernfs_rwsem from dentry revalidation
fs/kernfs/dir.c | 76 +++++++++++++++---------------------- fs/kernfs/kernfs-internal.h | 9 ++--- 2 files changed, 35 insertions(+), 50 deletions(-)
base-commit: 7079a12d7506b07fb53b54a664bfad5fa9b16d70
Signature · 1 line
2.53.0-Meta