Skip to content

kernfs: remove kernfs_rwsem from dentry revalidation

Shakeel Butt cgroups driver-core lkml 1 version Applied

Status from this message: Applied to the vfs-7.4.kernfs branch of the vfs/vfs.git tree.

Get series with b4
b4 am [email protected]

Versions newest first

  1. v1 Latest4 patches 8 repliesApplied

Patches in v1 Applied

#Subject and filesSizeRepliesApplied
1/4kernfs: Use VFS lookup context in d_revalidate()fs/kernfs/dir.c+17 -220 replies
2/4kernfs: Prepare directory revisions for lockless readsfs/kernfs/kernfs-internal.h+4 -50 replies
3/4kernfs: Avoid namespace dereference in d_revalidate()fs/kernfs/dir.c+3 -30 replies
4/4kernfs: Remove kernfs_rwsem from dentry revalidationfs/kernfs/dir.c+20 -296 replies

Reviews trailers per patch

None of the 4 patches has a Reviewed-by or Acked-by of their own; the series has one for all of them.

PatchIan Kent
SeriesR-b
1/4
2/4
3/4
4/4

Trailers given in replies are filled; those carried in the patches from earlier versions are outlined with dashes.

Read the v1 thread

Cover letter [PATCH 0/4] kernfs: remove kernfs_rwsem from dentry revalidation

At Meta, we are seeing important system daemons that poll cgroupfs and sysfs geth stuck in kernfs_dop_revalidate() for minutes. The two that hurt most are the ones we can least afford to lose: oomd, which decides what to kill when a machine runs out of memory, and below[1], which records the telemetry used to understand what happened afterwards.

kernfs_dop_revalidate() takes kernfs_rwsem for read once per path component of every walk into a kernfs mount. Linux rwsems do not permit reader lock stealing once a writer is queued, so one writer -- a cgroup created or destroyed, a device renamed -- parks the entire incoming reader stream in uninterruptible sleep. Daemons polling cgroup files in a loop are exactly the workload that turns this into a convoy, and cgroup churn is exactly what a busy machine does.

Nothing the callback reads needs the semaphore. kn->active is an atomic_t already tested lock-free elsewhere, kn->__parent and kn->name are RCU pointers, kn->ns can be compared rather than dereferenced, and parent->dir.rev is a plain counter.

1/4 uses the parent inode and name the VFS already passes to
->d_revalidate() rather than recovering them from mutable dentry fields, comparing the name by explicit length
2/4 annotates the directory revision counter for lockless access 3/4 compares namespace tags by pointer
4/4 removes kernfs_rwsem from the callback

LOOKUP_RCU still returns -ECHILD. kernfs_iop_permission() forces every walk out of RCU-walk before children are revalidated, so lifting it here would have no effect until that path is fixed; left to a separate series.

Readers walking cgroupfs and sysfs while another thread churns cgroups, renames netdevs and adds/removes devices, on an 8-CPU VM:

                    kernfs_rwsem read   contentions   revalidate among
                        acquisitions                  top call sites
  before                  48,593,360     1,744,517    #1 and #2
  after                    1,280,280       429,846    absent

Reader path-walk throughput improved 35-53% over the same workload.

Tested against an unpatched control of the same tree, built and booted with KASAN, KCSAN (default and STRICT), PROVE_LOCKING, PROVE_RCU, DEBUG_ATOMIC_SLEEP and LOCK_STAT. The deactivated, renamed and namespace-moved reject paths and negative-dentry invalidation all behave as before. KCSAN_STRICT over 180s reports no data race involving kernfs_dop_revalidate() or any field it reads, and there are no KASAN, lockdep or might-sleep reports across millions of concurrent path walks. The only kernfs KCSAN reports are in kernfs_refresh_inode(), present identically on the control and addressed separately.

Shakeel Butt (4):
kernfs: Use VFS lookup context in d_revalidate()
kernfs: Prepare directory revisions for lockless reads
kernfs: Avoid namespace dereference in d_revalidate()
kernfs: Remove kernfs_rwsem from dentry revalidation

 fs/kernfs/dir.c             | 76 +++++++++++++++----------------------
 fs/kernfs/kernfs-internal.h |  9 ++---
 2 files changed, 35 insertions(+), 50 deletions(-)

base-commit: 7079a12d7506b07fb53b54a664bfad5fa9b16d70

Signature · 1 line
2.53.0-Meta