Skip to contentSkip to conversation

kernfs: stop taking kernfs_iattr_rwsem for writing in mkdir/rmdir

PATCH New Shakeel Butt 1 participant 7 messages Nov 18

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

cover kernfs: stop taking kernfs_iattr_rwsem for writing in mkdir/rmdir

In the Meta fleet, cgroup mkdir and rmdir can stall cgroup operations on the whole host. They hold cgroup_mutex and kernfs_rwsem while they wait for kernfs_iattr_rwsem, which every path walk takes for reading, and they take it once or twice for every file of the cgroup. systemd, clone3() into a cgroup, BPF queries and the OOM daemon all wait behind them.

- Our lock monitor saw a kernfs_rwsem writer blocked on kernfs_iattr_rwsem several thousand times on a large number of hosts in 29 days (in mkdir and rmdir), for up to 1.8 s at a time.

- Under memory and CPU pressure the waits become host-wide stalls. In one week, we observed several cases where the OOM daemon and the monitoring agent were stuck for up to 22 minutes. In October, one batch job caused these stalls on a large number of hosts. On the sampled hosts with current kernels, the cgroup_mutex owner was always waiting for kernfs_iattr_rwsem (in mkdir and rmdir). One specific stall lasted 90 minutes and ended in a reboot.

The write lock is not needed for most of what it covers:

- kernfs_rwsem already serialises the parent's two counters and KERNFS_REMOVING (patches 3 and 4).
- The parent's times get a seqlock of their own (patch 5).
- Only inode setup needs to set a directory's i_nlink, so removal can clear it without the lock (patches 2 and 6).

The lock also kept times and link counts stable while getattr and permission copied them into the shared inode. Patches 1 and 2 make getattr report them from the node and stop copying them, so stat() cannot return a stale or torn value.

After the series, mkdir and rmdir take no kernfs_iattr_rwsem for writing. Write acquisitions, from lock_stat:

before after

  5 dummy netdevs created and removed       1850       0
  100 cgroup mkdirs and 100 rmdirs          7500       0

Latency in a VM with 8 vCPUs, while 32 threads walk cgroupfs paths (worst of three 20 s runs; mkdir and rmdir by a SCHED_FIFO thread):

before after
p50 p99 max p50 p99 max

  cgroup mkdir     2.8ms   9.0ms  20.1ms    25us   37us   84us
  cgroup rmdir     4.2ms  12.7ms  26.6ms    28us   38us  107us

A write that only needs cgroup_mutex waited up to 23 ms before and up to 34 us after. With 8 threads and no writer, stat() and access() on a path whose directories have iattrs are 12-14% faster, since the refresh copies less.

Tested with the kernfs selftests on lockdep + KASAN and KCSAN kernels; with extra checks of link counts, times and IN_DELETE_SELF; with concurrent mkdir/rmdir, setattr, stat, readdir and netdev churn; and with test-only delays that force the stale stat(), lost time update, backwards time, overwritten setattr and stat()-during-removal races. KCSAN reports nothing new, and its set_nlink() reports from the refresh are gone.

Based on vfs-7.5.kernfs. It also applies to v7.4-rc3.

Shakeel Butt (6):
kernfs: report times and link count from the node in getattr
kernfs: don't copy times and link count into the inode on refresh
kernfs: don't take kernfs_iattr_rwsem for the directory counters
kernfs: don't take kernfs_iattr_rwsem to publish KERNFS_REMOVING
kernfs: update a directory's times without kernfs_iattr_rwsem
kernfs: don't take kernfs_iattr_rwsem in __kernfs_remove()

 fs/kernfs/dir.c             |  48 +++++---------
 fs/kernfs/inode.c           | 125 ++++++++++++++++++++++++++++++------
 fs/kernfs/kernfs-internal.h |   2 +
 3 files changed, 124 insertions(+), 51 deletions(-)

base-commit: ccf2a4692fed0fa77751d32d2690585127a152c5

Signature · 1 line
2.53.1-Meta
#1 lore raw

PATCH 1/6 kernfs: report times and link count from the node in getattr

getattr and permission copy a node's attributes into its inode while holding kernfs_iattr_rwsem for reading, so several of them can write the same inode at once. That is safe only because every update of these attributes takes the lock for writing, so they all write the same values. Commit aa1d058d48f2 ("kernfs: dont take i_lock on inode attr read") relies on this. Setting up a node's first iattrs from setxattr does not take the lock, but a refresh that misses them writes no times.

The next patches let a directory's times and link count change without that lock. A slow refresh could then write an old value after a newer one, and stat() could return a stale or torn time or link count.

Make getattr report the times and link count from the node instead of the inode. Permission checks use neither. Nothing changes yet, except that a stat() racing a node's first setxattr can report the new times.

---
 fs/kernfs/inode.c | 32 ++++++++++++++++++++++++++++++--
 1 file changed, 30 insertions(+), 2 deletions(-)
fs/kernfs/inode.c +30 -2
index 53b1a2796e0b..3616161a56cb 100644
@@ -162,9 +162,22 @@ static inline void set_inode_attr(struct inode *inode,
 	inode_set_ctime_to_ts(inode, attrs->ia_ctime);
 }
 
+/*
+ * Get the link count of a directory: its subdirectories plus two.
+ * Return false for other nodes and for a directory being removed.
+ */
+static bool kernfs_dir_nlink(struct kernfs_node *kn, unsigned int *nlink)
+{
+	if (kernfs_type(kn) != KERNFS_DIR || (kn->flags & KERNFS_REMOVING))
+		return false;
+	*nlink = kn->dir.subdirs + 2;
+	return true;
+}
+
 static void kernfs_refresh_inode(struct kernfs_node *kn, struct inode *inode)
 {
 	struct kernfs_iattrs *attrs;
+	unsigned int nlink;
 
 	inode->i_mode = kn->mode;
 	attrs = kernfs_iattrs_noalloc(kn);
@@ -175,8 +188,8 @@ static void kernfs_refresh_inode(struct kernfs_node *kn, struct inode *inode)
 		 */
 		set_inode_attr(inode, attrs);
 
-	if (kernfs_type(kn) == KERNFS_DIR && !(kn->flags & KERNFS_REMOVING))
-		set_nlink(inode, kn->dir.subdirs + 2);
+	if (kernfs_dir_nlink(kn, &nlink))
+		set_nlink(inode, nlink);
 }
 
 int kernfs_iop_getattr(const struct mnt_idmap *idmap,
@@ -186,10 +199,25 @@ int kernfs_iop_getattr(const struct mnt_idmap *idmap,
 	struct inode *inode = d_inode(path->dentry);
 	struct kernfs_node *kn = inode->i_private;
 	struct kernfs_root *root = kernfs_root(kn);
+	struct kernfs_iattrs *attrs;
+	unsigned int nlink;
 
 	down_read(&root->kernfs_iattr_rwsem);
 	kernfs_refresh_inode(kn, inode);
 	generic_fillattr(&nop_mnt_idmap, request_mask, inode, stat);
+
+	/*
+	 * Other refreshes may store older times or link counts in the
+	 * inode at the same time.  Report the values from the node.
+	 */
+	attrs = kernfs_iattrs_noalloc(kn);
+	if (attrs) {
+		stat->atime = attrs->ia_atime;
+		stat->mtime = attrs->ia_mtime;
+		stat->ctime = attrs->ia_ctime;
+	}
+	if (kernfs_dir_nlink(kn, &nlink))
+		stat->nlink = nlink;
 	up_read(&root->kernfs_iattr_rwsem);
 
 	return 0;
Signature · 1 line
2.53.1-Meta
#2 lore raw

PATCH 2/6 kernfs: don't copy times and link count into the inode on refresh

In reply to Shakeel Butt

Every permission check and getattr copies a node's times, if it has iattrs, and a directory's link count into the shared inode. Since the previous patch, getattr reports both from the node, and permission checks use only the mode, owner and group. The only other users of a kernfs inode's times are the mmap write fault and the atime update, which just update the inode's copy. The inode's link count matters only to the IN_DELETE_SELF check, which needs 0 once __kernfs_remove() clears it.

Copy only the mode, owner and group, which also takes work off every permission check. Set a directory's link count when its inode is set up. Every caller that sets up a directory inode holds kernfs_rwsem, and __kernfs_remove() holds it for writing while it clears the count, so the two cannot race. Assert that, and note in the kernfs_get_inode() comment that it is not for directories; its one user outside kernfs, cgroup_may_write(), gets a file.

getattr fell back to the inode's link count for a directory being removed. That count is no longer kept up to date, so report 0 instead:
KERNFS_REMOVING is never cleared, and lookups already fail.

---
 fs/kernfs/inode.c | 32 ++++++++++++++++++--------------
 1 file changed, 18 insertions(+), 14 deletions(-)
fs/kernfs/inode.c +18 -14
index 3616161a56cb..2d8da8b4e7da 100644
@@ -157,27 +157,23 @@ static inline void set_inode_attr(struct inode *inode,
 {
 	inode->i_uid = attrs->ia_uid;
 	inode->i_gid = attrs->ia_gid;
-	inode_set_atime_to_ts(inode, attrs->ia_atime);
-	inode_set_mtime_to_ts(inode, attrs->ia_mtime);
-	inode_set_ctime_to_ts(inode, attrs->ia_ctime);
 }
 
 /*
- * Get the link count of a directory: its subdirectories plus two.
- * Return false for other nodes and for a directory being removed.
+ * Get a directory's link count: its subdirectories plus two, or 0 once
+ * its removal has started.  Return false for other nodes.
  */
 static bool kernfs_dir_nlink(struct kernfs_node *kn, unsigned int *nlink)
 {
-	if (kernfs_type(kn) != KERNFS_DIR || (kn->flags & KERNFS_REMOVING))
+	if (kernfs_type(kn) != KERNFS_DIR)
 		return false;
-	*nlink = kn->dir.subdirs + 2;
+	*nlink = (kn->flags & KERNFS_REMOVING) ? 0 : kn->dir.subdirs + 2;
 	return true;
 }
 
 static void kernfs_refresh_inode(struct kernfs_node *kn, struct inode *inode)
 {
 	struct kernfs_iattrs *attrs;
-	unsigned int nlink;
 
 	inode->i_mode = kn->mode;
 	attrs = kernfs_iattrs_noalloc(kn);
@@ -187,9 +183,6 @@ static void kernfs_refresh_inode(struct kernfs_node *kn, struct inode *inode)
 		 * persistent copy in kernfs_node.
 		 */
 		set_inode_attr(inode, attrs);
-
-	if (kernfs_dir_nlink(kn, &nlink))
-		set_nlink(inode, nlink);
 }
 
 int kernfs_iop_getattr(const struct mnt_idmap *idmap,
@@ -207,8 +200,8 @@ int kernfs_iop_getattr(const struct mnt_idmap *idmap,
 	generic_fillattr(&nop_mnt_idmap, request_mask, inode, stat);
 
 	/*
-	 * Other refreshes may store older times or link counts in the
-	 * inode at the same time.  Report the values from the node.
+	 * The inode's times and link count are not kept up to date.
+	 * Report them from the node.
 	 */
 	attrs = kernfs_iattrs_noalloc(kn);
 	if (attrs) {
@@ -225,6 +218,8 @@ int kernfs_iop_getattr(const struct mnt_idmap *idmap,
 
 static void kernfs_init_inode(struct kernfs_node *kn, struct inode *inode)
 {
+	unsigned int nlink;
+
 	kernfs_get(kn);
 	inode->i_private = kn;
 	inode->i_mapping->a_ops = &ram_aops;
@@ -238,6 +233,14 @@ static void kernfs_init_inode(struct kernfs_node *kn, struct inode *inode)
 	/* initialize inode according to type */
 	switch (kernfs_type(kn)) {
 	case KERNFS_DIR:
+		/*
+		 * The link count is set only here; getattr reports it from
+		 * the node.  Holding kernfs_rwsem keeps this from racing
+		 * with __kernfs_remove() clearing it.
+		 */
+		lockdep_assert_held(&kernfs_root(kn)->kernfs_rwsem);
+		if (kernfs_dir_nlink(kn, &nlink))
+			set_nlink(inode, nlink);
 		inode->i_op = &kernfs_dir_iops;
 		inode->i_fop = &kernfs_dir_fops;
 		if (kn->flags & KERNFS_EMPTY_DIR)
@@ -267,7 +270,8 @@ static void kernfs_init_inode(struct kernfs_node *kn, struct inode *inode)
  *	locked.
  *
  *	Locking:
- *	Kernel thread context (may sleep).
+ *	Kernel thread context (may sleep).  Not for directories: kernfs
+ *	sets up their inodes itself, with kernfs_rwsem held.
  *
  *	Return:
  *	Pointer to allocated inode on success, %NULL on failure.
Signature · 1 line
2.53.1-Meta
#3 lore raw

PATCH 3/6 kernfs: don't take kernfs_iattr_rwsem for the directory counters

In reply to Shakeel Butt

kernfs_link_sibling() and kernfs_unlink_sibling() take kernfs_iattr_rwsem for writing to update the parent's dir.subdirs and dir.rev. Every permission check and getattr takes the lock for reading, so under load the writer waits, while it holds kernfs_rwsem and, for cgroups, cgroup_mutex.

Both writers hold kernfs_rwsem for writing, which is enough. Commit 0559f63057f9 ("kernfs: fix missing kernfs_iattr_rwsem locking") added the lock to keep the link count stable for getattr, which now reads it from the node; the refresh no longer copies it. The other readers, kernfs_dir_changed(), kernfs_set_rev(), kobject_has_children() and resctrl, never took the lock. A racing stat() sees the old or the new link count. While a subdirectory is renamed within the directory, it can also see one less, as it could before.

Drop the lock, mark the accesses, and assert that kernfs_rwsem is held.

kernfs_iattr_rwsem write acquisitions (lock_stat), before -> after:

5 dummy netdevs created and removed 1850 -> 1110
100 cgroup mkdirs and 100 rmdirs 7500 -> 4500

---
 fs/kernfs/dir.c   | 12 ++++++------
 fs/kernfs/inode.c |  3 ++-
 2 files changed, 8 insertions(+), 7 deletions(-)
fs/kernfs/dir.c +6 -6
index fb1dcf35d900..ef23eac684c6 100644
@@ -387,6 +387,8 @@ static int kernfs_link_sibling(struct kernfs_node *kn)
 	struct kernfs_node *kn_parent;
 	struct rb_node **node;
 
+	lockdep_assert_held_write(&kernfs_root(kn)->kernfs_rwsem);
+
 	kn_parent = kernfs_parent(kn);
 	node = &kn_parent->dir.children.rb_node;
 
@@ -410,11 +412,9 @@ static int kernfs_link_sibling(struct kernfs_node *kn)
 	rb_insert_color(&kn->rb, &kn_parent->dir.children);
 
 	/* successfully added, account subdir number */
-	down_write(&kernfs_root(kn)->kernfs_iattr_rwsem);
 	if (kernfs_type(kn) == KERNFS_DIR)
-		kn_parent->dir.subdirs++;
+		WRITE_ONCE(kn_parent->dir.subdirs, kn_parent->dir.subdirs + 1);
 	kernfs_inc_rev(kn_parent);
-	up_write(&kernfs_root(kn)->kernfs_iattr_rwsem);
 
 	return 0;
 }
@@ -436,15 +436,15 @@ static bool kernfs_unlink_sibling(struct kernfs_node *kn)
 {
 	struct kernfs_node *kn_parent;
 
+	lockdep_assert_held_write(&kernfs_root(kn)->kernfs_rwsem);
+
 	if (RB_EMPTY_NODE(&kn->rb))
 		return false;
 
 	kn_parent = kernfs_parent(kn);
-	down_write(&kernfs_root(kn)->kernfs_iattr_rwsem);
 	if (kernfs_type(kn) == KERNFS_DIR)
-		kn_parent->dir.subdirs--;
+		WRITE_ONCE(kn_parent->dir.subdirs, kn_parent->dir.subdirs - 1);
 	kernfs_inc_rev(kn_parent);
-	up_write(&kernfs_root(kn)->kernfs_iattr_rwsem);
 
 	rb_erase(&kn->rb, &kn_parent->dir.children);
 	RB_CLEAR_NODE(&kn->rb);
fs/kernfs/inode.c +2 -1
index 2d8da8b4e7da..2f463c0354a5 100644
@@ -167,7 +167,8 @@ static bool kernfs_dir_nlink(struct kernfs_node *kn, unsigned int *nlink)
 {
 	if (kernfs_type(kn) != KERNFS_DIR)
 		return false;
-	*nlink = (kn->flags & KERNFS_REMOVING) ? 0 : kn->dir.subdirs + 2;
+	*nlink = (kn->flags & KERNFS_REMOVING) ? 0 :
+		 READ_ONCE(kn->dir.subdirs) + 2;
 	return true;
 }
 
Signature · 1 line
2.53.1-Meta
#4 lore raw

PATCH 4/6 kernfs: don't take kernfs_iattr_rwsem to publish KERNFS_REMOVING

In reply to Shakeel Butt

__kernfs_remove() takes kernfs_iattr_rwsem for writing while it marks a subtree KERNFS_REMOVING, inside kernfs_rwsem. Commit 507d8ce13f5b ("kernfs: Don't set_nlink for directories being removed") added it so that a refresh would not set the i_nlink of a directory whose removal has started. The refresh no longer sets i_nlink, so the lock is not needed.

The flag is only set here, under kernfs_rwsem. Inode setup reads it under kernfs_rwsem. getattr reads it without and reports the old link count or 0, either of which is right for a stat() racing the removal.

Drop the lock, mark the store and the load, and read the flags once in kernfs_dir_nlink(), as the type bits share the word.

kernfs_iattr_rwsem write acquisitions (lock_stat), before -> after:

5 dummy netdevs created and removed 1110 -> 740
100 cgroup mkdirs and 100 rmdirs 4500 -> 3000

---
 fs/kernfs/dir.c   | 10 ++++++----
 fs/kernfs/inode.c |  7 +++++--
 2 files changed, 11 insertions(+), 6 deletions(-)
fs/kernfs/dir.c +6 -4
index ef23eac684c6..890aca5d8d70 100644
@@ -1604,15 +1604,17 @@ static void __kernfs_remove(struct kernfs_node *kn)
 
 	pr_debug("kernfs %s: removing\n", kernfs_rcu_name(kn));
 
-	/* prevent new usage by marking all nodes removing and deactivating */
-	down_write(&kernfs_root(kn)->kernfs_iattr_rwsem);
+	/*
+	 * Prevent new usage by marking all nodes removing and deactivating.
+	 * KERNFS_REMOVING is only set here, under kernfs_rwsem.
+	 * kernfs_dir_nlink() reads it without that lock.
+	 */
 	pos = NULL;
 	while ((pos = kernfs_next_descendant_post(pos, kn))) {
-		pos->flags |= KERNFS_REMOVING;
+		WRITE_ONCE(pos->flags, pos->flags | KERNFS_REMOVING);
 		if (kernfs_active(pos))
 			atomic_add(KN_DEACTIVATED_BIAS, &pos->active);
 	}
-	up_write(&kernfs_root(kn)->kernfs_iattr_rwsem);
 
 	/* deactivate and unlink the subtree node-by-node */
 	do {
fs/kernfs/inode.c +5 -2
index 2f463c0354a5..ca234f69882b 100644
@@ -165,9 +165,12 @@ static inline void set_inode_attr(struct inode *inode,
  */
 static bool kernfs_dir_nlink(struct kernfs_node *kn, unsigned int *nlink)
 {
-	if (kernfs_type(kn) != KERNFS_DIR)
+	/* KERNFS_REMOVING can be set concurrently; read the flags once. */
+	unsigned short flags = READ_ONCE(kn->flags);
+
+	if ((flags & KERNFS_TYPE_MASK) != KERNFS_DIR)
 		return false;
-	*nlink = (kn->flags & KERNFS_REMOVING) ? 0 :
+	*nlink = (flags & KERNFS_REMOVING) ? 0 :
 		 READ_ONCE(kn->dir.subdirs) + 2;
 	return true;
 }
Signature · 1 line
2.53.1-Meta
#5 lore raw

PATCH 5/6 kernfs: update a directory's times without kernfs_iattr_rwsem

In reply to Shakeel Butt

kernfs_add_one() and __kernfs_remove() update the parent's ctime and mtime under kernfs_iattr_rwsem for writing, inside kernfs_rwsem. This is where cgroup mkdirs block on Meta fleet: the writer waits for every path walk's permission check while it holds kernfs_rwsem and cgroup_mutex.

The lock keeps the times stable for getattr, now their only reader, and a timespec64 is too wide for a single store. Give the three times a seqlock instead. setattr takes it in addition to kernfs_iattr_rwsem, adding or removing an entry takes only the seqlock, and getattr retries if a write was in progress. Adding or removing an entry reads the clock with the seqlock held, so updates store their times in the order they happen. A racing stat() can see the new link count with the old times, or the reverse.

A directory without iattrs still skips the update. So that an update racing with setting up the iattrs is not lost, take the seqlock of new iattrs before publishing them and set their times after. An update that found no iattrs ran before the times were set, and a writer that finds the new iattrs waits for them. mkdir no longer takes kernfs_iattr_rwsem for writing.

kernfs_iattr_rwsem write acquisitions (lock_stat), before -> after:

5 dummy netdevs created and removed 740 -> 370
100 cgroup mkdirs and 100 rmdirs 3000 -> 1500

---
 fs/kernfs/dir.c             | 26 ++----------
 fs/kernfs/inode.c           | 85 ++++++++++++++++++++++++++++++-------
 fs/kernfs/kernfs-internal.h |  2 +
 3 files changed, 75 insertions(+), 38 deletions(-)
fs/kernfs/dir.c +4 -22
index 890aca5d8d70..8686219b826f 100644
@@ -825,7 +825,6 @@ struct kernfs_node *kernfs_find_and_get_node_by_id(struct kernfs_root *root,
 int kernfs_add_one(struct kernfs_node *kn)
 {
 	struct kernfs_root *root = kernfs_root(kn);
-	struct kernfs_iattrs *ps_iattr;
 	struct kernfs_node *parent;
 	bool has_ns;
 	int ret;
@@ -853,16 +852,7 @@ int kernfs_add_one(struct kernfs_node *kn)
 	if (ret)
 		goto out_unlock;
 
-	/* Update timestamps on the parent */
-	down_write(&root->kernfs_iattr_rwsem);
-
-	ps_iattr = parent->iattr;
-	if (ps_iattr) {
-		ktime_get_real_ts64(&ps_iattr->ia_ctime);
-		ps_iattr->ia_mtime = ps_iattr->ia_ctime;
-	}
-
-	up_write(&root->kernfs_iattr_rwsem);
+	kernfs_dir_update_times(parent);
 
 	/*
 	 * Activate the new node unless CREATE_DEACTIVATED is requested.
@@ -1635,20 +1625,12 @@ static void __kernfs_remove(struct kernfs_node *kn)
 		 * to decide who's responsible for cleanups.
 		 */
 		if (!parent || kernfs_unlink_sibling(pos)) {
-			struct kernfs_iattrs *ps_iattr =
-				parent ? parent->iattr : NULL;
-
 			down_write(&kernfs_root(kn)->kernfs_iattr_rwsem);
-
 			kernfs_clear_inode_nlink(pos);
-
-			/* update timestamps on the parent */
-			if (ps_iattr) {
-				ktime_get_real_ts64(&ps_iattr->ia_ctime);
-				ps_iattr->ia_mtime = ps_iattr->ia_ctime;
-			}
-
 			up_write(&kernfs_root(kn)->kernfs_iattr_rwsem);
+
+			if (parent)
+				kernfs_dir_update_times(parent);
 			kernfs_put(pos);
 		}
 
fs/kernfs/inode.c +69 -16

120 lines not on this page · raw message

fs/kernfs/kernfs-internal.h +2 -0
index 2e364b626c1e..9bba1fc8acfd 100644
@@ -25,6 +25,7 @@ struct kernfs_iattrs {
 	struct timespec64	ia_atime;
 	struct timespec64	ia_mtime;
 	struct timespec64	ia_ctime;
+	seqlock_t		ia_time_lock;	/* protects the times above */
 
 	struct list_head	xattrs;
 	struct simple_xattr_limits xattr_limits;
@@ -188,6 +189,7 @@ int kernfs_iop_getattr(const struct mnt_idmap *idmap,
 		       u32 request_mask, unsigned int query_flags);
 ssize_t kernfs_iop_listxattr(struct dentry *dentry, char *buf, size_t size);
 int __kernfs_setattr(struct kernfs_node *kn, const struct iattr *iattr);
+void kernfs_dir_update_times(struct kernfs_node *kn);
 
 /*
  * dir.c
Signature · 1 line
2.53.1-Meta
#6 lore raw

PATCH 6/6 kernfs: don't take kernfs_iattr_rwsem in __kernfs_remove()

In reply to Shakeel Butt

__kernfs_remove() takes kernfs_iattr_rwsem for writing around kernfs_clear_inode_nlink() for every removed node. A cgroup rmdir removes the directory and all its files, so it takes the lock once per file, each time behind the readers. Our lock monitor saw rmdirs wait in __kernfs_remove() for several seconds.

Commit eea5d2bb34ba ("kernfs: Send IN_DELETE_SELF and IN_IGNORED") clears i_nlink inside this section, so a refresh in progress could not set a removed directory's i_nlink again, the race described in commit 507d8ce13f5b ("kernfs: Don't set_nlink for directories being removed"). The refresh no longer sets i_nlink. Only inode setup does, under kernfs_rwsem, which __kernfs_remove() holds for writing.

A racing fstat() of a file can read its i_nlink while it is cleared, and sees 1 or 0, as on other filesystems. A directory reports its link count from the node.

Drop the lock. rmdir no longer takes kernfs_iattr_rwsem for writing.

kernfs_iattr_rwsem write acquisitions (lock_stat), before -> after:

5 dummy netdevs created and removed 370 -> 0
100 cgroup mkdirs and 100 rmdirs 1500 -> 0

---
 fs/kernfs/dir.c | 2 --
 1 file changed, 2 deletions(-)
fs/kernfs/dir.c +0 -2
index 8686219b826f..c1402a461405 100644
@@ -1625,9 +1625,7 @@ static void __kernfs_remove(struct kernfs_node *kn)
 		 * to decide who's responsible for cleanups.
 		 */
 		if (!parent || kernfs_unlink_sibling(pos)) {
-			down_write(&kernfs_root(kn)->kernfs_iattr_rwsem);
 			kernfs_clear_inode_nlink(pos);
-			up_write(&kernfs_root(kn)->kernfs_iattr_rwsem);
 
 			if (parent)
 				kernfs_dir_update_times(parent);
Signature · 1 line
2.53.1-Meta