Skip to contentSkip to conversation

cgroup: allocate the inode for cgroup_may_write() before cgroup_mutex

PATCH New Shakeel Butt 1 participant 1 message Nov 18

Subsystems (4)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 cgroup: allocate the inode for cgroup_may_write() before cgroup_mutex

In the Meta fleet, on 6.16 kernels, the memcg OOM killer ran from cgroup_may_write() with cgroup_mutex and cgroup_threadgroup_rwsem held. The callers were systemd in containers and build tools, short of memory in their own cgroup. Our lock monitor also saw cgroup_mutex held across memcg reclaim there, for up to 152 ms.

cgroup_may_write() checks permission on the inode of a cgroup.procs file. kernfs frees that inode with its last reference, so it is often not cached, and kernfs_get_inode() then allocates one, charged to the caller's memory cgroup, with cgroup_mutex held. A cgroup.procs write usually holds cgroup_threadgroup_rwsem too. While the allocation reclaims, every cgroup operation, fork and exit on the host waits. clone3(CLONE_INTO_CGROUP) allocates the same way.

Allocate the inode before taking cgroup_mutex, and add it under kernfs_rwsem with kernfs_get_inode_locked(), as kernfs lookup does. The cgroup.procs write allocates after dropping the file's active reference, because rmdir waits for active references with cgroup_mutex held. clone3() checks that the fd is a cgroup2 directory before allocating on its superblock.

Worst latency in a VM while a task in a full 256 MB memory cgroup moves a process between two cgroups (60 s, worst of four runs):

before after

  fork and exit            4.9 s     14 ms
  mkdir and rmdir          4.9 s    8.4 ms
  cgroup_mutex held        4.9 s    8.3 ms

Before, each run spent 2 to 16 s in memcg reclaim with cgroup_mutex held. After, none.

---
This is based on my series "kernfs: stop taking kernfs_iattr_rwsem for writing in mkdir/rmdir" [1], on top of vfs-7.5.kernfs, as both change the comment of kernfs_get_inode().

[1] https://lore.kernel.org/all/[email protected]/

 fs/kernfs/inode.c      |  28 ++++-----
 include/linux/kernfs.h |   4 +-
 kernel/cgroup/cgroup.c | 128 +++++++++++++++++++++++++++--------------
 3 files changed, 98 insertions(+), 62 deletions(-)
fs/kernfs/inode.c +11 -17
index 580095e0fad9..71e19d32e392 100644
@@ -318,30 +318,24 @@ static void kernfs_init_inode(struct kernfs_node *kn, struct inode *inode)
 }
 
 /**
- *	kernfs_get_inode - get inode for kernfs_node
- *	@sb: super block
- *	@kn: kernfs_node to allocate inode for
+ *	kernfs_get_inode - get the inode of a kernfs_node
+ *	@inode: inode allocated with new_inode() on the target superblock
+ *	@kn: kernfs_node to get the inode for
  *
- *	Get inode for @kn.  If such inode doesn't exist, a new inode is
- *	allocated and basics are initialized.  New inode is returned
- *	locked.
+ *	kernfs_get_inode_locked() for callers outside kernfs.  Allocate
+ *	@inode before taking your own locks, as the allocation can wait for
+ *	reclaim.  @inode is consumed.
  *
  *	Locking:
- *	Kernel thread context (may sleep).  Not for directories: kernfs
- *	sets up their inodes itself, with kernfs_rwsem held.
+ *	Kernel thread context (may sleep).  Takes kernfs_rwsem for reading.
  *
  *	Return:
- *	Pointer to allocated inode on success, %NULL on failure.
+ *	The inode of @kn, or %NULL if @kn is no longer active.
  */
-struct inode *kernfs_get_inode(struct super_block *sb, struct kernfs_node *kn)
+struct inode *kernfs_get_inode(struct inode *inode, struct kernfs_node *kn)
 {
-	struct inode *inode;
-
-	inode = iget_locked(sb, kernfs_ino(kn));
-	if (inode && (inode_state_read_once(inode) & I_NEW))
-		kernfs_init_inode(kn, inode);
-
-	return inode;
+	guard(rwsem_read)(&kernfs_root(kn)->kernfs_rwsem);
+	return kernfs_get_inode_locked(inode, kn);
 }
 
 static int kernfs_inode_test(struct inode *inode, void *data)
include/linux/kernfs.h +2 -2
index 803294a67e47..cda9467693da 100644
@@ -429,7 +429,7 @@ void kernfs_put(struct kernfs_node *kn);
 
 struct kernfs_node *kernfs_node_from_dentry(struct dentry *dentry);
 struct kernfs_root *kernfs_root_from_sb(struct super_block *sb);
-struct inode *kernfs_get_inode(struct super_block *sb, struct kernfs_node *kn);
+struct inode *kernfs_get_inode(struct inode *inode, struct kernfs_node *kn);
 
 struct dentry *kernfs_node_dentry(struct kernfs_node *kn,
 				  struct super_block *sb);
@@ -528,7 +528,7 @@ static inline struct kernfs_root *kernfs_root_from_sb(struct super_block *sb)
 { return NULL; }
 
 static inline struct inode *
-kernfs_get_inode(struct super_block *sb, struct kernfs_node *kn)
+kernfs_get_inode(struct inode *inode, struct kernfs_node *kn)
 { return NULL; }
 
 static inline struct kernfs_root *
kernel/cgroup/cgroup.c +85 -43

238 lines not on this page · raw message

base-commit: ccf2a4692fed0fa77751d32d2690585127a152c5

Signature · 1 line
2.53.1-Meta