cgroup: allocate the inode for cgroup_may_write() before cgroup_mutex
Subsystems (4)
PATCH 1/1 cgroup: allocate the inode for cgroup_may_write() before cgroup_mutex
In the Meta fleet, on 6.16 kernels, the memcg OOM killer ran from cgroup_may_write() with cgroup_mutex and cgroup_threadgroup_rwsem held. The callers were systemd in containers and build tools, short of memory in their own cgroup. Our lock monitor also saw cgroup_mutex held across memcg reclaim there, for up to 152 ms.
cgroup_may_write() checks permission on the inode of a cgroup.procs file. kernfs frees that inode with its last reference, so it is often not cached, and kernfs_get_inode() then allocates one, charged to the caller's memory cgroup, with cgroup_mutex held. A cgroup.procs write usually holds cgroup_threadgroup_rwsem too. While the allocation reclaims, every cgroup operation, fork and exit on the host waits. clone3(CLONE_INTO_CGROUP) allocates the same way.
Allocate the inode before taking cgroup_mutex, and add it under kernfs_rwsem with kernfs_get_inode_locked(), as kernfs lookup does. The cgroup.procs write allocates after dropping the file's active reference, because rmdir waits for active references with cgroup_mutex held. clone3() checks that the fd is a cgroup2 directory before allocating on its superblock.
Worst latency in a VM while a task in a full 256 MB memory cgroup moves a process between two cgroups (60 s, worst of four runs):
before after
fork and exit 4.9 s 14 ms mkdir and rmdir 4.9 s 8.4 ms cgroup_mutex held 4.9 s 8.3 ms
Before, each run spent 2 to 16 s in memcg reclaim with cgroup_mutex held. After, none.
---
This is based on my series "kernfs: stop taking kernfs_iattr_rwsem for writing in mkdir/rmdir" [1], on top of vfs-7.5.kernfs, as both change the comment of kernfs_get_inode().
[1] https://lore.kernel.org/all/[email protected]/
fs/kernfs/inode.c | 28 ++++----- include/linux/kernfs.h | 4 +- kernel/cgroup/cgroup.c | 128 +++++++++++++++++++++++++++-------------- 3 files changed, 98 insertions(+), 62 deletions(-)
fs/kernfs/inode.c +11 -17
@@ -318,30 +318,24 @@ static void kernfs_init_inode(struct kernfs_node *kn, struct inode *inode) }/** -* kernfs_get_inode - get inode for kernfs_node -* @sb: super block -* @kn: kernfs_node to allocate inode for +* kernfs_get_inode - get the inode of a kernfs_node +* @inode: inode allocated with new_inode() on the target superblock +* @kn: kernfs_node to get the inode for * -* Get inode for @kn. If such inode doesn't exist, a new inode is -* allocated and basics are initialized. New inode is returned -* locked. +* kernfs_get_inode_locked() for callers outside kernfs. Allocate +* @inode before taking your own locks, as the allocation can wait for +* reclaim. @inode is consumed. * * Locking: -* Kernel thread context (may sleep). Not for directories: kernfs -* sets up their inodes itself, with kernfs_rwsem held. +* Kernel thread context (may sleep). Takes kernfs_rwsem for reading. * * Return: -* Pointer to allocated inode on success, %NULL on failure. +* The inode of @kn, or %NULL if @kn is no longer active. */ -struct inode *kernfs_get_inode (struct super_block *sb,struct kernfs_node *kn) +struct inode *kernfs_get_inode (struct inode *inode,struct kernfs_node *kn) { -struct inode *inode; - - inode =iget_locked (sb,kernfs_ino (kn)); -if (inode && (inode_state_read_once (inode) &I_NEW )) -kernfs_init_inode (kn, inode); - -return inode; +guard (rwsem_read)(&kernfs_root (kn)->kernfs_rwsem); +return kernfs_get_inode_locked (inode, kn); }static int kernfs_inode_test (struct inode *inode,void *data)
include/linux/kernfs.h +2 -2
@@ -429,7 +429,7 @@ void kernfs_put(struct kernfs_node *kn);struct kernfs_node *kernfs_node_from_dentry (struct dentry *dentry);struct kernfs_root *kernfs_root_from_sb (struct super_block *sb); -struct inode *kernfs_get_inode (struct super_block *sb,struct kernfs_node *kn); +struct inode *kernfs_get_inode (struct inode *inode,struct kernfs_node *kn);struct dentry *kernfs_node_dentry (struct kernfs_node *kn,struct super_block *sb);
@@ -528,7 +528,7 @@ static inline struct kernfs_root *kernfs_root_from_sb(struct super_block *sb) {return NULL ; }static inline struct inode * -kernfs_get_inode (struct super_block *sb,struct kernfs_node *kn) +kernfs_get_inode (struct inode *inode,struct kernfs_node *kn) {return NULL ; } static inline struct kernfs_root *
kernel/cgroup/cgroup.c +85 -43
238 lines not on this page · raw message
base-commit: ccf2a4692fed0fa77751d32d2690585127a152c5
Signature · 1 line
2.53.1-Meta