kernfs: don't hold kernfs_rwsem across dir_emit()
Subsystems (4)
v2 cover kernfs: don't hold kernfs_rwsem across dir_emit()
kernfs_fop_readdir() takes kernfs_rwsem for reading and holds it for the whole listing, dir_emit() included. dir_emit() copies into a userspace buffer, so it can fault, and under memory pressure that fault goes to reclaim.
We are hitting this case on the Meta fleet very regularly. Many times it is below[1], a monitoring daemon. It walks the cgroup tree and faults on its own getdents(2) buffer with the lock held for reading.
below: page allocation stall for 120 secs: order:0, mode:0x140dca(GFP_HIGHUSER_MOVABLE|__GFP_ZERO|__GFP_COMP) nodemask=(null),cpuset=hostcritical.slice,mems_allowed=0 Call Trace: <TASK> dump_stack_lvl+0x5d/0x80 __alloc_frozen_pages_noprof+0x5f4d/0x6300 ? memcg_list_lru_alloc+0x73/0x320 ? ima_file_check+0xd0/0x7d0 vma_alloc_folio_noprof+0x145/0x560 handle_mm_fault+0x17c9/0x2720 ? find_vma+0x27/0x30 do_user_addr_fault+0x39f/0x6e0 exc_page_fault+0x8f/0x110 asm_exc_page_fault+0x22/0x30 RIP: 0010:filldir64+0xd7/0x1a0 [Code:/RSP:/RAX:..R15: register block elided] kernfs_fop_readdir+0x2de/0x420 iterate_dir+0x8c/0x1f0 __se_sys_getdents64+0x61/0xe0 ? copy_page_from_iter+0x860/0x860 do_syscall_64+0x6a/0x250 entry_SYSCALL_64_after_hwframe+0x4b/0x53 </TASK>
kernfs_rwsem is the lock every create, remove and rename in the hierarchy needs, and sysfs and cgroupfs have one per machine. More importantly, the userspace OOM killers such as systemd-oomd also traverse the cgroupfs hierarchy and can get stuck behind below, which is stuck in reclaim. They are supposed to relieve memory pressure on the system, but can get stuck themselves.
The fix is to not hold the lock across dir_emit(). Copy the name while the lock is held, drop the lock, emit, and take it again.
That opens a window. With the lock dropped, the entry the listing stopped on can be removed or renamed before the listing picks up again, and readdir only remembers its place as a name hash. Two entries in one directory can share a hash, so the hash alone does not say where the listing stopped.
Patch 1 makes the fallback search land after the missing entry rather than on either side of it. Patch 2 keeps the name of the entry the listing is on and resumes on the full key. Patch 3 is the fix. Patch 4 adds tests.
Patch 1 is worth having on its own. Without the rest, the same search can land on the entry before the missing one and report it a second time, between two getdents(2) calls.
The series applies to the vfs-7.4.kernfs branch of the vfs tree.
Changes since v1 [2]:
- Split the old patch 2 in two, per Tejun. Patch 2 adds the name copy and the full-key resume with the locking unchanged; patch 3 only drops the lock.
[1] https://github.com/facebookincubator/below
[2] https://lore.kernel.org/all/[email protected]/
Shakeel Butt (4):
kernfs: don't repeat or skip an entry when readdir resumes
kernfs: resume readdir on the full entry key
kernfs: don't hold kernfs_rwsem across dir_emit()
selftests: cover readdir resuming at a removed entry
fs/kernfs/dir.c | 85 ++++++-- .../selftests/filesystems/kernfs_test.c | 200 ++++++++++++++++++ 2 files changed, 263 insertions(+), 22 deletions(-)
base-commit: 7f24d0f60b1087dbc28d68398957c18774e5a7d8
Signature · 1 line
2.53.0-Meta
PATCH v2 1/4 kernfs: don't repeat or skip an entry when readdir resumes
readdir keeps its place as a name hash in ctx->pos and pins the entry in file->private_data. If that entry is gone when the listing comes back, kernfs_dir_pos() searches the rbtree for the hash and keeps whatever node the descent stopped on. That is the entry before or after the missing one, depending on the shape of the tree.
Landing before it repeats an entry the previous getdents(2) call already reported. Landing after it, kernfs_dir_next_pos() calls rb_next() and steps over an unreported entry.
With children A(10), B(20), C(30):
report A, ctx->pos = 10
A removed
kernfs_dir_next_pos(10, A)
A is gone, the search for 10 stops at B
rb_next(B) -> C, so B is never reported
Only the repeat happens today, between two getdents(2) calls. The skip needs the pinned entry to go away inside one call, which the next patch allows when it drops kernfs_rwsem around dir_emit().
Before the commit 4e4d6d860b93 the descent kept a node only on the way left, which is a search for the first entry at or after the hash. That commit moved the assignment to the top of the loop, where it runs on right turns too. Restore that search, and step forward only when the pinned entry is still there rather than when the hash matches, since two entries in one directory can share a hash.
While here, kernfs_dir_next_pos() called the hash @ino. It is never an inode number, so name it @hash.
--- fs/kernfs/dir.c | 43 ++++++++++++++++++++++++++++--------------- 1 file changed, 28 insertions(+), 15 deletions(-)
fs/kernfs/dir.c +28 -15
@@ -1903,9 +1903,16 @@ static int kernfs_dir_fop_release(struct inode *inode, struct file *filp)return 0 ; } +/* +* Find where a listing left off. @resumed says whether @pos is still that +* entry; if not, the first entry at or after @hash is returned instead. +*/ static struct kernfs_node *kernfs_dir_pos (const struct ns_common *ns, -struct kernfs_node *parent,loff_t hash,struct kernfs_node *pos) +struct kernfs_node *parent,loff_t hash,struct kernfs_node *pos, +bool *resumed) { +if (resumed) + *resumed = false;if (pos) {int valid =kernfs_active (pos) && rcu_access_pointer(pos->__parent) == parent &&
@@ -1913,23 +1920,26 @@ static struct kernfs_node *kernfs_dir_pos(const struct ns_common *ns,kernfs_put (pos);if (!valid) pos =NULL ; +else if (resumed) + *resumed = true; }if (!pos && (hash >1 ) && (hash <INT_MAX )) {struct rb_node *node = parent->dir.children.rb_node; -u64 ns_id =kernfs_ns_id (ns); + +/* +* Keep a node only on the way left, so the search ends on the +* first entry at or after @hash. The empty name sorts before +* every entry sharing the hash, so it lands on the first. +*/ while (node) { - pos =rb_to_kn (node); +struct kernfs_node *kn =rb_to_kn (node); -if (hash < pos->hash) +if (kernfs_name_compare (hash,"" , ns, kn) <0 ) { + pos = kn; node = node->rb_left; -else if (hash > pos->hash) + }else { node = node->rb_right; -else if (ns_id <kernfs_ns_id (pos->ns)) - node = node->rb_left; -else if (ns_id >kernfs_ns_id (pos->ns)) - node = node->rb_right; -else -break ; + } } }/* Skip over entries which are dying/dead or in the wrong namespace */
@@ -1945,10 +1955,13 @@ static struct kernfs_node *kernfs_dir_pos(const struct ns_common *ns, }static struct kernfs_node *kernfs_dir_next_pos (const struct ns_common *ns, -struct kernfs_node *parent,ino_t ino,struct kernfs_node *pos) +struct kernfs_node *parent,loff_t hash,struct kernfs_node *pos) { - pos =kernfs_dir_pos (ns, parent, ino, pos); -if (pos) { +bool resumed; + + pos =kernfs_dir_pos (ns, parent, hash, pos, &resumed); +/* Step over @pos only if it survived; two entries can share a hash. */ +if (pos && resumed) { do { struct rb_node *node = rb_next(&pos->rb); if (!node)
@@ -1978,7 +1991,7 @@ static int kernfs_fop_readdir(struct file *file, struct dir_context *ctx)if (kernfs_ns_enabled (parent)) ns =kernfs_info (dentry->d_sb)->ns; -for (pos =kernfs_dir_pos (ns, parent, ctx->pos, pos); +for (pos =kernfs_dir_pos (ns, parent, ctx->pos, pos,NULL ); pos; pos =kernfs_dir_next_pos (ns, parent, ctx->pos, pos)) {const char *name =kernfs_rcu_name (pos);
Signature · 1 line
2.53.0-Meta
PATCH v2 2/4 kernfs: resume readdir on the full entry key
In reply to Shakeel Buttreaddir resumes at the entry pinned in file->private_data, and falls back to the hash in ctx->pos when that entry is gone. Two entries in one directory can share a hash, so the hash alone does not say where the listing stopped.
Copy the name under the lock, emit the copy, and keep it as the resume key. The pinned entry counts as the same entry only while its name still matches, and the fallback search starts after (hash, ns, name) rather than at the first entry with the hash.
No behaviour change yet. kernfs_rwsem is held across the loop, so nothing can move between two entries of one call, and a resume across getdents(2) calls has no saved name and keys on the hash as before.
--- fs/kernfs/dir.c | 50 ++++++++++++++++++++++++++++++++++++------------- 1 file changed, 37 insertions(+), 13 deletions(-)
fs/kernfs/dir.c +37 -13
@@ -1905,18 +1905,24 @@ static int kernfs_dir_fop_release(struct inode *inode, struct file *filp)/* * Find where a listing left off. @resumed says whether @pos is still that -* entry; if not, the first entry at or after @hash is returned instead. +* entry; if not, the search falls back to @hash, keyed by @name if given. */ static struct kernfs_node *kernfs_dir_pos (const struct ns_common *ns,struct kernfs_node *parent,loff_t hash,struct kernfs_node *pos, -bool *resumed) +const char *name,bool *resumed) {if (resumed) *resumed = false;if (pos) { +/* +* A rename keeps the hash if the new name hashes the same, so +* check @name too. Otherwise the caller would step over the +* entry now sitting where @pos used to be. +*/ int valid =kernfs_active (pos) &&rcu_access_pointer (pos->__parent) == parent && - hash == pos->hash; + hash == pos->hash && + (!name || !strcmp (name,kernfs_rcu_name (pos)));kernfs_put (pos);if (!valid) pos =NULL ;
@@ -1928,13 +1934,13 @@ static struct kernfs_node *kernfs_dir_pos(const struct ns_common *ns,/* * Keep a node only on the way left, so the search ends on the -* first entry at or after @hash. The empty name sorts before -* every entry sharing the hash, so it lands on the first. +* first entry after the key. An empty @name sorts before all +* entries sharing the hash, so it lands on the first of them. */ while (node) {struct kernfs_node *kn =rb_to_kn (node); -if (kernfs_name_compare (hash,"" , ns, kn) <0 ) { +if (kernfs_name_compare (hash, name ?:"" , ns, kn) <0 ) { pos = kn; node = node->rb_left; }else {
@@ -1955,12 +1961,13 @@ static struct kernfs_node *kernfs_dir_pos(const struct ns_common *ns, }static struct kernfs_node *kernfs_dir_next_pos (const struct ns_common *ns, -struct kernfs_node *parent,loff_t hash,struct kernfs_node *pos) +struct kernfs_node *parent,loff_t hash,struct kernfs_node *pos, +const char *name) {bool resumed; - pos =kernfs_dir_pos (ns, parent, hash, pos, &resumed); -/* Step over @pos only if it survived; two entries can share a hash. */ + pos =kernfs_dir_pos (ns, parent, hash, pos, name, &resumed); +/* Step over @pos only if it survived; @name finds the spot if not. */ if (pos && resumed) { do { struct rb_node *node = rb_next(&pos->rb);
@@ -1979,25 +1986,42 @@ static int kernfs_fop_readdir(struct file *file, struct dir_context *ctx)struct dentry *dentry = file->f_path.dentry;struct kernfs_node *parent =kernfs_dentry_node (dentry);struct kernfs_node *pos = file->private_data; + char *name __free(kfree) = NULL;struct kernfs_root *root;const struct ns_common *ns =NULL ;if (!dir_emit_dots (file, ctx))return 0 ; +/* +* One buffer for the call, holding the name of the entry the listing +* is on. PATH_MAX: kernfs bounds no single name. +*/ + name =kmalloc (PATH_MAX ,GFP_KERNEL ); +if (!name) +return -ENOMEM ; + root =kernfs_root (parent);down_read (&root->kernfs_rwsem);if (kernfs_ns_enabled (parent)) ns =kernfs_info (dentry->d_sb)->ns; -for (pos =kernfs_dir_pos (ns, parent, ctx->pos, pos,NULL ); +for (pos =kernfs_dir_pos (ns, parent, ctx->pos, pos,NULL ,NULL ); pos; - pos =kernfs_dir_next_pos (ns, parent, ctx->pos, pos)) { -const char *name =kernfs_rcu_name (pos); + pos =kernfs_dir_next_pos (ns, parent, ctx->pos, pos, name)) {unsigned int type =fs_umode_to_dtype (pos->mode); -int len =strlen (name);ino_t ino =kernfs_ino (pos); +int len; + +/* +* The copy is also the resume key, so a truncated name would +* resume here again. getname() caps a path, so only an +* in-kernel caller can get here; end the listing instead. +*/ + len =strscpy (name,kernfs_rcu_name (pos),PATH_MAX ); +if (WARN_ON_ONCE (len <0 )) +break ; ctx->pos = pos->hash; file->private_data = pos;
Signature · 1 line
2.53.0-Meta
PATCH v2 3/4 kernfs: don't hold kernfs_rwsem across dir_emit()
In reply to Shakeel Buttkernfs_fop_readdir() holds kernfs_rwsem for reading across the whole listing, dir_emit() included. dir_emit() copies to userspace, so it can fault into reclaim while holding the lock that every create, remove and rename in the hierarchy needs. sysfs and cgroupfs have one per machine.
Under memory pressure the monitoring daemons fault on their own getdents(2) buffer with it held:
below: page allocation stall for 120 secs: order:0,
mode:0x140dca(GFP_HIGHUSER_MOVABLE|__GFP_ZERO|__GFP_COMP)
nodemask=(null),cpuset=hostcritical.slice,mems_allowed=0
Call Trace:
<TASK>
dump_stack_lvl+0x5d/0x80
__alloc_frozen_pages_noprof+0x5f4d/0x6300
? memcg_list_lru_alloc+0x73/0x320
? ima_file_check+0xd0/0x7d0
vma_alloc_folio_noprof+0x145/0x560
handle_mm_fault+0x17c9/0x2720
? find_vma+0x27/0x30
do_user_addr_fault+0x39f/0x6e0
exc_page_fault+0x8f/0x110
asm_exc_page_fault+0x22/0x30
RIP: 0010:filldir64+0xd7/0x1a0
[Code:/RSP:/RAX:..R15: register block elided]
kernfs_fop_readdir+0x2de/0x420
iterate_dir+0x8c/0x1f0
__se_sys_getdents64+0x61/0xe0
? copy_page_from_iter+0x860/0x860
do_syscall_64+0x6a/0x250
entry_SYSCALL_64_after_hwframe+0x4b/0x53
</TASK>Commit 9aab10a0249e ("kernfs: Don't re-lock kernfs_root::kernfs_rwsem in kernfs_fop_readdir().") took the lock drop out because dir_emit() was handed kernfs_node::name, which a rename can free. The previous patch emits a copy instead, so drop the lock around dir_emit().
A listing is no longer atomic within one getdents(2) call, which for most sysfs and cgroup directories is all of it. POSIX leaves that unspecified for an entry added or removed since opendir(3).
--- fs/kernfs/dir.c | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-)
fs/kernfs/dir.c +7 -3
@@ -2027,10 +2027,14 @@ static int kernfs_fop_readdir(struct file *file, struct dir_context *ctx) file->private_data = pos;kernfs_get (pos); -if (!dir_emit (ctx, name, len, ino, type)) { -up_read (&root->kernfs_rwsem); +/* +* dir_emit() can fault, so run it unlocked. @pos is pinned +* above and kernfs_dir_pos() rechecks it on the way back. +*/ +up_read (&root->kernfs_rwsem); +if (!dir_emit (ctx, name, len, ino, type))return 0 ; - } +down_read (&root->kernfs_rwsem); }up_read (&root->kernfs_rwsem); file->private_data =NULL ;
Signature · 1 line
2.53.0-Meta
PATCH v2 4/4 selftests: cover readdir resuming at a removed entry
In reply to Shakeel ButtThe kernfs tests list only directories that do not change, so nothing covers the entry a listing stopped on being gone when it comes back.
readdir_resume_at_removed_entry takes a telldir(3) cookie for every entry, then removes each entry in turn, seeks to its cookie and reads the rest; nothing reported before it may come back. This is a resume between two getdents(2) calls, and it fails without "kernfs: don't repeat or skip an entry when readdir resumes".
readdir_resume_vs_internal_remove is a resume inside one call, which "kernfs: don't hold kernfs_rwsem across dir_emit()" opens. It churns cgroup.subtree_control rather than calling rmdir(2), which cannot reach that window because iterate_dir() holds the listed directory's i_rwsem for the whole listing. It is a stress test, has not been seen to catch the ordering bug, and keeps the window busy for lockdep and KASAN. It fails if the churn died and skips if the listings never overlapped it, so it cannot pass having listed a static directory.
--- .../selftests/filesystems/kernfs_test.c | 200 ++++++++++++++++++ 1 file changed, 200 insertions(+)
tools/testing/selftests/filesystems/kernfs_test.c +200 -0
@@ -8,6 +8,7 @@#include <limits.h> #include <net/if.h> #include <sched.h> +#include <signal.h> #include <stdio.h> #include <stdlib.h> #include <string.h>
@@ -17,6 +18,7 @@#include <sys/mount.h> #include <sys/socket.h> #include <sys/stat.h> +#include <sys/syscall.h> #include <sys/xattr.h> #include "kselftest_harness.h"
@@ -472,6 +474,204 @@ TEST_F(kernfs_cgroup, readdir_no_duplicates)EXPECT_STRNE (names[i], names[j]); } +#define RESUME_DIRS 24 + +/* +* Resuming at an entry that has gone must carry on after it, never before. +* Take a cookie for every entry, then remove each one, seek to its cookie +* and read the rest; nothing already reported may come back. +*/ +TEST_F(kernfs_cgroup, readdir_resume_at_removed_entry) +{ +/* The cgroup's own control files are listed alongside ours. */ +char names[128 ][NAME_MAX +1 ]; +long pos[128 ]; +char path[PATH_MAX ]; +struct dirent *de; +int n =0 , i, j; +DIR *d; + +for (i =0 ; i <RESUME_DIRS ; i++) { +snprintf (path,sizeof (path),"%s/e%02d" , self->scratch, i); +ASSERT_EQ (mkdir (path,0755 ),0 ); + } + +/* Record the cookie before reading each entry, with its name. */ + d =opendir (self->scratch); +ASSERT_NE (d,NULL ); +while (1 ) { +long here =telldir (d); + + de =readdir (d); +if (!de) +break ; +if (!strcmp (de->d_name,"." ) || !strcmp (de->d_name,".." )) +continue ; +ASSERT_LT (n, (int )ARRAY_SIZE (pos)); + pos[n] = here; +strncpy (names[n], de->d_name,NAME_MAX ); + names[n][NAME_MAX ] ='\0' ; + n++; + } +closedir (d); +ASSERT_GT (n,1 ); + +for (i =0 ; i < n; i++) { +/* Only the directories we made can be removed and put back. */ +if (strncmp (names[i],"e" ,1 )) +continue ; + +snprintf (path,sizeof (path),"%s/%s" , self->scratch, names[i]); +ASSERT_EQ (rmdir (path),0 ); + +/* Reopen so the seek has to reach the kernel. */ + d =opendir (self->scratch); +ASSERT_NE (d,NULL ); +seekdir (d, pos[i]); +while ((de =readdir (d))) { +if (!strcmp (de->d_name,"." ) || !strcmp (de->d_name,".." )) +continue ; +for (j =0 ; j < i; j++) + ASSERT_STRNE(de->d_name, names[j]) +TH_LOG ("resuming at %s (gone) went back to %s" , + names[i], names[j]); + } +closedir (d); + +ASSERT_EQ (mkdir (path,0755 ),0 ); + } + +for (i =0 ; i <RESUME_DIRS ; i++) { +snprintf (path,sizeof (path),"%s/e%02d" , self->scratch, i); +EXPECT_EQ (rmdir (path),0 ); + } +} + +#define CHURN_ROUNDS 400 +#define CHURN_BUFSZ 512/* small, so a listing takes several calls */ + +/* +* The files appear at the end of the enabling write and go at the start of +* the disabling one, so the window where they exist is the short one. +*/ +#define CHURN_DWELL_ON 2000 +#define CHURN_DWELL_OFF 200 + +struct kernfs_dirent64 { +unsigned long long d_ino; +long long d_off; +unsigned short d_reclen; +unsigned char d_type; +char d_name[]; +}; + +/* +* The same resume, but inside one getdents(2) call. rmdir(2) cannot reach +* that window because iterate_dir() holds the listed directory's i_rwsem +* for the whole listing; cgroup.subtree_control can, having no VFS +* operation on the names it adds and removes. The files that are not the +* controller's stay throughout, so each must appear exactly once. +* +* A stress test: it has not been seen to catch the ordering bug, and is +* here to keep the unlocked window under load for lockdep and KASAN. +*/ +TEST_F(kernfs_cgroup, readdir_resume_vs_internal_remove) +{ + char buf[CHURN_BUFSZ] __attribute__((aligned(8))); +char stable[128 ][NAME_MAX +1 ]; +int nstable =0 , i, r; +int withctl =0 , without =0 ; +int seen[128 ], status; +pid_t churner; +DIR *d; + +/* With the controller off, whatever is left is what must persist. */ +ASSERT_EQ (write_file (self->scratch_sc, self->disable),0 ); + d =opendir (self->child); +ASSERT_NE (d,NULL ); +for (;;) { +struct dirent *de =readdir (d); + +if (!de) +break ; +if (!strcmp (de->d_name,"." ) || !strcmp (de->d_name,".." )) +continue ; +ASSERT_LT (nstable, (int )ARRAY_SIZE (stable)); +strncpy (stable[nstable], de->d_name,NAME_MAX ); + stable[nstable][NAME_MAX ] ='\0' ; + nstable++; + } +closedir (d); +ASSERT_GT (nstable,0 ); + + churner =fork (); +ASSERT_GE (churner,0 ); +if (churner ==0 ) { +for (;;) { +if (write_file (self->scratch_sc, self->enable)) +_exit (10 ); +usleep (CHURN_DWELL_ON ); +if (write_file (self->scratch_sc, self->disable)) +_exit (11 ); +usleep (CHURN_DWELL_OFF ); + } + } + +for (r =0 ; r <CHURN_ROUNDS ; r++) { +int fd =open (self->child,O_RDONLY |O_DIRECTORY ); +int extra =0 ; +int n; + +ASSERT_GE (fd,0 ); +memset (seen,0 ,sizeof (seen)); + +while ((n =syscall (SYS_getdents64, fd, buf,sizeof (buf))) >0 ) { +int off =0 ; + +while (off < n) { +struct kernfs_dirent64 *de = (void *)(buf + off); +bool known = false; + + off += de->d_reclen; +for (i =0 ; i < nstable; i++) +if (!strcmp (de->d_name, stable[i])) { + seen[i]++; + known = true; + } +if (!known &&strcmp (de->d_name,"." ) && +strcmp (de->d_name,".." )) + extra++; + } + } +ASSERT_GE (n,0 ); +EXPECT_EQ (close (fd),0 ); + +if (extra) + withctl++; +else + without++; + +for (i =0 ; i < nstable; i++) + ASSERT_EQ(seen[i], 1) +TH_LOG ("round %d: %s seen %d times" , + r, stable[i], seen[i]); + } + +/* The churn must have been running, or the listings prove nothing. */ +EXPECT_EQ (kill (churner,SIGKILL ),0 ); +ASSERT_EQ (waitpid (churner, &status,0 ), churner); + ASSERT_TRUE(WIFSIGNALED(status) && WTERMSIG(status) == SIGKILL) +TH_LOG ("churner exited on its own: status %d" , status); + +/* +* They also have to have overlapped it. How much depends on the +* machine, so say the race could not be arranged rather than fail. +*/ +if (!withctl || !without) +SKIP (return,"listings did not span the churn: %d with, %d without" , + withctl, without); +} +/* * A telldir() cookie must resolve back to the same entry after seekdir(). * kernfs encodes the cookie as the node's name hash, so this covers
Signature · 1 line
2.53.0-Meta
Christian Brauner
maintainerQuoted 53 lines
kernfs_fop_readdir()takeskernfs_rwsemfor reading and holds it for the whole listing,dir_emit()included.dir_emit()copies into a userspace buffer, so it can fault, and under memory pressure that fault goes to reclaim.We are hitting this case on the Meta fleet very regularly. Many times it is below[1], a monitoring daemon. It walks the cgroup tree and faults on its own getdents(2) buffer with the lock held for reading.
below: page allocation stall for 120 secs: order:0, mode:0x140dca(GFP_HIGHUSER_MOVABLE|__GFP_ZERO|__GFP_COMP) nodemask=(null),cpuset=hostcritical.slice,mems_allowed=0 Call Trace: <TASK> dump_stack_lvl+0x5d/0x80 __alloc_frozen_pages_noprof+0x5f4d/0x6300 ? memcg_list_lru_alloc+0x73/0x320 ? ima_file_check+0xd0/0x7d0 vma_alloc_folio_noprof+0x145/0x560 handle_mm_fault+0x17c9/0x2720 ? find_vma+0x27/0x30 do_user_addr_fault+0x39f/0x6e0 exc_page_fault+0x8f/0x110 asm_exc_page_fault+0x22/0x30 RIP: 0010:filldir64+0xd7/0x1a0 [Code:/RSP:/RAX:..R15: register block elided] kernfs_fop_readdir+0x2de/0x420 iterate_dir+0x8c/0x1f0 __se_sys_getdents64+0x61/0xe0 ? copy_page_from_iter+0x860/0x860 do_syscall_64+0x6a/0x250 entry_SYSCALL_64_after_hwframe+0x4b/0x53 </TASK>
kernfs_rwsemis the lock every create, remove and rename in the hierarchy needs, and sysfs and cgroupfs have one per machine. More importantly, the userspace OOM killers such as systemd-oomd also traverse the cgroupfs hierarchy and can get stuck behind below, which is stuck in reclaim. They are supposed to relieve memory pressure on the system, but can get stuck themselves.The fix is to not hold the lock across
dir_emit(). Copy the name while the lock is held, drop the lock, emit, and take it again.That opens a window. With the lock dropped, the entry the listing stopped on can be removed or renamed before the listing picks up again, and readdir only remembers its place as a name hash. Two entries in one directory can share a hash, so the hash alone does not say where the listing stopped.
Patch 1 makes the fallback search land after the missing entry rather than on either side of it. Patch 2 keeps the name of the entry the listing is on and resumes on the full key. Patch 3 is the fix. Patch 4 adds tests.
Quoted 5 lines
Patch 1 is worth having on its own. Without the rest, the same search can land on the entry before the missing one and report it a second time, between two getdents(2) calls.
The series applies to the vfs-7.4.kernfs branch of the vfs tree.
Looks fine to me although it spaghettifies the code quite a bit. I'll pull it but wait for Tejun to give it a nod.
Quoted 8 lines · Shakeel Butt
kernfs: don't hold
kernfs_rwsemacrossdir_emit()
kernfs_fop_readdir()takeskernfs_rwsemfor reading and holds it for the whole listing,dir_emit()included.dir_emit()copies into a userspace buffer, so it can fault, and under memory pressure that fault goes to reclaim.[...]
Applied to the vfs-7.4.kernfs branch of the vfs/vfs.git tree. Patches in the vfs-7.4.kernfs branch should appear in linux-next soon.
Please report any outstanding bugs that were missed during review in a new review to the original patch series allowing us to drop it.
It's encouraged to provide Acked-bys and Reviewed-bys even though the patch has now been applied. If possible patch trailers will be updated.
Note that commit hashes shown below are subject to change due to rebase, trailer updates or similar. If in doubt, please check the listed branch.
tree: https://git.kernel.org/pub/scm/linux/kernel/git/vfs/vfs.git branch: vfs-7.4.kernfs
[1/4] kernfs: don't repeat or skip an entry when readdir resumes
https://git.kernel.org/vfs/vfs/c/5ebcb578adee [2/4] kernfs: resume readdir on the full entry key https://git.kernel.org/vfs/vfs/c/45126ed80bb1 [3/4] kernfs: don't hold kernfs_rwsem across dir_emit() https://git.kernel.org/vfs/vfs/c/341c65579de5 [4/4] selftests: cover readdir resuming at a removed entry https://git.kernel.org/vfs/vfs/c/a4354906005d