make unused huge shrinker memcg aware
Subsystems (3)
v3 cover make unused huge shrinker memcg aware
From: Qi Zheng <[email protected]>
Changes in v3:
- add a fix patch to fix missed removal of super_fs_objects_eligible()
- move the original shrinklist addition logic after all checks are completed, and split it into a separate patch. (suggested by Baolin)
- simplify the shmem_unused_huge_requeue() (suggested by Baolin)
- keep the move_back label in shmem_unused_huge_shrink() (suggested by Baolin)
- rebase onto the next-20260731
Changes in v2:
- temporarily add the dependent patch from Usama to the series for review convenience
- remove shrinklist_scan and shrinklist_isolated from struct shmem_inode_info, and re-implement the logic by resuing the same info->shrinklist (suggested by Baolin)
- add more comments (suggested by Andrew)
- fix missing initialization of info->shrinklist_memcg (pointed by sashiko)
- rebase onto the next-20260717
Qi Zheng (3):
fs: fix missed removal of super_fs_objects_eligible()
mm: shmem: move unused huge shrinklist queuing past the truncation
check
mm: shmem: make unused huge shrinker memcg aware
fs/super.c | 18 +- include/linux/shmem_fs.h | 12 +- mm/shmem.c | 361 +++++++++++++++++++++++++++++---------- 3 files changed, 283 insertions(+), 108 deletions(-)
Signature · 1 line
2.54.0
PATCH v3 1/3 fs: fix missed removal of super_fs_objects_eligible()
From: Qi Zheng <[email protected]>
Commit 0ef8faff490be ("fs: push nr_cached_objects memcg gating into individual filesystems") forgot to drop the super_fs_objects_eligible(), just fix it.
--- fs/super.c | 18 ++---------------- 1 file changed, 2 insertions(+), 16 deletions(-)
fs/super.c +2 -16
@@ -171,19 +171,6 @@ static void super_wake(struct super_block *sb, unsigned int flag)wake_up_var (&sb->s_flags); } -/* -* The s_op->nr_cached_objects hooks (used for example by btrfs and xfs) -* operate on filesystem-global state and ignore sc->memcg. Driving them -* from per-memcg shrink_slab_memcg() invocations only burns CPU walking -* per-cpu counters and queueing duplicate work: the actual reclaim happens on -* the global path (kswapd or root direct reclaim) regardless. Restrict them -* to that path. -*/ -static inline bool super_fs_objects_eligible (struct shrink_control *sc) -{ -return !sc->memcg ||mem_cgroup_is_root (sc->memcg); -} -/* * One thing we have to be careful of with a per-sb shrinker is that we don't * drop the last active reference to the superblock from within the shrinker.
@@ -213,7 +200,7 @@ static unsigned long super_cache_scan(struct shrinker *shrink,if (!super_trylock_shared (sb))return SHRINK_STOP ; -if (sb->s_op->nr_cached_objects &&super_fs_objects_eligible (sc)) +if (sb->s_op->nr_cached_objects) fs_objects = sb->s_op->nr_cached_objects (sb, sc); inodes =list_lru_shrink_count (&sb->s_inode_lru, sc);
@@ -274,8 +261,7 @@ static unsigned long super_cache_count(struct shrinker *shrink,return 0 ;smp_rmb (); -if (sb->s_op && sb->s_op->nr_cached_objects && -super_fs_objects_eligible (sc)) +if (sb->s_op && sb->s_op->nr_cached_objects) total_objects = sb->s_op->nr_cached_objects (sb, sc); total_objects +=list_lru_shrink_count (&sb->s_dentry_lru, sc);
Signature · 1 line
2.54.0
Andrew Morton
maintainerRe: [PATCH v3 1/3] fs: fix missed removal of super_fs_objects_eligible()
Quoted 3 lines · On Mon, 3 Aug 2026 16:46:33 +0800 Qi Zheng <[email protected]> wrote:
Commit
0ef8faff490be("fs: pushnr_cached_objectsmemcg gating into individual filesystems") forgot to drop thesuper_fs_objects_eligible(), just fix it.
What are the userspace-visible runtime effects of this bug? Please include this info with all bugfix patches.
Should we cc:stable?
Thanks.
Re: [PATCH v3 1/3] fs: fix missed removal of super_fs_objects_eligible()
Quoted 8 lines · Andrew Morton
Quoted 3 lines · On Mon, 3 Aug 2026 16:46:33 +0800 Qi Zheng <[email protected]> wrote:
Commit
0ef8faff490be("fs: pushnr_cached_objectsmemcg gating into individual filesystems") forgot to drop thesuper_fs_objects_eligible(), just fix it.What are the userspace-visible runtime effects of this bug? Please include this info with all bugfix patches.
It's seems this part was missed during the push to the linux-next
branch, see https://lore.kernel.org/all/[email protected]/.
I expect Christian to help fix this. Including this patch in the current series is just to provide a complete picture and make reviewing easier.
Thanks,
Qi
Quoted 4 lines
Should we cc:stable?
Thanks.
PATCH v3 2/3 mm: shmem: move unused huge shrinklist queuing past the truncation check
In reply to Qi ZhengFrom: Qi Zheng <[email protected]>
The shmem_get_folio_gfp() adds the inode to the unused huge shrinker list at the alloced label, but a subsequent truncation check may still fail and remove the folio, leaving the inode on the list with a stale folio.
The original code works because the shrinker re-looks-up the folio and drops stale entries, but it is cleaner to queue the inode only after all checks that might remove the folio have passed.
So just make the pure structural move with no functional change, and it serves as preparation for the memcg-aware shrinker conversion.
--- mm/shmem.c | 48 +++++++++++++++++++++++++++--------------------- 1 file changed, 27 insertions(+), 21 deletions(-)
mm/shmem.c +27 -21
@@ -2535,27 +2535,6 @@ static int shmem_get_folio_gfp(struct inode *inode, pgoff_t index, alloced: alloced = true; -if (folio_test_large (folio) && -DIV_ROUND_UP (i_size_read (inode),PAGE_SIZE ) < -folio_next_index (folio)) { -struct shmem_sb_info *sbinfo =SHMEM_SB (inode->i_sb); -struct shmem_inode_info *info =SHMEM_I (inode); -/* -* Part of the large folio is beyond i_size: subject -* to shrink under memory pressure. -*/ -spin_lock (&sbinfo->shrinklist_lock); -/* -* _careful to defend against unlocked access to -* ->shrink_list in shmem_unused_huge_shrink() -*/ -if (list_empty_careful (&info->shrinklist)) { -list_add_tail (&info->shrinklist, - &sbinfo->shrinklist); - sbinfo->shrinklist_len++; - } -spin_unlock (&sbinfo->shrinklist_lock); - }if (sgp ==SGP_WRITE )folio_set_referenced (folio);
@@ -2585,6 +2564,33 @@ static int shmem_get_folio_gfp(struct inode *inode, pgoff_t index, error = -EINVAL ; goto unlock; } + +/* +* Queue the inode on the shrink list only after all checks that might +* remove the folio have passed. Otherwise the inode could be left on +* the shrinker list with a stale folio. +*/ +if (alloced &&folio_test_large (folio) && +DIV_ROUND_UP (i_size_read (inode),PAGE_SIZE ) <folio_next_index (folio)) { +struct shmem_sb_info *sbinfo =SHMEM_SB (inode->i_sb); +struct shmem_inode_info *info =SHMEM_I (inode); +/* +* Part of the large folio is beyond i_size: subject +* to shrink under memory pressure. +*/ +spin_lock (&sbinfo->shrinklist_lock); +/* +* _careful to defend against unlocked access to +* ->shrink_list in shmem_unused_huge_shrink() +*/ +if (list_empty_careful (&info->shrinklist)) { +list_add_tail (&info->shrinklist, + &sbinfo->shrinklist); + sbinfo->shrinklist_len++; + } +spin_unlock (&sbinfo->shrinklist_lock); + } + out: *foliop = folio;return 0 ;
Signature · 1 line
2.54.0
Re: [PATCH v3 2/3] mm: shmem: move unused huge shrinklist queuing past the truncation check
Quoted 6 lines · Qi Zheng
From: Qi Zheng <[email protected]>
The
shmem_get_folio_gfp()adds the inode to the unused huge shrinker list at the alloced label, but a subsequent truncation check may still fail and remove the folio, leaving the inode on the list with a stale folio.
Quoted 8 lines
The original code works because the shrinker re-looks-up the folio and drops stale entries, but it is cleaner to queue the inode only after all checks that might remove the folio have passed.
So just make the pure structural move with no functional change, and it serves as preparation for the memcg-aware shrinker conversion.
Signed-off-by Qi Zheng <[email protected]>
missed adding:
Re: [PATCH v3 2/3] mm: shmem: move unused huge shrinklist queuing past the truncation check
In reply to Qi ZhengQuoted 10 lines · Qi Zheng
From: Qi Zheng <[email protected]>
The
shmem_get_folio_gfp()adds the inode to the unused huge shrinker list at the alloced label, but a subsequent truncation check may still fail and remove the folio, leaving the inode on the list with a stale folio.The original code works because the shrinker re-looks-up the folio and drops stale entries, but it is cleaner to queue the inode only after all checks that might remove the folio have passed.
Quoted 5 lines
So just make the pure structural move with no functional change, and it serves as preparation for the memcg-aware shrinker conversion.
Signed-off-by Qi Zheng <[email protected]>---
LGTM. Thanks.
PATCH v3 3/3 mm: shmem: make unused huge shrinker memcg aware
In reply to Qi ZhengFrom: Qi Zheng <[email protected]>
The shmem unused huge shrinker keeps a per-superblock list of inodes whose tail huge folio extends beyond i_size. Since that list is not memcg aware, reclaim triggered by one memcg can scan inodes from the whole superblock and split shmem huge folios charged to unrelated memcgs.
Convert the shrink list to a memcg-aware list_lru. Queue each inode on the list_lru sublist matching the memcg and node of the current tail huge folio, so non-root memcg reclaim only walks candidates charged to the reclaiming memcg. Global reclaim, root memcg reclaim and shmem quota reclaim keep global semantics.
The list_lru still tracks inodes while the actual split target is the current tail huge folio, so validate the folio memcg/node during scan. If the folio no longer matches the reclaim context or splitting cannot proceed, requeue the inode according to the current tail folio; if the inode is no longer shrinkable, drop the scan entry.
This can be tested with the shrinker debugfs interface by allocating 32 tmpfs tail THPs in each of two memcgs, then scanning the sb-tmpfs shrinker with memcg A's cgroup id:
before A scan after A scan base A=64M, B=64M A=64M, B=64M (per-memcg count is skipped) patched A=64M, B=64M A=0, B=64M
--- include/linux/shmem_fs.h | 12 +- mm/shmem.c | 341 +++++++++++++++++++++++++++++---------- 2 files changed, 268 insertions(+), 85 deletions(-)
include/linux/shmem_fs.h +9 -3
@@ -11,6 +11,7 @@#include <linux/fs_parser.h> #include <linux/userfaultfd_k.h> #include <linux/bits.h> +#include <linux/list_lru.h> /* inode in-kernel data */
@@ -54,6 +55,11 @@ struct shmem_inode_info {struct dquot __rcu *i_dquot[MAXQUOTAS ];#endif struct inode vfs_inode; + +#ifdef CONFIG_TRANSPARENT_HUGEPAGE +struct mem_cgroup *shrinklist_memcg; +int shrinklist_nid; +#endif };#define SHMEM_FL_USER_VISIBLE (FS_FL_USER_VISIBLE | FS_CASEFOLD_FL)
@@ -83,9 +89,9 @@ struct shmem_sb_info {ino_t next_ino;/* The next per-sb inode number to use */ ino_t __percpu *ino_batch;/* The next per-cpu inode number to use */ struct mempolicy *mpol;/* default memory policy for mappings */ -spinlock_t shrinklist_lock;/* Protects shrinklist */ -struct list_head shrinklist;/* List of shinkable inodes */ -unsigned long shrinklist_len;/* Length of shrinklist */ +#ifdef CONFIG_TRANSPARENT_HUGEPAGE +struct list_lru shrinklist;/* List of shrinkable inodes */ +#endif struct shmem_quota_limits qlimits;/* Default quota limits */ struct simple_xattr_cache xa_cache; };
mm/shmem.c +259 -82
445 lines not on this page · raw message
Signature · 1 line
2.54.0
Baolin Wang
reviewerRe: [PATCH v3 3/3] mm: shmem: make unused huge shrinker memcg aware
Quoted 23 lines · Qi Zheng
From: Qi Zheng <[email protected]>
The shmem unused huge shrinker keeps a per-superblock list of inodes whose tail huge folio extends beyond
i_size. Since that list is not memcg aware, reclaim triggered by one memcg can scan inodes from the whole superblock and split shmem huge folios charged to unrelated memcgs.Convert the shrink list to a memcg-aware
list_lru. Queue each inode on thelist_lrusublist matching the memcg and node of the current tail huge folio, so non-root memcg reclaim only walks candidates charged to the reclaiming memcg. Global reclaim, root memcg reclaim and shmem quota reclaim keep global semantics.The
list_lrustill tracks inodes while the actual split target is the current tail huge folio, so validate the folio memcg/node during scan. If the folio no longer matches the reclaim context or splitting cannot proceed, requeue the inode according to the current tail folio; if the inode is no longer shrinkable, drop the scan entry.This can be tested with the shrinker debugfs interface by allocating 32 tmpfs tail THPs in each of two memcgs, then scanning the sb-tmpfs shrinker with memcg A's cgroup id:
Quoted 6 lines
before A scan after A scan base A=64M, B=64M A=64M, B=64M (per-memcg count is skipped) patched A=64M, B=64M A=0, B=64MSigned-off-by Qi Zheng <[email protected]>---
Apart from the two issues Sashiko pointed out (which you've already replied to), nothing else jumped out at me. Thanks.
Re: [PATCH v3 3/3] mm: shmem: make unused huge shrinker memcg aware
Quoted 26 lines · Baolin Wang
Quoted 23 lines · Qi Zheng
From: Qi Zheng <[email protected]>
The shmem unused huge shrinker keeps a per-superblock list of inodes whose tail huge folio extends beyond
i_size. Since that list is not memcg aware, reclaim triggered by one memcg can scan inodes from the whole superblock and split shmem huge folios charged to unrelated memcgs.Convert the shrink list to a memcg-aware
list_lru. Queue each inode on thelist_lrusublist matching the memcg and node of the current tail huge folio, so non-root memcg reclaim only walks candidates charged to the reclaiming memcg. Global reclaim, root memcg reclaim and shmem quota reclaim keep global semantics.The
list_lrustill tracks inodes while the actual split target is the current tail huge folio, so validate the folio memcg/node during scan. If the folio no longer matches the reclaim context or splitting cannot proceed, requeue the inode according to the current tail folio; if the inode is no longer shrinkable, drop the scan entry.This can be tested with the shrinker debugfs interface by allocating 32 tmpfs tail THPs in each of two memcgs, then scanning the sb-tmpfs shrinker with memcg A's cgroup id:
Quoted 8 lines
Quoted 6 lines
before A scan after A scan
base A=64M, B=64M A=64M, B=64M (per-memcg count is skipped) patched A=64M, B=64M A=0, B=64MSigned-off-by Qi Zheng <[email protected]>---
Apart from the two issues Sashiko pointed out (which you've already replied to), nothing else jumped out at me. Thanks.
Thanks for the review! I will fix those two issues as I replied, and then send out v4.
Quoted 1 line · Qi Zheng
From: Qi Zheng <[email protected]>
I'm missing description and motivation here.
This is only about shrinking huge pages that span end of shmem files.
Is this really a problem? And if so, why?
Signature · 3 lines
Cheers, David
Hi David,
Quoted 4 lines · David Hildenbrand (Arm)
Quoted 1 line · Qi Zheng
From: Qi Zheng <[email protected]>
I'm missing description and motivation here.
My bad, since v1 was just a single patch, I got lazy and didn't bother adding a cover letter description later on.
Quoted 4 lines
This is only about shrinking huge pages that span end of shmem files.
Is this really a problem? And if so, why?
Yes, this is a real-world problem that we encountered in production.
The root cause is that shmem unused shrinker used to be non-memcg-aware. This could lead to a scenario where reclaim triggered by one memcg A reclaims the shmem of another memcg B, causing unexpected impact on it.
Even worse, memcg A might have no reclaimable shmem at all, making this completely useless work and incurring some performance overhead.
such as:
tid 11340 comm scanner locked a page for 182264 us! kstack:
unlock_page+1
split_huge_page_to_list+3135
shmem_unused_huge_shrink+767
super_cache_scan+329
do_shrink_slab+291
shrink_slab+533
shrink_node+400
do_try_to_free_pages+206
try_to_free_mem_cgroup_pages+262
try_charge_memcg+591
mem_cgroup_charge+136
__handle_mm_fault+2431
handle_mm_fault+194
do_user_addr_fault+462
__do_page_fault+176
do_page_fault+48
page_fault+62Later, with Usama's patch [1], the shmem unused shrinker is no longer triggered during memcg-level reclaim. But this actually doesn't make sense either, since we can clearly just reclaim from memcg A individuallty.
[1]. https://lore.kernel.org/all/[email protected]/
Thanks,
Qi
Quoted 1 line
Andrew Morton
maintainerQuoted 1 line · On Mon, 3 Aug 2026 16:46:32 +0800 Qi Zheng <[email protected]> wrote:
What David said.
Also, AI review might have found a couple of things:
https://sashiko.dev/#/patchset/[email protected]
Hi Andrew,
Quoted 8 lines · Andrew Morton
Quoted 1 line · On Mon, 3 Aug 2026 16:46:32 +0800 Qi Zheng <[email protected]> wrote:
What David said.
Also, AI review might have found a couple of things:
https://sashiko.dev/#/patchset/[email protected]
I saw it, and it sounds reasonable.
Quoted 11 lines
Review
Quoted 10 lines
(quoted diff) +10
-0+static struct mem_cgroup * +shmem_unused_huge_alloc_lru(struct shmem_sb_info *sbinfo, struct folio *folio, + gfp_t gfp) +{ + struct mem_cgroup *memcg; + int ret; + + memcg = get_mem_cgroup_from_folio(folio); + if (!memcg) + return NULL;Could this leak the
mem_cgroupstructure if a file is truncated?
Quoted 8 lines
When
shmem_unused_huge_add()calls this function, it stores the memcg reference ininfo->shrinklist_memcg.
If an unprivileged user truncates the file to 0 bytes,shmem_truncate_range()frees the folio, but it does not remove the inode from the shrinklist or drop this memcg reference.
Without memory pressure to trigger the shrinker, themem_cgroupmight remain pinned indefinitely until the file is deleted, potentially allowing zombie memcgs to accumulate on the system.
Indeed, will also call shmem_unused_huge_del() in shmem_truncate_range() to take the inode off the shrinklist in time.
Quoted 18 lines
[ ... ]
Quoted 17 lines
(quoted diff) +17
-0+static bool is_shmem_unused_huge_match(struct folio *folio, + struct shrink_control *sc) +{ + struct mem_cgroup *memcg = NULL; + bool match; + + /* + * Only non-root memcg reclaim needs to match the folio charge against + * sc->memcg. Skip the folio memcg check for the following cases: + * 1. shmem quota reclaim (sc == NULL) + * 2. global shrinker reclaim + * 3. root memcg reclaim + */ + if (!sc || !sc->memcg || mem_cgroup_is_root(sc->memcg)) + return true; + + if (folio_nid(folio) != sc->nid)
Quoted 8 lines
Review
Quoted 1 line
(quoted diff) +1
-0+ return false;Does this early return bypass the NUMA node check for global and root memcg reclaim?
The comment indicates this is only intended to skip the folio memcg check. However, returning true immediately also bypasses thesc->nidcheck. If a huge folio is migrated to a different NUMA node, kswapd or direct reclaim might incorrectly split folios on the wrong node instead of requeuing the inode to the correct node's list.
Indeed, root memcg reclaim also needs to check if the nid matches.
Will move the nid check earlier in v4:
if (!sc)
return true;
if (folio_nid(folio) != sc->nid)
return false;
if (!sc->memcg || mem_cgroup_is_root(sc->memcg))
return true;Thanks,
Qi