Eliminate Dying Memory Cgroup
Status from this message: [merged mm-stable] mm-lru-add-vm_warn_on_once_folio-to-lru-maintenance-helpers.patch removed from -mm tree
Get series with b4
b4 am [email protected]Versions newest first · linked by author and title
- v6 Latest33 patches 19 repliesApplied 33/33
- v532 patches 34 repliesSuperseded
- v431 patches 16 repliesSuperseded
Earlier versions · 3
| Version | Posted | Patches | Replies | Status |
|---|---|---|---|---|
| v3 | 30 patches | 60 replies | Superseded | |
| v2 | 28 patches | 130 replies | Superseded | |
| v1 | 26 patches | 77 replies | Superseded |
Changes in v6 from the cover letter
- refactor mod_memcg_state() and mod_memcg_lruvec_state() (suggested by Yosry Ahmed)
- use non-atomic method to reparent non-hierarchical stats (suggested by Yosry Ahmed)
- remove the redundant declaration in [PATCH v5 29/32] (pointed by Shakeel Butt)
- collect Acked-bys
Patches in v6 All 33 applied
Reviews trailers per patch
All 33 patches have a Reviewed-by or Acked-by.
| Patch | Roman Gushchin | Johannes Weiner | Harry Yoo | Chen Ridong | Shakeel Butt | David Hildenbrand (Red Hat) | Muchun Song | Nhat Pham | Chengming Zhou | Usama Arif |
|---|---|---|---|---|---|---|---|---|---|---|
| 1/33 | A-b | A-b | R-b | R-b | A-b | |||||
| 2/33 | A-b | R-b | A-b | |||||||
| 3/33 | A-b | A-b | R-b | R-b | A-b | A-b | ||||
| 4/33 | A-b | R-b | R-b | A-b | A-b | |||||
| 5/33 | A-b | R-b | A-b | |||||||
| 6/33 | A-b | R-b | R-b | A-b | ||||||
| 7/33 | A-b | R-b | A-b | |||||||
| 8/33 | R-b | A-b | ||||||||
| 9/33 | A-b | R-b | A-b | |||||||
| 10/33 | A-b | R-b | A-b | |||||||
| 11/33 | A-b | R-b | A-b | |||||||
| 12/33 | A-b | R-b | A-b | |||||||
| 13/33 | A-b | R-b | A-b | |||||||
| 14/33 | R-b | A-b | ||||||||
| 15/33 | A-b | R-b | A-b | |||||||
| 16/33 | A-b | R-b | A-b | |||||||
| 17/33 | A-b | R-b | A-b | A-b | A-b | |||||
| 18/33 | A-b | R-b | A-b | A-b | ||||||
| 19/33 | R-b | A-b | ||||||||
| 20/33 | A-b | R-b | A-b | A-b | R-b | |||||
| 21/33 | A-b | R-b | A-b | |||||||
| 22/33 | A-b | R-b | A-b | |||||||
| 23/33 | R-b | A-b | A-b | |||||||
| 24/33 | A-b | A-b | ||||||||
| 25/33 | A-b | R-b | A-b | A-b | ||||||
| 26/33 | A-b R-b | |||||||||
| 27/33 | A-b | R-b | A-b | R-b | ||||||
| 28/33 | A-b | A-b | ||||||||
| 29/33 | A-b | |||||||||
| 30/33 | A-b | |||||||||
| 31/33 | A-b | A-b | ||||||||
| 32/33 | A-b | |||||||||
| 33/33 | A-b | A-b | A-b |
Trailers given in replies are filled; those carried in the patches from earlier versions are outlined with dashes. Scroll the table sideways for all 10 reviewers.
Cover letter [PATCH v6 00/33] Eliminate Dying Memory Cgroup
From: Qi Zheng <[email protected]>
Changes in v6:
- refactor mod_memcg_state() and mod_memcg_lruvec_state() (suggested by Yosry Ahmed)
- use non-atomic method to reparent non-hierarchical stats (suggested by Yosry Ahmed)
- remove the redundant declaration in [PATCH v5 29/32] (pointed by Shakeel Butt)
- collect Acked-bys
Changes in v5:
- fix build error in [PATCH v4 24/31] (reported by kernel test robot)
- move the declaration of reparent_memcg_state_local() and reparent_memcg_lruvec_state_local() into mm/memcontrol-v1.h file. (suggested by Shakeel Butt, and due to some function dependencies, I keep reparent_memcg_lruvec_state_local() in place)
- convert objcg to be per-memcg per-node type
- hold the per-node lru locks to reparent per-node objcg and lru folios, which fix the potential lockdep warning
(pointed by Usama Arif and suggested by Shakeel Butt)
- use try_charge_memcg() directly in charge_memcg()
(suggested by Shakeel Butt)
- collect Acked-bys
- rebase onto the next-20260224
Changes in v4:
- fix commit message in [PATCH v3 23/30] (pointed by Baoquan He)
- move lruvec_lock_irq() and firends to mm/memcontrol.c to fix the compilation error in [PATCH v4 24/31] (reported by LKP)
- include parent_lruvec() within the RCU lock in lru_note_cost_unlock_irq() in [PATCH v4 24/31] (pointed by Harry Yoo)
- move the declaration of lru_reparent_memcg() to swap.h (suggested by Muchun Song)
- fix lru size update logic in lru_gen_reparent_memcg() in [PATCH v4 26/31] (pointed and suggested by Harry Yoo)
- add [PATCH v4 28/31] to use lruvec_lru_size() to get the number of lru pages in count_shadow_nodes() (suggested by Shakeel Butt)
- fix reparenting logic of lruvec_stats->state_local in [PATCH v4 29/31] (pointed by Shakeel Butt)
- change these non-hierarchical stats to atomic_long_t type to avoid race between mem_cgroup_stat_aggregate() and reparent_state_local() in [PATCH v4 29/31]
- make css_killed_work_fn() to be called in rcu work, and use rcu lock + CSS_IS_DYING check to avoid race between mod_memcg_state()/mod_memcg_lruvec_state()
(suggested by Shakeel Butt)
- collect Acked-bys and Reviewed-bys
- rebase onto the next-20260128
Changes in v3:
- modify the commit message in [PATCH v2 04/28], [PATCH v2 06/28], [PATCH v2 13/28], [PATCH v2 24/28] and [PATCH v2 27/28] (suggested by David Hildenbrand, Chen Ridong and Johannes Weiner)
- change code style in [PATCH v3 8/30], [PATCH v3 15/30] and [PATCH v3 27/30] (suggested by Johannes Weiner and Shakeel Butt)
- use get_mem_cgroup_from_folio() + mem_cgroup_put() to replace holding rcu lock in [PATCH v3 14/30] and [PATCH v3 19/30]
(pointed by Johannes Weiner)
- add a comment to folio_split_queue_lock() in [PATCH v3 17/30] (suggested by Shakeel Butt)
- modify the comment above folio_lruvec() in [PATCH v3 24/30] (suggested by Johannes Weiner)
- fix rcu lock issue in lru_note_cost_refault()
(pointed by Shakeel Butt)
- add [PATCH v3 28/30] to fix non-hierarchical memcg1_stats issues (pointed by Yosry Ahmed)
- fix lru_zone_size issue in [PATCH v2 24/28] and [PATCH v2 25/28]
- collect Acked-bys and Reviewed-bys
- rebase onto the next-20260114
Changes in v2:
- add [PATCH v2 04/28] and remove local_irq_disable() in evict_folios() (pointed by Harry Yoo)
- recheck objcg in [PATCH v2 07/28] (pointed by Harry Yoo)
- modify the commit message in [PATCH v2 12/28] and [PATCH v2 21/28] (pointed by Harry Yoo)
- use rcu lock to protect mm_state in [PATCH v2 14/28] (pointed by Harry Yoo)
- fix bad unlock balance warning in [PATCH v2 23/28]
- change nr_pages type to long in [PATCH v2 25/28] (pointed by Harry Yoo)
- incease mm_state->seq during reparenting to make mm walker work properly in [PATCH v2 25/28] (pointed by Harry Yoo)
- add [PATCH v2 18/28] to fix WARNING in folio_memcg() (pointed by Harry Yoo)
- collect Reviewed-bys
- rebase onto the next-20251216
Changes in v1:
- drop [PATCH RFC 02/28]
- drop THP split queue related part, which has been merged as a separate patchset[2]
- prevent memory cgroup release in folio_split_queue_lock{_irqsave}() in [PATCH v1 16/26]
- Separate the reparenting function of traditional LRU folios to [PATCH v1 22/26]
- adapted to the MGLRU scenarios in [PATCH v1 23/26]
- refactor memcg_reparent_objcgs() in [PATCH v1 24/26]
- collect Acked-bys and Reviewed-bys
- rebase onto the next-20251028
Hi all,
Introduction
============
This patchset is intended to transfer the LRU pages to the object cgroup without holding a reference to the original memory cgroup in order to address the issue of the dying memory cgroup. A consensus has already been reached regarding this approach recently [1].
Background
==========
The issue of a dying memory cgroup refers to a situation where a memory cgroup is no longer being used by users, but memory (the metadata associated with memory cgroups) remains allocated to it. This situation may potentially result in memory leaks or inefficiencies in memory reclamation and has persisted as an issue for several years. Any memory allocation that endures longer than the lifespan (from the users' perspective) of a memory cgroup can lead to the issue of dying memory cgroup. We have exerted greater efforts to tackle this problem by introducing the infrastructure of object cgroup [2].
Presently, numerous types of objects (slab objects, non-slab kernel allocations, per-CPU objects) are charged to the object cgroup without holding a reference to the original memory cgroup. The final allocations for LRU pages (anonymous pages and file pages) are charged at allocation time and continues to hold a reference to the original memory cgroup until reclaimed.
File pages are more complex than anonymous pages as they can be shared among different memory cgroups and may persist beyond the lifespan of the memory cgroup. The long-term pinning of file pages to memory cgroups is a widespread issue that causes recurring problems in practical scenarios [3]. File pages remain unreclaimed for extended periods. Additionally, they are accessed by successive instances (second, third, fourth, etc.) of the same job, which is restarted into a new cgroup each time. As a result, unreclaimable dying memory cgroups accumulate, leading to memory wastage and significantly reducing the efficiency of page reclamation.
Fundamentals
============
A folio will no longer pin its corresponding memory cgroup. It is necessary to ensure that the memory cgroup or the lruvec associated with the memory cgroup is not released when a user obtains a pointer to the memory cgroup or lruvec returned by folio_memcg() or folio_lruvec(). Users are required to hold the RCU read lock or acquire a reference to the memory cgroup associated with the folio to prevent its release if they are not concerned about the binding stability between the folio and its corresponding memory cgroup. However, some users of folio_lruvec() (i.e., the lruvec lock) desire a stable binding between the folio and its corresponding memory cgroup. An approach is needed to ensure the stability of the binding while the lruvec lock is held, and to detect the situation of holding the incorrect lruvec lock when there is a race condition during memory cgroup reparenting. The following four steps are taken to achieve these goals.
1. The first step to be taken is to identify all users of both functions (folio_memcg() and folio_lruvec()) who are not concerned about binding stability and implement appropriate measures (such as holding a RCU read lock or temporarily obtaining a reference to the memory cgroup for a brief period) to prevent the release of the memory cgroup.
2. Secondly, the following refactoring of folio_lruvec_lock() demonstrates how to ensure the binding stability from the user's perspective of folio_lruvec().
struct lruvec *folio_lruvec_lock(struct folio *folio)
{
struct lruvec *lruvec; rcu_read_lock();
retry:
lruvec = folio_lruvec(folio);
spin_lock(&lruvec->lru_lock);
if (unlikely(lruvec_memcg(lruvec) != folio_memcg(folio))) {
spin_unlock(&lruvec->lru_lock);
goto retry;
}
return lruvec;}
From the perspective of memory cgroup removal, the entire reparenting process (altering the binding relationship between folio and its memory cgroup and moving the LRU lists to its parental memory cgroup) should be carried out under both the lruvec lock of the memory cgroup being removed and the lruvec lock of its parent.
3. Finally, transfer the LRU pages to the object cgroup without holding a reference to the original memory cgroup.
Effect
======
Finally, it can be observed that the quantity of dying memory cgroups will not experience a significant increase if the following test script is executed to reproduce the issue.
```bash
#!/bin/bash
# Create a temporary file 'temp' filled with zero bytes
dd if=/dev/zero of=temp bs=4096 count=1
# Display memory-cgroup info from /proc/cgroups
cat /proc/cgroups | grep memory
for i in {0..2000}
do
mkdir /sys/fs/cgroup/memory/test$i
echo $$ > /sys/fs/cgroup/memory/test$i/cgroup.procs
# Append 'temp' file content to 'log'
cat temp >> log
echo $$ > /sys/fs/cgroup/memory/cgroup.procs
# Potentially create a dying memory cgroup
rmdir /sys/fs/cgroup/memory/test$i
done
# Display memory-cgroup info after test
cat /proc/cgroups | grep memory
rm -f temp log
```Comments and suggestions are welcome!
Thanks,
Qi
[1].https://lore.kernel.org/linux-mm/Z6OkXXYDorPrBvEQ@hm-sls2/ [2].https://lwn.net/Articles/895431/ [3].https://github.com/systemd/systemd/pull/36827
Muchun Song (22):
mm: memcontrol: remove dead code of checking parent memory cgroup
mm: workingset: use folio_lruvec() in workingset_refault()
mm: rename unlock_page_lruvec_irq and its variants
mm: vmscan: refactor move_folios_to_lru()
mm: memcontrol: allocate object cgroup for non-kmem case
mm: memcontrol: return root object cgroup for root memory cgroup
mm: memcontrol: prevent memory cgroup release in
get_mem_cgroup_from_folio()
buffer: prevent memory cgroup release in folio_alloc_buffers()
writeback: prevent memory cgroup release in writeback module
mm: memcontrol: prevent memory cgroup release in
count_memcg_folio_events()
mm: page_io: prevent memory cgroup release in page_io module
mm: migrate: prevent memory cgroup release in folio_migrate_mapping()
mm: mglru: prevent memory cgroup release in mglru
mm: memcontrol: prevent memory cgroup release in
mem_cgroup_swap_full()
mm: workingset: prevent memory cgroup release in lru_gen_eviction()
mm: workingset: prevent lruvec release in workingset_refault()
mm: zswap: prevent lruvec release in zswap_folio_swapin()
mm: swap: prevent lruvec release in lru_gen_clear_refs()
mm: workingset: prevent lruvec release in workingset_activation()
mm: memcontrol: prepare for reparenting LRU pages for lruvec lock
mm: memcontrol: eliminate the problem of dying memory cgroup for LRU
folios
mm: lru: add VM_WARN_ON_ONCE_FOLIO to lru maintenance helpers
Qi Zheng (11):
mm: vmscan: prepare for the refactoring the move_folios_to_lru()
mm: thp: prevent memory cgroup release in
folio_split_queue_lock{_irqsave}()
mm: zswap: prevent memory cgroup release in zswap_compress()
mm: do not open-code lruvec lock
mm: vmscan: prepare for reparenting traditional LRU folios
mm: vmscan: prepare for reparenting MGLRU folios
mm: memcontrol: refactor memcg_reparent_objcgs()
mm: workingset: use lruvec_lru_size() to get the number of lru pages
mm: memcontrol: refactor mod_memcg_state() and
mod_memcg_lruvec_state()
mm: memcontrol: prepare for reparenting non-hierarchical stats
mm: memcontrol: convert objcg to be per-memcg per-node type
fs/buffer.c | 4 +- fs/fs-writeback.c | 22 +- include/linux/memcontrol.h | 189 +++++----- include/linux/mm_inline.h | 6 + include/linux/mmzone.h | 17 + include/linux/sched.h | 2 +- include/linux/swap.h | 25 +- include/trace/events/writeback.h | 3 + kernel/cgroup/cgroup.c | 9 +- mm/compaction.c | 43 ++- mm/huge_memory.c | 22 +- mm/memcontrol-v1.c | 31 +- mm/memcontrol-v1.h | 7 + mm/memcontrol.c | 585 ++++++++++++++++++++----------- mm/migrate.c | 2 + mm/mlock.c | 2 +- mm/page_io.c | 8 +- mm/percpu.c | 2 +- mm/shrinker.c | 6 +- mm/swap.c | 59 +++- mm/vmscan.c | 273 +++++++++++---- mm/workingset.c | 30 +- mm/zswap.c | 5 + 23 files changed, 917 insertions(+), 435 deletions(-)
Signature · 1 line
2.20.1