Eliminate Dying Memory Cgroup
Subsystems (18)
Qi Zheng
reviewerv5 cover Eliminate Dying Memory Cgroup
From: Qi Zheng <[email protected]>
Changes in v5:
- fix build error in [PATCH v4 24/31] (reported by kernel test robot)
- move the declaration of reparent_memcg_state_local() and reparent_memcg_lruvec_state_local() into mm/memcontrol-v1.h file. (suggested by Shakeel Butt, and due to some function dependencies, I keep reparent_memcg_lruvec_state_local() in place)
- convert objcg to be per-memcg per-node type
- hold the per-node lru locks to reparent per-node objcg and lru folios, which fix the potential lockdep warning
(pointed by Usama Arif and suggested by Shakeel Butt)
- use try_charge_memcg() directly in charge_memcg()
(suggested by Shakeel Butt)
- collect Acked-bys
- rebase onto the next-20260224
Changes in v4:
- fix commit message in [PATCH v3 23/30] (pointed by Baoquan He)
- move lruvec_lock_irq() and firends to mm/memcontrol.c to fix the compilation error in [PATCH v4 24/31] (reported by LKP)
- include parent_lruvec() within the RCU lock in lru_note_cost_unlock_irq() in [PATCH v4 24/31] (pointed by Harry Yoo)
- move the declaration of lru_reparent_memcg() to swap.h (suggested by Muchun Song)
- fix lru size update logic in lru_gen_reparent_memcg() in [PATCH v4 26/31] (pointed and suggested by Harry Yoo)
- add [PATCH v4 28/31] to use lruvec_lru_size() to get the number of lru pages in count_shadow_nodes() (suggested by Shakeel Butt)
- fix reparenting logic of lruvec_stats->state_local in [PATCH v4 29/31] (pointed by Shakeel Butt)
- change these non-hierarchical stats to atomic_long_t type to avoid race between mem_cgroup_stat_aggregate() and reparent_state_local() in [PATCH v4 29/31]
- make css_killed_work_fn() to be called in rcu work, and use rcu lock + CSS_IS_DYING check to avoid race between mod_memcg_state()/mod_memcg_lruvec_state()
(suggested by Shakeel Butt)
- collect Acked-bys and Reviewed-bys
- rebase onto the next-20260128
Changes in v3:
- modify the commit message in [PATCH v2 04/28], [PATCH v2 06/28], [PATCH v2 13/28], [PATCH v2 24/28] and [PATCH v2 27/28] (suggested by David Hildenbrand, Chen Ridong and Johannes Weiner)
- change code style in [PATCH v3 8/30], [PATCH v3 15/30] and [PATCH v3 27/30] (suggested by Johannes Weiner and Shakeel Butt)
- use get_mem_cgroup_from_folio() + mem_cgroup_put() to replace holding rcu lock in [PATCH v3 14/30] and [PATCH v3 19/30]
(pointed by Johannes Weiner)
- add a comment to folio_split_queue_lock() in [PATCH v3 17/30] (suggested by Shakeel Butt)
- modify the comment above folio_lruvec() in [PATCH v3 24/30] (suggested by Johannes Weiner)
- fix rcu lock issue in lru_note_cost_refault()
(pointed by Shakeel Butt)
- add [PATCH v3 28/30] to fix non-hierarchical memcg1_stats issues (pointed by Yosry Ahmed)
- fix lru_zone_size issue in [PATCH v2 24/28] and [PATCH v2 25/28]
- collect Acked-bys and Reviewed-bys
- rebase onto the next-20260114
Changes in v2:
- add [PATCH v2 04/28] and remove local_irq_disable() in evict_folios() (pointed by Harry Yoo)
- recheck objcg in [PATCH v2 07/28] (pointed by Harry Yoo)
- modify the commit message in [PATCH v2 12/28] and [PATCH v2 21/28] (pointed by Harry Yoo)
- use rcu lock to protect mm_state in [PATCH v2 14/28] (pointed by Harry Yoo)
- fix bad unlock balance warning in [PATCH v2 23/28]
- change nr_pages type to long in [PATCH v2 25/28] (pointed by Harry Yoo)
- incease mm_state->seq during reparenting to make mm walker work properly in [PATCH v2 25/28] (pointed by Harry Yoo)
- add [PATCH v2 18/28] to fix WARNING in folio_memcg() (pointed by Harry Yoo)
- collect Reviewed-bys
- rebase onto the next-20251216
Changes in v1:
- drop [PATCH RFC 02/28]
- drop THP split queue related part, which has been merged as a separate patchset[2]
- prevent memory cgroup release in folio_split_queue_lock{_irqsave}() in [PATCH v1 16/26]
- Separate the reparenting function of traditional LRU folios to [PATCH v1 22/26]
- adapted to the MGLRU scenarios in [PATCH v1 23/26]
- refactor memcg_reparent_objcgs() in [PATCH v1 24/26]
- collect Acked-bys and Reviewed-bys
- rebase onto the next-20251028
Hi all,
Introduction
============
This patchset is intended to transfer the LRU pages to the object cgroup without holding a reference to the original memory cgroup in order to address the issue of the dying memory cgroup. A consensus has already been reached regarding this approach recently [1].
Background
==========
The issue of a dying memory cgroup refers to a situation where a memory cgroup is no longer being used by users, but memory (the metadata associated with memory cgroups) remains allocated to it. This situation may potentially result in memory leaks or inefficiencies in memory reclamation and has persisted as an issue for several years. Any memory allocation that endures longer than the lifespan (from the users' perspective) of a memory cgroup can lead to the issue of dying memory cgroup. We have exerted greater efforts to tackle this problem by introducing the infrastructure of object cgroup [2].
Presently, numerous types of objects (slab objects, non-slab kernel allocations, per-CPU objects) are charged to the object cgroup without holding a reference to the original memory cgroup. The final allocations for LRU pages (anonymous pages and file pages) are charged at allocation time and continues to hold a reference to the original memory cgroup until reclaimed.
File pages are more complex than anonymous pages as they can be shared among different memory cgroups and may persist beyond the lifespan of the memory cgroup. The long-term pinning of file pages to memory cgroups is a widespread issue that causes recurring problems in practical scenarios [3]. File pages remain unreclaimed for extended periods. Additionally, they are accessed by successive instances (second, third, fourth, etc.) of the same job, which is restarted into a new cgroup each time. As a result, unreclaimable dying memory cgroups accumulate, leading to memory wastage and significantly reducing the efficiency of page reclamation.
Fundamentals
============
A folio will no longer pin its corresponding memory cgroup. It is necessary to ensure that the memory cgroup or the lruvec associated with the memory cgroup is not released when a user obtains a pointer to the memory cgroup or lruvec returned by folio_memcg() or folio_lruvec(). Users are required to hold the RCU read lock or acquire a reference to the memory cgroup associated with the folio to prevent its release if they are not concerned about the binding stability between the folio and its corresponding memory cgroup. However, some users of folio_lruvec() (i.e., the lruvec lock) desire a stable binding between the folio and its corresponding memory cgroup. An approach is needed to ensure the stability of the binding while the lruvec lock is held, and to detect the situation of holding the incorrect lruvec lock when there is a race condition during memory cgroup reparenting. The following four steps are taken to achieve these goals.
1. The first step to be taken is to identify all users of both functions (folio_memcg() and folio_lruvec()) who are not concerned about binding stability and implement appropriate measures (such as holding a RCU read lock or temporarily obtaining a reference to the memory cgroup for a brief period) to prevent the release of the memory cgroup.
2. Secondly, the following refactoring of folio_lruvec_lock() demonstrates how to ensure the binding stability from the user's perspective of folio_lruvec().
struct lruvec *folio_lruvec_lock(struct folio *folio)
{
struct lruvec *lruvec; rcu_read_lock();
retry:
lruvec = folio_lruvec(folio);
spin_lock(&lruvec->lru_lock);
if (unlikely(lruvec_memcg(lruvec) != folio_memcg(folio))) {
spin_unlock(&lruvec->lru_lock);
goto retry;
}
return lruvec;}
From the perspective of memory cgroup removal, the entire reparenting process (altering the binding relationship between folio and its memory cgroup and moving the LRU lists to its parental memory cgroup) should be carried out under both the lruvec lock of the memory cgroup being removed and the lruvec lock of its parent.
3. Finally, transfer the LRU pages to the object cgroup without holding a reference to the original memory cgroup.
Effect
======
Finally, it can be observed that the quantity of dying memory cgroups will not experience a significant increase if the following test script is executed to reproduce the issue.
```bash
#!/bin/bash
# Create a temporary file 'temp' filled with zero bytes
dd if=/dev/zero of=temp bs=4096 count=1
# Display memory-cgroup info from /proc/cgroups
cat /proc/cgroups | grep memory
for i in {0..2000}
do
mkdir /sys/fs/cgroup/memory/test$i
echo $$ > /sys/fs/cgroup/memory/test$i/cgroup.procs
# Append 'temp' file content to 'log'
cat temp >> log
echo $$ > /sys/fs/cgroup/memory/cgroup.procs
# Potentially create a dying memory cgroup
rmdir /sys/fs/cgroup/memory/test$i
done
# Display memory-cgroup info after test
cat /proc/cgroups | grep memory
rm -f temp log
```Comments and suggestions are welcome!
Thanks,
Qi
[1].https://lore.kernel.org/linux-mm/Z6OkXXYDorPrBvEQ@hm-sls2/ [2].https://lwn.net/Articles/895431/ [3].https://github.com/systemd/systemd/pull/36827
Muchun Song (22):
mm: memcontrol: remove dead code of checking parent memory cgroup
mm: workingset: use folio_lruvec() in workingset_refault()
mm: rename unlock_page_lruvec_irq and its variants
mm: vmscan: refactor move_folios_to_lru()
mm: memcontrol: allocate object cgroup for non-kmem case
mm: memcontrol: return root object cgroup for root memory cgroup
mm: memcontrol: prevent memory cgroup release in
get_mem_cgroup_from_folio()
buffer: prevent memory cgroup release in folio_alloc_buffers()
writeback: prevent memory cgroup release in writeback module
mm: memcontrol: prevent memory cgroup release in
count_memcg_folio_events()
mm: page_io: prevent memory cgroup release in page_io module
mm: migrate: prevent memory cgroup release in folio_migrate_mapping()
mm: mglru: prevent memory cgroup release in mglru
mm: memcontrol: prevent memory cgroup release in
mem_cgroup_swap_full()
mm: workingset: prevent memory cgroup release in lru_gen_eviction()
mm: workingset: prevent lruvec release in workingset_refault()
mm: zswap: prevent lruvec release in zswap_folio_swapin()
mm: swap: prevent lruvec release in lru_gen_clear_refs()
mm: workingset: prevent lruvec release in workingset_activation()
mm: memcontrol: prepare for reparenting LRU pages for lruvec lock
mm: memcontrol: eliminate the problem of dying memory cgroup for LRU
folios
mm: lru: add VM_WARN_ON_ONCE_FOLIO to lru maintenance helpers
Qi Zheng (10):
mm: vmscan: prepare for the refactoring the move_folios_to_lru()
mm: thp: prevent memory cgroup release in
folio_split_queue_lock{_irqsave}()
mm: zswap: prevent memory cgroup release in zswap_compress()
mm: do not open-code lruvec lock
mm: vmscan: prepare for reparenting traditional LRU folios
mm: vmscan: prepare for reparenting MGLRU folios
mm: memcontrol: refactor memcg_reparent_objcgs()
mm: workingset: use lruvec_lru_size() to get the number of lru pages
mm: memcontrol: prepare for reparenting non-hierarchical stats
mm: memcontrol: convert objcg to be per-memcg per-node type
fs/buffer.c | 4 +- fs/fs-writeback.c | 22 +- include/linux/memcontrol.h | 191 ++++++----- include/linux/mm_inline.h | 6 + include/linux/mmzone.h | 17 + include/linux/sched.h | 2 +- include/linux/swap.h | 25 +- include/trace/events/writeback.h | 3 + kernel/cgroup/cgroup.c | 8 +- mm/compaction.c | 43 ++- mm/huge_memory.c | 22 +- mm/memcontrol-v1.c | 31 +- mm/memcontrol-v1.h | 8 + mm/memcontrol.c | 562 ++++++++++++++++++++----------- mm/migrate.c | 2 + mm/mlock.c | 2 +- mm/page_io.c | 8 +- mm/percpu.c | 2 +- mm/shrinker.c | 6 +- mm/swap.c | 59 +++- mm/vmscan.c | 273 +++++++++++---- mm/workingset.c | 30 +- mm/zswap.c | 5 + 23 files changed, 904 insertions(+), 427 deletions(-)
Signature · 1 line
2.20.1
Qi Zheng
reviewerPATCH v5 1/32 mm: memcontrol: remove dead code of checking parent memory cgroup
From: Muchun Song <[email protected]>
Since the no-hierarchy mode has been deprecated after the commit:
commit bef8620cd8e0 ("mm: memcg: deprecate the non-hierarchical mode").
As a result, parent_mem_cgroup() will not return NULL except when passing the root memcg, and the root memcg cannot be offline. Hence, it's safe to remove the check on the returned value of parent_mem_cgroup(). Remove the corresponding dead code.
--- mm/memcontrol.c | 5 ----- mm/shrinker.c | 6 +----- 2 files changed, 1 insertion(+), 10 deletions(-)
mm/memcontrol.c +0 -5
@@ -3424,9 +3424,6 @@ static void memcg_offline_kmem(struct mem_cgroup *memcg)return ; parent =parent_mem_cgroup (memcg); -if (!parent) - parent = root_mem_cgroup; -memcg_reparent_list_lrus (memcg, parent);/*
@@ -3706,8 +3703,6 @@ struct mem_cgroup *mem_cgroup_private_id_get_online(struct mem_cgroup *memcg, unbreak ; } memcg =parent_mem_cgroup (memcg); -if (!memcg) - memcg = root_mem_cgroup; }return memcg; }
mm/shrinker.c +1 -5
@@ -286,14 +286,10 @@ void reparent_shrinker_deferred(struct mem_cgroup *memcg) {int nid, index, offset;long nr; -struct mem_cgroup *parent; +struct mem_cgroup *parent =parent_mem_cgroup (memcg);struct shrinker_info *child_info, *parent_info;struct shrinker_info_unit *child_unit, *parent_unit; - parent =parent_mem_cgroup (memcg); -if (!parent) - parent = root_mem_cgroup; -/* Prevent from concurrent shrinker_info expand */ mutex_lock (&shrinker_mutex);for_each_node (nid) {
Signature · 1 line
2.20.1
Qi Zheng
reviewerPATCH v5 2/32 mm: workingset: use folio_lruvec() in workingset_refault()
In reply to Qi ZhengFrom: Muchun Song <[email protected]>
Use folio_lruvec() to simplify the code.
--- mm/workingset.c | 7 +------ 1 file changed, 1 insertion(+), 6 deletions(-)
mm/workingset.c +1 -6
@@ -541,8 +541,6 @@ bool workingset_test_recent(void *shadow, bool file, bool *workingset,void workingset_refault (struct folio *folio,void *shadow) {bool file =folio_is_file_lru (folio); -struct pglist_data *pgdat; -struct mem_cgroup *memcg;struct lruvec *lruvec;bool workingset;long nr;
@@ -564,10 +562,7 @@ void workingset_refault(struct folio *folio, void *shadow)* locked to guarantee folio_memcg() stability throughout. */ nr =folio_nr_pages (folio); - memcg =folio_memcg (folio); - pgdat =folio_pgdat (folio); - lruvec =mem_cgroup_lruvec (memcg, pgdat); - + lruvec =folio_lruvec (folio);mod_lruvec_state (lruvec,WORKINGSET_REFAULT_BASE + file, nr);if (!workingset_test_recent (shadow, file, &workingset, true))
Signature · 1 line
2.20.1
Qi Zheng
reviewerPATCH v5 3/32 mm: rename unlock_page_lruvec_irq and its variants
In reply to Qi ZhengFrom: Muchun Song <[email protected]>
It is inappropriate to use folio_lruvec_lock() variants in conjunction with unlock_page_lruvec() variants, as this involves the inconsistent operation of locking a folio while unlocking a page. To rectify this, the functions unlock_page_lruvec{_irq, _irqrestore} are renamed to lruvec_unlock{_irq,_irqrestore}.
--- include/linux/memcontrol.h | 10 +++++----- mm/compaction.c | 14 +++++++------- mm/huge_memory.c | 2 +- mm/mlock.c | 2 +- mm/swap.c | 12 ++++++------ mm/vmscan.c | 4 ++-- 6 files changed, 22 insertions(+), 22 deletions(-)
include/linux/memcontrol.h +5 -5
@@ -1480,17 +1480,17 @@ static inline struct lruvec *parent_lruvec(struct lruvec *lruvec)return mem_cgroup_lruvec (memcg,lruvec_pgdat (lruvec)); } -static inline void unlock_page_lruvec (struct lruvec *lruvec) +static inline void lruvec_unlock (struct lruvec *lruvec) {spin_unlock (&lruvec->lru_lock); } -static inline void unlock_page_lruvec_irq (struct lruvec *lruvec) +static inline void lruvec_unlock_irq (struct lruvec *lruvec) {spin_unlock_irq (&lruvec->lru_lock); } -static inline void unlock_page_lruvec_irqrestore (struct lruvec *lruvec, +static inline void lruvec_unlock_irqrestore (struct lruvec *lruvec,unsigned long flags) {spin_unlock_irqrestore (&lruvec->lru_lock, flags);
@@ -1512,7 +1512,7 @@ static inline struct lruvec *folio_lruvec_relock_irq(struct folio *folio,if (folio_matches_lruvec (folio, locked_lruvec))return locked_lruvec; -unlock_page_lruvec_irq (locked_lruvec); +lruvec_unlock_irq (locked_lruvec); }return folio_lruvec_lock_irq (folio);
@@ -1526,7 +1526,7 @@ static inline void folio_lruvec_relock_irqsave(struct folio *folio,if (folio_matches_lruvec (folio, *lruvecp))return ; -unlock_page_lruvec_irqrestore (*lruvecp, *flags); +lruvec_unlock_irqrestore (*lruvecp, *flags); } *lruvecp =folio_lruvec_lock_irqsave (folio, flags);
mm/compaction.c +7 -7
@@ -913,7 +913,7 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn,*/ if (!(low_pfn %COMPACT_CLUSTER_MAX )) {if (locked) { -unlock_page_lruvec_irqrestore (locked, flags); +lruvec_unlock_irqrestore (locked, flags); locked =NULL ; }
@@ -964,7 +964,7 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn, }/* for alloc_contig case */ if (locked) { -unlock_page_lruvec_irqrestore (locked, flags); +lruvec_unlock_irqrestore (locked, flags); locked =NULL ; }
@@ -1053,7 +1053,7 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn,if (unlikely (page_has_movable_ops (page)) && !PageMovableOpsIsolated (page)) {if (locked) { -unlock_page_lruvec_irqrestore (locked, flags); +lruvec_unlock_irqrestore (locked, flags); locked =NULL ; }
@@ -1158,7 +1158,7 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn,/* If we already hold the lock, we can skip some rechecking */ if (lruvec != locked) {if (locked) -unlock_page_lruvec_irqrestore (locked, flags); +lruvec_unlock_irqrestore (locked, flags);compact_lock_irqsave (&lruvec->lru_lock, &flags, cc); locked = lruvec;
@@ -1226,7 +1226,7 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn, isolate_fail_put:/* Avoid potential deadlock in freeing page under lru_lock */ if (locked) { -unlock_page_lruvec_irqrestore (locked, flags); +lruvec_unlock_irqrestore (locked, flags); locked =NULL ; }folio_put (folio);
@@ -1242,7 +1242,7 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn,*/ if (nr_isolated) {if (locked) { -unlock_page_lruvec_irqrestore (locked, flags); +lruvec_unlock_irqrestore (locked, flags); locked =NULL ; }putback_movable_pages (&cc->migratepages);
@@ -1274,7 +1274,7 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn, isolate_abort:if (locked) -unlock_page_lruvec_irqrestore (locked, flags); +lruvec_unlock_irqrestore (locked, flags);if (folio) {folio_set_lru (folio);folio_put (folio);
mm/huge_memory.c +1 -1
@@ -3902,7 +3902,7 @@ static int __folio_freeze_and_split_unmapped(struct folio *folio, unsigned int nfolio_ref_unfreeze (folio,folio_cache_ref_count (folio) +1 );if (do_lru) -unlock_page_lruvec (lruvec); +lruvec_unlock (lruvec);if (ci)swap_cluster_unlock (ci);
mm/mlock.c +1 -1
@@ -205,7 +205,7 @@ static void mlock_folio_batch(struct folio_batch *fbatch) }if (lruvec) -unlock_page_lruvec_irq (lruvec); +lruvec_unlock_irq (lruvec);folios_put (fbatch); }
mm/swap.c +6 -6
@@ -91,7 +91,7 @@ static void page_cache_release(struct folio *folio)__page_cache_release (folio, &lruvec, &flags);if (lruvec) -unlock_page_lruvec_irqrestore (lruvec, flags); +lruvec_unlock_irqrestore (lruvec, flags); }void __folio_put (struct folio *folio)
@@ -175,7 +175,7 @@ static void folio_batch_move_lru(struct folio_batch *fbatch, move_fn_t move_fn) }if (lruvec) -unlock_page_lruvec_irqrestore (lruvec, flags); +lruvec_unlock_irqrestore (lruvec, flags);folios_put (fbatch); }
@@ -349,7 +349,7 @@ void folio_activate(struct folio *folio) lruvec =folio_lruvec_lock_irq (folio);lru_activate (lruvec, folio); -unlock_page_lruvec_irq (lruvec); +lruvec_unlock_irq (lruvec); folio_set_lru(folio); }#endif
@@ -963,7 +963,7 @@ void folios_put_refs(struct folio_batch *folios, unsigned int *refs)if (folio_is_zone_device (folio)) {if (lruvec) { -unlock_page_lruvec_irqrestore (lruvec, flags); +lruvec_unlock_irqrestore (lruvec, flags); lruvec =NULL ; }if (folio_ref_sub_and_test (folio, nr_refs))
@@ -977,7 +977,7 @@ void folios_put_refs(struct folio_batch *folios, unsigned int *refs)/* hugetlb has its own memcg */ if (folio_test_hugetlb (folio)) {if (lruvec) { -unlock_page_lruvec_irqrestore (lruvec, flags); +lruvec_unlock_irqrestore (lruvec, flags); lruvec =NULL ; }free_huge_folio (folio);
@@ -991,7 +991,7 @@ void folios_put_refs(struct folio_batch *folios, unsigned int *refs) j++; }if (lruvec) -unlock_page_lruvec_irqrestore (lruvec, flags); +lruvec_unlock_irqrestore (lruvec, flags);if (!j) {folio_batch_reinit (folios);return ;
mm/vmscan.c +2 -2
@@ -1835,7 +1835,7 @@ bool folio_isolate_lru(struct folio *folio)folio_get (folio); lruvec =folio_lruvec_lock_irq (folio);lruvec_del_folio (lruvec, folio); -unlock_page_lruvec_irq (lruvec); +lruvec_unlock_irq (lruvec); ret = true; }
@@ -7861,7 +7861,7 @@ void check_move_unevictable_folios(struct folio_batch *fbatch)if (lruvec) {__count_vm_events (UNEVICTABLE_PGRESCUED , pgrescued);__count_vm_events (UNEVICTABLE_PGSCANNED , pgscanned); -unlock_page_lruvec_irq (lruvec); +lruvec_unlock_irq (lruvec); }else if (pgscanned) {count_vm_events (UNEVICTABLE_PGSCANNED , pgscanned); }
Signature · 1 line
2.20.1
Qi Zheng
reviewerPATCH v5 4/32 mm: vmscan: prepare for the refactoring the move_folios_to_lru()
In reply to Qi ZhengFrom: Qi Zheng <[email protected]>
Once we refactor move_folios_to_lru(), its callers will no longer have to hold the lruvec lock; For shrink_inactive_list(), shrink_active_list() and evict_folios(), IRQ disabling is only needed for __count_vm_events() and __mod_node_page_state().
To avoid using local_irq_disable() on the PREEMPT_RT kernel, let's make all callers of move_folios_to_lru() use IRQ-safed count_vm_events() and mod_node_page_state().
--- mm/vmscan.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
mm/vmscan.c +3 -3
@@ -2025,7 +2025,7 @@ static unsigned long shrink_inactive_list(unsigned long nr_to_scan,mod_lruvec_state (lruvec,PGDEMOTE_KSWAPD +reclaimer_offset (sc), stat.nr_demoted); -__mod_node_page_state (pgdat,NR_ISOLATED_ANON + file, -nr_taken); +mod_node_page_state (pgdat,NR_ISOLATED_ANON + file, -nr_taken); item =PGSTEAL_KSWAPD +reclaimer_offset (sc);mod_lruvec_state (lruvec, item, nr_reclaimed);mod_lruvec_state (lruvec,PGSTEAL_ANON + file, nr_reclaimed);
@@ -2171,10 +2171,10 @@ static void shrink_active_list(unsigned long nr_to_scan, nr_activate =move_folios_to_lru (lruvec, &l_active); nr_deactivate =move_folios_to_lru (lruvec, &l_inactive); -__count_vm_events (PGDEACTIVATE , nr_deactivate); +count_vm_events (PGDEACTIVATE , nr_deactivate);count_memcg_events (lruvec_memcg (lruvec),PGDEACTIVATE , nr_deactivate); -__mod_node_page_state (pgdat,NR_ISOLATED_ANON + file, -nr_taken); +mod_node_page_state (pgdat,NR_ISOLATED_ANON + file, -nr_taken);lru_note_cost_unlock_irq (lruvec, file,0 , nr_rotated); trace_mm_vmscan_lru_shrink_active(pgdat->node_id, nr_taken, nr_activate,
Signature · 1 line
2.20.1
Qi Zheng
reviewerPATCH v5 5/32 mm: vmscan: refactor move_folios_to_lru()
In reply to Qi ZhengFrom: Muchun Song <[email protected]>
In a subsequent patch, we'll reparent the LRU folios. The folios that are moved to the appropriate LRU list can undergo reparenting during the move_folios_to_lru() process. Hence, it's incorrect for the caller to hold a lruvec lock. Instead, we should utilize the more general interface of folio_lruvec_relock_irq() to obtain the correct lruvec lock.
This patch involves only code refactoring and doesn't introduce any functional changes.
--- mm/vmscan.c | 46 +++++++++++++++++++++------------------------- 1 file changed, 21 insertions(+), 25 deletions(-)
mm/vmscan.c +21 -25
@@ -1889,24 +1889,27 @@ static bool too_many_isolated(struct pglist_data *pgdat, int file,/* * move_folios_to_lru() moves folios from private @list to appropriate LRU list. * -* Returns the number of pages moved to the given lruvec. +* Returns the number of pages moved to the appropriate lruvec. +* +* Note: The caller must not hold any lruvec lock. */ -static unsigned int move_folios_to_lru (struct lruvec *lruvec, -struct list_head *list) +static unsigned int move_folios_to_lru (struct list_head *list) {int nr_pages, nr_moved =0 ; +struct lruvec *lruvec =NULL ;struct folio_batch free_folios;folio_batch_init (&free_folios);while (!list_empty (list)) {struct folio *folio =lru_to_folio (list); + lruvec =folio_lruvec_relock_irq (folio, lruvec);VM_BUG_ON_FOLIO (folio_test_lru (folio), folio);list_del (&folio->lru);if (unlikely (!folio_evictable (folio))) { -spin_unlock_irq (&lruvec->lru_lock); +lruvec_unlock_irq (lruvec);folio_putback_lru (folio); -spin_lock_irq (&lruvec->lru_lock); + lruvec =NULL ;continue ; }
@@ -1928,19 +1931,15 @@ static unsigned int move_folios_to_lru(struct lruvec *lruvec,folio_unqueue_deferred_split (folio);if (folio_batch_add (&free_folios, folio) ==0 ) { -spin_unlock_irq (&lruvec->lru_lock); +lruvec_unlock_irq (lruvec);mem_cgroup_uncharge_folios (&free_folios);free_unref_folios (&free_folios); -spin_lock_irq (&lruvec->lru_lock); + lruvec =NULL ; }continue ; } -/* -* All pages were isolated from the same lruvec (and isolation -* inhibits memcg migration). -*/ VM_BUG_ON_FOLIO (!folio_matches_lruvec (folio, lruvec), folio);lruvec_add_folio (lruvec, folio); nr_pages =folio_nr_pages (folio);
@@ -1949,11 +1948,12 @@ static unsigned int move_folios_to_lru(struct lruvec *lruvec,workingset_age_nonresident (lruvec, nr_pages); } +if (lruvec) +lruvec_unlock_irq (lruvec); +if (free_folios.nr) { -spin_unlock_irq (&lruvec->lru_lock);mem_cgroup_uncharge_folios (&free_folios);free_unref_folios (&free_folios); -spin_lock_irq (&lruvec->lru_lock); }return nr_moved;
@@ -2020,8 +2020,7 @@ static unsigned long shrink_inactive_list(unsigned long nr_to_scan, nr_reclaimed =shrink_folio_list (&folio_list, pgdat, sc, &stat, false,lruvec_memcg (lruvec)); -spin_lock_irq (&lruvec->lru_lock); -move_folios_to_lru (lruvec, &folio_list); +move_folios_to_lru (&folio_list);mod_lruvec_state (lruvec,PGDEMOTE_KSWAPD +reclaimer_offset (sc), stat.nr_demoted);
@@ -2030,6 +2029,7 @@ static unsigned long shrink_inactive_list(unsigned long nr_to_scan,mod_lruvec_state (lruvec, item, nr_reclaimed);mod_lruvec_state (lruvec,PGSTEAL_ANON + file, nr_reclaimed); +spin_lock_irq (&lruvec->lru_lock);lru_note_cost_unlock_irq (lruvec, file, stat.nr_pageout, nr_scanned - nr_reclaimed);
@@ -2166,16 +2166,14 @@ static void shrink_active_list(unsigned long nr_to_scan,/* * Move folios back to the lru list. */ -spin_lock_irq (&lruvec->lru_lock); - - nr_activate =move_folios_to_lru (lruvec, &l_active); - nr_deactivate =move_folios_to_lru (lruvec, &l_inactive); + nr_activate =move_folios_to_lru (&l_active); + nr_deactivate =move_folios_to_lru (&l_inactive);count_vm_events (PGDEACTIVATE , nr_deactivate);count_memcg_events (lruvec_memcg (lruvec),PGDEACTIVATE , nr_deactivate); -mod_node_page_state (pgdat,NR_ISOLATED_ANON + file, -nr_taken); +spin_lock_irq (&lruvec->lru_lock);lru_note_cost_unlock_irq (lruvec, file,0 , nr_rotated);trace_mm_vmscan_lru_shrink_active (pgdat->node_id, nr_taken, nr_activate, nr_deactivate, nr_rotated, sc->priority, file);
@@ -4731,14 +4729,14 @@ static int evict_folios(unsigned long nr_to_scan, struct lruvec *lruvec,set_mask_bits (&folio->flags.f,LRU_REFS_FLAGS ,BIT (PG_active)); } -spin_lock_irq (&lruvec->lru_lock); - -move_folios_to_lru (lruvec, &list); +move_folios_to_lru (&list); walk = current->reclaim_state->mm_walk;if (walk && walk->batched) { walk->lruvec = lruvec; +spin_lock_irq (&lruvec->lru_lock);reset_batch_size (walk); +spin_unlock_irq (&lruvec->lru_lock); } mod_lruvec_state(lruvec, PGDEMOTE_KSWAPD + reclaimer_offset(sc),
@@ -4748,8 +4746,6 @@ static int evict_folios(unsigned long nr_to_scan, struct lruvec *lruvec,mod_lruvec_state (lruvec, item, reclaimed);mod_lruvec_state (lruvec,PGSTEAL_ANON + type, reclaimed); -spin_unlock_irq (&lruvec->lru_lock); -list_splice_init (&clean, &list);if (!list_empty (&list)) {
Signature · 1 line
2.20.1
Qi Zheng
reviewerPATCH v5 6/32 mm: memcontrol: allocate object cgroup for non-kmem case
In reply to Qi ZhengFrom: Muchun Song <[email protected]>
To allow LRU page reparenting, the objcg infrastructure is no longer solely applicable to the kmem case. In this patch, we extend the scope of the objcg infrastructure beyond the kmem case, enabling LRU folios to reuse it for folio charging purposes.
It should be noted that LRU folios are not accounted for at the root level, yet the folio->memcg_data points to the root_mem_cgroup. Hence, the folio->memcg_data of LRU folios always points to a valid pointer. However, the root_mem_cgroup does not possess an object cgroup. Therefore, we also allocate an object cgroup for the root_mem_cgroup.
--- mm/memcontrol.c | 51 +++++++++++++++++++++++-------------------------- 1 file changed, 24 insertions(+), 27 deletions(-)
mm/memcontrol.c +24 -27
@@ -207,10 +207,10 @@ static struct obj_cgroup *obj_cgroup_alloc(void)return objcg; } -static void memcg_reparent_objcgs (struct mem_cgroup *memcg, -struct mem_cgroup *parent) +static void memcg_reparent_objcgs (struct mem_cgroup *memcg) {struct obj_cgroup *objcg, *iter; +struct mem_cgroup *parent =parent_mem_cgroup (memcg); objcg =rcu_replace_pointer (memcg->objcg,NULL , true);
@@ -3387,30 +3387,17 @@ void folio_split_memcg_refs(struct folio *folio, unsigned old_order,css_get_many (&__folio_memcg (folio)->css, new_refs); } -static int memcg_online_kmem (struct mem_cgroup *memcg) +static void memcg_online_kmem (struct mem_cgroup *memcg) { -struct obj_cgroup *objcg; -if (mem_cgroup_kmem_disabled ()) -return 0 ; +return ;if (unlikely (mem_cgroup_is_root (memcg))) -return 0 ; - - objcg =obj_cgroup_alloc (); -if (!objcg) -return -ENOMEM ; - - objcg->memcg = memcg; -rcu_assign_pointer (memcg->objcg, objcg); -obj_cgroup_get (objcg); - memcg->orig_objcg = objcg; +return ;static_branch_enable (&memcg_kmem_online_key); memcg->kmemcg_id = memcg->id.id; - -return 0 ; }static void memcg_offline_kmem (struct mem_cgroup *memcg)
@@ -3425,12 +3412,6 @@ static void memcg_offline_kmem(struct mem_cgroup *memcg) parent =parent_mem_cgroup (memcg);memcg_reparent_list_lrus (memcg, parent); - -/* -* Objcg's reparenting must be after list_lru's, make sure list_lru -* helpers won't use parent's list_lru until child is drained. -*/ -memcg_reparent_objcgs (memcg, parent); }#ifdef CONFIG_CGROUP_WRITEBACK
@@ -3931,9 +3912,9 @@ mem_cgroup_css_alloc(struct cgroup_subsys_state *parent_css)static int mem_cgroup_css_online (struct cgroup_subsys_state *css) {struct mem_cgroup *memcg =mem_cgroup_from_css (css); +struct obj_cgroup *objcg; -if (memcg_online_kmem (memcg)) - goto remove_id; +memcg_online_kmem (memcg);/* * A memcg must be visible for expand_shrinker_info()
@@ -3943,6 +3924,15 @@ static int mem_cgroup_css_online(struct cgroup_subsys_state *css)if (alloc_shrinker_info (memcg)) goto offline_kmem; + objcg =obj_cgroup_alloc (); +if (!objcg) + goto free_shrinker; + + objcg->memcg = memcg; +rcu_assign_pointer (memcg->objcg, objcg); +obj_cgroup_get (objcg); + memcg->orig_objcg = objcg; +if (unlikely (mem_cgroup_is_root (memcg)) && !mem_cgroup_disabled ())queue_delayed_work (system_dfl_wq, &stats_flush_dwork,FLUSH_TIME );
@@ -3965,9 +3955,10 @@ static int mem_cgroup_css_online(struct cgroup_subsys_state *css)xa_store (&mem_cgroup_private_ids, memcg->id.id, memcg,GFP_KERNEL );return 0 ; +free_shrinker: +free_shrinker_info (memcg); offline_kmem:memcg_offline_kmem (memcg); -remove_id:mem_cgroup_private_id_remove (memcg);return -ENOMEM ; }
@@ -3985,6 +3976,12 @@ static void mem_cgroup_css_offline(struct cgroup_subsys_state *css)memcg_offline_kmem (memcg);reparent_deferred_split_queue (memcg); +/* +* The reparenting of objcg must be after the reparenting of the +* list_lru and deferred_split_queue above, which ensures that they will +* not mistakenly get the parent list_lru and deferred_split_queue. +*/ +memcg_reparent_objcgs (memcg);reparent_shrinker_deferred (memcg);wb_memcg_offline (memcg);lru_gen_offline_memcg (memcg);
Signature · 1 line
2.20.1
Qi Zheng
reviewerPATCH v5 7/32 mm: memcontrol: return root object cgroup for root memory cgroup
In reply to Qi ZhengFrom: Muchun Song <[email protected]>
Memory cgroup functions such as get_mem_cgroup_from_folio() and get_mem_cgroup_from_mm() return a valid memory cgroup pointer, even for the root memory cgroup. In contrast, the situation for object cgroups has been different.
Previously, the root object cgroup couldn't be returned because it didn't exist. Now that a valid root object cgroup exists, for the sake of consistency, it's necessary to align the behavior of object-cgroup-related operations with that of memory cgroup APIs.
--- include/linux/memcontrol.h | 26 +++++++++++++++++----- mm/memcontrol.c | 45 ++++++++++++++++++++------------------ mm/percpu.c | 2 +- 3 files changed, 45 insertions(+), 28 deletions(-)
include/linux/memcontrol.h +20 -6
@@ -333,6 +333,7 @@ struct mem_cgroup {#define MEMCG_CHARGE_BATCH 64Uextern struct mem_cgroup *root_mem_cgroup; +extern struct obj_cgroup *root_obj_cgroup;enum page_memcg_data_flags {/* page->memcg_data is a pointer to an slabobj_ext vector */
@@ -549,6 +550,11 @@ static inline bool mem_cgroup_is_root(struct mem_cgroup *memcg)return (memcg == root_mem_cgroup); } +static inline bool obj_cgroup_is_root (const struct obj_cgroup *objcg) +{ +return objcg == root_obj_cgroup; +} +static inline bool mem_cgroup_disabled (void ) {return !cgroup_subsys_enabled (memory_cgrp_subsys);
@@ -775,23 +781,26 @@ struct mem_cgroup *mem_cgroup_from_css(struct cgroup_subsys_state *css){static inline bool obj_cgroup_tryget (struct obj_cgroup *objcg) { +if (obj_cgroup_is_root (objcg)) +return true;return percpu_ref_tryget (&objcg->refcnt); } -static inline void obj_cgroup_get (struct obj_cgroup *objcg) +static inline void obj_cgroup_get_many (struct obj_cgroup *objcg, +unsigned long nr) { -percpu_ref_get (&objcg->refcnt); +if (!obj_cgroup_is_root (objcg)) +percpu_ref_get_many (&objcg->refcnt, nr); } -static inline void obj_cgroup_get_many (struct obj_cgroup *objcg, -unsigned long nr) +static inline void obj_cgroup_get (struct obj_cgroup *objcg) { -percpu_ref_get_many (&objcg->refcnt, nr); +obj_cgroup_get_many (objcg,1 ); }static inline void obj_cgroup_put (struct obj_cgroup *objcg) { -if (objcg) +if (objcg && !obj_cgroup_is_root (objcg))percpu_ref_put (&objcg->refcnt); }
@@ -1088,6 +1097,11 @@ static inline bool mem_cgroup_is_root(struct mem_cgroup *memcg)return true; } +static inline bool obj_cgroup_is_root (const struct obj_cgroup *objcg) +{ +return true; +} +static inline bool mem_cgroup_disabled (void ) {return true;
mm/memcontrol.c +24 -21
119 lines not on this page · raw message
mm/percpu.c +1 -1
@@ -1622,7 +1622,7 @@ static bool pcpu_memcg_pre_alloc_hook(size_t size, gfp_t gfp,return true; objcg =current_obj_cgroup (); -if (!objcg) +if (!objcg ||obj_cgroup_is_root (objcg))return true;if (obj_cgroup_charge (objcg, gfp,pcpu_obj_full_size (size)))
Signature · 1 line
2.20.1
Qi Zheng
reviewerPATCH v5 8/32 mm: memcontrol: prevent memory cgroup release in get_mem_cgroup_from_folio()
In reply to Qi ZhengFrom: Muchun Song <[email protected]>
In the near future, a folio will no longer pin its corresponding memory cgroup. To ensure safety, it will only be appropriate to hold the rcu read lock or acquire a reference to the memory cgroup returned by folio_memcg(), thereby preventing it from being released.
In the current patch, the rcu read lock is employed to safeguard against the release of the memory cgroup in get_mem_cgroup_from_folio().
This serves as a preparatory measure for the reparenting of the LRU pages.
--- mm/memcontrol.c | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-)
mm/memcontrol.c +7 -3
@@ -997,14 +997,18 @@ struct mem_cgroup *get_mem_cgroup_from_current(void)*/ struct mem_cgroup *get_mem_cgroup_from_folio (struct folio *folio) { -struct mem_cgroup *memcg =folio_memcg (folio); +struct mem_cgroup *memcg;if (mem_cgroup_disabled ())return NULL ; +if (!folio_memcg_charged (folio)) +return root_mem_cgroup; +rcu_read_lock (); -if (!memcg ||WARN_ON_ONCE (!css_tryget (&memcg->css))) - memcg = root_mem_cgroup; +do { + memcg =folio_memcg (folio); + }while (unlikely (!css_tryget (&memcg->css)));rcu_read_unlock ();return memcg; }
Signature · 1 line
2.20.1
Qi Zheng
reviewerPATCH v5 9/32 buffer: prevent memory cgroup release in folio_alloc_buffers()
In reply to Qi ZhengFrom: Muchun Song <[email protected]>
In the near future, a folio will no longer pin its corresponding memory cgroup. To ensure safety, it will only be appropriate to hold the rcu read lock or acquire a reference to the memory cgroup returned by folio_memcg(), thereby preventing it from being released.
In the current patch, the function get_mem_cgroup_from_folio() is employed to safeguard against the release of the memory cgroup. This serves as a preparatory measure for the reparenting of the LRU pages.
--- fs/buffer.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
fs/buffer.c +2 -2
@@ -922,8 +922,7 @@ struct buffer_head *folio_alloc_buffers(struct folio *folio, unsigned long size,long offset;struct mem_cgroup *memcg, *old_memcg; -/* The folio lock pins the memcg */ - memcg =folio_memcg (folio); + memcg =get_mem_cgroup_from_folio (folio); old_memcg =set_active_memcg (memcg); head =NULL ;
@@ -944,6 +943,7 @@ struct buffer_head *folio_alloc_buffers(struct folio *folio, unsigned long size, } out:set_active_memcg (old_memcg); +mem_cgroup_put (memcg);return head;/* * In case anything failed, we just free everything we got.
Signature · 1 line
2.20.1
Qi Zheng
reviewerPATCH v5 10/32 writeback: prevent memory cgroup release in writeback module
In reply to Qi ZhengFrom: Muchun Song <[email protected]>
In the near future, a folio will no longer pin its corresponding memory cgroup. To ensure safety, it will only be appropriate to hold the rcu read lock or acquire a reference to the memory cgroup returned by folio_memcg(), thereby preventing it from being released.
In the current patch, the function get_mem_cgroup_css_from_folio() and the rcu read lock are employed to safeguard against the release of the memory cgroup.
This serves as a preparatory measure for the reparenting of the LRU pages.
--- fs/fs-writeback.c | 22 +++++++++++----------- include/linux/memcontrol.h | 9 +++++++-- include/trace/events/writeback.h | 3 +++ mm/memcontrol.c | 14 ++++++++------ 4 files changed, 29 insertions(+), 19 deletions(-)
fs/fs-writeback.c +11 -11
@@ -280,15 +280,13 @@ void __inode_attach_wb(struct inode *inode, struct folio *folio)if (inode_cgwb_enabled (inode)) {struct cgroup_subsys_state *memcg_css; -if (folio) { - memcg_css =mem_cgroup_css_from_folio (folio); - wb =wb_get_create (bdi, memcg_css,GFP_ATOMIC ); - }else { -/* must pin memcg_css, see wb_get_create() */ +/* must pin memcg_css, see wb_get_create() */ +if (folio) + memcg_css =get_mem_cgroup_css_from_folio (folio); +else memcg_css =task_get_css (current, memory_cgrp_id); - wb =wb_get_create (bdi, memcg_css,GFP_ATOMIC ); -css_put (memcg_css); - } + wb =wb_get_create (bdi, memcg_css,GFP_ATOMIC ); +css_put (memcg_css); }if (!wb)
@@ -979,16 +977,16 @@ void wbc_account_cgroup_owner(struct writeback_control *wbc, struct folio *folioif (!wbc->wb || wbc->no_cgroup_owner)return ; - css =mem_cgroup_css_from_folio (folio); + css =get_mem_cgroup_css_from_folio (folio);/* dead cgroups shouldn't contribute to inode ownership arbitration */ if (!css_is_online (css)) -return ; + goto out; id = css->id;if (id == wbc->wb_id) { wbc->wb_bytes += bytes; -return ; + goto out; }if (id == wbc->wb_lcand_id)
@@ -1001,6 +999,8 @@ void wbc_account_cgroup_owner(struct writeback_control *wbc, struct folio *folio wbc->wb_tcand_bytes += bytes; else wbc->wb_tcand_bytes -= min(bytes, wbc->wb_tcand_bytes); +out: +css_put (css); }EXPORT_SYMBOL_GPL (wbc_account_cgroup_owner);
include/linux/memcontrol.h +7 -2
@@ -895,7 +895,7 @@ static inline bool mm_match_cgroup(struct mm_struct *mm,return match; } -struct cgroup_subsys_state *mem_cgroup_css_from_folio (struct folio *folio); +struct cgroup_subsys_state *get_mem_cgroup_css_from_folio (struct folio *folio);ino_t page_cgroup_ino (struct page *page);static inline bool mem_cgroup_online (struct mem_cgroup *memcg)
@@ -1564,9 +1564,14 @@ static inline void mem_cgroup_track_foreign_dirty(struct folio *folio,if (mem_cgroup_disabled ())return ; +if (!folio_memcg_charged (folio)) +return ; + +rcu_read_lock (); memcg =folio_memcg (folio); -if (unlikely (memcg && &memcg->css != wb->memcg_css)) +if (unlikely (&memcg->css != wb->memcg_css))mem_cgroup_track_foreign_dirty_slowpath (folio, wb); +rcu_read_unlock (); }void mem_cgroup_flush_foreign (struct bdi_writeback *wb);
include/trace/events/writeback.h +3 -0
@@ -294,7 +294,10 @@ TRACE_EVENT(track_foreign_dirty, __entry->ino = inode ? inode->i_ino :0 ; __entry->memcg_id = wb->memcg_css->id; __entry->cgroup_ino =__trace_wb_assign_cgroup (wb); + +rcu_read_lock (); __entry->page_cgroup_ino =cgroup_ino (folio_memcg (folio)->css.cgroup); +rcu_read_unlock (); ), TP_printk("bdi %s[%llu]: ino=%lu memcg_id=%u cgroup_ino=%lu page_cgroup_ino=%lu",
mm/memcontrol.c +8 -6
@@ -244,7 +244,7 @@ DEFINE_STATIC_KEY_FALSE(memcg_bpf_enabled_key);EXPORT_SYMBOL (memcg_bpf_enabled_key);/** -* mem_cgroup_css_from_folio - css of the memcg associated with a folio +* get_mem_cgroup_css_from_folio - acquire a css of the memcg associated with a folio * @folio: folio of interest * * If memcg is bound to the default hierarchy, css of the memcg associated
@@ -254,14 +254,16 @@ EXPORT_SYMBOL(memcg_bpf_enabled_key);* If memcg is bound to a traditional hierarchy, the css of root_mem_cgroup * is returned. */ -struct cgroup_subsys_state *mem_cgroup_css_from_folio (struct folio *folio) +struct cgroup_subsys_state *get_mem_cgroup_css_from_folio (struct folio *folio) { -struct mem_cgroup *memcg =folio_memcg (folio); +struct mem_cgroup *memcg; -if (!memcg || !cgroup_subsys_on_dfl (memory_cgrp_subsys)) - memcg = root_mem_cgroup; +if (!cgroup_subsys_on_dfl (memory_cgrp_subsys)) +return &root_mem_cgroup->css; -return &memcg->css; + memcg =get_mem_cgroup_from_folio (folio); + +return memcg ? &memcg->css : &root_mem_cgroup->css; }/**
Signature · 1 line
2.20.1
Qi Zheng
reviewerPATCH v5 11/32 mm: memcontrol: prevent memory cgroup release in count_memcg_folio_events()
In reply to Qi ZhengFrom: Muchun Song <[email protected]>
In the near future, a folio will no longer pin its corresponding memory cgroup. To ensure safety, it will only be appropriate to hold the rcu read lock or acquire a reference to the memory cgroup returned by folio_memcg(), thereby preventing it from being released.
In the current patch, the rcu read lock is employed to safeguard against the release of the memory cgroup in count_memcg_folio_events().
This serves as a preparatory measure for the reparenting of the LRU pages.
--- include/linux/memcontrol.h | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-)
include/linux/memcontrol.h +8 -3
@@ -976,10 +976,15 @@ void count_memcg_events(struct mem_cgroup *memcg, enum vm_event_item idx,static inline void count_memcg_folio_events (struct folio *folio,enum vm_event_item idx,unsigned long nr) { -struct mem_cgroup *memcg =folio_memcg (folio); +struct mem_cgroup *memcg; -if (memcg) -count_memcg_events (memcg, idx, nr); +if (!folio_memcg_charged (folio)) +return ; + +rcu_read_lock (); + memcg =folio_memcg (folio); +count_memcg_events (memcg, idx, nr); +rcu_read_unlock (); } static inline void count_memcg_events_mm(struct mm_struct *mm,
Signature · 1 line
2.20.1
Qi Zheng
reviewerPATCH v5 12/32 mm: page_io: prevent memory cgroup release in page_io module
In reply to Qi ZhengFrom: Muchun Song <[email protected]>
In the near future, a folio will no longer pin its corresponding memory cgroup. To ensure safety, it will only be appropriate to hold the rcu read lock or acquire a reference to the memory cgroup returned by folio_memcg(), thereby preventing it from being released.
In the current patch, the rcu read lock is employed to safeguard against the release of the memory cgroup in swap_writeout() and bio_associate_blkg_from_page().
This serves as a preparatory measure for the reparenting of the LRU pages.
--- mm/page_io.c | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-)
mm/page_io.c +6 -2
@@ -276,10 +276,14 @@ int swap_writeout(struct folio *folio, struct swap_iocb **swap_plug)count_mthp_stat (folio_order (folio),MTHP_STAT_ZSWPOUT ); goto out_unlock; } + +rcu_read_lock ();if (!mem_cgroup_zswap_writeback_enabled (folio_memcg (folio))) { +rcu_read_unlock ();folio_mark_dirty (folio);return AOP_WRITEPAGE_ACTIVATE ; } +rcu_read_unlock ();__swap_writepage (folio, swap_plug);return 0 ;
@@ -307,11 +311,11 @@ static void bio_associate_blkg_from_page(struct bio *bio, struct folio *folio)struct cgroup_subsys_state *css;struct mem_cgroup *memcg; - memcg =folio_memcg (folio); -if (!memcg) +if (!folio_memcg_charged (folio))return ;rcu_read_lock (); + memcg =folio_memcg (folio); css =cgroup_e_css (memcg->css.cgroup, &io_cgrp_subsys);bio_associate_blkg_from_css (bio, css);rcu_read_unlock ();
Signature · 1 line
2.20.1
Qi Zheng
reviewerPATCH v5 13/32 mm: migrate: prevent memory cgroup release in folio_migrate_mapping()
In reply to Qi ZhengFrom: Muchun Song <[email protected]>
In the near future, a folio will no longer pin its corresponding memory cgroup. To ensure safety, it will only be appropriate to hold the rcu read lock or acquire a reference to the memory cgroup returned by folio_memcg(), thereby preventing it from being released.
In __folio_migrate_mapping(), the rcu read lock is employed to safeguard against the release of the memory cgroup in folio_migrate_mapping().
This serves as a preparatory measure for the reparenting of the LRU pages.
--- mm/migrate.c | 2 ++ 1 file changed, 2 insertions(+)
mm/migrate.c +2 -0
@@ -672,6 +672,7 @@ static int __folio_migrate_mapping(struct address_space *mapping,struct lruvec *old_lruvec, *new_lruvec;struct mem_cgroup *memcg; +rcu_read_lock (); memcg =folio_memcg (folio); old_lruvec =mem_cgroup_lruvec (memcg, oldzone->zone_pgdat); new_lruvec =mem_cgroup_lruvec (memcg, newzone->zone_pgdat);
@@ -699,6 +700,7 @@ static int __folio_migrate_mapping(struct address_space *mapping,mod_lruvec_state (new_lruvec,NR_FILE_DIRTY , nr);__mod_zone_page_state (newzone,NR_ZONE_WRITE_PENDING , nr); } +rcu_read_unlock (); }local_irq_enable ();
Signature · 1 line
2.20.1
Qi Zheng
reviewerPATCH v5 14/32 mm: mglru: prevent memory cgroup release in mglru
In reply to Qi ZhengFrom: Muchun Song <[email protected]>
In the near future, a folio will no longer pin its corresponding memory cgroup. To ensure safety, it will only be appropriate to hold the rcu read lock or acquire a reference to the memory cgroup returned by folio_memcg(), thereby preventing it from being released.
In the current patch, the rcu read lock is employed to safeguard against the release of the memory cgroup in mglru.
This serves as a preparatory measure for the reparenting of the LRU pages.
--- mm/vmscan.c | 22 ++++++++++++++++------ 1 file changed, 16 insertions(+), 6 deletions(-)
mm/vmscan.c +16 -6
@@ -3443,8 +3443,10 @@ static struct folio *get_pfn_folio(unsigned long pfn, struct mem_cgroup *memcg,if (folio_nid (folio) != pgdat->node_id)return NULL ; +rcu_read_lock ();if (folio_memcg (folio) != memcg) -return NULL ; + folio =NULL ; +rcu_read_unlock ();return folio; }
@@ -4202,12 +4204,12 @@ bool lru_gen_look_around(struct page_vma_mapped_walk *pvmw)unsigned long addr = pvmw->address;struct vm_area_struct *vma = pvmw->vma;struct folio *folio =pfn_folio (pvmw->pfn); -struct mem_cgroup *memcg =folio_memcg (folio); +struct mem_cgroup *memcg;struct pglist_data *pgdat =folio_pgdat (folio); -struct lruvec *lruvec =mem_cgroup_lruvec (memcg, pgdat); -struct lru_gen_mm_state *mm_state =get_mm_state (lruvec); -DEFINE_MAX_SEQ (lruvec); -int gen =lru_gen_from_seq (max_seq); +struct lruvec *lruvec; +struct lru_gen_mm_state *mm_state; +unsigned long max_seq; +int gen;lockdep_assert_held (pvmw->ptl);VM_WARN_ON_ONCE_FOLIO (folio_test_lru (folio), folio);
@@ -4242,6 +4244,12 @@ bool lru_gen_look_around(struct page_vma_mapped_walk *pvmw) } } + memcg =get_mem_cgroup_from_folio (folio); + lruvec =mem_cgroup_lruvec (memcg, pgdat); + max_seq =READ_ONCE ((lruvec)->lrugen.max_seq); + gen =lru_gen_from_seq (max_seq); + mm_state =get_mm_state (lruvec); +lazy_mmu_mode_enable (); pte -= (addr - start) /PAGE_SIZE ;
@@ -4282,6 +4290,8 @@ bool lru_gen_look_around(struct page_vma_mapped_walk *pvmw)if (mm_state &&suitable_to_scan (i, young))update_bloom_filter (mm_state, max_seq, pvmw->pmd); +mem_cgroup_put (memcg); +return true; }
Signature · 1 line
2.20.1
Qi Zheng
reviewerPATCH v5 15/32 mm: memcontrol: prevent memory cgroup release in mem_cgroup_swap_full()
In reply to Qi ZhengFrom: Muchun Song <[email protected]>
In the near future, a folio will no longer pin its corresponding memory cgroup. To ensure safety, it will only be appropriate to hold the rcu read lock or acquire a reference to the memory cgroup returned by folio_memcg(), thereby preventing it from being released.
In the current patch, the rcu read lock is employed to safeguard against the release of the memory cgroup in mem_cgroup_swap_full().
This serves as a preparatory measure for the reparenting of the LRU pages.
--- mm/memcontrol.c | 18 ++++++++++-------- 1 file changed, 10 insertions(+), 8 deletions(-)
mm/memcontrol.c +10 -8
@@ -5345,27 +5345,29 @@ long mem_cgroup_get_nr_swap_pages(struct mem_cgroup *memcg)bool mem_cgroup_swap_full (struct folio *folio) {struct mem_cgroup *memcg; +bool ret = false;VM_BUG_ON_FOLIO (!folio_test_locked (folio), folio);if (vm_swap_full ())return true; -if (do_memsw_account ()) -return false; +if (do_memsw_account () || !folio_memcg_charged (folio)) +return ret; +rcu_read_lock (); memcg =folio_memcg (folio); -if (!memcg) -return false; -for (; !mem_cgroup_is_root (memcg); memcg =parent_mem_cgroup (memcg)) {unsigned long usage =page_counter_read (&memcg->swap);if (usage *2 >=READ_ONCE (memcg->swap.high) || - usage *2 >=READ_ONCE (memcg->swap.max)) -return true; + usage *2 >=READ_ONCE (memcg->swap.max)) { + ret = true; +break ; + } } +rcu_read_unlock (); -return false; +return ret; }static int __init setup_swap_account (char *s)
Signature · 1 line
2.20.1
Qi Zheng
reviewerPATCH v5 16/32 mm: workingset: prevent memory cgroup release in lru_gen_eviction()
In reply to Qi ZhengFrom: Muchun Song <[email protected]>
In the near future, a folio will no longer pin its corresponding memory cgroup. To ensure safety, it will only be appropriate to hold the rcu read lock or acquire a reference to the memory cgroup returned by folio_memcg(), thereby preventing it from being released.
In the current patch, the rcu read lock is employed to safeguard against the release of the memory cgroup in lru_gen_eviction().
This serves as a preparatory measure for the reparenting of the LRU pages.
--- mm/workingset.c | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-)
mm/workingset.c +7 -2
@@ -244,12 +244,15 @@ static void *lru_gen_eviction(struct folio *folio)int refs =folio_lru_refs (folio);bool workingset =folio_test_workingset (folio);int tier =lru_tier_from_refs (refs, workingset); -struct mem_cgroup *memcg =folio_memcg (folio); +struct mem_cgroup *memcg;struct pglist_data *pgdat =folio_pgdat (folio); +unsigned short memcg_id;BUILD_BUG_ON (LRU_GEN_WIDTH +LRU_REFS_WIDTH >BITS_PER_LONG -max (EVICTION_SHIFT ,EVICTION_SHIFT_ANON )); +rcu_read_lock (); + memcg =folio_memcg (folio); lruvec =mem_cgroup_lruvec (memcg, pgdat); lrugen = &lruvec->lrugen; min_seq =READ_ONCE (lrugen->min_seq[type]);
@@ -257,8 +260,10 @@ static void *lru_gen_eviction(struct folio *folio) hist =lru_hist_from_seq (min_seq);atomic_long_add (delta, &lrugen->evicted[hist][type][tier]); + memcg_id =mem_cgroup_private_id (memcg); +rcu_read_unlock (); -return pack_shadow (mem_cgroup_private_id (memcg), pgdat, token, workingset, type); +return pack_shadow (memcg_id, pgdat, token, workingset, type); }/*
Signature · 1 line
2.20.1
Qi Zheng
reviewerPATCH v5 17/32 mm: thp: prevent memory cgroup release in folio_split_queue_lock{_irqsave}()
In reply to Qi ZhengFrom: Qi Zheng <[email protected]>
In the near future, a folio will no longer pin its corresponding memory cgroup. To ensure safety, it will only be appropriate to hold the rcu read lock or acquire a reference to the memory cgroup returned by folio_memcg(), thereby preventing it from being released.
In the current patch, the rcu read lock is employed to safeguard against the release of the memory cgroup in folio_split_queue_lock{_irqsave}().
--- mm/huge_memory.c | 20 ++++++++++++++++++-- 1 file changed, 18 insertions(+), 2 deletions(-)
mm/huge_memory.c +18 -2
@@ -1157,13 +1157,29 @@ split_queue_lock_irqsave(int nid, struct mem_cgroup *memcg, unsigned long *flagsstatic struct deferred_split *folio_split_queue_lock (struct folio *folio) { -return split_queue_lock (folio_nid (folio),folio_memcg (folio)); +struct deferred_split *queue; + +rcu_read_lock (); + queue =split_queue_lock (folio_nid (folio),folio_memcg (folio)); +/* +* The memcg destruction path is acquiring the split queue lock for +* reparenting. Once you have it locked, it's safe to drop the rcu lock. +*/ +rcu_read_unlock (); + +return queue; }static struct deferred_split *folio_split_queue_lock_irqsave (struct folio *folio,unsigned long *flags) { -return split_queue_lock_irqsave (folio_nid (folio),folio_memcg (folio), flags); +struct deferred_split *queue; + +rcu_read_lock (); + queue =split_queue_lock_irqsave (folio_nid (folio),folio_memcg (folio), flags); +rcu_read_unlock (); + +return queue; }static inline void split_queue_unlock (struct deferred_split *queue)
Signature · 1 line
2.20.1
Qi Zheng
reviewerPATCH v5 18/32 mm: zswap: prevent memory cgroup release in zswap_compress()
In reply to Qi ZhengFrom: Qi Zheng <[email protected]>
In the near future, a folio will no longer pin its corresponding memory cgroup. To ensure safety, it will only be appropriate to hold the rcu read lock or acquire a reference to the memory cgroup returned by folio_memcg(), thereby preventing it from being released.
In the current patch, the rcu read lock is employed to safeguard against the release of the memory cgroup in zswap_compress().
--- mm/zswap.c | 3 +++ 1 file changed, 3 insertions(+)
mm/zswap.c +3 -0
@@ -893,11 +893,14 @@ static bool zswap_compress(struct page *page, struct zswap_entry *entry,* to the active LRU list in the case. */ if (comp_ret || !dlen || dlen >=PAGE_SIZE ) { +rcu_read_lock ();if (!mem_cgroup_zswap_writeback_enabled (folio_memcg (page_folio (page)))) { +rcu_read_unlock (); comp_ret = comp_ret ? comp_ret : -EINVAL ; goto unlock; } +rcu_read_unlock (); comp_ret =0 ; dlen =PAGE_SIZE ; dst =kmap_local_page (page);
Signature · 1 line
2.20.1
Qi Zheng
reviewerPATCH v5 19/32 mm: workingset: prevent lruvec release in workingset_refault()
In reply to Qi ZhengFrom: Muchun Song <[email protected]>
In the near future, a folio will no longer pin its corresponding memory cgroup. So an lruvec returned by folio_lruvec() could be released without the rcu read lock or a reference to its memory cgroup.
In the current patch, the rcu read lock is employed to safeguard against the release of the lruvec in workingset_refault().
This serves as a preparatory measure for the reparenting of the LRU pages.
--- mm/workingset.c | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-)
mm/workingset.c +6 -2
@@ -546,6 +546,7 @@ bool workingset_test_recent(void *shadow, bool file, bool *workingset,void workingset_refault (struct folio *folio,void *shadow) {bool file =folio_is_file_lru (folio); +struct mem_cgroup *memcg;struct lruvec *lruvec;bool workingset;long nr;
@@ -567,11 +568,12 @@ void workingset_refault(struct folio *folio, void *shadow)* locked to guarantee folio_memcg() stability throughout. */ nr =folio_nr_pages (folio); - lruvec =folio_lruvec (folio); + memcg =get_mem_cgroup_from_folio (folio); + lruvec =mem_cgroup_lruvec (memcg,folio_pgdat (folio));mod_lruvec_state (lruvec,WORKINGSET_REFAULT_BASE + file, nr);if (!workingset_test_recent (shadow, file, &workingset, true)) -return ; + goto out;folio_set_active (folio);workingset_age_nonresident (lruvec, nr);
@@ -587,6 +589,8 @@ void workingset_refault(struct folio *folio, void *shadow)lru_note_cost_refault (folio);mod_lruvec_state (lruvec,WORKINGSET_RESTORE_BASE + file, nr); } +out: +mem_cgroup_put (memcg); }/**
Signature · 1 line
2.20.1
Qi Zheng
reviewerPATCH v5 20/32 mm: zswap: prevent lruvec release in zswap_folio_swapin()
In reply to Qi ZhengFrom: Muchun Song <[email protected]>
In the near future, a folio will no longer pin its corresponding memory cgroup. So an lruvec returned by folio_lruvec() could be released without the rcu read lock or a reference to its memory cgroup.
In the current patch, the rcu read lock is employed to safeguard against the release of the lruvec in zswap_folio_swapin().
This serves as a preparatory measure for the reparenting of the LRU pages.
--- mm/zswap.c | 2 ++ 1 file changed, 2 insertions(+)
mm/zswap.c +2 -0
@@ -664,8 +664,10 @@ void zswap_folio_swapin(struct folio *folio)struct lruvec *lruvec;if (folio) { +rcu_read_lock (); lruvec =folio_lruvec (folio);atomic_long_inc (&lruvec->zswap_lruvec_state.nr_disk_swapins); +rcu_read_unlock (); } }
Signature · 1 line
2.20.1
Qi Zheng
reviewerPATCH v5 21/32 mm: swap: prevent lruvec release in lru_gen_clear_refs()
In reply to Qi ZhengFrom: Muchun Song <[email protected]>
In the near future, a folio will no longer pin its corresponding memory cgroup. So an lruvec returned by folio_lruvec() could be released without the rcu read lock or a reference to its memory cgroup.
In the current patch, the rcu read lock is employed to safeguard against the release of the lruvec in lru_gen_clear_refs().
This serves as a preparatory measure for the reparenting of the LRU pages.
--- mm/swap.c | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-)
mm/swap.c +5 -3
@@ -412,18 +412,20 @@ static void lru_gen_inc_refs(struct folio *folio)static bool lru_gen_clear_refs (struct folio *folio) { -struct lru_gen_folio *lrugen;int gen =folio_lru_gen (folio);int type =folio_is_file_lru (folio); +unsigned long seq;if (gen <0 )return true;set_mask_bits (&folio->flags.f,LRU_REFS_FLAGS |BIT (PG_workingset),0 ); - lrugen = &folio_lruvec (folio)->lrugen; +rcu_read_lock (); + seq =READ_ONCE (folio_lruvec (folio)->lrugen.min_seq[type]); +rcu_read_unlock ();/* whether can do without shuffling under the LRU lock */ -return gen ==lru_gen_from_seq (READ_ONCE (lrugen->min_seq[type])); +return gen ==lru_gen_from_seq (seq); }#else /* !CONFIG_LRU_GEN */
Signature · 1 line
2.20.1
Qi Zheng
reviewerPATCH v5 22/32 mm: workingset: prevent lruvec release in workingset_activation()
In reply to Qi ZhengFrom: Muchun Song <[email protected]>
In the near future, a folio will no longer pin its corresponding memory cgroup. So an lruvec returned by folio_lruvec() could be released without the rcu read lock or a reference to its memory cgroup.
In the current patch, the rcu read lock is employed to safeguard against the release of the lruvec in workingset_activation().
This serves as a preparatory measure for the reparenting of the LRU pages.
--- mm/workingset.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)
mm/workingset.c +4 -1
@@ -603,8 +603,11 @@ void workingset_activation(struct folio *folio)* Filter non-memcg pages here, e.g. unmap can call * mark_page_accessed() on VDSO pages. */ -if (mem_cgroup_disabled () ||folio_memcg_charged (folio)) +if (mem_cgroup_disabled () ||folio_memcg_charged (folio)) { +rcu_read_lock ();workingset_age_nonresident (folio_lruvec (folio),folio_nr_pages (folio)); +rcu_read_unlock (); + } }/*
Signature · 1 line
2.20.1
Qi Zheng
reviewerPATCH v5 23/32 mm: do not open-code lruvec lock
In reply to Qi ZhengFrom: Qi Zheng <[email protected]>
Now we have lruvec_unlock(), lruvec_unlock_irq() and lruvec_unlock_irqrestore(), but no the paired lruvec_lock(), lruvec_lock_irq() and lruvec_lock_irqsave().
There is currently no use case for lruvec_lock_irqsave(), so only introduce lruvec_lock_irq(), and change all open-code places to use this helper function. This looks cleaner and prepares for reparenting LRU pages, preventing user from missing RCU lock calls due to open-code lruvec lock.
--- include/linux/memcontrol.h | 5 +++++ mm/vmscan.c | 38 +++++++++++++++++++------------------- 2 files changed, 24 insertions(+), 19 deletions(-)
include/linux/memcontrol.h +5 -0
@@ -1499,6 +1499,11 @@ static inline struct lruvec *parent_lruvec(struct lruvec *lruvec)return mem_cgroup_lruvec (memcg,lruvec_pgdat (lruvec)); } +static inline void lruvec_lock_irq (struct lruvec *lruvec) +{ +spin_lock_irq (&lruvec->lru_lock); +} +static inline void lruvec_unlock (struct lruvec *lruvec) {spin_unlock (&lruvec->lru_lock);
mm/vmscan.c +19 -19
@@ -2002,7 +2002,7 @@ static unsigned long shrink_inactive_list(unsigned long nr_to_scan,lru_add_drain (); -spin_lock_irq (&lruvec->lru_lock); +lruvec_lock_irq (lruvec); nr_taken =isolate_lru_folios (nr_to_scan, lruvec, &folio_list, &nr_scanned, sc, lru);
@@ -2012,7 +2012,7 @@ static unsigned long shrink_inactive_list(unsigned long nr_to_scan,mod_lruvec_state (lruvec, item, nr_scanned);mod_lruvec_state (lruvec,PGSCAN_ANON + file, nr_scanned); -spin_unlock_irq (&lruvec->lru_lock); +lruvec_unlock_irq (lruvec);if (nr_taken ==0 )return 0 ;
@@ -2029,7 +2029,7 @@ static unsigned long shrink_inactive_list(unsigned long nr_to_scan,mod_lruvec_state (lruvec, item, nr_reclaimed);mod_lruvec_state (lruvec,PGSTEAL_ANON + file, nr_reclaimed); -spin_lock_irq (&lruvec->lru_lock); +lruvec_lock_irq (lruvec);lru_note_cost_unlock_irq (lruvec, file, stat.nr_pageout, nr_scanned - nr_reclaimed);
@@ -2108,7 +2108,7 @@ static void shrink_active_list(unsigned long nr_to_scan,lru_add_drain (); -spin_lock_irq (&lruvec->lru_lock); +lruvec_lock_irq (lruvec); nr_taken =isolate_lru_folios (nr_to_scan, lruvec, &l_hold, &nr_scanned, sc, lru);
@@ -2117,7 +2117,7 @@ static void shrink_active_list(unsigned long nr_to_scan,mod_lruvec_state (lruvec,PGREFILL , nr_scanned); -spin_unlock_irq (&lruvec->lru_lock); +lruvec_unlock_irq (lruvec);while (!list_empty (&l_hold)) {struct folio *folio;
@@ -2173,7 +2173,7 @@ static void shrink_active_list(unsigned long nr_to_scan,count_memcg_events (lruvec_memcg (lruvec),PGDEACTIVATE , nr_deactivate);mod_node_page_state (pgdat,NR_ISOLATED_ANON + file, -nr_taken); -spin_lock_irq (&lruvec->lru_lock); +lruvec_lock_irq (lruvec);lru_note_cost_unlock_irq (lruvec, file,0 , nr_rotated);trace_mm_vmscan_lru_shrink_active (pgdat->node_id, nr_taken, nr_activate, nr_deactivate, nr_rotated, sc->priority, file);
@@ -3794,9 +3794,9 @@ static void walk_mm(struct mm_struct *mm, struct lru_gen_mm_walk *walk) }if (walk->batched) { -spin_lock_irq (&lruvec->lru_lock); +lruvec_lock_irq (lruvec);reset_batch_size (walk); -spin_unlock_irq (&lruvec->lru_lock); +lruvec_unlock_irq (lruvec); }cond_resched ();
@@ -3956,7 +3956,7 @@ static bool inc_max_seq(struct lruvec *lruvec, unsigned long seq, int swappinessif (seq <READ_ONCE (lrugen->max_seq))return false; -spin_lock_irq (&lruvec->lru_lock); +lruvec_lock_irq (lruvec);VM_WARN_ON_ONCE (!seq_is_valid (lruvec));
@@ -3971,7 +3971,7 @@ static bool inc_max_seq(struct lruvec *lruvec, unsigned long seq, int swappinessif (inc_min_seq (lruvec, type, swappiness))continue ; -spin_unlock_irq (&lruvec->lru_lock); +lruvec_unlock_irq (lruvec);cond_resched (); goto restart; }
@@ -4006,7 +4006,7 @@ static bool inc_max_seq(struct lruvec *lruvec, unsigned long seq, int swappiness/* make sure preceding modifications appear */ smp_store_release (&lrugen->max_seq, lrugen->max_seq +1 ); unlock: -spin_unlock_irq (&lruvec->lru_lock); +lruvec_unlock_irq (lruvec);return success; }
@@ -4697,7 +4697,7 @@ static int evict_folios(unsigned long nr_to_scan, struct lruvec *lruvec,struct mem_cgroup *memcg =lruvec_memcg (lruvec);struct pglist_data *pgdat =lruvec_pgdat (lruvec); -spin_lock_irq (&lruvec->lru_lock); +lruvec_lock_irq (lruvec); scanned =isolate_folios (nr_to_scan, lruvec, sc, swappiness, &type, &list);
@@ -4706,7 +4706,7 @@ static int evict_folios(unsigned long nr_to_scan, struct lruvec *lruvec,if (evictable_min_seq (lrugen->min_seq, swappiness) +MIN_NR_GENS > lrugen->max_seq) scanned =0 ; -spin_unlock_irq (&lruvec->lru_lock); +lruvec_unlock_irq (lruvec);if (list_empty (&list))return scanned;
@@ -4744,9 +4744,9 @@ static int evict_folios(unsigned long nr_to_scan, struct lruvec *lruvec, walk = current->reclaim_state->mm_walk;if (walk && walk->batched) { walk->lruvec = lruvec; -spin_lock_irq (&lruvec->lru_lock); +lruvec_lock_irq (lruvec);reset_batch_size (walk); -spin_unlock_irq (&lruvec->lru_lock); +lruvec_unlock_irq (lruvec); } mod_lruvec_state(lruvec, PGDEMOTE_KSWAPD + reclaimer_offset(sc),
@@ -5178,7 +5178,7 @@ static void lru_gen_change_state(bool enabled)for_each_node (nid) {struct lruvec *lruvec =get_lruvec (memcg, nid); -spin_lock_irq (&lruvec->lru_lock); +lruvec_lock_irq (lruvec);VM_WARN_ON_ONCE (!seq_is_valid (lruvec));VM_WARN_ON_ONCE (!state_is_valid (lruvec));
@@ -5186,12 +5186,12 @@ static void lru_gen_change_state(bool enabled) lruvec->lrugen.enabled = enabled;while (!(enabled ?fill_evictable (lruvec) :drain_evictable (lruvec))) { -spin_unlock_irq (&lruvec->lru_lock); +lruvec_unlock_irq (lruvec);cond_resched (); -spin_lock_irq (&lruvec->lru_lock); +lruvec_lock_irq (lruvec); } -spin_unlock_irq (&lruvec->lru_lock); +lruvec_unlock_irq (lruvec); }cond_resched ();
Signature · 1 line
2.20.1
Qi Zheng
reviewerPATCH v5 24/32 mm: memcontrol: prepare for reparenting LRU pages for lruvec lock
In reply to Qi ZhengFrom: Muchun Song <[email protected]>
The following diagram illustrates how to ensure the safety of the folio lruvec lock when LRU folios undergo reparenting.
In the folio_lruvec_lock(folio) function:
```
rcu_read_lock();
retry:
lruvec = folio_lruvec(folio);
/* There is a possibility of folio reparenting at this point. */
spin_lock(&lruvec->lru_lock);
if (unlikely(lruvec_memcg(lruvec) != folio_memcg(folio))) {
/*
* The wrong lruvec lock was acquired, and a retry is required.
* This is because the folio resides on the parent memcg lruvec
* list.
*/
spin_unlock(&lruvec->lru_lock);
goto retry;
}
/* Reaching here indicates that folio_memcg() is stable. */
```In the memcg_reparent_objcgs(memcg) function:
```
spin_lock(&lruvec->lru_lock);
spin_lock(&lruvec_parent->lru_lock);
/* Transfer folios from the lruvec list to the parent's. */
spin_unlock(&lruvec_parent->lru_lock);
spin_unlock(&lruvec->lru_lock);
```After acquiring the lruvec lock, it is necessary to verify whether the folio has been reparented. If reparenting has occurred, the new lruvec lock must be reacquired. During the LRU folio reparenting process, the lruvec lock will also be acquired (this will be implemented in a subsequent patch). Therefore, folio_memcg() remains unchanged while the lruvec lock is held.
Given that lruvec_memcg(lruvec) is always equal to folio_memcg(folio) after the lruvec lock is acquired, the lruvec_memcg_debug() check is redundant. Hence, it is removed.
This patch serves as a preparation for the reparenting of LRU folios.
--- include/linux/memcontrol.h | 34 ++++++++++++------------ include/linux/swap.h | 3 +-- mm/compaction.c | 29 ++++++++++++++++----- mm/memcontrol.c | 53 +++++++++++++++++++------------------- mm/swap.c | 6 ++++- 5 files changed, 73 insertions(+), 52 deletions(-)
include/linux/memcontrol.h +17 -17
@@ -742,7 +742,15 @@ static inline struct lruvec *mem_cgroup_lruvec(struct mem_cgroup *memcg,* folio_lruvec - return lruvec for isolating/putting an LRU folio * @folio: Pointer to the folio. * -* This function relies on folio->mem_cgroup being stable. +* Call with rcu_read_lock() held to ensure the lifetime of the returned lruvec. +* Note that this alone will NOT guarantee the stability of the folio->lruvec +* association; the folio can be reparented to an ancestor if this races with +* cgroup deletion. +* +* Use folio_lruvec_lock() to ensure both lifetime and stability of the binding. +* Once a lruvec is locked, folio_lruvec() can be called on other folios, and +* their binding is stable if the returned lruvec matches the one the caller has +* locked. Useful for lock batching. */ static inline struct lruvec *folio_lruvec (struct folio *folio) {
@@ -765,15 +773,6 @@ struct lruvec *folio_lruvec_lock_irq(struct folio *folio);struct lruvec *folio_lruvec_lock_irqsave (struct folio *folio,unsigned long *flags); -#ifdef CONFIG_DEBUG_VM -void lruvec_memcg_debug (struct lruvec *lruvec,struct folio *folio); -#else -static inline -void lruvec_memcg_debug (struct lruvec *lruvec,struct folio *folio) -{ -} -#endif -static inline struct mem_cgroup *mem_cgroup_from_css (struct cgroup_subsys_state *css){ return css ? container_of(css, struct mem_cgroup, css) : NULL;
@@ -1199,11 +1198,6 @@ static inline struct lruvec *folio_lruvec(struct folio *folio)return &pgdat->__lruvec; } -static inline -void lruvec_memcg_debug (struct lruvec *lruvec,struct folio *folio) -{ -} -static inline struct mem_cgroup *parent_mem_cgroup (struct mem_cgroup *memcg) {return NULL ;
@@ -1262,6 +1256,7 @@ static inline struct lruvec *folio_lruvec_lock(struct folio *folio) {struct pglist_data *pgdat =folio_pgdat (folio); +rcu_read_lock ();spin_lock (&pgdat->__lruvec.lru_lock);return &pgdat->__lruvec; }
@@ -1270,6 +1265,7 @@ static inline struct lruvec *folio_lruvec_lock_irq(struct folio *folio) {struct pglist_data *pgdat =folio_pgdat (folio); +rcu_read_lock ();spin_lock_irq (&pgdat->__lruvec.lru_lock);return &pgdat->__lruvec; }
@@ -1279,6 +1275,7 @@ static inline struct lruvec *folio_lruvec_lock_irqsave(struct folio *folio, {struct pglist_data *pgdat =folio_pgdat (folio); +rcu_read_lock ();spin_lock_irqsave (&pgdat->__lruvec.lru_lock, *flagsp);return &pgdat->__lruvec; }
@@ -1501,23 +1498,26 @@ static inline struct lruvec *parent_lruvec(struct lruvec *lruvec)static inline void lruvec_lock_irq (struct lruvec *lruvec) { +rcu_read_lock ();spin_lock_irq (&lruvec->lru_lock); }static inline void lruvec_unlock (struct lruvec *lruvec) {spin_unlock (&lruvec->lru_lock); +rcu_read_unlock (); }static inline void lruvec_unlock_irq (struct lruvec *lruvec) {spin_unlock_irq (&lruvec->lru_lock); +rcu_read_unlock (); } -static inline void lruvec_unlock_irqrestore (struct lruvec *lruvec, -unsigned long flags) +static inline void lruvec_unlock_irqrestore (struct lruvec *lruvec,unsigned long flags) {spin_unlock_irqrestore (&lruvec->lru_lock, flags); +rcu_read_unlock (); }/* Test requires a stable folio->memcg binding, see folio_memcg() */
include/linux/swap.h +1 -2
@@ -312,8 +312,7 @@ extern unsigned long totalreserve_pages;/* linux/mm/swap.c */ void lru_note_cost_unlock_irq (struct lruvec *lruvec,bool file, - unsigned int nr_io, unsigned int nr_rotated) - __releases(lruvec->lru_lock); +unsigned int nr_io,unsigned int nr_rotated);void lru_note_cost_refault (struct folio *);void folio_add_lru (struct folio *);void folio_add_lru_vma (struct folio *,struct vm_area_struct *);
mm/compaction.c +23 -6
@@ -518,6 +518,24 @@ static bool compact_lock_irqsave(spinlock_t *lock, unsigned long *flags,return true; } +static struct lruvec * +compact_folio_lruvec_lock_irqsave (struct folio *folio,unsigned long *flags, +struct compact_control *cc) +{ +struct lruvec *lruvec; + +rcu_read_lock (); +retry: + lruvec =folio_lruvec (folio); +compact_lock_irqsave (&lruvec->lru_lock, flags, cc); +if (unlikely (lruvec_memcg (lruvec) !=folio_memcg (folio))) { +spin_unlock_irqrestore (&lruvec->lru_lock, *flags); + goto retry; + } + +return lruvec; +} +/* * Compaction requires the taking of some coarse locks that are potentially * very heavily contended. The lock should be periodically unlocked to avoid
@@ -839,7 +857,7 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn, {pg_data_t *pgdat = cc->zone->zone_pgdat;unsigned long nr_scanned =0 , nr_isolated =0 ; -struct lruvec *lruvec; +struct lruvec *lruvec =NULL ;unsigned long flags =0 ;struct lruvec *locked =NULL ;struct folio *folio =NULL ;
@@ -1153,18 +1171,17 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn,if (!folio_test_clear_lru (folio)) goto isolate_fail_put; - lruvec =folio_lruvec (folio); +if (locked) + lruvec =folio_lruvec (folio);/* If we already hold the lock, we can skip some rechecking */ -if (lruvec != locked) { +if (lruvec != locked || !locked) {if (locked)lruvec_unlock_irqrestore (locked, flags); -compact_lock_irqsave (&lruvec->lru_lock, &flags, cc); + lruvec =compact_folio_lruvec_lock_irqsave (folio, &flags, cc); locked = lruvec; -lruvec_memcg_debug (lruvec, folio); -/* * Try get exclusive access under lock. If marked for * skip, the scan is aborted unless the current context
mm/memcontrol.c +27 -26
93 lines not on this page · raw message
mm/swap.c +5 -1
@@ -240,6 +240,7 @@ void folio_rotate_reclaimable(struct folio *folio)void lru_note_cost_unlock_irq (struct lruvec *lruvec,bool file,unsigned int nr_io,unsigned int nr_rotated) __releases(lruvec->lru_lock) +__releases (rcu) {unsigned long cost;
@@ -253,6 +254,7 @@ void lru_note_cost_unlock_irq(struct lruvec *lruvec, bool file, cost = nr_io *SWAP_CLUSTER_MAX + nr_rotated;if (!cost) {spin_unlock_irq (&lruvec->lru_lock); +rcu_read_unlock ();return ; }
@@ -285,8 +287,10 @@ void lru_note_cost_unlock_irq(struct lruvec *lruvec, bool file,spin_unlock_irq (&lruvec->lru_lock); lruvec =parent_lruvec (lruvec); -if (!lruvec) +if (!lruvec) { +rcu_read_unlock ();break ; + }spin_lock_irq (&lruvec->lru_lock); } }
Signature · 1 line
2.20.1
Qi Zheng
reviewerPATCH v5 25/32 mm: vmscan: prepare for reparenting traditional LRU folios
In reply to Qi ZhengFrom: Qi Zheng <[email protected]>
To resolve the dying memcg issue, we need to reparent LRU folios of child memcg to its parent memcg. For traditional LRU list, each lruvec of every memcg comprises four LRU lists. Due to the symmetry of the LRU lists, it is feasible to transfer the LRU lists from a memcg to its parent memcg during the reparenting process.
This commit implements the specific function, which will be used during the reparenting process.
--- include/linux/swap.h | 21 +++++++++++++++++++++ mm/swap.c | 33 +++++++++++++++++++++++++++++++++ mm/vmscan.c | 19 ------------------- 3 files changed, 54 insertions(+), 19 deletions(-)
include/linux/swap.h +21 -0
@@ -548,6 +548,8 @@ static inline int mem_cgroup_swappiness(struct mem_cgroup *memcg) return READ_ONCE(memcg->swappiness); } + +void lru_reparent_memcg (struct mem_cgroup *memcg,struct mem_cgroup *parent,int nid);#else static inline int mem_cgroup_swappiness (struct mem_cgroup *mem) {
@@ -612,5 +614,24 @@ static inline bool mem_cgroup_swap_full(struct folio *folio) }#endif +/* for_each_managed_zone_pgdat - helper macro to iterate over all managed zones in a pgdat up to +* and including the specified highidx +* @zone: The current zone in the iterator +* @pgdat: The pgdat which node_zones are being iterated +* @idx: The index variable +* @highidx: The index of the highest zone to return +* +* This macro iterates through all managed zones up to and including the specified highidx. +* The zone iterator enters an invalid state after macro call and must be reinitialized +* before it can be used again. +*/ +#define for_each_managed_zone_pgdat (zone, pgdat, idx, highidx) \ + for ((idx) = 0, (zone) = (pgdat)->node_zones; \ + (idx) <= (highidx); \ + (idx)++, (zone)++) \ + if (!managed_zone(zone)) \ + continue; \ + else +#endif /* __KERNEL__*/ #endif /* _LINUX_SWAP_H */
mm/swap.c +33 -0
@@ -1090,6 +1090,39 @@ void folio_batch_remove_exceptionals(struct folio_batch *fbatch) fbatch->nr = j; } +#ifdef CONFIG_MEMCG +static void lruvec_reparent_lru (struct lruvec *child_lruvec, +struct lruvec *parent_lruvec, +enum lru_list lru,int nid) +{ +int zid; +struct zone *zone; + +if (lru !=LRU_UNEVICTABLE ) +list_splice_tail_init (&child_lruvec->lists[lru], &parent_lruvec->lists[lru]); + +for_each_managed_zone_pgdat (zone,NODE_DATA (nid), zid,MAX_NR_ZONES -1 ) { +unsigned long size =mem_cgroup_get_zone_lru_size (child_lruvec, lru, zid); + +mem_cgroup_update_lru_size (parent_lruvec, lru, zid, size); + } +} + +void lru_reparent_memcg (struct mem_cgroup *memcg,struct mem_cgroup *parent,int nid) +{ +enum lru_list lru; +struct lruvec *child_lruvec, *parent_lruvec; + + child_lruvec =mem_cgroup_lruvec (memcg,NODE_DATA (nid)); + parent_lruvec =mem_cgroup_lruvec (parent,NODE_DATA (nid)); + parent_lruvec->anon_cost += child_lruvec->anon_cost; + parent_lruvec->file_cost += child_lruvec->file_cost; + +for_each_lru (lru) +lruvec_reparent_lru (child_lruvec, parent_lruvec, lru, nid); +} +#endif +static const struct ctl_table swap_sysctl_table[] = { { .procname ="page-cluster" ,
mm/vmscan.c +0 -19
@@ -269,25 +269,6 @@ static int sc_swappiness(struct scan_control *sc, struct mem_cgroup *memcg) }#endif -/* for_each_managed_zone_pgdat - helper macro to iterate over all managed zones in a pgdat up to -* and including the specified highidx -* @zone: The current zone in the iterator -* @pgdat: The pgdat which node_zones are being iterated -* @idx: The index variable -* @highidx: The index of the highest zone to return -* -* This macro iterates through all managed zones up to and including the specified highidx. -* The zone iterator enters an invalid state after macro call and must be reinitialized -* before it can be used again. -*/ -#define for_each_managed_zone_pgdat (zone, pgdat, idx, highidx) \ - for ((idx) = 0, (zone) = (pgdat)->node_zones; \ - (idx) <= (highidx); \ - (idx)++, (zone)++) \ - if (!managed_zone(zone)) \ - continue; \ - else -static void set_task_reclaim_state (struct task_struct *task,struct reclaim_state *rs) {
Signature · 1 line
2.20.1
Qi Zheng
reviewerPATCH v5 26/32 mm: vmscan: prepare for reparenting MGLRU folios
In reply to Qi ZhengFrom: Qi Zheng <[email protected]>
Similar to traditional LRU folios, in order to solve the dying memcg problem, we also need to reparenting MGLRU folios to the parent memcg when memcg offline.
However, there are the following challenges:
1. Each lruvec has between MIN_NR_GENS and MAX_NR_GENS generations, the number of generations of the parent and child memcg may be different, so we cannot simply transfer MGLRU folios in the child memcg to the parent memcg as we did for traditional LRU folios.
2. The generation information is stored in folio->flags, but we cannot traverse these folios while holding the lru lock, otherwise it may cause softlockup.
3. In walk_update_folio(), the gen of folio and corresponding lru size may be updated, but the folio is not immediately moved to the corresponding lru list. Therefore, there may be folios of different generations on an LRU list.
4. In lru_gen_del_folio(), the generation to which the folio belongs is found based on the generation information in folio->flags, and the corresponding LRU size will be updated. Therefore, we need to update the lru size correctly during reparenting, otherwise the lru size may be updated incorrectly in lru_gen_del_folio().
Finally, this patch chose a compromise method, which is to splice the lru list in the child memcg to the lru list of the same generation in the parent memcg during reparenting. And in order to ensure that the parent memcg has the same generation, we need to increase the generations in the parent memcg to the MAX_NR_GENS before reparenting.
Of course, the same generation has different meanings in the parent and child memcg, this will cause confusion in the hot and cold information of folios. But other than that, this method is simple enough, the lru size is correct, and there is no need to consider some concurrency issues (such as lru_gen_del_folio()).
To prepare for the above work, this commit implements the specific functions, which will be used during reparenting.
--- include/linux/mmzone.h | 17 +++++ mm/vmscan.c | 142 +++++++++++++++++++++++++++++++++++++++++ 2 files changed, 159 insertions(+)
include/linux/mmzone.h +17 -0
@@ -637,6 +637,9 @@ void lru_gen_online_memcg(struct mem_cgroup *memcg);void lru_gen_offline_memcg (struct mem_cgroup *memcg);void lru_gen_release_memcg (struct mem_cgroup *memcg);void lru_gen_soft_reclaim (struct mem_cgroup *memcg,int nid); +void max_lru_gen_memcg (struct mem_cgroup *memcg,int nid); +bool recheck_lru_gen_max_memcg (struct mem_cgroup *memcg,int nid); +void lru_gen_reparent_memcg (struct mem_cgroup *memcg,struct mem_cgroup *parent,int nid);#else /* !CONFIG_LRU_GEN */
@@ -677,6 +680,20 @@ static inline void lru_gen_soft_reclaim(struct mem_cgroup *memcg, int nid) { } +static inline void max_lru_gen_memcg (struct mem_cgroup *memcg,int nid) +{ +} + +static inline bool recheck_lru_gen_max_memcg (struct mem_cgroup *memcg,int nid) +{ +return true; +} + +static inline +void lru_gen_reparent_memcg (struct mem_cgroup *memcg,struct mem_cgroup *parent,int nid) +{ +} +#endif /* CONFIG_LRU_GEN */ struct lruvec {
mm/vmscan.c +142 -0
148 lines not on this page · raw message
Signature · 1 line
2.20.1
Qi Zheng
reviewerPATCH v5 27/32 mm: memcontrol: refactor memcg_reparent_objcgs()
In reply to Qi ZhengFrom: Qi Zheng <[email protected]>
Refactor the memcg_reparent_objcgs() to facilitate subsequent reparenting LRU folios here.
--- mm/memcontrol.c | 29 ++++++++++++++++++++++++----- 1 file changed, 24 insertions(+), 5 deletions(-)
mm/memcontrol.c +24 -5
@@ -209,15 +209,12 @@ static struct obj_cgroup *obj_cgroup_alloc(void)return objcg; } -static void memcg_reparent_objcgs (struct mem_cgroup *memcg) +static inline struct obj_cgroup *__memcg_reparent_objcgs (struct mem_cgroup *memcg, +struct mem_cgroup *parent) {struct obj_cgroup *objcg, *iter; -struct mem_cgroup *parent =parent_mem_cgroup (memcg); objcg =rcu_replace_pointer (memcg->objcg,NULL , true); - -spin_lock_irq (&objcg_lock); -/* 1) Ready to reparent active objcg. */ list_add (&objcg->list, &memcg->objcg_list);/* 2) Reparent active objcg and already reparented objcgs to parent. */
@@ -226,7 +223,29 @@ static void memcg_reparent_objcgs(struct mem_cgroup *memcg)/* 3) Move already reparented objcgs to the parent's list */ list_splice (&memcg->objcg_list, &parent->objcg_list); +return objcg; +} + +static inline void reparent_locks (struct mem_cgroup *memcg,struct mem_cgroup *parent) +{ +spin_lock_irq (&objcg_lock); +} + +static inline void reparent_unlocks (struct mem_cgroup *memcg,struct mem_cgroup *parent) +{spin_unlock_irq (&objcg_lock); +} + +static void memcg_reparent_objcgs (struct mem_cgroup *memcg) +{ +struct obj_cgroup *objcg; +struct mem_cgroup *parent =parent_mem_cgroup (memcg); + +reparent_locks (memcg, parent); + + objcg =__memcg_reparent_objcgs (memcg, parent); + +reparent_unlocks (memcg, parent);percpu_ref_kill (&objcg->refcnt); }
Signature · 1 line
2.20.1
Qi Zheng
reviewerPATCH v5 28/32 mm: workingset: use lruvec_lru_size() to get the number of lru pages
In reply to Qi ZhengFrom: Qi Zheng <[email protected]>
For cgroup v2, count_shadow_nodes() is the only place to read non-hierarchical stats (lruvec_stats->state_local). To avoid the need to consider cgroup v2 during subsequent non-hierarchical stats reparenting, use lruvec_lru_size() instead of lruvec_page_state_local() to get the number of lru pages.
For NR_SLAB_RECLAIMABLE_B and NR_SLAB_UNRECLAIMABLE_B cases, it appears that the statistics here have already been problematic for a while since slab pages have been reparented. So just ignore it for now.
--- include/linux/swap.h | 1 + mm/vmscan.c | 3 +-- mm/workingset.c | 5 +++-- 3 files changed, 5 insertions(+), 4 deletions(-)
include/linux/swap.h +1 -0
@@ -354,6 +354,7 @@ extern void swap_setup(void);extern unsigned long zone_reclaimable_pages (struct zone *zone);extern unsigned long try_to_free_pages (struct zonelist *zonelist,int order,gfp_t gfp_mask,nodemask_t *mask); +unsigned long lruvec_lru_size (struct lruvec *lruvec,enum lru_list lru,int zone_idx);#define MEMCG_RECLAIM_MAY_SWAP (1 << 1)#define MEMCG_RECLAIM_PROACTIVE (1 << 2)
mm/vmscan.c +1 -2
@@ -390,8 +390,7 @@ unsigned long zone_reclaimable_pages(struct zone *zone)* @lru: lru to use * @zone_idx: zones to consider (use MAX_NR_ZONES - 1 for the whole LRU list) */ -static unsigned long lruvec_lru_size (struct lruvec *lruvec,enum lru_list lru, -int zone_idx) +unsigned long lruvec_lru_size (struct lruvec *lruvec,enum lru_list lru,int zone_idx) {unsigned long size =0 ;int zid;
mm/workingset.c +3 -2
@@ -691,9 +691,10 @@ static unsigned long count_shadow_nodes(struct shrinker *shrinker,mem_cgroup_flush_stats_ratelimited (sc->memcg); lruvec =mem_cgroup_lruvec (sc->memcg,NODE_DATA (sc->nid)); +for (pages =0 , i =0 ; i <NR_LRU_LISTS ; i++) - pages +=lruvec_page_state_local (lruvec, -NR_LRU_BASE + i); + pages +=lruvec_lru_size (lruvec, i,MAX_NR_ZONES -1 ); + pages +=lruvec_page_state_local ( lruvec,NR_SLAB_RECLAIMABLE_B ) >>PAGE_SHIFT ; pages += lruvec_page_state_local(
Signature · 1 line
2.20.1
Qi Zheng
reviewerPATCH v5 29/32 mm: memcontrol: prepare for reparenting non-hierarchical stats
In reply to Qi ZhengFrom: Qi Zheng <[email protected]>
To resolve the dying memcg issue, we need to reparent LRU folios of child memcg to its parent memcg. This could cause problems for non-hierarchical stats.
As Yosry Ahmed pointed out:
``` In short, if memory is charged to a dying cgroup at the time of reparenting, when the memory gets uncharged the stats updates will occur at the parent. This will update both hierarchical and non-hierarchical stats of the parent, which would corrupt the parent's non-hierarchical stats (because those counters were never incremented when the memory was charged). ```
Now we have the following two types of non-hierarchical stats, and they are only used in CONFIG_MEMCG_V1:
a. memcg->vmstats->state_local[i]
b. pn->lruvec_stats->state_local[i]
To ensure that these non-hierarchical stats work properly, we need to reparent these non-hierarchical stats after reparenting LRU folios. To this end, this commit makes the following preparations:
1. implement reparent_state_local() to reparent non-hierarchical stats
2. make css_killed_work_fn() to be called in rcu work, and implement get_non_dying_memcg_start() and get_non_dying_memcg_end() to avoid race between mod_memcg_state()/mod_memcg_lruvec_state()
and reparent_state_local()
3. change these non-hierarchical stats to atomic_long_t type to avoid race between mem_cgroup_stat_aggregate() and reparent_state_local()
--- include/linux/memcontrol.h | 1 + kernel/cgroup/cgroup.c | 8 +-- mm/memcontrol-v1.c | 16 ++++++ mm/memcontrol-v1.h | 8 +++ mm/memcontrol.c | 115 ++++++++++++++++++++++++++++++++++--- 5 files changed, 136 insertions(+), 12 deletions(-)
include/linux/memcontrol.h +1 -0
@@ -957,6 +957,7 @@ static inline void mod_memcg_page_state(struct page *page,unsigned long memcg_events (struct mem_cgroup *memcg,int event);unsigned long memcg_page_state (struct mem_cgroup *memcg,int idx); +unsigned long memcg_page_state_output (struct mem_cgroup *memcg,int item);bool memcg_stat_item_valid (int idx);bool memcg_vm_event_item_valid (enum vm_event_item idx);
kernel/cgroup/cgroup.c +4 -4
@@ -6044,8 +6044,8 @@ int cgroup_mkdir(struct kernfs_node *parent_kn, const char *name, umode_t mode)*/ static void css_killed_work_fn (struct work_struct *work) { -struct cgroup_subsys_state *css = - container_of(work, struct cgroup_subsys_state, destroy_work); +struct cgroup_subsys_state *css =container_of (to_rcu_work (work), + struct cgroup_subsys_state, destroy_rwork);cgroup_lock ();
@@ -6066,8 +6066,8 @@ static void css_killed_ref_fn(struct percpu_ref *ref) container_of(ref, struct cgroup_subsys_state, refcnt);if (atomic_dec_and_test (&css->online_cnt)) { -INIT_WORK (&css->destroy_work, css_killed_work_fn); -queue_work (cgroup_offline_wq, &css->destroy_work); +INIT_RCU_WORK (&css->destroy_rwork, css_killed_work_fn); +queue_rcu_work (cgroup_offline_wq, &css->destroy_rwork); } }
mm/memcontrol-v1.c +16 -0
@@ -1897,6 +1897,22 @@ static const unsigned int memcg1_events[] = { PGMAJFAULT, }; +void reparent_memcg1_state_local (struct mem_cgroup *memcg,struct mem_cgroup *parent) +{ +int i; + +for (i =0 ; i <ARRAY_SIZE (memcg1_stats); i++) +reparent_memcg_state_local (memcg, parent, memcg1_stats[i]); +} + +void reparent_memcg1_lruvec_state_local (struct mem_cgroup *memcg,struct mem_cgroup *parent) +{ +int i; + +for (i =0 ; i <NR_LRU_LISTS ; i++) +reparent_memcg_lruvec_state_local (memcg, parent, i); +} +void memcg1_stat_format (struct mem_cgroup *memcg,struct seq_buf *s) {unsigned long memory, memsw;
mm/memcontrol-v1.h +8 -0
@@ -45,6 +45,7 @@ static inline bool do_memsw_account(void)unsigned long memcg_events_local (struct mem_cgroup *memcg,int event);unsigned long memcg_page_state_local (struct mem_cgroup *memcg,int idx); +void mod_memcg_page_state_local (struct mem_cgroup *memcg,int idx,unsigned long val);unsigned long memcg_page_state_local_output (struct mem_cgroup *memcg,int item);bool memcg1_alloc_events (struct mem_cgroup *memcg);void memcg1_free_events (struct mem_cgroup *memcg);
@@ -77,6 +78,13 @@ void memcg1_uncharge_batch(struct mem_cgroup *memcg, unsigned long pgpgout, unsigned long nr_memory, int nid);void memcg1_stat_format (struct mem_cgroup *memcg,struct seq_buf *s); +void reparent_memcg1_state_local (struct mem_cgroup *memcg,struct mem_cgroup *parent); +void reparent_memcg1_lruvec_state_local (struct mem_cgroup *memcg,struct mem_cgroup *parent); + +void reparent_memcg_state_local (struct mem_cgroup *memcg, +struct mem_cgroup *parent,int idx); +void reparent_memcg_lruvec_state_local (struct mem_cgroup *memcg, +struct mem_cgroup *parent,int idx);void memcg1_account_kmem (struct mem_cgroup *memcg,int nr_pages);static inline bool memcg1_tcpmem_active (struct mem_cgroup *memcg)
mm/memcontrol.c +107 -8
187 lines not on this page · raw message
Signature · 1 line
2.20.1
Yosry Ahmed
maintainerRe: [PATCH v5 29/32] mm: memcontrol: prepare for reparenting non-hierarchical stats
Quoted 15 lines
mm/memcontrol.c +11
-0@@ -473,6 +493,29 @@ unsigned long lruvec_page_state_local(struct lruvec *lruvec,return x; } +#ifdef CONFIG_MEMCG_V1 +void reparent_memcg_lruvec_state_local (struct mem_cgroup *memcg, +struct mem_cgroup *parent,int idx) +{ +int i =memcg_stats_index (idx); +int nid; + +if (WARN_ONCE (BAD_STAT_IDX (i),"%s: missing stat item %d\n" , __func__, idx)) +return ; + +for_each_node (nid) {
mm/memcontrol.c unsigned long lruvec_page_state_local(struct lruvec *lruvec, · line 516
Quoted 10 lines
mm/memcontrol.c +10
-0+struct lruvec *child_lruvec =mem_cgroup_lruvec (memcg,NODE_DATA (nid)); +struct lruvec *parent_lruvec =mem_cgroup_lruvec (parent,NODE_DATA (nid)); +struct mem_cgroup_per_node *parent_pn; +unsigned long value =lruvec_page_state_local (child_lruvec, idx); + + parent_pn = container_of(parent_lruvec, struct mem_cgroup_per_node, lruvec); + +atomic_long_add (value, &(parent_pn->lruvec_stats->state_local[i])); + } +}
Did you measure the impact of making state_local atomic on the flush path? It's a slow path but we've seen pain from it being too slow before, because it extends the critical section of the rstat flush lock.
Can we keep this non-atomic and use mod_memcg_lruvec_state() here? It will update the stat on the local counter and it will be added to state_local in the flush path when needed. We can even force another flush in reparent_state_local () after reparenting is completed, if we want to avoid leaving a potentially large stat update pending, as it can be missed by mem_cgroup_flush_stats_ratelimited().
Same for reparent_memcg_state_local(), we can probably use mod_memcg_state()?
Shakeel Butt
maintainerRe: [PATCH v5 29/32] mm: memcontrol: prepare for reparenting non-hierarchical stats
Quoted 22 lines · Yosry Ahmed
Quoted 22 lines
mm/memcontrol.c +18
-0@@ -473,6 +493,29 @@ unsigned long lruvec_page_state_local(struct lruvec *lruvec,return x; } +#ifdef CONFIG_MEMCG_V1 +void reparent_memcg_lruvec_state_local (struct mem_cgroup *memcg, +struct mem_cgroup *parent,int idx) +{ +int i =memcg_stats_index (idx); +int nid; + +if (WARN_ONCE (BAD_STAT_IDX (i),"%s: missing stat item %d\n" , __func__, idx)) +return ; + +for_each_node (nid) { +struct lruvec *child_lruvec =mem_cgroup_lruvec (memcg,NODE_DATA (nid)); +struct lruvec *parent_lruvec =mem_cgroup_lruvec (parent,NODE_DATA (nid)); +struct mem_cgroup_per_node *parent_pn; +unsigned long value =lruvec_page_state_local (child_lruvec, idx); + + parent_pn = container_of(parent_lruvec, struct mem_cgroup_per_node, lruvec); +
Quoted 8 lines
mm/memcontrol.c unsigned long lruvec_page_state_local(struct lruvec *lruvec, · line 516
Quoted 3 lines
mm/memcontrol.c +3
-0+atomic_long_add (value, &(parent_pn->lruvec_stats->state_local[i])); + } +}Did you measure the impact of making
state_localatomic on the flush path? It's a slow path but we've seen pain from it being too slow before, because it extends the critical section of the rstat flush lock.
Qi, please measure the impact on flushing and if no impact then no need to do anything as I don't want anymore churn in this series.
Quoted 9 lines
Can we keep this non-atomic and use
mod_memcg_lruvec_state()here? It will update the stat on the local counter and it will be added tostate_localin the flush path when needed. We can even force another flush inreparent_state_local() after reparenting is completed, if we want to avoid leaving a potentially large stat update pending, as it can be missed bymem_cgroup_flush_stats_ratelimited().Same for
reparent_memcg_state_local(), we can probably usemod_memcg_state()?
Yosry, do you mind sending the patch you are thinking about over this series?
Qi Zheng
reviewerRe: [PATCH v5 29/32] mm: memcontrol: prepare for reparenting non-hierarchical stats
Quoted 32 lines · Shakeel Butt
Quoted 22 lines · Yosry Ahmed
Quoted 22 lines
mm/memcontrol.c +18
-0@@ -473,6 +493,29 @@ unsigned long lruvec_page_state_local(struct lruvec *lruvec,return x; } +#ifdef CONFIG_MEMCG_V1 +void reparent_memcg_lruvec_state_local (struct mem_cgroup *memcg, +struct mem_cgroup *parent,int idx) +{ +int i =memcg_stats_index (idx); +int nid; + +if (WARN_ONCE (BAD_STAT_IDX (i),"%s: missing stat item %d\n" , __func__, idx)) +return ; + +for_each_node (nid) { +struct lruvec *child_lruvec =mem_cgroup_lruvec (memcg,NODE_DATA (nid)); +struct lruvec *parent_lruvec =mem_cgroup_lruvec (parent,NODE_DATA (nid)); +struct mem_cgroup_per_node *parent_pn; +unsigned long value =lruvec_page_state_local (child_lruvec, idx); + + parent_pn = container_of(parent_lruvec, struct mem_cgroup_per_node, lruvec); +Quoted 8 lines
mm/memcontrol.c unsigned long lruvec_page_state_local(struct lruvec *lruvec, · line 516
Quoted 3 lines
mm/memcontrol.c +3
-0+atomic_long_add (value, &(parent_pn->lruvec_stats->state_local[i])); + } +}Did you measure the impact of making
state_localatomic on the flush path? It's a slow path but we've seen pain from it being too slow before, because it extends the critical section of the rstat flush lock.
Quoted 1 line
Qi, please measure the impact on flushing and if no impact then no need to do
OK, will try to do it.
Quoted 1 line
anything as I don't want anymore churn in this series.
Agree.
Quoted 1 line
Yosry Ahmed
maintainerRe: [PATCH v5 29/32] mm: memcontrol: prepare for reparenting non-hierarchical stats
In reply to Shakeel ButtQuoted 16 lines
Quoted 4 lines
Did you measure the impact of making
state_localatomic on the flush path? It's a slow path but we've seen pain from it being too slow before, because it extends the critical section of the rstat flush lock.Qi, please measure the impact on flushing and if no impact then no need to do anything as I don't want anymore churn in this series.
Quoted 8 lines
Can we keep this non-atomic and use
mod_memcg_lruvec_state()here? It will update the stat on the local counter and it will be added tostate_localin the flush path when needed. We can even force another flush inreparent_state_local() after reparenting is completed, if we want to avoid leaving a potentially large stat update pending, as it can be missed bymem_cgroup_flush_stats_ratelimited().
Quoted 3 lines
Quoted 1 line
Same for
reparent_memcg_state_local(), we can probably usemod_memcg_state()?Yosry, do you mind sending the patch you are thinking about over this series?
Honestly, I'd rather squash it into this patch if possible. It avoids churn in the history (switch to atomics and back), and is arguably simpler than checking for regressions in the flush path.
What I have in mind is the diff below (build tested only). Qi, would you be able to test this? It applies directly on this patch in mm-new:
mm/memcontrol.c +0 -0
@@ -234,11 +234,18 @@ static inline void reparent_state_local(struct
mem_cgroup *memcg, struct mem_cgr
if (cgroup_subsys_on_dfl(memory_cgrp_subsys))
return;+ /*
+ * Reparent stats exposed non-hierarchically. Flush @memcg's
stats first to
+ * read its stats accurately , and conservatively flush @parent's stats
+ * after reparenting to avoid hiding a potentially large stat update
+ * (e.g. from callers of mem_cgroup_flush_stats_ratelimited()).
+ */
__mem_cgroup_flush_stats(memcg, true);- /* The following counts are all non-hierarchical and need to
be reparented. */
reparent_memcg1_state_local(memcg, parent);
reparent_memcg1_lruvec_state_local(memcg, parent);+
+ __mem_cgroup_flush_stats(parent, true);
}
#else
static inline void reparent_state_local(struct mem_cgroup *memcg,
struct mem_cgroup *parent)
@@ -442,7 +449,7 @@ struct lruvec_stats {
long state[NR_MEMCG_NODE_STAT_ITEMS];
/* Non-hierarchical (CPU aggregated) state */- atomic_long_t state_local[NR_MEMCG_NODE_STAT_ITEMS];
+ long state_local[NR_MEMCG_NODE_STAT_ITEMS];
/* Pending child counts during tree propagation */
long state_pending[NR_MEMCG_NODE_STAT_ITEMS];
@@ -485,7 +492,7 @@ unsigned long lruvec_page_state_local(struct lruvec *lruvec,
return 0;
pn = container_of(lruvec, struct mem_cgroup_per_node, lruvec);
- x = atomic_long_read(&(pn->lruvec_stats->state_local[i]));
+ x = READ_ONCE(pn->lruvec_stats->state_local[i]);
#ifdef CONFIG_SMP
if (x < 0)
x = 0;
@@ -493,6 +500,10 @@ unsigned long lruvec_page_state_local(struct
lruvec *lruvec,
return x;}
+static void mod_memcg_lruvec_state(struct lruvec *lruvec,
+ enum node_stat_item idx,
+ int val);
+
#ifdef CONFIG_MEMCG_V1
void reparent_memcg_lruvec_state_local(struct mem_cgroup *memcg,
struct mem_cgroup *parent, int idx)
@@ -506,12 +517,10 @@ void reparent_memcg_lruvec_state_local(struct mem_cgroup *memcg,
for_each_node(nid) {
struct lruvec *child_lruvec = mem_cgroup_lruvec(memcg,NODE_DATA(nid));
struct lruvec *parent_lruvec =mem_cgroup_lruvec(parent, NODE_DATA(nid));
- struct mem_cgroup_per_node *parent_pn;
unsigned long value =lruvec_page_state_local(child_lruvec, idx);
- parent_pn = container_of(parent_lruvec, structmem_cgroup_per_node, lruvec);
-
- atomic_long_add(value,
&(parent_pn->lruvec_stats->state_local[i]));
+ mod_memcg_lruvec_state(child_lruvec, idx, -value);
+ mod_memcg_lruvec_state(parent_lruvec, idx, value);
}
}
#endif
@@ -598,7 +607,7 @@ struct memcg_vmstats {
unsigned long events[NR_MEMCG_EVENTS];
/* Non-hierarchical (CPU aggregated) page state & events */
- atomic_long_t state_local[MEMCG_VMSTAT_SIZE];
+ long state_local[MEMCG_VMSTAT_SIZE];
unsigned long events_local[NR_MEMCG_EVENTS];/* Pending child counts during tree propagation */
@@ -835,7 +844,7 @@ unsigned long memcg_page_state_local(struct mem_cgroup *memcg, int idx)
if (WARN_ONCE(BAD_STAT_IDX(i), "%s: missing stat item %d\n",__func__, idx))
return 0;
- x = atomic_long_read(&(memcg->vmstats->state_local[i]));
+ x = READ_ONCE(memcg->vmstats->state_local[i]);
#ifdef CONFIG_SMP
if (x < 0)
x = 0;
@@ -852,7 +861,8 @@ void reparent_memcg_state_local(struct mem_cgroup *memcg,
if (WARN_ONCE(BAD_STAT_IDX(i), "%s: missing stat item %d\n",
__func__, idx))
return;- atomic_long_add(value, &(parent->vmstats->state_local[i])); + mod_memcg_state(memcg, idx, -value); + mod_memcg_state(parent, idx, value);
}
#endif
@@ -4174,8 +4184,6 @@ struct aggregate_control {
long *aggregate;
/* pointer to the non-hierarchichal (CPU aggregated) counters */
long *local;
- /* pointer to the atomic non-hierarchichal (CPU aggregated) counters */
- atomic_long_t *alocal;
/* pointer to the pending child counters during tree propagation */
long *pending;
/* pointer to the parent's pending counters, could be NULL */
@@ -4213,12 +4221,8 @@ static void mem_cgroup_stat_aggregate(struct
aggregate_control *ac)
}
/* Aggregate counts on this level and propagate upwards */- if (delta_cpu) {
- if (ac->local)
- ac->local[i] += delta_cpu;
- else if (ac->alocal)
- atomic_long_add(delta_cpu, &(ac->alocal[i]));
- }
+ if (delta_cpu)
+ ac->local[i] += delta_cpu;
if (delta) {
ac->aggregate[i] += delta;@@ -4289,8 +4293,7 @@ static void mem_cgroup_css_rstat_flush(struct cgroup_subsys_state *css, int cpu)
ac = (struct aggregate_control) {
.aggregate = memcg->vmstats->state,
- .local = NULL,
- .alocal = memcg->vmstats->state_local,
+ .local = memcg->vmstats->state_local,
.pending = memcg->vmstats->state_pending,
.ppending = parent ? parent->vmstats->state_pending : NULL,
.cstat = statc->state,@@ -4323,8 +4326,7 @@ static void mem_cgroup_css_rstat_flush(struct cgroup_subsys_state *css, int cpu)
ac = (struct aggregate_control) {
.aggregate = lstats->state,
- .local = NULL,
- .alocal = lstats->state_local,
+ .local = lstats->state_local,
.pending = lstats->state_pending,
.ppending = plstats ? plstats->state_pending : NULL,
.cstat = lstatc->state,Shakeel Butt
maintainerRe: [PATCH v5 29/32] mm: memcontrol: prepare for reparenting non-hierarchical stats
Quoted 16 lines · Yosry Ahmed
Quoted 16 lines
Quoted 4 lines
Did you measure the impact of making
state_localatomic on the flush path? It's a slow path but we've seen pain from it being too slow before, because it extends the critical section of the rstat flush lock.Qi, please measure the impact on flushing and if no impact then no need to do anything as I don't want anymore churn in this series.
Quoted 8 lines
Can we keep this non-atomic and use
mod_memcg_lruvec_state()here? It will update the stat on the local counter and it will be added tostate_localin the flush path when needed. We can even force another flush inreparent_state_local() after reparenting is completed, if we want to avoid leaving a potentially large stat update pending, as it can be missed bymem_cgroup_flush_stats_ratelimited().
Quoted 7 lines
Quoted 3 lines
Quoted 1 line
Same for
reparent_memcg_state_local(), we can probably usemod_memcg_state()?Yosry, do you mind sending the patch you are thinking about over this series?
Honestly, I'd rather squash it into this patch if possible. It avoids churn in the history (switch to atomics and back), and is arguably simpler than checking for regressions in the flush path.
Yup, let's squash it into the original patch. Please add your sign-off tag.
Quoted 3 lines
What I have in mind is the diff below (build tested only). Qi, would you be able to test this? It applies directly on this patch in mm-new:
Qi, please squash this diff into the patch and test. You might need to change the subsequent patches. Once you are done with testing, you can post the diffs for those in reply to those patches and we will ask Andrew to squash into orinigal ones.
The diff looks good to me though.
Quoted 158 lines
mm/memcontrol.c +0
-0@@ -234,11 +234,18 @@ static inline void reparent_state_local(struct
mem_cgroup*memcg, structmem_cgrif (cgroup_subsys_on_dfl(memory_cgrp_subsys)) return;mm/memcontrol.c +2
-0+/* +* Reparent stats exposed non-hierarchically. Flush @memcg's stats first to
mm/memcontrol.c +4
-1+* read its stats accurately , and conservatively flush @parent's stats +* after reparenting to avoid hiding a potentially large stat update +* (e.g. from callers of mem_cgroup_flush_stats_ratelimited()). +*/ __mem_cgroup_flush_stats (memcg, true); -/* The following counts are all non-hierarchical and need to be reparented. */
reparent_memcg1_state_local(memcg, parent); reparent_memcg1_lruvec_state_local(memcg, parent);mm/memcontrol.c +2
-0+ +__mem_cgroup_flush_stats (parent, true); }#else static inline void reparent_state_local(struct mem_cgroup *memcg,struct
mem_cgroup*parent)mm/memcontrol.c +6
-2@@ -442,7 +449,7 @@ struct lruvec_stats {long state[NR_MEMCG_NODE_STAT_ITEMS ];/* Non-hierarchical (CPU aggregated) state */ -atomic_long_t state_local[NR_MEMCG_NODE_STAT_ITEMS ]; +long state_local[NR_MEMCG_NODE_STAT_ITEMS ];/* Pending child counts during tree propagation */ long state_pending[NR_MEMCG_NODE_STAT_ITEMS ];@@ -485,7 +492,7 @@ unsigned long lruvec_page_state_local(struct lruvec *lruvec,return 0 ; pn = container_of(lruvec, struct mem_cgroup_per_node, lruvec); - x =atomic_long_read (&(pn->lruvec_stats->state_local[i])); + x =READ_ONCE (pn->lruvec_stats->state_local[i]);#ifdef CONFIG_SMP if (x <0 ) x =0 ;@@ -493,6 +500,10 @@ unsigned long lruvec_page_state_local(structlruvec *lruvec, return x; } +static void mod_memcg_lruvec_state (struct lruvec *lruvec, +enum node_stat_item idx, +int val); +#ifdef CONFIG_MEMCG_V1 void reparent_memcg_lruvec_state_local(struct mem_cgroup *memcg,struct
mem_cgroup*parent, int idx)mm/memcontrol.c +0
-4@@ -506,12 +517,10 @@ void reparent_memcg_lruvec_state_local(struct mem_cgroup *memcg, for_each_node(nid) {struct lruvec *child_lruvec =mem_cgroup_lruvec (memcg,NODE_DATA (nid));struct lruvec *parent_lruvec =mem_cgroup_lruvec (parent,NODE_DATA (nid)); -struct mem_cgroup_per_node *parent_pn;unsigned long value =lruvec_page_state_local (child_lruvec, idx); - parent_pn = container_of(parent_lruvec, struct mem_cgroup_per_node, lruvec); - - atomic_long_add(value,&(parent_pn->lruvec_stats->state_local[i]));
mm/memcontrol.c +4
-2+mod_memcg_lruvec_state (child_lruvec, idx, -value); + mod_memcg_lruvec_state(parent_lruvec, idx, value); } }#endif @@ -598,7 +607,7 @@ struct memcg_vmstats {unsigned long events[NR_MEMCG_EVENTS ];/* Non-hierarchical (CPU aggregated) page state & events */ -atomic_long_t state_local[MEMCG_VMSTAT_SIZE ]; +long state_local[MEMCG_VMSTAT_SIZE ];unsigned long events_local[NR_MEMCG_EVENTS ];/* Pending child counts during tree propagation */ @@ -835,7 +844,7 @@ unsigned long memcg_page_state_local(struct mem_cgroup *memcg, int idx) if (WARN_ONCE(BAD_STAT_IDX(i), "%s: missing stat item %d\n", __func__, idx))return 0 ; - x =atomic_long_read (&(memcg->vmstats->state_local[i])); + x =READ_ONCE (memcg->vmstats->state_local[i]);#ifdef CONFIG_SMP if (x < 0) x = 0;mm/memcontrol.c +4
-9@@ -852,7 +861,8 @@ void reparent_memcg_state_local(struct mem_cgroup *memcg,if (WARN_ONCE (BAD_STAT_IDX (i),"%s: missing stat item %d\n" , __func__, idx))return ; - atomic_long_add(value, &(parent->vmstats->state_local[i])); +mod_memcg_state (memcg, idx, -value); + mod_memcg_state(parent, idx, value); }#endif @@ -4174,8 +4184,6 @@ struct aggregate_control {long *aggregate;/* pointer to the non-hierarchichal (CPU aggregated) counters */ long *local; -/* pointer to the atomic non-hierarchichal (CPU aggregated) counters */ -atomic_long_t *alocal;/* pointer to the pending child counters during tree propagation */ long *pending;/* pointer to the parent's pending counters, could be NULL */ @@ -4213,12 +4221,8 @@ static void mem_cgroup_stat_aggregate(struct aggregate_control *ac) }/* Aggregate counts on this level and propagate upwards */ -if (delta_cpu) { -if (ac->local) - ac->local[i] += delta_cpu; -else if (ac->alocal) -atomic_long_add (delta_cpu, &(ac->alocal[i])); - } +if (delta_cpu) + ac->local[i] += delta_cpu;if (delta) {ac->aggregate[i] += delta;mm/memcontrol.c +1
-2@@ -4289,8 +4293,7 @@ static void mem_cgroup_css_rstat_flush(struct cgroup_subsys_state *css, int cpu) ac = (struct aggregate_control ) { .aggregate = memcg->vmstats->state, - .local =NULL , - .alocal = memcg->vmstats->state_local, + .local = memcg->vmstats->state_local, .pending = memcg->vmstats->state_pending, .ppending = parent ? parent->vmstats->state_pending :NULL ,.cstat = statc->state,mm/memcontrol.c +1
-2@@ -4323,8 +4326,7 @@ static void mem_cgroup_css_rstat_flush(struct cgroup_subsys_state *css, int cpu) ac = (struct aggregate_control ) { .aggregate = lstats->state, - .local =NULL , - .alocal = lstats->state_local, + .local = lstats->state_local, .pending = lstats->state_pending, .ppending = plstats ? plstats->state_pending :NULL ,.cstat = lstatc->state,
Yosry Ahmed
maintainerRe: [PATCH v5 29/32] mm: memcontrol: prepare for reparenting non-hierarchical stats
Quoted 20 lines · Shakeel Butt
Quoted 18 lines · Yosry Ahmed
Quoted 18 lines
Quoted 4 lines
Did you measure the impact of making
state_localatomic on the flush path? It's a slow path but we've seen pain from it being too slow before, because it extends the critical section of the rstat flush lock.Qi, please measure the impact on flushing and if no impact then no need to do anything as I don't want anymore churn in this series.
Quoted 9 lines
Can we keep this non-atomic and use
mod_memcg_lruvec_state()here? It will update the stat on the local counter and it will be added tostate_localin the flush path when needed. We can even force another flush inreparent_state_local() after reparenting is completed, if we want to avoid leaving a potentially large stat update pending, as it can be missed bymem_cgroup_flush_stats_ratelimited().Same for
reparent_memcg_state_local(), we can probably usemod_memcg_state()?
Quoted 7 lines
Quoted 5 lines
Quoted 1 line
Yosry, do you mind sending the patch you are thinking about over this series?
Honestly, I'd rather squash it into this patch if possible. It avoids churn in the history (switch to atomics and back), and is arguably simpler than checking for regressions in the flush path.
Yup, let's squash it into the original patch. Please add your sign-off tag.
Sure. Qi/Andrew, feel free to add these tags if you squash the diff below:
Quoted 9 lines
Quoted 4 lines
What I have in mind is the diff below (build tested only). Qi, would you be able to test this? It applies directly on this patch in mm-new:
Qi, please squash this diff into the patch and test. You might need to change the subsequent patches. Once you are done with testing, you can post the diffs for those in reply to those patches and we will ask Andrew to squash into orinigal ones.
FWIW, after applying this diff, the rest of the series applies cleanly. So I think we won't need diffs for other patches.
Quoted 1 line
The diff looks good to me though.
Thanks for taking a look!
Qi Zheng
reviewerRe: [PATCH v5 29/32] mm: memcontrol: prepare for reparenting non-hierarchical stats
In reply to Yosry AhmedHi Yosry,
Quoted 18 lines · Yosry Ahmed
Quoted 18 lines
Quoted 4 lines
Did you measure the impact of making
state_localatomic on the flush path? It's a slow path but we've seen pain from it being too slow before, because it extends the critical section of the rstat flush lock.Qi, please measure the impact on flushing and if no impact then no need to do anything as I don't want anymore churn in this series.
Quoted 9 lines
Can we keep this non-atomic and use
mod_memcg_lruvec_state()here? It will update the stat on the local counter and it will be added tostate_localin the flush path when needed. We can even force another flush inreparent_state_local() after reparenting is completed, if we want to avoid leaving a potentially large stat update pending, as it can be missed bymem_cgroup_flush_stats_ratelimited().Same for
reparent_memcg_state_local(), we can probably usemod_memcg_state()?
Quoted 8 lines
Quoted 1 line
Yosry, do you mind sending the patch you are thinking about over this series?
Honestly, I'd rather squash it into this patch if possible. It avoids churn in the history (switch to atomics and back), and is arguably simpler than checking for regressions in the flush path.
What I have in mind is the diff below (build tested only). Qi, would you be able to test this? It applies directly on this patch in mm-new:
Thank you so much for doing this! I'm willing to squash and test.
And I found some issues with the diff:
Quoted 74 lines
mm/memcontrol.c +0
-0@@ -234,11 +234,18 @@ static inline void reparent_state_local(struct
mem_cgroup*memcg, structmem_cgrif (cgroup_subsys_on_dfl(memory_cgrp_subsys)) return;mm/memcontrol.c +2
-0+/* +* Reparent stats exposed non-hierarchically. Flush @memcg's stats first to
mm/memcontrol.c +4
-1+* read its stats accurately , and conservatively flush @parent's stats +* after reparenting to avoid hiding a potentially large stat update +* (e.g. from callers of mem_cgroup_flush_stats_ratelimited()). +*/ __mem_cgroup_flush_stats (memcg, true); -/* The following counts are all non-hierarchical and need to be reparented. */
reparent_memcg1_state_local(memcg, parent); reparent_memcg1_lruvec_state_local(memcg, parent);mm/memcontrol.c +2
-0+ +__mem_cgroup_flush_stats (parent, true); }#else static inline void reparent_state_local(struct mem_cgroup *memcg,struct
mem_cgroup*parent)mm/memcontrol.c +6
-2@@ -442,7 +449,7 @@ struct lruvec_stats {long state[NR_MEMCG_NODE_STAT_ITEMS ];/* Non-hierarchical (CPU aggregated) state */ -atomic_long_t state_local[NR_MEMCG_NODE_STAT_ITEMS ]; +long state_local[NR_MEMCG_NODE_STAT_ITEMS ];/* Pending child counts during tree propagation */ long state_pending[NR_MEMCG_NODE_STAT_ITEMS ];@@ -485,7 +492,7 @@ unsigned long lruvec_page_state_local(struct lruvec *lruvec,return 0 ; pn = container_of(lruvec, struct mem_cgroup_per_node, lruvec); - x =atomic_long_read (&(pn->lruvec_stats->state_local[i])); + x =READ_ONCE (pn->lruvec_stats->state_local[i]);#ifdef CONFIG_SMP if (x <0 ) x =0 ;@@ -493,6 +500,10 @@ unsigned long lruvec_page_state_local(structlruvec *lruvec, return x; } +static void mod_memcg_lruvec_state (struct lruvec *lruvec, +enum node_stat_item idx, +int val); +#ifdef CONFIG_MEMCG_V1 void reparent_memcg_lruvec_state_local(struct mem_cgroup *memcg,struct
mem_cgroup*parent, int idx)mm/memcontrol.c +0
-4@@ -506,12 +517,10 @@ void reparent_memcg_lruvec_state_local(struct mem_cgroup *memcg, for_each_node(nid) {struct lruvec *child_lruvec =mem_cgroup_lruvec (memcg,NODE_DATA (nid));struct lruvec *parent_lruvec =mem_cgroup_lruvec (parent,NODE_DATA (nid)); -struct mem_cgroup_per_node *parent_pn;unsigned long value =lruvec_page_state_local (child_lruvec, idx); - parent_pn = container_of(parent_lruvec, struct mem_cgroup_per_node, lruvec); - - atomic_long_add(value,
Quoted 2 lines
&(parent_pn->lruvec_stats->state_local[i]));
+mod_memcg_lruvec_state(child_lruvec, idx, -value);
We can't use mod_memcg_lruvec_state() here, because child memcg has already been set CSS_DYING. So in mod_memcg_lruvec_state(), we will get parent memcg.
It seems we need to reimplement a function or add a parameter to mod_memcg_lruvec_state() to solve the problem. What do you think?
Quoted 25 lines
mm/memcontrol.c +3
-2+ mod_memcg_lruvec_state(parent_lruvec, idx, value); } }#endif @@ -598,7 +607,7 @@ struct memcg_vmstats {unsigned long events[NR_MEMCG_EVENTS ];/* Non-hierarchical (CPU aggregated) page state & events */ -atomic_long_t state_local[MEMCG_VMSTAT_SIZE ]; +long state_local[MEMCG_VMSTAT_SIZE ];unsigned long events_local[NR_MEMCG_EVENTS ];/* Pending child counts during tree propagation */ @@ -835,7 +844,7 @@ unsigned long memcg_page_state_local(struct mem_cgroup *memcg, int idx) if (WARN_ONCE(BAD_STAT_IDX(i), "%s: missing stat item %d\n", __func__, idx))return 0 ; - x =atomic_long_read (&(memcg->vmstats->state_local[i])); + x =READ_ONCE (memcg->vmstats->state_local[i]);#ifdef CONFIG_SMP if (x < 0) x = 0;
mm/memcontrol.c void reparent_memcg_state_local(struct mem_cgroup *memcg, · line 865
Quoted 6 lines
mm/memcontrol.c +1
-1@@ -852,7 +861,8 @@ void reparent_memcg_state_local(struct mem_cgroup *memcg,if (WARN_ONCE (BAD_STAT_IDX (i),"%s: missing stat item %d\n" , __func__, idx))return ; -atomic_long_add (value, &(parent->vmstats->state_local[i])); +mod_memcg_state (memcg, idx, -value);
Same as mod_memcg_lruvec_state().
Thanks,
Qi
Quoted 51 lines
mm/memcontrol.c +3
-8+ mod_memcg_state(parent, idx, value); }#endif @@ -4174,8 +4184,6 @@ struct aggregate_control {long *aggregate;/* pointer to the non-hierarchichal (CPU aggregated) counters */ long *local; -/* pointer to the atomic non-hierarchichal (CPU aggregated) counters */ -atomic_long_t *alocal;/* pointer to the pending child counters during tree propagation */ long *pending;/* pointer to the parent's pending counters, could be NULL */ @@ -4213,12 +4221,8 @@ static void mem_cgroup_stat_aggregate(struct aggregate_control *ac) }/* Aggregate counts on this level and propagate upwards */ -if (delta_cpu) { -if (ac->local) - ac->local[i] += delta_cpu; -else if (ac->alocal) -atomic_long_add (delta_cpu, &(ac->alocal[i])); - } +if (delta_cpu) + ac->local[i] += delta_cpu;if (delta) {ac->aggregate[i] += delta;mm/memcontrol.c +1
-2@@ -4289,8 +4293,7 @@ static void mem_cgroup_css_rstat_flush(struct cgroup_subsys_state *css, int cpu) ac = (struct aggregate_control ) { .aggregate = memcg->vmstats->state, - .local =NULL , - .alocal = memcg->vmstats->state_local, + .local = memcg->vmstats->state_local, .pending = memcg->vmstats->state_pending, .ppending = parent ? parent->vmstats->state_pending :NULL ,.cstat = statc->state,mm/memcontrol.c +1
-2@@ -4323,8 +4326,7 @@ static void mem_cgroup_css_rstat_flush(struct cgroup_subsys_state *css, int cpu) ac = (struct aggregate_control ) { .aggregate = lstats->state, - .local =NULL , - .alocal = lstats->state_local, + .local = lstats->state_local, .pending = lstats->state_pending, .ppending = plstats ? plstats->state_pending :NULL ,.cstat = lstatc->state,
Qi Zheng
reviewerRe: [PATCH v5 29/32] mm: memcontrol: prepare for reparenting non-hierarchical stats
Quoted 118 lines · Qi Zheng
Hi Yosry,
Quoted 18 lines · Yosry Ahmed
Quoted 18 lines
Quoted 4 lines
Did you measure the impact of making
state_localatomic on the flush path? It's a slow path but we've seen pain from it being too slow before, because it extends the critical section of the rstat flush lock.Qi, please measure the impact on flushing and if no impact then no need to do anything as I don't want anymore churn in this series.
Quoted 9 lines
Can we keep this non-atomic and use
mod_memcg_lruvec_state()here? It will update the stat on the local counter and it will be added tostate_localin the flush path when needed. We can even force another flush inreparent_state_local() after reparenting is completed, if we want to avoid leaving a potentially large stat update pending, as it can be missed bymem_cgroup_flush_stats_ratelimited().Same for
reparent_memcg_state_local(), we can probably usemod_memcg_state()?Quoted 8 lines
Quoted 1 line
Yosry, do you mind sending the patch you are thinking about over this series?
Honestly, I'd rather squash it into this patch if possible. It avoids churn in the history (switch to atomics and back), and is arguably simpler than checking for regressions in the flush path.
What I have in mind is the diff below (build tested only). Qi, would you be able to test this? It applies directly on this patch in mm-new:
Thank you so much for doing this! I'm willing to squash and test.
And I found some issues with the diff:
Quoted 73 lines
mm/memcontrol.c +0
-0@@ -234,11 +234,18 @@ static inline void reparent_state_local(struct
mem_cgroup*memcg, structmem_cgr
if (cgroup_subsys_on_dfl(memory_cgrp_subsys)) return;+ /*
+ * Reparent stats exposed non-hierarchically. Flush @memcg's stats first to
+ * read its stats accurately , and conservatively flush @parent's stats
+ * after reparenting to avoid hiding a potentially large stat update
+ * (e.g. from callers ofmem_cgroup_flush_stats_ratelimited()). + */
__mem_cgroup_flush_stats(memcg, true);- /* The following counts are all non-hierarchical and need to be reparented. */
reparent_memcg1_state_local(memcg, parent);reparent_memcg1_lruvec_state_local(memcg, parent); +
+__mem_cgroup_flush_stats(parent, true);
}
#else
static inline voidreparent_state_local(struct mem_cgroup *memcg, struct mem_cgroup *parent)mm/memcontrol.c +2
-2@@ -442,7 +449,7 @@ struct lruvec_stats { long state[NR_MEMCG_NODE_STAT_ITEMS]; /* Non-hierarchical (CPU aggregated) state */ - atomic_long_t state_local[NR_MEMCG_NODE_STAT_ITEMS]; + long state_local[NR_MEMCG_NODE_STAT_ITEMS]; /* Pending child counts during tree propagation */ long state_pending[NR_MEMCG_NODE_STAT_ITEMS];@@ -485,7 +492,7 @@ unsigned long lruvec_page_state_local(structlruvec *lruvec, return 0; pn = container_of(lruvec, struct mem_cgroup_per_node, lruvec); - x = atomic_long_read(&(pn->lruvec_stats->state_local[i])); + x = READ_ONCE(pn->lruvec_stats->state_local[i]); #ifdef CONFIG_SMP if (x < 0)x = 0;
mm/memcontrol.c +4
-0@@ -493,6 +500,10 @@ unsigned long lruvec_page_state_local(structlruvec *lruvec, return x; } +static void mod_memcg_lruvec_state (struct lruvec *lruvec, + enum node_stat_item idx, + int val); + #ifdef CONFIG_MEMCG_V1 void reparent_memcg_lruvec_state_local(struct mem_cgroup *memcg,struct
mem_cgroup*parent, int idx)mm/memcontrol.c +0
-0@@ -506,12 +517,10 @@ void reparent_memcg_lruvec_state_local(structmem_cgroup *memcg, for_each_node(nid) { struct lruvec *child_lruvec = mem_cgroup_lruvec(memcg,NODE_DATA (nid));struct lruvec
*parent_lruvec=mem_cgroup_lruvec(parent, NODE_DATA(nid));
- structmem_cgroup_per_node*parent_pn; unsigned long value =lruvec_page_state_local(child_lruvec, idx);Quoted 6 lines
-
parent_pn=container_of(parent_lruvec, struct mem_cgroup_per_node, lruvec);
-
-atomic_long_add(value, &(parent_pn->lruvec_stats->state_local[i]));
+mod_memcg_lruvec_state(child_lruvec, idx, -value);We can't use
mod_memcg_lruvec_state()here, because child memcg has already been setCSS_DYING. So inmod_memcg_lruvec_state(), we will get parent memcg.
Quoted 2 lines
It seems we need to reimplement a function or add a parameter to
mod_memcg_lruvec_state()to solve the problem. What do you think?
Since child memcg is about to disappear, perhaps we can just add value to parent memcg without handling the child memcg. Make sense?
Quoted 94 lines
Quoted 26 lines
+
mod_memcg_lruvec_state(parent_lruvec, idx, value); }
}
#endifmm/memcontrol.c +2
-2@@ -598,7 +607,7 @@ struct memcg_vmstats { unsigned long events[NR_MEMCG_EVENTS]; /* Non-hierarchical (CPU aggregated) page state & events */ - atomic_long_t state_local[MEMCG_VMSTAT_SIZE]; + long state_local[MEMCG_VMSTAT_SIZE]; unsigned long events_local[NR_MEMCG_EVENTS]; /* Pending child counts during tree propagation */@@ -835,7 +844,7 @@ unsigned long memcg_page_state_local(struct mem_cgroup *memcg, int idx) if (WARN_ONCE(BAD_STAT_IDX(i), "%s: missing stat item %d\n", __func__, idx)) return 0; - x = atomic_long_read(&(memcg->vmstats->state_local[i])); + x = READ_ONCE(memcg->vmstats->state_local[i]); #ifdef CONFIG_SMPif (x < 0)
x = 0;mm/memcontrol.c void reparent_memcg_state_local(struct mem_cgroup · line 866
Quoted 7 lines
mm/memcontrol.c +1
-1@@ -852,7 +861,8 @@ void reparent_memcg_state_local(struct mem_cgroup *memcg, if (WARN_ONCE(BAD_STAT_IDX(i), "%s: missing stat item %d\n", __func__, idx)) return; - atomic_long_add(value, &(parent->vmstats->state_local[i])); + mod_memcg_state(memcg, idx, -value);Same as
mod_memcg_lruvec_state().Thanks,
QiQuoted 54 lines
mm/memcontrol.c +1
-0+ mod_memcg_state(parent, idx, value); }#endif
mm/memcontrol.c +0
-1@@ -4174,8 +4184,6 @@ struct aggregate_control { long *aggregate; /* pointer to the non-hierarchichal (CPU aggregated) counters */ long *local; - /* pointer to the atomic non-hierarchichal (CPU aggregated)counters */
-atomic_long_t*alocal;
/* pointer to the pending child counters during tree propagation */ long *pending;
/* pointer to the parent's pending counters, could be NULL */mm/memcontrol.c +2
-6@@ -4213,12 +4221,8 @@ static void mem_cgroup_stat_aggregate(struct aggregate_control *ac) } /* Aggregate counts on this level and propagate upwards */ - if (delta_cpu) { - if (ac->local) - ac->local[i] += delta_cpu; - else if (ac->alocal) - atomic_long_add(delta_cpu, &(ac- >alocal[i])); - } + if (delta_cpu) + ac->local[i] += delta_cpu;if (delta) { ac->aggregate[i] += delta;
mm/memcontrol.c +1
-2@@ -4289,8 +4293,7 @@ static void mem_cgroup_css_rstat_flush(struct cgroup_subsys_state *css, int cpu) ac = (struct aggregate_control) { .aggregate = memcg->vmstats->state, - .local = NULL, - .alocal = memcg->vmstats->state_local, + .local = memcg->vmstats->state_local, .pending = memcg->vmstats->state_pending, .ppending = parent ? parent->vmstats->state_pending : NULL,.cstat =
statc->state,mm/memcontrol.c +1
-2@@ -4323,8 +4326,7 @@ static void mem_cgroup_css_rstat_flush(struct cgroup_subsys_state *css, int cpu) ac = (struct aggregate_control) { .aggregate = lstats->state, - .local = NULL, - .alocal = lstats->state_local, + .local = lstats->state_local, .pending = lstats->state_pending,.ppending = plstats ? plstats-
Quoted 1 line
state_pending: NULL,.cstat =
lstatc->state,
Yosry Ahmed
maintainerRe: [PATCH v5 29/32] mm: memcontrol: prepare for reparenting non-hierarchical stats
[..]
Quoted 22 lines
Quoted 22 lines
Quoted 15 lines
(quoted diff) +0
-4@@ -506,12 +517,10 @@ void reparent_memcg_lruvec_state_local(struct mem_cgroup *memcg, for_each_node(nid) { struct lruvec *child_lruvec = mem_cgroup_lruvec(memcg, NODE_DATA(nid)); struct lruvec *parent_lruvec = mem_cgroup_lruvec(parent, NODE_DATA(nid)); - struct mem_cgroup_per_node *parent_pn; unsigned long value = lruvec_page_state_local(child_lruvec, idx); - parent_pn = container_of(parent_lruvec, struct mem_cgroup_per_node, lruvec); - - atomic_long_add(value,Quoted 2 lines
&(parent_pn->lruvec_stats->state_local[i]));
+mod_memcg_lruvec_state(child_lruvec, idx, -value);We can't use
mod_memcg_lruvec_state()here, because child memcg has already been setCSS_DYING. So inmod_memcg_lruvec_state(), we will get parent memcg.
Quoted 5 lines
Quoted 2 lines
It seems we need to reimplement a function or add a parameter to
mod_memcg_lruvec_state()to solve the problem. What do you think?Since child memcg is about to disappear, perhaps we can just add value to parent memcg without handling the child memcg. Make sense?
Ugh yes, I missed that, thanks.
I don't think we can just leave the child's memcg wrong. Aside from the fact that I would be nervous if access to those stats is still possible after it's offlined (e.g. can userspace already have the stats file open, or maybe some in-kernel code uses it), there's a bigger issue.
When the child cgroup is released, css_release_work_fn() will flush its stats and then it will be double counted at the parent.
Maybe refactor the part sof mod_memcg_lruvec_state() and mod_memcg_state () without get_non_dying_memcg_{start/end}() into helpers, and call that directly from the reparenting functions? Adding a boolean argument to mod_memcg_lruvec_state() and mod_memcg_state() will add a lot of churn, and naked boolean arguments are not ideal.
Qi Zheng
reviewerRe: [PATCH v5 29/32] mm: memcontrol: prepare for reparenting non-hierarchical stats
Quoted 37 lines · Yosry Ahmed
[..]
Quoted 22 lines
Quoted 22 lines
Quoted 15 lines
(quoted diff) +0
-4@@ -506,12 +517,10 @@ void reparent_memcg_lruvec_state_local(struct mem_cgroup *memcg, for_each_node(nid) { struct lruvec *child_lruvec = mem_cgroup_lruvec(memcg, NODE_DATA(nid)); struct lruvec *parent_lruvec = mem_cgroup_lruvec(parent, NODE_DATA(nid)); - struct mem_cgroup_per_node *parent_pn; unsigned long value = lruvec_page_state_local(child_lruvec, idx); - parent_pn = container_of(parent_lruvec, struct mem_cgroup_per_node, lruvec); - - atomic_long_add(value,Quoted 2 lines
&(parent_pn->lruvec_stats->state_local[i]));
+mod_memcg_lruvec_state(child_lruvec, idx, -value);We can't use
mod_memcg_lruvec_state()here, because child memcg has already been setCSS_DYING. So inmod_memcg_lruvec_state(), we will get parent memcg.Quoted 5 lines
Quoted 2 lines
It seems we need to reimplement a function or add a parameter to
mod_memcg_lruvec_state()to solve the problem. What do you think?Since child memcg is about to disappear, perhaps we can just add value to parent memcg without handling the child memcg. Make sense?
Ugh yes, I missed that, thanks.
I don't think we can just leave the child's memcg wrong. Aside from the fact that I would be nervous if access to those stats is still possible after it's offlined (e.g. can userspace already have the stats file open, or maybe some in-kernel code uses it), there's a bigger issue.
Quoted 6 lines
When the child cgroup is released,
css_release_work_fn()will flush its stats and then it will be double counted at the parent.Maybe refactor the part sof
mod_memcg_lruvec_state()andmod_memcg_state() without get_non_dying_memcg_{start/end}() into helpers, and call that directly from the reparenting functions? Adding
OK, will do.
Quoted 2 lines
a boolean argument to
mod_memcg_lruvec_state()andmod_memcg_state()will add a lot of churn, and naked boolean arguments are not ideal.
Qi Zheng
reviewerRe: [PATCH v5 29/32] mm: memcontrol: prepare for reparenting non-hierarchical stats
In reply to Yosry AhmedHi Yosry,
Quoted 19 lines · Yosry Ahmed
Quoted 19 lines
mm/memcontrol.c +15
-0@@ -473,6 +493,29 @@ unsigned long lruvec_page_state_local(struct lruvec *lruvec,return x; } +#ifdef CONFIG_MEMCG_V1 +void reparent_memcg_lruvec_state_local (struct mem_cgroup *memcg, +struct mem_cgroup *parent,int idx) +{ +int i =memcg_stats_index (idx); +int nid; + +if (WARN_ONCE (BAD_STAT_IDX (i),"%s: missing stat item %d\n" , __func__, idx)) +return ; + +for_each_node (nid) { +struct lruvec *child_lruvec =mem_cgroup_lruvec (memcg,NODE_DATA (nid)); +struct lruvec *parent_lruvec =mem_cgroup_lruvec (parent,NODE_DATA (nid)); +struct mem_cgroup_per_node *parent_pn; +unsigned long value =lruvec_page_state_local (child_lruvec, idx);
Quoted 8 lines
mm/memcontrol.c unsigned long lruvec_page_state_local(struct lruvec *lruvec, · line 516
Quoted 6 lines
mm/memcontrol.c +6
-0+ + parent_pn = container_of(parent_lruvec, struct mem_cgroup_per_node, lruvec); + +atomic_long_add (value, &(parent_pn->lruvec_stats->state_local[i])); + } +}Did you measure the impact of making
state_localatomic on the flush
No, because this is not a performance-critical path, I did not perform performance testing. :(
Quoted 3 lines
path? It's a slow path but we've seen pain from it being too slow before, because it extends the critical section of the rstat flush lock.
Got it. Do you have any test cases that previously identified this problem?
Quoted 9 lines
Can we keep this non-atomic and use
mod_memcg_lruvec_state()here? It will update the stat on the local counter and it will be added tostate_localin the flush path when needed. We can even force another flush inreparent_state_local() after reparenting is completed, if we want to avoid leaving a potentially large stat update pending, as it can be missed bymem_cgroup_flush_stats_ratelimited().Same for
reparent_memcg_state_local(), we can probably usemod_memcg_state()?
Thank you for providing optimization ideas; I will take a closer look.
Re: [PATCH v5 29/32] mm: memcontrol: prepare for reparenting non-hierarchical stats
In reply to Qi ZhengQuoted 36 lines · Qi Zheng
From: Qi Zheng <[email protected]>
To resolve the dying memcg issue, we need to reparent LRU folios of child memcg to its parent memcg. This could cause problems for non-hierarchical stats.
As Yosry Ahmed pointed out:
``` In short, if memory is charged to a dying cgroup at the time of reparenting, when the memory gets uncharged the stats updates will occur at the parent. This will update both hierarchical and non-hierarchical stats of the parent, which would corrupt the parent's non-hierarchical stats (because those counters were never incremented when the memory was charged). ```Now we have the following two types of non-hierarchical stats, and they are only used in
CONFIG_MEMCG_V1:a. memcg->vmstats->state_local[i]
b. pn->lruvec_stats->state_local[i]To ensure that these non-hierarchical stats work properly, we need to reparent these non-hierarchical stats after reparenting LRU folios. To this end, this commit makes the following preparations:
1. implement
reparent_state_local()to reparent non-hierarchical stats
2. makecss_killed_work_fn()to be called in rcu work, and implementget_non_dying_memcg_start()andget_non_dying_memcg_end()to avoid race between mod_memcg_state()/mod_memcg_lruvec_state()
andreparent_state_local()
3. change these non-hierarchical stats toatomic_long_ttype to avoid race betweenmem_cgroup_stat_aggregate()andreparent_state_local()Signed-off-by Qi Zheng <[email protected]>
[...]
mm/memcontrol-v1.h static inline bool do_memsw_account(void) · line 48
Quoted 7 lines
mm/memcontrol-v1.h +1
-0@@ -45,6 +45,7 @@ static inline bool do_memsw_account(void)unsigned long memcg_events_local (struct mem_cgroup *memcg,int event);unsigned long memcg_page_state_local (struct mem_cgroup *memcg,int idx); +void mod_memcg_page_state_local (struct mem_cgroup *memcg,int idx,unsigned long val);
The above seems like addition by mistake.
After fixing this, you can add:
Qi Zheng
reviewerRe: [PATCH v5 29/32] mm: memcontrol: prepare for reparenting non-hierarchical stats
Quoted 44 lines · Shakeel Butt
Quoted 36 lines · Qi Zheng
From: Qi Zheng <[email protected]>
To resolve the dying memcg issue, we need to reparent LRU folios of child memcg to its parent memcg. This could cause problems for non-hierarchical stats.
As Yosry Ahmed pointed out:
``` In short, if memory is charged to a dying cgroup at the time of reparenting, when the memory gets uncharged the stats updates will occur at the parent. This will update both hierarchical and non-hierarchical stats of the parent, which would corrupt the parent's non-hierarchical stats (because those counters were never incremented when the memory was charged). ```Now we have the following two types of non-hierarchical stats, and they are only used in
CONFIG_MEMCG_V1:a. memcg->vmstats->state_local[i]
b. pn->lruvec_stats->state_local[i]To ensure that these non-hierarchical stats work properly, we need to reparent these non-hierarchical stats after reparenting LRU folios. To this end, this commit makes the following preparations:
1. implement
reparent_state_local()to reparent non-hierarchical stats
2. makecss_killed_work_fn()to be called in rcu work, and implementget_non_dying_memcg_start()andget_non_dying_memcg_end()to avoid race between mod_memcg_state()/mod_memcg_lruvec_state()
andreparent_state_local()
3. change these non-hierarchical stats toatomic_long_ttype to avoid race betweenmem_cgroup_stat_aggregate()andreparent_state_local()Signed-off-by Qi Zheng <[email protected]>[...]
Quoted 4 lines
mm/memcontrol-v1.h +0
-0@@ -45,6 +45,7 @@ static inline bool do_memsw_account(void)
Quoted 5 lines
mm/memcontrol-v1.h static inline bool do_memsw_account(void) · line 48
Quoted 3 lines
mm/memcontrol-v1.h +1
-0unsigned long memcg_events_local (struct mem_cgroup *memcg,int event);unsigned long memcg_page_state_local (struct mem_cgroup *memcg,int idx); +void mod_memcg_page_state_local (struct mem_cgroup *memcg,int idx,unsigned long val);The above seems like addition by mistake.
Yes, my mistake, need to be deleted directly. ;)
Quoted 4 lines
After fixing this, you can add:
Acked-by Shakeel Butt <[email protected]>
Thanks!
Quoted 1 line
Qi Zheng
reviewerPATCH v5 29/32 mm: memcontrol: prepare for reparenting non-hierarchical stats
In reply to Qi ZhengFrom: Qi Zheng <[email protected]>
To resolve the dying memcg issue, we need to reparent LRU folios of child memcg to its parent memcg. This could cause problems for non-hierarchical stats.
As Yosry Ahmed pointed out:
``` In short, if memory is charged to a dying cgroup at the time of reparenting, when the memory gets uncharged the stats updates will occur at the parent. This will update both hierarchical and non-hierarchical stats of the parent, which would corrupt the parent's non-hierarchical stats (because those counters were never incremented when the memory was charged). ```
Now we have the following two types of non-hierarchical stats, and they are only used in CONFIG_MEMCG_V1:
a. memcg->vmstats->state_local[i]
b. pn->lruvec_stats->state_local[i]
To ensure that these non-hierarchical stats work properly, we need to reparent these non-hierarchical stats after reparenting LRU folios. To this end, this commit makes the following preparations:
1. implement reparent_state_local() to reparent non-hierarchical stats
2. make css_killed_work_fn() to be called in rcu work, and implement get_non_dying_memcg_start() and get_non_dying_memcg_end() to avoid race between mod_memcg_state()/mod_memcg_lruvec_state()
and reparent_state_local()
--- kernel/cgroup/cgroup.c | 8 +-- mm/memcontrol-v1.c | 16 +++++ mm/memcontrol-v1.h | 7 ++ mm/memcontrol.c | 146 +++++++++++++++++++++++++++++++++++++++++ 4 files changed, 173 insertions(+), 4 deletions(-)
kernel/cgroup/cgroup.c +4 -4
@@ -6044,8 +6044,8 @@ int cgroup_mkdir(struct kernfs_node *parent_kn, const char *name, umode_t mode)*/ static void css_killed_work_fn (struct work_struct *work) { -struct cgroup_subsys_state *css = - container_of(work, struct cgroup_subsys_state, destroy_work); +struct cgroup_subsys_state *css =container_of (to_rcu_work (work), + struct cgroup_subsys_state, destroy_rwork);cgroup_lock ();
@@ -6066,8 +6066,8 @@ static void css_killed_ref_fn(struct percpu_ref *ref) container_of(ref, struct cgroup_subsys_state, refcnt);if (atomic_dec_and_test (&css->online_cnt)) { -INIT_WORK (&css->destroy_work, css_killed_work_fn); -queue_work (cgroup_offline_wq, &css->destroy_work); +INIT_RCU_WORK (&css->destroy_rwork, css_killed_work_fn); +queue_rcu_work (cgroup_offline_wq, &css->destroy_rwork); } }
mm/memcontrol-v1.c +16 -0
@@ -1897,6 +1897,22 @@ static const unsigned int memcg1_events[] = { PGMAJFAULT, }; +void reparent_memcg1_state_local (struct mem_cgroup *memcg,struct mem_cgroup *parent) +{ +int i; + +for (i =0 ; i <ARRAY_SIZE (memcg1_stats); i++) +reparent_memcg_state_local (memcg, parent, memcg1_stats[i]); +} + +void reparent_memcg1_lruvec_state_local (struct mem_cgroup *memcg,struct mem_cgroup *parent) +{ +int i; + +for (i =0 ; i <NR_LRU_LISTS ; i++) +reparent_memcg_lruvec_state_local (memcg, parent, i); +} +void memcg1_stat_format (struct mem_cgroup *memcg,struct seq_buf *s) {unsigned long memory, memsw;
mm/memcontrol-v1.h +7 -0
@@ -77,6 +77,13 @@ void memcg1_uncharge_batch(struct mem_cgroup *memcg, unsigned long pgpgout, unsigned long nr_memory, int nid);void memcg1_stat_format (struct mem_cgroup *memcg,struct seq_buf *s); +void reparent_memcg1_state_local (struct mem_cgroup *memcg,struct mem_cgroup *parent); +void reparent_memcg1_lruvec_state_local (struct mem_cgroup *memcg,struct mem_cgroup *parent); + +void reparent_memcg_state_local (struct mem_cgroup *memcg, +struct mem_cgroup *parent,int idx); +void reparent_memcg_lruvec_state_local (struct mem_cgroup *memcg, +struct mem_cgroup *parent,int idx);void memcg1_account_kmem (struct mem_cgroup *memcg,int nr_pages);static inline bool memcg1_tcpmem_active (struct mem_cgroup *memcg)
mm/memcontrol.c +146 -0
170 lines not on this page · raw message
Signature · 1 line
2.20.1
Andrew Morton
maintainerRe: [PATCH v5 update 29/32] mm: memcontrol: prepare for reparenting non-hierarchical stats
Quoted 25 lines · On Sat, 28 Feb 2026 15:25:56 +0800 Qi Zheng <[email protected]> wrote:
To resolve the dying memcg issue, we need to reparent LRU folios of child memcg to its parent memcg. This could cause problems for non-hierarchical stats.
As Yosry Ahmed pointed out:
``` In short, if memory is charged to a dying cgroup at the time of reparenting, when the memory gets uncharged the stats updates will occur at the parent. This will update both hierarchical and non-hierarchical stats of the parent, which would corrupt the parent's non-hierarchical stats (because those counters were never incremented when the memory was charged). ```Now we have the following two types of non-hierarchical stats, and they are only used in
CONFIG_MEMCG_V1:a. memcg->vmstats->state_local[i]
b. pn->lruvec_stats->state_local[i]To ensure that these non-hierarchical stats work properly, we need to reparent these non-hierarchical stats after reparenting LRU folios. To this end, this commit makes the following preparations:
Quoted 5 lines
1. implement
reparent_state_local()to reparent non-hierarchical stats
2. makecss_killed_work_fn()to be called in rcu work, and implementget_non_dying_memcg_start()andget_non_dying_memcg_end()to avoid race between mod_memcg_state()/mod_memcg_lruvec_state()
andreparent_state_local()
That's a big update (delta patch is below).
What did it all do?
I've bookmarked these reviewer comments:
https://lkml.kernel.org/r/CAO9r8zOhZgrym6oSrtg7b+HmNHEfWuAzZ0i8eYhm5-OEnfFLdw@mail.gmail.com
https://lkml.kernel.org/r/[email protected]
https://lkml.kernel.org/r/[email protected]
Were they all addressed in some fashion?
include/linux/memcontrol.h | 1 mm/memcontrol-v1.h | 1 mm/memcontrol.c | 89 ++++++++++++++++++++++++++--------- 3 files changed, 68 insertions(+), 23 deletions(-)
include/linux/memcontrol.h renamed from include/linux/memcontrol.h~mm-memcontrol-prepare-for-reparenting-non-hierarchical-stats-update +0 -1
@@ -956,7 +956,6 @@ static inline void mod_memcg_page_state(unsigned long memcg_events (struct mem_cgroup *memcg,int event);unsigned long memcg_page_state (struct mem_cgroup *memcg,int idx); -unsigned long memcg_page_state_output (struct mem_cgroup *memcg,int item);bool memcg_stat_item_valid (int idx);bool memcg_vm_event_item_valid (enum vm_event_item idx);
mm/memcontrol.c renamed from mm/memcontrol.c~mm-memcontrol-prepare-for-reparenting-non-hierarchical-stats-update +68 -21
174 lines not on this page · raw message
mm/memcontrol-v1.h renamed from mm/memcontrol-v1.h~mm-memcontrol-prepare-for-reparenting-non-hierarchical-stats-update +0 -1
@@ -45,7 +45,6 @@ static inline bool do_memsw_account(voidunsigned long memcg_events_local (struct mem_cgroup *memcg,int event);unsigned long memcg_page_state_local (struct mem_cgroup *memcg,int idx); -void mod_memcg_page_state_local (struct mem_cgroup *memcg,int idx,unsigned long val);unsigned long memcg_page_state_local_output (struct mem_cgroup *memcg,int item);bool memcg1_alloc_events (struct mem_cgroup *memcg);void memcg1_free_events (struct mem_cgroup *memcg);
_
Qi Zheng
reviewerRe: [PATCH v5 update 29/32] mm: memcontrol: prepare for reparenting non-hierarchical stats
Hi Andrew,
Quoted 33 lines · Andrew Morton
Quoted 25 lines · On Sat, 28 Feb 2026 15:25:56 +0800 Qi Zheng <[email protected]> wrote:
To resolve the dying memcg issue, we need to reparent LRU folios of child memcg to its parent memcg. This could cause problems for non-hierarchical stats.
As Yosry Ahmed pointed out:
``` In short, if memory is charged to a dying cgroup at the time of reparenting, when the memory gets uncharged the stats updates will occur at the parent. This will update both hierarchical and non-hierarchical stats of the parent, which would corrupt the parent's non-hierarchical stats (because those counters were never incremented when the memory was charged). ```Now we have the following two types of non-hierarchical stats, and they are only used in
CONFIG_MEMCG_V1:a. memcg->vmstats->state_local[i]
b. pn->lruvec_stats->state_local[i]To ensure that these non-hierarchical stats work properly, we need to reparent these non-hierarchical stats after reparenting LRU folios. To this end, this commit makes the following preparations:
Quoted 5 lines
1. implement
reparent_state_local()to reparent non-hierarchical stats
2. makecss_killed_work_fn()to be called in rcu work, and implementget_non_dying_memcg_start()andget_non_dying_memcg_end()to avoid race between mod_memcg_state()/mod_memcg_lruvec_state()
andreparent_state_local()
Quoted 3 lines
That's a big update (delta patch is below).
What did it all do?
Oh sorry, I forgot the changlog.
Quoted 5 lines
I've bookmarked these reviewer comments:
https://lkml.kernel.org/r/CAO9r8zOhZgrym6oSrtg7b+HmNHEfWuAzZ0i8eYhm5-OEnfFLdw@mail.gmail.com
https://lkml.kernel.org/r/[email protected]
Based on Yosry Ahmed's suggestion and diff, the method has been changed to non-atomic mode, so there is no need to test the perfomance impact of atomic mode.
Quoted 1 line
This redundant declaration has been removed.
Thanks,
Qi
Quoted 220 lines
Were they all addressed in some fashion?
include/linux/memcontrol.h | 1 mm/memcontrol-v1.h | 1 mm/memcontrol.c | 89 ++++++++++++++++++++++++++--------- 3 files changed, 68 insertions(+), 23 deletions(-)include/linux/memcontrol.h renamed from include/linux/memcontrol.h~mm-memcontrol-prepare-for-reparenting-non-hierarchical-stats-update +0
-1@@ -956,7 +956,6 @@ static inline void mod_memcg_page_state(unsigned long memcg_events (struct mem_cgroup *memcg,int event);unsigned long memcg_page_state (struct mem_cgroup *memcg,int idx); -unsigned long memcg_page_state_output (struct mem_cgroup *memcg,int item);bool memcg_stat_item_valid (int idx);bool memcg_vm_event_item_valid (enum vm_event_item idx);mm/memcontrol.c renamed from mm/memcontrol.c~mm-memcontrol-prepare-for-reparenting-non-hierarchical-stats-update +68
-21@@ -234,11 +234,19 @@ static inline void reparent_state_local(if (cgroup_subsys_on_dfl (memory_cgrp_subsys))return ; +/* +* Reparent stats exposed non-hierarchically. Flush @memcg's stats first +* to read its stats accurately , and conservatively flush @parent's +* stats after reparenting to avoid hiding a potentially large stat +* update (e.g. from callers of mem_cgroup_flush_stats_ratelimited()). +*/ __mem_cgroup_flush_stats (memcg, true);/* The following counts are all non-hierarchical and need to be reparented. */ reparent_memcg1_state_local (memcg, parent);reparent_memcg1_lruvec_state_local (memcg, parent); + + __mem_cgroup_flush_stats(parent, true); }#else static inline void reparent_state_local (struct mem_cgroup *memcg,struct mem_cgroup *parent)@@ -442,7 +450,7 @@ struct lruvec_stats {long state[NR_MEMCG_NODE_STAT_ITEMS ];/* Non-hierarchical (CPU aggregated) state */ -atomic_long_t state_local[NR_MEMCG_NODE_STAT_ITEMS ]; +long state_local[NR_MEMCG_NODE_STAT_ITEMS ];/* Pending child counts during tree propagation */ long state_pending[NR_MEMCG_NODE_STAT_ITEMS ];@@ -485,7 +493,7 @@ unsigned long lruvec_page_state_local(streturn 0 ; pn = container_of(lruvec, struct mem_cgroup_per_node, lruvec); - x =atomic_long_read (&(pn->lruvec_stats->state_local[i])); + x =READ_ONCE (pn->lruvec_stats->state_local[i]);#ifdef CONFIG_SMP if (x <0 ) x =0 ;@@ -494,6 +502,9 @@ unsigned long lruvec_page_state_local(st }#ifdef CONFIG_MEMCG_V1 +static void __mod_memcg_lruvec_state (struct lruvec *lruvec, +enum node_stat_item idx,int val); +void reparent_memcg_lruvec_state_local (struct mem_cgroup *memcg,struct mem_cgroup *parent,int idx) {@@ -506,12 +517,10 @@ void reparent_memcg_lruvec_state_local(sfor_each_node (nid) {struct lruvec *child_lruvec =mem_cgroup_lruvec (memcg,NODE_DATA (nid));struct lruvec *parent_lruvec =mem_cgroup_lruvec (parent,NODE_DATA (nid)); -struct mem_cgroup_per_node *parent_pn;unsigned long value =lruvec_page_state_local (child_lruvec, idx); - parent_pn = container_of(parent_lruvec, struct mem_cgroup_per_node, lruvec); - -atomic_long_add (value, &(parent_pn->lruvec_stats->state_local[i])); +__mod_memcg_lruvec_state (child_lruvec, idx, -value); +__mod_memcg_lruvec_state (parent_lruvec, idx, value); } }#endif @@ -598,7 +607,7 @@ struct memcg_vmstats {unsigned long events[NR_MEMCG_EVENTS ];/* Non-hierarchical (CPU aggregated) page state & events */ -atomic_long_t state_local[MEMCG_VMSTAT_SIZE ]; +long state_local[MEMCG_VMSTAT_SIZE ];unsigned long events_local[NR_MEMCG_EVENTS ];/* Pending child counts during tree propagation */ @@ -835,7 +844,7 @@ unsigned long memcg_page_state_local(strif (WARN_ONCE (BAD_STAT_IDX (i),"%s: missing stat item %d\n" , __func__, idx))return 0 ; - x =atomic_long_read (&(memcg->vmstats->state_local[i])); + x =READ_ONCE (memcg->vmstats->state_local[i]);#ifdef CONFIG_SMP if (x <0 ) x =0 ;@@ -843,6 +852,51 @@ unsigned long memcg_page_state_local(strreturn x; } +static void __mod_memcg_state (struct mem_cgroup *memcg, +enum memcg_stat_item idx,int val) +{ +int i =memcg_stats_index (idx); +int cpu; + +if (mem_cgroup_disabled ()) +return ; + + cpu =get_cpu (); + +this_cpu_add (memcg->vmstats_percpu->state[i], val); + val =memcg_state_val_in_pages (idx, val); +memcg_rstat_updated (memcg, val, cpu); +trace_mod_memcg_state (memcg, idx, val); + +put_cpu (); +} + +static void __mod_memcg_lruvec_state (struct lruvec *lruvec, +enum node_stat_item idx,int val) +{ +struct mem_cgroup_per_node *pn; +struct mem_cgroup *memcg; +int i =memcg_stats_index (idx); +int cpu; + + pn = container_of(lruvec, struct mem_cgroup_per_node, lruvec); + memcg = pn->memcg; + + cpu =get_cpu (); + +/* Update memcg */ +this_cpu_add (memcg->vmstats_percpu->state[i], val); + +/* Update lruvec */ +this_cpu_add (pn->lruvec_stats_percpu->state[i], val); + + val =memcg_state_val_in_pages (idx, val); +memcg_rstat_updated (memcg, val, cpu); +trace_mod_memcg_lruvec_state (memcg, idx, val); + +put_cpu (); +} +void reparent_memcg_state_local (struct mem_cgroup *memcg,struct mem_cgroup *parent,int idx) {@@ -852,7 +906,8 @@ void reparent_memcg_state_local(struct mif (WARN_ONCE (BAD_STAT_IDX (i),"%s: missing stat item %d\n" , __func__, idx))return ; - atomic_long_add(value, &(parent->vmstats->state_local[i])); +__mod_memcg_state (memcg, idx, -value); + __mod_memcg_state(parent, idx, value); }#endif @@ -4174,8 +4229,6 @@ struct aggregate_control {long *aggregate;/* pointer to the non-hierarchichal (CPU aggregated) counters */ long *local; -/* pointer to the atomic non-hierarchichal (CPU aggregated) counters */ -atomic_long_t *alocal;/* pointer to the pending child counters during tree propagation */ long *pending;/* pointer to the parent's pending counters, could be NULL */ @@ -4213,12 +4266,8 @@ static void mem_cgroup_stat_aggregate(st }/* Aggregate counts on this level and propagate upwards */ -if (delta_cpu) { -if (ac->local) - ac->local[i] += delta_cpu; -else if (ac->alocal) -atomic_long_add (delta_cpu, &(ac->alocal[i])); - } +if (delta_cpu) + ac->local[i] += delta_cpu;if (delta) { ac->aggregate[i] += delta;@@ -4289,8 +4338,7 @@ static void mem_cgroup_css_rstat_flush(s ac = (struct aggregate_control ) { .aggregate = memcg->vmstats->state, - .local =NULL , - .alocal = memcg->vmstats->state_local, + .local = memcg->vmstats->state_local, .pending = memcg->vmstats->state_pending, .ppending = parent ? parent->vmstats->state_pending :NULL , .cstat = statc->state,@@ -4323,8 +4371,7 @@ static void mem_cgroup_css_rstat_flush(s ac = (struct aggregate_control ) { .aggregate = lstats->state, - .local =NULL , - .alocal = lstats->state_local, + .local = lstats->state_local, .pending = lstats->state_pending, .ppending = plstats ? plstats->state_pending :NULL , .cstat = lstatc->state,mm/memcontrol-v1.h renamed from mm/memcontrol-v1.h~mm-memcontrol-prepare-for-reparenting-non-hierarchical-stats-update +0
-1@@ -45,7 +45,6 @@ static inline bool do_memsw_account(voidunsigned long memcg_events_local (struct mem_cgroup *memcg,int event);unsigned long memcg_page_state_local (struct mem_cgroup *memcg,int idx); -void mod_memcg_page_state_local (struct mem_cgroup *memcg,int idx,unsigned long val);unsigned long memcg_page_state_local_output (struct mem_cgroup *memcg,int item);bool memcg1_alloc_events (struct mem_cgroup *memcg);void memcg1_free_events (struct mem_cgroup *memcg);_
Yosry Ahmed
maintainerRe: [PATCH v5 update 29/32] mm: memcontrol: prepare for reparenting non-hierarchical stats
In reply to Qi Zheng[..]
Quoted 39 lines
mm/memcontrol.c +35
-0@@ -763,6 +851,64 @@ unsigned long memcg_page_state_local(struct mem_cgroup *memcg, int idx)#endif return x; } + +static void __mod_memcg_state (struct mem_cgroup *memcg, +enum memcg_stat_item idx,int val) +{ +int i =memcg_stats_index (idx); +int cpu; + +if (mem_cgroup_disabled ()) +return ; + + cpu =get_cpu (); + +this_cpu_add (memcg->vmstats_percpu->state[i], val); + val =memcg_state_val_in_pages (idx, val); +memcg_rstat_updated (memcg, val, cpu); +trace_mod_memcg_state (memcg, idx, val); + +put_cpu (); +} + +static void __mod_memcg_lruvec_state (struct lruvec *lruvec, +enum node_stat_item idx,int val) +{ +struct mem_cgroup_per_node *pn; +struct mem_cgroup *memcg; +int i =memcg_stats_index (idx); +int cpu; + + pn = container_of(lruvec, struct mem_cgroup_per_node, lruvec); + memcg = pn->memcg; + + cpu =get_cpu (); + +/* Update memcg */ +this_cpu_add (memcg->vmstats_percpu->state[i], val);
mm/memcontrol.c unsigned long memcg_page_state_local(struct mem_cgroup *memcg, int idx) · line 898
Quoted 10 lines
mm/memcontrol.c +10
-0+ +/* Update lruvec */ +this_cpu_add (pn->lruvec_stats_percpu->state[i], val); + + val =memcg_state_val_in_pages (idx, val); +memcg_rstat_updated (memcg, val, cpu); +trace_mod_memcg_lruvec_state (memcg, idx, val); + +put_cpu (); +}
I don't think we should end up with two copies of __mod_memcg_state/mod_memcg_state() and __mod_memcg_lruvec_state/mod_memcg_lruvec_state(). I meant to refactor mod_memcg_state() to call __mod_memcg_state(), where the latter does not call get_non_dying_memcg_{start/end}(). Same for mod_memcg_lruvec_state().
Quoted 20 lines
mm/memcontrol.c +13
-1+ +void reparent_memcg_state_local (struct mem_cgroup *memcg, +struct mem_cgroup *parent,int idx) +{ +int i =memcg_stats_index (idx); +unsigned long value =memcg_page_state_local (memcg, idx); + +if (WARN_ONCE (BAD_STAT_IDX (i),"%s: missing stat item %d\n" , __func__, idx)) +return ; + +__mod_memcg_state (memcg, idx, -value); +__mod_memcg_state (parent, idx, value); +}#endif static void mod_memcg_lruvec_state(struct lruvec *lruvec, --2.20.1
Qi Zheng
reviewerRe: [PATCH v5 update 29/32] mm: memcontrol: prepare for reparenting non-hierarchical stats
Hi Yosry,
Quoted 49 lines · Yosry Ahmed
[..]
Quoted 48 lines
(quoted diff) +44
-0@@ -763,6 +851,64 @@ unsigned long memcg_page_state_local(struct mem_cgroup *memcg, int idx) #endif return x; } + +static void __mod_memcg_state(struct mem_cgroup *memcg, + enum memcg_stat_item idx, int val) +{ + int i = memcg_stats_index(idx); + int cpu; + + if (mem_cgroup_disabled()) + return; + + cpu = get_cpu(); + + this_cpu_add(memcg->vmstats_percpu->state[i], val); + val = memcg_state_val_in_pages(idx, val); + memcg_rstat_updated(memcg, val, cpu); + trace_mod_memcg_state(memcg, idx, val); + + put_cpu(); +} + +static void __mod_memcg_lruvec_state(struct lruvec *lruvec, + enum node_stat_item idx, int val) +{ + struct mem_cgroup_per_node *pn; + struct mem_cgroup *memcg; + int i = memcg_stats_index(idx); + int cpu; + + pn = container_of(lruvec, struct mem_cgroup_per_node, lruvec); + memcg = pn->memcg; + + cpu = get_cpu(); + + /* Update memcg */ + this_cpu_add(memcg->vmstats_percpu->state[i], val); + + /* Update lruvec */ + this_cpu_add(pn->lruvec_stats_percpu->state[i], val); + + val = memcg_state_val_in_pages(idx, val); + memcg_rstat_updated(memcg, val, cpu); + trace_mod_memcg_lruvec_state(memcg, idx, val); + + put_cpu();
Quoted 8 lines
Review unsigned long memcg_page_state_local(struct mem_cgroup *memcg, int idx) · line 898
Quoted 1 line
(quoted diff) +1
-0+}I don't think we should end up with two copies of __mod_memcg_state/mod_memcg_state() and __mod_memcg_lruvec_state/mod_memcg_lruvec_state(). I meant to refactor
mod_memcg_state()to call__mod_memcg_state(), where the latter does not call get_non_dying_memcg_{start/end}(). Same formod_memcg_lruvec_state().
Okay, like the following? But this would require modifications to [PATCH v5 31/32]. If there are no problems, I will send the updated patch to [PATCH v5 29/32] and [PATCH v5 31/32].
mm/memcontrol.c +2 -1
@@ -528,7 +528,8 @@ unsigned long lruvec_page_state_local(struct lruvec *lruvec,#ifdef CONFIG_MEMCG_V1 static void __mod_memcg_lruvec_state (struct lruvec *lruvec, -enum node_stat_item idx,int val); +enum node_stat_item idx,int val, +bool reparent);void reparent_memcg_lruvec_state_local (struct mem_cgroup *memcg,struct mem_cgroup *parent,int idx)
@@ -544,8 +545,8 @@ void reparent_memcg_lruvec_state_local(struct mem_cgroup *memcg, struct lruvec *parent_lruvec =
mem_cgroup_lruvec(parent, NODE_DATA(nid));
unsigned long value =lruvec_page_state_local(child_lruvec, idx);
- __mod_memcg_lruvec_state(child_lruvec, idx, -value);
- __mod_memcg_lruvec_state(parent_lruvec, idx, value);
+ __mod_memcg_lruvec_state(child_lruvec, idx, -value, true);
+ __mod_memcg_lruvec_state(parent_lruvec, idx, value, true);
} }
#endif
@@ -831,14 +832,9 @@ static inline void get_non_dying_memcg_end(void)
}
#endif
-/**
- * mod_memcg_state - update cgroup memory statistics
- * @memcg: the memory cgroup
- * @idx: the stat item - can be enum memcg_stat_item or enum node_stat_item
- * @val: delta to add to the counter, can be negative
- */
-void mod_memcg_state(struct mem_cgroup *memcg, enum memcg_stat_item idx,
- int val)
+static void __mod_memcg_state(struct mem_cgroup *memcg,
+ enum memcg_stat_item idx, int val,
+ bool reparent)
{
int i = memcg_stats_index(idx);
int cpu;
@@ -846,24 +842,38 @@ void mod_memcg_state(struct mem_cgroup *memcg, enum memcg_stat_item idx,
if (mem_cgroup_disabled())
return;- if (WARN_ONCE(BAD_STAT_IDX(i), "%s: missing stat item %d\n", __func__, idx))
+ if (!reparent && WARN_ONCE(BAD_STAT_IDX(i), "%s: missing stat item %d\n", __func__, idx))
return;
cpu = get_cpu();- memcg = get_non_dying_memcg_start(memcg);
+ if (!reparent)
+ memcg = get_non_dying_memcg_start(memcg);
this_cpu_add(memcg->vmstats_percpu->state[i], val);
val = memcg_state_val_in_pages(idx, val);
memcg_rstat_updated(memcg, val, cpu);- get_non_dying_memcg_end();
+ if (!reparent)
+ get_non_dying_memcg_end();
trace_mod_memcg_state(memcg, idx, val);
put_cpu();}
+/**
+ * mod_memcg_state - update cgroup memory statistics
+ * @memcg: the memory cgroup
+ * @idx: the stat item - can be enum memcg_stat_item or enum node_stat_item
+ * @val: delta to add to the counter, can be negative
+ */
+void mod_memcg_state(struct mem_cgroup *memcg, enum memcg_stat_item idx,
+ int val)
+{
+ __mod_memcg_state(memcg, idx, val, false);
+}
+
#ifdef CONFIG_MEMCG_V1
/* idx can be of type enum memcg_stat_item or node_stat_item. */
unsigned long memcg_page_state_local(struct mem_cgroup *memcg, int idx)
@@ -882,51 +892,6 @@ unsigned long memcg_page_state_local(struct mem_cgroup *memcg, int idx)
return x;
}
-static void __mod_memcg_state(struct mem_cgroup *memcg,
- enum memcg_stat_item idx, int val)
-{
- int i = memcg_stats_index(idx);
- int cpu;
-
- if (mem_cgroup_disabled())
- return;
-
- cpu = get_cpu();
-
- this_cpu_add(memcg->vmstats_percpu->state[i], val); - val = memcg_state_val_in_pages(idx, val); - memcg_rstat_updated(memcg, val, cpu); - trace_mod_memcg_state(memcg, idx, val);
-
- put_cpu();
-}
-
-static void __mod_memcg_lruvec_state(struct lruvec *lruvec,
- enum node_stat_item idx, int val)
-{
- struct mem_cgroup_per_node *pn; - struct mem_cgroup *memcg; - int i = memcg_stats_index(idx); - int cpu;
-
- pn = container_of(lruvec, struct mem_cgroup_per_node, lruvec);
- memcg = pn->memcg;
-
- cpu = get_cpu();
-
- /* Update memcg */
- this_cpu_add(memcg->vmstats_percpu->state[i], val);
-
- /* Update lruvec */
- this_cpu_add(pn->lruvec_stats_percpu->state[i], val);
-
- val = memcg_state_val_in_pages(idx, val); - memcg_rstat_updated(memcg, val, cpu); - trace_mod_memcg_lruvec_state(memcg, idx, val);
-
- put_cpu();
-}
-
void reparent_memcg_state_local(struct mem_cgroup *memcg,
struct mem_cgroup *parent, int idx)
{
@@ -936,14 +901,14 @@ void reparent_memcg_state_local(struct mem_cgroup *memcg,
if (WARN_ONCE(BAD_STAT_IDX(i), "%s: missing stat item %d\n",__func__, idx))
return;
- __mod_memcg_state(memcg, idx, -value); - __mod_memcg_state(parent, idx, value); + __mod_memcg_state(memcg, idx, -value, true); + __mod_memcg_state(parent, idx, value, true);
}
#endif
-static void mod_memcg_lruvec_state(struct lruvec *lruvec,
- enum node_stat_item idx,
- int val)
+static void __mod_memcg_lruvec_state(struct lruvec *lruvec,
+ enum node_stat_item idx, int val,
+ bool reparent)
{
struct pglist_data *pgdat = lruvec_pgdat(lruvec);
struct mem_cgroup_per_node *pn;
@@ -951,7 +916,7 @@ static void mod_memcg_lruvec_state(struct lruvec *lruvec,
int i = memcg_stats_index(idx);
int cpu;- if (WARN_ONCE(BAD_STAT_IDX(i), "%s: missing stat item %d\n", __func__, idx))
+ if (!reparent && WARN_ONCE(BAD_STAT_IDX(i), "%s: missing stat item %d\n", __func__, idx))
return;
pn = container_of(lruvec, struct mem_cgroup_per_node, lruvec);@@ -959,8 +924,10 @@ static void mod_memcg_lruvec_state(struct lruvec *lruvec,
cpu = get_cpu();
- memcg = get_non_dying_memcg_start(memcg);
- pn = memcg->nodeinfo[pgdat->node_id];
+ if (!reparent) {+ memcg = get_non_dying_memcg_start(memcg);
+ pn = memcg->nodeinfo[pgdat->node_id];
+ }
/* Update memcg */
this_cpu_add(memcg->vmstats_percpu->state[i], val);
@@ -969,13 +936,21 @@ static void mod_memcg_lruvec_state(struct lruvec *lruvec,
val = memcg_state_val_in_pages(idx, val);
memcg_rstat_updated(memcg, val, cpu);- get_non_dying_memcg_end();
+ if (!reparent)
+ get_non_dying_memcg_end();
trace_mod_memcg_lruvec_state(memcg, idx, val);
put_cpu();}
+static void mod_memcg_lruvec_state(struct lruvec *lruvec,
+ enum node_stat_item idx,
+ int val)
+{
+ __mod_memcg_lruvec_state(lruvec, idx, val, false);
+}
+
/**
* mod_lruvec_state - update lruvec memory statistics
* @lruvec: the lruvec
Quoted 1 line
Yosry Ahmed
maintainerRe: [PATCH v5 update 29/32] mm: memcontrol: prepare for reparenting non-hierarchical stats
Quoted 1 line · Qi Zheng
Hi Yosry,
[..]
Quoted 11 lines
Quoted 7 lines
I don't think we should end up with two copies of __mod_memcg_state/mod_memcg_state() and __mod_memcg_lruvec_state/mod_memcg_lruvec_state(). I meant to refactor
mod_memcg_state()to call__mod_memcg_state(), where the latter does not call get_non_dying_memcg_{start/end}(). Same formod_memcg_lruvec_state().Okay, like the following? But this would require modifications to [PATCH v5 31/32]. If there are no problems, I will send the updated patch to [PATCH v5 29/32] and [PATCH v5 31/32].
I cannot apply the diff, seems a bit corrupted.
But ideally, instead of a @reparent argument, we just have __mod_memcg_lruvec_state() and __mod_memcg_state() do the work without getting parent of dead memcgs, and then mod_memcg_lruvec_state() and mod_memcg_state() just call them after get_non_dying_memcg_start().
What about this (untested), it should apply on top of 'mm: memcontrol:
eliminate the problem of dying memory cgroup for LRU folios' in mm-new, so maybe it needs to be broken down across different patches:
mm/memcontrol.c +38 -76
@@ -527,7 +527,7 @@ unsigned long lruvec_page_state_local(struct lruvec *lruvec, }#ifdef CONFIG_MEMCG_V1 -static void __mod_memcg_lruvec_state (struct lruvec *lruvec, +static void __mod_memcg_lruvec_state (struct mem_cgroup_per_node *pn,enum node_stat_item idx,int val); void reparent_memcg_lruvec_state_local(struct mem_cgroup *memcg,
@@ -536,16 +536,17 @@ void reparent_memcg_lruvec_state_local(struct mem_cgroup *memcg,int i =memcg_stats_index (idx);int nid; -if (WARN_ONCE (BAD_STAT_IDX (i),"%s: missing stat item %d\n" , __func__, idx)) -return ; -for_each_node (nid) {struct lruvec *child_lruvec =mem_cgroup_lruvec (memcg,NODE_DATA (nid));struct lruvec *parent_lruvec =mem_cgroup_lruvec (parent,NODE_DATA (nid));unsigned long value =lruvec_page_state_local (child_lruvec, idx); +struct mem_cgroup_per_node *child_pn, *parent_pn; -__mod_memcg_lruvec_state (child_lruvec, idx, -value); -__mod_memcg_lruvec_state (parent_lruvec, idx, value); + child_pn = container_of(child_lruvec, struct mem_cgroup_per_node, lruvec); + parent_pn = container_of(parent_lruvec, struct mem_cgroup_per_node, lruvec); + +__mod_memcg_lruvec_state (child_pn, idx, -value); +__mod_memcg_lruvec_state (parent_pn, idx, value); } }#endif
@@ -831,39 +832,42 @@ static inline void get_non_dying_memcg_end(void) }#endif -/** -* mod_memcg_state - update cgroup memory statistics -* @memcg: the memory cgroup -* @idx: the stat item - can be enum memcg_stat_item or enum node_stat_item -* @val: delta to add to the counter, can be negative -*/ -void mod_memcg_state (struct mem_cgroup *memcg,enum memcg_stat_item idx, -int val) +static void __mod_memcg_state (struct mem_cgroup *memcg, +enum memcg_stat_item idx,int val) {int i =memcg_stats_index (idx);int cpu; -if (mem_cgroup_disabled ()) -return ; -if (WARN_ONCE (BAD_STAT_IDX (i),"%s: missing stat item %d\n" , __func__, idx))return ; cpu =get_cpu (); - memcg =get_non_dying_memcg_start (memcg); -this_cpu_add (memcg->vmstats_percpu->state[i], val); val =memcg_state_val_in_pages (idx, val);memcg_rstat_updated (memcg, val, cpu); - -get_non_dying_memcg_end (); -trace_mod_memcg_state (memcg, idx, val);put_cpu (); } +/** +* mod_memcg_state - update cgroup memory statistics +* @memcg: the memory cgroup +* @idx: the stat item - can be enum memcg_stat_item or enum node_stat_item +* @val: delta to add to the counter, can be negative +*/ +void mod_memcg_state (struct mem_cgroup *memcg,enum memcg_stat_item idx, +int val) +{ +if (mem_cgroup_disabled ()) +return ; + + memcg =get_non_dying_memcg_start (memcg); +__mod_memcg_state (memcg, idx, val); +get_non_dying_memcg_end (); +} +#ifdef CONFIG_MEMCG_V1 /* idx can be of type enum memcg_stat_item or node_stat_item. */ unsigned long memcg_page_state_local (struct mem_cgroup *memcg,int idx)
@@ -882,35 +886,26 @@ unsigned long memcg_page_state_local(struct mem_cgroup *memcg, int idx) return x; } -static void __mod_memcg_state (struct mem_cgroup *memcg, -enum memcg_stat_item idx,int val) +void reparent_memcg_state_local (struct mem_cgroup *memcg, +struct mem_cgroup *parent,int idx) {int i =memcg_stats_index (idx); -int cpu; - -if (mem_cgroup_disabled ()) -return ; - - cpu =get_cpu (); - -this_cpu_add (memcg->vmstats_percpu->state[i], val); - val =memcg_state_val_in_pages (idx, val); -memcg_rstat_updated (memcg, val, cpu); -trace_mod_memcg_state (memcg, idx, val); +unsigned long value =memcg_page_state_local (memcg, idx); -put_cpu (); +__mod_memcg_state (memcg, idx, -value); +__mod_memcg_state (parent, idx, value); } +#endif -static void __mod_memcg_lruvec_state (struct lruvec *lruvec, +static void __mod_memcg_lruvec_state (struct mem_cgroup_per_node *pn,enum node_stat_item idx,int val) { -struct mem_cgroup_per_node *pn; -struct mem_cgroup *memcg; +struct mem_cgroup *memcg = pn->memcg;int i =memcg_stats_index (idx);int cpu; - pn = container_of(lruvec, struct mem_cgroup_per_node, lruvec); - memcg = pn->memcg; +if (WARN_ONCE (BAD_STAT_IDX (i),"%s: missing stat item %d\n" , __func__, idx)) +return ; cpu =get_cpu ();
@@ -927,20 +922,6 @@ static void __mod_memcg_lruvec_state(struct lruvec *lruvec,put_cpu (); } -void reparent_memcg_state_local (struct mem_cgroup *memcg, -struct mem_cgroup *parent,int idx) -{ -int i =memcg_stats_index (idx); -unsigned long value =memcg_page_state_local (memcg, idx); - -if (WARN_ONCE (BAD_STAT_IDX (i),"%s: missing stat item %d\n" , __func__, idx)) -return ; - -__mod_memcg_state (memcg, idx, -value); -__mod_memcg_state (parent, idx, value); -} -#endif -static void mod_memcg_lruvec_state (struct lruvec *lruvec,enum node_stat_item idx,int val)
@@ -948,32 +929,13 @@ static void mod_memcg_lruvec_state(struct lruvec *lruvec,struct pglist_data *pgdat =lruvec_pgdat (lruvec);struct mem_cgroup_per_node *pn;struct mem_cgroup *memcg; -int i =memcg_stats_index (idx); -int cpu; - -if (WARN_ONCE (BAD_STAT_IDX (i),"%s: missing stat item %d\n" , __func__, idx)) -return ; pn = container_of(lruvec, struct mem_cgroup_per_node, lruvec); - memcg = pn->memcg; - - cpu =get_cpu (); - - memcg =get_non_dying_memcg_start (memcg); + memcg =get_non_dying_memcg_start (pn->memcg); pn = memcg->nodeinfo[pgdat->node_id]; - -/* Update memcg */ -this_cpu_add (memcg->vmstats_percpu->state[i], val); -/* Update lruvec */ -this_cpu_add (pn->lruvec_stats_percpu->state[i], val); - val =memcg_state_val_in_pages (idx, val); -memcg_rstat_updated (memcg, val, cpu); - +__mod_memcg_lruvec_state (pn, idx, val);get_non_dying_memcg_end (); -trace_mod_memcg_lruvec_state (memcg, idx, val); - -put_cpu (); } /
Qi Zheng
reviewerRe: [PATCH v5 update 29/32] mm: memcontrol: prepare for reparenting non-hierarchical stats
Quoted 17 lines · Yosry Ahmed
Quoted 1 line · Qi Zheng
Hi Yosry,
[..]
Quoted 11 lines
Quoted 7 lines
I don't think we should end up with two copies of __mod_memcg_state/mod_memcg_state() and __mod_memcg_lruvec_state/mod_memcg_lruvec_state(). I meant to refactor
mod_memcg_state()to call__mod_memcg_state(), where the latter does not call get_non_dying_memcg_{start/end}(). Same formod_memcg_lruvec_state().Okay, like the following? But this would require modifications to [PATCH v5 31/32]. If there are no problems, I will send the updated patch to [PATCH v5 29/32] and [PATCH v5 31/32].
I cannot apply the diff, seems a bit corrupted.
Quoted 8 lines
But ideally, instead of a @reparent argument, we just have
__mod_memcg_lruvec_state()and__mod_memcg_state()do the work without getting parent of dead memcgs, and thenmod_memcg_lruvec_state()andmod_memcg_state()just call them afterget_non_dying_memcg_start().What about this (untested), it should apply on top of 'mm: memcontrol:
eliminate the problem of dying memory cgroup for LRU folios' in mm-new, so maybe it needs to be broken down across different patches:
It looks feasible. I will apply and test it. If there are no problems, I will send the update patch.
Thanks,
Qi
Quoted 198 lines
mm/memcontrol.c +38
-76@@ -527,7 +527,7 @@ unsigned long lruvec_page_state_local(struct lruvec *lruvec, }#ifdef CONFIG_MEMCG_V1 -static void __mod_memcg_lruvec_state (struct lruvec *lruvec, +static void __mod_memcg_lruvec_state (struct mem_cgroup_per_node *pn,enum node_stat_item idx,int val); void reparent_memcg_lruvec_state_local(struct mem_cgroup *memcg,@@ -536,16 +536,17 @@ void reparent_memcg_lruvec_state_local(struct mem_cgroup *memcg,int i =memcg_stats_index (idx);int nid; -if (WARN_ONCE (BAD_STAT_IDX (i),"%s: missing stat item %d\n" , __func__, idx)) -return ; -for_each_node (nid) {struct lruvec *child_lruvec =mem_cgroup_lruvec (memcg,NODE_DATA (nid));struct lruvec *parent_lruvec =mem_cgroup_lruvec (parent,NODE_DATA (nid));unsigned long value =lruvec_page_state_local (child_lruvec, idx); +struct mem_cgroup_per_node *child_pn, *parent_pn; -__mod_memcg_lruvec_state (child_lruvec, idx, -value); -__mod_memcg_lruvec_state (parent_lruvec, idx, value); + child_pn = container_of(child_lruvec, struct mem_cgroup_per_node, lruvec); + parent_pn = container_of(parent_lruvec, struct mem_cgroup_per_node, lruvec); + +__mod_memcg_lruvec_state (child_pn, idx, -value); +__mod_memcg_lruvec_state (parent_pn, idx, value); } }#endif @@ -831,39 +832,42 @@ static inline void get_non_dying_memcg_end(void) }#endif -/** -* mod_memcg_state - update cgroup memory statistics -* @memcg: the memory cgroup -* @idx: the stat item - can be enum memcg_stat_item or enum node_stat_item -* @val: delta to add to the counter, can be negative -*/ -void mod_memcg_state (struct mem_cgroup *memcg,enum memcg_stat_item idx, -int val) +static void __mod_memcg_state (struct mem_cgroup *memcg, +enum memcg_stat_item idx,int val) {int i =memcg_stats_index (idx);int cpu; -if (mem_cgroup_disabled ()) -return ; -if (WARN_ONCE (BAD_STAT_IDX (i),"%s: missing stat item %d\n" , __func__, idx))return ; cpu =get_cpu (); - memcg =get_non_dying_memcg_start (memcg); -this_cpu_add (memcg->vmstats_percpu->state[i], val); val =memcg_state_val_in_pages (idx, val);memcg_rstat_updated (memcg, val, cpu); - -get_non_dying_memcg_end (); -trace_mod_memcg_state (memcg, idx, val);put_cpu (); } +/** +* mod_memcg_state - update cgroup memory statistics +* @memcg: the memory cgroup +* @idx: the stat item - can be enum memcg_stat_item or enum node_stat_item +* @val: delta to add to the counter, can be negative +*/ +void mod_memcg_state (struct mem_cgroup *memcg,enum memcg_stat_item idx, +int val) +{ +if (mem_cgroup_disabled ()) +return ; + + memcg =get_non_dying_memcg_start (memcg); +__mod_memcg_state (memcg, idx, val); +get_non_dying_memcg_end (); +} +#ifdef CONFIG_MEMCG_V1 /* idx can be of type enum memcg_stat_item or node_stat_item. */ unsigned long memcg_page_state_local (struct mem_cgroup *memcg,int idx)@@ -882,35 +886,26 @@ unsigned long memcg_page_state_local(struct mem_cgroup *memcg, int idx) return x; } -static void __mod_memcg_state (struct mem_cgroup *memcg, -enum memcg_stat_item idx,int val) +void reparent_memcg_state_local (struct mem_cgroup *memcg, +struct mem_cgroup *parent,int idx) {int i =memcg_stats_index (idx); -int cpu; - -if (mem_cgroup_disabled ()) -return ; - - cpu =get_cpu (); - -this_cpu_add (memcg->vmstats_percpu->state[i], val); - val =memcg_state_val_in_pages (idx, val); -memcg_rstat_updated (memcg, val, cpu); -trace_mod_memcg_state (memcg, idx, val); +unsigned long value =memcg_page_state_local (memcg, idx); -put_cpu (); +__mod_memcg_state (memcg, idx, -value); +__mod_memcg_state (parent, idx, value); } +#endif -static void __mod_memcg_lruvec_state (struct lruvec *lruvec, +static void __mod_memcg_lruvec_state (struct mem_cgroup_per_node *pn,enum node_stat_item idx,int val) { -struct mem_cgroup_per_node *pn; -struct mem_cgroup *memcg; +struct mem_cgroup *memcg = pn->memcg;int i =memcg_stats_index (idx);int cpu; - pn = container_of(lruvec, struct mem_cgroup_per_node, lruvec); - memcg = pn->memcg; +if (WARN_ONCE (BAD_STAT_IDX (i),"%s: missing stat item %d\n" , __func__, idx)) +return ; cpu =get_cpu ();@@ -927,20 +922,6 @@ static void __mod_memcg_lruvec_state(struct lruvec *lruvec,put_cpu (); } -void reparent_memcg_state_local (struct mem_cgroup *memcg, -struct mem_cgroup *parent,int idx) -{ -int i =memcg_stats_index (idx); -unsigned long value =memcg_page_state_local (memcg, idx); - -if (WARN_ONCE (BAD_STAT_IDX (i),"%s: missing stat item %d\n" , __func__, idx)) -return ; - -__mod_memcg_state (memcg, idx, -value); -__mod_memcg_state (parent, idx, value); -} -#endif -static void mod_memcg_lruvec_state (struct lruvec *lruvec,enum node_stat_item idx,int val)@@ -948,32 +929,13 @@ static void mod_memcg_lruvec_state(struct lruvec *lruvec,struct pglist_data *pgdat =lruvec_pgdat (lruvec);struct mem_cgroup_per_node *pn;struct mem_cgroup *memcg; -int i =memcg_stats_index (idx); -int cpu; - -if (WARN_ONCE (BAD_STAT_IDX (i),"%s: missing stat item %d\n" , __func__, idx)) -return ; pn = container_of(lruvec, struct mem_cgroup_per_node, lruvec); - memcg = pn->memcg; - - cpu =get_cpu (); - - memcg =get_non_dying_memcg_start (memcg); + memcg =get_non_dying_memcg_start (pn->memcg); pn = memcg->nodeinfo[pgdat->node_id]; - -/* Update memcg */ -this_cpu_add (memcg->vmstats_percpu->state[i], val); -/* Update lruvec */ -this_cpu_add (pn->lruvec_stats_percpu->state[i], val); - val =memcg_state_val_in_pages (idx, val); -memcg_rstat_updated (memcg, val, cpu); - +__mod_memcg_lruvec_state (pn, idx, val);get_non_dying_memcg_end (); -trace_mod_memcg_lruvec_state (memcg, idx, val); - -put_cpu (); } /
Re: [PATCH v5 update 29/32] mm: memcontrol: prepare for reparenting non-hierarchical stats
In reply to Yosry AhmedQuoted 17 lines · Yosry Ahmed
Quoted 1 line · Qi Zheng
Hi Yosry,
[..]
Quoted 11 lines
Quoted 7 lines
I don't think we should end up with two copies of __mod_memcg_state/mod_memcg_state() and __mod_memcg_lruvec_state/mod_memcg_lruvec_state(). I meant to refactor
mod_memcg_state()to call__mod_memcg_state(), where the latter does not call get_non_dying_memcg_{start/end}(). Same formod_memcg_lruvec_state().Okay, like the following? But this would require modifications to [PATCH v5 31/32]. If there are no problems, I will send the updated patch to [PATCH v5 29/32] and [PATCH v5 31/32].
I cannot apply the diff, seems a bit corrupted.
Quoted 9 lines
But ideally, instead of a @reparent argument, we just have
__mod_memcg_lruvec_state()and__mod_memcg_state()do the work without getting parent of dead memcgs, and thenmod_memcg_lruvec_state()andmod_memcg_state()just call them afterget_non_dying_memcg_start().What about this (untested), it should apply on top of 'mm: memcontrol:
eliminate the problem of dying memory cgroup for LRU folios' in mm-new, so maybe it needs to be broken down across different patches:
I applied and tested it, so the final updated patches is as follows, If there are no problems, I will send out the official patches.
[PATCH v5 update v2 29/32]:
``` Author: Qi Zheng <[email protected]> Date: Mon Jan 5 19:59:28 2026 +0800 mm: memcontrol: prepare for reparenting non-hierarchical stats To resolve the dying memcg issue, we need to reparent LRU folios of child memcg to its parent memcg. This could cause problems for non-hierarchical stats. As Yosry Ahmed pointed out: ```
In short, if memory is charged to a dying cgroup at the time of reparenting, when the memory gets uncharged the stats updates will
occur
at the parent. This will update both hierarchical and non-hierarchical stats of the parent, which would corrupt the parent's non-hierarchical stats (because those counters were never incremented when the
memory was
charged).
```
Now we have the following two types of non-hierarchical stats, and they are only used in CONFIG_MEMCG_V1:
a. memcg->vmstats->state_local[i]
b. pn->lruvec_stats->state_local[i]
To ensure that these non-hierarchical stats work properly, we need to reparent these non-hierarchical stats after reparenting LRU folios. To this end, this commit makes the following preparations:
1. implement reparent_state_local() to reparent non-hierarchical stats
2. make css_killed_work_fn() to be called in rcu work, and implement get_non_dying_memcg_start() and get_non_dying_memcg_end() to
avoid race
between mod_memcg_state()/mod_memcg_lruvec_state()
and reparent_state_local()
kernel/cgroup/cgroup.c +4 -4
@@ -6044,8 +6044,8 @@ int cgroup_mkdir(struct kernfs_node *parent_kn, const char *name, umode_t mode)*/ static void css_killed_work_fn (struct work_struct *work) { -struct cgroup_subsys_state *css = - container_of(work, struct cgroup_subsys_state, destroy_work); +struct cgroup_subsys_state *css =container_of (to_rcu_work (work), + struct cgroup_subsys_state, destroy_rwork);cgroup_lock ();
@@ -6066,8 +6066,8 @@ static void css_killed_ref_fn(struct percpu_ref *ref) container_of(ref, struct cgroup_subsys_state, refcnt);if (atomic_dec_and_test (&css->online_cnt)) { -INIT_WORK (&css->destroy_work, css_killed_work_fn); -queue_work (cgroup_offline_wq, &css->destroy_work); +INIT_RCU_WORK (&css->destroy_rwork, css_killed_work_fn); +queue_rcu_work (cgroup_offline_wq, &css->destroy_rwork); } }
mm/memcontrol-v1.c +16 -0
@@ -1897,6 +1897,22 @@ static const unsigned int memcg1_events[] = { PGMAJFAULT, }; +void reparent_memcg1_state_local (struct mem_cgroup *memcg,struct mem_cgroup *parent) +{ +int i; + +for (i =0 ; i <ARRAY_SIZE (memcg1_stats); i++) +reparent_memcg_state_local (memcg, parent, memcg1_stats[i]); +} + +void reparent_memcg1_lruvec_state_local (struct mem_cgroup *memcg,struct mem_cgroup *parent) +{ +int i; + +for (i =0 ; i <NR_LRU_LISTS ; i++) +reparent_memcg_lruvec_state_local (memcg, parent, i); +} +void memcg1_stat_format (struct mem_cgroup *memcg,struct seq_buf *s) {unsigned long memory, memsw;
mm/memcontrol-v1.h +7 -0
@@ -77,6 +77,13 @@ void memcg1_uncharge_batch(struct mem_cgroup *memcg, unsigned long pgpgout, unsigned long nr_memory, int nid);void memcg1_stat_format (struct mem_cgroup *memcg,struct seq_buf *s); +void reparent_memcg1_state_local (struct mem_cgroup *memcg,struct mem_cgroup *parent); +void reparent_memcg1_lruvec_state_local (struct mem_cgroup *memcg,struct mem_cgroup *parent); + +void reparent_memcg_state_local (struct mem_cgroup *memcg, +struct mem_cgroup *parent,int idx); +void reparent_memcg_lruvec_state_local (struct mem_cgroup *memcg, +struct mem_cgroup *parent,int idx);void memcg1_account_kmem (struct mem_cgroup *memcg,int nr_pages);static inline bool memcg1_tcpmem_active (struct mem_cgroup *memcg)
mm/memcontrol.c +28 -0
@@ -226,6 +226,34 @@ static inline struct obj_cgroup *__memcg_reparent_objcgs(struct mem_cgroup *memcreturn objcg; } +#ifdef CONFIG_MEMCG_V1 +static void __mem_cgroup_flush_stats (struct mem_cgroup *memcg,bool force); + +static inline void reparent_state_local (struct mem_cgroup *memcg,struct mem_cgroup *parent) +{ +if (cgroup_subsys_on_dfl (memory_cgrp_subsys)) +return ; + +/* +* Reparent stats exposed non-hierarchically. Flush @memcg's stats first +* to read its stats accurately , and conservatively flush @parent's +* stats after reparenting to avoid hiding a potentially large stat +* update (e.g. from callers of mem_cgroup_flush_stats_ratelimited()). +*/ +__mem_cgroup_flush_stats (memcg, true); + +/* The following counts are all non-hierarchical and need to be reparented. */ +reparent_memcg1_state_local (memcg, parent); +reparent_memcg1_lruvec_state_local (memcg, parent); + +__mem_cgroup_flush_stats (parent, true); +} +#else +static inline void reparent_state_local (struct mem_cgroup *memcg,struct mem_cgroup *parent) +{ +} +#endif + static inline void reparent_locks(struct mem_cgroup *memcg, struct
mem_cgroup *parent)
{
spin_lock_irq(&objcg_lock);
@@ -473,6 +501,30 @@ unsigned long lruvec_page_state_local(struct lruvec *lruvec,
return x;}
+#ifdef CONFIG_MEMCG_V1
+static void __mod_memcg_lruvec_state(struct mem_cgroup_per_node *pn,
+ enum node_stat_item idx, int val);
+
+void reparent_memcg_lruvec_state_local(struct mem_cgroup *memcg,
+ struct mem_cgroup *parent, int idx)
+{
+ int nid;
+
+ for_each_node(nid) {
+ struct lruvec *child_lruvec = mem_cgroup_lruvec(memcg, NODE_DATA(nid)); + struct lruvec *parent_lruvec = mem_cgroup_lruvec(parent, NODE_DATA(nid)); + unsigned long value = lruvec_page_state_local(child_lruvec, idx); + struct mem_cgroup_per_node *child_pn, *parent_pn;
+
+ child_pn = container_of(child_lruvec, struct mem_cgroup_per_node, lruvec);
+ parent_pn = container_of(parent_lruvec, struct mem_cgroup_per_node, lruvec);
+
+ __mod_memcg_lruvec_state(child_pn, idx, -value);
+ __mod_memcg_lruvec_state(parent_pn, idx, value);
+ }
+}
+#endif
+
/* Subset of vm_event_item to report for memcg event stats */ static const unsigned int memcg_vm_event_stat[] = {
#ifdef CONFIG_MEMCG_V1
@@ -718,21 +770,48 @@ static int memcg_state_val_in_pages(int idx, int val)
return max(val * unit / PAGE_SIZE, 1UL);
}
-/**
- * mod_memcg_state - update cgroup memory statistics
- * @memcg: the memory cgroup
- * @idx: the stat item - can be enum memcg_stat_item or enum node_stat_item
- * @val: delta to add to the counter, can be negative
+#ifdef CONFIG_MEMCG_V1
+/*
+ * Used in mod_memcg_state() and mod_memcg_lruvec_state() to avoid race with
+ * reparenting of non-hierarchical state_locals.
*/
-void mod_memcg_state(struct mem_cgroup *memcg, enum memcg_stat_item idx,
- int val)
+static inline struct mem_cgroup *get_non_dying_memcg_start(struct mem_cgroup *memcg)
{
- int i = memcg_stats_index(idx); - int cpu; + if (cgroup_subsys_on_dfl(memory_cgrp_subsys))
+ return memcg;
- if (mem_cgroup_disabled())
+ rcu_read_lock();
+
+ while (memcg_is_dying(memcg))
+ memcg = parent_mem_cgroup(memcg);
+
+ return memcg;
+}
+
+static inline void get_non_dying_memcg_end(void)
+{
+ if (cgroup_subsys_on_dfl(memory_cgrp_subsys))
return;
+ rcu_read_unlock();
+}
+#else
+static inline struct mem_cgroup *get_non_dying_memcg_start(struct mem_cgroup *memcg) +{
+ return memcg;
+}
+
+static inline void get_non_dying_memcg_end(void)
+{
+}
+#endif
+
+static void __mod_memcg_state(struct mem_cgroup *memcg,
+ enum memcg_stat_item idx, int val)
+{
+ int i = memcg_stats_index(idx);
+ int cpu;
+
if (WARN_ONCE(BAD_STAT_IDX(i), "%s: missing stat item %d\n",__func__, idx))
return;
@@ -746,6 +825,21 @@ void mod_memcg_state(struct mem_cgroup *memcg, enum memcg_stat_item idx,
put_cpu();
}
+/**
+ * mod_memcg_state - update cgroup memory statistics
+ * @memcg: the memory cgroup
+ * @idx: the stat item - can be enum memcg_stat_item or enum node_stat_item
+ * @val: delta to add to the counter, can be negative
+ */
+void mod_memcg_state(struct mem_cgroup *memcg, enum memcg_stat_item idx,
+ int val)
+{
+ if (mem_cgroup_disabled())
+ return;
+
+ __mod_memcg_state(memcg, idx, val);
+}
+
#ifdef CONFIG_MEMCG_V1
/* idx can be of type enum memcg_stat_item or node_stat_item. */
unsigned long memcg_page_state_local(struct mem_cgroup *memcg, int idx)
@@ -763,23 +857,27 @@ unsigned long memcg_page_state_local(struct mem_cgroup *memcg, int idx)
#endif
return x; }
+
+void reparent_memcg_state_local(struct mem_cgroup *memcg,
+ struct mem_cgroup *parent, int idx)
+{
+ unsigned long value = memcg_page_state_local(memcg, idx);
+
+ __mod_memcg_state(memcg, idx, -value);
+ __mod_memcg_state(parent, idx, value);
+}
#endif
-static void mod_memcg_lruvec_state(struct lruvec *lruvec,
- enum node_stat_item idx,
- int val)
+static void __mod_memcg_lruvec_state(struct mem_cgroup_per_node *pn,
+ enum node_stat_item idx, int val)
{
- struct mem_cgroup_per_node *pn;
- struct mem_cgroup *memcg;
+ struct mem_cgroup *memcg = pn->memcg;
int i = memcg_stats_index(idx);
int cpu;
if (WARN_ONCE(BAD_STAT_IDX(i), "%s: missing stat item %d\n",
__func__, idx))
return;- pn = container_of(lruvec, struct mem_cgroup_per_node, lruvec);
- memcg = pn->memcg;
-
cpu = get_cpu();
/* Update memcg */
@@ -795,6 +893,17 @@ static void mod_memcg_lruvec_state(struct lruvec *lruvec,
put_cpu();}
+static void mod_memcg_lruvec_state(struct lruvec *lruvec,
+ enum node_stat_item idx,
+ int val)
+{
+ struct mem_cgroup_per_node *pn;
+
+ pn = container_of(lruvec, struct mem_cgroup_per_node, lruvec);
+
+ __mod_memcg_lruvec_state(pn, idx, val);
+}
+
/**
* mod_lruvec_state - update lruvec memory statistics
* @lruvec: the lruvec
``` [PATCH v5 update 31/32]: ``` commit d5db0b0193c02231bb3d7938245cfd00d25a3bb4 Author: Muchun Song <[email protected]> Date: Tue Apr 15 10:45:31 2025 +0800
mm: memcontrol: eliminate the problem of dying memory cgroup for
LRU folios
Now that everything is set up, switch folio->memcg_data pointers to objcgs, update the accessors, and execute reparenting on cgroup death.
Finally, folio->memcg_data of LRU folios and kmem folios will always point to an object cgroup pointer. The folio->memcg_data of slab folios will point to an vector of object cgroups.
include/linux/memcontrol.h +25 -52
131 lines not on this page · raw message
mm/memcontrol-v1.c +6 -4
@@ -613,6 +613,7 @@ void memcg1_commit_charge(struct folio *folio, struct mem_cgroup *memcg)void memcg1_swapout (struct folio *folio,swp_entry_t entry) {struct mem_cgroup *memcg, *swap_memcg; +struct obj_cgroup *objcg;unsigned int nr_entries;VM_BUG_ON_FOLIO (folio_test_lru (folio), folio);
@@ -624,12 +625,13 @@ void memcg1_swapout(struct folio *folio, swp_entry_t entry)if (!do_memsw_account ())return ; - memcg =folio_memcg (folio); - -VM_WARN_ON_ONCE_FOLIO (!memcg, folio); -if (!memcg) + objcg =folio_objcg (folio); +VM_WARN_ON_ONCE_FOLIO (!objcg, folio); +if (!objcg)return ; +rcu_read_lock (); + memcg =obj_cgroup_memcg (objcg);/* * In case the memcg owning these pages has been offlined and
doesn't
* have an ID allocated to it anymore, charge the closest online
@@ -644,7 +646,7 @@ void memcg1_swapout(struct folio *folio, swp_entry_t entry)
folio_unqueue_deferred_split(folio);
folio->memcg_data = 0;- if (!mem_cgroup_is_root(memcg))
+ if (!obj_cgroup_is_root(objcg))
page_counter_uncharge(&memcg->memory, nr_entries);
if (memcg != swap_memcg) {
@@ -665,7 +667,8 @@ void memcg1_swapout(struct folio *folio, swp_entry_t entry)
preempt_enable_nested();
memcg1_check_events(memcg, folio_nid(folio));- css_put(&memcg->css); + rcu_read_unlock(); + obj_cgroup_put(objcg);
}
/*
mm/memcontrol.c +34 -4
@@ -255,13 +255,17 @@ static inline void reparent_state_local(struct mem_cgroup *memcg, struct mem_cgr }#endif -static inline void reparent_locks (struct mem_cgroup *memcg,struct mem_cgroup *parent) +static inline void reparent_locks (struct mem_cgroup *memcg,struct mem_cgroup *parent,int nid) {spin_lock_irq (&objcg_lock); +spin_lock_nested (&mem_cgroup_lruvec (memcg,NODE_DATA (nid))->lru_lock,1 ); +spin_lock_nested (&mem_cgroup_lruvec (parent,NODE_DATA (nid))->lru_lock,2 ); } -static inline void reparent_unlocks (struct mem_cgroup *memcg,struct mem_cgroup *parent) +static inline void reparent_unlocks (struct mem_cgroup *memcg,struct mem_cgroup *parent,int nid) { +spin_unlock (&mem_cgroup_lruvec (parent,NODE_DATA (nid))->lru_lock); +spin_unlock (&mem_cgroup_lruvec (memcg,NODE_DATA (nid))->lru_lock);spin_unlock_irq (&objcg_lock); }
@@ -272,14 +276,31 @@ static void memcg_reparent_objcgs(struct mem_cgroup *memcg)int nid;for_each_node (nid) { -reparent_locks (memcg, parent); +retry: +if (lru_gen_enabled ()) +max_lru_gen_memcg (parent, nid); + +reparent_locks (memcg, parent, nid); + +if (lru_gen_enabled ()) { +if (!recheck_lru_gen_max_memcg (parent, nid)) { +reparent_unlocks (memcg, parent, nid); +cond_resched (); + goto retry; + } +lru_gen_reparent_memcg (memcg, parent, nid); + }else { +lru_reparent_memcg (memcg, parent, nid); + } objcg =__memcg_reparent_objcgs (memcg, parent, nid); -reparent_unlocks (memcg, parent); +reparent_unlocks (memcg, parent, nid);percpu_ref_kill (&objcg->refcnt); } + +reparent_state_local (memcg, parent); }/*
@@ -824,6 +845,7 @@ static void __mod_memcg_state(struct mem_cgroup *memcg,this_cpu_add (memcg->vmstats_percpu->state[i], val); val =memcg_state_val_in_pages (idx, val);memcg_rstat_updated (memcg, val, cpu); +trace_mod_memcg_state (memcg, idx, val);put_cpu ();
@@ -841,7 +863,9 @@ void mod_memcg_state(struct mem_cgroup *memcg, enum memcg_stat_item idx,if (mem_cgroup_disabled ())return ; + memcg =get_non_dying_memcg_start (memcg);__mod_memcg_state (memcg, idx, val); +get_non_dying_memcg_end (); }#ifdef CONFIG_MEMCG_V1
@@ -901,11 +925,17 @@ static void mod_memcg_lruvec_state(struct lruvec *lruvec, enum node_stat_item idx, int val) { +struct pglist_data *pgdat =lruvec_pgdat (lruvec);struct mem_cgroup_per_node *pn; +struct mem_cgroup *memcg; pn = container_of(lruvec, struct mem_cgroup_per_node, lruvec); + memcg =get_non_dying_memcg_start (pn->memcg); + pn = memcg->nodeinfo[pgdat->node_id];__mod_memcg_lruvec_state (pn, idx, val); + +get_non_dying_memcg_end (); }/**
@@ -1128,6 +1158,8 @@ struct mem_cgroup *get_mem_cgroup_from_current(void)/** * get_mem_cgroup_from_folio - Obtain a reference on a given folio's
memcg.
* @folio: folio from which memcg should be extracted.
+ *
+ * See folio_memcg() for folio->objcg/memcg binding rules.
*/
struct mem_cgroup *get_mem_cgroup_from_folio(struct folio *folio)
{
@@ -2769,17 +2801,17 @@ static inline int try_charge(struct mem_cgroup *memcg, gfp_t gfp_mask,
return try_charge_memcg(memcg, gfp_mask, nr_pages);
}
-static void commit_charge(struct folio *folio, struct mem_cgroup *memcg) +static void commit_charge(struct folio *folio, struct obj_cgroup *objcg)
{
VM_BUG_ON_FOLIO(folio_memcg_charged(folio), folio);
/*- * Any of the following ensures page's memcg stability:
+ * Any of the following ensures folio's objcg stability:
*
* - the page lock
* - LRU isolation
* - exclusive reference
*/
- folio->memcg_data = (unsigned long)memcg;
+ folio->memcg_data = (unsigned long)objcg;
}
#ifdef CONFIG_MEMCG_NMI_SAFETY_REQUIRES_ATOMIC
@@ -2893,6 +2925,17 @@ static struct obj_cgroup *__get_obj_cgroup_from_memcg(struct mem_cgroup *memcg)
return NULL;
}
+static inline struct obj_cgroup *get_obj_cgroup_from_memcg(struct mem_cgroup *memcg) +{
+ struct obj_cgroup *objcg;
+
+ rcu_read_lock(); + objcg = __get_obj_cgroup_from_memcg(memcg); + rcu_read_unlock();
+
+ return objcg;
+}
+
static struct obj_cgroup *current_objcg_update(void)
{
struct mem_cgroup *memcg;
@@ -2994,17 +3037,10 @@ struct obj_cgroup *get_obj_cgroup_from_folio(struct folio *folio)
{
struct obj_cgroup *objcg;- if (!memcg_kmem_online())
- return NULL;
-
- if (folio_memcg_kmem(folio)) {
- objcg = __folio_objcg(folio);
+ objcg = folio_objcg(folio);
+ if (objcg)
obj_cgroup_get(objcg);
- } else {
- rcu_read_lock();
- objcg = __get_obj_cgroup_from_memcg(__folio_memcg(folio));
- rcu_read_unlock();
- }
+
return objcg;}
@@ -3520,7 +3556,7 @@ void folio_split_memcg_refs(struct folio *folio, unsigned old_order,
return;
new_refs = (1 << (old_order - new_order)) - 1;- css_get_many(&__folio_memcg(folio)->css, new_refs);
+ obj_cgroup_get_many(folio_objcg(folio), new_refs);
}
static void memcg_online_kmem(struct mem_cgroup *memcg)
@@ -4949,16 +4985,20 @@ void mem_cgroup_calculate_protection(struct mem_cgroup *root,
static int charge_memcg(struct folio *folio, struct mem_cgroup *memcg,
gfp_t gfp)
{
- int ret;
-
- ret = try_charge(memcg, gfp, folio_nr_pages(folio));
- if (ret)
- goto out;
+ int ret = 0;
+ struct obj_cgroup *objcg;
- css_get(&memcg->css); - commit_charge(folio, memcg); + objcg = get_obj_cgroup_from_memcg(memcg); + /* Do not account at the root objcg level. */ + if (!obj_cgroup_is_root(objcg))
+ ret = try_charge_memcg(memcg, gfp, folio_nr_pages(folio));
+ if (ret) {
+ obj_cgroup_put(objcg);
+ return ret;
+ }
+ commit_charge(folio, objcg);
memcg1_commit_charge(folio, memcg);
-out:
+
return ret;}
@@ -5044,7 +5084,7 @@ int mem_cgroup_swapin_charge_folio(struct folio *folio, struct mm_struct *mm,
}
struct uncharge_gather {
- struct mem_cgroup *memcg;
+ struct obj_cgroup *objcg;
unsigned long nr_memory;
unsigned long pgpgout;
unsigned long nr_kmem;@@ -5058,58 +5098,52 @@ static inline void uncharge_gather_clear(struct uncharge_gather *ug)
static void uncharge_batch(const struct uncharge_gather *ug) {
+ struct mem_cgroup *memcg;
+
+ rcu_read_lock();
+ memcg = obj_cgroup_memcg(ug->objcg);
if (ug->nr_memory) {
- memcg_uncharge(ug->memcg, ug->nr_memory);
+ memcg_uncharge(memcg, ug->nr_memory);
if (ug->nr_kmem) {
- mod_memcg_state(ug->memcg, MEMCG_KMEM, -ug->nr_kmem);
- memcg1_account_kmem(ug->memcg, -ug->nr_kmem);
+ mod_memcg_state(memcg, MEMCG_KMEM, -ug->nr_kmem);
+ memcg1_account_kmem(memcg, -ug->nr_kmem);
}
- memcg1_oom_recover(ug->memcg);
+ memcg1_oom_recover(memcg);
}- memcg1_uncharge_batch(ug->memcg, ug->pgpgout, ug->nr_memory, ug->nid); + memcg1_uncharge_batch(memcg, ug->pgpgout, ug->nr_memory, ug->nid); + rcu_read_unlock();
/* drop reference from uncharge_folio */
- css_put(&ug->memcg->css);
+ obj_cgroup_put(ug->objcg);
}
static void uncharge_folio(struct folio *folio, struct uncharge_gather
*ug)
{
long nr_pages;
- struct mem_cgroup *memcg;
struct obj_cgroup *objcg;
VM_BUG_ON_FOLIO(folio_test_lru(folio), folio);
/*
* Nobody should be changing or seriously looking at
- * folio memcg or objcg at this point, we have fully
- * exclusive access to the folio.
+ * folio objcg at this point, we have fully exclusive
+ * access to the folio.
*/
- if (folio_memcg_kmem(folio)) {
- objcg = __folio_objcg(folio);
- /*
- * This get matches the put at the end of the function and
- * kmem pages do not hold memcg references anymore.
- */- memcg = get_mem_cgroup_from_objcg(objcg);
- } else {
- memcg = __folio_memcg(folio);
- }
-
- if (!memcg)
+ objcg = folio_objcg(folio);
+ if (!objcg)
return;- if (ug->memcg != memcg) {
- if (ug->memcg) {
+ if (ug->objcg != objcg) {
+ if (ug->objcg) {
uncharge_batch(ug);
uncharge_gather_clear(ug);
}
- ug->memcg = memcg;
+ ug->objcg = objcg;
ug->nid = folio_nid(folio);- /* pairs with css_put in uncharge_batch */
- css_get(&memcg->css);
+ /* pairs with obj_cgroup_put in uncharge_batch */
+ obj_cgroup_get(objcg);
}
nr_pages = folio_nr_pages(folio);
@@ -5117,20 +5151,17 @@ static void uncharge_folio(struct folio *folio, struct uncharge_gather *ug)
if (folio_memcg_kmem(folio)) {
ug->nr_memory += nr_pages;
ug->nr_kmem += nr_pages;
-
- folio->memcg_data = 0;
- obj_cgroup_put(objcg);
} else {
/* LRU pages aren't accounted at the root level */
- if (!mem_cgroup_is_root(memcg))
+ if (!obj_cgroup_is_root(objcg))
ug->nr_memory += nr_pages;
ug->pgpgout++;
WARN_ON_ONCE(folio_unqueue_deferred_split(folio));
- folio->memcg_data = 0;
}- css_put(&memcg->css); + folio->memcg_data = 0; + obj_cgroup_put(objcg);
}
void __mem_cgroup_uncharge(struct folio *folio)
@@ -5154,7 +5185,7 @@ void __mem_cgroup_uncharge_folios(struct folio_batch *folios)
uncharge_gather_clear(&ug);
for (i = 0; i < folios->nr; i++)
uncharge_folio(folios->folios[i], &ug);
- if (ug.memcg)
+ if (ug.objcg)
uncharge_batch(&ug);}
@@ -5171,6 +5202,7 @@ void __mem_cgroup_uncharge_folios(struct folio_batch *folios)
void mem_cgroup_replace_folio(struct folio *old, struct folio *new)
{
struct mem_cgroup *memcg;
+ struct obj_cgroup *objcg;
long nr_pages = folio_nr_pages(new);
VM_BUG_ON_FOLIO(!folio_test_locked(old), old);
@@ -5185,21 +5217,24 @@ void mem_cgroup_replace_folio(struct folio *old, struct folio *new)
if (folio_memcg_charged(new))
return;- memcg = folio_memcg(old);
- VM_WARN_ON_ONCE_FOLIO(!memcg, old);
- if (!memcg)
+ objcg = folio_objcg(old);
+ VM_WARN_ON_ONCE_FOLIO(!objcg, old);
+ if (!objcg)
return;+ rcu_read_lock();
+ memcg = obj_cgroup_memcg(objcg);
/* Force-charge the new page. The old one will be freed soon */
- if (!mem_cgroup_is_root(memcg)) {
+ if (!obj_cgroup_is_root(objcg)) {
page_counter_charge(&memcg->memory, nr_pages);
if (do_memsw_account())
page_counter_charge(&memcg->memsw, nr_pages);
}- css_get(&memcg->css);
- commit_charge(new, memcg);
+ obj_cgroup_get(objcg);
+ commit_charge(new, objcg);
memcg1_commit_charge(new, memcg);+ rcu_read_unlock();
}
/**
@@ -5215,7 +5250,7 @@ void mem_cgroup_replace_folio(struct folio *old, struct folio *new)
*/
void mem_cgroup_migrate(struct folio *old, struct folio *new)
{
- struct mem_cgroup *memcg;
+ struct obj_cgroup *objcg;
VM_BUG_ON_FOLIO(!folio_test_locked(old), old);
VM_BUG_ON_FOLIO(!folio_test_locked(new), new);
@@ -5226,18 +5261,18 @@ void mem_cgroup_migrate(struct folio *old, struct folio *new)
if (mem_cgroup_disabled())
return;- memcg = folio_memcg(old);
+ objcg = folio_objcg(old);
/*- * Note that it is normal to see !memcg for a hugetlb folio.
+ * Note that it is normal to see !objcg for a hugetlb folio.
* For e.g, it could have been allocated whenmemory_hugetlb_accounting
* was not selected.
*/
- VM_WARN_ON_ONCE_FOLIO(!folio_test_hugetlb(old) && !memcg, old);
- if (!memcg)
+ VM_WARN_ON_ONCE_FOLIO(!folio_test_hugetlb(old) && !objcg, old);
+ if (!objcg)
return;- /* Transfer the charge and the css ref */ - commit_charge(new, memcg); + /* Transfer the charge and the objcg ref */ + commit_charge(new, objcg);
/* Warning should never happen, so don't worry about refcount
non-0 */
WARN_ON_ONCE(folio_unqueue_deferred_split(old));
@@ -5420,22 +5455,27 @@ int __mem_cgroup_try_charge_swap(struct folio *folio, swp_entry_t entry)
unsigned int nr_pages = folio_nr_pages(folio);
struct page_counter *counter;
struct mem_cgroup *memcg;+ struct obj_cgroup *objcg;
if (do_memsw_account())
return 0;- memcg = folio_memcg(folio);
-
- VM_WARN_ON_ONCE_FOLIO(!memcg, folio);
- if (!memcg)
+ objcg = folio_objcg(folio);
+ VM_WARN_ON_ONCE_FOLIO(!objcg, folio);
+ if (!objcg)
return 0;+ rcu_read_lock();
+ memcg = obj_cgroup_memcg(objcg);
if (!entry.val) {
memcg_memory_event(memcg, MEMCG_SWAP_FAIL);
+ rcu_read_unlock();
return 0;
}
memcg = mem_cgroup_private_id_get_online(memcg, nr_pages);+ /* memcg is pined by memcg ID. */
+ rcu_read_unlock();
if (!mem_cgroup_is_root(memcg) &&
!page_counter_try_charge(&memcg->swap, nr_pages, &counter)) {```
Yosry Ahmed
maintainerRe: [PATCH v5 update 29/32] mm: memcontrol: prepare for reparenting non-hierarchical stats
Quoted 25 lines · Qi Zheng
Quoted 22 lines · Yosry Ahmed
Quoted 1 line · Qi Zheng
Hi Yosry,
[..]
Quoted 11 lines
Quoted 7 lines
I don't think we should end up with two copies of __mod_memcg_state/mod_memcg_state() and __mod_memcg_lruvec_state/mod_memcg_lruvec_state(). I meant to refactor
mod_memcg_state()to call__mod_memcg_state(), where the latter does not call get_non_dying_memcg_{start/end}(). Same formod_memcg_lruvec_state().Okay, like the following? But this would require modifications to [PATCH v5 31/32]. If there are no problems, I will send the updated patch to [PATCH v5 29/32] and [PATCH v5 31/32].
I cannot apply the diff, seems a bit corrupted.
But ideally, instead of a @reparent argument, we just have
__mod_memcg_lruvec_state()and__mod_memcg_state()do the work without getting parent of dead memcgs, and thenmod_memcg_lruvec_state()andmod_memcg_state()just call them afterget_non_dying_memcg_start().
Quoted 7 lines
Quoted 4 lines
What about this (untested), it should apply on top of 'mm: memcontrol:
eliminate the problem of dying memory cgroup for LRU folios' in mm-new, so maybe it needs to be broken down across different patches:I applied and tested it, so the final updated patches is as follows, If there are no problems, I will send out the official patches.
If I am not mistaken, Andrew prefers fixups to what he already has in mm-new (Andrew, please correct me if I am wrong).
Andrew Morton
maintainerRe: [PATCH v5 update 29/32] mm: memcontrol: prepare for reparenting non-hierarchical stats
Quoted 10 lines · On Wed, 4 Mar 2026 13:57:41 +0000 Yosry Ahmed <[email protected]> wrote:
Quoted 7 lines
Quoted 4 lines
What about this (untested), it should apply on top of 'mm: memcontrol:
eliminate the problem of dying memory cgroup for LRU folios' in mm-new, so maybe it needs to be broken down across different patches:I applied and tested it, so the final updated patches is as follows, If there are no problems, I will send out the official patches.
If I am not mistaken, Andrew prefers fixups to what he already has in mm-new (Andrew, please correct me if I am wrong).
Yes, if the changes are reasonably small and the code has already undergone significant review.
Although the mm-new branch is quite speculative/early so I guess this is less important there.
Adding a sprinkle of -fix patches can be a pain all round, so nowadays if someone sends a replacement series I'll generate and send a what-you-changed-since-last-time diff. So
- we can check that the diff matches the changelogged updates
- reviewers don't have to re-review everything
- the author can eyeball it and think "yup, I meant to change that".
I believe this series is due for quite a few updates so a full v6 resend series would be appropriate. I'll generate the how-you-changed-mm.git email from that.
Yosry Ahmed
maintainerRe: [PATCH v5 update 29/32] mm: memcontrol: prepare for reparenting non-hierarchical stats
Quoted 24 lines · Andrew Morton
Quoted 10 lines · On Wed, 4 Mar 2026 13:57:41 +0000 Yosry Ahmed <[email protected]> wrote:
Quoted 7 lines
Quoted 4 lines
What about this (untested), it should apply on top of 'mm: memcontrol:
eliminate the problem of dying memory cgroup for LRU folios' in mm-new, so maybe it needs to be broken down across different patches:I applied and tested it, so the final updated patches is as follows, If there are no problems, I will send out the official patches.
If I am not mistaken, Andrew prefers fixups to what he already has in mm-new (Andrew, please correct me if I am wrong).
Yes, if the changes are reasonably small and the code has already undergone significant review.
Although the mm-new branch is quite speculative/early so I guess this is less important there.
Adding a sprinkle of -fix patches can be a pain all round, so nowadays if someone sends a replacement series I'll generate and send a what-you-changed-since-last-time diff. So
Quoted 7 lines
- we can check that the diff matches the changelogged updates
- reviewers don't have to re-review everything
- the author can eyeball it and think "yup, I meant to change that".I believe this series is due for quite a few updates so a full v6 resend series would be appropriate. I'll generate the how-you-changed-mm.git email from that.
Thanks for chiming in. Qi, if you send a new version, I think separating refactoring (and moving, if needed) mod_memcg_state() and mod_memcg_lruvec_state() into a separate patch will make things easier to review.
Qi Zheng
reviewerRe: [PATCH v5 update 29/32] mm: memcontrol: prepare for reparenting non-hierarchical stats
Quoted 29 lines · Yosry Ahmed
Quoted 28 lines · Andrew Morton
Quoted 10 lines · On Wed, 4 Mar 2026 13:57:41 +0000 Yosry Ahmed <[email protected]> wrote:
Quoted 7 lines
Quoted 4 lines
What about this (untested), it should apply on top of 'mm: memcontrol:
eliminate the problem of dying memory cgroup for LRU folios' in mm-new, so maybe it needs to be broken down across different patches:I applied and tested it, so the final updated patches is as follows, If there are no problems, I will send out the official patches.
If I am not mistaken, Andrew prefers fixups to what he already has in mm-new (Andrew, please correct me if I am wrong).
Yes, if the changes are reasonably small and the code has already undergone significant review.
Although the mm-new branch is quite speculative/early so I guess this is less important there.
Adding a sprinkle of -fix patches can be a pain all round, so nowadays if someone sends a replacement series I'll generate and send a what-you-changed-since-last-time diff. So
- we can check that the diff matches the changelogged updates
- reviewers don't have to re-review everything
- the author can eyeball it and think "yup, I meant to change that".
Quoted 8 lines
Quoted 3 lines
I believe this series is due for quite a few updates so a full v6 resend series would be appropriate. I'll generate the how-you-changed-mm.git email from that.
Thanks for chiming in. Qi, if you send a new version, I think separating refactoring (and moving, if needed)
mod_memcg_state()andmod_memcg_lruvec_state()into a separate patch will make things easier to review.
OK, will do.
Andrew Morton
maintainerRe: [PATCH v5 update 29/32] mm: memcontrol: prepare for reparenting non-hierarchical stats
In reply to Qi Zhengkernel/cgroup/cgroup.c int cgroup_mkdir(struct kernfs_node *parent_kn, · line 6048
Quoted 10 lines · On Wed, 4 Mar 2026 15:56:42 +0800 Qi Zheng <[email protected]> wrote:
kernel/cgroup/cgroup.c +0
-2@@ -6044,8 +6044,8 @@ int cgroup_mkdir(struct kernfs_node *parent_kn,const char *name, umode_t mode) */ static void css_killed_work_fn (struct work_struct *work) { -struct cgroup_subsys_state *css = - container_of(work, struct cgroup_subsys_state, destroy_work);
Lots of wordwrapping in this patch.
Qi Zheng
reviewerRe: [PATCH v5 update 29/32] mm: memcontrol: prepare for reparenting non-hierarchical stats
Quoted 14 lines · Andrew Morton
kernel/cgroup/cgroup.c int cgroup_mkdir(struct kernfs_node *parent_kn, · line 6048
Quoted 10 lines · On Wed, 4 Mar 2026 15:56:42 +0800 Qi Zheng <[email protected]> wrote:
kernel/cgroup/cgroup.c +0
-2@@ -6044,8 +6044,8 @@ int cgroup_mkdir(struct kernfs_node *parent_kn,const char *name, umode_t mode) */ static void css_killed_work_fn (struct work_struct *work) { -struct cgroup_subsys_state *css = - container_of(work, struct cgroup_subsys_state, destroy_work);Lots of wordwrapping in this patch.
Oh, I will check this more carefully in v6.
Thanks,
Qi
Qi Zheng
reviewerPATCH v5 30/32 mm: memcontrol: convert objcg to be per-memcg per-node type
In reply to Qi ZhengFrom: Qi Zheng <[email protected]>
Convert objcg to be per-memcg per-node type, so that when reparent LRU folios later, we can hold the lru lock at the node level, thus avoiding holding too many lru locks at once.
--- include/linux/memcontrol.h | 22 ++++++----- include/linux/sched.h | 2 +- mm/memcontrol.c | 75 ++++++++++++++++++++++++-------------- 3 files changed, 61 insertions(+), 38 deletions(-)
include/linux/memcontrol.h +12 -10
@@ -116,6 +116,16 @@ struct mem_cgroup_per_node {unsigned long lru_zone_size[MAX_NR_ZONES ][NR_LRU_LISTS ];struct mem_cgroup_reclaim_iter iter; +/* +* objcg is wiped out as a part of the objcg repaprenting process. +* orig_objcg preserves a pointer (and a reference) to the original +* objcg until the end of live of memcg. +*/ +struct obj_cgroup __rcu *objcg; +struct obj_cgroup *orig_objcg; +/* list of inherited objcgs, protected by objcg_lock */ +struct list_head objcg_list; +#ifdef CONFIG_MEMCG_NMI_SAFETY_REQUIRES_ATOMIC /* slab stats for nmi context */ atomic_t slab_reclaimable;
@@ -180,6 +190,7 @@ struct obj_cgroup {struct list_head list;/* protected by objcg_lock */ struct rcu_head rcu; }; +bool is_root; };/*
@@ -258,15 +269,6 @@ struct mem_cgroup {seqlock_t socket_pressure_seqlock;#endif int kmemcg_id; -/* -* memcg->objcg is wiped out as a part of the objcg repaprenting -* process. memcg->orig_objcg preserves a pointer (and a reference) -* to the original objcg until the end of live of memcg. -*/ -struct obj_cgroup __rcu *objcg; -struct obj_cgroup *orig_objcg; -/* list of inherited objcgs, protected by objcg_lock */ -struct list_head objcg_list;struct memcg_vmstats_percpu __percpu *vmstats_percpu;
@@ -552,7 +554,7 @@ static inline bool mem_cgroup_is_root(struct mem_cgroup *memcg)static inline bool obj_cgroup_is_root (const struct obj_cgroup *objcg) { -return objcg == root_obj_cgroup; +return objcg->is_root; }static inline bool mem_cgroup_disabled (void )
include/linux/sched.h +1 -1
@@ -1533,7 +1533,7 @@ struct task_struct {/* Used by memcontrol for targeted memcg charge: */ struct mem_cgroup *active_memcg; -/* Cache for current->cgroups->memcg->objcg lookups: */ +/* Cache for current->cgroups->memcg->nodeinfo[nid]->objcg lookups: */ struct obj_cgroup *objcg;#endif
mm/memcontrol.c +48 -27
166 lines not on this page · raw message
Signature · 1 line
2.20.1
Qi Zheng
reviewerPATCH v5 30/32 mm: memcontrol: convert objcg to be per-memcg per-node type
From: Qi Zheng <[email protected]>
Convert objcg to be per-memcg per-node type, so that when reparent LRU folios later, we can hold the lru lock at the node level, thus avoiding holding too many lru locks at once.
---
changlog:
- fix a missing root_obj_cgroup conversion and completely delete root_obj_cgroup.
include/linux/memcontrol.h | 23 +++++------ include/linux/sched.h | 2 +- mm/memcontrol.c | 79 +++++++++++++++++++++++--------------- 3 files changed, 62 insertions(+), 42 deletions(-)
include/linux/memcontrol.h +12 -11
@@ -116,6 +116,16 @@ struct mem_cgroup_per_node {unsigned long lru_zone_size[MAX_NR_ZONES ][NR_LRU_LISTS ];struct mem_cgroup_reclaim_iter iter; +/* +* objcg is wiped out as a part of the objcg repaprenting process. +* orig_objcg preserves a pointer (and a reference) to the original +* objcg until the end of live of memcg. +*/ +struct obj_cgroup __rcu *objcg; +struct obj_cgroup *orig_objcg; +/* list of inherited objcgs, protected by objcg_lock */ +struct list_head objcg_list; +#ifdef CONFIG_MEMCG_NMI_SAFETY_REQUIRES_ATOMIC /* slab stats for nmi context */ atomic_t slab_reclaimable;
@@ -180,6 +190,7 @@ struct obj_cgroup {struct list_head list;/* protected by objcg_lock */ struct rcu_head rcu; }; +bool is_root; };/*
@@ -258,15 +269,6 @@ struct mem_cgroup {seqlock_t socket_pressure_seqlock;#endif int kmemcg_id; -/* -* memcg->objcg is wiped out as a part of the objcg repaprenting -* process. memcg->orig_objcg preserves a pointer (and a reference) -* to the original objcg until the end of live of memcg. -*/ -struct obj_cgroup __rcu *objcg; -struct obj_cgroup *orig_objcg; -/* list of inherited objcgs, protected by objcg_lock */ -struct list_head objcg_list;struct memcg_vmstats_percpu __percpu *vmstats_percpu;
@@ -333,7 +335,6 @@ struct mem_cgroup {#define MEMCG_CHARGE_BATCH 64Uextern struct mem_cgroup *root_mem_cgroup; -extern struct obj_cgroup *root_obj_cgroup;enum page_memcg_data_flags {/* page->memcg_data is a pointer to an slabobj_ext vector */
@@ -552,7 +553,7 @@ static inline bool mem_cgroup_is_root(struct mem_cgroup *memcg)static inline bool obj_cgroup_is_root (const struct obj_cgroup *objcg) { -return objcg == root_obj_cgroup; +return objcg->is_root; }static inline bool mem_cgroup_disabled (void )
include/linux/sched.h +1 -1
@@ -1533,7 +1533,7 @@ struct task_struct {/* Used by memcontrol for targeted memcg charge: */ struct mem_cgroup *active_memcg; -/* Cache for current->cgroups->memcg->objcg lookups: */ +/* Cache for current->cgroups->memcg->nodeinfo[nid]->objcg lookups: */ struct obj_cgroup *objcg;#endif
mm/memcontrol.c +49 -30
182 lines not on this page · raw message
Signature · 1 line
2.20.1
Re: [PATCH v5 update 30/32] mm: memcontrol: convert objcg to be per-memcg per-node type
Quoted 22 lines · Qi Zheng
From: Qi Zheng <[email protected]>
Convert objcg to be per-memcg per-node type, so that when reparent LRU folios later, we can hold the lru lock at the node level, thus avoiding holding too many lru locks at once.
Signed-off-by Qi Zheng <[email protected]>---
changlog:
- fix a missingroot_obj_cgroupconversion and completely deleteroot_obj_cgroup.include/linux/memcontrol.h | 23 +++++------ include/linux/sched.h | 2 +- mm/memcontrol.c | 79 +++++++++++++++++++++++--------------- 3 files changed, 62 insertions(+), 42 deletions(-)include/linux/memcontrol.h +0
-0
include/linux/memcontrol.h struct mem_cgroup_per_node { · line 125
Quoted 10 lines
include/linux/memcontrol.h +7
-0@@ -116,6 +116,16 @@ struct mem_cgroup_per_node {unsigned long lru_zone_size[MAX_NR_ZONES ][NR_LRU_LISTS ];struct mem_cgroup_reclaim_iter iter; +/* +* objcg is wiped out as a part of the objcg repaprenting process. +* orig_objcg preserves a pointer (and a reference) to the original +* objcg until the end of live of memcg. +*/ +struct obj_cgroup __rcu *objcg; +struct obj_cgroup *orig_objcg;
The layout of struct mem_cgroup_per_node is very performance sensitive. Please couple of performance benchmarks after rearranging the fields particularly the above two pointers together at the start of the struct.
Otherwise:
Qi Zheng
reviewerRe: [PATCH v5 update 30/32] mm: memcontrol: convert objcg to be per-memcg per-node type
Quoted 26 lines · Shakeel Butt
Quoted 25 lines · Qi Zheng
From: Qi Zheng <[email protected]>
Convert objcg to be per-memcg per-node type, so that when reparent LRU folios later, we can hold the lru lock at the node level, thus avoiding holding too many lru locks at once.
Signed-off-by Qi Zheng <[email protected]>---
changlog:
- fix a missingroot_obj_cgroupconversion and completely deleteroot_obj_cgroup.include/linux/memcontrol.h | 23 +++++------ include/linux/sched.h | 2 +- mm/memcontrol.c | 79 +++++++++++++++++++++++--------------- 3 files changed, 62 insertions(+), 42 deletions(-)include/linux/memcontrol.h +0
-0@@ -116,6 +116,16 @@ struct mem_cgroup_per_node {unsigned long lru_zone_size[MAX_NR_ZONES ][NR_LRU_LISTS ];struct mem_cgroup_reclaim_iter iter;
Quoted 11 lines
include/linux/memcontrol.h struct mem_cgroup_per_node { · line 125
Quoted 7 lines
include/linux/memcontrol.h +7
-0+/* +* objcg is wiped out as a part of the objcg repaprenting process. +* orig_objcg preserves a pointer (and a reference) to the original +* objcg until the end of live of memcg. +*/ +struct obj_cgroup __rcu *objcg; +struct obj_cgroup *orig_objcg;The layout of struct
mem_cgroup_per_nodeis very performance sensitive. Please couple of performance benchmarks after rearranging the fields particularly the above two pointers together at the start of the struct.
Got it, will try to do it. Additionally, it would be even better if there were existing test cases available. ;)
Quoted 4 lines
Otherwise:
Acked-by Shakeel Butt <[email protected]>
Thanks!
Qi Zheng
reviewerRe: [PATCH v5 update 30/32] mm: memcontrol: convert objcg to be per-memcg per-node type
In reply to Shakeel ButtHi Shakeel,
Quoted 26 lines · Shakeel Butt
Quoted 25 lines · Qi Zheng
From: Qi Zheng <[email protected]>
Convert objcg to be per-memcg per-node type, so that when reparent LRU folios later, we can hold the lru lock at the node level, thus avoiding holding too many lru locks at once.
Signed-off-by Qi Zheng <[email protected]>---
changlog:
- fix a missingroot_obj_cgroupconversion and completely deleteroot_obj_cgroup.include/linux/memcontrol.h | 23 +++++------ include/linux/sched.h | 2 +- mm/memcontrol.c | 79 +++++++++++++++++++++++--------------- 3 files changed, 62 insertions(+), 42 deletions(-)include/linux/memcontrol.h +0
-0@@ -116,6 +116,16 @@ struct mem_cgroup_per_node {unsigned long lru_zone_size[MAX_NR_ZONES ][NR_LRU_LISTS ];struct mem_cgroup_reclaim_iter iter;
Quoted 10 lines
include/linux/memcontrol.h struct mem_cgroup_per_node { · line 125
Quoted 7 lines
include/linux/memcontrol.h +7
-0+/* +* objcg is wiped out as a part of the objcg repaprenting process. +* orig_objcg preserves a pointer (and a reference) to the original +* objcg until the end of live of memcg. +*/ +struct obj_cgroup __rcu *objcg; +struct obj_cgroup *orig_objcg;The layout of struct
mem_cgroup_per_nodeis very performance sensitive. Please couple of performance benchmarks after rearranging the fields particularly the
I ran a simple test using will-it-scale page_fault1, and the results are as follows:
tasks | baseline | with-patch | diff | %
-------------------------------------------------------------
1 | 216394 | 217537 | +1143 | +0.53% 2 | 419895 | 421510 | +1615 | +0.38% 3 | 626275 | 629533 | +3258 | +0.52% 4 | 833146 | 830481 | -2665 | -0.32% 5 | 1037597 | 1034334 | -3263 | -0.31% 6 | 1229398 | 1231656 | +2258 | +0.18% 7 | 1425040 | 1424432 | -608 | -0.04% 8 | 1603738 | 1626028 | +22290 | +1.39% 9 | 1797881 | 1803901 | +6020 | +0.33% 10 | 1976264 | 1966342 | -9922 | -0.50% 11 | 2150073 | 2159328 | +9255 | +0.43% 12 | 2304435 | 2321128 | +16693 | +0.72% 13 | 2524668 | 2538582 | +13914 | +0.55% 14 | 2763854 | 2743098 | -20756 | -0.75% 15 | 2938939 | 2961671 | +22732 | +0.77% 16 | 3152497 | 3181420 | +28923 | +0.92% 17 | 3333936 | 3370667 | +36731 | +1.10% 18 | 3531225 | 3552907 | +21682 | +0.61% 19 | 3732347 | 3769889 | +37542 | +1.01% 20 | 3917502 | 3951226 | +33724 | +0.86% 21 | 4118040 | 4134952 | +16912 | +0.41% 22 | 4268932 | 4312343 | +43411 | +1.02% 23 | 4438411 | 4484408 | +45997 | +1.04% 24 | 4618400 | 4683531 | +65131 | +1.41%
No obvious degradation observed, so I'll keep this arrangement in v6.
Quoted 5 lines
above two pointers together at the start of the struct.
Otherwise:
Acked-by Shakeel Butt <[email protected]>
Thanks! I'll add this tag in v6.
Shakeel Butt
maintainerRe: [PATCH v5 update 30/32] mm: memcontrol: convert objcg to be per-memcg per-node type
In reply to Qi ZhengQuoted 6 lines · Qi Zheng
From: Qi Zheng <[email protected]>
Convert objcg to be per-memcg per-node type, so that when reparent LRU folios later, we can hold the lru lock at the node level, thus avoiding holding too many lru locks at once.
Quoted 6 lines
Signed-off-by Qi Zheng <[email protected]>---
changlog:
- fix a missingroot_obj_cgroupconversion and completely deleteroot_obj_cgroup.
After this patch, do we care that page/folio/slab points to the objcg of the same node as them for a given memcg?
Qi Zheng
reviewerRe: [PATCH v5 update 30/32] mm: memcontrol: convert objcg to be per-memcg per-node type
Quoted 14 lines · Shakeel Butt
Quoted 6 lines · Qi Zheng
From: Qi Zheng <[email protected]>
Convert objcg to be per-memcg per-node type, so that when reparent LRU folios later, we can hold the lru lock at the node level, thus avoiding holding too many lru locks at once.
Quoted 6 lines
Signed-off-by Qi Zheng <[email protected]>---
changlog:
- fix a missingroot_obj_cgroupconversion and completely deleteroot_obj_cgroup.
Quoted 2 lines
After this patch, do we care that page/folio/slab points to the objcg of the same node as them for a given memcg?
Maybe not. My only concern is whether the kernel has a way of determining whether two folios belong to the same memcg by checking if the objcg pointers are equal. If so, it needs to be changed to check if objcg->memcg are equal.
Quoted 1 line
Qi Zheng
reviewerPATCH v5 31/32 mm: memcontrol: eliminate the problem of dying memory cgroup for LRU folios
In reply to Qi ZhengFrom: Muchun Song <[email protected]>
Now that everything is set up, switch folio->memcg_data pointers to objcgs, update the accessors, and execute reparenting on cgroup death.
Finally, folio->memcg_data of LRU folios and kmem folios will always point to an object cgroup pointer. The folio->memcg_data of slab folios will point to an vector of object cgroups.
--- include/linux/memcontrol.h | 77 +++++--------- mm/memcontrol-v1.c | 15 +-- mm/memcontrol.c | 199 ++++++++++++++++++++++--------------- 3 files changed, 154 insertions(+), 137 deletions(-)
include/linux/memcontrol.h +25 -52
@@ -371,9 +371,6 @@ enum objext_flags {#define OBJEXTS_FLAGS_MASK (__NR_OBJEXTS_FLAGS - 1)#ifdef CONFIG_MEMCG - -static inline bool folio_memcg_kmem (struct folio *folio); -/* * After the initialization objcg->memcg is always pointing at * a valid memcg, but can be atomically swapped to the parent memcg.
@@ -387,43 +384,19 @@ static inline struct mem_cgroup *obj_cgroup_memcg(struct obj_cgroup *objcg) }/* -* __folio_memcg - Get the memory cgroup associated with a non-kmem folio -* @folio: Pointer to the folio. -* -* Returns a pointer to the memory cgroup associated with the folio, -* or NULL. This function assumes that the folio is known to have a -* proper memory cgroup pointer. It's not safe to call this function -* against some type of folios, e.g. slab folios or ex-slab folios or -* kmem folios. -*/ -static inline struct mem_cgroup *__folio_memcg (struct folio *folio) -{ -unsigned long memcg_data = folio->memcg_data; - -VM_BUG_ON_FOLIO (folio_test_slab (folio), folio); -VM_BUG_ON_FOLIO (memcg_data &MEMCG_DATA_OBJEXTS , folio); -VM_BUG_ON_FOLIO (memcg_data &MEMCG_DATA_KMEM , folio); - -return (struct mem_cgroup *)(memcg_data & ~OBJEXTS_FLAGS_MASK ); -} - -/* -* __folio_objcg - get the object cgroup associated with a kmem folio. +* folio_objcg - get the object cgroup associated with a folio. * @folio: Pointer to the folio. * * Returns a pointer to the object cgroup associated with the folio, * or NULL. This function assumes that the folio is known to have a -* proper object cgroup pointer. It's not safe to call this function -* against some type of folios, e.g. slab folios or ex-slab folios or -* LRU folios. +* proper object cgroup pointer. */ -static inline struct obj_cgroup *__folio_objcg (struct folio *folio) +static inline struct obj_cgroup *folio_objcg (struct folio *folio) {unsigned long memcg_data = folio->memcg_data;VM_BUG_ON_FOLIO (folio_test_slab (folio), folio);VM_BUG_ON_FOLIO (memcg_data &MEMCG_DATA_OBJEXTS , folio); -VM_BUG_ON_FOLIO (!(memcg_data &MEMCG_DATA_KMEM ), folio);return (struct obj_cgroup *)(memcg_data & ~OBJEXTS_FLAGS_MASK ); }
@@ -437,21 +410,30 @@ static inline struct obj_cgroup *__folio_objcg(struct folio *folio)* proper memory cgroup pointer. It's not safe to call this function * against some type of folios, e.g. slab folios or ex-slab folios. * -* For a non-kmem folio any of the following ensures folio and memcg binding -* stability: +* For a folio any of the following ensures folio and objcg binding stability: * * - the folio lock * - LRU isolation * - exclusive reference * -* For a kmem folio a caller should hold an rcu read lock to protect memcg -* associated with a kmem folio from being released. +* Based on the stable binding of folio and objcg, for a folio any of the +* following ensures folio and memcg binding stability: +* +* - cgroup_mutex +* - the lruvec lock +* +* If the caller only want to ensure that the page counters of memcg are +* updated correctly, ensure that the binding stability of folio and objcg +* is sufficient. +* +* Note: The caller should hold an rcu read lock or cgroup_mutex to protect +* memcg associated with a folio from being released. */ static inline struct mem_cgroup *folio_memcg (struct folio *folio) { -if (folio_memcg_kmem (folio)) -return obj_cgroup_memcg (__folio_objcg (folio)); -return __folio_memcg (folio); +struct obj_cgroup *objcg =folio_objcg (folio); + +return objcg ?obj_cgroup_memcg (objcg) :NULL ; }/*
@@ -475,15 +457,10 @@ static inline bool folio_memcg_charged(struct folio *folio)* has an associated memory cgroup pointer or an object cgroups vector or * an object cgroup. * -* For a non-kmem folio any of the following ensures folio and memcg binding -* stability: +* The page and objcg or memcg binding rules can refer to folio_memcg(). * -* - the folio lock -* - LRU isolation -* - exclusive reference -* -* For a kmem folio a caller should hold an rcu read lock to protect memcg -* associated with a kmem folio from being released. +* A caller should hold an rcu read lock to protect memcg associated with a +* page from being released. */ static inline struct mem_cgroup *folio_memcg_check (struct folio *folio) {
@@ -492,18 +469,14 @@ static inline struct mem_cgroup *folio_memcg_check(struct folio *folio)* for slabs, READ_ONCE() should be used here. */ unsigned long memcg_data =READ_ONCE (folio->memcg_data); +struct obj_cgroup *objcg;if (memcg_data &MEMCG_DATA_OBJEXTS )return NULL ; -if (memcg_data &MEMCG_DATA_KMEM ) { -struct obj_cgroup *objcg; - - objcg = (void *)(memcg_data & ~OBJEXTS_FLAGS_MASK ); -return obj_cgroup_memcg (objcg); - } + objcg = (void *)(memcg_data & ~OBJEXTS_FLAGS_MASK ); -return (struct mem_cgroup *)(memcg_data & ~OBJEXTS_FLAGS_MASK ); +return objcg ?obj_cgroup_memcg (objcg) :NULL ; }static inline struct mem_cgroup *page_memcg_check (struct page *page)
mm/memcontrol-v1.c +9 -6
@@ -613,6 +613,7 @@ void memcg1_commit_charge(struct folio *folio, struct mem_cgroup *memcg)void memcg1_swapout (struct folio *folio,swp_entry_t entry) {struct mem_cgroup *memcg, *swap_memcg; +struct obj_cgroup *objcg;unsigned int nr_entries;VM_BUG_ON_FOLIO (folio_test_lru (folio), folio);
@@ -624,12 +625,13 @@ void memcg1_swapout(struct folio *folio, swp_entry_t entry)if (!do_memsw_account ())return ; - memcg =folio_memcg (folio); - -VM_WARN_ON_ONCE_FOLIO (!memcg, folio); -if (!memcg) + objcg =folio_objcg (folio); +VM_WARN_ON_ONCE_FOLIO (!objcg, folio); +if (!objcg)return ; +rcu_read_lock (); + memcg =obj_cgroup_memcg (objcg);/* * In case the memcg owning these pages has been offlined and doesn't * have an ID allocated to it anymore, charge the closest online
@@ -644,7 +646,7 @@ void memcg1_swapout(struct folio *folio, swp_entry_t entry)folio_unqueue_deferred_split (folio); folio->memcg_data =0 ; -if (!mem_cgroup_is_root (memcg)) +if (!obj_cgroup_is_root (objcg))page_counter_uncharge (&memcg->memory, nr_entries);if (memcg != swap_memcg) {
@@ -665,7 +667,8 @@ void memcg1_swapout(struct folio *folio, swp_entry_t entry)preempt_enable_nested ();memcg1_check_events (memcg,folio_nid (folio)); -css_put (&memcg->css); +rcu_read_unlock (); +obj_cgroup_put (objcg); }/*
mm/memcontrol.c +120 -79
411 lines not on this page · raw message
Signature · 1 line
2.20.1
Re: [PATCH v5 31/32] mm: memcontrol: eliminate the problem of dying memory cgroup for LRU folios
Quoted 11 lines · Qi Zheng
From: Muchun Song <[email protected]>
Now that everything is set up, switch
folio->memcg_datapointers to objcgs, update the accessors, and execute reparenting on cgroup death.Finally,
folio->memcg_dataof LRU folios and kmem folios will always point to an object cgroup pointer. Thefolio->memcg_dataof slab folios will point to an vector of object cgroups.Signed-off-by Muchun Song <[email protected]>Signed-off-by Qi Zheng <[email protected]>
Qi Zheng
reviewerPATCH v5 32/32 mm: lru: add VM_WARN_ON_ONCE_FOLIO to lru maintenance helpers
In reply to Qi ZhengFrom: Muchun Song <[email protected]>
We must ensure the folio is deleted from or added to the correct lruvec list. So, add VM_WARN_ON_ONCE_FOLIO() to catch invalid users. The VM_BUG_ON_PAGE() in move_pages_to_lru() can be removed as add_page_to_lru_list() will perform the necessary check.
--- include/linux/mm_inline.h | 6 ++++++ mm/vmscan.c | 1 - 2 files changed, 6 insertions(+), 1 deletion(-)
include/linux/mm_inline.h +6 -0
@@ -342,6 +342,8 @@ void lruvec_add_folio(struct lruvec *lruvec, struct folio *folio) {enum lru_list lru =folio_lru_list (folio); +VM_WARN_ON_ONCE_FOLIO (!folio_matches_lruvec (folio, lruvec), folio); +if (lru_gen_add_folio (lruvec, folio, false))return ;
@@ -356,6 +358,8 @@ void lruvec_add_folio_tail(struct lruvec *lruvec, struct folio *folio) {enum lru_list lru =folio_lru_list (folio); +VM_WARN_ON_ONCE_FOLIO (!folio_matches_lruvec (folio, lruvec), folio); +if (lru_gen_add_folio (lruvec, folio, true))return ;
@@ -370,6 +374,8 @@ void lruvec_del_folio(struct lruvec *lruvec, struct folio *folio) {enum lru_list lru =folio_lru_list (folio); +VM_WARN_ON_ONCE_FOLIO (!folio_matches_lruvec (folio, lruvec), folio); +if (lru_gen_del_folio (lruvec, folio, false))return ;
mm/vmscan.c +0 -1
@@ -1920,7 +1920,6 @@ static unsigned int move_folios_to_lru(struct list_head *list)continue ; } -VM_BUG_ON_FOLIO (!folio_matches_lruvec (folio, lruvec), folio);lruvec_add_folio (lruvec, folio); nr_pages =folio_nr_pages (folio); nr_moved += nr_pages;
Signature · 1 line
2.20.1
Andrew Morton
maintainerQuoted 4 lines · On Wed, 25 Feb 2026 15:48:33 +0800 Qi Zheng <[email protected]> wrote:
This patchset is intended to transfer the LRU pages to the object cgroup without holding a reference to the original memory cgroup in order to address the issue of the dying memory cgroup. A consensus has already been reached regarding this approach recently [1].
Thanks, I've added this to mm.git's mm-new branch.
All being well I'll move it into the mm-unstable branch a few days hence.
Then into mm-stable around -rc5.
Then into mainline for 7.1-rc1.
Quoted 8 lines · On Wed, 25 Feb 2026 13:57:06 -0800 Andrew Morton <[email protected]>…
Quoted 4 lines · On Wed, 25 Feb 2026 15:48:33 +0800 Qi Zheng <[email protected]> wrote:
This patchset is intended to transfer the LRU pages to the object cgroup without holding a reference to the original memory cgroup in order to address the issue of the dying memory cgroup. A consensus has already been reached regarding this approach recently [1].
Thanks, I've added this to mm.git's mm-new branch.
Great. I found the commit message on mm-new lacks the conventional "This patch (of X):" line, though. Andrew, could you please add that line?
Thanks,
SJ
[...]