Skip to contentSkip to conversation

mm: read root memcg stats from global counters instead of flushing

PATCH In review Joanne Koong 3 participants 13 messages Nov 14 – Nov 17

Subsystems (7)
lore
Share & tools
lore
#0 lore raw

v3 cover mm: read root memcg stats from global counters instead of flushing

Reclaim, refault detection, the zswap shrinker, and DAMON quota goals all flush the memcg rstat stats before reading them. When the target is the root memcg (eg global reclaim), that flushes every cgroup in the system, which is expensive. Most of these flushes are ratelimited today, but the stats they read can be stale for seconds.

For the root memcg, the call sites can read global counters directly instead of going through rstat. Global counters are always up to date and don't need flushing. For stats that only track memory charged to a memcg (eg LRU sizes, workingset, rotation, and reclaim writeback counts), the root memcg's hierarchical value equals the node counter.

Of the excluded stats, kernel stack, secondary page tables and WORKINGSET_NODERECLAIM only differ from the node counters because accounted kernel memory allocated by tasks in the root cgroup isn't charged to the root memcg. If the root memcg's stats should include that memory, then those could read the node counters as well. This is left for a separate subsequent patchset though since that changes what the root memcg reports and needs more upstream consensus.

Three root flushes remain. Reclaim also reads the stats of the child memcgs it scans, and count_shadow_nodes() and filemap_cachestat() need the root memcg's own (local) stats or the stats of arbitrary memcgs, which means global counters can't be used as a substitute there. This will be addressed in a subsequent patchset, as well as getting rid of ratelimiting for most call sites.

On an 80-CPU machine running proactive reclaim on the root cgroup, a root flush took 69us on average with little cgroup activity and ~0.9ms with 128 memcgs repeatedly allocating and freeing memory (wall time, including waiting for the rstat lock). Reading a node counter doesn't add any cost: it took the same amount of time as reading the rstat counter (~150ns, which includes bpftrace probe overhead).

For more context on the background of this series, please see the discussion about memcg stats flushing in [1].

Changelog
---------
Changes since v2 [3]:
- Change from denylist (patch 2) to allowlist (Sashiko, me, Michal)
- Add SJ's reviewed-by from v1 for the damon patch, which I forgot to add to v2

Changes since v1 [2]:
- Read the node counters for the root memcg by default in lruvec_page_state()/lruvec_page_state_monotonic(), with explicit exclusions, instead of adding lruvec_lru_page_state() helpers that callers opt in to (Yosry). As a result, the root memcg's memory.numa_stat now also reports the node counters for the non-excluded stats
- Split the workingset_test_recent() flush skip out into its own patch (patch 3)
- The DAMON patch (now patch 4) only skips the flush, as lruvec_page_state() reads the node counters for the root memcg
- Rebased onto current mm-new

[1] https://lore.kernel.org/linux-fsdevel/CAJnrk1aHdBm4CofvKfx7B3H-uF2xRXZUQs47J_4xeRmRZ4hcZg@mail.gmail.com/
[2] https://lore.kernel.org/linux-mm/[email protected]/
[3] https://lore.kernel.org/linux-mm/[email protected]/

Joanne Koong (4):
mm: zswap: fix shrinker count for the root memcg
mm: memcg: read root lruvec stats from the node counters
mm: workingset: skip the root memcg flush in workingset_test_recent() mm/damon/core: use node counters instead of flushing the root memcg

 mm/damon/core.c |  7 +++++-
 mm/memcontrol.c | 58 +++++++++++++++++++++++++++++++++++++++++++++++--
 mm/workingset.c |  5 ++++-
 mm/zswap.c      |  7 ++++--
 4 files changed, 71 insertions(+), 6 deletions(-)

base-commit: 31a696f9c37defab0f28f774e00ac7f135889ec3

Signature · 1 line
2.52.1
#1 lore raw

PATCH v3 1/4 mm: zswap: fix shrinker count for the root memcg

zswap_shrinker_count() scales the number of freeable entries on a memcg's zswap LRU by the memcg's compression ratio. That compression ratio is computed from the memcg's MEMCG_ZSWAP_B and MEMCG_ZSWAPPED stats.

However, for the root memcg, the zswap entries owned by the root memcg itself are never accounted for in the stats (obj_cgroup_charge_zswap() skips the root objcg, and before the root memcg had an objcg, its entries weren't charged at all). The stats only reflect the entries owned by the root memcg's descendants. When the root memcg's descendants have no zswap entries (eg when the whole workload runs in the root cgroup), the shrinker reports nothing to reclaim for the root memcg, and its zswap entries are never written back by the shrinker. When its descendants do have entries, the root memcg's entries are scaled by the descendants' compression ratio instead of a ratio that includes them.

Fix this by using the zswap pool-wide counters for the root memcg instead. The zswap pool-wide counters cover every zswap entry, including those owned by the root memcg.

This also gets rid of having to flush the root memcg's stats, which flushes every cgroup in the system.

In a VM running a reclaim-heavy workload in the root cgroup with the zswap shrinker enabled, the shrinker wrote back no pages before this change (zswpwb stayed at 0) and 1016 pages with it.

Fixes b5ba474f3f51 ("zswap: shrink zswap pool based on memory pressure")
---
 mm/zswap.c | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)
mm/zswap.c +5 -2
index a2e0314db1cf..6e1cd3e482f7 100644
@@ -1330,9 +1330,12 @@ static unsigned long zswap_shrinker_count(struct shrinker *shrinker,
 	 * runtime-disabled: we can get sc->memcg == NULL, which is ok
 	 * for the lruvec, but not for memcg_page_state().
 	 *
-	 * Without memcg, use the zswap pool-wide metrics.
+	 * Without memcg, or for the root memcg, use the zswap pool-wide
+	 * metrics. The root memcg's stats don't count the entries it owns
+	 * itself (obj_cgroup_charge_zswap() skips the root objcg), whereas the
+	 * pool-wide metrics cover every entry and don't need a flush.
 	 */
-	if (!mem_cgroup_disabled()) {
+	if (!mem_cgroup_disabled() && !mem_cgroup_is_root(memcg)) {
 		mem_cgroup_flush_stats_ratelimited(memcg);
 		nr_backing = memcg_page_state(memcg, MEMCG_ZSWAP_B) >> PAGE_SHIFT;
 		nr_stored = memcg_page_state(memcg, MEMCG_ZSWAPPED);
Signature · 1 line
2.52.1
#2 lore raw

Re: [PATCH v3 1/4] mm: zswap: fix shrinker count for the root memcg

Sashiko has reviewed this patch and found no issues.
Quoted 15 lines

zswap_shrinker_count() scales the number of freeable entries on a memcg's zswap LRU by the memcg's compression ratio. That compression ratio is computed from the memcg's MEMCG_ZSWAP_B and MEMCG_ZSWAPPED stats.

However, for the root memcg, the zswap entries owned by the root memcg itself are never accounted for in the stats (obj_cgroup_charge_zswap() skips the root objcg, and before the root memcg had an objcg, its entries weren't charged at all). The stats only reflect the entries owned by the root memcg's descendants. When the root memcg's descendants have no zswap entries (eg when the whole workload runs in the root cgroup), the shrinker reports nothing to reclaim for the root memcg, and its zswap entries are never written back by the shrinker. When its descendants do have entries, the root memcg's entries are scaled by the descendants' compression ratio instead of a ratio that includes them.

Quoted 6 lines

Sashiko has reviewed this patch and found no issues.

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
#4 lore raw

PATCH v3 2/4 mm: memcg: read root lruvec stats from the node counters

In reply to Joanne Koong

For the root memcg, lruvec_page_state() returns the rstat value, which needs a flush to be up to date. Flushing the root memcg is expensive, as it flushes every memcg in the system.

Most memcg node stats only track memory charged to a memcg, so the root memcg's hierarchical value equals the node counter, which is always up to date and doesn't need a flush. For those stats, have lruvec_page_state() and lruvec_page_state_monotonic() read the node counter for the root memcg, as they already do when memcg is disabled. Callers that only read the root memcg's own lruvec then no longer need to flush it, which the following patches take advantage of. It also makes the root memcg's stats more accurate, as callers that ratelimit the flush can otherwise read values that are seconds old.

The exceptions are the kernel memory stats (slab, kernel stack, page tables, secondary page tables and vmalloc), hugetlb, and WORKINGSET_NODERECLAIM, which counts reclaimed xarray nodes (slab objects). Their node counters also count memory that isn't charged to a memcg, which falls into two categories:

a) the allocation isn't accounted to any memcg (eg kernel page tables, or hugetlb when hugetlb accounting is off)
b) the allocation is accounted, but it comes from a task running in the root cgroup, and the root memcg isn't charged for it

As such, these stats still need to read the rstat value, since the node counter value can be higher than the rstat value.

One user-visible change is that the root memcg's memory.numa_stat (and the hierarchical_* entries of the cgroup v1 memory.numa_stat) now reports the node counters for the non-excluded stats, as it reads lruvec_page_state(). This is more accurate, but as a result, it may not exactly match the root memcg's memory.stat, which still reads the rstat values.

---
 mm/memcontrol.c | 58 +++++++++++++++++++++++++++++++++++++++++++++++--
 1 file changed, 56 insertions(+), 2 deletions(-)
mm/memcontrol.c +56 -2
index 091401cecf39..196c0382181b 100644
@@ -504,13 +504,67 @@ struct lruvec_stats {
 	long state_pending[NR_MEMCG_NODE_STAT_ITEMS];
 };
 
+/*
+ * Whether to read the node counter for @idx instead of @lruvec's rstat value.
+ * This is only done for the root memcg for stats where every update also
+ * updates a memcg's stats, so the root memcg's hierarchical value equals the
+ * node counter. The node counter is preferred since it is always up to date,
+ * and avoids an rstat flush which for the root memcg flushes every cgroup in
+ * the system.
+ */
+static bool lruvec_use_node_stat(const struct lruvec *lruvec,
+				 enum node_stat_item idx)
+{
+	if (!mem_cgroup_is_root(lruvec_memcg(lruvec)))
+		return false;
+
+	switch (idx) {
+	case NR_INACTIVE_ANON:
+	case NR_ACTIVE_ANON:
+	case NR_INACTIVE_FILE:
+	case NR_ACTIVE_FILE:
+	case NR_UNEVICTABLE:
+	case WORKINGSET_REFAULT_ANON:
+	case WORKINGSET_REFAULT_FILE:
+	case WORKINGSET_ACTIVATE_ANON:
+	case WORKINGSET_ACTIVATE_FILE:
+	case WORKINGSET_RESTORE_ANON:
+	case WORKINGSET_RESTORE_FILE:
+#ifdef CONFIG_NUMA_BALANCING
+	case PGPROMOTE_SUCCESS:
+#endif
+	case PGDEMOTE_KSWAPD:
+	case PGDEMOTE_DIRECT:
+	case PGDEMOTE_KHUGEPAGED:
+	case PGDEMOTE_PROACTIVE:
+	case PGSTEAL_KSWAPD:
+	case PGSTEAL_DIRECT:
+	case PGSTEAL_KHUGEPAGED:
+	case PGSTEAL_PROACTIVE:
+	case PGSTEAL_ANON:
+	case PGSTEAL_FILE:
+	case PGSCAN_KSWAPD:
+	case PGSCAN_DIRECT:
+	case PGSCAN_KHUGEPAGED:
+	case PGSCAN_PROACTIVE:
+	case PGSCAN_ANON:
+	case PGSCAN_FILE:
+	case PGROTATE_ANON:
+	case PGROTATE_FILE:
+	case PGREFILL:
+		return true;
+	default:
+		return false;
+	}
+}
+
 unsigned long lruvec_page_state(const struct lruvec *lruvec, enum node_stat_item idx)
 {
 	const struct mem_cgroup_per_node *pn;
 	long x;
 	int i;
 
-	if (mem_cgroup_disabled())
+	if (mem_cgroup_disabled() || lruvec_use_node_stat(lruvec, idx))
 		return node_page_state(lruvec_pgdat(lruvec), idx);
 
 	i = memcg_stats_index(idx);
@@ -551,7 +605,7 @@ unsigned long lruvec_page_state_monotonic(const struct lruvec *lruvec,
 	const struct mem_cgroup_per_node *pn;
 	int i;
 
-	if (mem_cgroup_disabled())
+	if (mem_cgroup_disabled() || lruvec_use_node_stat(lruvec, idx))
 		return node_page_state_monotonic(lruvec_pgdat(lruvec), idx);
 
 	i = memcg_stats_index(idx);
Signature · 1 line
2.52.1
#5 lore raw

Re: [PATCH v3 2/4] mm: memcg: read root lruvec stats from the node counters

Thank you for your contribution! Sashiko AI review found 1 potential issue to consider: - [Medium] mm: memcg: missing NR_VMSCAN_WRITE and folio stats in lruvec_use_node_stat() --

Thank you for your contribution! Sashiko AI review found 1 potential issue to consider:
- [Medium] mm: memcg: missing NR_VMSCAN_WRITE and folio stats in lruvec_use_node_stat()
--

commit 95f274e85e4bf7a42c96a69c0a4509c8d43a4417
Author: Joanne Koong <[email protected]>

mm: memcg: read root lruvec stats from the node counters

This commit makes lruvec_page_state() and lruvec_page_state_monotonic() read the node counter directly for the root memcg on stats whose node counter equals the root memcg's hierarchical value.

Quoted 27 lines
mm/memcontrol.c +19 -0
index 091401cecf397..196c0382181b1 100644
@@ -504,13 +504,67 @@ struct lruvec_stats {
 	long state_pending[NR_MEMCG_NODE_STAT_ITEMS];
 };
 
+/*
+ * Whether to read the node counter for @idx instead of @lruvec's rstat value.
+ * This is only done for the root memcg for stats where every update also
+ * updates a memcg's stats, so the root memcg's hierarchical value equals the
+ * node counter. The node counter is preferred since it is always up to date,
+ * and avoids an rstat flush which for the root memcg flushes every cgroup in
+ * the system.
+ */
+static bool lruvec_use_node_stat(const struct lruvec *lruvec,
+				 enum node_stat_item idx)
+{
+	if (!mem_cgroup_is_root(lruvec_memcg(lruvec)))
+		return false;
+
+	switch (idx) {
+	case NR_INACTIVE_ANON:
+	case NR_ACTIVE_ANON:
+	case NR_INACTIVE_FILE:
+	case NR_ACTIVE_FILE:

mm/memcontrol.c struct lruvec_stats { · line 535

Quoted 10 lines
mm/memcontrol.c +10 -0
+	case NR_UNEVICTABLE:
+	case WORKINGSET_REFAULT_ANON:
+	case WORKINGSET_REFAULT_FILE:
+	case WORKINGSET_ACTIVATE_ANON:
+	case WORKINGSET_ACTIVATE_FILE:
+	case WORKINGSET_RESTORE_ANON:
+	case WORKINGSET_RESTORE_FILE:
+#ifdef CONFIG_NUMA_BALANCING
+	case PGPROMOTE_SUCCESS:
+#endif

[Severity: Medium]
Should lruvec_use_node_stat() also include NR_VMSCAN_WRITE, NR_SWAPCACHE, and the other charged-folio stats between WORKINGSET_RESTORE_FILE and PGPROMOTE_SUCCESS in memcg_node_stat_items[]?

The switch statement drops the entire contiguous block from WORKINGSET_NODERECLAIM through NR_SWAPCACHE rather than only the kernel memory items (WORKINGSET_NODERECLAIM and NR_VMALLOC..NR_SECONDARY_PAGETABLE):

mm/memcontrol.c:memcg_node_stat_items[] {
    ...
	WORKINGSET_RESTORE_FILE,
	WORKINGSET_NODERECLAIM,
	NR_ANON_MAPPED,
	NR_FILE_MAPPED,
	NR_FILE_PAGES,
	NR_FILE_DIRTY,
	NR_WRITEBACK,
	NR_SHMEM,
	NR_SHMEM_THPS,
	NR_FILE_THPS,
	NR_ANON_THPS,
	NR_VMSCAN_WRITE,
	NR_VMALLOC,
	NR_KERNEL_STACK_KB,
	NR_PAGETABLE,
	NR_SECONDARY_PAGETABLE,

#ifdef CONFIG_SWAP
NR_SWAPCACHE,
#endif
#ifdef CONFIG_NUMA_BALANCING
PGPROMOTE_SUCCESS,

#endif
    ...
}

In prepare_scan_control() (the only caller of lruvec_page_state_monotonic()), anon IO cost is computed as WORKINGSET_RESTORE_ANON + NR_VMSCAN_WRITE. Because NR_VMSCAN_WRITE is only updated via lruvec_stat_mod_folio() in pageout() and is omitted here, root-lruvec reclaim mixes live node counters for PGROTATE_ANON and WORKINGSET_RESTORE_ANON with the rstat counter for NR_VMSCAN_WRITE. Similarly, reclaimable_anon_is_low() on root_mem_cgroup compares live node counters for NR_INACTIVE_ANON + NR_ACTIVE_ANON against the unflushed rstat counter for NR_SWAPCACHE.

NR_ANON_MAPPED, NR_ANON_THPS, NR_SHMEM, NR_SHMEM_THPS, NR_FILE_THPS, and NR_FILE_DIRTY are also only updated on charged folios, whereas NR_FILE_PAGES (updated via node_stat_mod_folio() for swapcache), NR_WRITEBACK (updated via node_stat_mod_folio() for NFS unstable writes), and NR_FILE_MAPPED (updated for uncharged pages in vm_insert_page()) can exceed the root memcg's rstat value and would need to be noted as exceptions if excluded.

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=2
Reviewed-by on v3 2/4
#6 lore raw

Re: [PATCH v3 2/4] mm: memcg: read root lruvec stats from the node counters

In reply to Joanne Koong

Hi Joanne,

Quoted 5 lines · Joanne Koong

The exceptions are the kernel memory stats (slab, kernel stack, page tables, secondary page tables and vmalloc), hugetlb, and WORKINGSET_NODERECLAIM, which counts reclaimed xarray nodes (slab objects). Their node counters also count memory that isn't charged to a memcg, which falls into two categories:

This still describes the v2 denylist. With the allowlist, the root memcg also keeps reading rstat for NR_ANON_MAPPED, NR_FILE_MAPPED, NR_SHMEM et al, and some of those differ for reasons outside the two categories here, like swapcache being counted differently in the node counter, for example. Could the changelog say what is on the list instead? So that would be the stats that only change through mod_lruvec_state(), with everything else still reading rstat.

mm/memcontrol.c struct lruvec_stats { · line 555

Quoted 4 lines
mm/memcontrol.c +4 -0
+	case PGROTATE_ANON:
+	case PGROTATE_FILE:
+	case PGREFILL:
+		return true;

NR_VMSCAN_WRITE isn't on the list, but prepare_scan_control() adds it to WORKINGSET_RESTORE_ANON, so once the root flush in reclaim goes away, root reclaim would mix a live node counter with a stale rstat value. Unless I'm mistaken it should go here as well.

Once those are fixed, please feel free to add:

Thanks!

Chris

#7 lore raw

PATCH v3 3/4 mm: workingset: skip the root memcg flush in workingset_test_recent()

In reply to Joanne Koong

We can skip the rstat flush in workingset_test_recent() when the eviction memcg is the root memcg. lruvec_page_state() now reads the LRU sizes of the root memcg from the node counters, which are always up to date, and workingset_test_recent() only reads the eviction memcg's own lruvec. This avoids an expensive flush on the refault path, as flushing the root memcg flushes every memcg in the system.

For reclaim however, we still need to keep the flush (for now) since global and root reclaim also read the rstat stats of the child memcgs they scan, which the flush is needed for in order to keep up to date.

On an 80-CPU machine, a root flush took 69us on average with little cgroup activity and ~0.9ms with 128 memcgs actively allocating (wall time, including waiting for the rstat lock). Reading a node counter doesn't add any cost. It took the same amount of time as reading the rstat counter (~150ns, which includes bpftrace probe overhead).

---
 mm/workingset.c | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)
mm/workingset.c +4 -1
index 1504f91cdca5..383f951eaad4 100644
@@ -483,10 +483,13 @@ bool workingset_test_recent(void *shadow, bool file, bool *workingset,
 	 * section (for e.g, in cachestat) - these callers need to skip flushing
 	 * stats (via the flush argument).
 	 *
+	 * For the root memcg, the LRU sizes are read directly from the node
+	 * counters (see lruvec_page_state()).
+	 *
 	 * XXX: With per-memcg flushing and thresholding, is ratelimiting
 	 * still needed here?
 	 */
-	if (flush)
+	if (flush && !mem_cgroup_is_root(eviction_memcg))
 		mem_cgroup_flush_stats_ratelimited(eviction_memcg);
 
 	eviction_lruvec = mem_cgroup_lruvec(eviction_memcg, pgdat);
Signature · 1 line
2.52.1
#8 lore raw

Re: [PATCH v3 3/4] mm: workingset: skip the root memcg flush in workingset_test_recent()

Sashiko has reviewed this patch and found no issues.
Quoted 11 lines

We can skip the rstat flush in workingset_test_recent() when the eviction memcg is the root memcg. lruvec_page_state() now reads the LRU sizes of the root memcg from the node counters, which are always up to date, and workingset_test_recent() only reads the eviction memcg's own lruvec. This avoids an expensive flush on the refault path, as flushing the root memcg flushes every memcg in the system.

For reclaim however, we still need to keep the flush (for now) since global and root reclaim also read the rstat stats of the child memcgs they scan, which the flush is needed for in order to keep up to date.

Quoted 7 lines

On an 80-CPU machine, a root flush took 69us on average with little cgroup activity and ~0.9ms with 128 memcgs actively allocating (wall time, including waiting for the rstat lock). Reading a node counter doesn't add any cost. It took the same amount of time as reading the rstat counter (~150ns, which includes bpftrace probe overhead).

Sashiko has reviewed this patch and found no issues.

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=3
Reviewed-by Tested-by on v3 3/4
#9 lore raw

Re: [PATCH v3 3/4] mm: workingset: skip the root memcg flush in workingset_test_recent()

In reply to Joanne Koong

Hi Joanne,

Quoted 3 lines · Joanne Koong
mm/workingset.c +1 -1
-	if (flush)
+	if (flush && !mem_cgroup_is_root(eviction_memcg))
		mem_cgroup_flush_stats_ratelimited(eviction_memcg);

Not a big deal, but in the v2 thread you mentioned you would try a mem_cgroup_flush_lru_stats_ratelimited() helper for this, per Michal's suggestion. That still seems like a good idea given this skip and the one in DAMON rely on everything read after them staying in lruvec_use_node_stat(), and at least speaking for myself, keeping the root check close to the list in memcontrol.c makes that easier to see. It doesn't need
to hold this patch up though, and I've tested that refaults in the root cgroup no longer flush, so either way:

Thanks!

Chris

#10 lore raw

PATCH v3 4/4 mm/damon/core: use node counters instead of flushing the root memcg

In reply to Joanne Koong

damos_get_node_memcg_used_bp() flushes the memcg stats before reading the LRU sizes of the goal's memcg on the goal's node. For the root memcg, that flushes every cgroup in the system. The flush is not ratelimited, which means it happens every time the goal is evaluated.

lruvec_page_state() now reads the LRU sizes of the root memcg from the node counters, which are always up to date, so skip the flush for the root memcg.

---
 mm/damon/core.c | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)
mm/damon/core.c +6 -1
index 0a056949604f..14f2a36c1025 100644
@@ -3057,7 +3057,12 @@ static unsigned long damos_get_node_memcg_used_bp(
 			return 10000;
 	}
 
-	mem_cgroup_flush_stats(memcg);
+	/*
+	 * The root memcg's LRU sizes are read directly from the node counters
+	 * (see lruvec_page_state()).
+	 */
+	if (!mem_cgroup_is_root(memcg))
+		mem_cgroup_flush_stats(memcg);
 	lruvec = mem_cgroup_lruvec(memcg, NODE_DATA(goal->nid));
 	used_pages = lruvec_page_state(lruvec, NR_ACTIVE_ANON);
 	used_pages += lruvec_page_state(lruvec, NR_INACTIVE_ANON);
Signature · 1 line
2.52.1
#11 lore raw

Re: [PATCH v3 4/4] mm/damon/core: use node counters instead of flushing the root memcg

Sashiko has reviewed this patch and found no issues.
Quoted 11 lines

damos_get_node_memcg_used_bp() flushes the memcg stats before reading the LRU sizes of the goal's memcg on the goal's node. For the root memcg, that flushes every cgroup in the system. The flush is not ratelimited, which means it happens every time the goal is evaluated.

lruvec_page_state() now reads the LRU sizes of the root memcg from the node counters, which are always up to date, so skip the flush for the root memcg.

Sashiko has reviewed this patch and found no issues.

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=4