Skip to contentSkip to conversation

mm/vmscan: move tracepoints to mm/ and add MGLRU tracepoints

PATCH Superseded Ridong Chen 3 participants 16 messages Nov 3 – Nov 4

Subsystems (8)
Memory management BPF Tracing cgroup +4 more subsystems

v6 of 7 Superseded Latest is v7 →

lore
Share & tools
lore
#0 lore raw

v6 cover mm/vmscan: move tracepoints to mm/ and add MGLRU tracepoints

From: Ridong Chen <[email protected]>

Background
==========

MGLRU currently has no tracepoints of its own. The scan and evict paths reuse the classic-LRU tracepoints (trace_mm_vmscan_lru_isolate() and trace_mm_vmscan_lru_shrink_inactive()), which predate MGLRU and carry no generation, sequence or memcg context. A trace of a running system therefore cannot tell which memcg a given scan belongs to, how far reclaim has progressed through the generations, or when a new generation is created.

While reviewing an earlier version that added two MGLRU tracepoints, Steven suggested moving include/trace/events/vmscan.h to mm/trace_vmscan.h [1], as several other subsystems already do. Once the header lives under mm/, struct scan_control can be moved there too, letting the trace events access its fields directly instead of having each callsite copy the scalars out by hand. This version merges the two efforts into one series: the restructuring first, then the two new tracepoints built on top of it.

Implementation
==============

Part 1, restructuring (patches 1-4):

1) drop the unused vmscan trace include from memcontrol.c;
2) move include/trace/events/vmscan.h to mm/trace_vmscan.h, setting

     TRACE_INCLUDE_PATH to "." and adding -I$(src) for vmscan.o;
  3) move struct scan_control into a dedicated mm/vmscan.h with an
     ordinary include guard, which trace_vmscan.h pulls in;

4) pass scan_control to the vmscan tracepoints and pick the fields out in TP_fast_assign: the reclaim-begin events, the LRU isolate/shrink events, mm_vmscan_reclaim_pages and mm_vmscan_balance_pgdat_end.

Part 2, MGLRU tracepoints (patches 5-7):

5) add mm_mglru_scan_folios on the scan path, next to the classic mm_vmscan_lru_isolate, reporting the same counters plus nr_sorted and the nid, memcg id, max_seq, tier and min_seq of the lruvec;
6) add lruvec_gen_size() to mm/vmscan.h, returning the number of pages of one type in the generation of a given seq, and convert the two open-coded users in vmscan.c;
7) add mm_mglru_inc_max_seq on the aging path, reporting the nid, memcg id, the new max_seq, the anon/file min_seq and the per-generation page counts of both types, from the oldest generation to the newest as in the debugfs lru_gen file.

Both new events only take pointers (lruvec, sc) plus the local counters, and derive everything else in TP_fast_assign(), so the reclaim paths do no extra work when they are disabled.

Effect
======

Part 1 moves the loads of the tracepoint arguments out of the reclaim paths and into the trace event callbacks, which only run when the event is enabled. On x86_64 with gcc 11.4, the callers in mm/vmscan.o shrink accordingly (bytes), e.g.:

  isolate_lru_folios    4499 -> 4335 (-164)
  __node_reclaim        2402 -> 2220 (-182)
  isolate_folios        8706 -> 8577 (-129)
  shrink_active_list    3669 -> 3544 (-125)
  evict_folios          4858 -> 4769  (-89)

There is no functional change in part 1. TP_STRUCT__entry and TP_printk of the existing events are untouched, so their tracefs format is identical. This was confirmed by diffing the tracefs format files before and after, and by capturing the direct/memcg/node reclaim-begin events under load in a QEMU guest (order, gfp_flags and memcg_id all match). BPF programs attached via raw_tp/tp_btf do see the new TP_PROTO and need to be adjusted; tracepoints are not a stable ABI (see Documentation/bpf/bpf_design_QA.rst).

highest_zoneidx and nr_reclaimed stay separate arguments: the former is a balance_pgdat() parameter (sc.reclaim_idx may differ from it by the time the end event fires), and the latter is the return value of shrink_folio_list(), which does not update sc->nr_reclaimed.

Part 2 makes the full aging-to-eviction window observable per memcg:
aging advances max_seq (the leading edge), scanning consumes the oldest generations, and the min_seq/max_seq pair on each event shows how the generation window moves over time. A sample trace, captured under memory.reclaim of a test memcg, with the classic-LRU tracepoints left enabled to show how they interleave:

mm_vmscan_lru_isolate: classzone=4 order=0 nr_requested=64 nr_scanned=36 nr_skipped=0 nr_taken=36 lru=inactive_file
mm_mglru_scan_folios: nid=0 memcg_id=1833 classzone=4 order=0 nr_requested=64 nr_scanned=36 nr_sorted=0 nr_skipped=0 nr_taken=36 lru=inactive_file max_seq=3 tier=3 min_seq=0 mm_vmscan_lru_shrink_inactive: nid=0 nr_scanned=36 nr_reclaimed=31 nr_dirty=1 nr_writeback=0 nr_congested=0 nr_immediate=0 nr_activate_anon=0 nr_activate_file=1 nr_ref_keep=4 nr_unmap_fail=0 priority=2 flags=RECLAIM_WB_FILE|RECLAIM_WB_ASYNC
mm_mglru_inc_max_seq: nid=0 memcg_id=1833 max_seq=4 anon_min_seq=1 file_min_seq=2 nr_anon={0x0,0x802,0x0,0x0} nr_file={0x0,0x357,0x0,0x0}
mm_vmscan_lru_isolate: classzone=4 order=0 nr_requested=64 nr_scanned=64 nr_skipped=0 nr_taken=64 lru=inactive_file
mm_mglru_scan_folios: nid=0 memcg_id=1833 classzone=4 order=0 nr_requested=64 nr_scanned=64 nr_sorted=0 nr_skipped=0 nr_taken=64 lru=inactive_file max_seq=4 tier=3 min_seq=2 mm_vmscan_lru_shrink_inactive: nid=0 nr_scanned=64 nr_reclaimed=64 nr_dirty=0 nr_writeback=0 nr_congested=0 nr_immediate=0 nr_activate_anon=0 nr_activate_file=0 nr_ref_keep=0 nr_unmap_fail=0 priority=2 flags=RECLAIM_WB_FILE|RECLAIM_WB_ASYNC

The mm_mglru_scan_folios lines carry the node, memcg id and the generation window (max_seq/min_seq) that the bare mm_vmscan_lru_isolate lines above them cannot - those cannot even say which memcg they came from. The mm_mglru_inc_max_seq line then shows a new generation being created (max_seq 3 -> 4) with the per-generation page counts for both types, listed from the oldest generation (seq 1) to the newest (seq 4), and the following scan already reports the moved window (max_seq=4 min_seq=2).

Testing
=======

The output of mm_mglru_inc_max_seq matches the debugfs lru_gen file, e.g. right after aging a test memcg:

mm_mglru_inc_max_seq: nid=0 memcg_id=1833 max_seq=4 anon_min_seq=1 file_min_seq=1 nr_anon={0x0,0x4002,0x0,0x0} nr_file={0x2412,0x0,0x0,0x0}

  # cat /sys/kernel/debug/lru_gen
  memcg 1833 /tpt
   node     0
            1       4541          0        9234
            2       4541      16386           0
            3       4541          0           0
            4         76          0           0

[1]: https://lore.kernel.org/linux-mm/20260916095122.50cbb620@robin/

This series was developed with the assistance of an LLM-based coding assistant.

---
Changes in v6:
- Rebase onto mm-new; MAINTAINERS already listed include/trace/events/vmscan.h under MEMORY MANAGEMENT - RECLAIM,
patch 2 now renames that entry.
- New patch 6: add lruvec_gen_size(), ported from Barry's swappiness
series [3], in mm/vmscan.h so the tracepoint can use it too, and
convert lruvec_evictable_size() and the debugfs lru_gen output
(Barry).
- Patch 7: order nr_anon/nr_file by seq, from the oldest generation
to the newest, the same order as the debugfs lru_gen file, instead
of by generation slot (Barry, Baoquan).
- Collect Reviewed-by/Acked-by/Tested-by tags from Baoquan, Muchun
and Barry.
- Link to v5: https://lore.kernel.org/linux-mm/[email protected]/

Changes in v5:
- Squash v4 patches 4-7 into a single patch 4 (Barry).
- Link to v4: https://lore.kernel.org/linux-mm/[email protected]/

Changes in v4:
- Merge the MGLRU tracepoint series [2] into this one, on top of the restructuring, and drop the RFC tag.
- Patch 2: use TRACE_INCLUDE_PATH "." with CFLAGS_vmscan.o += -I$(src)
instead of a relative path (Steven).
- Patches 8-9: pass lruvec/sc and compute the fields in
TP_fast_assign() instead of at the call site (Steven).
- Patches 8-9: add nid, derived from the lruvec (Baoquan).
- Patch 8: group the int fields at the end of the entry to avoid
padding on 64-bit (Steven).
- Drop "mm/mglru: factor out lru_gen_seq_nr_pages()" from [2]: the per-generation page count is now summed inside TP_fast_assign(), so
the helper has no user. The Reviewed-by tags given to [2] are not
carried over, as patches 8-9 changed substantially.
- Link to RFC v3: https://lore.kernel.org/all/[email protected]/

Changes in RFC v3:
- Use "Assisted-by: LLM" (Muchun).
- Note the BPF raw_tp/tp_btf impact in the changelogs (Sashiko).
- Changelog wording fixes and minor indentation fixes.
- Link to RFC v2: https://lore.kernel.org/all/[email protected]/

Changes in RFC v2:
- Convert the remaining vmscan tracepoints too.
- Move struct scan_control to mm/vmscan.h (Baoquan).

[2]: https://lore.kernel.org/all/[email protected]/
[3]: https://lore.kernel.org/linux-mm/[email protected]/

Ridong Chen (7):
mm/memcontrol: drop unused vmscan tracepoint include mm/vmscan: move vmscan tracepoints to a local header mm/vmscan: move struct scan_control to a dedicated header mm/vmscan: pass scan_control to the vmscan tracepoints mm/mglru: add tracepoint for scan_folios() mm/mglru: add lruvec_gen_size() helper
mm/mglru: add tracepoint for inc_max_seq()

 MAINTAINERS                                   |   5 +-
 mm/Makefile                                   |   1 +
 mm/memcontrol.c                               |   2 -
 mm/shrinker.c                                 |   2 +-
 .../events/vmscan.h => mm/trace_vmscan.h      | 192 ++++++++++++++----
 mm/vmscan.c                                   | 158 +++-----------
 mm/vmscan.h                                   | 134 ++++++++++++
 7 files changed, 317 insertions(+), 177 deletions(-)
 rename include/trace/events/vmscan.h => mm/trace_vmscan.h (76%)
 create mode 100644 mm/vmscan.h

base-commit: 0e02eb17f263e4a3ae6bf1fe45122c23361f0cf7

Signature · 1 line
2.34.1
#1 lore raw

PATCH v6 1/7 mm/memcontrol: drop unused vmscan tracepoint include

From: Ridong Chen <[email protected]>

memcontrol.c includes <trace/events/vmscan.h> but does not use any of the tracepoints or macros it defines: all tracepoints invoked here come from memcg.h and writeback.h.

Drop it ahead of moving the vmscan trace header under mm/, after which <trace/events/vmscan.h> will no longer exist.

No functional change.

---
 mm/memcontrol.c | 2 --
 1 file changed, 2 deletions(-)
mm/memcontrol.c +0 -2
index ac6c43ea8c5a..5bad742dc381 100644
@@ -77,8 +77,6 @@
 #include <trace/events/memcg.h>
 #undef CREATE_TRACE_POINTS
 
-#include <trace/events/vmscan.h>
-
 struct cgroup_subsys memory_cgrp_subsys __read_mostly;
 EXPORT_SYMBOL(memory_cgrp_subsys);
 
Signature · 1 line
2.34.1
#2 lore raw

PATCH v6 2/7 mm/vmscan: move vmscan tracepoints to a local header

In reply to Ridong Chen

From: Ridong Chen <[email protected]>

Move include/trace/events/vmscan.h to mm/trace_vmscan.h so that the tracepoints can be extended to reference vmscan-internal structures such as struct scan_control, which is not visible outside mm/. This mirrors what several other subsystems already do with their trace headers (e.g. fs/fuse/fuse_trace.h, fs/gfs2/trace_gfs2.h, fs/lockd/trace.h). As they do, set TRACE_INCLUDE_PATH to "." and add mm/ to the include path of vmscan.o, the only user that defines CREATE_TRACE_POINTS.

No functional change.

---
 MAINTAINERS                                        | 3 ++-
 mm/Makefile                                        | 1 +
 mm/shrinker.c                                      | 2 +-
 include/trace/events/vmscan.h => mm/trace_vmscan.h | 4 ++++
 mm/vmscan.c                                        | 2 +-
 5 files changed, 9 insertions(+), 3 deletions(-)
 rename include/trace/events/vmscan.h => mm/trace_vmscan.h (99%)
MAINTAINERS +2 -1
index f4a6b5735794..95ff3743cec8 100644
@@ -17708,6 +17708,7 @@ F:	Documentation/admin-guide/mm/multigen_lru.rst
 F:	Documentation/mm/multigen_lru.rst
 F:	include/linux/mm_inline.h
 F:	include/linux/mmzone.h
+F:	mm/trace_vmscan.h
 F:	mm/vmscan.c
 F:	mm/workingset.c
 
@@ -17928,7 +17929,7 @@ R:	Shakeel Butt <[email protected]>
 R:	Lorenzo Stoakes <[email protected]>
 L:	[email protected]
 S:	Maintained
-F:	include/trace/events/vmscan.h
+F:	mm/trace_vmscan.h
 F:	mm/vmscan.c
 F:	mm/workingset.c
 
mm/Makefile +1 -0
index 2a3ec53d62ee..277b97f7f8d7 100644
@@ -32,6 +32,7 @@ KCOV_INSTRUMENT_vmstat.o := n
 KCOV_INSTRUMENT_failslab.o := n
 
 CFLAGS_init-mm.o += -Wno-override-init
+CFLAGS_vmscan.o += -I$(src)
 
 mmu-y			:= nommu.o
 mmu-$(CONFIG_MMU)	:= highmem.o memory.o mincore.o \
mm/shrinker.c +1 -1
index 4fb424e721bc..4fbdade6d2f6 100644
@@ -3,7 +3,7 @@
 #include <linux/rwsem.h>
 #include <linux/shrinker.h>
 #include <linux/rculist.h>
-#include <trace/events/vmscan.h>
+#include "trace_vmscan.h"
 
 #include "internal.h"
 
mm/trace_vmscan.h renamed from include/trace/events/vmscan.h +4 -0
similarity index 99%
rename from include/trace/events/vmscan.h
rename to mm/trace_vmscan.h
index 8a872990b4be..ad90f0e9bf2e 100644
@@ -648,4 +648,8 @@ TRACE_EVENT(mm_vmscan_kswapd_clear_hopeless,
 #endif /* _TRACE_VMSCAN_H */
 
 /* This part must be outside protection */
+#undef TRACE_INCLUDE_FILE
+#undef TRACE_INCLUDE_PATH
+#define TRACE_INCLUDE_PATH .
+#define TRACE_INCLUDE_FILE trace_vmscan
 #include <trace/define_trace.h>
mm/vmscan.c +1 -1
index 7ddef3b488b3..b51fae79c6ab 100644
@@ -71,7 +71,7 @@
 #include "swap.h"
 
 #define CREATE_TRACE_POINTS
-#include <trace/events/vmscan.h>
+#include "trace_vmscan.h"
 
 struct scan_control {
 	/* How many pages shrink_list() should reclaim */
Signature · 1 line
2.34.1
#3 lore raw

PATCH v6 3/7 mm/vmscan: move struct scan_control to a dedicated header

In reply to Ridong Chen

From: Ridong Chen <[email protected]>

The definition of struct scan_control has lived inline at the top of mm/vmscan.c. Move it out to a new mm/vmscan.h so the reclaim-internal state can be referenced from other reclaim translation units without duplicating the definition.

The header uses an ordinary include guard and pulls in only what the struct needs (nodemask_t, gfp_t and struct reclaim_state).

mm/trace_vmscan.h includes the new header directly so that any translation unit pulling in the trace events (mm/vmscan.c, mm/shrinker.c) sees the full definition of struct scan_control regardless of include order.

No functional change.

---
 MAINTAINERS       |   2 +
 mm/trace_vmscan.h |   2 +
 mm/vmscan.c       | 108 ------------------------------------------
 mm/vmscan.h       | 118 ++++++++++++++++++++++++++++++++++++++++++++++
 4 files changed, 122 insertions(+), 108 deletions(-)
 create mode 100644 mm/vmscan.h
MAINTAINERS +2 -0
index 95ff3743cec8..47e4673b04ee 100644
@@ -17710,6 +17710,7 @@ F:	include/linux/mm_inline.h
 F:	include/linux/mmzone.h
 F:	mm/trace_vmscan.h
 F:	mm/vmscan.c
+F:	mm/vmscan.h
 F:	mm/workingset.c
 
 MEMORY MANAGEMENT - MIGRATION
@@ -17931,6 +17932,7 @@ L:	[email protected]
 S:	Maintained
 F:	mm/trace_vmscan.h
 F:	mm/vmscan.c
+F:	mm/vmscan.h
 F:	mm/workingset.c
 
 MEMORY MANAGEMENT - RMAP (REVERSE MAPPING)
mm/trace_vmscan.h +2 -0
index ad90f0e9bf2e..3917e1b62dfa 100644
@@ -11,6 +11,8 @@
 #include <linux/memcontrol.h>
 #include <trace/events/mmflags.h>
 
+#include "vmscan.h"
+
 #define RECLAIM_WB_ANON		0x0001u
 #define RECLAIM_WB_FILE		0x0002u
 #define RECLAIM_WB_MIXED	0x0010u
mm/vmscan.c +0 -108

114 lines not on this page · raw message

mm/vmscan.h new file +118 -0

118 lines not on this page · raw message

Signature · 1 line
2.34.1
#4 lore raw

PATCH v6 4/7 mm/vmscan: pass scan_control to the vmscan tracepoints

In reply to Ridong Chen

From: Ridong Chen <[email protected]>

Several vmscan tracepoints spell out reclaim state as individual arguments, all of which the caller already holds in struct scan_control. Now that the struct is visible from the trace header, pass sc directly and pick the fields out in TP_fast_assign instead of spelling them out at every callsite:

- mm_vmscan_direct_reclaim_begin, mm_vmscan_memcg_reclaim_begin and mm_vmscan_node_reclaim_begin: gfp_mask and order.

- mm_vmscan_lru_isolate, mm_vmscan_lru_shrink_inactive and mm_vmscan_lru_shrink_active: reclaim_idx, order and priority.

- mm_vmscan_reclaim_pages: nr_scanned. nr_reclaimed stays a separate
argument: it is the return value of shrink_folio_list(), which does not update sc->nr_reclaimed.

- mm_vmscan_balance_pgdat_end: order and nr_reclaimed. highest_zoneidx stays a separate argument: it is a balance_pgdat() parameter, and sc.reclaim_idx may differ from it by the time the end event fires.

No functional change. The tracefs event format (TP_STRUCT__entry and TP_printk) is unchanged, so tracefs-based consumers are unaffected. BPF programs attached via raw_tp/tp_btf see the new TP_PROTO and need to be adjusted; tracepoints are not a stable ABI (see Documentation/bpf/bpf_design_QA.rst).

---
 mm/trace_vmscan.h | 74 +++++++++++++++++++++++------------------------
 mm/vmscan.c       | 26 ++++++++---------
 2 files changed, 48 insertions(+), 52 deletions(-)
mm/trace_vmscan.h +36 -38
index 3917e1b62dfa..b2ad83e5f25e 100644
@@ -124,10 +124,9 @@ TRACE_EVENT(mm_vmscan_balance_pgdat_begin,
 
 TRACE_EVENT(mm_vmscan_balance_pgdat_end,
 
-	TP_PROTO(int nid, int order, int highest_zoneidx,
-		 unsigned long nr_reclaimed),
+	TP_PROTO(int nid, struct scan_control *sc, int highest_zoneidx),
 
-	TP_ARGS(nid, order, highest_zoneidx, nr_reclaimed),
+	TP_ARGS(nid, sc, highest_zoneidx),
 
 	TP_STRUCT__entry(
 		__field(int, nid)
@@ -138,9 +137,9 @@ TRACE_EVENT(mm_vmscan_balance_pgdat_end,
 
 	TP_fast_assign(
 		__entry->nid = nid;
-		__entry->order = order;
+		__entry->order = sc->order;
 		__entry->highest_zoneidx = highest_zoneidx;
-		__entry->nr_reclaimed = nr_reclaimed;
+		__entry->nr_reclaimed = sc->nr_reclaimed;
 	),
 
 	TP_printk("nid=%d order=%d highest_zoneidx=%-8s nr_reclaimed=%lu",
@@ -178,9 +177,9 @@ TRACE_EVENT(mm_vmscan_wakeup_kswapd,
 
 DECLARE_EVENT_CLASS(mm_vmscan_direct_reclaim_begin_template,
 
-	TP_PROTO(gfp_t gfp_flags, int order, struct mem_cgroup *memcg),
+	TP_PROTO(struct scan_control *sc, struct mem_cgroup *memcg),
 
-	TP_ARGS(gfp_flags, order, memcg),
+	TP_ARGS(sc, memcg),
 
 	TP_STRUCT__entry(
 		__field(	unsigned long,	gfp_flags	)
@@ -189,8 +188,8 @@ DECLARE_EVENT_CLASS(mm_vmscan_direct_reclaim_begin_template,
 	),
 
 	TP_fast_assign(
-		__entry->gfp_flags	= (__force unsigned long)gfp_flags;
-		__entry->order		= order;
+		__entry->gfp_flags	= (__force unsigned long)sc->gfp_mask;
+		__entry->order		= sc->order;
 		__entry->memcg_id	= mem_cgroup_id(memcg);
 	),
 
@@ -204,17 +203,17 @@ DECLARE_EVENT_CLASS(mm_vmscan_direct_reclaim_begin_template,
 
 DEFINE_EVENT(mm_vmscan_direct_reclaim_begin_template, mm_vmscan_direct_reclaim_begin,
 
-	TP_PROTO(gfp_t gfp_flags, int order, struct mem_cgroup *memcg),
+	TP_PROTO(struct scan_control *sc, struct mem_cgroup *memcg),
 
-	TP_ARGS(gfp_flags, order, memcg)
+	TP_ARGS(sc, memcg)
 );
 
 #ifdef CONFIG_MEMCG
 DEFINE_EVENT(mm_vmscan_direct_reclaim_begin_template, mm_vmscan_memcg_reclaim_begin,
 
-	TP_PROTO(gfp_t gfp_flags, int order, struct mem_cgroup *memcg),
+	TP_PROTO(struct scan_control *sc, struct mem_cgroup *memcg),
 
-	TP_ARGS(gfp_flags, order, memcg)
+	TP_ARGS(sc, memcg)
 );
 #endif /* CONFIG_MEMCG */
 
@@ -350,15 +349,14 @@ TRACE_EVENT(mm_shrink_slab_end,
 );
 
 TRACE_EVENT(mm_vmscan_lru_isolate,
-	TP_PROTO(int highest_zoneidx,
-		int order,
-		unsigned long nr_requested,
-		unsigned long nr_scanned,
-		unsigned long nr_skipped,
-		unsigned long nr_taken,
-		int lru),
+	TP_PROTO(struct scan_control *sc,
+		 unsigned long nr_requested,
+		 unsigned long nr_scanned,
+		 unsigned long nr_skipped,
+		 unsigned long nr_taken,
+		 int lru),
 
-	TP_ARGS(highest_zoneidx, order, nr_requested, nr_scanned, nr_skipped, nr_taken, lru),
+	TP_ARGS(sc, nr_requested, nr_scanned, nr_skipped, nr_taken, lru),
 
 	TP_STRUCT__entry(
 		__field(int, highest_zoneidx)
@@ -371,8 +369,8 @@ TRACE_EVENT(mm_vmscan_lru_isolate,
 	),
 
 	TP_fast_assign(
-		__entry->highest_zoneidx = highest_zoneidx;
-		__entry->order = order;
+		__entry->highest_zoneidx = sc->reclaim_idx;
+		__entry->order = sc->order;
 		__entry->nr_requested = nr_requested;
 		__entry->nr_scanned = nr_scanned;
 		__entry->nr_skipped = nr_skipped;
@@ -419,11 +417,11 @@ TRACE_EVENT(mm_vmscan_write_folio,
 
 TRACE_EVENT(mm_vmscan_reclaim_pages,
 
-	TP_PROTO(int nid,
-		unsigned long nr_scanned, unsigned long nr_reclaimed,
-		struct reclaim_stat *stat),
+	TP_PROTO(int nid, struct scan_control *sc,
+		 unsigned long nr_reclaimed,
+		 struct reclaim_stat *stat),
 
-	TP_ARGS(nid, nr_scanned, nr_reclaimed, stat),
+	TP_ARGS(nid, sc, nr_reclaimed, stat),
 
 	TP_STRUCT__entry(
 		__field(int, nid)
@@ -441,7 +439,7 @@ TRACE_EVENT(mm_vmscan_reclaim_pages,
 
 	TP_fast_assign(
 		__entry->nid = nid;
-		__entry->nr_scanned = nr_scanned;
+		__entry->nr_scanned = sc->nr_scanned;
 		__entry->nr_reclaimed = nr_reclaimed;
 		__entry->nr_dirty = stat->nr_dirty;
 		__entry->nr_writeback = stat->nr_writeback;
@@ -466,9 +464,9 @@ TRACE_EVENT(mm_vmscan_lru_shrink_inactive,
 
 	TP_PROTO(int nid,
 		unsigned long nr_scanned, unsigned long nr_reclaimed,
-		struct reclaim_stat *stat, int priority, int file),
+		struct reclaim_stat *stat, struct scan_control *sc, int file),
 
-	TP_ARGS(nid, nr_scanned, nr_reclaimed, stat, priority, file),
+	TP_ARGS(nid, nr_scanned, nr_reclaimed, stat, sc, file),
 
 	TP_STRUCT__entry(
 		__field(int, nid)
@@ -498,7 +496,7 @@ TRACE_EVENT(mm_vmscan_lru_shrink_inactive,
 		__entry->nr_activate1 = stat->nr_activate[1];
 		__entry->nr_ref_keep = stat->nr_ref_keep;
 		__entry->nr_unmap_fail = stat->nr_unmap_fail;
-		__entry->priority = priority;
+		__entry->priority = sc->priority;
 		__entry->reclaim_flags = trace_reclaim_flags(file);
 	),
 
@@ -517,9 +515,9 @@ TRACE_EVENT(mm_vmscan_lru_shrink_active,
 
 	TP_PROTO(int nid, unsigned long nr_taken,
 		unsigned long nr_active, unsigned long nr_deactivated,
-		unsigned long nr_referenced, int priority, int file),
+		unsigned long nr_referenced, struct scan_control *sc, int file),
 
-	TP_ARGS(nid, nr_taken, nr_active, nr_deactivated, nr_referenced, priority, file),
+	TP_ARGS(nid, nr_taken, nr_active, nr_deactivated, nr_referenced, sc, file),
 
 	TP_STRUCT__entry(
 		__field(int, nid)
@@ -537,7 +535,7 @@ TRACE_EVENT(mm_vmscan_lru_shrink_active,
 		__entry->nr_active = nr_active;
 		__entry->nr_deactivated = nr_deactivated;
 		__entry->nr_referenced = nr_referenced;
-		__entry->priority = priority;
+		__entry->priority = sc->priority;
 		__entry->reclaim_flags = trace_reclaim_flags(file);
 	),
 
@@ -551,9 +549,9 @@ TRACE_EVENT(mm_vmscan_lru_shrink_active,
 
 TRACE_EVENT(mm_vmscan_node_reclaim_begin,
 
-	TP_PROTO(int nid, int order, gfp_t gfp_flags),
+	TP_PROTO(int nid, struct scan_control *sc),
 
-	TP_ARGS(nid, order, gfp_flags),
+	TP_ARGS(nid, sc),
 
 	TP_STRUCT__entry(
 		__field(int, nid)
@@ -563,8 +561,8 @@ TRACE_EVENT(mm_vmscan_node_reclaim_begin,
 
 	TP_fast_assign(
 		__entry->nid = nid;
-		__entry->order = order;
-		__entry->gfp_flags = (__force unsigned long)gfp_flags;
+		__entry->order = sc->order;
+		__entry->gfp_flags = (__force unsigned long)sc->gfp_mask;
 	),
 
 	TP_printk("nid=%d order=%d gfp_flags=%s",
mm/vmscan.c +12 -14
index 88d9d15da893..f3ec19352e6d 100644
@@ -1764,7 +1764,7 @@ static unsigned long isolate_lru_folios(unsigned long nr_to_scan,
 		}
 	}
 	*nr_scanned = total_scan;
-	trace_mm_vmscan_lru_isolate(sc->reclaim_idx, sc->order, nr_to_scan,
+	trace_mm_vmscan_lru_isolate(sc, nr_to_scan,
 				    total_scan, skipped, nr_taken, lru);
 	update_lru_sizes(lruvec, lru, nr_zone_taken);
 	return nr_taken;
@@ -2054,7 +2054,7 @@ static unsigned long shrink_inactive_list(unsigned long nr_to_scan,
 
 	handle_reclaim_writeback(nr_taken, pgdat, sc, &stat);
 	trace_mm_vmscan_lru_shrink_inactive(pgdat->node_id,
-			nr_scanned, nr_reclaimed, &stat, sc->priority, file);
+			nr_scanned, nr_reclaimed, &stat, sc, file);
 	return nr_reclaimed;
 }
 
@@ -2161,7 +2161,7 @@ static void shrink_active_list(unsigned long nr_to_scan,
 		mod_lruvec_state(lruvec, PGROTATE_ANON + file, nr_rotated);
 
 	trace_mm_vmscan_lru_shrink_active(pgdat->node_id, nr_taken, nr_activate,
-			nr_deactivate, nr_rotated, sc->priority, file);
+			nr_deactivate, nr_rotated, sc, file);
 }
 
 static unsigned int reclaim_folio_list(struct list_head *folio_list,
@@ -2184,7 +2184,7 @@ static unsigned int reclaim_folio_list(struct list_head *folio_list,
 		list_del(&folio->lru);
 		folio_putback_lru(folio);
 	}
-	trace_mm_vmscan_reclaim_pages(pgdat->node_id, sc.nr_scanned, nr_reclaimed, &stat);
+	trace_mm_vmscan_reclaim_pages(pgdat->node_id, &sc, nr_reclaimed, &stat);
 
 	return nr_reclaimed;
 }
@@ -4830,9 +4830,9 @@ static int scan_folios(unsigned long nr_to_scan, struct lruvec *lruvec,
 	mod_lruvec_state(lruvec, item, isolated);
 	mod_lruvec_state(lruvec, PGREFILL, sorted);
 	mod_lruvec_state(lruvec, PGSCAN_ANON + type, isolated);
-	trace_mm_vmscan_lru_isolate(sc->reclaim_idx, sc->order, nr_to_scan,
-				scanned, skipped, isolated,
-				type ? LRU_INACTIVE_FILE : LRU_INACTIVE_ANON);
+	trace_mm_vmscan_lru_isolate(sc, nr_to_scan,
+				    scanned, skipped, isolated,
+				    type ? LRU_INACTIVE_FILE : LRU_INACTIVE_ANON);
 
 	*isolatedp = isolated;
 	return scanned;
@@ -4974,7 +4974,7 @@ static int evict_folios(unsigned long nr_to_scan, struct lruvec *lruvec,
 	if (isolated)
 		handle_reclaim_writeback(isolated, pgdat, sc, &stat);
 	trace_mm_vmscan_lru_shrink_inactive(pgdat->node_id,
-			type_scanned, reclaimed, &stat, sc->priority,
+			type_scanned, reclaimed, &stat, sc,
 			type ? LRU_INACTIVE_FILE : LRU_INACTIVE_ANON);
 
 	list_for_each_entry_safe_reverse(folio, next, &list, lru) {
@@ -6870,7 +6870,7 @@ unsigned long try_to_free_pages(struct zonelist *zonelist, int order,
 		return 1;
 
 	set_task_reclaim_state(current, &sc.reclaim_state);
-	trace_mm_vmscan_direct_reclaim_begin(sc.gfp_mask, order, NULL);
+	trace_mm_vmscan_direct_reclaim_begin(&sc, NULL);
 
 	nr_reclaimed = do_try_to_free_pages(zonelist, &sc);
 
@@ -6911,7 +6911,7 @@ unsigned long try_to_free_mem_cgroup_pages(struct mem_cgroup *memcg,
 	struct zonelist *zonelist = node_zonelist(numa_node_id(), sc.gfp_mask);
 
 	set_task_reclaim_state(current, &sc.reclaim_state);
-	trace_mm_vmscan_memcg_reclaim_begin(sc.gfp_mask, 0, memcg);
+	trace_mm_vmscan_memcg_reclaim_begin(&sc, memcg);
 	noreclaim_flag = memalloc_noreclaim_save();
 
 	nr_reclaimed = do_try_to_free_pages(zonelist, &sc);
@@ -7388,8 +7388,7 @@ static int balance_pgdat(pg_data_t *pgdat, int order, int highest_zoneidx)
 	psi_memstall_leave(&pflags);
 	set_task_reclaim_state(current, NULL);
 
-	trace_mm_vmscan_balance_pgdat_end(pgdat->node_id, sc.order,
-					  highest_zoneidx, sc.nr_reclaimed);
+	trace_mm_vmscan_balance_pgdat_end(pgdat->node_id, &sc, highest_zoneidx);
 
 	/*
 	 * Return the order kswapd stopped reclaiming at as
@@ -7867,8 +7866,7 @@ static unsigned long __node_reclaim(struct pglist_data *pgdat,
 	unsigned int noreclaim_flag;
 	unsigned long pflags;
 
-	trace_mm_vmscan_node_reclaim_begin(pgdat->node_id, sc->order,
-					   sc->gfp_mask);
+	trace_mm_vmscan_node_reclaim_begin(pgdat->node_id, sc);
 
 	cond_resched();
 	if (!sc->proactive)
Signature · 1 line
2.34.1
#5 lore raw

PATCH v6 5/7 mm/mglru: add tracepoint for scan_folios()

In reply to Ridong Chen

From: Ridong Chen <[email protected]>

MGLRU's scan_folios() emits the classic-LRU tracepoint trace_mm_vmscan_lru_isolate(), which predates MGLRU. It reports the scan/isolate counts and the LRU type, but carries no generation or memcg context, so a trace of an MGLRU run cannot tell which memcg a given scan belongs to, nor how far reclaim has progressed through the generations.

Add mm_mglru_scan_folios next to it, reporting the same counters plus nr_sorted (folios moved to a younger generation by sort_folio()) and the context the classic tracepoint lacks: the node id and memcg id of the lruvec, and the max_seq, tier and min_seq of the type being scanned.

Now that struct scan_control is visible to the trace header, pass sc, lruvec, type and tier straight in and derive everything else in TP_fast_assign(), so nothing is computed at the call site when the event is disabled. The int fields are grouped at the end of the entry to avoid padding on 64-bit.

scan_folios() is the MGLRU-specific layer where folios are actually scanned, so it has no classic-LRU counterpart. The existing tracepoint is left unchanged.

---
 mm/trace_vmscan.h | 66 +++++++++++++++++++++++++++++++++++++++++++++++
 mm/vmscan.c       |  2 ++
 2 files changed, 68 insertions(+)
mm/trace_vmscan.h +66 -0
index b2ad83e5f25e..8148b8224422 100644
@@ -392,6 +392,72 @@ TRACE_EVENT(mm_vmscan_lru_isolate,
 		__print_symbolic(__entry->lru, LRU_NAMES))
 );
 
+#ifdef CONFIG_LRU_GEN
+TRACE_EVENT(mm_mglru_scan_folios,
+
+	TP_PROTO(struct lruvec *lruvec,
+		 struct scan_control *sc,
+		 unsigned long nr_requested,
+		 unsigned long nr_scanned,
+		 unsigned long nr_sorted,
+		 unsigned long nr_skipped,
+		 unsigned long nr_taken,
+		 int type,
+		 int tier),
+
+	TP_ARGS(lruvec, sc, nr_requested, nr_scanned, nr_sorted, nr_skipped,
+		nr_taken, type, tier),
+
+	TP_STRUCT__entry(
+		__field(u64, memcg_id)
+		__field(unsigned long, nr_requested)
+		__field(unsigned long, nr_scanned)
+		__field(unsigned long, nr_sorted)
+		__field(unsigned long, nr_skipped)
+		__field(unsigned long, nr_taken)
+		__field(unsigned long, max_seq)
+		__field(unsigned long, min_seq)
+		__field(int, nid)
+		__field(int, highest_zoneidx)
+		__field(int, order)
+		__field(int, lru)
+		__field(int, tier)
+	),
+
+	TP_fast_assign(
+		__entry->memcg_id = mem_cgroup_id(lruvec_memcg(lruvec));
+		__entry->nr_requested = nr_requested;
+		__entry->nr_scanned = nr_scanned;
+		__entry->nr_sorted = nr_sorted;
+		__entry->nr_skipped = nr_skipped;
+		__entry->nr_taken = nr_taken;
+		__entry->max_seq = lruvec->lrugen.max_seq;
+		__entry->min_seq = lruvec->lrugen.min_seq[type];
+		__entry->nid = lruvec_pgdat(lruvec)->node_id;
+		__entry->highest_zoneidx = sc->reclaim_idx;
+		__entry->order = sc->order;
+		__entry->lru = type ? LRU_INACTIVE_FILE : LRU_INACTIVE_ANON;
+		__entry->tier = tier;
+	),
+
+	TP_printk("nid=%d memcg_id=%llu classzone=%d order=%d nr_requested=%lu nr_scanned=%lu nr_sorted=%lu nr_skipped=%lu nr_taken=%lu lru=%s max_seq=%lu tier=%d min_seq=%lu",
+		__entry->nid,
+		__entry->memcg_id,
+		__entry->highest_zoneidx,
+		__entry->order,
+		__entry->nr_requested,
+		__entry->nr_scanned,
+		__entry->nr_sorted,
+		__entry->nr_skipped,
+		__entry->nr_taken,
+		__print_symbolic(__entry->lru, LRU_NAMES),
+		__entry->max_seq,
+		__entry->tier,
+		__entry->min_seq)
+);
+
+#endif /* CONFIG_LRU_GEN */
+
 TRACE_EVENT(mm_vmscan_write_folio,
 
 	TP_PROTO(struct folio *folio),
mm/vmscan.c +2 -0
index f3ec19352e6d..3054175717ab 100644
@@ -4833,6 +4833,8 @@ static int scan_folios(unsigned long nr_to_scan, struct lruvec *lruvec,
 	trace_mm_vmscan_lru_isolate(sc, nr_to_scan,
 				    scanned, skipped, isolated,
 				    type ? LRU_INACTIVE_FILE : LRU_INACTIVE_ANON);
+	trace_mm_mglru_scan_folios(lruvec, sc, nr_to_scan, scanned, sorted,
+				   skipped, isolated, type, tier);
 
 	*isolatedp = isolated;
 	return scanned;
Signature · 1 line
2.34.1
#6 lore raw

PATCH v6 6/7 mm/mglru: add lruvec_gen_size() helper

In reply to Ridong Chen

From: Ridong Chen <[email protected]>

The number of pages of a given type in a generation is computed by summing lrugen->nr_pages[gen][type][] over all zones and clamping each per-zone counter to >= 0, since the counters are updated locklessly and can transiently go negative. This loop is open-coded in lruvec_evictable_size() and in the debugfs lru_gen output.

Add lruvec_gen_size() to compute it from a seq, and convert both users. Put it in mm/vmscan.h so that the vmscan tracepoints, which include that header, can use it as well.
An upcoming tracepoint and the MGLRU swappiness rework will also use it.

No functional change.

---
 mm/vmscan.c | 18 ++++++------------
 mm/vmscan.h | 16 ++++++++++++++++
 2 files changed, 22 insertions(+), 12 deletions(-)
mm/vmscan.c +6 -12
index 3054175717ab..1581b73aec56 100644
@@ -4207,18 +4207,15 @@ static void set_initial_priority(struct pglist_data *pgdat, struct scan_control
 
 static unsigned long lruvec_evictable_size(struct lruvec *lruvec, int swappiness)
 {
-	int gen, type, zone;
+	int type;
 	unsigned long seq, total = 0;
 	struct lru_gen_folio *lrugen = &lruvec->lrugen;
 	DEFINE_MAX_SEQ(lruvec);
 	DEFINE_MIN_SEQ(lruvec);
 
 	for_each_evictable_type(type, swappiness) {
-		for (seq = min_seq[type]; seq <= max_seq; seq++) {
-			gen = lru_gen_from_seq(seq);
-			for (zone = 0; zone < MAX_NR_ZONES; zone++)
-				total += max(READ_ONCE(lrugen->nr_pages[gen][type][zone]), 0L);
-		}
+		for (seq = min_seq[type]; seq <= max_seq; seq++)
+			total += lruvec_gen_size(lrugen, type, seq);
 	}
 
 	return total;
@@ -5723,20 +5720,17 @@ static int lru_gen_seq_show(struct seq_file *m, void *v)
 		seq = 0;
 
 	for (; seq <= max_seq; seq++) {
-		int type, zone;
+		int type;
 		int gen = lru_gen_from_seq(seq);
 		unsigned long birth = READ_ONCE(lruvec->lrugen.timestamps[gen]);
 
 		seq_printf(m, " %10lu %10u", seq, jiffies_to_msecs(jiffies - birth));
 
 		for (type = 0; type < ANON_AND_FILE; type++) {
-			unsigned long size = 0;
 			char mark = full && seq < min_seq[type] ? 'x' : ' ';
 
-			for (zone = 0; zone < MAX_NR_ZONES; zone++)
-				size += max(READ_ONCE(lrugen->nr_pages[gen][type][zone]), 0L);
-
-			seq_printf(m, " %10lu%c", size, mark);
+			seq_printf(m, " %10lu%c",
+				   lruvec_gen_size(lrugen, type, seq), mark);
 		}
 
 		seq_putc(m, '\n');
mm/vmscan.h +16 -0
index adb22e80d760..c6b583fc7493 100644
@@ -3,6 +3,7 @@
 #define _MM_VMSCAN_H
 
 #include <linux/gfp_types.h>
+#include <linux/mm_inline.h>
 #include <linux/nodemask.h>
 #include <linux/swap.h>
 #include <linux/types.h>
@@ -115,4 +116,19 @@ struct scan_control {
 	struct reclaim_state reclaim_state;
 };
 
+#ifdef CONFIG_LRU_GEN
+/* The number of pages of @type in the generation @seq, summed over all zones */
+static inline unsigned long lruvec_gen_size(struct lru_gen_folio *lrugen,
+					    int type, unsigned long seq)
+{
+	int gen = lru_gen_from_seq(seq);
+	unsigned long size = 0;
+
+	for (int zone = 0; zone < MAX_NR_ZONES; zone++)
+		size += max(READ_ONCE(lrugen->nr_pages[gen][type][zone]), 0L);
+
+	return size;
+}
+#endif /* CONFIG_LRU_GEN */
+
 #endif /* _MM_VMSCAN_H */
Signature · 1 line
2.34.1
reviewer Reviewed-by on v6 6/7
#7 lore raw

Re: [PATCH v6 6/7] mm/mglru: add lruvec_gen_size() helper

Quoted 13 lines · Ridong Chen

From: Ridong Chen <[email protected]>

The number of pages of a given type in a generation is computed by summing lrugen->nr_pages[gen][type][] over all zones and clamping each per-zone counter to >= 0, since the counters are updated locklessly and can transiently go negative. This loop is open-coded in lruvec_evictable_size() and in the debugfs lru_gen output.

Add lruvec_gen_size() to compute it from a seq, and convert both users. Put it in mm/vmscan.h so that the vmscan tracepoints, which include that header, can use it as well.
An upcoming tracepoint and the MGLRU swappiness rework will also use it.

Quoted 8 lines

No functional change.

---
 mm/vmscan.c | 18 ++++++------------
 mm/vmscan.h | 16 ++++++++++++++++
 2 files changed, 22 insertions(+), 12 deletions(-)

LGTM,

Quoted 86 lines
mm/vmscan.c +6 -12
index 3054175717ab..1581b73aec56 100644
@@ -4207,18 +4207,15 @@ static void set_initial_priority(struct pglist_data *pgdat, struct scan_control
 
 static unsigned long lruvec_evictable_size(struct lruvec *lruvec, int swappiness)
 {
-	int gen, type, zone;
+	int type;
 	unsigned long seq, total = 0;
 	struct lru_gen_folio *lrugen = &lruvec->lrugen;
 	DEFINE_MAX_SEQ(lruvec);
 	DEFINE_MIN_SEQ(lruvec);
 
 	for_each_evictable_type(type, swappiness) {
-		for (seq = min_seq[type]; seq <= max_seq; seq++) {
-			gen = lru_gen_from_seq(seq);
-			for (zone = 0; zone < MAX_NR_ZONES; zone++)
-				total += max(READ_ONCE(lrugen->nr_pages[gen][type][zone]), 0L);
-		}
+		for (seq = min_seq[type]; seq <= max_seq; seq++)
+			total += lruvec_gen_size(lrugen, type, seq);
 	}
 
 	return total;
@@ -5723,20 +5720,17 @@ static int lru_gen_seq_show(struct seq_file *m, void *v)
 		seq = 0;
 
 	for (; seq <= max_seq; seq++) {
-		int type, zone;
+		int type;
 		int gen = lru_gen_from_seq(seq);
 		unsigned long birth = READ_ONCE(lruvec->lrugen.timestamps[gen]);
 
 		seq_printf(m, " %10lu %10u", seq, jiffies_to_msecs(jiffies - birth));
 
 		for (type = 0; type < ANON_AND_FILE; type++) {
-			unsigned long size = 0;
 			char mark = full && seq < min_seq[type] ? 'x' : ' ';
 
-			for (zone = 0; zone < MAX_NR_ZONES; zone++)
-				size += max(READ_ONCE(lrugen->nr_pages[gen][type][zone]), 0L);
-
-			seq_printf(m, " %10lu%c", size, mark);
+			seq_printf(m, " %10lu%c",
+				   lruvec_gen_size(lrugen, type, seq), mark);
 		}
 
 		seq_putc(m, '\n');
mm/vmscan.h +16 -0
index adb22e80d760..c6b583fc7493 100644
@@ -3,6 +3,7 @@
 #define _MM_VMSCAN_H
 
 #include <linux/gfp_types.h>
+#include <linux/mm_inline.h>
 #include <linux/nodemask.h>
 #include <linux/swap.h>
 #include <linux/types.h>
@@ -115,4 +116,19 @@ struct scan_control {
 	struct reclaim_state reclaim_state;
 };
 
+#ifdef CONFIG_LRU_GEN
+/* The number of pages of @type in the generation @seq, summed over all zones */
+static inline unsigned long lruvec_gen_size(struct lru_gen_folio *lrugen,
+					    int type, unsigned long seq)
+{
+	int gen = lru_gen_from_seq(seq);
+	unsigned long size = 0;
+
+	for (int zone = 0; zone < MAX_NR_ZONES; zone++)
+		size += max(READ_ONCE(lrugen->nr_pages[gen][type][zone]), 0L);
+
+	return size;
+}
+#endif /* CONFIG_LRU_GEN */
+
 #endif /* _MM_VMSCAN_H */
Signature · 1 line
2.34.1
reviewer Reviewed-by on v6 6/7
#8 lore raw

Re: [PATCH v6 6/7] mm/mglru: add lruvec_gen_size() helper

In reply to Ridong Chen
Quoted 14 lines · Ridong Chen

From: Ridong Chen <[email protected]>

The number of pages of a given type in a generation is computed by summing lrugen->nr_pages[gen][type][] over all zones and clamping each per-zone counter to >= 0, since the counters are updated locklessly and can transiently go negative. This loop is open-coded in lruvec_evictable_size() and in the debugfs lru_gen output.

Add lruvec_gen_size() to compute it from a seq, and convert both users. Put it in mm/vmscan.h so that the vmscan tracepoints, which include that header, can use it as well.
An upcoming tracepoint and the MGLRU swappiness rework will also use it.

Quoted 5 lines

No functional change.

---

#9 lore raw

PATCH v6 7/7 mm/mglru: add tracepoint for inc_max_seq()

In reply to Ridong Chen

From: Ridong Chen <[email protected]>

Aging in MGLRU advances max_seq via inc_max_seq(), creating a new youngest generation. There is currently no tracepoint on this path, so the moment a new generation is created, and how the min_seq of each type trails behind it, cannot be observed as it happens.

Add mm_mglru_inc_max_seq, emitted right after max_seq is bumped while the lruvec lock is still held, with the node id, memcg id, the new max_seq, the anon and file min_seq, and the number of pages in each generation for both types, as computed by lruvec_gen_size().

The nr_anon/nr_file arrays are ordered by seq rather than by generation
slot: entry i holds the generation max_seq - MAX_NR_GENS + 1 + i, i.e. they go from the oldest generation to the newest, the same order as the debugfs lru_gen file, so the output can be read directly without mapping slots back to seqs. Entries for seqs below the corresponding min_seq are not in use and normally read as 0. The arrays are printed with __print_array(), so the output is hex and its length follows MAX_NR_GENS automatically.

Only the lruvec is passed in; all of the above is collected in TP_fast_assign(), so inc_max_seq() does no extra work when the event is disabled. Paired with mm_mglru_scan_folios it makes the full aging-to-eviction window observable per memcg.

---
 mm/trace_vmscan.h | 46 ++++++++++++++++++++++++++++++++++++++++++++++
 mm/vmscan.c       |  2 ++
 2 files changed, 48 insertions(+)
mm/trace_vmscan.h +46 -0
index 8148b8224422..40d6d90f1d7a 100644
@@ -456,6 +456,52 @@ TRACE_EVENT(mm_mglru_scan_folios,
 		__entry->min_seq)
 );
 
+TRACE_EVENT(mm_mglru_inc_max_seq,
+
+	TP_PROTO(struct lruvec *lruvec),
+
+	TP_ARGS(lruvec),
+
+	TP_STRUCT__entry(
+		__field(u64, memcg_id)
+		__field(unsigned long, max_seq)
+		__field(unsigned long, anon_min_seq)
+		__field(unsigned long, file_min_seq)
+		__array(unsigned long, nr_anon, MAX_NR_GENS)
+		__array(unsigned long, nr_file, MAX_NR_GENS)
+		__field(int, nid)
+	),
+
+	TP_fast_assign(
+		struct lru_gen_folio *lrugen = &lruvec->lrugen;
+		unsigned long max_seq = lrugen->max_seq;
+		int i;
+
+		__entry->memcg_id = mem_cgroup_id(lruvec_memcg(lruvec));
+		__entry->max_seq = max_seq;
+		__entry->anon_min_seq = lrugen->min_seq[LRU_GEN_ANON];
+		__entry->file_min_seq = lrugen->min_seq[LRU_GEN_FILE];
+		/* from the oldest generation to the newest, as debugfs lru_gen does */
+		for (i = 0; i < MAX_NR_GENS; i++) {
+			unsigned long seq = max_seq - MAX_NR_GENS + 1 + i;
+
+			__entry->nr_anon[i] =
+				lruvec_gen_size(lrugen, LRU_GEN_ANON, seq);
+			__entry->nr_file[i] =
+				lruvec_gen_size(lrugen, LRU_GEN_FILE, seq);
+		}
+		__entry->nid = lruvec_pgdat(lruvec)->node_id;
+	),
+
+	TP_printk("nid=%d memcg_id=%llu max_seq=%lu anon_min_seq=%lu file_min_seq=%lu nr_anon=%s nr_file=%s",
+		__entry->nid,
+		__entry->memcg_id,
+		__entry->max_seq,
+		__entry->anon_min_seq,
+		__entry->file_min_seq,
+		__print_array(__entry->nr_anon, MAX_NR_GENS, sizeof(unsigned long)),
+		__print_array(__entry->nr_file, MAX_NR_GENS, sizeof(unsigned long)))
+);
 #endif /* CONFIG_LRU_GEN */
 
 TRACE_EVENT(mm_vmscan_write_folio,
mm/vmscan.c +2 -0
index 1581b73aec56..841fefa68d1f 100644
@@ -4115,6 +4115,8 @@ static bool inc_max_seq(struct lruvec *lruvec, unsigned long seq, int swappiness
 	WRITE_ONCE(lrugen->timestamps[next], jiffies);
 	/* make sure preceding modifications appear */
 	smp_store_release(&lrugen->max_seq, lrugen->max_seq + 1);
+
+	trace_mm_mglru_inc_max_seq(lruvec);
 unlock:
 	lruvec_unlock_irq(lruvec);
 
Signature · 1 line
2.34.1
reviewer
#10 lore raw

Re: [PATCH v6 7/7] mm/mglru: add tracepoint for inc_max_seq()

Quoted 7 lines · Ridong Chen

From: Ridong Chen <[email protected]>

Aging in MGLRU advances max_seq via inc_max_seq(), creating a new youngest generation. There is currently no tracepoint on this path, so the moment a new generation is created, and how the min_seq of each type trails behind it, cannot be observed as it happens.

We might be more interested in why aging is triggered rather than the exact moment when max_seq is advanced.

With your patch, we get:

~ # cat /sys/kernel/tracing/trace | grep mglru_inc_max_seq

           a.out-93      [001] d..1.   278.276539:
mm_mglru_inc_max_seq: nid=0 memcg_id=1 max_seq=4 anon_min_seq=2
file_min_seq=1 nr_anon={0x0,0xf016,0x12,0x0}
nr_file={0x1886,0xb,0x4c,0x0}
         kswapd0-50      [000] d..1.   278.422400:
mm_mglru_inc_max_seq: nid=0 memcg_id=1 max_seq=5 anon_min_seq=2
file_min_seq=2 nr_anon={0x5,0xccb6,0x3439,0x0}
nr_file={0x4ae,0x4f,0x55,0x0}
         kswapd0-50      [000] d..1.   278.508003:
mm_mglru_inc_max_seq: nid=0 memcg_id=1 max_seq=6 anon_min_seq=3
file_min_seq=4 nr_anon={0xccb6,0x3d29,0x1,0x0}
nr_file={0x0,0x59,0x7,0x0}
         kswapd0-50      [000] d..1.   280.046056:
mm_mglru_inc_max_seq: nid=0 memcg_id=1 max_seq=7 anon_min_seq=4
file_min_seq=5 nr_anon={0x3bf3,0xcbbd,0x23e,0x0}
nr_file={0x0,0xb,0x1,0x0}
         kswapd0-50      [000] d..1.   280.124876:
mm_mglru_inc_max_seq: nid=0 memcg_id=1 max_seq=8 anon_min_seq=5
file_min_seq=6 nr_anon={0xcafd,0x3e31,0x0,0x0}
nr_file={0x0,0x5,0x7,0x0}
           a.out-96      [001] d..1.   290.629014:
mm_mglru_inc_max_seq: nid=0 memcg_id=1 max_seq=9 anon_min_seq=7
file_min_seq=6 nr_anon={0x0,0x10615,0xb,0x0}
nr_file={0x331,0xb,0x9a,0x0}
         kswapd0-50      [000] d..1.   290.653098:
mm_mglru_inc_max_seq: nid=0 memcg_id=1 max_seq=10 anon_min_seq=7
file_min_seq=7 nr_anon={0x15,0xef23,0x18c4,0x0}
nr_file={0xb,0xb8,0x55,0x0}
           a.out-96      [001] d..1.   290.684875:

mm_mglru_inc_max_seq: nid=0 memcg_id=1 max_seq=11 anon_min_seq=8 file_min_seq=9 nr_anon={0xeee3,0x1afb,0x2,0x0} nr_file={0x0,0x76,0x4,0x0}

You can see that the youngest generation is always zero, because you're just getting the newly added youngest generation :-)

This somehow suggests that the information is redundant, as there is always a zero there.

I suspect we'll need an additional tracepoint to show why aging is triggered and at what sc->priority it happens.

But do you know why you trace the moment just after aging is completed, rather than the reason why aging is needed?

Best,
Regards

#11 lore raw

Re: [PATCH v6 7/7] mm/mglru: add tracepoint for inc_max_seq()

Quoted 44 lines · Barry Song
Quoted 7 lines · Ridong Chen

From: Ridong Chen <[email protected]>

Aging in MGLRU advances max_seq via inc_max_seq(), creating a new youngest generation. There is currently no tracepoint on this path, so the moment a new generation is created, and how the min_seq of each type trails behind it, cannot be observed as it happens.

We might be more interested in why aging is triggered rather than the exact moment when max_seq is advanced.

With your patch, we get:

~ # cat /sys/kernel/tracing/trace | grep mglru_inc_max_seq

            a.out-93      [001] d..1.   278.276539:
mm_mglru_inc_max_seq: nid=0 memcg_id=1 max_seq=4 anon_min_seq=2
file_min_seq=1 nr_anon={0x0,0xf016,0x12,0x0}
nr_file={0x1886,0xb,0x4c,0x0}
          kswapd0-50      [000] d..1.   278.422400:
mm_mglru_inc_max_seq: nid=0 memcg_id=1 max_seq=5 anon_min_seq=2
file_min_seq=2 nr_anon={0x5,0xccb6,0x3439,0x0}
nr_file={0x4ae,0x4f,0x55,0x0}
          kswapd0-50      [000] d..1.   278.508003:
mm_mglru_inc_max_seq: nid=0 memcg_id=1 max_seq=6 anon_min_seq=3
file_min_seq=4 nr_anon={0xccb6,0x3d29,0x1,0x0}
nr_file={0x0,0x59,0x7,0x0}
          kswapd0-50      [000] d..1.   280.046056:
mm_mglru_inc_max_seq: nid=0 memcg_id=1 max_seq=7 anon_min_seq=4
file_min_seq=5 nr_anon={0x3bf3,0xcbbd,0x23e,0x0}
nr_file={0x0,0xb,0x1,0x0}
          kswapd0-50      [000] d..1.   280.124876:
mm_mglru_inc_max_seq: nid=0 memcg_id=1 max_seq=8 anon_min_seq=5
file_min_seq=6 nr_anon={0xcafd,0x3e31,0x0,0x0}
nr_file={0x0,0x5,0x7,0x0}
            a.out-96      [001] d..1.   290.629014:
mm_mglru_inc_max_seq: nid=0 memcg_id=1 max_seq=9 anon_min_seq=7
file_min_seq=6 nr_anon={0x0,0x10615,0xb,0x0}
nr_file={0x331,0xb,0x9a,0x0}
          kswapd0-50      [000] d..1.   290.653098:
mm_mglru_inc_max_seq: nid=0 memcg_id=1 max_seq=10 anon_min_seq=7
file_min_seq=7 nr_anon={0x15,0xef23,0x18c4,0x0}
nr_file={0xb,0xb8,0x55,0x0}
            a.out-96      [001] d..1.   290.684875:
Quoted 8 lines

mm_mglru_inc_max_seq: nid=0 memcg_id=1 max_seq=11 anon_min_seq=8 file_min_seq=9 nr_anon={0xeee3,0x1afb,0x2,0x0} nr_file={0x0,0x76,0x4,0x0}

You can see that the youngest generation is always zero, because you're just getting the newly added youngest generation :-)

Hi Barry,

Thank you very much for testing and reviewing.

Quoted 9 lines

This somehow suggests that the information is redundant, as there is always a zero there.

I suspect we'll need an additional tracepoint to show why aging is triggered and at what sc->priority it happens.

But do you know why you trace the moment just after aging is completed, rather than the reason why aging is needed?

My original goal was to pair it with mm_mglru_scan_folios, to see how the aging window of each memcg moves. The per-generation anon/file nr_pages were added following your suggestion on v1. I put it at the end of inc_max_seq() because of the restart there, as max_seq and min_seq are only final after that.

You make a very good point that the reason for aging is also important. So I'd like to discuss which way is better:

1) Keep mm_mglru_inc_max_seq, and add another tracepoint at the aging decision in try_to_shrink_lruvec(), reporting why aging is needed and at what sc->priority.

2) Replace mm_mglru_inc_max_seq with a single tracepoint in try_to_shrink_lruvec(), right before try_to_inc_max_seq(), reporting the reason, sc->priority, swappiness, max_seq/min_seq and the per-generation nr_pages before aging. The youngest generation is not empty at that point, and the new window after aging can still be seen from the following mm_mglru_scan_folios.

Which one do you prefer?

Signature · 2 lines
Best regards
Ridong
reviewer
#12 lore raw

Re: [PATCH v6 7/7] mm/mglru: add tracepoint for inc_max_seq()

Quoted 67 lines · Ridong Chen
Quoted 44 lines · Barry Song
Quoted 7 lines · Ridong Chen

From: Ridong Chen <[email protected]>

Aging in MGLRU advances max_seq via inc_max_seq(), creating a new youngest generation. There is currently no tracepoint on this path, so the moment a new generation is created, and how the min_seq of each type trails behind it, cannot be observed as it happens.

We might be more interested in why aging is triggered rather than the exact moment when max_seq is advanced.

With your patch, we get:

~ # cat /sys/kernel/tracing/trace | grep mglru_inc_max_seq

            a.out-93      [001] d..1.   278.276539:
mm_mglru_inc_max_seq: nid=0 memcg_id=1 max_seq=4 anon_min_seq=2
file_min_seq=1 nr_anon={0x0,0xf016,0x12,0x0}
nr_file={0x1886,0xb,0x4c,0x0}
          kswapd0-50      [000] d..1.   278.422400:
mm_mglru_inc_max_seq: nid=0 memcg_id=1 max_seq=5 anon_min_seq=2
file_min_seq=2 nr_anon={0x5,0xccb6,0x3439,0x0}
nr_file={0x4ae,0x4f,0x55,0x0}
          kswapd0-50      [000] d..1.   278.508003:
mm_mglru_inc_max_seq: nid=0 memcg_id=1 max_seq=6 anon_min_seq=3
file_min_seq=4 nr_anon={0xccb6,0x3d29,0x1,0x0}
nr_file={0x0,0x59,0x7,0x0}
          kswapd0-50      [000] d..1.   280.046056:
mm_mglru_inc_max_seq: nid=0 memcg_id=1 max_seq=7 anon_min_seq=4
file_min_seq=5 nr_anon={0x3bf3,0xcbbd,0x23e,0x0}
nr_file={0x0,0xb,0x1,0x0}
          kswapd0-50      [000] d..1.   280.124876:
mm_mglru_inc_max_seq: nid=0 memcg_id=1 max_seq=8 anon_min_seq=5
file_min_seq=6 nr_anon={0xcafd,0x3e31,0x0,0x0}
nr_file={0x0,0x5,0x7,0x0}
            a.out-96      [001] d..1.   290.629014:
mm_mglru_inc_max_seq: nid=0 memcg_id=1 max_seq=9 anon_min_seq=7
file_min_seq=6 nr_anon={0x0,0x10615,0xb,0x0}
nr_file={0x331,0xb,0x9a,0x0}
          kswapd0-50      [000] d..1.   290.653098:
mm_mglru_inc_max_seq: nid=0 memcg_id=1 max_seq=10 anon_min_seq=7
file_min_seq=7 nr_anon={0x15,0xef23,0x18c4,0x0}
nr_file={0xb,0xb8,0x55,0x0}
            a.out-96      [001] d..1.   290.684875:
Quoted 8 lines

mm_mglru_inc_max_seq: nid=0 memcg_id=1 max_seq=11 anon_min_seq=8 file_min_seq=9 nr_anon={0xeee3,0x1afb,0x2,0x0} nr_file={0x0,0x76,0x4,0x0}

You can see that the youngest generation is always zero, because you're just getting the newly added youngest generation :-)

Hi Barry,

Thank you very much for testing and reviewing.

Quoted 6 lines

This somehow suggests that the information is redundant, as there is always a zero there.

I suspect we'll need an additional tracepoint to show why aging is triggered and at what sc->priority it happens.

Quoted 8 lines
Quoted 3 lines

But do you know why you trace the moment just after aging is completed, rather than the reason why aging is needed?

My original goal was to pair it with mm_mglru_scan_folios, to see how the aging window of each memcg moves. The per-generation anon/file nr_pages were added following your suggestion on v1. I put it at the end of inc_max_seq() because of the restart there, as max_seq and min_seq are only final after that.

The per-generation anon/file nr_pages are perfectly good, as they can also help explain why aging is needed, especially now that we're introducing new code for imbalance-based aging.

Quoted 8 lines

You make a very good point that the reason for aging is also important. So I'd like to discuss which way is better:

1) Keep mm_mglru_inc_max_seq, and add another tracepoint at the aging decision in try_to_shrink_lruvec(), reporting why aging is needed and at what sc->priority.

Quoted 8 lines

2) Replace mm_mglru_inc_max_seq with a single tracepoint in try_to_shrink_lruvec(), right before try_to_inc_max_seq(), reporting the reason, sc->priority, swappiness, max_seq/min_seq and the per-generation nr_pages before aging. The youngest generation is not empty at that point, and the new window after aging can still be seen from the following mm_mglru_scan_folios.

Which one do you prefer?

Maybe we should keep option 2 for now. Understanding why we are aging and at what priority is much more important, especially since we're still introducing some imbalance-based aging code.

It should be quite simple, just a matter of moving the code around a little.

If we later find a good reason to have option 1, we can always add it, right?

Best,
Barry

#13 lore raw

Re: [PATCH v6 7/7] mm/mglru: add tracepoint for inc_max_seq()

Quoted 104 lines · Barry Song
Quoted 67 lines · Ridong Chen
Quoted 44 lines · Barry Song
Quoted 7 lines · Ridong Chen

From: Ridong Chen <[email protected]>

Aging in MGLRU advances max_seq via inc_max_seq(), creating a new youngest generation. There is currently no tracepoint on this path, so the moment a new generation is created, and how the min_seq of each type trails behind it, cannot be observed as it happens.

We might be more interested in why aging is triggered rather than the exact moment when max_seq is advanced.

With your patch, we get:

~ # cat /sys/kernel/tracing/trace | grep mglru_inc_max_seq

             a.out-93      [001] d..1.   278.276539:
mm_mglru_inc_max_seq: nid=0 memcg_id=1 max_seq=4 anon_min_seq=2
file_min_seq=1 nr_anon={0x0,0xf016,0x12,0x0}
nr_file={0x1886,0xb,0x4c,0x0}
           kswapd0-50      [000] d..1.   278.422400:
mm_mglru_inc_max_seq: nid=0 memcg_id=1 max_seq=5 anon_min_seq=2
file_min_seq=2 nr_anon={0x5,0xccb6,0x3439,0x0}
nr_file={0x4ae,0x4f,0x55,0x0}
           kswapd0-50      [000] d..1.   278.508003:
mm_mglru_inc_max_seq: nid=0 memcg_id=1 max_seq=6 anon_min_seq=3
file_min_seq=4 nr_anon={0xccb6,0x3d29,0x1,0x0}
nr_file={0x0,0x59,0x7,0x0}
           kswapd0-50      [000] d..1.   280.046056:
mm_mglru_inc_max_seq: nid=0 memcg_id=1 max_seq=7 anon_min_seq=4
file_min_seq=5 nr_anon={0x3bf3,0xcbbd,0x23e,0x0}
nr_file={0x0,0xb,0x1,0x0}
           kswapd0-50      [000] d..1.   280.124876:
mm_mglru_inc_max_seq: nid=0 memcg_id=1 max_seq=8 anon_min_seq=5
file_min_seq=6 nr_anon={0xcafd,0x3e31,0x0,0x0}
nr_file={0x0,0x5,0x7,0x0}
             a.out-96      [001] d..1.   290.629014:
mm_mglru_inc_max_seq: nid=0 memcg_id=1 max_seq=9 anon_min_seq=7
file_min_seq=6 nr_anon={0x0,0x10615,0xb,0x0}
nr_file={0x331,0xb,0x9a,0x0}
           kswapd0-50      [000] d..1.   290.653098:
mm_mglru_inc_max_seq: nid=0 memcg_id=1 max_seq=10 anon_min_seq=7
file_min_seq=7 nr_anon={0x15,0xef23,0x18c4,0x0}
nr_file={0xb,0xb8,0x55,0x0}
             a.out-96      [001] d..1.   290.684875:
Quoted 8 lines

mm_mglru_inc_max_seq: nid=0 memcg_id=1 max_seq=11 anon_min_seq=8 file_min_seq=9 nr_anon={0xeee3,0x1afb,0x2,0x0} nr_file={0x0,0x76,0x4,0x0}

You can see that the youngest generation is always zero, because you're just getting the newly added youngest generation :-)

Hi Barry,

Thank you very much for testing and reviewing.

Quoted 6 lines

This somehow suggests that the information is redundant, as there is always a zero there.

I suspect we'll need an additional tracepoint to show why aging is triggered and at what sc->priority it happens.

Quoted 8 lines
Quoted 3 lines

But do you know why you trace the moment just after aging is completed, rather than the reason why aging is needed?

My original goal was to pair it with mm_mglru_scan_folios, to see how the aging window of each memcg moves. The per-generation anon/file nr_pages were added following your suggestion on v1. I put it at the end of inc_max_seq() because of the restart there, as max_seq and min_seq are only final after that.

The per-generation anon/file nr_pages are perfectly good, as they can also help explain why aging is needed, especially now that we're introducing new code for imbalance-based aging.

Quoted 8 lines

You make a very good point that the reason for aging is also important. So I'd like to discuss which way is better:

1) Keep mm_mglru_inc_max_seq, and add another tracepoint at the aging decision in try_to_shrink_lruvec(), reporting why aging is needed and at what sc->priority.

Quoted 8 lines

2) Replace mm_mglru_inc_max_seq with a single tracepoint in try_to_shrink_lruvec(), right before try_to_inc_max_seq(), reporting the reason, sc->priority, swappiness, max_seq/min_seq and the per-generation nr_pages before aging. The youngest generation is not empty at that point, and the new window after aging can still be seen from the following mm_mglru_scan_folios.

Which one do you prefer?

Maybe we should keep option 2 for now. Understanding why we are aging and at what priority is much more important, especially since we're still introducing some imbalance-based aging code.

Quoted 6 lines

It should be quite simple, just a matter of moving the code around a little.

If we later find a good reason to have option 1, we can always add it, right?

Hi Barry,

Agreed, I'll go with option 2. Two things I'd like to confirm first:

1) To report the reason, should_run_aging() would need to return it instead of a bool, e.g. an enum like:

enum lru_gen_aging_reason {
AGING_NONE,
AGING_NO_EVICTABLE, /* eviction is not possible anymore */
AGING_MIN_GENS, /* better to age, eviction still possible */
};

Or just report sc->priority, swappiness, max_seq/min_seq and the per-generation nr_pages, and let users derive the reason from them? The former is more direct, the latter keeps should_run_aging() unchanged.

2) try_to_shrink_lruvec() has changed quite a lot recently, e.g.
Kairui's reclaim loop rework, and your swappiness series also touches it, including the aging part. Is it stable enough now to carry a tracepoint there?

Signature · 2 lines
Best regards
Ridong
reviewer
#14 lore raw

Re: [PATCH v6 7/7] mm/mglru: add tracepoint for inc_max_seq()

On Tue, Nov 3, 2026 at 8:23 PM Ridong Chen <[email protected]> wrote:
[...]

Quoted 14 lines

Hi Barry,

Agreed, I'll go with option 2. Two things I'd like to confirm first:

1) To report the reason, should_run_aging() would need to return it instead of a bool, e.g. an enum like:

enum lru_gen_aging_reason {
AGING_NONE,
AGING_NO_EVICTABLE, /* eviction is not possible anymore */
AGING_MIN_GENS, /* better to age, eviction still possible */
};

Quoted 4 lines

Or just report sc->priority, swappiness, max_seq/min_seq and the per-generation nr_pages, and let users derive the reason from them? The former is more direct, the latter keeps should_run_aging() unchanged.

Right. We can let users infer the reason from max_seq, min_seq, and nr_pages in each generation. This shouldn't be difficult, especially since should_run_aging() may change. We don't need to spell out the exact reason. Just present users with the overall state of the lru_gen.

Quoted 5 lines

2) try_to_shrink_lruvec() has changed quite a lot recently, e.g. Kairui's reclaim loop rework, and your swappiness series also touches it, including the aging part. Is it stable enough now to carry a tracepoint there?

Yes, I think aging will always be needed, regardless of how we change the code, so this should remain stable.

Best,
Barry

#15 lore raw

Re: [PATCH v6 7/7] mm/mglru: add tracepoint for inc_max_seq()

Quoted 28 lines · Barry Song

On Tue, Nov 3, 2026 at 8:23 PM Ridong Chen <[email protected]> wrote:
[...]

Quoted 14 lines

Hi Barry,

Agreed, I'll go with option 2. Two things I'd like to confirm first:

1) To report the reason, should_run_aging() would need to return it instead of a bool, e.g. an enum like:

enum lru_gen_aging_reason {
AGING_NONE,
AGING_NO_EVICTABLE, /* eviction is not possible anymore */
AGING_MIN_GENS, /* better to age, eviction still possible */
};

Quoted 4 lines

Or just report sc->priority, swappiness, max_seq/min_seq and the per-generation nr_pages, and let users derive the reason from them? The former is more direct, the latter keeps should_run_aging() unchanged.

Right. We can let users infer the reason from max_seq, min_seq, and nr_pages in each generation. This shouldn't be difficult, especially since should_run_aging() may change. We don't need to spell out the exact reason. Just present users with the overall state of the lru_gen.

Quoted 1 line
Quoted 8 lines
Quoted 4 lines

2) try_to_shrink_lruvec() has changed quite a lot recently, e.g. Kairui's reclaim loop rework, and your swappiness series also touches it, including the aging part. Is it stable enough now to carry a tracepoint there?

Yes, I think aging will always be needed, regardless of how we change the code, so this should remain stable.

Thank you for your patience. That makes sense to me.

Will update.

Signature · 2 lines
Best regards
Ridong