Skip to contentSkip to conversation

mm/lruvec: trace LRU add drains and drain-all requests

PATCH Superseded JP Kobryn 6 participants 11 messages Jun 10 – Jun 26

Subsystems (2)

v3 of 5 Superseded Latest is v5 →

lore
Share & tools
lore

JP Kobryn

author
#0 lore raw

PATCH v3 1/1 mm/lruvec: trace LRU add drains and drain-all requests

LRU add batches can be drained before they reach capacity. This can be a source of LRU lock contention, but it is not currently possible to attribute these drains to callers with existing tracepoints.

Add mm_lru_add_drain to report the CPU and lru_add batch count when an lru_add batch is drained. This allows tracing to distinguish full drains from partial drains and attribute them to the calling stack.

Add mm_lru_add_drain_all to capture callers of __lru_add_drain_all and whether they set the force flag for all CPUs. The tracepoint resembles the signature of the enclosing function, but is needed because of potential inlining.

---
 include/trace/events/pagemap.h | 37 ++++++++++++++++++++++++++++++++++
 mm/swap.c                      |  7 ++++++-
 2 files changed, 43 insertions(+), 1 deletion(-)
include/trace/events/pagemap.h +37 -0
index 171524d3526d..ff3da07ccb40 100644
@@ -77,6 +77,43 @@ TRACE_EVENT(mm_lru_activate,
 	TP_printk("folio=%p pfn=0x%lx", __entry->folio, __entry->pfn)
 );
 
+TRACE_EVENT(mm_lru_add_drain,
+
+	TP_PROTO(int cpu, unsigned int nr),
+
+	TP_ARGS(cpu, nr),
+
+	TP_STRUCT__entry(
+		__field(int,		cpu	)
+		__field(unsigned int,	nr	)
+	),
+
+	TP_fast_assign(
+		__entry->cpu	= cpu;
+		__entry->nr	= nr;
+	),
+
+	TP_printk("cpu=%d nr=%u", __entry->cpu, __entry->nr)
+);
+
+TRACE_EVENT(mm_lru_add_drain_all,
+
+	TP_PROTO(bool force_all_cpus),
+
+	TP_ARGS(force_all_cpus),
+
+	TP_STRUCT__entry(
+		__field(bool,	force_all_cpus	)
+	),
+
+	TP_fast_assign(
+		__entry->force_all_cpus	= force_all_cpus;
+	),
+
+	TP_printk("force_all_cpus=%s",
+		__entry->force_all_cpus ? "true" : "false")
+);
+
 #endif /* _TRACE_PAGEMAP_H */
 
 /* This part must be outside protection */
mm/swap.c +6 -1
index 588f50d8f1a8..e14b7612f896 100644
@@ -694,9 +694,12 @@ void lru_add_drain_cpu(int cpu)
 {
 	struct cpu_fbatches *fbatches = &per_cpu(cpu_fbatches, cpu);
 	struct folio_batch *fbatch = &fbatches->lru_add;
+	unsigned int nr_folios_add = folio_batch_count(fbatch);
 
-	if (folio_batch_count(fbatch))
+	if (nr_folios_add) {
 		folio_batch_move_lru(fbatch, lru_add);
+		trace_mm_lru_add_drain(cpu, nr_folios_add);
+	}
 
 	fbatch = &fbatches->lru_move_tail;
 	/* Disabling interrupts below acts as a compiler barrier. */
@@ -869,6 +872,8 @@ static inline void __lru_add_drain_all(bool force_all_cpus)
 	if (WARN_ON(!mm_percpu_wq))
 		return;
 
+	trace_mm_lru_add_drain_all(force_all_cpus);
+
 	/*
 	 * Guarantee folio_batch counter stores visible by this CPU
 	 * are visible to other CPUs before loading the current drain
Signature · 1 line
2.54.0
Reviewed-by on v3 1/1
#1 lore raw
Quoted 9 lines · JP Kobryn

LRU add batches can be drained before they reach capacity. This can be a source of LRU lock contention, but it is not currently possible to attribute these drains to callers with existing tracepoints.

Add mm_lru_add_drain to report the CPU and lru_add batch count when an lru_add batch is drained. This allows tracing to distinguish full drains from partial drains and attribute them to the calling stack.

Quoted 6 lines

Add mm_lru_add_drain_all to capture callers of __lru_add_drain_all and whether they set the force flag for all CPUs. The tracepoint resembles the signature of the enclosing function, but is needed because of potential inlining.

Some minor nits:

[...]

mm/swap.c void lru_add_drain_cpu(int cpu) · line 702

Quoted 7 lines
mm/swap.c +4 -1
+       unsigned int nr_folios_add = folio_batch_count(fbatch);
 
-       if (folio_batch_count(fbatch))
+       if (nr_folios_add) {
                folio_batch_move_lru(fbatch, lru_add);
+               trace_mm_lru_add_drain(cpu, nr_folios_add);
+       }

Would "nr_folios" work here, given the surrounding lru_add context? Alternatively, nr_folios_added might make the meaning a little clearer.

Best Regards
Barry

Acked-by on v3 1/1
#2 lore raw
In reply to JP Kobryn
Quoted 8 lines · JP Kobryn

LRU add batches can be drained before they reach capacity. This can be a source of LRU lock contention, but it is not currently possible to attribute these drains to callers with existing tracepoints.

Add mm_lru_add_drain to report the CPU and lru_add batch count when an lru_add batch is drained. This allows tracing to distinguish full drains from partial drains and attribute them to the calling stack.

Quoted 6 lines

Add mm_lru_add_drain_all to capture callers of __lru_add_drain_all and whether they set the force flag for all CPUs. The tracepoint resembles the signature of the enclosing function, but is needed because of potential inlining.

#3 lore raw
In reply to JP Kobryn
Quoted 86 lines · JP Kobryn

LRU add batches can be drained before they reach capacity. This can be a source of LRU lock contention, but it is not currently possible to attribute these drains to callers with existing tracepoints.

Add mm_lru_add_drain to report the CPU and lru_add batch count when an lru_add batch is drained. This allows tracing to distinguish full drains from partial drains and attribute them to the calling stack.

Add mm_lru_add_drain_all to capture callers of __lru_add_drain_all and whether they set the force flag for all CPUs. The tracepoint resembles the signature of the enclosing function, but is needed because of potential inlining.

---
 include/trace/events/pagemap.h | 37 ++++++++++++++++++++++++++++++++++
 mm/swap.c                      |  7 ++++++-
 2 files changed, 43 insertions(+), 1 deletion(-)
include/trace/events/pagemap.h +37 -0
index 171524d3526d..ff3da07ccb40 100644
@@ -77,6 +77,43 @@ TRACE_EVENT(mm_lru_activate,
 	TP_printk("folio=%p pfn=0x%lx", __entry->folio, __entry->pfn)
 );
 
+TRACE_EVENT(mm_lru_add_drain,
+
+	TP_PROTO(int cpu, unsigned int nr),
+
+	TP_ARGS(cpu, nr),
+
+	TP_STRUCT__entry(
+		__field(int,		cpu	)
+		__field(unsigned int,	nr	)
+	),
+
+	TP_fast_assign(
+		__entry->cpu	= cpu;
+		__entry->nr	= nr;
+	),
+
+	TP_printk("cpu=%d nr=%u", __entry->cpu, __entry->nr)
+);
+
+TRACE_EVENT(mm_lru_add_drain_all,
+
+	TP_PROTO(bool force_all_cpus),
+
+	TP_ARGS(force_all_cpus),
+
+	TP_STRUCT__entry(
+		__field(bool,	force_all_cpus	)
+	),
+
+	TP_fast_assign(
+		__entry->force_all_cpus	= force_all_cpus;
+	),
+
+	TP_printk("force_all_cpus=%s",
+		__entry->force_all_cpus ? "true" : "false")
+);
+
 #endif /* _TRACE_PAGEMAP_H */
 
 /* This part must be outside protection */
mm/swap.c +4 -1
index 588f50d8f1a8..e14b7612f896 100644
@@ -694,9 +694,12 @@ void lru_add_drain_cpu(int cpu)
 {
 	struct cpu_fbatches *fbatches = &per_cpu(cpu_fbatches, cpu);
 	struct folio_batch *fbatch = &fbatches->lru_add;
+	unsigned int nr_folios_add = folio_batch_count(fbatch);
 
-	if (folio_batch_count(fbatch))
+	if (nr_folios_add) {
 		folio_batch_move_lru(fbatch, lru_add);
+		trace_mm_lru_add_drain(cpu, nr_folios_add);
+	}
 
 	fbatch = &fbatches->lru_move_tail;
 	/* Disabling interrupts below acts as a compiler barrier. */

mm/swap.c static inline void __lru_add_drain_all(bool force_all_cpus) · line 879

Quoted 8 lines
mm/swap.c +2 -0
@@ -869,6 +872,8 @@ static inline void __lru_add_drain_all(bool force_all_cpus)
 	if (WARN_ON(!mm_percpu_wq))
 		return;
 
+	trace_mm_lru_add_drain_all(force_all_cpus);
+
 	/*
 	 * Guarantee folio_batch counter stores visible by this CPU
 	 * are visible to other CPUs before loading the current drain

Given that trace events can quickly become stable ABI [1], are we really sure we want to add this?

[1] https://lore.kernel.org/r/[email protected]

Signature · 3 lines
Cheers,

David
#4 lore raw
Quoted 90 lines · David Hildenbrand (Arm)
Quoted 89 lines · JP Kobryn

LRU add batches can be drained before they reach capacity. This can be a source of LRU lock contention, but it is not currently possible to attribute these drains to callers with existing tracepoints.

Add mm_lru_add_drain to report the CPU and lru_add batch count when an lru_add batch is drained. This allows tracing to distinguish full drains from partial drains and attribute them to the calling stack.

Add mm_lru_add_drain_all to capture callers of __lru_add_drain_all and whether they set the force flag for all CPUs. The tracepoint resembles the signature of the enclosing function, but is needed because of potential inlining.

---
 include/trace/events/pagemap.h | 37 ++++++++++++++++++++++++++++++++++
 mm/swap.c                      |  7 ++++++-
 2 files changed, 43 insertions(+), 1 deletion(-)
include/trace/events/pagemap.h +37 -0
index 171524d3526d..ff3da07ccb40 100644
@@ -77,6 +77,43 @@ TRACE_EVENT(mm_lru_activate,
 	TP_printk("folio=%p pfn=0x%lx", __entry->folio, __entry->pfn)
 );
 
+TRACE_EVENT(mm_lru_add_drain,
+
+	TP_PROTO(int cpu, unsigned int nr),
+
+	TP_ARGS(cpu, nr),
+
+	TP_STRUCT__entry(
+		__field(int,		cpu	)
+		__field(unsigned int,	nr	)
+	),
+
+	TP_fast_assign(
+		__entry->cpu	= cpu;
+		__entry->nr	= nr;
+	),
+
+	TP_printk("cpu=%d nr=%u", __entry->cpu, __entry->nr)
+);
+
+TRACE_EVENT(mm_lru_add_drain_all,
+
+	TP_PROTO(bool force_all_cpus),
+
+	TP_ARGS(force_all_cpus),
+
+	TP_STRUCT__entry(
+		__field(bool,	force_all_cpus	)
+	),
+
+	TP_fast_assign(
+		__entry->force_all_cpus	= force_all_cpus;
+	),
+
+	TP_printk("force_all_cpus=%s",
+		__entry->force_all_cpus ? "true" : "false")
+);
+
 #endif /* _TRACE_PAGEMAP_H */
 
 /* This part must be outside protection */
mm/swap.c +4 -1
index 588f50d8f1a8..e14b7612f896 100644
@@ -694,9 +694,12 @@ void lru_add_drain_cpu(int cpu)
 {
 	struct cpu_fbatches *fbatches = &per_cpu(cpu_fbatches, cpu);
 	struct folio_batch *fbatch = &fbatches->lru_add;
+	unsigned int nr_folios_add = folio_batch_count(fbatch);
 
-	if (folio_batch_count(fbatch))
+	if (nr_folios_add) {
 		folio_batch_move_lru(fbatch, lru_add);
+		trace_mm_lru_add_drain(cpu, nr_folios_add);
+	}
 
 	fbatch = &fbatches->lru_move_tail;
 	/* Disabling interrupts below acts as a compiler barrier. */
@@ -869,6 +872,8 @@ static inline void __lru_add_drain_all(bool force_all_cpus)
 	if (WARN_ON(!mm_percpu_wq))
 		return;
 
Quoted 8 lines

mm/swap.c static inline void __lru_add_drain_all(bool force_all_cpus) · line 879

Quoted 5 lines
mm/swap.c +2 -0
+	trace_mm_lru_add_drain_all(force_all_cpus);
+
 	/*
 	 * Guarantee folio_batch counter stores visible by this CPU
 	 * are visible to other CPUs before loading the current drain

Given that trace events can quickly become stable ABI [1], are we really sure we want to add this?

Yes, I think so as this is useful to get insights into lru cache draining. Trace events being stable or not is secondary IMHO. If in future we rearchitect the lru page handling where there is no cache draining anymore, we can make these a noops.

Quoted 7 lines

[1] https://lore.kernel.org/r/[email protected]

Signature · 3 lines
Cheers,

David
#5 lore raw
Quoted 100 lines · Shakeel Butt
Quoted 90 lines · David Hildenbrand (Arm)
Quoted 89 lines · JP Kobryn

LRU add batches can be drained before they reach capacity. This can be a source of LRU lock contention, but it is not currently possible to attribute these drains to callers with existing tracepoints.

Add mm_lru_add_drain to report the CPU and lru_add batch count when an lru_add batch is drained. This allows tracing to distinguish full drains from partial drains and attribute them to the calling stack.

Add mm_lru_add_drain_all to capture callers of __lru_add_drain_all and whether they set the force flag for all CPUs. The tracepoint resembles the signature of the enclosing function, but is needed because of potential inlining.

---
 include/trace/events/pagemap.h | 37 ++++++++++++++++++++++++++++++++++
 mm/swap.c                      |  7 ++++++-
 2 files changed, 43 insertions(+), 1 deletion(-)
include/trace/events/pagemap.h +37 -0
index 171524d3526d..ff3da07ccb40 100644
@@ -77,6 +77,43 @@ TRACE_EVENT(mm_lru_activate,
 	TP_printk("folio=%p pfn=0x%lx", __entry->folio, __entry->pfn)
 );
 
+TRACE_EVENT(mm_lru_add_drain,
+
+	TP_PROTO(int cpu, unsigned int nr),
+
+	TP_ARGS(cpu, nr),
+
+	TP_STRUCT__entry(
+		__field(int,		cpu	)
+		__field(unsigned int,	nr	)
+	),
+
+	TP_fast_assign(
+		__entry->cpu	= cpu;
+		__entry->nr	= nr;
+	),
+
+	TP_printk("cpu=%d nr=%u", __entry->cpu, __entry->nr)
+);
+
+TRACE_EVENT(mm_lru_add_drain_all,
+
+	TP_PROTO(bool force_all_cpus),
+
+	TP_ARGS(force_all_cpus),
+
+	TP_STRUCT__entry(
+		__field(bool,	force_all_cpus	)
+	),
+
+	TP_fast_assign(
+		__entry->force_all_cpus	= force_all_cpus;
+	),
+
+	TP_printk("force_all_cpus=%s",
+		__entry->force_all_cpus ? "true" : "false")
+);
+
 #endif /* _TRACE_PAGEMAP_H */
 
 /* This part must be outside protection */
mm/swap.c +4 -1
index 588f50d8f1a8..e14b7612f896 100644
@@ -694,9 +694,12 @@ void lru_add_drain_cpu(int cpu)
 {
 	struct cpu_fbatches *fbatches = &per_cpu(cpu_fbatches, cpu);
 	struct folio_batch *fbatch = &fbatches->lru_add;
+	unsigned int nr_folios_add = folio_batch_count(fbatch);
 
-	if (folio_batch_count(fbatch))
+	if (nr_folios_add) {
 		folio_batch_move_lru(fbatch, lru_add);
+		trace_mm_lru_add_drain(cpu, nr_folios_add);
+	}
 
 	fbatch = &fbatches->lru_move_tail;
 	/* Disabling interrupts below acts as a compiler barrier. */
@@ -869,6 +872,8 @@ static inline void __lru_add_drain_all(bool force_all_cpus)
 	if (WARN_ON(!mm_percpu_wq))
 		return;
 
Quoted 8 lines

mm/swap.c static inline void __lru_add_drain_all(bool force_all_cpus) · line 879

Quoted 5 lines
mm/swap.c +2 -0
+	trace_mm_lru_add_drain_all(force_all_cpus);
+
 	/*
 	 * Guarantee folio_batch counter stores visible by this CPU
 	 * are visible to other CPUs before loading the current drain

Given that trace events can quickly become stable ABI [1], are we really sure we want to add this?

Quoted 4 lines

Yes, I think so as this is useful to get insights into lru cache draining. Trace events being stable or not is secondary IMHO. If in future we rearchitect the lru page handling where there is no cache draining anymore, we can make these a noops.

Yeah and I don't recall ever that a change to a mm tracepoint would ever break someone who'd complain and we'd have to revert it. These are niche enough. So I think the risk is low.

Quoted 7 lines
Quoted 7 lines

[1] https://lore.kernel.org/r/[email protected]

Signature · 3 lines
Cheers,

David
#6 lore raw
Quoted 6 lines · Vlastimil Babka (SUSE)
Quoted 5 lines · Shakeel Butt
Quoted 3 lines · David Hildenbrand (Arm)

Given that trace events can quickly become stable ABI [1], are we really sure we want to add this?

Quoted 7 lines
Quoted 4 lines

Yes, I think so as this is useful to get insights into lru cache draining. Trace events being stable or not is secondary IMHO. If in future we rearchitect the lru page handling where there is no cache draining anymore, we can make these a noops.

Yeah and I don't recall ever that a change to a mm tracepoint would ever break someone who'd complain and we'd have to revert it.

Really? :)

Read the context of the link I posted once more.

Signature · 3 lines
Cheers,

David
#7 lore raw
Quoted 12 lines · David Hildenbrand (Arm)
Quoted 11 lines · Vlastimil Babka (SUSE)
Quoted 9 lines · Shakeel Butt
Quoted 3 lines · David Hildenbrand (Arm)

Given that trace events can quickly become stable ABI [1], are we really sure we want to add this?

Yes, I think so as this is useful to get insights into lru cache draining. Trace events being stable or not is secondary IMHO. If in future we rearchitect the lru page handling where there is no cache draining anymore, we can make these a noops.

Quoted 5 lines
Quoted 2 lines

Yeah and I don't recall ever that a change to a mm tracepoint would ever break someone who'd complain and we'd have to revert it.

Really? :)

Read the context of the link I posted once more.

Ah, I see. I've only read the single mail from Steven that referred to the old powertop breakage and didn't notice the context.

But I don't think these worries should stop us from adding easily usable tracepoints.

#8 lore raw
Quoted 7 lines · Vlastimil Babka (SUSE)
Quoted 6 lines · David Hildenbrand (Arm)
Quoted 3 lines · Vlastimil Babka (SUSE)

Yeah and I don't recall ever that a change to a mm tracepoint would ever break someone who'd complain and we'd have to revert it.

Really? :)

Quoted 7 lines
Quoted 1 line

Read the context of the link I posted once more.

Ah, I see. I've only read the single mail from Steven that referred to the old powertop breakage and didn't notice the context.

But I don't think these worries should stop us from adding easily usable tracepoints.

Steve explained a way how apparently scheduler people are handling it without trace events.

You can always remove/modify tracepoints, but not trace events.

Anyhow, just wanted to mention it, because so far MM didn't rally know about this implication.

Signature · 3 lines
Cheers,

David
author
#9 lore raw
Quoted 16 lines · David Hildenbrand (Arm)
Quoted 7 lines · Vlastimil Babka (SUSE)
Quoted 6 lines · David Hildenbrand (Arm)
Quoted 3 lines · Vlastimil Babka (SUSE)

Yeah and I don't recall ever that a change to a mm tracepoint would ever break someone who'd complain and we'd have to revert it.

Really? :)

Quoted 7 lines
Quoted 1 line

Read the context of the link I posted once more.

Ah, I see. I've only read the single mail from Steven that referred to the old powertop breakage and didn't notice the context.

But I don't think these worries should stop us from adding easily usable tracepoints.

Quoted 8 lines

Steve explained a way how apparently scheduler people are handling it without trace events.

You can always remove/modify tracepoints, but not trace events.

Anyhow, just wanted to mention it, because so far MM didn't rally know about this implication.

Thanks for pointing this out. I'll sent v4 using DECLARE_TRACE() to avoid creating a new event.

maintainer
#10 lore raw
In reply to Vlastimil Babka (SUSE)
Quoted 3 lines · On Wed, 17 Jun 2026 20:18:57 +0200 "Vlastimil Babka (SUSE)"…

Yeah and I don't recall ever that a change to a mm tracepoint would ever break someone who'd complain and we'd have to revert it. These are niche enough. So I think the risk is low.

Note, we have literally thousands of trace events already, so the chances of one being required by an application is rather low. Especially since access still requires root access, which limits it to administration tooling.

That said, if you know of a tool that uses trace events, then those that it is likely to use can become an ABI. For mm trace evnets, rasdaemon is the tool to worry about.

-- Steve