mm/lruvec: trace LRU add drains and drain-all requests
Subsystems (2)
PATCH v3 1/1 mm/lruvec: trace LRU add drains and drain-all requests
LRU add batches can be drained before they reach capacity. This can be a source of LRU lock contention, but it is not currently possible to attribute these drains to callers with existing tracepoints.
Add mm_lru_add_drain to report the CPU and lru_add batch count when an lru_add batch is drained. This allows tracing to distinguish full drains from partial drains and attribute them to the calling stack.
Add mm_lru_add_drain_all to capture callers of __lru_add_drain_all and whether they set the force flag for all CPUs. The tracepoint resembles the signature of the enclosing function, but is needed because of potential inlining.
--- include/trace/events/pagemap.h | 37 ++++++++++++++++++++++++++++++++++ mm/swap.c | 7 ++++++- 2 files changed, 43 insertions(+), 1 deletion(-)
include/trace/events/pagemap.h +37 -0
@@ -77,6 +77,43 @@ TRACE_EVENT(mm_lru_activate,TP_printk ("folio=%p pfn=0x%lx" , __entry->folio, __entry->pfn) ); +TRACE_EVENT (mm_lru_add_drain, + + TP_PROTO(int cpu, unsigned int nr), + +TP_ARGS (cpu, nr), + +TP_STRUCT__entry ( + __field(int, cpu ) + __field(unsigned int, nr ) + ), + +TP_fast_assign ( + __entry->cpu = cpu; + __entry->nr = nr; + ), + +TP_printk ("cpu=%d nr=%u" , __entry->cpu, __entry->nr) +); + +TRACE_EVENT (mm_lru_add_drain_all, + + TP_PROTO(bool force_all_cpus), + +TP_ARGS (force_all_cpus), + +TP_STRUCT__entry ( +__field (bool, force_all_cpus ) + ), + +TP_fast_assign ( + __entry->force_all_cpus = force_all_cpus; + ), + +TP_printk ("force_all_cpus=%s" , + __entry->force_all_cpus ?"true" :"false" ) +); +#endif /* _TRACE_PAGEMAP_H */ /* This part must be outside protection */
mm/swap.c +6 -1
@@ -694,9 +694,12 @@ void lru_add_drain_cpu(int cpu) {struct cpu_fbatches *fbatches = &per_cpu (cpu_fbatches, cpu);struct folio_batch *fbatch = &fbatches->lru_add; +unsigned int nr_folios_add =folio_batch_count (fbatch); -if (folio_batch_count (fbatch)) +if (nr_folios_add) {folio_batch_move_lru (fbatch, lru_add); +trace_mm_lru_add_drain (cpu, nr_folios_add); + } fbatch = &fbatches->lru_move_tail;/* Disabling interrupts below acts as a compiler barrier. */
@@ -869,6 +872,8 @@ static inline void __lru_add_drain_all(bool force_all_cpus)if (WARN_ON (!mm_percpu_wq))return ; +trace_mm_lru_add_drain_all (force_all_cpus); +/* * Guarantee folio_batch counter stores visible by this CPU * are visible to other CPUs before loading the current drain
Signature · 1 line
2.54.0
Quoted 9 lines · JP Kobryn
LRU add batches can be drained before they reach capacity. This can be a source of LRU lock contention, but it is not currently possible to attribute these drains to callers with existing tracepoints.
Add
mm_lru_add_drainto report the CPU andlru_addbatch count when anlru_addbatch is drained. This allows tracing to distinguish full drains from partial drains and attribute them to the calling stack.
Quoted 6 lines
Add
mm_lru_add_drain_allto capture callers of__lru_add_drain_alland whether they set the force flag for all CPUs. The tracepoint resembles the signature of the enclosing function, but is needed because of potential inlining.Signed-off-by JP Kobryn <[email protected]>
Some minor nits:
[...]
mm/swap.c void lru_add_drain_cpu(int cpu) · line 702
Quoted 7 lines
mm/swap.c +4
-1+unsigned int nr_folios_add =folio_batch_count (fbatch); -if (folio_batch_count (fbatch)) +if (nr_folios_add) {folio_batch_move_lru (fbatch, lru_add); +trace_mm_lru_add_drain (cpu, nr_folios_add); + }
Would "nr_folios" work here, given the surrounding lru_add context? Alternatively, nr_folios_added might make the meaning a little clearer.
Best Regards
Barry
Quoted 8 lines · JP Kobryn
LRU add batches can be drained before they reach capacity. This can be a source of LRU lock contention, but it is not currently possible to attribute these drains to callers with existing tracepoints.
Add
mm_lru_add_drainto report the CPU andlru_addbatch count when anlru_addbatch is drained. This allows tracing to distinguish full drains from partial drains and attribute them to the calling stack.
Quoted 6 lines
Add
mm_lru_add_drain_allto capture callers of__lru_add_drain_alland whether they set the force flag for all CPUs. The tracepoint resembles the signature of the enclosing function, but is needed because of potential inlining.Signed-off-by JP Kobryn <[email protected]>
Quoted 86 lines · JP Kobryn
LRU add batches can be drained before they reach capacity. This can be a source of LRU lock contention, but it is not currently possible to attribute these drains to callers with existing tracepoints.
Add
mm_lru_add_drainto report the CPU andlru_addbatch count when anlru_addbatch is drained. This allows tracing to distinguish full drains from partial drains and attribute them to the calling stack.Add
mm_lru_add_drain_allto capture callers of__lru_add_drain_alland whether they set the force flag for all CPUs. The tracepoint resembles the signature of the enclosing function, but is needed because of potential inlining.Signed-off-by JP Kobryn <[email protected]>--- include/trace/events/pagemap.h | 37 ++++++++++++++++++++++++++++++++++ mm/swap.c | 7 ++++++- 2 files changed, 43 insertions(+), 1 deletion(-)include/trace/events/pagemap.h +37
-0@@ -77,6 +77,43 @@ TRACE_EVENT(mm_lru_activate,TP_printk ("folio=%p pfn=0x%lx" , __entry->folio, __entry->pfn) ); +TRACE_EVENT (mm_lru_add_drain, + + TP_PROTO(int cpu, unsigned int nr), + +TP_ARGS (cpu, nr), + +TP_STRUCT__entry ( + __field(int, cpu ) + __field(unsigned int, nr ) + ), + +TP_fast_assign ( + __entry->cpu = cpu; + __entry->nr = nr; + ), + +TP_printk ("cpu=%d nr=%u" , __entry->cpu, __entry->nr) +); + +TRACE_EVENT (mm_lru_add_drain_all, + + TP_PROTO(bool force_all_cpus), + +TP_ARGS (force_all_cpus), + +TP_STRUCT__entry ( +__field (bool, force_all_cpus ) + ), + +TP_fast_assign ( + __entry->force_all_cpus = force_all_cpus; + ), + +TP_printk ("force_all_cpus=%s" , + __entry->force_all_cpus ?"true" :"false" ) +); +#endif /* _TRACE_PAGEMAP_H */ /* This part must be outside protection */ mm/swap.c +4
-1@@ -694,9 +694,12 @@ void lru_add_drain_cpu(int cpu) {struct cpu_fbatches *fbatches = &per_cpu (cpu_fbatches, cpu);struct folio_batch *fbatch = &fbatches->lru_add; +unsigned int nr_folios_add =folio_batch_count (fbatch); -if (folio_batch_count (fbatch)) +if (nr_folios_add) {folio_batch_move_lru (fbatch, lru_add); +trace_mm_lru_add_drain (cpu, nr_folios_add); + } fbatch = &fbatches->lru_move_tail;/* Disabling interrupts below acts as a compiler barrier. */
mm/swap.c static inline void __lru_add_drain_all(bool force_all_cpus) · line 879
Quoted 8 lines
mm/swap.c +2
-0@@ -869,6 +872,8 @@ static inline void __lru_add_drain_all(bool force_all_cpus)if (WARN_ON (!mm_percpu_wq))return ; +trace_mm_lru_add_drain_all (force_all_cpus); +/* * Guarantee folio_batch counter stores visible by this CPU * are visible to other CPUs before loading the current drain
Given that trace events can quickly become stable ABI [1], are we really sure we want to add this?
[1] https://lore.kernel.org/r/[email protected]
Signature · 3 lines
Cheers, David
Quoted 90 lines · David Hildenbrand (Arm)
Quoted 89 lines · JP Kobryn
LRU add batches can be drained before they reach capacity. This can be a source of LRU lock contention, but it is not currently possible to attribute these drains to callers with existing tracepoints.
Add
mm_lru_add_drainto report the CPU andlru_addbatch count when anlru_addbatch is drained. This allows tracing to distinguish full drains from partial drains and attribute them to the calling stack.Add
mm_lru_add_drain_allto capture callers of__lru_add_drain_alland whether they set the force flag for all CPUs. The tracepoint resembles the signature of the enclosing function, but is needed because of potential inlining.Signed-off-by JP Kobryn <[email protected]>--- include/trace/events/pagemap.h | 37 ++++++++++++++++++++++++++++++++++ mm/swap.c | 7 ++++++- 2 files changed, 43 insertions(+), 1 deletion(-)include/trace/events/pagemap.h +37
-0@@ -77,6 +77,43 @@ TRACE_EVENT(mm_lru_activate,TP_printk ("folio=%p pfn=0x%lx" , __entry->folio, __entry->pfn) ); +TRACE_EVENT (mm_lru_add_drain, + + TP_PROTO(int cpu, unsigned int nr), + +TP_ARGS (cpu, nr), + +TP_STRUCT__entry ( + __field(int, cpu ) + __field(unsigned int, nr ) + ), + +TP_fast_assign ( + __entry->cpu = cpu; + __entry->nr = nr; + ), + +TP_printk ("cpu=%d nr=%u" , __entry->cpu, __entry->nr) +); + +TRACE_EVENT (mm_lru_add_drain_all, + + TP_PROTO(bool force_all_cpus), + +TP_ARGS (force_all_cpus), + +TP_STRUCT__entry ( +__field (bool, force_all_cpus ) + ), + +TP_fast_assign ( + __entry->force_all_cpus = force_all_cpus; + ), + +TP_printk ("force_all_cpus=%s" , + __entry->force_all_cpus ?"true" :"false" ) +); +#endif /* _TRACE_PAGEMAP_H */ /* This part must be outside protection */ mm/swap.c +4
-1@@ -694,9 +694,12 @@ void lru_add_drain_cpu(int cpu) {struct cpu_fbatches *fbatches = &per_cpu (cpu_fbatches, cpu);struct folio_batch *fbatch = &fbatches->lru_add; +unsigned int nr_folios_add =folio_batch_count (fbatch); -if (folio_batch_count (fbatch)) +if (nr_folios_add) {folio_batch_move_lru (fbatch, lru_add); +trace_mm_lru_add_drain (cpu, nr_folios_add); + } fbatch = &fbatches->lru_move_tail;/* Disabling interrupts below acts as a compiler barrier. */ @@ -869,6 +872,8 @@ static inline void __lru_add_drain_all(bool force_all_cpus)if (WARN_ON (!mm_percpu_wq))return ;
Quoted 8 lines
mm/swap.c static inline void __lru_add_drain_all(bool force_all_cpus) · line 879
Quoted 5 lines
mm/swap.c +2
-0+trace_mm_lru_add_drain_all (force_all_cpus); +/* * Guarantee folio_batch counter stores visible by this CPU * are visible to other CPUs before loading the current drain Given that trace events can quickly become stable ABI [1], are we really sure we want to add this?
Yes, I think so as this is useful to get insights into lru cache draining. Trace events being stable or not is secondary IMHO. If in future we rearchitect the lru page handling where there is no cache draining anymore, we can make these a noops.
Quoted 7 lines
[1] https://lore.kernel.org/r/[email protected]
Signature · 3 lines
Cheers, David
Quoted 100 lines · Shakeel Butt
Quoted 90 lines · David Hildenbrand (Arm)
Quoted 89 lines · JP Kobryn
LRU add batches can be drained before they reach capacity. This can be a source of LRU lock contention, but it is not currently possible to attribute these drains to callers with existing tracepoints.
Add
mm_lru_add_drainto report the CPU andlru_addbatch count when anlru_addbatch is drained. This allows tracing to distinguish full drains from partial drains and attribute them to the calling stack.Add
mm_lru_add_drain_allto capture callers of__lru_add_drain_alland whether they set the force flag for all CPUs. The tracepoint resembles the signature of the enclosing function, but is needed because of potential inlining.Signed-off-by JP Kobryn <[email protected]>--- include/trace/events/pagemap.h | 37 ++++++++++++++++++++++++++++++++++ mm/swap.c | 7 ++++++- 2 files changed, 43 insertions(+), 1 deletion(-)include/trace/events/pagemap.h +37
-0@@ -77,6 +77,43 @@ TRACE_EVENT(mm_lru_activate,TP_printk ("folio=%p pfn=0x%lx" , __entry->folio, __entry->pfn) ); +TRACE_EVENT (mm_lru_add_drain, + + TP_PROTO(int cpu, unsigned int nr), + +TP_ARGS (cpu, nr), + +TP_STRUCT__entry ( + __field(int, cpu ) + __field(unsigned int, nr ) + ), + +TP_fast_assign ( + __entry->cpu = cpu; + __entry->nr = nr; + ), + +TP_printk ("cpu=%d nr=%u" , __entry->cpu, __entry->nr) +); + +TRACE_EVENT (mm_lru_add_drain_all, + + TP_PROTO(bool force_all_cpus), + +TP_ARGS (force_all_cpus), + +TP_STRUCT__entry ( +__field (bool, force_all_cpus ) + ), + +TP_fast_assign ( + __entry->force_all_cpus = force_all_cpus; + ), + +TP_printk ("force_all_cpus=%s" , + __entry->force_all_cpus ?"true" :"false" ) +); +#endif /* _TRACE_PAGEMAP_H */ /* This part must be outside protection */ mm/swap.c +4
-1@@ -694,9 +694,12 @@ void lru_add_drain_cpu(int cpu) {struct cpu_fbatches *fbatches = &per_cpu (cpu_fbatches, cpu);struct folio_batch *fbatch = &fbatches->lru_add; +unsigned int nr_folios_add =folio_batch_count (fbatch); -if (folio_batch_count (fbatch)) +if (nr_folios_add) {folio_batch_move_lru (fbatch, lru_add); +trace_mm_lru_add_drain (cpu, nr_folios_add); + } fbatch = &fbatches->lru_move_tail;/* Disabling interrupts below acts as a compiler barrier. */ @@ -869,6 +872,8 @@ static inline void __lru_add_drain_all(bool force_all_cpus)if (WARN_ON (!mm_percpu_wq))return ;Quoted 8 lines
mm/swap.c static inline void __lru_add_drain_all(bool force_all_cpus) · line 879
Quoted 5 lines
mm/swap.c +2
-0+trace_mm_lru_add_drain_all (force_all_cpus); +/* * Guarantee folio_batch counter stores visible by this CPU * are visible to other CPUs before loading the current drain Given that trace events can quickly become stable ABI [1], are we really sure we want to add this?
Quoted 4 lines
Yes, I think so as this is useful to get insights into lru cache draining. Trace events being stable or not is secondary IMHO. If in future we rearchitect the lru page handling where there is no cache draining anymore, we can make these a noops.
Yeah and I don't recall ever that a change to a mm tracepoint would ever break someone who'd complain and we'd have to revert it. These are niche enough. So I think the risk is low.
Quoted 7 lines
Quoted 7 lines
[1] https://lore.kernel.org/r/[email protected]
Signature · 3 lines
Cheers, David
Quoted 6 lines · Vlastimil Babka (SUSE)
Quoted 5 lines · Shakeel Butt
Quoted 3 lines · David Hildenbrand (Arm)
Given that trace events can quickly become stable ABI [1], are we really sure we want to add this?
Quoted 7 lines
Quoted 4 lines
Yes, I think so as this is useful to get insights into lru cache draining. Trace events being stable or not is secondary IMHO. If in future we rearchitect the lru page handling where there is no cache draining anymore, we can make these a noops.
Yeah and I don't recall ever that a change to a mm tracepoint would ever break someone who'd complain and we'd have to revert it.
Really? :)
Read the context of the link I posted once more.
Signature · 3 lines
Cheers, David
Quoted 12 lines · David Hildenbrand (Arm)
Quoted 11 lines · Vlastimil Babka (SUSE)
Quoted 9 lines · Shakeel Butt
Quoted 3 lines · David Hildenbrand (Arm)
Given that trace events can quickly become stable ABI [1], are we really sure we want to add this?
Yes, I think so as this is useful to get insights into lru cache draining. Trace events being stable or not is secondary IMHO. If in future we rearchitect the lru page handling where there is no cache draining anymore, we can make these a noops.
Quoted 5 lines
Quoted 2 lines
Yeah and I don't recall ever that a change to a mm tracepoint would ever break someone who'd complain and we'd have to revert it.
Really? :)
Read the context of the link I posted once more.
Ah, I see. I've only read the single mail from Steven that referred to the old powertop breakage and didn't notice the context.
But I don't think these worries should stop us from adding easily usable tracepoints.
Quoted 7 lines · Vlastimil Babka (SUSE)
Quoted 6 lines · David Hildenbrand (Arm)
Quoted 3 lines · Vlastimil Babka (SUSE)
Yeah and I don't recall ever that a change to a mm tracepoint would ever break someone who'd complain and we'd have to revert it.
Really? :)
Quoted 7 lines
Quoted 1 line
Read the context of the link I posted once more.
Ah, I see. I've only read the single mail from Steven that referred to the old powertop breakage and didn't notice the context.
But I don't think these worries should stop us from adding easily usable tracepoints.
Steve explained a way how apparently scheduler people are handling it without trace events.
You can always remove/modify tracepoints, but not trace events.
Anyhow, just wanted to mention it, because so far MM didn't rally know about this implication.
Signature · 3 lines
Cheers, David
Quoted 16 lines · David Hildenbrand (Arm)
Quoted 7 lines · Vlastimil Babka (SUSE)
Quoted 6 lines · David Hildenbrand (Arm)
Quoted 3 lines · Vlastimil Babka (SUSE)
Yeah and I don't recall ever that a change to a mm tracepoint would ever break someone who'd complain and we'd have to revert it.
Really? :)
Quoted 7 lines
Quoted 1 line
Read the context of the link I posted once more.
Ah, I see. I've only read the single mail from Steven that referred to the old powertop breakage and didn't notice the context.
But I don't think these worries should stop us from adding easily usable tracepoints.
Quoted 8 lines
Steve explained a way how apparently scheduler people are handling it without trace events.
You can always remove/modify tracepoints, but not trace events.
Anyhow, just wanted to mention it, because so far MM didn't rally know about this implication.
Thanks for pointing this out. I'll sent v4 using DECLARE_TRACE() to avoid creating a new event.
Steven Rostedt
maintainerQuoted 3 lines · On Wed, 17 Jun 2026 20:18:57 +0200 "Vlastimil Babka (SUSE)"…
Yeah and I don't recall ever that a change to a mm tracepoint would ever break someone who'd complain and we'd have to revert it. These are niche enough. So I think the risk is low.
Note, we have literally thousands of trace events already, so the chances of one being required by an application is rather low. Especially since access still requires root access, which limits it to administration tooling.
That said, if you know of a tool that uses trace events, then those that it is likely to use can become an ABI. For mm trace evnets, rasdaemon is the tool to worry about.
-- Steve