Skip to contentSkip to conversation

mm/vmstat: spread vmstat_update requeue across the stat interval

PATCH Superseded Breno Leitao 8 participants 24 messages Apr 1 – Apr 9

Subsystems (2)

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 mm/vmstat: spread vmstat_update requeue across the stat interval

vmstat_update uses round_jiffies_relative() when re-queuing itself, which aligns all CPUs' timers to the same second boundary. When many CPUs have pending PCP pages to drain, they all call decay_pcp_high() -> free_pcppages_bulk() simultaneously, serializing on zone->lock and hitting contention.

Introduce vmstat_spread_delay() which distributes each CPU's vmstat_update evenly across the stat interval instead of aligning them.

This does not increase the number of timer interrupts — each CPU still fires once per interval. The timers are simply staggered rather than aligned. Additionally, vmstat_work is DEFERRABLE_WORK, so it does not wake idle CPUs regardless of scheduling; the spread only affects CPUs that are already active

perf lock contention shows 7.5x reduction in zone->lock contention (872 -> 117 contentions, 199ms -> 81ms total wait) on a 72-CPU aarch64 system under memory pressure.

Tested on a 72-CPU aarch64 system using stress-ng --vm to generate memory allocation bursts. Lock contention was measured with:

perf lock contention -a -b -S free_pcppages_bulk

Results with KASAN enabled:

free_pcppages_bulk contention (KASAN):

  +--------------+----------+----------+
  | Metric       | No fix   | With fix |
  +--------------+----------+----------+
  | Contentions  |      872 |      117 |
  | Total wait   | 199.43ms | 80.76ms  |
  | Max wait     |   4.19ms | 35.76ms  |
  +--------------+----------+----------+

Results without KASAN:

free_pcppages_bulk contention (no KASAN):

  +--------------+----------+----------+
  | Metric       | No fix   | With fix |
  +--------------+----------+----------+
  | Contentions  |      240 |      133 |
  | Total wait   |  34.01ms | 24.61ms  |
  | Max wait     |   965us  |  1.35ms  |
  +--------------+----------+----------+
---
 mm/vmstat.c | 25 ++++++++++++++++++++++++-
 1 file changed, 24 insertions(+), 1 deletion(-)
mm/vmstat.c +24 -1
index 2370c6fb1fcd..2e94bd765606 100644
@@ -2032,6 +2032,29 @@ static int vmstat_refresh(const struct ctl_table *table, int write,
 }
 #endif /* CONFIG_PROC_FS */
 
+/*
+ * Return a per-cpu delay that spreads vmstat_update work across the stat
+ * interval.  Without this, round_jiffies_relative() aligns every CPU's
+ * timer to the same second boundary, causing a thundering-herd on
+ * zone->lock when multiple CPUs drain PCP pages simultaneously via
+ * decay_pcp_high() -> free_pcppages_bulk().
+ */
+static unsigned long vmstat_spread_delay(void)
+{
+	unsigned long interval = sysctl_stat_interval;
+	unsigned int nr_cpus = num_online_cpus();
+
+	if (nr_cpus <= 1)
+		return round_jiffies_relative(interval);
+
+	/*
+	 * Spread per-cpu vmstat work evenly across the interval.  Don't
+	 * use round_jiffies_relative() here -- it would snap every CPU
+	 * back to the same second boundary, defeating the spread.
+	 */
+	return interval + (interval * (smp_processor_id() % nr_cpus)) / nr_cpus;
+}
+
 static void vmstat_update(struct work_struct *w)
 {
 	if (refresh_cpu_vm_stats(true)) {
@@ -2042,7 +2065,7 @@ static void vmstat_update(struct work_struct *w)
 		 */
 		queue_delayed_work_on(smp_processor_id(), mm_percpu_wq,
 				this_cpu_ptr(&vmstat_work),
-				round_jiffies_relative(sysctl_stat_interval));
+				vmstat_spread_delay());
 	}
 }
 

---
base-commit: cf7c3c02fdd0dfccf4d6611714273dcb538af2cb
change-id: 20260401-vmstat-048e0feaf344

Best regards,

Signature · 1 line
Breno Leitao <[email protected]>
Acked-by on v1 1/1
#1 lore raw
Quoted 46 lines · Breno Leitao

vmstat_update uses round_jiffies_relative() when re-queuing itself, which aligns all CPUs' timers to the same second boundary. When many CPUs have pending PCP pages to drain, they all call decay_pcp_high() -> free_pcppages_bulk() simultaneously, serializing on zone->lock and hitting contention.

Introduce vmstat_spread_delay() which distributes each CPU's vmstat_update evenly across the stat interval instead of aligning them.

This does not increase the number of timer interrupts — each CPU still fires once per interval. The timers are simply staggered rather than aligned. Additionally, vmstat_work is DEFERRABLE_WORK, so it does not wake idle CPUs regardless of scheduling; the spread only affects CPUs that are already active

perf lock contention shows 7.5x reduction in zone->lock contention (872 -> 117 contentions, 199ms -> 81ms total wait) on a 72-CPU aarch64 system under memory pressure.

Tested on a 72-CPU aarch64 system using stress-ng --vm to generate memory allocation bursts. Lock contention was measured with:

perf lock contention -a -b -S free_pcppages_bulk

Results with KASAN enabled:

free_pcppages_bulk contention (KASAN):

  +--------------+----------+----------+
  | Metric       | No fix   | With fix |
  +--------------+----------+----------+
  | Contentions  |      872 |      117 |
  | Total wait   | 199.43ms | 80.76ms  |
  | Max wait     |   4.19ms | 35.76ms  |
  +--------------+----------+----------+

Results without KASAN:

free_pcppages_bulk contention (no KASAN):

  +--------------+----------+----------+
  | Metric       | No fix   | With fix |
  +--------------+----------+----------+
  | Contentions  |      240 |      133 |
  | Total wait   |  34.01ms | 24.61ms  |
  | Max wait     |   965us  |  1.35ms  |
  +--------------+----------+----------+
Quoted 1 line

Nice!

Quoted 23 lines
---
 mm/vmstat.c | 25 ++++++++++++++++++++++++-
 1 file changed, 24 insertions(+), 1 deletion(-)
mm/vmstat.c +11 -0
index 2370c6fb1fcd..2e94bd765606 100644
@@ -2032,6 +2032,29 @@ static int vmstat_refresh(const struct ctl_table *table, int write,
 }
 #endif /* CONFIG_PROC_FS */
 
+/*
+ * Return a per-cpu delay that spreads vmstat_update work across the stat
+ * interval.  Without this, round_jiffies_relative() aligns every CPU's
+ * timer to the same second boundary, causing a thundering-herd on
+ * zone->lock when multiple CPUs drain PCP pages simultaneously via
+ * decay_pcp_high() -> free_pcppages_bulk().
+ */
+static unsigned long vmstat_spread_delay(void)
+{
+	unsigned long interval = sysctl_stat_interval;
+	unsigned int nr_cpus = num_online_cpus();

mm/vmstat.c static int vmstat_refresh(const struct ctl_table *table, int write, · line 2055

Quoted 10 lines
mm/vmstat.c +10 -0
+
+	if (nr_cpus <= 1)
+		return round_jiffies_relative(interval);
+
+	/*
+	 * Spread per-cpu vmstat work evenly across the interval.  Don't
+	 * use round_jiffies_relative() here -- it would snap every CPU
+	 * back to the same second boundary, defeating the spread.
+	 */
+	return interval + (interval * (smp_processor_id() % nr_cpus)) / nr_cpus;

smp_processor_id() <= nr_cpus, so

return interval + interval*cpu/nr_cpus

should be equivalent, no?

Other than that,

#2 lore raw

Hello Johannes,

Quoted 18 lines · Johannes Weiner
Quoted 4 lines · Breno Leitao
mm/vmstat.c +4 -0
+static unsigned long vmstat_spread_delay(void)
+{
+	unsigned long interval = sysctl_stat_interval;
+	unsigned int nr_cpus = num_online_cpus();

mm/vmstat.c static int vmstat_refresh(const struct ctl_table *table, int write, · line 2055

Quoted 10 lines
mm/vmstat.c +10 -0
+
+	if (nr_cpus <= 1)
+		return round_jiffies_relative(interval);
+
+	/*
+	 * Spread per-cpu vmstat work evenly across the interval.  Don't
+	 * use round_jiffies_relative() here -- it would snap every CPU
+	 * back to the same second boundary, defeating the spread.
+	 */
+	return interval + (interval * (smp_processor_id() % nr_cpus)) / nr_cpus;

smp_processor_id() <= nr_cpus, so

Quoted 3 lines

return interval + interval*cpu/nr_cpus

should be equivalent, no?

nr_cpus is the number of online CPUs, while smp_processor_id() is the CPU id.

If you offline a CPU, then smp_processor_id() might be bigger than num_online_cpus()

My goal was to linearly shift the timer and avoid creating gaps when removing certain CPUs.

Thanks for the review,
--breno

#3 lore raw
Quoted 25 lines · Breno Leitao

Hello Johannes,

Quoted 18 lines · Johannes Weiner
Quoted 4 lines · Breno Leitao
mm/vmstat.c +4 -0
+static unsigned long vmstat_spread_delay(void)
+{
+	unsigned long interval = sysctl_stat_interval;
+	unsigned int nr_cpus = num_online_cpus();

mm/vmstat.c static int vmstat_refresh(const struct ctl_table *table, int write, · line 2055

Quoted 10 lines
mm/vmstat.c +10 -0
+
+	if (nr_cpus <= 1)
+		return round_jiffies_relative(interval);
+
+	/*
+	 * Spread per-cpu vmstat work evenly across the interval.  Don't
+	 * use round_jiffies_relative() here -- it would snap every CPU
+	 * back to the same second boundary, defeating the spread.
+	 */
+	return interval + (interval * (smp_processor_id() % nr_cpus)) / nr_cpus;

smp_processor_id() <= nr_cpus, so

Quoted 3 lines

return interval + interval*cpu/nr_cpus

should be equivalent, no?

Quoted 8 lines

nr_cpus is the number of online CPUs, while smp_processor_id() is the CPU id.

If you offline a CPU, then smp_processor_id() might be bigger than num_online_cpus()

My goal was to linearly shift the timer and avoid creating gaps when removing certain CPUs.

Ah makes sense. Plus you'd spill into the next interval otherwise.

#4 lore raw
In reply to Breno Leitao
Quoted 11 lines · Breno Leitao

free_pcppages_bulk contention (KASAN):

  +--------------+----------+----------+
  | Metric       | No fix   | With fix |
  +--------------+----------+----------+
  | Contentions  |      872 |      117 |
  | Total wait   | 199.43ms | 80.76ms  |
  | Max wait     |   4.19ms | 35.76ms  |
  +--------------+----------+----------+

Results without KASAN:

Quoted 8 lines

free_pcppages_bulk contention (no KASAN):

  +--------------+----------+----------+
  | Metric       | No fix   | With fix |
  +--------------+----------+----------+
  | Contentions  |      240 |      133 |
  | Total wait   |  34.01ms | 24.61ms  |
  | Max wait     |   965us  |  1.35ms  |
  +--------------+----------+----------+

Sorry, the Max wait time is inverted on both cases.

free_pcppages_bulk contention (KASAN):

  +--------------+----------+----------+
  | Metric       | No fix   | With fix |
  +--------------+----------+----------+
  | Contentions  |      872 |      117 |
  | Total wait   | 199.43ms | 80.76ms  |
  | Max wait     |  35.76ms | 4.19ms   |
  +--------------+----------+----------+

Results without KASAN:

free_pcppages_bulk contention (no KASAN):

  +--------------+----------+----------+
  | Metric       | No fix   | With fix |
  +--------------+----------+----------+
  | Contentions  |      240 |      133 |
  | Total wait   |  34.01ms | 24.61ms  |
  | Max wait     |   1.35ms |   965us  |
  +--------------+----------+----------+
Acked-by on v1 1/1
#5 lore raw
In reply to Breno Leitao
Quoted 8 lines · Breno Leitao

vmstat_update uses round_jiffies_relative() when re-queuing itself, which aligns all CPUs' timers to the same second boundary. When many CPUs have pending PCP pages to drain, they all call decay_pcp_high() -> free_pcppages_bulk() simultaneously, serializing on zone->lock and hitting contention.

Introduce vmstat_spread_delay() which distributes each CPU's vmstat_update evenly across the stat interval instead of aligning them.

Nice idea.

Quoted 9 lines

This does not increase the number of timer interrupts — each CPU still fires once per interval. The timers are simply staggered rather than aligned. Additionally, vmstat_work is DEFERRABLE_WORK, so it does not wake idle CPUs regardless of scheduling; the spread only affects CPUs that are already active

perf lock contention shows 7.5x reduction in zone->lock contention (872 -> 117 contentions, 199ms -> 81ms total wait) on a 72-CPU aarch64 system under memory pressure.

Wow. That's huge improvement.

Quoted 28 lines

Tested on a 72-CPU aarch64 system using stress-ng --vm to generate memory allocation bursts. Lock contention was measured with:

perf lock contention -a -b -S free_pcppages_bulk

Results with KASAN enabled:

free_pcppages_bulk contention (KASAN):

  +--------------+----------+----------+
  | Metric       | No fix   | With fix |
  +--------------+----------+----------+
  | Contentions  |      872 |      117 |
  | Total wait   | 199.43ms | 80.76ms  |
  | Max wait     |   4.19ms | 35.76ms  |
  +--------------+----------+----------+

Results without KASAN:

free_pcppages_bulk contention (no KASAN):

  +--------------+----------+----------+
  | Metric       | No fix   | With fix |
  +--------------+----------+----------+
  | Contentions  |      240 |      133 |
  | Total wait   |  34.01ms | 24.61ms  |
  | Max wait     |   965us  |  1.35ms  |
  +--------------+----------+----------+
Quoted 1 line
Signature · 1 line
  Kiryl Shutsemau / Kirill A. Shutemov
#6 lore raw
In reply to Breno Leitao
Quoted 86 lines · On Wed, 01 Apr 2026 06:57:50 -0700 Breno Leitao <[email protected]> wrote:

vmstat_update uses round_jiffies_relative() when re-queuing itself, which aligns all CPUs' timers to the same second boundary. When many CPUs have pending PCP pages to drain, they all call decay_pcp_high() -> free_pcppages_bulk() simultaneously, serializing on zone->lock and hitting contention.

Introduce vmstat_spread_delay() which distributes each CPU's vmstat_update evenly across the stat interval instead of aligning them.

This does not increase the number of timer interrupts — each CPU still fires once per interval. The timers are simply staggered rather than aligned. Additionally, vmstat_work is DEFERRABLE_WORK, so it does not wake idle CPUs regardless of scheduling; the spread only affects CPUs that are already active

perf lock contention shows 7.5x reduction in zone->lock contention (872 -> 117 contentions, 199ms -> 81ms total wait) on a 72-CPU aarch64 system under memory pressure.

Tested on a 72-CPU aarch64 system using stress-ng --vm to generate memory allocation bursts. Lock contention was measured with:

perf lock contention -a -b -S free_pcppages_bulk

Results with KASAN enabled:

free_pcppages_bulk contention (KASAN):

  +--------------+----------+----------+
  | Metric       | No fix   | With fix |
  +--------------+----------+----------+
  | Contentions  |      872 |      117 |
  | Total wait   | 199.43ms | 80.76ms  |
  | Max wait     |   4.19ms | 35.76ms  |
  +--------------+----------+----------+

Results without KASAN:

free_pcppages_bulk contention (no KASAN):

  +--------------+----------+----------+
  | Metric       | No fix   | With fix |
  +--------------+----------+----------+
  | Contentions  |      240 |      133 |
  | Total wait   |  34.01ms | 24.61ms  |
  | Max wait     |   965us  |  1.35ms  |
  +--------------+----------+----------+
---
 mm/vmstat.c | 25 ++++++++++++++++++++++++-
 1 file changed, 24 insertions(+), 1 deletion(-)
mm/vmstat.c +23 -0
index 2370c6fb1fcd..2e94bd765606 100644
@@ -2032,6 +2032,29 @@ static int vmstat_refresh(const struct ctl_table *table, int write,
 }
 #endif /* CONFIG_PROC_FS */
 
+/*
+ * Return a per-cpu delay that spreads vmstat_update work across the stat
+ * interval.  Without this, round_jiffies_relative() aligns every CPU's
+ * timer to the same second boundary, causing a thundering-herd on
+ * zone->lock when multiple CPUs drain PCP pages simultaneously via
+ * decay_pcp_high() -> free_pcppages_bulk().
+ */
+static unsigned long vmstat_spread_delay(void)
+{
+	unsigned long interval = sysctl_stat_interval;
+	unsigned int nr_cpus = num_online_cpus();
+
+	if (nr_cpus <= 1)
+		return round_jiffies_relative(interval);
+
+	/*
+	 * Spread per-cpu vmstat work evenly across the interval.  Don't
+	 * use round_jiffies_relative() here -- it would snap every CPU
+	 * back to the same second boundary, defeating the spread.
+	 */
+	return interval + (interval * (smp_processor_id() % nr_cpus)) / nr_cpus;
+}
+
 static void vmstat_update(struct work_struct *w)
 {
 	if (refresh_cpu_vm_stats(true)) {

mm/vmstat.c static void vmstat_update(struct work_struct *w) · line 2068

Quoted 5 lines
mm/vmstat.c +1 -1
@@ -2042,7 +2065,7 @@ static void vmstat_update(struct work_struct *w)
 		 */
 		queue_delayed_work_on(smp_processor_id(), mm_percpu_wq,
 				this_cpu_ptr(&vmstat_work),
-				round_jiffies_relative(sysctl_stat_interval));
+				vmstat_spread_delay());

This is awesome! Maybe this needs to be done to vmstat_shepherd() as well?

vmstat_shepherd() still queues work with delay 0 on all CPUs that need_update() in its for_each_online_cpu() loop:

      if (!delayed_work_pending(dw) && need_update(cpu))
          queue_delayed_work_on(cpu, mm_percpu_wq, dw, 0);

So when the shepherd fires, it kicks all dormant CPUs' vmstat workers simultaneously.

Under sustained memory pressure on a large system, I think the shepherd fires every sysctl_stat_interval and could re-trigger the same lock contention?

Quoted 13 lines
mm/vmstat.c +0 -1
 	}
 }
 
 
---

base-commit: cf7c3c02fdd0dfccf4d6611714273dcb538af2cb
change-id: 20260401-vmstat-048e0feaf344

Best regards,

Signature · 1 line
Breno Leitao <[email protected]>
#7 lore raw
Quoted 102 lines · Usama Arif
Quoted 86 lines · On Wed, 01 Apr 2026 06:57:50 -0700 Breno Leitao <[email protected]> wrote:

vmstat_update uses round_jiffies_relative() when re-queuing itself, which aligns all CPUs' timers to the same second boundary. When many CPUs have pending PCP pages to drain, they all call decay_pcp_high() -> free_pcppages_bulk() simultaneously, serializing on zone->lock and hitting contention.

Introduce vmstat_spread_delay() which distributes each CPU's vmstat_update evenly across the stat interval instead of aligning them.

This does not increase the number of timer interrupts — each CPU still fires once per interval. The timers are simply staggered rather than aligned. Additionally, vmstat_work is DEFERRABLE_WORK, so it does not wake idle CPUs regardless of scheduling; the spread only affects CPUs that are already active

perf lock contention shows 7.5x reduction in zone->lock contention (872 -> 117 contentions, 199ms -> 81ms total wait) on a 72-CPU aarch64 system under memory pressure.

Tested on a 72-CPU aarch64 system using stress-ng --vm to generate memory allocation bursts. Lock contention was measured with:

perf lock contention -a -b -S free_pcppages_bulk

Results with KASAN enabled:

free_pcppages_bulk contention (KASAN):

  +--------------+----------+----------+
  | Metric       | No fix   | With fix |
  +--------------+----------+----------+
  | Contentions  |      872 |      117 |
  | Total wait   | 199.43ms | 80.76ms  |
  | Max wait     |   4.19ms | 35.76ms  |
  +--------------+----------+----------+

Results without KASAN:

free_pcppages_bulk contention (no KASAN):

  +--------------+----------+----------+
  | Metric       | No fix   | With fix |
  +--------------+----------+----------+
  | Contentions  |      240 |      133 |
  | Total wait   |  34.01ms | 24.61ms  |
  | Max wait     |   965us  |  1.35ms  |
  +--------------+----------+----------+
---
 mm/vmstat.c | 25 ++++++++++++++++++++++++-
 1 file changed, 24 insertions(+), 1 deletion(-)
mm/vmstat.c +23 -0
index 2370c6fb1fcd..2e94bd765606 100644
@@ -2032,6 +2032,29 @@ static int vmstat_refresh(const struct ctl_table *table, int write,
 }
 #endif /* CONFIG_PROC_FS */
 
+/*
+ * Return a per-cpu delay that spreads vmstat_update work across the stat
+ * interval.  Without this, round_jiffies_relative() aligns every CPU's
+ * timer to the same second boundary, causing a thundering-herd on
+ * zone->lock when multiple CPUs drain PCP pages simultaneously via
+ * decay_pcp_high() -> free_pcppages_bulk().
+ */
+static unsigned long vmstat_spread_delay(void)
+{
+	unsigned long interval = sysctl_stat_interval;
+	unsigned int nr_cpus = num_online_cpus();
+
+	if (nr_cpus <= 1)
+		return round_jiffies_relative(interval);
+
+	/*
+	 * Spread per-cpu vmstat work evenly across the interval.  Don't
+	 * use round_jiffies_relative() here -- it would snap every CPU
+	 * back to the same second boundary, defeating the spread.
+	 */
+	return interval + (interval * (smp_processor_id() % nr_cpus)) / nr_cpus;
+}
+
 static void vmstat_update(struct work_struct *w)
 {
 	if (refresh_cpu_vm_stats(true)) {

mm/vmstat.c static void vmstat_update(struct work_struct *w) · line 2068

Quoted 5 lines
mm/vmstat.c +1 -1
@@ -2042,7 +2065,7 @@ static void vmstat_update(struct work_struct *w)
 		 */
 		queue_delayed_work_on(smp_processor_id(), mm_percpu_wq,
 				this_cpu_ptr(&vmstat_work),
-				round_jiffies_relative(sysctl_stat_interval));
+				vmstat_spread_delay());

This is awesome! Maybe this needs to be done to vmstat_shepherd() as well?

vmstat_shepherd() still queues work with delay 0 on all CPUs that need_update() in its for_each_online_cpu() loop:

      if (!delayed_work_pending(dw) && need_update(cpu))
          queue_delayed_work_on(cpu, mm_percpu_wq, dw, 0);
Quoted 6 lines

So when the shepherd fires, it kicks all dormant CPUs' vmstat workers simultaneously.

Under sustained memory pressure on a large system, I think the shepherd fires every sysctl_stat_interval and could re-trigger the same lock contention?

Good point - incorporating similar spreading logic in vmstat_shepherd() would indeed address the simultaneous queueing issue you've described.

Should I include this in a v2 of this patch, or would you prefer it as a separate follow-up patch?

Acked-by
#8 lore raw
Quoted 110 lines · Breno Leitao
Quoted 102 lines · Usama Arif
Quoted 86 lines · On Wed, 01 Apr 2026 06:57:50 -0700 Breno Leitao <[email protected]> wrote:

vmstat_update uses round_jiffies_relative() when re-queuing itself, which aligns all CPUs' timers to the same second boundary. When many CPUs have pending PCP pages to drain, they all call decay_pcp_high() -> free_pcppages_bulk() simultaneously, serializing on zone->lock and hitting contention.

Introduce vmstat_spread_delay() which distributes each CPU's vmstat_update evenly across the stat interval instead of aligning them.

This does not increase the number of timer interrupts — each CPU still fires once per interval. The timers are simply staggered rather than aligned. Additionally, vmstat_work is DEFERRABLE_WORK, so it does not wake idle CPUs regardless of scheduling; the spread only affects CPUs that are already active

perf lock contention shows 7.5x reduction in zone->lock contention (872 -> 117 contentions, 199ms -> 81ms total wait) on a 72-CPU aarch64 system under memory pressure.

Tested on a 72-CPU aarch64 system using stress-ng --vm to generate memory allocation bursts. Lock contention was measured with:

perf lock contention -a -b -S free_pcppages_bulk

Results with KASAN enabled:

free_pcppages_bulk contention (KASAN):

  +--------------+----------+----------+
  | Metric       | No fix   | With fix |
  +--------------+----------+----------+
  | Contentions  |      872 |      117 |
  | Total wait   | 199.43ms | 80.76ms  |
  | Max wait     |   4.19ms | 35.76ms  |
  +--------------+----------+----------+

Results without KASAN:

free_pcppages_bulk contention (no KASAN):

  +--------------+----------+----------+
  | Metric       | No fix   | With fix |
  +--------------+----------+----------+
  | Contentions  |      240 |      133 |
  | Total wait   |  34.01ms | 24.61ms  |
  | Max wait     |   965us  |  1.35ms  |
  +--------------+----------+----------+
---
 mm/vmstat.c | 25 ++++++++++++++++++++++++-
 1 file changed, 24 insertions(+), 1 deletion(-)
mm/vmstat.c +23 -0
index 2370c6fb1fcd..2e94bd765606 100644
@@ -2032,6 +2032,29 @@ static int vmstat_refresh(const struct ctl_table *table, int write,
 }
 #endif /* CONFIG_PROC_FS */
 
+/*
+ * Return a per-cpu delay that spreads vmstat_update work across the stat
+ * interval.  Without this, round_jiffies_relative() aligns every CPU's
+ * timer to the same second boundary, causing a thundering-herd on
+ * zone->lock when multiple CPUs drain PCP pages simultaneously via
+ * decay_pcp_high() -> free_pcppages_bulk().
+ */
+static unsigned long vmstat_spread_delay(void)
+{
+	unsigned long interval = sysctl_stat_interval;
+	unsigned int nr_cpus = num_online_cpus();
+
+	if (nr_cpus <= 1)
+		return round_jiffies_relative(interval);
+
+	/*
+	 * Spread per-cpu vmstat work evenly across the interval.  Don't
+	 * use round_jiffies_relative() here -- it would snap every CPU
+	 * back to the same second boundary, defeating the spread.
+	 */
+	return interval + (interval * (smp_processor_id() % nr_cpus)) / nr_cpus;
+}
+
 static void vmstat_update(struct work_struct *w)
 {
 	if (refresh_cpu_vm_stats(true)) {

mm/vmstat.c static void vmstat_update(struct work_struct *w) · line 2068

Quoted 5 lines
mm/vmstat.c +1 -1
@@ -2042,7 +2065,7 @@ static void vmstat_update(struct work_struct *w)
 		 */
 		queue_delayed_work_on(smp_processor_id(), mm_percpu_wq,
 				this_cpu_ptr(&vmstat_work),
-				round_jiffies_relative(sysctl_stat_interval));
+				vmstat_spread_delay());

This is awesome! Maybe this needs to be done to vmstat_shepherd() as well?

vmstat_shepherd() still queues work with delay 0 on all CPUs that need_update() in its for_each_online_cpu() loop:

      if (!delayed_work_pending(dw) && need_update(cpu))
          queue_delayed_work_on(cpu, mm_percpu_wq, dw, 0);
Quoted 6 lines

So when the shepherd fires, it kicks all dormant CPUs' vmstat workers simultaneously.

Under sustained memory pressure on a large system, I think the shepherd fires every sysctl_stat_interval and could re-trigger the same lock contention?

Quoted 5 lines

Good point - incorporating similar spreading logic in vmstat_shepherd() would indeed address the simultaneous queueing issue you've described.

Should I include this in a v2 of this patch, or would you prefer it as a separate follow-up patch?

I think it can be a separate follow-up patch, but no strong preference. For this patch:

#9 lore raw
Quoted 16 lines · Usama Arif
Quoted 16 lines
Quoted 15 lines

This is awesome! Maybe this needs to be done to vmstat_shepherd() as well?

vmstat_shepherd() still queues work with delay 0 on all CPUs that need_update() in its for_each_online_cpu() loop:

      if (!delayed_work_pending(dw) && need_update(cpu))
          queue_delayed_work_on(cpu, mm_percpu_wq, dw, 0);

So when the shepherd fires, it kicks all dormant CPUs' vmstat workers simultaneously.

Under sustained memory pressure on a large system, I think the shepherd fires every sysctl_stat_interval and could re-trigger the same lock contention?

Quoted 7 lines
Quoted 5 lines

Good point - incorporating similar spreading logic in vmstat_shepherd() would indeed address the simultaneous queueing issue you've described.

Should I include this in a v2 of this patch, or would you prefer it as a separate follow-up patch?

I think it can be a separate follow-up patch, but no strong preference.

Thanks!

I will send a follow-up patch soon.
--breno

#10 lore raw
In reply to Breno Leitao
Quoted 46 lines · Breno Leitao

vmstat_update uses round_jiffies_relative() when re-queuing itself, which aligns all CPUs' timers to the same second boundary. When many CPUs have pending PCP pages to drain, they all call decay_pcp_high() -> free_pcppages_bulk() simultaneously, serializing on zone->lock and hitting contention.

Introduce vmstat_spread_delay() which distributes each CPU's vmstat_update evenly across the stat interval instead of aligning them.

This does not increase the number of timer interrupts — each CPU still fires once per interval. The timers are simply staggered rather than aligned. Additionally, vmstat_work is DEFERRABLE_WORK, so it does not wake idle CPUs regardless of scheduling; the spread only affects CPUs that are already active

perf lock contention shows 7.5x reduction in zone->lock contention (872 -> 117 contentions, 199ms -> 81ms total wait) on a 72-CPU aarch64 system under memory pressure.

Tested on a 72-CPU aarch64 system using stress-ng --vm to generate memory allocation bursts. Lock contention was measured with:

perf lock contention -a -b -S free_pcppages_bulk

Results with KASAN enabled:

free_pcppages_bulk contention (KASAN):

  +--------------+----------+----------+
  | Metric       | No fix   | With fix |
  +--------------+----------+----------+
  | Contentions  |      872 |      117 |
  | Total wait   | 199.43ms | 80.76ms  |
  | Max wait     |   4.19ms | 35.76ms  |
  +--------------+----------+----------+

Results without KASAN:

free_pcppages_bulk contention (no KASAN):

  +--------------+----------+----------+
  | Metric       | No fix   | With fix |
  +--------------+----------+----------+
  | Contentions  |      240 |      133 |
  | Total wait   |  34.01ms | 24.61ms  |
  | Max wait     |   965us  |  1.35ms  |
  +--------------+----------+----------+
Quoted 1 line

Cool!

I noticed __round_jiffies_relative() exists and the description looks like it's meant for exactly this use case?

Quoted 23 lines
---
 mm/vmstat.c | 25 ++++++++++++++++++++++++-
 1 file changed, 24 insertions(+), 1 deletion(-)
mm/vmstat.c +11 -0
index 2370c6fb1fcd..2e94bd765606 100644
@@ -2032,6 +2032,29 @@ static int vmstat_refresh(const struct ctl_table *table, int write,
 }
 #endif /* CONFIG_PROC_FS */
 
+/*
+ * Return a per-cpu delay that spreads vmstat_update work across the stat
+ * interval.  Without this, round_jiffies_relative() aligns every CPU's
+ * timer to the same second boundary, causing a thundering-herd on
+ * zone->lock when multiple CPUs drain PCP pages simultaneously via
+ * decay_pcp_high() -> free_pcppages_bulk().
+ */
+static unsigned long vmstat_spread_delay(void)
+{
+	unsigned long interval = sysctl_stat_interval;
+	unsigned int nr_cpus = num_online_cpus();

mm/vmstat.c static int vmstat_refresh(const struct ctl_table *table, int write, · line 2055

Quoted 10 lines
mm/vmstat.c +10 -0
+
+	if (nr_cpus <= 1)
+		return round_jiffies_relative(interval);
+
+	/*
+	 * Spread per-cpu vmstat work evenly across the interval.  Don't
+	 * use round_jiffies_relative() here -- it would snap every CPU
+	 * back to the same second boundary, defeating the spread.
+	 */
+	return interval + (interval * (smp_processor_id() % nr_cpus)) / nr_cpus;

Hm doesn't this mean that lower id cpus will consistently fire in shorter intervals and higher id in longer intervals? What we want is same interval but differently offset, no?

Quoted 23 lines
mm/vmstat.c +3 -2
+}
+
 static void vmstat_update(struct work_struct *w)
 {
 	if (refresh_cpu_vm_stats(true)) {
@@ -2042,7 +2065,7 @@ static void vmstat_update(struct work_struct *w)
 		 */
 		queue_delayed_work_on(smp_processor_id(), mm_percpu_wq,
 				this_cpu_ptr(&vmstat_work),
-				round_jiffies_relative(sysctl_stat_interval));
+				vmstat_spread_delay());
 	}
 }
 
 
---

base-commit: cf7c3c02fdd0dfccf4d6611714273dcb538af2cb
change-id: 20260401-vmstat-048e0feaf344

Best regards,

Signature · 1 line
Breno Leitao <[email protected]>
#11 lore raw
Quoted 47 lines · Vlastimil Babka (SUSE)
Quoted 46 lines · Breno Leitao

vmstat_update uses round_jiffies_relative() when re-queuing itself, which aligns all CPUs' timers to the same second boundary. When many CPUs have pending PCP pages to drain, they all call decay_pcp_high() -> free_pcppages_bulk() simultaneously, serializing on zone->lock and hitting contention.

Introduce vmstat_spread_delay() which distributes each CPU's vmstat_update evenly across the stat interval instead of aligning them.

This does not increase the number of timer interrupts — each CPU still fires once per interval. The timers are simply staggered rather than aligned. Additionally, vmstat_work is DEFERRABLE_WORK, so it does not wake idle CPUs regardless of scheduling; the spread only affects CPUs that are already active

perf lock contention shows 7.5x reduction in zone->lock contention (872 -> 117 contentions, 199ms -> 81ms total wait) on a 72-CPU aarch64 system under memory pressure.

Tested on a 72-CPU aarch64 system using stress-ng --vm to generate memory allocation bursts. Lock contention was measured with:

perf lock contention -a -b -S free_pcppages_bulk

Results with KASAN enabled:

free_pcppages_bulk contention (KASAN):

  +--------------+----------+----------+
  | Metric       | No fix   | With fix |
  +--------------+----------+----------+
  | Contentions  |      872 |      117 |
  | Total wait   | 199.43ms | 80.76ms  |
  | Max wait     |   4.19ms | 35.76ms  |
  +--------------+----------+----------+

Results without KASAN:

free_pcppages_bulk contention (no KASAN):

  +--------------+----------+----------+
  | Metric       | No fix   | With fix |
  +--------------+----------+----------+
  | Contentions  |      240 |      133 |
  | Total wait   |  34.01ms | 24.61ms  |
  | Max wait     |   965us  |  1.35ms  |
  +--------------+----------+----------+
Quoted 6 lines
Quoted 1 line

Cool!

I noticed __round_jiffies_relative() exists and the description looks like it's meant for exactly this use case?

On closer look, using round_jiffies_relative() as before your patch means it's calling __round_jiffies_relative(j, raw_smp_processor_id()) so that's already doing this spread internally. You're also relying smp_processor_id() so it's not about using a different cpu id.

But your patch has better results, why? I still think it's not doing what it intends - I think it makes every cpu have different interval length (up to twice the original length), not skew. Is it that, or that the 3 jiffies skew per cpu used in round_jiffies_common() is insufficient? Or it a bug in its skew implementation?

Ideally once that's clear, the findings could be used to improve round_jiffies_common() and hopefully there's nothing here that's vmstat specific.

Thanks,
Vlastimil

Quoted 62 lines
Quoted 23 lines
---
 mm/vmstat.c | 25 ++++++++++++++++++++++++-
 1 file changed, 24 insertions(+), 1 deletion(-)
mm/vmstat.c +11 -0
index 2370c6fb1fcd..2e94bd765606 100644
@@ -2032,6 +2032,29 @@ static int vmstat_refresh(const struct ctl_table *table, int write,
 }
 #endif /* CONFIG_PROC_FS */
 
+/*
+ * Return a per-cpu delay that spreads vmstat_update work across the stat
+ * interval.  Without this, round_jiffies_relative() aligns every CPU's
+ * timer to the same second boundary, causing a thundering-herd on
+ * zone->lock when multiple CPUs drain PCP pages simultaneously via
+ * decay_pcp_high() -> free_pcppages_bulk().
+ */
+static unsigned long vmstat_spread_delay(void)
+{
+	unsigned long interval = sysctl_stat_interval;
+	unsigned int nr_cpus = num_online_cpus();

mm/vmstat.c static int vmstat_refresh(const struct ctl_table *table, int write, · line 2055

Quoted 10 lines
mm/vmstat.c +10 -0
+
+	if (nr_cpus <= 1)
+		return round_jiffies_relative(interval);
+
+	/*
+	 * Spread per-cpu vmstat work evenly across the interval.  Don't
+	 * use round_jiffies_relative() here -- it would snap every CPU
+	 * back to the same second boundary, defeating the spread.
+	 */
+	return interval + (interval * (smp_processor_id() % nr_cpus)) / nr_cpus;

Hm doesn't this mean that lower id cpus will consistently fire in shorter intervals and higher id in longer intervals? What we want is same interval but differently offset, no?

Quoted 23 lines
mm/vmstat.c +3 -2
+}
+
 static void vmstat_update(struct work_struct *w)
 {
 	if (refresh_cpu_vm_stats(true)) {
@@ -2042,7 +2065,7 @@ static void vmstat_update(struct work_struct *w)
 		 */
 		queue_delayed_work_on(smp_processor_id(), mm_percpu_wq,
 				this_cpu_ptr(&vmstat_work),
-				round_jiffies_relative(sysctl_stat_interval));
+				vmstat_spread_delay());
 	}
 }
 
 
---

base-commit: cf7c3c02fdd0dfccf4d6611714273dcb538af2cb
change-id: 20260401-vmstat-048e0feaf344

Best regards,

Signature · 1 line
Breno Leitao <[email protected]>
#12 lore raw
Quoted 17 lines
Quoted 5 lines

Cool!

I noticed __round_jiffies_relative() exists and the description looks like it's meant for exactly this use case?

On closer look, using round_jiffies_relative() as before your patch means it's calling __round_jiffies_relative(j, raw_smp_processor_id()) so that's already doing this spread internally. You're also relying smp_processor_id() so it's not about using a different cpu id.

But your patch has better results, why? I still think it's not doing what it intends - I think it makes every cpu have different interval length (up to twice the original length), not skew. Is it that, or that the 3 jiffies skew per cpu used in round_jiffies_common() is insufficient? Or it a bug in its skew implementation?

Quoted 3 lines

Ideally once that's clear, the findings could be used to improve round_jiffies_common() and hopefully there's nothing here that's vmstat specific.

Excellent observation. I believe there are two key differences:

1) The interval duration now varies per CPU. Specifically, vmstat_update() is scheduled at sysctl_stat_interval*2 for the highest CPU with my proposed change, rather than a uniform sysctl_stat_interval across all CPUs. (as you raised in the first email)

2) round_jiffies_relative() applies a 3-jiffies shift per CPU, whereas vmstat_spread_delay distributes all CPUs across the full second interval. (My tests were on HZ=1000)

I'll investigate this further to provide more concrete data.

Thanks for the review,
--breno

#13 lore raw
Quoted 28 lines · Breno Leitao
Quoted 17 lines
Quoted 5 lines

Cool!

I noticed __round_jiffies_relative() exists and the description looks like it's meant for exactly this use case?

On closer look, using round_jiffies_relative() as before your patch means it's calling __round_jiffies_relative(j, raw_smp_processor_id()) so that's already doing this spread internally. You're also relying smp_processor_id() so it's not about using a different cpu id.

But your patch has better results, why? I still think it's not doing what it intends - I think it makes every cpu have different interval length (up to twice the original length), not skew. Is it that, or that the 3 jiffies skew per cpu used in round_jiffies_common() is insufficient? Or it a bug in its skew implementation?

Quoted 3 lines

Ideally once that's clear, the findings could be used to improve round_jiffies_common() and hopefully there's nothing here that's vmstat specific.

Excellent observation. I believe there are two key differences:

1) The interval duration now varies per CPU. Specifically, vmstat_update() is scheduled at sysctl_stat_interval*2 for the highest CPU with my proposed change, rather than a uniform sysctl_stat_interval across all CPUs. (as you raised in the first email)

Quoted 5 lines

2) round_jiffies_relative() applies a 3-jiffies shift per CPU, whereas vmstat_spread_delay distributes all CPUs across the full second interval. (My tests were on HZ=1000)

I'll investigate this further to provide more concrete data.

After further investigation, I can confirm that both factors mentioned above contribute to the performance improvement.

However, we certainly don't want scenario (1) where the delay varies per CPU, resulting in the last CPU having vmstat_update() scheduled every 2 seconds instead of 1 second.

I've implemented a patch following Dmitry's suggestion, and the performance gains are measurable.

Here's my testing methodology:

1) Use ftrace to measure the execution time of refresh_cpu_vm_stats()
* Applied a custom instrumentation patch [1]

2) Execute stress-ng:
* stress-ng --vm 72 --vm-bytes 11256M --vm-method all --timeout 60s ; cat /sys/kernel/debug/tracing/trace

3) Parse the output using a Python script [2]

While the results are not as dramatic as initially reported (since approach (1) was good but incorrect), the improvement is still substantial:

  ┌─────────┬────────────┬────────────┬───────┐
  │ Metric  │ upstream*  │    fix**   │ Delta │
  ├─────────┼────────────┼────────────┼───────┤
  │ samples │ 36,981     │ 37,267     │ ~same │
  ├─────────┼────────────┼────────────┼───────┤
  │ avg     │ 31,511 ns  │ 21,337 ns  │ -32%  │
  ├─────────┼────────────┼────────────┼───────┤
  │ p50     │ 2,644 ns   │ 2,925 ns   │ ~same │
  ├─────────┼────────────┼────────────┼───────┤
  │ p99     │ 382,083 ns │ 304,357 ns │ -20%  │
  ├─────────┼────────────┼────────────┼───────┤
  │ max     │ 72.6 ms    │ 16.0 ms    │ -78%  │
  └─────────┴────────────┴────────────┴───────┘

* Upstream is based on linux-next commit f3e6330d7fe42 ("Add linux-next specific files for 20260407") ** "fix" contains the patch below:

commit 41b7aaa1a51f07fc1f0db0614d140fbca78463d3

Author: Breno Leitao <[email protected]>
Date:   Tue Apr 7 07:56:35 2026 -0700

mm/vmstat: spread per-cpu vmstat work to reduce zone->lock contention

vmstat_shepherd() queues all per-cpu vmstat_update work with zero delay, and vmstat_update() re-queues itself with round_jiffies_relative(), which clusters timers near the same second boundary due to the small per-CPU spread in round_jiffies_common(). On many-CPU systems this causes thundering-herd contention on zone->lock when multiple CPUs simultaneously call refresh_cpu_vm_stats() -> decay_pcp_high() -> free_pcppages_bulk().

Introduce vmstat_spread_delay() to assign each CPU a unique offset distributed evenly across sysctl_stat_interval. The shepherd uses this when initially queuing per-cpu work, and vmstat_update re-queues with a plain sysctl_stat_interval to preserve the spread (round_jiffies_relative would snap CPUs back to the same boundary).

mm/vmstat.c +22 -2
index 3704f6ca7a268..8d93eee3b1f75 100644
@@ -2040,6 +2040,22 @@ static int vmstat_refresh(const struct ctl_table *table, int write,
 }
 #endif /* CONFIG_PROC_FS */
 
+/*
+ * Return a per-cpu initial delay that spreads vmstat_update work evenly
+ * across the stat interval, so that CPUs do not all fire at the same
+ * second boundary.
+ */
+static unsigned long vmstat_spread_delay(int cpu)
+{
+	unsigned long interval = sysctl_stat_interval;
+	unsigned int nr_cpus = num_online_cpus();
+
+	if (nr_cpus <= 1)
+		return 0;
+
+	return (interval * (cpu % nr_cpus)) / nr_cpus;
+}
+
 static void vmstat_update(struct work_struct *w)
 {
 	if (refresh_cpu_vm_stats(true)) {
@@ -2047,10 +2063,13 @@ static void vmstat_update(struct work_struct *w)
 		 * Counters were updated so we expect more updates
 		 * to occur in the future. Keep on running the
 		 * update worker thread.
+		 * Avoid round_jiffies_relative() here -- it would snap
+		 * every CPU back to the same second boundary, undoing
+		 * the initial spread from vmstat_shepherd.
 		 */
 		queue_delayed_work_on(smp_processor_id(), mm_percpu_wq,
 				this_cpu_ptr(&vmstat_work),
-				round_jiffies_relative(sysctl_stat_interval));
+				sysctl_stat_interval);
 	}
 }
 
@@ -2148,7 +2167,8 @@ static void vmstat_shepherd(struct work_struct *w)
 				continue;
 
 			if (!delayed_work_pending(dw) && need_update(cpu))
-				queue_delayed_work_on(cpu, mm_percpu_wq, dw, 0);
+				queue_delayed_work_on(cpu, mm_percpu_wq, dw,
+						      vmstat_spread_delay(cpu));
 		}
 
 		cond_resched();
#14 lore raw
Quoted 33 lines · Breno Leitao
Quoted 32 lines · Breno Leitao
Quoted 17 lines
Quoted 5 lines

Cool!

I noticed __round_jiffies_relative() exists and the description looks like it's meant for exactly this use case?

On closer look, using round_jiffies_relative() as before your patch means it's calling __round_jiffies_relative(j, raw_smp_processor_id()) so that's already doing this spread internally. You're also relying smp_processor_id() so it's not about using a different cpu id.

But your patch has better results, why? I still think it's not doing what it intends - I think it makes every cpu have different interval length (up to twice the original length), not skew. Is it that, or that the 3 jiffies skew per cpu used in round_jiffies_common() is insufficient? Or it a bug in its skew implementation?

Quoted 3 lines

Ideally once that's clear, the findings could be used to improve round_jiffies_common() and hopefully there's nothing here that's vmstat specific.

Excellent observation. I believe there are two key differences:

1) The interval duration now varies per CPU. Specifically, vmstat_update() is scheduled at sysctl_stat_interval*2 for the highest CPU with my proposed change, rather than a uniform sysctl_stat_interval across all CPUs. (as you raised in the first email)

2) round_jiffies_relative() applies a 3-jiffies shift per CPU, whereas vmstat_spread_delay distributes all CPUs across the full second interval. (My tests were on HZ=1000)

Quoted 8 lines
Quoted 1 line

I'll investigate this further to provide more concrete data.

After further investigation, I can confirm that both factors mentioned above contribute to the performance improvement.

However, we certainly don't want scenario (1) where the delay varies per CPU, resulting in the last CPU having vmstat_update() scheduled every 2 seconds instead of 1 second.

Indeed.

Quoted 18 lines

I've implemented a patch following Dmitry's suggestion, and the performance gains are measurable.

Here's my testing methodology:

1) Use ftrace to measure the execution time of refresh_cpu_vm_stats()
* Applied a custom instrumentation patch [1]

2) Execute stress-ng:
* stress-ng --vm 72 --vm-bytes 11256M --vm-method all --timeout 60s ; cat /sys/kernel/debug/tracing/trace

3) Parse the output using a Python script [2]

While the results are not as dramatic as initially reported (since approach (1) was good but incorrect), the improvement is still substantial:

Quoted 13 lines
  ┌─────────┬────────────┬────────────┬───────┐
  │ Metric  │ upstream*  │    fix**   │ Delta │
  ├─────────┼────────────┼────────────┼───────┤
  │ samples │ 36,981     │ 37,267     │ ~same │
  ├─────────┼────────────┼────────────┼───────┤
  │ avg     │ 31,511 ns  │ 21,337 ns  │ -32%  │
  ├─────────┼────────────┼────────────┼───────┤
  │ p50     │ 2,644 ns   │ 2,925 ns   │ ~same │
  ├─────────┼────────────┼────────────┼───────┤
  │ p99     │ 382,083 ns │ 304,357 ns │ -20%  │
  ├─────────┼────────────┼────────────┼───────┤
  │ max     │ 72.6 ms    │ 16.0 ms    │ -78%  │
  └─────────┴────────────┴────────────┴───────┘

So you have 72 cpus, the vmstat interval is 1s, and what's the CONFIG_HZ? If it's 1000, it means 13 jiffies per cpu. Would changing the round_jiffies_common() implementation to add 13 jiffies per cpu instead of 3 have the same effect?

Quoted 22 lines

* Upstream is based on linux-next commit f3e6330d7fe42 ("Add linux-next specific files for 20260407") ** "fix" contains the patch below:

commit 41b7aaa1a51f07fc1f0db0614d140fbca78463d3

Author: Breno Leitao <[email protected]>
Date:   Tue Apr 7 07:56:35 2026 -0700

mm/vmstat: spread per-cpu vmstat work to reduce zone->lock contention

vmstat_shepherd() queues all per-cpu vmstat_update work with zero delay, and vmstat_update() re-queues itself with round_jiffies_relative(), which clusters timers near the same second boundary due to the small per-CPU spread in round_jiffies_common(). On many-CPU systems this causes thundering-herd contention on zone->lock when multiple CPUs simultaneously call refresh_cpu_vm_stats() -> decay_pcp_high() -> free_pcppages_bulk().

Quoted 7 lines

Introduce vmstat_spread_delay() to assign each CPU a unique offset distributed evenly across sysctl_stat_interval. The shepherd uses this when initially queuing per-cpu work, and vmstat_update re-queues with a plain sysctl_stat_interval to preserve the spread (round_jiffies_relative would snap CPUs back to the same boundary).

I think this approach could have the following problems:

- the initially spread delays can drift over time, there's nothing keeping them in sync
- not using round_jiffies_relative() means firing at other times than other timers that are using the rounding, so this could be working against the power savings effects of rounding
- it's a vmstat-specific workaround for some yet unclear underlying suboptimality that's likely not vmstat specific

Quoted 53 lines
mm/vmstat.c +22 -2
index 3704f6ca7a268..8d93eee3b1f75 100644
@@ -2040,6 +2040,22 @@ static int vmstat_refresh(const struct ctl_table *table, int write,
 }
 #endif /* CONFIG_PROC_FS */
 
+/*
+ * Return a per-cpu initial delay that spreads vmstat_update work evenly
+ * across the stat interval, so that CPUs do not all fire at the same
+ * second boundary.
+ */
+static unsigned long vmstat_spread_delay(int cpu)
+{
+	unsigned long interval = sysctl_stat_interval;
+	unsigned int nr_cpus = num_online_cpus();
+
+	if (nr_cpus <= 1)
+		return 0;
+
+	return (interval * (cpu % nr_cpus)) / nr_cpus;
+}
+
 static void vmstat_update(struct work_struct *w)
 {
 	if (refresh_cpu_vm_stats(true)) {
@@ -2047,10 +2063,13 @@ static void vmstat_update(struct work_struct *w)
 		 * Counters were updated so we expect more updates
 		 * to occur in the future. Keep on running the
 		 * update worker thread.
+		 * Avoid round_jiffies_relative() here -- it would snap
+		 * every CPU back to the same second boundary, undoing
+		 * the initial spread from vmstat_shepherd.
 		 */
 		queue_delayed_work_on(smp_processor_id(), mm_percpu_wq,
 				this_cpu_ptr(&vmstat_work),
-				round_jiffies_relative(sysctl_stat_interval));
+				sysctl_stat_interval);
 	}
 }
 
@@ -2148,7 +2167,8 @@ static void vmstat_shepherd(struct work_struct *w)
 				continue;
 
 			if (!delayed_work_pending(dw) && need_update(cpu))
-				queue_delayed_work_on(cpu, mm_percpu_wq, dw, 0);
+				queue_delayed_work_on(cpu, mm_percpu_wq, dw,
+						      vmstat_spread_delay(cpu));
 		}
 
 		cond_resched();
#15 lore raw
Quoted 12 lines · Vlastimil Babka (SUSE)
Quoted 8 lines · Breno Leitao
Quoted 4 lines · Breno Leitao

1) The interval duration now varies per CPU. Specifically, vmstat_update() is scheduled at sysctl_stat_interval*2 for the highest CPU with my proposed change, rather than a uniform sysctl_stat_interval across all CPUs. (as you raised in the first email)

However, we certainly don't want scenario (1) where the delay varies per CPU, resulting in the last CPU having vmstat_update() scheduled every 2 seconds instead of 1 second.

Indeed.

....

Quoted 2 lines

So you have 72 cpus, the vmstat interval is 1s, and what's the CONFIG_HZ?

Yes, CONFIG_HZ=1000 in my configuration.

Quoted 3 lines

If it's 1000, it means 13 jiffies per cpu. Would changing the round_jiffies_common() implementation to add 13 jiffies per cpu instead of 3 have the same effect?

That approach would increase the spread in round_jiffies_common(), but it brings us back to scenario 1 - each CPU would have a variable rescheduling delay, which defeats the purpose of maintaining consistent timing.

Quoted 29 lines

commit 41b7aaa1a51f07fc1f0db0614d140fbca78463d3

Author: Breno Leitao <[email protected]>
Date:   Tue Apr 7 07:56:35 2026 -0700

mm/vmstat: spread per-cpu vmstat work to reduce zone->lock contention

vmstat_shepherd() queues all per-cpu vmstat_update work with zero delay, and vmstat_update() re-queues itself with round_jiffies_relative(), which clusters timers near the same second boundary due to the small per-CPU spread in round_jiffies_common(). On many-CPU systems this causes thundering-herd contention on zone->lock when multiple CPUs simultaneously call refresh_cpu_vm_stats() -> decay_pcp_high() -> free_pcppages_bulk().

Introduce vmstat_spread_delay() to assign each CPU a unique offset distributed evenly across sysctl_stat_interval. The shepherd uses this when initially queuing per-cpu work, and vmstat_update re-queues with a plain sysctl_stat_interval to preserve the spread (round_jiffies_relative would snap CPUs back to the same boundary).

I think this approach could have the following problems:

Quoted 7 lines

- the initially spread delays can drift over time, there's nothing keeping them in sync
- not using round_jiffies_relative() means firing at other times than other timers that are using the rounding, so this could be working against the power savings effects of rounding
- it's a vmstat-specific workaround for some yet unclear underlying suboptimality that's likely not vmstat specific

I believe the issue is that vmstat's current use of round_jiffies_relative() fundamentally solve a different problem than the one we trying to solve. The round_jiffies_relative() mechanism is designed for a different purpose, as documented:

"""
By rounding these timers to whole seconds, all such timers will fire at the same time, rather than at various times spread out. The goal of this is to have the CPU wake up less, which saves power.
"""

Backing up a bit, let me summarize the problem:

1) vmstat_shepherd() starts all vmstat_update workers simultaneously (or nearly so)

2) All vmstat_update() workers then reschedule themselves for the next second with only 216ms of variance across 72 cores

Upon further reflection, issue (2) wouldn't exist if we simply avoided starting all workqueues simultaneously in the first place.

That said, I created a hacky code to find how often the vmstat_update is called, and it is not uncommon to see cases where vmstat_update is called a few jiffies after it was called.

Debugging further, I found a race between vmstat_shepherd() trying to schedule vmstat_update() even when it is already running, which happens a lot, given they both run at the

commit 86e27727b6cc9fbe8bf67818a065105fd30bc7e8 (HEAD -> b4/vmstat)

Author: Breno Leitao <[email protected]>
Date:   Wed Apr 8 08:11:01 2026 -0700

mm/vmstat: warn when shepherd schedules vmstat_update while it is running

vmstat_shepherd checks delayed_work_pending() to decide whether to queue vmstat_update for a CPU. However, delayed_work_pending() only tests the WORK_STRUCT_PENDING_BIT, which is cleared as soon as a worker thread picks up the work for execution.

This means that while vmstat_update is actively running on a CPU, delayed_work_pending() returns false, and the shepherd may queue another invocation if need_update() also returns true (which it can, since per-cpu counters are still being flushed mid-execution).

Add a WARN_ONCE to make this race visible: fire a warning when the shepherd is about to queue vmstat_update for a CPU where it is currently executing, i.e., work_busy() reports WORK_BUSY_RUNNING but delayed_work_pending() does not prevent the re-queue.

mm/vmstat.c +5 -1
index 2370c6fb1fcd6..8d53242e7aa66 100644
@@ -2139,8 +2139,12 @@ static void vmstat_shepherd(struct work_struct *w)
                        if (cpu_is_isolated(cpu))
                                continue;
 
-                       if (!delayed_work_pending(dw) && need_update(cpu))
+                       if (!delayed_work_pending(dw) && need_update(cpu)) {
+                               WARN_ONCE(work_busy(&dw->work) & WORK_BUSY_RUNNING,
+                                         "cpu%d: vmstat_update already running, scheduling again\n",
+                                         cpu);
                                queue_delayed_work_on(cpu, mm_percpu_wq, dw, 0);
+                       }
                }
 
                cond_resched();

The fix is a one-line change: !delayed_work_pending(dw) → !work_busy(&dw->work)

#16 lore raw
Quoted 21 lines · Breno Leitao

On Wed, Apr 08, 2026 at 12:13:04PM +0200, Vlastimil Babka (SUSE) wrote:

mm/vmstat.c +5 -1
index 2370c6fb1fcd6..8d53242e7aa66 100644
@@ -2139,8 +2139,12 @@ static void vmstat_shepherd(struct work_struct *w)
                        if (cpu_is_isolated(cpu))
                                continue;
 
-                       if (!delayed_work_pending(dw) && need_update(cpu))
+                       if (!delayed_work_pending(dw) && need_update(cpu)) {
+                               WARN_ONCE(work_busy(&dw->work) & WORK_BUSY_RUNNING,
+                                         "cpu%d: vmstat_update already running, scheduling again\n",
+                                         cpu);
                                queue_delayed_work_on(cpu, mm_percpu_wq, dw, 0);
+                       }
                }
 
                cond_resched();
Quoted 1 line

The fix is a one-line change: !delayed_work_pending(dw) → !work_busy(&dw->work)

In my testing, this race condition occurs more frequently than expected, likely due to the timer configurations we've been discussing throughout this thread.

I developed a diagnostic patch to monitor vmstat_update worker scheduling frequency, and the results show consistently low values. Avoiding rescheduling a worker that is already also reduces the contention on stress-ng test case.

commit d725f0664b70aa5c677215b0fc1abc0117aaf114

Author: Breno Leitao <[email protected]>
Date:   Wed Apr 8 09:01:02 2026 -0700

mm/vmstat: fix vmstat_shepherd double-scheduling vmstat_update

vmstat_shepherd uses delayed_work_pending() to check whether vmstat_update is already scheduled for a given CPU before queuing it. However, delayed_work_pending() only tests WORK_STRUCT_PENDING_BIT, which is cleared the moment a worker thread picks up the work to execute it.

This means that while vmstat_update is actively running on a CPU, delayed_work_pending() returns false. If need_update() also returns true at that point (per-cpu counters not yet zeroed mid-flush), the shepherd queues a second invocation with delay=0, causing vmstat_update to run again immediately after finishing.

On a 72-CPU system this race is readily observable: before the fix, many CPUs show invocation gaps well below 500 jiffies (the minimum round_jiffies_relative() can produce), with the most extreme cases reaching 0 jiffies—vmstat_update called twice within the same jiffy.

Fix this by replacing delayed_work_pending() with work_busy(), which returns non-zero for both WORK_BUSY_PENDING (timer armed or work queued) and WORK_BUSY_RUNNING (work currently executing). The shepherd now correctly skips a CPU in all busy states.

After the fix, all sub-jiffy and most sub-100-jiffie gaps disappear. The remaining early invocations have gaps in the 700–999 jiffie range, attributable to round_jiffies_relative() aligning to a nearer jiffie-second boundary rather than to this race.

Each spurious vmstat_update invocation has a measurable side effect:
refresh_cpu_vm_stats() calls decay_pcp_high() for every zone, which drains idle per-CPU pages back to the buddy allocator via free_pcppages_bulk(), taking the zone spinlock each time. Eliminating the double-scheduling therefore reduces zone lock contention directly. On a 72-CPU stress-ng workload measured with perf lock contention:

free_pcppages_bulk contention count: ~55% reduction
free_pcppages_bulk total wait time: ~57% reduction
free_pcppages_bulk max wait time: ~47% reduction

Note: work_busy() is inherently racy—between the check and the subsequent queue_delayed_work_on() call, vmstat_update can finish execution, leaving the work neither pending nor running. In that narrow window the shepherd can still queue a second invocation. After the fix, this residual race is rare and produces only occasional small gaps, a significant improvement over the systematic double-scheduling seen with delayed_work_pending().

mm/vmstat.c +1 -1
index d59eff1582547..5489549241b51 100644
@@ -2156,7 +2156,7 @@ static void vmstat_shepherd(struct work_struct *w)
 			if (cpu_is_isolated(cpu))
 				continue;
 
-			if (!delayed_work_pending(dw) && need_update(cpu))
+			if (!work_busy(&dw->work) && need_update(cpu))
 				queue_delayed_work_on(cpu, mm_percpu_wq, dw, 0);
 		}
 
reviewer Reviewed-by
#17 lore raw
Quoted 73 lines · Breno Leitao
Quoted 21 lines · Breno Leitao

On Wed, Apr 08, 2026 at 12:13:04PM +0200, Vlastimil Babka (SUSE) wrote:

mm/vmstat.c +5 -1
index 2370c6fb1fcd6..8d53242e7aa66 100644
@@ -2139,8 +2139,12 @@ static void vmstat_shepherd(struct work_struct *w)
                        if (cpu_is_isolated(cpu))
                                continue;
 
-                       if (!delayed_work_pending(dw) && need_update(cpu))
+                       if (!delayed_work_pending(dw) && need_update(cpu)) {
+                               WARN_ONCE(work_busy(&dw->work) & WORK_BUSY_RUNNING,
+                                         "cpu%d: vmstat_update already running, scheduling again\n",
+                                         cpu);
                                queue_delayed_work_on(cpu, mm_percpu_wq, dw, 0);
+                       }
                }
 
                cond_resched();
Quoted 1 line

The fix is a one-line change: !delayed_work_pending(dw) → !work_busy(&dw->work)

In my testing, this race condition occurs more frequently than expected, likely due to the timer configurations we've been discussing throughout this thread.

I developed a diagnostic patch to monitor vmstat_update worker scheduling frequency, and the results show consistently low values. Avoiding rescheduling a worker that is already also reduces the contention on stress-ng test case.

commit d725f0664b70aa5c677215b0fc1abc0117aaf114

Author: Breno Leitao <[email protected]>
Date:   Wed Apr 8 09:01:02 2026 -0700

mm/vmstat: fix vmstat_shepherd double-scheduling vmstat_update

vmstat_shepherd uses delayed_work_pending() to check whether vmstat_update is already scheduled for a given CPU before queuing it. However, delayed_work_pending() only tests WORK_STRUCT_PENDING_BIT, which is cleared the moment a worker thread picks up the work to execute it.

This means that while vmstat_update is actively running on a CPU, delayed_work_pending() returns false. If need_update() also returns true at that point (per-cpu counters not yet zeroed mid-flush), the shepherd queues a second invocation with delay=0, causing vmstat_update to run again immediately after finishing.

On a 72-CPU system this race is readily observable: before the fix, many CPUs show invocation gaps well below 500 jiffies (the minimum round_jiffies_relative() can produce), with the most extreme cases reaching 0 jiffies—vmstat_update called twice within the same jiffy.

Fix this by replacing delayed_work_pending() with work_busy(), which returns non-zero for both WORK_BUSY_PENDING (timer armed or work queued) and WORK_BUSY_RUNNING (work currently executing). The shepherd now correctly skips a CPU in all busy states.

After the fix, all sub-jiffy and most sub-100-jiffie gaps disappear. The remaining early invocations have gaps in the 700–999 jiffie range, attributable to round_jiffies_relative() aligning to a nearer jiffie-second boundary rather than to this race.

Each spurious vmstat_update invocation has a measurable side effect:
refresh_cpu_vm_stats() calls decay_pcp_high() for every zone, which drains idle per-CPU pages back to the buddy allocator via free_pcppages_bulk(), taking the zone spinlock each time. Eliminating the double-scheduling therefore reduces zone lock contention directly. On a 72-CPU stress-ng workload measured with perf lock contention:

Quoted 3 lines

free_pcppages_bulk contention count: ~55% reduction
free_pcppages_bulk total wait time: ~57% reduction
free_pcppages_bulk max wait time: ~47% reduction

Great!

Quoted 9 lines

Note: work_busy() is inherently racy—between the check and the subsequent queue_delayed_work_on() call, vmstat_update can finish execution, leaving the work neither pending nor running. In that narrow window the shepherd can still queue a second invocation. After the fix, this residual race is rare and produces only occasional small gaps, a significant improvement over the systematic double-scheduling seen with delayed_work_pending().

Quoted 13 lines
mm/vmstat.c +1 -1
index d59eff1582547..5489549241b51 100644
@@ -2156,7 +2156,7 @@ static void vmstat_shepherd(struct work_struct *w)
 			if (cpu_is_isolated(cpu))
 				continue;
 
-			if (!delayed_work_pending(dw) && need_update(cpu))
+			if (!work_busy(&dw->work) && need_update(cpu))
 				queue_delayed_work_on(cpu, mm_percpu_wq, dw, 0);
 		}
 
Reviewed-by
#18 lore raw
In reply to Breno Leitao

[...]

Quoted 24 lines

commit d725f0664b70aa5c677215b0fc1abc0117aaf114

Author: Breno Leitao <[email protected]>
Date:   Wed Apr 8 09:01:02 2026 -0700

mm/vmstat: fix vmstat_shepherd double-scheduling vmstat_update

vmstat_shepherd uses delayed_work_pending() to check whether vmstat_update is already scheduled for a given CPU before queuing it. However, delayed_work_pending() only tests WORK_STRUCT_PENDING_BIT, which is cleared the moment a worker thread picks up the work to execute it.

This means that while vmstat_update is actively running on a CPU, delayed_work_pending() returns false. If need_update() also returns true at that point (per-cpu counters not yet zeroed mid-flush), the shepherd queues a second invocation with delay=0, causing vmstat_update to run again immediately after finishing.

On a 72-CPU system this race is readily observable: before the fix, many CPUs show invocation gaps well below 500 jiffies (the minimum round_jiffies_relative() can produce), with the most extreme cases reaching 0 jiffies—vmstat_update called twice within the same jiffy.

Quoted 4 lines

Fix this by replacing delayed_work_pending() with work_busy(), which returns non-zero for both WORK_BUSY_PENDING (timer armed or work queued) and WORK_BUSY_RUNNING (work currently executing). The shepherd now correctly skips a CPU in all busy states.

Great find. Thanks for taking the time to track this down.

Quoted 6 lines
mm/vmstat.c +0 -0
index d59eff1582547..5489549241b51 100644

mm/vmstat.c static void vmstat_shepherd(struct work_struct *w) · line 2162

Quoted 8 lines
mm/vmstat.c +1 -1
@@ -2156,7 +2156,7 @@ static void vmstat_shepherd(struct work_struct *w)
 			if (cpu_is_isolated(cpu))
 				continue;
 
-			if (!delayed_work_pending(dw) && need_update(cpu))
+			if (!work_busy(&dw->work) && need_update(cpu))
 				queue_delayed_work_on(cpu, mm_percpu_wq, dw, 0);
 		}
 
#19 lore raw
In reply to Breno Leitao
Quoted 19 lines · Breno Leitao
Quoted 12 lines · Vlastimil Babka (SUSE)
Quoted 8 lines · Breno Leitao
Quoted 4 lines · Breno Leitao

1) The interval duration now varies per CPU. Specifically, vmstat_update() is scheduled at sysctl_stat_interval*2 for the highest CPU with my proposed change, rather than a uniform sysctl_stat_interval across all CPUs. (as you raised in the first email)

However, we certainly don't want scenario (1) where the delay varies per CPU, resulting in the last CPU having vmstat_update() scheduled every 2 seconds instead of 1 second.

Indeed.

....

Quoted 2 lines

So you have 72 cpus, the vmstat interval is 1s, and what's the CONFIG_HZ?

Yes, CONFIG_HZ=1000 in my configuration.

Quoted 8 lines
Quoted 3 lines

If it's 1000, it means 13 jiffies per cpu. Would changing the round_jiffies_common() implementation to add 13 jiffies per cpu instead of 3 have the same effect?

That approach would increase the spread in round_jiffies_common(), but it brings us back to scenario 1 - each CPU would have a variable rescheduling delay, which defeats the purpose of maintaining consistent timing.

I think it's not the case and round_jiffies_common() prevents it, more below.

Quoted 42 lines
Quoted 29 lines

commit 41b7aaa1a51f07fc1f0db0614d140fbca78463d3

Author: Breno Leitao <[email protected]>
Date:   Tue Apr 7 07:56:35 2026 -0700

mm/vmstat: spread per-cpu vmstat work to reduce zone->lock contention

vmstat_shepherd() queues all per-cpu vmstat_update work with zero delay, and vmstat_update() re-queues itself with round_jiffies_relative(), which clusters timers near the same second boundary due to the small per-CPU spread in round_jiffies_common(). On many-CPU systems this causes thundering-herd contention on zone->lock when multiple CPUs simultaneously call refresh_cpu_vm_stats() -> decay_pcp_high() -> free_pcppages_bulk().

Introduce vmstat_spread_delay() to assign each CPU a unique offset distributed evenly across sysctl_stat_interval. The shepherd uses this when initially queuing per-cpu work, and vmstat_update re-queues with a plain sysctl_stat_interval to preserve the spread (round_jiffies_relative would snap CPUs back to the same boundary).

I think this approach could have the following problems:

Quoted 7 lines

- the initially spread delays can drift over time, there's nothing keeping them in sync
- not using round_jiffies_relative() means firing at other times than other timers that are using the rounding, so this could be working against the power savings effects of rounding
- it's a vmstat-specific workaround for some yet unclear underlying suboptimality that's likely not vmstat specific

I believe the issue is that vmstat's current use of round_jiffies_relative() fundamentally solve a different problem than the one we trying to solve. The round_jiffies_relative() mechanism is designed for a different purpose, as documented:

Quoted 5 lines

"""
By rounding these timers to whole seconds, all such timers will fire at the same time, rather than at various times spread out. The goal of this is to have the CPU wake up less, which saves power.
"""

This is true, but only in the context of a single CPU, to minimize its wakeups. What it doesn't say is that every CPU has its "whole second" moment shifted relatively to others. Which doesn't contradict the above, except that "whole seconds" without further details can be misleading.

I've finally looked at round_jiffies_common() in detail and after initial confusion I believe it doesn't introduce different delays for different cpus. One observation is that in the middle we round "j" to HZ, thus a whole second value, and then

        /* now that we have rounded, subtract the extra skew again */
        j -= cpu * 3;

so the final jiffies target is slightly different for every cpu. This alone could indeed shorten delays for higher cpu numbers, so crucially there's also:

        j += cpu * 3;

*before* the rounding step. The rounding can be also down instead of up if the remainder is less than 1/4s.

So I'm now rather convinced that it works fine. Suppose we have a delayed work queuing itself every second on cpu C. It will be executed for the second (since boot) N at jiffies == (N*HZ - 3*C), then this will reschedule it for the second N+1 at jiffies == ((N+1)*HZ - 3*C) (as long as it's finished the previous iteration and requeued itself in 1/4 second, otherwise it will be N+2). So I believe it will maintain the spread among cpus without drifting over time, and not introduce any variable delay depending on the cpu id.

Quoted 7 lines

Backing up a bit, let me summarize the problem:

1) vmstat_shepherd() starts all vmstat_update workers simultaneously (or nearly so)

2) All vmstat_update() workers then reschedule themselves for the next second with only 216ms of variance across 72 cores

My point in suggesting 13 of 3 per cpu was to find out whether the 216ms spread is really insufficient for vmstat - I would be surprised if it was the case.

Quoted 2 lines

Upon further reflection, issue (2) wouldn't exist if we simply avoided starting all workqueues simultaneously in the first place.

I still doubt it would work without continuously controlling for the drift (which round_jiffies_common() does). I think it the existence of the shared lock and initially some random perturbations would eventually result in them becoming synchronized, like metronomes sitting on the same table.

But thanks to your new findings below this is all hopefully now moot :)

Quoted 51 lines

That said, I created a hacky code to find how often the vmstat_update is called, and it is not uncommon to see cases where vmstat_update is called a few jiffies after it was called.

Debugging further, I found a race between vmstat_shepherd() trying to schedule vmstat_update() even when it is already running, which happens a lot, given they both run at the

commit 86e27727b6cc9fbe8bf67818a065105fd30bc7e8 (HEAD -> b4/vmstat)

Author: Breno Leitao <[email protected]>
Date:   Wed Apr 8 08:11:01 2026 -0700

mm/vmstat: warn when shepherd schedules vmstat_update while it is running

vmstat_shepherd checks delayed_work_pending() to decide whether to queue vmstat_update for a CPU. However, delayed_work_pending() only tests the WORK_STRUCT_PENDING_BIT, which is cleared as soon as a worker thread picks up the work for execution.

This means that while vmstat_update is actively running on a CPU, delayed_work_pending() returns false, and the shepherd may queue another invocation if need_update() also returns true (which it can, since per-cpu counters are still being flushed mid-execution).

Add a WARN_ONCE to make this race visible: fire a warning when the shepherd is about to queue vmstat_update for a CPU where it is currently executing, i.e., work_busy() reports WORK_BUSY_RUNNING but delayed_work_pending() does not prevent the re-queue.

mm/vmstat.c +5 -1
index 2370c6fb1fcd6..8d53242e7aa66 100644
@@ -2139,8 +2139,12 @@ static void vmstat_shepherd(struct work_struct *w)
                        if (cpu_is_isolated(cpu))
                                continue;
 
-                       if (!delayed_work_pending(dw) && need_update(cpu))
+                       if (!delayed_work_pending(dw) && need_update(cpu)) {
+                               WARN_ONCE(work_busy(&dw->work) & WORK_BUSY_RUNNING,
+                                         "cpu%d: vmstat_update already running, scheduling again\n",
+                                         cpu);
                                queue_delayed_work_on(cpu, mm_percpu_wq, dw, 0);
+                       }
                }
 
                cond_resched();

The fix is a one-line change: !delayed_work_pending(dw) → !work_busy(&dw->work)

#20 lore raw
In reply to Vlastimil Babka (SUSE)

On Wed, Apr 01, 2026 at 07:46:35PM +0200, Vlastimil Babka (SUSE) wrote:

[...]

Quoted 17 lines
Quoted 17 lines
(quoted diff) +17 -0
+/*
+ * Return a per-cpu delay that spreads vmstat_update work across the stat
+ * interval.  Without this, round_jiffies_relative() aligns every CPU's
+ * timer to the same second boundary, causing a thundering-herd on
+ * zone->lock when multiple CPUs drain PCP pages simultaneously via
+ * decay_pcp_high() -> free_pcppages_bulk().
+ */
+static unsigned long vmstat_spread_delay(void)
+{
+	unsigned long interval = sysctl_stat_interval;
+	unsigned int nr_cpus = num_online_cpus();
+
+	if (nr_cpus <= 1)
+		return round_jiffies_relative(interval);
+
+	/*
+	 * Spread per-cpu vmstat work evenly across the interval.  Don't
Quoted 8 lines

Review

Quoted 4 lines
(quoted diff) +4 -0
+	 * use round_jiffies_relative() here -- it would snap every CPU
+	 * back to the same second boundary, defeating the spread.
+	 */
+	return interval + (interval * (smp_processor_id() % nr_cpus)) / nr_cpus;

Hm doesn't this mean that lower id cpus will consistently fire in shorter intervals and higher id in longer intervals? What we want is same interval but differently offset, no?

Yes, I think that's a valid concern, this effectively skews the interval rather than just introducing a phase offset.

I initially thought this might explain the increase in max wait, but it turns out the columns were just swapped.

Spreading the initial scheduling and then requeueing with a constant interval sounds like a reasonable alternative, e.g. below.

From 56ed7e17b32f0a7ce433caed87650b0de8246c4e Mon Sep 17 00:00:00 2001
From: Dmitry Ilvokhin <[email protected]>
Date: Thu, 2 Apr 2026 04:49:06 -0700
Subject: [PATCH] mm/vmstat: stagger per-cpu vmstat updates to avoid zone->lock
contention

Fix by spreading the shepherd's initial wakeup across the stat plain sysctl_stat_interval to preserve the stagger. Every CPU still fires once per interval, same frequency, different phase.

---
 mm/vmstat.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)
mm/vmstat.c +3 -2
index 2370c6fb1fcd..aee99786718a 100644
@@ -2042,7 +2042,7 @@ static void vmstat_update(struct work_struct *w)
 		 */
 		queue_delayed_work_on(smp_processor_id(), mm_percpu_wq,
 				this_cpu_ptr(&vmstat_work),
-				round_jiffies_relative(sysctl_stat_interval));
+				sysctl_stat_interval);
 	}
 }
 
@@ -2140,7 +2140,8 @@ static void vmstat_shepherd(struct work_struct *w)
 				continue;
 
 			if (!delayed_work_pending(dw) && need_update(cpu))
-				queue_delayed_work_on(cpu, mm_percpu_wq, dw, 0);
+				queue_delayed_work_on(cpu, mm_percpu_wq, dw,
+					(sysctl_stat_interval * cpu) / nr_cpu_ids);
 		}
 
 		cond_resched();
Signature · 1 line
2.52.0
reviewer Acked-by on v1 1/1
#21 lore raw
In reply to Breno Leitao
Quoted 46 lines · Breno Leitao

vmstat_update uses round_jiffies_relative() when re-queuing itself, which aligns all CPUs' timers to the same second boundary. When many CPUs have pending PCP pages to drain, they all call decay_pcp_high() -> free_pcppages_bulk() simultaneously, serializing on zone->lock and hitting contention.

Introduce vmstat_spread_delay() which distributes each CPU's vmstat_update evenly across the stat interval instead of aligning them.

This does not increase the number of timer interrupts — each CPU still fires once per interval. The timers are simply staggered rather than aligned. Additionally, vmstat_work is DEFERRABLE_WORK, so it does not wake idle CPUs regardless of scheduling; the spread only affects CPUs that are already active

perf lock contention shows 7.5x reduction in zone->lock contention (872 -> 117 contentions, 199ms -> 81ms total wait) on a 72-CPU aarch64 system under memory pressure.

Tested on a 72-CPU aarch64 system using stress-ng --vm to generate memory allocation bursts. Lock contention was measured with:

perf lock contention -a -b -S free_pcppages_bulk

Results with KASAN enabled:

free_pcppages_bulk contention (KASAN):

  +--------------+----------+----------+
  | Metric       | No fix   | With fix |
  +--------------+----------+----------+
  | Contentions  |      872 |      117 |
  | Total wait   | 199.43ms | 80.76ms  |
  | Max wait     |   4.19ms | 35.76ms  |
  +--------------+----------+----------+

Results without KASAN:

free_pcppages_bulk contention (no KASAN):

  +--------------+----------+----------+
  | Metric       | No fix   | With fix |
  +--------------+----------+----------+
  | Contentions  |      240 |      133 |
  | Total wait   |  34.01ms | 24.61ms  |
  | Max wait     |   965us  |  1.35ms  |
  +--------------+----------+----------+
Quoted 1 line

Makes sense

Thanks!

Quoted 56 lines
---
 mm/vmstat.c | 25 ++++++++++++++++++++++++-
 1 file changed, 24 insertions(+), 1 deletion(-)
mm/vmstat.c +24 -2
index 2370c6fb1fcd..2e94bd765606 100644
@@ -2032,6 +2032,29 @@ static int vmstat_refresh(const struct ctl_table *table, int write,
 }
 #endif /* CONFIG_PROC_FS */
 
+/*
+ * Return a per-cpu delay that spreads vmstat_update work across the stat
+ * interval.  Without this, round_jiffies_relative() aligns every CPU's
+ * timer to the same second boundary, causing a thundering-herd on
+ * zone->lock when multiple CPUs drain PCP pages simultaneously via
+ * decay_pcp_high() -> free_pcppages_bulk().
+ */
+static unsigned long vmstat_spread_delay(void)
+{
+	unsigned long interval = sysctl_stat_interval;
+	unsigned int nr_cpus = num_online_cpus();
+
+	if (nr_cpus <= 1)
+		return round_jiffies_relative(interval);
+
+	/*
+	 * Spread per-cpu vmstat work evenly across the interval.  Don't
+	 * use round_jiffies_relative() here -- it would snap every CPU
+	 * back to the same second boundary, defeating the spread.
+	 */
+	return interval + (interval * (smp_processor_id() % nr_cpus)) / nr_cpus;
+}
+
 static void vmstat_update(struct work_struct *w)
 {
 	if (refresh_cpu_vm_stats(true)) {
@@ -2042,7 +2065,7 @@ static void vmstat_update(struct work_struct *w)
 		 */
 		queue_delayed_work_on(smp_processor_id(), mm_percpu_wq,
 				this_cpu_ptr(&vmstat_work),
-				round_jiffies_relative(sysctl_stat_interval));
+				vmstat_spread_delay());
 	}
 }
 
 
---

base-commit: cf7c3c02fdd0dfccf4d6611714273dcb538af2cb
change-id: 20260401-vmstat-048e0feaf344

Best regards,

Signature · 1 line
Breno Leitao <[email protected]>
Signature · 2 lines
Michal Hocko
SUSE Labs
#22 lore raw
In reply to Breno Leitao
Quoted 8 lines · Breno Leitao

vmstat_update uses round_jiffies_relative() when re-queuing itself, which aligns all CPUs' timers to the same second boundary. When many CPUs have pending PCP pages to drain, they all call decay_pcp_high() -> free_pcppages_bulk() simultaneously, serializing on zone->lock and hitting contention.

Introduce vmstat_spread_delay() which distributes each CPU's vmstat_update evenly across the stat interval instead of aligning them.

But, uh, round_jiffies_relative() is _supposed_ to do that! Look at both the documentation and implementation. Why isn't it working?

#23 lore raw

Hello Matthew,

Quoted 7 lines · Matthew Wilcox
Quoted 6 lines · Breno Leitao

vmstat_update uses round_jiffies_relative() when re-queuing itself, which aligns all CPUs' timers to the same second boundary. When many CPUs have pending PCP pages to drain, they all call decay_pcp_high() -> free_pcppages_bulk() simultaneously, serializing on zone->lock and hitting contention.

Quoted 5 lines
Quoted 2 lines

Introduce vmstat_spread_delay() which distributes each CPU's vmstat_update evenly across the stat interval instead of aligning them.

But, uh, round_jiffies_relative() is _supposed_ to do that! Look at both the documentation and implementation. Why isn't it working?

The documentation is a bit trick on me, sorry. This is what I read:

"By rounding these timers to whole seconds, all such timers will fire
at the same time, rather than at various times spread out. The goal of this is to have the CPU wake up less, which saves power."

This is the full documentation:

/**
* round_jiffies_relative - function to round jiffies to a full second
* @j: the time in (relative) jiffies that should be rounded
*
* round_jiffies_relative() rounds a time delta in the future (in jiffies)
* up or down to (approximately) full seconds. This is useful for timers
* for which the exact time they fire does not matter too much, as long as
* they fire approximately every X seconds.
*
* By rounding these timers to whole seconds, all such timers will fire
* at the same time, rather than at various times spread out. The goal
* of this is to have the CPU wake up less, which saves power. *
* The return value is the rounded version of the @j parameter. */
unsigned long round_jiffies_relative(unsigned long j)

{
        return __round_jiffies_relative(j, raw_smp_processor_id());
}

EXPORT_SYMBOL_GPL(round_jiffies_relative);