Skip to contentSkip to conversation

mm/vmstat: fix vmstat_shepherd double-scheduling vmstat_update

PATCH Applied Breno Leitao 5 participants 8 messages Apr 9 – Apr 10

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH v2 1/1 mm/vmstat: fix vmstat_shepherd double-scheduling vmstat_update

vmstat_shepherd uses delayed_work_pending() to check whether vmstat_update is already scheduled for a given CPU before queuing it. However, delayed_work_pending() only tests WORK_STRUCT_PENDING_BIT, which is cleared the moment a worker thread picks up the work to execute it.

This means that while vmstat_update is actively running on a CPU, delayed_work_pending() returns false. If need_update() also returns true at that point (per-cpu counters not yet zeroed mid-flush), the shepherd queues a second invocation with delay=0, causing vmstat_update to run again immediately after finishing.

On a 72-CPU system this race is readily observable: before the fix, many CPUs show invocation gaps well below 500 jiffies (the minimum round_jiffies_relative() can produce), with the most extreme cases reaching 0 jiffies—vmstat_update called twice within the same jiffy.

Fix this by replacing delayed_work_pending() with work_busy(), which returns non-zero for both WORK_BUSY_PENDING (timer armed or work queued) and WORK_BUSY_RUNNING (work currently executing). The shepherd now correctly skips a CPU in all busy states.

After the fix, all sub-jiffy and most sub-100-jiffie gaps disappear. The remaining early invocations have gaps in the 700–999 jiffie range, attributable to round_jiffies_relative() aligning to a nearer jiffie-second boundary rather than to this race.

Each spurious vmstat_update invocation has a measurable side effect:
refresh_cpu_vm_stats() calls decay_pcp_high() for every zone, which drains idle per-CPU pages back to the buddy allocator via free_pcppages_bulk(), taking the zone spinlock each time. Eliminating the double-scheduling therefore reduces zone lock contention directly. On a 72-CPU stress-ng workload measured with perf lock contention:

  free_pcppages_bulk contention count:  ~55% reduction
  free_pcppages_bulk total wait time:   ~57% reduction
  free_pcppages_bulk max wait time:     ~47% reduction

Note: work_busy() is inherently racy—between the check and the subsequent queue_delayed_work_on() call, vmstat_update can finish execution, leaving the work neither pending nor running. In that narrow window the shepherd can still queue a second invocation. After the fix, this residual race is rare and produces only occasional small gaps, a significant improvement over the systematic double-scheduling seen with delayed_work_pending().

Fixes 7b8da4c7f07774 ("vmstat: get rid of the ugly cpu_stat_off variable")

---
Changes in v2:
- Instead of changing the timings, do not double-schedule.
- Link to v1: https://patch.msgid.link/[email protected]

---
 mm/vmstat.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
mm/vmstat.c +1 -1
index 2370c6fb1fcd6..cc5fdc0d0f298 100644
@@ -2139,7 +2139,7 @@ static void vmstat_shepherd(struct work_struct *w)
 			if (cpu_is_isolated(cpu))
 				continue;
 
-			if (!delayed_work_pending(dw) && need_update(cpu))
+			if (!work_busy(&dw->work) && need_update(cpu))
 				queue_delayed_work_on(cpu, mm_percpu_wq, dw, 0);
 		}
 

---
base-commit: cf7c3c02fdd0dfccf4d6611714273dcb538af2cb
change-id: 20260401-vmstat-048e0feaf344

Best regards,

Signature · 1 line
Breno Leitao <[email protected]>
maintainer
#1 lore raw
Quoted 5 lines · On Thu, 09 Apr 2026 05:26:36 -0700 Breno Leitao <[email protected]> wrote:

vmstat_shepherd uses delayed_work_pending() to check whether vmstat_update is already scheduled for a given CPU before queuing it. However, delayed_work_pending() only tests WORK_STRUCT_PENDING_BIT, which is cleared the moment a worker thread picks up the work to execute it.

Thanks, I tentatively added this to the 7.1-rc1 queue, to upstream ~2 weeks hence.

#2 lore raw
In reply to Breno Leitao
Quoted 46 lines · Breno Leitao

vmstat_shepherd uses delayed_work_pending() to check whether vmstat_update is already scheduled for a given CPU before queuing it. However, delayed_work_pending() only tests WORK_STRUCT_PENDING_BIT, which is cleared the moment a worker thread picks up the work to execute it.

This means that while vmstat_update is actively running on a CPU, delayed_work_pending() returns false. If need_update() also returns true at that point (per-cpu counters not yet zeroed mid-flush), the shepherd queues a second invocation with delay=0, causing vmstat_update to run again immediately after finishing.

On a 72-CPU system this race is readily observable: before the fix, many CPUs show invocation gaps well below 500 jiffies (the minimum round_jiffies_relative() can produce), with the most extreme cases reaching 0 jiffies—vmstat_update called twice within the same jiffy.

Fix this by replacing delayed_work_pending() with work_busy(), which returns non-zero for both WORK_BUSY_PENDING (timer armed or work queued) and WORK_BUSY_RUNNING (work currently executing). The shepherd now correctly skips a CPU in all busy states.

After the fix, all sub-jiffy and most sub-100-jiffie gaps disappear. The remaining early invocations have gaps in the 700–999 jiffie range, attributable to round_jiffies_relative() aligning to a nearer jiffie-second boundary rather than to this race.

Each spurious vmstat_update invocation has a measurable side effect:
refresh_cpu_vm_stats() calls decay_pcp_high() for every zone, which drains idle per-CPU pages back to the buddy allocator via free_pcppages_bulk(), taking the zone spinlock each time. Eliminating the double-scheduling therefore reduces zone lock contention directly. On a 72-CPU stress-ng workload measured with perf lock contention:

  free_pcppages_bulk contention count:  ~55% reduction
  free_pcppages_bulk total wait time:   ~57% reduction
  free_pcppages_bulk max wait time:     ~47% reduction

Note: work_busy() is inherently racy—between the check and the subsequent queue_delayed_work_on() call, vmstat_update can finish execution, leaving the work neither pending nor running. In that narrow window the shepherd can still queue a second invocation. After the fix, this residual race is rare and produces only occasional small gaps, a significant improvement over the systematic double-scheduling seen with delayed_work_pending().

Quoted 3 lines
Fixes 7b8da4c7f07774 ("vmstat: get rid of the ugly cpu_stat_off variable")

Looks like v2 posting raced with Dmitry adding R-b tag to the posting in v1 thread:

https://lore.kernel.org/all/[email protected]/

Quoted 30 lines

---
Changes in v2:
- Instead of changing the timings, do not double-schedule.
- Link to v1: https://patch.msgid.link/[email protected]

---
 mm/vmstat.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
mm/vmstat.c +1 -2
index 2370c6fb1fcd6..cc5fdc0d0f298 100644
@@ -2139,7 +2139,7 @@ static void vmstat_shepherd(struct work_struct *w)
 			if (cpu_is_isolated(cpu))
 				continue;
 
-			if (!delayed_work_pending(dw) && need_update(cpu))
+			if (!work_busy(&dw->work) && need_update(cpu))
 				queue_delayed_work_on(cpu, mm_percpu_wq, dw, 0);
 		}
 
 
---

base-commit: cf7c3c02fdd0dfccf4d6611714273dcb538af2cb
change-id: 20260401-vmstat-048e0feaf344

Best regards,

Signature · 1 line
Breno Leitao <[email protected]>
#3 lore raw
Quoted 9 lines · Vlastimil Babka (SUSE)
Quoted 3 lines · Breno Leitao
Fixes 7b8da4c7f07774 ("vmstat: get rid of the ugly cpu_stat_off variable")

Looks like v2 posting raced with Dmitry adding R-b tag to the posting in v1 thread:

https://lore.kernel.org/all/[email protected]/

Dmitry, would you mind reviewing this version as well to keep the review trail complete?

This is essentially the same patch you already reviewed.

Thanks,
--breno

reviewer Acked-by on v2 1/1
#6 lore raw
In reply to Breno Leitao
Quoted 46 lines · Breno Leitao

vmstat_shepherd uses delayed_work_pending() to check whether vmstat_update is already scheduled for a given CPU before queuing it. However, delayed_work_pending() only tests WORK_STRUCT_PENDING_BIT, which is cleared the moment a worker thread picks up the work to execute it.

This means that while vmstat_update is actively running on a CPU, delayed_work_pending() returns false. If need_update() also returns true at that point (per-cpu counters not yet zeroed mid-flush), the shepherd queues a second invocation with delay=0, causing vmstat_update to run again immediately after finishing.

On a 72-CPU system this race is readily observable: before the fix, many CPUs show invocation gaps well below 500 jiffies (the minimum round_jiffies_relative() can produce), with the most extreme cases reaching 0 jiffies—vmstat_update called twice within the same jiffy.

Fix this by replacing delayed_work_pending() with work_busy(), which returns non-zero for both WORK_BUSY_PENDING (timer armed or work queued) and WORK_BUSY_RUNNING (work currently executing). The shepherd now correctly skips a CPU in all busy states.

After the fix, all sub-jiffy and most sub-100-jiffie gaps disappear. The remaining early invocations have gaps in the 700–999 jiffie range, attributable to round_jiffies_relative() aligning to a nearer jiffie-second boundary rather than to this race.

Each spurious vmstat_update invocation has a measurable side effect:
refresh_cpu_vm_stats() calls decay_pcp_high() for every zone, which drains idle per-CPU pages back to the buddy allocator via free_pcppages_bulk(), taking the zone spinlock each time. Eliminating the double-scheduling therefore reduces zone lock contention directly. On a 72-CPU stress-ng workload measured with perf lock contention:

  free_pcppages_bulk contention count:  ~55% reduction
  free_pcppages_bulk total wait time:   ~57% reduction
  free_pcppages_bulk max wait time:     ~47% reduction

Note: work_busy() is inherently racy—between the check and the subsequent queue_delayed_work_on() call, vmstat_update can finish execution, leaving the work neither pending nor running. In that narrow window the shepherd can still queue a second invocation. After the fix, this residual race is rare and produces only occasional small gaps, a significant improvement over the systematic double-scheduling seen with delayed_work_pending().

Quoted 3 lines
Fixes 7b8da4c7f07774 ("vmstat: get rid of the ugly cpu_stat_off variable")

I like this much better. Thanks for digging deeper into this.

Quoted 30 lines

---
Changes in v2:
- Instead of changing the timings, do not double-schedule.
- Link to v1: https://patch.msgid.link/[email protected]

---
 mm/vmstat.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
mm/vmstat.c +1 -2
index 2370c6fb1fcd6..cc5fdc0d0f298 100644
@@ -2139,7 +2139,7 @@ static void vmstat_shepherd(struct work_struct *w)
 			if (cpu_is_isolated(cpu))
 				continue;
 
-			if (!delayed_work_pending(dw) && need_update(cpu))
+			if (!work_busy(&dw->work) && need_update(cpu))
 				queue_delayed_work_on(cpu, mm_percpu_wq, dw, 0);
 		}
 
 
---

base-commit: cf7c3c02fdd0dfccf4d6611714273dcb538af2cb
change-id: 20260401-vmstat-048e0feaf344

Best regards,

Signature · 1 line
Breno Leitao <[email protected]>
Signature · 2 lines
Michal Hocko
SUSE Labs
Reviewed-by on v2 1/1
#7 lore raw
In reply to Breno Leitao
Quoted 46 lines · Breno Leitao

vmstat_shepherd uses delayed_work_pending() to check whether vmstat_update is already scheduled for a given CPU before queuing it. However, delayed_work_pending() only tests WORK_STRUCT_PENDING_BIT, which is cleared the moment a worker thread picks up the work to execute it.

This means that while vmstat_update is actively running on a CPU, delayed_work_pending() returns false. If need_update() also returns true at that point (per-cpu counters not yet zeroed mid-flush), the shepherd queues a second invocation with delay=0, causing vmstat_update to run again immediately after finishing.

On a 72-CPU system this race is readily observable: before the fix, many CPUs show invocation gaps well below 500 jiffies (the minimum round_jiffies_relative() can produce), with the most extreme cases reaching 0 jiffies—vmstat_update called twice within the same jiffy.

Fix this by replacing delayed_work_pending() with work_busy(), which returns non-zero for both WORK_BUSY_PENDING (timer armed or work queued) and WORK_BUSY_RUNNING (work currently executing). The shepherd now correctly skips a CPU in all busy states.

After the fix, all sub-jiffy and most sub-100-jiffie gaps disappear. The remaining early invocations have gaps in the 700–999 jiffie range, attributable to round_jiffies_relative() aligning to a nearer jiffie-second boundary rather than to this race.

Each spurious vmstat_update invocation has a measurable side effect:
refresh_cpu_vm_stats() calls decay_pcp_high() for every zone, which drains idle per-CPU pages back to the buddy allocator via free_pcppages_bulk(), taking the zone spinlock each time. Eliminating the double-scheduling therefore reduces zone lock contention directly. On a 72-CPU stress-ng workload measured with perf lock contention:

  free_pcppages_bulk contention count:  ~55% reduction
  free_pcppages_bulk total wait time:   ~57% reduction
  free_pcppages_bulk max wait time:     ~47% reduction

Note: work_busy() is inherently racy—between the check and the subsequent queue_delayed_work_on() call, vmstat_update can finish execution, leaving the work neither pending nor running. In that narrow window the shepherd can still queue a second invocation. After the fix, this residual race is rare and produces only occasional small gaps, a significant improvement over the systematic double-scheduling seen with delayed_work_pending().

Quoted 3 lines
Fixes 7b8da4c7f07774 ("vmstat: get rid of the ugly cpu_stat_off variable")
#4 lore raw
In reply to Breno Leitao
Quoted 9 lines · Breno Leitao
Quoted 8 lines · Vlastimil Babka (SUSE)
Quoted 3 lines · Breno Leitao
Fixes 7b8da4c7f07774 ("vmstat: get rid of the ugly cpu_stat_off variable")

Looks like v2 posting raced with Dmitry adding R-b tag to the posting in v1 thread:

Quoted 6 lines
Quoted 1 line

https://lore.kernel.org/all/[email protected]/

Dmitry, would you mind reviewing this version as well to keep the review trail complete?

This is essentially the same patch you already reviewed.

Andrew seems to have already picked up my Reviewed-by in mm-new.

There are also several other tags there, but most of them were given for v1, while the current patch (v2) differs, so they may no longer apply.

I've added my Reviewed-by again for this version.

#5 lore raw
Quoted 15 lines · Dmitry Ilvokhin
Quoted 14 lines · Breno Leitao
Quoted 9 lines · Vlastimil Babka (SUSE)
Quoted 3 lines · Breno Leitao
Fixes 7b8da4c7f07774 ("vmstat: get rid of the ugly cpu_stat_off variable")

Looks like v2 posting raced with Dmitry adding R-b tag to the posting in v1 thread:

https://lore.kernel.org/all/[email protected]/

Dmitry, would you mind reviewing this version as well to keep the review trail complete?

Quoted 8 lines
Quoted 1 line

This is essentially the same patch you already reviewed.

Andrew seems to have already picked up my Reviewed-by in mm-new.

There are also several other tags there, but most of them were given for v1, while the current patch (v2) differs, so they may no longer apply.

I've added my Reviewed-by again for this version.

Thanks Dmitry!