mm/vmstat: fix vmstat_shepherd double-scheduling vmstat_update
Subsystems (2)
PATCH v2 1/1 mm/vmstat: fix vmstat_shepherd double-scheduling vmstat_update
vmstat_shepherd uses delayed_work_pending() to check whether vmstat_update is already scheduled for a given CPU before queuing it. However, delayed_work_pending() only tests WORK_STRUCT_PENDING_BIT, which is cleared the moment a worker thread picks up the work to execute it.
This means that while vmstat_update is actively running on a CPU, delayed_work_pending() returns false. If need_update() also returns true at that point (per-cpu counters not yet zeroed mid-flush), the shepherd queues a second invocation with delay=0, causing vmstat_update to run again immediately after finishing.
On a 72-CPU system this race is readily observable: before the fix, many CPUs show invocation gaps well below 500 jiffies (the minimum round_jiffies_relative() can produce), with the most extreme cases reaching 0 jiffies—vmstat_update called twice within the same jiffy.
Fix this by replacing delayed_work_pending() with work_busy(), which returns non-zero for both WORK_BUSY_PENDING (timer armed or work queued) and WORK_BUSY_RUNNING (work currently executing). The shepherd now correctly skips a CPU in all busy states.
After the fix, all sub-jiffy and most sub-100-jiffie gaps disappear. The remaining early invocations have gaps in the 700–999 jiffie range, attributable to round_jiffies_relative() aligning to a nearer jiffie-second boundary rather than to this race.
Each spurious vmstat_update invocation has a measurable side effect:refresh_cpu_vm_stats() calls decay_pcp_high() for every zone, which drains idle per-CPU pages back to the buddy allocator via free_pcppages_bulk(), taking the zone spinlock each time. Eliminating the double-scheduling therefore reduces zone lock contention directly. On a 72-CPU stress-ng workload measured with perf lock contention:
free_pcppages_bulk contention count: ~55% reduction free_pcppages_bulk total wait time: ~57% reduction free_pcppages_bulk max wait time: ~47% reduction
Note: work_busy() is inherently racy—between the check and the subsequent queue_delayed_work_on() call, vmstat_update can finish execution, leaving the work neither pending nor running. In that narrow window the shepherd can still queue a second invocation. After the fix, this residual race is rare and produces only occasional small gaps, a significant improvement over the systematic double-scheduling seen with delayed_work_pending().
---
Changes in v2:
- Instead of changing the timings, do not double-schedule.
- Link to v1: https://patch.msgid.link/[email protected]
--- mm/vmstat.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
mm/vmstat.c +1 -1
@@ -2139,7 +2139,7 @@ static void vmstat_shepherd(struct work_struct *w)if (cpu_is_isolated (cpu))continue ; -if (!delayed_work_pending (dw) &&need_update (cpu)) +if (!work_busy (&dw->work) &&need_update (cpu))queue_delayed_work_on (cpu, mm_percpu_wq, dw,0 ); }
---
base-commit: cf7c3c02fdd0dfccf4d6611714273dcb538af2cb
change-id: 20260401-vmstat-048e0feaf344
Best regards,
Signature · 1 line
Breno Leitao <[email protected]>
Andrew Morton
maintainerQuoted 5 lines · On Thu, 09 Apr 2026 05:26:36 -0700 Breno Leitao <[email protected]> wrote:
vmstat_shepherdusesdelayed_work_pending()to check whethervmstat_updateis already scheduled for a given CPU before queuing it. However,delayed_work_pending()only testsWORK_STRUCT_PENDING_BIT, which is cleared the moment a worker thread picks up the work to execute it.
Thanks, I tentatively added this to the 7.1-rc1 queue, to upstream ~2 weeks hence.
Vlastimil Babka (SUSE)
reviewerQuoted 46 lines · Breno Leitao
vmstat_shepherdusesdelayed_work_pending()to check whethervmstat_updateis already scheduled for a given CPU before queuing it. However,delayed_work_pending()only testsWORK_STRUCT_PENDING_BIT, which is cleared the moment a worker thread picks up the work to execute it.This means that while
vmstat_updateis actively running on a CPU,delayed_work_pending()returns false. Ifneed_update()also returns true at that point (per-cpu counters not yet zeroed mid-flush), the shepherd queues a second invocation with delay=0, causingvmstat_updateto run again immediately after finishing.On a 72-CPU system this race is readily observable: before the fix, many CPUs show invocation gaps well below 500 jiffies (the minimum
round_jiffies_relative()can produce), with the most extreme cases reaching 0 jiffies—vmstat_update called twice within the same jiffy.Fix this by replacing
delayed_work_pending()withwork_busy(), which returns non-zero for bothWORK_BUSY_PENDING(timer armed or work queued) andWORK_BUSY_RUNNING(work currently executing). The shepherd now correctly skips a CPU in all busy states.After the fix, all sub-jiffy and most sub-100-jiffie gaps disappear. The remaining early invocations have gaps in the 700–999 jiffie range, attributable to
round_jiffies_relative()aligning to a nearer jiffie-second boundary rather than to this race.Each spurious
vmstat_updateinvocation has a measurable side effect:refresh_cpu_vm_stats()callsdecay_pcp_high()for every zone, which drains idle per-CPU pages back to the buddy allocator viafree_pcppages_bulk(), taking the zone spinlock each time. Eliminating the double-scheduling therefore reduces zone lock contention directly. On a 72-CPU stress-ng workload measured with perf lock contention:free_pcppages_bulk contention count: ~55% reduction free_pcppages_bulk total wait time: ~57% reduction free_pcppages_bulk max wait time: ~47% reductionNote:
work_busy()is inherently racy—between the check and the subsequentqueue_delayed_work_on()call,vmstat_updatecan finish execution, leaving the work neither pending nor running. In that narrow window the shepherd can still queue a second invocation. After the fix, this residual race is rare and produces only occasional small gaps, a significant improvement over the systematic double-scheduling seen withdelayed_work_pending().
Quoted 3 lines
Fixes 7b8da4c7f07774 ("vmstat: get rid of the ugly cpu_stat_off variable")Signed-off-by Breno Leitao <[email protected]>Reviewed-by Vlastimil Babka (SUSE) <[email protected]>
Looks like v2 posting raced with Dmitry adding R-b tag to the posting in v1 thread:
https://lore.kernel.org/all/[email protected]/
Quoted 30 lines
---
Changes in v2:
- Instead of changing the timings, do not double-schedule.
- Link to v1: https://patch.msgid.link/[email protected]--- mm/vmstat.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)mm/vmstat.c +1
-2@@ -2139,7 +2139,7 @@ static void vmstat_shepherd(struct work_struct *w)if (cpu_is_isolated (cpu))continue ; -if (!delayed_work_pending (dw) &&need_update (cpu)) +if (!work_busy (&dw->work) &&need_update (cpu))queue_delayed_work_on (cpu, mm_percpu_wq, dw,0 ); } ---base-commit:
cf7c3c02fdd0dfccf4d6611714273dcb538af2cb
change-id: 20260401-vmstat-048e0feaf344Best regards,
Signature · 1 line
Breno Leitao <[email protected]>
Quoted 9 lines · Vlastimil Babka (SUSE)
Quoted 3 lines · Breno Leitao
Fixes 7b8da4c7f07774 ("vmstat: get rid of the ugly cpu_stat_off variable")Signed-off-by Breno Leitao <[email protected]>Reviewed-by Vlastimil Babka (SUSE) <[email protected]>Looks like v2 posting raced with Dmitry adding R-b tag to the posting in v1 thread:
Dmitry, would you mind reviewing this version as well to keep the review trail complete?
This is essentially the same patch you already reviewed.
Thanks,
--breno
Quoted 9 lines · Breno Leitao
Quoted 8 lines · Vlastimil Babka (SUSE)
Quoted 3 lines · Breno Leitao
Fixes 7b8da4c7f07774 ("vmstat: get rid of the ugly cpu_stat_off variable")Signed-off-by Breno Leitao <[email protected]>Reviewed-by Vlastimil Babka (SUSE) <[email protected]>Looks like v2 posting raced with Dmitry adding R-b tag to the posting in v1 thread:
Quoted 6 lines
Quoted 1 line
Dmitry, would you mind reviewing this version as well to keep the review trail complete?
This is essentially the same patch you already reviewed.
Andrew seems to have already picked up my Reviewed-by in mm-new.
There are also several other tags there, but most of them were given for v1, while the current patch (v2) differs, so they may no longer apply.
I've added my Reviewed-by again for this version.
Quoted 15 lines · Dmitry Ilvokhin
Quoted 14 lines · Breno Leitao
Quoted 9 lines · Vlastimil Babka (SUSE)
Quoted 3 lines · Breno Leitao
Fixes 7b8da4c7f07774 ("vmstat: get rid of the ugly cpu_stat_off variable")Signed-off-by Breno Leitao <[email protected]>Reviewed-by Vlastimil Babka (SUSE) <[email protected]>Looks like v2 posting raced with Dmitry adding R-b tag to the posting in v1 thread:
Dmitry, would you mind reviewing this version as well to keep the review trail complete?
Quoted 8 lines
Quoted 1 line
This is essentially the same patch you already reviewed.
Andrew seems to have already picked up my Reviewed-by in mm-new.
There are also several other tags there, but most of them were given for v1, while the current patch (v2) differs, so they may no longer apply.
I've added my Reviewed-by again for this version.
Thanks Dmitry!
Quoted 46 lines · Breno Leitao
vmstat_shepherdusesdelayed_work_pending()to check whethervmstat_updateis already scheduled for a given CPU before queuing it. However,delayed_work_pending()only testsWORK_STRUCT_PENDING_BIT, which is cleared the moment a worker thread picks up the work to execute it.This means that while
vmstat_updateis actively running on a CPU,delayed_work_pending()returns false. Ifneed_update()also returns true at that point (per-cpu counters not yet zeroed mid-flush), the shepherd queues a second invocation with delay=0, causingvmstat_updateto run again immediately after finishing.On a 72-CPU system this race is readily observable: before the fix, many CPUs show invocation gaps well below 500 jiffies (the minimum
round_jiffies_relative()can produce), with the most extreme cases reaching 0 jiffies—vmstat_update called twice within the same jiffy.Fix this by replacing
delayed_work_pending()withwork_busy(), which returns non-zero for bothWORK_BUSY_PENDING(timer armed or work queued) andWORK_BUSY_RUNNING(work currently executing). The shepherd now correctly skips a CPU in all busy states.After the fix, all sub-jiffy and most sub-100-jiffie gaps disappear. The remaining early invocations have gaps in the 700–999 jiffie range, attributable to
round_jiffies_relative()aligning to a nearer jiffie-second boundary rather than to this race.Each spurious
vmstat_updateinvocation has a measurable side effect:refresh_cpu_vm_stats()callsdecay_pcp_high()for every zone, which drains idle per-CPU pages back to the buddy allocator viafree_pcppages_bulk(), taking the zone spinlock each time. Eliminating the double-scheduling therefore reduces zone lock contention directly. On a 72-CPU stress-ng workload measured with perf lock contention:free_pcppages_bulk contention count: ~55% reduction free_pcppages_bulk total wait time: ~57% reduction free_pcppages_bulk max wait time: ~47% reductionNote:
work_busy()is inherently racy—between the check and the subsequentqueue_delayed_work_on()call,vmstat_updatecan finish execution, leaving the work neither pending nor running. In that narrow window the shepherd can still queue a second invocation. After the fix, this residual race is rare and produces only occasional small gaps, a significant improvement over the systematic double-scheduling seen withdelayed_work_pending().
Quoted 3 lines
Fixes 7b8da4c7f07774 ("vmstat: get rid of the ugly cpu_stat_off variable")Signed-off-by Breno Leitao <[email protected]>Reviewed-by Vlastimil Babka (SUSE) <[email protected]>
I like this much better. Thanks for digging deeper into this.
Quoted 30 lines
---
Changes in v2:
- Instead of changing the timings, do not double-schedule.
- Link to v1: https://patch.msgid.link/[email protected]--- mm/vmstat.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)mm/vmstat.c +1
-2@@ -2139,7 +2139,7 @@ static void vmstat_shepherd(struct work_struct *w)if (cpu_is_isolated (cpu))continue ; -if (!delayed_work_pending (dw) &&need_update (cpu)) +if (!work_busy (&dw->work) &&need_update (cpu))queue_delayed_work_on (cpu, mm_percpu_wq, dw,0 ); } ---base-commit:
cf7c3c02fdd0dfccf4d6611714273dcb538af2cb
change-id: 20260401-vmstat-048e0feaf344Best regards,
Signature · 1 line
Breno Leitao <[email protected]>
Signature · 2 lines
Michal Hocko SUSE Labs
Quoted 46 lines · Breno Leitao
vmstat_shepherdusesdelayed_work_pending()to check whethervmstat_updateis already scheduled for a given CPU before queuing it. However,delayed_work_pending()only testsWORK_STRUCT_PENDING_BIT, which is cleared the moment a worker thread picks up the work to execute it.This means that while
vmstat_updateis actively running on a CPU,delayed_work_pending()returns false. Ifneed_update()also returns true at that point (per-cpu counters not yet zeroed mid-flush), the shepherd queues a second invocation with delay=0, causingvmstat_updateto run again immediately after finishing.On a 72-CPU system this race is readily observable: before the fix, many CPUs show invocation gaps well below 500 jiffies (the minimum
round_jiffies_relative()can produce), with the most extreme cases reaching 0 jiffies—vmstat_update called twice within the same jiffy.Fix this by replacing
delayed_work_pending()withwork_busy(), which returns non-zero for bothWORK_BUSY_PENDING(timer armed or work queued) andWORK_BUSY_RUNNING(work currently executing). The shepherd now correctly skips a CPU in all busy states.After the fix, all sub-jiffy and most sub-100-jiffie gaps disappear. The remaining early invocations have gaps in the 700–999 jiffie range, attributable to
round_jiffies_relative()aligning to a nearer jiffie-second boundary rather than to this race.Each spurious
vmstat_updateinvocation has a measurable side effect:refresh_cpu_vm_stats()callsdecay_pcp_high()for every zone, which drains idle per-CPU pages back to the buddy allocator viafree_pcppages_bulk(), taking the zone spinlock each time. Eliminating the double-scheduling therefore reduces zone lock contention directly. On a 72-CPU stress-ng workload measured with perf lock contention:free_pcppages_bulk contention count: ~55% reduction free_pcppages_bulk total wait time: ~57% reduction free_pcppages_bulk max wait time: ~47% reductionNote:
work_busy()is inherently racy—between the check and the subsequentqueue_delayed_work_on()call,vmstat_updatecan finish execution, leaving the work neither pending nor running. In that narrow window the shepherd can still queue a second invocation. After the fix, this residual race is rare and produces only occasional small gaps, a significant improvement over the systematic double-scheduling seen withdelayed_work_pending().
Quoted 3 lines
Fixes 7b8da4c7f07774 ("vmstat: get rid of the ugly cpu_stat_off variable")Signed-off-by Breno Leitao <[email protected]>Reviewed-by Vlastimil Babka (SUSE) <[email protected]>