vmstat bug fixes for nohz_full and isolated CPUs
Subsystems (4)
v3 cover vmstat bug fixes for nohz_full and isolated CPUs
This patch series addresses the following two problems:
1. A customer provided evidence indicating that a process was stalled in direct reclaim:
- The process was trapped in throttle_direct_reclaim(). The function wait_event_killable() was called to wait condition allow_direct_reclaim(pgdat) for current node to be true. The allow_direct_reclaim(pgdat) examined the number of free pages on the node by zone_page_state() which just returns value in zone->vm_stat[NR_FREE_PAGES].
- On node #1, zone->vm_stat[NR_FREE_PAGES] was 0. However, the freelist on this node was not empty.
- This inconsistent of vmstat value was caused by percpu vmstat on nohz_full cpus. Every increment/decrement of vmstat is performed on percpu vmstat counter at first, then pooled diffs are cumulated to the zone's vmstat counter in timely manner. However, on nohz_full cpus (in case of this customer's system, 48 of 52 cpus) these pooled diffs were not cumulated once the cpu had no event on it so that the cpu started sleeping infinitely.
I checked percpu vmstat and found there were total 69 counts not cumulated to the zone's vmstat counter yet.
- In this situation, kswapd did not help the trapped process. In pgdat_balanced(), zone_wakermark_ok_safe() examined the number of free pages on the node by zone_page_state_snapshot() which checks pending counts on percpu vmstat.
Therefore kswapd could know there were 69 free pages correctly. Since zone->_watermark = {8, 20, 32}, kswapd did not work because 69 was greater than 32 as high watermark.
2. With a task that busy loops on a given CPU, the kworker interruption to execute vmstat_update is undesired and may exceed latency thresholds for certain applications.
First issue is solved by using _snapshot version of the counters on allow_direct_reclaim.
Second issue is fixed by disabling periodic vmstat updates for nohz_full CPUs.
Thanks to Aaron Tomlin for diagnosing issue 1 and writing the initial patch series.
Performance details for the kworker interruption:
With workloads that are running on isolated cpus with nohz full mode to shield off any kernel interruption. For example, a VM running a time sensitive application with a 50us maximum acceptable interruption (use case: soft PLC).
oslat 1094.456862: sys_mlock(start: 7f7ed0000b60, len: 1000)
oslat 1094.456971: workqueue_queue_work: ... function=vmstat_update ...
oslat 1094.456974: sched_switch: prev_comm=oslat ... ==> next_comm=kworker/5:1 ...
kworker 1094.456978: sched_switch: prev_comm=kworker/5:1 ==> next_comm=oslat ...
The example above shows an additional 7us for the
oslat -> kworker -> oslat
switches. In the case of a virtualized CPU, and the vmstat_update interruption in the host (of a qemu-kvm vcpu), the latency penalty observed in the guest is higher than 50us, violating the acceptable latency threshold.
The isolated vCPU can perform operations that modify per-CPU page counters, for example to complete I/O operations:
CPU 11/KVM-9540 [001] dNh1. 2314.248584: mod_zone_page_state <-__folio_end_writeback
CPU 11/KVM-9540 [001] dNh1. 2314.248585: <stack trace> => 0xffffffffc042b083
=> mod_zone_page_state
=> __folio_end_writeback
=> folio_end_writeback
=> iomap_finish_ioend
=> blk_mq_end_request_batch
=> nvme_irq
=> __handle_irq_event_percpu
=> handle_irq_event
=> handle_edge_irq
=> __common_interrupt
=> common_interrupt
=> asm_common_interrupt
=> vmx_do_interrupt_nmi_irqoff
=> vmx_handle_exit_irqoff
=> vcpu_enter_guest
=> vcpu_run
=> kvm_arch_vcpu_ioctl_run
=> kvm_vcpu_ioctl
=> __x64_sys_ioctl
=> do_syscall_64
=> entry_SYSCALL_64_after_hwframe
v3: improve changelog (Michal Hocko)
v2: use cpu_is_isolated (Michal Hocko)
opencode schedule_on_each_cpu (Michal Hocko)
mm/vmscan.c | 2 +- mm/vmstat.c | 39 +++++++++++++++++++++++++++++++++++---- 2 files changed, 36 insertions(+), 5 deletions(-)
PATCH v3 1/3 vmstat: allow_direct_reclaim should use zone_page_state_snapshot
A customer provided evidence indicating that a process was stalled in direct reclaim:
- The process was trapped in throttle_direct_reclaim(). The function wait_event_killable() was called to wait condition allow_direct_reclaim(pgdat) for current node to be true. The allow_direct_reclaim(pgdat) examined the number of free pages on the node by zone_page_state() which just returns value in zone->vm_stat[NR_FREE_PAGES].
- On node #1, zone->vm_stat[NR_FREE_PAGES] was 0. However, the freelist on this node was not empty.
- This inconsistent of vmstat value was caused by percpu vmstat on nohz_full cpus. Every increment/decrement of vmstat is performed on percpu vmstat counter at first, then pooled diffs are cumulated to the zone's vmstat counter in timely manner. However, on nohz_full cpus (in case of this customer's system, 48 of 52 cpus) these pooled diffs were not cumulated once the cpu had no event on it so that the cpu started sleeping infinitely.
I checked percpu vmstat and found there were total 69 counts not cumulated to the zone's vmstat counter yet.
- In this situation, kswapd did not help the trapped process. In pgdat_balanced(), zone_wakermark_ok_safe() examined the number of free pages on the node by zone_page_state_snapshot() which checks pending counts on percpu vmstat.
Therefore kswapd could know there were 69 free pages correctly. Since zone->_watermark = {8, 20, 32}, kswapd did not work because 69 was greater than 32 as high watermark.
Change allow_direct_reclaim to use zone_page_state_snapshot, which allows a more precise version of the vmstat counters to be used.
allow_direct_reclaim will only be called from try_to_free_pages, which is not a hot path.
Testing: Due to difficulties accessing the system, it has not been possible for the reproducer to test the patch (however its clear from available data and analysis that it should fix it).
---
mm/vmscan.c +1 -1
@@ -6887,7 +6887,7 @@ static bool allow_direct_reclaim(pg_datacontinue ; pfmemalloc_reserve +=min_wmark_pages (zone); - free_pages +=zone_page_state (zone,NR_FREE_PAGES ); + free_pages +=zone_page_state_snapshot (zone,NR_FREE_PAGES ); }/* If there are no reserves (unexpected config) then do not throttle */
PATCH v3 2/3 vmstat: skip periodic vmstat update for isolated CPUs
In reply to Marcelo TosattiProblem: The interruption caused by vmstat_update is undesirable for certain applications.
With workloads that are running on isolated cpus with nohz full mode to shield off any kernel interruption. For example, a VM running a time sensitive application with a 50us maximum acceptable interruption (use case: soft PLC).
oslat 1094.456862: sys_mlock(start: 7f7ed0000b60, len: 1000)
oslat 1094.456971: workqueue_queue_work: ... function=vmstat_update ...
oslat 1094.456974: sched_switch: prev_comm=oslat ... ==> next_comm=kworker/5:1 ...
kworker 1094.456978: sched_switch: prev_comm=kworker/5:1 ==> next_comm=oslat ...
The example above shows an additional 7us for the
oslat -> kworker -> oslat
switches. In the case of a virtualized CPU, and the vmstat_update interruption in the host (of a qemu-kvm vcpu), the latency penalty observed in the guest is higher than 50us, violating the acceptable latency threshold.
The isolated vCPU can perform operations that modify per-CPU page counters, for example to complete I/O operations:
CPU 11/KVM-9540 [001] dNh1. 2314.248584: mod_zone_page_state <-__folio_end_writeback
CPU 11/KVM-9540 [001] dNh1. 2314.248585: <stack trace> => 0xffffffffc042b083
=> mod_zone_page_state
=> __folio_end_writeback
=> folio_end_writeback
=> iomap_finish_ioend
=> blk_mq_end_request_batch
=> nvme_irq
=> __handle_irq_event_percpu
=> handle_irq_event
=> handle_edge_irq
=> __common_interrupt
=> common_interrupt
=> asm_common_interrupt
=> vmx_do_interrupt_nmi_irqoff
=> vmx_handle_exit_irqoff
=> vcpu_enter_guest
=> vcpu_run
=> kvm_arch_vcpu_ioctl_run
=> kvm_vcpu_ioctl
=> __x64_sys_ioctl
=> do_syscall_64
=> entry_SYSCALL_64_after_hwframe
In kernel users of vmstat counters either require the precise value and they are using zone_page_state_snapshot interface or they can live with an imprecision as the regular flushing can happen at arbitrary time and cumulative error can grow (see calculate_normal_threshold).
Quoted 1 line
From that POV the regular flushing can be postponed for CPUs that have
been isolated from the kernel interference without critical infrastructure ever noticing. Skip regular flushing from vmstat_shepherd for all isolated CPUs to avoid interference with the isolated workload.
Suggested by Michal Hocko.
---
v3: improve changelog (Michal Hocko)
v2: use cpu_is_isolated (Michal Hocko)
mm/vmstat.c +15 -0
@@ -28,6 +28,7 @@#include <linux/mm_inline.h> #include <linux/page_ext.h> #include <linux/page_owner.h> +#include <linux/sched/isolation.h> #include "internal.h"
@@ -2022,6 +2023,20 @@ static void vmstat_shepherd(struct work_for_each_online_cpu (cpu) {struct delayed_work *dw = &per_cpu (vmstat_work, cpu); +/* +* In kernel users of vmstat counters either require the precise value and +* they are using zone_page_state_snapshot interface or they can live with +* an imprecision as the regular flushing can happen at arbitrary time and +* cumulative error can grow (see calculate_normal_threshold). +* +* From that POV the regular flushing can be postponed for CPUs that have +* been isolated from the kernel interference without critical +* infrastructure ever noticing. Skip regular flushing from vmstat_shepherd +* for all isolated CPUs to avoid interference with the isolated workload. +*/ +if (cpu_is_isolated (cpu)) +continue ; +if (!delayed_work_pending (dw) &&need_update (cpu))queue_delayed_work_on (cpu, mm_percpu_wq, dw,0 );
PATCH v3 3/3 mm/vmstat: do not refresh stats for isolated CPUs
In reply to Marcelo Tosattischedule_work_on API uses the workqueue mechanism to queue a work item on a queue. A kernel thread, which runs on the target CPU, executes those work items.
Therefore, when using the schedule_work_on API, it is necessary for the kworker kernel thread to be scheduled in, for the work function to be executed.
Time sensitive applications such as SoftPLCs
(https://tum-esi.github.io/publications-list/PDF/2022-ETFA-How_Real_Time_Are_Virtual_PLCs.pdf),
have their response times affected by such interruptions.
The /proc/sys/vm/stat_refresh file was originally introduced with the goal to:
"Provide /proc/sys/vm/stat_refresh to force an immediate update of
per-cpu into global vmstats: useful to avoid a sleep(2) or whatever before checking counts when testing. Originally added to work around a bug which left counts stranded indefinitely on a cpu going idle (an inaccuracy magnified when small below-batch numbers represent "huge" amounts of memory), but I believe that bug is now fixed: nonetheless, this is still a useful knob."
Other than the potential interruption to a time sensitive application, if using SCHED_FIFO or SCHED_RR priority on the isolated CPU, then system hangs can occur:
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=978688
To avoid the problems above, do not schedule the work to synchronize per-CPU mm counters on isolated CPUs. Given the possibility for breaking existing userspace applications, avoid returning errors from access to /proc/sys/vm/stat_refresh.
---
v3: improve changelog (Michal Hocko)
v2: opencode schedule_on_each_cpu (Michal Hocko)
mm/vmstat.c +24 -4
@@ -1881,8 +1881,13 @@ int vmstat_refresh(struct ctl_table *tab void *buffer, size_t *lenp, loff_t *ppos) { long val; -int err;int i; +int cpu; +struct work_struct __percpu *works; + + works = alloc_percpu(struct work_struct); +if (!works) +return -ENOMEM ;/* * The regular update, every sysctl_stat_interval, may come later
@@ -1896,9 +1901,24 @@ int vmstat_refresh(struct ctl_table *tab* transiently negative values, report an error here if any of * the stats is negative, so we know to go looking for imbalance. */ - err =schedule_on_each_cpu (refresh_vm_stats); -if (err) -return err; +cpus_read_lock (); +for_each_online_cpu (cpu) { +struct work_struct *work; + +if (cpu_is_isolated (cpu)) +continue ; + work =per_cpu_ptr (works, cpu); +INIT_WORK (work, refresh_vm_stats); +schedule_work_on (cpu, work); + } + +for_each_online_cpu (cpu) { +if (cpu_is_isolated (cpu)) +continue ; +flush_work (per_cpu_ptr (works, cpu)); + } +cpus_read_unlock (); +free_percpu (works);for (i =0 ; i <NR_VM_ZONE_STAT_ITEMS ; i++) {/* * Skip checking stats known to go negative occasionally.
Re: [PATCH v3 3/3] mm/vmstat: do not refresh stats for isolated CPUs
Quoted 23 lines · Marcelo Tosatti
schedule_work_onAPI uses the workqueue mechanism to queue a work item on a queue. A kernel thread, which runs on the target CPU, executes those work items.Therefore, when using the
schedule_work_onAPI, it is necessary for the kworker kernel thread to be scheduled in, for the work function to be executed.Time sensitive applications such as SoftPLCs
(https://tum-esi.github.io/publications-list/PDF/2022-ETFA-How_Real_Time_Are_Virtual_PLCs.pdf),
have their response times affected by such interruptions.The
/proc/sys/vm/stat_refreshfile was originally introduced with the goal to:"Provide
/proc/sys/vm/stat_refreshto force an immediate update of
per-cpu into global vmstats: useful to avoid a sleep(2) or whatever before checking counts when testing. Originally added to work around a bug which left counts stranded indefinitely on a cpu going idle (an inaccuracy magnified when small below-batch numbers represent "huge" amounts of memory), but I believe that bug is now fixed: nonetheless, this is still a useful knob."
Quoted 3 lines
Other than the potential interruption to a time sensitive application, if using
SCHED_FIFOorSCHED_RRpriority on the isolated CPU, then system hangs can occur:
The same thing can happen without isolated CPUs and this patch doesn't help at all.
And this is an example of that...
Quoted 6 lines
To avoid the problems above, do not schedule the work to synchronize per-CPU mm counters on isolated CPUs. Given the possibility for breaking existing userspace applications, avoid returning errors from access to
/proc/sys/vm/stat_refresh.Signed-off-by Marcelo Tosatti <[email protected]>
It would be really helpful to not post new versions while discussion of the previous one is still not done.
Anyway
This is silently changing semantic and I do not think you have actually shown this is a real life problem. To me it sounds like a theoretical issue at most and it can be worked around by disalowing to use this interface from userspace. stat_refresh is mostly for debugging purposes and I strongly doubt it is ever used in environments you refer to in this series.
Quoted 53 lines
---
v3: improve changelog (Michal Hocko)
v2: opencodeschedule_on_each_cpu(Michal Hocko)Index:
linux-vmstat-remote/mm/vmstat.c
===================================================================mm/vmstat.c +24
-4@@ -1881,8 +1881,13 @@ int vmstat_refresh(struct ctl_table *tab void *buffer, size_t *lenp, loff_t *ppos) { long val; -int err;int i; +int cpu; +struct work_struct __percpu *works; + + works = alloc_percpu(struct work_struct); +if (!works) +return -ENOMEM ;/* * The regular update, every sysctl_stat_interval, may come later @@ -1896,9 +1901,24 @@ int vmstat_refresh(struct ctl_table *tab* transiently negative values, report an error here if any of * the stats is negative, so we know to go looking for imbalance. */ - err =schedule_on_each_cpu (refresh_vm_stats); -if (err) -return err; +cpus_read_lock (); +for_each_online_cpu (cpu) { +struct work_struct *work; + +if (cpu_is_isolated (cpu)) +continue ; + work =per_cpu_ptr (works, cpu); +INIT_WORK (work, refresh_vm_stats); +schedule_work_on (cpu, work); + } + +for_each_online_cpu (cpu) { +if (cpu_is_isolated (cpu)) +continue ; +flush_work (per_cpu_ptr (works, cpu)); + } +cpus_read_unlock (); +free_percpu (works);for (i =0 ; i <NR_VM_ZONE_STAT_ITEMS ; i++) {/* * Skip checking stats known to go negative occasionally.
Signature · 2 lines
Michal Hocko SUSE Labs
Re: [PATCH v3 3/3] mm/vmstat: do not refresh stats for isolated CPUs
Quoted 42 lines · Michal Hocko
Quoted 23 lines · Marcelo Tosatti
schedule_work_onAPI uses the workqueue mechanism to queue a work item on a queue. A kernel thread, which runs on the target CPU, executes those work items.Therefore, when using the
schedule_work_onAPI, it is necessary for the kworker kernel thread to be scheduled in, for the work function to be executed.Time sensitive applications such as SoftPLCs
(https://tum-esi.github.io/publications-list/PDF/2022-ETFA-How_Real_Time_Are_Virtual_PLCs.pdf),
have their response times affected by such interruptions.The
/proc/sys/vm/stat_refreshfile was originally introduced with the goal to:"Provide
/proc/sys/vm/stat_refreshto force an immediate update of
per-cpu into global vmstats: useful to avoid a sleep(2) or whatever before checking counts when testing. Originally added to work around a bug which left counts stranded indefinitely on a cpu going idle (an inaccuracy magnified when small below-batch numbers represent "huge" amounts of memory), but I believe that bug is now fixed: nonetheless, this is still a useful knob."Quoted 3 lines
Other than the potential interruption to a time sensitive application, if using
SCHED_FIFOorSCHED_RRpriority on the isolated CPU, then system hangs can occur:The same thing can happen without isolated CPUs and this patch doesn't help at all.
And this is an example of that...
Quoted 6 lines
To avoid the problems above, do not schedule the work to synchronize per-CPU mm counters on isolated CPUs. Given the possibility for breaking existing userspace applications, avoid returning errors from access to
/proc/sys/vm/stat_refresh.Signed-off-by Marcelo Tosatti <[email protected]>
Quoted 8 lines
It would be really helpful to not post new versions while discussion of the previous one is still not done.
Anyway
Nacked-by Michal Hocko <[email protected]>This is silently changing semantic and I do not think you have actually shown this is a real life problem.
https://bugzilla.redhat.com/show_bug.cgi?id=1921601
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=978688
Quoted 2 lines
To me it sounds like a theoretical
issue
Its not (see data above).
Quoted 4 lines
at most and it can be worked around by disalowing to use this interface from userspace.
stat_refreshis mostly for debugging purposes and I strongly doubt it is ever used in environments you refer to in this series.
Based on experience, I strongly believe people will run latency sensitive apps and end up reading/writing this file.
Michal Hocko
reviewerRe: [PATCH v3 3/3] mm/vmstat: do not refresh stats for isolated CPUs
Quoted 46 lines · Marcelo Tosatti
Quoted 45 lines · Michal Hocko
Quoted 23 lines · Marcelo Tosatti
schedule_work_onAPI uses the workqueue mechanism to queue a work item on a queue. A kernel thread, which runs on the target CPU, executes those work items.Therefore, when using the
schedule_work_onAPI, it is necessary for the kworker kernel thread to be scheduled in, for the work function to be executed.Time sensitive applications such as SoftPLCs
(https://tum-esi.github.io/publications-list/PDF/2022-ETFA-How_Real_Time_Are_Virtual_PLCs.pdf),
have their response times affected by such interruptions.The
/proc/sys/vm/stat_refreshfile was originally introduced with the goal to:"Provide
/proc/sys/vm/stat_refreshto force an immediate update of
per-cpu into global vmstats: useful to avoid a sleep(2) or whatever before checking counts when testing. Originally added to work around a bug which left counts stranded indefinitely on a cpu going idle (an inaccuracy magnified when small below-batch numbers represent "huge" amounts of memory), but I believe that bug is now fixed: nonetheless, this is still a useful knob."Quoted 3 lines
Other than the potential interruption to a time sensitive application, if using
SCHED_FIFOorSCHED_RRpriority on the isolated CPU, then system hangs can occur:The same thing can happen without isolated CPUs and this patch doesn't help at all.
And this is an example of that...
Quoted 6 lines
To avoid the problems above, do not schedule the work to synchronize per-CPU mm counters on isolated CPUs. Given the possibility for breaking existing userspace applications, avoid returning errors from access to
/proc/sys/vm/stat_refresh.Signed-off-by Marcelo Tosatti <[email protected]>It would be really helpful to not post new versions while discussion of the previous one is still not done.
Quoted 7 lines
Quoted 5 lines
Anyway
Nacked-by Michal Hocko <[email protected]>This is silently changing semantic and I do not think you have actually shown this is a real life problem.
Yet anothe new reference. Is the story complete now?
Again, nothing isolcpu related. Simple RR/realtime cpu hoggers that this patch has not way to deal with. Why do you keep referencing it?
Quoted 4 lines
Quoted 2 lines
To me it sounds like a theoretical
issueIts not (see data above).
No it't not. Either your sosreport need precise counters and stat_refresh is exactly what it should be doing or it can live with imprecision and then it should avoid it. Full stop. There is no reason to tweak the existing semantic to half bake isolcpu case while the fundamental problem is still there.
Signature · 2 lines
Michal Hocko SUSE Labs