Skip to contentSkip to conversation

vmstat bug fixes for nohz_full CPUs

PATCH Superseded Marcelo Tosatti 3 participants 14 messages May 30, 2023 – Jun 5, 2023

Subsystems (5)

v1 of 3 Superseded 1 of 4 applied Latest is v3 →

lore
Share & tools
lore
#0 lore raw

cover vmstat bug fixes for nohz_full CPUs

This patch series addresses the following two problems:

1. A customer provided evidence indicating that a process was stalled in direct reclaim:

- The process was trapped in throttle_direct_reclaim(). The function wait_event_killable() was called to wait condition allow_direct_reclaim(pgdat) for current node to be true. The allow_direct_reclaim(pgdat) examined the number of free pages on the node by zone_page_state() which just returns value in zone->vm_stat[NR_FREE_PAGES].

- On node #1, zone->vm_stat[NR_FREE_PAGES] was 0. However, the freelist on this node was not empty.

- This inconsistent of vmstat value was caused by percpu vmstat on nohz_full cpus. Every increment/decrement of vmstat is performed on percpu vmstat counter at first, then pooled diffs are cumulated to the zone's vmstat counter in timely manner. However, on nohz_full cpus (in case of this customer's system, 48 of 52 cpus) these pooled diffs were not cumulated once the cpu had no event on it so that the cpu started sleeping infinitely.
I checked percpu vmstat and found there were total 69 counts not cumulated to the zone's vmstat counter yet.

- In this situation, kswapd did not help the trapped process. In pgdat_balanced(), zone_wakermark_ok_safe() examined the number of free pages on the node by zone_page_state_snapshot() which checks pending counts on percpu vmstat.
Therefore kswapd could know there were 69 free pages correctly. Since zone->_watermark = {8, 20, 32}, kswapd did not work because 69 was greater than 32 as high watermark.

2. With a task that busy loops on a given CPU, the kworker interruption to execute vmstat_update is undesired and may exceed latency thresholds for certain applications.

First issue is solved by using _snapshot version of the counters on allow_direct_reclaim.

Second issue is fixed by disabling periodic vmstat updates for nohz_full CPUs.

Thanks to Aaron Tomlin for diagnosing issue 1 and writing the initial patch series.

Performance details for the kworker interruption:

oslat 1094.456862: sys_mlock(start: 7f7ed0000b60, len: 1000)
oslat 1094.456971: workqueue_queue_work: ... function=vmstat_update ...
oslat 1094.456974: sched_switch: prev_comm=oslat ... ==> next_comm=kworker/5:1 ...
kworker 1094.456978: sched_switch: prev_comm=kworker/5:1 ==> next_comm=oslat ...

The example above shows an additional 7us for the

        oslat -> kworker -> oslat

switches. In the case of a virtualized CPU, and the vmstat_update interruption in the host (of a qemu-kvm vcpu), the latency penalty observed in the guest is higher than 50us, violating the acceptable latency threshold for certain applications.

 include/linux/workqueue.h |    1 +
 kernel/workqueue.c        |   50 ++++++++++++++++++++++++++++++++++++++++++++++++++
 mm/vmscan.c               |    2 +-
 mm/vmstat.c               |   11 +++++++++++
 4 files changed, 63 insertions(+), 1 deletion(-)
#1 lore raw

PATCH 1/4 vmstat: allow_direct_reclaim should use zone_page_state_snapshot

A customer provided evidence indicating that a process was stalled in direct reclaim:

- The process was trapped in throttle_direct_reclaim(). The function wait_event_killable() was called to wait condition allow_direct_reclaim(pgdat) for current node to be true. The allow_direct_reclaim(pgdat) examined the number of free pages on the node by zone_page_state() which just returns value in zone->vm_stat[NR_FREE_PAGES].

- On node #1, zone->vm_stat[NR_FREE_PAGES] was 0. However, the freelist on this node was not empty.

- This inconsistent of vmstat value was caused by percpu vmstat on nohz_full cpus. Every increment/decrement of vmstat is performed on percpu vmstat counter at first, then pooled diffs are cumulated to the zone's vmstat counter in timely manner. However, on nohz_full cpus (in case of this customer's system, 48 of 52 cpus) these pooled diffs were not cumulated once the cpu had no event on it so that the cpu started sleeping infinitely.
I checked percpu vmstat and found there were total 69 counts not cumulated to the zone's vmstat counter yet.

- In this situation, kswapd did not help the trapped process. In pgdat_balanced(), zone_wakermark_ok_safe() examined the number of free pages on the node by zone_page_state_snapshot() which checks pending counts on percpu vmstat.
Therefore kswapd could know there were 69 free pages correctly. Since zone->_watermark = {8, 20, 32}, kswapd did not work because 69 was greater than 32 as high watermark.

Change allow_direct_reclaim to use zone_page_state_snapshot, which allows a more precise version of the vmstat counters to be used.

allow_direct_reclaim will only be called from try_to_free_pages, which is not a hot path.

Testing: Due to difficulties accessing the system, it has not been possible for the reproducer to test the patch (however its clear from available data and analysis that it should fix it).

---

mm/vmscan.c +1 -1
Index: linux-vmstat-remote/mm/vmscan.c
===================================================================
@@ -6887,7 +6887,7 @@ static bool allow_direct_reclaim(pg_data
 			continue;
 
 		pfmemalloc_reserve += min_wmark_pages(zone);
-		free_pages += zone_page_state(zone, NR_FREE_PAGES);
+		free_pages += zone_page_state_snapshot(zone, NR_FREE_PAGES);
 	}
 
 	/* If there are no reserves (unexpected config) then do not throttle */
reviewer
#2 lore raw

Re: [PATCH 1/4] vmstat: allow_direct_reclaim should use zone_page_state_snapshot

Andrew, as I've stated in previously, this sounds like a reasonable standalone fix worth taking independently of the rest.

Quoted 61 lines · Marcelo Tosatti

A customer provided evidence indicating that a process was stalled in direct reclaim:

- The process was trapped in throttle_direct_reclaim(). The function wait_event_killable() was called to wait condition allow_direct_reclaim(pgdat) for current node to be true. The allow_direct_reclaim(pgdat) examined the number of free pages on the node by zone_page_state() which just returns value in zone->vm_stat[NR_FREE_PAGES].

- On node #1, zone->vm_stat[NR_FREE_PAGES] was 0. However, the freelist on this node was not empty.

- This inconsistent of vmstat value was caused by percpu vmstat on nohz_full cpus. Every increment/decrement of vmstat is performed on percpu vmstat counter at first, then pooled diffs are cumulated to the zone's vmstat counter in timely manner. However, on nohz_full cpus (in case of this customer's system, 48 of 52 cpus) these pooled diffs were not cumulated once the cpu had no event on it so that the cpu started sleeping infinitely.
I checked percpu vmstat and found there were total 69 counts not cumulated to the zone's vmstat counter yet.

- In this situation, kswapd did not help the trapped process. In pgdat_balanced(), zone_wakermark_ok_safe() examined the number of free pages on the node by zone_page_state_snapshot() which checks pending counts on percpu vmstat.
Therefore kswapd could know there were 69 free pages correctly. Since zone->_watermark = {8, 20, 32}, kswapd did not work because 69 was greater than 32 as high watermark.

Change allow_direct_reclaim to use zone_page_state_snapshot, which allows a more precise version of the vmstat counters to be used.

allow_direct_reclaim will only be called from try_to_free_pages, which is not a hot path.

Testing: Due to difficulties accessing the system, it has not been possible for the reproducer to test the patch (however its clear from available data and analysis that it should fix it).

---

Index: linux-vmstat-remote/mm/vmscan.c
===================================================================

mm/vmscan.c +1 -1
@@ -6887,7 +6887,7 @@ static bool allow_direct_reclaim(pg_data
 			continue;
 
 		pfmemalloc_reserve += min_wmark_pages(zone);
-		free_pages += zone_page_state(zone, NR_FREE_PAGES);
+		free_pages += zone_page_state_snapshot(zone, NR_FREE_PAGES);
 	}
 
 	/* If there are no reserves (unexpected config) then do not throttle */
Signature · 2 lines
Michal Hocko
SUSE Labs
#3 lore raw

PATCH 2/4 vmstat: skip periodic vmstat update for nohz full CPUs

In reply to Marcelo Tosatti

The interruption caused by vmstat_update is undesirable for certain aplications:

oslat 1094.456862: sys_mlock(start: 7f7ed0000b60, len: 1000)
oslat 1094.456971: workqueue_queue_work: ... function=vmstat_update ...
oslat 1094.456974: sched_switch: prev_comm=oslat ... ==> next_comm=kworker/5:1 ...
kworker 1094.456978: sched_switch: prev_comm=kworker/5:1 ==> next_comm=oslat ...

The example above shows an additional 7us for the

       	oslat -> kworker -> oslat

switches. In the case of a virtualized CPU, and the vmstat_update interruption in the host (of a qemu-kvm vcpu), the latency penalty observed in the guest is higher than 50us, violating the acceptable latency threshold.

Skip periodic updates for nohz full CPUs. Any callers who need precise values should use a snapshot of the per-CPU counters, or use the global counters with measures to handle errors up to thresholds (see calculate_normal_threshold).

Suggested by Michal Hocko.

---

mm/vmstat.c +11 -0
Index: linux-vmstat-remote/mm/vmstat.c
===================================================================
@@ -28,6 +28,7 @@
 #include <linux/mm_inline.h>
 #include <linux/page_ext.h>
 #include <linux/page_owner.h>
+#include <linux/tick.h>
 
 #include "internal.h"
 
@@ -2022,6 +2023,16 @@ static void vmstat_shepherd(struct work_
 	for_each_online_cpu(cpu) {
 		struct delayed_work *dw = &per_cpu(vmstat_work, cpu);
 
+		/*
+		 * Skip periodic updates for nohz full CPUs.
+		 * Any callers who need precise values should use
+		 * a snapshot of the per-CPU counters, or use the global
+		 * counters with measures to handle errors up to
+		 * thresholds (see calculate_normal_threshold).
+		 */
+		if (tick_nohz_full_cpu(cpu))
+			continue;
+
 		if (!delayed_work_pending(dw) && need_update(cpu))
 			queue_delayed_work_on(cpu, mm_percpu_wq, dw, 0);
 
reviewer
#4 lore raw

Re: [PATCH 2/4] vmstat: skip periodic vmstat update for nohz full CPUs

mm/vmstat.c static void vmstat_shepherd(struct work_ · line 2034

Quoted 13 lines · Marcelo Tosatti
mm/vmstat.c +9 -0
@@ -2022,6 +2023,16 @@ static void vmstat_shepherd(struct work_
 	for_each_online_cpu(cpu) {
 		struct delayed_work *dw = &per_cpu(vmstat_work, cpu);
 
+		/*
+		 * Skip periodic updates for nohz full CPUs.
+		 * Any callers who need precise values should use
+		 * a snapshot of the per-CPU counters, or use the global
+		 * counters with measures to handle errors up to
+		 * thresholds (see calculate_normal_threshold).
+		 */
+		if (tick_nohz_full_cpu(cpu))
+			continue;

In other code path we have used cpu_is_isolated, is there any reason to diverge from that here? Isn't this effectivelly the same kind of problem?

Quoted 5 lines
mm/vmstat.c +1 -0
+
 		if (!delayed_work_pending(dw) && need_update(cpu))
 			queue_delayed_work_on(cpu, mm_percpu_wq, dw, 0);
 
Signature · 2 lines
Michal Hocko
SUSE Labs
#5 lore raw

Re: [PATCH 2/4] vmstat: skip periodic vmstat update for nohz full CPUs

Quoted 5 lines · Michal Hocko
Quoted 4 lines · Marcelo Tosatti
mm/vmstat.c +0 -0
@@ -2022,6 +2023,16 @@ static void vmstat_shepherd(struct work_
 	for_each_online_cpu(cpu) {
 		struct delayed_work *dw = &per_cpu(vmstat_work, cpu);
 
Quoted 13 lines

mm/vmstat.c static void vmstat_shepherd(struct work_ · line 2034

Quoted 9 lines
mm/vmstat.c +9 -0
+		/*
+		 * Skip periodic updates for nohz full CPUs.
+		 * Any callers who need precise values should use
+		 * a snapshot of the per-CPU counters, or use the global
+		 * counters with measures to handle errors up to
+		 * thresholds (see calculate_normal_threshold).
+		 */
+		if (tick_nohz_full_cpu(cpu))
+			continue;

In other code path we have used cpu_is_isolated, is there any reason to diverge from that here? Isn't this effectivelly the same kind of problem?

Changed to cpu_is_isolated, thanks.

#6 lore raw

PATCH 3/4 workqueue: add schedule_on_each_cpumask helper

In reply to Marcelo Tosatti

Add a schedule_on_each_cpumask function, equivalent to schedule_on_each_cpu but accepting a cpumask to operate.

---

kernel/workqueue.c +50 -0
Index: linux-vmstat-remote/kernel/workqueue.c
===================================================================
@@ -3455,6 +3455,56 @@ int schedule_on_each_cpu(work_func_t fun
 	return 0;
 }
 
+
+/**
+ * schedule_on_each_cpumask - execute a function synchronously on each
+ * CPU in "cpumask", for those which are online.
+ *
+ * @func: the function to call
+ * @mask: the CPUs which to call function on
+ *
+ * schedule_on_each_cpu() executes @func on each specified CPU that is online,
+ * using the system workqueue and blocks until all such CPUs have completed.
+ * schedule_on_each_cpu() is very slow.
+ *
+ * Return:
+ * 0 on success, -errno on failure.
+ */
+int schedule_on_each_cpumask(work_func_t func, cpumask_t *cpumask)
+{
+	int cpu;
+	struct work_struct __percpu *works;
+	cpumask_var_t effmask;
+
+	works = alloc_percpu(struct work_struct);
+	if (!works)
+		return -ENOMEM;
+
+	if (!alloc_cpumask_var(&effmask, GFP_KERNEL)) {
+		free_percpu(works);
+		return -ENOMEM;
+	}
+
+	cpumask_and(effmask, cpumask, cpu_online_mask);
+
+	cpus_read_lock();
+
+	for_each_cpu(cpu, effmask) {
+		struct work_struct *work = per_cpu_ptr(works, cpu);
+
+		INIT_WORK(work, func);
+		schedule_work_on(cpu, work);
+	}
+
+	for_each_cpu(cpu, effmask)
+		flush_work(per_cpu_ptr(works, cpu));
+
+	cpus_read_unlock();
+	free_percpu(works);
+	free_cpumask_var(effmask);
+	return 0;
+}
+
 /**
  * execute_in_process_context - reliably execute the routine with user context
  * @fn:		the function to execute
include/linux/workqueue.h +1 -0
Index: linux-vmstat-remote/include/linux/workqueue.h
===================================================================
@@ -450,6 +450,7 @@ extern void __flush_workqueue(struct wor
 extern void drain_workqueue(struct workqueue_struct *wq);
 
 extern int schedule_on_each_cpu(work_func_t func);
+extern int schedule_on_each_cpumask(work_func_t func, cpumask_t *cpumask);
 
 int execute_in_process_context(work_func_t fn, struct execute_work *);
 
maintainer
#7 lore raw

Re: [PATCH 3/4] workqueue: add schedule_on_each_cpumask helper

Quoted 40 lines · On Tue, 30 May 2023 11:52:37 -0300 Marcelo Tosatti <[email protected]> wrote:

Add a schedule_on_each_cpumask function, equivalent to schedule_on_each_cpu but accepting a cpumask to operate.

---

Index: linux-vmstat-remote/kernel/workqueue.c
===================================================================

kernel/workqueue.c +25 -0
@@ -3455,6 +3455,56 @@ int schedule_on_each_cpu(work_func_t fun
 	return 0;
 }
 
+
+/**
+ * schedule_on_each_cpumask - execute a function synchronously on each
+ * CPU in "cpumask", for those which are online.
+ *
+ * @func: the function to call
+ * @mask: the CPUs which to call function on
+ *
+ * schedule_on_each_cpu() executes @func on each specified CPU that is online,
+ * using the system workqueue and blocks until all such CPUs have completed.
+ * schedule_on_each_cpu() is very slow.
+ *
+ * Return:
+ * 0 on success, -errno on failure.
+ */
+int schedule_on_each_cpumask(work_func_t func, cpumask_t *cpumask)
+{
+	int cpu;
+	struct work_struct __percpu *works;
+	cpumask_var_t effmask;
+
+	works = alloc_percpu(struct work_struct);
+	if (!works)
+		return -ENOMEM;
+

kernel/workqueue.c int schedule_on_each_cpu(work_func_t fun · line 3492

Quoted 10 lines
kernel/workqueue.c +10 -0
+	if (!alloc_cpumask_var(&effmask, GFP_KERNEL)) {
+		free_percpu(works);
+		return -ENOMEM;
+	}
+
+	cpumask_and(effmask, cpumask, cpu_online_mask);
+
+	cpus_read_lock();
+
+	for_each_cpu(cpu, effmask) {

Should we check here that the cpu is still online?

Quoted 21 lines
kernel/workqueue.c +15 -0
+		struct work_struct *work = per_cpu_ptr(works, cpu);
+
+		INIT_WORK(work, func);
+		schedule_work_on(cpu, work);
+	}
+
+	for_each_cpu(cpu, effmask)
+		flush_work(per_cpu_ptr(works, cpu));
+
+	cpus_read_unlock();
+	free_percpu(works);
+	free_cpumask_var(effmask);
+	return 0;
+}
+
 /**
  * execute_in_process_context - reliably execute the routine with user context
  * @fn:		the function to execute

include/linux/workqueue.h extern void __flush_workqueue(struct wor · line 453

Quoted 4 lines
include/linux/workqueue.h +1 -0
@@ -450,6 +450,7 @@ extern void __flush_workqueue(struct wor
 extern void drain_workqueue(struct workqueue_struct *wq);
 
 extern int schedule_on_each_cpu(work_func_t func);
+extern int schedule_on_each_cpumask(work_func_t func, cpumask_t *cpumask);

May as well make schedule_on_each_cpu() call schedule_on_each_cpumask()? Save a bit of text, and they're hardly performance-critical to that extent.

#8 lore raw

Re: [PATCH 3/4] workqueue: add schedule_on_each_cpumask helper

Quoted 78 lines · Andrew Morton
Quoted 40 lines · On Tue, 30 May 2023 11:52:37 -0300 Marcelo Tosatti <[email protected]> wrote:

Add a schedule_on_each_cpumask function, equivalent to schedule_on_each_cpu but accepting a cpumask to operate.

---

Index: linux-vmstat-remote/kernel/workqueue.c
===================================================================

kernel/workqueue.c +25 -0
@@ -3455,6 +3455,56 @@ int schedule_on_each_cpu(work_func_t fun
 	return 0;
 }
 
+
+/**
+ * schedule_on_each_cpumask - execute a function synchronously on each
+ * CPU in "cpumask", for those which are online.
+ *
+ * @func: the function to call
+ * @mask: the CPUs which to call function on
+ *
+ * schedule_on_each_cpu() executes @func on each specified CPU that is online,
+ * using the system workqueue and blocks until all such CPUs have completed.
+ * schedule_on_each_cpu() is very slow.
+ *
+ * Return:
+ * 0 on success, -errno on failure.
+ */
+int schedule_on_each_cpumask(work_func_t func, cpumask_t *cpumask)
+{
+	int cpu;
+	struct work_struct __percpu *works;
+	cpumask_var_t effmask;
+
+	works = alloc_percpu(struct work_struct);
+	if (!works)
+		return -ENOMEM;
+

kernel/workqueue.c int schedule_on_each_cpu(work_func_t fun · line 3492

Quoted 10 lines
kernel/workqueue.c +10 -0
+	if (!alloc_cpumask_var(&effmask, GFP_KERNEL)) {
+		free_percpu(works);
+		return -ENOMEM;
+	}
+
+	cpumask_and(effmask, cpumask, cpu_online_mask);
+
+	cpus_read_lock();
+
+	for_each_cpu(cpu, effmask) {

Should we check here that the cpu is still online?

Quoted 23 lines
kernel/workqueue.c +15 -0
+		struct work_struct *work = per_cpu_ptr(works, cpu);
+
+		INIT_WORK(work, func);
+		schedule_work_on(cpu, work);
+	}
+
+	for_each_cpu(cpu, effmask)
+		flush_work(per_cpu_ptr(works, cpu));
+
+	cpus_read_unlock();
+	free_percpu(works);
+	free_cpumask_var(effmask);
+	return 0;
+}
+
 /**
  * execute_in_process_context - reliably execute the routine with user context
  * @fn:		the function to execute
include/linux/workqueue.h +0 -0
@@ -450,6 +450,7 @@ extern void __flush_workqueue(struct wor
 extern void drain_workqueue(struct workqueue_struct *wq);
 
Quoted 6 lines

include/linux/workqueue.h extern void __flush_workqueue(struct wor · line 453

Quoted 2 lines
include/linux/workqueue.h +1 -0
 extern int schedule_on_each_cpu(work_func_t func);
+extern int schedule_on_each_cpumask(work_func_t func, cpumask_t *cpumask);

May as well make schedule_on_each_cpu() call schedule_on_each_cpumask()? Save a bit of text, and they're hardly performance-critical to that extent.

Agree, will wait for Michal's review before resending -v2.

workqueue: add schedule_on_each_cpumask helper

Add a schedule_on_each_cpumask function, equivalent to schedule_on_each_cpu but accepting a cpumask to operate.

---

v2: - cpu_online_mask reference should happen with cpus_read_lock held (Andrew Morton)
- have schedule_on_each_cpu call _cpumask variant (Andrew Morton)

kernel/workqueue.c +31 -2
Index: linux-vmstat-remote/kernel/workqueue.c
===================================================================
@@ -3431,27 +3431,56 @@ EXPORT_SYMBOL(cancel_delayed_work_sync);
  */
 int schedule_on_each_cpu(work_func_t func)
 {
+	return schedule_on_each_cpumask(func, cpu_possible_mask);
+}
+
+
+/**
+ * schedule_on_each_cpumask - execute a function synchronously on each
+ * CPU in "cpumask", for those which are online.
+ *
+ * @func: the function to call
+ * @mask: the CPUs which to call function on
+ *
+ * schedule_on_each_cpu() executes @func on each specified CPU that is online,
+ * using the system workqueue and blocks until all such CPUs have completed.
+ * schedule_on_each_cpu() is very slow.
+ *
+ * Return:
+ * 0 on success, -errno on failure.
+ */
+int schedule_on_each_cpumask(work_func_t func, cpumask_t *cpumask)
+{
 	int cpu;
 	struct work_struct __percpu *works;
+	cpumask_var_t effmask;
 
 	works = alloc_percpu(struct work_struct);
 	if (!works)
 		return -ENOMEM;
 
+	if (!alloc_cpumask_var(&effmask, GFP_KERNEL)) {
+		free_percpu(works);
+		return -ENOMEM;
+	}
+
 	cpus_read_lock();
 
-	for_each_online_cpu(cpu) {
+	cpumask_and(effmask, cpumask, cpu_online_mask);
+
+	for_each_cpu(cpu, effmask) {
 		struct work_struct *work = per_cpu_ptr(works, cpu);
 
 		INIT_WORK(work, func);
 		schedule_work_on(cpu, work);
 	}
 
-	for_each_online_cpu(cpu)
+	for_each_cpu(cpu, effmask)
 		flush_work(per_cpu_ptr(works, cpu));
 
 	cpus_read_unlock();
 	free_percpu(works);
+	free_cpumask_var(effmask);
 	return 0;
 }
 
include/linux/workqueue.h +1 -0
Index: linux-vmstat-remote/include/linux/workqueue.h
===================================================================
@@ -450,6 +450,7 @@ extern void __flush_workqueue(struct wor
 extern void drain_workqueue(struct workqueue_struct *wq);
 
 extern int schedule_on_each_cpu(work_func_t func);
+extern int schedule_on_each_cpumask(work_func_t func, cpumask_t *cpumask);
 
 int execute_in_process_context(work_func_t fn, struct execute_work *);
 
reviewer
#9 lore raw

Re: [PATCH 3/4] workqueue: add schedule_on_each_cpumask helper

In reply to Marcelo Tosatti

You should be CCing WQ maintainers on changes like this one (now added).

Quoted 2 lines · Marcelo Tosatti

Add a schedule_on_each_cpumask function, equivalent to schedule_on_each_cpu but accepting a cpumask to operate.

IMHO it is preferable to add a new function along with its user so that the usecase is more clear.

Quoted 37 lines

---

Index: linux-vmstat-remote/kernel/workqueue.c
===================================================================

kernel/workqueue.c +25 -0
@@ -3455,6 +3455,56 @@ int schedule_on_each_cpu(work_func_t fun
 	return 0;
 }
 
+
+/**
+ * schedule_on_each_cpumask - execute a function synchronously on each
+ * CPU in "cpumask", for those which are online.
+ *
+ * @func: the function to call
+ * @mask: the CPUs which to call function on
+ *
+ * schedule_on_each_cpu() executes @func on each specified CPU that is online,
+ * using the system workqueue and blocks until all such CPUs have completed.
+ * schedule_on_each_cpu() is very slow.
+ *
+ * Return:
+ * 0 on success, -errno on failure.
+ */
+int schedule_on_each_cpumask(work_func_t func, cpumask_t *cpumask)
+{
+	int cpu;
+	struct work_struct __percpu *works;
+	cpumask_var_t effmask;
+
+	works = alloc_percpu(struct work_struct);
+	if (!works)
+		return -ENOMEM;
+

kernel/workqueue.c int schedule_on_each_cpu(work_func_t fun · line 3492

Quoted 10 lines
kernel/workqueue.c +10 -0
+	if (!alloc_cpumask_var(&effmask, GFP_KERNEL)) {
+		free_percpu(works);
+		return -ENOMEM;
+	}
+
+	cpumask_and(effmask, cpumask, cpu_online_mask);
+
+	cpus_read_lock();
+
+	for_each_cpu(cpu, effmask) {

Is the cpu_online_mask dance really necessary? Why cannot you simply do for_each_online_cpu here? flush_work on unqueued work item should just return, no?

Also there is no synchronization with the cpu hotplug so cpu_online_mask can change under your feet so this construct seem unsafe to me.

Quoted 31 lines
kernel/workqueue.c +15 -0
+		struct work_struct *work = per_cpu_ptr(works, cpu);
+
+		INIT_WORK(work, func);
+		schedule_work_on(cpu, work);
+	}
+
+	for_each_cpu(cpu, effmask)
+		flush_work(per_cpu_ptr(works, cpu));
+
+	cpus_read_unlock();
+	free_percpu(works);
+	free_cpumask_var(effmask);
+	return 0;
+}
+
 /**
  * execute_in_process_context - reliably execute the routine with user context
  * @fn:		the function to execute

Index: linux-vmstat-remote/include/linux/workqueue.h
===================================================================

include/linux/workqueue.h +1 -0
@@ -450,6 +450,7 @@ extern void __flush_workqueue(struct wor
 extern void drain_workqueue(struct workqueue_struct *wq);
 
 extern int schedule_on_each_cpu(work_func_t func);
+extern int schedule_on_each_cpumask(work_func_t func, cpumask_t *cpumask);
 
 int execute_in_process_context(work_func_t fn, struct execute_work *);
 
Signature · 2 lines
Michal Hocko
SUSE Labs
#10 lore raw

Re: [PATCH 3/4] workqueue: add schedule_on_each_cpumask helper

Quoted 59 lines · Michal Hocko

You should be CCing WQ maintainers on changes like this one (now added).

Quoted 2 lines · Marcelo Tosatti

Add a schedule_on_each_cpumask function, equivalent to schedule_on_each_cpu but accepting a cpumask to operate.

IMHO it is preferable to add a new function along with its user so that the usecase is more clear.

Quoted 37 lines

---

Index: linux-vmstat-remote/kernel/workqueue.c
===================================================================

kernel/workqueue.c +25 -0
@@ -3455,6 +3455,56 @@ int schedule_on_each_cpu(work_func_t fun
 	return 0;
 }
 
+
+/**
+ * schedule_on_each_cpumask - execute a function synchronously on each
+ * CPU in "cpumask", for those which are online.
+ *
+ * @func: the function to call
+ * @mask: the CPUs which to call function on
+ *
+ * schedule_on_each_cpu() executes @func on each specified CPU that is online,
+ * using the system workqueue and blocks until all such CPUs have completed.
+ * schedule_on_each_cpu() is very slow.
+ *
+ * Return:
+ * 0 on success, -errno on failure.
+ */
+int schedule_on_each_cpumask(work_func_t func, cpumask_t *cpumask)
+{
+	int cpu;
+	struct work_struct __percpu *works;
+	cpumask_var_t effmask;
+
+	works = alloc_percpu(struct work_struct);
+	if (!works)
+		return -ENOMEM;
+

kernel/workqueue.c int schedule_on_each_cpu(work_func_t fun · line 3492

Quoted 10 lines
kernel/workqueue.c +10 -0
+	if (!alloc_cpumask_var(&effmask, GFP_KERNEL)) {
+		free_percpu(works);
+		return -ENOMEM;
+	}
+
+	cpumask_and(effmask, cpumask, cpu_online_mask);
+
+	cpus_read_lock();
+
+	for_each_cpu(cpu, effmask) {

Is the cpu_online_mask dance really necessary?

Quoted 1 line

Why cannot you simply do for_each_online_cpu here?

Are you suggesting to do:

	for_each_online_cpu(cpu) {
		if cpu is not in cpumask
			continue;
		...
	}

This does not seem efficient.

Quoted 2 lines

flush_work on unqueued work item should just
return, no?

Apparently not:

commit 0e8d6a9336b487a1dd6f1991ff376e669d4c87c6

Author: Thomas Gleixner <[email protected]>
Date:   Wed Apr 12 22:07:28 2017 +0200

workqueue: Provide work_on_cpu_safe()

work_on_cpu() is not protected against CPU hotplug. For code which requires to be either executed on an online CPU or to fail if the CPU is not available the callsite would have to protect against CPU hotplug.

Provide a function which does get/put_online_cpus() around the call to work_on_cpu() and fails the call with -ENODEV if the target CPU is not online.

Quoted 2 lines

Also there is no synchronization with the cpu hotplug so cpu_online_mask can change under your feet so this construct seem unsafe to me.

Yes, fixed by patch in response to Andrew's comment.

reviewer
#11 lore raw

Re: [PATCH 3/4] workqueue: add schedule_on_each_cpumask helper

Quoted 1 line · Marcelo Tosatti

On Fri, Jun 02, 2023 at 12:48:23PM +0200, Michal Hocko wrote:

[...]

Quoted 17 lines
Quoted 13 lines

kernel/workqueue.c int schedule_on_each_cpu(work_func_t fun · line 3492

Quoted 10 lines
kernel/workqueue.c +10 -0
+	if (!alloc_cpumask_var(&effmask, GFP_KERNEL)) {
+		free_percpu(works);
+		return -ENOMEM;
+	}
+
+	cpumask_and(effmask, cpumask, cpu_online_mask);
+
+	cpus_read_lock();
+
+	for_each_cpu(cpu, effmask) {

Is the cpu_online_mask dance really necessary?

Quoted 1 line

Why cannot you simply do for_each_online_cpu here?

Are you suggesting to do:

Quoted 7 lines
	for_each_online_cpu(cpu) {
		if cpu is not in cpumask
			continue;
		...
	}

This does not seem efficient.

Are you sure this is less sufficient than a memory allocation?

Signature · 2 lines
Michal Hocko
SUSE Labs
#12 lore raw

PATCH 4/4 mm/vmstat: do not refresh stats for nohz_full CPUs

In reply to Marcelo Tosatti

The interruption caused by queueing work on nohz_full CPUs is undesirable for certain aplications.

Fix by not refreshing per-CPU stats of nohz_full CPUs.

---

mm/vmstat.c +22 -1
Index: linux-vmstat-remote/mm/vmstat.c
===================================================================
@@ -1877,12 +1877,31 @@ static void refresh_vm_stats(struct work
 	refresh_cpu_vm_stats(true);
 }
 
+#ifdef CONFIG_NO_HZ_FULL
+static inline const cpumask_t *tickless_cpumask(void)
+{
+	return tick_nohz_full_mask;
+}
+#else
+static cpumask_t empty_cpumask;
+static inline const cpumask_t *tickless_cpumask(void)
+{
+	return &empty_cpumask;
+}
+#endif
+
 int vmstat_refresh(struct ctl_table *table, int write,
 		   void *buffer, size_t *lenp, loff_t *ppos)
 {
 	long val;
 	int err;
 	int i;
+	cpumask_var_t dstmask;
+
+	if (!alloc_cpumask_var(&dstmask, GFP_KERNEL))
+		return -ENOMEM;
+
+	cpumask_andnot(dstmask, cpu_possible_mask, tickless_cpumask());
 
 	/*
 	 * The regular update, every sysctl_stat_interval, may come later
@@ -1896,7 +1915,9 @@ int vmstat_refresh(struct ctl_table *tab
 	 * transiently negative values, report an error here if any of
 	 * the stats is negative, so we know to go looking for imbalance.
 	 */
-	err = schedule_on_each_cpu(refresh_vm_stats);
+	err = schedule_on_each_cpumask(refresh_vm_stats, dstmask);
+	free_cpumask_var(dstmask);
+
 	if (err)
 		return err;
 	for (i = 0; i < NR_VM_ZONE_STAT_ITEMS; i++) {
reviewer
#13 lore raw

Re: [PATCH 4/4] mm/vmstat: do not refresh stats for nohz_full CPUs

Quoted 32 lines · Marcelo Tosatti

The interruption caused by queueing work on nohz_full CPUs is undesirable for certain aplications.

Fix by not refreshing per-CPU stats of nohz_full CPUs.

---

Index: linux-vmstat-remote/mm/vmstat.c
===================================================================

mm/vmstat.c +13 -0
@@ -1877,12 +1877,31 @@ static void refresh_vm_stats(struct work
 	refresh_cpu_vm_stats(true);
 }
 
+#ifdef CONFIG_NO_HZ_FULL
+static inline const cpumask_t *tickless_cpumask(void)
+{
+	return tick_nohz_full_mask;
+}
+#else
+static cpumask_t empty_cpumask;
+static inline const cpumask_t *tickless_cpumask(void)
+{
+	return &empty_cpumask;
+}
+#endif
+
 int vmstat_refresh(struct ctl_table *table, int write,
 		   void *buffer, size_t *lenp, loff_t *ppos)

mm/vmstat.c static void refresh_vm_stats(struct work · line 1904

Quoted 10 lines
mm/vmstat.c +6 -0
 {
 	long val;
 	int err;
 	int i;
+	cpumask_var_t dstmask;
+
+	if (!alloc_cpumask_var(&dstmask, GFP_KERNEL))
+		return -ENOMEM;
+
+	cpumask_andnot(dstmask, cpu_possible_mask, tickless_cpumask());

Similar feedback to the patch 2. wouldn't it make more sense to opencode schedule_on_each_cpu here and use cpu_is_isolated instead?

Quoted 15 lines
mm/vmstat.c +3 -1
 	/*
 	 * The regular update, every sysctl_stat_interval, may come later
@@ -1896,7 +1915,9 @@ int vmstat_refresh(struct ctl_table *tab
 	 * transiently negative values, report an error here if any of
 	 * the stats is negative, so we know to go looking for imbalance.
 	 */
-	err = schedule_on_each_cpu(refresh_vm_stats);
+	err = schedule_on_each_cpumask(refresh_vm_stats, dstmask);
+	free_cpumask_var(dstmask);
+
 	if (err)
 		return err;
 	for (i = 0; i < NR_VM_ZONE_STAT_ITEMS; i++) {
Signature · 2 lines
Michal Hocko
SUSE Labs