Introduce QPW for per-cpu operations

Some places in the kernel implement a parallel programming strategy consisting on local_locks() for most of the work, and some rare remote operations are scheduled on target cpu. This keeps cache bouncing low since cacheline tends to be mostly local, and avoids the cost of locks in non-RT kernels, even though the very few remote operations will be expensive due to scheduling overhead.

Some places in the kernel implement a parallel programming strategy consisting on local_locks() for most of the work, and some rare remote operations are scheduled on target cpu. This keeps cache bouncing low since cacheline tends to be mostly local, and avoids the cost of locks in non-RT kernels, even though the very few remote operations will be expensive due to scheduling overhead.

On the other hand, for RT workloads this can represent a problem:
scheduling work on remote cpu that are executing low latency tasks is undesired and can introduce unexpected deadline misses.

It's interesting, though, that local_lock()s in RT kernels become spinlock(). We can make use of those to avoid scheduling work on a remote cpu by directly updating another cpu's per_cpu structure, while holding it's spinlock().

In order to do that, it's necessary to introduce a new set of functions to make it possible to get another cpu's per-cpu "local" lock (qpw_{un,}lock*) and also the corresponding queue_percpu_work_on() and flush_percpu_work() helpers to run the remote work.

Users of non-RT kernels but with low latency requirements can select similar functionality by using the CONFIG_QPW compile time option.

On CONFIG_QPW disabled kernels, no changes are expected, as every one of the introduced helpers work the exactly same as the current implementation:

qpw_{un,}lock*()        ->  local_{un,}lock*() (ignores cpu parameter)
queue_percpu_work_on()  ->  queue_work_on()
flush_percpu_work()     ->  flush_work()

For QPW enabled kernels, though, qpw_{un,}lock*() will use the extra cpu parameter to select the correct per-cpu structure to work on, and acquire the spinlock for that cpu.

queue_percpu_work_on() will just call the requested function in the current cpu, which will operate in another cpu's per-cpu object. Since the local_locks() become spinlock()s in QPW enabled kernels, we are safe doing that.

flush_percpu_work() then becomes a no-op since no work is actually scheduled on a remote cpu.

Some minimal code rework is needed in order to make this mechanism work:
The calls for local_{un,}lock*() on the functions that are currently scheduled on remote cpus need to be replaced by qpw_{un,}lock_n*(), so in QPW enabled kernels they can reference a different cpu. It's also necessary to use a qpw_struct instead of a work_struct, but it just contains a work struct and, in CONFIG_QPW, the target cpu.

This should have almost no impact on non-CONFIG_QPW kernels: few this_cpu_ptr() will become per_cpu_ptr(,smp_processor_id()).

On CONFIG_QPW kernels, this should avoid deadlines misses by removing scheduling noise.

Signed-off-by Leonardo Bras <[email protected]>

Signed-off-by Marcelo Tosatti <[email protected]>

---
 Documentation/admin-guide/kernel-parameters.txt |   10 +
 Documentation/locking/qpwlocks.rst              |   63 +++++++
 MAINTAINERS                                     |    6 
 include/linux/qpw.h                             |  190 ++++++++++++++++++++++++
 init/Kconfig                                    |   35 ++++
 kernel/Makefile                                 |    2 
 kernel/qpw.c                                    |   26 +++
 7 files changed, 332 insertions(+)
 create mode 100644 include/linux/qpw.h
 create mode 100644 kernel/qpw.c

Index: slab/Documentation/admin-guide/kernel-parameters.txt
===================================================================

Documentation/admin-guide/kernel-parameters.txt +10 -0

@@ -2819,6 +2819,16 @@ Kernel parameters
 
 			The format of <cpu-list> is described above.
 
+	qpw=		[KNL,SMP] Select a behavior on per-CPU resource sharing
+			and remote interference mechanism on a kernel built with
+			CONFIG_QPW.
+			Format: { "0" | "1" }
+			0 - local_lock() + queue_work_on(remote_cpu)
+			1 - spin_lock() for both local and remote operations
+
+			Selecting 1 may be interesting for systems that want
+			to avoid interruption & context switches from IPIs.
+
 	iucv=		[HW,NET]
 
 	ivrs_ioapic	[HW,X86-64]

Index: slab/MAINTAINERS
===================================================================

MAINTAINERS +0 -0

 QT1010 MEDIA DRIVER
 L:	[email protected]
 S:	Orphan

Index: slab/include/linux/qpw.h
===================================================================

include/linux/qpw.h new file +190 -0

@@ -0,0 +1,190 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _LINUX_QPW_H
+#define _LINUX_QPW_H
+
+#include "linux/spinlock.h"
+#include "linux/local_lock.h"
+#include "linux/workqueue.h"
+
+#ifndef CONFIG_QPW
+
+typedef local_lock_t qpw_lock_t;
+typedef local_trylock_t qpw_trylock_t;
+
+struct qpw_struct {
+	struct work_struct work;
+};
+
+#define qpw_lock_init(lock)			\
+	local_lock_init(lock)
+
+#define qpw_trylock_init(lock)			\
+	local_trylock_init(lock)
+
+#define qpw_lock(lock, cpu)			\
+	local_lock(lock)
+
+#define qpw_lock_irqsave(lock, flags, cpu)	\
+	local_lock_irqsave(lock, flags)
+
+#define qpw_trylock(lock, cpu)			\
+	local_trylock(lock)
+
+#define qpw_trylock_irqsave(lock, flags, cpu)	\
+	local_trylock_irqsave(lock, flags)
+
+#define qpw_unlock(lock, cpu)			\
+	local_unlock(lock)
+
+#define qpw_unlock_irqrestore(lock, flags, cpu)	\
+	local_unlock_irqrestore(lock, flags)
+
+#define qpw_lockdep_assert_held(lock)		\
+	lockdep_assert_held(lock)
+
+#define queue_percpu_work_on(c, wq, qpw)	\
+	queue_work_on(c, wq, &(qpw)->work)
+
+#define flush_percpu_work(qpw)			\
+	flush_work(&(qpw)->work)
+
+#define qpw_get_cpu(qpw)	smp_processor_id()
+
+#define qpw_is_cpu_remote(cpu)		(false)
+
+#define INIT_QPW(qpw, func, c)			\
+	INIT_WORK(&(qpw)->work, (func))
+
+#else /* CONFIG_QPW */
+
+DECLARE_STATIC_KEY_MAYBE(CONFIG_QPW_DEFAULT, qpw_sl);
+
+typedef union {
+	spinlock_t sl;
+	local_lock_t ll;
+} qpw_lock_t;
+
+typedef union {
+	spinlock_t sl;
+	local_trylock_t ll;
+} qpw_trylock_t;
+
+struct qpw_struct {
+	struct work_struct work;
+	int cpu;
+};
+
+#define qpw_lock_init(lock)								\
+	do {										\
+		if (static_branch_maybe(CONFIG_QPW_DEFAULT, &qpw_sl))			\
+			spin_lock_init(lock.sl);					\
+		else									\
+			local_lock_init(lock.ll);					\
+	} while (0)
+
+#define qpw_trylock_init(lock)								\
+	do {										\
+		if (static_branch_maybe(CONFIG_QPW_DEFAULT, &qpw_sl))			\
+			spin_lock_init(lock.sl);					\
+		else									\
+			local_trylock_init(lock.ll);					\
+	} while (0)
+
+#define qpw_lock(lock, cpu)								\
+	do {										\
+		if (static_branch_maybe(CONFIG_QPW_DEFAULT, &qpw_sl))			\
+			spin_lock(per_cpu_ptr(lock.sl, cpu));				\
+		else									\
+			local_lock(lock.ll);						\
+	} while (0)
+
+#define qpw_lock_irqsave(lock, flags, cpu)						\
+	do {										\
+		if (static_branch_maybe(CONFIG_QPW_DEFAULT, &qpw_sl))			\
+			spin_lock_irqsave(per_cpu_ptr(lock.sl, cpu), flags);		\
+		else									\
+			local_lock_irqsave(lock.ll, flags);				\
+	} while (0)
+
+#define qpw_trylock(lock, cpu)								\
+	({										\
+		int t;									\
+		if (static_branch_maybe(CONFIG_QPW_DEFAULT, &qpw_sl))			\
+			t = spin_trylock(per_cpu_ptr(lock.sl, cpu));			\
+		else									\
+			t = local_trylock(lock.ll);					\
+		t;									\
+	})
+
+#define qpw_trylock_irqsave(lock, flags, cpu)						\
+	({										\
+		int t;									\
+		if (static_branch_maybe(CONFIG_QPW_DEFAULT, &qpw_sl))			\
+			t = spin_trylock_irqsave(per_cpu_ptr(lock.sl, cpu), flags);	\
+		else									\
+			t = local_trylock_irqsave(lock.ll, flags);			\
+		t;									\
+	})
+
+#define qpw_unlock(lock, cpu)								\
+	do {										\
+		if (static_branch_maybe(CONFIG_QPW_DEFAULT, &qpw_sl)) {			\
+			spin_unlock(per_cpu_ptr(lock.sl, cpu));				\
+		} else {								\
+			local_unlock(lock.ll);						\
+		}									\
+	} while (0)
+
+#define qpw_unlock_irqrestore(lock, flags, cpu)						\
+	do {										\
+		if (static_branch_maybe(CONFIG_QPW_DEFAULT, &qpw_sl))			\
+			spin_unlock_irqrestore(per_cpu_ptr(lock.sl, cpu), flags);	\
+		else									\
+			local_unlock_irqrestore(lock.ll, flags);			\
+	} while (0)
+
+#define qpw_lockdep_assert_held(lock)							\
+	do {										\
+		if (static_branch_maybe(CONFIG_QPW_DEFAULT, &qpw_sl))			\
+			lockdep_assert_held(this_cpu_ptr(lock.sl));			\
+		else									\
+			lockdep_assert_held(this_cpu_ptr(lock.ll));			\
+	} while (0)
+
+#define queue_percpu_work_on(c, wq, qpw)						\
+	do {										\
+		int __c = c;								\
+		struct qpw_struct *__qpw = (qpw);					\
+		if (static_branch_maybe(CONFIG_QPW_DEFAULT, &qpw_sl)) {			\
+			WARN_ON((__c) != __qpw->cpu);					\
+			__qpw->work.func(&__qpw->work);					\
+		} else {								\
+			queue_work_on(__c, wq, &(__qpw)->work);				\
+		}									\
+	} while (0)
+
+/*
+ * Does nothing if QPW is set to use spinlock, as the task is already done at the
+ * time queue_percpu_work_on() returns.
+ */
+#define flush_percpu_work(qpw)								\
+	do {										\
+		struct qpw_struct *__qpw = (qpw);					\
+		if (!static_branch_maybe(CONFIG_QPW_DEFAULT, &qpw_sl)) {		\
+			flush_work(&__qpw->work);					\
+		}									\
+	} while (0)
+
+#define qpw_get_cpu(w)			container_of((w), struct qpw_struct, work)->cpu
+
+#define qpw_is_cpu_remote(cpu)		((cpu) != smp_processor_id())
+
+#define INIT_QPW(qpw, func, c)								\
+	do {										\
+		struct qpw_struct *__qpw = (qpw);					\
+		INIT_WORK(&__qpw->work, (func));					\
+		__qpw->cpu = (c);							\
+	} while (0)
+
+#endif /* CONFIG_QPW */
+#endif /* LINUX_QPW_H */

Index: slab/init/Kconfig
===================================================================

init/Kconfig +35 -0

@@ -747,6 +747,41 @@ config CPU_ISOLATION
 
 	  Say Y if unsure.
 
+config QPW
+	bool "Queue per-CPU Work"
+	depends on SMP || COMPILE_TEST
+	default n
+	help
+	  Allow changing the behavior on per-CPU resource sharing with cache,
+	  from the regular local_locks() + queue_work_on(remote_cpu) to using
+	  per-CPU spinlocks on both local and remote operations.
+
+	  This is useful to give user the option on reducing IPIs to CPUs, and
+	  thus reduce interruptions and context switches. On the other hand, it
+	  increases generated code and will use atomic operations if spinlocks
+	  are selected.
+
+	  If set, will use the default behavior set in QPW_DEFAULT unless boot
+	  parameter qpw is passed with a different behavior.
+
+	  If unset, will use the local_lock() + queue_work_on() strategy,
+	  regardless of the boot parameter or QPW_DEFAULT.
+
+	  Say N if unsure.
+
+config QPW_DEFAULT
+	bool "Use per-CPU spinlocks by default"
+	depends on QPW
+	default n
+	help
+	  If set, will use per-CPU spinlocks as default behavior for per-CPU
+	  remote operations.
+
+	  If unset, will use local_lock() + queue_work_on(cpu) as default
+	  behavior for remote operations.
+
+	  Say N if unsure
+
 source "kernel/rcu/Kconfig"
 
 config IKCONFIG

Index: slab/kernel/Makefile
===================================================================

kernel/Makefile +2 -0

@@ -140,6 +140,8 @@ obj-$(CONFIG_WATCH_QUEUE) += watch_queue
 obj-$(CONFIG_RESOURCE_KUNIT_TEST) += resource_kunit.o
 obj-$(CONFIG_SYSCTL_KUNIT_TEST) += sysctl-test.o
 
+obj-$(CONFIG_QPW) += qpw.o
+
 CFLAGS_kstack_erase.o += $(DISABLE_KSTACK_ERASE)
 CFLAGS_kstack_erase.o += $(call cc-option,-mgeneral-regs-only)
 obj-$(CONFIG_KSTACK_ERASE) += kstack_erase.o

Index: slab/kernel/qpw.c
===================================================================

kernel/qpw.c new file +26 -0

@@ -0,0 +1,26 @@
+// SPDX-License-Identifier: GPL-2.0
+#include "linux/export.h"
+#include <linux/sched.h>
+#include <linux/qpw.h>
+#include <linux/string.h>
+
+DEFINE_STATIC_KEY_MAYBE(CONFIG_QPW_DEFAULT, qpw_sl);
+EXPORT_SYMBOL(qpw_sl);
+
+static int __init qpw_setup(char *str)
+{
+	int opt;
+
+	if (!get_option(&str, &opt)) {
+		pr_warn("QPW: invalid qpw parameter: %s, ignoring.\n", str);
+		return 0;
+	}
+
+	if (opt)
+		static_branch_enable(&qpw_sl);
+	else
+		static_branch_disable(&qpw_sl);
+
+	return 0;
+}
+__setup("qpw=", qpw_setup);

Index: slab/Documentation/locking/qpwlocks.rst
===================================================================

Documentation/locking/qpwlocks.rst new file +32 -0

@@ -0,0 +1,63 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+=========
+QPW locks
+=========
+
+Some places in the kernel implement a parallel programming strategy
+consisting on local_locks() for most of the work, and some rare remote
+operations are scheduled on target cpu. This keeps cache bouncing low since
+cacheline tends to be mostly local, and avoids the cost of locks in non-RT
+kernels, even though the very few remote operations will be expensive due
+to scheduling overhead.
+
+On the other hand, for RT workloads this can represent a problem:
+scheduling work on remote cpu that are executing low latency tasks
+is undesired and can introduce unexpected deadline misses.
+
+QPW locks help to convert sites that use local_locks (for cpu local operations)
+and queue_work_on (for queueing work remotely, to be executed
+locally on the owner cpu of the lock) to QPW locks.
+
+The lock is declared qpw_lock_t type.
+The lock is initialized with qpw_lock_init.
+The lock is locked with qpw_lock (takes a lock and cpu as a parameter).
+The lock is unlocked with qpw_unlock (takes a lock and cpu as a parameter).
+
+The qpw_lock_irqsave function disables interrupts and saves current interrupt state,
+cpu as a parameter.
+
+For trylock variant, there is the qpw_trylock_t type, initialized with
+qpw_trylock_init. Then the corresponding qpw_trylock and
+qpw_trylock_irqsave.

MAINTAINERS F: Documentation/networking/device_drive · line 21295

Quoted 9 lines

===================================================================

MAINTAINERS +2 -0

@@ -21291,6 +21291,12 @@ F:	Documentation/networking/device_drive
 F:	drivers/bus/fsl-mc/
 F:	include/uapi/linux/fsl_mc.h
 
+QPW
+M:	Leonardo Bras <[email protected]>

Thanks for keeping that up :)
Could you please change this line to

+M: Leonardo Bras <[email protected]>

As I don't have access to Red Hat's mail anymore. The signoffs on each commit should be fine to keep :)

MAINTAINERS F: Documentation/networking/device_drive · line 21299

Quoted 5 lines

MAINTAINERS +4 -0

+S:	Supported
+F:	include/linux/qpw.h
+F:	kernel/qpw.c
+

Should we also add the Documentation file as well?

+F: Documentation/locking/qpwlocks.rst

MAINTAINERS F: Documentation/networking/device_drive

Quoted 6 lines

MAINTAINERS +6 -0

+The queue work related functions (analogous to queue_work_on and flush_work) are:
+queue_percpu_work_on and flush_percpu_work.
+
+The behaviour of the QPW functions is as follows:
+
+* !CONFIG_PREEMPT_RT and !CONFIG_QPW (or CONFIG_QPW and qpw=off kernel

I don't think PREEMPT_RT is needed here (maybe it was copied from the previous QPW version which was dependent on PREEMPT_RT?)

MAINTAINERS

Quoted 10 lines

MAINTAINERS +10 -0

+boot parameter):
+        - qpw_lock:                     local_lock
+        - qpw_lock_irqsave:             local_lock_irqsave
+        - qpw_trylock:                  local_trylock
+        - qpw_trylock_irqsave:          local_trylock_irqsave
+        - qpw_unlock:                   local_unlock
+        - queue_percpu_work_on:         queue_work_on
+        - flush_percpu_work:            flush_work
+
+* CONFIG_PREEMPT_RT or CONFIG_QPW (and CONFIG_QPW_DEFAULT or qpw=on kernel

Same here

MAINTAINERS

Quoted 13 lines

MAINTAINERS +11 -0

+boot parameter),
+        - qpw_lock:                     spin_lock
+        - qpw_lock_irqsave:             spin_lock_irqsave
+        - qpw_trylock:                  spin_trylock
+        - qpw_trylock_irqsave:          spin_trylock_irqsave
+        - qpw_unlock:                   spin_unlock
+        - queue_percpu_work_on:         executes work function on caller cpu
+        - flush_percpu_work:            empty
+
+qpw_get_cpu(work_struct), to be called from within qpw work function,
+returns the target cpu.

Other than that, LGTM!

Reviewed-by Leonardo Bras <[email protected]>

Thanks!
Leo

Hi Leonardo,

Quoted 13 lines · Leonardo Bras

MAINTAINERS F: Documentation/networking/device_drive · line 21295

Quoted 9 lines

===================================================================

MAINTAINERS +2 -0

@@ -21291,6 +21291,12 @@ F:	Documentation/networking/device_drive
 F:	drivers/bus/fsl-mc/
 F:	include/uapi/linux/fsl_mc.h
 
+QPW
+M:	Leonardo Bras <[email protected]>

Thanks for keeping that up :)
Could you please change this line to

Quoted 4 lines

+M: Leonardo Bras <[email protected]>

As I don't have access to Red Hat's mail anymore. The signoffs on each commit should be fine to keep :)

Done.

Quoted 9 lines

MAINTAINERS F: Documentation/networking/device_drive · line 21299

Quoted 5 lines

MAINTAINERS +4 -0

+S:	Supported
+F:	include/linux/qpw.h
+F:	kernel/qpw.c
+

Should we also add the Documentation file as well?

+F: Documentation/locking/qpwlocks.rst

Done.

Quoted 9 lines

MAINTAINERS F: Documentation/networking/device_drive

Quoted 6 lines

MAINTAINERS +6 -0

+The queue work related functions (analogous to queue_work_on and flush_work) are:
+queue_percpu_work_on and flush_percpu_work.
+
+The behaviour of the QPW functions is as follows:
+
+* !CONFIG_PREEMPT_RT and !CONFIG_QPW (or CONFIG_QPW and qpw=off kernel

I don't think PREEMPT_RT is needed here (maybe it was copied from the previous QPW version which was dependent on PREEMPT_RT?)

Ah, OK, my bad. Well, shouldnt CONFIG_PREEMPT_RT select CONFIG_QPW and CONFIG_QPW_DEFAULT=y ?

Make use of the new qpw_{un,}lock*() and queue_percpu_work_on() interface to improve performance & latency on PREEMPT_RT kernels.

For functions that may be scheduled in a different cpu, replace local_{un,}lock*() by qpw_{un,}lock*(), and replace schedule_work_on() by queue_percpu_work_on(). The same happens for flush_work() and flush_percpu_work().

The change requires allocation of qpw_structs instead of a work_structs, and changing parameters of a few functions to include the cpu parameter.

This should bring no relevant performance impact on non-RT kernels:

For functions that may be scheduled in a different cpu, the local_*lock's this_cpu_ptr() becomes a per_cpu_ptr(smp_processor_id()).

Signed-off-by Leonardo Bras <[email protected]>

Signed-off-by Marcelo Tosatti <[email protected]>

---
 mm/internal.h   |    4 +-
 mm/mlock.c      |   71 ++++++++++++++++++++++++++++++++------------
 mm/page_alloc.c |    2 -
 mm/swap.c       |   90 +++++++++++++++++++++++++++++++-------------------------
 4 files changed, 108 insertions(+), 59 deletions(-)

Index: slab/mm/mlock.c
===================================================================

mm/mlock.c +15 -9

@@ -25,17 +25,16 @@
 #include <linux/memcontrol.h>
 #include <linux/mm_inline.h>
 #include <linux/secretmem.h>
+#include <linux/qpw.h>
 
 #include "internal.h"
 
 struct mlock_fbatch {
-	local_lock_t lock;
+	qpw_lock_t lock;
 	struct folio_batch fbatch;
 };
 
-static DEFINE_PER_CPU(struct mlock_fbatch, mlock_fbatch) = {
-	.lock = INIT_LOCAL_LOCK(lock),
-};
+static DEFINE_PER_CPU(struct mlock_fbatch, mlock_fbatch);
 
 bool can_do_mlock(void)
 {

@@ -209,18 +208,25 @@ static void mlock_folio_batch(struct fol
 	folios_put(fbatch);
 }
 
-void mlock_drain_local(void)
+void mlock_drain_cpu(int cpu)
 {
 	struct folio_batch *fbatch;
 
-	local_lock(&mlock_fbatch.lock);
-	fbatch = this_cpu_ptr(&mlock_fbatch.fbatch);
+	qpw_lock(&mlock_fbatch.lock, cpu);
+	fbatch = per_cpu_ptr(&mlock_fbatch.fbatch, cpu);
 	if (folio_batch_count(fbatch))
 		mlock_folio_batch(fbatch);
-	local_unlock(&mlock_fbatch.lock);
+	qpw_unlock(&mlock_fbatch.lock, cpu);
 }
 
-void mlock_drain_remote(int cpu)
+void mlock_drain_local(void)
+{
+	migrate_disable();
+	mlock_drain_cpu(smp_processor_id());
+	migrate_enable();
+}
+
+void mlock_drain_offline(int cpu)
 {
 	struct folio_batch *fbatch;
 

mm/mlock.c +10 -3

+	qpw_lock(&mlock_fbatch.lock, cpu);
+	fbatch = per_cpu_ptr(&mlock_fbatch.fbatch, cpu);
 
 	if (!folio_test_set_mlocked(folio)) {
 		int nr_pages = folio_nr_pages(folio);

@@ -257,7 +266,8 @@ void mlock_folio(struct folio *folio)
 	if (!folio_batch_add(fbatch, mlock_lru(folio)) ||
 	    !folio_may_be_lru_cached(folio) || lru_cache_disabled())
 		mlock_folio_batch(fbatch);
-	local_unlock(&mlock_fbatch.lock);
+	qpw_unlock(&mlock_fbatch.lock, cpu);
+	migrate_enable();
 }
 
 /**

@@ -268,9 +278,13 @@ void mlock_new_folio(struct folio *folio
 {
 	struct folio_batch *fbatch;
 	int nr_pages = folio_nr_pages(folio);
+	int cpu;
+
+	migrate_disable();
+	cpu = smp_processor_id();
+	qpw_lock(&mlock_fbatch.lock, cpu);
 
-	local_lock(&mlock_fbatch.lock);
-	fbatch = this_cpu_ptr(&mlock_fbatch.fbatch);
+	fbatch = per_cpu_ptr(&mlock_fbatch.fbatch, cpu);
 	folio_set_mlocked(folio);
 
 	zone_stat_mod_folio(folio, NR_MLOCK, nr_pages);

mm/mlock.c +23 -3

 }
 
 /**

@@ -290,9 +305,13 @@ void mlock_new_folio(struct folio *folio
 void munlock_folio(struct folio *folio)
 {
 	struct folio_batch *fbatch;
+	int cpu;
 
-	local_lock(&mlock_fbatch.lock);
-	fbatch = this_cpu_ptr(&mlock_fbatch.fbatch);
+	migrate_disable();
+	cpu = smp_processor_id();
+	qpw_lock(&mlock_fbatch.lock, cpu);
+
+	fbatch = per_cpu_ptr(&mlock_fbatch.fbatch, cpu);
 	/*
 	 * folio_test_clear_mlocked(folio) must be left to __munlock_folio(),
 	 * which will check whether the folio is multiply mlocked.

@@ -301,7 +320,8 @@ void munlock_folio(struct folio *folio)
 	if (!folio_batch_add(fbatch, folio) ||
 	    !folio_may_be_lru_cached(folio) || lru_cache_disabled())
 		mlock_folio_batch(fbatch);
-	local_unlock(&mlock_fbatch.lock);
+	qpw_unlock(&mlock_fbatch.lock, cpu);
+	migrate_enable();
 }
 
 static inline unsigned int folio_mlock_step(struct folio *folio,

@@ -823,3 +843,18 @@ void user_shm_unlock(size_t size, struct
 	spin_unlock(&shmlock_user_lock);
 	put_ucounts(ucounts);
 }
+
+int __init mlock_init(void)
+{
+	unsigned int cpu;
+
+	for_each_possible_cpu(cpu) {
+		struct mlock_fbatch *fbatch = &per_cpu(mlock_fbatch, cpu);
+
+		qpw_lock_init(&fbatch->lock);
+	}
+
+	return 0;
+}
+
+module_init(mlock_init);

Index: slab/mm/swap.c
===================================================================

mm/swap.c +4 -7

@@ -35,7 +35,7 @@
 #include <linux/uio.h>
 #include <linux/hugetlb.h>
 #include <linux/page_idle.h>
-#include <linux/local_lock.h>
+#include <linux/qpw.h>
 #include <linux/buffer_head.h>
 
 #include "internal.h"

@@ -52,7 +52,7 @@ struct cpu_fbatches {
 	 * The following folio batches are grouped together because they are protected
 	 * by disabling preemption (and interrupts remain enabled).
 	 */
-	local_lock_t lock;
+	qpw_lock_t lock;
 	struct folio_batch lru_add;
 	struct folio_batch lru_deactivate_file;
 	struct folio_batch lru_deactivate;

@@ -61,14 +61,11 @@ struct cpu_fbatches {
 	struct folio_batch lru_activate;
 #endif
 	/* Protecting the following batches which require disabling interrupts */
-	local_lock_t lock_irq;
+	qpw_lock_t lock_irq;
 	struct folio_batch lru_move_tail;
 };
 
-static DEFINE_PER_CPU(struct cpu_fbatches, cpu_fbatches) = {
-	.lock = INIT_LOCAL_LOCK(lock),
-	.lock_irq = INIT_LOCAL_LOCK(lock_irq),
-};
+static DEFINE_PER_CPU(struct cpu_fbatches, cpu_fbatches);
 
 static void __page_cache_release(struct folio *folio, struct lruvec **lruvecp,
 		unsigned long *flagsp)

mm/swap.c +7 -6

+	cpu = smp_processor_id();
 	if (disable_irq)
-		local_lock_irqsave(&cpu_fbatches.lock_irq, flags);
+		qpw_lock_irqsave(&cpu_fbatches.lock_irq, flags, cpu);
 	else
-		local_lock(&cpu_fbatches.lock);
+		qpw_lock(&cpu_fbatches.lock, cpu);
 
-	if (!folio_batch_add(this_cpu_ptr(fbatch), folio) ||
+	if (!folio_batch_add(per_cpu_ptr(fbatch, cpu), folio) ||
 			!folio_may_be_lru_cached(folio) || lru_cache_disabled())
-		folio_batch_move_lru(this_cpu_ptr(fbatch), move_fn);
+		folio_batch_move_lru(per_cpu_ptr(fbatch, cpu), move_fn);
 
 	if (disable_irq)
-		local_unlock_irqrestore(&cpu_fbatches.lock_irq, flags);
+		qpw_unlock_irqrestore(&cpu_fbatches.lock_irq, flags, cpu);
 	else
-		local_unlock(&cpu_fbatches.lock);
+		qpw_unlock(&cpu_fbatches.lock, cpu);
 }
 
 #define folio_batch_add_and_move(folio, op)		\

mm/swap.c +6 -5

+	int cpu = smp_processor_id();
 
-	local_lock(&cpu_fbatches.lock);
-	fbatch = this_cpu_ptr(&cpu_fbatches.lru_add);
+	qpw_lock(&cpu_fbatches.lock, cpu);
+	fbatch = per_cpu_ptr(&cpu_fbatches.lru_add, cpu);
 
 	/*
 	 * Search backwards on the optimistic assumption that the folio being

@@ -381,7 +381,7 @@ static void __lru_cache_activate_folio(s
 		}
 	}
 
-	local_unlock(&cpu_fbatches.lock);
+	qpw_unlock(&cpu_fbatches.lock, cpu);
 }
 
 #ifdef CONFIG_LRU_GEN

@@ -653,9 +653,9 @@ void lru_add_drain_cpu(int cpu)
 		unsigned long flags;
 
 		/* No harm done if a racing interrupt already did this */
-		local_lock_irqsave(&cpu_fbatches.lock_irq, flags);
+		qpw_lock_irqsave(&cpu_fbatches.lock_irq, flags, cpu);
 		folio_batch_move_lru(fbatch, lru_move_tail);
-		local_unlock_irqrestore(&cpu_fbatches.lock_irq, flags);
+		qpw_unlock_irqrestore(&cpu_fbatches.lock_irq, flags, cpu);
 	}
 
 	fbatch = &fbatches->lru_deactivate_file;

mm/swap.c +7 -5

+	int cpu = smp_processor_id();
+
+	qpw_lock(&cpu_fbatches.lock, cpu);
+	lru_add_drain_cpu(cpu);
+	qpw_unlock(&cpu_fbatches.lock, cpu);
+	mlock_drain_cpu(cpu);
 }
 
 /*

@@ -745,30 +747,32 @@ void lru_add_drain(void)
  * the same cpu. It shouldn't be a problem in !SMP case since
  * the core is only one and the locks will disable preemption.
  */
-static void lru_add_mm_drain(void)
+static void lru_add_mm_drain(int cpu)
 {
-	local_lock(&cpu_fbatches.lock);
-	lru_add_drain_cpu(smp_processor_id());
-	local_unlock(&cpu_fbatches.lock);
-	mlock_drain_local();

mm/swap.c +13 -9

+	int cpu = smp_processor_id();
+
+	qpw_lock(&cpu_fbatches.lock, cpu);
+	lru_add_drain_cpu(cpu);
 	drain_local_pages(zone);
-	local_unlock(&cpu_fbatches.lock);
-	mlock_drain_local();
+	qpw_unlock(&cpu_fbatches.lock, cpu);
+	mlock_drain_cpu(cpu);
 }
 
 #ifdef CONFIG_SMP
 
-static DEFINE_PER_CPU(struct work_struct, lru_add_drain_work);
+static DEFINE_PER_CPU(struct qpw_struct, lru_add_drain_qpw);
 
-static void lru_add_drain_per_cpu(struct work_struct *dummy)
+static void lru_add_drain_per_cpu(struct work_struct *w)
 {
-	lru_add_mm_drain();
+	lru_add_mm_drain(qpw_get_cpu(w));
 }
 
 static DEFINE_PER_CPU(struct work_struct, bh_add_drain_work);

@@ -883,12 +887,12 @@ static inline void __lru_add_drain_all(b
 	cpumask_clear(&has_mm_work);
 	cpumask_clear(&has_bh_work);
 	for_each_online_cpu(cpu) {
-		struct work_struct *mm_work = &per_cpu(lru_add_drain_work, cpu);
+		struct qpw_struct *mm_qpw = &per_cpu(lru_add_drain_qpw, cpu);
 		struct work_struct *bh_work = &per_cpu(bh_add_drain_work, cpu);
 
 		if (cpu_needs_mm_drain(cpu)) {
-			INIT_WORK(mm_work, lru_add_drain_per_cpu);
-			queue_work_on(cpu, mm_percpu_wq, mm_work);
+			INIT_QPW(mm_qpw, lru_add_drain_per_cpu, cpu);
+			queue_percpu_work_on(cpu, mm_percpu_wq, mm_qpw);
 			__cpumask_set_cpu(cpu, &has_mm_work);
 		}
 

@@ -900,7 +904,7 @@ static inline void __lru_add_drain_all(b
 	}
 
 	for_each_cpu(cpu, &has_mm_work)
-		flush_work(&per_cpu(lru_add_drain_work, cpu));
+		flush_percpu_work(&per_cpu(lru_add_drain_qpw, cpu));
 
 	for_each_cpu(cpu, &has_bh_work)
 		flush_work(&per_cpu(bh_add_drain_work, cpu));

mm/swap.c +9 -0

+	lru_add_mm_drain(smp_processor_id());
 	invalidate_bh_lrus_cpu();
 #endif
 }

@@ -1124,6 +1128,7 @@ static const struct ctl_table swap_sysct
 void __init swap_setup(void)
 {
 	unsigned long megs = PAGES_TO_MB(totalram_pages());
+	unsigned int cpu;
 
 	/* Use a smaller cluster for small-memory machines */
 	if (megs < 16)

@@ -1136,4 +1141,11 @@ void __init swap_setup(void)
 	 */
 
 	register_sysctl_init("vm", swap_sysctl_table);
+
+	for_each_possible_cpu(cpu) {
+		struct cpu_fbatches *fbatches = &per_cpu(cpu_fbatches, cpu);
+
+		qpw_lock_init(&fbatches->lock);
+		qpw_lock_init(&fbatches->lock_irq);
+	}
 }

Index: slab/mm/internal.h
===================================================================

mm/internal.h +3 -1

@@ -1061,10 +1061,12 @@ static inline void munlock_vma_folio(str
 		munlock_folio(folio);
 }
 
+int __init mlock_init(void);
 void mlock_new_folio(struct folio *folio);
 bool need_mlock_drain(int cpu);
 void mlock_drain_local(void);
-void mlock_drain_remote(int cpu);
+void mlock_drain_cpu(int cpu);
+void mlock_drain_offline(int cpu);
 
 extern pmd_t maybe_pmd_mkwrite(pmd_t pmd, struct vm_area_struct *vma);
 

Index: slab/mm/page_alloc.c
===================================================================

Review

Quoted 1 line

(quoted diff) +1 -0

+	cpu = smp_processor_id();

Wondering if for these cases it would make sense to have something like:

qpw_get_local_cpu() and
qpw_put_local_cpu()

so we could encapsulate these migrate_{en,dis}able() and the smp_processor_id().

Or even,

int qpw_local_lock() {
	migrate_disable();
	cpu = smp_processor_id();
	qpw_lock(..., cpu);

	return cpu;
}

and

qpw_local_unlock(cpu){
	qpw_unlock(...,cpu);
	migrate_enable();
} 

so it's more direct to convert the local-only cases.

What do you think?

Review

Quoted 3 lines

{

(quoted diff) +0 -2

-	local_lock(&cpu_fbatches.lock);
-	lru_add_drain_cpu(smp_processor_id());

and here ?

mm/swap.c void lru_cache_disable(void) · line 953

Quoted 5 lines

mm/swap.c +0 -1

@@ -950,7 +954,7 @@ void lru_cache_disable(void)
 #ifdef CONFIG_SMP
 	__lru_add_drain_all(true);
 #else
-	lru_add_mm_drain();

and here, I wonder

mm/page_alloc.c static int page_alloc_cpu_dead(unsigned · line 6257

Quoted 5 lines

mm/page_alloc.c +0 -0

 	drain_pages(cpu);
 
 	/*

TBH, I am still trying to understand if we need the migrate_{en,dis}able():
- There is a data dependency beween cpu being filled and being used.
- If we get the cpu, and then migrate to a different cpu, the operation will still be executed with the data from that starting cpu

- But maybe the compiler tries to optize this because the processor number can be on a register and of easy access, which would break this.

Maybe a READ_ONCE() on smp_processor_id() should suffice?

Other than that, all the conversions done look correct.

That being said, I understand very little about mm code, so let's hope we get proper feedback from those who do :)

Thanks!
Leo

Quoted 22 lines · Leonardo Bras

Review

Quoted 1 line

(quoted diff) +1 -0

+	cpu = smp_processor_id();

Wondering if for these cases it would make sense to have something like:

qpw_get_local_cpu() and
qpw_put_local_cpu()

so we could encapsulate these migrate_{en,dis}able() and the smp_processor_id().

Or even,

int qpw_local_lock() {
	migrate_disable();
	cpu = smp_processor_id();
	qpw_lock(..., cpu);

	return cpu;
}

and

Quoted 8 lines

qpw_local_unlock(cpu){
	qpw_unlock(...,cpu);
	migrate_enable();
} 

so it's more direct to convert the local-only cases.

What do you think?

Switched to local_qpw_lock variants.

Quoted 5 lines

Review

Quoted 3 lines

{

(quoted diff) +0 -2

-	local_lock(&cpu_fbatches.lock);
-	lru_add_drain_cpu(smp_processor_id());

and here ?

Fixed lack of migrate_disable/migrate_enable, thanks!

Quoted 7 lines

mm/swap.c void lru_cache_disable(void) · line 953

Quoted 5 lines

mm/swap.c +0 -1

@@ -950,7 +954,7 @@ void lru_cache_disable(void)
 #ifdef CONFIG_SMP
 	__lru_add_drain_all(true);
 #else
-	lru_add_mm_drain();

and here, I wonder

This is !CONFIG_SMP, so smp_processor_id is always 0.

Quoted 6 lines

Quoted 5 lines

mm/page_alloc.c +0 -0

 	drain_pages(cpu);
 
 	/*

Quoted 4 lines

TBH, I am still trying to understand if we need the migrate_{en,dis}able():
- There is a data dependency beween cpu being filled and being used.
- If we get the cpu, and then migrate to a different cpu, the operation will still be executed with the data from that starting cpu

Yes, but on a remote CPU. What prevents the original CPU from accessing its per-CPU local data, therefore racing with the code executing on the remote CPU.

Quoted 15 lines

- But maybe the compiler tries to optize this because the processor number can be on a register and of easy access, which would break this.

Maybe a READ_ONCE() on smp_processor_id() should suffice?

Other than that, all the conversions done look correct.

That being said, I understand very little about mm code, so let's hope we get proper feedback from those who do :)

Thanks!
Leo

Make use of the new qpw_{un,}lock*() and queue_percpu_work_on() interface to improve performance & latency on PREEMPT_RT kernels.

For functions that may be scheduled in a different cpu, replace local_{un,}lock*() by qpw_{un,}lock*(), and replace schedule_work_on() by queue_percpu_work_on(). The same happens for flush_work() and flush_percpu_work().

This change requires allocation of qpw_structs instead of a work_structs, and changing parameters of a few functions to include the cpu parameter.

This should bring no relevant performance impact on non-RT kernels:

For functions that may be scheduled in a different cpu, the local_*lock's this_cpu_ptr() becomes a per_cpu_ptr(smp_processor_id()).

Signed-off-by Leonardo Bras <[email protected]>

Signed-off-by Marcelo Tosatti <[email protected]>

---
 mm/slub.c |  218 ++++++++++++++++++++++++++++++++++++++++----------------------
 1 file changed, 142 insertions(+), 76 deletions(-)

Index: slab/mm/slub.c
===================================================================

mm/slub.c +81 -49

@@ -49,6 +49,7 @@
 #include <linux/irq_work.h>
 #include <linux/kprobes.h>
 #include <linux/debugfs.h>
+#include <linux/qpw.h>
 #include <trace/events/kmem.h>
 
 #include "internal.h"

@@ -128,7 +129,7 @@
  *   For debug caches, all allocations are forced to go through a list_lock
  *   protected region to serialize against concurrent validation.
  *
- *   cpu_sheaves->lock (local_trylock)
+ *   cpu_sheaves->lock (qpw_trylock)
  *
  *   This lock protects fastpath operations on the percpu sheaves. On !RT it
  *   only disables preemption and does no atomic operations. As long as the main

@@ -156,7 +157,7 @@
  *   Interrupts are disabled as part of list_lock or barn lock operations, or
  *   around the slab_lock operation, in order to make the slab allocator safe
  *   to use in the context of an irq.
- *   Preemption is disabled as part of local_trylock operations.
+ *   Preemption is disabled as part of qpw_trylock operations.
  *   kmalloc_nolock() and kfree_nolock() are safe in NMI context but see
  *   their limitations.
  *

@@ -417,7 +418,7 @@ struct slab_sheaf {
 };
 
 struct slub_percpu_sheaves {
-	local_trylock_t lock;
+	qpw_trylock_t lock;
 	struct slab_sheaf *main; /* never NULL when unlocked */
 	struct slab_sheaf *spare; /* empty or full, may be NULL */
 	struct slab_sheaf *rcu_free; /* for batching kfree_rcu() */

@@ -479,7 +480,7 @@ static nodemask_t slab_nodes;
 static struct workqueue_struct *flushwq;
 
 struct slub_flush_work {
-	struct work_struct work;
+	struct qpw_struct qpw;
 	struct kmem_cache *s;
 	bool skip;
 };

@@ -2826,7 +2827,7 @@ static void __kmem_cache_free_bulk(struc
  *
  * returns true if at least partially flushed
  */
-static bool sheaf_flush_main(struct kmem_cache *s)
+static bool sheaf_flush_main(struct kmem_cache *s, int cpu)
 {
 	struct slub_percpu_sheaves *pcs;
 	unsigned int batch, remaining;

@@ -2835,10 +2836,10 @@ static bool sheaf_flush_main(struct kmem
 	bool ret = false;
 
 next_batch:
-	if (!local_trylock(&s->cpu_sheaves->lock))
+	if (!qpw_trylock(&s->cpu_sheaves->lock, cpu))
 		return ret;
 
-	pcs = this_cpu_ptr(s->cpu_sheaves);
+	pcs = per_cpu_ptr(s->cpu_sheaves, cpu);
 	sheaf = pcs->main;
 
 	batch = min(PCS_BATCH_MAX, sheaf->size);

@@ -2848,7 +2849,7 @@ next_batch:
 
 	remaining = sheaf->size;
 
-	local_unlock(&s->cpu_sheaves->lock);
+	qpw_unlock(&s->cpu_sheaves->lock, cpu);
 
 	__kmem_cache_free_bulk(s, batch, &objects[0]);
 

@@ -2932,13 +2933,13 @@ static void rcu_free_sheaf_nobarn(struct
  * flushing operations are rare so let's keep it simple and flush to slabs
  * directly, skipping the barn
  */
-static void pcs_flush_all(struct kmem_cache *s)
+static void pcs_flush_all(struct kmem_cache *s, int cpu)
 {
 	struct slub_percpu_sheaves *pcs;
 	struct slab_sheaf *spare, *rcu_free;
 
-	local_lock(&s->cpu_sheaves->lock);
-	pcs = this_cpu_ptr(s->cpu_sheaves);
+	qpw_lock(&s->cpu_sheaves->lock, cpu);
+	pcs = per_cpu_ptr(s->cpu_sheaves, cpu);
 
 	spare = pcs->spare;
 	pcs->spare = NULL;

@@ -2946,7 +2947,7 @@ static void pcs_flush_all(struct kmem_ca
 	rcu_free = pcs->rcu_free;
 	pcs->rcu_free = NULL;
 
-	local_unlock(&s->cpu_sheaves->lock);
+	qpw_unlock(&s->cpu_sheaves->lock, cpu);
 
 	if (spare) {
 		sheaf_flush_unused(s, spare);

@@ -2956,7 +2957,7 @@ static void pcs_flush_all(struct kmem_ca
 	if (rcu_free)
 		call_rcu(&rcu_free->rcu_head, rcu_free_sheaf_nobarn);
 
-	sheaf_flush_main(s);
+	sheaf_flush_main(s, cpu);
 }
 
 static void __pcs_flush_all_cpu(struct kmem_cache *s, unsigned int cpu)

@@ -3881,13 +3882,13 @@ static void flush_cpu_sheaves(struct wor
 {
 	struct kmem_cache *s;
 	struct slub_flush_work *sfw;
+	int cpu = qpw_get_cpu(w);
 
-	sfw = container_of(w, struct slub_flush_work, work);
-
+	sfw = &per_cpu(slub_flush, cpu);
 	s = sfw->s;
 
 	if (cache_has_sheaves(s))
-		pcs_flush_all(s);
+		pcs_flush_all(s, cpu);
 }
 
 static void flush_all_cpus_locked(struct kmem_cache *s)

@@ -3904,17 +3905,17 @@ static void flush_all_cpus_locked(struct
 			sfw->skip = true;
 			continue;
 		}
-		INIT_WORK(&sfw->work, flush_cpu_sheaves);
+		INIT_QPW(&sfw->qpw, flush_cpu_sheaves, cpu);
 		sfw->skip = false;
 		sfw->s = s;
-		queue_work_on(cpu, flushwq, &sfw->work);
+		queue_percpu_work_on(cpu, flushwq, &sfw->qpw);
 	}
 
 	for_each_online_cpu(cpu) {
 		sfw = &per_cpu(slub_flush, cpu);
 		if (sfw->skip)
 			continue;
-		flush_work(&sfw->work);
+		flush_percpu_work(&sfw->qpw);
 	}
 
 	mutex_unlock(&flush_lock);

@@ -3933,17 +3934,18 @@ static void flush_rcu_sheaf(struct work_
 	struct slab_sheaf *rcu_free;
 	struct slub_flush_work *sfw;
 	struct kmem_cache *s;
+	int cpu = qpw_get_cpu(w);
 
-	sfw = container_of(w, struct slub_flush_work, work);
+	sfw = &per_cpu(slub_flush, cpu);
 	s = sfw->s;
 
-	local_lock(&s->cpu_sheaves->lock);
-	pcs = this_cpu_ptr(s->cpu_sheaves);
+	qpw_lock(&s->cpu_sheaves->lock, cpu);
+	pcs = per_cpu_ptr(s->cpu_sheaves, cpu);
 
 	rcu_free = pcs->rcu_free;
 	pcs->rcu_free = NULL;
 
-	local_unlock(&s->cpu_sheaves->lock);
+	qpw_unlock(&s->cpu_sheaves->lock, cpu);
 
 	if (rcu_free)
 		call_rcu(&rcu_free->rcu_head, rcu_free_sheaf_nobarn);

@@ -3968,14 +3970,14 @@ void flush_rcu_sheaves_on_cache(struct k
 		 * sure the __kfree_rcu_sheaf() finished its call_rcu()
 		 */
 
-		INIT_WORK(&sfw->work, flush_rcu_sheaf);
+		INIT_QPW(&sfw->qpw, flush_rcu_sheaf, cpu);
 		sfw->s = s;
-		queue_work_on(cpu, flushwq, &sfw->work);
+		queue_percpu_work_on(cpu, flushwq, &sfw->qpw);
 	}
 
 	for_each_online_cpu(cpu) {
 		sfw = &per_cpu(slub_flush, cpu);
-		flush_work(&sfw->work);
+		flush_percpu_work(&sfw->qpw);
 	}
 
 	mutex_unlock(&flush_lock);

@@ -4472,22 +4474,24 @@ bool slab_post_alloc_hook(struct kmem_ca
  *
  * Must be called with the cpu_sheaves local lock locked. If successful, returns
  * the pcs pointer and the local lock locked (possibly on a different cpu than
- * initially called). If not successful, returns NULL and the local lock
- * unlocked.
+ * initially called), and migration disabled. If not successful, returns NULL
+ * and the local lock unlocked, with migration enabled.
  */
 static struct slub_percpu_sheaves *
-__pcs_replace_empty_main(struct kmem_cache *s, struct slub_percpu_sheaves *pcs, gfp_t gfp)
+__pcs_replace_empty_main(struct kmem_cache *s, struct slub_percpu_sheaves *pcs, gfp_t gfp,
+			 int *cpu)
 {
 	struct slab_sheaf *empty = NULL;
 	struct slab_sheaf *full;
 	struct node_barn *barn;
 	bool can_alloc;
 
-	lockdep_assert_held(this_cpu_ptr(&s->cpu_sheaves->lock));
+	qpw_lockdep_assert_held(&s->cpu_sheaves->lock);
 
 	/* Bootstrap or debug cache, back off */
 	if (unlikely(!cache_has_sheaves(s))) {
-		local_unlock(&s->cpu_sheaves->lock);
+		qpw_unlock(&s->cpu_sheaves->lock, *cpu);
+		migrate_enable();
 		return NULL;
 	}
 

@@ -4498,7 +4502,8 @@ __pcs_replace_empty_main(struct kmem_cac
 
 	barn = get_barn(s);
 	if (!barn) {
-		local_unlock(&s->cpu_sheaves->lock);
+		qpw_unlock(&s->cpu_sheaves->lock, *cpu);
+		migrate_enable();
 		return NULL;
 	}
 

@@ -4524,7 +4529,8 @@ __pcs_replace_empty_main(struct kmem_cac
 		}
 	}
 
-	local_unlock(&s->cpu_sheaves->lock);
+	qpw_unlock(&s->cpu_sheaves->lock, *cpu);
+	migrate_enable();
 
 	if (!can_alloc)
 		return NULL;

@@ -4550,7 +4556,9 @@ __pcs_replace_empty_main(struct kmem_cac
 	 * we can reach here only when gfpflags_allow_blocking
 	 * so this must not be an irq
 	 */
-	local_lock(&s->cpu_sheaves->lock);
+	migrate_disable();
+	*cpu = smp_processor_id();
+	qpw_lock(&s->cpu_sheaves->lock, *cpu);
 	pcs = this_cpu_ptr(s->cpu_sheaves);
 
 	/*

@@ -4593,6 +4601,7 @@ void *alloc_from_pcs(struct kmem_cache *
 	struct slub_percpu_sheaves *pcs;
 	bool node_requested;
 	void *object;
+	int cpu;
 
 #ifdef CONFIG_NUMA
 	if (static_branch_unlikely(&strict_numa) &&

@@ -4627,13 +4636,17 @@ void *alloc_from_pcs(struct kmem_cache *
 		return NULL;
 	}
 
-	if (!local_trylock(&s->cpu_sheaves->lock))
+	migrate_disable();
+	cpu = smp_processor_id();
+	if (!qpw_trylock(&s->cpu_sheaves->lock, cpu)) {
+		migrate_enable();
 		return NULL;
+	}
 
 	pcs = this_cpu_ptr(s->cpu_sheaves);
 
 	if (unlikely(pcs->main->size == 0)) {
-		pcs = __pcs_replace_empty_main(s, pcs, gfp);
+		pcs = __pcs_replace_empty_main(s, pcs, gfp, &cpu);
 		if (unlikely(!pcs))
 			return NULL;
 	}

@@ -4647,7 +4660,8 @@ void *alloc_from_pcs(struct kmem_cache *
 		 * the current allocation or previous freeing process.
 		 */
 		if (page_to_nid(virt_to_page(object)) != node) {
-			local_unlock(&s->cpu_sheaves->lock);
+			qpw_unlock(&s->cpu_sheaves->lock, cpu);
+			migrate_enable();
 			stat(s, ALLOC_NODE_MISMATCH);
 			return NULL;
 		}

@@ -4655,7 +4669,8 @@ void *alloc_from_pcs(struct kmem_cache *
 
 	pcs->main->size--;
 
-	local_unlock(&s->cpu_sheaves->lock);
+	qpw_unlock(&s->cpu_sheaves->lock, cpu);
+	migrate_enable();
 
 	stat(s, ALLOC_FASTPATH);
 

@@ -4670,10 +4685,15 @@ unsigned int alloc_from_pcs_bulk(struct
 	struct slab_sheaf *main;
 	unsigned int allocated = 0;
 	unsigned int batch;
+	int cpu;
 
 next_batch:
-	if (!local_trylock(&s->cpu_sheaves->lock))
+	migrate_disable();
+	cpu = smp_processor_id();
+	if (!qpw_trylock(&s->cpu_sheaves->lock, cpu)) {
+		migrate_enable();
 		return allocated;
+	}
 
 	pcs = this_cpu_ptr(s->cpu_sheaves);
 

@@ -4683,7 +4703,8 @@ next_batch:
 		struct node_barn *barn;
 
 		if (unlikely(!cache_has_sheaves(s))) {
-			local_unlock(&s->cpu_sheaves->lock);
+			qpw_unlock(&s->cpu_sheaves->lock, cpu);
+			migrate_enable();
 			return allocated;
 		}
 

@@ -4694,7 +4715,8 @@ next_batch:
 
 		barn = get_barn(s);
 		if (!barn) {
-			local_unlock(&s->cpu_sheaves->lock);
+			qpw_unlock(&s->cpu_sheaves->lock, cpu);
+			migrate_enable();
 			return allocated;
 		}
 

@@ -4709,7 +4731,8 @@ next_batch:
 
 		stat(s, BARN_GET_FAIL);
 
-		local_unlock(&s->cpu_sheaves->lock);
+		qpw_unlock(&s->cpu_sheaves->lock, cpu);
+		migrate_enable();
 
 		/*
 		 * Once full sheaves in barn are depleted, let the bulk

@@ -4727,7 +4750,8 @@ do_alloc:
 	main->size -= batch;
 	memcpy(p, main->objects + main->size, batch * sizeof(void *));
 
-	local_unlock(&s->cpu_sheaves->lock);
+	qpw_unlock(&s->cpu_sheaves->lock, cpu);
+	migrate_enable();
 
 	stat_add(s, ALLOC_FASTPATH, batch);
 

@@ -4877,6 +4901,7 @@ kmem_cache_prefill_sheaf(struct kmem_cac
 	struct slub_percpu_sheaves *pcs;
 	struct slab_sheaf *sheaf = NULL;
 	struct node_barn *barn;
+	int cpu;
 
 	if (unlikely(!size))
 		return NULL;

@@ -4906,7 +4931,9 @@ kmem_cache_prefill_sheaf(struct kmem_cac
 		return sheaf;
 	}
 
-	local_lock(&s->cpu_sheaves->lock);
+	migrate_disable();
+	cpu = smp_processor_id();
+	qpw_lock(&s->cpu_sheaves->lock, cpu);
 	pcs = this_cpu_ptr(s->cpu_sheaves);
 
 	if (pcs->spare) {

@@ -4925,7 +4952,8 @@ kmem_cache_prefill_sheaf(struct kmem_cac
 			stat(s, BARN_GET_FAIL);
 	}
 
-	local_unlock(&s->cpu_sheaves->lock);
+	qpw_unlock(&s->cpu_sheaves->lock, cpu);
+	migrate_enable();
 
 
 	if (!sheaf)

@@ -4961,6 +4989,7 @@ void kmem_cache_return_sheaf(struct kmem
 {
 	struct slub_percpu_sheaves *pcs;
 	struct node_barn *barn;
+	int cpu;
 
 	if (unlikely((sheaf->capacity != s->sheaf_capacity)
 		     || sheaf->pfmemalloc)) {

@@ -4969,7 +4998,9 @@ void kmem_cache_return_sheaf(struct kmem
 		return;
 	}
 
-	local_lock(&s->cpu_sheaves->lock);
+	migrate_disable();
+	cpu = smp_processor_id();
+	qpw_lock(&s->cpu_sheaves->lock, cpu);
 	pcs = this_cpu_ptr(s->cpu_sheaves);
 	barn = get_barn(s);
 

@@ -4979,7 +5010,8 @@ void kmem_cache_return_sheaf(struct kmem
 		stat(s, SHEAF_RETURN_FAST);
 	}
 
-	local_unlock(&s->cpu_sheaves->lock);
+	qpw_unlock(&s->cpu_sheaves->lock, cpu);
+	migrate_enable();
 
 	if (!sheaf)
 		return;

@@ -5507,9 +5539,9 @@ slab_empty:
  */
 static void __pcs_install_empty_sheaf(struct kmem_cache *s,
 		struct slub_percpu_sheaves *pcs, struct slab_sheaf *empty,
-		struct node_barn *barn)
+		struct node_barn *barn, int cpu)
 {
-	lockdep_assert_held(this_cpu_ptr(&s->cpu_sheaves->lock));
+	qpw_lockdep_assert_held(&s->cpu_sheaves->lock);
 
 	/* This is what we expect to find if nobody interrupted us. */
 	if (likely(!pcs->spare)) {

mm/slub.c +56 -22

+ * and migration counter decreased.
  */
 static struct slub_percpu_sheaves *
 __pcs_replace_full_main(struct kmem_cache *s, struct slub_percpu_sheaves *pcs,
-			bool allow_spin)
+			bool allow_spin, int *cpu)
 {
 	struct slab_sheaf *empty;
 	struct node_barn *barn;
 	bool put_fail;
 
 restart:
-	lockdep_assert_held(this_cpu_ptr(&s->cpu_sheaves->lock));
+	qpw_lockdep_assert_held(&s->cpu_sheaves->lock);
 
 	/* Bootstrap or debug cache, back off */
 	if (unlikely(!cache_has_sheaves(s))) {
-		local_unlock(&s->cpu_sheaves->lock);
+		qpw_unlock(&s->cpu_sheaves->lock, *cpu);
+		migrate_enable();
 		return NULL;
 	}
 
 	barn = get_barn(s);
 	if (!barn) {
-		local_unlock(&s->cpu_sheaves->lock);
+		qpw_unlock(&s->cpu_sheaves->lock, *cpu);
+		migrate_enable();
 		return NULL;
 	}
 

@@ -5607,7 +5642,8 @@ restart:
 		stat(s, BARN_PUT_FAIL);
 
 		pcs->spare = NULL;
-		local_unlock(&s->cpu_sheaves->lock);
+		qpw_unlock(&s->cpu_sheaves->lock, *cpu);
+		migrate_enable();
 
 		sheaf_flush_unused(s, to_flush);
 		empty = to_flush;

@@ -5623,7 +5659,8 @@ restart:
 	put_fail = true;
 
 alloc_empty:
-	local_unlock(&s->cpu_sheaves->lock);
+	qpw_unlock(&s->cpu_sheaves->lock, *cpu);
+	migrate_enable();
 
 	/*
 	 * alloc_empty_sheaf() doesn't support !allow_spin and it's

@@ -5640,11 +5677,17 @@ alloc_empty:
 	if (put_fail)
 		 stat(s, BARN_PUT_FAIL);
 
-	if (!sheaf_flush_main(s))
+	migrate_disable();
+	*cpu = smp_processor_id();
+	if (!sheaf_flush_main(s, *cpu)) {
+		migrate_enable();
 		return NULL;
+	}
 
-	if (!local_trylock(&s->cpu_sheaves->lock))
+	if (!qpw_trylock(&s->cpu_sheaves->lock, *cpu)) {
+		migrate_enable();
 		return NULL;
+	}
 
 	pcs = this_cpu_ptr(s->cpu_sheaves);
 

@@ -5659,13 +5702,14 @@ alloc_empty:
 	return pcs;
 
 got_empty:
-	if (!local_trylock(&s->cpu_sheaves->lock)) {
+	if (!qpw_trylock(&s->cpu_sheaves->lock, *cpu)) {
+		migrate_enable();
 		barn_put_empty_sheaf(barn, empty);
 		return NULL;
 	}
 
 	pcs = this_cpu_ptr(s->cpu_sheaves);
-	__pcs_install_empty_sheaf(s, pcs, empty, barn);
+	__pcs_install_empty_sheaf(s, pcs, empty, barn, *cpu);
 
 	return pcs;
 }

@@ -5678,22 +5722,28 @@ static __fastpath_inline
 bool free_to_pcs(struct kmem_cache *s, void *object, bool allow_spin)
 {
 	struct slub_percpu_sheaves *pcs;
+	int cpu;
 
-	if (!local_trylock(&s->cpu_sheaves->lock))
+	migrate_disable();
+	cpu = smp_processor_id();
+	if (!qpw_trylock(&s->cpu_sheaves->lock, cpu)) {
+		migrate_enable();
 		return false;
+	}
 
 	pcs = this_cpu_ptr(s->cpu_sheaves);
 
 	if (unlikely(pcs->main->size == s->sheaf_capacity)) {
 
-		pcs = __pcs_replace_full_main(s, pcs, allow_spin);
+		pcs = __pcs_replace_full_main(s, pcs, allow_spin, &cpu);
 		if (unlikely(!pcs))
 			return false;
 	}
 
 	pcs->main->objects[pcs->main->size++] = object;
 
-	local_unlock(&s->cpu_sheaves->lock);
+	qpw_unlock(&s->cpu_sheaves->lock, cpu);
+	migrate_enable();
 
 	stat(s, FREE_FASTPATH);
 

@@ -5777,14 +5827,19 @@ bool __kfree_rcu_sheaf(struct kmem_cache
 {
 	struct slub_percpu_sheaves *pcs;
 	struct slab_sheaf *rcu_sheaf;
+	int cpu;
 
 	if (WARN_ON_ONCE(IS_ENABLED(CONFIG_PREEMPT_RT)))
 		return false;
 
 	lock_map_acquire_try(&kfree_rcu_sheaf_map);
 
-	if (!local_trylock(&s->cpu_sheaves->lock))
+	migrate_disable();
+	cpu = smp_processor_id();
+	if (!qpw_trylock(&s->cpu_sheaves->lock, cpu)) {
+		migrate_enable();
 		goto fail;
+	}
 
 	pcs = this_cpu_ptr(s->cpu_sheaves);
 

@@ -5795,7 +5850,8 @@ bool __kfree_rcu_sheaf(struct kmem_cache
 
 		/* Bootstrap or debug cache, fall back */
 		if (unlikely(!cache_has_sheaves(s))) {
-			local_unlock(&s->cpu_sheaves->lock);
+			qpw_unlock(&s->cpu_sheaves->lock, cpu);
+			migrate_enable();
 			goto fail;
 		}
 

@@ -5807,7 +5863,8 @@ bool __kfree_rcu_sheaf(struct kmem_cache
 
 		barn = get_barn(s);
 		if (!barn) {
-			local_unlock(&s->cpu_sheaves->lock);
+			qpw_unlock(&s->cpu_sheaves->lock, cpu);
+			migrate_enable();
 			goto fail;
 		}
 

@@ -5818,15 +5875,18 @@ bool __kfree_rcu_sheaf(struct kmem_cache
 			goto do_free;
 		}
 
-		local_unlock(&s->cpu_sheaves->lock);
+		qpw_unlock(&s->cpu_sheaves->lock, cpu);
+		migrate_enable();
 
 		empty = alloc_empty_sheaf(s, GFP_NOWAIT);
 
 		if (!empty)
 			goto fail;
 
-		if (!local_trylock(&s->cpu_sheaves->lock)) {
+		migrate_disable();
+		if (!qpw_trylock(&s->cpu_sheaves->lock, cpu)) {
 			barn_put_empty_sheaf(barn, empty);
+			migrate_enable();
 			goto fail;
 		}
 

@@ -5862,7 +5922,8 @@ do_free:
 	if (rcu_sheaf)
 		call_rcu(&rcu_sheaf->rcu_head, rcu_free_sheaf);
 
-	local_unlock(&s->cpu_sheaves->lock);
+	qpw_unlock(&s->cpu_sheaves->lock, cpu);
+	migrate_enable();
 
 	stat(s, FREE_RCU_SHEAF);
 	lock_map_release(&kfree_rcu_sheaf_map);

@@ -5889,6 +5950,7 @@ static void free_to_pcs_bulk(struct kmem
 	void *remote_objects[PCS_BATCH_MAX];
 	unsigned int remote_nr = 0;
 	int node = numa_mem_id();
+	int cpu;
 
 next_remote_batch:
 	while (i < size) {

@@ -5918,7 +5980,9 @@ next_remote_batch:
 		goto flush_remote;
 
 next_batch:
-	if (!local_trylock(&s->cpu_sheaves->lock))
+	migrate_disable();
+	cpu = smp_processor_id();
+	if (!qpw_trylock(&s->cpu_sheaves->lock, cpu))
 		goto fallback;
 
 	pcs = this_cpu_ptr(s->cpu_sheaves);

@@ -5961,7 +6025,8 @@ do_free:
 	memcpy(main->objects + main->size, p, batch * sizeof(void *));
 	main->size += batch;
 
-	local_unlock(&s->cpu_sheaves->lock);
+	qpw_unlock(&s->cpu_sheaves->lock, cpu);
+	migrate_enable();
 
 	stat_add(s, FREE_FASTPATH, batch);
 

@@ -5977,7 +6042,8 @@ do_free:
 	return;
 
 no_empty:
-	local_unlock(&s->cpu_sheaves->lock);
+	qpw_unlock(&s->cpu_sheaves->lock, cpu);
+	migrate_enable();
 
 	/*
 	 * if we depleted all empty sheaves in the barn or there are too

The problem:
Some places in the kernel implement a parallel programming strategy consisting on local_locks() for most of the work, and some rare remote operations are scheduled on target cpu. This keeps cache bouncing low since cacheline tends to be mostly local, and avoids the cost of locks in non-RT kernels, even though the very few remote operations will be expensive due to scheduling overhead.

On the other hand, for RT workloads this can represent a problem: getting an important workload scheduled out to deal with remote requests is sure to introduce unexpected deadline misses.

The idea:
Currently with PREEMPT_RT=y, local_locks() become per-cpu spinlocks. In this case, instead of scheduling work on a remote cpu, it should be safe to grab that remote cpu's per-cpu spinlock and run the required work locally. That major cost, which is un/locking in every local function, already happens in PREEMPT_RT.

Also, there is no need to worry about extra cache bouncing:
The cacheline invalidation already happens due to schedule_work_on().

This will avoid schedule_work_on(), and thus avoid scheduling-out an RT workload.

Proposed solution:
A new interface called Queue PerCPU Work (QPW), which should replace Work Queue in the above mentioned use case.

If PREEMPT_RT=n this interfaces just wraps the current

local_locks + WorkQueue behavior, so no expected change in runtime.

If PREEMPT_RT=y, or CONFIG_QPW=y, queue_percpu_work_on(cpu,...) will

lock that cpu's per-cpu structure and perform work on it locally. This is possible because on functions that can be used for performing remote work on remote per-cpu structures, the local_lock (which is already a this_cpu spinlock()), will be replaced by a qpw_spinlock(), which is able to get the per_cpu spinlock() for the cpu passed as parameter.

RFC->v1:

- Introduce CONFIG_QPW and qpw= kernel boot option to enable remote spinlocking and execution even on !CONFIG_PREEMPT_RT kernels (Leonardo Bras).
- Move buffer_head draining to separate workqueue (Marcelo Tosatti).
- Convert mlock per-CPU page lists to QPW (Marcelo Tosatti).
- Drop memcontrol convertion (as isolated CPUs are not targets of queue_work_on anymore).
- Rebase SLUB against Vlastimil's slab/next.
- Add basic document for QPW (Waiman Long).

The following testcase triggers lru_add_drain_all on an isolated CPU (that does sys_write to a file before entering its realtime loop).

/*
* Simulates a low latency loop program that is interrupted
* due to lru_add_drain_all. To trigger lru_add_drain_all, run:
*
* blockdev --flushbufs /dev/sdX
*
*/
#define _GNU_SOURCE
#include <fcntl.h>
#include <stdio.h>
#include <stdlib.h>
#include <sys/mman.h>
#include <string.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <stdlib.h>
#include <stdarg.h>
#include <pthread.h>
#include <sched.h>
#include <unistd.h>

int cpu;

static void *run(void *arg)

{
	pthread_t current_thread;
	cpu_set_t cpuset;
	int ret, nrloops;
	struct sched_param sched_p;
	pid_t pid;
	int fd;
	char buf[] = "xxxxxxxxxxx";

	CPU_ZERO(&cpuset);
	CPU_SET(cpu, &cpuset);

	current_thread = pthread_self();    
	ret = pthread_setaffinity_np(current_thread, sizeof(cpu_set_t), &cpuset);
	if (ret) {
		perror("pthread_setaffinity_np failed\n");
		exit(0);
	}

	memset(&sched_p, 0, sizeof(struct sched_param));
	sched_p.sched_priority = 1;
	pid = gettid();
	ret = sched_setscheduler(pid, SCHED_FIFO, &sched_p);
	if (ret) {
		perror("sched_setscheduler");
		exit(0);
	}

	fd = open("/tmp/tmpfile", O_RDWR|O_CREAT|O_TRUNC);
	if (fd == -1) {
		perror("open");
		exit(0);
	}

	ret = write(fd, buf, sizeof(buf));
	if (ret == -1) {
		perror("write");
		exit(0);
	}

	do { 
		nrloops = nrloops+2;
		nrloops--;
	} while (1);
}

int main(int argc, char *argv[])

{
        int fd, ret;
	pthread_t thread;
	long val;
	char *endptr, *str;
	struct sched_param sched_p;
	pid_t pid;

	if (argc != 2) {
		printf("usage: %s cpu-nr\n", argv[0]);
		printf("where CPU number is the CPU to pin thread to\n");
		exit(0);
	}
	str = argv[1];
	cpu = strtol(str, &endptr, 10);
	if (cpu < 0) {
		printf("strtol returns %d\n", cpu);
		exit(0);
	}
	printf("cpunr=%d\n", cpu);

	memset(&sched_p, 0, sizeof(struct sched_param));
	sched_p.sched_priority = 1;
	pid = getpid();
	ret = sched_setscheduler(pid, SCHED_FIFO, &sched_p);
	if (ret) {
		perror("sched_setscheduler");
		exit(0);

	}

	pthread_create(&thread, NULL, run, NULL);

	sleep(5000);

	pthread_join(thread, NULL);
}

The problem:
Some places in the kernel implement a parallel programming strategy consisting on local_locks() for most of the work, and some rare remote operations are scheduled on target cpu. This keeps cache bouncing low since cacheline tends to be mostly local, and avoids the cost of locks in non-RT kernels, even though the very few remote operations will be expensive due to scheduling overhead.

On the other hand, for RT workloads this can represent a problem: getting an important workload scheduled out to deal with remote requests is sure to introduce unexpected deadline misses.

The idea:
Currently with PREEMPT_RT=y, local_locks() become per-cpu spinlocks. In this case, instead of scheduling work on a remote cpu, it should be safe to grab that remote cpu's per-cpu spinlock and run the required work locally. That major cost, which is un/locking in every local function, already happens in PREEMPT_RT.

Also, there is no need to worry about extra cache bouncing:
The cacheline invalidation already happens due to schedule_work_on().

This will avoid schedule_work_on(), and thus avoid scheduling-out an RT workload.

Proposed solution:
A new interface called Queue PerCPU Work (QPW), which should replace Work Queue in the above mentioned use case.

If PREEMPT_RT=n this interfaces just wraps the current local_locks + WorkQueue behavior, so no expected change in runtime.

If PREEMPT_RT=y, or CONFIG_QPW=y, queue_percpu_work_on(cpu,...) will lock that cpu's per-cpu structure and perform work on it locally. This is possible because on functions that can be used for performing remote work on remote per-cpu structures, the local_lock (which is already a this_cpu spinlock()), will be replaced by a qpw_spinlock(), which is able to get the per_cpu spinlock() for the cpu passed as parameter.

Quoted 32 lines · Marcelo Tosatti

The problem:
Some places in the kernel implement a parallel programming strategy consisting on local_locks() for most of the work, and some rare remote operations are scheduled on target cpu. This keeps cache bouncing low since cacheline tends to be mostly local, and avoids the cost of locks in non-RT kernels, even though the very few remote operations will be expensive due to scheduling overhead.

On the other hand, for RT workloads this can represent a problem: getting an important workload scheduled out to deal with remote requests is sure to introduce unexpected deadline misses.

The idea:
Currently with PREEMPT_RT=y, local_locks() become per-cpu spinlocks. In this case, instead of scheduling work on a remote cpu, it should be safe to grab that remote cpu's per-cpu spinlock and run the required work locally. That major cost, which is un/locking in every local function, already happens in PREEMPT_RT.

Also, there is no need to worry about extra cache bouncing:
The cacheline invalidation already happens due to schedule_work_on().

This will avoid schedule_work_on(), and thus avoid scheduling-out an RT workload.

Proposed solution:
A new interface called Queue PerCPU Work (QPW), which should replace Work Queue in the above mentioned use case.

If PREEMPT_RT=n this interfaces just wraps the current local_locks + WorkQueue behavior, so no expected change in runtime.

Quoted 6 lines

If PREEMPT_RT=y, or CONFIG_QPW=y, queue_percpu_work_on(cpu,...) will lock that cpu's per-cpu structure and perform work on it locally. This is possible because on functions that can be used for performing remote work on remote per-cpu structures, the local_lock (which is already a this_cpu spinlock()), will be replaced by a qpw_spinlock(), which is able to get the per_cpu spinlock() for the cpu passed as parameter.

What about !PREEMPT_RT? We have people running isolated workloads and these sorts of pcp disruptions are really unwelcome as well. They do not have requirements as strong as RT workloads but the underlying fundamental problem is the same. Frederic (now CCed) is working on moving those pcp book keeping activities to be executed to the return to the userspace which should be taking care of both RT and non-RT configurations AFAICS.

Quoted 40 lines · Michal Hocko

Quoted 32 lines · Marcelo Tosatti

The problem:
Some places in the kernel implement a parallel programming strategy consisting on local_locks() for most of the work, and some rare remote operations are scheduled on target cpu. This keeps cache bouncing low since cacheline tends to be mostly local, and avoids the cost of locks in non-RT kernels, even though the very few remote operations will be expensive due to scheduling overhead.

On the other hand, for RT workloads this can represent a problem: getting an important workload scheduled out to deal with remote requests is sure to introduce unexpected deadline misses.

The idea:
Currently with PREEMPT_RT=y, local_locks() become per-cpu spinlocks. In this case, instead of scheduling work on a remote cpu, it should be safe to grab that remote cpu's per-cpu spinlock and run the required work locally. That major cost, which is un/locking in every local function, already happens in PREEMPT_RT.

Also, there is no need to worry about extra cache bouncing:
The cacheline invalidation already happens due to schedule_work_on().

This will avoid schedule_work_on(), and thus avoid scheduling-out an RT workload.

Proposed solution:
A new interface called Queue PerCPU Work (QPW), which should replace Work Queue in the above mentioned use case.

If PREEMPT_RT=n this interfaces just wraps the current local_locks + WorkQueue behavior, so no expected change in runtime.

Quoted 6 lines

If PREEMPT_RT=y, or CONFIG_QPW=y, queue_percpu_work_on(cpu,...) will lock that cpu's per-cpu structure and perform work on it locally. This is possible because on functions that can be used for performing remote work on remote per-cpu structures, the local_lock (which is already a this_cpu spinlock()), will be replaced by a qpw_spinlock(), which is able to get the per_cpu spinlock() for the cpu passed as parameter.

Quoted 7 lines

What about !PREEMPT_RT? We have people running isolated workloads and these sorts of pcp disruptions are really unwelcome as well. They do not have requirements as strong as RT workloads but the underlying fundamental problem is the same. Frederic (now CCed) is working on moving those pcp book keeping activities to be executed to the return to the userspace which should be taking care of both RT and non-RT configurations AFAICS.

Michal,

For !PREEMPT_RT, _if_ you select CONFIG_QPW=y, then there is a kernel boot option qpw=y/n, which controls whether the behaviour will be similar (the spinlock is taken on local_lock, similar to PREEMPT_RT).

If CONFIG_QPW=n, or kernel boot option qpw=n, then only local_lock (and remote work via work_queue) is used.

What "pcp book keeping activities" you refer to ? I don't see how moving certain activities that happen under SLUB or LRU spinlocks to happen before return to userspace changes things related to avoidance of CPU interruption ?

Thanks

Quoted 55 lines · Marcelo Tosatti

Quoted 40 lines · Michal Hocko

Quoted 32 lines · Marcelo Tosatti

The problem:
Some places in the kernel implement a parallel programming strategy consisting on local_locks() for most of the work, and some rare remote operations are scheduled on target cpu. This keeps cache bouncing low since cacheline tends to be mostly local, and avoids the cost of locks in non-RT kernels, even though the very few remote operations will be expensive due to scheduling overhead.

On the other hand, for RT workloads this can represent a problem: getting an important workload scheduled out to deal with remote requests is sure to introduce unexpected deadline misses.

The idea:
Currently with PREEMPT_RT=y, local_locks() become per-cpu spinlocks. In this case, instead of scheduling work on a remote cpu, it should be safe to grab that remote cpu's per-cpu spinlock and run the required work locally. That major cost, which is un/locking in every local function, already happens in PREEMPT_RT.

Also, there is no need to worry about extra cache bouncing:
The cacheline invalidation already happens due to schedule_work_on().

This will avoid schedule_work_on(), and thus avoid scheduling-out an RT workload.

Proposed solution:
A new interface called Queue PerCPU Work (QPW), which should replace Work Queue in the above mentioned use case.

If PREEMPT_RT=n this interfaces just wraps the current local_locks + WorkQueue behavior, so no expected change in runtime.

Quoted 6 lines

If PREEMPT_RT=y, or CONFIG_QPW=y, queue_percpu_work_on(cpu,...) will lock that cpu's per-cpu structure and perform work on it locally. This is possible because on functions that can be used for performing remote work on remote per-cpu structures, the local_lock (which is already a this_cpu spinlock()), will be replaced by a qpw_spinlock(), which is able to get the per_cpu spinlock() for the cpu passed as parameter.

Quoted 7 lines

What about !PREEMPT_RT? We have people running isolated workloads and these sorts of pcp disruptions are really unwelcome as well. They do not have requirements as strong as RT workloads but the underlying fundamental problem is the same. Frederic (now CCed) is working on moving those pcp book keeping activities to be executed to the return to the userspace which should be taking care of both RT and non-RT configurations AFAICS.

Michal,

For !PREEMPT_RT, _if_ you select CONFIG_QPW=y, then there is a kernel boot option qpw=y/n, which controls whether the behaviour will be similar (the spinlock is taken on local_lock, similar to PREEMPT_RT).

Quoted 2 lines

If CONFIG_QPW=n, or kernel boot option qpw=n, then only local_lock (and remote work via work_queue) is used.

OK, this is not true. There is only CONFIG_QPW and the qpw=yes/no kernel boot option for control.

CONFIG_PREEMPT_RT should probably select CONFIG_QPW=y and CONFIG_QPW_DEFAULT=y.

Quoted 8 lines

What "pcp book keeping activities" you refer to ? I don't see how moving certain activities that happen under SLUB or LRU spinlocks to happen before return to userspace changes things related to avoidance of CPU interruption ?

Thanks

On Tue, Feb 10, 2026 at 03:01:10PM +0100, Michal Hocko wrote:

Quoted 7 lines

What about !PREEMPT_RT? We have people running isolated workloads and these sorts of pcp disruptions are really unwelcome as well. They do not have requirements as strong as RT workloads but the underlying fundamental problem is the same. Frederic (now CCed) is working on moving those pcp book keeping activities to be executed to the return to the userspace which should be taking care of both RT and non-RT configurations AFAICS.

Michal,

For !PREEMPT_RT, _if_ you select CONFIG_QPW=y, then there is a kernel boot option qpw=y/n, which controls whether the behaviour will be similar (the spinlock is taken on local_lock, similar to PREEMPT_RT).

If CONFIG_QPW=n, or kernel boot option qpw=n, then only local_lock (and remote work via work_queue) is used.

What "pcp book keeping activities" you refer to ? I don't see how moving certain activities that happen under SLUB or LRU spinlocks to happen before return to userspace changes things related to avoidance of CPU interruption ?

Quoted 1 line · Marcelo Tosatti

On Tue, Feb 10, 2026 at 03:01:10PM +0100, Michal Hocko wrote:

[...]

Quoted 8 lines

Quoted 7 lines

What about !PREEMPT_RT? We have people running isolated workloads and these sorts of pcp disruptions are really unwelcome as well. They do not have requirements as strong as RT workloads but the underlying fundamental problem is the same. Frederic (now CCed) is working on moving those pcp book keeping activities to be executed to the return to the userspace which should be taking care of both RT and non-RT configurations AFAICS.

Quoted 5 lines

Michal,

For !PREEMPT_RT, _if_ you select CONFIG_QPW=y, then there is a kernel boot option qpw=y/n, which controls whether the behaviour will be similar (the spinlock is taken on local_lock, similar to PREEMPT_RT).

My bad. I've misread the config space of this.

Quoted 7 lines

If CONFIG_QPW=n, or kernel boot option qpw=n, then only local_lock (and remote work via work_queue) is used.

What "pcp book keeping activities" you refer to ? I don't see how moving certain activities that happen under SLUB or LRU spinlocks to happen before return to userspace changes things related to avoidance of CPU interruption ?

Essentially delayed operations like pcp state flushing happens on return to the userspace on isolated CPUs. No locking changes are required as the work is still per-cpu.

In other words the approach Frederic is working on is to not change the locking of pcp delayed work but instead move that work into well defined place - i.e. return to the userspace.

Btw. have you measure the impact of preempt_disbale -> spinlock on hot paths like SLUB sheeves?

Quoted 11 lines · Michal Hocko

Quoted 1 line · Marcelo Tosatti

On Tue, Feb 10, 2026 at 03:01:10PM +0100, Michal Hocko wrote:

[...]

Quoted 8 lines

Quoted 7 lines

What about !PREEMPT_RT? We have people running isolated workloads and these sorts of pcp disruptions are really unwelcome as well. They do not have requirements as strong as RT workloads but the underlying fundamental problem is the same. Frederic (now CCed) is working on moving those pcp book keeping activities to be executed to the return to the userspace which should be taking care of both RT and non-RT configurations AFAICS.

Quoted 7 lines

Quoted 5 lines

Michal,

For !PREEMPT_RT, _if_ you select CONFIG_QPW=y, then there is a kernel boot option qpw=y/n, which controls whether the behaviour will be similar (the spinlock is taken on local_lock, similar to PREEMPT_RT).

My bad. I've misread the config space of this.

My bad, actually. Its only CONFIG_QPW on the current patchset.

Quoted 16 lines

Quoted 7 lines

If CONFIG_QPW=n, or kernel boot option qpw=n, then only local_lock (and remote work via work_queue) is used.

What "pcp book keeping activities" you refer to ? I don't see how moving certain activities that happen under SLUB or LRU spinlocks to happen before return to userspace changes things related to avoidance of CPU interruption ?

Essentially delayed operations like pcp state flushing happens on return to the userspace on isolated CPUs. No locking changes are required as the work is still per-cpu.

In other words the approach Frederic is working on is to not change the locking of pcp delayed work but instead move that work into well defined place - i.e. return to the userspace.

Quoted 2 lines

Btw. have you measure the impact of preempt_disbale -> spinlock on hot paths like SLUB sheeves?

Nope, i have not. What is/are the standard benchmarks for SLUB/SLAB allocation ?

Quoted 13 lines · Michal Hocko

Quoted 1 line · Marcelo Tosatti

On Tue, Feb 10, 2026 at 03:01:10PM +0100, Michal Hocko wrote:

[...]

Quoted 10 lines

Quoted 7 lines

What about !PREEMPT_RT? We have people running isolated workloads and these sorts of pcp disruptions are really unwelcome as well. They do not have requirements as strong as RT workloads but the underlying fundamental problem is the same. Frederic (now CCed) is working on moving those pcp book keeping activities to be executed to the return to the userspace which should be taking care of both RT and non-RT configurations AFAICS.

Michal,

Quoted 5 lines

Quoted 3 lines

For !PREEMPT_RT, _if_ you select CONFIG_QPW=y, then there is a kernel boot option qpw=y/n, which controls whether the behaviour will be similar (the spinlock is taken on local_lock, similar to PREEMPT_RT).

My bad. I've misread the config space of this.

My bad, actually. Its only CONFIG_QPW on the current patchset.

Quoted 1 line · Marcelo Tosatti

On Tue, Feb 10, 2026 at 03:01:10PM +0100, Michal Hocko wrote:

[...]

Quoted 8 lines

Quoted 7 lines

What about !PREEMPT_RT? We have people running isolated workloads and these sorts of pcp disruptions are really unwelcome as well. They do not have requirements as strong as RT workloads but the underlying fundamental problem is the same. Frederic (now CCed) is working on moving those pcp book keeping activities to be executed to the return to the userspace which should be taking care of both RT and non-RT configurations AFAICS.

Quoted 5 lines

Michal,

For !PREEMPT_RT, _if_ you select CONFIG_QPW=y, then there is a kernel boot option qpw=y/n, which controls whether the behaviour will be similar (the spinlock is taken on local_lock, similar to PREEMPT_RT).

My bad. I've misread the config space of this.

Quoted 7 lines

If CONFIG_QPW=n, or kernel boot option qpw=n, then only local_lock (and remote work via work_queue) is used.

What "pcp book keeping activities" you refer to ? I don't see how moving certain activities that happen under SLUB or LRU spinlocks to happen before return to userspace changes things related to avoidance of CPU interruption ?

Essentially delayed operations like pcp state flushing happens on return to the userspace on isolated CPUs. No locking changes are required as the work is still per-cpu.

In other words the approach Frederic is working on is to not change the locking of pcp delayed work but instead move that work into well defined place - i.e. return to the userspace.

Btw. have you measure the impact of preempt_disbale -> spinlock on hot paths like SLUB sheeves?

Quoted 31 lines · Michal Hocko

Quoted 1 line · Marcelo Tosatti

On Tue, Feb 10, 2026 at 03:01:10PM +0100, Michal Hocko wrote:

[...]

Quoted 8 lines

Quoted 7 lines

What about !PREEMPT_RT? We have people running isolated workloads and these sorts of pcp disruptions are really unwelcome as well. They do not have requirements as strong as RT workloads but the underlying fundamental problem is the same. Frederic (now CCed) is working on moving those pcp book keeping activities to be executed to the return to the userspace which should be taking care of both RT and non-RT configurations AFAICS.

Quoted 5 lines

Michal,

For !PREEMPT_RT, _if_ you select CONFIG_QPW=y, then there is a kernel boot option qpw=y/n, which controls whether the behaviour will be similar (the spinlock is taken on local_lock, similar to PREEMPT_RT).

My bad. I've misread the config space of this.

Quoted 7 lines

If CONFIG_QPW=n, or kernel boot option qpw=n, then only local_lock (and remote work via work_queue) is used.

What "pcp book keeping activities" you refer to ? I don't see how moving certain activities that happen under SLUB or LRU spinlocks to happen before return to userspace changes things related to avoidance of CPU interruption ?

Essentially delayed operations like pcp state flushing happens on return to the userspace on isolated CPUs. No locking changes are required as the work is still per-cpu.

Quoted 6 lines

In other words the approach Frederic is working on is to not change the locking of pcp delayed work but instead move that work into well defined place - i.e. return to the userspace.

Btw. have you measure the impact of preempt_disbale -> spinlock on hot paths like SLUB sheeves?

Hi Michal,

I have done some study on this (which I presented on Plumbers 2023):
https://lpc.events/event/17/contributions/1484/

Since they are per-cpu spinlocks, and the remote operations are not that frequent, as per design of the current approach, we are not supposed to see contention (I was not able to detect contention even after stress testing for weeks), nor relevant cacheline bouncing.

That being said, for RT local_locks already get per-cpu spinlocks, so there is only difference for !RT, which as you mention, does preemtp_disable():

The performance impact noticed was mostly about jumping around in executable code, as inlining spinlocks (test #2 on presentation) took care of most of the added extra cycles, adding about 4-14 extra cycles per lock/unlock cycle. (tested on memcg with kmalloc test)

Yeah, as expected there is some extra cycles, as we are doing extra atomic operations (even if in a local cacheline) in !RT case, but this could be enabled only if the user thinks this is an ok cost for reducing interruptions.

What do you think?

Quoted 62 lines · Leonardo Bras

Quoted 31 lines · Michal Hocko

Quoted 1 line · Marcelo Tosatti

On Tue, Feb 10, 2026 at 03:01:10PM +0100, Michal Hocko wrote:

[...]

Quoted 8 lines

Quoted 7 lines

What about !PREEMPT_RT? We have people running isolated workloads and these sorts of pcp disruptions are really unwelcome as well. They do not have requirements as strong as RT workloads but the underlying fundamental problem is the same. Frederic (now CCed) is working on moving those pcp book keeping activities to be executed to the return to the userspace which should be taking care of both RT and non-RT configurations AFAICS.

Quoted 5 lines

Michal,

For !PREEMPT_RT, _if_ you select CONFIG_QPW=y, then there is a kernel boot option qpw=y/n, which controls whether the behaviour will be similar (the spinlock is taken on local_lock, similar to PREEMPT_RT).

My bad. I've misread the config space of this.

Quoted 7 lines

If CONFIG_QPW=n, or kernel boot option qpw=n, then only local_lock (and remote work via work_queue) is used.

What "pcp book keeping activities" you refer to ? I don't see how moving certain activities that happen under SLUB or LRU spinlocks to happen before return to userspace changes things related to avoidance of CPU interruption ?

Essentially delayed operations like pcp state flushing happens on return to the userspace on isolated CPUs. No locking changes are required as the work is still per-cpu.

Quoted 6 lines

In other words the approach Frederic is working on is to not change the locking of pcp delayed work but instead move that work into well defined place - i.e. return to the userspace.

Btw. have you measure the impact of preempt_disbale -> spinlock on hot paths like SLUB sheeves?

Hi Michal,

I have done some study on this (which I presented on Plumbers 2023):
https://lpc.events/event/17/contributions/1484/

Since they are per-cpu spinlocks, and the remote operations are not that frequent, as per design of the current approach, we are not supposed to see contention (I was not able to detect contention even after stress testing for weeks), nor relevant cacheline bouncing.

That being said, for RT local_locks already get per-cpu spinlocks, so there is only difference for !RT, which as you mention, does preemtp_disable():

The performance impact noticed was mostly about jumping around in executable code, as inlining spinlocks (test #2 on presentation) took care of most of the added extra cycles, adding about 4-14 extra cycles per lock/unlock cycle. (tested on memcg with kmalloc test)

Yeah, as expected there is some extra cycles, as we are doing extra atomic operations (even if in a local cacheline) in !RT case, but this could be enabled only if the user thinks this is an ok cost for reducing interruptions.

Quoted 1 line

What do you think?

The fact that the behavior is opt-in for !RT is certainly a plus. I also do not expect the overhead to be really be really big. To me, a much more important question is which of the two approaches is easier to maintain long term. The pcp work needs to be done one way or the other. Whether we want to tweak locking or do it at a very well defined time is the bigger question.

Signature · 2 lines

Michal Hocko
SUSE Labs

Michal,

Again, i don't see how moving operations to happen at return to kernel would help (assuming you are talking about "context_tracking,x86: Defer some IPIs until a user->kernel transition").

You can't delay either kmalloc (removal of object from per-CPU freelist), or kfree (return of object from per-CPU freelist), or kmem_cache_shrink or kmem_cache_shrink to return to userspace.

What i missing something here? (or do you have something on your mind which i can't see).

Quoted 5 lines · Marcelo Tosatti

Michal,

Again, i don't see how moving operations to happen at return to kernel would help (assuming you are talking about "context_tracking,x86: Defer some IPIs until a user->kernel transition").

Nope, I am not talking about IPIs, although those are an example of pcp state as well. I am sorry I do not have a link handy, I am pretty sure Frederic will have that. Another example, though, was vmstat flushes that need to be pcp. There are many other examples.

[...]

Quoted 3 lines

You can't delay either kmalloc (removal of object from per-CPU freelist), or kfree (return of object from per-CPU freelist), or kmem_cache_shrink or kmem_cache_shrink to return to userspace.

Why?

Quoted 2 lines

What i missing something here? (or do you have something on your mind which i can't see).

I am really sorry for being really vague here. Let me try to draw a more abstract problem definition and let's see whether we are trying to solve the same problem here. Maybe not...

I believe the main usecase of the interest here is uninterrupted userspace execution

and delayed pcp work that migh disturb such workload
after it has returned to the userspace. Right?
That is usually hauskeeping work that for, performance reasons, doesn't happen in hot paths while the workload was executing in the kernel space.

There are more ways to deal with that. You can either change the hot path to not require deferred operation (tricky withtout introducing regressions for most workloads) or you can define a more suitable place to perform the housekeeping while still running in the kernel.

Your QWP work relies on local_lock -> spin_lock transition and performing the pcp work remotely so you do not need to disturb that remote cpu. Correct?

Alternative approach is to define a moment when the housekeeping operation is performed on that local cpu while still running in the kernel space - e.g. when returning to the userspace. Delayed work is then not necessary and userspace is not disrupted after returning to the userspace.

Do I make more sense or does the above sound like a complete gibberish?

Quoted 1 line · Michal Hocko

On Thu 19-02-26 12:27:23, Marcelo Tosatti wrote:

Quoted 21 lines

and delayed pcp work that migh disturb such workload
after it has returned to the userspace. Right?
That is usually hauskeeping work that for, performance reasons, doesn't happen in hot paths while the workload was executing in the kernel space.

There are more ways to deal with that. You can either change the hot path to not require deferred operation (tricky withtout introducing regressions for most workloads) or you can define a more suitable place to perform the housekeeping while still running in the kernel.

Your QWP work relies on local_lock -> spin_lock transition and performing the pcp work remotely so you do not need to disturb that remote cpu. Correct?

Alternative approach is to define a moment when the housekeeping operation is performed on that local cpu while still running in the kernel space - e.g. when returning to the userspace. Delayed work is then not necessary and userspace is not disrupted after returning to the userspace.

Quoted 1 line

Do I make more sense or does the above sound like a complete gibberish?

OK, sure, but can't see how you can do that with per-CPU caches for kmalloc, for example.

Quoted 2 lines · Marcelo Tosatti

Quoted 1 line · Michal Hocko

On Thu 19-02-26 12:27:23, Marcelo Tosatti wrote:

[...]

Quoted 21 lines

Quoted 21 lines

and delayed pcp work that migh disturb such workload
after it has returned to the userspace. Right?
That is usually hauskeeping work that for, performance reasons, doesn't happen in hot paths while the workload was executing in the kernel space.

There are more ways to deal with that. You can either change the hot path to not require deferred operation (tricky withtout introducing regressions for most workloads) or you can define a more suitable place to perform the housekeeping while still running in the kernel.

Your QWP work relies on local_lock -> spin_lock transition and performing the pcp work remotely so you do not need to disturb that remote cpu. Correct?

Alternative approach is to define a moment when the housekeeping operation is performed on that local cpu while still running in the kernel space - e.g. when returning to the userspace. Delayed work is then not necessary and userspace is not disrupted after returning to the userspace.

Quoted 4 lines

Quoted 1 line

Do I make more sense or does the above sound like a complete gibberish?

OK, sure, but can't see how you can do that with per-CPU caches for kmalloc, for example.

As we have discussed in other subthread. By flushing those pcp caches on the return to userspace. Those flushes are not needed immediately. They just need to happen to allow operations listed by Vlastimil to finish. Or to avoid the problem by not using them but that is a separate discussion.

I believe we can establish that any pcp delayed operation implemented through WQs can be flushed on the way to the userspace, right? The performance might be suboptimal but correctness will be preserved. So doing this on isolated CPUs could be an alternative to making changes to the pcp WQ handling.

I haven't checked the WQ code deeply but I believe it should be feasible to flush all pcp WQs with pending work on the isolated cpu when the isolated workload returns to the userspace. This way we wouldn't need to special case each and every one of them.

Quoted 5 lines · Marcelo Tosatti

Michal,

Again, i don't see how moving operations to happen at return to kernel would help (assuming you are talking about "context_tracking,x86: Defer some IPIs until a user->kernel transition").

Nope, I am not talking about IPIs, although those are an example of pcp state as well. I am sorry I do not have a link handy, I am pretty sure Frederic will have that. Another example, though, was vmstat flushes that need to be pcp. There are many other examples.

Quoted 11 lines · Michal Hocko

Quoted 5 lines · Marcelo Tosatti

Michal,

Again, i don't see how moving operations to happen at return to kernel would help (assuming you are talking about "context_tracking,x86: Defer some IPIs until a user->kernel transition").

Nope, I am not talking about IPIs, although those are an example of pcp state as well. I am sorry I do not have a link handy, I am pretty sure Frederic will have that. Another example, though, was vmstat flushes that need to be pcp. There are many other examples.

Here it is:

https://lore.kernel.org/all/[email protected]/

Thanks.

Quoted 17 lines · Frederic Weisbecker

Quoted 11 lines · Michal Hocko

Quoted 5 lines · Marcelo Tosatti

Michal,

Again, i don't see how moving operations to happen at return to kernel would help (assuming you are talking about "context_tracking,x86: Defer some IPIs until a user->kernel transition").

Nope, I am not talking about IPIs, although those are an example of pcp state as well. I am sorry I do not have a link handy, I am pretty sure Frederic will have that. Another example, though, was vmstat flushes that need to be pcp. There are many other examples.

Here it is:

https://lore.kernel.org/all/[email protected]/

Quoted 1 line

Thanks.

Frederic,

I think this is a valid solution, however on systems with many CPUs, in nohz_full, performing system calls, can't there be significant increase of lru_lock contention ? Consider 100+ CPUs performing many system calls which add 1 or 2 folios to per-CPU LRU lists.

Note: if you are confident about the above not being a problem, this approach looks good to me.

commit eb709b0d062efd653a61183af8e27b2711c3cf5c

Author: Shaohua Li <[email protected]>
Date:   Tue May 24 17:12:55 2011 -0700

mm: batch activate_page() to reduce lock contention

The zone->lru_lock is heavily contented in workload where activate_page() is frequently used. We could do batch activate_page() to reduce the lock contention. The batched pages will be added into zone list when the pool is full or page reclaim is trying to drain them.

For example, in a 4 socket 64 CPU system, create a sparse file and 64 processes, processes shared map to the file. Each process read access the whole file and then exit. The process exit will do unmap_vmas() and cause a lot of activate_page() call. In such workload, we saw about 58% total time reduction with below patch. Other workloads with a lot of activate_page also benefits a lot too.

...

    The most significent are:
      case-lru-file-readtwice     -11.69%
      case-mmap-pread-rand        -15.26%
      case-mmap-pread-seq         -69.72%

Some Gemini answers (question was "list of nohz_full usecases"):

2. Scientific Simulation & Research

Research institutions (like CERN, NASA, or national labs) use nohz_full for "tightly coupled" parallel workloads.

Workloads: Molecular dynamics, fluid dynamics (CFD), and weather forecasting (e.g., WRF models).

The "Barrier" Problem: In massive clusters using MPI (Message Passing Interface), all CPUs often have to reach a synchronization barrier before the next step of a simulation. If one CPU is delayed by a few milliseconds due to a timer tick, all other thousands of CPUs sit idle waiting for it. nohz_full prevents this "tail latency" from stalling the entire supercomputer.

...

4. Competitive Benchmarking & Kernel Development
Performance engineers use this mode to get "clean" numbers when testing new hardware or compilers.

Workloads: Core-to-core latency tests, cache-bandwidth benchmarks, and standard suites like SPEC CPU.

Goal: Eliminating the "noise" of the operating system so that the results reflect pure hardware performance.

Summary Table: Who uses nohz_full?

User Group       Primary Workload       Why they use it
Quant Firms      High-Frequency Trading To prevent micro-stutter during trade execution.
Research Labs    MPI-based Simulations  To avoid the "slowest node" stalling the whole cluster.
Telcos/ISP       5G/Packet Processing   To ensure wire-speed processing without interrupts.

Hardware Vendors Chip Validation To benchmark CPU performance without OS interference.

Here is how scientific simulations handle system calls:

1. The "Compute-Loop" (Low Syscall)

The core of a simulation (like a GROMACS molecular dynamics step) is just raw math: fetching data from RAM, doing floating-point arithmetic (AVX/SSE), and writing it back.

During the loop: The CPU stays in "Userspace" for millions of cycles without ever asking the kernel for help.

Why it works: Since there are no system calls, nohz_full can successfully turn off the timer tick, allowing the CPU to focus 100% on the math.

2. The "Communication-Phase" (High Syscall)

System calls usually happen only at the end of a computation block, when the simulation needs to talk to other nodes.

The Tools: MPI (Message Passing Interface) uses system calls like write, sendmsg, or specialized RDMA calls to move data across the network.

The Pattern: These simulations follow a "Burst" pattern—long periods of zero system calls (computation) followed by a quick burst of system calls (synchronization).

3. When are they "Syscall Intensive"?

There are specific parts of a simulation that are intensive, but researchers try to minimize them:

I/O Operations: Writing "checkpoints" or large trajectory files to disk (using write()). This is why high-end HPC systems use Asynchronous I/O or dedicated I/O nodes—to keep the compute cores from getting bogged down in system calls.

Memory Allocation: Constantly calling malloc/free involves the brk or mmap system calls. Optimized simulation tools pre-allocate all the memory they need at startup to avoid this.

Quoted 20 lines · Marcelo Tosatti

Quoted 18 lines · Frederic Weisbecker

Quoted 11 lines · Michal Hocko

Quoted 5 lines · Marcelo Tosatti

Michal,

Again, i don't see how moving operations to happen at return to kernel would help (assuming you are talking about "context_tracking,x86: Defer some IPIs until a user->kernel transition").

Nope, I am not talking about IPIs, although those are an example of pcp state as well. I am sorry I do not have a link handy, I am pretty sure Frederic will have that. Another example, though, was vmstat flushes that need to be pcp. There are many other examples.

Here it is:

https://lore.kernel.org/all/[email protected]/

Thanks.

Quoted 6 lines

Frederic,

I think this is a valid solution, however on systems with many CPUs, in nohz_full, performing system calls, can't there be significant increase of lru_lock contention ? Consider 100+ CPUs performing many system calls which add 1 or 2 folios to per-CPU LRU lists.

That's more a question for Michal or Vlastimil.

Quoted 8 lines

There are specific parts of a simulation that are intensive, but researchers try to minimize them:

I/O Operations: Writing "checkpoints" or large trajectory files to disk (using write()). This is why high-end HPC systems use Asynchronous I/O or dedicated I/O nodes—to keep the compute cores from getting bogged down in system calls.

Quoted 3 lines

Memory Allocation: Constantly calling malloc/free involves the brk or mmap system calls. Optimized simulation tools pre-allocate all the memory they need at startup to avoid this.

Ok. I asked a similar question and got this (you made me use an LLM for the first time btw, I held out for 4 years... I'm sure I can wait 4 more years until the next usage :o)

### 2. The "Slow Path" (System Calls / Syscalls)

Passing through the kernel (a syscall) is necessary in certain situations, but it is "expensive" because it forces a **context switch**, which flushes CPU caches.

* **Initialization:** During startup (MPI_Init), many syscalls are used to create sockets, map shared memory (mmap), and configure network interfaces.
* **Standard TCP/IP:** If you are not using a high-performance network (RDMA) but simple Ethernet instead, MPI must call send() and recv(), which are syscalls. The Linux kernel then takes over to manage the TCP/IP stack.
* **Sleep Mode (Blocking):** If an MPI process waits for a message for too long, it may decide to "go to sleep" to yield the CPU to another task via syscalls like futex() or poll().

**In summary:** MPI synchronization aims to be **100% User-Space** (via memory polling) to avoid syscall latency. It is precisely because MPI tries to bypass the kernel that we use nohz_full: we are asking the kernel not to even "knock on the CPU's door" with its clock interruptions.

On Wed, Feb 25, 2026 at 10:49:54PM +0100, Frederic Weisbecker wrote:

<snip>

Quoted 12 lines

Quoted 11 lines

There are specific parts of a simulation that are intensive, but researchers try to minimize them:

I/O Operations: Writing "checkpoints" or large trajectory files to disk (using write()). This is why high-end HPC systems use Asynchronous I/O or dedicated I/O nodes—to keep the compute cores from getting bogged down in system calls.

Memory Allocation: Constantly calling malloc/free involves the brk or mmap system calls. Optimized simulation tools pre-allocate all the memory they need at startup to avoid this.

Quoted 3 lines

Ok. I asked a similar question and got this (you made me use an LLM for the first time btw, I held out for 4 years... I'm sure I can wait 4 more years until the next usage :o)

You should use it more often, it can save a significant amount of time :-)

Quoted 8 lines

### 2. The "Slow Path" (System Calls / Syscalls)

Passing through the kernel (a syscall) is necessary in certain situations, but it is "expensive" because it forces a **context switch**, which flushes CPU caches.

* **Initialization:** During startup (MPI_Init), many syscalls are used to create sockets, map shared memory (mmap), and configure network interfaces.
* **Standard TCP/IP:** If you are not using a high-performance network (RDMA) but simple Ethernet instead, MPI must call send() and recv(), which are syscalls. The Linux kernel then takes over to manage the TCP/IP stack.
* **Sleep Mode (Blocking):** If an MPI process waits for a message for too long, it may decide to "go to sleep" to yield the CPU to another task via syscalls like futex() or poll().

Quoted 1 line

**In summary:** MPI synchronization aims to be **100% User-Space** (via memory polling) to avoid syscall latency. It is precisely because MPI tries to bypass the kernel that we use nohz_full: we are asking the kernel not to even "knock on the CPU's door" with its clock interruptions.

Of course, there is a cost to system calls. However, considering "low latency applications must necessarily remain in userspace, therefore lets optimize only for that case" is limiting IMHO.

Should avoid interruptions whenever possible, for isolated CPUs (in userspace _and_ kernelspace).