Introduce QPW for per-cpu operations (v3)
Subsystems (7)
v3 cover Introduce QPW for per-cpu operations (v3)
The problem:
Some places in the kernel implement a parallel programming strategy consisting on local_locks() for most of the work, and some rare remote operations are scheduled on target cpu. This keeps cache bouncing low since cacheline tends to be mostly local, and avoids the cost of locks in non-RT kernels, even though the very few remote operations will be expensive due to scheduling overhead.
On the other hand, for RT workloads this can represent a problem: getting an important workload scheduled out to deal with remote requests is sure to introduce unexpected deadline misses.
The idea:
Currently with PREEMPT_RT=y, local_locks() become per-cpu spinlocks. In this case, instead of scheduling work on a remote cpu, it should be safe to grab that remote cpu's per-cpu spinlock and run the required work locally. That major cost, which is un/locking in every local function, already happens in PREEMPT_RT.
Also, there is no need to worry about extra cache bouncing:
The cacheline invalidation already happens due to schedule_work_on().
This will avoid schedule_work_on(), and thus avoid scheduling-out an RT workload.
Proposed solution:
A new interface called Queue PerCPU Work (QPW), which should replace Work Queue in the above mentioned use case.
If CONFIG_QPW=n this interfaces just wraps the current local_locks + WorkQueue behavior, so no expected change in runtime.
If CONFIG_QPW=y, and qpw kernel boot option =1, queue_percpu_work_on(cpu,...) will lock that cpu's per-cpu structure and perform work on it locally. This is possible because on functions that can be used for performing remote work on remote per-cpu structures, the local_lock (which is already
a this_cpu spinlock()), will be replaced by a qpw_spinlock(), which is able to get the per_cpu spinlock() for the cpu passed as parameter.
v2->v3:
- Use preempt_disable/preempt_enable on !CONFIG_PREEMPT_RT (Vlastimil Babka).
- Improve documentation to include local_qpw_lock on operations table (Leonardo Bras).
- Enable qpw=1 automatically if CPU isolation is enabled (Vlastimil Babka).
v1->v2:
- Introduce local_qpw_lock and unlock functions, move preempt_disable/ preempt_enable to it (Leonardo Bras). This reduces performance overhead of the patch.
- Documentation and changelog typo fixes (Leonardo Bras).
- Fix places where preempt_disable/preempt_enable was not being correctly performed.
- Add performance measurements.
RFC->v1:
- Introduce CONFIG_QPW and qpw= kernel boot option to enable remote spinlocking and execution even on !CONFIG_PREEMPT_RT kernels (Leonardo Bras).
- Move buffer_head draining to separate workqueue (Marcelo Tosatti).
- Convert mlock per-CPU page lists to QPW (Marcelo Tosatti).
- Drop memcontrol convertion (as isolated CPUs are not targets of queue_work_on anymore).
- Rebase SLUB against Vlastimil's slab/next.
- Add basic document for QPW (Waiman Long).
The performance numbers, as measured by the following test program, are as follows:
CONFIG_PREEMPT_DYNAMIC=y
Unpatched kernel: 60 cycles Patched kernel, CONFIG_QPW=n: 62 cycles Patched kernel, CONFIG_QPW=y, qpw=0: 62 cycles Patched kernel, CONFIG_QPW=y, qpw=1: 75 cycles
CONFIG_PREEMPT_RT:
Unpatched kernel: 95 cycles Patched kernel, CONFIG_QPW=y, qpw=0: 99 cycles Patched kernel, CONFIG_QPW=y, qpw=1: 97 cycles
kmalloc_bench.c:
#include <linux/module.h>
#include <linux/kernel.h>
#include <linux/slab.h>
#include <linux/timex.h>
#include <linux/preempt.h>
#include <linux/irqflags.h>
#include <linux/vmalloc.h>
MODULE_LICENSE("GPL");MODULE_AUTHOR("Gemini AI");MODULE_DESCRIPTION("A simple kmalloc performance benchmark");
static int size = 64; // Default allocation size in bytes module_param(size, int, 0644);
static int iterations = 9000000; // Default number of iterations module_param(iterations, int, 0644);
static int __init kmalloc_bench_init(void) {
void **ptrs;
cycles_t start, end;
uint64_t total_cycles;
int i;
pr_info("kmalloc_bench: Starting test (size=%d, iterations=%d)\n", size, iterations);
// Allocate an array to store pointers to avoid immediate kfree-reuse optimization
ptrs = vmalloc(sizeof(void *) * iterations);
if (!ptrs) {
pr_err("kmalloc_bench: Failed to allocate pointer array\n");
return -ENOMEM;
}
preempt_disable();
start = get_cycles();
for (i = 0; i < iterations; i++) {
ptrs[i] = kmalloc(size, GFP_ATOMIC);
}
end = get_cycles();
total_cycles = end - start;
preempt_enable();
pr_info("kmalloc_bench: Total cycles for %d allocs: %llu\n", iterations, total_cycles);
pr_info("kmalloc_bench: Avg cycles per kmalloc: %llu\n", total_cycles / iterations);
// Cleanup
for (i = 0; i < iterations; i++) {
kfree(ptrs[i]);
}
vfree(ptrs);
return 0;
}static void __exit kmalloc_bench_exit(void) {
pr_info("kmalloc_bench: Module unloaded\n");
}module_init(kmalloc_bench_init);module_exit(kmalloc_bench_exit);
The following testcase triggers lru_add_drain_all on an isolated CPU (that does sys_write to a file before entering its realtime loop).
/*
* Simulates a low latency loop program that is interrupted
* due to lru_add_drain_all. To trigger lru_add_drain_all, run:
*
* blockdev --flushbufs /dev/sdX
*
*/
#define _GNU_SOURCE
#include <fcntl.h>
#include <stdio.h>
#include <stdlib.h>
#include <sys/mman.h>
#include <string.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <stdlib.h>
#include <stdarg.h>
#include <pthread.h>
#include <sched.h>
#include <unistd.h>
int cpu;
static void *run(void *arg)
{
pthread_t current_thread;
cpu_set_t cpuset;
int ret, nrloops;
struct sched_param sched_p;
pid_t pid;
int fd;
char buf[] = "xxxxxxxxxxx";
CPU_ZERO(&cpuset);
CPU_SET(cpu, &cpuset);
current_thread = pthread_self();
ret = pthread_setaffinity_np(current_thread, sizeof(cpu_set_t), &cpuset);
if (ret) {
perror("pthread_setaffinity_np failed\n");
exit(0);
}
memset(&sched_p, 0, sizeof(struct sched_param));
sched_p.sched_priority = 1;
pid = gettid();
ret = sched_setscheduler(pid, SCHED_FIFO, &sched_p);
if (ret) {
perror("sched_setscheduler");
exit(0);
}
fd = open("/tmp/tmpfile", O_RDWR|O_CREAT|O_TRUNC);
if (fd == -1) {
perror("open");
exit(0);
}
ret = write(fd, buf, sizeof(buf));
if (ret == -1) {
perror("write");
exit(0);
}
do {
nrloops = nrloops+2;
nrloops--;
} while (1);
}int main(int argc, char *argv[])
{
int fd, ret;
pthread_t thread;
long val;
char *endptr, *str;
struct sched_param sched_p;
pid_t pid;
if (argc != 2) {
printf("usage: %s cpu-nr\n", argv[0]);
printf("where CPU number is the CPU to pin thread to\n");
exit(0);
}
str = argv[1];
cpu = strtol(str, &endptr, 10);
if (cpu < 0) {
printf("strtol returns %d\n", cpu);
exit(0);
}
printf("cpunr=%d\n", cpu);
memset(&sched_p, 0, sizeof(struct sched_param));
sched_p.sched_priority = 1;
pid = getpid();
ret = sched_setscheduler(pid, SCHED_FIFO, &sched_p);
if (ret) {
perror("sched_setscheduler");
exit(0);
}
pthread_create(&thread, NULL, run, NULL);
sleep(5000);
pthread_join(thread, NULL);
}PATCH v3 1/4 Introducing qpw_lock() and per-cpu queue & flush work
Some places in the kernel implement a parallel programming strategy consisting on local_locks() for most of the work, and some rare remote operations are scheduled on target cpu. This keeps cache bouncing low since cacheline tends to be mostly local, and avoids the cost of locks in non-RT kernels, even though the very few remote operations will be expensive due to scheduling overhead.
On the other hand, for RT workloads this can represent a problem:
scheduling work on remote cpu that are executing low latency tasks is undesired and can introduce unexpected deadline misses.
It's interesting, though, that local_lock()s in RT kernels become spinlock(). We can make use of those to avoid scheduling work on a remote cpu by directly updating another cpu's per_cpu structure, while holding it's spinlock().
In order to do that, it's necessary to introduce a new set of functions to make it possible to get another cpu's per-cpu "local" lock (qpw_{un,}lock*) and also the corresponding queue_percpu_work_on() and flush_percpu_work() helpers to run the remote work.
Users of non-RT kernels but with low latency requirements can select similar functionality by using the CONFIG_QPW compile time option.
On CONFIG_QPW disabled kernels, no changes are expected, as every one of the introduced helpers work the exactly same as the current implementation:
qpw_{un,}lock*() -> local_{un,}lock*() (ignores cpu parameter)
queue_percpu_work_on() -> queue_work_on()
flush_percpu_work() -> flush_work()For QPW enabled kernels, though, qpw_{un,}lock*() will use the extra cpu parameter to select the correct per-cpu structure to work on, and acquire the spinlock for that cpu.
queue_percpu_work_on() will just call the requested function in the current cpu, which will operate in another cpu's per-cpu object. Since the local_locks() become spinlock()s in QPW enabled kernels, we are safe doing that.
flush_percpu_work() then becomes a no-op since no work is actually scheduled on a remote cpu.
Some minimal code rework is needed in order to make this mechanism work:
The calls for local_{un,}lock*() on the functions that are currently scheduled on remote cpus need to be replaced by qpw_{un,}lock_n*(), so in QPW enabled kernels they can reference a different cpu. It's also necessary to use a qpw_struct instead of a work_struct, but it just contains a work struct and, in CONFIG_QPW, the target cpu.
This should have almost no impact on non-CONFIG_QPW kernels: few this_cpu_ptr() will become per_cpu_ptr(,smp_processor_id()).
On CONFIG_QPW kernels, this should avoid deadlines misses by removing scheduling noise.
--- Documentation/admin-guide/kernel-parameters.txt | 10 Documentation/locking/qpwlocks.rst | 70 ++++++ MAINTAINERS | 7 include/linux/qpw.h | 256 ++++++++++++++++++++++++ init/Kconfig | 35 +++ kernel/Makefile | 2 kernel/qpw.c | 26 ++ 7 files changed, 406 insertions(+) create mode 100644 include/linux/qpw.h create mode 100644 kernel/qpw.c
Documentation/admin-guide/kernel-parameters.txt +10 -0
@@ -2841,6 +2841,16 @@ Kernel parameters The format of <cpu-list> is described above. + qpw= [KNL,SMP] Select a behavior on per-CPU resource sharing + and remote interference mechanism on a kernel built with + CONFIG_QPW. + Format: { "0" | "1" } + 0 - local_lock() + queue_work_on(remote_cpu) + 1 - spin_lock() for both local and remote operations + + Selecting 1 may be interesting for systems that want + to avoid interruption & context switches from IPIs. + iucv= [HW,NET] ivrs_ioapic [HW,X86-64]
MAINTAINERS +7 -0
@@ -21536,6 +21536,13 @@ F: Documentation/networking/device_drive F: drivers/bus/fsl-mc/ F: include/uapi/linux/fsl_mc.h +QPW +M: Leonardo Bras <[email protected]> +S: Supported +F: Documentation/locking/qpwlocks.rst +F: include/linux/qpw.h +F: kernel/qpw.c + QT1010 MEDIA DRIVER L: [email protected] S: Orphan
include/linux/qpw.h new file +264 -0
264 lines not on this page · raw message
init/Kconfig +35 -0
@@ -762,6 +762,41 @@ config CPU_ISOLATION Say Y if unsure. +config QPW + bool "Queue per-CPU Work" + depends on SMP || COMPILE_TEST + default n + help + Allow changing the behavior on per-CPU resource sharing with cache, + from the regular local_locks() + queue_work_on(remote_cpu) to using + per-CPU spinlocks on both local and remote operations. + + This is useful to give user the option on reducing IPIs to CPUs, and + thus reduce interruptions and context switches. On the other hand, it + increases generated code and will use atomic operations if spinlocks + are selected. + + If set, will use the default behavior set in QPW_DEFAULT unless boot + parameter qpw is passed with a different behavior. + + If unset, will use the local_lock() + queue_work_on() strategy, + regardless of the boot parameter or QPW_DEFAULT. + + Say N if unsure. + +config QPW_DEFAULT + bool "Use per-CPU spinlocks by default" + depends on QPW + default n + help + If set, will use per-CPU spinlocks as default behavior for per-CPU + remote operations. + + If unset, will use local_lock() + queue_work_on(cpu) as default + behavior for remote operations. + + Say N if unsure + source "kernel/rcu/Kconfig" config IKCONFIG
kernel/Makefile +2 -0
@@ -142,6 +142,8 @@ obj-$(CONFIG_WATCH_QUEUE) += watch_queue obj-$(CONFIG_RESOURCE_KUNIT_TEST) += resource_kunit.o obj-$(CONFIG_SYSCTL_KUNIT_TEST) += sysctl-test.o +obj-$(CONFIG_QPW) += qpw.o + CFLAGS_kstack_erase.o += $(DISABLE_KSTACK_ERASE) CFLAGS_kstack_erase.o += $(call cc-option,-mgeneral-regs-only) obj-$(CONFIG_KSTACK_ERASE) += kstack_erase.o
kernel/qpw.c new file +47 -0
47 lines not on this page · raw message
Documentation/locking/qpwlocks.rst new file +76 -0
76 lines not on this page · raw message
Re: [PATCH v3 1/4] Introducing qpw_lock() and per-cpu queue & flush work
Quoted 202 lines · Marcelo Tosatti
Some places in the kernel implement a parallel programming strategy consisting on
local_locks()for most of the work, and some rare remote operations are scheduled on target cpu. This keeps cache bouncing low since cacheline tends to be mostly local, and avoids the cost of locks in non-RT kernels, even though the very few remote operations will be expensive due to scheduling overhead.On the other hand, for RT workloads this can represent a problem:
scheduling work on remote cpu that are executing low latency tasks is undesired and can introduce unexpected deadline misses.It's interesting, though, that local_lock()s in RT kernels become
spinlock(). We can make use of those to avoid scheduling work on a remote cpu by directly updating another cpu'sper_cpustructure, while holding it'sspinlock().In order to do that, it's necessary to introduce a new set of functions to make it possible to get another cpu's per-cpu "local" lock (qpw_{un,}lock*) and also the corresponding
queue_percpu_work_on()andflush_percpu_work()helpers to run the remote work.Users of non-RT kernels but with low latency requirements can select similar functionality by using the
CONFIG_QPWcompile time option.On
CONFIG_QPWdisabled kernels, no changes are expected, as every one of the introduced helpers work the exactly same as the current implementation:qpw_{un,}lock*() -> local_{un,}lock*() (ignores cpu parameter) queue_percpu_work_on() -> queue_work_on() flush_percpu_work() -> flush_work()For QPW enabled kernels, though, qpw_{un,}lock*() will use the extra cpu parameter to select the correct per-cpu structure to work on, and acquire the spinlock for that cpu.
queue_percpu_work_on()will just call the requested function in the current cpu, which will operate in another cpu's per-cpu object. Since thelocal_locks()become spinlock()s in QPW enabled kernels, we are safe doing that.
flush_percpu_work()then becomes a no-op since no work is actually scheduled on a remote cpu.Some minimal code rework is needed in order to make this mechanism work:
The calls for local_{un,}lock*() on the functions that are currently scheduled on remote cpus need to be replaced by qpw_{un,}lock_n*(), so in QPW enabled kernels they can reference a different cpu. It's also necessary to use aqpw_structinstead of awork_struct, but it just contains a work struct and, inCONFIG_QPW, the target cpu.This should have almost no impact on non-CONFIG_QPW kernels: few
this_cpu_ptr()will become per_cpu_ptr(,smp_processor_id()).On
CONFIG_QPWkernels, this should avoid deadlines misses by removing scheduling noise.Signed-off-by Leonardo Bras <[email protected]>Signed-off-by Marcelo Tosatti <[email protected]>--- Documentation/admin-guide/kernel-parameters.txt | 10 Documentation/locking/qpwlocks.rst | 70 ++++++ MAINTAINERS | 7 include/linux/qpw.h | 256 ++++++++++++++++++++++++ init/Kconfig | 35 +++ kernel/Makefile | 2 kernel/qpw.c | 26 ++ 7 files changed, 406 insertions(+) create mode 100644 include/linux/qpw.h create mode 100644 kernel/qpw.cIndex:
linux/Documentation/admin-guide/kernel-parameters.txt
===================================================================Documentation/admin-guide/kernel-parameters.txt +10
-0@@ -2841,6 +2841,16 @@ Kernel parameters The format of <cpu-list> is described above. + qpw= [KNL,SMP] Select a behavior on per-CPU resource sharing + and remote interference mechanism on a kernel built with + CONFIG_QPW. + Format: { "0" | "1" } + 0 - local_lock() + queue_work_on(remote_cpu) + 1 - spin_lock() for both local and remote operations + + Selecting 1 may be interesting for systems that want + to avoid interruption & context switches from IPIs. + iucv= [HW,NET] ivrs_ioapic [HW,X86-64]Index: linux/MAINTAINERS
===================================================================MAINTAINERS +7
-0@@ -21536,6 +21536,13 @@ F: Documentation/networking/device_drive F: drivers/bus/fsl-mc/ F: include/uapi/linux/fsl_mc.h +QPW +M: Leonardo Bras <[email protected]> +S: Supported +F: Documentation/locking/qpwlocks.rst +F: include/linux/qpw.h +F: kernel/qpw.c + QT1010 MEDIA DRIVER L: [email protected] S: OrphanIndex:
linux/include/linux/qpw.h
===================================================================include/linux/qpw.h new file +88
-0@@ -0,0 +1,264 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _LINUX_QPW_H +#define _LINUX_QPW_H + +#include "linux/spinlock.h" +#include "linux/local_lock.h" +#include "linux/workqueue.h" + +#ifndef CONFIG_QPW + +typedef local_lock_t qpw_lock_t; +typedef local_trylock_t qpw_trylock_t; + +struct qpw_struct { + struct work_struct work; +}; + +#define qpw_lock_init(lock) \ + local_lock_init(lock) + +#define qpw_trylock_init(lock) \ + local_trylock_init(lock) + +#define qpw_lock(lock, cpu) \ + local_lock(lock) + +#define local_qpw_lock(lock) \ + local_lock(lock) + +#define qpw_lock_irqsave(lock, flags, cpu) \ + local_lock_irqsave(lock, flags) + +#define local_qpw_lock_irqsave(lock, flags) \ + local_lock_irqsave(lock, flags) + +#define qpw_trylock(lock, cpu) \ + local_trylock(lock) + +#define local_qpw_trylock(lock) \ + local_trylock(lock) + +#define qpw_trylock_irqsave(lock, flags, cpu) \ + local_trylock_irqsave(lock, flags) + +#define qpw_unlock(lock, cpu) \ + local_unlock(lock) + +#define local_qpw_unlock(lock) \ + local_unlock(lock) + +#define qpw_unlock_irqrestore(lock, flags, cpu) \ + local_unlock_irqrestore(lock, flags) + +#define local_qpw_unlock_irqrestore(lock, flags) \ + local_unlock_irqrestore(lock, flags) + +#define qpw_lockdep_assert_held(lock) \ + lockdep_assert_held(lock) + +#define queue_percpu_work_on(c, wq, qpw) \ + queue_work_on(c, wq, &(qpw)->work) + +#define flush_percpu_work(qpw) \ + flush_work(&(qpw)->work) + +#define qpw_get_cpu(qpw) smp_processor_id() + +#define qpw_is_cpu_remote(cpu) (false) + +#define INIT_QPW(qpw, func, c) \ + INIT_WORK(&(qpw)->work, (func)) + +#else /* CONFIG_QPW */ + +DECLARE_STATIC_KEY_MAYBE(CONFIG_QPW_DEFAULT, qpw_sl); + +typedef union { + spinlock_t sl; + local_lock_t ll; +} qpw_lock_t; + +typedef union { + spinlock_t sl; + local_trylock_t ll; +} qpw_trylock_t; + +struct qpw_struct { + struct work_struct work;
include/linux/qpw.h line 98
Quoted 10 lines
include/linux/qpw.h new file +10
-0+int cpu; +}; + +#ifdef CONFIG_PREEMPT_RT +#define preempt_or_migrate_disable migrate_disable +#define preempt_or_migrate_enable migrate_enable +#else +#define preempt_or_migrate_disable preempt_disable +#define preempt_or_migrate_enable preempt_enable +#endif
Nice!
Quoted 267 lines
include/linux/qpw.h new file +166
-0+ +#define qpw_lock_init (lock) \ + do { \ + if (static_branch_maybe(CONFIG_QPW_DEFAULT, &qpw_sl)) \ + spin_lock_init(lock.sl); \ + else \ + local_lock_init(lock.ll); \ + } while (0) + +#define qpw_trylock_init (lock) \ + do { \ + if (static_branch_maybe(CONFIG_QPW_DEFAULT, &qpw_sl)) \ + spin_lock_init(lock.sl); \ + else \ + local_trylock_init(lock.ll); \ + } while (0) + +#define qpw_lock (lock, cpu) \ + do { \ + if (static_branch_maybe(CONFIG_QPW_DEFAULT, &qpw_sl)) \ + spin_lock(per_cpu_ptr(lock.sl, cpu)); \ + else \ + local_lock(lock.ll); \ + } while (0) + +#define local_qpw_lock (lock) \ + do { \ + if (static_branch_maybe(CONFIG_QPW_DEFAULT, &qpw_sl)) { \ + preempt_or_migrate_disable(); \ + spin_lock(this_cpu_ptr(lock.sl)); \ + } else \ + local_lock(lock.ll); \ + } while (0) + +#define qpw_lock_irqsave (lock, flags, cpu) \ + do { \ + if (static_branch_maybe(CONFIG_QPW_DEFAULT, &qpw_sl)) \ + spin_lock_irqsave(per_cpu_ptr(lock.sl, cpu), flags); \ + else \ + local_lock_irqsave(lock.ll, flags); \ + } while (0) + +#define local_qpw_lock_irqsave (lock, flags) \ + do { \ + if (static_branch_maybe(CONFIG_QPW_DEFAULT, &qpw_sl)) { \ + preempt_or_migrate_disable(); \ + spin_lock_irqsave(this_cpu_ptr(lock.sl), flags); \ + } else \ + local_lock_irqsave(lock.ll, flags); \ + } while (0) + + +#define qpw_trylock (lock, cpu) \ + ({ \ + int t; \ + if (static_branch_maybe(CONFIG_QPW_DEFAULT, &qpw_sl)) \ + t = spin_trylock(per_cpu_ptr(lock.sl, cpu)); \ + else \ + t = local_trylock(lock.ll); \ + t; \ + }) + +#define local_qpw_trylock (lock) \ + ({ \ + int t; \ + if (static_branch_maybe(CONFIG_QPW_DEFAULT, &qpw_sl)) { \ + preempt_or_migrate_disable(); \ + t = spin_trylock(this_cpu_ptr(lock.sl)); \ + if (!t) \ + preempt_or_migrate_enable(); \ + } else \ + t = local_trylock(lock.ll); \ + t; \ + }) + +#define qpw_trylock_irqsave (lock, flags, cpu) \ + ({ \ + int t; \ + if (static_branch_maybe(CONFIG_QPW_DEFAULT, &qpw_sl)) \ + t = spin_trylock_irqsave(per_cpu_ptr(lock.sl, cpu), flags); \ + else \ + t = local_trylock_irqsave(lock.ll, flags); \ + t; \ + }) + +#define qpw_unlock (lock, cpu) \ + do { \ + if (static_branch_maybe(CONFIG_QPW_DEFAULT, &qpw_sl)) { \ + spin_unlock(per_cpu_ptr(lock.sl, cpu)); \ + } else { \ + local_unlock(lock.ll); \ + } \ + } while (0) + +#define local_qpw_unlock (lock) \ +do { \ + if (static_branch_maybe(CONFIG_QPW_DEFAULT, &qpw_sl)) { \ + spin_unlock(this_cpu_ptr(lock.sl)); \ + preempt_or_migrate_enable(); \ + } else { \ + local_unlock(lock.ll); \ + } \ +} while (0) + +#define qpw_unlock_irqrestore (lock, flags, cpu) \ + do { \ + if (static_branch_maybe(CONFIG_QPW_DEFAULT, &qpw_sl)) \ + spin_unlock_irqrestore(per_cpu_ptr(lock.sl, cpu), flags); \ + else \ + local_unlock_irqrestore(lock.ll, flags); \ + } while (0) + +#define local_qpw_unlock_irqrestore (lock, flags) \ + do { \ + if (static_branch_maybe(CONFIG_QPW_DEFAULT, &qpw_sl)) { \ + spin_unlock_irqrestore(this_cpu_ptr(lock.sl), flags); \ + preempt_or_migrate_enable(); \ + } else \ + local_unlock_irqrestore(lock.ll, flags); \ + } while (0) + +#define qpw_lockdep_assert_held (lock) \ + do { \ + if (static_branch_maybe(CONFIG_QPW_DEFAULT, &qpw_sl)) \ + lockdep_assert_held(this_cpu_ptr(lock.sl)); \ + else \ + lockdep_assert_held(this_cpu_ptr(lock.ll)); \ + } while (0) + +#define queue_percpu_work_on (c, wq, qpw) \ + do { \ + int __c = c; \ + struct qpw_struct *__qpw = (qpw); \ + if (static_branch_maybe(CONFIG_QPW_DEFAULT, &qpw_sl)) { \ + WARN_ON((__c) != __qpw->cpu); \ + __qpw->work.func(&__qpw->work); \ + } else { \ + queue_work_on(__c, wq, &(__qpw)->work); \ + } \ + } while (0) + +/* +* Does nothing if QPW is set to use spinlock, as the task is already done at the +* time queue_percpu_work_on() returns. +*/ +#define flush_percpu_work (qpw) \ + do { \ + struct qpw_struct *__qpw = (qpw); \ + if (!static_branch_maybe(CONFIG_QPW_DEFAULT, &qpw_sl)) { \ + flush_work(&__qpw->work); \ + } \ + } while (0) + +#define qpw_get_cpu (w) container_of((w), struct qpw_struct, work)->cpu + +#define qpw_is_cpu_remote (cpu) ((cpu) != smp_processor_id()) + +#define INIT_QPW (qpw, func, c) \ + do { \ + struct qpw_struct *__qpw = (qpw); \ + INIT_WORK(&__qpw->work, (func)); \ + __qpw->cpu = (c); \ + } while (0) + +#endif /* CONFIG_QPW */ +#endif /* LINUX_QPW_H */ Index: linux/init/Kconfig
===================================================================init/Kconfig +35
-0@@ -762,6 +762,41 @@ config CPU_ISOLATION Say Y if unsure. +config QPW + bool "Queue per-CPU Work" + depends on SMP || COMPILE_TEST + default n + help + Allow changing the behavior on per-CPU resource sharing with cache, + from the regular local_locks() + queue_work_on(remote_cpu) to using + per-CPU spinlocks on both local and remote operations. + + This is useful to give user the option on reducing IPIs to CPUs, and + thus reduce interruptions and context switches. On the other hand, it + increases generated code and will use atomic operations if spinlocks + are selected. + + If set, will use the default behavior set in QPW_DEFAULT unless boot + parameter qpw is passed with a different behavior. + + If unset, will use the local_lock() + queue_work_on() strategy, + regardless of the boot parameter or QPW_DEFAULT. + + Say N if unsure. + +config QPW_DEFAULT + bool "Use per-CPU spinlocks by default" + depends on QPW + default n + help + If set, will use per-CPU spinlocks as default behavior for per-CPU + remote operations. + + If unset, will use local_lock() + queue_work_on(cpu) as default + behavior for remote operations. + + Say N if unsure + source "kernel/rcu/Kconfig" config IKCONFIGIndex: linux/kernel/Makefile
===================================================================kernel/Makefile +2
-0@@ -142,6 +142,8 @@ obj-$(CONFIG_WATCH_QUEUE) += watch_queue obj-$(CONFIG_RESOURCE_KUNIT_TEST) += resource_kunit.o obj-$(CONFIG_SYSCTL_KUNIT_TEST) += sysctl-test.o +obj-$(CONFIG_QPW) += qpw.o + CFLAGS_kstack_erase.o += $(DISABLE_KSTACK_ERASE) CFLAGS_kstack_erase.o += $(call cc-option,-mgeneral-regs-only) obj-$(CONFIG_KSTACK_ERASE) += kstack_erase.oIndex:
linux/kernel/qpw.c
===================================================================kernel/qpw.c new file +37
-0@@ -0,0 +1,47 @@ +// SPDX-License-Identifier: GPL-2.0 +#include "linux/export.h" +#include <linux/sched.h> +#include <linux/qpw.h> +#include <linux/string.h> +#include <linux/sched/isolation.h> + +DEFINE_STATIC_KEY_MAYBE (CONFIG_QPW_DEFAULT , qpw_sl); +EXPORT_SYMBOL (qpw_sl); + +static bool qpw_param_specified; + +static int __init qpw_setup (char *str) +{ +int opt; + +if (!get_option (&str, &opt)) { +pr_warn ("QPW: invalid qpw parameter: %s, ignoring.\n" , str); +return 0 ; + } + +if (opt) +static_branch_enable (&qpw_sl); +else +static_branch_disable (&qpw_sl); + + qpw_param_specified = true; + +return 1 ; +} +__setup ("qpw=" , qpw_setup); + +/* +* Enable QPW if CPUs want to avoid kernel noise. +*/ +static int __init qpw_init (void ) +{
kernel/qpw.c line 47
Quoted 10 lines
kernel/qpw.c new file +10
-0+if (qpw_param_specified == true) +return 0 ; + +if (housekeeping_enabled (HK_TYPE_KERNEL_NOISE )) +static_branch_enable (&qpw_sl); + +return 0 ; +} + +late_initcall (qpw_init);
Awesome! Clean and efficient!
Quoted 71 lines
Index:
linux/Documentation/locking/qpwlocks.rst
===================================================================Documentation/locking/qpwlocks.rst new file +66
-0@@ -0,0 +1,76 @@ +.. SPDX-License-Identifier: GPL-2.0 + +========= +QPW locks +========= + +Some places in the kernel implement a parallel programming strategy +consisting on local_locks() for most of the work, and some rare remote +operations are scheduled on target cpu. This keeps cache bouncing low since +cacheline tends to be mostly local, and avoids the cost of locks in non-RT +kernels, even though the very few remote operations will be expensive due +to scheduling overhead. + +On the other hand, for RT workloads this can represent a problem: +scheduling work on remote cpu that are executing low latency tasks +is undesired and can introduce unexpected deadline misses. + +QPW locks help to convert sites that use local_locks (for cpu local operations) +and queue_work_on (for queueing work remotely, to be executed +locally on the owner cpu of the lock) to QPW locks. + +The lock is declared qpw_lock_t type. +The lock is initialized with qpw_lock_init. +The lock is locked with qpw_lock (takes a lock and cpu as a parameter). +The lock is unlocked with qpw_unlock (takes a lock and cpu as a parameter). + +The qpw_lock_irqsave function disables interrupts and saves current interrupt state, +cpu as a parameter. + +For trylock variant, there is the qpw_trylock_t type, initialized with +qpw_trylock_init. Then the corresponding qpw_trylock and +qpw_trylock_irqsave. + +work_struct should be replaced by qpw_struct, which contains a cpu parameter +(owner cpu of the lock), initialized by INIT_QPW. + +The queue work related functions (analogous to queue_work_on and flush_work) are: +queue_percpu_work_on and flush_percpu_work. + +The behaviour of the QPW functions is as follows: + +* !CONFIG_QPW (or CONFIG_QPW and qpw=off kernel boot parameter): + - qpw_lock: local_lock + - qpw_lock_irqsave: local_lock_irqsave + - qpw_trylock: local_trylock + - qpw_trylock_irqsave: local_trylock_irqsave + - qpw_unlock: local_unlock + - local_qpw_lock: local_lock + - local_qpw_trylock: local_trylock + - local_qpw_unlock: local_unlock + - queue_percpu_work_on: queue_work_on + - flush_percpu_work: flush_work + +* CONFIG_QPW (and CONFIG_QPW_DEFAULT=y or qpw=on kernel boot parameter), + - qpw_lock: spin_lock + - qpw_lock_irqsave: spin_lock_irqsave + - qpw_trylock: spin_trylock + - qpw_trylock_irqsave: spin_trylock_irqsave + - qpw_unlock: spin_unlock + - local_qpw_lock: preempt_disable OR migrate_disable + spin_lock + - local_qpw_trylock: preempt_disable OR migrate_disable + spin_trylock + - local_qpw_unlock: preempt_enable OR migrate_enable + spin_unlock + - queue_percpu_work_on: executes work function on caller cpu + - flush_percpu_work: empty + +qpw_get_cpu(work_struct), to be called from within qpw work function,
Documentation/locking/qpwlocks.rst line 76
Quoted 10 lines
Documentation/locking/qpwlocks.rst new file +10
-0+returns the target cpu. + +In addition to the locking functions above, there are the local locking +functions (local_qpw_lock, local_qpw_trylock and local_qpw_unlock). +These must only be used to access per-CPU data from the CPU that owns +that data, and not remotely. They disable preemption or migration +and don't require a cpu parameter. + +These should only be used when accessing per-CPU data of the local CPU. +
Awesome!
Thanks for this new version Marcelo!
Leo
PATCH v3 2/4 mm/swap: move bh draining into a separate workqueue
In reply to Marcelo TosattiSeparate the bh draining into a separate workqueue (from the mm lru draining), so that its possible to switch the mm lru draining to QPW.
To switch bh draining to QPW, it would be necessary to add a spinlock to addition of bhs to percpu cache, and that is a very hot path.
--- mm/swap.c | 52 +++++++++++++++++++++++++++++++++++++--------------- 1 file changed, 37 insertions(+), 15 deletions(-)
mm/swap.c +37 -15
@@ -745,12 +745,11 @@ void lru_add_drain(void)* the same cpu. It shouldn't be a problem in !SMP case since * the core is only one and the locks will disable preemption. */ -static void lru_add_and_bh_lrus_drain (void ) +static void lru_add_mm_drain (void ) {local_lock (&cpu_fbatches.lock);lru_add_drain_cpu (smp_processor_id ());local_unlock (&cpu_fbatches.lock); -invalidate_bh_lrus_cpu ();mlock_drain_local (); }
@@ -769,10 +768,17 @@ static DEFINE_PER_CPU(struct work_structstatic void lru_add_drain_per_cpu (struct work_struct *dummy) { -lru_add_and_bh_lrus_drain (); +lru_add_mm_drain (); } -static bool cpu_needs_drain (unsigned int cpu) +static DEFINE_PER_CPU(struct work_struct, bh_add_drain_work); + +static void bh_add_drain_per_cpu (struct work_struct *dummy) +{ +invalidate_bh_lrus_cpu (); +} + +static bool cpu_needs_mm_drain (unsigned int cpu) {struct cpu_fbatches *fbatches = &per_cpu (cpu_fbatches, cpu);
@@ -783,8 +789,12 @@ static bool cpu_needs_drain(unsigned intfolio_batch_count (&fbatches->lru_deactivate) ||folio_batch_count (&fbatches->lru_lazyfree) ||folio_batch_count (&fbatches->lru_activate) || -need_mlock_drain (cpu) || -has_bh_in_lru (cpu,NULL ); +need_mlock_drain (cpu); +} + +static bool cpu_needs_bh_drain (unsigned int cpu) +{ +return has_bh_in_lru (cpu,NULL ); }/*
@@ -807,7 +817,7 @@ static inline void __lru_add_drain_all(b* each CPU. */ static unsigned int lru_drain_gen; -static struct cpumask has_work; +static struct cpumask has_mm_work, has_bh_work;static DEFINE_MUTEX (lock);unsigned cpu, this_gen;
@@ -870,20 +880,31 @@ static inline void __lru_add_drain_all(bWRITE_ONCE (lru_drain_gen, lru_drain_gen +1 );smp_mb (); -cpumask_clear (&has_work); +cpumask_clear (&has_mm_work); +cpumask_clear (&has_bh_work);for_each_online_cpu (cpu) { -struct work_struct *work = &per_cpu (lru_add_drain_work, cpu); +struct work_struct *mm_work = &per_cpu (lru_add_drain_work, cpu); +struct work_struct *bh_work = &per_cpu (bh_add_drain_work, cpu); + +if (cpu_needs_mm_drain (cpu)) { +INIT_WORK (mm_work, lru_add_drain_per_cpu); +queue_work_on (cpu, mm_percpu_wq, mm_work); +__cpumask_set_cpu (cpu, &has_mm_work); + } -if (cpu_needs_drain (cpu)) { -INIT_WORK (work, lru_add_drain_per_cpu); -queue_work_on (cpu, mm_percpu_wq, work); -__cpumask_set_cpu (cpu, &has_work); +if (cpu_needs_bh_drain (cpu)) { +INIT_WORK (bh_work, bh_add_drain_per_cpu); +queue_work_on (cpu, mm_percpu_wq, bh_work); +__cpumask_set_cpu (cpu, &has_bh_work); } } -for_each_cpu (cpu, &has_work) +for_each_cpu (cpu, &has_mm_work)flush_work (&per_cpu (lru_add_drain_work, cpu)); +for_each_cpu (cpu, &has_bh_work) +flush_work (&per_cpu (bh_add_drain_work, cpu)); + done:mutex_unlock (&lock); }
@@ -929,7 +950,8 @@ void lru_cache_disable(void)#ifdef CONFIG_SMP __lru_add_drain_all (true);#else -lru_add_and_bh_lrus_drain (); +lru_add_mm_drain (); +invalidate_bh_lrus_cpu ();#endif }
PATCH v3 3/4 swap: apply new queue_percpu_work_on() interface
In reply to Marcelo TosattiMake use of the new qpw_{un,}lock*() and queue_percpu_work_on() interface to improve performance & latency.
For functions that may be scheduled in a different cpu, replace local_{un,}lock*() by qpw_{un,}lock*(), and replace schedule_work_on() by queue_percpu_work_on(). The same happens for flush_work() and flush_percpu_work().
The change requires allocation of qpw_structs instead of a work_structs, and changing parameters of a few functions to include the cpu parameter.
This should bring no relevant performance impact on non-QPW kernels:
For functions that may be scheduled in a different cpu, the local_*lock's this_cpu_ptr() becomes a per_cpu_ptr(smp_processor_id()).
--- mm/internal.h | 4 ++- mm/mlock.c | 51 ++++++++++++++++++++++++++++++----------- mm/page_alloc.c | 2 - mm/swap.c | 69 ++++++++++++++++++++++++++++++-------------------------- 4 files changed, 79 insertions(+), 47 deletions(-)
mm/mlock.c +38 -13
@@ -25,17 +25,16 @@#include <linux/memcontrol.h> #include <linux/mm_inline.h> #include <linux/secretmem.h> +#include <linux/qpw.h> #include "internal.h" struct mlock_fbatch { -local_lock_t lock; +qpw_lock_t lock;struct folio_batch fbatch; }; -static DEFINE_PER_CPU(struct mlock_fbatch, mlock_fbatch) = { - .lock =INIT_LOCAL_LOCK (lock), -}; +static DEFINE_PER_CPU(struct mlock_fbatch, mlock_fbatch);bool can_do_mlock (void ) {
@@ -209,18 +208,29 @@ static void mlock_folio_batch(struct folfolios_put (fbatch); } +void mlock_drain_cpu (int cpu) +{ +struct folio_batch *fbatch; + +qpw_lock (&mlock_fbatch.lock, cpu); + fbatch =per_cpu_ptr (&mlock_fbatch.fbatch, cpu); +if (folio_batch_count (fbatch)) +mlock_folio_batch (fbatch); +qpw_unlock (&mlock_fbatch.lock, cpu); +} +void mlock_drain_local (void ) {struct folio_batch *fbatch; -local_lock (&mlock_fbatch.lock); +local_qpw_lock (&mlock_fbatch.lock); fbatch =this_cpu_ptr (&mlock_fbatch.fbatch);if (folio_batch_count (fbatch))mlock_folio_batch (fbatch); -local_unlock (&mlock_fbatch.lock); +local_qpw_unlock (&mlock_fbatch.lock); } -void mlock_drain_remote (int cpu) +void mlock_drain_offline (int cpu) {struct folio_batch *fbatch;
@@ -243,7 +253,7 @@ void mlock_folio(struct folio *folio) {struct folio_batch *fbatch; -local_lock (&mlock_fbatch.lock); +local_qpw_lock (&mlock_fbatch.lock); fbatch =this_cpu_ptr (&mlock_fbatch.fbatch);if (!folio_test_set_mlocked (folio)) {
@@ -257,7 +267,7 @@ void mlock_folio(struct folio *folio)if (!folio_batch_add (fbatch,mlock_lru (folio)) || !folio_may_be_lru_cached (folio) ||lru_cache_disabled ())mlock_folio_batch (fbatch); -local_unlock (&mlock_fbatch.lock); +local_qpw_unlock (&mlock_fbatch.lock); }/**
@@ -269,7 +279,7 @@ void mlock_new_folio(struct folio *foliostruct folio_batch *fbatch;int nr_pages =folio_nr_pages (folio); -local_lock (&mlock_fbatch.lock); +local_qpw_lock (&mlock_fbatch.lock); fbatch =this_cpu_ptr (&mlock_fbatch.fbatch);folio_set_mlocked (folio);
@@ -280,7 +290,7 @@ void mlock_new_folio(struct folio *folioif (!folio_batch_add (fbatch,mlock_new (folio)) || !folio_may_be_lru_cached (folio) ||lru_cache_disabled ())mlock_folio_batch (fbatch); -local_unlock (&mlock_fbatch.lock); +local_qpw_unlock (&mlock_fbatch.lock); }/**
@@ -291,7 +301,7 @@ void munlock_folio(struct folio *folio) {struct folio_batch *fbatch; -local_lock (&mlock_fbatch.lock); +local_qpw_lock (&mlock_fbatch.lock); fbatch =this_cpu_ptr (&mlock_fbatch.fbatch);/* * folio_test_clear_mlocked(folio) must be left to __munlock_folio(),
@@ -301,7 +311,7 @@ void munlock_folio(struct folio *folio)if (!folio_batch_add (fbatch, folio) || !folio_may_be_lru_cached (folio) ||lru_cache_disabled ())mlock_folio_batch (fbatch); -local_unlock (&mlock_fbatch.lock); +local_qpw_unlock (&mlock_fbatch.lock); } static inline unsigned int folio_mlock_step(struct folio *folio,
@@ -823,3 +833,18 @@ void user_shm_unlock(size_t size, structspin_unlock (&shmlock_user_lock);put_ucounts (ucounts); } + +int __init mlock_init (void ) +{ +unsigned int cpu; + +for_each_possible_cpu (cpu) { +struct mlock_fbatch *fbatch = &per_cpu (mlock_fbatch, cpu); + +qpw_lock_init (&fbatch->lock); + } + +return 0 ; +} + +module_init (mlock_init);
mm/swap.c +37 -32
181 lines not on this page · raw message
mm/internal.h +3 -1
@@ -1140,10 +1140,12 @@ static inline void munlock_vma_folio(strmunlock_folio (folio); } +int __init mlock_init (void );void mlock_new_folio (struct folio *folio);bool need_mlock_drain (int cpu);void mlock_drain_local (void ); -void mlock_drain_remote (int cpu); +void mlock_drain_cpu (int cpu); +void mlock_drain_offline (int cpu);extern pmd_t maybe_pmd_mkwrite (pmd_t pmd,struct vm_area_struct *vma);
mm/page_alloc.c +1 -1
@@ -6285,7 +6285,7 @@ static int page_alloc_cpu_dead(unsignedstruct zone *zone;lru_add_drain_cpu (cpu); -mlock_drain_remote (cpu); +mlock_drain_offline (cpu);drain_pages (cpu);/*
Re: [PATCH v3 3/4] swap: apply new queue_percpu_work_on() interface
Quoted 11 lines · Marcelo Tosatti
Make use of the new qpw_{un,}lock*() and
queue_percpu_work_on()interface to improve performance & latency.For functions that may be scheduled in a different cpu, replace local_{un,}lock*() by qpw_{un,}lock*(), and replace
schedule_work_on()byqueue_percpu_work_on(). The same happens forflush_work()andflush_percpu_work().The change requires allocation of
qpw_structsinstead of awork_structs, and changing parameters of a few functions to include the cpu parameter.
Quoted 6 lines
This should bring no relevant performance impact on non-QPW kernels:
For functions that may be scheduled in a different cpu, the local_*lock'sthis_cpu_ptr()becomes a per_cpu_ptr(smp_processor_id()).Signed-off-by Leonardo Bras <[email protected]>Signed-off-by Marcelo Tosatti <[email protected]>
I can't review that thoroughly because it's not my area of expertise. But I'm fine the approach. Just please consolidate that qpw_* prefix based namespace.
Signature · 2 lines
Frederic Weisbecker SUSE Labs
PATCH v3 4/4 slub: apply new queue_percpu_work_on() interface
In reply to Marcelo TosattiMake use of the new qpw_{un,}lock*() and queue_percpu_work_on() interface to improve performance & latency.
For functions that may be scheduled in a different cpu, replace local_{un,}lock*() by qpw_{un,}lock*(), and replace schedule_work_on() by queue_percpu_work_on(). The same happens for flush_work() and flush_percpu_work().
This change requires allocation of qpw_structs instead of a work_structs, and changing parameters of a few functions to include the cpu parameter.
This should bring no relevant performance impact on non-QPW kernels:
For functions that may be scheduled in a different cpu, the local_*lock's this_cpu_ptr() becomes a per_cpu_ptr(smp_processor_id()).
--- mm/slub.c | 146 +++++++++++++++++++++++++++++++------------------------------- 1 file changed, 74 insertions(+), 72 deletions(-)
mm/slub.c +72 -70
@@ -50,6 +50,7 @@#include <linux/irq_work.h> #include <linux/kprobes.h> #include <linux/debugfs.h> +#include <linux/qpw.h> #include <trace/events/kmem.h> #include "internal.h"
@@ -129,7 +130,7 @@* For debug caches, all allocations are forced to go through a list_lock * protected region to serialize against concurrent validation. * -* cpu_sheaves->lock (local_trylock) +* cpu_sheaves->lock (qpw_trylock) * * This lock protects fastpath operations on the percpu sheaves. On !RT it * only disables preemption and does no atomic operations. As long as the main
@@ -157,7 +158,7 @@* Interrupts are disabled as part of list_lock or barn lock operations, or * around the slab_lock operation, in order to make the slab allocator safe * to use in the context of an irq. -* Preemption is disabled as part of local_trylock operations. +* Preemption is disabled as part of qpw_trylock operations. * kmalloc_nolock() and kfree_nolock() are safe in NMI context but see * their limitations. *
@@ -418,7 +419,7 @@ struct slab_sheaf { };struct slub_percpu_sheaves { -local_trylock_t lock; +qpw_trylock_t lock;struct slab_sheaf *main;/* never NULL when unlocked */ struct slab_sheaf *spare;/* empty or full, may be NULL */ struct slab_sheaf *rcu_free;/* for batching kfree_rcu() */
@@ -480,7 +481,7 @@ static nodemask_t slab_nodes;static struct workqueue_struct *flushwq;struct slub_flush_work { -struct work_struct work; +struct qpw_struct qpw;struct kmem_cache *s;bool skip; };
@@ -2859,16 +2860,14 @@ static void __kmem_cache_free_bulk(struc* * Returns how many objects are remaining to be flushed */ -static unsigned int __sheaf_flush_main_batch (struct kmem_cache *s) +static unsigned int __sheaf_flush_main_batch (struct kmem_cache *s,int cpu) {struct slub_percpu_sheaves *pcs;unsigned int batch, remaining;void *objects[PCS_BATCH_MAX ];struct slab_sheaf *sheaf; -lockdep_assert_held (this_cpu_ptr (&s->cpu_sheaves->lock)); - - pcs =this_cpu_ptr (s->cpu_sheaves); + pcs =per_cpu_ptr (s->cpu_sheaves, cpu); sheaf = pcs->main; batch =min (PCS_BATCH_MAX , sheaf->size);
@@ -2878,7 +2877,7 @@ static unsigned int __sheaf_flush_main_b remaining = sheaf->size; -local_unlock (&s->cpu_sheaves->lock); +qpw_unlock (&s->cpu_sheaves->lock, cpu);__kmem_cache_free_bulk (s, batch, &objects[0 ]);
@@ -2887,14 +2886,14 @@ static unsigned int __sheaf_flush_main_breturn remaining; } -static void sheaf_flush_main (struct kmem_cache *s) +static void sheaf_flush_main (struct kmem_cache *s,int cpu) {unsigned int remaining;do { -local_lock (&s->cpu_sheaves->lock); +qpw_lock (&s->cpu_sheaves->lock, cpu); - remaining =__sheaf_flush_main_batch (s); + remaining =__sheaf_flush_main_batch (s, cpu); }while (remaining); }
@@ -2908,11 +2907,13 @@ static bool sheaf_try_flush_main(structbool ret = false;do { -if (!local_trylock (&s->cpu_sheaves->lock)) +if (!local_qpw_trylock (&s->cpu_sheaves->lock))return ret; ret = true; - remaining =__sheaf_flush_main_batch (s); + +qpw_lockdep_assert_held (&s->cpu_sheaves->lock); + remaining =__sheaf_flush_main_batch (s,smp_processor_id ()); }while (remaining);
@@ -2989,13 +2990,13 @@ static void rcu_free_sheaf_nobarn(struct* flushing operations are rare so let's keep it simple and flush to slabs * directly, skipping the barn */ -static void pcs_flush_all (struct kmem_cache *s) +static void pcs_flush_all (struct kmem_cache *s,int cpu) {struct slub_percpu_sheaves *pcs;struct slab_sheaf *spare, *rcu_free; -local_lock (&s->cpu_sheaves->lock); - pcs =this_cpu_ptr (s->cpu_sheaves); +qpw_lock (&s->cpu_sheaves->lock, cpu); + pcs =per_cpu_ptr (s->cpu_sheaves, cpu); spare = pcs->spare; pcs->spare =NULL ;
@@ -3003,7 +3004,7 @@ static void pcs_flush_all(struct kmem_ca rcu_free = pcs->rcu_free; pcs->rcu_free =NULL ; -local_unlock (&s->cpu_sheaves->lock); +qpw_unlock (&s->cpu_sheaves->lock, cpu);if (spare) {sheaf_flush_unused (s, spare);
@@ -3013,7 +3014,7 @@ static void pcs_flush_all(struct kmem_caif (rcu_free)call_rcu (&rcu_free->rcu_head, rcu_free_sheaf_nobarn); -sheaf_flush_main (s); +sheaf_flush_main (s, cpu); }static void __pcs_flush_all_cpu (struct kmem_cache *s,unsigned int cpu)
@@ -3963,13 +3964,13 @@ static void flush_cpu_sheaves(struct wor {struct kmem_cache *s;struct slub_flush_work *sfw; +int cpu =qpw_get_cpu (w); - sfw = container_of(w, struct slub_flush_work, work); - + sfw = &per_cpu (slub_flush, cpu); s = sfw->s;if (cache_has_sheaves (s)) -pcs_flush_all (s); +pcs_flush_all (s, cpu); }static void flush_all_cpus_locked (struct kmem_cache *s)
@@ -3986,17 +3987,17 @@ static void flush_all_cpus_locked(struct sfw->skip = true;continue ; } -INIT_WORK (&sfw->work, flush_cpu_sheaves); +INIT_QPW (&sfw->qpw, flush_cpu_sheaves, cpu); sfw->skip = false; sfw->s = s; -queue_work_on (cpu, flushwq, &sfw->work); +queue_percpu_work_on (cpu, flushwq, &sfw->qpw); }for_each_online_cpu (cpu) { sfw = &per_cpu (slub_flush, cpu);if (sfw->skip)continue ; -flush_work (&sfw->work); +flush_percpu_work (&sfw->qpw); }mutex_unlock (&flush_lock);
@@ -4015,17 +4016,18 @@ static void flush_rcu_sheaf(struct work_struct slab_sheaf *rcu_free;struct slub_flush_work *sfw;struct kmem_cache *s; +int cpu =qpw_get_cpu (w); - sfw = container_of(w, struct slub_flush_work, work); + sfw = &per_cpu (slub_flush, cpu); s = sfw->s; -local_lock (&s->cpu_sheaves->lock); - pcs =this_cpu_ptr (s->cpu_sheaves); +qpw_lock (&s->cpu_sheaves->lock, cpu); + pcs =per_cpu_ptr (s->cpu_sheaves, cpu); rcu_free = pcs->rcu_free; pcs->rcu_free =NULL ; -local_unlock (&s->cpu_sheaves->lock); +qpw_unlock (&s->cpu_sheaves->lock, cpu);if (rcu_free)call_rcu (&rcu_free->rcu_head, rcu_free_sheaf_nobarn);
@@ -4050,14 +4052,14 @@ void flush_rcu_sheaves_on_cache(struct k* sure the __kfree_rcu_sheaf() finished its call_rcu() */ -INIT_WORK (&sfw->work, flush_rcu_sheaf); +INIT_QPW (&sfw->qpw, flush_rcu_sheaf, cpu); sfw->s = s; -queue_work_on (cpu, flushwq, &sfw->work); +queue_percpu_work_on (cpu, flushwq, &sfw->qpw); }for_each_online_cpu (cpu) { sfw = &per_cpu (slub_flush, cpu); -flush_work (&sfw->work); +flush_percpu_work (&sfw->qpw); }mutex_unlock (&flush_lock);
@@ -4565,11 +4567,11 @@ __pcs_replace_empty_main(struct kmem_cacstruct node_barn *barn;bool allow_spin; -lockdep_assert_held (this_cpu_ptr (&s->cpu_sheaves->lock)); +qpw_lockdep_assert_held (&s->cpu_sheaves->lock);/* Bootstrap or debug cache, back off */ if (unlikely (!cache_has_sheaves (s))) { -local_unlock (&s->cpu_sheaves->lock); +local_qpw_unlock (&s->cpu_sheaves->lock);return NULL ; }
@@ -4580,7 +4582,7 @@ __pcs_replace_empty_main(struct kmem_cac barn =get_barn (s);if (!barn) { -local_unlock (&s->cpu_sheaves->lock); +local_qpw_unlock (&s->cpu_sheaves->lock);return NULL ; }
@@ -4605,7 +4607,7 @@ __pcs_replace_empty_main(struct kmem_cac } } -local_unlock (&s->cpu_sheaves->lock); +local_qpw_unlock (&s->cpu_sheaves->lock); pcs =NULL ;if (!allow_spin)
@@ -4629,7 +4631,7 @@ __pcs_replace_empty_main(struct kmem_cacif (!full)return NULL ; -if (!local_trylock (&s->cpu_sheaves->lock)) +if (!local_qpw_trylock (&s->cpu_sheaves->lock)) goto barn_put; pcs =this_cpu_ptr (s->cpu_sheaves);
@@ -4708,7 +4710,7 @@ void *alloc_from_pcs(struct kmem_cache *return NULL ; } -if (!local_trylock (&s->cpu_sheaves->lock)) +if (!local_qpw_trylock (&s->cpu_sheaves->lock))return NULL ; pcs =this_cpu_ptr (s->cpu_sheaves);
@@ -4728,7 +4730,7 @@ void *alloc_from_pcs(struct kmem_cache ** the current allocation or previous freeing process. */ if (page_to_nid (virt_to_page (object)) != node) { -local_unlock (&s->cpu_sheaves->lock); +local_qpw_unlock (&s->cpu_sheaves->lock);stat (s,ALLOC_NODE_MISMATCH );return NULL ; }
@@ -4736,7 +4738,7 @@ void *alloc_from_pcs(struct kmem_cache * pcs->main->size--; -local_unlock (&s->cpu_sheaves->lock); +local_qpw_unlock (&s->cpu_sheaves->lock);stat (s,ALLOC_FASTPATH );
@@ -4753,7 +4755,7 @@ unsigned int alloc_from_pcs_bulk(structunsigned int batch; next_batch: -if (!local_trylock (&s->cpu_sheaves->lock)) +if (!local_qpw_trylock (&s->cpu_sheaves->lock))return allocated; pcs =this_cpu_ptr (s->cpu_sheaves);
@@ -4764,7 +4766,7 @@ next_batch:struct node_barn *barn;if (unlikely (!cache_has_sheaves (s))) { -local_unlock (&s->cpu_sheaves->lock); +local_qpw_unlock (&s->cpu_sheaves->lock);return allocated; }
@@ -4775,7 +4777,7 @@ next_batch: barn =get_barn (s);if (!barn) { -local_unlock (&s->cpu_sheaves->lock); +local_qpw_unlock (&s->cpu_sheaves->lock);return allocated; }
@@ -4790,7 +4792,7 @@ next_batch:stat (s,BARN_GET_FAIL ); -local_unlock (&s->cpu_sheaves->lock); +local_qpw_unlock (&s->cpu_sheaves->lock);/* * Once full sheaves in barn are depleted, let the bulk
@@ -4808,7 +4810,7 @@ do_alloc: main->size -= batch;memcpy (p, main->objects + main->size, batch *sizeof (void *)); -local_unlock (&s->cpu_sheaves->lock); +local_qpw_unlock (&s->cpu_sheaves->lock);stat_add (s,ALLOC_FASTPATH , batch);
@@ -4992,7 +4994,7 @@ kmem_cache_prefill_sheaf(struct kmem_cacreturn sheaf; } -local_lock (&s->cpu_sheaves->lock); +local_qpw_lock (&s->cpu_sheaves->lock); pcs =this_cpu_ptr (s->cpu_sheaves);if (pcs->spare) {
@@ -5011,7 +5013,7 @@ kmem_cache_prefill_sheaf(struct kmem_cacstat (s,BARN_GET_FAIL ); } -local_unlock (&s->cpu_sheaves->lock); +local_qpw_unlock (&s->cpu_sheaves->lock);if (!sheaf)
@@ -5055,7 +5057,7 @@ void kmem_cache_return_sheaf(struct kmemreturn ; } -local_lock (&s->cpu_sheaves->lock); +local_qpw_lock (&s->cpu_sheaves->lock); pcs =this_cpu_ptr (s->cpu_sheaves); barn =get_barn (s);
@@ -5065,7 +5067,7 @@ void kmem_cache_return_sheaf(struct kmemstat (s,SHEAF_RETURN_FAST ); } -local_unlock (&s->cpu_sheaves->lock); +local_qpw_unlock (&s->cpu_sheaves->lock);if (!sheaf)return ;
@@ -5595,7 +5597,7 @@ static void __pcs_install_empty_sheaf(st struct slub_percpu_sheaves *pcs, struct slab_sheaf *empty, struct node_barn *barn) { - lockdep_assert_held(this_cpu_ptr(&s->cpu_sheaves->lock)); + qpw_lockdep_assert_held(&s->cpu_sheaves->lock);/* This is what we expect to find if nobody interrupted us. */ if (likely (!pcs->spare)) {
@@ -5646,17 +5648,17 @@ __pcs_replace_full_main(struct kmem_cachbool put_fail; restart: -lockdep_assert_held (this_cpu_ptr (&s->cpu_sheaves->lock)); +qpw_lockdep_assert_held (&s->cpu_sheaves->lock);/* Bootstrap or debug cache, back off */ if (unlikely (!cache_has_sheaves (s))) { -local_unlock (&s->cpu_sheaves->lock); +local_qpw_unlock (&s->cpu_sheaves->lock);return NULL ; } barn =get_barn (s);if (!barn) { -local_unlock (&s->cpu_sheaves->lock); +local_qpw_unlock (&s->cpu_sheaves->lock);return NULL ; }
@@ -5693,7 +5695,7 @@ restart:stat (s,BARN_PUT_FAIL ); pcs->spare =NULL ; -local_unlock (&s->cpu_sheaves->lock); +local_qpw_unlock (&s->cpu_sheaves->lock);sheaf_flush_unused (s, to_flush); empty = to_flush;
@@ -5709,7 +5711,7 @@ restart: put_fail = true; alloc_empty: -local_unlock (&s->cpu_sheaves->lock); +local_qpw_unlock (&s->cpu_sheaves->lock);/* * alloc_empty_sheaf() doesn't support !allow_spin and it's
@@ -5729,7 +5731,7 @@ alloc_empty:if (!sheaf_try_flush_main (s))return NULL ; -if (!local_trylock (&s->cpu_sheaves->lock)) +if (!local_qpw_trylock (&s->cpu_sheaves->lock))return NULL ; pcs =this_cpu_ptr (s->cpu_sheaves);
@@ -5745,7 +5747,7 @@ alloc_empty:return pcs; got_empty: -if (!local_trylock (&s->cpu_sheaves->lock)) { +if (!local_qpw_trylock (&s->cpu_sheaves->lock)) {barn_put_empty_sheaf (barn, empty);return NULL ; }
@@ -5765,7 +5767,7 @@ bool free_to_pcs(struct kmem_cache *s, v {struct slub_percpu_sheaves *pcs; -if (!local_trylock (&s->cpu_sheaves->lock)) +if (!local_qpw_trylock (&s->cpu_sheaves->lock))return false; pcs =this_cpu_ptr (s->cpu_sheaves);
@@ -5779,7 +5781,7 @@ bool free_to_pcs(struct kmem_cache *s, v pcs->main->objects[pcs->main->size++] = object; -local_unlock (&s->cpu_sheaves->lock); +local_qpw_unlock (&s->cpu_sheaves->lock);stat (s,FREE_FASTPATH );
@@ -5869,7 +5871,7 @@ bool __kfree_rcu_sheaf(struct kmem_cachelock_map_acquire_try (&kfree_rcu_sheaf_map); -if (!local_trylock (&s->cpu_sheaves->lock)) +if (!local_qpw_trylock (&s->cpu_sheaves->lock)) goto fail; pcs =this_cpu_ptr (s->cpu_sheaves);
@@ -5881,7 +5883,7 @@ bool __kfree_rcu_sheaf(struct kmem_cache/* Bootstrap or debug cache, fall back */ if (unlikely (!cache_has_sheaves (s))) { -local_unlock (&s->cpu_sheaves->lock); +local_qpw_unlock (&s->cpu_sheaves->lock); goto fail; }
@@ -5893,7 +5895,7 @@ bool __kfree_rcu_sheaf(struct kmem_cache barn =get_barn (s);if (!barn) { -local_unlock (&s->cpu_sheaves->lock); +local_qpw_unlock (&s->cpu_sheaves->lock); goto fail; }
@@ -5904,14 +5906,14 @@ bool __kfree_rcu_sheaf(struct kmem_cache goto do_free; } -local_unlock (&s->cpu_sheaves->lock); +local_qpw_unlock (&s->cpu_sheaves->lock); empty =alloc_empty_sheaf (s,GFP_NOWAIT );if (!empty) goto fail; -if (!local_trylock (&s->cpu_sheaves->lock)) { +if (!local_qpw_trylock (&s->cpu_sheaves->lock)) {barn_put_empty_sheaf (barn, empty); goto fail; }
@@ -5942,13 +5944,13 @@ do_free: }/* -* we flush before local_unlock to make sure a racing +* we flush before local_qpw_unlock to make sure a racing * flush_all_rcu_sheaves() doesn't miss this sheaf */ if (rcu_sheaf)call_rcu (&rcu_sheaf->rcu_head, rcu_free_sheaf); -local_unlock (&s->cpu_sheaves->lock); +local_qpw_unlock (&s->cpu_sheaves->lock);stat (s,FREE_RCU_SHEAF );lock_map_release (&kfree_rcu_sheaf_map);
@@ -6004,7 +6006,7 @@ next_remote_batch: goto flush_remote; next_batch: -if (!local_trylock (&s->cpu_sheaves->lock)) +if (!local_qpw_trylock (&s->cpu_sheaves->lock)) goto fallback; pcs =this_cpu_ptr (s->cpu_sheaves);
@@ -6047,7 +6049,7 @@ do_free:memcpy (main->objects + main->size, p, batch *sizeof (void *)); main->size += batch; -local_unlock (&s->cpu_sheaves->lock); +local_qpw_unlock (&s->cpu_sheaves->lock);stat_add (s,FREE_FASTPATH , batch);
@@ -6063,7 +6065,7 @@ do_free:return ; no_empty: -local_unlock (&s->cpu_sheaves->lock); +local_qpw_unlock (&s->cpu_sheaves->lock);/* * if we depleted all empty sheaves in the barn or there are too
@@ -7468,7 +7470,7 @@ static int init_percpu_sheaves(struct km pcs =per_cpu_ptr (s->cpu_sheaves, cpu); -local_trylock_init (&pcs->lock); +qpw_trylock_init (&pcs->lock);/* * Bootstrap sheaf has zero size so fast-path allocation fails.
Re: [PATCH v3 4/4] slub: apply new queue_percpu_work_on() interface
Quoted 11 lines · Marcelo Tosatti
Make use of the new qpw_{un,}lock*() and
queue_percpu_work_on()interface to improve performance & latency.For functions that may be scheduled in a different cpu, replace local_{un,}lock*() by qpw_{un,}lock*(), and replace
schedule_work_on()byqueue_percpu_work_on(). The same happens forflush_work()andflush_percpu_work().This change requires allocation of
qpw_structsinstead of awork_structs, and changing parameters of a few functions to include the cpu parameter.
Quoted 6 lines
This should bring no relevant performance impact on non-QPW kernels:
For functions that may be scheduled in a different cpu, the local_*lock'sthis_cpu_ptr()becomes a per_cpu_ptr(smp_processor_id()).Signed-off-by Leonardo Bras <[email protected]>Signed-off-by Marcelo Tosatti <[email protected]>
Again I can't review the details but I'm fine with the approach once the namespace is fixed.
Signature · 2 lines
Frederic Weisbecker SUSE Labs
Quoted 261 lines · Marcelo Tosatti
The problem:
Some places in the kernel implement a parallel programming strategy consisting onlocal_locks()for most of the work, and some rare remote operations are scheduled on target cpu. This keeps cache bouncing low since cacheline tends to be mostly local, and avoids the cost of locks in non-RT kernels, even though the very few remote operations will be expensive due to scheduling overhead.On the other hand, for RT workloads this can represent a problem: getting an important workload scheduled out to deal with remote requests is sure to introduce unexpected deadline misses.
The idea:
Currently with PREEMPT_RT=y,local_locks()become per-cpu spinlocks. In this case, instead of scheduling work on a remote cpu, it should be safe to grab that remote cpu's per-cpu spinlock and run the required work locally. That major cost, which is un/locking in every local function, already happens inPREEMPT_RT.Also, there is no need to worry about extra cache bouncing:
The cacheline invalidation already happens due toschedule_work_on().This will avoid
schedule_work_on(), and thus avoid scheduling-out an RT workload.Proposed solution:
A new interface called Queue PerCPU Work (QPW), which should replace Work Queue in the above mentioned use case.If
CONFIG_QPW=nthis interfaces just wraps the currentlocal_locks+ WorkQueue behavior, so no expected change in runtime.If
CONFIG_QPW=y, and qpw kernel boot option =1,queue_percpu_work_on(cpu,...)will lock that cpu's per-cpu structure and perform work on it locally. This is possible because on functions that can be used for performing remote work on remote per-cpu structures, thelocal_lock(which is already
athis_cpuspinlock()), will be replaced by aqpw_spinlock(), which is able to get theper_cpuspinlock()for the cpu passed as parameter.
v2->v3:
- Use preempt_disable/preempt_enable on !CONFIG_PREEMPT_RT (Vlastimil Babka).
- Improve documentation to includelocal_qpw_lockon operations table (Leonardo Bras).
- Enable qpw=1 automatically if CPU isolation is enabled (Vlastimil Babka).
v1->v2:
- Introducelocal_qpw_lockand unlock functions, move preempt_disable/preempt_enableto it (Leonardo Bras). This reduces performance overhead of the patch.
- Documentation and changelog typo fixes (Leonardo Bras).
- Fix places where preempt_disable/preempt_enable was not being correctly performed.
- Add performance measurements.
RFC->v1:- Introduce
CONFIG_QPWand qpw= kernel boot option to enable remote spinlocking and execution even on !CONFIG_PREEMPT_RT kernels (Leonardo Bras).
- Movebuffer_headdraining to separate workqueue (Marcelo Tosatti).
- Convert mlock per-CPU page lists to QPW (Marcelo Tosatti).
- Drop memcontrol convertion (as isolated CPUs are not targets ofqueue_work_onanymore).
- Rebase SLUB against Vlastimil's slab/next.
- Add basic document for QPW (Waiman Long).The performance numbers, as measured by the following test program, are as follows:
CONFIG_PREEMPT_DYNAMIC=yUnpatched kernel: 60 cycles Patched kernel, CONFIG_QPW=n: 62 cycles Patched kernel, CONFIG_QPW=y, qpw=0: 62 cycles Patched kernel, CONFIG_QPW=y, qpw=1: 75 cycles
CONFIG_PREEMPT_RT:Unpatched kernel: 95 cycles Patched kernel, CONFIG_QPW=y, qpw=0: 99 cycles Patched kernel, CONFIG_QPW=y, qpw=1: 97 cycles
kmalloc_bench.c:
#include <linux/module.h>
#include <linux/kernel.h>
#include <linux/slab.h>
#include <linux/timex.h>
#include <linux/preempt.h>
#include <linux/irqflags.h>
#include <linux/vmalloc.h>
MODULE_LICENSE("GPL");MODULE_AUTHOR("Gemini AI");MODULE_DESCRIPTION("A simple kmalloc performance benchmark");static int size = 64; // Default allocation size in bytes
module_param(size, int, 0644);static int iterations = 9000000; // Default number of iterations
module_param(iterations, int, 0644);static int __init kmalloc_bench_init(void) { void **ptrs; cycles_t start, end; uint64_t total_cycles; int i; pr_info("kmalloc_bench: Starting test (size=%d, iterations=%d)\n", size, iterations); // Allocate an array to store pointers to avoid immediate kfree-reuse optimization ptrs = vmalloc(sizeof(void *) * iterations); if (!ptrs) { pr_err("kmalloc_bench: Failed to allocate pointer array\n"); return -ENOMEM; } preempt_disable(); start = get_cycles(); for (i = 0; i < iterations; i++) { ptrs[i] = kmalloc(size, GFP_ATOMIC); } end = get_cycles(); total_cycles = end - start; preempt_enable(); pr_info("kmalloc_bench: Total cycles for %d allocs: %llu\n", iterations, total_cycles); pr_info("kmalloc_bench: Avg cycles per kmalloc: %llu\n", total_cycles / iterations); // Cleanup for (i = 0; i < iterations; i++) { kfree(ptrs[i]); } vfree(ptrs); return 0; }static void __exit kmalloc_bench_exit(void) { pr_info("kmalloc_bench: Module unloaded\n"); }
module_init(kmalloc_bench_init);module_exit(kmalloc_bench_exit);The following testcase triggers
lru_add_drain_allon an isolated CPU (that doessys_writeto a file before entering its realtime loop)./*
* Simulates a low latency loop program that is interrupted
* due tolru_add_drain_all. To triggerlru_add_drain_all, run:
*
* blockdev --flushbufs/dev/sdX
*
*/
#define_GNU_SOURCE
#include <fcntl.h>
#include <stdio.h>
#include <stdlib.h>
#include <sys/mman.h>
#include <string.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <stdlib.h>
#include <stdarg.h>
#include <pthread.h>
#include <sched.h>
#include <unistd.h>int cpu;
static void *run(void *arg)
{ pthread_t current_thread; cpu_set_t cpuset; int ret, nrloops; struct sched_param sched_p; pid_t pid; int fd; char buf[] = "xxxxxxxxxxx"; CPU_ZERO(&cpuset); CPU_SET(cpu, &cpuset); current_thread = pthread_self(); ret = pthread_setaffinity_np(current_thread, sizeof(cpu_set_t), &cpuset); if (ret) { perror("pthread_setaffinity_np failed\n"); exit(0); } memset(&sched_p, 0, sizeof(struct sched_param)); sched_p.sched_priority = 1; pid = gettid(); ret = sched_setscheduler(pid, SCHED_FIFO, &sched_p); if (ret) { perror("sched_setscheduler"); exit(0); } fd = open("/tmp/tmpfile", O_RDWR|O_CREAT|O_TRUNC); if (fd == -1) { perror("open"); exit(0); } ret = write(fd, buf, sizeof(buf)); if (ret == -1) { perror("write"); exit(0); } do { nrloops = nrloops+2; nrloops--; } while (1); }int
main(int argc, char *argv[]){ int fd, ret; pthread_t thread; long val; char *endptr, *str; struct sched_param sched_p; pid_t pid; if (argc != 2) { printf("usage: %s cpu-nr\n", argv[0]); printf("where CPU number is the CPU to pin thread to\n"); exit(0); } str = argv[1]; cpu = strtol(str, &endptr, 10); if (cpu < 0) { printf("strtol returns %d\n", cpu); exit(0); } printf("cpunr=%d\n", cpu); memset(&sched_p, 0, sizeof(struct sched_param)); sched_p.sched_priority = 1; pid = getpid(); ret = sched_setscheduler(pid, SCHED_FIFO, &sched_p); if (ret) { perror("sched_setscheduler"); exit(0); } pthread_create(&thread, NULL, run, NULL); sleep(5000); pthread_join(thread, NULL); }
Hi,
Is there any plan to proceed with this patchset?
Thanks.
Quoted 261 lines · Marcelo Tosatti
The problem:
Some places in the kernel implement a parallel programming strategy consisting onlocal_locks()for most of the work, and some rare remote operations are scheduled on target cpu. This keeps cache bouncing low since cacheline tends to be mostly local, and avoids the cost of locks in non-RT kernels, even though the very few remote operations will be expensive due to scheduling overhead.On the other hand, for RT workloads this can represent a problem: getting an important workload scheduled out to deal with remote requests is sure to introduce unexpected deadline misses.
The idea:
Currently with PREEMPT_RT=y,local_locks()become per-cpu spinlocks. In this case, instead of scheduling work on a remote cpu, it should be safe to grab that remote cpu's per-cpu spinlock and run the required work locally. That major cost, which is un/locking in every local function, already happens inPREEMPT_RT.Also, there is no need to worry about extra cache bouncing:
The cacheline invalidation already happens due toschedule_work_on().This will avoid
schedule_work_on(), and thus avoid scheduling-out an RT workload.Proposed solution:
A new interface called Queue PerCPU Work (QPW), which should replace Work Queue in the above mentioned use case.If
CONFIG_QPW=nthis interfaces just wraps the currentlocal_locks+ WorkQueue behavior, so no expected change in runtime.If
CONFIG_QPW=y, and qpw kernel boot option =1,queue_percpu_work_on(cpu,...)will lock that cpu's per-cpu structure and perform work on it locally. This is possible because on functions that can be used for performing remote work on remote per-cpu structures, thelocal_lock(which is already
athis_cpuspinlock()), will be replaced by aqpw_spinlock(), which is able to get theper_cpuspinlock()for the cpu passed as parameter.
v2->v3:
- Use preempt_disable/preempt_enable on !CONFIG_PREEMPT_RT (Vlastimil Babka).
- Improve documentation to includelocal_qpw_lockon operations table (Leonardo Bras).
- Enable qpw=1 automatically if CPU isolation is enabled (Vlastimil Babka).
v1->v2:
- Introducelocal_qpw_lockand unlock functions, move preempt_disable/preempt_enableto it (Leonardo Bras). This reduces performance overhead of the patch.
- Documentation and changelog typo fixes (Leonardo Bras).
- Fix places where preempt_disable/preempt_enable was not being correctly performed.
- Add performance measurements.
RFC->v1:- Introduce
CONFIG_QPWand qpw= kernel boot option to enable remote spinlocking and execution even on !CONFIG_PREEMPT_RT kernels (Leonardo Bras).
- Movebuffer_headdraining to separate workqueue (Marcelo Tosatti).
- Convert mlock per-CPU page lists to QPW (Marcelo Tosatti).
- Drop memcontrol convertion (as isolated CPUs are not targets ofqueue_work_onanymore).
- Rebase SLUB against Vlastimil's slab/next.
- Add basic document for QPW (Waiman Long).The performance numbers, as measured by the following test program, are as follows:
CONFIG_PREEMPT_DYNAMIC=yUnpatched kernel: 60 cycles Patched kernel, CONFIG_QPW=n: 62 cycles Patched kernel, CONFIG_QPW=y, qpw=0: 62 cycles Patched kernel, CONFIG_QPW=y, qpw=1: 75 cycles
CONFIG_PREEMPT_RT:Unpatched kernel: 95 cycles Patched kernel, CONFIG_QPW=y, qpw=0: 99 cycles Patched kernel, CONFIG_QPW=y, qpw=1: 97 cycles
kmalloc_bench.c:
#include <linux/module.h>
#include <linux/kernel.h>
#include <linux/slab.h>
#include <linux/timex.h>
#include <linux/preempt.h>
#include <linux/irqflags.h>
#include <linux/vmalloc.h>
MODULE_LICENSE("GPL");MODULE_AUTHOR("Gemini AI");MODULE_DESCRIPTION("A simple kmalloc performance benchmark");static int size = 64; // Default allocation size in bytes
module_param(size, int, 0644);static int iterations = 9000000; // Default number of iterations
module_param(iterations, int, 0644);static int __init kmalloc_bench_init(void) { void **ptrs; cycles_t start, end; uint64_t total_cycles; int i; pr_info("kmalloc_bench: Starting test (size=%d, iterations=%d)\n", size, iterations); // Allocate an array to store pointers to avoid immediate kfree-reuse optimization ptrs = vmalloc(sizeof(void *) * iterations); if (!ptrs) { pr_err("kmalloc_bench: Failed to allocate pointer array\n"); return -ENOMEM; } preempt_disable(); start = get_cycles(); for (i = 0; i < iterations; i++) { ptrs[i] = kmalloc(size, GFP_ATOMIC); } end = get_cycles(); total_cycles = end - start; preempt_enable(); pr_info("kmalloc_bench: Total cycles for %d allocs: %llu\n", iterations, total_cycles); pr_info("kmalloc_bench: Avg cycles per kmalloc: %llu\n", total_cycles / iterations); // Cleanup for (i = 0; i < iterations; i++) { kfree(ptrs[i]); } vfree(ptrs); return 0; }static void __exit kmalloc_bench_exit(void) { pr_info("kmalloc_bench: Module unloaded\n"); }
module_init(kmalloc_bench_init);module_exit(kmalloc_bench_exit);The following testcase triggers
lru_add_drain_allon an isolated CPU (that doessys_writeto a file before entering its realtime loop)./*
* Simulates a low latency loop program that is interrupted
* due tolru_add_drain_all. To triggerlru_add_drain_all, run:
*
* blockdev --flushbufs/dev/sdX
*
*/
#define_GNU_SOURCE
#include <fcntl.h>
#include <stdio.h>
#include <stdlib.h>
#include <sys/mman.h>
#include <string.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <stdlib.h>
#include <stdarg.h>
#include <pthread.h>
#include <sched.h>
#include <unistd.h>int cpu;
static void *run(void *arg)
{ pthread_t current_thread; cpu_set_t cpuset; int ret, nrloops; struct sched_param sched_p; pid_t pid; int fd; char buf[] = "xxxxxxxxxxx"; CPU_ZERO(&cpuset); CPU_SET(cpu, &cpuset); current_thread = pthread_self(); ret = pthread_setaffinity_np(current_thread, sizeof(cpu_set_t), &cpuset); if (ret) { perror("pthread_setaffinity_np failed\n"); exit(0); } memset(&sched_p, 0, sizeof(struct sched_param)); sched_p.sched_priority = 1; pid = gettid(); ret = sched_setscheduler(pid, SCHED_FIFO, &sched_p); if (ret) { perror("sched_setscheduler"); exit(0); } fd = open("/tmp/tmpfile", O_RDWR|O_CREAT|O_TRUNC); if (fd == -1) { perror("open"); exit(0); } ret = write(fd, buf, sizeof(buf)); if (ret == -1) { perror("write"); exit(0); } do { nrloops = nrloops+2; nrloops--; } while (1); }int
main(int argc, char *argv[]){ int fd, ret; pthread_t thread; long val; char *endptr, *str; struct sched_param sched_p; pid_t pid; if (argc != 2) { printf("usage: %s cpu-nr\n", argv[0]); printf("where CPU number is the CPU to pin thread to\n"); exit(0); } str = argv[1]; cpu = strtol(str, &endptr, 10); if (cpu < 0) { printf("strtol returns %d\n", cpu); exit(0); } printf("cpunr=%d\n", cpu); memset(&sched_p, 0, sizeof(struct sched_param)); sched_p.sched_priority = 1; pid = getpid(); ret = sched_setscheduler(pid, SCHED_FIFO, &sched_p); if (ret) { perror("sched_setscheduler"); exit(0); } pthread_create(&thread, NULL, run, NULL); sleep(5000); pthread_join(thread, NULL); }
Quoted 4 lines · Frederic Weisbecker
Hi,
Is there any plan to proceed with this patchset?
Hello Frederic,
Yes, I plan to send a v4 soon.
(Pushed the changes to my gilab a few hours before receiving your email today :)
I just need to perform some tests and rewrite the cover letter
Thanks for your interest!
Leo
Quoted 7 lines · Leonardo Bras
Quoted 4 lines · Frederic Weisbecker
Hi,
Is there any plan to proceed with this patchset?
Quoted 5 lines
Hello Frederic,
Yes, I plan to send a v4 soon.
(Pushed the changes to my gilab a few hours before receiving your email today :)
Stars aligned ;-)
Quoted 4 lines
I just need to perform some tests and rewrite the cover letter
Thanks for your interest!
Leo
Thank you!
Signature · 2 lines
Frederic Weisbecker SUSE Labs
Quoted 5 lines · Frederic Weisbecker
Hi,
Is there any plan to proceed with this patchset?
Thanks.
Yes, will address Sashiko's comments and repost.