sched: make migrate_enable/migrate_disable inline
Subsystems (23)
v2 cover sched: make migrate_enable/migrate_disable inline
In this series, we make migrate_enable/migrate_disable inline to obtain better performance in some case.
In the first patch, we add the macro "COMPILE_OFFSETS" to all the asm-offset.c to avoid circular dependency in the 2nd patch.
In the 2nd patch, we generate the offset of nr_pinned in "struct rq" with rq-offsets.c, as the "struct rq" is defined internally and we need to access the "nr_pinned" field in migrate_enable and migrate_disable. Then, we move the definition of migrate_enable/migrate_disable from kernel/sched/core.c to include/linux/sched.h.
In the 3rd patch, we fix some typos in include/linux/preempt.h.
One of the beneficiaries of this series is BPF trampoline. Without this series, the migrate_enable/migrate_disable is hot when we run the benchmark for FENTRY, FEXIT, MODIFY_RETURN, etc:
54.63% bpf_prog_2dcccf652aac1793_bench_trigger_fentry [k]
bpf_prog_2dcccf652aac1793_bench_trigger_fentry
10.43% [kernel] [k] migrate_enable
10.07% bpf_trampoline_6442517037 [k] bpf_trampoline_6442517037 8.06% [kernel] [k] __bpf_prog_exit_recur
4.11% libc.so.6 [.] syscall
2.15% [kernel] [k] entry_SYSCALL_64
1.48% [kernel] [k] memchr_inv
1.32% [kernel] [k] fput
1.16% [kernel] [k] _copy_to_user
0.73% [kernel] [k] bpf_prog_test_run_raw_tp
Before this patch, the performance of BPF FENTRY is:
fentry : 113.030 ± 0.149M/s fentry : 112.501 ± 0.187M/s fentry : 112.828 ± 0.267M/s fentry : 115.287 ± 0.241M/s
After this patch, the performance of BPF FENTRY increases to:
fentry : 143.644 ± 0.670M/s fentry : 149.764 ± 0.362M/s fentry : 149.642 ± 0.156M/s fentry : 145.263 ± 0.221M/s
Changes since V1:
* use PERCPU_PTR() for this_rq_raw() if !CONFIG_SMP in the 2nd patch
Menglong Dong (3):
arch: add the macro COMPILE_OFFSETS to all the asm-offsets.c
sched: make migrate_enable/migrate_disable inline
sched: fix some typos in include/linux/preempt.h
Kbuild | 13 ++++- arch/alpha/kernel/asm-offsets.c | 1 + arch/arc/kernel/asm-offsets.c | 1 + arch/arm/kernel/asm-offsets.c | 2 + arch/arm64/kernel/asm-offsets.c | 1 + arch/csky/kernel/asm-offsets.c | 1 + arch/hexagon/kernel/asm-offsets.c | 1 + arch/loongarch/kernel/asm-offsets.c | 2 + arch/m68k/kernel/asm-offsets.c | 1 + arch/microblaze/kernel/asm-offsets.c | 1 + arch/mips/kernel/asm-offsets.c | 2 + arch/nios2/kernel/asm-offsets.c | 1 + arch/openrisc/kernel/asm-offsets.c | 1 + arch/parisc/kernel/asm-offsets.c | 1 + arch/powerpc/kernel/asm-offsets.c | 1 + arch/riscv/kernel/asm-offsets.c | 1 + arch/s390/kernel/asm-offsets.c | 1 + arch/sh/kernel/asm-offsets.c | 1 + arch/sparc/kernel/asm-offsets.c | 1 + arch/um/kernel/asm-offsets.c | 2 + arch/xtensa/kernel/asm-offsets.c | 1 + include/linux/preempt.h | 11 ++-- include/linux/sched.h | 77 ++++++++++++++++++++++++++++ kernel/bpf/verifier.c | 3 +- kernel/sched/core.c | 56 ++------------------ kernel/sched/rq-offsets.c | 12 +++++ 26 files changed, 134 insertions(+), 62 deletions(-) create mode 100644 kernel/sched/rq-offsets.c
Signature · 1 line
2.50.1
PATCH v2 1/3 arch: add the macro COMPILE_OFFSETS to all the asm-offsets.c
The include/generated/asm-offsets.h is generated in Kbuild during compiling from arch/SRCARCH/kernel/asm-offsets.c. When we want to generate another similar offset header file, circular dependency can happen.
For example, we want to generate a offset file include/generated/test.h, which is included in include/sched/sched.h. If we generate asm-offsets.h first, it will fail, as include/sched/sched.h is included in asm-offsets.c and include/generated/test.h doesn't exist; If we generate test.h first, it can't success neither, as include/generated/asm-offsets.h is included by it.
In x86_64, the macro COMPILE_OFFSETS is used to avoid such circular dependency. We can generate asm-offsets.h first, and if the COMPILE_OFFSETS is defined, we don't include the "generated/test.h".
And we define the macro COMPILE_OFFSETS for all the asm-offsets.c for this purpose.
--- arch/alpha/kernel/asm-offsets.c | 1 + arch/arc/kernel/asm-offsets.c | 1 + arch/arm/kernel/asm-offsets.c | 2 ++ arch/arm64/kernel/asm-offsets.c | 1 + arch/csky/kernel/asm-offsets.c | 1 + arch/hexagon/kernel/asm-offsets.c | 1 + arch/loongarch/kernel/asm-offsets.c | 2 ++ arch/m68k/kernel/asm-offsets.c | 1 + arch/microblaze/kernel/asm-offsets.c | 1 + arch/mips/kernel/asm-offsets.c | 2 ++ arch/nios2/kernel/asm-offsets.c | 1 + arch/openrisc/kernel/asm-offsets.c | 1 + arch/parisc/kernel/asm-offsets.c | 1 + arch/powerpc/kernel/asm-offsets.c | 1 + arch/riscv/kernel/asm-offsets.c | 1 + arch/s390/kernel/asm-offsets.c | 1 + arch/sh/kernel/asm-offsets.c | 1 + arch/sparc/kernel/asm-offsets.c | 1 + arch/um/kernel/asm-offsets.c | 2 ++ arch/xtensa/kernel/asm-offsets.c | 1 + 20 files changed, 24 insertions(+)
arch/alpha/kernel/asm-offsets.c +1 -0
@@ -4,6 +4,7 @@* This code generates raw asm output which is post-processed to extract * and format the required data. */ +#define COMPILE_OFFSETS #include <linux/types.h> #include <linux/stddef.h>
arch/arc/kernel/asm-offsets.c +1 -0
@@ -2,6 +2,7 @@/* * Copyright (C) 2004, 2007-2010, 2011-2012 Synopsys, Inc. (www.synopsys.com) */ +#define COMPILE_OFFSETS #include <linux/sched.h> #include <linux/mm.h>
arch/arm/kernel/asm-offsets.c +2 -0
@@ -7,6 +7,8 @@* This code generates raw asm output which is post-processed to extract * and format the required data. */ +#define COMPILE_OFFSETS +#include <linux/compiler.h> #include <linux/sched.h> #include <linux/mm.h>
arch/arm64/kernel/asm-offsets.c +1 -0
@@ -6,6 +6,7 @@* 2001-2002 Keith Owens * Copyright (C) 2012 ARM Ltd. */ +#define COMPILE_OFFSETS #include <linux/arm_sdei.h> #include <linux/sched.h>
arch/csky/kernel/asm-offsets.c +1 -0
@@ -1,5 +1,6 @@// SPDX-License-Identifier: GPL-2.0 // Copyright (C) 2018 Hangzhou C-SKY Microsystems co.,ltd. +#define COMPILE_OFFSETS #include <linux/sched.h> #include <linux/kernel_stat.h>
arch/hexagon/kernel/asm-offsets.c +1 -0
@@ -8,6 +8,7 @@* * Copyright (c) 2010-2012, The Linux Foundation. All rights reserved. */ +#define COMPILE_OFFSETS #include <linux/compat.h> #include <linux/types.h>
arch/loongarch/kernel/asm-offsets.c +2 -0
@@ -4,6 +4,8 @@* * Copyright (C) 2020-2022 Loongson Technology Corporation Limited */ +#define COMPILE_OFFSETS +#include <linux/types.h> #include <linux/sched.h> #include <linux/mm.h>
arch/m68k/kernel/asm-offsets.c +1 -0
@@ -9,6 +9,7 @@* #defines from the assembly-language output. */ +#define COMPILE_OFFSETS #define ASM_OFFSETS_C #include <linux/stddef.h>
arch/microblaze/kernel/asm-offsets.c +1 -0
@@ -7,6 +7,7 @@* License. See the file "COPYING" in the main directory of this archive * for more details. */ +#define COMPILE_OFFSETS #include <linux/init.h> #include <linux/stddef.h>
arch/mips/kernel/asm-offsets.c +2 -0
@@ -9,6 +9,8 @@* Kevin Kissell, [email protected] and Carsten Langgaard, [email protected] * Copyright (C) 2000 MIPS Technologies, Inc. */ +#define COMPILE_OFFSETS +#include <linux/compat.h> #include <linux/types.h> #include <linux/sched.h>
arch/nios2/kernel/asm-offsets.c +1 -0
@@ -2,6 +2,7 @@/* * Copyright (C) 2011 Tobias Klauser <[email protected]> */ +#define COMPILE_OFFSETS #include <linux/stddef.h> #include <linux/sched.h>
arch/openrisc/kernel/asm-offsets.c +1 -0
@@ -18,6 +18,7 @@* compile this file to assembler, and then extract the * #defines from the assembly-language output. */ +#define COMPILE_OFFSETS #include <linux/signal.h> #include <linux/sched.h>
arch/parisc/kernel/asm-offsets.c +1 -0
@@ -13,6 +13,7 @@* Copyright (C) 2002 Randolph Chung <tausq with parisc-linux.org> * Copyright (C) 2003 James Bottomley <jejb at parisc-linux.org> */ +#define COMPILE_OFFSETS #include <linux/types.h> #include <linux/sched.h>
arch/powerpc/kernel/asm-offsets.c +1 -0
@@ -8,6 +8,7 @@* compile this file to assembler, and then extract the * #defines from the assembly-language output. */ +#define COMPILE_OFFSETS #include <linux/compat.h> #include <linux/signal.h>
arch/riscv/kernel/asm-offsets.c +1 -0
@@ -3,6 +3,7 @@* Copyright (C) 2012 Regents of the University of California * Copyright (C) 2017 SiFive */ +#define COMPILE_OFFSETS #include <linux/kbuild.h> #include <linux/mm.h>
arch/s390/kernel/asm-offsets.c +1 -0
@@ -4,6 +4,7 @@* This code generates raw asm output which is post-processed to extract * and format the required data. */ +#define COMPILE_OFFSETS #include <linux/kbuild.h> #include <linux/sched.h>
arch/sh/kernel/asm-offsets.c +1 -0
@@ -8,6 +8,7 @@* compile this file to assembler, and then extract the * #defines from the assembly-language output. */ +#define COMPILE_OFFSETS #include <linux/stddef.h> #include <linux/types.h>
arch/sparc/kernel/asm-offsets.c +1 -0
@@ -10,6 +10,7 @@* * On sparc, thread_info data is static and TI_XXX offsets are computed by hand. */ +#define COMPILE_OFFSETS #include <linux/sched.h> #include <linux/mm_types.h>
arch/um/kernel/asm-offsets.c +2 -0
@@ -1 +1,3 @@ +#define COMPILE_OFFSETS +#include <sysdep/kernel-offsets.h>
arch/xtensa/kernel/asm-offsets.c +1 -0
@@ -11,6 +11,7 @@* * Chris Zankel <[email protected]> */ +#define COMPILE_OFFSETS #include <asm/processor.h> #include <asm/coprocessor.h>
Signature · 1 line
2.50.1
PATCH v2 2/3 sched: make migrate_enable/migrate_disable inline
In reply to Menglong DongFor now, migrate_enable and migrate_disable are global, which makes them become hotspots in some case. Take BPF for example, the function calling to migrate_enable and migrate_disable in BPF trampoline can introduce significant overhead, and following is the 'perf top' of FENTRY's benchmark (./tools/testing/selftests/bpf/bench trig-fentry):
54.63% bpf_prog_2dcccf652aac1793_bench_trigger_fentry [k]
bpf_prog_2dcccf652aac1793_bench_trigger_fentry
10.43% [kernel] [k] migrate_enable
10.07% bpf_trampoline_6442517037 [k] bpf_trampoline_6442517037 8.06% [kernel] [k] __bpf_prog_exit_recur
4.11% libc.so.6 [.] syscall
2.15% [kernel] [k] entry_SYSCALL_64
1.48% [kernel] [k] memchr_inv
1.32% [kernel] [k] fput
1.16% [kernel] [k] _copy_to_user
0.73% [kernel] [k] bpf_prog_test_run_raw_tp
So in this commit, we make migrate_enable/migrate_disable inline to obtain better performance. The struct rq is defined internally in kernel/sched/sched.h, and the field "nr_pinned" is accessed in migrate_enable/migrate_disable, which makes it hard to make them inline.
Alexei Starovoitov suggests to generate the offset of "nr_pinned" in [1], so we can define the migrate_enable/migrate_disable in include/linux/sched.h and access "this_rq()->nr_pinned" with "(void *)this_rq() + RQ_nr_pinned".
The offset of "nr_pinned" is generated in include/generated/rq-offsets.h by kernel/sched/rq-offsets.c.
Generally speaking, we move the definition of migrate_enable and migrate_disable to include/linux/sched.h from kernel/sched/core.c. The calling to __set_cpus_allowed_ptr() is leaved in __migrate_enable().
The "struct rq" is not available in include/linux/sched.h, so we can't access the "runqueues" with this_cpu_ptr(), as the compilation will fail in this_cpu_ptr() -> raw_cpu_ptr() -> __verify_pcpu_ptr():
typeof((ptr) + 0)
So we introduce the this_rq_raw() and access the runqueues with arch_raw_cpu_ptr() directly.
Before this patch, the performance of BPF FENTRY is:
fentry : 113.030 ± 0.149M/s fentry : 112.501 ± 0.187M/s fentry : 112.828 ± 0.267M/s fentry : 115.287 ± 0.241M/s
After this patch, the performance of BPF FENTRY increases to:
fentry : 143.644 ± 0.670M/s fentry : 149.764 ± 0.362M/s fentry : 149.642 ± 0.156M/s fentry : 145.263 ± 0.221M/s
---
v2:
- use PERCPU_PTR() for this_rq_raw() if !CONFIG_SMP
--- Kbuild | 13 ++++++- include/linux/preempt.h | 3 -- include/linux/sched.h | 77 +++++++++++++++++++++++++++++++++++++++ kernel/bpf/verifier.c | 3 +- kernel/sched/core.c | 56 ++-------------------------- kernel/sched/rq-offsets.c | 12 ++++++ 6 files changed, 106 insertions(+), 58 deletions(-) create mode 100644 kernel/sched/rq-offsets.c
Kbuild +12 -1
@@ -34,13 +34,24 @@ arch/$(SRCARCH)/kernel/asm-offsets.s: $(timeconst-file) $(bounds-file) $(offsets-file): arch/$(SRCARCH)/kernel/asm-offsets.s FORCE $(call filechk,offsets,__ASM_OFFSETS_H__) +# Generate rq-offsets.h + +rq-offsets-file := include/generated/rq-offsets.h + +targets += kernel/sched/rq-offsets.s + +kernel/sched/rq-offsets.s: $(offsets-file) + +$(rq-offsets-file): kernel/sched/rq-offsets.s FORCE + $(call filechk,offsets,__RQ_OFFSETS_H__) + # Check for missing system calls quiet_cmd_syscalls = CALL $< cmd_syscalls = $(CONFIG_SHELL) $< $(CC) $(c_flags) $(missing_syscalls_flags) PHONY += missing-syscalls -missing-syscalls: scripts/checksyscalls.sh $(offsets-file) +missing-syscalls: scripts/checksyscalls.sh $(rq-offsets-file) $(call cmd,syscalls) # Check the manual modification of atomic headers
include/linux/preempt.h +0 -3
@@ -424,8 +424,6 @@ static inline void preempt_notifier_init(struct preempt_notifier *notifier,* work-conserving schedulers. * */ -extern void migrate_disable (void ); -extern void migrate_enable (void );/** * preempt_disable_nested - Disable preemption inside a normally preempt disabled section
@@ -471,7 +469,6 @@ static __always_inline void preempt_enable_nested(void) DEFINE_LOCK_GUARD_0(preempt, preempt_disable(), preempt_enable()) DEFINE_LOCK_GUARD_0(preempt_notrace, preempt_disable_notrace(), preempt_enable_notrace()) -DEFINE_LOCK_GUARD_0(migrate, migrate_disable(), migrate_enable())#ifdef CONFIG_PREEMPT_DYNAMIC
include/linux/sched.h +77 -0
87 lines not on this page · raw message
kernel/bpf/verifier.c +1 -2
@@ -23855,8 +23855,7 @@ int bpf_check_attach_target(struct bpf_verifier_log *log,BTF_SET_START (btf_id_deny) BTF_ID_UNUSED#ifdef CONFIG_SMP -BTF_ID(func, migrate_disable) -BTF_ID(func, migrate_enable) +BTF_ID(func, __migrate_enable)#endif #if !definedCONFIG_PREEMPT_RCU && !definedCONFIG_TINY_RCU BTF_ID (func, rcu_read_unlock_strict)
kernel/sched/core.c +4 -52
75 lines not on this page · raw message
kernel/sched/rq-offsets.c new file +12 -0
@@ -0,0 +1,12 @@ +// SPDX-License-Identifier: GPL-2.0 +#define COMPILE_OFFSETS +#include <linux/kbuild.h> +#include <linux/types.h> +#include "sched.h" + +int main (void ) +{ +DEFINE (RQ_nr_pinned, offsetof(struct rq , nr_pinned)); + +return 0 ; +}
Signature · 1 line
2.50.1
Peter Zijlstra
maintainerkernel/sched/core.c EXPORT_TRACEPOINT_SYMBOL_GPL(sched_update_nr_running_tp); · line 122
Quoted 9 lines · Menglong Dong
kernel/sched/core.c +1
-0@@ -119,6 +119,7 @@ EXPORT_TRACEPOINT_SYMBOL_GPL(sched_update_nr_running_tp);EXPORT_TRACEPOINT_SYMBOL_GPL (sched_compute_energy_tp); DEFINE_PER_CPU_SHARED_ALIGNED(struct rq, runqueues); +EXPORT_SYMBOL_GPL (runqueues);
Oh no, absolutely not.
You never, ever, export a variable, and certainly not this one.
How about something like so?
I tried 'clever' things with export inline, but the compiler hates me, so the below is the best I could make work.
---
include/linux/sched.h +27 -6
@@ -2315,6 +2315,7 @@ static __always_inline void alloc_tag_re#define alloc_tag_restore (_tag, _old) do {} while (0)#endif +#ifndef MODULE #ifndef COMPILE_OFFSETS extern void __migrate_enable (void );
@@ -2328,7 +2329,7 @@ DECLARE_PER_CPU_SHARED_ALIGNED(struct rq#define this_rq_raw () PERCPU_PTR(&runqueues)#endif -static inline void migrate_enable (void ) +static inline void _migrate_enable (void ) {struct task_struct *p = current;
@@ -2363,7 +2364,7 @@ static inline void migrate_enable(void) (*(unsigned int *)((void *)this_rq_raw () + RQ_nr_pinned))--; } -static inline void migrate_disable (void ) +static inline void _migrate_disable (void ) {struct task_struct *p = current;
@@ -2382,10 +2383,30 @@ static inline void migrate_disable(void) (*(unsigned int *)((void *)this_rq_raw () + RQ_nr_pinned))++; p->migration_disabled =1 ; } -#else -static inline void migrate_disable (void ) { } -static inline void migrate_enable (void ) { } -#endif +#else /* !COMPILE_OFFSETS */ +static inline void _migrate_disable (void ) { } +static inline void _migrate_enable (void ) { } +#endif /* !COMPILE_OFFSETS */ + +#ifndef CREATE_MIGRATE_DISABLE +static inline void migrate_disable (void ) +{ +_migrate_disable (); +} + +static inline void migrate_enable (void ) +{ +_migrate_enable (); +} +#else /* CREATE_MIGRATE_DISABLE */ +extern void migrate_disable (void ); +extern void migrate_enable (void ); +#endif /* CREATE_MIGRATE_DISABLE */ + +#else /* !MODULE */ +extern void migrate_disable (void ); +extern void migrate_enable (void ); +#endif /* !MODULE */ DEFINE_LOCK_GUARD_0 (migrate,migrate_disable (),migrate_enable ())
kernel/sched/core.c +13 -2
@@ -7,6 +7,9 @@* Copyright (C) 1991-2002 Linus Torvalds * Copyright (C) 1998-2024 Ingo Molnar, Red Hat */ +#define CREATE_MIGRATE_DISABLE +#include <linux/sched.h> +#include <linux/highmem.h> #include <linux/hrtimer_api.h> #include <linux/ktime_api.h>
@@ -119,7 +122,6 @@ EXPORT_TRACEPOINT_SYMBOL_GPL(sched_updatEXPORT_TRACEPOINT_SYMBOL_GPL (sched_compute_energy_tp); DEFINE_PER_CPU_SHARED_ALIGNED(struct rq, runqueues); -EXPORT_SYMBOL_GPL (runqueues);#ifdef CONFIG_SCHED_PROXY_EXEC DEFINE_STATIC_KEY_TRUE (__sched_proxy_exec);
@@ -2382,6 +2384,11 @@ static void migrate_disable_switch(struc__do_set_cpus_allowed (p, &ac); } +void migrate_disable (void ) +{ +_migrate_disable (); +} +void __migrate_enable (void ) {struct task_struct *p = current;
@@ -2392,7 +2399,11 @@ void __migrate_enable(void)__set_cpus_allowed_ptr (p, &ac); } -EXPORT_SYMBOL_GPL (__migrate_enable); + +void migrate_enable (void ) +{ +_migrate_enable (); +}static inline bool rq_has_pinned_tasks (struct rq *rq) {
Peter Zijlstra
maintainerQuoted 14 lines · Peter Zijlstra
kernel/sched/core.c EXPORT_TRACEPOINT_SYMBOL_GPL(sched_update_nr_running_tp); · line 122
Quoted 9 lines · Menglong Dong
kernel/sched/core.c +1
-0@@ -119,6 +119,7 @@ EXPORT_TRACEPOINT_SYMBOL_GPL(sched_update_nr_running_tp);EXPORT_TRACEPOINT_SYMBOL_GPL (sched_compute_energy_tp); DEFINE_PER_CPU_SHARED_ALIGNED(struct rq, runqueues); +EXPORT_SYMBOL_GPL (runqueues);Oh no, absolutely not.
Quoted 6 lines
You never, ever, export a variable, and certainly not this one.
How about something like so?
I tried 'clever' things with export inline, but the compiler hates me, so the below is the best I could make work.
extern inline, that is, obviously...
Quoted 5 lines · Peter Zijlstra
Quoted 1 line
+EXPORT_SYMBOL_GPL(runqueues);
Oh no, absolutely not.
You never, ever, export a variable, and certainly not this one.
Tangential thought:
I think it would be possible to warn about non-function exports at build time, and maybe plug it in W=1 builds.
BR,
Jani.
Signature · 1 line
Jani Nikula, Intel
Peter Zijlstra
maintainerQuoted 12 lines · Jani Nikula
Quoted 5 lines · Peter Zijlstra
Quoted 1 line
+EXPORT_SYMBOL_GPL(runqueues);
Oh no, absolutely not.
You never, ever, export a variable, and certainly not this one.
Tangential thought:
I think it would be possible to warn about non-function exports at build time, and maybe plug it in W=1 builds.
Too much noise, there's a metric ton of variables exported. Sometimes its unavoidable.
I just try and avoid wherever possible.
Quoted 15 lines · Peter Zijlstra
kernel/sched/core.c EXPORT_TRACEPOINT_SYMBOL_GPL(sched_update_nr_running_tp); · line 122
Quoted 9 lines · Menglong Dong
kernel/sched/core.c +1
-0@@ -119,6 +119,7 @@ EXPORT_TRACEPOINT_SYMBOL_GPL(sched_update_nr_running_tp);EXPORT_TRACEPOINT_SYMBOL_GPL (sched_compute_energy_tp); DEFINE_PER_CPU_SHARED_ALIGNED(struct rq, runqueues); +EXPORT_SYMBOL_GPL (runqueues);Oh no, absolutely not.
Quoted 6 lines
You never, ever, export a variable, and certainly not this one.
How about something like so?
I tried 'clever' things with export inline, but the compiler hates me, so the below is the best I could make work.
I see. You mean that we don't export the various, and use the inlined version in vmlinux, and use the external version in modules, which I think is nice ;)
(I were not aware that we should export various :/)
I'll try your advice.
Thanks!
Menglong Dong
Quoted 110 lines
---
include/linux/sched.h +27
-6@@ -2315,6 +2315,7 @@ static __always_inline void alloc_tag_re#define alloc_tag_restore (_tag, _old) do {} while (0)#endif +#ifndef MODULE #ifndef COMPILE_OFFSETS extern void __migrate_enable (void );@@ -2328,7 +2329,7 @@ DECLARE_PER_CPU_SHARED_ALIGNED(struct rq#define this_rq_raw () PERCPU_PTR(&runqueues)#endif -static inline void migrate_enable (void ) +static inline void _migrate_enable (void ) {struct task_struct *p = current;@@ -2363,7 +2364,7 @@ static inline void migrate_enable(void) (*(unsigned int *)((void *)this_rq_raw () + RQ_nr_pinned))--; } -static inline void migrate_disable (void ) +static inline void _migrate_disable (void ) {struct task_struct *p = current;@@ -2382,10 +2383,30 @@ static inline void migrate_disable(void) (*(unsigned int *)((void *)this_rq_raw () + RQ_nr_pinned))++; p->migration_disabled =1 ; } -#else -static inline void migrate_disable (void ) { } -static inline void migrate_enable (void ) { } -#endif +#else /* !COMPILE_OFFSETS */ +static inline void _migrate_disable (void ) { } +static inline void _migrate_enable (void ) { } +#endif /* !COMPILE_OFFSETS */ + +#ifndef CREATE_MIGRATE_DISABLE +static inline void migrate_disable (void ) +{ +_migrate_disable (); +} + +static inline void migrate_enable (void ) +{ +_migrate_enable (); +} +#else /* CREATE_MIGRATE_DISABLE */ +extern void migrate_disable (void ); +extern void migrate_enable (void ); +#endif /* CREATE_MIGRATE_DISABLE */ + +#else /* !MODULE */ +extern void migrate_disable (void ); +extern void migrate_enable (void ); +#endif /* !MODULE */ DEFINE_LOCK_GUARD_0 (migrate,migrate_disable (),migrate_enable ())kernel/sched/core.c +13
-2@@ -7,6 +7,9 @@* Copyright (C) 1991-2002 Linus Torvalds * Copyright (C) 1998-2024 Ingo Molnar, Red Hat */ +#define CREATE_MIGRATE_DISABLE +#include <linux/sched.h> +#include <linux/highmem.h> #include <linux/hrtimer_api.h> #include <linux/ktime_api.h> @@ -119,7 +122,6 @@ EXPORT_TRACEPOINT_SYMBOL_GPL(sched_updatEXPORT_TRACEPOINT_SYMBOL_GPL (sched_compute_energy_tp); DEFINE_PER_CPU_SHARED_ALIGNED(struct rq, runqueues); -EXPORT_SYMBOL_GPL (runqueues);#ifdef CONFIG_SCHED_PROXY_EXEC DEFINE_STATIC_KEY_TRUE (__sched_proxy_exec);@@ -2382,6 +2384,11 @@ static void migrate_disable_switch(struc__do_set_cpus_allowed (p, &ac); } +void migrate_disable (void ) +{ +_migrate_disable (); +} +void __migrate_enable (void ) {struct task_struct *p = current;@@ -2392,7 +2399,11 @@ void __migrate_enable(void)__set_cpus_allowed_ptr (p, &ac); } -EXPORT_SYMBOL_GPL (__migrate_enable); + +void migrate_enable (void ) +{ +_migrate_enable (); +}static inline bool rq_has_pinned_tasks (struct rq *rq) {
Peter Zijlstra
maintainerQuoted 7 lines · Menglong Dong
The "struct rq" is not available in
include/linux/sched.h, so we can't access the "runqueues" withthis_cpu_ptr(), as the compilation will fail inthis_cpu_ptr()->raw_cpu_ptr()->__verify_pcpu_ptr():
typeof((ptr) + 0)So we introduce the
this_rq_raw()and access the runqueues witharch_raw_cpu_ptr()directly.
^ That, wants to be a comment near here:
Quoted 6 lines
include/linux/sched.h +2
-0@@ -2312,4 +2315,78 @@ static __always_inline void alloc_tag_restore(struct alloc_tag *tag, struct allo#define alloc_tag_restore (_tag, _old) do {} while (0)#endif +#ifndef COMPILE_OFFSETS +
include/linux/sched.h static __always_inline void alloc_tag_restore(struct alloc_tag *tag, struct allo · line 2329
Quoted 10 lines
include/linux/sched.h +10
-0+extern void __migrate_enable (void ); + +struct rq ; +DECLARE_PER_CPU_SHARED_ALIGNED(struct rq, runqueues); + +#ifdef CONFIG_SMP +#define this_rq_raw () arch_raw_cpu_ptr(&runqueues) +#else +#define this_rq_raw () PERCPU_PTR(&runqueues) +#endif
Because that arch_ thing really is weird.
include/linux/sched.h static __always_inline void alloc_tag_restore(struct alloc_tag *tag, struct allo · line 2382
Quoted 2 lines
include/linux/sched.h +2
-0+ (*(unsigned int *)((void *)this_rq_raw () + RQ_nr_pinned))--; + (*(unsigned int *)((void *)this_rq_raw () + RQ_nr_pinned))++;
And since you did a macro anyway, why not fold that magic in there, instead of duplicating it?
#define __this_rq_raw() ((void *)arch_raw_cpu_ptr(&runqueues)) #define this_rq_pinned() (*(unsigned int *)(__this_rq_raw() + RQ_nr_pinned))
this_rq_pinned()--; this_rq_pinned()++;
is nicer, no?
Quoted 17 lines · Peter Zijlstra
Quoted 7 lines · Menglong Dong
The "struct rq" is not available in
include/linux/sched.h, so we can't access the "runqueues" withthis_cpu_ptr(), as the compilation will fail inthis_cpu_ptr()->raw_cpu_ptr()->__verify_pcpu_ptr():
typeof((ptr) + 0)So we introduce the
this_rq_raw()and access the runqueues witharch_raw_cpu_ptr()directly.^ That, wants to be a comment near here:
Quoted 4 lines
include/linux/sched.h +0
-0@@ -2312,4 +2315,78 @@ static __always_inline void alloc_tag_restore(struct alloc_tag *tag, struct allo#define alloc_tag_restore (_tag, _old) do {} while (0)#endif
Quoted 14 lines
include/linux/sched.h static __always_inline void alloc_tag_restore(struct alloc_tag *tag, struct allo · line 2329
Quoted 12 lines
include/linux/sched.h +12
-0+#ifndef COMPILE_OFFSETS + +extern void __migrate_enable (void ); + +struct rq ; +DECLARE_PER_CPU_SHARED_ALIGNED(struct rq, runqueues); + +#ifdef CONFIG_SMP +#define this_rq_raw () arch_raw_cpu_ptr(&runqueues) +#else +#define this_rq_raw () PERCPU_PTR(&runqueues) +#endif Because that
arch_thing really is weird.
OK! I'll comment on this part.
Quoted 7 lines
include/linux/sched.h static __always_inline void alloc_tag_restore(struct alloc_tag *tag, struct allo · line 2382
Quoted 3 lines
include/linux/sched.h +2
-0+ (*(unsigned int *)((void *)this_rq_raw () + RQ_nr_pinned))--; + (*(unsigned int *)((void *)this_rq_raw () + RQ_nr_pinned))++;And since you did a macro anyway, why not fold that magic in there, instead of duplicating it?
Quoted 7 lines
#define
__this_rq_raw()((void *)arch_raw_cpu_ptr(&runqueues)) #definethis_rq_pinned()(*(unsigned int *)(__this_rq_raw() +RQ_nr_pinned))this_rq_pinned()--; this_rq_pinned()++;is nicer, no?
Yeah, much better!
PATCH v2 3/3 sched: fix some typos in include/linux/preempt.h
In reply to Menglong DongThere are some typos in the comments of migrate in include/linux/preempt.h:
elegible -> eligible
it's -> its
migirate_disable -> migrate_disable
abritrary -> arbitrary
Just fix them.
--- include/linux/preempt.h | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-)
include/linux/preempt.h +4 -4
@@ -372,7 +372,7 @@ static inline void preempt_notifier_init(struct preempt_notifier *notifier,/* * Migrate-Disable and why it is undesired. * -* When a preempted task becomes elegible to run under the ideal model (IOW it +* When a preempted task becomes eligible to run under the ideal model (IOW it * becomes one of the M highest priority tasks), it might still have to wait * for the preemptee's migrate_disable() section to complete. Thereby suffering * a reduction in bandwidth in the exact duration of the migrate_disable()
@@ -387,7 +387,7 @@ static inline void preempt_notifier_init(struct preempt_notifier *notifier,* - a lower priority tasks; which under preempt_disable() could've instantly * migrated away when another CPU becomes available, is now constrained * by the ability to push the higher priority task away, which might itself be -* in a migrate_disable() section, reducing it's available bandwidth. +* in a migrate_disable() section, reducing its available bandwidth. * * IOW it trades latency / moves the interference term, but it stays in the * system, and as long as it remains unbounded, the system is not fully
@@ -399,7 +399,7 @@ static inline void preempt_notifier_init(struct preempt_notifier *notifier,* PREEMPT_RT breaks a number of assumptions traditionally held. By forcing a * number of primitives into becoming preemptible, they would also allow * migration. This turns out to break a bunch of per-cpu usage. To this end, -* all these primitives employ migirate_disable() to restore this implicit +* all these primitives employ migrate_disable() to restore this implicit * assumption. * * This is a 'temporary' work-around at best. The correct solution is getting
@@ -407,7 +407,7 @@ static inline void preempt_notifier_init(struct preempt_notifier *notifier,* per-cpu locking or short preempt-disable regions. * * The end goal must be to get rid of migrate_disable(), alternatively we need -* a schedulability theory that does not depend on abritrary migration. +* a schedulability theory that does not depend on arbitrary migration. * * * Notes on the implementation.
Signature · 1 line
2.50.1