Skip to contentSkip to conversation

sched: make migrate_enable/migrate_disable inline

PATCH In review Menglong Dong 4 participants 12 messages Aug 28, 2025 – Sep 17, 2025

Subsystems (23)
Arm (32-bit) PowerPC MIPS BPF +19 more subsystems
lore
Share & tools
lore
#0 lore raw

v4 cover sched: make migrate_enable/migrate_disable inline

In this series, we make migrate_enable/migrate_disable inline to obtain better performance in some case.

In the first patch, we add the macro "COMPILE_OFFSETS" to all the asm-offset.c to avoid circular dependency in the 2nd patch.

In the 2nd patch, we generate the offset of nr_pinned in "struct rq" with rq-offsets.c, as the "struct rq" is defined internally and we need to access the "nr_pinned" field in migrate_enable and migrate_disable. Then, we move the definition of migrate_enable/migrate_disable from kernel/sched/core.c to include/linux/sched.h.

In the 3rd patch, we fix some typos in include/linux/preempt.h.

One of the beneficiaries of this series is BPF trampoline. Without this series, the migrate_enable/migrate_disable is hot when we run the benchmark for FENTRY, FEXIT, MODIFY_RETURN, etc:

54.63% bpf_prog_2dcccf652aac1793_bench_trigger_fentry [k]

                 bpf_prog_2dcccf652aac1793_bench_trigger_fentry

10.43% [kernel] [k] migrate_enable
10.07% bpf_trampoline_6442517037 [k] bpf_trampoline_6442517037 8.06% [kernel] [k] __bpf_prog_exit_recur
4.11% libc.so.6 [.] syscall
2.15% [kernel] [k] entry_SYSCALL_64
1.48% [kernel] [k] memchr_inv
1.32% [kernel] [k] fput
1.16% [kernel] [k] _copy_to_user
0.73% [kernel] [k] bpf_prog_test_run_raw_tp

Before this patch, the performance of BPF FENTRY is:

  fentry         :  113.030 ± 0.149M/s
  fentry         :  112.501 ± 0.187M/s
  fentry         :  112.828 ± 0.267M/s
  fentry         :  115.287 ± 0.241M/s

After this patch, the performance of BPF FENTRY increases to:

  fentry         :  143.644 ± 0.670M/s
  fentry         :  149.764 ± 0.362M/s
  fentry         :  149.642 ± 0.156M/s
  fentry         :  145.263 ± 0.221M/s

Changes since V3:
* some modification on the 2nd patch, as Alexei advised:
- rename CREATE_MIGRATE_DISABLE to INSTANTIATE_EXPORTED_MIGRATE_DISABLE
- add document for INSTANTIATE_EXPORTED_MIGRATE_DISABLE

Changes since V2:
* some modification on the 2nd patch, as Peter advised:
- don't export runqueues, define migrate_enable and migrate_disable in kernel/sched/core.c and use them for kernel modules instead
- define the macro this_rq_pinned()
- add some comment for this_rq_raw()

Changes since V1:
* use PERCPU_PTR() for this_rq_raw() if !CONFIG_SMP in the 2nd patch

Menglong Dong (3):
arch: add the macro COMPILE_OFFSETS to all the asm-offsets.c
sched: make migrate_enable/migrate_disable inline
sched: fix some typos in include/linux/preempt.h

 Kbuild                               |  13 ++-
 arch/alpha/kernel/asm-offsets.c      |   1 +
 arch/arc/kernel/asm-offsets.c        |   1 +
 arch/arm/kernel/asm-offsets.c        |   2 +
 arch/arm64/kernel/asm-offsets.c      |   1 +
 arch/csky/kernel/asm-offsets.c       |   1 +
 arch/hexagon/kernel/asm-offsets.c    |   1 +
 arch/loongarch/kernel/asm-offsets.c  |   2 +
 arch/m68k/kernel/asm-offsets.c       |   1 +
 arch/microblaze/kernel/asm-offsets.c |   1 +
 arch/mips/kernel/asm-offsets.c       |   2 +
 arch/nios2/kernel/asm-offsets.c      |   1 +
 arch/openrisc/kernel/asm-offsets.c   |   1 +
 arch/parisc/kernel/asm-offsets.c     |   1 +
 arch/powerpc/kernel/asm-offsets.c    |   1 +
 arch/riscv/kernel/asm-offsets.c      |   1 +
 arch/s390/kernel/asm-offsets.c       |   1 +
 arch/sh/kernel/asm-offsets.c         |   1 +
 arch/sparc/kernel/asm-offsets.c      |   1 +
 arch/um/kernel/asm-offsets.c         |   2 +
 arch/xtensa/kernel/asm-offsets.c     |   1 +
 include/linux/preempt.h              |  11 +--
 include/linux/sched.h                | 113 +++++++++++++++++++++++++++
 kernel/bpf/verifier.c                |   1 +
 kernel/sched/core.c                  |  63 ++++-----------
 kernel/sched/rq-offsets.c            |  12 +++
 26 files changed, 180 insertions(+), 57 deletions(-)
 create mode 100644 kernel/sched/rq-offsets.c
Signature · 1 line
2.51.0
#1 lore raw

PATCH v4 1/3 arch: add the macro COMPILE_OFFSETS to all the asm-offsets.c

The include/generated/asm-offsets.h is generated in Kbuild during compiling from arch/SRCARCH/kernel/asm-offsets.c. When we want to generate another similar offset header file, circular dependency can happen.

For example, we want to generate a offset file include/generated/test.h, which is included in include/sched/sched.h. If we generate asm-offsets.h first, it will fail, as include/sched/sched.h is included in asm-offsets.c and include/generated/test.h doesn't exist; If we generate test.h first, it can't success neither, as include/generated/asm-offsets.h is included by it.

In x86_64, the macro COMPILE_OFFSETS is used to avoid such circular dependency. We can generate asm-offsets.h first, and if the COMPILE_OFFSETS is defined, we don't include the "generated/test.h".

And we define the macro COMPILE_OFFSETS for all the asm-offsets.c for this purpose.

---
 arch/alpha/kernel/asm-offsets.c      | 1 +
 arch/arc/kernel/asm-offsets.c        | 1 +
 arch/arm/kernel/asm-offsets.c        | 2 ++
 arch/arm64/kernel/asm-offsets.c      | 1 +
 arch/csky/kernel/asm-offsets.c       | 1 +
 arch/hexagon/kernel/asm-offsets.c    | 1 +
 arch/loongarch/kernel/asm-offsets.c  | 2 ++
 arch/m68k/kernel/asm-offsets.c       | 1 +
 arch/microblaze/kernel/asm-offsets.c | 1 +
 arch/mips/kernel/asm-offsets.c       | 2 ++
 arch/nios2/kernel/asm-offsets.c      | 1 +
 arch/openrisc/kernel/asm-offsets.c   | 1 +
 arch/parisc/kernel/asm-offsets.c     | 1 +
 arch/powerpc/kernel/asm-offsets.c    | 1 +
 arch/riscv/kernel/asm-offsets.c      | 1 +
 arch/s390/kernel/asm-offsets.c       | 1 +
 arch/sh/kernel/asm-offsets.c         | 1 +
 arch/sparc/kernel/asm-offsets.c      | 1 +
 arch/um/kernel/asm-offsets.c         | 2 ++
 arch/xtensa/kernel/asm-offsets.c     | 1 +
 20 files changed, 24 insertions(+)
arch/alpha/kernel/asm-offsets.c +1 -0
index e9dad60b147f..1ebb05890499 100644
@@ -4,6 +4,7 @@
  * This code generates raw asm output which is post-processed to extract
  * and format the required data.
  */
+#define COMPILE_OFFSETS
 
 #include <linux/types.h>
 #include <linux/stddef.h>
arch/arc/kernel/asm-offsets.c +1 -0
index f77deb799175..2978da85fcb6 100644
@@ -2,6 +2,7 @@
 /*
  * Copyright (C) 2004, 2007-2010, 2011-2012 Synopsys, Inc. (www.synopsys.com)
  */
+#define COMPILE_OFFSETS
 
 #include <linux/sched.h>
 #include <linux/mm.h>
arch/arm/kernel/asm-offsets.c +2 -0
index 123f4a8ef446..2101938d27fc 100644
@@ -7,6 +7,8 @@
  * This code generates raw asm output which is post-processed to extract
  * and format the required data.
  */
+#define COMPILE_OFFSETS
+
 #include <linux/compiler.h>
 #include <linux/sched.h>
 #include <linux/mm.h>
arch/arm64/kernel/asm-offsets.c +1 -0
index 30d4bbe68661..b6367ff3a49c 100644
@@ -6,6 +6,7 @@
  *               2001-2002 Keith Owens
  * Copyright (C) 2012 ARM Ltd.
  */
+#define COMPILE_OFFSETS
 
 #include <linux/arm_sdei.h>
 #include <linux/sched.h>
arch/csky/kernel/asm-offsets.c +1 -0
index d1e903579473..5525c8e7e1d9 100644
@@ -1,5 +1,6 @@
 // SPDX-License-Identifier: GPL-2.0
 // Copyright (C) 2018 Hangzhou C-SKY Microsystems co.,ltd.
+#define COMPILE_OFFSETS
 
 #include <linux/sched.h>
 #include <linux/kernel_stat.h>
arch/hexagon/kernel/asm-offsets.c +1 -0
index 03a7063f9456..50eea9fa6f13 100644
@@ -8,6 +8,7 @@
  *
  * Copyright (c) 2010-2012, The Linux Foundation. All rights reserved.
  */
+#define COMPILE_OFFSETS
 
 #include <linux/compat.h>
 #include <linux/types.h>
arch/loongarch/kernel/asm-offsets.c +2 -0
index db1e4bb26b6a..3017c7157600 100644
@@ -4,6 +4,8 @@
  *
  * Copyright (C) 2020-2022 Loongson Technology Corporation Limited
  */
+#define COMPILE_OFFSETS
+
 #include <linux/types.h>
 #include <linux/sched.h>
 #include <linux/mm.h>
arch/m68k/kernel/asm-offsets.c +1 -0
index 906d73230537..67a1990f9d74 100644
@@ -9,6 +9,7 @@
  * #defines from the assembly-language output.
  */
 
+#define COMPILE_OFFSETS
 #define ASM_OFFSETS_C
 
 #include <linux/stddef.h>
arch/microblaze/kernel/asm-offsets.c +1 -0
index 104c3ac5f30c..b4b67d58e7f6 100644
@@ -7,6 +7,7 @@
  * License. See the file "COPYING" in the main directory of this archive
  * for more details.
  */
+#define COMPILE_OFFSETS
 
 #include <linux/init.h>
 #include <linux/stddef.h>
arch/mips/kernel/asm-offsets.c +2 -0
index 1e29efcba46e..5debd9a3854a 100644
@@ -9,6 +9,8 @@
  * Kevin Kissell, [email protected] and Carsten Langgaard, [email protected]
  * Copyright (C) 2000 MIPS Technologies, Inc.
  */
+#define COMPILE_OFFSETS
+
 #include <linux/compat.h>
 #include <linux/types.h>
 #include <linux/sched.h>
arch/nios2/kernel/asm-offsets.c +1 -0
index e3d9b7b6fb48..88190b503ce5 100644
@@ -2,6 +2,7 @@
 /*
  * Copyright (C) 2011 Tobias Klauser <[email protected]>
  */
+#define COMPILE_OFFSETS
 
 #include <linux/stddef.h>
 #include <linux/sched.h>
arch/openrisc/kernel/asm-offsets.c +1 -0
index 710651d5aaae..3cc826f2216b 100644
@@ -18,6 +18,7 @@
  * compile this file to assembler, and then extract the
  * #defines from the assembly-language output.
  */
+#define COMPILE_OFFSETS
 
 #include <linux/signal.h>
 #include <linux/sched.h>
arch/parisc/kernel/asm-offsets.c +1 -0
index 757816a7bd4b..9abfe65492c6 100644
@@ -13,6 +13,7 @@
  *    Copyright (C) 2002 Randolph Chung <tausq with parisc-linux.org>
  *    Copyright (C) 2003 James Bottomley <jejb at parisc-linux.org>
  */
+#define COMPILE_OFFSETS
 
 #include <linux/types.h>
 #include <linux/sched.h>
arch/powerpc/kernel/asm-offsets.c +1 -0
index b3048f6d3822..a4bc80b30410 100644
@@ -8,6 +8,7 @@
  * compile this file to assembler, and then extract the
  * #defines from the assembly-language output.
  */
+#define COMPILE_OFFSETS
 
 #include <linux/compat.h>
 #include <linux/signal.h>
arch/riscv/kernel/asm-offsets.c +1 -0
index 6e8c0d6feae9..7d42d3b8a32a 100644
@@ -3,6 +3,7 @@
  * Copyright (C) 2012 Regents of the University of California
  * Copyright (C) 2017 SiFive
  */
+#define COMPILE_OFFSETS
 
 #include <linux/kbuild.h>
 #include <linux/mm.h>
arch/s390/kernel/asm-offsets.c +1 -0
index 95ecad9c7d7d..a8915663e917 100644
@@ -4,6 +4,7 @@
  * This code generates raw asm output which is post-processed to extract
  * and format the required data.
  */
+#define COMPILE_OFFSETS
 
 #include <linux/kbuild.h>
 #include <linux/sched.h>
arch/sh/kernel/asm-offsets.c +1 -0
index a0322e832845..429b6a763146 100644
@@ -8,6 +8,7 @@
  * compile this file to assembler, and then extract the
  * #defines from the assembly-language output.
  */
+#define COMPILE_OFFSETS
 
 #include <linux/stddef.h>
 #include <linux/types.h>
arch/sparc/kernel/asm-offsets.c +1 -0
index 3d9b9855dce9..6e660bde48dd 100644
@@ -10,6 +10,7 @@
  *
  * On sparc, thread_info data is static and TI_XXX offsets are computed by hand.
  */
+#define COMPILE_OFFSETS
 
 #include <linux/sched.h>
 #include <linux/mm_types.h>
arch/um/kernel/asm-offsets.c +2 -0
index 1fb12235ab9c..a69873aa697f 100644
@@ -1 +1,3 @@
+#define COMPILE_OFFSETS
+
 #include <sysdep/kernel-offsets.h>
arch/xtensa/kernel/asm-offsets.c +1 -0
index da38de20ae59..cfbced95e944 100644
@@ -11,6 +11,7 @@
  *
  * Chris Zankel <[email protected]>
  */
+#define COMPILE_OFFSETS
 
 #include <asm/processor.h>
 #include <asm/coprocessor.h>
Signature · 1 line
2.51.0
#2 lore raw

PATCH v4 2/3 sched: make migrate_enable/migrate_disable inline

In reply to Menglong Dong

For now, migrate_enable and migrate_disable are global, which makes them become hotspots in some case. Take BPF for example, the function calling to migrate_enable and migrate_disable in BPF trampoline can introduce significant overhead, and following is the 'perf top' of FENTRY's benchmark (./tools/testing/selftests/bpf/bench trig-fentry):

54.63% bpf_prog_2dcccf652aac1793_bench_trigger_fentry [k]

                 bpf_prog_2dcccf652aac1793_bench_trigger_fentry

10.43% [kernel] [k] migrate_enable
10.07% bpf_trampoline_6442517037 [k] bpf_trampoline_6442517037 8.06% [kernel] [k] __bpf_prog_exit_recur
4.11% libc.so.6 [.] syscall
2.15% [kernel] [k] entry_SYSCALL_64
1.48% [kernel] [k] memchr_inv
1.32% [kernel] [k] fput
1.16% [kernel] [k] _copy_to_user
0.73% [kernel] [k] bpf_prog_test_run_raw_tp

So in this commit, we make migrate_enable/migrate_disable inline to obtain better performance. The struct rq is defined internally in kernel/sched/sched.h, and the field "nr_pinned" is accessed in migrate_enable/migrate_disable, which makes it hard to make them inline.

Alexei Starovoitov suggests to generate the offset of "nr_pinned" in [1], so we can define the migrate_enable/migrate_disable in include/linux/sched.h and access "this_rq()->nr_pinned" with "(void *)this_rq() + RQ_nr_pinned".

The offset of "nr_pinned" is generated in include/generated/rq-offsets.h by kernel/sched/rq-offsets.c.

Generally speaking, we move the definition of migrate_enable and migrate_disable to include/linux/sched.h from kernel/sched/core.c. The calling to __set_cpus_allowed_ptr() is leaved in ___migrate_enable().

The "struct rq" is not available in include/linux/sched.h, so we can't access the "runqueues" with this_cpu_ptr(), as the compilation will fail in this_cpu_ptr() -> raw_cpu_ptr() -> __verify_pcpu_ptr():
typeof((ptr) + 0)

So we introduce the this_rq_raw() and access the runqueues with arch_raw_cpu_ptr/PERCPU_PTR directly.

The variable "runqueues" is not visible in the kernel modules, and export it is not a good idea. As Peter Zijlstra advised in [2], we define and export migrate_enable/migrate_disable in kernel/sched/core.c too, and use them for the modules.

Before this patch, the performance of BPF FENTRY is:

  fentry         :  113.030 ± 0.149M/s
  fentry         :  112.501 ± 0.187M/s
  fentry         :  112.828 ± 0.267M/s
  fentry         :  115.287 ± 0.241M/s

After this patch, the performance of BPF FENTRY increases to:

  fentry         :  143.644 ± 0.670M/s
  fentry         :  149.764 ± 0.362M/s
  fentry         :  149.642 ± 0.156M/s
  fentry         :  145.263 ± 0.221M/s

---
v4:
- rename CREATE_MIGRATE_DISABLE to INSTANTIATE_EXPORTED_MIGRATE_DISABLE
- add document for INSTANTIATE_EXPORTED_MIGRATE_DISABLE

v3:
- don't export runqueues, define migrate_enable and migrate_disable in kernel/sched/core.c and use them for kernel modules instead
- define the macro this_rq_pinned()
- add some comment for this_rq_raw()

v2:
- use PERCPU_PTR() for this_rq_raw() if !CONFIG_SMP

---
 Kbuild                    |  13 ++++-
 include/linux/preempt.h   |   3 -
 include/linux/sched.h     | 113 ++++++++++++++++++++++++++++++++++++++
 kernel/bpf/verifier.c     |   1 +
 kernel/sched/core.c       |  63 +++++----------------
 kernel/sched/rq-offsets.c |  12 ++++
 6 files changed, 152 insertions(+), 53 deletions(-)
 create mode 100644 kernel/sched/rq-offsets.c
Kbuild +12 -1
index f327ca86990c..13324b4bbe23 100644
@@ -34,13 +34,24 @@ arch/$(SRCARCH)/kernel/asm-offsets.s: $(timeconst-file) $(bounds-file)
 $(offsets-file): arch/$(SRCARCH)/kernel/asm-offsets.s FORCE
 	$(call filechk,offsets,__ASM_OFFSETS_H__)
 
+# Generate rq-offsets.h
+
+rq-offsets-file := include/generated/rq-offsets.h
+
+targets += kernel/sched/rq-offsets.s
+
+kernel/sched/rq-offsets.s: $(offsets-file)
+
+$(rq-offsets-file): kernel/sched/rq-offsets.s FORCE
+	$(call filechk,offsets,__RQ_OFFSETS_H__)
+
 # Check for missing system calls
 
 quiet_cmd_syscalls = CALL    $<
       cmd_syscalls = $(CONFIG_SHELL) $< $(CC) $(c_flags) $(missing_syscalls_flags)
 
 PHONY += missing-syscalls
-missing-syscalls: scripts/checksyscalls.sh $(offsets-file)
+missing-syscalls: scripts/checksyscalls.sh $(rq-offsets-file)
 	$(call cmd,syscalls)
 
 # Check the manual modification of atomic headers
include/linux/preempt.h +0 -3
index 1fad1c8a4c76..92237c319035 100644
@@ -424,8 +424,6 @@ static inline void preempt_notifier_init(struct preempt_notifier *notifier,
  *       work-conserving schedulers.
  *
  */
-extern void migrate_disable(void);
-extern void migrate_enable(void);
 
 /**
  * preempt_disable_nested - Disable preemption inside a normally preempt disabled section
@@ -471,7 +469,6 @@ static __always_inline void preempt_enable_nested(void)
 
 DEFINE_LOCK_GUARD_0(preempt, preempt_disable(), preempt_enable())
 DEFINE_LOCK_GUARD_0(preempt_notrace, preempt_disable_notrace(), preempt_enable_notrace())
-DEFINE_LOCK_GUARD_0(migrate, migrate_disable(), migrate_enable())
 
 #ifdef CONFIG_PREEMPT_DYNAMIC
 
include/linux/sched.h +113 -0

123 lines not on this page · raw message

kernel/bpf/verifier.c +1 -0
index c4f69a9e9af6..de9078a9df3a 100644
@@ -23855,6 +23855,7 @@ int bpf_check_attach_target(struct bpf_verifier_log *log,
 BTF_SET_START(btf_id_deny)
 BTF_ID_UNUSED
 #ifdef CONFIG_SMP
+BTF_ID(func, ___migrate_enable)
 BTF_ID(func, migrate_disable)
 BTF_ID(func, migrate_enable)
 #endif
kernel/sched/core.c +14 -49

83 lines not on this page · raw message

kernel/sched/rq-offsets.c new file +12 -0
new file mode 100644
index 000000000000..a23747bbe25b
@@ -0,0 +1,12 @@
+// SPDX-License-Identifier: GPL-2.0
+#define COMPILE_OFFSETS
+#include <linux/kbuild.h>
+#include <linux/types.h>
+#include "sched.h"
+
+int main(void)
+{
+	DEFINE(RQ_nr_pinned, offsetof(struct rq, nr_pinned));
+
+	return 0;
+}
Signature · 1 line
2.51.0
maintainer
#3 lore raw
Quoted 7 lines · Menglong Dong

+/* The "struct rq" is not available here, so we can't access the
+ * "runqueues" with this_cpu_ptr(), as the compilation will fail in
+ * this_cpu_ptr() -> raw_cpu_ptr() -> __verify_pcpu_ptr():
+ * typeof((ptr) + 0)
+ *
+ * So use arch_raw_cpu_ptr()/PERCPU_PTR() directly here.
+ */

Please fix broken comment style while you fix that compile error.

#4 lore raw
Quoted 11 lines · Peter Zijlstra
Quoted 7 lines · Menglong Dong

+/* The "struct rq" is not available here, so we can't access the
+ * "runqueues" with this_cpu_ptr(), as the compilation will fail in
+ * this_cpu_ptr() -> raw_cpu_ptr() -> __verify_pcpu_ptr():
+ * typeof((ptr) + 0)
+ *
+ * So use arch_raw_cpu_ptr()/PERCPU_PTR() directly here.
+ */

Quoted 1 line

Please fix broken comment style while you fix that compile error.

+1

We switched to normal kernel comment style in bpf land quite some time ago, but old habits die hard and I have to keep reminding people to use normal comments in all new code.

#5 lore raw
In reply to Peter Zijlstra

On 2025/9/16 19:07 Peter Zijlstra <[email protected]> write:

Quoted 11 lines · Menglong Dong
Quoted 8 lines

+/* The "struct rq" is not available here, so we can't access the
+ * "runqueues" with this_cpu_ptr(), as the compilation will fail in
+ * this_cpu_ptr() -> raw_cpu_ptr() -> __verify_pcpu_ptr():
+ * typeof((ptr) + 0)
+ *
+ * So use arch_raw_cpu_ptr()/PERCPU_PTR() directly here.
+ */

Please fix broken comment style while you fix that compile error.

It's a little embarrassing. The compile error is caused by the commit 1b93c03fb319 ("rcu: add rcu_read_lock_dont_migrate()") in bpf-next tree, which uses migrate_enable/migrate_disable in include/linux/rcupdate.h but include the <linux/preempt.h>.

I can fix it by replace the linux/preempt.h with linux/sched.h, but should I fix it in this series? I mean, the commit 1b93c03fb319 doesn't exist in the tip for now :/

Thanks!
Menglong Dong

Quoted 2 lines
#6 lore raw
Quoted 14 lines · Menglong Dong

On 2025/9/16 19:07 Peter Zijlstra <[email protected]> write:

Quoted 11 lines · Menglong Dong
Quoted 8 lines

+/* The "struct rq" is not available here, so we can't access the
+ * "runqueues" with this_cpu_ptr(), as the compilation will fail in
+ * this_cpu_ptr() -> raw_cpu_ptr() -> __verify_pcpu_ptr():
+ * typeof((ptr) + 0)
+ *
+ * So use arch_raw_cpu_ptr()/PERCPU_PTR() directly here.
+ */

Please fix broken comment style while you fix that compile error.

Quoted 8 lines

It's a little embarrassing. The compile error is caused by the commit 1b93c03fb319 ("rcu: add rcu_read_lock_dont_migrate()") in bpf-next tree, which uses migrate_enable/migrate_disable in include/linux/rcupdate.h but include the <linux/preempt.h>.

I can fix it by replace the linux/preempt.h with linux/sched.h, but should I fix it in this series? I mean, the commit 1b93c03fb319 doesn't exist in the tip for now :/

If it's just a different include then go for it.
Make sure there are no nasty build issues during the merge window.

#7 lore raw

On 2025/9/17 09:29 Alexei Starovoitov <[email protected]> write:

Quoted 20 lines · Menglong Dong
Quoted 19 lines

On 2025/9/16 19:07 Peter Zijlstra <[email protected]> write:

Quoted 11 lines · Menglong Dong
Quoted 8 lines

+/* The "struct rq" is not available here, so we can't access the
+ * "runqueues" with this_cpu_ptr(), as the compilation will fail in
+ * this_cpu_ptr() -> raw_cpu_ptr() -> __verify_pcpu_ptr():
+ * typeof((ptr) + 0)
+ *
+ * So use arch_raw_cpu_ptr()/PERCPU_PTR() directly here.
+ */

Please fix broken comment style while you fix that compile error.

It's a little embarrassing. The compile error is caused by the commit 1b93c03fb319 ("rcu: add rcu_read_lock_dont_migrate()") in bpf-next tree, which uses migrate_enable/migrate_disable in include/linux/rcupdate.h but include the <linux/preempt.h>.

Quoted 6 lines
Quoted 3 lines

I can fix it by replace the linux/preempt.h with linux/sched.h, but should I fix it in this series? I mean, the commit 1b93c03fb319 doesn't exist in the tip for now :/

If it's just a different include then go for it.
Make sure there are no nasty build issues during the merge window.

OK!

Quoted 1 line
#8 lore raw

PATCH v4 3/3 sched: fix some typos in include/linux/preempt.h

In reply to Menglong Dong

There are some typos in the comments of migrate in include/linux/preempt.h:

elegible -> eligible
it's -> its
migirate_disable -> migrate_disable
abritrary -> arbitrary

Just fix them.

---
 include/linux/preempt.h | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)
include/linux/preempt.h +4 -4
index 92237c319035..102202185d7a 100644
@@ -372,7 +372,7 @@ static inline void preempt_notifier_init(struct preempt_notifier *notifier,
 /*
  * Migrate-Disable and why it is undesired.
  *
- * When a preempted task becomes elegible to run under the ideal model (IOW it
+ * When a preempted task becomes eligible to run under the ideal model (IOW it
  * becomes one of the M highest priority tasks), it might still have to wait
  * for the preemptee's migrate_disable() section to complete. Thereby suffering
  * a reduction in bandwidth in the exact duration of the migrate_disable()
@@ -387,7 +387,7 @@ static inline void preempt_notifier_init(struct preempt_notifier *notifier,
  * - a lower priority tasks; which under preempt_disable() could've instantly
  *   migrated away when another CPU becomes available, is now constrained
  *   by the ability to push the higher priority task away, which might itself be
- *   in a migrate_disable() section, reducing it's available bandwidth.
+ *   in a migrate_disable() section, reducing its available bandwidth.
  *
  * IOW it trades latency / moves the interference term, but it stays in the
  * system, and as long as it remains unbounded, the system is not fully
@@ -399,7 +399,7 @@ static inline void preempt_notifier_init(struct preempt_notifier *notifier,
  * PREEMPT_RT breaks a number of assumptions traditionally held. By forcing a
  * number of primitives into becoming preemptible, they would also allow
  * migration. This turns out to break a bunch of per-cpu usage. To this end,
- * all these primitives employ migirate_disable() to restore this implicit
+ * all these primitives employ migrate_disable() to restore this implicit
  * assumption.
  *
  * This is a 'temporary' work-around at best. The correct solution is getting
@@ -407,7 +407,7 @@ static inline void preempt_notifier_init(struct preempt_notifier *notifier,
  * per-cpu locking or short preempt-disable regions.
  *
  * The end goal must be to get rid of migrate_disable(), alternatively we need
- * a schedulability theory that does not depend on abritrary migration.
+ * a schedulability theory that does not depend on arbitrary migration.
  *
  *
  * Notes on the implementation.
Signature · 1 line
2.51.0
syzbot Tested-by [email protected] on the v4 series
#9 lore raw
In reply to Menglong Dong
syzbot ci has tested the following series

syzbot ci has tested the following series

[v4] sched: make migrate_enable/migrate_disable inline
https://lore.kernel.org/all/[email protected]
* [PATCH v4 1/3] arch: add the macro COMPILE_OFFSETS to all the asm-offsets.c
* [PATCH v4 2/3] sched: make migrate_enable/migrate_disable inline
* [PATCH v4 3/3] sched: fix some typos in include/linux/preempt.h

and found the following issue:
kernel build error

Full report is available here:
https://ci.syzbot.org/series/aac2d563-711a-4b4b-89c3-7365b1c03190

***

kernel build error

tree:      bpf-next
URL:       https://kernel.googlesource.com/pub/scm/linux/kernel/git/bpf/bpf-next.git
base:      d3abefe897408718799ae3bd06295b89b870a38e
arch:      amd64
compiler:  Debian clang version 20.1.7 (++20250616065708+6146a88f6049-1~exp1~20250616065826.132), Debian LLD 20.1.7
config:    https://ci.syzbot.org/builds/bb1748d9-0b48-4894-b30e-ea91020f1c78/config

./include/linux/rcupdate.h:968:3: error: call to undeclared function 'migrate_disable'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration] ./include/linux/rcupdate.h:976:3: error: call to undeclared function 'migrate_enable'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration] ./include/linux/sched.h:2402:20: error: static declaration of 'migrate_disable' follows non-static declaration
./include/linux/sched.h:2407:20: error: static declaration of 'migrate_enable' follows non-static declaration

***

If these findings have caused you to resend the series or submit a separate fix, please add the following tag to your commit message:

---
This report is generated by a bot. It may contain errors. syzbot ci engineers can be reached at [email protected].

#10 lore raw
In reply to Menglong Dong

On 2025/8/28 14:03 Menglong Dong <[email protected]> write:

Quoted 6 lines

In this series, we make migrate_enable/migrate_disable inline to obtain better performance in some case.

In the first patch, we add the macro "COMPILE_OFFSETS" to all the asm-offset.c to avoid circular dependency in the 2nd patch.

Quoted 7 lines

In the 2nd patch, we generate the offset of nr_pinned in "struct rq" with rq-offsets.c, as the "struct rq" is defined internally and we need to access the "nr_pinned" field in migrate_enable and migrate_disable. Then, we move the definition of migrate_enable/migrate_disable from kernel/sched/core.c to include/linux/sched.h.

In the 3rd patch, we fix some typos in include/linux/preempt.h.

Hi, everyone. Do we have any thoughts on this series?

Thanks!
Menglong Dong

Quoted 84 lines

One of the beneficiaries of this series is BPF trampoline. Without this series, the migrate_enable/migrate_disable is hot when we run the benchmark for FENTRY, FEXIT, MODIFY_RETURN, etc:

54.63% bpf_prog_2dcccf652aac1793_bench_trigger_fentry [k]

                 bpf_prog_2dcccf652aac1793_bench_trigger_fentry

10.43% [kernel] [k] migrate_enable
10.07% bpf_trampoline_6442517037 [k] bpf_trampoline_6442517037 8.06% [kernel] [k] __bpf_prog_exit_recur
4.11% libc.so.6 [.] syscall
2.15% [kernel] [k] entry_SYSCALL_64
1.48% [kernel] [k] memchr_inv
1.32% [kernel] [k] fput
1.16% [kernel] [k] _copy_to_user
0.73% [kernel] [k] bpf_prog_test_run_raw_tp

Before this patch, the performance of BPF FENTRY is:

  fentry         :  113.030 ± 0.149M/s
  fentry         :  112.501 ± 0.187M/s
  fentry         :  112.828 ± 0.267M/s
  fentry         :  115.287 ± 0.241M/s

After this patch, the performance of BPF FENTRY increases to:

  fentry         :  143.644 ± 0.670M/s
  fentry         :  149.764 ± 0.362M/s
  fentry         :  149.642 ± 0.156M/s
  fentry         :  145.263 ± 0.221M/s

Changes since V3:
* some modification on the 2nd patch, as Alexei advised:
- rename CREATE_MIGRATE_DISABLE to INSTANTIATE_EXPORTED_MIGRATE_DISABLE
- add document for INSTANTIATE_EXPORTED_MIGRATE_DISABLE

Changes since V2:
* some modification on the 2nd patch, as Peter advised:
- don't export runqueues, define migrate_enable and migrate_disable in kernel/sched/core.c and use them for kernel modules instead
- define the macro this_rq_pinned()
- add some comment for this_rq_raw()

Changes since V1:
* use PERCPU_PTR() for this_rq_raw() if !CONFIG_SMP in the 2nd patch

Menglong Dong (3):
arch: add the macro COMPILE_OFFSETS to all the asm-offsets.c
sched: make migrate_enable/migrate_disable inline
sched: fix some typos in include/linux/preempt.h

 Kbuild                               |  13 ++-
 arch/alpha/kernel/asm-offsets.c      |   1 +
 arch/arc/kernel/asm-offsets.c        |   1 +
 arch/arm/kernel/asm-offsets.c        |   2 +
 arch/arm64/kernel/asm-offsets.c      |   1 +
 arch/csky/kernel/asm-offsets.c       |   1 +
 arch/hexagon/kernel/asm-offsets.c    |   1 +
 arch/loongarch/kernel/asm-offsets.c  |   2 +
 arch/m68k/kernel/asm-offsets.c       |   1 +
 arch/microblaze/kernel/asm-offsets.c |   1 +
 arch/mips/kernel/asm-offsets.c       |   2 +
 arch/nios2/kernel/asm-offsets.c      |   1 +
 arch/openrisc/kernel/asm-offsets.c   |   1 +
 arch/parisc/kernel/asm-offsets.c     |   1 +
 arch/powerpc/kernel/asm-offsets.c    |   1 +
 arch/riscv/kernel/asm-offsets.c      |   1 +
 arch/s390/kernel/asm-offsets.c       |   1 +
 arch/sh/kernel/asm-offsets.c         |   1 +
 arch/sparc/kernel/asm-offsets.c      |   1 +
 arch/um/kernel/asm-offsets.c         |   2 +
 arch/xtensa/kernel/asm-offsets.c     |   1 +
 include/linux/preempt.h              |  11 +--
 include/linux/sched.h                | 113 +++++++++++++++++++++++++++
 kernel/bpf/verifier.c                |   1 +
 kernel/sched/core.c                  |  63 ++++-----------
 kernel/sched/rq-offsets.c            |  12 +++
 26 files changed, 180 insertions(+), 57 deletions(-)
 create mode 100644 kernel/sched/rq-offsets.c
Signature · 1 line
2.51.0
#11 lore raw
Quoted 14 lines · Menglong Dong

On 2025/8/28 14:03 Menglong Dong <[email protected]> write:

Quoted 12 lines

In this series, we make migrate_enable/migrate_disable inline to obtain better performance in some case.

In the first patch, we add the macro "COMPILE_OFFSETS" to all the asm-offset.c to avoid circular dependency in the 2nd patch.

In the 2nd patch, we generate the offset of nr_pinned in "struct rq" with rq-offsets.c, as the "struct rq" is defined internally and we need to access the "nr_pinned" field in migrate_enable and migrate_disable. Then, we move the definition of migrate_enable/migrate_disable from kernel/sched/core.c to include/linux/sched.h.

Quoted 3 lines
Quoted 1 line

In the 3rd patch, we fix some typos in include/linux/preempt.h.

Hi, everyone. Do we have any thoughts on this series?

Pls fix build errors and repost.