sched: Split out kernel/sched/fair_balance.c, numa_balancing.c and syscalls.c, plus other updates
cover sched: Split out kernel/sched/fair_balance.c, numa_balancing.c and syscalls.c, plus other updates
kernel/sched/core.c and fair.c have become a hodgepodge of scheduler functionality that obscures core scheduling and fair-balancing class scheduling functionality quite a bit.
Improve it all a bit by splitting syscall.c out of core.c, and fair_balance.c and numa_balancing.c out of fair.c:
# of Lines of code
BEFORE => AFTER
---------------
kernel/sched/sched.h 3493 => 3891
kernel/sched/core.c 12067 => 10308
kernel/sched/syscalls.c - => 1691kernel/sched/fair.c 13295 => 5707 kernel/sched/fair_balance.c - => 5103 kernel/sched/numa_balancing.c - => 2277
total 49650 => 49765
This is only a first-approximation split.
Thanks,
Ingo
=================>
Ingo Molnar (5):
sched: Split out kernel/sched/syscalls.c from kernel/sched/core.c
sched: Split out kernel/sched/fair_balance.c from kernel/sched/fair.c
sched: Split out kernel/sched/numa_balancing.c from kernel/sched/fair.c sched/fair: Remove NEXT_BUDDY
sched/fair: Rename set_next_buddy() to set_next_pick()
kernel/sched/Makefile | 3 + kernel/sched/core.c | 1951 +-------- kernel/sched/fair.c | 13184 +++++++++++++----------------------------------------------- kernel/sched/fair_balance.c | 5103 +++++++++++++++++++++++ kernel/sched/features.h | 7 - kernel/sched/numa_balancing.c | 2277 +++++++++++ kernel/sched/sched.h | 400 +- kernel/sched/syscalls.c | 1691 ++++++++ 8 files changed, 12367 insertions(+), 12249 deletions(-) create mode 100644 kernel/sched/fair_balance.c create mode 100644 kernel/sched/numa_balancing.c create mode 100644 kernel/sched/syscalls.c
Signature · 1 line
2.40.1
PATCH 1/5 sched: Split out kernel/sched/syscalls.c from kernel/sched/core.c
core.c has become rather large, move most scheduler syscall related functionality into a separate file, syscalls.c.
Move the alloc_user_cpus_ptr(), __rt_effective_prio(), rt_effective_prio(), uclamp_none(), uclamp_se_set() and uclamp_bucket_id() inlines to kernel/sched/sched.h.
Internally export the __sched_setscheduler(), __sched_setaffinity(), __setscheduler_prio(), set_load_weight(), enqueue_task(), dequeue_task(), check_class_changed(), splice_balance_callbacks() and balance_callbacks() methods to better facilitate this.
--- kernel/sched/Makefile | 1 + kernel/sched/core.c | 1950 ++++--------------------------------------------------------------- kernel/sched/sched.h | 106 +++- kernel/sched/syscalls.c | 1691 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 4 files changed, 1892 insertions(+), 1856 deletions(-)
kernel/sched/Makefile +1 -0
@@ -29,6 +29,7 @@ endif # build parallelizes well and finishes roughly at once: # obj-y += core.o +obj-y += syscalls.o obj-y += fair.o obj-y += build_policy.o obj-y += build_utility.o
kernel/sched/core.c +95 -1855
2,115 lines not on this page · raw message
kernel/sched/sched.h +105 -1
136 lines not on this page · raw message
kernel/sched/syscalls.c new file +1691 -0
742 lines not on this page · raw message
The rest of this diff is not shown. View the raw message.
Signature · 1 line
2.40.1
Part of the diff is not shown here. View the raw message.
kernel test robot
build botRe: [PATCH 1/5] sched: Split out kernel/sched/syscalls.c from kernel/sched/core.c
kernel test robot noticed the following build warnings:
Hi Ingo,
kernel test robot noticed the following build warnings:
[auto build test WARNING on tip/sched/core]
[also build test WARNING on linux/master linus/master peterz-queue/sched/core v6.9-rc2 next-20240405] [If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]
url: https://github.com/intel-lab-lkp/linux/commits/Ingo-Molnar/sched-Split-out-kernel-sched-syscalls-c-from-kernel-sched-core-c/20240407-164646 base: tip/sched/core patch link: https://lore.kernel.org/r/20240407084319.1462211-2-mingo%40kernel.org patch subject: [PATCH 1/5] sched: Split out kernel/sched/syscalls.c from kernel/sched/core.c config: i386-randconfig-061-20240408 (https://download.01.org/0day-ci/archive/20240408/[email protected]/config) compiler: gcc-13 (Ubuntu 13.2.0-4ubuntu3) 13.2.0 reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20240408/[email protected]/reproduce)
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/
sparse warnings: (new ones prefixed by >>)
>> kernel/sched/syscalls.c:197:22: sparse: sparse: incompatible types in comparison expression (different address spaces):
kernel/sched/syscalls.c:197:22: sparse: struct task_struct [noderef] __rcu *
kernel/sched/syscalls.c:197:22: sparse: struct task_struct *
kernel/sched/syscalls.c: note: in included file:
kernel/sched/sched.h:2172:25: sparse: sparse: incompatible types in comparison expression (different address spaces):
kernel/sched/sched.h:2172:25: sparse: struct task_struct [noderef] __rcu *
kernel/sched/sched.h:2172:25: sparse: struct task_struct *
kernel/sched/sched.h:2344:9: sparse: sparse: incompatible types in comparison expression (different address spaces):
kernel/sched/sched.h:2344:9: sparse: struct task_struct [noderef] __rcu *
kernel/sched/sched.h:2344:9: sparse: struct task_struct *
kernel/sched/sched.h:2172:25: sparse: sparse: incompatible types in comparison expression (different address spaces):
kernel/sched/sched.h:2172:25: sparse: struct task_struct [noderef] __rcu *
kernel/sched/sched.h:2172:25: sparse: struct task_struct *
kernel/sched/sched.h:2344:9: sparse: sparse: incompatible types in comparison expression (different address spaces):
kernel/sched/sched.h:2344:9: sparse: struct task_struct [noderef] __rcu *
kernel/sched/sched.h:2344:9: sparse: struct task_struct *
kernel/sched/syscalls.c:1228:5: sparse: sparse: context imbalance in 'dl_task_check_affinity' - wrong count at exit kernel/sched/syscalls.c:1331:9: sparse: sparse: context imbalance in 'sched_setaffinity' - different lock contexts for basic block kernel/sched/syscalls.c:1393:6: sparse: sparse: context imbalance in 'sched_getaffinity' - wrong count at exit
vim +197 kernel/sched/syscalls.c
186
187 /**
188 * idle_cpu - is a given CPU idle currently?
189 * @cpu: the processor in question.
190 *
191 * Return: 1 if the CPU is currently idle. 0 otherwise.
192 */
193 int idle_cpu(int cpu)
194 {
195 struct rq *rq = cpu_rq(cpu);
196
> 197 if (rq->curr != rq->idle)
198 return 0;
199
200 if (rq->nr_running)
201 return 0;
202
Signature · 2 lines
0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests/wiki
[tip: sched/core] sched/syscalls: Split out kernel/sched/syscalls.c from kernel/sched/core.c
In reply to Ingo MolnarThe following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: 04746ed80bcf3130951ed4d5c1bc5b0bcabdde22 Gitweb: https://git.kernel.org/tip/04746ed80bcf3130951ed4d5c1bc5b0bcabdde22 Author: Ingo Molnar <[email protected]> AuthorDate: Sun, 07 Apr 2024 10:43:15 +02:00 Committer: Ingo Molnar <[email protected]> CommitterDate: Mon, 27 May 2024 13:56:10 +02:00
sched/syscalls: Split out kernel/sched/syscalls.c from kernel/sched/core.c
core.c has become rather large, move most scheduler syscall related functionality into a separate file, syscalls.c.
This is about ~15% of core.c's raw linecount.
Move the alloc_user_cpus_ptr(), __rt_effective_prio(), rt_effective_prio(), uclamp_none(), uclamp_se_set() and uclamp_bucket_id() inlines to kernel/sched/sched.h.
Internally export the __sched_setscheduler(), __sched_setaffinity(), __setscheduler_prio(), set_load_weight(), enqueue_task(), dequeue_task(), check_class_changed(), splice_balance_callbacks() and balance_callbacks() methods to better facilitate this.
Move the new file's build to sched_policy.c, because it fits there semantically, but also because it's the smallest of the 4 build units under an allmodconfig build:
-rw-rw-r-- 1 mingo mingo 7.3M May 27 12:35 kernel/sched/core.i -rw-rw-r-- 1 mingo mingo 6.4M May 27 12:36 kernel/sched/build_utility.i -rw-rw-r-- 1 mingo mingo 6.3M May 27 12:36 kernel/sched/fair.i -rw-rw-r-- 1 mingo mingo 5.8M May 27 12:36 kernel/sched/build_policy.i
This better balances build time for scheduler subsystem rebuilds.
I build-tested this new file as a standalone syscalls.o file for a bit, to make sure all the encapsulations & abstractions are robust.
Also update/add my copyright notices to these files.
Build time measurements:
# -Before/+After:
kepler:~/tip> perf stat -e 'cycles,instructions,duration_time' --sync --repeat 5 --pre 'rm -f kernel/sched/*.o' m kernel/sched/built-in.a >/dev/null
Performance counter stats for 'm kernel/sched/built-in.a' (5 runs):
- 71,938,508,607 cycles ( +- 0.17% ) + 71,992,916,493 cycles ( +- 0.22% ) - 106,214,780,964 instructions # 1.48 insn per cycle ( +- 0.01% ) + 105,450,231,154 instructions # 1.46 insn per cycle ( +- 0.01% ) - 5,878,232,620 ns duration_time ( +- 0.38% ) + 5,290,085,069 ns duration_time ( +- 0.21% )
- 5.8782 +- 0.0221 seconds time elapsed ( +- 0.38% ) + 5.2901 +- 0.0111 seconds time elapsed ( +- 0.21% )
Build time improvement of -11.1% (duration_time) is expected: the parallel build time of the scheduler subsystem is determined by the largest, slowest to build object file, which is kernel/sched/core.o. By moving ~15% of its complexity into another build unit, we reduced build time by -11%.
Measured cycles spent on building is within its ~0.2% stddev noise envelope.
The -0.7% reduction in instructions spent on building the scheduler is statistically reliable and somewhat surprising - I can only speculate:
maybe compilers aren't that efficient at building & optimizing 10+ KLOC files (core.c), and it's an overall win to balance the linecount a bit.
Anyway, this might be a data point that suggests that reducing the linecount of our largest files will improve not just code readability and maintainability, but might also improve build times a bit.
Code generation got a bit worse, by 0.5kb text on an x86 defconfig build:
# -Before/+After:
kepler:~/tip> size vmlinux
text data bss dec hex filename
-26475475 10439178 1740804 38655457 24dd5e1 vmlinux
+26476003 10439178 1740804 38655985 24dd7f1 vmlinux
kepler:~/tip> size kernel/sched/built-in.a
text data bss dec hex filename
- 76056 30025 489 106570 1a04a kernel/sched/core.o (ex kernel/sched/built-in.a)
+ 63452 29453 489 93394 16cd2 kernel/sched/core.o (ex kernel/sched/built-in.a)
44299 2181 104 46584 b5f8 kernel/sched/fair.o (ex kernel/sched/built-in.a)
- 42764 3424 120 46308 b4e4 kernel/sched/build_policy.o (ex kernel/sched/built-in.a)
+ 55651 4044 120 59815 e9a7 kernel/sched/build_policy.o (ex kernel/sched/built-in.a)
44866 12655 2192 59713 e941 kernel/sched/build_utility.o (ex kernel/sched/built-in.a)
44866 12655 2192 59713 e941 kernel/sched/build_utility.o (ex kernel/sched/built-in.a)This is primarily due to the extra functions exported, and the size gets exaggerated somewhat by __pfx CFI function padding:
ffffffff810cc710 <__pfx_enqueue_task>: ffffffff810cc710: 90 nop ffffffff810cc711: 90 nop ffffffff810cc712: 90 nop ffffffff810cc713: 90 nop ffffffff810cc714: 90 nop ffffffff810cc715: 90 nop ffffffff810cc716: 90 nop ffffffff810cc717: 90 nop ffffffff810cc718: 90 nop ffffffff810cc719: 90 nop ffffffff810cc71a: 90 nop ffffffff810cc71b: 90 nop ffffffff810cc71c: 90 nop ffffffff810cc71d: 90 nop ffffffff810cc71e: 90 nop ffffffff810cc71f: 90 nop
AFAICS the cost is primarily not to core.o and fair.o though (which contain most performance sensitive scheduler functions), only to syscalls.o that get called with much lower frequency - so I think this is an acceptable trade-off for better code separation.
--- kernel/sched/build_policy.c | 1 +- kernel/sched/core.c | 1953 +--------------------------------- kernel/sched/sched.h | 106 +- kernel/sched/syscalls.c | 1699 ++++++++++++++++++++++++++++++- 4 files changed, 1902 insertions(+), 1857 deletions(-) create mode 100644 kernel/sched/syscalls.c
kernel/sched/build_policy.c +1 -0
@@ -52,3 +52,4 @@#include "cputime.c" #include "deadline.c" +#include "syscalls.c"
kernel/sched/core.c +97 -1856
2,126 lines not on this page · raw message
kernel/sched/sched.h +105 -1
136 lines not on this page · raw message
kernel/sched/syscalls.c new file +1699 -0
734 lines not on this page · raw message
The rest of this diff is not shown. View the raw message.
Part of the diff is not shown here. View the raw message.
PATCH 2/5 sched: Split out kernel/sched/fair_balance.c from kernel/sched/fair.c
In reply to Ingo MolnarMove the SMP load-balancing code into the new fair_balance.c file, because it's mostly self-contained code that comprised about 50% of the lines of code in fair.c.
Expose the sched_balance_softirq(), sched_balance_find_dst_group(), cpu_load_without(), cpu_runnable_without(), cpu_util_without(), sched_balance_newidle(), task_h_load(), throttled_lb_pair(), task_util(), task_util_est() and a number of other methods internally to better facilitate this code separation.
--- kernel/sched/Makefile | 1 + kernel/sched/core.c | 1 + kernel/sched/fair.c | 10366 +++++++++++++++----------------------------------------------- kernel/sched/fair_balance.c | 5103 +++++++++++++++++++++++++++++++ kernel/sched/sched.h | 256 ++ 5 files changed, 7909 insertions(+), 7818 deletions(-)
kernel/sched/Makefile +1 -0
@@ -31,5 +31,6 @@ endif obj-y += core.o obj-y += syscalls.o obj-y += fair.o +obj-y += fair_balance.o obj-y += build_policy.o obj-y += build_utility.o
kernel/sched/core.c +1 -0
@@ -8337,6 +8337,7 @@ void __init sched_init(void)balance_push_set (smp_processor_id (), false);#endif init_sched_fair_class (); +init_sched_fair_class_balance ();psi_init ();
kernel/sched/fair.c +2548 -7818
2,987 lines not on this page · raw message
kernel/sched/fair_balance.c new file +5103 -0
5,103 lines not on this page · raw message
kernel/sched/sched.h +256 -0
256 lines not on this page · raw message
The rest of this diff is not shown. View the raw message.
Signature · 1 line
2.40.1
Part of the diff is not shown here. View the raw message.
kernel test robot
build botRe: [PATCH 2/5] sched: Split out kernel/sched/fair_balance.c from kernel/sched/fair.c
kernel test robot noticed the following build errors:
Hi Ingo,
kernel test robot noticed the following build errors:
[auto build test ERROR on tip/sched/core]
[cannot apply to linux/master linus/master peterz-queue/sched/core v6.9-rc2] [If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]
url: https://github.com/intel-lab-lkp/linux/commits/Ingo-Molnar/sched-Split-out-kernel-sched-syscalls-c-from-kernel-sched-core-c/20240407-164646 base: tip/sched/core patch link: https://lore.kernel.org/r/20240407084319.1462211-3-mingo%40kernel.org patch subject: [PATCH 2/5] sched: Split out kernel/sched/fair_balance.c from kernel/sched/fair.c config: arc-defconfig (https://download.01.org/0day-ci/archive/20240407/[email protected]/config) compiler: arc-elf-gcc (GCC) 13.2.0 reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20240407/[email protected]/reproduce)
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/
All error/warnings (new ones prefixed by >>):
In file included from kernel/sched/core.c:87:
>> kernel/sched/sched.h:3611:22: warning: 'task_h_load' defined but not used [-Wunused-function]
3611 | static unsigned long task_h_load(struct task_struct *p)
| ^~~~~~~~~~~
--
arc-elf-ld: kernel/sched/fair.o: in function `switched_from_fair':
>> fair.c:(.text+0x1f0): undefined reference to `update_tg_load_avg' >> arc-elf-ld: fair.c:(.text+0x1f0): undefined reference to `update_tg_load_avg' arc-elf-ld: kernel/sched/fair.o: in function `rq_offline_fair': >> fair.c:(.text+0x766): undefined reference to `clear_tg_offline_cfs_rqs' >> arc-elf-ld: fair.c:(.text+0x766): undefined reference to `clear_tg_offline_cfs_rqs' arc-elf-ld: fair.c:(.text+0x774): undefined reference to `clear_tg_offline_cfs_rqs' arc-elf-ld: fair.c:(.text+0x774): undefined reference to `clear_tg_offline_cfs_rqs' arc-elf-ld: kernel/sched/fair.o: in function `switched_to_fair': fair.c:(.text+0x158a): undefined reference to `update_tg_load_avg' arc-elf-ld: fair.c:(.text+0x158a): undefined reference to `update_tg_load_avg' arc-elf-ld: kernel/sched/fair_balance.o: in function `update_load_avg': >> fair_balance.c:(.text+0x12c8): undefined reference to `update_tg_load_avg' >> arc-elf-ld: fair_balance.c:(.text+0x12c8): undefined reference to `update_tg_load_avg'
Signature · 2 lines
0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests/wiki
kernel test robot
build botRe: [PATCH 2/5] sched: Split out kernel/sched/fair_balance.c from kernel/sched/fair.c
In reply to Ingo Molnarkernel test robot noticed the following build errors:
Hi Ingo,
kernel test robot noticed the following build errors:
[auto build test ERROR on tip/sched/core]
[also build test ERROR on next-20240405]
[cannot apply to linux/master linus/master peterz-queue/sched/core v6.9-rc2] [If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]
url: https://github.com/intel-lab-lkp/linux/commits/Ingo-Molnar/sched-Split-out-kernel-sched-syscalls-c-from-kernel-sched-core-c/20240407-164646 base: tip/sched/core patch link: https://lore.kernel.org/r/20240407084319.1462211-3-mingo%40kernel.org patch subject: [PATCH 2/5] sched: Split out kernel/sched/fair_balance.c from kernel/sched/fair.c config: arm64-allnoconfig (https://download.01.org/0day-ci/archive/20240407/[email protected]/config) compiler: aarch64-linux-gcc (GCC) 13.2.0 reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20240407/[email protected]/reproduce)
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/
All errors (new ones prefixed by >>):
aarch64-linux-ld: Unexpected GOT/PLT entries detected!
aarch64-linux-ld: Unexpected run-time procedure linkages detected!
aarch64-linux-ld: kernel/sched/fair.o: in function `switched_from_fair':
fair.c:(.text+0x16c): undefined reference to `update_tg_load_avg'
aarch64-linux-ld: kernel/sched/fair.o: in function `rq_offline_fair':
fair.c:(.text+0x230): undefined reference to `clear_tg_offline_cfs_rqs'
>> aarch64-linux-ld: fair.c:(.text+0x24c): undefined reference to `clear_tg_offline_cfs_rqs' aarch64-linux-ld: kernel/sched/fair.o: in function `switched_to_fair': fair.c:(.text+0x87c): undefined reference to `update_tg_load_avg' aarch64-linux-ld: kernel/sched/fair_balance.o: in function `update_load_avg': fair_balance.c:(.text+0x1520): undefined reference to `update_tg_load_avg' >> aarch64-linux-ld: fair_balance.c:(.text+0x15ec): undefined reference to `update_tg_load_avg'
aarch64-linux-ld: fair_balance.c:(.text+0x1608): undefined reference to `update_tg_load_avg'
Signature · 2 lines
0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests/wiki
kernel test robot
build botRe: [PATCH 2/5] sched: Split out kernel/sched/fair_balance.c from kernel/sched/fair.c
In reply to Ingo Molnarkernel test robot noticed the following build warnings:
Hi Ingo,
kernel test robot noticed the following build warnings:
[auto build test WARNING on tip/sched/core]
[also build test WARNING on next-20240405]
[cannot apply to linux/master linus/master peterz-queue/sched/core v6.9-rc2] [If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]
url: https://github.com/intel-lab-lkp/linux/commits/Ingo-Molnar/sched-Split-out-kernel-sched-syscalls-c-from-kernel-sched-core-c/20240407-164646 base: tip/sched/core patch link: https://lore.kernel.org/r/20240407084319.1462211-3-mingo%40kernel.org patch subject: [PATCH 2/5] sched: Split out kernel/sched/fair_balance.c from kernel/sched/fair.c config: i386-randconfig-061-20240408 (https://download.01.org/0day-ci/archive/20240408/[email protected]/config) compiler: gcc-13 (Ubuntu 13.2.0-4ubuntu3) 13.2.0 reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20240408/[email protected]/reproduce)
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/
sparse warnings: (new ones prefixed by >>)
>> kernel/sched/fair_balance.c:2171:40: sparse: sparse: incorrect type in initializer (different address spaces) @@ expected struct sched_domain *child @@ got struct sched_domain [noderef] __rcu *child @@ kernel/sched/fair_balance.c:2171:40: sparse: expected struct sched_domain *child kernel/sched/fair_balance.c:2171:40: sparse: got struct sched_domain [noderef] __rcu *child >> kernel/sched/fair_balance.c:2793:22: sparse: sparse: incompatible types in comparison expression (different address spaces): kernel/sched/fair_balance.c:2793:22: sparse: struct task_struct [noderef] __rcu * kernel/sched/fair_balance.c:2793:22: sparse: struct task_struct * >> kernel/sched/fair_balance.c:4218:9: sparse: sparse: incorrect type in assignment (different address spaces) @@ expected struct sched_domain *[assigned] sd @@ got struct sched_domain [noderef] __rcu *parent @@ kernel/sched/fair_balance.c:4218:9: sparse: expected struct sched_domain *[assigned] sd kernel/sched/fair_balance.c:4218:9: sparse: got struct sched_domain [noderef] __rcu *parent >> kernel/sched/fair_balance.c:3866:44: sparse: sparse: incorrect type in initializer (different address spaces) @@ expected struct sched_domain *sd_parent @@ got struct sched_domain [noderef] __rcu *parent @@ kernel/sched/fair_balance.c:3866:44: sparse: expected struct sched_domain *sd_parent kernel/sched/fair_balance.c:3866:44: sparse: got struct sched_domain [noderef] __rcu *parent kernel/sched/fair_balance.c:4327:9: sparse: sparse: incorrect type in assignment (different address spaces) @@ expected struct sched_domain *[assigned] sd @@ got struct sched_domain [noderef] __rcu *parent @@ kernel/sched/fair_balance.c:4327:9: sparse: expected struct sched_domain *[assigned] sd kernel/sched/fair_balance.c:4327:9: sparse: got struct sched_domain [noderef] __rcu *parent kernel/sched/fair_balance.c:4972:9: sparse: sparse: incorrect type in assignment (different address spaces) @@ expected struct sched_domain *[assigned] sd @@ got struct sched_domain [noderef] __rcu *parent @@ kernel/sched/fair_balance.c:4972:9: sparse: expected struct sched_domain *[assigned] sd kernel/sched/fair_balance.c:4972:9: sparse: got struct sched_domain [noderef] __rcu *parent >> kernel/sched/fair_balance.c:1921:24: sparse: sparse: dereference of noderef expression
kernel/sched/fair_balance.c:4050:64: sparse: sparse: dereference of noderef expression kernel/sched/fair_balance.c:4050:64: sparse: sparse: dereference of noderef expression
vim +2171 kernel/sched/fair_balance.c
2168
2169 void update_group_capacity(struct sched_domain *sd, int cpu)
2170 {
> 2171 struct sched_domain *child = sd->child;
2172 struct sched_group *group, *sdg = sd->groups;
2173 unsigned long capacity, min_capacity, max_capacity;
2174 unsigned long interval;
2175
2176 interval = msecs_to_jiffies(sd->balance_interval);
2177 interval = clamp(interval, 1UL, max_load_balance_interval);
2178 sdg->sgc->next_update = jiffies + interval;
2179
2180 if (!child) {
2181 update_cpu_capacity(sd, cpu);
2182 return;
2183 }
2184
2185 capacity = 0;
2186 min_capacity = ULONG_MAX;
2187 max_capacity = 0;
2188
2189 if (child->flags & SD_OVERLAP) {
2190 /*
2191 * SD_OVERLAP domains cannot assume that child groups
2192 * span the current group.
2193 */
2194
2195 for_each_cpu(cpu, sched_group_span(sdg)) {
2196 unsigned long cpu_cap = capacity_of(cpu);
2197
2198 capacity += cpu_cap;
2199 min_capacity = min(cpu_cap, min_capacity);
2200 max_capacity = max(cpu_cap, max_capacity);
2201 }
2202 } else {
2203 /*
2204 * !SD_OVERLAP domains can assume that child groups
2205 * span the current group.
2206 */
2207
2208 group = child->groups;
2209 do {
2210 struct sched_group_capacity *sgc = group->sgc;
2211
2212 capacity += sgc->capacity;
2213 min_capacity = min(sgc->min_capacity, min_capacity);
2214 max_capacity = max(sgc->max_capacity, max_capacity);
2215 group = group->next;
2216 } while (group != child->groups);
2217 }
2218
2219 sdg->sgc->capacity = capacity;
2220 sdg->sgc->min_capacity = min_capacity;
2221 sdg->sgc->max_capacity = max_capacity;
2222 }
2223
Signature · 2 lines
0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests/wiki
PATCH 3/5 sched: Split out kernel/sched/numa_balancing.c from kernel/sched/fair.c
In reply to Ingo MolnarMuch of the NUMA balancing code already lives in a single #ifdef block - move it over into its own file: kernel/sched/numa_balancing.c.
Expose a handful of methods internally to facilitate this.
This further shrinks the rather large kernel/sched/fair.c file.
--- kernel/sched/Makefile | 1 + kernel/sched/fair.c | 2307 +------------------------------------------------------------ kernel/sched/numa_balancing.c | 2277 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ kernel/sched/sched.h | 40 +- 4 files changed, 2318 insertions(+), 2307 deletions(-)
kernel/sched/Makefile +1 -0
@@ -32,5 +32,6 @@ obj-y += core.o obj-y += syscalls.o obj-y += fair.o obj-y += fair_balance.o +obj-y += numa_balancing.o obj-y += build_policy.o obj-y += build_utility.o
kernel/sched/fair.c +1 -2306
2,337 lines not on this page · raw message
kernel/sched/numa_balancing.c new file +2277 -0
657 lines not on this page · raw message
The rest of this diff is not shown. View the raw message.
Signature · 1 line
2.40.1
Part of the diff is not shown here. View the raw message.
kernel test robot
build botRe: [PATCH 3/5] sched: Split out kernel/sched/numa_balancing.c from kernel/sched/fair.c
kernel test robot noticed the following build warnings:
Hi Ingo,
kernel test robot noticed the following build warnings:
[auto build test WARNING on tip/sched/core]
[also build test WARNING on next-20240405]
[cannot apply to linux/master linus/master peterz-queue/sched/core v6.9-rc2] [If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]
url: https://github.com/intel-lab-lkp/linux/commits/Ingo-Molnar/sched-Split-out-kernel-sched-syscalls-c-from-kernel-sched-core-c/20240407-164646 base: tip/sched/core patch link: https://lore.kernel.org/r/20240407084319.1462211-4-mingo%40kernel.org patch subject: [PATCH 3/5] sched: Split out kernel/sched/numa_balancing.c from kernel/sched/fair.c config: x86_64-randconfig-121-20240408 (https://download.01.org/0day-ci/archive/20240408/[email protected]/config) compiler: clang version 17.0.6 (https://github.com/llvm/llvm-project 6009708b4367171ccdbf4b5905cb6a803753fe18) reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20240408/[email protected]/reproduce)
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/
sparse warnings: (new ones prefixed by >>)
>> kernel/sched/numa_balancing.c:1654:13: sparse: sparse: incorrect type in assignment (different address spaces) @@ expected struct task_struct *tsk @@ got struct task_struct [noderef] __rcu * @@
kernel/sched/numa_balancing.c:1654:13: sparse: expected struct task_struct *tsk
kernel/sched/numa_balancing.c:1654:13: sparse: got struct task_struct [noderef] __rcu *
kernel/sched/numa_balancing.c: note: in included file (through include/linux/mmzone.h, include/linux/memory-tiers.h):
include/linux/page-flags.h:242:46: sparse: sparse: self-comparison always evaluates to false include/linux/page-flags.h:242:46: sparse: sparse: self-comparison always evaluates to false kernel/sched/numa_balancing.c:1600:9: sparse: sparse: context imbalance in 'task_numa_placement' - different lock contexts for basic block
vim +1654 kernel/sched/numa_balancing.c
1619
1620 static void task_numa_group(struct task_struct *p, int cpupid, int flags,
1621 int *priv)
1622 {
1623 struct numa_group *grp, *my_grp;
1624 struct task_struct *tsk;
1625 bool join = false;
1626 int cpu = cpupid_to_cpu(cpupid);
1627 int i;
1628
1629 if (unlikely(!deref_curr_numa_group(p))) {
1630 unsigned int size = sizeof(struct numa_group) +
1631 NR_NUMA_HINT_FAULT_STATS *
1632 nr_node_ids * sizeof(unsigned long);
1633
1634 grp = kzalloc(size, GFP_KERNEL | __GFP_NOWARN);
1635 if (!grp)
1636 return;
1637
1638 refcount_set(&grp->refcount, 1);
1639 grp->active_nodes = 1;
1640 grp->max_faults_cpu = 0;
1641 spin_lock_init(&grp->lock);
1642 grp->gid = p->pid;
1643
1644 for (i = 0; i < NR_NUMA_HINT_FAULT_STATS * nr_node_ids; i++)
1645 grp->faults[i] = p->numa_faults[i];
1646
1647 grp->total_faults = p->total_numa_faults;
1648
1649 grp->nr_tasks++;
1650 rcu_assign_pointer(p->numa_group, grp);
1651 }
1652
1653 rcu_read_lock();
> 1654 tsk = READ_ONCE(cpu_rq(cpu)->curr);
1655
1656 if (!cpupid_match_pid(tsk, cpupid))
1657 goto no_join;
1658
1659 grp = rcu_dereference(tsk->numa_group);
1660 if (!grp)
1661 goto no_join;
1662
1663 my_grp = deref_curr_numa_group(p);
1664 if (grp == my_grp)
1665 goto no_join;
1666
1667 /*
1668 * Only join the other group if its bigger; if we're the bigger group,
1669 * the other task will join us.
1670 */
1671 if (my_grp->nr_tasks > grp->nr_tasks)
1672 goto no_join;
1673
1674 /*
1675 * Tie-break on the grp address.
1676 */
1677 if (my_grp->nr_tasks == grp->nr_tasks && my_grp > grp)
1678 goto no_join;
1679
1680 /* Always join threads in the same process. */
1681 if (tsk->mm == current->mm)
1682 join = true;
1683
1684 /* Simple filter to avoid false positives due to PID collisions */
1685 if (flags & TNF_SHARED)
1686 join = true;
1687
1688 /* Update priv based on whether false sharing was detected */
1689 *priv = !join;
1690
1691 if (join && !get_numa_group(grp))
1692 goto no_join;
1693
1694 rcu_read_unlock();
1695
1696 if (!join)
1697 return;
1698
1699 WARN_ON_ONCE(irqs_disabled());
1700 double_lock_irq(&my_grp->lock, &grp->lock);
1701
1702 for (i = 0; i < NR_NUMA_HINT_FAULT_STATS * nr_node_ids; i++) {
1703 my_grp->faults[i] -= p->numa_faults[i];
1704 grp->faults[i] += p->numa_faults[i];
1705 }
1706 my_grp->total_faults -= p->total_numa_faults;
1707 grp->total_faults += p->total_numa_faults;
1708
1709 my_grp->nr_tasks--;
1710 grp->nr_tasks++;
1711
1712 spin_unlock(&my_grp->lock);
1713 spin_unlock_irq(&grp->lock);
1714
1715 rcu_assign_pointer(p->numa_group, grp);
1716
1717 put_numa_group(my_grp);
1718 return;
1719
1720 no_join:
1721 rcu_read_unlock();
1722 return;
1723 }
1724
Signature · 2 lines
0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests/wiki
PATCH 4/5 sched/fair: Remove NEXT_BUDDY
In reply to Ingo MolnarIt has been turned off in 2009 and hasn't been enabled since, 15 years ought to be enough to remove it.
--- kernel/sched/fair.c | 11 ----------- kernel/sched/features.h | 7 ------- 2 files changed, 18 deletions(-)
kernel/sched/fair.c +0 -11
@@ -1901,13 +1901,6 @@ set_next_entity(struct cfs_rq *cfs_rq, struct sched_entity *se)static struct sched_entity *pick_next_entity (struct cfs_rq *cfs_rq) { -/* -* Enabling NEXT_BUDDY will affect latency but not fairness. -*/ -if (sched_feat (NEXT_BUDDY ) && - cfs_rq->next &&entity_eligible (cfs_rq, cfs_rq->next)) -return cfs_rq->next; -return pick_eevdf (cfs_rq); }
@@ -4671,10 +4664,6 @@ static void check_preempt_wakeup_fair(struct rq *rq, struct task_struct *p, intif (unlikely (throttled_hierarchy (cfs_rq_of (pse))))return ; -if (sched_feat (NEXT_BUDDY ) && !(wake_flags &WF_FORK )) { -set_next_buddy (pse); - } -/* * We can come here with TIF_NEED_RESCHED already set from new task * wake up path.
kernel/sched/features.h +0 -7
@@ -8,13 +8,6 @@ SCHED_FEAT(PLACE_LAG, true) SCHED_FEAT(PLACE_DEADLINE_INITIAL, true)SCHED_FEAT (RUN_TO_PARITY , true) -/* -* Prefer to schedule the task we woke last (assuming it failed -* wakeup-preemption), since its likely going to consume data we -* touched, increases cache locality. - */ -SCHED_FEAT (NEXT_BUDDY , false) -/* * Consider buddies to be cache hot, decreases the likeliness of a * cache buddy being migrated away, increases cache locality.
Signature · 1 line
2.40.1
PATCH 5/5 sched/fair: Rename set_next_buddy() to set_next_pick()
In reply to Ingo MolnarThis is a mechanism to set the next task_pick target, 'buddy' is too ambiguous and refers to a historic feature we don't have anymore.
--- kernel/sched/fair.c | 28 +++++++++++++--------------- 1 file changed, 13 insertions(+), 15 deletions(-)
kernel/sched/fair.c +13 -15
@@ -3200,7 +3200,16 @@ enqueue_task_fair(struct rq *rq, struct task_struct *p, int flags)hrtick_update (rq); } -static void set_next_buddy (struct sched_entity *se); +static void set_next_pick (struct sched_entity *se) +{ +for_each_sched_entity (se) { +if (SCHED_WARN_ON (!se->on_rq)) +return ; +if (se_is_idle (se)) +return ; +cfs_rq_of (se)->next = se; + } +}/* * The dequeue_task method is called before nr_running is
@@ -3240,7 +3249,7 @@ static void dequeue_task_fair(struct rq *rq, struct task_struct *p, int flags)* p is sleeping when it is within its sched_slice. */ if (task_sleep && se && !throttled_hierarchy (cfs_rq)) -set_next_buddy (se); +set_next_pick (se);break ; } flags |=DEQUEUE_SLEEP ;
@@ -4631,17 +4640,6 @@ balance_fair(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)static inline void set_task_max_allowed_capacity (struct task_struct *p) {}#endif /* CONFIG_SMP */ -static void set_next_buddy (struct sched_entity *se) -{ -for_each_sched_entity (se) { -if (SCHED_WARN_ON (!se->on_rq)) -return ; -if (se_is_idle (se)) -return ; -cfs_rq_of (se)->next = se; - } -} -/* * Preempt the current task with a newly woken task if needed: */
@@ -4769,7 +4767,7 @@ pick_next_task_fair(struct rq *rq, struct task_struct *prev, struct rq_flags *rf goto simple;/* -* Because of the set_next_buddy() in dequeue_task_fair() it is rather +* Because of the set_next_pick() in dequeue_task_fair() it is rather * likely that a next task is from the same cgroup as the current. * * Therefore attempt to avoid putting and setting the entire cgroup
@@ -4957,7 +4955,7 @@ static bool yield_to_task_fair(struct rq *rq, struct task_struct *p)return false;/* Tell the scheduler that we'd really like se to run next. */ -set_next_buddy (se); +set_next_pick (se);yield_task_fair (rq);
Signature · 1 line
2.40.1
Peter Zijlstra
maintainerRe: [PATCH 5/5] sched/fair: Rename set_next_buddy() to set_next_pick()
Quoted 45 lines · Ingo Molnar
This is a mechanism to set the next
task_picktarget, 'buddy' is too ambiguous and refers to a historic feature we don't have anymore.Signed-off-by Ingo Molnar <[email protected]>--- kernel/sched/fair.c | 28 +++++++++++++--------------- 1 file changed, 13 insertions(+), 15 deletions(-)kernel/sched/fair.c +11
-3@@ -3200,7 +3200,16 @@ enqueue_task_fair(struct rq *rq, struct task_struct *p, int flags)hrtick_update (rq); } -static void set_next_buddy (struct sched_entity *se); +static void set_next_pick (struct sched_entity *se) +{ +for_each_sched_entity (se) { +if (SCHED_WARN_ON (!se->on_rq)) +return ; +if (se_is_idle (se)) +return ; +cfs_rq_of (se)->next = se; + } +}/* * The dequeue_task method is called before nr_running is @@ -3240,7 +3249,7 @@ static void dequeue_task_fair(struct rq *rq, struct task_struct *p, int flags)* p is sleeping when it is within its sched_slice. */ if (task_sleep && se && !throttled_hierarchy (cfs_rq)) -set_next_buddy (se); +set_next_pick (se);break ; } flags |=DEQUEUE_SLEEP ;@@ -4631,17 +4640,6 @@ balance_fair(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)static inline void set_task_max_allowed_capacity (struct task_struct *p) {}#endif /* CONFIG_SMP */ -static void set_next_buddy (struct sched_entity *se)
kernel/sched/fair.c balance_fair(struct rq *rq, struct task_struct *prev, struct rq_flags *rf) · line 4644
Quoted 10 lines
kernel/sched/fair.c +0
-10-{ -for_each_sched_entity (se) { -if (SCHED_WARN_ON (!se->on_rq)) -return ; -if (se_is_idle (se)) -return ; -cfs_rq_of (se)->next = se; - } -} -
Hurmmm.. afaict the only actual user of cfs_rq->next left is task_hot(), no? Is that thing worth it?
That is, should we not totally nuke the thing?
kernel/sched/fair.c pick_next_task_fair(struct rq *rq, struct task_struct *prev, struct rq_flags *rf · line 4772
Quoted 11 lines
kernel/sched/fair.c +1
-1/* * Preempt the current task with a newly woken task if needed: */ @@ -4769,7 +4767,7 @@ pick_next_task_fair(struct rq *rq, struct task_struct *prev, struct rq_flags *rf goto simple;/* -* Because of the set_next_buddy() in dequeue_task_fair() it is rather +* Because of the set_next_pick() in dequeue_task_fair() it is rather * likely that a next task is from the same cgroup as the current. *
So, given you killed the ->next consideration in pick, isn't this comment 'misleading' at best?
Re: [PATCH 5/5] sched/fair: Rename set_next_buddy() to set_next_pick()
Quoted 45 lines · Peter Zijlstra
Quoted 44 lines · Ingo Molnar
This is a mechanism to set the next
task_picktarget, 'buddy' is too ambiguous and refers to a historic feature we don't have anymore.Signed-off-by Ingo Molnar <[email protected]>--- kernel/sched/fair.c | 28 +++++++++++++--------------- 1 file changed, 13 insertions(+), 15 deletions(-)kernel/sched/fair.c +11
-2@@ -3200,7 +3200,16 @@ enqueue_task_fair(struct rq *rq, struct task_struct *p, int flags)hrtick_update (rq); } -static void set_next_buddy (struct sched_entity *se); +static void set_next_pick (struct sched_entity *se) +{ +for_each_sched_entity (se) { +if (SCHED_WARN_ON (!se->on_rq)) +return ; +if (se_is_idle (se)) +return ; +cfs_rq_of (se)->next = se; + } +}/* * The dequeue_task method is called before nr_running is @@ -3240,7 +3249,7 @@ static void dequeue_task_fair(struct rq *rq, struct task_struct *p, int flags)* p is sleeping when it is within its sched_slice. */ if (task_sleep && se && !throttled_hierarchy (cfs_rq)) -set_next_buddy (se); +set_next_pick (se);break ; } flags |=DEQUEUE_SLEEP ;@@ -4631,17 +4640,6 @@ balance_fair(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)static inline void set_task_max_allowed_capacity (struct task_struct *p) {}#endif /* CONFIG_SMP */
Quoted 14 lines
kernel/sched/fair.c balance_fair(struct rq *rq, struct task_struct *prev, struct rq_flags *rf) · line 4644
Quoted 11 lines
kernel/sched/fair.c +0
-11-static void set_next_buddy (struct sched_entity *se) -{ -for_each_sched_entity (se) { -if (SCHED_WARN_ON (!se->on_rq)) -return ; -if (se_is_idle (se)) -return ; -cfs_rq_of (se)->next = se; - } -} -Hurmmm.. afaict the only actual user of
cfs_rq->nextleft istask_hot(), no? Is that thing worth it?
Yeah, so:
1)
While you are correct in the context of my patch, I think that might be a bug - the yield_to() methods are intending to use ->next:
/* Tell the scheduler that we'd really like se to run next. */
set_next_buddy(se);
yield_task_fair(rq);... and yield_to() would rather fundamentally rely on ->next overriding the next-task-pick selection, but it won't due to NEXT_BUDDY being false:
static struct sched_entity *pick_next_entity(struct cfs_rq *cfs_rq)
{
/*
* Enabling NEXT_BUDDY will affect latency but not fairness.
*/
if (sched_feat(NEXT_BUDDY) &&
cfs_rq->next && entity_eligible(cfs_rq, cfs_rq->next))
return cfs_rq->next;Quoted 1 line
That is, should we not totally nuke the thing?
I don't think we want to nuke it - there's 3 users:
- yield()
- CFS bandwidth
- wakeup
I think the yield() and CFS bandwidth ones are genuine, but non-working due to NEXT_BUDDY at 0. Wakeup was the original intended NEXT_BUDDY logic, but it got turned off due to some performance or latency considerations that might or might not be valid & relevant today.
2)
Even the task_hot() use of ->next isn't spurious: if a task has been marked as run-next, then presumably the current task is descheduling and we should probably not tear its ->next away in load-balancing.
3)
Side note: a set rq->next should probably reduce a candidate runqueue's weight both in periodic load-balancing and in idle-balancing, by rq->curr's weight or so?
So what I think we should do is to keep ->next and fix all its intended uses, and make it all unconditional by removing both NEXT_BUDDY and CACHE_HOT_BUDDY. I can cook up a series if you agree in principle.
Thanks,
Ingo
Peter Zijlstra
maintainerRe: [PATCH 5/5] sched/fair: Rename set_next_buddy() to set_next_pick()
Quoted 19 lines · Ingo Molnar
I don't think we want to nuke it - there's 3 users:
-
yield()
- CFS bandwidth
- wakeupI think the
yield()and CFS bandwidth ones are genuine, but non-working due toNEXT_BUDDYat 0. Wakeup was the original intendedNEXT_BUDDYlogic, but it got turned off due to some performance or latency considerations that might or might not be valid & relevant today.2)
Even the
task_hot()use of ->next isn't spurious: if a task has been marked as run-next, then presumably the current task is descheduling and we should probably not tear its ->next away in load-balancing.3)
Quoted 7 lines
Side note: a set
rq->nextshould probably reduce a candidate runqueue's weight both in periodic load-balancing and in idle-balancing, by rq->curr's weight or so?So what I think we should do is to keep ->next and fix all its intended uses, and make it all unconditional by removing both
NEXT_BUDDYandCACHE_HOT_BUDDY. I can cook up a series if you agree in principle.
So yes on fixing those yield_to() and cfs_bandwidth thingies, but put then under a new knob -- if the fix regresses we can simply flip it.
Ack on removing the current knobs, for them not having been changed in forever.