selftests: sched_ext: Add sched_ext as proper selftest target
Subsystems (4)
PATCH v2 1/1 selftests: sched_ext: Add sched_ext as proper selftest target
From: Björn Töpel <[email protected]>
The sched_ext selftests is missing proper cross-compilation support, a proper target entry, and out-of-tree build support.
When building the kselftest suite, e.g.:
make ARCH=riscv CROSS_COMPILE=riscv64-linux-gnu- \
SKIP_TARGETS="" O=/output/foo -C tools/testing/selftests installThe expectation is that the sched_ext is included, cross-built, and placed into /output/foo.
Add CROSS_COMPILE, OUTPUT, and TARGETS support to the sched_ext selftest. Also, remove some variables that were unused by the Makefile.
---
v2: * Removed the duplicated LLVM prefix parsing (David)
* Made sure make clean didn't do a complete mess (David)
* Added sched_ext to default skip (Shuah)
---tools/testing/selftests/Makefile | 9 +--
tools/testing/selftests/sched_ext/Makefile | 80 +++++++++++----------- 2 files changed, 45 insertions(+), 44 deletions(-)
tools/testing/selftests/Makefile +5 -4
@@ -88,6 +88,7 @@ TARGETS += rlimits TARGETS += rseq TARGETS += rtc TARGETS += rust +TARGETS += sched_ext TARGETS += seccomp TARGETS += sgx TARGETS += sigaltstack
@@ -129,10 +130,10 @@ ifeq ($(filter net/lib,$(TARGETS)),) endif endif -# User can optionally provide a TARGETS skiplist. By default we skip -# BPF since it has cutting edge build time dependencies which require -# more effort to install. -SKIP_TARGETS ?= bpf +# User can optionally provide a TARGETS skiplist. By default we skip +# targets using BPF since it has cutting edge build time dependencies +# which require more effort to install. +SKIP_TARGETS ?= bpf sched_ext ifneq ($(SKIP_TARGETS),) TMP := $(filter-out $(SKIP_TARGETS), $(TARGETS)) override TARGETS := $(TMP)
tools/testing/selftests/sched_ext/Makefile +40 -40
150 lines not on this page · raw message
base-commit: 8cf0b93919e13d1e8d4466eb4080a4c4d9d66d7b
Signature · 1 line
2.43.0
Quoted 7 lines · Björn Töpel
When building the kselftest suite, e.g.:
make ARCH=riscv CROSS_COMPILE=riscv64-linux-gnu- \ SKIP_TARGETS="" O=/output/foo -C tools/testing/selftests installThe expectation is that the
sched_extis included, cross-built, and placed into /output/foo.
When building for arm64 with this applied on top of mainline or -next I'm seeing:
make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- -C tools/testing/selftests TARGETS=sched_ext SKIP_TARGETS="
give
CLNG-BPF create_dsq.bpf.o
In file included from create_dsq.bpf.c:9:/home/broonie/git/linux/tools/sched_ext/include/scx/common.bpf.h:33:17: error: use of undeclared identifier 'SCX_DSQ_FLAG_BUILTIN'
33 | _Static_assert(SCX_DSQ_FLAG_BUILTIN,
| ^
and more (my system clang is clang 20). It's also failing with a native x86_64 build in the same way. I've run "make headers_install", it looks like clang is not getting told about the copy of the headers installed in ./usr/include:
clang -g -D__TARGET_ARCH_x86 -mlittle-endian -I/home/broonie/git/linux/tools/testing/selftests/sched_ext/include -I/home/broonie/git/linux/tools/testing/selftests/sched_ext/include/bpf-compat -I/home/broonie/git/linux/tools/testing/selftests/sched_ext/build/include -I/home/broonie/git/linux/tools/include/uapi -I/home/broonie/git/linux/tools/sched_ext/include -I/home/broonie/git/linux/include -idirafter /usr/lib/llvm-20/lib/clang/20/include -idirafter /usr/local/include -idirafter /usr/include/x86_64-linux-gnu -idirafter /usr/include -Wall -Wno-compare-distinct-pointer-types -Wno-incompatible-function-pointer-types -O2 -mcpu=v3 -target bpf -c create_dsq.bpf.c -o /home/broonie/git/linux/tools/testing/selftests/sched_ext/build/obj/sched_ext/create_dsq.bpf.o
Quoted 7 lines
Add
CROSS_COMPILE, OUTPUT, and TARGETS support to thesched_extselftest. Also, remove some variables that were unused by theMakefile.tools/testing/selftests/sched_ext/Makefile +3
-0+ifneq ($(CROSS_COMPILE),) +DEFAULT_BPFTOOL := $(OUTPUT_DIR)/host/sbin/bpftool +HOST_OBJ_DIR := $(OBJ_DIR)/host/bpftool
tools/testing/selftests/sched_ext/Makefile GENHDR := $(GENDIR)/autoconf.h · line 46
Quoted 10 lines
tools/testing/selftests/sched_ext/Makefile +10
-0+HOST_LIBBPF_OUTPUT := $(OBJ_DIR)/host/libbpf/ +HOST_LIBBPF_DESTDIR := $(OUTPUT_DIR)/host/ +HOST_DESTDIR := $(OUTPUT_DIR)/host/ +else +DEFAULT_BPFTOOL := $(OUTPUT_DIR)/sbin/bpftool +HOST_OBJ_DIR := $(OBJ_DIR)/bpftool +HOST_LIBBPF_OUTPUT := $(OBJ_DIR)/libbpf/ +HOST_LIBBPF_DESTDIR := $(OUTPUT_DIR)/ +HOST_DESTDIR := $(OUTPUT_DIR)/ +endif
This won't detect a cross compile when building using LLVM as that doesn't need to set CROSS_COMPILE, it can use the same binaries for all targets. There's runes in the Makefile for the mm selftests for identifying the target architecture, though actually I'm wondering if it's worth just always building the host copy and never having to worry if the target build is a cross build or not? It's obvious overhead in the native case though if we actually need the target copy.
Quoted 4 lines · Mark Brown
Quoted 1 line · Björn Töpel
When building the kselftest suite, e.g.:
Quoted 8 lines
Quoted 5 lines
make ARCH=riscv CROSS_COMPILE=riscv64-linux-gnu- \ SKIP_TARGETS="" O=/output/foo -C tools/testing/selftests installThe expectation is that the
sched_extis included, cross-built, and placed into /output/foo.When building for arm64 with this applied on top of mainline or -next I'm seeing:
Thanks for taking it for a spin!
Quoted 9 lines
make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- -C
tools/testing/selftestsTARGETS=sched_ext SKIP_TARGETS="give
CLNG-BPF create_dsq.bpf.o
In file included from create_dsq.bpf.c:9:/home/broonie/git/linux/tools/sched_ext/include/scx/common.bpf.h:33:17: error: use of undeclared identifier 'SCX_DSQ_FLAG_BUILTIN'
33 | _Static_assert(SCX_DSQ_FLAG_BUILTIN,
| ^
This is most likely due to incorrect VMLINUX_BTF_PATHS, so that vmlinux.h is incorrectly generated. Try grepping for SCX_DSQ_FLAG_BUILTIN in vmlinux.h.
Quoted 6 lines
and more (my system clang is clang 20). It's also failing with a native
x86_64build in the same way. I've run "makeheaders_install", it looks like clang is not getting told about the copy of the headers installed in ./usr/include:clang -g -D__TARGET_ARCH_x86 -mlittle-endian -I/home/broonie/git/linux/tools/testing/selftests/sched_ext/include -I/home/broonie/git/linux/tools/testing/selftests/sched_ext/include/bpf-compat -I/home/broonie/git/linux/tools/testing/selftests/sched_ext/build/include -I/home/broonie/git/linux/tools/include/uapi -I/home/broonie/git/linux/tools/sched_ext/include -I/home/broonie/git/linux/include -idirafter
/usr/lib/llvm-20/lib/clang/20/include-idirafter/usr/local/include-idirafter/usr/include/x86_64-linux-gnu-idirafter/usr/include-Wall -Wno-compare-distinct-pointer-types -Wno-incompatible-function-pointer-types -O2 -mcpu=v3 -target bpf -ccreate_dsq.bpf.c-o/home/broonie/git/linux/tools/testing/selftests/sched_ext/build/obj/sched_ext/create_dsq.bpf.o
The sched_ext BPF programs relies on a vmlinux.h, which is generated using bpftool and the vmlinux with BTF information. Have you built a kernel with BTF support?
Quoted 18 lines
Quoted 7 lines
Add
CROSS_COMPILE, OUTPUT, and TARGETS support to thesched_extselftest. Also, remove some variables that were unused by theMakefile.tools/testing/selftests/sched_ext/Makefile +3
-0+ifneq ($(CROSS_COMPILE),) +DEFAULT_BPFTOOL := $(OUTPUT_DIR)/host/sbin/bpftool +HOST_OBJ_DIR := $(OBJ_DIR)/host/bpftooltools/testing/selftests/sched_ext/Makefile GENHDR := $(GENDIR)/autoconf.h · line 46
Quoted 10 lines
tools/testing/selftests/sched_ext/Makefile +10
-0+HOST_LIBBPF_OUTPUT := $(OBJ_DIR)/host/libbpf/ +HOST_LIBBPF_DESTDIR := $(OUTPUT_DIR)/host/ +HOST_DESTDIR := $(OUTPUT_DIR)/host/ +else +DEFAULT_BPFTOOL := $(OUTPUT_DIR)/sbin/bpftool +HOST_OBJ_DIR := $(OBJ_DIR)/bpftool +HOST_LIBBPF_OUTPUT := $(OBJ_DIR)/libbpf/ +HOST_LIBBPF_DESTDIR := $(OUTPUT_DIR)/ +HOST_DESTDIR := $(OUTPUT_DIR)/ +endif
Quoted 7 lines
This won't detect a cross compile when building using LLVM as that doesn't need to set
CROSS_COMPILE, it can use the same binaries for all targets. There's runes in theMakefilefor the mm selftests for identifying the target architecture, though actually I'm wondering if it's worth just always building the host copy and never having to worry if the target build is a cross build or not? It's obvious overhead in the native case though if we actually need the target copy.
Yeah, that would indeed simplify things! I'll spin a v3 with that (and wait for more feedback).
Thanks for testing the patch!
Björn
Quoted 9 lines · Björn Töpel
Quoted 7 lines · Mark Brown
On Mon, Oct 07, 2024 at 09:31:32AM +0200, Björn Töpel wrote:
CLNG-BPF create_dsq.bpf.o
In file included from create_dsq.bpf.c:9:/home/broonie/git/linux/tools/sched_ext/include/scx/common.bpf.h:33:17: error: use of undeclared identifier 'SCX_DSQ_FLAG_BUILTIN'
33 | _Static_assert(SCX_DSQ_FLAG_BUILTIN,
| ^
Quoted 3 lines
This is most likely due to incorrect
VMLINUX_BTF_PATHS, so thatvmlinux.his incorrectly generated. Try grepping forSCX_DSQ_FLAG_BUILTINinvmlinux.h.
Yeah, it's not in the generated files:
broonie@finisterre:~/git/linux$ grep SCX_DSQ_FLAG_BUILTIN ./tools/testing/selftests/sched_ext/build/include/vmlinux.h ./tools/testing/selftests/sched_ext/build/obj/bpftool/vmlinux.h broonie@finisterre:~/git/linux$
I didn't actually build a kernel, if the build system needs a kernel it's just silently not detected that it's missing?
Quoted 8 lines · Björn Töpel
Quoted 2 lines · Mark Brown
When building for arm64 with this applied on top of mainline or -next I'm seeing:
Thanks for taking it for a spin!
Quoted 1 line
make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- -C
tools/testing/selftestsTARGETS=sched_ext SKIP_TARGETS="
Oh, and for arm64 as previously noted the sched_ext config fragment odesn't DTRT, merge_config.sh says:
Value requested for CONFIG_SCHED_DEBUG not in final .config Requested value: CONFIG_SCHED_DEBUG=y
Actual value:
Value requested for CONFIG_SCHED_CLASS_EXT not in final .config Requested value: CONFIG_SCHED_CLASS_EXT=y
Actual value:
Value requested for CONFIG_EXT_GROUP_SCHED not in final .config Requested value: CONFIG_EXT_GROUP_SCHED=y
Actual value:
Value requested for CONFIG_DEBUG_INFO not in final .config Requested value: CONFIG_DEBUG_INFO=y
Actual value:
Value requested for CONFIG_DEBUG_INFO_BTF not in final .config Requested value: CONFIG_DEBUG_INFO_BTF=y
Actual value:
Quoted 14 lines · Mark Brown
Quoted 9 lines · Björn Töpel
Quoted 7 lines · Mark Brown
On Mon, Oct 07, 2024 at 09:31:32AM +0200, Björn Töpel wrote:
CLNG-BPF create_dsq.bpf.o
In file included from create_dsq.bpf.c:9:/home/broonie/git/linux/tools/sched_ext/include/scx/common.bpf.h:33:17: error: use of undeclared identifier 'SCX_DSQ_FLAG_BUILTIN'
33 | _Static_assert(SCX_DSQ_FLAG_BUILTIN,
| ^Quoted 3 lines
This is most likely due to incorrect
VMLINUX_BTF_PATHS, so thatvmlinux.his incorrectly generated. Try grepping forSCX_DSQ_FLAG_BUILTINinvmlinux.h.
Quoted 7 lines
Yeah, it's not in the generated files:
broonie@finisterre:~/git/linux$ grep SCX_DSQ_FLAG_BUILTIN ./tools/testing/selftests/sched_ext/build/include/vmlinux.h ./tools/testing/selftests/sched_ext/build/obj/bpftool/vmlinux.h broonie@finisterre:~/git/linux$I didn't actually build a kernel, if the build system needs a kernel it's just silently not detected that it's missing?
It tries to find a kernel with BTF:
| VMLINUX_BTF_PATHS ?= $(if $(O),$(O)/vmlinux) \ | $(if $(KBUILD_OUTPUT),$(KBUILD_OUTPUT)/vmlinux) \ | ../../../../vmlinux \ | /sys/kernel/btf/vmlinux \ | /boot/vmlinux-$(shell uname -r)
Similar to all the other selftests using BPF.
(Oh, and at some point the BPF parts should be in lib.mk...)
Björn
Quoted 12 lines · Björn Töpel
Quoted 2 lines · Mark Brown
I didn't actually build a kernel, if the build system needs a kernel it's just silently not detected that it's missing?
It tries to find a kernel with BTF:
| VMLINUX_BTF_PATHS ?= $(if $(O),$(O)/vmlinux) \ | $(if $(KBUILD_OUTPUT),$(KBUILD_OUTPUT)/vmlinux) \ | ../../../../vmlinux \ | /sys/kernel/btf/vmlinux \ | /boot/vmlinux-$(shell uname -r)
Quoted 3 lines
Similar to all the other selftests using BPF.
(Oh, and at some point the BPF parts should be in lib.mk...)
Ah, that'll have found my host kernel. There should probably be some sort of check that it's at least targeting the correct architecture since this *will* go wrong.
Quoted 3 lines · Björn Töpel
The
sched_extBPF programs relies on avmlinux.h, which is generated using bpftool and the vmlinux with BTF information. Have you built a kernel with BTF support?
OK, so having beaten the kernel config into shape and using GCC rather than clang for my build I did get this to build fine. I think the main gotchas are the issues with the arm64 defconfig not supporting BTF and the fact that the Makefile silently picked up the host kernel.