perf build: Improve header handling for BPF skeleton cross-builds
cover perf build: Improve header handling for BPF skeleton cross-builds
I am currently cross-compiling perf for ARM64. During the build of the eBPF skeleton (compiled by Clang), the build fails due to missing header files. The error message is as follows:
/usr/include/linux/ioctl.h:5:10: fatal error: 'asm/ioctl.h' file not found
5 | #include <asm/ioctl.h>
| ^~~~~~~~~~~~~Leo suggested resolving this issue by installing the following packages on the host:
$ sudo apt-get install gcc-aarch64-linux-gnu g++-aarch64-linux-gnu $ sudo apt-get install libc6-dev-aarch64-cross linux-libc-dev-aarch64-cross $ sudo apt-get install libc6-dev-arm64-cross linux-libc-dev-arm64-cross
With these packages installed, the required header files can be found in the following host paths (defined by the CLANG_SYS_INCLUDES variable), and perf can indeed be built successfully:
*** Makefile.perf:1203: ***
CLANG_SYS_INCLUDES=-idirafter /usr/lib/llvm-18/lib/clang/18/include
-idirafter /usr/local/include
-idirafter /usr/bin/../lib/gcc-cross/aarch64-linux-gnu/14/../../../../aarch64-linux-gnu/include -idirafter /usr/include/aarch64-linux-gnu
-idirafter /usr/include
However, I do not believe that relying on the host build environment is the best approach, for several reasons:
a) These commands install UAPI header files on the host, especially linux-libc-dev-aarch64-cross and linux-libc-dev-arm64-cross. These headers originate from the kernel source tree’s include/uapi/ and arch/arm64/include/uapi/ directories, and their versions are tied to the *HOST* kernel version. If the target kernel version is different, mismatches may cause compilation errors or even runtime failures.
b) Even if perf can be compiled and run successfully now, there is no guarantee that the kernel source headers will always match the host-installed UAPI headers as the upstream kernel evolves.
c) In scenarios where the host acts as a general build server and needs to build multiple target kernel versions, it is not possible to ensure that the host UAPI headers are compatible with all target versions.
d) On the other hand, CLANG_SYS_INCLUDES does include host headers, but it uses -idirafter instead of -I. This means the host headers have lower priority. This change was introduced in commit a2af0f6b8ef7 ("perf build: Add system include paths to BPF builds"); as noted in the commit message, the preferred approach is to use kernel source headers rather than potentially older ones from the system.
Therefore, although installing the corresponding packages on the host can resolve the immediate build issue, I do not think this should be considered the only solution, nor should it have the highest priority.
Instead, I think the header search order should be as follows.
- Prefer kernel source headers
[RFC 2/2] perf build: Prefer kernel source headers for BPF skeletons With this approach, even as the kernel version evolves, the headers used during the build will always match the target kernel version.
- Allow users to specify header search paths matching the target kernel version (eg. via EXTRA_BPF_FLAGS)
[RFC 1/2] perf build: Allow passing extra Clang flags via EXTRA_BPF_FLAGS With this approach, users can explicitly specify the appropriate header search paths via EXTRA_BPF_FLAGS.
- Fall back to the host build environment only if necessary Reliance on the host build environment should be the last resort.
In summary, while relying on the host build environment can address the current build issue, I believe it is not the optimal solution.
hupu (2):
perf build: Allow passing extra Clang flags via EXTRA_BPF_FLAGS perf build: Prefer kernel source headers for BPF skeletons
tools/perf/Makefile.perf | 13 +++++++++++-- 1 file changed, 11 insertions(+), 2 deletions(-)
Signature · 1 line
2.43.0
RFC 1/2 perf build: Allow passing extra Clang flags via EXTRA_BPF_FLAGS
Add support for EXTRA_BPF_FLAGS in the eBPF skeleton build, allowing users to pass additional Clang options such as --sysroot or custom include paths when cross-compiling perf.
This is primarily intended for cross-build scenarios where the default host include paths do not match the target kernel version.
Example usage:
make perf ARCH="arm64" EXTRA_BPF_FLAGS="--sysroot=..."
Change history:
v5:
- Update commit message
v4:
- Apply EXTRA_BPF_FLAGS directly to BPF skeleton build command instead of modifying CLANG_OPTIONS
v3:
- Move the variable description comment to the top of Makefile.perf
- Update commit message
v2:
- Rename EXTRA_CLANG_FLAGS to EXTRA_BPF_FLAGS
- Update commit message
v1:
- Introduce EXTRA_CLANG_FLAGS to allow passing extra Clang options
--- tools/perf/Makefile.perf | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)
tools/perf/Makefile.perf +4 -1
@@ -35,6 +35,9 @@ include ../scripts/utilities.mak # # Define EXTRA_CFLAGS=-m64 or EXTRA_CFLAGS=-m32 as appropriate for cross-builds. # +# Define EXTRA_BPF_FLAGS="--sysroot=<path>" or other custom include paths for +# cross-compiling BPF skeletons +# # Define EXCLUDE_EXTLIBS=-lmylib to exclude libmylib from the auto-generated # EXTLIBS. #
@@ -1250,7 +1253,7 @@ endif $(SKEL_TMP_OUT)/%.bpf.o: $(OUTPUT)PERF-VERSION-FILE util/bpf_skel/perf_version.h | $(SKEL_TMP_OUT) $(SKEL_TMP_OUT)/%.bpf.o: util/bpf_skel/%.bpf.c $(LIBBPF) $(SKEL_OUT)/vmlinux.h $(QUIET_CLANG)$(CLANG) -g -O2 -fno-stack-protector --target=bpf \ - $(CLANG_OPTIONS) $(BPF_INCLUDE) $(TOOLS_UAPI_INCLUDE) \ + $(CLANG_OPTIONS) $(EXTRA_BPF_FLAGS) $(BPF_INCLUDE) $(TOOLS_UAPI_INCLUDE) \ -include $(OUTPUT)PERF-VERSION-FILE -include util/bpf_skel/perf_version.h \ -c $(filter util/bpf_skel/%.bpf.c,$^) -o $@
Signature · 1 line
2.43.0
Re: [RFC 1/2] perf build: Allow passing extra Clang flags via EXTRA_BPF_FLAGS
Hello,
Quoted 25 lines · hupu
Add support for
EXTRA_BPF_FLAGSin the eBPF skeleton build, allowing users to pass additional Clang options such as --sysroot or custom include paths when cross-compiling perf.This is primarily intended for cross-build scenarios where the default host include paths do not match the target kernel version.
Example usage:
make perf ARCH="arm64" EXTRA_BPF_FLAGS="--sysroot=..."Change history:
v5:
- Update commit message
v4:
- ApplyEXTRA_BPF_FLAGSdirectly to BPF skeleton build command instead of modifyingCLANG_OPTIONS
v3:
- Move the variable description comment to the top of Makefile.perf
- Update commit message
v2:
- RenameEXTRA_CLANG_FLAGStoEXTRA_BPF_FLAGS
- Update commit message
v1:
- IntroduceEXTRA_CLANG_FLAGSto allow passing extra Clang options
Quoted 1 line
Signed-off-by hupu <[email protected]>
Thanks,
Namhyung
Quoted 30 lines
--- tools/perf/Makefile.perf | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)tools/perf/Makefile.perf +4
-1@@ -35,6 +35,9 @@ include ../scripts/utilities.mak # # Define EXTRA_CFLAGS=-m64 or EXTRA_CFLAGS=-m32 as appropriate for cross-builds. # +# Define EXTRA_BPF_FLAGS="--sysroot=<path>" or other custom include paths for +# cross-compiling BPF skeletons +# # Define EXCLUDE_EXTLIBS=-lmylib to exclude libmylib from the auto-generated # EXTLIBS. #@@ -1250,7 +1253,7 @@ endif $(SKEL_TMP_OUT)/%.bpf.o: $(OUTPUT)PERF-VERSION-FILE util/bpf_skel/perf_version.h | $(SKEL_TMP_OUT) $(SKEL_TMP_OUT)/%.bpf.o: util/bpf_skel/%.bpf.c $(LIBBPF) $(SKEL_OUT)/vmlinux.h $(QUIET_CLANG)$(CLANG) -g -O2 -fno-stack-protector --target=bpf \ - $(CLANG_OPTIONS) $(BPF_INCLUDE) $(TOOLS_UAPI_INCLUDE) \ + $(CLANG_OPTIONS) $(EXTRA_BPF_FLAGS) $(BPF_INCLUDE) $(TOOLS_UAPI_INCLUDE) \ -include $(OUTPUT)PERF-VERSION-FILE -include util/bpf_skel/perf_version.h \ -c $(filter util/bpf_skel/%.bpf.c,$^) -o $@Signature · 1 line
2.43.0
Re: [RFC 1/2] perf build: Allow passing extra Clang flags via EXTRA_BPF_FLAGS
Quoted 29 lines · Namhyung Kim
Hello,
Quoted 25 lines · hupu
Add support for
EXTRA_BPF_FLAGSin the eBPF skeleton build, allowing users to pass additional Clang options such as --sysroot or custom include paths when cross-compiling perf.This is primarily intended for cross-build scenarios where the default host include paths do not match the target kernel version.
Example usage:
make perf ARCH="arm64" EXTRA_BPF_FLAGS="--sysroot=..."Change history:
v5:
- Update commit message
v4:
- ApplyEXTRA_BPF_FLAGSdirectly to BPF skeleton build command instead of modifyingCLANG_OPTIONS
v3:
- Move the variable description comment to the top of Makefile.perf
- Update commit message
v2:
- RenameEXTRA_CLANG_FLAGStoEXTRA_BPF_FLAGS
- Update commit message
v1:
- IntroduceEXTRA_CLANG_FLAGSto allow passing extra Clang options
Quoted 7 lines
Quoted 1 line
Signed-off-by hupu <[email protected]>Reviewed-by Namhyung Kim <[email protected]>Thanks,
Namhyung
Hi Namhyung,
Sorry to bother you.
I noticed that the patch adding support for EXTRA_BPF_FLAGS, which was previously reviewed by you, does not seem to be included in the latest released version. I just wanted to follow up and check on the current status of this patch, in case it was missed or delayed for some reason.
Thanks a lot for your time and review.
Thanks
hupu
Arnaldo Carvalho de Melo
maintainerRe: [RFC 1/2] perf build: Allow passing extra Clang flags via EXTRA_BPF_FLAGS
Quoted 39 lines · hupu
Quoted 29 lines · Namhyung Kim
Hello,
Quoted 25 lines · hupu
Add support for
EXTRA_BPF_FLAGSin the eBPF skeleton build, allowing users to pass additional Clang options such as --sysroot or custom include paths when cross-compiling perf.This is primarily intended for cross-build scenarios where the default host include paths do not match the target kernel version.
Example usage:
make perf ARCH="arm64" EXTRA_BPF_FLAGS="--sysroot=..."Change history:
v5:
- Update commit message
v4:
- ApplyEXTRA_BPF_FLAGSdirectly to BPF skeleton build command instead of modifyingCLANG_OPTIONS
v3:
- Move the variable description comment to the top of Makefile.perf
- Update commit message
v2:
- RenameEXTRA_CLANG_FLAGStoEXTRA_BPF_FLAGS
- Update commit message
v1:
- IntroduceEXTRA_CLANG_FLAGSto allow passing extra Clang optionsQuoted 4 lines
Quoted 1 line
Signed-off-by hupu <[email protected]>Reviewed-by Namhyung Kim <[email protected]>Hi Namhyung,
Sorry to bother you.
Quoted 7 lines
I noticed that the patch adding support for
EXTRA_BPF_FLAGS, which was previously reviewed by you, does not seem to be included in the latest released version. I just wanted to follow up and check on the current status of this patch, in case it was missed or delayed for some reason.Thanks a lot for your time and review.
Applied this one,
- Arnaldo
RFC 2/2 perf build: Prefer kernel source headers for BPF skeletons
In reply to hupuWhen building eBPF skeletons with Clang, prefer header files provided by the kernel source tree over headers installed on the host system.
While host-installed UAPI headers can be sufficient today, they are tightly coupled to the host environment and may diverge from the target kernel version over time as the kernel evolves. Relying on kernel source headers helps ensure that the headers used during the build remain consistent with the target kernel version and reduces dependence on the host build environment.
This change adds the kernel source UAPI and generated header paths to the BPF include list and places them before the system include paths, while still keeping host headers (CLANG_SYS_INCLUDES) as a fallback.
--- tools/perf/Makefile.perf | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-)
tools/perf/Makefile.perf +7 -1
@@ -1203,7 +1203,13 @@ endif CLANG_OPTIONS = -Wall CLANG_SYS_INCLUDES = $(call get_sys_includes,$(CLANG),$(CLANG_TARGET_ARCH)) -BPF_INCLUDE := -I$(SKEL_TMP_OUT)/.. -I$(LIBBPF_INCLUDE) $(CLANG_SYS_INCLUDES) + +KHDR_INCLUDES := -I$(abspath $(OUTPUT)../../usr/include) \ +-I$(abspath $(OUTPUT)../../arch/$(SRCARCH)/include/generated/uapi) \ +-I$(abspath $(OUTPUT)../../arch/$(SRCARCH)/include/uapi) + +BPF_INCLUDE := -I$(SKEL_TMP_OUT)/.. -I$(LIBBPF_INCLUDE) \ + $(KHDR_INCLUDES) $(CLANG_SYS_INCLUDES) TOOLS_UAPI_INCLUDE := -I$(srctree)/tools/include/uapi ifneq ($(WERROR),0)
Signature · 1 line
2.43.0
Re: [RFC 2/2] perf build: Prefer kernel source headers for BPF skeletons
Hi Maintainers,
Sorry for the noise.
This patch was sent nearly three weeks ago, but I have not received any feedback yet.
I am resending this email as a gentle reminder in case it was missed.
Thank you very much for your time and review.
Best regards,
hupu