Skip to contentSkip to conversation

perf build: Improve header handling for BPF skeleton cross-builds

RFC In review hupu 3 participants 7 messages Dec 23, 2025 – Feb 11 perf events

lore
Share & tools
lore

hupu

author
#0 lore raw

cover perf build: Improve header handling for BPF skeleton cross-builds

I am currently cross-compiling perf for ARM64. During the build of the eBPF skeleton (compiled by Clang), the build fails due to missing header files. The error message is as follows:

/usr/include/linux/ioctl.h:5:10: fatal error: 'asm/ioctl.h' file not found

    5 | #include <asm/ioctl.h>
      |          ^~~~~~~~~~~~~

Leo suggested resolving this issue by installing the following packages on the host:

  $ sudo apt-get install gcc-aarch64-linux-gnu g++-aarch64-linux-gnu
  $ sudo apt-get install libc6-dev-aarch64-cross linux-libc-dev-aarch64-cross
  $ sudo apt-get install libc6-dev-arm64-cross linux-libc-dev-arm64-cross

With these packages installed, the required header files can be found in the following host paths (defined by the CLANG_SYS_INCLUDES variable), and perf can indeed be built successfully:

*** Makefile.perf:1203: ***
CLANG_SYS_INCLUDES=-idirafter /usr/lib/llvm-18/lib/clang/18/include
-idirafter /usr/local/include
-idirafter /usr/bin/../lib/gcc-cross/aarch64-linux-gnu/14/../../../../aarch64-linux-gnu/include -idirafter /usr/include/aarch64-linux-gnu
-idirafter /usr/include

However, I do not believe that relying on the host build environment is the best approach, for several reasons:

a) These commands install UAPI header files on the host, especially linux-libc-dev-aarch64-cross and linux-libc-dev-arm64-cross. These headers originate from the kernel source tree’s include/uapi/ and arch/arm64/include/uapi/ directories, and their versions are tied to the *HOST* kernel version. If the target kernel version is different, mismatches may cause compilation errors or even runtime failures.

b) Even if perf can be compiled and run successfully now, there is no guarantee that the kernel source headers will always match the host-installed UAPI headers as the upstream kernel evolves.

c) In scenarios where the host acts as a general build server and needs to build multiple target kernel versions, it is not possible to ensure that the host UAPI headers are compatible with all target versions.

d) On the other hand, CLANG_SYS_INCLUDES does include host headers, but it uses -idirafter instead of -I. This means the host headers have lower priority. This change was introduced in commit a2af0f6b8ef7 ("perf build: Add system include paths to BPF builds"); as noted in the commit message, the preferred approach is to use kernel source headers rather than potentially older ones from the system.

Therefore, although installing the corresponding packages on the host can resolve the immediate build issue, I do not think this should be considered the only solution, nor should it have the highest priority.

Instead, I think the header search order should be as follows.

- Prefer kernel source headers
[RFC 2/2] perf build: Prefer kernel source headers for BPF skeletons With this approach, even as the kernel version evolves, the headers used during the build will always match the target kernel version.

- Allow users to specify header search paths matching the target kernel version (eg. via EXTRA_BPF_FLAGS)
[RFC 1/2] perf build: Allow passing extra Clang flags via EXTRA_BPF_FLAGS With this approach, users can explicitly specify the appropriate header search paths via EXTRA_BPF_FLAGS.

- Fall back to the host build environment only if necessary Reliance on the host build environment should be the last resort.

In summary, while relying on the host build environment can address the current build issue, I believe it is not the optimal solution.

hupu (2):
perf build: Allow passing extra Clang flags via EXTRA_BPF_FLAGS perf build: Prefer kernel source headers for BPF skeletons

 tools/perf/Makefile.perf | 13 +++++++++++--
 1 file changed, 11 insertions(+), 2 deletions(-)
Signature · 1 line
2.43.0
author
#1 lore raw

RFC 1/2 perf build: Allow passing extra Clang flags via EXTRA_BPF_FLAGS

Add support for EXTRA_BPF_FLAGS in the eBPF skeleton build, allowing users to pass additional Clang options such as --sysroot or custom include paths when cross-compiling perf.

This is primarily intended for cross-build scenarios where the default host include paths do not match the target kernel version.

Example usage:

    make perf ARCH="arm64" EXTRA_BPF_FLAGS="--sysroot=..."

Change history:
v5:
- Update commit message
v4:
- Apply EXTRA_BPF_FLAGS directly to BPF skeleton build command instead of modifying CLANG_OPTIONS
v3:
- Move the variable description comment to the top of Makefile.perf
- Update commit message
v2:
- Rename EXTRA_CLANG_FLAGS to EXTRA_BPF_FLAGS
- Update commit message
v1:
- Introduce EXTRA_CLANG_FLAGS to allow passing extra Clang options

---
 tools/perf/Makefile.perf | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)
tools/perf/Makefile.perf +4 -1
index b3f481a626af..8b8647fd4f41 100644
@@ -35,6 +35,9 @@ include ../scripts/utilities.mak
 #
 # Define EXTRA_CFLAGS=-m64 or EXTRA_CFLAGS=-m32 as appropriate for cross-builds.
 #
+# Define EXTRA_BPF_FLAGS="--sysroot=<path>" or other custom include paths for
+# cross-compiling BPF skeletons
+#
 # Define EXCLUDE_EXTLIBS=-lmylib to exclude libmylib from the auto-generated
 # EXTLIBS.
 #
@@ -1250,7 +1253,7 @@ endif
 $(SKEL_TMP_OUT)/%.bpf.o: $(OUTPUT)PERF-VERSION-FILE util/bpf_skel/perf_version.h | $(SKEL_TMP_OUT)
 $(SKEL_TMP_OUT)/%.bpf.o: util/bpf_skel/%.bpf.c $(LIBBPF) $(SKEL_OUT)/vmlinux.h
 	$(QUIET_CLANG)$(CLANG) -g -O2 -fno-stack-protector --target=bpf \
-	  $(CLANG_OPTIONS) $(BPF_INCLUDE) $(TOOLS_UAPI_INCLUDE) \
+	  $(CLANG_OPTIONS) $(EXTRA_BPF_FLAGS) $(BPF_INCLUDE) $(TOOLS_UAPI_INCLUDE) \
 	  -include $(OUTPUT)PERF-VERSION-FILE -include util/bpf_skel/perf_version.h \
 	  -c $(filter util/bpf_skel/%.bpf.c,$^) -o $@
 
Signature · 1 line
2.43.0
maintainer Reviewed-by on v1 1/2
#2 lore raw

Re: [RFC 1/2] perf build: Allow passing extra Clang flags via EXTRA_BPF_FLAGS

Hello,

Quoted 25 lines · hupu

Add support for EXTRA_BPF_FLAGS in the eBPF skeleton build, allowing users to pass additional Clang options such as --sysroot or custom include paths when cross-compiling perf.

This is primarily intended for cross-build scenarios where the default host include paths do not match the target kernel version.

Example usage:

    make perf ARCH="arm64" EXTRA_BPF_FLAGS="--sysroot=..."

Change history:
v5:
- Update commit message
v4:
- Apply EXTRA_BPF_FLAGS directly to BPF skeleton build command instead of modifying CLANG_OPTIONS
v3:
- Move the variable description comment to the top of Makefile.perf
- Update commit message
v2:
- Rename EXTRA_CLANG_FLAGS to EXTRA_BPF_FLAGS
- Update commit message
v1:
- Introduce EXTRA_CLANG_FLAGS to allow passing extra Clang options

Quoted 1 line

Thanks,
Namhyung

Quoted 30 lines
---
 tools/perf/Makefile.perf | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)
tools/perf/Makefile.perf +4 -1
index b3f481a626af..8b8647fd4f41 100644
@@ -35,6 +35,9 @@ include ../scripts/utilities.mak
 #
 # Define EXTRA_CFLAGS=-m64 or EXTRA_CFLAGS=-m32 as appropriate for cross-builds.
 #
+# Define EXTRA_BPF_FLAGS="--sysroot=<path>" or other custom include paths for
+# cross-compiling BPF skeletons
+#
 # Define EXCLUDE_EXTLIBS=-lmylib to exclude libmylib from the auto-generated
 # EXTLIBS.
 #
@@ -1250,7 +1253,7 @@ endif
 $(SKEL_TMP_OUT)/%.bpf.o: $(OUTPUT)PERF-VERSION-FILE util/bpf_skel/perf_version.h | $(SKEL_TMP_OUT)
 $(SKEL_TMP_OUT)/%.bpf.o: util/bpf_skel/%.bpf.c $(LIBBPF) $(SKEL_OUT)/vmlinux.h
 	$(QUIET_CLANG)$(CLANG) -g -O2 -fno-stack-protector --target=bpf \
-	  $(CLANG_OPTIONS) $(BPF_INCLUDE) $(TOOLS_UAPI_INCLUDE) \
+	  $(CLANG_OPTIONS) $(EXTRA_BPF_FLAGS) $(BPF_INCLUDE) $(TOOLS_UAPI_INCLUDE) \
 	  -include $(OUTPUT)PERF-VERSION-FILE -include util/bpf_skel/perf_version.h \
 	  -c $(filter util/bpf_skel/%.bpf.c,$^) -o $@
 
Signature · 1 line
2.43.0
author
#3 lore raw

Re: [RFC 1/2] perf build: Allow passing extra Clang flags via EXTRA_BPF_FLAGS

Quoted 29 lines · Namhyung Kim

Hello,

Quoted 25 lines · hupu

Add support for EXTRA_BPF_FLAGS in the eBPF skeleton build, allowing users to pass additional Clang options such as --sysroot or custom include paths when cross-compiling perf.

This is primarily intended for cross-build scenarios where the default host include paths do not match the target kernel version.

Example usage:

    make perf ARCH="arm64" EXTRA_BPF_FLAGS="--sysroot=..."

Change history:
v5:
- Update commit message
v4:
- Apply EXTRA_BPF_FLAGS directly to BPF skeleton build command instead of modifying CLANG_OPTIONS
v3:
- Move the variable description comment to the top of Makefile.perf
- Update commit message
v2:
- Rename EXTRA_CLANG_FLAGS to EXTRA_BPF_FLAGS
- Update commit message
v1:
- Introduce EXTRA_CLANG_FLAGS to allow passing extra Clang options

Quoted 7 lines
Quoted 1 line

Thanks,
Namhyung

Hi Namhyung,

Sorry to bother you.

I noticed that the patch adding support for EXTRA_BPF_FLAGS, which was previously reviewed by you, does not seem to be included in the latest released version. I just wanted to follow up and check on the current status of this patch, in case it was missed or delayed for some reason.

Thanks a lot for your time and review.

Thanks
hupu

#4 lore raw

Re: [RFC 1/2] perf build: Allow passing extra Clang flags via EXTRA_BPF_FLAGS

Quoted 39 lines · hupu
Quoted 29 lines · Namhyung Kim

Hello,

Quoted 25 lines · hupu

Add support for EXTRA_BPF_FLAGS in the eBPF skeleton build, allowing users to pass additional Clang options such as --sysroot or custom include paths when cross-compiling perf.

This is primarily intended for cross-build scenarios where the default host include paths do not match the target kernel version.

Example usage:

    make perf ARCH="arm64" EXTRA_BPF_FLAGS="--sysroot=..."

Change history:
v5:
- Update commit message
v4:
- Apply EXTRA_BPF_FLAGS directly to BPF skeleton build command instead of modifying CLANG_OPTIONS
v3:
- Move the variable description comment to the top of Makefile.perf
- Update commit message
v2:
- Rename EXTRA_CLANG_FLAGS to EXTRA_BPF_FLAGS
- Update commit message
v1:
- Introduce EXTRA_CLANG_FLAGS to allow passing extra Clang options

Quoted 4 lines
Quoted 1 line

Hi Namhyung,

Sorry to bother you.

Quoted 7 lines

I noticed that the patch adding support for EXTRA_BPF_FLAGS, which was previously reviewed by you, does not seem to be included in the latest released version. I just wanted to follow up and check on the current status of this patch, in case it was missed or delayed for some reason.

Thanks a lot for your time and review.

Applied this one,

- Arnaldo

author
#5 lore raw

RFC 2/2 perf build: Prefer kernel source headers for BPF skeletons

In reply to hupu

When building eBPF skeletons with Clang, prefer header files provided by the kernel source tree over headers installed on the host system.

While host-installed UAPI headers can be sufficient today, they are tightly coupled to the host environment and may diverge from the target kernel version over time as the kernel evolves. Relying on kernel source headers helps ensure that the headers used during the build remain consistent with the target kernel version and reduces dependence on the host build environment.

This change adds the kernel source UAPI and generated header paths to the BPF include list and places them before the system include paths, while still keeping host headers (CLANG_SYS_INCLUDES) as a fallback.

---
 tools/perf/Makefile.perf | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)
tools/perf/Makefile.perf +7 -1
index 8b8647fd4f41..57c73ab98703 100644
@@ -1203,7 +1203,13 @@ endif
 
 CLANG_OPTIONS = -Wall
 CLANG_SYS_INCLUDES = $(call get_sys_includes,$(CLANG),$(CLANG_TARGET_ARCH))
-BPF_INCLUDE := -I$(SKEL_TMP_OUT)/.. -I$(LIBBPF_INCLUDE) $(CLANG_SYS_INCLUDES)
+
+KHDR_INCLUDES := -I$(abspath $(OUTPUT)../../usr/include) \
+-I$(abspath $(OUTPUT)../../arch/$(SRCARCH)/include/generated/uapi) \
+-I$(abspath $(OUTPUT)../../arch/$(SRCARCH)/include/uapi)
+
+BPF_INCLUDE := -I$(SKEL_TMP_OUT)/.. -I$(LIBBPF_INCLUDE) \
+		$(KHDR_INCLUDES) $(CLANG_SYS_INCLUDES)
 TOOLS_UAPI_INCLUDE := -I$(srctree)/tools/include/uapi
 
 ifneq ($(WERROR),0)
Signature · 1 line
2.43.0
author
#6 lore raw

Re: [RFC 2/2] perf build: Prefer kernel source headers for BPF skeletons

Hi Maintainers,

Sorry for the noise.

This patch was sent nearly three weeks ago, but I have not received any feedback yet.
I am resending this email as a gentle reminder in case it was missed.

Thank you very much for your time and review.

Best regards,
hupu