Skip to contentSkip to conversation

perf build: Allow passing extra Clang flags via EXTRA_CLANG_FLAGS

RFC In review hupu 4 participants 44 messages Oct 13, 2025 – Dec 17, 2025 perf events

lore
Share & tools
lore

hupu

author
#0 lore raw

RFC 1/1 perf build: Allow passing extra Clang flags via EXTRA_CLANG_FLAGS

When cross-compiling perf with the BPF option enabled, Clang is used during the build process. Some environments may require additional compiler options such as --sysroot or custom include paths. This patch adds a new make variable, EXTRA_CLANG_FLAGS, which appends user-specified options to CLANG_OPTIONS during BPF skeleton builds.

Example:

EXTRA_CLANG_FLAGS="--sysroot=$SYSROOT"
make perf ARCH=arm64 EXTRA_CLANG_FLAGS="$EXTRA_CLANG_FLAGS"

This enables more flexible cross-compilation setups without modifying Makefile.perf directly.

---
 tools/perf/Makefile.perf | 5 +++++
 1 file changed, 5 insertions(+)
tools/perf/Makefile.perf +5 -0
index 47c906b807ef..b0da7314a0d8 100644
@@ -1249,6 +1249,11 @@ else
 	$(Q)cp "$(VMLINUX_H)" $@
 endif
 
+# Allow users to specify additional Clang options (e.g. --sysroot)
+# when cross-compiling BPF skeletons, enabling more flexible
+# build configurations.
+CLANG_OPTIONS += $(EXTRA_CLANG_FLAGS)
+
 $(SKEL_TMP_OUT)/%.bpf.o: $(OUTPUT)PERF-VERSION-FILE util/bpf_skel/perf_version.h | $(SKEL_TMP_OUT)
 $(SKEL_TMP_OUT)/%.bpf.o: util/bpf_skel/%.bpf.c $(LIBBPF) $(SKEL_OUT)/vmlinux.h
 	$(QUIET_CLANG)$(CLANG) -g -O2 -fno-stack-protector --target=bpf \
Signature · 1 line
2.43.0
author
#1 lore raw

Hi Maintainer,

I encountered some issues while cross-compiling perf (ARCH=arm64), particularly when enabling the BPF option. During the build, Clang fails to compile BPF programs due to missing header files. The relevant error messages are as follows:

---------------------------------------
/usr/bin/ld: skipping incompatible /home/hupu/work/tools/crosstool-ng/output/aarch64-dumpstack-linux-gnu/bin/../aarch64-dumpstack-linux-gnu/sysroot/usr/lib/aarch64-linux-gnu/libzstd.a when searching for -lzstd
/usr/bin/ld: skipping incompatible /home/hupu/work/tools/crosstool-ng/output/aarch64-dumpstack-linux-gnu/bin/../aarch64-dumpstack-linux-gnu/sysroot/usr/lib/libc.a when searching for -lc
/usr/bin/ld: /usr/lib/gcc/x86_64-linux-gnu/13/../../../x86_64-linux-gnu/libcrypto.a(libcrypto-lib-dso_dlfcn.o):
in function `dlfcn_globallookup':
(.text+0x1f): warning: Using 'dlopen' in statically linked
applications requires at runtime the shared libraries from the glibc
version used for linking
/usr/bin/ld: /usr/lib/gcc/x86_64-linux-gnu/13/../../../x86_64-linux-gnu/libcrypto.a(libcrypto-lib-bio_addr.o):
in function `BIO_lookup_ex':
(.text+0xdbc): warning: Using 'getaddrinfo' in statically linked
applications requires at runtime the shared libraries from the glibc
version used for linking
/usr/bin/ld: /usr/lib/gcc/x86_64-linux-gnu/13/../../../x86_64-linux-gnu/libcrypto.a(libcrypto-lib-bio_sock.o):
in function `BIO_gethostbyname':
(.text+0x85): warning: Using 'gethostbyname' in statically linked
applications requires at runtime the shared libraries from the glibc
version used for linking

  CLANG   /home/hupu/work/code/explore/output/build-mainline/tools/perf/util/bpf_skel/.tmp/bpf_prog_profiler.bpf.o
  CLANG   /home/hupu/work/code/explore/output/build-mainline/tools/perf/util/bpf_skel/.tmp/bperf_leader.bpf.o
  CLANG   /home/hupu/work/code/explore/output/build-mainline/tools/perf/util/bpf_skel/.tmp/bperf_follower.bpf.o
  CLANG   /home/hupu/work/code/explore/output/build-mainline/tools/perf/util/bpf_skel/.tmp/bperf_cgroup.bpf.o

In file included from <built-in>:2:
In file included from ./util/bpf_skel/perf_version.h:6:
In file included from <built-in>:2:
In file included from ./util/bpf_skel/perf_version.h:6:
In file included from /home/hupu/work/code/explore/output/build-mainline/tools/perf/util/bpf_skel/.tmp/../vmlinux.h:7:
In file included from /home/hupu/work/code/explore/linux-mainline/tools/include/uapi/linux/perf_event.h:19:
/usr/include/linux/ioctl.h:5:10: fatal error: 'asm/ioctl.h' file not found
In file included from
/home/hupu/work/code/explore/output/build-mainline/tools/perf/util/bpf_skel/.tmp/../vmlinux.h:5 | #include <asm/ioctl.h>

      |          ^~~~~~~~~~~~~

7:
In file included from /home/hupu/work/code/explore/linux-mainline/tools/include/uapi/linux/perf_event.h:19:
In file included from <built-in>:2:
In file included from ./util/bpf_skel/perf_version.h:6:
In file included from /home/hupu/work/code/explore/output/build-mainline/tools/perf/util/bpf_skel/.tmp/../vmlinux.h:7:
In file included from /home/hupu/work/code/explore/linux-mainline/tools/include/uapi/linux/perf_event.h:19:
/usr/include/linux/ioctl.h:5:10: fatal error: 'asm/ioctl.h' file not found

    5 | #include <asm/ioctl.h>
      |          ^~~~~~~~~~~~~
/usr/include/linux/ioctl.h:5:10: fatal error: 'asm/ioctl.h' file not found
    5 | #include <asm/ioctl.h>
      |          ^~~~~~~~~~~~~

1 error generated.
make[3]: *** [Makefile.perf:1259:
/home/hupu/work/code/explore/output/build-mainline/tools/perf/util/bpf_skel/.tmp/bperf_leader.bpf.o] Error 1
make[3]: *** Waiting for unfinished jobs....
In file included from <built-in>:2:
In file included from ./util/bpf_skel/perf_version.h:6:
In file included from /home/hupu/work/code/explore/output/build-mainline/tools/perf/util/bpf_skel/.tmp/../vmlinux.h:7:
In file included from /home/hupu/work/code/explore/linux-mainline/tools/include/uapi/linux/perf_event.h:19:
/usr/include/linux/ioctl.h:5:10: fatal error: 'asm/ioctl.h' file not found

    5 | #include <asm/ioctl.h>
      |          ^~~~~~~~~~~~~

1 error generated.
make[3]: *** [Makefile.perf:1259:
/home/hupu/work/code/explore/output/build-mainline/tools/perf/util/bpf_skel/.tmp/bperf_cgroup.bpf.o] Error 1
1 error generated.
1 error generated.
make[3]: *** [Makefile.perf:1259:
/home/hupu/work/code/explore/output/build-mainline/tools/perf/util/bpf_skel/.tmp/bpf_prog_profiler.bpf.o] Error 1
make[3]: *** [Makefile.perf:1259:
/home/hupu/work/code/explore/output/build-mainline/tools/perf/util/bpf_skel/.tmp/bperf_follower.bpf.o] Error 1
make[2]: *** [Makefile.perf:289: sub-make] Error 2
make[1]: *** [Makefile:76: all] Error 2
make: *** [Makefile:93: perf] Error 2
---------------------------------------

This happens because the eBPF part of perf is compiled with Clang, which by default searches only the host /usr/include/ paths for headers, not the cross-compilation sysroot. As a result, it cannot find the ARM64-specific asm/ioctl.h.

To address this issue, I wrote a patch that introduces a new make variable, EXTRA_CLANG_FLAGS. This allows passing custom Clang options via the command line during the BPF skeleton build, for example --sysroot or custom include paths, without modifying Makefile.perf itself.

Example usage:
EXTRA_CLANG_FLAGS="--sysroot=$SYSROOT"
make perf ARCH=arm64 EXTRA_CLANG_FLAGS="$EXTRA_CLANG_FLAGS"

I have tested this approach and it works. However, I am not sure if this is the optimal solution, so I would like to discuss this further with you and get your feedback.

Thanks,
hupu

reviewer
#2 lore raw
Quoted 7 lines · hupu

Hi Maintainer,

I encountered some issues while cross-compiling perf (ARCH=arm64), particularly when enabling the BPF option. During the build, Clang fails to compile BPF programs due to missing header files. The relevant error messages are as follows:

Thanks for reporting this issue! There has been some recent work in this area by Leo:
https://lore.kernel.org/lkml/[email protected]/
The patches are in Linux v6.18-rc1. Perhaps you could try repeating your build with this tree and reporting issues?

Thanks,
Ian

#3 lore raw
Quoted 9 lines · Ian Rogers
Quoted 7 lines · hupu

Hi Maintainer,

I encountered some issues while cross-compiling perf (ARCH=arm64), particularly when enabling the BPF option. During the build, Clang fails to compile BPF programs due to missing header files. The relevant error messages are as follows:

Quoted 5 lines

Thanks for reporting this issue! There has been some recent work in this area by Leo:
https://lore.kernel.org/lkml/[email protected]/
The patches are in Linux v6.18-rc1. Perhaps you could try repeating your build with this tree and reporting issues?

I am not 100% sure, could you execute install kernel headers and then build perf ?

make headers_install

Thanks,
Leo

author
#4 lore raw

Hi Leo,
Thank you for your reply.

Quoted 17 lines · Leo Yan
Quoted 9 lines · Ian Rogers
Quoted 7 lines · hupu

Hi Maintainer,

I encountered some issues while cross-compiling perf (ARCH=arm64), particularly when enabling the BPF option. During the build, Clang fails to compile BPF programs due to missing header files. The relevant error messages are as follows:

Quoted 5 lines

Thanks for reporting this issue! There has been some recent work in this area by Leo:
https://lore.kernel.org/lkml/[email protected]/
The patches are in Linux v6.18-rc1. Perhaps you could try repeating your build with this tree and reporting issues?

Quoted 5 lines

I am not 100% sure, could you execute install kernel headers and then build perf ?

make headers_install

I am currently building perf for arm64 in an Ubuntu environment using a cross toolchain, rather than compiling the entire perf directly with Clang. Clang is only invoked during the build process when the BPF option is enabled — as shown below where bpf is detected as on:

Auto-detecting system features:

...                                   libdw: [ on  ]
...                                   glibc: [ on  ]
...                                  libelf: [ on  ]
...                                 libnuma: [ on  ]
...                  numa_num_possible_cpus: [ on  ]
...                               libpython: [ OFF ]
...                             libcapstone: [ on  ]
...                               llvm-perf: [ OFF ]
...                                    zlib: [ on  ]
...                                    lzma: [ on  ]
...                               get_cpuid: [ OFF ]
...                                     bpf: [ on  ]
...                                  libaio: [ on  ]
...                                 libzstd: [ on  ]

I have already run the following command and successfully installed the related headers, but the same error still occurs when building perf. I am not sure if I also need to specify additional parameters during the build process to ensure these headers can be found correctly?

make ARCH=arm64 headers_install

Please let me know if you need any more details or testing.

Thanks,
hupu

#5 lore raw

Hi hupu,

On Tue, Oct 14, 2025 at 10:31:55AM +0800, hupu wrote:

[...]

Quoted 10 lines
Quoted 5 lines

I am not 100% sure, could you execute install kernel headers and then build perf ?

make headers_install

I am currently building perf for arm64 in an Ubuntu environment using a cross toolchain, rather than compiling the entire perf directly with Clang. Clang is only invoked during the build process when the BPF option is enabled — as shown below where bpf is detected as on:

Have you installed the GCC cross packages ?

 $ sudo apt-get install gcc-aarch64-linux-gnu g++-aarch64-linux-gnu
 $ sudo apt-get install libc6-dev-aarch64-cross linux-libc-dev-aarch64-cross
 $ sudo apt-get install libc6-dev-arm64-cross linux-libc-dev-arm64-cross

My understanding is arm64 cross compilation tries to find headers in the path /usr/aarch64-linux-gnu/include/ (I confirmed this on Ubuntu/Debian distros). After install GCC cross packages, the headers should appear in the folder.

Thanks,
Leo

author
#6 lore raw

Hi Leo,
Thank you for your reply.

Quoted 12 lines · Leo Yan

Have you installed the GCC cross packages ?

 $ sudo apt-get install gcc-aarch64-linux-gnu g++-aarch64-linux-gnu
 $ sudo apt-get install libc6-dev-aarch64-cross linux-libc-dev-aarch64-cross
 $ sudo apt-get install libc6-dev-arm64-cross linux-libc-dev-arm64-cross

My understanding is arm64 cross compilation tries to find headers in the path /usr/aarch64-linux-gnu/include/ (I confirmed this on Ubuntu/Debian distros). After install GCC cross packages, the headers should appear in the folder.

I hadn’t installed the packages you mentioned earlier, but after running the installation commands you provided, I was indeed able to successfully build perf.

In fact, I’m currently working on creating an SDK package, which includes a cross-toolchain that I built myself using crosstool-NG. My initial idea was to install certain third-party libraries (such as the packages you mentioned) into the cross-toolchain’s sysroot directory. With this approach, even when developing on different host machines, we could simply specify the header search path (pointing to the cross-toolchain’s sysroot directory) during compilation, and the build should succeed without requiring any additional package installation on the system.

Based on this, I think allowing users to extend some options via EXTRA_CLANG_FLAGS could be a flexible way to handle such cases. However, this is just my personal thought and might not be entirely correct, so I’d like to hear your advice.

Thanks,
hupu

#7 lore raw

Hi,

On Wed, Oct 15, 2025 at 07:47:04PM +0800, hupu wrote:

[...]

Quoted 3 lines

I hadn’t installed the packages you mentioned earlier, but after running the installation commands you provided, I was indeed able to successfully build perf.

Great!

Quoted 10 lines

In fact, I’m currently working on creating an SDK package, which includes a cross-toolchain that I built myself using crosstool-NG. My initial idea was to install certain third-party libraries (such as the packages you mentioned) into the cross-toolchain’s sysroot directory. With this approach, even when developing on different host machines, we could simply specify the header search path (pointing to the cross-toolchain’s sysroot directory) during compilation, and the build should succeed without requiring any additional package installation on the system.

Quoted 4 lines

Based on this, I think allowing users to extend some options via EXTRA_CLANG_FLAGS could be a flexible way to handle such cases. However, this is just my personal thought and might not be entirely correct, so I’d like to hear your advice.

In the end, it is up to your target.

- If you just want an enviornment for cross build (no matter arm64 or other archs), my preference is to rely on multi-arch packages.

For example, you can install arm64 version's packages on a Ubuntu x86_64 system. The benefit is you don't need to maintain any toolchain/packages, all are prepared by distro. See [1] for details.

- If you are working on a build system (like buildroot or OpenEmbedded) for releasing a SDK. In this case, I agree that we should include headers and libs provided by the SDK.

Either way, I don't think EXTRA_CLANG_FLAGS is right thing to do, as this flag is only for Clang and it does not cover any case for lib linkage. For SDK case, please check if PKG_CONFIG_SYSROOT_DIR and PKG_CONFIG_LIBDIR are helpful (The doc [2] records info for these variables, but I never verified it).

Thanks,
Leo

[1] https://github.com/perfwiki/main/blob/main/docs/arm64-cross-compilation-dockerfile.md
[2] tools/perf/Documentation/Build.txt

author
#8 lore raw
In reply to hupu

Hi Leo, Ian, and fellow maintainers.

Quoted 32 lines · hupu

Hi Leo,
Thank you for your reply.

Quoted 12 lines · Leo Yan

Have you installed the GCC cross packages ?

 $ sudo apt-get install gcc-aarch64-linux-gnu g++-aarch64-linux-gnu
 $ sudo apt-get install libc6-dev-aarch64-cross linux-libc-dev-aarch64-cross
 $ sudo apt-get install libc6-dev-arm64-cross linux-libc-dev-arm64-cross

My understanding is arm64 cross compilation tries to find headers in the path /usr/aarch64-linux-gnu/include/ (I confirmed this on Ubuntu/Debian distros). After install GCC cross packages, the headers should appear in the folder.

I hadn’t installed the packages you mentioned earlier, but after running the installation commands you provided, I was indeed able to successfully build perf.

In fact, I’m currently working on creating an SDK package, which includes a cross-toolchain that I built myself using crosstool-NG. My initial idea was to install certain third-party libraries (such as the packages you mentioned) into the cross-toolchain’s sysroot directory. With this approach, even when developing on different host machines, we could simply specify the header search path (pointing to the cross-toolchain’s sysroot directory) during compilation, and the build should succeed without requiring any additional package installation on the system.

Quoted 5 lines

Based on this, I think allowing users to extend some options via EXTRA_CLANG_FLAGS could be a flexible way to handle such cases. However, this is just my personal thought and might not be entirely correct, so I’d like to hear your advice.

Although installing the GCC cross packages allows me to build perf successfully, I still prefer to be able to pass additional Clang flags via EXTRA_CLANG_FLAGS, as this approach feels more flexible to me.

I look forward to continuing the discussion on this topic.

Thanks,
hupu

maintainer
#9 lore raw

Hello,

Quoted 42 lines · hupu

Hi Leo, Ian, and fellow maintainers.

Quoted 32 lines · hupu

Hi Leo,
Thank you for your reply.

Quoted 12 lines · Leo Yan

Have you installed the GCC cross packages ?

 $ sudo apt-get install gcc-aarch64-linux-gnu g++-aarch64-linux-gnu
 $ sudo apt-get install libc6-dev-aarch64-cross linux-libc-dev-aarch64-cross
 $ sudo apt-get install libc6-dev-arm64-cross linux-libc-dev-arm64-cross

My understanding is arm64 cross compilation tries to find headers in the path /usr/aarch64-linux-gnu/include/ (I confirmed this on Ubuntu/Debian distros). After install GCC cross packages, the headers should appear in the folder.

I hadn’t installed the packages you mentioned earlier, but after running the installation commands you provided, I was indeed able to successfully build perf.

In fact, I’m currently working on creating an SDK package, which includes a cross-toolchain that I built myself using crosstool-NG. My initial idea was to install certain third-party libraries (such as the packages you mentioned) into the cross-toolchain’s sysroot directory. With this approach, even when developing on different host machines, we could simply specify the header search path (pointing to the cross-toolchain’s sysroot directory) during compilation, and the build should succeed without requiring any additional package installation on the system.

Quoted 5 lines

Based on this, I think allowing users to extend some options via EXTRA_CLANG_FLAGS could be a flexible way to handle such cases. However, this is just my personal thought and might not be entirely correct, so I’d like to hear your advice.

Quoted 5 lines

Although installing the GCC cross packages allows me to build perf successfully, I still prefer to be able to pass additional Clang flags via EXTRA_CLANG_FLAGS, as this approach feels more flexible to me.

I look forward to continuing the discussion on this topic.

Doesn't EXTRA_CFLAGS work for you?

Thanks,
Namhyung

author
#10 lore raw

Hi Namhyung,
Thanks for your reply.

Quoted 46 lines · Namhyung Kim

Hello,

Quoted 42 lines · hupu

Hi Leo, Ian, and fellow maintainers.

Quoted 37 lines · hupu

Hi Leo,
Thank you for your reply.

Quoted 12 lines · Leo Yan

Have you installed the GCC cross packages ?

 $ sudo apt-get install gcc-aarch64-linux-gnu g++-aarch64-linux-gnu
 $ sudo apt-get install libc6-dev-aarch64-cross linux-libc-dev-aarch64-cross
 $ sudo apt-get install libc6-dev-arm64-cross linux-libc-dev-arm64-cross

My understanding is arm64 cross compilation tries to find headers in the path /usr/aarch64-linux-gnu/include/ (I confirmed this on Ubuntu/Debian distros). After install GCC cross packages, the headers should appear in the folder.

I hadn’t installed the packages you mentioned earlier, but after running the installation commands you provided, I was indeed able to successfully build perf.

In fact, I’m currently working on creating an SDK package, which includes a cross-toolchain that I built myself using crosstool-NG. My initial idea was to install certain third-party libraries (such as the packages you mentioned) into the cross-toolchain’s sysroot directory. With this approach, even when developing on different host machines, we could simply specify the header search path (pointing to the cross-toolchain’s sysroot directory) during compilation, and the build should succeed without requiring any additional package installation on the system.

Based on this, I think allowing users to extend some options via EXTRA_CLANG_FLAGS could be a flexible way to handle such cases. However, this is just my personal thought and might not be entirely correct, so I’d like to hear your advice.

Quoted 8 lines
Quoted 5 lines

Although installing the GCC cross packages allows me to build perf successfully, I still prefer to be able to pass additional Clang flags via EXTRA_CLANG_FLAGS, as this approach feels more flexible to me.

I look forward to continuing the discussion on this topic.

Doesn't EXTRA_CFLAGS work for you?

I’ve tried using EXTRA_CFLAGS and confirm that it doesn’t work. When I build perf with the following command, it still reports errors:
make perf ARCH=$ARCH CROSS_COMPILE=$CROSS_COMPILE
EXTRA_CFLAGS="--sysroot=$SYSROOT"

In fact, by checking the code in tools/perf/Makefile.perf, we can see that the EXTRA_CFLAGS variable is not passed to Clang at all:

$(SKEL_TMP_OUT)/%.bpf.o: util/bpf_skel/%.bpf.c $(LIBBPF) $(SKEL_OUT)/vmlinux.h $(QUIET_CLANG)$(CLANG) -g -O2 -fno-stack-protector --target=bpf \
$(CLANG_OPTIONS) $(BPF_INCLUDE) $(TOOLS_UAPI_INCLUDE) \
-include $(OUTPUT)PERF-VERSION-FILE -include util/bpf_skel/perf_version.h \
-c $(filter util/bpf_skel/%.bpf.c,$^) -o $@

As shown above, EXTRA_CFLAGS only affects the GCC cross-compilation process and has no impact on the Clang build process. That’s why I proposed introducing an EXTRA_CLANG_FLAGS variable to provide a way to control Clang’s compilation behavior more flexibly.

Thanks,
hupu

maintainer
#11 lore raw
Quoted 56 lines · hupu

Hi Namhyung,
Thanks for your reply.

Quoted 52 lines · Namhyung Kim

Hello,

Quoted 47 lines · hupu

Hi Leo, Ian, and fellow maintainers.

Quoted 32 lines · hupu

Hi Leo,
Thank you for your reply.

Quoted 12 lines · Leo Yan

Have you installed the GCC cross packages ?

 $ sudo apt-get install gcc-aarch64-linux-gnu g++-aarch64-linux-gnu
 $ sudo apt-get install libc6-dev-aarch64-cross linux-libc-dev-aarch64-cross
 $ sudo apt-get install libc6-dev-arm64-cross linux-libc-dev-arm64-cross

My understanding is arm64 cross compilation tries to find headers in the path /usr/aarch64-linux-gnu/include/ (I confirmed this on Ubuntu/Debian distros). After install GCC cross packages, the headers should appear in the folder.

I hadn’t installed the packages you mentioned earlier, but after running the installation commands you provided, I was indeed able to successfully build perf.

In fact, I’m currently working on creating an SDK package, which includes a cross-toolchain that I built myself using crosstool-NG. My initial idea was to install certain third-party libraries (such as the packages you mentioned) into the cross-toolchain’s sysroot directory. With this approach, even when developing on different host machines, we could simply specify the header search path (pointing to the cross-toolchain’s sysroot directory) during compilation, and the build should succeed without requiring any additional package installation on the system.

Quoted 5 lines

Based on this, I think allowing users to extend some options via EXTRA_CLANG_FLAGS could be a flexible way to handle such cases. However, this is just my personal thought and might not be entirely correct, so I’d like to hear your advice.

Although installing the GCC cross packages allows me to build perf successfully, I still prefer to be able to pass additional Clang flags via EXTRA_CLANG_FLAGS, as this approach feels more flexible to me.

I look forward to continuing the discussion on this topic.

Quoted 8 lines
Quoted 2 lines

Doesn't EXTRA_CFLAGS work for you?

I’ve tried using EXTRA_CFLAGS and confirm that it doesn’t work. When I build perf with the following command, it still reports errors:
make perf ARCH=$ARCH CROSS_COMPILE=$CROSS_COMPILE
EXTRA_CFLAGS="--sysroot=$SYSROOT"

Thanks for checking this.

Quoted 11 lines

In fact, by checking the code in tools/perf/Makefile.perf, we can see that the EXTRA_CFLAGS variable is not passed to Clang at all:

$(SKEL_TMP_OUT)/%.bpf.o: util/bpf_skel/%.bpf.c $(LIBBPF) $(SKEL_OUT)/vmlinux.h $(QUIET_CLANG)$(CLANG) -g -O2 -fno-stack-protector --target=bpf \
$(CLANG_OPTIONS) $(BPF_INCLUDE) $(TOOLS_UAPI_INCLUDE) \
-include $(OUTPUT)PERF-VERSION-FILE -include util/bpf_skel/perf_version.h \
-c $(filter util/bpf_skel/%.bpf.c,$^) -o $@

Quoted 4 lines

As shown above, EXTRA_CFLAGS only affects the GCC cross-compilation process and has no impact on the Clang build process. That’s why I proposed introducing an EXTRA_CLANG_FLAGS variable to provide a way to control Clang’s compilation behavior more flexibly.

One thing I care is that we can set clang as a default compiler using CC variable then meaning of EXTRA_CFLAGS and EXTRA_CLANG_FLAGS may be confusing. I guess EXTRA_BPF_FLAGS is clearer that we want to pass something when it builds BPF programs.

Thanks,
Namhyung

author
#12 lore raw

Hi Namhyung,
Thanks for your valuable feedback.

Quoted 24 lines · Namhyung Kim
Quoted 5 lines

I’ve tried using EXTRA_CFLAGS and confirm that it doesn’t work. When I build perf with the following command, it still reports errors:
make perf ARCH=$ARCH CROSS_COMPILE=$CROSS_COMPILE
EXTRA_CFLAGS="--sysroot=$SYSROOT"

Thanks for checking this.

Quoted 11 lines

In fact, by checking the code in tools/perf/Makefile.perf, we can see that the EXTRA_CFLAGS variable is not passed to Clang at all:

$(SKEL_TMP_OUT)/%.bpf.o: util/bpf_skel/%.bpf.c $(LIBBPF) $(SKEL_OUT)/vmlinux.h $(QUIET_CLANG)$(CLANG) -g -O2 -fno-stack-protector --target=bpf \
$(CLANG_OPTIONS) $(BPF_INCLUDE) $(TOOLS_UAPI_INCLUDE) \
-include $(OUTPUT)PERF-VERSION-FILE -include util/bpf_skel/perf_version.h \
-c $(filter util/bpf_skel/%.bpf.c,$^) -o $@

Quoted 4 lines

As shown above, EXTRA_CFLAGS only affects the GCC cross-compilation process and has no impact on the Clang build process. That’s why I proposed introducing an EXTRA_CLANG_FLAGS variable to provide a way to control Clang’s compilation behavior more flexibly.

Quoted 5 lines

One thing I care is that we can set clang as a default compiler using CC variable then meaning of EXTRA_CFLAGS and EXTRA_CLANG_FLAGS may be confusing. I guess EXTRA_BPF_FLAGS is clearer that we want to pass something when it builds BPF programs.

Indeed, I agree with your suggestion to rename the macro from EXTRA_CLANG_FLAGS to EXTRA_BPF_FLAGS, as it better reflects that the options are only intended for building BPF programs.

I will soon send the updated v2 patch with this change.

Thanks,
hupu

author
#13 lore raw

PATCH 1/1 perf build: Support passing extra Clang options via EXTRA_BPF_FLAGS

When cross-compiling perf with BPF enabled, Clang is invoked during the build. Some cross-compilation environments require additional compiler options, such as --sysroot or custom include paths.

This patch introduces a new Make variable, EXTRA_BPF_FLAGS. During BPF skeleton builds, it appends user-provided options to CLANG_OPTIONS, allowing extra Clang flags to be set without modifying Makefile.perf directly.

Example usage:
EXTRA_BPF_FLAGS="--sysroot=$SYSROOT"
make perf ARCH="$ARCH" EXTRA_BPF_FLAGS="$EXTRA_BPF_FLAGS"

Change history:
v2:
- Rename EXTRA_CLANG_FLAGS to EXTRA_BPF_FLAGS
- Update commit message
v1:
- Introduce EXTRA_CLANG_FLAGS to allow passing extra Clang options

---
 tools/perf/Makefile.perf | 5 +++++
 1 file changed, 5 insertions(+)
tools/perf/Makefile.perf +5 -0
index 47c906b807ef..f1f2efdbab8c 100644
@@ -1249,6 +1249,11 @@ else
 	$(Q)cp "$(VMLINUX_H)" $@
 endif
 
+# Allow users to specify additional Clang options (e.g. --sysroot)
+# when cross-compiling BPF skeletons, enabling more flexible
+# build configurations.
+CLANG_OPTIONS += $(EXTRA_BPF_FLAGS)
+
 $(SKEL_TMP_OUT)/%.bpf.o: $(OUTPUT)PERF-VERSION-FILE util/bpf_skel/perf_version.h | $(SKEL_TMP_OUT)
 $(SKEL_TMP_OUT)/%.bpf.o: util/bpf_skel/%.bpf.c $(LIBBPF) $(SKEL_OUT)/vmlinux.h
 	$(QUIET_CLANG)$(CLANG) -g -O2 -fno-stack-protector --target=bpf \
Signature · 1 line
2.43.0
maintainer
#14 lore raw

Re: [PATCH] perf build: Support passing extra Clang options via EXTRA_BPF_FLAGS

Quoted 4 lines · hupu

When cross-compiling perf with BPF enabled, Clang is invoked during the build. Some cross-compilation environments require additional compiler options, such as --sysroot or custom include paths.

Quoted 8 lines

This patch introduces a new Make variable, EXTRA_BPF_FLAGS. During BPF skeleton builds, it appends user-provided options to CLANG_OPTIONS, allowing extra Clang flags to be set without modifying Makefile.perf directly.

Example usage:
EXTRA_BPF_FLAGS="--sysroot=$SYSROOT"
make perf ARCH="$ARCH" EXTRA_BPF_FLAGS="$EXTRA_BPF_FLAGS"

Why not just:

make perf ARCH="arm64" EXTRA_BPF_FLAGS="--sysroot=..."

Quoted 9 lines

Change history:
v2:
- Rename EXTRA_CLANG_FLAGS to EXTRA_BPF_FLAGS
- Update commit message
v1:
- Introduce EXTRA_CLANG_FLAGS to allow passing extra Clang options

Leo, are you ok with this?

Quoted 9 lines
---
 tools/perf/Makefile.perf | 5 +++++
 1 file changed, 5 insertions(+)
tools/perf/Makefile.perf +0 -0
index 47c906b807ef..f1f2efdbab8c 100644

tools/perf/Makefile.perf else · line 1254

Quoted 6 lines
tools/perf/Makefile.perf +3 -0
@@ -1249,6 +1249,11 @@ else
 	$(Q)cp "$(VMLINUX_H)" $@
 endif
 
+# Allow users to specify additional Clang options (e.g. --sysroot)
+# when cross-compiling BPF skeletons, enabling more flexible
+# build configurations.

Can you please move this comment or add new one at the top of the file along with EXTRA_CFLAGS?

Thanks,
Namhyung

Quoted 8 lines
tools/perf/Makefile.perf +2 -0
+CLANG_OPTIONS += $(EXTRA_BPF_FLAGS)
+
 $(SKEL_TMP_OUT)/%.bpf.o: $(OUTPUT)PERF-VERSION-FILE util/bpf_skel/perf_version.h | $(SKEL_TMP_OUT)
 $(SKEL_TMP_OUT)/%.bpf.o: util/bpf_skel/%.bpf.c $(LIBBPF) $(SKEL_OUT)/vmlinux.h
 	$(QUIET_CLANG)$(CLANG) -g -O2 -fno-stack-protector --target=bpf \
Signature · 1 line
2.43.0
author
#15 lore raw

Re: [PATCH] perf build: Support passing extra Clang options via EXTRA_BPF_FLAGS

Hi Namhyung,
Thanks for your review.

Quoted 43 lines · Namhyung Kim
Quoted 4 lines · hupu

When cross-compiling perf with BPF enabled, Clang is invoked during the build. Some cross-compilation environments require additional compiler options, such as --sysroot or custom include paths.

Quoted 8 lines

This patch introduces a new Make variable, EXTRA_BPF_FLAGS. During BPF skeleton builds, it appends user-provided options to CLANG_OPTIONS, allowing extra Clang flags to be set without modifying Makefile.perf directly.

Example usage:
EXTRA_BPF_FLAGS="--sysroot=$SYSROOT"
make perf ARCH="$ARCH" EXTRA_BPF_FLAGS="$EXTRA_BPF_FLAGS"

Why not just:

make perf ARCH="arm64" EXTRA_BPF_FLAGS="--sysroot=..."

Quoted 9 lines

Change history:
v2:
- Rename EXTRA_CLANG_FLAGS to EXTRA_BPF_FLAGS
- Update commit message
v1:
- Introduce EXTRA_CLANG_FLAGS to allow passing extra Clang options

Leo, are you ok with this?

Quoted 12 lines
---
 tools/perf/Makefile.perf | 5 +++++
 1 file changed, 5 insertions(+)
tools/perf/Makefile.perf +0 -0
index 47c906b807ef..f1f2efdbab8c 100644
@@ -1249,6 +1249,11 @@ else
      $(Q)cp "$(VMLINUX_H)" $@
 endif
 
Quoted 7 lines

tools/perf/Makefile.perf else · line 1254

Quoted 3 lines
tools/perf/Makefile.perf +3 -0
+# Allow users to specify additional Clang options (e.g. --sysroot)
+# when cross-compiling BPF skeletons, enabling more flexible
+# build configurations.

Can you please move this comment or add new one at the top of the file along with EXTRA_CFLAGS?

Alright, I will prepare a v3 patch based on your review comments.

Thanks,
hupu

author
#16 lore raw

Re: [PATCH] perf build: Support passing extra Clang options via EXTRA_BPF_FLAGS

When cross-compiling perf with BPF enabled, Clang is invoked during the build. Some cross-compilation environments require additional compiler options, such as --sysroot or custom include paths.

This patch introduces a new Make variable, EXTRA_BPF_FLAGS. During BPF skeleton builds, it appends user-provided options to CLANG_OPTIONS, allowing extra Clang flags to be set without modifying Makefile.perf directly.

Example usage:

    make perf ARCH="arm64" EXTRA_BPF_FLAGS="--sysroot=..."

Change history:
v3:
- Move the variable description comment to the top of Makefile.perf
- Update commit message
v2:
- Rename EXTRA_CLANG_FLAGS to EXTRA_BPF_FLAGS
- Update commit message
v1:
- Introduce EXTRA_CLANG_FLAGS to allow passing extra Clang options

---
 tools/perf/Makefile.perf | 5 +++++
 1 file changed, 5 insertions(+)
tools/perf/Makefile.perf +0 -0
index 47c906b807ef..c1a66cf3d1a7 100644
@@ -35,6 +35,9 @@ include ../scripts/utilities.mak
 #
 # Define EXTRA_CFLAGS=-m64 or EXTRA_CFLAGS=-m32 as appropriate for

cross-builds.
#
+# Define EXTRA_BPF_FLAGS="--sysroot=<path>" or other custom include paths for +# cross-compiling BPF skeletons
+#
# Define EXCLUDE_EXTLIBS=-lmylib to exclude libmylib from the auto-generated # EXTLIBS.
#
@@ -1249,6 +1252,8 @@ else
$(Q)cp "$(VMLINUX_H)" $@
endif

+CLANG_OPTIONS += $(EXTRA_BPF_FLAGS)
+
$(SKEL_TMP_OUT)/%.bpf.o: $(OUTPUT)PERF-VERSION-FILE
util/bpf_skel/perf_version.h | $(SKEL_TMP_OUT)
$(SKEL_TMP_OUT)/%.bpf.o: util/bpf_skel/%.bpf.c $(LIBBPF) $(SKEL_OUT)/vmlinux.h
$(QUIET_CLANG)$(CLANG) -g -O2 -fno-stack-protector --target=bpf \

Signature · 1 line
2.43.0
#17 lore raw

Re: [PATCH] perf build: Support passing extra Clang options via EXTRA_BPF_FLAGS

In reply to Namhyung Kim

On Mon, Oct 20, 2025 at 01:16:46PM +0900, Namhyung Kim wrote:

[...]

Quoted 4 lines · hupu
Quoted 3 lines

When cross-compiling perf with BPF enabled, Clang is invoked during the build. Some cross-compilation environments require additional compiler options, such as --sysroot or custom include paths.

[...]

Quoted 1 line

Leo, are you ok with this?

To be clear, now we are not talking cross build for perf program or any targeting a CPU arch, it is a build failure for eBPF program.

This patch does not make clear why we cannot build eBPF program in self-contained way. E.g., after installed kernel headers, why Makefile misses to include installed headers when build eBPF program.

I am not saying we cannot specify an external sysroot path, but before proceed that, we should make sure the build can work with self-contained headers.

Thanks,
Leo

author
#18 lore raw

Re: [PATCH] perf build: Support passing extra Clang options via EXTRA_BPF_FLAGS

Hi Leo,
Thank you for your reply and for taking the time to discuss this in detail.

Quoted 10 lines · Leo Yan

On Mon, Oct 20, 2025 at 01:16:46PM +0900, Namhyung Kim wrote:

[...]

Quoted 4 lines · hupu
Quoted 3 lines

When cross-compiling perf with BPF enabled, Clang is invoked during the build. Some cross-compilation environments require additional compiler options, such as --sysroot or custom include paths.

Quoted 7 lines

[...]

Quoted 1 line

Leo, are you ok with this?

To be clear, now we are not talking cross build for perf program or any targeting a CPU arch, it is a build failure for eBPF program.

I’d like to clarify the background and scenario once more:
I’m building an SDK that includes a cross-compilation toolchain for the target architecture along with a copy of the kernel source tree. The goal is to make this SDK usable on any host system to build software, including perf with eBPF enabled, without requiring the host to install any additional packages. This is a common requirement in embedded environments, where we often cannot control or modify the host system setup.

Quoted 12 lines · Leo Yan

Have you installed the GCC cross packages ?

 $ sudo apt-get install gcc-aarch64-linux-gnu g++-aarch64-linux-gnu
 $ sudo apt-get install libc6-dev-aarch64-cross linux-libc-dev-aarch64-cross
 $ sudo apt-get install libc6-dev-arm64-cross linux-libc-dev-arm64-cross

My understanding is arm64 cross compilation tries to find headers in the path /usr/aarch64-linux-gnu/include/ (I confirmed this on Ubuntu/Debian distros). After install GCC cross packages, the headers should appear in the folder.

As you mentioned earlier, installing the GCC cross packages on the host does allow perf to be cross-built successfully. This works in my current setup (Ubuntu host kernel 6.14, SDK kernel source 6.18).

However, this approach has two key drawbacks:
a) Limited portability
If the SDK is moved to a different host, these packages must be installed again for it to work, breaking the “plug-and-play” goal and increasing deployment complexity.
b) Kernel version mismatch risk
The headers from these packages are determined by the kernel version in the host’s distribution repository, not the kernel version in the SDK. For example, the host apt repository might ship headers from kernel 5.0 while the SDK contains kernel 6.18; differences in UAPI files (e.g., linux/bpf.h) could cause eBPF programs to fail to build due to missing APIs or structure changes. This risk is greater with fast-evolving subsystems like eBPF, even if it does not happen in my current environment.

This is why I don’t consider relying on host-installed packages as an optimal solution for my case.

Quoted 4 lines

This patch does not make clear why we cannot build eBPF program in self-contained way. E.g., after installed kernel headers, why Makefile misses to include installed headers when build eBPF program.

I may not have explained myself clearly. As noted above, installing the headers you mentioned does allow perf to build successfully, but this is not the approach I ideally want, and it is not the most suitable option for my current context.

The aim of my patch is to explicitly direct the build system to use the SDK’s own header files (via --sysroot or an additional -I path), ensuring that:
a) The build works without installing any host packages;
b) The headers used always match the SDK’s kernel version, eliminating potential version mismatch problems.

I believe this approach meets the principle of self-contained builds and aligns well with real-world embedded development needs.

In addition, allowing additional parameters to be passed to clang via EXTRA_BPF_FLAGS would provide greater flexibility in the build process.

Looking forward to further discussion.

Thanks,
hupu

author
#19 lore raw

Re: [PATCH] perf build: Support passing extra Clang options via EXTRA_BPF_FLAGS

It is worth noting that this is not a special requirement unique to my case, but rather a common need in embedded development scenarios.

Thanks,
hupu

Quoted 98 lines · hupu

Hi Leo,
Thank you for your reply and for taking the time to discuss this in detail.

Quoted 10 lines · Leo Yan

On Mon, Oct 20, 2025 at 01:16:46PM +0900, Namhyung Kim wrote:

[...]

Quoted 4 lines · hupu
Quoted 3 lines

When cross-compiling perf with BPF enabled, Clang is invoked during the build. Some cross-compilation environments require additional compiler options, such as --sysroot or custom include paths.

Quoted 7 lines

[...]

Quoted 1 line

Leo, are you ok with this?

To be clear, now we are not talking cross build for perf program or any targeting a CPU arch, it is a build failure for eBPF program.

I’d like to clarify the background and scenario once more:
I’m building an SDK that includes a cross-compilation toolchain for the target architecture along with a copy of the kernel source tree. The goal is to make this SDK usable on any host system to build software, including perf with eBPF enabled, without requiring the host to install any additional packages. This is a common requirement in embedded environments, where we often cannot control or modify the host system setup.

Quoted 12 lines · Leo Yan

Have you installed the GCC cross packages ?

 $ sudo apt-get install gcc-aarch64-linux-gnu g++-aarch64-linux-gnu
 $ sudo apt-get install libc6-dev-aarch64-cross linux-libc-dev-aarch64-cross
 $ sudo apt-get install libc6-dev-arm64-cross linux-libc-dev-arm64-cross

My understanding is arm64 cross compilation tries to find headers in the path /usr/aarch64-linux-gnu/include/ (I confirmed this on Ubuntu/Debian distros). After install GCC cross packages, the headers should appear in the folder.

As you mentioned earlier, installing the GCC cross packages on the host does allow perf to be cross-built successfully. This works in my current setup (Ubuntu host kernel 6.14, SDK kernel source 6.18).

However, this approach has two key drawbacks:
a) Limited portability
If the SDK is moved to a different host, these packages must be installed again for it to work, breaking the “plug-and-play” goal and increasing deployment complexity.
b) Kernel version mismatch risk
The headers from these packages are determined by the kernel version in the host’s distribution repository, not the kernel version in the SDK. For example, the host apt repository might ship headers from kernel 5.0 while the SDK contains kernel 6.18; differences in UAPI files (e.g., linux/bpf.h) could cause eBPF programs to fail to build due to missing APIs or structure changes. This risk is greater with fast-evolving subsystems like eBPF, even if it does not happen in my current environment.

This is why I don’t consider relying on host-installed packages as an optimal solution for my case.

Quoted 4 lines

This patch does not make clear why we cannot build eBPF program in self-contained way. E.g., after installed kernel headers, why Makefile misses to include installed headers when build eBPF program.

I may not have explained myself clearly. As noted above, installing the headers you mentioned does allow perf to build successfully, but this is not the approach I ideally want, and it is not the most suitable option for my current context.

The aim of my patch is to explicitly direct the build system to use the SDK’s own header files (via --sysroot or an additional -I path), ensuring that:
a) The build works without installing any host packages;
b) The headers used always match the SDK’s kernel version, eliminating potential version mismatch problems.

I believe this approach meets the principle of self-contained builds and aligns well with real-world embedded development needs.

In addition, allowing additional parameters to be passed to clang via EXTRA_BPF_FLAGS would provide greater flexibility in the build process.

Looking forward to further discussion.

Thanks,
hupu

author
#20 lore raw

Re: [PATCH] perf build: Support passing extra Clang options via EXTRA_BPF_FLAGS

Hi Leo,
Thank you for your reply.

Quoted 8 lines
Quoted 7 lines
Quoted 3 lines

To be clear, now we are not talking cross build for perf program or any targeting a CPU arch, it is a build failure for eBPF program.

I’d like to clarify the background and scenario once more:
I’m building an SDK that includes a cross-compilation toolchain for the target architecture along with a copy of the kernel source tree.

Quoted 8 lines

I am not preventing to use toolchains in you mentioned SDK. I just prefer to give priority the headers provided by the kernel source.

Seems to me, a more reasonable series would be:

- Fix the eBPF program build with using self-contained headers;
- Extend to support external headers.

I’m sorry, but I believe there may be some misunderstanding between us regarding the term "self-contained headers". May I ask what exactly you mean by "self-contained headers" in your message above?

From my perspective, it could refer to one of the following:

a) The header files installed via the following apt commands:

Quoted 5 lines

Have you installed the GCC cross packages ?

 $ sudo apt-get install gcc-aarch64-linux-gnu g++-aarch64-linux-gnu
 $ sudo apt-get install libc6-dev-aarch64-cross linux-libc-dev-aarch64-cross
 $ sudo apt-get install libc6-dev-arm64-cross linux-libc-dev-arm64-cross

b) The headers from the Linux kernel source tree.
c) The headers located under the sysroot path of the cross-toolchain.

At the moment, I’m not entirely sure which one you’re referring to, and this uncertainty might have led to some misalignment in our previous discussion. Therefore, it would be very helpful if you could kindly clarify this point.

Thanks,
hupu

author
#21 lore raw

Re: [PATCH] perf build: Support passing extra Clang options via EXTRA_BPF_FLAGS

Hi Leo and Namhyung,
Sorry for the late response, and thanks for the guidance on prioritizing self-contained headers from the kernel source.

Quoted 8 lines
Quoted 7 lines
Quoted 3 lines

To be clear, now we are not talking cross build for perf program or any targeting a CPU arch, it is a build failure for eBPF program.

I’d like to clarify the background and scenario once more:
I’m building an SDK that includes a cross-compilation toolchain for the target architecture along with a copy of the kernel source tree.

Quoted 8 lines

I am not preventing to use toolchains in you mentioned SDK. I just prefer to give priority the headers provided by the kernel source.

Seems to me, a more reasonable series would be:

- Fix the eBPF program build with using self-contained headers;
- Extend to support external headers.

I added the following diagnostic output to Makefile.perf to print CLANG_OPTIONS, BPF_INCLUDE, and TOOLS_UAPI_INCLUDE during compilation, to identify the header search paths used when building the eBPF program.

tools/perf/Makefile.perf +6 -0
index 47c906b807ef..33fe2d8f16c6 100644
@@ -1249,6 +1249,12 @@ else
        $(Q)cp "$(VMLINUX_H)" $@
 endif
 
+$(warning AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA)
+$(warning CLANG_OPTIONS: $(CLANG_OPTIONS))
+$(warning BPF_INCLUDE: $(BPF_INCLUDE))
+$(warning TOOLS_UAPI_INCLUDE: $(TOOLS_UAPI_INCLUDE))
+$(warning AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA)
+
 $(SKEL_TMP_OUT)/%.bpf.o: $(OUTPUT)PERF-VERSION-FILE

util/bpf_skel/perf_version.h | $(SKEL_TMP_OUT)
$(SKEL_TMP_OUT)/%.bpf.o: util/bpf_skel/%.bpf.c $(LIBBPF) $(SKEL_OUT)/vmlinux.h

        $(QUIET_CLANG)$(CLANG) -g -O2 -fno-stack-protector --target=bpf \

The compilation output is as follows:

Makefile.perf:1252: AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
Makefile.perf:1253: CLANG_OPTIONS: -Wall -Werror
Makefile.perf:1254: BPF_INCLUDE:
-I/home/hupu/work/code/explore/output/build-mainline/tools/perf/util/bpf_skel/.tmp/.. -I/home/hupu/work/code/explore/output/build-mainline/tools/perf/libbpf/include -idirafter /usr/lib/llvm-18/lib/clang/18/include -idirafter /usr/local/include -idirafter /usr/include
Makefile.perf:1255: TOOLS_UAPI_INCLUDE:
-I/home/hupu/work/code/explore/linux-mainline/tools/include/uapi Makefile.perf:1256: AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA

From the above BPF_INCLUDE and TOOLS_UAPI_INCLUDE prints, perf was not prioritizing the kernel’s self-contained headers when compiling the eBPF program.

I then added KHDR_INCLUDES to BPF_INCLUDE, pointing it to the kernel’s self-contained header directory so that the build prefers headers provided by the kernel.

tools/perf/Makefile.perf +2 -2
index 47c906b807ef..65c6e871988b 100644
@@ -1202,7 +1202,8 @@ endif
 
 CLANG_OPTIONS = -Wall
 CLANG_SYS_INCLUDES = $(call get_sys_includes,$(CLANG),$(CLANG_TARGET_ARCH))
-BPF_INCLUDE := -I$(SKEL_TMP_OUT)/.. -I$(LIBBPF_INCLUDE) $(CLANG_SYS_INCLUDES)
+KHDR_INCLUDES := $(abspath $(OUTPUT)/../../usr/include)
+BPF_INCLUDE := -I$(SKEL_TMP_OUT)/.. -I$(LIBBPF_INCLUDE)
-I$(KHDR_INCLUDES) $(CLANG_SYS_INCLUDES)
 TOOLS_UAPI_INCLUDE := -I$(srctree)/tools/include/uapi
 

ifneq ($(WERROR),0)

With this change verified, perf compiles successfully even without explicitly specifying the cross-toolchain sysroot in the compile command.

Therefore, before sending PATCH v4, I would appreciate your review and any suggestions.

Thanks,
hupu

author
#22 lore raw

Re: [PATCH] perf build: Support passing extra Clang options via EXTRA_BPF_FLAGS

Hi Leo and Namhyung,

Quoted 23 lines · hupu

I then added KHDR_INCLUDES to BPF_INCLUDE, pointing it to the kernel’s self-contained header directory so that the build prefers headers provided by the kernel.

tools/perf/Makefile.perf +2 -2
index 47c906b807ef..65c6e871988b 100644
@@ -1202,7 +1202,8 @@ endif
 
 CLANG_OPTIONS = -Wall
 CLANG_SYS_INCLUDES = $(call get_sys_includes,$(CLANG),$(CLANG_TARGET_ARCH))
-BPF_INCLUDE := -I$(SKEL_TMP_OUT)/.. -I$(LIBBPF_INCLUDE) $(CLANG_SYS_INCLUDES)
+KHDR_INCLUDES := $(abspath $(OUTPUT)/../../usr/include)
+BPF_INCLUDE := -I$(SKEL_TMP_OUT)/.. -I$(LIBBPF_INCLUDE)
-I$(KHDR_INCLUDES) $(CLANG_SYS_INCLUDES)
 TOOLS_UAPI_INCLUDE := -I$(srctree)/tools/include/uapi
 
 ifneq ($(WERROR),0)
Quoted 4 lines

With this change verified, perf compiles successfully even without explicitly specifying the cross-toolchain sysroot in the compile command.

As an additional suggestion, I’d prefer to keep both the above patch and the previously discussed PATCH v3, which would mean submitting two PRs. From my perspective, allowing users to pass custom compilation options via EXTRA_BPF_FLAGS is more flexible than only configuring headers, so I’d like to hear your thoughts.

Thanks,
hupu

author
#23 lore raw

Re: [PATCH] perf build: Support passing extra Clang options via EXTRA_BPF_FLAGS

RESEND

Quoted 40 lines · hupu

Hi Leo and Namhyung,

Quoted 23 lines · hupu

I then added KHDR_INCLUDES to BPF_INCLUDE, pointing it to the kernel’s self-contained header directory so that the build prefers headers provided by the kernel.

tools/perf/Makefile.perf +2 -2
index 47c906b807ef..65c6e871988b 100644
@@ -1202,7 +1202,8 @@ endif
 
 CLANG_OPTIONS = -Wall
 CLANG_SYS_INCLUDES = $(call get_sys_includes,$(CLANG),$(CLANG_TARGET_ARCH))
-BPF_INCLUDE := -I$(SKEL_TMP_OUT)/.. -I$(LIBBPF_INCLUDE) $(CLANG_SYS_INCLUDES)
+KHDR_INCLUDES := $(abspath $(OUTPUT)/../../usr/include)
+BPF_INCLUDE := -I$(SKEL_TMP_OUT)/.. -I$(LIBBPF_INCLUDE)
-I$(KHDR_INCLUDES) $(CLANG_SYS_INCLUDES)
 TOOLS_UAPI_INCLUDE := -I$(srctree)/tools/include/uapi
 
 ifneq ($(WERROR),0)
Quoted 4 lines

With this change verified, perf compiles successfully even without explicitly specifying the cross-toolchain sysroot in the compile command.

As an additional suggestion, I’d prefer to keep both the above patch and the previously discussed PATCH v3, which would mean submitting two PRs. From my perspective, allowing users to pass custom compilation options via EXTRA_BPF_FLAGS is more flexible than only configuring headers, so I’d like to hear your thoughts.

Thanks,
hupu

maintainer
#24 lore raw

Re: [PATCH] perf build: Support passing extra Clang options via EXTRA_BPF_FLAGS

Hello,

Quoted 19 lines · hupu

RESEND

Quoted 16 lines · hupu

Hi Leo and Namhyung,

Quoted 12 lines · hupu

I then added KHDR_INCLUDES to BPF_INCLUDE, pointing it to the kernel’s self-contained header directory so that the build prefers headers provided by the kernel.

tools/perf/Makefile.perf +0 -0
index 47c906b807ef..65c6e871988b 100644
@@ -1202,7 +1202,8 @@ endif
 
Quoted 6 lines
Quoted 6 lines

tools/perf/Makefile.perf endif · line 1206

Quoted 6 lines
tools/perf/Makefile.perf +2 -2
 CLANG_OPTIONS = -Wall
 CLANG_SYS_INCLUDES = $(call get_sys_includes,$(CLANG),$(CLANG_TARGET_ARCH))
-BPF_INCLUDE := -I$(SKEL_TMP_OUT)/.. -I$(LIBBPF_INCLUDE) $(CLANG_SYS_INCLUDES)
+KHDR_INCLUDES := $(abspath $(OUTPUT)/../../usr/include)
+BPF_INCLUDE := -I$(SKEL_TMP_OUT)/.. -I$(LIBBPF_INCLUDE)
-I$(KHDR_INCLUDES) $(CLANG_SYS_INCLUDES)

I think your patch was wrapped around.

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
tools/perf/Makefile.perf +0 -0
 TOOLS_UAPI_INCLUDE := -I$(srctree)/tools/include/uapi
 
 ifneq ($(WERROR),0)

With this change verified, perf compiles successfully even without explicitly specifying the cross-toolchain sysroot in the compile command.

I'm not familiar with the usr directory in the kernel source. Will it contain some generated headers?

Quoted 7 lines
Quoted 7 lines
Quoted 1 line

As an additional suggestion, I’d prefer to keep both the above patch and the previously discussed PATCH v3, which would mean submitting two PRs. From my perspective, allowing users to pass custom compilation options via EXTRA_BPF_FLAGS is more flexible than only configuring headers, so I’d like to hear your thoughts.

Yep, I'm ok with the extra flag variable.

Thanks,
Namhyung

author
#25 lore raw

Re: [PATCH] perf build: Support passing extra Clang options via EXTRA_BPF_FLAGS

Hi Namhyung,
Thank you very much for your reply.

Quoted 23 lines · Namhyung Kim

Hello,

Quoted 19 lines · hupu

RESEND

Quoted 16 lines · hupu

Hi Leo and Namhyung,

Quoted 12 lines · hupu

I then added KHDR_INCLUDES to BPF_INCLUDE, pointing it to the kernel’s self-contained header directory so that the build prefers headers provided by the kernel.

tools/perf/Makefile.perf +0 -0
index 47c906b807ef..65c6e871988b 100644
@@ -1202,7 +1202,8 @@ endif
 
Quoted 9 lines
Quoted 6 lines
Quoted 6 lines

tools/perf/Makefile.perf endif · line 1206

Quoted 6 lines
tools/perf/Makefile.perf +2 -2
 CLANG_OPTIONS = -Wall
 CLANG_SYS_INCLUDES = $(call get_sys_includes,$(CLANG),$(CLANG_TARGET_ARCH))
-BPF_INCLUDE := -I$(SKEL_TMP_OUT)/.. -I$(LIBBPF_INCLUDE) $(CLANG_SYS_INCLUDES)
+KHDR_INCLUDES := $(abspath $(OUTPUT)/../../usr/include)
+BPF_INCLUDE := -I$(SKEL_TMP_OUT)/.. -I$(LIBBPF_INCLUDE)
-I$(KHDR_INCLUDES) $(CLANG_SYS_INCLUDES)

I think your patch was wrapped around.

Oh, yes — that line was automatically wrapped by the mail system because it exceeded 80 characters.
I’ve slightly adjusted it so that it can be displayed correctly in email clients, as shown below:

tools/perf/Makefile.perf +3 -1
index 47c906b807ef..eaccaae87e3c 100644
@@ -1202,7 +1202,9 @@ endif
 
 CLANG_OPTIONS = -Wall
 CLANG_SYS_INCLUDES = $(call get_sys_includes,$(CLANG),$(CLANG_TARGET_ARCH))
-BPF_INCLUDE := -I$(SKEL_TMP_OUT)/.. -I$(LIBBPF_INCLUDE) $(CLANG_SYS_INCLUDES)
+KHDR_INCLUDES := $(abspath $(OUTPUT)/../../usr/include)
+BPF_INCLUDE := -I$(SKEL_TMP_OUT)/.. -I$(LIBBPF_INCLUDE) \
+               -I$(KHDR_INCLUDES) $(CLANG_SYS_INCLUDES)
 TOOLS_UAPI_INCLUDE := -I$(srctree)/tools/include/uapi
 
 ifneq ($(WERROR),0)
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
tools/perf/Makefile.perf +0 -0
 TOOLS_UAPI_INCLUDE := -I$(srctree)/tools/include/uapi
 
 ifneq ($(WERROR),0)
Quoted 7 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

With this change verified, perf compiles successfully even without explicitly specifying the cross-toolchain sysroot in the compile command.

I'm not familiar with the usr directory in the kernel source. Will it contain some generated headers?

Yes, the headers in KHDR_INCLUDES are directly copied from the kernel source tree (sometimes with very minor modifications). They are exactly the headers required during perf compilation.

Quoted 8 lines
Quoted 7 lines
Quoted 7 lines
Quoted 1 line

As an additional suggestion, I’d prefer to keep both the above patch and the previously discussed PATCH v3, which would mean submitting two PRs. From my perspective, allowing users to pass custom compilation options via EXTRA_BPF_FLAGS is more flexible than only configuring headers, so I’d like to hear your thoughts.

Quoted 5 lines

Yep, I'm ok with the extra flag variable.

Thanks,
Namhyung

Leo, what are your thoughts on this? I’d appreciate hearing your suggestions.

Thanks,
hupu

author
#26 lore raw

Re: [PATCH] perf build: Support passing extra Clang options via EXTRA_BPF_FLAGS

Hi Leo and Namhyung,

Quoted 7 lines · hupu
Quoted 7 lines
Quoted 6 lines
Quoted 6 lines

As an additional suggestion, I’d prefer to keep both the above patch and the previously discussed PATCH v3, which would mean submitting two PRs. From my perspective, allowing users to pass custom compilation options via EXTRA_BPF_FLAGS is more flexible than only configuring headers, so I’d like to hear your thoughts.

Quoted 8 lines
Quoted 5 lines

Yep, I'm ok with the extra flag variable.

Thanks,
Namhyung

Leo, what are your thoughts on this? I’d appreciate hearing your suggestions.

Based on our previous discussion, I have prepared two PATCHes. In this email, let’s continue our discussion on whether we should use the extra flag variable. For the topic of using the kernel’s native self-contained headers, please refer to my other email at the link below.

https://lore.kernel.org/all/[email protected]/

Thanks,
hupu

author
#27 lore raw

PATCH 1/1 perf build: Support passing extra Clang options via EXTRA_BPF_FLAGS

When cross-compiling perf with BPF enabled, Clang is invoked during the build. Some cross-compilation environments require additional compiler options, such as --sysroot or custom include paths.

This patch introduces a new Make variable, EXTRA_BPF_FLAGS. During BPF skeleton builds, it appends user-provided options to CLANG_OPTIONS, allowing extra Clang flags to be set without modifying Makefile.perf directly.

Example usage:

    make perf ARCH="arm64" EXTRA_BPF_FLAGS="--sysroot=..."

Change history:
v3:
- Move the variable description comment to the top of Makefile.perf
- Update commit message
v2:
- Rename EXTRA_CLANG_FLAGS to EXTRA_BPF_FLAGS
- Update commit message
v1:
- Introduce EXTRA_CLANG_FLAGS to allow passing extra Clang options

---
 tools/perf/Makefile.perf | 5 +++++
 1 file changed, 5 insertions(+)
tools/perf/Makefile.perf +5 -0
index 47c906b807ef..c1a66cf3d1a7 100644
@@ -35,6 +35,9 @@ include ../scripts/utilities.mak
 #
 # Define EXTRA_CFLAGS=-m64 or EXTRA_CFLAGS=-m32 as appropriate for cross-builds.
 #
+# Define EXTRA_BPF_FLAGS="--sysroot=<path>" or other custom include paths for
+# cross-compiling BPF skeletons
+#
 # Define EXCLUDE_EXTLIBS=-lmylib to exclude libmylib from the auto-generated
 # EXTLIBS.
 #
@@ -1249,6 +1252,8 @@ else
 	$(Q)cp "$(VMLINUX_H)" $@
 endif
 
+CLANG_OPTIONS += $(EXTRA_BPF_FLAGS)
+
 $(SKEL_TMP_OUT)/%.bpf.o: $(OUTPUT)PERF-VERSION-FILE util/bpf_skel/perf_version.h | $(SKEL_TMP_OUT)
 $(SKEL_TMP_OUT)/%.bpf.o: util/bpf_skel/%.bpf.c $(LIBBPF) $(SKEL_OUT)/vmlinux.h
 	$(QUIET_CLANG)$(CLANG) -g -O2 -fno-stack-protector --target=bpf \
Signature · 1 line
2.43.0
maintainer
#28 lore raw

Re: [PATCH] perf build: Support passing extra Clang options via EXTRA_BPF_FLAGS

Hello,

Quoted 42 lines · hupu

When cross-compiling perf with BPF enabled, Clang is invoked during the build. Some cross-compilation environments require additional compiler options, such as --sysroot or custom include paths.

This patch introduces a new Make variable, EXTRA_BPF_FLAGS. During BPF skeleton builds, it appends user-provided options to CLANG_OPTIONS, allowing extra Clang flags to be set without modifying Makefile.perf directly.

Example usage:

    make perf ARCH="arm64" EXTRA_BPF_FLAGS="--sysroot=..."

Change history:
v3:
- Move the variable description comment to the top of Makefile.perf
- Update commit message
v2:
- Rename EXTRA_CLANG_FLAGS to EXTRA_BPF_FLAGS
- Update commit message
v1:
- Introduce EXTRA_CLANG_FLAGS to allow passing extra Clang options

---
 tools/perf/Makefile.perf | 5 +++++
 1 file changed, 5 insertions(+)
tools/perf/Makefile.perf +3 -0
index 47c906b807ef..c1a66cf3d1a7 100644
@@ -35,6 +35,9 @@ include ../scripts/utilities.mak
 #
 # Define EXTRA_CFLAGS=-m64 or EXTRA_CFLAGS=-m32 as appropriate for cross-builds.
 #
+# Define EXTRA_BPF_FLAGS="--sysroot=<path>" or other custom include paths for
+# cross-compiling BPF skeletons
+#
 # Define EXCLUDE_EXTLIBS=-lmylib to exclude libmylib from the auto-generated
 # EXTLIBS.
 #

tools/perf/Makefile.perf else · line 1255

Quoted 4 lines
tools/perf/Makefile.perf +1 -0
@@ -1249,6 +1252,8 @@ else
 	$(Q)cp "$(VMLINUX_H)" $@
 endif
 
+CLANG_OPTIONS += $(EXTRA_BPF_FLAGS)

While it seems CLANG_OPTIONS is used only for BPF skeletons, I think it's better to add it directly to the below lines instead of adding it to CLANG_OPTIONS like following.

tools/perf/Makefile.perf else · line 1259

Quoted 4 lines
tools/perf/Makefile.perf +1 -0
+
 $(SKEL_TMP_OUT)/%.bpf.o: $(OUTPUT)PERF-VERSION-FILE util/bpf_skel/perf_version.h | $(SKEL_TMP_OUT)
 $(SKEL_TMP_OUT)/%.bpf.o: util/bpf_skel/%.bpf.c $(LIBBPF) $(SKEL_OUT)/vmlinux.h
 	$(QUIET_CLANG)$(CLANG) -g -O2 -fno-stack-protector --target=bpf \
	  $(CLANG_OPTIONS) $(EXTRA_BPF_FLAGS) $(BPF_INCLUDE) $(TOOLS_UAPI_INCLUDE) \

Thanks,
Namhyung

author
#29 lore raw

Re: [PATCH] perf build: Support passing extra Clang options via EXTRA_BPF_FLAGS

Hi Namhyung,
Thank you very much for your valuable suggestion.

Quoted 7 lines · Namhyung Kim
Quoted 2 lines

+CLANG_OPTIONS += $(EXTRA_BPF_FLAGS)

While it seems CLANG_OPTIONS is used only for BPF skeletons, I think it's better to add it directly to the below lines instead of adding it to CLANG_OPTIONS like following.

Quoted 7 lines
Quoted 4 lines

+
$(SKEL_TMP_OUT)/%.bpf.o: $(OUTPUT)PERF-VERSION-FILE util/bpf_skel/perf_version.h | $(SKEL_TMP_OUT) $(SKEL_TMP_OUT)/%.bpf.o: util/bpf_skel/%.bpf.c $(LIBBPF) $(SKEL_OUT)/vmlinux.h

      $(QUIET_CLANG)$(CLANG) -g -O2 -fno-stack-protector --target=bpf \
          $(CLANG_OPTIONS) $(EXTRA_BPF_FLAGS) $(BPF_INCLUDE) $(TOOLS_UAPI_INCLUDE) \

I agree that this approach makes the logic clearer, and I will update the v4 patch to follow your recommendation.

Thanks again for your review and guidance.

Thanks,
hupu

author
#30 lore raw

Re: [PATCH] perf build: Support passing extra Clang options via EXTRA_BPF_FLAGS

When cross-compiling perf with BPF enabled, Clang is invoked during the build. Some cross-compilation environments require additional compiler options, such as --sysroot or custom include paths.

This patch introduces a new Make variable, EXTRA_BPF_FLAGS. During BPF skeleton builds, it appends user-provided options to CLANG_OPTIONS, allowing extra Clang flags to be set without modifying Makefile.perf directly.

Example usage:

    make perf ARCH="arm64" EXTRA_BPF_FLAGS="--sysroot=..."

Change history:
v4:
- Apply EXTRA_BPF_FLAGS directly to BPF skeleton build command instead of modifying CLANG_OPTIONS
v3:
- Move the variable description comment to the top of Makefile.perf
- Update commit message
v2:
- Rename EXTRA_CLANG_FLAGS to EXTRA_BPF_FLAGS
- Update commit message
v1:
- Introduce EXTRA_CLANG_FLAGS to allow passing extra Clang options

---
 tools/perf/Makefile.perf | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)
tools/perf/Makefile.perf +0 -0
index 47c906b807ef..e1cf6accaa4c 100644
@@ -35,6 +35,9 @@ include ../scripts/utilities.mak
 #
 # Define EXTRA_CFLAGS=-m64 or EXTRA_CFLAGS=-m32 as appropriate for

cross-builds.
#
+# Define EXTRA_BPF_FLAGS="--sysroot=<path>" or other custom include paths for +# cross-compiling BPF skeletons
+#
# Define EXCLUDE_EXTLIBS=-lmylib to exclude libmylib from the auto-generated # EXTLIBS.
#
@@ -1252,7 +1255,7 @@ endif
$(SKEL_TMP_OUT)/%.bpf.o: $(OUTPUT)PERF-VERSION-FILE
util/bpf_skel/perf_version.h | $(SKEL_TMP_OUT)
$(SKEL_TMP_OUT)/%.bpf.o: util/bpf_skel/%.bpf.c $(LIBBPF) $(SKEL_OUT)/vmlinux.h
$(QUIET_CLANG)$(CLANG) -g -O2 -fno-stack-protector --target=bpf \
- $(CLANG_OPTIONS) $(BPF_INCLUDE) $(TOOLS_UAPI_INCLUDE) \

+   $(CLANG_OPTIONS) $(EXTRA_BPF_FLAGS) $(BPF_INCLUDE) $(TOOLS_UAPI_INCLUDE) \
    -include $(OUTPUT)PERF-VERSION-FILE -include util/bpf_skel/perf_version.h \
    -c $(filter util/bpf_skel/%.bpf.c,$^) -o $@
Signature · 1 line
2.43.0
#31 lore raw

Re: [PATCH] perf build: Support passing extra Clang options via EXTRA_BPF_FLAGS

On Tue, Nov 25, 2025 at 09:07:26PM +0800, hupu wrote:

[...]

Quoted 14 lines
tools/perf/Makefile.perf +0 -0
@@ -35,6 +35,9 @@ include ../scripts/utilities.mak
 #
 # Define EXTRA_CFLAGS=-m64 or EXTRA_CFLAGS=-m32 as appropriate for

cross-builds.
#

tools/perf/Makefile.perf +3 -0
+# Define EXTRA_BPF_FLAGS="--sysroot=<path>" or other custom include paths for
+# cross-compiling BPF skeletons
+#
 # Define EXCLUDE_EXTLIBS=-lmylib to exclude libmylib from the auto-generated
 # EXTLIBS.
 #

tools/perf/Makefile.perf endif · line 1259

Quoted 6 lines
tools/perf/Makefile.perf +1 -1
@@ -1252,7 +1255,7 @@ endif
 $(SKEL_TMP_OUT)/%.bpf.o: $(OUTPUT)PERF-VERSION-FILE
 util/bpf_skel/perf_version.h | $(SKEL_TMP_OUT)
 $(SKEL_TMP_OUT)/%.bpf.o: util/bpf_skel/%.bpf.c $(LIBBPF) $(SKEL_OUT)/vmlinux.h
  $(QUIET_CLANG)$(CLANG) -g -O2 -fno-stack-protector --target=bpf \
-   $(CLANG_OPTIONS) $(BPF_INCLUDE) $(TOOLS_UAPI_INCLUDE) \
+   $(CLANG_OPTIONS) $(EXTRA_BPF_FLAGS) $(BPF_INCLUDE) $(TOOLS_UAPI_INCLUDE) \

I am concerned for this change.

How can you only build eBPF skel program with "--sysroot" but not applying the same build env on perf binary and associated libs (libperf, bpftool, etc) ?

For a convinced solution, I'd expect we can apply PKG_CONFIG_SYSROOT_DIR for both normal perf build and eBPF skel build. More important, please provide steps for how to build perf with a SDK.

Thanks,
Leo

author
#32 lore raw

Re: [PATCH] perf build: Support passing extra Clang options via EXTRA_BPF_FLAGS

Hi Leo,
Thank you for your reply.

Quoted 28 lines · Leo Yan

On Tue, Nov 25, 2025 at 09:07:26PM +0800, hupu wrote:

[...]

Quoted 14 lines
tools/perf/Makefile.perf +0 -0
@@ -35,6 +35,9 @@ include ../scripts/utilities.mak
 #
 # Define EXTRA_CFLAGS=-m64 or EXTRA_CFLAGS=-m32 as appropriate for

cross-builds.
#

tools/perf/Makefile.perf +3 -0
+# Define EXTRA_BPF_FLAGS="--sysroot=<path>" or other custom include paths for
+# cross-compiling BPF skeletons
+#
 # Define EXCLUDE_EXTLIBS=-lmylib to exclude libmylib from the auto-generated
 # EXTLIBS.
 #

tools/perf/Makefile.perf endif · line 1259

Quoted 6 lines
tools/perf/Makefile.perf +1 -1
@@ -1252,7 +1255,7 @@ endif
 $(SKEL_TMP_OUT)/%.bpf.o: $(OUTPUT)PERF-VERSION-FILE
 util/bpf_skel/perf_version.h | $(SKEL_TMP_OUT)
 $(SKEL_TMP_OUT)/%.bpf.o: util/bpf_skel/%.bpf.c $(LIBBPF) $(SKEL_OUT)/vmlinux.h
  $(QUIET_CLANG)$(CLANG) -g -O2 -fno-stack-protector --target=bpf \
-   $(CLANG_OPTIONS) $(BPF_INCLUDE) $(TOOLS_UAPI_INCLUDE) \
+   $(CLANG_OPTIONS) $(EXTRA_BPF_FLAGS) $(BPF_INCLUDE) $(TOOLS_UAPI_INCLUDE) \

I am concerned for this change.

Quoted 7 lines

How can you only build eBPF skel program with "--sysroot" but not applying the same build env on perf binary and associated libs (libperf, bpftool, etc) ?

For a convinced solution, I'd expect we can apply PKG_CONFIG_SYSROOT_DIR for both normal perf build and eBPF skel build. More important, please

I understand your concern, and I will explain it with reference to the current source code.

The eBPF skeleton is compiled via clang --target=bpf, and its header file search paths mainly come from BPF_INCLUDE and TOOLS_UAPI_INCLUDE. It also uses '-idirafter' to include the host’s /usr/local/include and /usr/include directories in the search path. This process is not directly coupled with cross-compilation managed via pkg-config, which means PKG_CONFIG_SYSROOT_DIR does not affect how the skeleton resolves its headers.

1252 $(SKEL_TMP_OUT)/%.bpf.o: $(OUTPUT)PERF-VERSION-FILE util/bpf_skel/perf_version.h | $(SKEL_TMP_OUT)
1253 $(SKEL_TMP_OUT)/%.bpf.o: util/bpf_skel/%.bpf.c $(LIBBPF) $(SKEL_OUT)/vmlinux.h
1254 $(QUIET_CLANG)$(CLANG) -g -O2 -fno-stack-protector --target=bpf \
1255 $(CLANG_OPTIONS) $(BPF_INCLUDE) $(TOOLS_UAPI_INCLUDE) \
1256 -include $(OUTPUT)PERF-VERSION-FILE -include
util/bpf_skel/perf_version.h \
1257 -c $(filter util/bpf_skel/%.bpf.c,$^) -o $@

I added some debugging output to inspect the contents of BPF_INCLUDE:

BPF_INCLUDE:
-I/home/hupu/work/code/explore/output/build-mainline/tools/perf/util/bpf_skel/.tmp/.. -I/home/hupu/work/code/explore/output/build-mainline/tools/perf/libbpf/include -idirafter /usr/lib/llvm-18/lib/clang/18/include
-idirafter /usr/local/include
-idirafter /usr/include

The perf binary and its associated libraries (such as libperf) are built with the cross toolchain. Their dependency headers and library paths are determined by CFLAGS/LDFLAGS and resolved via pkg-config. In this case, PKG_CONFIG_SYSROOT_DIR is indeed the correct control point.

802 $(OUTPUT)perf: $(PERFLIBS) $(PERF_IN)
803 $(QUIET_LINK)$(CC) $(CFLAGS) $(LDFLAGS) \
804 $(PERF_IN) $(LIBS) -o $@

971 $(LIBPERF): FORCE | $(LIBPERF_OUTPUT)
972 $(Q)$(MAKE) -C $(LIBPERF_DIR) O=$(LIBPERF_OUTPUT) \
973 DESTDIR=$(LIBPERF_DESTDIR) prefix= subdir= \
974 $@ install_headers

From the current build logic, there is no strict requirement for the skeleton and the perf binary to use exactly the same headers, because we cannot guarantee that the headers in the host’s /usr/local/include and /usr/include are fully consistent with the kernel source.

If the goal is to enforce header consistency, my other proposed patch — “perf build: Use self-contained headers from kernel source when compiling” — explicitly ensures that the skeleton uses UAPI headers from the kernel source tree, thus aligning the skeleton’s headers with those used by the perf binary.

Finally, introducing EXTRA_BPF_FLAGS offers a controlled, optional extension point for the skeleton build, rather than solely passing --sysroot. This allows users to add additional options such as -I or -D to address specific include resolution needs in certain environments. It is more flexible than hardcoding paths, and does not modify the existing pkg-config managed search paths used for user-space builds.

Quoted 3 lines

for both normal perf build and eBPF skel build. More important, please provide steps for how to build perf with a SDK.

(I apologize in advance — I may paste some build log output below. This could make the email appear somewhat cluttered, and may also cause line wrapping or truncation in some mail clients.)

If EXTRA_BPF_FLAGS is not specified during the perf build (example
command shown below), the compilation will fail:
make perf ARCH=arm64 CROSS_COMPILE=aarch64-dumpstack-linux-gnu- LDFLAGS="-L/home/hupu/work/tools/x-tools/aarch64-dumpstack-linux-gnu/bin/../aarch64-dumpstack-linux-gnu/sysroot/usr/lib/aarch64-linux-gnu -L/home/hupu/work/tools/x-tools/aarch64-dumpstack-linux-gnu/bin/../aarch64-dumpstack-linux-gnu/sysroot/usr/lib -static" O=/home/hupu/work/code/explore/output/build-mainline -j8

The full build failure log is as follows:

21:38:16 (00:00:00) - INFO : Starting build perf tools for arm64 ...
21:38:16 (00:00:00) - INFO : make perf ARCH=arm64
CROSS_COMPILE=aarch64-dumpstack-linux-gnu- LDFLAGS="-L/home/hupu/work/tools/x-tools/aarch64-dumpstack-linux-gnu/bin/../aarch64-dumpstack-linux-gnu/sysroot/usr/lib/aarch64-linux-gnu -L/home/hupu/work/tools/x-tools/aarch64-dumpstack-linux-gnu/bin/../aarch64-dumpstack-linux-gnu/sysroot/usr/lib -static" O=/home/hupu/work/code/explore/output/build-mainline -j8
BUILD: Doing 'make -j4' parallel build
Warning: Kernel ABI header differences:
diff -u tools/arch/arm64/include/asm/cputype.h
arch/arm64/include/asm/cputype.h
diff -u tools/perf/trace/beauty/include/uapi/linux/mount.h
include/uapi/linux/mount.h
Makefile.config:590: No elfutils/debuginfod.h found, no debuginfo
server support, please install
libdebuginfod-dev/elfutils-debuginfod-client-devel or equivalent
Makefile.config:633: No sys/sdt.h found, no SDT events are defined,
please install systemtap-sdt-devel or systemtap-sdt-dev
Makefile.config:881: No 'Python.h' was found: disables Python support
- please install python-devel/python-dev
Makefile.config:989: No libllvm 13+ found, slower source file
resolution, please install llvm-devel/llvm-dev
Makefile.config:1099: No libbabeltrace found, disables 'perf data' CTF
format support, please install
libbabeltrace-dev[el]/libbabeltrace-ctf-dev
Makefile.config:1142: No alternatives command found, you need to set
JDIR= to point to the root of your Java directory
Makefile.config:1173: libpfm4 not found, disables libpfm4 support.
Please install libpfm-devel or libpfm4-dev

Auto-detecting system features:

...                                   libdw: [ on  ]
...                                   glibc: [ on  ]
...                                  libelf: [ on  ]
...                                 libnuma: [ on  ]
...                  numa_num_possible_cpus: [ on  ]
...                               libpython: [ OFF ]
...                             libcapstone: [ on  ]
...                               llvm-perf: [ OFF ]
...                                    zlib: [ on  ]
...                                    lzma: [ on  ]
...                               get_cpuid: [ OFF ]
...                                     bpf: [ on  ]
...                                  libaio: [ on  ]
...                                 libzstd: [ on  ]
  GEN     /home/hupu/work/code/explore/output/build-mainline/tools/perf/common-cmds.h
  CC      /home/hupu/work/code/explore/output/build-mainline/tools/perf/dlfilters/dlfilter-test-api-v0.o
  GEN     /home/hupu/work/code/explore/output/build-mainline/tools/perf/arch/arm64/include/generated/asm/sysreg-defs.h
  PERF_VERSION = 6.18.rc7.g30f09200cc4a
  CC      /home/hupu/work/code/explore/output/build-mainline/tools/perf/dlfilters/dlfilter-test-api-v2.o
  CC      /home/hupu/work/code/explore/output/build-mainline/tools/perf/dlfilters/dlfilter-show-cycles.o
  MKDIR   /home/hupu/work/code/explore/output/build-mainline/tools/perf/libapi/fd/
  CC      /home/hupu/work/code/explore/output/build-mainline/tools/perf/libapi/fd/array.o
  SYSHDR  /home/hupu/work/code/explore/output/build-mainline/tools/perf/libperf/arch/arm64/include/generated/uapi/asm/unistd_64.h
  INSTALL /home/hupu/work/code/explore/output/build-mainline/tools/perf/libperf/include/perf/bpf_perf.h
  INSTALL /home/hupu/work/code/explore/output/build-mainline/tools/perf/libperf/include/perf/core.h
  INSTALL /home/hupu/work/code/explore/output/build-mainline/tools/perf/libperf/include/perf/cpumap.h
  INSTALL /home/hupu/work/code/explore/output/build-mainline/tools/perf/libperf/include/perf/threadmap.h
  INSTALL /home/hupu/work/code/explore/output/build-mainline/tools/perf/libperf/include/perf/evlist.h
  LD      /home/hupu/work/code/explore/output/build-mainline/tools/perf/libapi/fd/libapi-in.o
  INSTALL /home/hupu/work/code/explore/output/build-mainline/tools/perf/libperf/include/perf/evsel.h
  INSTALL /home/hupu/work/code/explore/output/build-mainline/tools/perf/libperf/include/perf/event.h
  INSTALL /home/hupu/work/code/explore/output/build-mainline/tools/perf/libperf/include/perf/mmap.h
  MKDIR   /home/hupu/work/code/explore/output/build-mainline/tools/perf/libapi/fs/
  INSTALL /home/hupu/work/code/explore/output/build-mainline/tools/perf/libperf/include/internal/cpumap.h
  CC      /home/hupu/work/code/explore/output/build-mainline/tools/perf/libapi/fs/fs.o
  INSTALL /home/hupu/work/code/explore/output/build-mainline/tools/perf/libperf/include/internal/evlist.h
  INSTALL /home/hupu/work/code/explore/output/build-mainline/tools/perf/libsubcmd/include/subcmd/exec-cmd.h
  INSTALL /home/hupu/work/code/explore/output/build-mainline/tools/perf/libperf/include/internal/evsel.h
  INSTALL /home/hupu/work/code/explore/output/build-mainline/tools/perf/libsubcmd/include/subcmd/help.h
  INSTALL /home/hupu/work/code/explore/output/build-mainline/tools/perf/libperf/include/internal/lib.h
  INSTALL /home/hupu/work/code/explore/output/build-mainline/tools/perf/libperf/include/internal/mmap.h
  INSTALL /home/hupu/work/code/explore/output/build-mainline/tools/perf/libsubcmd/include/subcmd/pager.h
  INSTALL /home/hupu/work/code/explore/output/build-mainline/tools/perf/libperf/include/internal/rc_check.h
  INSTALL /home/hupu/work/code/explore/output/build-mainline/tools/perf/libsubcmd/include/subcmd/parse-options.h
  INSTALL /home/hupu/work/code/explore/output/build-mainline/tools/perf/libperf/include/internal/threadmap.h
  INSTALL /home/hupu/work/code/explore/output/build-mainline/tools/perf/libperf/include/internal/xyarray.h
  INSTALL /home/hupu/work/code/explore/output/build-mainline/tools/perf/libsubcmd/include/subcmd/run-command.h
  CC      /home/hupu/work/code/explore/output/build-mainline/tools/perf/libperf/core.o
  CC      /home/hupu/work/code/explore/output/build-mainline/tools/perf/libsubcmd/exec-cmd.o
  INSTALL libsubcmd_headers
  INSTALL /home/hupu/work/code/explore/output/build-mainline/tools/perf/libapi/include/api/cpu.h
  CC      /home/hupu/work/code/explore/output/build-mainline/tools/perf/libapi/cpu.o
  CC      /home/hupu/work/code/explore/output/build-mainline/tools/perf/libperf/cpumap.o
  CC      /home/hupu/work/code/explore/output/build-mainline/tools/perf/libapi/debug.o
  MKDIR   /home/hupu/work/code/explore/output/build-mainline/tools/perf/libapi/fs/
  CC      /home/hupu/work/code/explore/output/build-mainline/tools/perf/libapi/fs/tracing_path.o
  CC      /home/hupu/work/code/explore/output/build-mainline/tools/perf/libsubcmd/help.o
  CC      /home/hupu/work/code/explore/output/build-mainline/tools/perf/libapi/fs/cgroup.o
  CC      /home/hupu/work/code/explore/output/build-mainline/tools/perf/libapi/str_error_r.o
  LD      /home/hupu/work/code/explore/output/build-mainline/tools/perf/libapi/fs/libapi-in.o
  INSTALL libperf_headers
  LD      /home/hupu/work/code/explore/output/build-mainline/tools/perf/libapi/libapi-in.o
  INSTALL /home/hupu/work/code/explore/output/build-mainline/tools/perf/libapi/include/api/debug.h
  INSTALL /home/hupu/work/code/explore/output/build-mainline/tools/perf/libapi/include/api/io.h
  INSTALL /home/hupu/work/code/explore/output/build-mainline/tools/perf/libapi/include/api/io_dir.h
  INSTALL /home/hupu/work/code/explore/output/build-mainline/tools/perf/libapi/include/api/fd/array.h
  INSTALL /home/hupu/work/code/explore/output/build-mainline/tools/perf/libapi/include/api/fs/fs.h
  INSTALL /home/hupu/work/code/explore/output/build-mainline/tools/perf/libapi/include/api/fs/tracing_path.h
  AR      /home/hupu/work/code/explore/output/build-mainline/tools/perf/libapi/libapi.a
  INSTALL libapi_headers
  CC      /home/hupu/work/code/explore/output/build-mainline/tools/perf/libperf/threadmap.o
  CC      /home/hupu/work/code/explore/output/build-mainline/tools/perf/libsymbol/kallsyms.o
  LD      /home/hupu/work/code/explore/output/build-mainline/tools/perf/libsymbol/libsymbol-in.o
  INSTALL /home/hupu/work/code/explore/output/build-mainline/tools/perf/libsymbol/include/symbol/kallsyms.h
  AR      /home/hupu/work/code/explore/output/build-mainline/tools/perf/libsymbol/libsymbol.a
  INSTALL libsymbol_headers
  CC      /home/hupu/work/code/explore/output/build-mainline/tools/perf/libsubcmd/pager.o
  CC      /home/hupu/work/code/explore/output/build-mainline/tools/perf/libperf/evsel.o
  CC      /home/hupu/work/code/explore/output/build-mainline/tools/perf/libsubcmd/parse-options.o
  GEN     /home/hupu/work/code/explore/output/build-mainline/tools/perf/libbpf/bpf_helper_defs.h
  INSTALL /home/hupu/work/code/explore/output/build-mainline/tools/perf/libbpf/include/bpf/bpf.h
  INSTALL /home/hupu/work/code/explore/output/build-mainline/tools/perf/libbpf/include/bpf/libbpf.h
  INSTALL /home/hupu/work/code/explore/output/build-mainline/tools/perf/libbpf/include/bpf/btf.h
  INSTALL /home/hupu/work/code/explore/output/build-mainline/tools/perf/libbpf/include/bpf/libbpf_common.h
  INSTALL /home/hupu/work/code/explore/output/build-mainline/tools/perf/libbpf/include/bpf/libbpf_legacy.h
  INSTALL /home/hupu/work/code/explore/output/build-mainline/tools/perf/libbpf/include/bpf/bpf_helpers.h
  INSTALL /home/hupu/work/code/explore/output/build-mainline/tools/perf/libbpf/include/bpf/bpf_tracing.h
  INSTALL /home/hupu/work/code/explore/output/build-mainline/tools/perf/libbpf/include/bpf/bpf_endian.h
  INSTALL /home/hupu/work/code/explore/output/build-mainline/tools/perf/libbpf/include/bpf/bpf_core_read.h
  INSTALL /home/hupu/work/code/explore/output/build-mainline/tools/perf/libbpf/include/bpf/skel_internal.h
  INSTALL /home/hupu/work/code/explore/output/build-mainline/tools/perf/libbpf/include/bpf/libbpf_version.h
  INSTALL /home/hupu/work/code/explore/output/build-mainline/tools/perf/libbpf/include/bpf/usdt.bpf.h
  INSTALL /home/hupu/work/code/explore/output/build-mainline/tools/perf/libbpf/include/bpf/bpf_helper_defs.h
  CC      /home/hupu/work/code/explore/output/build-mainline/tools/perf/libbpf/staticobjs/libbpf.o
  CC      /home/hupu/work/code/explore/output/build-mainline/tools/perf/libperf/evlist.o
  CC      /home/hupu/work/code/explore/output/build-mainline/tools/perf/libsubcmd/run-command.o
  CC      /home/hupu/work/code/explore/output/build-mainline/tools/perf/libsubcmd/sigchain.o
  CC      /home/hupu/work/code/explore/output/build-mainline/tools/perf/libsubcmd/subcmd-config.o
  LD      /home/hupu/work/code/explore/output/build-mainline/tools/perf/libsubcmd/libsubcmd-in.o
  AR      /home/hupu/work/code/explore/output/build-mainline/tools/perf/libsubcmd/libsubcmd.a
  CC      /home/hupu/work/code/explore/output/build-mainline/tools/perf/libbpf/staticobjs/bpf.o
  CC      /home/hupu/work/code/explore/output/build-mainline/tools/perf/libperf/mmap.o
  CC      /home/hupu/work/code/explore/output/build-mainline/tools/perf/libbpf/staticobjs/nlattr.o
  INSTALL libbpf_headers
  GEN     perf-archive
  CC      /home/hupu/work/code/explore/output/build-mainline/tools/perf/libperf/zalloc.o
  CC      /home/hupu/work/code/explore/output/build-mainline/tools/perf/libbpf/staticobjs/btf.o
  GEN     perf-iostat
  CC      /home/hupu/work/code/explore/output/build-mainline/tools/perf/libperf/xyarray.o
  CC      /home/hupu/work/code/explore/output/build-mainline/tools/perf/libperf/lib.o
  LINK    /home/hupu/work/code/explore/output/build-mainline/tools/perf/dlfilters/dlfilter-test-api-v0.so
  CC      /home/hupu/work/code/explore/output/build-mainline/tools/perf/libbpf/staticobjs/libbpf_utils.o
  LD      /home/hupu/work/code/explore/output/build-mainline/tools/perf/libperf/libperf-in.o
  AR      /home/hupu/work/code/explore/output/build-mainline/tools/perf/libperf/libperf.a
  CC      /home/hupu/work/code/explore/output/build-mainline/tools/perf/libbpf/staticobjs/netlink.o
  LINK    /home/hupu/work/code/explore/output/build-mainline/tools/perf/dlfilters/dlfilter-test-api-v2.so
  CC      /home/hupu/work/code/explore/output/build-mainline/tools/perf/libbpf/staticobjs/bpf_prog_linfo.o
  LINK    /home/hupu/work/code/explore/output/build-mainline/tools/perf/dlfilters/dlfilter-show-cycles.so
  CC      /home/hupu/work/code/explore/output/build-mainline/tools/perf/libbpf/staticobjs/libbpf_probes.o
  CC      /home/hupu/work/code/explore/output/build-mainline/tools/perf/libbpf/staticobjs/hashmap.o
  CC      /home/hupu/work/code/explore/output/build-mainline/tools/perf/libbpf/staticobjs/btf_dump.o
  CC      /home/hupu/work/code/explore/output/build-mainline/tools/perf/libbpf/staticobjs/ringbuf.o
  CC      /home/hupu/work/code/explore/output/build-mainline/tools/perf/libbpf/staticobjs/strset.o
  CC      /home/hupu/work/code/explore/output/build-mainline/tools/perf/libbpf/staticobjs/linker.o
  CC      /home/hupu/work/code/explore/output/build-mainline/tools/perf/libbpf/staticobjs/gen_loader.o
  CC      /home/hupu/work/code/explore/output/build-mainline/tools/perf/libbpf/staticobjs/relo_core.o
  CC      /home/hupu/work/code/explore/output/build-mainline/tools/perf/libbpf/staticobjs/usdt.o
  CC      /home/hupu/work/code/explore/output/build-mainline/tools/perf/libbpf/staticobjs/zip.o
  CC      /home/hupu/work/code/explore/output/build-mainline/tools/perf/libbpf/staticobjs/elf.o
  CC      /home/hupu/work/code/explore/output/build-mainline/tools/perf/libbpf/staticobjs/features.o
  CC      /home/hupu/work/code/explore/output/build-mainline/tools/perf/libbpf/staticobjs/btf_iter.o
  CC      /home/hupu/work/code/explore/output/build-mainline/tools/perf/libbpf/staticobjs/btf_relocate.o
  LD      /home/hupu/work/code/explore/output/build-mainline/tools/perf/libbpf/staticobjs/libbpf-in.o
  LINK    /home/hupu/work/code/explore/output/build-mainline/tools/perf/libbpf/libbpf.a

Auto-detecting system features:
... clang-bpf-co-re: [ on ]
... llvm: [ OFF ]
... libcap: [ on ]
... libbfd: [ OFF ]

  MKDIR   /home/hupu/work/code/explore/output/build-mainline/tools/perf/util/bpf_skel/.tmp/bootstrap/libbpf/include/bpf
  MKDIR   /home/hupu/work/code/explore/output/build-mainline/tools/perf/util/bpf_skel/.tmp/bootstrap/
  MKDIR   /home/hupu/work/code/explore/output/build-mainline/tools/perf/util/bpf_skel/.tmp/bootstrap/libbpf/
  INSTALL /home/hupu/work/code/explore/output/build-mainline/tools/perf/util/bpf_skel/.tmp/bootstrap/libbpf/include/bpf/hashmap.h
  INSTALL /home/hupu/work/code/explore/output/build-mainline/tools/perf/util/bpf_skel/.tmp/bootstrap/libbpf/include/bpf/relo_core.h
  INSTALL /home/hupu/work/code/explore/output/build-mainline/tools/perf/util/bpf_skel/.tmp/bootstrap/libbpf/include/bpf/libbpf_internal.h
  GEN     /home/hupu/work/code/explore/output/build-mainline/tools/perf/util/bpf_skel/.tmp/bootstrap/libbpf/bpf_helper_defs.h
  INSTALL /home/hupu/work/code/explore/output/build-mainline/tools/perf/util/bpf_skel/.tmp/bootstrap/libbpf/include/bpf/bpf.h
  INSTALL /home/hupu/work/code/explore/output/build-mainline/tools/perf/util/bpf_skel/.tmp/bootstrap/libbpf/include/bpf/libbpf.h
  INSTALL /home/hupu/work/code/explore/output/build-mainline/tools/perf/util/bpf_skel/.tmp/bootstrap/libbpf/include/bpf/btf.h
  INSTALL /home/hupu/work/code/explore/output/build-mainline/tools/perf/util/bpf_skel/.tmp/bootstrap/libbpf/include/bpf/libbpf_common.h
  INSTALL /home/hupu/work/code/explore/output/build-mainline/tools/perf/util/bpf_skel/.tmp/bootstrap/libbpf/include/bpf/libbpf_legacy.h
  INSTALL /home/hupu/work/code/explore/output/build-mainline/tools/perf/util/bpf_skel/.tmp/bootstrap/libbpf/include/bpf/bpf_helpers.h
  INSTALL /home/hupu/work/code/explore/output/build-mainline/tools/perf/util/bpf_skel/.tmp/bootstrap/libbpf/include/bpf/bpf_tracing.h
  INSTALL /home/hupu/work/code/explore/output/build-mainline/tools/perf/util/bpf_skel/.tmp/bootstrap/libbpf/include/bpf/bpf_endian.h
  INSTALL /home/hupu/work/code/explore/output/build-mainline/tools/perf/util/bpf_skel/.tmp/bootstrap/libbpf/include/bpf/bpf_core_read.h
  INSTALL /home/hupu/work/code/explore/output/build-mainline/tools/perf/util/bpf_skel/.tmp/bootstrap/libbpf/include/bpf/skel_internal.h
  INSTALL /home/hupu/work/code/explore/output/build-mainline/tools/perf/util/bpf_skel/.tmp/bootstrap/libbpf/include/bpf/libbpf_version.h
  INSTALL /home/hupu/work/code/explore/output/build-mainline/tools/perf/util/bpf_skel/.tmp/bootstrap/libbpf/include/bpf/usdt.bpf.h
  INSTALL /home/hupu/work/code/explore/output/build-mainline/tools/perf/util/bpf_skel/.tmp/bootstrap/libbpf/include/bpf/bpf_helper_defs.h
  INSTALL libbpf_headers
  CC      /home/hupu/work/code/explore/output/build-mainline/tools/perf/util/bpf_skel/.tmp/bootstrap/libbpf/staticobjs/libbpf.o
  CC      /home/hupu/work/code/explore/output/build-mainline/tools/perf/util/bpf_skel/.tmp/bootstrap/libbpf/staticobjs/bpf.o
  CC      /home/hupu/work/code/explore/output/build-mainline/tools/perf/util/bpf_skel/.tmp/bootstrap/libbpf/staticobjs/nlattr.o
  CC      /home/hupu/work/code/explore/output/build-mainline/tools/perf/util/bpf_skel/.tmp/bootstrap/libbpf/staticobjs/btf.o
  CC      /home/hupu/work/code/explore/output/build-mainline/tools/perf/util/bpf_skel/.tmp/bootstrap/libbpf/staticobjs/libbpf_utils.o
  CC      /home/hupu/work/code/explore/output/build-mainline/tools/perf/util/bpf_skel/.tmp/bootstrap/libbpf/staticobjs/netlink.o
  CC      /home/hupu/work/code/explore/output/build-mainline/tools/perf/util/bpf_skel/.tmp/bootstrap/libbpf/staticobjs/bpf_prog_linfo.o
  CC      /home/hupu/work/code/explore/output/build-mainline/tools/perf/util/bpf_skel/.tmp/bootstrap/libbpf/staticobjs/libbpf_probes.o
  CC      /home/hupu/work/code/explore/output/build-mainline/tools/perf/util/bpf_skel/.tmp/bootstrap/libbpf/staticobjs/hashmap.o
  CC      /home/hupu/work/code/explore/output/build-mainline/tools/perf/util/bpf_skel/.tmp/bootstrap/libbpf/staticobjs/btf_dump.o
  CC      /home/hupu/work/code/explore/output/build-mainline/tools/perf/util/bpf_skel/.tmp/bootstrap/libbpf/staticobjs/ringbuf.o
  CC      /home/hupu/work/code/explore/output/build-mainline/tools/perf/util/bpf_skel/.tmp/bootstrap/libbpf/staticobjs/strset.o
  CC      /home/hupu/work/code/explore/output/build-mainline/tools/perf/util/bpf_skel/.tmp/bootstrap/libbpf/staticobjs/linker.o
  CC      /home/hupu/work/code/explore/output/build-mainline/tools/perf/util/bpf_skel/.tmp/bootstrap/libbpf/staticobjs/gen_loader.o
  CC      /home/hupu/work/code/explore/output/build-mainline/tools/perf/util/bpf_skel/.tmp/bootstrap/libbpf/staticobjs/relo_core.o
  CC      /home/hupu/work/code/explore/output/build-mainline/tools/perf/util/bpf_skel/.tmp/bootstrap/libbpf/staticobjs/usdt.o
  CC      /home/hupu/work/code/explore/output/build-mainline/tools/perf/util/bpf_skel/.tmp/bootstrap/libbpf/staticobjs/zip.o
  CC      /home/hupu/work/code/explore/output/build-mainline/tools/perf/util/bpf_skel/.tmp/bootstrap/libbpf/staticobjs/elf.o
  CC      /home/hupu/work/code/explore/output/build-mainline/tools/perf/util/bpf_skel/.tmp/bootstrap/libbpf/staticobjs/features.o
  CC      /home/hupu/work/code/explore/output/build-mainline/tools/perf/util/bpf_skel/.tmp/bootstrap/libbpf/staticobjs/btf_iter.o
  CC      /home/hupu/work/code/explore/output/build-mainline/tools/perf/util/bpf_skel/.tmp/bootstrap/libbpf/staticobjs/btf_relocate.o
  LD      /home/hupu/work/code/explore/output/build-mainline/tools/perf/util/bpf_skel/.tmp/bootstrap/libbpf/staticobjs/libbpf-in.o
  LINK    /home/hupu/work/code/explore/output/build-mainline/tools/perf/util/bpf_skel/.tmp/bootstrap/libbpf/libbpf.a
  CC      /home/hupu/work/code/explore/output/build-mainline/tools/perf/util/bpf_skel/.tmp/bootstrap/main.o
  CC      /home/hupu/work/code/explore/output/build-mainline/tools/perf/util/bpf_skel/.tmp/bootstrap/common.o
  CC      /home/hupu/work/code/explore/output/build-mainline/tools/perf/util/bpf_skel/.tmp/bootstrap/json_writer.o
  CC      /home/hupu/work/code/explore/output/build-mainline/tools/perf/util/bpf_skel/.tmp/bootstrap/gen.o
  CC      /home/hupu/work/code/explore/output/build-mainline/tools/perf/util/bpf_skel/.tmp/bootstrap/btf.o
  CC      /home/hupu/work/code/explore/output/build-mainline/tools/perf/util/bpf_skel/.tmp/bootstrap/sign.o
  LINK    /home/hupu/work/code/explore/output/build-mainline/tools/perf/util/bpf_skel/.tmp/bootstrap/bpftool

/usr/bin/ld: skipping incompatible /home/hupu/work/tools/x-tools/aarch64-dumpstack-linux-gnu/bin/../aarch64-dumpstack-linux-gnu/sysroot/usr/lib/aarch64-linux-gnu/libelf.a when searching for -lelf
/usr/bin/ld: skipping incompatible /home/hupu/work/tools/x-tools/aarch64-dumpstack-linux-gnu/bin/../aarch64-dumpstack-linux-gnu/sysroot/usr/lib/aarch64-linux-gnu/libelf.a when searching for -lelf
/usr/bin/ld: skipping incompatible /home/hupu/work/tools/x-tools/aarch64-dumpstack-linux-gnu/bin/../aarch64-dumpstack-linux-gnu/sysroot/usr/lib/aarch64-linux-gnu/libz.a when searching for -lz
/usr/bin/ld: skipping incompatible /home/hupu/work/tools/x-tools/aarch64-dumpstack-linux-gnu/bin/../aarch64-dumpstack-linux-gnu/sysroot/usr/lib/aarch64-linux-gnu/libz.a when searching for -lz
/usr/bin/ld: skipping incompatible /home/hupu/work/tools/x-tools/aarch64-dumpstack-linux-gnu/bin/../aarch64-dumpstack-linux-gnu/sysroot/usr/lib/aarch64-linux-gnu/libcrypto.a when searching for -lcrypto
/usr/bin/ld: skipping incompatible /home/hupu/work/tools/x-tools/aarch64-dumpstack-linux-gnu/bin/../aarch64-dumpstack-linux-gnu/sysroot/usr/lib/aarch64-linux-gnu/libcrypto.a when searching for -lcrypto
/usr/bin/ld: skipping incompatible /home/hupu/work/tools/x-tools/aarch64-dumpstack-linux-gnu/bin/../aarch64-dumpstack-linux-gnu/sysroot/usr/lib/aarch64-linux-gnu/libzstd.a when searching for -lzstd
/usr/bin/ld: skipping incompatible /home/hupu/work/tools/x-tools/aarch64-dumpstack-linux-gnu/bin/../aarch64-dumpstack-linux-gnu/sysroot/usr/lib/aarch64-linux-gnu/libzstd.a when searching for -lzstd
/usr/bin/ld: skipping incompatible /home/hupu/work/tools/x-tools/aarch64-dumpstack-linux-gnu/bin/../aarch64-dumpstack-linux-gnu/sysroot/usr/lib/libc.a when searching for -lc
/usr/bin/ld: /usr/lib/gcc/x86_64-linux-gnu/13/../../../x86_64-linux-gnu/libcrypto.a(libcrypto-lib-dso_dlfcn.o):
in function `dlfcn_globallookup':
(.text+0x1f): warning: Using 'dlopen' in statically linked
applications requires at runtime the shared libraries from the glibc
version used for linking
/usr/bin/ld: /usr/lib/gcc/x86_64-linux-gnu/13/../../../x86_64-linux-gnu/libcrypto.a(libcrypto-lib-bio_addr.o):
in function `BIO_lookup_ex':
(.text+0xdbc): warning: Using 'getaddrinfo' in statically linked
applications requires at runtime the shared libraries from the glibc
version used for linking
/usr/bin/ld: /usr/lib/gcc/x86_64-linux-gnu/13/../../../x86_64-linux-gnu/libcrypto.a(libcrypto-lib-bio_sock.o):
in function `BIO_gethostbyname':
(.text+0x85): warning: Using 'gethostbyname' in statically linked
applications requires at runtime the shared libraries from the glibc
version used for linking

  CLANG   /home/hupu/work/code/explore/output/build-mainline/tools/perf/util/bpf_skel/.tmp/bpf_prog_profiler.bpf.o
  CLANG   /home/hupu/work/code/explore/output/build-mainline/tools/perf/util/bpf_skel/.tmp/bperf_leader.bpf.o
  CLANG   /home/hupu/work/code/explore/output/build-mainline/tools/perf/util/bpf_skel/.tmp/bperf_follower.bpf.o
  CLANG   /home/hupu/work/code/explore/output/build-mainline/tools/perf/util/bpf_skel/.tmp/bperf_cgroup.bpf.o

In file included from <built-in>:2:
In file included from ./util/bpf_skel/perf_version.h:6:
In file included from /home/hupu/work/code/explore/output/build-mainline/tools/perf/util/bpf_skel/.tmp/../vmlinux.h:7:
In file included from /home/hupu/work/code/explore/linux-mainline/tools/include/uapi/linux/perf_event.h:19:
/usr/include/linux/ioctl.h:5:10: fatal error: 'asm/ioctl.h' file not found

    5 | #include <asm/ioctl.h>
      |          ^~~~~~~~~~~~~

In file included from <built-in>:2:
In file included from ./util/bpf_skel/perf_version.h:6:
In file included from /home/hupu/work/code/explore/output/build-mainline/tools/perf/util/bpf_skel/.tmp/../vmlinux.h:7:
In file included from /home/hupu/work/code/explore/linux-mainline/tools/include/uapi/linux/perf_event.h:19:
/usr/include/linux/ioctl.h:5:10: fatal error: 'asm/ioctl.h' file not found

    5 | #include <asm/ioctl.h>
      |          ^~~~~~~~~~~~~

In file included from <built-in>:2:
In file included from ./util/bpf_skel/perf_version.h:6:
In file included from /home/hupu/work/code/explore/output/build-mainline/tools/perf/util/bpf_skel/.tmp/../vmlinux.h:7:
In file included from /home/hupu/work/code/explore/linux-mainline/tools/include/uapi/linux/perf_event.h:19:
/usr/include/linux/ioctl.h:5:10: fatal error: 'asm/ioctl.h' file not found

    5 | #include <asm/ioctl.h>
      |          ^~~~~~~~~~~~~

1 error generated.
In file included from <built-in>:2:
In file included from ./util/bpf_skel/perf_version.h:6:
In file included from /home/hupu/work/code/explore/output/build-mainline/tools/perf/util/bpf_skel/.tmp/../vmlinux.h:7:
In file included from /home/hupu/work/code/explore/linux-mainline/tools/include/uapi/linux/perf_event.h:19:
/usr/include/linux/ioctl.h:5:10: fatal error: 'asm/ioctl.h' file not found

    5 | #include <asm/ioctl.h>
      |          ^~~~~~~~~~~~~

1 error generated.
make[3]: *** [Makefile.perf:1254:
/home/hupu/work/code/explore/output/build-mainline/tools/perf/util/bpf_skel/.tmp/bpf_prog_profiler.bpf.o] Error 1
make[3]: *** Waiting for unfinished jobs....
make[3]: *** [Makefile.perf:1254:
/home/hupu/work/code/explore/output/build-mainline/tools/perf/util/bpf_skel/.tmp/bperf_leader.bpf.o] Error 1
1 error generated.
make[3]: *** [Makefile.perf:1254:
/home/hupu/work/code/explore/output/build-mainline/tools/perf/util/bpf_skel/.tmp/bperf_follower.bpf.o] Error 1
1 error generated.
make[3]: *** [Makefile.perf:1254:
/home/hupu/work/code/explore/output/build-mainline/tools/perf/util/bpf_skel/.tmp/bperf_cgroup.bpf.o] Error 1
make[2]: *** [Makefile.perf:289: sub-make] Error 2
make[1]: *** [Makefile:76: all] Error 2
make: *** [Makefile:93: perf] Error 2

However, when applying the patch “perf build: Support passing extra Clang options via EXTRA_BPF_FLAGS”, running the following command successfully builds perf:

make perf ARCH=arm64 CROSS_COMPILE=aarch64-dumpstack-linux-gnu- EXTRA_BPF_FLAGS="--sysroot=/home/hupu/work/tools/x-tools/aarch64-dumpstack-linux-gnu/bin/../aarch64-dumpstack-linux-gnu/sysroot" LDFLAGS="-L/home/hupu/work/tools/x-tools/aarch64-dumpstack-linux-gnu/bin/../aarch64-dumpstack-linux-gnu/sysroot/usr/lib/aarch64-linux-gnu -L/home/hupu/work/tools/x-tools/aarch64-dumpstack-linux-gnu/bin/../aarch64-dumpstack-linux-gnu/sysroot/usr/lib -static" O=/home/hupu/work/code/explore/output/build-mainline -j8

Similarly, when applying the patch “perf build: Use self-contained headers from kernel source when compiling”, perf can also be successfully built without specifying EXTRA_BPF_FLAGS:

make perf ARCH=arm64 CROSS_COMPILE=aarch64-dumpstack-linux-gnu- LDFLAGS="-L/home/hupu/work/tools/x-tools/aarch64-dumpstack-linux-gnu/bin/../aarch64-dumpstack-linux-gnu/sysroot/usr/lib/aarch64-linux-gnu -L/home/hupu/work/tools/x-tools/aarch64-dumpstack-linux-gnu/bin/../aarch64-dumpstack-linux-gnu/sysroot/usr/lib -static" O=/home/hupu/work/code/explore/output/build-mainline -j8

Therefore, both patches are valid approaches to successfully build perf.

Thanks,
hupu

author
#33 lore raw

Re: [PATCH] perf build: Support passing extra Clang options via EXTRA_BPF_FLAGS

Hi Leo,

Just RESEND this for your attention.

Thanks,
hupu

author
#34 lore raw

Re: [PATCH] perf build: Support passing extra Clang options via EXTRA_BPF_FLAGS

RESEND for your attention.

author
#35 lore raw

Re: [PATCH] perf build: Support passing extra Clang options via EXTRA_BPF_FLAGS

RESEND AGAIN

Quoted 2 lines · hupu

RESEND for your attention.

#36 lore raw

Re: [PATCH] perf build: Support passing extra Clang options via EXTRA_BPF_FLAGS

In reply to hupu

On Wed, Nov 26, 2025 at 09:44:02PM +0800, hupu wrote:

Please don't spam on mailing list as you did. It is really bad practice. You could find resources [1][2] to learn upstreaming and co-work on the ML.

Quoted 7 lines

The eBPF skeleton is compiled via clang --target=bpf, and its header file search paths mainly come from BPF_INCLUDE and TOOLS_UAPI_INCLUDE. It also uses '-idirafter' to include the host’s /usr/local/include and /usr/include directories in the search path. This process is not directly coupled with cross-compilation managed via pkg-config, which means PKG_CONFIG_SYSROOT_DIR does not affect how the skeleton resolves its headers.

Based on my limited knowledge, Clang does not provide its own headers. It needs to rely on GCC's headers for compilation. I do see the Makefile does right thing for finding headers:

Makefile.perf:1203: ***
CLANG_SYS_INCLUDES=-idirafter /usr/lib/llvm-18/lib/clang/18/include
-idirafter /usr/local/include
-idirafter /usr/bin/../lib/gcc-cross/aarch64-linux-gnu/14/../../../../aarch64-linux-gnu/include -idirafter /usr/include/aarch64-linux-gnu
-idirafter /usr/include

It is mess to add some random include paths and feed to clang. We already have provided a reliable way for building eBPF skelton program
- keep in mind, eBPF skeleton program is not any aarch64 cross compilation, we just use clang for building bpf target.

My understanding is you don't have a sane setting up on your local building env.

Thanks,
Leo

[1] https://static.linaro.org/connect/hkg18/presentations/hkg18-tr02.pdf
[2] https://static.linaro.org/connect/hkg18/presentations/hkg18-tr03.pdf

author
#37 lore raw

Re: [PATCH] perf build: Support passing extra Clang options via EXTRA_BPF_FLAGS

Hi Leo,

Quoted 7 lines · Leo Yan

On Wed, Nov 26, 2025 at 09:44:02PM +0800, hupu wrote:

Please don't spam on mailing list as you did. It is really bad practice. You could find resources [1][2] to learn upstreaming and co-work on the ML.

I apologize for the inconvenience caused by my frequent emails. I only hoped to bring this nearly two-month-long discussion to a conclusion sooner.

Quoted 19 lines
Quoted 7 lines

The eBPF skeleton is compiled via clang --target=bpf, and its header file search paths mainly come from BPF_INCLUDE and TOOLS_UAPI_INCLUDE. It also uses '-idirafter' to include the host’s /usr/local/include and /usr/include directories in the search path. This process is not directly coupled with cross-compilation managed via pkg-config, which means PKG_CONFIG_SYSROOT_DIR does not affect how the skeleton resolves its headers.

Based on my limited knowledge, Clang does not provide its own headers. It needs to rely on GCC's headers for compilation. I do see the Makefile does right thing for finding headers:

Makefile.perf:1203: ***
CLANG_SYS_INCLUDES=-idirafter /usr/lib/llvm-18/lib/clang/18/include
-idirafter /usr/local/include
-idirafter /usr/bin/../lib/gcc-cross/aarch64-linux-gnu/14/../../../../aarch64-linux-gnu/include -idirafter /usr/include/aarch64-linux-gnu
-idirafter /usr/include

Quoted 8 lines

It is mess to add some random include paths and feed to clang. We already have provided a reliable way for building eBPF skelton program
- keep in mind, eBPF skeleton program is not any aarch64 cross compilation, we just use clang for building bpf target.

My understanding is you don't have a sane setting up on your local building env.

Although I have explained my view in earlier emails, it seems the discussion has returned to the initial state.

Indeed, as you mentioned in the early discussion, running the following commands on the host to install certain packages can successfully compile perf:

Quoted 3 lines
$ sudo apt-get install gcc-aarch64-linux-gnu g++-aarch64-linux-gnu
$ sudo apt-get install libc6-dev-aarch64-cross linux-libc-dev-aarch64-cross
$ sudo apt-get install libc6-dev-arm64-cross linux-libc-dev-arm64-cross

However, I don’t think relying on the host build environment is the best approach, for several reasons:

a) These commands install UAPI header files on the host, especially linux-libc-dev-aarch64-cross and linux-libc-dev-arm64-cross. These headers originate from the kernel source tree’s include/uapi/ and arch/arm64/include/uapi/ directories, and their versions are tied to the *HOST* kernel version. If the target kernel version is different, mismatches may cause compilation errors or even runtime failures.

b) Even if perf can be compiled and run successfully now, there is no guarantee that the kernel source headers will always match the host-installed UAPI headers as the upstream kernel evolves.

c) In scenarios where the host acts as a general build server and needs to build multiple target kernel versions, it is not possible to ensure that the host UAPI headers are compatible with all target versions.

d) As you pointed out, CLANG_SYS_INCLUDES does include host headers, but it uses -idirafter instead of -I. This means the host headers have lower priority. This change was introduced in commit a2af0f6b8ef7 ("perf build: Add system include paths to BPF builds"); as noted in the commit message, the preferred approach is to use kernel source headers rather than potentially older ones from the system.

Based on this, I propose the following include order:
- Prefer kernel source headers
[RFC] perf build: Use self-contained headers from kernel source when compiling
https://lore.kernel.org/all/[email protected]/

- Allow users to specify header search paths matching the target kernel version (eg. via EXTRA_BPF_FLAGS)
[PATCH] perf build: Support passing extra Clang options via EXTRA_BPF_FLAGS
https://lore.kernel.org/all/[email protected]/

- Fall back to the host build environment only if necessary

In summary, while relying on the host build environment can fix the current build issue, I believe it is not the optimal solution.

Anyway, regardless of the final decision, I will respect it. Before concluding, I hope we can explore this further and involve more maintainers in the discussion. As for your comment, "you don't have a sane setting up on your local building env", I am not yet fully convinced by this reason.

Lastly, I once again apologize for any disturbance caused by my frequent emails, and I sincerely thank both you and Namhyung for your involvement throughout this process. I also hope this discussion will attract more attention so that additional maintainers can participate.

Thanks,
hupu

maintainer
#38 lore raw

Re: [PATCH] perf build: Support passing extra Clang options via EXTRA_BPF_FLAGS

Hello,

Quoted 5 lines · hupu

Hi Leo,

Quoted 3 lines · Leo Yan

On Wed, Nov 26, 2025 at 09:44:02PM +0800, hupu wrote:

Quoted 8 lines
Quoted 4 lines

Please don't spam on mailing list as you did. It is really bad practice. You could find resources [1][2] to learn upstreaming and co-work on the ML.

I apologize for the inconvenience caused by my frequent emails. I only hoped to bring this nearly two-month-long discussion to a conclusion sooner.

Sorry for the delay. But sometimes it takes time to get a consensus in an open discussion. And it's been in a merge window + travelling for LPC 2025. Please don't hurry up.

Quoted 30 lines
Quoted 21 lines
Quoted 9 lines

The eBPF skeleton is compiled via clang --target=bpf, and its header file search paths mainly come from BPF_INCLUDE and TOOLS_UAPI_INCLUDE. It also uses '-idirafter' to include the host’s /usr/local/include and /usr/include directories in the search path. This process is not directly coupled with cross-compilation managed via pkg-config, which means PKG_CONFIG_SYSROOT_DIR does not affect how the skeleton resolves its headers.

Based on my limited knowledge, Clang does not provide its own headers. It needs to rely on GCC's headers for compilation. I do see the Makefile does right thing for finding headers:

Makefile.perf:1203: ***
CLANG_SYS_INCLUDES=-idirafter /usr/lib/llvm-18/lib/clang/18/include
-idirafter /usr/local/include
-idirafter /usr/bin/../lib/gcc-cross/aarch64-linux-gnu/14/../../../../aarch64-linux-gnu/include -idirafter /usr/include/aarch64-linux-gnu
-idirafter /usr/include

Quoted 8 lines

It is mess to add some random include paths and feed to clang. We already have provided a reliable way for building eBPF skelton program
- keep in mind, eBPF skeleton program is not any aarch64 cross compilation, we just use clang for building bpf target.

My understanding is you don't have a sane setting up on your local building env.

Quoted 9 lines

Although I have explained my view in earlier emails, it seems the discussion has returned to the initial state.

Indeed, as you mentioned in the early discussion, running the following commands on the host to install certain packages can successfully compile perf:

Quoted 3 lines
$ sudo apt-get install gcc-aarch64-linux-gnu g++-aarch64-linux-gnu
$ sudo apt-get install libc6-dev-aarch64-cross linux-libc-dev-aarch64-cross
$ sudo apt-get install libc6-dev-arm64-cross linux-libc-dev-arm64-cross

At this point, I'm confused whether we are talking about general cross-build or just BPF skeleton. I agree with Leo that the skeleton build should not require any host specific information rather than vmlinux.h.

Quoted 34 lines

However, I don’t think relying on the host build environment is the best approach, for several reasons:

a) These commands install UAPI header files on the host, especially linux-libc-dev-aarch64-cross and linux-libc-dev-arm64-cross. These headers originate from the kernel source tree’s include/uapi/ and arch/arm64/include/uapi/ directories, and their versions are tied to the *HOST* kernel version. If the target kernel version is different, mismatches may cause compilation errors or even runtime failures.

b) Even if perf can be compiled and run successfully now, there is no guarantee that the kernel source headers will always match the host-installed UAPI headers as the upstream kernel evolves.

c) In scenarios where the host acts as a general build server and needs to build multiple target kernel versions, it is not possible to ensure that the host UAPI headers are compatible with all target versions.

d) As you pointed out, CLANG_SYS_INCLUDES does include host headers, but it uses -idirafter instead of -I. This means the host headers have lower priority. This change was introduced in commit a2af0f6b8ef7 ("perf build: Add system include paths to BPF builds"); as noted in the commit message, the preferred approach is to use kernel source headers rather than potentially older ones from the system.

Based on this, I propose the following include order:
- Prefer kernel source headers
[RFC] perf build: Use self-contained headers from kernel source when compiling
https://lore.kernel.org/all/[email protected]/

Quoted 4 lines

- Allow users to specify header search paths matching the target kernel version (eg. via EXTRA_BPF_FLAGS)
[PATCH] perf build: Support passing extra Clang options via EXTRA_BPF_FLAGS
https://lore.kernel.org/all/[email protected]/

I'm ok with this part - not only to pass header search paths, it can do anything user wants to add to the compiler. You may want to send it out in a separate thread with all comments addressed.

Quoted 13 lines

- Fall back to the host build environment only if necessary

In summary, while relying on the host build environment can fix the current build issue, I believe it is not the optimal solution.

Anyway, regardless of the final decision, I will respect it. Before concluding, I hope we can explore this further and involve more maintainers in the discussion. As for your comment, "you don't have a sane setting up on your local building env", I am not yet fully convinced by this reason.

Quoted 4 lines

Lastly, I once again apologize for any disturbance caused by my frequent emails, and I sincerely thank both you and Namhyung for your involvement throughout this process. I also hope this discussion will attract more attention so that additional maintainers can participate.

Thanks for your understanding,
Namhyung

author
#39 lore raw

Re: [PATCH] perf build: Support passing extra Clang options via EXTRA_BPF_FLAGS

Hi Namhyung,
Thank you very much for your reply.

Quoted 4 lines · Namhyung Kim
Quoted 4 lines

Indeed, as you mentioned in the early discussion, running the following commands on the host to install certain packages can successfully compile perf:

Quoted 9 lines
Quoted 3 lines
Quoted 3 lines
$ sudo apt-get install gcc-aarch64-linux-gnu g++-aarch64-linux-gnu
$ sudo apt-get install libc6-dev-aarch64-cross linux-libc-dev-aarch64-cross
$ sudo apt-get install libc6-dev-arm64-cross linux-libc-dev-arm64-cross

At this point, I'm confused whether we are talking about general cross-build or just BPF skeleton. I agree with Leo that the skeleton build should not require any host specific information rather than vmlinux.h.

I agree with this point. This is also what I was trying to express
earlier: although installing some additional packages on the host can indeed make perf build successfully, in my view this approach is not an ideal solution.

Quoted 34 lines
Quoted 34 lines

However, I don’t think relying on the host build environment is the best approach, for several reasons:

a) These commands install UAPI header files on the host, especially linux-libc-dev-aarch64-cross and linux-libc-dev-arm64-cross. These headers originate from the kernel source tree’s include/uapi/ and arch/arm64/include/uapi/ directories, and their versions are tied to the *HOST* kernel version. If the target kernel version is different, mismatches may cause compilation errors or even runtime failures.

b) Even if perf can be compiled and run successfully now, there is no guarantee that the kernel source headers will always match the host-installed UAPI headers as the upstream kernel evolves.

c) In scenarios where the host acts as a general build server and needs to build multiple target kernel versions, it is not possible to ensure that the host UAPI headers are compatible with all target versions.

d) As you pointed out, CLANG_SYS_INCLUDES does include host headers, but it uses -idirafter instead of -I. This means the host headers have lower priority. This change was introduced in commit a2af0f6b8ef7 ("perf build: Add system include paths to BPF builds"); as noted in the commit message, the preferred approach is to use kernel source headers rather than potentially older ones from the system.

Based on this, I propose the following include order:
- Prefer kernel source headers
[RFC] perf build: Use self-contained headers from kernel source when compiling
https://lore.kernel.org/all/[email protected]/

Quoted 9 lines
Quoted 4 lines

- Allow users to specify header search paths matching the target kernel version (eg. via EXTRA_BPF_FLAGS)
[PATCH] perf build: Support passing extra Clang options via EXTRA_BPF_FLAGS
https://lore.kernel.org/all/[email protected]/

I'm ok with this part - not only to pass header search paths, it can do anything user wants to add to the compiler. You may want to send it out in a separate thread with all comments addressed.

Thank you very much for this suggestion. I understand that you would prefer to continue the discussion in a new, separate email thread, and I am happy to do so.

Before proceeding, I would like to clarify your intention to make sure I understand it correctly and avoid any misunderstanding. Since the discussion above involves two patches with different functionalities, I would like to confirm whether you would prefer me to combine these two patches into a single one, or to discuss and resend both patches together in a new separate email thread.

Thanks again for your time and guidance.
hupu

maintainer
#40 lore raw

Re: [PATCH] perf build: Support passing extra Clang options via EXTRA_BPF_FLAGS

Hello,

Quoted 71 lines · hupu

Hi Namhyung,
Thank you very much for your reply.

Quoted 4 lines · Namhyung Kim
Quoted 4 lines

Indeed, as you mentioned in the early discussion, running the following commands on the host to install certain packages can successfully compile perf:

Quoted 9 lines
Quoted 3 lines
Quoted 3 lines
$ sudo apt-get install gcc-aarch64-linux-gnu g++-aarch64-linux-gnu
$ sudo apt-get install libc6-dev-aarch64-cross linux-libc-dev-aarch64-cross
$ sudo apt-get install libc6-dev-arm64-cross linux-libc-dev-arm64-cross

At this point, I'm confused whether we are talking about general cross-build or just BPF skeleton. I agree with Leo that the skeleton build should not require any host specific information rather than vmlinux.h.

I agree with this point. This is also what I was trying to express
earlier: although installing some additional packages on the host can indeed make perf build successfully, in my view this approach is not an ideal solution.

Quoted 34 lines
Quoted 34 lines

However, I don’t think relying on the host build environment is the best approach, for several reasons:

a) These commands install UAPI header files on the host, especially linux-libc-dev-aarch64-cross and linux-libc-dev-arm64-cross. These headers originate from the kernel source tree’s include/uapi/ and arch/arm64/include/uapi/ directories, and their versions are tied to the *HOST* kernel version. If the target kernel version is different, mismatches may cause compilation errors or even runtime failures.

b) Even if perf can be compiled and run successfully now, there is no guarantee that the kernel source headers will always match the host-installed UAPI headers as the upstream kernel evolves.

c) In scenarios where the host acts as a general build server and needs to build multiple target kernel versions, it is not possible to ensure that the host UAPI headers are compatible with all target versions.

d) As you pointed out, CLANG_SYS_INCLUDES does include host headers, but it uses -idirafter instead of -I. This means the host headers have lower priority. This change was introduced in commit a2af0f6b8ef7 ("perf build: Add system include paths to BPF builds"); as noted in the commit message, the preferred approach is to use kernel source headers rather than potentially older ones from the system.

Based on this, I propose the following include order:
- Prefer kernel source headers
[RFC] perf build: Use self-contained headers from kernel source when compiling
https://lore.kernel.org/all/[email protected]/

Quoted 9 lines
Quoted 4 lines

- Allow users to specify header search paths matching the target kernel version (eg. via EXTRA_BPF_FLAGS)
[PATCH] perf build: Support passing extra Clang options via EXTRA_BPF_FLAGS
https://lore.kernel.org/all/[email protected]/

I'm ok with this part - not only to pass header search paths, it can do anything user wants to add to the compiler. You may want to send it out in a separate thread with all comments addressed.

Thank you very much for this suggestion. I understand that you would prefer to continue the discussion in a new, separate email thread, and I am happy to do so.

Quoted 6 lines

Before proceeding, I would like to clarify your intention to make sure I understand it correctly and avoid any misunderstanding. Since the discussion above involves two patches with different functionalities, I would like to confirm whether you would prefer me to combine these two patches into a single one, or to discuss and resend both patches together in a new separate email thread.

No, I just want EXTRA_BPF_FLAGS part. Preferring the kernel source would need more discussion.

Thanks,
Namhyung

#41 lore raw

Re: [PATCH] perf build: Support passing extra Clang options via EXTRA_BPF_FLAGS

In reply to hupu

Hi hupu,

On Mon, Oct 20, 2025 at 08:51:52PM +0800, hupu wrote:

[...]

Quoted 7 lines
Quoted 3 lines

To be clear, now we are not talking cross build for perf program or any targeting a CPU arch, it is a build failure for eBPF program.

I’d like to clarify the background and scenario once more:
I’m building an SDK that includes a cross-compilation toolchain for the target architecture along with a copy of the kernel source tree.

I am not preventing to use toolchains in you mentioned SDK. I just prefer to give priority the headers provided by the kernel source.

Seems to me, a more reasonable series would be:

- Fix the eBPF program build with using self-contained headers;
- Extend to support external headers.

Thanks,
Leo

author
#42 lore raw

PATCH 1/1 perf build: Support passing extra Clang options via EXTRA_BPF_FLAGS

In reply to hupu

When cross-compiling perf with BPF enabled, Clang is invoked during the build. Some cross-compilation environments require additional compiler options, such as --sysroot or custom include paths.

This patch introduces a new Make variable, EXTRA_BPF_FLAGS. During BPF skeleton builds, it appends user-provided options to CLANG_OPTIONS, allowing extra Clang flags to be set without modifying Makefile.perf directly.

Example usage:
EXTRA_BPF_FLAGS="--sysroot=$SYSROOT"
make perf ARCH="$ARCH" EXTRA_BPF_FLAGS="$EXTRA_BPF_FLAGS"

Change history:
v2:
- Rename EXTRA_CLANG_FLAGS to EXTRA_BPF_FLAGS
- Update commit message
v1:
- Introduce EXTRA_CLANG_FLAGS to allow passing extra Clang options

---
 tools/perf/Makefile.perf | 5 +++++
 1 file changed, 5 insertions(+)
tools/perf/Makefile.perf +5 -0
index 47c906b807ef..f1f2efdbab8c 100644
@@ -1249,6 +1249,11 @@ else
 	$(Q)cp "$(VMLINUX_H)" $@
 endif
 
+# Allow users to specify additional Clang options (e.g. --sysroot)
+# when cross-compiling BPF skeletons, enabling more flexible
+# build configurations.
+CLANG_OPTIONS += $(EXTRA_BPF_FLAGS)
+
 $(SKEL_TMP_OUT)/%.bpf.o: $(OUTPUT)PERF-VERSION-FILE util/bpf_skel/perf_version.h | $(SKEL_TMP_OUT)
 $(SKEL_TMP_OUT)/%.bpf.o: util/bpf_skel/%.bpf.c $(LIBBPF) $(SKEL_OUT)/vmlinux.h
 	$(QUIET_CLANG)$(CLANG) -g -O2 -fno-stack-protector --target=bpf \
Signature · 1 line
2.43.0
author
#43 lore raw
In reply to Ian Rogers

Hi lan,
Thank you very much for your reply.

Quoted 10 lines · Ian Rogers
Quoted 7 lines · hupu

Hi Maintainer,

I encountered some issues while cross-compiling perf (ARCH=arm64), particularly when enabling the BPF option. During the build, Clang fails to compile BPF programs due to missing header files. The relevant error messages are as follows:

Quoted 6 lines

Thanks for reporting this issue! There has been some recent work in this area by Leo:
https://lore.kernel.org/lkml/[email protected]/
The patches are in Linux v6.18-rc1. Perhaps you could try repeating your build with this tree and reporting issues?

My local code is already updated to the latest version, and the current commit is exactly at tag: v6.18-rc1:

commit 3a8660878839faadb4f1a6dd72c3179c1df56787 (tag: v6.18-rc1, origin/master, origin/HEAD)

Author: Linus Torvalds <[email protected]>
Date:   Sun Oct 12 13:42:36 2025 -0700
    Linux 6.18-rc1

However, building with this version still gives the same error as before — when Clang tries to compile the BPF programs, it fails due to a missing asm/ioctl.h header file.

----------------------------------------------------------
INSTALL libsubcmd_headers
INSTALL libsymbol_headers
SYSHDR /home/hupu/work/code/explore/output/build-mainline/tools/perf/libperf/arch/arm64/include/generated/uapi/asm/unistd_64.h INSTALL libapi_headers
INSTALL libperf_headers
CC /home/hupu/work/code/explore/output/build-mainline/tools/perf/libperf/evsel.o
INSTALL libbpf_headers

  CLANG   /home/hupu/work/code/explore/output/build-mainline/tools/perf/util/bpf_skel/.tmp/bpf_prog_profiler.bpf.o
  CLANG   /home/hupu/work/code/explore/output/build-mainline/tools/perf/util/bpf_skel/.tmp/bperf_leader.bpf.o
  CLANG   /home/hupu/work/code/explore/output/build-mainline/tools/perf/util/bpf_skel/.tmp/bperf_follower.bpf.o

In file included from <built-in>:2:
In file included from ./util/bpf_skel/perf_version.h:6:
In file included from /home/hupu/work/code/explore/output/build-mainline/tools/perf/util/bpf_skel/.tmp/../vmlinux.h:7:
In file included from /home/hupu/work/code/explore/linux-mainline/tools/include/uapi/linux/perf_event.h:19:
/usr/include/linux/ioctl.h:5:10: fatal error: 'asm/ioctl.h' file not found

    5 | #include <asm/ioctl.h>
      |          ^~~~~~~~~~~~~

In file included from <built-in>:2:
In file included from ./util/bpf_skel/perf_version.h:6:
In file included from /home/hupu/work/code/explore/output/build-mainline/tools/perf/util/bpf_skel/.tmp/../vmlinux.h:7:
In file included from /home/hupu/work/code/explore/linux-mainline/tools/include/uapi/linux/perf_event.h:19:
/usr/include/linux/ioctl.h:5:10: fatal error: 'asm/ioctl.h' file not found

    5 | #include <asm/ioctl.h>
      |          ^~~~~~~~~~~~~

In file included from <built-in>:2:
In file included from ./util/bpf_skel/perf_version.h:6:
In file included from /home/hupu/work/code/explore/output/build-mainline/tools/perf/util/bpf_skel/.tmp/../vmlinux.h:7:
In file included from /home/hupu/work/code/explore/linux-mainline/tools/include/uapi/linux/perf_event.h:19:
/usr/include/linux/ioctl.h:5:10: fatal error: 'asm/ioctl.h' file not found

    5 | #include <asm/ioctl.h>
      |          ^~~~~~~~~~~~~

11 error generated.
error generated.
make[3]: *** [Makefile.perf:1259:
/home/hupu/work/code/explore/output/build-mainline/tools/perf/util/bpf_skel/.tmp/bperf_leader.bpf.o] Error 1
make[3]: *** Waiting for unfinished jobs....
make[3]: *** [Makefile.perf:1259:
/home/hupu/work/code/explore/output/build-mainline/tools/perf/util/bpf_skel/.tmp/bpf_prog_profiler.bpf.o] Error 1
1 error generated.
make[3]: *** [Makefile.perf:1259:
/home/hupu/work/code/explore/output/build-mainline/tools/perf/util/bpf_skel/.tmp/bperf_follower.bpf.o] Error 1
LD /home/hupu/work/code/explore/output/build-mainline/tools/perf/libperf/libperf-in.o
AR /home/hupu/work/code/explore/output/build-mainline/tools/perf/libperf/libperf.a
make[2]: *** [Makefile.perf:289: sub-make] Error 2
make[1]: *** [Makefile:76: all] Error 2
make: *** [Makefile:93: perf] Error 2
----------------------------------------------------------

However, when I apply the previous PATCH and explicitly specify the sysroot of the cross toolchain via the EXTRA_CLANG_FLAGS setting, the build succeeds:

SYSROOT=$(${CROSS_COMPILE}gcc --print-sysroot) EXTRA_CLANG_FLAGS="--sysroot=$SYSROOT" make perf ARCH=$ARCH CROSS_COMPILE=$CROSS_COMPILE
EXTRA_CLANG_FLAGS="$EXTRA_CLANG_FLAGS"

Please let me know if you need any more details or testing.

Thanks,
hupu