Skip to contentSkip to conversation

Fast path feature detection fixes

PATCH In review Arnaldo Carvalho de Melo 3 participants 7 messages Dec 11, 2024 – Dec 12, 2024

lore
Share & tools
lore
#0 lore raw

cover Fast path feature detection fixes

Hi,

This is on top of a previous patch I sent today for libunwind
opt-in fallout, I'll continue checking if what is being enabled when test-all.c builds is the right set of features to be declared as detected on this fast path method.

- Arnaldo

Arnaldo Carvalho de Melo (2):
tools build feature: Add some comments to explain the FEATURE_TESTS logic tools build feature: Don't set feature-libcap=1 if libcap-devel isn't available

 tools/build/Makefile.feature | 37 ++++++++++++++++++++++++++++++++++--
 tools/build/feature/Makefile |  2 +-
 2 files changed, 36 insertions(+), 3 deletions(-)
Signature · 1 line
2.47.0
#1 lore raw

PATCH 1/2 tools build feature: Add some comments to explain the FEATURE_TESTS logic

From: Arnaldo Carvalho de Melo <[email protected]>

The tools/build/feature/test-all.c works in conjunction with the tools/build/Makefile.feature FEATURE_TESTS_BASIC and FEATURE_TESTS_EXTRA contents, so that if test-all.c manages to be built, we go on and iterate all entries in FEATURE_TESTS_BASIC + FEATURE_TESTS_EXTRA setting then to 1.

To test this:

  $ rm -rf /tmp/b ; mkdir /tmp/b ; make -C tools/perf O=/tmp/b feature-dump
  $ cat /tmp/b/feature/test-all.make.output
  $ ldd /tmp/b/feature/test-all.bin
	linux-vdso.so.1 (0x00007f2a47a67000)
	libdw.so.1 => /lib64/libdw.so.1 (0x00007f2a477cf000)
	libpython3.12.so.1.0 => /lib64/libpython3.12.so.1.0 (0x00007f2a471fe000)
	libm.so.6 => /lib64/libm.so.6 (0x00007f2a4711a000)
	libtraceevent.so.1 => /lib64/libtraceevent.so.1 (0x00007f2a470f2000)
	libtracefs.so.1 => /lib64/libtracefs.so.1 (0x00007f2a470cb000)
	libcrypto.so.3 => /lib64/libcrypto.so.3 (0x00007f2a46c1b000)
	libz.so.1 => /lib64/libz.so.1 (0x00007f2a46bf8000)
	libbabeltrace-ctf.so.1 => /lib64/libbabeltrace-ctf.so.1 (0x00007f2a46bad000)
	libcapstone.so.5 => /lib64/libcapstone.so.5 (0x00007f2a464b8000)
	libopencsd_c_api.so.1 => /lib64/libopencsd_c_api.so.1 (0x00007f2a464a8000)
	libopencsd.so.1 => /lib64/libopencsd.so.1 (0x00007f2a46422000)
	libelf.so.1 => /lib64/libelf.so.1 (0x00007f2a46406000)
	libnuma.so.1 => /lib64/libnuma.so.1 (0x00007f2a463f6000)
	libslang.so.2 => /lib64/libslang.so.2 (0x00007f2a46113000)
	libperl.so.5.38 => /lib64/libperl.so.5.38 (0x00007f2a45d74000)
	libc.so.6 => /lib64/libc.so.6 (0x00007f2a45b83000)
	liblzma.so.5 => /lib64/liblzma.so.5 (0x00007f2a45b50000)
	libzstd.so.1 => /lib64/libzstd.so.1 (0x00007f2a45a91000)
	libbz2.so.1 => /lib64/libbz2.so.1 (0x00007f2a45a7b000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f2a47a69000)
	libbabeltrace.so.1 => /lib64/libbabeltrace.so.1 (0x00007f2a45a6b000)
	libpopt.so.0 => /lib64/libpopt.so.0 (0x00007f2a45a5b000)
	libuuid.so.1 => /lib64/libuuid.so.1 (0x00007f2a45a51000)
	libgmodule-2.0.so.0 => /lib64/libgmodule-2.0.so.0 (0x00007f2a45a4a000)
	libglib-2.0.so.0 => /lib64/libglib-2.0.so.0 (0x00007f2a458fa000)
	libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007f2a45696000)
	libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f2a45668000)
	libcrypt.so.2 => /lib64/libcrypt.so.2 (0x00007f2a45630000)
	libpcre2-8.so.0 => /lib64/libpcre2-8.so.0 (0x00007f2a45590000)
  $ head /tmp/b/FEATURE-DUMP
  feature-backtrace=1
  feature-libdw=1
  feature-eventfd=1
  feature-fortify-source=1
  feature-get_current_dir_name=1
  feature-gettid=1
  feature-glibc=1
  feature-libbfd=1
  feature-libbfd-buildid=1
  feature-libcap=1
  $

There are inconsistencies that are being audited, as can be seen above with the libcap case, that is not linked with test-all.bin nor is present in test-all.c, so shouldn't be set as present. Further patches are going to address those inconsistencies, but lets document this a bit more to reduce the chances of this happening again.

Cc Adrian Hunter <[email protected]>
Cc Ian Rogers <[email protected]>
Cc James Clark <[email protected]>
Cc Jiri Olsa <[email protected]>
Cc Kan Liang <[email protected]>
Cc Namhyung Kim <[email protected]>
---
 tools/build/Makefile.feature | 35 +++++++++++++++++++++++++++++++++++
 1 file changed, 35 insertions(+)
tools/build/Makefile.feature +35 -0
index 80563154318601ac..52600e4d33af8712 100644
@@ -28,6 +28,41 @@ endef
 #   the rule that uses them - an example for that is the 'bionic'
 #   feature check. ]
 #
+# These + the ones in FEATURE_TESTS_EXTRA are included in
+# tools/build/feature/test-all.c and we try to build it all together
+# then setting all those features to '1' meaning they are all enabled.
+#
+# There are things like fortify-source that will be set to 1 because test-all
+# is built with the flags needed to test if its enabled, resulting in
+#
+#   $ rm -rf /tmp/b ; mkdir /tmp/b ; make -C tools/perf O=/tmp/b feature-dump
+#   $ grep fortify-source /tmp/b/FEATURE-DUMP
+#   feature-fortify-source=1
+#   $
+#
+#   All the others should have lines in tools/build/feature/test-all.c like:
+#
+#    #define main main_test_disassembler_init_styled
+#    # include "test-disassembler-init-styled.c"
+#    #undef main
+#
+#    #define main main_test_libzstd
+#    # include "test-libzstd.c"
+#    #undef main
+#
+#    int main(int argc, char *argv[])
+#    {
+#      main_test_disassembler_four_args();
+#      main_test_libzstd();
+#      return 0;
+#    }
+#
+#    If the sample above works, then we end up with these lines in the FEATURE-DUMP
+#    file:
+#
+#    feature-disassembler-four-args=1
+#    feature-libzstd=1
+#
 FEATURE_TESTS_BASIC :=                  \
         backtrace                       \
         libdw                           \
Signature · 1 line
2.47.0
Reviewed-by on v1 1/2
#2 lore raw

Re: [PATCH 1/2] tools build feature: Add some comments to explain the FEATURE_TESTS logic

Quoted 8 lines · Arnaldo Carvalho de Melo

From: Arnaldo Carvalho de Melo <[email protected]>

The tools/build/feature/test-all.c works in conjunction with the tools/build/Makefile.feature FEATURE_TESTS_BASIC and FEATURE_TESTS_EXTRA contents, so that if test-all.c manages to be built, we go on and iterate all entries in FEATURE_TESTS_BASIC + FEATURE_TESTS_EXTRA setting then to 1.

nit: s/then/them/

Quoted 54 lines

To test this:

  $ rm -rf /tmp/b ; mkdir /tmp/b ; make -C tools/perf O=/tmp/b feature-dump
  $ cat /tmp/b/feature/test-all.make.output
  $ ldd /tmp/b/feature/test-all.bin
        linux-vdso.so.1 (0x00007f2a47a67000)
        libdw.so.1 => /lib64/libdw.so.1 (0x00007f2a477cf000)
        libpython3.12.so.1.0 => /lib64/libpython3.12.so.1.0 (0x00007f2a471fe000)
        libm.so.6 => /lib64/libm.so.6 (0x00007f2a4711a000)
        libtraceevent.so.1 => /lib64/libtraceevent.so.1 (0x00007f2a470f2000)
        libtracefs.so.1 => /lib64/libtracefs.so.1 (0x00007f2a470cb000)
        libcrypto.so.3 => /lib64/libcrypto.so.3 (0x00007f2a46c1b000)
        libz.so.1 => /lib64/libz.so.1 (0x00007f2a46bf8000)
        libbabeltrace-ctf.so.1 => /lib64/libbabeltrace-ctf.so.1 (0x00007f2a46bad000)
        libcapstone.so.5 => /lib64/libcapstone.so.5 (0x00007f2a464b8000)
        libopencsd_c_api.so.1 => /lib64/libopencsd_c_api.so.1 (0x00007f2a464a8000)
        libopencsd.so.1 => /lib64/libopencsd.so.1 (0x00007f2a46422000)
        libelf.so.1 => /lib64/libelf.so.1 (0x00007f2a46406000)
        libnuma.so.1 => /lib64/libnuma.so.1 (0x00007f2a463f6000)
        libslang.so.2 => /lib64/libslang.so.2 (0x00007f2a46113000)
        libperl.so.5.38 => /lib64/libperl.so.5.38 (0x00007f2a45d74000)
        libc.so.6 => /lib64/libc.so.6 (0x00007f2a45b83000)
        liblzma.so.5 => /lib64/liblzma.so.5 (0x00007f2a45b50000)
        libzstd.so.1 => /lib64/libzstd.so.1 (0x00007f2a45a91000)
        libbz2.so.1 => /lib64/libbz2.so.1 (0x00007f2a45a7b000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f2a47a69000)
        libbabeltrace.so.1 => /lib64/libbabeltrace.so.1 (0x00007f2a45a6b000)
        libpopt.so.0 => /lib64/libpopt.so.0 (0x00007f2a45a5b000)
        libuuid.so.1 => /lib64/libuuid.so.1 (0x00007f2a45a51000)
        libgmodule-2.0.so.0 => /lib64/libgmodule-2.0.so.0 (0x00007f2a45a4a000)
        libglib-2.0.so.0 => /lib64/libglib-2.0.so.0 (0x00007f2a458fa000)
        libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007f2a45696000)
        libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f2a45668000)
        libcrypt.so.2 => /lib64/libcrypt.so.2 (0x00007f2a45630000)
        libpcre2-8.so.0 => /lib64/libpcre2-8.so.0 (0x00007f2a45590000)
  $ head /tmp/b/FEATURE-DUMP
  feature-backtrace=1
  feature-libdw=1
  feature-eventfd=1
  feature-fortify-source=1
  feature-get_current_dir_name=1
  feature-gettid=1
  feature-glibc=1
  feature-libbfd=1
  feature-libbfd-buildid=1
  feature-libcap=1
  $

There are inconsistencies that are being audited, as can be seen above with the libcap case, that is not linked with test-all.bin nor is present in test-all.c, so shouldn't be set as present. Further patches are going to address those inconsistencies, but lets document this a bit more to reduce the chances of this happening again.

Quoted 7 lines
Cc Adrian Hunter <[email protected]>
Cc Ian Rogers <[email protected]>
Cc James Clark <[email protected]>
Cc Jiri Olsa <[email protected]>
Cc Kan Liang <[email protected]>
Cc Namhyung Kim <[email protected]>

Thanks,
Ian

Quoted 53 lines
---
 tools/build/Makefile.feature | 35 +++++++++++++++++++++++++++++++++++
 1 file changed, 35 insertions(+)
tools/build/Makefile.feature +35 -1
index 80563154318601ac..52600e4d33af8712 100644
@@ -28,6 +28,41 @@ endef
 #   the rule that uses them - an example for that is the 'bionic'
 #   feature check. ]
 #
+# These + the ones in FEATURE_TESTS_EXTRA are included in
+# tools/build/feature/test-all.c and we try to build it all together
+# then setting all those features to '1' meaning they are all enabled.
+#
+# There are things like fortify-source that will be set to 1 because test-all
+# is built with the flags needed to test if its enabled, resulting in
+#
+#   $ rm -rf /tmp/b ; mkdir /tmp/b ; make -C tools/perf O=/tmp/b feature-dump
+#   $ grep fortify-source /tmp/b/FEATURE-DUMP
+#   feature-fortify-source=1
+#   $
+#
+#   All the others should have lines in tools/build/feature/test-all.c like:
+#
+#    #define main main_test_disassembler_init_styled
+#    # include "test-disassembler-init-styled.c"
+#    #undef main
+#
+#    #define main main_test_libzstd
+#    # include "test-libzstd.c"
+#    #undef main
+#
+#    int main(int argc, char *argv[])
+#    {
+#      main_test_disassembler_four_args();
+#      main_test_libzstd();
+#      return 0;
+#    }
+#
+#    If the sample above works, then we end up with these lines in the FEATURE-DUMP
+#    file:
+#
+#    feature-disassembler-four-args=1
+#    feature-libzstd=1
+#
 FEATURE_TESTS_BASIC :=                  \
         backtrace                       \
         libdw                           \
--

2.47.0

#3 lore raw

PATCH 2/2 tools build feature: Don't set feature-libcap=1 if libcap-devel isn't available

In reply to Arnaldo Carvalho de Melo

From: Arnaldo Carvalho de Melo <[email protected]>

libcap isn't tested in the tools/build/feature/test-all.c fast path feature detection process, so don't set it as available if test-all manages to build.

There are other users of this feature detection mechanism, and they explicitely ask for libcap to be tested, so are not affected by this patch, for instance, with this patch in place:

  $ make -C tools/bpf/bpftool/ clean
  <SNIP>
  make: Leaving directory '/home/acme/git/perf-tools-next/tools/bpf/bpftool'
  ⬢ [acme@toolbox perf-tools-next]$ make -C tools/bpf/bpftool/
  make: Entering directory '/home/acme/git/perf-tools-next/tools/bpf/bpftool'

Auto-detecting system features:

  ...                         clang-bpf-co-re: [ on  ]
  ...                                    llvm: [ on  ]
  ...                                  libcap: [ on  ]
  ...                                  libbfd: [ on  ]
  ...                             libelf-zstd: [ on  ]
  <SNIP>
    LINK    bpftool
  make: Leaving directory '/home/acme/git/perf-tools-next/tools/bpf/bpftool'
  $
  $ sudo rpm -e libcap-devel
  $ make -C tools/bpf/bpftool/
  <SNIP>
  make: Entering directory '/home/acme/git/perf-tools-next/tools/bpf/bpftool'

Auto-detecting system features:
... clang-bpf-co-re: [ on ]
... llvm: [ on ]
... libcap: [ OFF ]
... libbfd: [ on ]
... libelf-zstd: [ on ]

$

Cc Adrian Hunter <[email protected]>
Cc Ian Rogers <[email protected]>
Cc James Clark <[email protected]>
Cc Jiri Olsa <[email protected]>
Cc Kan Liang <[email protected]>
Cc Namhyung Kim <[email protected]>
Cc Quentin Monnet <[email protected]>
---
 tools/build/Makefile.feature | 2 --
 tools/build/feature/Makefile | 2 +-
 2 files changed, 1 insertion(+), 3 deletions(-)
tools/build/Makefile.feature +0 -2
index 52600e4d33af8712..e25cdb7db40e3f3d 100644
@@ -73,7 +73,6 @@ FEATURE_TESTS_BASIC :=                  \
         glibc                           \
         libbfd                          \
         libbfd-buildid			\
-        libcap                          \
         libelf                          \
         libelf-getphdrnum               \
         libelf-gelf_getnote             \
@@ -150,7 +149,6 @@ FEATURE_DISPLAY ?=              \
          glibc                  \
          libbfd                 \
          libbfd-buildid		\
-         libcap                 \
          libelf                 \
          libnuma                \
          numa_num_possible_cpus \
tools/build/feature/Makefile +1 -1
index 043dfd00fce72d8f..cbf751b6f0f76225 100644
@@ -110,7 +110,7 @@ all: $(FILES)
 __BUILD = $(CC) $(CFLAGS) -MD -Wall -Werror -o $@ $(patsubst %.bin,%.c,$(@F)) $(LDFLAGS)
   BUILD = $(__BUILD) > $(@:.bin=.make.output) 2>&1
   BUILD_BFD = $(BUILD) -DPACKAGE='"perf"' -lbfd -ldl
-  BUILD_ALL = $(BUILD) -fstack-protector-all -O2 -D_FORTIFY_SOURCE=2 -ldw -lelf -lnuma -lelf -lslang $(FLAGS_PERL_EMBED) $(FLAGS_PYTHON_EMBED) -DPACKAGE='"perf"' -lbfd -ldl -lz -llzma -lzstd -lcap
+  BUILD_ALL = $(BUILD) -fstack-protector-all -O2 -D_FORTIFY_SOURCE=2 -ldw -lelf -lnuma -lelf -lslang $(FLAGS_PERL_EMBED) $(FLAGS_PYTHON_EMBED) -DPACKAGE='"perf"' -lbfd -ldl -lz -llzma -lzstd
 
 __BUILDXX = $(CXX) $(CXXFLAGS) -MD -Wall -Werror -o $@ $(patsubst %.bin,%.cpp,$(@F)) $(LDFLAGS)
   BUILDXX = $(__BUILDXX) > $(@:.bin=.make.output) 2>&1
Signature · 1 line
2.47.0
Reviewed-by on v1 2/2
#4 lore raw

Re: [PATCH 2/2] tools build feature: Don't set feature-libcap=1 if libcap-devel isn't available

Quoted 41 lines · Arnaldo Carvalho de Melo

From: Arnaldo Carvalho de Melo <[email protected]>

libcap isn't tested in the tools/build/feature/test-all.c fast path feature detection process, so don't set it as available if test-all manages to build.

There are other users of this feature detection mechanism, and they explicitely ask for libcap to be tested, so are not affected by this patch, for instance, with this patch in place:

  $ make -C tools/bpf/bpftool/ clean
  <SNIP>
  make: Leaving directory '/home/acme/git/perf-tools-next/tools/bpf/bpftool'
  ⬢ [acme@toolbox perf-tools-next]$ make -C tools/bpf/bpftool/
  make: Entering directory '/home/acme/git/perf-tools-next/tools/bpf/bpftool'

Auto-detecting system features:

  ...                         clang-bpf-co-re: [ on  ]
  ...                                    llvm: [ on  ]
  ...                                  libcap: [ on  ]
  ...                                  libbfd: [ on  ]
  ...                             libelf-zstd: [ on  ]
  <SNIP>
    LINK    bpftool
  make: Leaving directory '/home/acme/git/perf-tools-next/tools/bpf/bpftool'
  $
  $ sudo rpm -e libcap-devel
  $ make -C tools/bpf/bpftool/
  <SNIP>
  make: Entering directory '/home/acme/git/perf-tools-next/tools/bpf/bpftool'

Auto-detecting system features:
... clang-bpf-co-re: [ on ]
... llvm: [ on ]
... libcap: [ OFF ]
... libbfd: [ on ]
... libelf-zstd: [ on ]

$

Quoted 8 lines
Cc Adrian Hunter <[email protected]>
Cc Ian Rogers <[email protected]>
Cc James Clark <[email protected]>
Cc Jiri Olsa <[email protected]>
Cc Kan Liang <[email protected]>
Cc Namhyung Kim <[email protected]>
Cc Quentin Monnet <[email protected]>

Thanks,
Ian

#5 lore raw

Re: [PATCH 2/2] tools build feature: Don't set feature-libcap=1 if libcap-devel isn't available

2024-12-11 20:25 UTC-0800 ~ Ian Rogers <[email protected]>

Quoted 52 lines
Quoted 41 lines · Arnaldo Carvalho de Melo

From: Arnaldo Carvalho de Melo <[email protected]>

libcap isn't tested in the tools/build/feature/test-all.c fast path feature detection process, so don't set it as available if test-all manages to build.

There are other users of this feature detection mechanism, and they explicitely ask for libcap to be tested, so are not affected by this patch, for instance, with this patch in place:

  $ make -C tools/bpf/bpftool/ clean
  <SNIP>
  make: Leaving directory '/home/acme/git/perf-tools-next/tools/bpf/bpftool'
  ⬢ [acme@toolbox perf-tools-next]$ make -C tools/bpf/bpftool/
  make: Entering directory '/home/acme/git/perf-tools-next/tools/bpf/bpftool'

Auto-detecting system features:

  ...                         clang-bpf-co-re: [ on  ]
  ...                                    llvm: [ on  ]
  ...                                  libcap: [ on  ]
  ...                                  libbfd: [ on  ]
  ...                             libelf-zstd: [ on  ]
  <SNIP>
    LINK    bpftool
  make: Leaving directory '/home/acme/git/perf-tools-next/tools/bpf/bpftool'
  $
  $ sudo rpm -e libcap-devel
  $ make -C tools/bpf/bpftool/
  <SNIP>
  make: Entering directory '/home/acme/git/perf-tools-next/tools/bpf/bpftool'

Auto-detecting system features:
... clang-bpf-co-re: [ on ]
... llvm: [ on ]
... libcap: [ OFF ]
... libbfd: [ on ]
... libelf-zstd: [ on ]

$

Quoted 8 lines
Cc Adrian Hunter <[email protected]>
Cc Ian Rogers <[email protected]>
Cc James Clark <[email protected]>
Cc Jiri Olsa <[email protected]>
Cc Kan Liang <[email protected]>
Cc Namhyung Kim <[email protected]>
Cc Quentin Monnet <[email protected]>
Quoted 1 line

Thanks Arnaldo for checking the bpftool build :)
Quentin

#6 lore raw

Re: [PATCH 2/2] tools build feature: Don't set feature-libcap=1 if libcap-devel isn't available

Quoted 53 lines · Quentin Monnet

2024-12-11 20:25 UTC-0800 ~ Ian Rogers <[email protected]>

Quoted 52 lines
Quoted 49 lines · Arnaldo Carvalho de Melo

From: Arnaldo Carvalho de Melo <[email protected]>

libcap isn't tested in the tools/build/feature/test-all.c fast path feature detection process, so don't set it as available if test-all manages to build.

There are other users of this feature detection mechanism, and they explicitely ask for libcap to be tested, so are not affected by this patch, for instance, with this patch in place:

  $ make -C tools/bpf/bpftool/ clean
  <SNIP>
  make: Leaving directory '/home/acme/git/perf-tools-next/tools/bpf/bpftool'
  ⬢ [acme@toolbox perf-tools-next]$ make -C tools/bpf/bpftool/
  make: Entering directory '/home/acme/git/perf-tools-next/tools/bpf/bpftool'

Auto-detecting system features:

  ...                         clang-bpf-co-re: [ on  ]
  ...                                    llvm: [ on  ]
  ...                                  libcap: [ on  ]
  ...                                  libbfd: [ on  ]
  ...                             libelf-zstd: [ on  ]
  <SNIP>
    LINK    bpftool
  make: Leaving directory '/home/acme/git/perf-tools-next/tools/bpf/bpftool'
  $
  $ sudo rpm -e libcap-devel
  $ make -C tools/bpf/bpftool/
  <SNIP>
  make: Entering directory '/home/acme/git/perf-tools-next/tools/bpf/bpftool'

Auto-detecting system features:
... clang-bpf-co-re: [ on ]
... llvm: [ on ]
... libcap: [ OFF ]
... libbfd: [ on ]
... libelf-zstd: [ on ]

$

Cc Adrian Hunter <[email protected]>
Cc Ian Rogers <[email protected]>
Cc James Clark <[email protected]>
Cc Jiri Olsa <[email protected]>
Cc Kan Liang <[email protected]>
Cc Namhyung Kim <[email protected]>
Cc Quentin Monnet <[email protected]>
Quoted 6 lines
Quoted 1 line

Thanks Arnaldo for checking the bpftool build :)

Thanks! Added to the cset.

- Arnaldo