Skip to contentSkip to conversation

support/scripts/check-host-libs: add new check on host binaries/libs

PATCH In review Thomas Petazzoni via buildroot 1 participant 4 messages Aug 24 – Aug 25 Kbuild

lore
Share & tools
lore
#0 lore raw

PATCH v2 1/1 support/scripts/check-host-libs: add new check on host binaries/libs

One frequent issue in Buildroot is that when building host libraries or applications, the build system of the package detects some libraries provided by the system, and happily links to them, without Buildroot knowing. Sometimes this doesn't cause any problem, but sometimes this causes issues, and we're regularly eliminating such mis-detection by forcing those packages to not detect the system libraries that have not been built by Buildroot.

The new script check-host-libs added in this commit, which is executed during the host-finalize step at the end of the build is an attempt at detecting at least some of these situations.

What it does is that at the end of the build, it verifies that all binaries and libraries in $(HOST_DIR) only have shared library dependencies on libraries that are in Buildroot $(HOST_DIR), to the exception of the C library, for which we of course use the system C library.

For example, if the binary output/host/bin/plop is linked against libpng, but libpng was not built and installed by Buildroot, the build will now fail with:

ERROR: in /home/thomas/projets/buildroot/output/host/bin/plop, libpng16.so.16 unknown
make: *** [Makefile:715: host-finalize] Error 1

The script includes an allowlist of libraries provided by the C library. It is potentially possible that this list might need to be extended to cover all systems/distributions/C libraries, but only wider testing of this script will help detect such cases.

It is worth mentioning that for now this script is executed only once at the end of the build. This means that if a package A gets built, detects and uses a system library libfoo and uses it, and then by chance later Buildroot package B builds and installs libfoo into HOST_DIR/lib, this script will believe that package A is correct, as it finds libfoo in HOST_DIR/lib, even though while package A was being built, the libfoo being detected was the system one. Detecting this would require running check-host-libs at the end of each package build, but that would imply re-checking over and over again all host binaries/libraries, which could have a noticeable impact on the build time. So for now, we simply check at the end of the build, which should already help to detect a lot of interesting bogus situations.

---
It would be very useful if a few people could apply this patch to their local tree, run their usual build, and see how it behaves. This way, I can get some feedback to address the most obvious issues before it gets merged and starts causing build failures in the autobuilders.

The v1 of this patch was sent on Sep 20, 2022, almost 4 years ago :-)

Changes since v1:
- Replaced the per-file file --mime-type checks with direct ELF magic detection using Bash read -N 4, significantly reducing scan time.
- Switched file traversal to NUL-delimited find -print0 output, safely handling paths containing whitespace.
- Suppressed harmless readelf errors for valid ELF object files without a dynamic section, such as the Go test fixtures mentioned during review.
- Added librt.so*, libutil.so*, and libresolv.so* to the system-library allowlist.
- Quoted file and host-directory paths where appropriate.
- Fixed shellcheck issues

---
 Makefile                        |  1 +
 support/scripts/check-host-libs | 37 +++++++++++++++++++++++++++++++++
 2 files changed, 38 insertions(+)
 create mode 100755 support/scripts/check-host-libs
Makefile +1 -0
index 4e59c7b87f..890575f462 100644
@@ -748,6 +748,7 @@ host-finalize: $(PACKAGES) $(HOST_DIR) $(HOST_DIR_SYMLINK)
 		PER_PACKAGE_DIR=$(PER_PACKAGE_DIR) \
 		$(TOPDIR)/support/scripts/fix-rpath staging
 	$(call ppd-fixup-paths,$(BASE_DIR))
+	./support/scripts/check-host-libs $(HOST_DIR)
 
 .PHONY: staging-finalize
 staging-finalize: $(STAGING_DIR_SYMLINK)
support/scripts/check-host-libs new file +37 -0
new file mode 100755
index 0000000000..79f60024c4
@@ -0,0 +1,37 @@
+#!/bin/bash
+
+HOST_DIR=$1
+
+if test -z "${HOST_DIR}" ; then
+    echo "usage: check-host-libs HOST_DIR"
+    exit 1
+fi
+
+bailout="no"
+ELF=$'\x7fELF'
+
+while IFS= read -r -d '' f; do
+    read -r -N 4 magic < "${f}"
+    if test "${magic}" != "${ELF}" ; then
+        continue
+    fi
+    for lib in $(LC_ALL=C readelf -d "${f}" 2>/dev/null | \
+        sed -n 's,.*Shared library: \[\(.*\)\].*,\1,p'); do
+        case ${lib} in
+        libc.so*|libm.so*|libstdc++.so*|libpthread.so*|libgcc_s.so*|libdl.so*|ld-*|libgomp.so*|libcrypt.so*|libcrypto.so*|libatomic.so*|librt.so*|libutil.so*|libresolv.so*)
+            continue
+            ;;
+        *)
+            if test -e "${HOST_DIR}/lib/${lib}" ; then
+                continue
+            fi
+            echo "ERROR: in ${f}, ${lib} unknown"
+            bailout="yes"
+            ;;
+        esac
+    done
+done < <(find "${HOST_DIR}"/*bin "${HOST_DIR}"/lib* -type f -print0)
+
+if test "${bailout}" = "yes" ; then
+    exit 1
+fi
Signature · 6 lines
2.55.0

_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
#1 lore raw

Hi Thomas,

Quoted 43 lines · Thomas Petazzoni via buildroot

One frequent issue in Buildroot is that when building host libraries or applications, the build system of the package detects some libraries provided by the system, and happily links to them, without Buildroot knowing. Sometimes this doesn't cause any problem, but sometimes this causes issues, and we're regularly eliminating such mis-detection by forcing those packages to not detect the system libraries that have not been built by Buildroot.

The new script check-host-libs added in this commit, which is executed during the host-finalize step at the end of the build is an attempt at detecting at least some of these situations.

What it does is that at the end of the build, it verifies that all binaries and libraries in $(HOST_DIR) only have shared library dependencies on libraries that are in Buildroot $(HOST_DIR), to the exception of the C library, for which we of course use the system C library.

For example, if the binary output/host/bin/plop is linked against libpng, but libpng was not built and installed by Buildroot, the build will now fail with:

ERROR: in /home/thomas/projets/buildroot/output/host/bin/plop, libpng16.so.16 unknown
make: *** [Makefile:715: host-finalize] Error 1

The script includes an allowlist of libraries provided by the C library. It is potentially possible that this list might need to be extended to cover all systems/distributions/C libraries, but only wider testing of this script will help detect such cases.

It is worth mentioning that for now this script is executed only once at the end of the build. This means that if a package A gets built, detects and uses a system library libfoo and uses it, and then by chance later Buildroot package B builds and installs libfoo into HOST_DIR/lib, this script will believe that package A is correct, as it finds libfoo in HOST_DIR/lib, even though while package A was being built, the libfoo being detected was the system one. Detecting this would require running check-host-libs at the end of each package build, but that would imply re-checking over and over again all host binaries/libraries, which could have a noticeable impact on the build time. So for now, we simply check at the end of the build, which should already help to detect a lot of interesting bogus situations.

Quoted 6 lines

---
It would be very useful if a few people could apply this patch to their local tree, run their usual build, and see how it behaves. This way, I can get some feedback to address the most obvious issues before it gets merged and starts causing build failures in the autobuilders.

I gave a try to your checker on some personal project, and I get plenty of warnings:

Quoted 1 line
Quoted 1 line
Quoted 1 line

Finalizing host directory

./support/scripts/check-host-libs /home/alexis/src/neon-beat/neon-beat-controller/output/host
ERROR: in /home/alexis/src/neon-beat/neon-beat-controller/output/host/bin/systemd-cgtop, libsystemd-shared-256.so unknown
ERROR: in /home/alexis/src/neon-beat/neon-beat-controller/output/host/bin/systemd-creds, libsystemd-shared-256.so unknown
ERROR: in /home/alexis/src/neon-beat/neon-beat-controller/output/host/bin/systemd-cgls, libsystemd-shared-256.so unknown
ERROR: in /home/alexis/src/neon-beat/neon-beat-controller/output/host/bin/systemd-escape, libsystemd-shared-256.so unknown
ERROR: in /home/alexis/src/neon-beat/neon-beat-controller/output/host/bin/systemd-stdio-bridge, libsystemd-shared-256.so unknown
ERROR: in /home/alexis/src/neon-beat/neon-beat-controller/output/host/bin/systemd-tty-ask-password-agent, libsystemd-shared-256.so unknown
ERROR: in /home/alexis/src/neon-beat/neon-beat-controller/output/host/bin/cpack, libidn2.so.0 unknown
ERROR: in /home/alexis/src/neon-beat/neon-beat-controller/output/host/bin/cpack, libmd.so.0 unknown
ERROR: in /home/alexis/src/neon-beat/neon-beat-controller/output/host/bin/systemd-cat, libsystemd-shared-256.so unknown
ERROR: in /home/alexis/src/neon-beat/neon-beat-controller/output/host/bin/systemd-detect-virt, libsystemd-shared-256.so unknown
ERROR: in /home/alexis/src/neon-beat/neon-beat-controller/output/host/bin/systemd-hwdb, libsystemd-shared-256.so unknown
ERROR: in /home/alexis/src/neon-beat/neon-beat-controller/output/host/bin/systemd-mount, libsystemd-shared-256.so unknown
ERROR: in /home/alexis/src/neon-beat/neon-beat-controller/output/host/bin/systemd-vpick, libsystemd-shared-256.so unknown
ERROR: in /home/alexis/src/neon-beat/neon-beat-controller/output/host/bin/systemd-run, libsystemd-shared-256.so unknown
ERROR: in /home/alexis/src/neon-beat/neon-beat-controller/output/host/bin/varlinkctl, libsystemd-shared-256.so unknown
ERROR: in /home/alexis/src/neon-beat/neon-beat-controller/output/host/bin/systemd-socket-activate, libsystemd-shared-256.so unknown
ERROR: in /home/alexis/src/neon-beat/neon-beat-controller/output/host/bin/floppyd, libX11.so.6 unknown
ERROR: in /home/alexis/src/neon-beat/neon-beat-controller/output/host/bin/systemd-ac-power, libsystemd-shared-256.so unknown
ERROR: in /home/alexis/src/neon-beat/neon-beat-controller/output/host/bin/systemd-path, libsystemd-shared-256.so unknown
ERROR: in /home/alexis/src/neon-beat/neon-beat-controller/output/host/bin/systemctl, libsystemd-shared-256.so unknown
ERROR: in /home/alexis/src/neon-beat/neon-beat-controller/output/host/bin/cmake, libidn2.so.0 unknown
ERROR: in /home/alexis/src/neon-beat/neon-beat-controller/output/host/bin/cmake, libmd.so.0 unknown
ERROR: in /home/alexis/src/neon-beat/neon-beat-controller/output/host/bin/systemd-delta, libsystemd-shared-256.so unknown
ERROR: in /home/alexis/src/neon-beat/neon-beat-controller/output/host/bin/busctl, libsystemd-shared-256.so unknown
ERROR: in /home/alexis/src/neon-beat/neon-beat-controller/output/host/bin/systemd-id128, libsystemd-shared-256.so unknown
ERROR: in /home/alexis/src/neon-beat/neon-beat-controller/output/host/bin/udevadm, libsystemd-shared-256.so unknown
ERROR: in /home/alexis/src/neon-beat/neon-beat-controller/output/host/bin/systemd-notify, libsystemd-shared-256.so unknown
ERROR: in /home/alexis/src/neon-beat/neon-beat-controller/output/host/bin/systemd-machine-id-setup, libsystemd-shared-256.so unknown
ERROR: in /home/alexis/src/neon-beat/neon-beat-controller/output/host/bin/systemd-ask-password, libsystemd-shared-256.so unknown
ERROR: in /home/alexis/src/neon-beat/neon-beat-controller/output/host/bin/systemd-nspawn, libsystemd-shared-256.so unknown
ERROR: in /home/alexis/src/neon-beat/neon-beat-controller/output/host/bin/journalctl, libsystemd-shared-256.so unknown
ERROR: in /home/alexis/src/neon-beat/neon-beat-controller/output/host/bin/ctest, libidn2.so.0 unknown
ERROR: in /home/alexis/src/neon-beat/neon-beat-controller/output/host/bin/ctest, libmd.so.0 unknown
ERROR: in /home/alexis/src/neon-beat/neon-beat-controller/output/host/bin/systemd-tmpfiles, libsystemd-shared-256.so unknown
ERROR: in /home/alexis/src/neon-beat/neon-beat-controller/output/host/sbin/sfdisk, libreadline.so.8 unknown
ERROR: in /home/alexis/src/neon-beat/neon-beat-controller/output/host/sbin/fdisk, libreadline.so.8 unknown
ERROR: in /home/alexis/src/neon-beat/neon-beat-controller/output/host/lib/udev/iocost, libsystemd-shared-256.so unknown
ERROR: in /home/alexis/src/neon-beat/neon-beat-controller/output/host/lib/udev/mtd_probe, libsystemd-shared-256.so unknown
ERROR: in /home/alexis/src/neon-beat/neon-beat-controller/output/host/lib/udev/dmi_memory_id, libsystemd-shared-256.so unknown
ERROR: in /home/alexis/src/neon-beat/neon-beat-controller/output/host/lib/udev/v4l_id, libsystemd-shared-256.so unknown
ERROR: in /home/alexis/src/neon-beat/neon-beat-controller/output/host/lib/udev/cdrom_id, libsystemd-shared-256.so unknown
ERROR: in /home/alexis/src/neon-beat/neon-beat-controller/output/host/lib/udev/ata_id, libsystemd-shared-256.so unknown
ERROR: in /home/alexis/src/neon-beat/neon-beat-controller/output/host/lib/udev/fido_id, libsystemd-shared-256.so unknown
ERROR: in /home/alexis/src/neon-beat/neon-beat-controller/output/host/lib/udev/scsi_id, libsystemd-shared-256.so unknown
ERROR: in /home/alexis/src/neon-beat/neon-beat-controller/output/host/lib/systemd/system-generators/systemd-run-generator, libsystemd-shared-256.so unknown
ERROR: in /home/alexis/src/neon-beat/neon-beat-controller/output/host/lib/systemd/system-generators/systemd-debug-generator, libsystemd-shared-256.so unknown
ERROR: in /home/alexis/src/neon-beat/neon-beat-controller/output/host/lib/systemd/system-generators/systemd-ssh-generator, libsystemd-shared-256.so unknown
ERROR: in /home/alexis/src/neon-beat/neon-beat-controller/output/host/lib/systemd/system-generators/systemd-getty-generator, libsystemd-shared-256.so unknown
ERROR: in /home/alexis/src/neon-beat/neon-beat-controller/output/host/lib/systemd/system-generators/systemd-fstab-generator, libsystemd-shared-256.so unknown
ERROR: in /home/alexis/src/neon-beat/neon-beat-controller/output/host/lib/systemd/system-generators/systemd-system-update-generator, libsystemd-shared-256.so unknown
ERROR: in /home/alexis/src/neon-beat/neon-beat-controller/output/host/lib/systemd/system-generators/systemd-tpm2-generator, libsystemd-shared-256.so unknown
ERROR: in /home/alexis/src/neon-beat/neon-beat-controller/output/host/lib/systemd/systemd-network-generator, libsystemd-shared-256.so unknown
ERROR: in /home/alexis/src/neon-beat/neon-beat-controller/output/host/lib/systemd/systemd-socket-proxyd, libsystemd-shared-256.so unknown
ERROR: in /home/alexis/src/neon-beat/neon-beat-controller/output/host/lib/systemd/systemd-ssh-proxy, libsystemd-shared-256.so unknown
ERROR: in /home/alexis/src/neon-beat/neon-beat-controller/output/host/lib/systemd/systemd-sleep, libsystemd-shared-256.so unknown
ERROR: in /home/alexis/src/neon-beat/neon-beat-controller/output/host/lib/systemd/systemd-executor, libsystemd-core-256.so unknown
ERROR: in /home/alexis/src/neon-beat/neon-beat-controller/output/host/lib/systemd/systemd-executor, libsystemd-shared-256.so unknown
ERROR: in /home/alexis/src/neon-beat/neon-beat-controller/output/host/lib/systemd/systemd-reply-password, libsystemd-shared-256.so unknown
ERROR: in /home/alexis/src/neon-beat/neon-beat-controller/output/host/lib/systemd/systemd-cgroups-agent, libsystemd-shared-256.so unknown
ERROR: in /home/alexis/src/neon-beat/neon-beat-controller/output/host/lib/systemd/systemd-fsck, libsystemd-shared-256.so unknown
ERROR: in /home/alexis/src/neon-beat/neon-beat-controller/output/host/lib/systemd/systemd-battery-check, libsystemd-shared-256.so unknown
ERROR: in /home/alexis/src/neon-beat/neon-beat-controller/output/host/lib/systemd/systemd, libsystemd-core-256.so unknown
ERROR: in /home/alexis/src/neon-beat/neon-beat-controller/output/host/lib/systemd/systemd, libsystemd-shared-256.so unknown
ERROR: in /home/alexis/src/neon-beat/neon-beat-controller/output/host/lib/systemd/libsystemd-core-256.so, libsystemd-shared-256.so unknown
ERROR: in /home/alexis/src/neon-beat/neon-beat-controller/output/host/lib/systemd/systemd-shutdown, libsystemd-shared-256.so unknown
ERROR: in /home/alexis/src/neon-beat/neon-beat-controller/output/host/lib/systemd/systemd-sysctl, libsystemd-shared-256.so unknown
ERROR: in /home/alexis/src/neon-beat/neon-beat-controller/output/host/lib/systemd/systemd-sulogin-shell, libsystemd-shared-256.so unknown
ERROR: in /home/alexis/src/neon-beat/neon-beat-controller/output/host/lib/systemd/systemd-makefs, libsystemd-shared-256.so unknown
ERROR: in /home/alexis/src/neon-beat/neon-beat-controller/output/host/lib/systemd/systemd-remount-fs, libsystemd-shared-256.so unknown
ERROR: in /home/alexis/src/neon-beat/neon-beat-controller/output/host/lib/systemd/systemd-growfs, libsystemd-shared-256.so unknown
ERROR: in /home/alexis/src/neon-beat/neon-beat-controller/output/host/lib/systemd/systemd-update-done, libsystemd-shared-256.so unknown
ERROR: in /home/alexis/src/neon-beat/neon-beat-controller/output/host/lib/systemd/systemd-boot-check-no-failures, libsystemd-shared-256.so unknown
ERROR: in /home/alexis/src/neon-beat/neon-beat-controller/output/host/lib/systemd/systemd-journald, libsystemd-shared-256.so unknown
ERROR: in /home/alexis/src/neon-beat/neon-beat-controller/output/host/lib/erlang/lib/wx-2.4.1/priv/wxe_driver.so, libwx_gtk3u_stc-3.2.so.0 unknown
ERROR: in /home/alexis/src/neon-beat/neon-beat-controller/output/host/lib/erlang/lib/wx-2.4.1/priv/wxe_driver.so, libwx_gtk3u_xrc-3.2.so.0 unknown
ERROR: in /home/alexis/src/neon-beat/neon-beat-controller/output/host/lib/erlang/lib/wx-2.4.1/priv/wxe_driver.so, libwx_gtk3u_html-3.2.so.0 unknown
ERROR: in /home/alexis/src/neon-beat/neon-beat-controller/output/host/lib/erlang/lib/wx-2.4.1/priv/wxe_driver.so, libwx_baseu_xml-3.2.so.0 unknown
ERROR: in /home/alexis/src/neon-beat/neon-beat-controller/output/host/lib/erlang/lib/wx-2.4.1/priv/wxe_driver.so, libwx_gtk3u_core-3.2.so.0 unknown
ERROR: in /home/alexis/src/neon-beat/neon-beat-controller/output/host/lib/erlang/lib/wx-2.4.1/priv/wxe_driver.so, libwx_baseu-3.2.so.0 unknown
ERROR: in /home/alexis/src/neon-beat/neon-beat-controller/output/host/lib/erlang/lib/wx-2.4.1/priv/wxe_driver.so, libwx_gtk3u_gl-3.2.so.0 unknown
ERROR: in /home/alexis/src/neon-beat/neon-beat-controller/output/host/lib/erlang/lib/wx-2.4.1/priv/wxe_driver.so, libwx_gtk3u_aui-3.2.so.0 unknown
ERROR: in /home/alexis/src/neon-beat/neon-beat-controller/output/host/lib/erlang/lib/wx-2.4.1/priv/wxe_driver.so, libwx_gtk3u_webview-3.2.so.0 unknown
ERROR: in /home/alexis/src/neon-beat/neon-beat-controller/output/host/lib/erlang/lib/wx-2.4.1/priv/erl_gl.so, libGL.so.1 unknown
ERROR: in /home/alexis/src/neon-beat/neon-beat-controller/output/host/lib/erlang/lib/wx-2.4.1/priv/erl_gl.so, libGLU.so.1 unknown
ERROR: in /home/alexis/src/neon-beat/neon-beat-controller/output/host/lib/erlang/lib/odbc-2.14.2/priv/bin/odbcserver, libodbc.so.2 unknown
ERROR: in /home/alexis/src/neon-beat/neon-beat-controller/output/host/lib/python3.12/lib-dynload/_gdbm.cpython-312-x86_64-linux-gnu.so, libgdbm.so.6 unknown
ERROR: in /home/alexis/src/neon-beat/neon-beat-controller/output/host/lib/python3.12/lib-dynload/readline.cpython-312-x86_64-linux-gnu.so, libreadline.so.8 unknown
ERROR: in /home/alexis/src/neon-beat/neon-beat-controller/output/host/lib/python3.12/lib-dynload/_dbm.cpython-312-x86_64-linux-gnu.so, libgdbm_compat.so.4 unknown make[2]: *** [Makefile:747: host-finalize] Error 1
make[1]: *** [Makefile:23: _all] Error 2
make: *** [Makefile:24: neon-beat-controller] Error 2

I did not take a look at all of those, but the first one on systemd-cgtop looks wrong.

  $ ldd /home/alexis/src/neon-beat/neon-beat-controller/output/host/bin/systemd-cgtop
    linux-vdso.so.1 (0x00007fe696105000)
    libsystemd-shared-256.so => /home/alexis/src/neon-beat/neon-beat-controller/output/host/bin/../lib/systemd/libsystemd-shared-256.so (0x00007fe695c00000)
    libc.so.6 => /usr/lib/libc.so.6 (0x00007fe695800000)
    libcap.so.2 => /home/alexis/src/neon-beat/neon-beat-controller/output/host/bin/../lib/systemd/../../lib/libcap.so.2 (0x00007fe69609d000)
    libcrypt.so.2 => /home/alexis/src/neon-beat/neon-beat-controller/output/host/bin/../lib/systemd/../../lib/libcrypt.so.2 (0x00007fe69605d000)
    libmount.so.1 => /home/alexis/src/neon-beat/neon-beat-controller/output/host/bin/../lib/systemd/../../lib/libmount.so.1 (0x00007fe695fe6000)
    libm.so.6 => /usr/lib/libm.so.6 (0x00007fe695ac9000)
    /lib64/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007fe696107000)
    libblkid.so.1 => /home/alexis/src/neon-beat/neon-beat-controller/output/host/bin/../lib/systemd/../../lib/libblkid.so.1 (0x00007fe695a65000)

$ls -l output/host/lib/systemd/libsystemd-shared-256.so
-rwxr-xr-x 1 alexis alexis 3713824 Aug 25 09:11 output/host/lib/systemd/libsystemd-shared-256.so

$readelf -d output/host/bin/systemd-cgtop

Dynamic section at offset 0x8ae0 contains 29 entries:
Tag Type Name/Value

   0x0000000000000001 (NEEDED)             Shared library: [libsystemd-shared-256.so]
   0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]
   0x000000000000001d (RUNPATH)            Library runpath: [$ORIGIN/../lib:$ORIGIN/../lib/systemd]
   0x000000000000000c (INIT)               0x3000
   0x000000000000000d (FINI)               0x64a0
   0x0000000000000019 (INIT_ARRAY)         0x9910
   0x000000000000001b (INIT_ARRAYSZ)       8 (bytes)
   0x000000000000001a (FINI_ARRAY)         0x9918

[...]

The main issue seems to be due to libsystemd-shared-256.so being in $(HOST_DIR)/lib/systemd and not $(HOST_DIR)/lib. Shouldn't check-host-libs also parse this RUNPATH ?

Alexis

_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot

#2 lore raw
In reply to Thomas Petazzoni via buildroot

Thomas, All,

On 2026-08-24 23:34 +0200, Thomas Petazzoni spake thusly:
[--SNIP--]

Quoted 5 lines

What it does is that at the end of the build, it verifies that all binaries and libraries in $(HOST_DIR) only have shared library dependencies on libraries that are in Buildroot $(HOST_DIR), to the exception of the C library, for which we of course use the system C library.

[--SNIP--]

I was wondering why you would not extend check-host-rpath, see my tentative, totally untested patch at the end...

Also, check-host-rpath is run after each package install, so it will catch executables installed without their dependent libraries, should they later be installed.

Otherwise, some comments below...

support/scripts/check-host-libs line 3

Quoted 9 lines
support/scripts/check-host-libs new file +3 -0
new file mode 100755
index 0000000000..79f60024c4
@@ -0,0 +1,37 @@
+#!/bin/bash
+
+HOST_DIR=$1

Double-quote variable expansion (shellcheck should have reported that one, I think).

[--SNIP--]

support/scripts/check-host-libs line 10

Quoted 1 line
support/scripts/check-host-libs new file +1 -0
+bailout="no"

There is a construct that I tend to use nowadas, which is to use true/false, rather than 0/1 or yes/no, because that can be reused without a test;

success=true
for loop; do
if [ conditiion ]; then
success=false
fi
done
${success}

[--SNIP--]

support/scripts/check-host-libs line 20

Quoted 1 line
support/scripts/check-host-libs new file +1 -0
+        case ${lib} in

Ditto, double-quote around variable expansion.

And here's a tentative, totally untested patch to introduce that in check-host-rpath (indented so that patchwork does not see it):

diff --git a/support/scripts/check-host-rpath b/support/scripts/check-host-rpath index 41aa0aa1ed..e1fdf16829 100755
--- a/support/scripts/check-host-rpath
+++ b/support/scripts/check-host-rpath
@@ -21,6 +21,14 @@ main() {
while read file; do
is_elf "${file}" || continue
elf_needs_rpath "${file}" "${hostdir}" || continue
+ missing_libs="$(get_missing_libs "${file}" "${hostdir}")"
+ if [ "${missing_libs}" ]; then
+ ret=1
+ printf "***\n"
+ printf "*** ERROR: package %s uses libs not in HOST_DIR:\n" "${pkg}"
+ # shellcheck disable=SC2086 # we need the word splitting
+ printf ' - %s\n' ${missing_libs}
+ fi
check_elf_has_rpath "${file}" "${hostdir}" "${perpackagedir}" && continue if [ ${ret} -eq 0 ]; then
ret=1
@@ -57,16 +65,44 @@ is_elf() {
elf_needs_rpath() {
local file="${1}"
local hostdir="${2}"
+
+ [ -n "$(get_libs "${file}" "${hostdir}")" ]
+}
+
+# This function returns all non-toolchain libs tha tare not in HOST_DIR +get_missing_libs() {
+ local file="${1}"
+ local hostdir="${2}"
+ local lib
+
+ get_libs "${file}" "${hostdir}" \
+ | while read lib; do
+ if [ -not -e "${hostdir}/lib/${lib}" ]; then
+ printf '%s\n' "${lib}"
+ fi
+ done
+}
+
+# This function returns the list of non-toolchain libraries that an ELF +# file has as DT_NEEDED
+get_libs() {
+ local file="${1}"
+ local hostdir="${2}"
local lib

while read lib; do
- [ -e "${hostdir}/lib/${lib}" ] && return 0
+ case "${lib}" in
+ libc.so*|libm.so*|libstdc++.so*|libpthread.so*|libgcc_s.so*|libdl.so*|ld-*|libgomp.so*|libcrypt.so*|libcrypto.so*|libatomic.so*|librt.so*|libutil.so*|libresolv.so*)
+ continue
+ ;;
+ *)
+ printf '%s\n' "${lib}"
+ ;;
+ esac
done < <( readelf -d "${file}" 2>/dev/null \
|sed -r -e '/^.* \(NEEDED\) .*Shared library: \[(.+)\]$/!d;' \
-e 's//\1/;' \
)
-
- return 1
}

# This function checks whether at least one of the RPATH of the given

Signature · 10 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
#3 lore raw

Hello,

Quoted 6 lines · Yann E. MORIN

I was wondering why you would not extend check-host-rpath, see my tentative, totally untested patch at the end...

Also, check-host-rpath is run after each package install, so it will catch executables installed without their dependent libraries, should they later be installed.

Ah, good idea. Then maybe check-host-rpath should be renamed into check-host-bins or something like that, because it would no longer check just RPATH, but other aspects of host-installed bins and libs?

Quoted 4 lines
Quoted 1 line

+HOST_DIR=$1

Double-quote variable expansion (shellcheck should have reported that one, I think).

I just passed "./utils/docker-run shellcheck" on it, maybe there's some additional shellcheck options to pass to have stricter checks.

Quoted 11 lines

There is a construct that I tend to use nowadas, which is to use true/false, rather than 0/1 or yes/no, because that can be reused without a test;

success=true
for loop; do
if [ conditiion ]; then
success=false
fi
done
${success}

OK, nice.

Quoted 60 lines

And here's a tentative, totally untested patch to introduce that in check-host-rpath (indented so that patchwork does not see it):

diff --git a/support/scripts/check-host-rpath b/support/scripts/check-host-rpath index 41aa0aa1ed..e1fdf16829 100755
--- a/support/scripts/check-host-rpath
+++ b/support/scripts/check-host-rpath
@@ -21,6 +21,14 @@ main() {
while read file; do
is_elf "${file}" || continue
elf_needs_rpath "${file}" "${hostdir}" || continue
+ missing_libs="$(get_missing_libs "${file}" "${hostdir}")"
+ if [ "${missing_libs}" ]; then
+ ret=1
+ printf "***\n"
+ printf "*** ERROR: package %s uses libs not in HOST_DIR:\n" "${pkg}"
+ # shellcheck disable=SC2086 # we need the word splitting
+ printf ' - %s\n' ${missing_libs}
+ fi
check_elf_has_rpath "${file}" "${hostdir}" "${perpackagedir}" && continue if [ ${ret} -eq 0 ]; then
ret=1
@@ -57,16 +65,44 @@ is_elf() {
elf_needs_rpath() {
local file="${1}"
local hostdir="${2}"
+
+ [ -n "$(get_libs "${file}" "${hostdir}")" ]
+}
+
+# This function returns all non-toolchain libs tha tare not in HOST_DIR +get_missing_libs() {
+ local file="${1}"
+ local hostdir="${2}"
+ local lib
+
+ get_libs "${file}" "${hostdir}" \
+ | while read lib; do
+ if [ -not -e "${hostdir}/lib/${lib}" ]; then
+ printf '%s\n' "${lib}"
+ fi
+ done
+}
+
+# This function returns the list of non-toolchain libraries that an ELF +# file has as DT_NEEDED
+get_libs() {
+ local file="${1}"
+ local hostdir="${2}"
local lib

while read lib; do
- [ -e "${hostdir}/lib/${lib}" ] && return 0
+ case "${lib}" in
+ libc.so*|libm.so*|libstdc++.so*|libpthread.so*|libgcc_s.so*|libdl.so*|ld-*|libgomp.so*|libcrypt.so*|libcrypto.so*|libatomic.so*|librt.so*|libutil.so*|libresolv.so*)
+ continue
+ ;;
+ *)
+ printf '%s\n' "${lib}"
+ ;;

Quoted 8 lines

+ esac
done < <( readelf -d "${file}" 2>/dev/null \
|sed -r -e '/^.* \(NEEDED\) .*Shared library: \[(.+)\]$/!d;' \
-e 's//\1/;' \
)
-
- return 1
}

OK, I'll review this and try to get back with a v3... hopefully earlier than in 4 years!

Thanks for the prompt review and feedback!

Thomas

Signature · 7 lines
Thomas Petazzoni, co-owner and CEO, Bootlin
Embedded Linux and Kernel engineering and training
https://bootlin.com
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot