Skip to contentSkip to conversation

python3: Fix platform triplet detection

PATCH In review Khem Raj 2 participants 6 messages Oct 22, 2024 – Oct 28, 2024

lore
Share & tools
lore

Khem Raj

author
#0 lore raw

PATCH v3 1/3 python3: Fix platform triplet detection

Currently, platform is based upon build_os detection which works ok for native builds and where build_os and host_os are same i.e. glibc based systems, but it fails for musl based systems to load correct modules which are shared libraries e.g. python3-pydantic-core

This is already fixed upstream in 3.13+ but we need this backported at minimum to get 3.12 or 3.11 working with musl

see
https://gitlab.alpinelinux.org/alpine/aports/-/issues/13227
https://github.com/meta-homeassistant/meta-homeassistant/issues/89
https://github.com/pypa/auditwheel/issues/349
https://github.com/PyO3/maturin/issues/1559

---
v2: rebased
v3: rebased

 ...r-platform-triplet-detection-code-ad.patch | 498 ++++++++++++++++++
 .../recipes-devtools/python/python3_3.12.6.bb |   1 +
 2 files changed, 499 insertions(+)
 create mode 100644 meta/recipes-devtools/python/python3/0001-gh-95855-Refactor-platform-triplet-detection-code-ad.patch
meta/recipes-devtools/python/python3/0001-gh-95855-Refactor-platform-triplet-detection-code-ad.patch new file +498 -0
new file mode 100644
index 00000000000..863fcc2d36b
@@ -0,0 +1,498 @@
+rom c163d7f0b67a568e9b64eeb9c1cbbaa127818596 Mon Sep 17 00:00:00 2001
+From: Jeffery To <[email protected]>
+Date: Thu, 24 Aug 2023 20:22:50 +0800
+Subject: [PATCH] gh-95855: Refactor platform triplet detection code, add
+ detection for MIPS soft float and musl libc (#107221)
+
+- Move platform triplet detection code into Misc/platform_triplet.c
+- Refactor MIPS detection, use defined(__mips64) to detect MIPS64
+- Compute libc values in separate section
+- Add detection for MIPS soft float
+- Add detection for musl
+
+musl supports SPE with its soft-float ABI:
+https://git.musl-libc.org/cgit/musl/commit/?id=7be59733d71ada3a32a98622507399253f1d5e48
+
+Original patch by Christian Heimes.
+
+Upstream-Status: Backport [https://github.com/python/cpython/commit/c163d7f0b67a568e9b64eeb9c1cbbaa127818596]
+Signed-off-by: Khem Raj <[email protected]>
+Co-authored-by: Christian Heimes <[email protected]>
+Co-authored-by: Erlend E. Aasland <[email protected]>
+---
+ ...3-07-25-02-30-00.gh-issue-95855.wA7rAf.rst |   2 +
+ Misc/platform_triplet.c                       | 255 ++++++++++++++++++
+ configure                                     | 192 +------------
+ configure.ac                                  | 192 +------------
+ 4 files changed, 265 insertions(+), 376 deletions(-)
+ create mode 100644 Misc/NEWS.d/next/Build/2023-07-25-02-30-00.gh-issue-95855.wA7rAf.rst
+ create mode 100644 Misc/platform_triplet.c
+
+--- /dev/null
++++ b/Misc/NEWS.d/next/Build/2023-07-25-02-30-00.gh-issue-95855.wA7rAf.rst
+@@ -0,0 +1,2 @@
++Refactor platform triplet detection code and add detection for MIPS soft
++float and musl libc.
+--- /dev/null
++++ b/Misc/platform_triplet.c
+@@ -0,0 +1,255 @@
++/* Detect platform triplet from builtin defines
++ * cc -E Misc/platform_triplet.c | grep '^PLATFORM_TRIPLET=' | tr -d ' '
++ */
++#undef bfin
++#undef cris
++#undef fr30
++#undef linux
++#undef hppa
++#undef hpux
++#undef i386
++#undef mips
++#undef powerpc
++#undef sparc
++#undef unix
++#if defined(__ANDROID__)
++    # Android is not a multiarch system.
++#elif defined(__linux__)
++/*
++ * BEGIN of Linux block
++ */
++// Detect libc (based on config.guess)
++# include <features.h>
++# if defined(__UCLIBC__)
++#  error uclibc not supported
++# elif defined(__dietlibc__)
++#  error dietlibc not supported
++# elif defined(__GLIBC__)
++#  define LIBC gnu
++#  define LIBC_X32 gnux32
++#  if defined(__ARM_PCS_VFP)
++#   define LIBC_ARM gnueabihf
++#  else
++#   define LIBC_ARM gnueabi
++#  endif
++#  if defined(__loongarch__)
++#   if defined(__loongarch_soft_float)
++#    define LIBC_LA gnusf
++#   elif defined(__loongarch_single_float)
++#    define LIBC_LA gnuf32
++#   elif defined(__loongarch_double_float)
++#    define LIBC_LA gnu
++#   else
++#    error unknown loongarch floating-point base abi
++#   endif
++#  endif
++#  if defined(_MIPS_SIM)
++#   if defined(__mips_hard_float)
++#    if _MIPS_SIM == _ABIO32
++#     define LIBC_MIPS gnu
++#    elif _MIPS_SIM == _ABIN32
++#     define LIBC_MIPS gnuabin32
++#    elif _MIPS_SIM == _ABI64
++#     define LIBC_MIPS gnuabi64
++#    else
++#     error unknown mips sim value
++#    endif
++#   else
++#    if _MIPS_SIM == _ABIO32
++#     define LIBC_MIPS gnusf
++#    elif _MIPS_SIM == _ABIN32
++#     define LIBC_MIPS gnuabin32sf
++#    elif _MIPS_SIM == _ABI64
++#     define LIBC_MIPS gnuabi64sf
++#    else
++#     error unknown mips sim value
++#    endif
++#   endif
++#  endif
++#  if defined(__SPE__)
++#   define LIBC_PPC gnuspe
++#  else
++#   define LIBC_PPC gnu
++#  endif
++# else
++// Heuristic to detect musl libc
++#  include <stdarg.h>
++#  ifdef __DEFINED_va_list
++#   define LIBC musl
++#   define LIBC_X32 muslx32
++#   if defined(__ARM_PCS_VFP)
++#    define LIBC_ARM musleabihf
++#   else
++#    define LIBC_ARM musleabi
++#   endif
++#   if defined(__loongarch__)
++#    if defined(__loongarch_soft_float)
++#     define LIBC_LA muslsf
++#    elif defined(__loongarch_single_float)
++#     define LIBC_LA muslf32
++#    elif defined(__loongarch_double_float)
++#     define LIBC_LA musl
++#    else
++#     error unknown loongarch floating-point base abi
++#    endif
++#   endif
++#   if defined(_MIPS_SIM)
++#    if defined(__mips_hard_float)
++#     if _MIPS_SIM == _ABIO32
++#      define LIBC_MIPS musl
++#     elif _MIPS_SIM == _ABIN32
++#      define LIBC_MIPS musln32
++#     elif _MIPS_SIM == _ABI64
++#      define LIBC_MIPS musl
++#     else
++#      error unknown mips sim value
++#     endif
++#    else
++#     if _MIPS_SIM == _ABIO32
++#      define LIBC_MIPS muslsf
++#     elif _MIPS_SIM == _ABIN32
++#      define LIBC_MIPS musln32sf
++#     elif _MIPS_SIM == _ABI64
++#      define LIBC_MIPS muslsf
++#     else
++#      error unknown mips sim value
++#     endif
++#    endif
++#   endif
++#   if defined(_SOFT_FLOAT) || defined(__NO_FPRS__)
++#    define LIBC_PPC muslsf
++#   else
++#    define LIBC_PPC musl
++#   endif
++#  else
++#   error unknown libc
++#  endif
++# endif
++
++# if defined(__x86_64__) && defined(__LP64__)
++PLATFORM_TRIPLET=x86_64-linux-LIBC
++# elif defined(__x86_64__) && defined(__ILP32__)
++PLATFORM_TRIPLET=x86_64-linux-LIBC_X32
++# elif defined(__i386__)
++PLATFORM_TRIPLET=i386-linux-LIBC
++# elif defined(__aarch64__) && defined(__AARCH64EL__)
++#  if defined(__ILP32__)
++PLATFORM_TRIPLET=aarch64_ilp32-linux-LIBC
++#  else
++PLATFORM_TRIPLET=aarch64-linux-LIBC
++#  endif
++# elif defined(__aarch64__) && defined(__AARCH64EB__)
++#  if defined(__ILP32__)
++PLATFORM_TRIPLET=aarch64_be_ilp32-linux-LIBC
++#  else
++PLATFORM_TRIPLET=aarch64_be-linux-LIBC
++#  endif
++# elif defined(__alpha__)
++PLATFORM_TRIPLET=alpha-linux-LIBC
++# elif defined(__ARM_EABI__)
++#  if defined(__ARMEL__)
++PLATFORM_TRIPLET=arm-linux-LIBC_ARM
++#  else
++PLATFORM_TRIPLET=armeb-linux-LIBC_ARM
++#  endif
++# elif defined(__hppa__)
++PLATFORM_TRIPLET=hppa-linux-LIBC
++# elif defined(__ia64__)
++PLATFORM_TRIPLET=ia64-linux-LIBC
++# elif defined(__loongarch__) && defined(__loongarch_lp64)
++PLATFORM_TRIPLET=loongarch64-linux-LIBC_LA
++# elif defined(__m68k__) && !defined(__mcoldfire__)
++PLATFORM_TRIPLET=m68k-linux-LIBC
++# elif defined(__mips__)
++#  if defined(__mips_isa_rev) && (__mips_isa_rev >=6)
++#   if defined(_MIPSEL) && defined(__mips64)
++PLATFORM_TRIPLET=mipsisa64r6el-linux-LIBC_MIPS
++#   elif defined(_MIPSEL)
++PLATFORM_TRIPLET=mipsisa32r6el-linux-LIBC_MIPS
++#   elif defined(__mips64)
++PLATFORM_TRIPLET=mipsisa64r6-linux-LIBC_MIPS
++#   else
++PLATFORM_TRIPLET=mipsisa32r6-linux-LIBC_MIPS
++#   endif
++#  else
++#   if defined(_MIPSEL) && defined(__mips64)
++PLATFORM_TRIPLET=mips64el-linux-LIBC_MIPS
++#   elif defined(_MIPSEL)
++PLATFORM_TRIPLET=mipsel-linux-LIBC_MIPS
++#   elif defined(__mips64)
++PLATFORM_TRIPLET=mips64-linux-LIBC_MIPS
++#   else
++PLATFORM_TRIPLET=mips-linux-LIBC_MIPS
++#   endif
++#  endif
++# elif defined(__or1k__)
++PLATFORM_TRIPLET=or1k-linux-LIBC
++# elif defined(__powerpc64__)
++#  if defined(__LITTLE_ENDIAN__)
++PLATFORM_TRIPLET=powerpc64le-linux-LIBC
++#  else
++PLATFORM_TRIPLET=powerpc64-linux-LIBC
++#  endif
++# elif defined(__powerpc__)
++PLATFORM_TRIPLET=powerpc-linux-LIBC_PPC
++# elif defined(__s390x__)
++PLATFORM_TRIPLET=s390x-linux-LIBC
++# elif defined(__s390__)
++PLATFORM_TRIPLET=s390-linux-LIBC
++# elif defined(__sh__) && defined(__LITTLE_ENDIAN__)
++PLATFORM_TRIPLET=sh4-linux-LIBC
++# elif defined(__sparc__) && defined(__arch64__)
++PLATFORM_TRIPLET=sparc64-linux-LIBC
++# elif defined(__sparc__)
++PLATFORM_TRIPLET=sparc-linux-LIBC
++# elif defined(__riscv)
++#  if __riscv_xlen == 32
++PLATFORM_TRIPLET=riscv32-linux-LIBC
++#  elif __riscv_xlen == 64
++PLATFORM_TRIPLET=riscv64-linux-LIBC
++#  else
++#   error unknown platform triplet
++#  endif
++# else
++#   error unknown platform triplet
++# endif
++/*
++ * END of Linux block
++ */
++#elif defined(__FreeBSD_kernel__)
++# if defined(__LP64__)
++PLATFORM_TRIPLET=x86_64-kfreebsd-gnu
++# elif defined(__i386__)
++PLATFORM_TRIPLET=i386-kfreebsd-gnu
++# else
++#   error unknown platform triplet
++# endif
++#elif defined(__gnu_hurd__)
++PLATFORM_TRIPLET=i386-gnu
++#elif defined(__APPLE__)
++PLATFORM_TRIPLET=darwin
++#elif defined(__VXWORKS__)
++PLATFORM_TRIPLET=vxworks
++#elif defined(__wasm32__)
++#  if defined(__EMSCRIPTEN__)
++PLATFORM_TRIPLET=wasm32-emscripten
++#  elif defined(__wasi__)
++#    if defined(_REENTRANT)
++PLATFORM_TRIPLET=wasm32-wasi-threads
++#    else
++PLATFORM_TRIPLET=wasm32-wasi
++#    endif
++#  else
++#    error unknown wasm32 platform
++#  endif
++#elif defined(__wasm64__)
++#  if defined(__EMSCRIPTEN__)
++PLATFORM_TRIPLET=wasm64-emscripten
++#  elif defined(__wasi__)
++PLATFORM_TRIPLET=wasm64-wasi
++#  else
++#    error unknown wasm64 platform
++#  endif
++#else
++# error unknown platform triplet
++#endif
+--- a/configure.ac
++++ b/configure.ac
+@@ -927,198 +927,14 @@ fi
+ 
+ 
+ AC_MSG_CHECKING([for the platform triplet based on compiler characteristics])
+-cat > conftest.c <<EOF
+-#undef bfin
+-#undef cris
+-#undef fr30
+-#undef linux
+-#undef hppa
+-#undef hpux
+-#undef i386
+-#undef mips
+-#undef powerpc
+-#undef sparc
+-#undef unix
+-#if defined(__ANDROID__)
+-    # Android is not a multiarch system.
+-#elif defined(__linux__)
+-# if defined(__x86_64__) && defined(__LP64__)
+-        x86_64-linux-gnu
+-# elif defined(__x86_64__) && defined(__ILP32__)
+-        x86_64-linux-gnux32
+-# elif defined(__i386__)
+-        i386-linux-gnu
+-# elif defined(__aarch64__) && defined(__AARCH64EL__)
+-#  if defined(__ILP32__)
+-        aarch64_ilp32-linux-gnu
+-#  else
+-        aarch64-linux-gnu
+-#  endif
+-# elif defined(__aarch64__) && defined(__AARCH64EB__)
+-#  if defined(__ILP32__)
+-        aarch64_be_ilp32-linux-gnu
+-#  else
+-        aarch64_be-linux-gnu
+-#  endif
+-# elif defined(__alpha__)
+-        alpha-linux-gnu
+-# elif defined(__ARM_EABI__) && defined(__ARM_PCS_VFP)
+-#  if defined(__ARMEL__)
+-        arm-linux-gnueabihf
+-#  else
+-        armeb-linux-gnueabihf
+-#  endif
+-# elif defined(__ARM_EABI__) && !defined(__ARM_PCS_VFP)
+-#  if defined(__ARMEL__)
+-        arm-linux-gnueabi
+-#  else
+-        armeb-linux-gnueabi
+-#  endif
+-# elif defined(__hppa__)
+-        hppa-linux-gnu
+-# elif defined(__ia64__)
+-        ia64-linux-gnu
+-# elif defined(__loongarch__)
+-#  if defined(__loongarch_lp64)
+-#   if defined(__loongarch_soft_float)
+-        loongarch64-linux-gnusf
+-#   elif defined(__loongarch_single_float)
+-        loongarch64-linux-gnuf32
+-#   elif defined(__loongarch_double_float)
+-        loongarch64-linux-gnu
+-#   else
+-#    error unknown platform triplet
+-#   endif
+-#  else
+-#   error unknown platform triplet
+-#  endif
+-# elif defined(__m68k__) && !defined(__mcoldfire__)
+-        m68k-linux-gnu
+-# elif defined(__mips_hard_float) && defined(__mips_isa_rev) && (__mips_isa_rev >=6) && defined(_MIPSEL)
+-#  if _MIPS_SIM == _ABIO32
+-        mipsisa32r6el-linux-gnu
+-#  elif _MIPS_SIM == _ABIN32
+-        mipsisa64r6el-linux-gnuabin32
+-#  elif _MIPS_SIM == _ABI64
+-        mipsisa64r6el-linux-gnuabi64
+-#  else
+-#   error unknown platform triplet
+-#  endif
+-# elif defined(__mips_hard_float) && defined(__mips_isa_rev) && (__mips_isa_rev >=6)
+-#  if _MIPS_SIM == _ABIO32
+-        mipsisa32r6-linux-gnu
+-#  elif _MIPS_SIM == _ABIN32
+-        mipsisa64r6-linux-gnuabin32
+-#  elif _MIPS_SIM == _ABI64
+-        mipsisa64r6-linux-gnuabi64
+-#  else
+-#   error unknown platform triplet
+-#  endif
+-# elif defined(__mips_hard_float) && defined(_MIPSEL)
+-#  if _MIPS_SIM == _ABIO32
+-        mipsel-linux-gnu
+-#  elif _MIPS_SIM == _ABIN32
+-        mips64el-linux-gnuabin32
+-#  elif _MIPS_SIM == _ABI64
+-        mips64el-linux-gnuabi64
+-#  else
+-#   error unknown platform triplet
+-#  endif
+-# elif defined(__mips_hard_float)
+-#  if _MIPS_SIM == _ABIO32
+-        mips-linux-gnu
+-#  elif _MIPS_SIM == _ABIN32
+-        mips64-linux-gnuabin32
+-#  elif _MIPS_SIM == _ABI64
+-        mips64-linux-gnuabi64
+-#  else
+-#   error unknown platform triplet
+-#  endif
+-# elif defined(__or1k__)
+-        or1k-linux-gnu
+-# elif defined(__powerpc__) && defined(__SPE__)
+-        powerpc-linux-gnuspe
+-# elif defined(__powerpc64__)
+-#  if defined(__LITTLE_ENDIAN__)
+-        powerpc64le-linux-gnu
+-#  else
+-        powerpc64-linux-gnu
+-#  endif
+-# elif defined(__powerpc__)
+-        powerpc-linux-gnu
+-# elif defined(__s390x__)
+-        s390x-linux-gnu
+-# elif defined(__s390__)
+-        s390-linux-gnu
+-# elif defined(__sh__) && defined(__LITTLE_ENDIAN__)
+-        sh4-linux-gnu
+-# elif defined(__sparc__) && defined(__arch64__)
+-        sparc64-linux-gnu
+-# elif defined(__sparc__)
+-        sparc-linux-gnu
+-# elif defined(__riscv)
+-#  if __riscv_xlen == 32
+-        riscv32-linux-gnu
+-#  elif __riscv_xlen == 64
+-        riscv64-linux-gnu
+-#  else
+-#   error unknown platform triplet
+-#  endif
+-# else
+-#   error unknown platform triplet
+-# endif
+-#elif defined(__FreeBSD_kernel__)
+-# if defined(__LP64__)
+-        x86_64-kfreebsd-gnu
+-# elif defined(__i386__)
+-        i386-kfreebsd-gnu
+-# else
+-#   error unknown platform triplet
+-# endif
+-#elif defined(__gnu_hurd__)
+-        i386-gnu
+-#elif defined(__APPLE__)
+-        darwin
+-#elif defined(__VXWORKS__)
+-        vxworks
+-#elif defined(__wasm32__)
+-#  if defined(__EMSCRIPTEN__)
+-	wasm32-emscripten
+-#  elif defined(__wasi__)
+-#    if defined(_REENTRANT)
+-	wasm32-wasi-threads
+-#    else
+-	wasm32-wasi
+-#    endif
+-#  else
+-#    error unknown wasm32 platform
+-#  endif
+-#elif defined(__wasm64__)
+-#  if defined(__EMSCRIPTEN__)
+-	wasm64-emscripten
+-#  elif defined(__wasi__)
+-	wasm64-wasi
+-#  else
+-#    error unknown wasm64 platform
+-#  endif
+-#else
+-# error unknown platform triplet
+-#endif
+-
+-EOF
+-
+-if $CPP $CPPFLAGS conftest.c >conftest.out 2>/dev/null; then
+-  PLATFORM_TRIPLET=`grep -v '^#' conftest.out | grep -v '^ *$' | tr -d ' 	'`
+-  case "$build_os" in
+-  linux-musl*)
+-    PLATFORM_TRIPLET=`echo "$PLATFORM_TRIPLET" | sed 's/linux-gnu/linux-musl/'`
+-    ;;
+-  esac
++if $CPP $CPPFLAGS $srcdir/Misc/platform_triplet.c >conftest.out 2>/dev/null; then
++  PLATFORM_TRIPLET=`grep '^PLATFORM_TRIPLET=' conftest.out | tr -d ' 	'`
++  PLATFORM_TRIPLET="${PLATFORM_TRIPLET@%:@PLATFORM_TRIPLET=}"
+   AC_MSG_RESULT([$PLATFORM_TRIPLET])
+ else
+   AC_MSG_RESULT([none])
+ fi
+-rm -f conftest.c conftest.out
++rm -f conftest.out
+ 
+ AC_MSG_CHECKING([for multiarch])
+ AS_CASE([$ac_sys_system],
meta/recipes-devtools/python/python3_3.12.6.bb +1 -0
index 858124d104f..da1a0afad4c 100644
@@ -34,6 +34,7 @@ SRC_URI = "http://www.python.org/ftp/python/${PV}/Python-${PV}.tar.xz \
 	   file://0001-test_deadlock-skip-problematic-test.patch \
 	   file://0001-test_active_children-skip-problematic-test.patch \
            file://0001-test_readline-skip-limited-history-test.patch \
+           file://0001-gh-95855-Refactor-platform-triplet-detection-code-ad.patch \
            "
 
 SRC_URI:append:class-native = " \
author
#1 lore raw

PATCH v3 2/3 python3-packaging: Ignore old glibc tests on musl

These tests expect glibc to be the system C library ignore them for musl

---
v2: Use bitbake variable for ignored tests
v3: apply chmod to make the script executable

 meta/recipes-devtools/python/python3-packaging/run-ptest   | 3 ---
 .../recipes-devtools/python/python3-packaging/run-ptest.in | 2 ++
 meta/recipes-devtools/python/python3-packaging_24.1.bb     | 7 ++++++-
 3 files changed, 8 insertions(+), 4 deletions(-)
 delete mode 100644 meta/recipes-devtools/python/python3-packaging/run-ptest
 create mode 100644 meta/recipes-devtools/python/python3-packaging/run-ptest.in
meta/recipes-devtools/python/python3-packaging/run-ptest deleted +0 -3
deleted file mode 100644
index 8d2017d39ce..00000000000
@@ -1,3 +0,0 @@
-#!/bin/sh
-
-pytest --automake
meta/recipes-devtools/python/python3-packaging/run-ptest.in new file +2 -0
new file mode 100644
index 00000000000..508538463bb
@@ -0,0 +1,2 @@
+#!/bin/sh
+pytest --automake IGNOREDTESTS
meta/recipes-devtools/python/python3-packaging_24.1.bb +6 -1
index 20bcd55fbad..6986880ac51 100644
@@ -5,7 +5,7 @@ LIC_FILES_CHKSUM = "file://LICENSE;md5=faadaedca9251a90b205c9167578ce91"
 
 SRC_URI[sha256sum] = "026ed72c8ed3fcce5bf8950572258698927fd1dbda10a5e981cdf0ac37f4f002"
 
-SRC_URI += "file://run-ptest"
+SRC_URI += "file://run-ptest.in"
 
 inherit pypi python_flit_core ptest
 
@@ -15,6 +15,9 @@ BBCLASSEXTEND = "native nativesdk"
 DEPENDS:remove:class-native = "python3-build-native"
 RDEPENDS:${PN} += "python3-profile"
 
+SKIPLIST ?= ""
+SKIPLIST:libc-musl = "--deselect tests/test_manylinux.py::test_is_manylinux_compatible_old --ignore=tests/test_tags.py"
+
 do_compile:class-native () {
     python_flit_core_do_manual_build
 }
@@ -23,6 +26,8 @@ do_install_ptest() {
     cp -r ${S}/tests ${D}${PTEST_PATH}/
     # We don't need this script which is used to build the binaries
     rm -f ${D}${PTEST_PATH}/tests/manylinux/build.sh
+    sed -e 's|IGNOREDTESTS|${SKIPLIST}|' ${UNPACKDIR}/run-ptest.in > ${D}${PTEST_PATH}/run-ptest
+    chmod 0755 ${D}${PTEST_PATH}/run-ptest
 }
 
 RDEPENDS:${PN}-ptest = "\
author
#2 lore raw

PATCH v3 3/3 python3-libarchive-c: Avoid using find_library python API

In reply to Khem Raj

find_library API depends on platform to provide ldconfig, ld, gcc, objdump etc, so either we add these dependencies or avoid them by computing the libarchive library name during build, which we can be done.

This ensures that ffi module works with musl as well as glibc equally as musl does not provide ldconfig like glibc does

---
v2: rebased
v3: rebased

 meta/recipes-devtools/python/python3-libarchive-c_5.1.bb | 8 ++++++++
 1 file changed, 8 insertions(+)
meta/recipes-devtools/python/python3-libarchive-c_5.1.bb +8 -0
index 3116c6b62ec..510e2d9159c 100644
@@ -16,6 +16,14 @@ inherit pypi setuptools3 ptest
 
 SRC_URI[sha256sum] = "7bcce24ea6c0fa3bc62468476c6d2f6264156db2f04878a372027c10615a2721"
 
+DEPENDS += "patchelf-native libarchive"
+# Avoid using find_library API which needs ldconfig and ld/objdump
+# https://docs.python.org/3/library/ctypes.html#ctypes-reference
+#
+do_configure:append() {
+    sed -i -e "s|find_library('archive')|'${libdir}/$(patchelf --print-soname ${STAGING_LIBDIR}/libarchive.so)'|" ${S}/libarchive/ffi.py
+}
+
 RDEPENDS:${PN} += "\
   libarchive \
   python3-ctypes \
#3 lore raw

Re: [OE-core] [PATCH v3 3/3] python3-libarchive-c: Avoid using find_library python API

Quoted 3 lines · Khem Raj via lists.openembedded.org

find_library API depends on platform to provide ldconfig, ld, gcc, objdump etc, so either we add these dependencies or avoid them by computing the libarchive library name during build, which we can be done.

OH GOD THAT IS HORRIBLE.

"On Linux, find_library() tries to run external programs (/sbin/ldconfig, gcc, objdump and ld) to find the library file. It returns the filename of the library file.”

meta/recipes-devtools/python/python3-libarchive-c_5.1.bb inherit pypi setuptools3 ptest · line 25

Quoted 7 lines
meta/recipes-devtools/python/python3-libarchive-c_5.1.bb +7 -0
+DEPENDS += "patchelf-native libarchive"
+# Avoid using find_library API which needs ldconfig and ld/objdump
+# https://docs.python.org/3/library/ctypes.html#ctypes-reference
+#
+do_configure:append() {
+    sed -i -e "s|find_library('archive')|'${libdir}/$(patchelf --print-soname ${STAGING_LIBDIR}/libarchive.so)'|" ${S}/libarchive/ffi.py
+}

Can we patch the source instead:

-libarchive_path = os.environ.get('LIBARCHIVE') or find_library('archive') +libarchive_path = @LIBARCHIVE_SONAME@

And then just sed that expression with the right value? I don’t like magic seds because they can transparently not apply and regress in the future.

Also, thoughts on a solution that doesn’t involve ldconfig so it works on musl would be appreciated. Can you tell the musl loader to dump its search paths somehow? We should really file a bug with python…

Ross

author
#4 lore raw

Re: [OE-core] [PATCH v3 3/3] python3-libarchive-c: Avoid using find_library python API

Quoted 20 lines · Ross Burton
Quoted 3 lines · Khem Raj via lists.openembedded.org

find_library API depends on platform to provide ldconfig, ld, gcc, objdump etc, so either we add these dependencies or avoid them by computing the libarchive library name during build, which we can be done.

OH GOD THAT IS HORRIBLE.

"On Linux, find_library() tries to run external programs (/sbin/ldconfig, gcc, objdump and ld) to find the library file. It returns the filename of the library file.”

meta/recipes-devtools/python/python3-libarchive-c_5.1.bb inherit pypi setuptools3 ptest · line 25

Quoted 7 lines
meta/recipes-devtools/python/python3-libarchive-c_5.1.bb +7 -0
+DEPENDS += "patchelf-native libarchive"
+# Avoid using find_library API which needs ldconfig and ld/objdump
+# https://docs.python.org/3/library/ctypes.html#ctypes-reference
+#
+do_configure:append() {
+    sed -i -e "s|find_library('archive')|'${libdir}/$(patchelf --print-soname ${STAGING_LIBDIR}/libarchive.so)'|" ${S}/libarchive/ffi.py
+}

Can we patch the source instead:

Review

Quoted 2 lines
(quoted diff) +1 -1
-libarchive_path = os.environ.get('LIBARCHIVE') or find_library('archive')
+libarchive_path = @LIBARCHIVE_SONAME@

This would mean we do not respect LIBARCHIVE envvar which is a documented way of calling
a non-default libarchive.so perhaps we can do

libarchive_path = os.environ.get('LIBARCHIVE') or @LIBARCHIVE_SONAME@

unless we lift the whole expression out into sed op

Quoted 7 lines

And then just sed that expression with the right value? I don’t like magic seds because they can transparently not apply and regress in the future.

Also, thoughts on a solution that doesn’t involve ldconfig so it works on musl would be appreciated. Can you tell the musl loader to dump its search paths somehow? We should really file a bug with python…

Ross

#5 lore raw

Re: [OE-core] [PATCH v3 3/3] python3-libarchive-c: Avoid using find_library python API

Quoted 22 lines · Khem Raj
Quoted 20 lines · Ross Burton
Quoted 3 lines · Khem Raj via lists.openembedded.org

find_library API depends on platform to provide ldconfig, ld, gcc, objdump etc, so either we add these dependencies or avoid them by computing the libarchive library name during build, which we can be done.

OH GOD THAT IS HORRIBLE.

"On Linux, find_library() tries to run external programs (/sbin/ldconfig, gcc, objdump and ld) to find the library file. It returns the filename of the library file.”

meta/recipes-devtools/python/python3-libarchive-c_5.1.bb inherit pypi setuptools3 ptest · line 25

Quoted 7 lines
meta/recipes-devtools/python/python3-libarchive-c_5.1.bb +7 -0
+DEPENDS += "patchelf-native libarchive"
+# Avoid using find_library API which needs ldconfig and ld/objdump
+# https://docs.python.org/3/library/ctypes.html#ctypes-reference
+#
+do_configure:append() {
+    sed -i -e "s|find_library('archive')|'${libdir}/$(patchelf --print-soname ${STAGING_LIBDIR}/libarchive.so)'|" ${S}/libarchive/ffi.py
+}

Can we patch the source instead:

Quoted 8 lines

Review

Quoted 2 lines
(quoted diff) +1 -1
-libarchive_path = os.environ.get('LIBARCHIVE') or find_library('archive')
+libarchive_path = @LIBARCHIVE_SONAME@

This would mean we do not respect LIBARCHIVE envvar which is a documented way of calling
a non-default libarchive.so perhaps we can do

libarchive_path = os.environ.get('LIBARCHIVE') or @LIBARCHIVE_SONAME@

Yes, that works. Just something that would actually fail to apply instead of silently doing nothing if upstream rewrote the code.

Ross