Support Python freethreading
cover Support Python freethreading
This patchset allows turning on Python freethreading by using a new DISTRO_FEATURES setting:
DISTRO_FEATURES += "python_freethreading"
and optionally:
DISTRO_FEATURES_NATIVE += "python_freethreading"
The change to python3-dir.bbclass is crucial, as all python module build helper classes rely on it.
I also added native variants of some settings because another series against meta-openembedded is needed to complete this one. In that, the change to the fontforge recipe relies on the native setting for setting BUILD_LDFLAGS.
All of the changes included here were successfully build tested, except for the gdb-cross-canadian change. That one failed, but it's included for completeness.
Quite a few of python modules were build tested for native and target builds, including python3-pytorch from my meta-python-ai with reduced RDEPENDS because e.g. python3-onnx there failed to build with freethreading enabled.
All in all, this series at least allows to get things going. Though maybe not for Yocto 6.0, as it's quite late.
Best regards,
Zoltán Böszörményi
RFC PATCH 1/14 python3-dir.bbclass: Support python freethreading
With a new DISTRO_FEATURE ("python_freethreading"), set PYTHON_ABI.
Added a new variable called PYTHON_MAINVERSION which is a combination of PYTHON_BASEVERSION and PYTHON_ABI.
Set PYTHON_DIR so it also contains the ABI. Python build helper classes use this variable, so it's important to set it correctly.
Support different settings for target and native.
--- meta/classes-recipe/python3-dir.bbclass | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-)
meta/classes-recipe/python3-dir.bbclass +7 -2
@@ -5,7 +5,12 @@ # PYTHON_BASEVERSION = "3.14" -PYTHON_ABI = "" -PYTHON_DIR = "python${PYTHON_BASEVERSION}" +PYTHON_ABI = "${@bb.utils.contains('DISTRO_FEATURES', 'python_freethreading', 't', '', d)}" +PYTHON_ABI_NATIVE = "${@bb.utils.contains('DISTRO_FEATURES_NATIVE', 'python_freethreading', 't', '', d)}" +PYTHON_MAINVERSION = "${PYTHON_BASEVERSION}${PYTHON_ABI}" +PYTHON_MAINVERSION_NATIVE = "${PYTHON_BASEVERSION}${PYTHON_ABI_NATIVE}" +PYTHON_DIR = "python${PYTHON_MAINVERSION}" +PYTHON_DIR_NATIVE = "python${PYTHON_MAINVERSION_NATIVE}" PYTHON_PN = "python3" PYTHON_SITEPACKAGES_DIR = "${libdir}/${PYTHON_DIR}/site-packages" +PYTHON_SITEPACKAGES_DIR_NATIVE = "${libdir}/${PYTHON_DIR_NATIVE}/site-packages"
Signature · 1 line
2.53.0
RFC PATCH 2/14 python3: Adapt recipe to freethreading
In reply to Zoltán BöszörményiSet the previously added PACKAGECONFIG[freethreading] based on the new DISTRO_FEATURE=python_freethreading.
Now inherit python3-dir. The PYTHON_MAJMIN private variable was removed, and its usages were replaced with the proper variables from python3-dir.bbclass to simplify the settings and for consistency.
--- .../recipes-devtools/python/python3_3.14.3.bb | 61 ++++++++++--------- 1 file changed, 32 insertions(+), 29 deletions(-)
meta/recipes-devtools/python/python3_3.14.3.bb +32 -29
@@ -47,19 +47,17 @@ UPSTREAM_CHECK_REGEX = "[Pp]ython-(?P<pver>\d+(\.\d+)+).tar" CVE_PRODUCT = "python:python python_software_foundation:python cpython" -PYTHON_MAJMIN = "3.14" - S = "${UNPACKDIR}/Python-${PV}" BBCLASSEXTEND = "native nativesdk" -inherit autotools pkgconfig qemu ptest multilib_header update-alternatives +inherit autotools pkgconfig qemu python3-dir ptest multilib_header update-alternatives MULTILIB_SUFFIX = "${@d.getVar('base_libdir',1).split('/')[-1]}" ALTERNATIVE:${PN}-dev = "python3-config" -ALTERNATIVE_LINK_NAME[python3-config] = "${bindir}/python${PYTHON_MAJMIN}-config" -ALTERNATIVE_TARGET[python3-config] = "${bindir}/python${PYTHON_MAJMIN}-config-${MULTILIB_SUFFIX}" +ALTERNATIVE_LINK_NAME[python3-config] = "${bindir}/${PYTHON_DIR}-config" +ALTERNATIVE_TARGET[python3-config] = "${bindir}/${PYTHON_DIR}-config-${MULTILIB_SUFFIX}" DEPENDS = "\ autoconf-archive-native \
@@ -86,7 +84,7 @@ EXTRA_OECONF:append:class-native = " --bindir=${bindir}/${PN}" EXTRA_OECONF:append:class-target = " --with-build-python=nativepython3 PLATFORM_TRIPLET=${HOST_ARCH}-${HOST_OS}" EXTRA_OECONF:append:class-nativesdk = " --with-build-python=nativepython3" -export CROSSPYTHONPATH = "${STAGING_LIBDIR_NATIVE}/python${PYTHON_MAJMIN}/lib-dynload/" +export CROSSPYTHONPATH = "${STAGING_LIBDIR_NATIVE}/${PYTHON_DIR}/lib-dynload/" EXTRANATIVEPATH += "python3-native"
@@ -111,7 +109,10 @@ CACHED_CONFIGUREVARS:append:libc-musl = "\ " # PGO currently causes builds to not be reproducible so disable by default, see YOCTO #13407 -PACKAGECONFIG ??= "editline gdbm ${@bb.utils.filter('DISTRO_FEATURES', 'lto', d)}" +PACKAGECONFIG ??= "editline gdbm \ + ${@bb.utils.filter('DISTRO_FEATURES', 'lto', d)} \ + ${@bb.utils.contains('DISTRO_FEATURES', 'python_freethreading', 'freethreading', '', d)} \ +" PACKAGECONFIG[freethreading] = "--disable-gil,--enable-gil" PACKAGECONFIG[readline] = "--with-readline=readline,,readline,,,editline" PACKAGECONFIG[editline] = "--with-readline=editline,,libedit,,,readline"
@@ -178,7 +179,7 @@ do_install:prepend() { } do_install:append:class-target() { - oe_multilib_header python${PYTHON_MAJMIN}/pyconfig.h + oe_multilib_header ${PYTHON_DIR}/pyconfig.h } do_install:append:class-native() {
@@ -203,20 +204,20 @@ do_install:append:class-native() { # Nothing should be looking into ${B} for python3-native sed -i -e 's:${B}:/build/path/unavailable/:g' \ - ${D}/${libdir}/python${PYTHON_MAJMIN}/config-${PYTHON_MAJMIN}${PYTHON_ABI}*/Makefile + ${D}/${libdir}/${PYTHON_DIR}/config-${PYTHON_MAINVERSION}*/Makefile # disable the lookup in user's site-packages globally - sed -i 's#ENABLE_USER_SITE = None#ENABLE_USER_SITE = False#' ${D}${libdir}/python${PYTHON_MAJMIN}/site.py + sed -i 's#ENABLE_USER_SITE = None#ENABLE_USER_SITE = False#' ${D}${libdir}/${PYTHON_DIR}/site.py # python3-config needs to be in /usr/bin and not in a subdir of it to work properly mv ${D}/${bindir}/${PN}/python*config ${D}/${bindir}/ } do_install:append() { - for c in ${D}/${libdir}/python${PYTHON_MAJMIN}/_sysconfigdata*.py; do + for c in ${D}/${libdir}/${PYTHON_DIR}/_sysconfigdata*.py; do python3 ${UNPACKDIR}/reformat_sysconfig.py $c done - rm -f ${D}${libdir}/python${PYTHON_MAJMIN}/__pycache__/_sysconfigdata*.cpython* + rm -f ${D}${libdir}/${PYTHON_DIR}/__pycache__/_sysconfigdata*.cpython* mkdir -p ${D}${libdir}/python-sysconfigdata sysconfigfile=`find ${D} -name _sysconfig*.py`
@@ -224,8 +225,8 @@ do_install:append() { -e "s,^ 'LIBDIR'.*, 'LIBDIR': '${STAGING_LIBDIR}'\,,g" \ -e "s,^ 'INCLUDEDIR'.*, 'INCLUDEDIR': '${STAGING_INCDIR}'\,,g" \ -e "s,^ 'CONFINCLUDEDIR'.*, 'CONFINCLUDEDIR': '${STAGING_INCDIR}'\,,g" \ - -e "s,^ 'INCLUDEPY'.*, 'INCLUDEPY': '${STAGING_INCDIR}/python${PYTHON_MAJMIN}'\,,g" \ - -e "s,^ 'CONFINCLUDEPY'.*, 'CONFINCLUDEPY': '${STAGING_INCDIR}/python${PYTHON_MAJMIN}'\,,g" \ + -e "s,^ 'INCLUDEPY'.*, 'INCLUDEPY': '${STAGING_INCDIR}/${PYTHON_DIR}'\,,g" \ + -e "s,^ 'CONFINCLUDEPY'.*, 'CONFINCLUDEPY': '${STAGING_INCDIR}/${PYTHON_DIR}'\,,g" \ -e "s,${B},/build/path/unavailable/,g" \ $sysconfigfile cp $sysconfigfile ${D}${libdir}/python-sysconfigdata/_sysconfigdata.py
@@ -241,7 +242,7 @@ do_install:append:class-nativesdk () { for PYTHSCRIPT in `grep -rIl ${bindir}/python ${D}${bindir}`; do sed -i -e '1s|^#!.*|#!/usr/bin/env python3|' $PYTHSCRIPT done - create_wrapper ${D}${bindir}/python${PYTHON_MAJMIN} TERMINFO_DIRS='${sysconfdir}/terminfo:/etc/terminfo:/usr/share/terminfo:/usr/share/misc/terminfo:/lib/terminfo' PYTHONNOUSERSITE='1' + create_wrapper ${D}${bindir}/${PYTHON_DIR} TERMINFO_DIRS='${sysconfdir}/terminfo:/etc/terminfo:/usr/share/terminfo:/usr/share/misc/terminfo:/lib/terminfo' PYTHONNOUSERSITE='1' } do_install_ptest:append:class-target:libc-musl () {
@@ -257,7 +258,7 @@ SYSROOT_PREPROCESS_FUNCS:append:class-nativesdk = " provide_target_config_script provide_target_config_script() { install -d ${SYSROOT_DESTDIR}${prefix}/python-target-config/ install ${D}/${bindir}/python3-config ${SYSROOT_DESTDIR}/${prefix}/python-target-config/ - install ${D}/${bindir}/python${PYTHON_MAJMIN}-config ${SYSROOT_DESTDIR}/${prefix}/python-target-config/ + install ${D}/${bindir}/${PYTHON_DIR}-config ${SYSROOT_DESTDIR}/${prefix}/python-target-config/ } SYSROOT_DIRS += "${prefix}/python-target-config/"
@@ -280,13 +281,13 @@ py_package_preprocess () { -e 's:${RECIPE_SYSROOT_NATIVE}::g' \ -e 's:${RECIPE_SYSROOT}::g' \ -e 's:${BASE_WORKDIR}/${MULTIMACH_TARGET_SYS}::g' \ - ${PKGD}/${libdir}/python${PYTHON_MAJMIN}/config-${PYTHON_MAJMIN}${PYTHON_ABI}*/Makefile \ - ${PKGD}/${libdir}/python${PYTHON_MAJMIN}/_sysconfigdata*.py \ - ${PKGD}/${bindir}/python${PYTHON_MAJMIN}-config + ${PKGD}/${libdir}/${PYTHON_DIR}/config-${PYTHON_MAINVERSION}*/Makefile \ + ${PKGD}/${libdir}/${PYTHON_DIR}/_sysconfigdata*.py \ + ${PKGD}/${bindir}/${PYTHON_DIR}-config # Reformat _sysconfigdata after modifying it so that it remains # reproducible - for c in ${PKGD}/${libdir}/python${PYTHON_MAJMIN}/_sysconfigdata*.py; do + for c in ${PKGD}/${libdir}/${PYTHON_DIR}/_sysconfigdata*.py; do python3 ${UNPACKDIR}/reformat_sysconfig.py $c done
@@ -301,7 +302,7 @@ py_package_preprocess () { -c "from py_compile import compile; compile('$sysconfigfile', optimize=2)" cd - - mv ${PKGD}/${bindir}/python${PYTHON_MAJMIN}-config ${PKGD}/${bindir}/python${PYTHON_MAJMIN}-config-${MULTILIB_SUFFIX} + mv ${PKGD}/${bindir}/${PYTHON_DIR}-config ${PKGD}/${bindir}/${PYTHON_DIR}-config-${MULTILIB_SUFFIX} #Remove the unneeded copy of target sysconfig data rm -rf ${PKGD}/${libdir}/python-sysconfigdata
@@ -313,7 +314,7 @@ py_package_preprocess () { -e 's:${RECIPE_SYSROOT_NATIVE}::g' \ -e 's:${RECIPE_SYSROOT}::g' \ -e 's:${BASE_WORKDIR}/${MULTIMACH_TARGET_SYS}::g' \ - ${PKGD}/${libdir}/python${PYTHON_MAJMIN}/_sysconfig_vars*.json + ${PKGD}/${libdir}/${PYTHON_DIR}/_sysconfig_vars*.json } # We want bytecode precompiled .py files (.pyc's) by default
@@ -414,7 +415,7 @@ do_create_manifest() { cp ${UNPACKDIR}/get_module_deps3.py ${WORKDIR} cd ${WORKDIR} # This needs to be executed by python-native and NOT by HOST's python - nativepython3 create_manifest3.py ${PYTHON_MAJMIN} + nativepython3 create_manifest3.py ${PYTHON_MAINVERSION} cp python3-manifest.json.new ${THISDIR}/python3/python3-manifest.json }
@@ -429,11 +430,12 @@ RRECOMMENDS:${PN}-crypt:append:class-nativesdk = " ${MLPREFIX}openssl ${MLPREFIX # For historical reasons PN is empty and provided by python3-modules FILES:${PN} = "" +FILES:${PN}-core += "${bindir}/python${PYTHON_MAINVERSION} ${bindir}/python${PYTHON_BASEVERSION}" RPROVIDES:${PN}-modules = "${PN}" -FILES:${PN}-pydoc += "${bindir}/pydoc${PYTHON_MAJMIN} ${bindir}/pydoc3" -FILES:${PN}-idle += "${bindir}/idle3 ${bindir}/idle${PYTHON_MAJMIN}" -FILES:${PN}-tkinter += "${libdir}/python${PYTHON_MAJMIN}/lib-dynload/_tkinter.*.so" +FILES:${PN}-pydoc += "${bindir}/pydoc${PYTHON_MAINVERSION} ${bindir}/pydoc3" +FILES:${PN}-idle += "${bindir}/idle3 ${bindir}/idle${PYTHON_MAINVERSION}" +FILES:${PN}-tkinter += "${libdir}/${PYTHON_DIR}/lib-dynload/_tkinter.*.so" # provide python-pyvenv from python3-venv RPROVIDES:${PN}-venv += "${MLPREFIX}python3-pyvenv"
@@ -441,13 +443,14 @@ RPROVIDES:${PN}-venv += "${MLPREFIX}python3-pyvenv" # package libpython3 PACKAGES =+ "libpython3 libpython3-staticdev" FILES:libpython3 = "${libdir}/libpython*.so.*" -FILES:libpython3-staticdev += "${libdir}/python${PYTHON_MAJMIN}/config-${PYTHON_MAJMIN}-*/libpython${PYTHON_MAJMIN}.a" +FILES:libpython3-staticdev += "${libdir}/${PYTHON_DIR}/config-${PYTHON_MAINVERSION}-*/lib${PYTHON_DIR}.a" INSANE_SKIP:${PN}-dev += "dev-elf" INSANE_SKIP:${PN}-ptest = "dev-deps" # catch all the rest (unsorted) PACKAGES += "${PN}-misc" RDEPENDS:${PN}-misc += "\ + bash \ ${PN}-audio \ ${PN}-codecs \ ${PN}-core \
@@ -459,7 +462,7 @@ RDEPENDS:${PN}-misc += "\ RDEPENDS:${PN}-modules:append:class-target = " ${MLPREFIX}python3-misc" RDEPENDS:${PN}-modules:append:class-nativesdk = " ${MLPREFIX}python3-misc" RDEPENDS:${PN}-modules:append:class-target = " ${@bb.utils.contains('PACKAGECONFIG', 'gdbm', '${MLPREFIX}python3-gdbm', '', d)}" -FILES:${PN}-misc = "${libdir}/python${PYTHON_MAJMIN} ${libdir}/python${PYTHON_MAJMIN}/lib-dynload" +FILES:${PN}-misc = "${libdir}/${PYTHON_DIR} ${libdir}/${PYTHON_DIR}/lib-dynload" # catch manpage PACKAGES += "${PN}-man"
@@ -499,5 +502,5 @@ RDEPENDS:${PN}-tests:append:class-nativesdk = " ${MLPREFIX}bash" # Python's tests contain large numbers of files we don't need in the recipe sysroots SYSROOT_PREPROCESS_FUNCS += " py3_sysroot_cleanup" py3_sysroot_cleanup () { - rm -rf ${SYSROOT_DESTDIR}${libdir}/python${PYTHON_MAJMIN}/test + rm -rf ${SYSROOT_DESTDIR}${libdir}/${PYTHON_DIR}/test }
Signature · 1 line
2.53.0
RFC PATCH 3/14 python3native.bbclass: Fix settings
In reply to Zoltán BöszörményiWith PYTHON_DIR now containing PYTHON_ABI, don't use the latter.
--- meta/classes-recipe/python3native.bbclass | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
meta/classes-recipe/python3native.bbclass +2 -2
@@ -20,8 +20,8 @@ export STAGING_LIBDIR # find_package(PythonLibs REQUIRED) # which ends up using libs/includes from build host # Therefore pre-empt that effort -export PYTHON_LIBRARY = "${STAGING_LIBDIR}/lib${PYTHON_DIR}${PYTHON_ABI}.so" -export PYTHON_INCLUDE_DIR = "${STAGING_INCDIR}/${PYTHON_DIR}${PYTHON_ABI}" +export PYTHON_LIBRARY = "${STAGING_LIBDIR}/lib${PYTHON_DIR}.so" +export PYTHON_INCLUDE_DIR = "${STAGING_INCDIR}/${PYTHON_DIR}" # suppress host user's site-packages dirs. export PYTHONNOUSERSITE = "1"
Signature · 1 line
2.53.0
RFC PATCH 4/14 python_pyo3.bbclass: Pass ABI flag and PYTHON_MAINVERSION
In reply to Zoltán BöszörményiSetting the ABI flag correctly is necessary for modules using python_maturin.bbclass and others based on pyo3.
--- meta/classes-recipe/python_pyo3.bbclass | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
meta/classes-recipe/python_pyo3.bbclass +3 -1
@@ -12,7 +12,8 @@ inherit cargo python3-dir siteinfo export PYO3_CROSS = "1" -export PYO3_CROSS_PYTHON_VERSION = "${PYTHON_BASEVERSION}" +export PYO3_CROSS_PYTHON_VERSION = "${PYTHON_MAINVERSION}" +export ABIFLAGS = "${PYTHON_ABI}" export PYO3_CROSS_LIB_DIR = "${STAGING_LIBDIR}" export CARGO_BUILD_TARGET = "${RUST_HOST_SYS}" export RUSTFLAGS
@@ -25,6 +26,7 @@ implementation=CPython version=${PYTHON_BASEVERSION} shared=true abi3=false +abiflags=${PYTHON_ABI} lib_name=${PYTHON_DIR} lib_dir=${STAGING_LIBDIR} pointer_width=${SITEINFO_BITS}
Signature · 1 line
2.53.0
RFC PATCH 5/14 clang: Use PYTHON_MAINVERSION for python module versioning
In reply to Zoltán BöszörményiUse the new complete main version setting instead of just PYTHON_BASEVERSION that does not include PYTHON_ABI.
--- meta/recipes-devtools/clang/clang_git.bb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
meta/recipes-devtools/clang/clang_git.bb +1 -1
@@ -46,7 +46,7 @@ PACKAGECONFIG[lld] = "-DCLANG_DEFAULT_LINKER=lld,,," PACKAGECONFIG[lto] = "-DLLVM_ENABLE_LTO=Full -DLLVM_BINUTILS_INCDIR=${STAGING_INCDIR},,binutils," PACKAGECONFIG[thin-lto] = "-DLLVM_ENABLE_LTO=Thin -DLLVM_BINUTILS_INCDIR=${STAGING_INCDIR},,binutils," PACKAGECONFIG[unwindlib] = "-DCLANG_DEFAULT_UNWINDLIB=libunwind,-DCLANG_DEFAULT_UNWINDLIB=libgcc,," -PACKAGECONFIG[libclang-python] = "-DCLANG_PYTHON_BINDINGS_VERSIONS=${PYTHON_BASEVERSION},," +PACKAGECONFIG[libclang-python] = "-DCLANG_PYTHON_BINDINGS_VERSIONS=${PYTHON_MAINVERSION},," OECMAKE_SOURCEPATH = "${S}/clang"
Signature · 1 line
2.53.0
RFC PATCH 6/14 gdb, gdb-cross-canadian: Fix settings for python freethreading
In reply to Zoltán BöszörményiPYTHON_DIR now contains PYTHON_ABI, don't use the latter. Simplify the linked library name to use -l${PYTHON_DIR}.
--- meta/recipes-devtools/gdb/gdb-cross-canadian.inc | 4 ++-- meta/recipes-devtools/gdb/gdb_17.1.bb | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-)
meta/recipes-devtools/gdb/gdb-cross-canadian.inc +2 -2
@@ -22,8 +22,8 @@ do_configure:prepend() { cat > ${WORKDIR}/python << EOF #! /bin/sh case "\$2" in - --includes) echo "-I${STAGING_INCDIR}/${PYTHON_DIR}${PYTHON_ABI}/" ;; - --ldflags) echo "-Wl,-rpath-link,${STAGING_LIBDIR}/.. -Wl,-rpath,${libdir}/.. -lpthread -ldl -lutil -lm -lpython${PYTHON_BASEVERSION}${PYTHON_ABI}" ;; + --includes) echo "-I${STAGING_INCDIR}/${PYTHON_DIR}/" ;; + --ldflags) echo "-Wl,-rpath-link,${STAGING_LIBDIR}/.. -Wl,-rpath,${libdir}/.. -lpthread -ldl -lutil -lm -l${PYTHON_DIR}" ;; --exec-prefix) echo "${exec_prefix}" ;; *) exit 1 ;; esac
meta/recipes-devtools/gdb/gdb_17.1.bb +2 -2
@@ -26,8 +26,8 @@ do_configure:prepend() { cat > ${WORKDIR}/python << EOF #!/bin/sh case "\$2" in - --includes) echo "-I${STAGING_INCDIR}/${PYTHON_DIR}${PYTHON_ABI}/" ;; - --ldflags) echo "-Wl,-rpath-link,${STAGING_LIBDIR}/.. -Wl,-rpath,${libdir}/.. -lpthread -ldl -lutil -lm -lpython${PYTHON_BASEVERSION}${PYTHON_ABI}" ;; + --includes) echo "-I${STAGING_INCDIR}/${PYTHON_DIR}/" ;; + --ldflags) echo "-Wl,-rpath-link,${STAGING_LIBDIR}/.. -Wl,-rpath,${libdir}/.. -lpthread -ldl -lutil -lm -l${PYTHON_DIR}" ;; --exec-prefix) echo "${exec_prefix}" ;; *) exit 1 ;; esac
Signature · 1 line
2.53.0
RFC PATCH 7/14 boost: Fix settings for python freethreading
In reply to Zoltán BöszörményiPYTHON_DIR now contains PYTHON_ABI, fix the settings accordingly to still find both python builds.
--- meta/recipes-support/boost/boost.inc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
meta/recipes-support/boost/boost.inc +1 -1
@@ -174,7 +174,7 @@ do_configure() { # If we want Python then we need to tell Boost *exactly* where to find it if ${@bb.utils.contains('BOOST_LIBS', 'python', 'true', 'false', d)}; then - echo "using python : ${PYTHON_BASEVERSION} : ${STAGING_DIR_HOST}${bindir}/python3 : ${STAGING_DIR_HOST}${includedir}/${PYTHON_DIR}${PYTHON_ABI} : ${STAGING_DIR_HOST}${libdir}/${PYTHON_DIR} ;" >> ${WORKDIR}/user-config.jam + echo "using python : ${PYTHON_BASEVERSION} : ${STAGING_DIR_HOST}${bindir}/python3 : ${STAGING_DIR_HOST}${includedir}/${PYTHON_DIR} : ${STAGING_DIR_HOST}${libdir}/python${PYTHON_BASEVERSION} ;" >> ${WORKDIR}/user-config.jam fi if ${@bb.utils.contains('BOOST_LIBS', 'mpi', 'true', 'false', d)}; then
Signature · 1 line
2.53.0
RFC PATCH 8/14 lttng-tools: Simplify setting PYTHON_INCLUDE
In reply to Zoltán BöszörményiPYTHON_DIR now contains PYTHON_ABI, use it.
--- meta/recipes-kernel/lttng/lttng-tools_2.14.1.bb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
meta/recipes-kernel/lttng/lttng-tools_2.14.1.bb +1 -1
@@ -35,7 +35,7 @@ INSANE_SKIP:${PN}-ptest += "dev-deps" PYTHON_OPTION = "am_cv_python_pyexecdir='${PYTHON_SITEPACKAGES_DIR}' \ am_cv_python_pythondir='${PYTHON_SITEPACKAGES_DIR}' \ - PYTHON_INCLUDE='-I${STAGING_INCDIR}/python${PYTHON_BASEVERSION}${PYTHON_ABI}' \ + PYTHON_INCLUDE='-I${STAGING_INCDIR}/${PYTHON_DIR}' \ " PACKAGECONFIG ??= "${LTTNGUST} kmod" PACKAGECONFIG[python] = "--enable-python-bindings ${PYTHON_OPTION},,python3 swig-native"
Signature · 1 line
2.53.0
RFC PATCH 9/14 lttng-ust: Simplify setting PYTHON_INCLUDE
In reply to Zoltán BöszörményiPYTHON_DIR now contains PYTHON_ABI, use it.
--- meta/recipes-kernel/lttng/lttng-ust_2.14.0.bb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
meta/recipes-kernel/lttng/lttng-ust_2.14.0.bb +1 -1
@@ -8,7 +8,7 @@ LIC_FILES_CHKSUM = "file://LICENSE;md5=3b04e8f34dbcf08198c6618d05e8fe7b" PYTHON_OPTION = "am_cv_python_pyexecdir='${PYTHON_SITEPACKAGES_DIR}' \ am_cv_python_pythondir='${PYTHON_SITEPACKAGES_DIR}' \ - PYTHON_INCLUDE='-I${STAGING_INCDIR}/python${PYTHON_BASEVERSION}${PYTHON_ABI}' \ + PYTHON_INCLUDE='-I${STAGING_INCDIR}/${PYTHON_DIR}' \ " inherit autotools lib_package manpages python3native pkgconfig
Signature · 1 line
2.53.0
RFC PATCH 10/14 libcap-ng-python: Use PYTHON_DIR for FILES
In reply to Zoltán BöszörményiPYTHON_DIR now contains PYTHON_ABI, use it for consistency.
Move the installed site-packages directory to the correct location.
--- meta/recipes-support/libcap-ng/libcap-ng-python_0.9.1.bb | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)
meta/recipes-support/libcap-ng/libcap-ng-python_0.9.1.bb +4 -1
@@ -12,6 +12,9 @@ EXTRA_OECONF += "--with-python3" do_install() { oe_runmake 'DESTDIR=${D}' install -C ${B}/bindings/python3 + if [ ! -d ${D}${libdir}/${PYTHON_DIR} ]; then + mv ${D}${libdir}/python${PYTHON_BASEVERSION} ${D}${libdir}/${PYTHON_DIR} + fi } -FILES:${PN} = "${libdir}/python${PYTHON_BASEVERSION}" +FILES:${PN} = "${libdir}/${PYTHON_DIR}"
Signature · 1 line
2.53.0
RFC PATCH 11/14 hwlatdetect: Use PYTHON_DIR to support python freethreading
In reply to Zoltán Böszörményipython${PYTHON_BASEVERSION} is not enough now, PYTHON_DIR is needed for consistency.
--- meta/recipes-rt/rt-tests/hwlatdetect_git.bb | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
meta/recipes-rt/rt-tests/hwlatdetect_git.bb +3 -3
@@ -8,7 +8,7 @@ LIC_FILES_CHKSUM = "file://COPYING;md5=751419260aa954499f7abaabaa882bbe" require rt-tests.inc inherit python3-dir -EXTRA_OEMAKE += "PYLIB=${libdir}/python${PYTHON_BASEVERSION}/dist-packages" +EXTRA_OEMAKE += "PYLIB=${libdir}/${PYTHON_DIR}/dist-packages" do_compile() { oe_runmake hwlatdetect
@@ -18,9 +18,9 @@ do_install() { oe_runmake install_hwlatdetect DESTDIR=${D} SBINDIR=${sbindir} \ MANDIR=${mandir} INCLUDEDIR=${includedir} - sed -i -e '1s,#!.*python.*,#!${bindir}/python3,' ${D}${libdir}/python${PYTHON_BASEVERSION}/dist-packages/hwlatdetect.py + sed -i -e '1s,#!.*python.*,#!${bindir}/python3,' ${D}${libdir}/${PYTHON_DIR}/dist-packages/hwlatdetect.py } -FILES:${PN} += "${libdir}/python${PYTHON_BASEVERSION}/dist-packages/hwlatdetect.py" +FILES:${PN} += "${libdir}/${PYTHON_DIR}/dist-packages/hwlatdetect.py" RDEPENDS:${PN} = "python3-core " RRECOMMENDS:${PN} = "kernel-module-hwlat-detector"
Signature · 1 line
2.53.0
RFC PATCH 12/14 rpm: Support Python freethreading
In reply to Zoltán BöszörményiAdded a patch to stop using the stable CPython API (of 3.7!) as it prevents building rpm against a Python build with freethreading enabled.
--- .../files/0001-rpm-Don-t-use-stable-API.patch | 37 +++++++++++++++++++ meta/recipes-devtools/rpm/rpm_4.20.1.bb | 1 + 2 files changed, 38 insertions(+) create mode 100644 meta/recipes-devtools/rpm/files/0001-rpm-Don-t-use-stable-API.patch
meta/recipes-devtools/rpm/files/0001-rpm-Don-t-use-stable-API.patch new file +37 -0
@@ -0,0 +1,37 @@ +From 2fb202730b49538bbe6d1439d419c18a89eb4c91 Mon Sep 17 00:00:00 2001 +From: =?UTF-8?q?Zolt=C3=A1n=20B=C3=B6sz=C3=B6rm=C3=A9nyi?= + <[email protected]> +Date: Wed, 25 Mar 2026 08:20:13 +0100 +Subject: [PATCH] rpm: Don't use stable API +MIME-Version: 1.0 +Content-Type: text/plain; charset=UTF-8 +Content-Transfer-Encoding: 8bit + +With Python freethreading enabled, these explicit errors are +triggered in the Python headers: + + #error "The limited API is not currently supported in the free-threaded build" + +Signed-off-by: Zoltán Böszörményi <[email protected]> +Upstream-Status: Inappropriate [python freethreading specific] +--- + python/CMakeLists.txt | 3 --- + 1 file changed, 3 deletions(-) + +diff --git a/python/CMakeLists.txt b/python/CMakeLists.txt +index acc003ad6..3f21205c8 100644 +--- a/python/CMakeLists.txt ++++ b/python/CMakeLists.txt +@@ -18,9 +18,6 @@ Python3_add_library(_rpm + spec-py.c spec-py.h + ) + +-# Select Python stable ABI +-target_compile_definitions(_rpm PRIVATE Py_LIMITED_API=0x03070000) +- + target_link_libraries(_rpm PRIVATE librpmio librpm librpmbuild librpmsign) + + install(TARGETS _rpm +-- +2.53.0 +
meta/recipes-devtools/rpm/rpm_4.20.1.bb +1 -0
@@ -39,6 +39,7 @@ SRC_URI = "git://github.com/rpm-software-management/rpm;branch=rpm-4.20.x;protoc file://0002-rpmio-rpmglob.c-avoid-using-GLOB_BRACE-if-undefined-.patch \ file://0001-CMakeLists.txt-set-libdir-to-CMAKE_INSTALL_FULL_LIBD.patch \ file://0001-CMakeLists.txt-Fix-checking-for-CFLAGS.patch \ + file://0001-rpm-Don-t-use-stable-API.patch \ " PE = "1"
Signature · 1 line
2.53.0
RFC PATCH 13/14 xcb-proto: Support Python freethreading
In reply to Zoltán Böszörményixcb-proto installs its python module into the ABI-less site-packages directory, then in turn libxcb won't find it.
Move the installed site-packages directory to the correct location.
--- meta/recipes-graphics/xorg-proto/xcb-proto_1.17.0.bb | 6 ++++++ 1 file changed, 6 insertions(+)
meta/recipes-graphics/xorg-proto/xcb-proto_1.17.0.bb +6 -0
@@ -19,6 +19,12 @@ SRC_URI[sha256sum] = "2c1bacd2110f4799f74de6ebb714b94cf6f80fb112316b1219480fd225 inherit autotools pkgconfig python3native +do_install:append () { + if [ ! -d ${D}${libdir}/${PYTHON_DIR} ]; then + mv ${D}${libdir}/python${PYTHON_BASEVERSION} ${D}${libdir}/${PYTHON_DIR} + fi +} + PACKAGES += "python-xcbgen" FILES:${PN} = ""
Signature · 1 line
2.53.0
RFC PATCH 14/14 libxcb: Support Python freethreading
In reply to Zoltán BöszörményiAdded xcb-proto-native as a new build dependency and inherit python3native.
This fixes the build issues when Python freethreading is enabled.
--- meta/recipes-graphics/xorg-lib/libxcb_1.17.0.bb | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-)
meta/recipes-graphics/xorg-lib/libxcb_1.17.0.bb +2 -4
@@ -16,19 +16,17 @@ SRC_URI[sha256sum] = "599ebf9996710fea71622e6e184f3a8ad5b43d0e5fa8c4e407123c88a5 BBCLASSEXTEND = "native nativesdk" -DEPENDS = "xcb-proto xorgproto libxau libpthread-stubs libxdmcp" +DEPENDS = "xcb-proto-native xcb-proto xorgproto libxau libpthread-stubs libxdmcp" PACKAGES_DYNAMIC = "^${PN}-.*" FILES:${PN} = "${libdir}/libxcb.so.*" -inherit autotools pkgconfig features_check +inherit autotools pkgconfig python3native features_check # The libxau and others requires x11 in DISTRO_FEATURES REQUIRED_DISTRO_FEATURES = "x11" -export PYTHON = "python3" - do_install:append () { chown root.root ${D}${datadir}/doc/${BPN}/tutorial -R }
Signature · 1 line
2.53.0
Quoted 4 lines · Zoltán Böszörményi
This patchset allows turning on Python freethreading by using a new
DISTRO_FEATURESsetting:
Quoted 8 lines
DISTRO_FEATURES+= "python_freethreading"and optionally:
DISTRO_FEATURES_NATIVE+= "python_freethreading"The change to python3-dir.bbclass is crucial, as all python module build helper classes rely on it.
This is fine as an RFC, but otherwise, given the large amount of tweaks required, and having to choose between one or the other (which complicates testing), I would wait and see how other distros provide a free-threading python. Ideally it should be co-installable with regular python.
Alex
2026. 03. 25. 13:58 keltezéssel, Alexander Kanavin írta:
Quoted 10 lines · Zoltán Böszörményi
Quoted 9 lines
This patchset allows turning on Python freethreading by using a new
DISTRO_FEATURESsetting:
DISTRO_FEATURES+= "python_freethreading"and optionally:
DISTRO_FEATURES_NATIVE+= "python_freethreading"
Quoted 7 lines
Quoted 2 lines
The change to python3-dir.bbclass is crucial, as all python module build helper classes rely on it.
This is fine as an RFC, but otherwise, given the large amount of tweaks required, and having to choose between one or the other (which complicates testing), I would wait and see how other distros provide a free-threading python. Ideally it should be co-installable with regular python.
As I said, this would double the amount of python module recipe builds.
See the Red Hat bugzilla ticket I created for a different reason (old bitbake needs old python, still a parallel installation use case):
https://bugzilla.redhat.com/show_bug.cgi?id=2418374
"We don't have the human power to maintain several separate Python stacks packaged as RPM packages in Fedora."
Python 3.14.x is present in Fedora 43, only without freethreading.
I expect they won't do a parallel installation with freethreading, either, due to the same reason. When they switch, it will be all or nothing.
Quoted 9 lines · Zoltan Boszormenyi via lists.openembedded.org
This patchset allows turning on Python freethreading by using a new
DISTRO_FEATURESsetting:
DISTRO_FEATURES+= "python_freethreading"and optionally:
DISTRO_FEATURES_NATIVE+= "python_freethreading"
Quoted 8 lines
The change to python3-dir.bbclass is crucial, as all python module build helper classes rely on it.
I also added native variants of some settings because another series against meta-openembedded is needed to complete this one. In that, the change to the fontforge recipe relies on the native setting for setting
BUILD_LDFLAGS.
Looking at those other patches in the series, they don't seem to be conditional on the distro feature.
Are those changes valid for a build with and without freethreading ?
That's the type of information, along with the benefits of enabling this would be information expected in a 0/N introductory patch.
Cheers,
Bruce
Quoted 27 lines
All of the changes included here were successfully build tested, except for the gdb-cross-canadian change. That one failed, but it's included for completeness.
Quite a few of python modules were build tested for native and target builds, including python3-pytorch from my meta-python-ai with reduced RDEPENDS because e.g. python3-onnx there failed to build with freethreading enabled.
All in all, this series at least allows to get things going. Though maybe not for Yocto 6.0, as it's quite late.
Best regards,
Zoltán Böszörményi-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group. View/Reply Online (#233874):
https://lists.openembedded.org/g/openembedded-core/message/233874
Mute This Topic: https://lists.openembedded.org/mt/118499141/1050810
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [ [email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
Signature · 3 lines
- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end - "Use the force Harry" - Gandalf, Star Trek II
2026. 03. 25. 14:01 keltezéssel, Bruce Ashfield írta:
Quoted 22 lines · On Wed, Mar 25, 2026 at 7:31 AM Zoltan Boszormenyi via lists.openembedded.org…
This patchset allows turning on Python freethreading by using a new
DISTRO_FEATURESsetting:
DISTRO_FEATURES+= "python_freethreading"and optionally:
DISTRO_FEATURES_NATIVE+= "python_freethreading"The change to python3-dir.bbclass is crucial, as all python module build helper classes rely on it.
I also added native variants of some settings because another series against meta-openembedded is needed to complete this one. In that, the change to the fontforge recipe relies on the native setting for setting
BUILD_LDFLAGS.
Quoted 4 lines
Looking at those other patches in the series, they don't seem to be conditional on the distro feature.
Are those changes valid for a build with and without freethreading ?
Yes!
The main change is in python3-dir.bbclass, and the recipe tweaks are just adaptations to the fact that only ${PYTHON_DIR} is needed, not ${PYTHON_DIR}${PYTHON_ABI} from that point.
I was thinking that the introductory mail stressed this enough.
FWIW, less than 30 recipes was spelling the latter out in openembedded-core and meta-openembedded combined.
Quoted 40 lines
That's the type of information, along with the benefits of enabling this would be information expected in a 0/N introductory patch.
Cheers,
Bruce
All of the changes included here were successfully build tested, except for the gdb-cross-canadian change. That one failed, but it's included for completeness.
Quite a few of python modules were build tested for native and target builds, including python3-pytorch from my meta-python-ai with reduced RDEPENDS because e.g. python3-onnx there failed to build with freethreading enabled.
All in all, this series at least allows to get things going. Though maybe not for Yocto 6.0, as it's quite late.
Best regards,
Zoltán Böszörményi-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group. View/Reply Online (#233874):
https://lists.openembedded.org/g/openembedded-core/message/233874
Mute This Topic: https://lists.openembedded.org/mt/118499141/1050810
Group Owner: [email protected] <mailto:openembedded-core%[email protected]>
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-Signature · 2 lines
- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end - "Use the force Harry" - Gandalf, Star Trek II
Quoted 31 lines · Böszörményi Zoltán
2026. 03. 25. 14:01 keltezéssel, Bruce Ashfield írta:
Quoted 3 lines
On Wed, Mar 25, 2026 at 7:31 AM Zoltan Boszormenyi via
lists.openembedded.org
Quoted 1 line
<http://lists.openembedded.org> <zboszor=
Quoted 19 lines · [email protected]> wrote:
This patchset allows turning on Python freethreading by using a new
DISTRO_FEATURESsetting:
DISTRO_FEATURES+= "python_freethreading"and optionally:
DISTRO_FEATURES_NATIVE+= "python_freethreading"The change to python3-dir.bbclass is crucial, as all python module build helper classes rely on it.
I also added native variants of some settings because another series against meta-openembedded is needed to complete this one. In that, the change to the fontforge recipe relies on the native setting for setting
BUILD_LDFLAGS.Quoted 4 lines
Looking at those other patches in the series, they don't seem to be conditional on the distro feature.
Are those changes valid for a build with and without freethreading ?
Quoted 7 lines
Yes!
The main change is in python3-dir.bbclass, and the recipe tweaks are just adaptations to the fact that only ${PYTHON_DIR} is needed, not ${PYTHON_DIR}${PYTHON_ABI} from that point.
I was thinking that the introductory mail stressed this enough.
Obviously not :)
I didn't get that impression at all from reading the introductory message or looking at the patches themselves.
You have the overview knowledge of the feature, why it is useful, how it impacts existing recipes, how the migration would work, what is the compatibility, etc.
The patch reader doesn't, so that all needs to be clearly supplied.
Bruce
Quoted 49 lines
FWIW, less than 30 recipes was spelling the latter out in openembedded-core and meta-openembedded combined.
Quoted 23 lines
That's the type of information, along with the benefits of enabling this would be information expected in a 0/N introductory patch.
Cheers,
Bruce
All of the changes included here were successfully build tested, except for the gdb-cross-canadian change. That one failed, but it's included for completeness.
Quite a few of python modules were build tested for native and target builds, including python3-pytorch from my meta-python-ai with reduced RDEPENDS because e.g. python3-onnx there failed to build with freethreading enabled.
All in all, this series at least allows to get things going. Though maybe not for Yocto 6.0, as it's quite late.
Best regards,
Zoltán BöszörményiQuoted 8 lines
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group. View/Reply Online (#233874):
https://lists.openembedded.org/g/openembedded-core/message/233874
Mute This Topic: https://lists.openembedded.org/mt/118499141/1050810
Group Owner: [email protected] <mailto:openembedded-core%[email protected]> Unsubscribe:https://lists.openembedded.org/g/openembedded-core/unsub
Quoted 7 lines
[[email protected]] -=-=-=-=-=-=-=-=-=-=-=---
- Thou shalt not follow the NULL pointer, for chaos and madness awaitthee at its end
Quoted 2 lines
- "Use the force Harry" - Gandalf, Star Trek II
Signature · 3 lines
- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end - "Use the force Harry" - Gandalf, Star Trek II
2026. 03. 25. 14:17 keltezéssel, Bruce Ashfield írta:
Quoted 36 lines · Böszörményi Zoltán
2026. 03. 25. 14:01 keltezéssel, Bruce Ashfield írta:
>
>
> On Wed, Mar 25, 2026 at 7:31 AM Zoltan Boszormenyi via lists.openembedded.org
<http://lists.openembedded.org>
> <http://lists.openembedded.org> <[email protected]> wrote:
>
> This patchset allows turning on Python freethreading by using
> a newDISTRO_FEATURESsetting:
>
> DISTRO_FEATURES += "python_freethreading"
>
> and optionally:
>
> DISTRO_FEATURES_NATIVE += "python_freethreading"
>
> The change to python3-dir.bbclass is crucial, as all python module > build helper classes rely on it.
>
> I also added native variants of some settings because another
> series against meta-openembedded is needed to complete this one. > In that, the change to the fontforge recipe relies on the native > setting for settingBUILD_LDFLAGS.
>
>
> Looking at those other patches in the series, they don't seem to
> be conditional on the distro feature.
>
> Are those changes valid for a build with and without freethreading ?Yes!
Quoted 8 lines
The main change is in python3-dir.bbclass, and the recipe tweaks are just adaptations to the fact that only ${PYTHON_DIR} is needed, not ${PYTHON_DIR}${PYTHON_ABI} from that point.
I was thinking that the introductory mail stressed this enough.
Obviously not :)
Obviously. :)
Quoted 6 lines
I didn't get that impression at all from reading the introductory message or looking at the patches themselves.
You have the overview knowledge of the feature, why it is useful, how it impacts existing recipes, how the migration would work, what is the compatibility, etc.
I think the Python 3.14 release notes has enough details. See also https://www.phoronix.com/news/Python-3.14
With the two series I sent, only a single recipe is left (cmpi-bindings) in openembedded-core and meta-openembedded that uses
the PYTHON_ABI setting explicitly.
Migration (of python scripts) should not be a problem. If it worked with 3.14 as is, it works with freethreading enabled.
Compatibility is poor for a lot of modules, that clearly spell this out in their pyproject.toml or PKG-INFO:
"Programming Language :: Python :: Free Threading :: 1 - Unstable"
or something not quite production level support.
It's still experimental for 3rd party modules, despite being officially supported for the interpreter and built-in modules in 3.14.
This is the reason this was an RFC series.
Anyway, the reason I was experimenting with this was that with heavily threaded python code, an import storm can cause a crash in 3.12.11 or the manually updated 3.13.12 in Yocto 5.3 in importlib._bootstrap according to the backtrace.
I would like to see if Python 3.14 (with and without freethreading) fixes such crashes.
Quoted 2 lines
The patch reader doesn't, so that all needs to be clearly supplied.
Maybe next time for v2 after I have a complete image successfully built with freethreading enabled.
Quoted 57 lines
Bruce
FWIW, less than 30 recipes was spelling the latter out in openembedded-core and meta-openembedded combined.
> That's the type of information, along with the benefits of enabling
> this would be information expected in a 0/N introductory patch.
>
> Cheers,
>
> Bruce
>
>
> All of the changes included here were successfully build tested,
> except for the gdb-cross-canadian change. That one failed, but
> it's included for completeness.
>
> Quite a few of python modules were build tested for native and
> target builds, including python3-pytorch from my meta-python-ai
> with reduced RDEPENDS because e.g. python3-onnx there failed
> to build with freethreading enabled.
>
> All in all, this series at least allows to get things going.
> Though maybe not for Yocto 6.0, as it's quite late.
>
> Best regards,
> Zoltán Böszörményi
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#233874):
> https://lists.openembedded.org/g/openembedded-core/message/233874
> Mute This Topic: https://lists.openembedded.org/mt/118499141/1050810
> Group Owner: [email protected] <mailto:openembedded-core%[email protected]>
> <mailto:openembedded-core%[email protected] <mailto:openembedded-core%[email protected]>>
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
> [[email protected]]
> -=-=-=-=-=-=-=-=-=-=-=-
>
>
>
> --
> - Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end > - "Use the force Harry" - Gandalf, Star Trek II
>Signature · 2 lines
- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end - "Use the force Harry" - Gandalf, Star Trek II
Quoted 58 lines · Böszörményi Zoltán
2026. 03. 25. 14:17 keltezéssel, Bruce Ashfield írta:
Quoted 3 lines
On Wed, Mar 25, 2026 at 9:13 AM Böszörményi Zoltán <[email protected]>
Quoted 5 lines · wrote:
2026. 03. 25. 14:01 keltezéssel, Bruce Ashfield írta: > > > On Wed, Mar 25, 2026 at 7:31 AM Zoltan Boszormenyi vialists.openembedded.org
Quoted 2 lines
<http://lists.openembedded.org>
> <http://lists.openembedded.org> <zboszor=Quoted 11 lines · [email protected]> wrote:
Quoted 11 lines
This patchset allows turning on Python freethreading by using a new
DISTRO_FEATURESsetting:
DISTRO_FEATURES+= "python_freethreading"and optionally:
DISTRO_FEATURES_NATIVE+= "python_freethreading"The change to python3-dir.bbclass is crucial, as all python
module
Quoted 4 lines
Quoted 4 lines
build helper classes rely on it.
I also added native variants of some settings because another series against meta-openembedded is needed to complete this
one.
Quoted 1 line
Quoted 1 line
In that, the change to the fontforge recipe relies on the
native
Quoted 7 lines
Quoted 7 lines
setting for setting
BUILD_LDFLAGS.Looking at those other patches in the series, they don't seem to be conditional on the distro feature.
Are those changes valid for a build with and without freethreading
?
Quoted 11 lines
Yes!
The main change is in python3-dir.bbclass, and the recipe tweaks are just adaptations to the fact that only ${PYTHON_DIR} is needed, not ${PYTHON_DIR}${PYTHON_ABI} from that point.
I was thinking that the introductory mail stressed this enough.
Obviously not :)
Obviously. :)
Quoted 3 lines
I didn't get that impression at all from reading the introductory message or looking at the patches themselves.
Quoted 6 lines
Quoted 3 lines
You have the overview knowledge of the feature, why it is useful, how it impacts existing recipes, how the migration would work, what is the compatibility, etc.
I think the Python 3.14 release notes has enough details. See also https://www.phoronix.com/news/Python-3.14
The information needs to be distilled into your commit, not left to the patch reader to go off and read release notes. That's the point I'm trying to make.
Quoted 20 lines
With the two series I sent, only a single recipe is left (cmpi-bindings) in openembedded-core and meta-openembedded that uses
thePYTHON_ABIsetting explicitly.Migration (of python scripts) should not be a problem. If it worked with 3.14 as is, it works with freethreading enabled.
Compatibility is poor for a lot of modules, that clearly spell this out in their pyproject.toml or PKG-INFO:
"Programming Language :: Python :: Free Threading :: 1 - Unstable"
or something not quite production level support.
It's still experimental for 3rd party modules, despite being officially supported for the interpreter and built-in modules in 3.14.This is the reason this was an RFC series.
Quoted 8 lines
Anyway, the reason I was experimenting with this was that with heavily threaded python code, an import storm can cause a crash in 3.12.11 or the manually updated 3.13.12 in Yocto 5.3 in importlib._bootstrap according to the backtrace.
I would like to see if Python 3.14 (with and without freethreading) fixes such crashes.
Definitely worth putting in the RFC as well, the motivation for the change matters!
Quoted 7 lines
Quoted 3 lines
The patch reader doesn't, so that all needs to be clearly supplied.
Maybe next time for v2 after I have a complete image successfully built with freethreading enabled.
Ack'd
Bruce
Quoted 65 lines
Quoted 12 lines
Bruce
FWIW, less than 30 recipes was spelling the latter out in openembedded-core and meta-openembedded combined.
> That's the type of information, along with the benefits of enabling > this would be information expected in a 0/N introductory patch. >
Quoted 6 lines
> Cheers,
>
> Bruce
>
>
> All of the changes included here were successfully buildtested,
Quoted 11 lines
Quoted 11 lines
except for the gdb-cross-canadian change. That one failed, but it's included for completeness.
Quite a few of python modules were build tested for native and target builds, including python3-pytorch from my meta-python-ai with reduced RDEPENDS because e.g. python3-onnx there failed to build with freethreading enabled.
All in all, this series at least allows to get things going. Though maybe not for Yocto 6.0, as it's quite late.
Quoted 8 lines
Quoted 8 lines
Best regards, Zoltán Böszörményi -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#233874):https://lists.openembedded.org/g/openembedded-core/message/233874
Mute This Topic:https://lists.openembedded.org/mt/118499141/1050810
Quoted 5 lines
Quoted 1 line
Group Owner: [email protected]
<mailto:openembedded-core%[email protected]>
> <mailto:openembedded-core%[email protected]
<mailto:openembedded-core%[email protected]>>
> Unsubscribe:https://lists.openembedded.org/g/openembedded-core/unsub
Quoted 7 lines
Quoted 7 lines
[[email protected]] -=-=-=-=-=-=-=-=-=-=-=---
- Thou shalt not follow the NULL pointer, for chaos and madnessawait thee at its end
Quoted 7 lines
Quoted 2 lines
- "Use the force Harry" - Gandalf, Star Trek II
--
- Thou shalt not follow the NULL pointer, for chaos and madness awaitthee at its end
Quoted 2 lines
- "Use the force Harry" - Gandalf, Star Trek II
Signature · 3 lines
- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end - "Use the force Harry" - Gandalf, Star Trek II
2026. 03. 25. 15:17 keltezéssel, Bruce Ashfield írta:
Quoted 102 lines · Böszörményi Zoltán
2026. 03. 25. 14:17 keltezéssel, Bruce Ashfield írta:
>
>
> On Wed, Mar 25, 2026 at 9:13 AM Böszörményi Zoltán <[email protected]> wrote:
>
> 2026. 03. 25. 14:01 keltezéssel, Bruce Ashfield írta:
> >
> >
> > On Wed, Mar 25, 2026 at 7:31 AM Zoltan Boszormenyi via lists.openembedded.org <http://lists.openembedded.org>
> <http://lists.openembedded.org>
> > <http://lists.openembedded.org> <[email protected]> wrote:
> >
> > This patchset allows turning on Python freethreading by using > > a newDISTRO_FEATURESsetting:
> >
> > DISTRO_FEATURES += "python_freethreading"
> >
> > and optionally:
> >
> > DISTRO_FEATURES_NATIVE += "python_freethreading"
> >
> > The change to python3-dir.bbclass is crucial, as all python module > > build helper classes rely on it.
> >
> > I also added native variants of some settings because another > > series against meta-openembedded is needed to complete this one. > > In that, the change to the fontforge recipe relies on the native > > setting for settingBUILD_LDFLAGS.
> >
> >
> > Looking at those other patches in the series, they don't seem to > > be conditional on the distro feature.
> >
> > Are those changes valid for a build with and without freethreading ? >
> Yes!
>
> The main change is in python3-dir.bbclass, and the recipe tweaks > are just adaptations to the fact that only ${PYTHON_DIR} is needed, > not ${PYTHON_DIR}${PYTHON_ABI} from that point.
>
> I was thinking that the introductory mail stressed this enough. >
>
> Obviously not :)Obviously. :)
> I didn't get that impression at all from reading the introductory > message or looking at the patches themselves.
>
> You have the overview knowledge of the feature, why it is > useful, how it impacts existing recipes, how the migration would > work, what is the compatibility, etc.I think the Python 3.14 release notes has enough details. See also https://www.phoronix.com/news/Python-3.14
The information needs to be distilled into your commit, not left to the patch reader to go off and read release notes. That's the point I'm trying to make.
With the two series I sent, only a single recipe is left (cmpi-bindings) in openembedded-core and meta-openembedded that uses
thePYTHON_ABIsetting explicitly.Migration (of python scripts) should not be a problem. If it worked with 3.14 as is, it works with freethreading enabled.
Compatibility is poor for a lot of modules, that clearly spell this out in their pyproject.toml or PKG-INFO:
"Programming Language :: Python :: Free Threading :: 1 - Unstable"
or something not quite production level support.
It's still experimental for 3rd party modules, despite being officially supported for the interpreter and built-in modules in 3.14.This is the reason this was an RFC series.
Anyway, the reason I was experimenting with this was that with heavily threaded python code, an import storm can cause a crash in 3.12.11 or the manually updated 3.13.12 in Yocto 5.3 in importlib._bootstrap according to the backtrace.
I would like to see if Python 3.14 (with and without freethreading) fixes such crashes.
Definitely worth putting in the RFC as well, the motivation for the change matters!
Quoted 6 lines
Quoted 2 lines
The patch reader doesn't, so that all needs to be clearly supplied.
Maybe next time for v2 after I have a complete image successfully built with freethreading enabled.
No such luck.
lldb in clang won't build because its swig source wants to play games with PyObject->ob_refcnt directly instead of using the proper macros. That struct member does not exist when freethreading is enabled. In fact, the whole struct _object (the PyObject implementation) is different when freethreading is enabled.
The llvm sources, even in the newly released 22.x and main do not have any reference to the Py_GIL_DISABLED macro, so as a whole, it does not support Python freethreading.
I can revisit this series a few year later when llvm adds the necessary support, which can be backported to older clang versions in my meta-clang-revival.
This concludes my little experiment.
That being said, I think the cleanup in the recipes is a good direction. PYTHON_DIR should include PYTHON_ABI in python3-dir.bbclass because build helper classes rely on it. Then PYTHON_DIR must be used consistently in recipes that need it, even if PYTHON_ABI stays empty.
Quoted 75 lines
Ack'd
Bruce
Quoted 29 lines
Bruce
FWIW, less than 30 recipes was spelling the latter out in openembedded-core and meta-openembedded combined.
> That's the type of information, along with the benefits of enabling > this would be information expected in a 0/N introductory patch. >
> Cheers,
>
> Bruce
>
>
> All of the changes included here were successfully build tested, > except for the gdb-cross-canadian change. That one failed, but > it's included for completeness.
>
> Quite a few of python modules were build tested for native and > target builds, including python3-pytorch from my meta-python-ai > with reduced RDEPENDS because e.g. python3-onnx there failed > to build with freethreading enabled.
>
> All in all, this series at least allows to get things going. > Though maybe not for Yocto 6.0, as it's quite late. >
> Best regards,Quoted 8 lines
> Zoltán Böszörményi
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#233874):
> https://lists.openembedded.org/g/openembedded-core/message/233874
> Mute This Topic: https://lists.openembedded.org/mt/118499141/1050810
> Group Owner: [email protected]<mailto:openembedded-core%[email protected]>
> <mailto:openembedded-core%[email protected] <mailto:openembedded-core%[email protected]>>
> > <mailto:openembedded-core%[email protected] <mailto:openembedded-core%[email protected]>
> <mailto:openembedded-core%[email protected] <mailto:openembedded-core%[email protected]>>>
> > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
> > [[email protected]]
> > -=-=-=-=-=-=-=-=-=-=-=-
> >
> >
> >
> > --
> > - Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end
> > - "Use the force Harry" - Gandalf, Star Trek II
> >
>
>
>
> --
> - Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end > - "Use the force Harry" - Gandalf, Star Trek II
>Signature · 2 lines
- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end - "Use the force Harry" - Gandalf, Star Trek II
A message that is not in the archive (sent off-list or not yet received): [email protected]
2026. 03. 26. 6:29 keltezéssel, Zoltan Boszormenyi via lists.openembedded.org írta:
Quoted 126 lines
2026. 03. 25. 15:17 keltezéssel, Bruce Ashfield írta:
Quoted 102 lines · Böszörményi Zoltán
2026. 03. 25. 14:17 keltezéssel, Bruce Ashfield írta:
>
>
> On Wed, Mar 25, 2026 at 9:13 AM Böszörményi Zoltán <[email protected]> wrote:
>
> 2026. 03. 25. 14:01 keltezéssel, Bruce Ashfield írta:
> >
> >
> > On Wed, Mar 25, 2026 at 7:31 AM Zoltan Boszormenyi via lists.openembedded.org <http://lists.openembedded.org>
> <http://lists.openembedded.org>
> > <http://lists.openembedded.org> <[email protected]> wrote:
> >
> > This patchset allows turning on Python freethreading by using > > a newDISTRO_FEATURESsetting:
> >
> > DISTRO_FEATURES += "python_freethreading"
> >
> > and optionally:
> >
> > DISTRO_FEATURES_NATIVE += "python_freethreading"
> >
> > The change to python3-dir.bbclass is crucial, as all python module > > build helper classes rely on it.
> >
> > I also added native variants of some settings because another > > series against meta-openembedded is needed to complete this one. > > In that, the change to the fontforge recipe relies on the native > > setting for settingBUILD_LDFLAGS.
> >
> >
> > Looking at those other patches in the series, they don't seem to > > be conditional on the distro feature.
> >
> > Are those changes valid for a build with and without freethreading ? >
> Yes!
>
> The main change is in python3-dir.bbclass, and the recipe tweaks > are just adaptations to the fact that only ${PYTHON_DIR} is needed, > not ${PYTHON_DIR}${PYTHON_ABI} from that point.
>
> I was thinking that the introductory mail stressed this enough. >
>
> Obviously not :)Obviously. :)
> I didn't get that impression at all from reading the introductory > message or looking at the patches themselves.
>
> You have the overview knowledge of the feature, why it is > useful, how it impacts existing recipes, how the migration would > work, what is the compatibility, etc.I think the Python 3.14 release notes has enough details. See also https://www.phoronix.com/news/Python-3.14
The information needs to be distilled into your commit, not left to the patch reader to go off and read release notes. That's the point I'm trying to make.
With the two series I sent, only a single recipe is left (cmpi-bindings) in openembedded-core and meta-openembedded that uses
thePYTHON_ABIsetting explicitly.Migration (of python scripts) should not be a problem. If it worked with 3.14 as is, it works with freethreading enabled.
Compatibility is poor for a lot of modules, that clearly spell this out in their pyproject.toml or PKG-INFO:
"Programming Language :: Python :: Free Threading :: 1 - Unstable"
or something not quite production level support.
It's still experimental for 3rd party modules, despite being officially supported for the interpreter and built-in modules in 3.14.This is the reason this was an RFC series.
Anyway, the reason I was experimenting with this was that with heavily threaded python code, an import storm can cause a crash in 3.12.11 or the manually updated 3.13.12 in Yocto 5.3 in importlib._bootstrap according to the backtrace.
I would like to see if Python 3.14 (with and without freethreading) fixes such crashes.
Definitely worth putting in the RFC as well, the motivation for the change matters!
Quoted 6 lines
> The patch reader doesn't, so that all needs to be clearly > supplied.
Maybe next time for v2 after I have a complete image successfully built with freethreading enabled.
No such luck.
lldb in clang won't build because its swig source wants to play games with
PyObject->ob_refcntdirectly instead of using the proper macros. That struct member does not exist when freethreading is enabled. In fact, the whole struct_object(the PyObject implementation) is different when freethreading is enabled.The llvm sources, even in the newly released 22.x and main do not have any reference to the
Py_GIL_DISABLEDmacro, so as a whole, it does not support Python freethreading.I can revisit this series a few year later when llvm adds the necessary support, which can be backported to older clang versions in my meta-clang-revival.
Quoted 6 lines
This concludes my little experiment.
That being said, I think the cleanup in the recipes is a good direction.
PYTHON_DIRshould includePYTHON_ABIin python3-dir.bbclass because build helper classes rely on it. ThenPYTHON_DIRmust be used consistently in recipes that need it, even ifPYTHON_ABIstays empty.
Which means I can re-send the whole series as a cleanup or preparation series without the DISTRO_FEATURES enablement part in python3-dir.bbclass.
That one can stay RFC as a standalone patch, separate from the cleanup series.
I will re-test my build without enabling freethreading before I would do that.
Quoted 71 lines
Quoted 46 lines
Ack'd
Bruce
>
> Bruce
>
>
>
> FWIW, less than 30 recipes was spelling the latter out in
> openembedded-core and meta-openembedded combined.
>
> > That's the type of information, along with the benefits of enabling > > this would be information expected in a 0/N introductory patch. > >
> > Cheers,
> >
> > Bruce
> >
> >
> > All of the changes included here were successfully build tested, > > except for the gdb-cross-canadian change. That one failed, but > > it's included for completeness.
> >
> > Quite a few of python modules were build tested for native and > > target builds, including python3-pytorch from my meta-python-ai > > with reduced RDEPENDS because e.g. python3-onnx there failed > > to build with freethreading enabled.
> >
> > All in all, this series at least allows to get things going. > > Though maybe not for Yocto 6.0, as it's quite late.
> >
> > Best regards,
> > Zoltán Böszörményi
> >
> >
> >
> >
> >
> > --
> > - Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end
> > - "Use the force Harry" - Gandalf, Star Trek IIQuoted 15 lines
> >
>
>
>
> --
> - Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end > - "Use the force Harry" - Gandalf, Star Trek II
>Signature · 2 lines
- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end - "Use the force Harry" - Gandalf, Star Trek II-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#233951): https://lists.openembedded.org/g/openembedded-core/message/233951
Mute This Topic: https://lists.openembedded.org/mt/118499141/3617728
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-