Skip to contentSkip to conversation

Support Python freethreading

RFC PATCH In review Zoltán Böszörményi 3 participants 24 messages Mar 25 – Mar 26 1 message not in the archive

lore
Share & tools
lore
#0 lore raw

cover Support Python freethreading

This patchset allows turning on Python freethreading by using a new DISTRO_FEATURES setting:

DISTRO_FEATURES += "python_freethreading"

and optionally:

DISTRO_FEATURES_NATIVE += "python_freethreading"

The change to python3-dir.bbclass is crucial, as all python module build helper classes rely on it.

I also added native variants of some settings because another series against meta-openembedded is needed to complete this one. In that, the change to the fontforge recipe relies on the native setting for setting BUILD_LDFLAGS.

All of the changes included here were successfully build tested, except for the gdb-cross-canadian change. That one failed, but it's included for completeness.

Quite a few of python modules were build tested for native and target builds, including python3-pytorch from my meta-python-ai with reduced RDEPENDS because e.g. python3-onnx there failed to build with freethreading enabled.

All in all, this series at least allows to get things going. Though maybe not for Yocto 6.0, as it's quite late.

Best regards,
Zoltán Böszörményi

#1 lore raw

RFC PATCH 1/14 python3-dir.bbclass: Support python freethreading

With a new DISTRO_FEATURE ("python_freethreading"), set PYTHON_ABI.

Added a new variable called PYTHON_MAINVERSION which is a combination of PYTHON_BASEVERSION and PYTHON_ABI.

Set PYTHON_DIR so it also contains the ABI. Python build helper classes use this variable, so it's important to set it correctly.

Support different settings for target and native.

---
 meta/classes-recipe/python3-dir.bbclass | 9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)
meta/classes-recipe/python3-dir.bbclass +7 -2
index 3eb0dea9b2..2b81b776cb 100644
@@ -5,7 +5,12 @@
 #
 
 PYTHON_BASEVERSION = "3.14"
-PYTHON_ABI = ""
-PYTHON_DIR = "python${PYTHON_BASEVERSION}"
+PYTHON_ABI = "${@bb.utils.contains('DISTRO_FEATURES', 'python_freethreading', 't', '', d)}"
+PYTHON_ABI_NATIVE = "${@bb.utils.contains('DISTRO_FEATURES_NATIVE', 'python_freethreading', 't', '', d)}"
+PYTHON_MAINVERSION = "${PYTHON_BASEVERSION}${PYTHON_ABI}"
+PYTHON_MAINVERSION_NATIVE = "${PYTHON_BASEVERSION}${PYTHON_ABI_NATIVE}"
+PYTHON_DIR = "python${PYTHON_MAINVERSION}"
+PYTHON_DIR_NATIVE = "python${PYTHON_MAINVERSION_NATIVE}"
 PYTHON_PN = "python3"
 PYTHON_SITEPACKAGES_DIR = "${libdir}/${PYTHON_DIR}/site-packages"
+PYTHON_SITEPACKAGES_DIR_NATIVE = "${libdir}/${PYTHON_DIR_NATIVE}/site-packages"
Signature · 1 line
2.53.0
#2 lore raw

RFC PATCH 2/14 python3: Adapt recipe to freethreading

In reply to Zoltán Böszörményi

Set the previously added PACKAGECONFIG[freethreading] based on the new DISTRO_FEATURE=python_freethreading.

Now inherit python3-dir. The PYTHON_MAJMIN private variable was removed, and its usages were replaced with the proper variables from python3-dir.bbclass to simplify the settings and for consistency.

---
 .../recipes-devtools/python/python3_3.14.3.bb | 61 ++++++++++---------
 1 file changed, 32 insertions(+), 29 deletions(-)
meta/recipes-devtools/python/python3_3.14.3.bb +32 -29
index 7a1ad1e386..59e4412b71 100644
@@ -47,19 +47,17 @@ UPSTREAM_CHECK_REGEX = "[Pp]ython-(?P<pver>\d+(\.\d+)+).tar"
 
 CVE_PRODUCT = "python:python python_software_foundation:python cpython"
 
-PYTHON_MAJMIN = "3.14"
-
 S = "${UNPACKDIR}/Python-${PV}"
 
 BBCLASSEXTEND = "native nativesdk"
 
-inherit autotools pkgconfig qemu ptest multilib_header update-alternatives
+inherit autotools pkgconfig qemu python3-dir ptest multilib_header update-alternatives
 
 MULTILIB_SUFFIX = "${@d.getVar('base_libdir',1).split('/')[-1]}"
 
 ALTERNATIVE:${PN}-dev = "python3-config"
-ALTERNATIVE_LINK_NAME[python3-config] = "${bindir}/python${PYTHON_MAJMIN}-config"
-ALTERNATIVE_TARGET[python3-config] = "${bindir}/python${PYTHON_MAJMIN}-config-${MULTILIB_SUFFIX}"
+ALTERNATIVE_LINK_NAME[python3-config] = "${bindir}/${PYTHON_DIR}-config"
+ALTERNATIVE_TARGET[python3-config] = "${bindir}/${PYTHON_DIR}-config-${MULTILIB_SUFFIX}"
 
 DEPENDS = "\
     autoconf-archive-native \
@@ -86,7 +84,7 @@ EXTRA_OECONF:append:class-native = " --bindir=${bindir}/${PN}"
 EXTRA_OECONF:append:class-target = " --with-build-python=nativepython3 PLATFORM_TRIPLET=${HOST_ARCH}-${HOST_OS}"
 EXTRA_OECONF:append:class-nativesdk = " --with-build-python=nativepython3"
 
-export CROSSPYTHONPATH = "${STAGING_LIBDIR_NATIVE}/python${PYTHON_MAJMIN}/lib-dynload/"
+export CROSSPYTHONPATH = "${STAGING_LIBDIR_NATIVE}/${PYTHON_DIR}/lib-dynload/"
 
 EXTRANATIVEPATH += "python3-native"
 
@@ -111,7 +109,10 @@ CACHED_CONFIGUREVARS:append:libc-musl = "\
 "
 
 # PGO currently causes builds to not be reproducible so disable by default, see YOCTO #13407
-PACKAGECONFIG ??= "editline gdbm ${@bb.utils.filter('DISTRO_FEATURES', 'lto', d)}"
+PACKAGECONFIG ??= "editline gdbm \
+    ${@bb.utils.filter('DISTRO_FEATURES', 'lto', d)} \
+    ${@bb.utils.contains('DISTRO_FEATURES', 'python_freethreading', 'freethreading', '', d)} \
+"
 PACKAGECONFIG[freethreading] = "--disable-gil,--enable-gil"
 PACKAGECONFIG[readline] = "--with-readline=readline,,readline,,,editline"
 PACKAGECONFIG[editline] = "--with-readline=editline,,libedit,,,readline"
@@ -178,7 +179,7 @@ do_install:prepend() {
 }
 
 do_install:append:class-target() {
-        oe_multilib_header python${PYTHON_MAJMIN}/pyconfig.h
+        oe_multilib_header ${PYTHON_DIR}/pyconfig.h
 }
 
 do_install:append:class-native() {
@@ -203,20 +204,20 @@ do_install:append:class-native() {
 
         # Nothing should be looking into ${B} for python3-native
         sed -i -e 's:${B}:/build/path/unavailable/:g' \
-                ${D}/${libdir}/python${PYTHON_MAJMIN}/config-${PYTHON_MAJMIN}${PYTHON_ABI}*/Makefile
+                ${D}/${libdir}/${PYTHON_DIR}/config-${PYTHON_MAINVERSION}*/Makefile
 
         # disable the lookup in user's site-packages globally
-        sed -i 's#ENABLE_USER_SITE = None#ENABLE_USER_SITE = False#' ${D}${libdir}/python${PYTHON_MAJMIN}/site.py
+        sed -i 's#ENABLE_USER_SITE = None#ENABLE_USER_SITE = False#' ${D}${libdir}/${PYTHON_DIR}/site.py
 
         # python3-config needs to be in /usr/bin and not in a subdir of it to work properly
         mv ${D}/${bindir}/${PN}/python*config ${D}/${bindir}/
 }
 
 do_install:append() {
-        for c in ${D}/${libdir}/python${PYTHON_MAJMIN}/_sysconfigdata*.py; do
+        for c in ${D}/${libdir}/${PYTHON_DIR}/_sysconfigdata*.py; do
             python3 ${UNPACKDIR}/reformat_sysconfig.py $c
         done
-        rm -f ${D}${libdir}/python${PYTHON_MAJMIN}/__pycache__/_sysconfigdata*.cpython*
+        rm -f ${D}${libdir}/${PYTHON_DIR}/__pycache__/_sysconfigdata*.cpython*
 
         mkdir -p ${D}${libdir}/python-sysconfigdata
         sysconfigfile=`find ${D} -name _sysconfig*.py`
@@ -224,8 +225,8 @@ do_install:append() {
                 -e "s,^ 'LIBDIR'.*, 'LIBDIR': '${STAGING_LIBDIR}'\,,g" \
                 -e "s,^ 'INCLUDEDIR'.*, 'INCLUDEDIR': '${STAGING_INCDIR}'\,,g" \
                 -e "s,^ 'CONFINCLUDEDIR'.*, 'CONFINCLUDEDIR': '${STAGING_INCDIR}'\,,g" \
-                -e "s,^ 'INCLUDEPY'.*, 'INCLUDEPY': '${STAGING_INCDIR}/python${PYTHON_MAJMIN}'\,,g" \
-                -e "s,^ 'CONFINCLUDEPY'.*, 'CONFINCLUDEPY': '${STAGING_INCDIR}/python${PYTHON_MAJMIN}'\,,g" \
+                -e "s,^ 'INCLUDEPY'.*, 'INCLUDEPY': '${STAGING_INCDIR}/${PYTHON_DIR}'\,,g" \
+                -e "s,^ 'CONFINCLUDEPY'.*, 'CONFINCLUDEPY': '${STAGING_INCDIR}/${PYTHON_DIR}'\,,g" \
                 -e "s,${B},/build/path/unavailable/,g" \
                 $sysconfigfile
         cp $sysconfigfile ${D}${libdir}/python-sysconfigdata/_sysconfigdata.py
@@ -241,7 +242,7 @@ do_install:append:class-nativesdk () {
     for PYTHSCRIPT in `grep -rIl ${bindir}/python ${D}${bindir}`; do
          sed -i -e '1s|^#!.*|#!/usr/bin/env python3|' $PYTHSCRIPT
     done
-    create_wrapper ${D}${bindir}/python${PYTHON_MAJMIN} TERMINFO_DIRS='${sysconfdir}/terminfo:/etc/terminfo:/usr/share/terminfo:/usr/share/misc/terminfo:/lib/terminfo' PYTHONNOUSERSITE='1'
+    create_wrapper ${D}${bindir}/${PYTHON_DIR} TERMINFO_DIRS='${sysconfdir}/terminfo:/etc/terminfo:/usr/share/terminfo:/usr/share/misc/terminfo:/lib/terminfo' PYTHONNOUSERSITE='1'
 }
 
 do_install_ptest:append:class-target:libc-musl () {
@@ -257,7 +258,7 @@ SYSROOT_PREPROCESS_FUNCS:append:class-nativesdk = " provide_target_config_script
 provide_target_config_script() {
         install -d ${SYSROOT_DESTDIR}${prefix}/python-target-config/
         install ${D}/${bindir}/python3-config ${SYSROOT_DESTDIR}/${prefix}/python-target-config/
-        install ${D}/${bindir}/python${PYTHON_MAJMIN}-config ${SYSROOT_DESTDIR}/${prefix}/python-target-config/
+        install ${D}/${bindir}/${PYTHON_DIR}-config ${SYSROOT_DESTDIR}/${prefix}/python-target-config/
 }
 SYSROOT_DIRS += "${prefix}/python-target-config/"
 
@@ -280,13 +281,13 @@ py_package_preprocess () {
                 -e 's:${RECIPE_SYSROOT_NATIVE}::g' \
                 -e 's:${RECIPE_SYSROOT}::g' \
                 -e 's:${BASE_WORKDIR}/${MULTIMACH_TARGET_SYS}::g' \
-                ${PKGD}/${libdir}/python${PYTHON_MAJMIN}/config-${PYTHON_MAJMIN}${PYTHON_ABI}*/Makefile \
-                ${PKGD}/${libdir}/python${PYTHON_MAJMIN}/_sysconfigdata*.py \
-                ${PKGD}/${bindir}/python${PYTHON_MAJMIN}-config
+                ${PKGD}/${libdir}/${PYTHON_DIR}/config-${PYTHON_MAINVERSION}*/Makefile \
+                ${PKGD}/${libdir}/${PYTHON_DIR}/_sysconfigdata*.py \
+                ${PKGD}/${bindir}/${PYTHON_DIR}-config
 
         # Reformat _sysconfigdata after modifying it so that it remains
         # reproducible
-        for c in ${PKGD}/${libdir}/python${PYTHON_MAJMIN}/_sysconfigdata*.py; do
+        for c in ${PKGD}/${libdir}/${PYTHON_DIR}/_sysconfigdata*.py; do
             python3 ${UNPACKDIR}/reformat_sysconfig.py $c
         done
 
@@ -301,7 +302,7 @@ py_package_preprocess () {
              -c "from py_compile import compile; compile('$sysconfigfile', optimize=2)"
         cd -
 
-        mv ${PKGD}/${bindir}/python${PYTHON_MAJMIN}-config ${PKGD}/${bindir}/python${PYTHON_MAJMIN}-config-${MULTILIB_SUFFIX}
+        mv ${PKGD}/${bindir}/${PYTHON_DIR}-config ${PKGD}/${bindir}/${PYTHON_DIR}-config-${MULTILIB_SUFFIX}
 
         #Remove the unneeded copy of target sysconfig data
         rm -rf ${PKGD}/${libdir}/python-sysconfigdata
@@ -313,7 +314,7 @@ py_package_preprocess () {
                 -e 's:${RECIPE_SYSROOT_NATIVE}::g' \
                 -e 's:${RECIPE_SYSROOT}::g' \
                 -e 's:${BASE_WORKDIR}/${MULTIMACH_TARGET_SYS}::g' \
-		${PKGD}/${libdir}/python${PYTHON_MAJMIN}/_sysconfig_vars*.json
+		${PKGD}/${libdir}/${PYTHON_DIR}/_sysconfig_vars*.json
 }
 
 # We want bytecode precompiled .py files (.pyc's) by default
@@ -414,7 +415,7 @@ do_create_manifest() {
     cp ${UNPACKDIR}/get_module_deps3.py ${WORKDIR}
     cd ${WORKDIR}
     # This needs to be executed by python-native and NOT by HOST's python
-    nativepython3 create_manifest3.py ${PYTHON_MAJMIN}
+    nativepython3 create_manifest3.py ${PYTHON_MAINVERSION}
     cp python3-manifest.json.new ${THISDIR}/python3/python3-manifest.json
 }
 
@@ -429,11 +430,12 @@ RRECOMMENDS:${PN}-crypt:append:class-nativesdk = " ${MLPREFIX}openssl ${MLPREFIX
 
 # For historical reasons PN is empty and provided by python3-modules
 FILES:${PN} = ""
+FILES:${PN}-core += "${bindir}/python${PYTHON_MAINVERSION} ${bindir}/python${PYTHON_BASEVERSION}"
 RPROVIDES:${PN}-modules = "${PN}"
 
-FILES:${PN}-pydoc += "${bindir}/pydoc${PYTHON_MAJMIN} ${bindir}/pydoc3"
-FILES:${PN}-idle += "${bindir}/idle3 ${bindir}/idle${PYTHON_MAJMIN}"
-FILES:${PN}-tkinter += "${libdir}/python${PYTHON_MAJMIN}/lib-dynload/_tkinter.*.so"
+FILES:${PN}-pydoc += "${bindir}/pydoc${PYTHON_MAINVERSION} ${bindir}/pydoc3"
+FILES:${PN}-idle += "${bindir}/idle3 ${bindir}/idle${PYTHON_MAINVERSION}"
+FILES:${PN}-tkinter += "${libdir}/${PYTHON_DIR}/lib-dynload/_tkinter.*.so"
 
 # provide python-pyvenv from python3-venv
 RPROVIDES:${PN}-venv += "${MLPREFIX}python3-pyvenv"
@@ -441,13 +443,14 @@ RPROVIDES:${PN}-venv += "${MLPREFIX}python3-pyvenv"
 # package libpython3
 PACKAGES =+ "libpython3 libpython3-staticdev"
 FILES:libpython3 = "${libdir}/libpython*.so.*"
-FILES:libpython3-staticdev += "${libdir}/python${PYTHON_MAJMIN}/config-${PYTHON_MAJMIN}-*/libpython${PYTHON_MAJMIN}.a"
+FILES:libpython3-staticdev += "${libdir}/${PYTHON_DIR}/config-${PYTHON_MAINVERSION}-*/lib${PYTHON_DIR}.a"
 INSANE_SKIP:${PN}-dev += "dev-elf"
 INSANE_SKIP:${PN}-ptest = "dev-deps"
 
 # catch all the rest (unsorted)
 PACKAGES += "${PN}-misc"
 RDEPENDS:${PN}-misc += "\
+    bash \
     ${PN}-audio \
     ${PN}-codecs \
     ${PN}-core \
@@ -459,7 +462,7 @@ RDEPENDS:${PN}-misc += "\
 RDEPENDS:${PN}-modules:append:class-target = " ${MLPREFIX}python3-misc"
 RDEPENDS:${PN}-modules:append:class-nativesdk = " ${MLPREFIX}python3-misc"
 RDEPENDS:${PN}-modules:append:class-target = " ${@bb.utils.contains('PACKAGECONFIG', 'gdbm', '${MLPREFIX}python3-gdbm', '', d)}"
-FILES:${PN}-misc = "${libdir}/python${PYTHON_MAJMIN} ${libdir}/python${PYTHON_MAJMIN}/lib-dynload"
+FILES:${PN}-misc = "${libdir}/${PYTHON_DIR} ${libdir}/${PYTHON_DIR}/lib-dynload"
 
 # catch manpage
 PACKAGES += "${PN}-man"
@@ -499,5 +502,5 @@ RDEPENDS:${PN}-tests:append:class-nativesdk = " ${MLPREFIX}bash"
 # Python's tests contain large numbers of files we don't need in the recipe sysroots
 SYSROOT_PREPROCESS_FUNCS += " py3_sysroot_cleanup"
 py3_sysroot_cleanup () {
-	rm -rf ${SYSROOT_DESTDIR}${libdir}/python${PYTHON_MAJMIN}/test
+	rm -rf ${SYSROOT_DESTDIR}${libdir}/${PYTHON_DIR}/test
 }
Signature · 1 line
2.53.0
#3 lore raw

RFC PATCH 3/14 python3native.bbclass: Fix settings

In reply to Zoltán Böszörményi

With PYTHON_DIR now containing PYTHON_ABI, don't use the latter.

---
 meta/classes-recipe/python3native.bbclass | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
meta/classes-recipe/python3native.bbclass +2 -2
index da1283d6b3..755a6c156c 100644
@@ -20,8 +20,8 @@ export STAGING_LIBDIR
 # find_package(PythonLibs REQUIRED)
 # which ends up using libs/includes from build host
 # Therefore pre-empt that effort
-export PYTHON_LIBRARY = "${STAGING_LIBDIR}/lib${PYTHON_DIR}${PYTHON_ABI}.so"
-export PYTHON_INCLUDE_DIR = "${STAGING_INCDIR}/${PYTHON_DIR}${PYTHON_ABI}"
+export PYTHON_LIBRARY = "${STAGING_LIBDIR}/lib${PYTHON_DIR}.so"
+export PYTHON_INCLUDE_DIR = "${STAGING_INCDIR}/${PYTHON_DIR}"
 
 # suppress host user's site-packages dirs.
 export PYTHONNOUSERSITE = "1"
Signature · 1 line
2.53.0
#4 lore raw

RFC PATCH 4/14 python_pyo3.bbclass: Pass ABI flag and PYTHON_MAINVERSION

In reply to Zoltán Böszörményi

Setting the ABI flag correctly is necessary for modules using python_maturin.bbclass and others based on pyo3.

---
 meta/classes-recipe/python_pyo3.bbclass | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
meta/classes-recipe/python_pyo3.bbclass +3 -1
index 7f5a00f584..c618d711ac 100644
@@ -12,7 +12,8 @@
 inherit cargo python3-dir siteinfo
 
 export PYO3_CROSS = "1"
-export PYO3_CROSS_PYTHON_VERSION = "${PYTHON_BASEVERSION}"
+export PYO3_CROSS_PYTHON_VERSION = "${PYTHON_MAINVERSION}"
+export ABIFLAGS = "${PYTHON_ABI}"
 export PYO3_CROSS_LIB_DIR = "${STAGING_LIBDIR}"
 export CARGO_BUILD_TARGET = "${RUST_HOST_SYS}"
 export RUSTFLAGS
@@ -25,6 +26,7 @@ implementation=CPython
 version=${PYTHON_BASEVERSION}
 shared=true
 abi3=false
+abiflags=${PYTHON_ABI}
 lib_name=${PYTHON_DIR}
 lib_dir=${STAGING_LIBDIR}
 pointer_width=${SITEINFO_BITS}
Signature · 1 line
2.53.0
#5 lore raw

RFC PATCH 5/14 clang: Use PYTHON_MAINVERSION for python module versioning

In reply to Zoltán Böszörményi

Use the new complete main version setting instead of just PYTHON_BASEVERSION that does not include PYTHON_ABI.

---
 meta/recipes-devtools/clang/clang_git.bb | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
meta/recipes-devtools/clang/clang_git.bb +1 -1
index e10c327a2a..d5f8aa777f 100644
@@ -46,7 +46,7 @@ PACKAGECONFIG[lld] = "-DCLANG_DEFAULT_LINKER=lld,,,"
 PACKAGECONFIG[lto] = "-DLLVM_ENABLE_LTO=Full -DLLVM_BINUTILS_INCDIR=${STAGING_INCDIR},,binutils,"
 PACKAGECONFIG[thin-lto] = "-DLLVM_ENABLE_LTO=Thin -DLLVM_BINUTILS_INCDIR=${STAGING_INCDIR},,binutils,"
 PACKAGECONFIG[unwindlib] = "-DCLANG_DEFAULT_UNWINDLIB=libunwind,-DCLANG_DEFAULT_UNWINDLIB=libgcc,,"
-PACKAGECONFIG[libclang-python] = "-DCLANG_PYTHON_BINDINGS_VERSIONS=${PYTHON_BASEVERSION},,"
+PACKAGECONFIG[libclang-python] = "-DCLANG_PYTHON_BINDINGS_VERSIONS=${PYTHON_MAINVERSION},,"
 
 OECMAKE_SOURCEPATH = "${S}/clang"
 
Signature · 1 line
2.53.0
#6 lore raw

RFC PATCH 6/14 gdb, gdb-cross-canadian: Fix settings for python freethreading

In reply to Zoltán Böszörményi

PYTHON_DIR now contains PYTHON_ABI, don't use the latter. Simplify the linked library name to use -l${PYTHON_DIR}.

---
 meta/recipes-devtools/gdb/gdb-cross-canadian.inc | 4 ++--
 meta/recipes-devtools/gdb/gdb_17.1.bb            | 4 ++--
 2 files changed, 4 insertions(+), 4 deletions(-)
meta/recipes-devtools/gdb/gdb-cross-canadian.inc +2 -2
index 7b4a7719e4..c4c62d02af 100644
@@ -22,8 +22,8 @@ do_configure:prepend() {
 cat > ${WORKDIR}/python << EOF
 #! /bin/sh
 case "\$2" in
-        --includes) echo "-I${STAGING_INCDIR}/${PYTHON_DIR}${PYTHON_ABI}/" ;;
-        --ldflags) echo "-Wl,-rpath-link,${STAGING_LIBDIR}/.. -Wl,-rpath,${libdir}/.. -lpthread -ldl -lutil -lm -lpython${PYTHON_BASEVERSION}${PYTHON_ABI}" ;;
+        --includes) echo "-I${STAGING_INCDIR}/${PYTHON_DIR}/" ;;
+        --ldflags) echo "-Wl,-rpath-link,${STAGING_LIBDIR}/.. -Wl,-rpath,${libdir}/.. -lpthread -ldl -lutil -lm -l${PYTHON_DIR}" ;;
         --exec-prefix) echo "${exec_prefix}" ;;
         *) exit 1 ;;
 esac
meta/recipes-devtools/gdb/gdb_17.1.bb +2 -2
index 9c6db4ca2c..a5d96f01eb 100644
@@ -26,8 +26,8 @@ do_configure:prepend() {
 		cat > ${WORKDIR}/python << EOF
 #!/bin/sh
 case "\$2" in
-	--includes) echo "-I${STAGING_INCDIR}/${PYTHON_DIR}${PYTHON_ABI}/" ;;
-	--ldflags) echo "-Wl,-rpath-link,${STAGING_LIBDIR}/.. -Wl,-rpath,${libdir}/.. -lpthread -ldl -lutil -lm -lpython${PYTHON_BASEVERSION}${PYTHON_ABI}" ;;
+	--includes) echo "-I${STAGING_INCDIR}/${PYTHON_DIR}/" ;;
+	--ldflags) echo "-Wl,-rpath-link,${STAGING_LIBDIR}/.. -Wl,-rpath,${libdir}/.. -lpthread -ldl -lutil -lm -l${PYTHON_DIR}" ;;
 	--exec-prefix) echo "${exec_prefix}" ;;
 	*) exit 1 ;;
 esac
Signature · 1 line
2.53.0
#7 lore raw

RFC PATCH 7/14 boost: Fix settings for python freethreading

In reply to Zoltán Böszörményi

PYTHON_DIR now contains PYTHON_ABI, fix the settings accordingly to still find both python builds.

---
 meta/recipes-support/boost/boost.inc | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
meta/recipes-support/boost/boost.inc +1 -1
index 64a57ddfb2..c76fd7a0c6 100644
@@ -174,7 +174,7 @@ do_configure() {
 
 	# If we want Python then we need to tell Boost *exactly* where to find it
 	if ${@bb.utils.contains('BOOST_LIBS', 'python', 'true', 'false', d)}; then
-		echo "using python : ${PYTHON_BASEVERSION} : ${STAGING_DIR_HOST}${bindir}/python3 : ${STAGING_DIR_HOST}${includedir}/${PYTHON_DIR}${PYTHON_ABI} : ${STAGING_DIR_HOST}${libdir}/${PYTHON_DIR} ;" >> ${WORKDIR}/user-config.jam
+		echo "using python : ${PYTHON_BASEVERSION} : ${STAGING_DIR_HOST}${bindir}/python3 : ${STAGING_DIR_HOST}${includedir}/${PYTHON_DIR} : ${STAGING_DIR_HOST}${libdir}/python${PYTHON_BASEVERSION} ;" >> ${WORKDIR}/user-config.jam
 	fi
 
 	if ${@bb.utils.contains('BOOST_LIBS', 'mpi', 'true', 'false', d)}; then
Signature · 1 line
2.53.0
#8 lore raw

RFC PATCH 8/14 lttng-tools: Simplify setting PYTHON_INCLUDE

In reply to Zoltán Böszörményi

PYTHON_DIR now contains PYTHON_ABI, use it.

---
 meta/recipes-kernel/lttng/lttng-tools_2.14.1.bb | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
meta/recipes-kernel/lttng/lttng-tools_2.14.1.bb +1 -1
index 3a3f2cff2c..ef80c53e9c 100644
@@ -35,7 +35,7 @@ INSANE_SKIP:${PN}-ptest += "dev-deps"
 
 PYTHON_OPTION = "am_cv_python_pyexecdir='${PYTHON_SITEPACKAGES_DIR}' \
                  am_cv_python_pythondir='${PYTHON_SITEPACKAGES_DIR}' \
-                 PYTHON_INCLUDE='-I${STAGING_INCDIR}/python${PYTHON_BASEVERSION}${PYTHON_ABI}' \
+                 PYTHON_INCLUDE='-I${STAGING_INCDIR}/${PYTHON_DIR}' \
 "
 PACKAGECONFIG ??= "${LTTNGUST} kmod"
 PACKAGECONFIG[python] = "--enable-python-bindings ${PYTHON_OPTION},,python3 swig-native"
Signature · 1 line
2.53.0
#9 lore raw

RFC PATCH 9/14 lttng-ust: Simplify setting PYTHON_INCLUDE

In reply to Zoltán Böszörményi

PYTHON_DIR now contains PYTHON_ABI, use it.

---
 meta/recipes-kernel/lttng/lttng-ust_2.14.0.bb | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
meta/recipes-kernel/lttng/lttng-ust_2.14.0.bb +1 -1
index 1a15c5b420..5c1c74b7a8 100644
@@ -8,7 +8,7 @@ LIC_FILES_CHKSUM = "file://LICENSE;md5=3b04e8f34dbcf08198c6618d05e8fe7b"
 
 PYTHON_OPTION = "am_cv_python_pyexecdir='${PYTHON_SITEPACKAGES_DIR}' \
                  am_cv_python_pythondir='${PYTHON_SITEPACKAGES_DIR}' \
-                 PYTHON_INCLUDE='-I${STAGING_INCDIR}/python${PYTHON_BASEVERSION}${PYTHON_ABI}' \
+                 PYTHON_INCLUDE='-I${STAGING_INCDIR}/${PYTHON_DIR}' \
 "
 
 inherit autotools lib_package manpages python3native pkgconfig
Signature · 1 line
2.53.0
#10 lore raw

RFC PATCH 10/14 libcap-ng-python: Use PYTHON_DIR for FILES

In reply to Zoltán Böszörményi

PYTHON_DIR now contains PYTHON_ABI, use it for consistency.

Move the installed site-packages directory to the correct location.

---
 meta/recipes-support/libcap-ng/libcap-ng-python_0.9.1.bb | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)
meta/recipes-support/libcap-ng/libcap-ng-python_0.9.1.bb +4 -1
index f702056f02..4e01c320fd 100644
@@ -12,6 +12,9 @@ EXTRA_OECONF += "--with-python3"
 
 do_install() {
     oe_runmake 'DESTDIR=${D}' install -C ${B}/bindings/python3
+    if [ ! -d ${D}${libdir}/${PYTHON_DIR} ]; then
+        mv ${D}${libdir}/python${PYTHON_BASEVERSION} ${D}${libdir}/${PYTHON_DIR}
+    fi
 }
 
-FILES:${PN} = "${libdir}/python${PYTHON_BASEVERSION}"
+FILES:${PN} = "${libdir}/${PYTHON_DIR}"
Signature · 1 line
2.53.0
#11 lore raw

RFC PATCH 11/14 hwlatdetect: Use PYTHON_DIR to support python freethreading

In reply to Zoltán Böszörményi

python${PYTHON_BASEVERSION} is not enough now, PYTHON_DIR is needed for consistency.

---
 meta/recipes-rt/rt-tests/hwlatdetect_git.bb | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
meta/recipes-rt/rt-tests/hwlatdetect_git.bb +3 -3
index 2dcc80965a..d37909cd87 100644
@@ -8,7 +8,7 @@ LIC_FILES_CHKSUM = "file://COPYING;md5=751419260aa954499f7abaabaa882bbe"
 require rt-tests.inc
 inherit python3-dir
 
-EXTRA_OEMAKE += "PYLIB=${libdir}/python${PYTHON_BASEVERSION}/dist-packages"
+EXTRA_OEMAKE += "PYLIB=${libdir}/${PYTHON_DIR}/dist-packages"
 
 do_compile() {
 	oe_runmake hwlatdetect
@@ -18,9 +18,9 @@ do_install() {
         oe_runmake install_hwlatdetect DESTDIR=${D} SBINDIR=${sbindir} \
 	           MANDIR=${mandir} INCLUDEDIR=${includedir}
 
-        sed -i -e '1s,#!.*python.*,#!${bindir}/python3,' ${D}${libdir}/python${PYTHON_BASEVERSION}/dist-packages/hwlatdetect.py
+        sed -i -e '1s,#!.*python.*,#!${bindir}/python3,' ${D}${libdir}/${PYTHON_DIR}/dist-packages/hwlatdetect.py
 }
 
-FILES:${PN} += "${libdir}/python${PYTHON_BASEVERSION}/dist-packages/hwlatdetect.py"
+FILES:${PN} += "${libdir}/${PYTHON_DIR}/dist-packages/hwlatdetect.py"
 RDEPENDS:${PN} = "python3-core "
 RRECOMMENDS:${PN} = "kernel-module-hwlat-detector"
Signature · 1 line
2.53.0
#12 lore raw

RFC PATCH 12/14 rpm: Support Python freethreading

In reply to Zoltán Böszörményi

Added a patch to stop using the stable CPython API (of 3.7!) as it prevents building rpm against a Python build with freethreading enabled.

---
 .../files/0001-rpm-Don-t-use-stable-API.patch | 37 +++++++++++++++++++
 meta/recipes-devtools/rpm/rpm_4.20.1.bb       |  1 +
 2 files changed, 38 insertions(+)
 create mode 100644 meta/recipes-devtools/rpm/files/0001-rpm-Don-t-use-stable-API.patch
meta/recipes-devtools/rpm/files/0001-rpm-Don-t-use-stable-API.patch new file +37 -0
new file mode 100644
index 0000000000..3d82fdab3f
@@ -0,0 +1,37 @@
+From 2fb202730b49538bbe6d1439d419c18a89eb4c91 Mon Sep 17 00:00:00 2001
+From: =?UTF-8?q?Zolt=C3=A1n=20B=C3=B6sz=C3=B6rm=C3=A9nyi?=
+ <[email protected]>
+Date: Wed, 25 Mar 2026 08:20:13 +0100
+Subject: [PATCH] rpm: Don't use stable API
+MIME-Version: 1.0
+Content-Type: text/plain; charset=UTF-8
+Content-Transfer-Encoding: 8bit
+
+With Python freethreading enabled, these explicit errors are
+triggered in the Python headers:
+
+  #error "The limited API is not currently supported in the free-threaded build"
+
+Signed-off-by: Zoltán Böszörményi <[email protected]>
+Upstream-Status: Inappropriate [python freethreading specific]
+---
+ python/CMakeLists.txt | 3 ---
+ 1 file changed, 3 deletions(-)
+
+diff --git a/python/CMakeLists.txt b/python/CMakeLists.txt
+index acc003ad6..3f21205c8 100644
+--- a/python/CMakeLists.txt
++++ b/python/CMakeLists.txt
+@@ -18,9 +18,6 @@ Python3_add_library(_rpm
+ 	spec-py.c spec-py.h
+ )
+ 
+-# Select Python stable ABI
+-target_compile_definitions(_rpm PRIVATE Py_LIMITED_API=0x03070000)
+-
+ target_link_libraries(_rpm PRIVATE librpmio librpm librpmbuild librpmsign)
+ 
+ install(TARGETS _rpm
+-- 
+2.53.0
+
meta/recipes-devtools/rpm/rpm_4.20.1.bb +1 -0
index ba967ec1fa..05be3933e0 100644
@@ -39,6 +39,7 @@ SRC_URI = "git://github.com/rpm-software-management/rpm;branch=rpm-4.20.x;protoc
            file://0002-rpmio-rpmglob.c-avoid-using-GLOB_BRACE-if-undefined-.patch \
            file://0001-CMakeLists.txt-set-libdir-to-CMAKE_INSTALL_FULL_LIBD.patch \
            file://0001-CMakeLists.txt-Fix-checking-for-CFLAGS.patch \
+           file://0001-rpm-Don-t-use-stable-API.patch \
            "
 
 PE = "1"
Signature · 1 line
2.53.0
#13 lore raw

RFC PATCH 13/14 xcb-proto: Support Python freethreading

In reply to Zoltán Böszörményi

xcb-proto installs its python module into the ABI-less site-packages directory, then in turn libxcb won't find it.

Move the installed site-packages directory to the correct location.

---
 meta/recipes-graphics/xorg-proto/xcb-proto_1.17.0.bb | 6 ++++++
 1 file changed, 6 insertions(+)
meta/recipes-graphics/xorg-proto/xcb-proto_1.17.0.bb +6 -0
index 2a78a87e77..dd42dba374 100644
@@ -19,6 +19,12 @@ SRC_URI[sha256sum] = "2c1bacd2110f4799f74de6ebb714b94cf6f80fb112316b1219480fd225
 
 inherit autotools pkgconfig python3native
 
+do_install:append () {
+    if [ ! -d ${D}${libdir}/${PYTHON_DIR} ]; then
+        mv ${D}${libdir}/python${PYTHON_BASEVERSION} ${D}${libdir}/${PYTHON_DIR}
+    fi
+}
+
 PACKAGES += "python-xcbgen"
 
 FILES:${PN} = ""
Signature · 1 line
2.53.0
#14 lore raw

RFC PATCH 14/14 libxcb: Support Python freethreading

In reply to Zoltán Böszörményi

Added xcb-proto-native as a new build dependency and inherit python3native.

This fixes the build issues when Python freethreading is enabled.

---
 meta/recipes-graphics/xorg-lib/libxcb_1.17.0.bb | 6 ++----
 1 file changed, 2 insertions(+), 4 deletions(-)
meta/recipes-graphics/xorg-lib/libxcb_1.17.0.bb +2 -4
index fcd8aad303..ac2fcd4cc8 100644
@@ -16,19 +16,17 @@ SRC_URI[sha256sum] = "599ebf9996710fea71622e6e184f3a8ad5b43d0e5fa8c4e407123c88a5
 
 BBCLASSEXTEND = "native nativesdk"
 
-DEPENDS = "xcb-proto xorgproto libxau libpthread-stubs libxdmcp"
+DEPENDS = "xcb-proto-native xcb-proto xorgproto libxau libpthread-stubs libxdmcp"
 
 PACKAGES_DYNAMIC = "^${PN}-.*"
 
 FILES:${PN} = "${libdir}/libxcb.so.*"
 
-inherit autotools pkgconfig features_check
+inherit autotools pkgconfig python3native features_check
 
 # The libxau and others requires x11 in DISTRO_FEATURES
 REQUIRED_DISTRO_FEATURES = "x11"
 
-export PYTHON = "python3"
-
 do_install:append () {
 	chown root.root ${D}${datadir}/doc/${BPN}/tutorial -R
 }
Signature · 1 line
2.53.0
#15 lore raw
In reply to Zoltán Böszörményi
Quoted 4 lines · Zoltán Böszörményi

This patchset allows turning on Python freethreading by using a new DISTRO_FEATURES setting:

Quoted 8 lines

DISTRO_FEATURES += "python_freethreading"

and optionally:

DISTRO_FEATURES_NATIVE += "python_freethreading"

The change to python3-dir.bbclass is crucial, as all python module build helper classes rely on it.

This is fine as an RFC, but otherwise, given the large amount of tweaks required, and having to choose between one or the other (which complicates testing), I would wait and see how other distros provide a free-threading python. Ideally it should be co-installable with regular python.

Alex

#16 lore raw

2026. 03. 25. 13:58 keltezéssel, Alexander Kanavin írta:

Quoted 10 lines · Zoltán Böszörményi
Quoted 9 lines

This patchset allows turning on Python freethreading by using a new DISTRO_FEATURES setting:

DISTRO_FEATURES += "python_freethreading"

and optionally:

DISTRO_FEATURES_NATIVE += "python_freethreading"

Quoted 7 lines
Quoted 2 lines

The change to python3-dir.bbclass is crucial, as all python module build helper classes rely on it.

This is fine as an RFC, but otherwise, given the large amount of tweaks required, and having to choose between one or the other (which complicates testing), I would wait and see how other distros provide a free-threading python. Ideally it should be co-installable with regular python.

As I said, this would double the amount of python module recipe builds.

See the Red Hat bugzilla ticket I created for a different reason (old bitbake needs old python, still a parallel installation use case):

https://bugzilla.redhat.com/show_bug.cgi?id=2418374

"We don't have the human power to maintain several separate Python stacks packaged as RPM packages in Fedora."

Python 3.14.x is present in Fedora 43, only without freethreading.
I expect they won't do a parallel installation with freethreading, either, due to the same reason. When they switch, it will be all or nothing.

#17 lore raw
In reply to Zoltán Böszörményi
Quoted 9 lines · Zoltan Boszormenyi via lists.openembedded.org

This patchset allows turning on Python freethreading by using a new DISTRO_FEATURES setting:

DISTRO_FEATURES += "python_freethreading"

and optionally:

DISTRO_FEATURES_NATIVE += "python_freethreading"

Quoted 8 lines

The change to python3-dir.bbclass is crucial, as all python module build helper classes rely on it.

I also added native variants of some settings because another series against meta-openembedded is needed to complete this one. In that, the change to the fontforge recipe relies on the native setting for setting BUILD_LDFLAGS.

Looking at those other patches in the series, they don't seem to be conditional on the distro feature.

Are those changes valid for a build with and without freethreading ?

That's the type of information, along with the benefits of enabling this would be information expected in a 0/N introductory patch.

Cheers,

Bruce

Quoted 27 lines

All of the changes included here were successfully build tested, except for the gdb-cross-canadian change. That one failed, but it's included for completeness.

Quite a few of python modules were build tested for native and target builds, including python3-pytorch from my meta-python-ai with reduced RDEPENDS because e.g. python3-onnx there failed to build with freethreading enabled.

All in all, this series at least allows to get things going. Though maybe not for Yocto 6.0, as it's quite late.

Best regards,
Zoltán Böszörményi

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group. View/Reply Online (#233874):
https://lists.openembedded.org/g/openembedded-core/message/233874
Mute This Topic: https://lists.openembedded.org/mt/118499141/1050810
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [ [email protected]]
-=-=-=-=-=-=-=-=-=-=-=-

Signature · 3 lines
- Thou shalt not follow the NULL pointer, for chaos and madness await thee
at its end
- "Use the force Harry" - Gandalf, Star Trek II
#18 lore raw

2026. 03. 25. 14:01 keltezéssel, Bruce Ashfield írta:

Quoted 22 lines · On Wed, Mar 25, 2026 at 7:31 AM Zoltan Boszormenyi via lists.openembedded.org…

This patchset allows turning on Python freethreading by using a new DISTRO_FEATURES setting:

DISTRO_FEATURES += "python_freethreading"

and optionally:

DISTRO_FEATURES_NATIVE += "python_freethreading"

The change to python3-dir.bbclass is crucial, as all python module build helper classes rely on it.

I also added native variants of some settings because another series against meta-openembedded is needed to complete this one. In that, the change to the fontforge recipe relies on the native setting for setting BUILD_LDFLAGS.

Quoted 4 lines

Looking at those other patches in the series, they don't seem to be conditional on the distro feature.

Are those changes valid for a build with and without freethreading ?

Yes!

The main change is in python3-dir.bbclass, and the recipe tweaks are just adaptations to the fact that only ${PYTHON_DIR} is needed, not ${PYTHON_DIR}${PYTHON_ABI} from that point.

I was thinking that the introductory mail stressed this enough.

FWIW, less than 30 recipes was spelling the latter out in openembedded-core and meta-openembedded combined.

Quoted 40 lines

That's the type of information, along with the benefits of enabling this would be information expected in a 0/N introductory patch.

Cheers,

Bruce

All of the changes included here were successfully build tested, except for the gdb-cross-canadian change. That one failed, but it's included for completeness.

Quite a few of python modules were build tested for native and target builds, including python3-pytorch from my meta-python-ai with reduced RDEPENDS because e.g. python3-onnx there failed to build with freethreading enabled.

All in all, this series at least allows to get things going. Though maybe not for Yocto 6.0, as it's quite late.

Best regards,
Zoltán Böszörményi

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group. View/Reply Online (#233874):
https://lists.openembedded.org/g/openembedded-core/message/233874
Mute This Topic: https://lists.openembedded.org/mt/118499141/1050810
Group Owner: [email protected] <mailto:openembedded-core%[email protected]>
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-

Signature · 2 lines
- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end
- "Use the force Harry" - Gandalf, Star Trek II
#19 lore raw
Quoted 31 lines · Böszörményi Zoltán

2026. 03. 25. 14:01 keltezéssel, Bruce Ashfield írta:

Quoted 3 lines

On Wed, Mar 25, 2026 at 7:31 AM Zoltan Boszormenyi via

lists.openembedded.org

Quoted 1 line

<http://lists.openembedded.org> <zboszor=

Quoted 19 lines · [email protected]> wrote:

This patchset allows turning on Python freethreading by using a new DISTRO_FEATURES setting:

DISTRO_FEATURES += "python_freethreading"

and optionally:

DISTRO_FEATURES_NATIVE += "python_freethreading"

The change to python3-dir.bbclass is crucial, as all python module build helper classes rely on it.

I also added native variants of some settings because another series against meta-openembedded is needed to complete this one. In that, the change to the fontforge recipe relies on the native setting for setting BUILD_LDFLAGS.

Quoted 4 lines

Looking at those other patches in the series, they don't seem to be conditional on the distro feature.

Are those changes valid for a build with and without freethreading ?

Quoted 7 lines

Yes!

The main change is in python3-dir.bbclass, and the recipe tweaks are just adaptations to the fact that only ${PYTHON_DIR} is needed, not ${PYTHON_DIR}${PYTHON_ABI} from that point.

I was thinking that the introductory mail stressed this enough.

Obviously not :)

I didn't get that impression at all from reading the introductory message or looking at the patches themselves.

You have the overview knowledge of the feature, why it is useful, how it impacts existing recipes, how the migration would work, what is the compatibility, etc.

The patch reader doesn't, so that all needs to be clearly supplied.

Bruce

Quoted 49 lines

FWIW, less than 30 recipes was spelling the latter out in openembedded-core and meta-openembedded combined.

Quoted 23 lines

That's the type of information, along with the benefits of enabling this would be information expected in a 0/N introductory patch.

Cheers,

Bruce

All of the changes included here were successfully build tested, except for the gdb-cross-canadian change. That one failed, but it's included for completeness.

Quite a few of python modules were build tested for native and target builds, including python3-pytorch from my meta-python-ai with reduced RDEPENDS because e.g. python3-onnx there failed to build with freethreading enabled.

All in all, this series at least allows to get things going. Though maybe not for Yocto 6.0, as it's quite late.

Best regards,
Zoltán Böszörményi

Quoted 8 lines

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group. View/Reply Online (#233874):
https://lists.openembedded.org/g/openembedded-core/message/233874
Mute This Topic: https://lists.openembedded.org/mt/118499141/1050810
Group Owner: [email protected] <mailto:openembedded-core%[email protected]> Unsubscribe:

https://lists.openembedded.org/g/openembedded-core/unsub

Quoted 7 lines
    [[email protected]]
    -=-=-=-=-=-=-=-=-=-=-=-

--
- Thou shalt not follow the NULL pointer, for chaos and madness await

thee at its end

Quoted 2 lines

- "Use the force Harry" - Gandalf, Star Trek II

Signature · 3 lines
- Thou shalt not follow the NULL pointer, for chaos and madness await thee
at its end
- "Use the force Harry" - Gandalf, Star Trek II
#20 lore raw

2026. 03. 25. 14:17 keltezéssel, Bruce Ashfield írta:

Quoted 36 lines · Böszörményi Zoltán

2026. 03. 25. 14:01 keltezéssel, Bruce Ashfield írta:
>
>
> On Wed, Mar 25, 2026 at 7:31 AM Zoltan Boszormenyi via lists.openembedded.org
<http://lists.openembedded.org>
> <http://lists.openembedded.org> <[email protected]> wrote:
>
>     This patchset allows turning on Python freethreading by using
>     a new DISTRO_FEATURES setting:
>
>     DISTRO_FEATURES += "python_freethreading"
>
>     and optionally:
>
>     DISTRO_FEATURES_NATIVE += "python_freethreading"
>
>     The change to python3-dir.bbclass is crucial, as all python module >     build helper classes rely on it.
>
>     I also added native variants of some settings because another
>     series against meta-openembedded is needed to complete this one. >     In that, the change to the fontforge recipe relies on the native >     setting for setting BUILD_LDFLAGS.
>
>
> Looking at those other patches in the series, they don't seem to
> be conditional on the distro feature.
>
> Are those changes valid for a build with and without freethreading ?

Yes!

Quoted 8 lines

The main change is in python3-dir.bbclass, and the recipe tweaks are just adaptations to the fact that only ${PYTHON_DIR} is needed, not ${PYTHON_DIR}${PYTHON_ABI} from that point.

I was thinking that the introductory mail stressed this enough.

Obviously not :)

Obviously. :)

Quoted 6 lines

I didn't get that impression at all from reading the introductory message or looking at the patches themselves.

You have the overview knowledge of the feature, why it is useful, how it impacts existing recipes, how the migration would work, what is the compatibility, etc.

I think the Python 3.14 release notes has enough details. See also https://www.phoronix.com/news/Python-3.14

With the two series I sent, only a single recipe is left (cmpi-bindings) in openembedded-core and meta-openembedded that uses
the PYTHON_ABI setting explicitly.

Migration (of python scripts) should not be a problem. If it worked with 3.14 as is, it works with freethreading enabled.

Compatibility is poor for a lot of modules, that clearly spell this out in their pyproject.toml or PKG-INFO:

    "Programming Language :: Python :: Free Threading :: 1 - Unstable"

or something not quite production level support.
It's still experimental for 3rd party modules, despite being officially supported for the interpreter and built-in modules in 3.14.

This is the reason this was an RFC series.

Anyway, the reason I was experimenting with this was that with heavily threaded python code, an import storm can cause a crash in 3.12.11 or the manually updated 3.13.12 in Yocto 5.3 in importlib._bootstrap according to the backtrace.

I would like to see if Python 3.14 (with and without freethreading) fixes such crashes.

Quoted 2 lines

The patch reader doesn't, so that all needs to be clearly supplied.

Maybe next time for v2 after I have a complete image successfully built with freethreading enabled.

Quoted 57 lines

Bruce

FWIW, less than 30 recipes was spelling the latter out in openembedded-core and meta-openembedded combined.

> That's the type of information, along with the benefits of enabling
> this would be information expected in a 0/N introductory patch.
>
> Cheers,
>
> Bruce
>
>
>     All of the changes included here were successfully build tested,
>     except for the gdb-cross-canadian change. That one failed, but
>     it's included for completeness.
>
>     Quite a few of python modules were build tested for native and
>     target builds, including python3-pytorch from my meta-python-ai
>     with reduced RDEPENDS because e.g. python3-onnx there failed
>     to build with freethreading enabled.
>
>     All in all, this series at least allows to get things going.
>     Though maybe not for Yocto 6.0, as it's quite late.
>
>     Best regards,
>     Zoltán Böszörményi
>
>     -=-=-=-=-=-=-=-=-=-=-=-
>     Links: You receive all messages sent to this group.
>     View/Reply Online (#233874):
> https://lists.openembedded.org/g/openembedded-core/message/233874
>     Mute This Topic: https://lists.openembedded.org/mt/118499141/1050810
>     Group Owner: [email protected] <mailto:openembedded-core%[email protected]>
>     <mailto:openembedded-core%[email protected] <mailto:openembedded-core%[email protected]>>
>     Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
>     [[email protected]]
>     -=-=-=-=-=-=-=-=-=-=-=-
>
>
>
> --
> - Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end > - "Use the force Harry" - Gandalf, Star Trek II
>

Signature · 2 lines
- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end
- "Use the force Harry" - Gandalf, Star Trek II
#21 lore raw
Quoted 58 lines · Böszörményi Zoltán

2026. 03. 25. 14:17 keltezéssel, Bruce Ashfield írta:

Quoted 3 lines

On Wed, Mar 25, 2026 at 9:13 AM Böszörményi Zoltán <[email protected]>

Quoted 5 lines · wrote:
    2026. 03. 25. 14:01 keltezéssel, Bruce Ashfield írta:
    >
    >
    > On Wed, Mar 25, 2026 at 7:31 AM Zoltan Boszormenyi via

lists.openembedded.org

Quoted 2 lines

<http://lists.openembedded.org>
> <http://lists.openembedded.org> <zboszor=

Quoted 11 lines · [email protected]> wrote:
Quoted 11 lines

This patchset allows turning on Python freethreading by using a new DISTRO_FEATURES setting:

DISTRO_FEATURES += "python_freethreading"

and optionally:

DISTRO_FEATURES_NATIVE += "python_freethreading"

The change to python3-dir.bbclass is crucial, as all python

module

Quoted 4 lines
Quoted 4 lines

build helper classes rely on it.

I also added native variants of some settings because another series against meta-openembedded is needed to complete this

one.

Quoted 1 line
Quoted 1 line

In that, the change to the fontforge recipe relies on the

native

Quoted 7 lines
Quoted 7 lines

setting for setting BUILD_LDFLAGS.

Looking at those other patches in the series, they don't seem to be conditional on the distro feature.

Are those changes valid for a build with and without freethreading

?

Quoted 11 lines

Yes!

The main change is in python3-dir.bbclass, and the recipe tweaks are just adaptations to the fact that only ${PYTHON_DIR} is needed, not ${PYTHON_DIR}${PYTHON_ABI} from that point.

I was thinking that the introductory mail stressed this enough.

Obviously not :)

Obviously. :)

Quoted 3 lines

I didn't get that impression at all from reading the introductory message or looking at the patches themselves.

Quoted 6 lines
Quoted 3 lines

You have the overview knowledge of the feature, why it is useful, how it impacts existing recipes, how the migration would work, what is the compatibility, etc.

I think the Python 3.14 release notes has enough details. See also https://www.phoronix.com/news/Python-3.14

The information needs to be distilled into your commit, not left to the patch reader to go off and read release notes. That's the point I'm trying to make.

Quoted 20 lines

With the two series I sent, only a single recipe is left (cmpi-bindings) in openembedded-core and meta-openembedded that uses
the PYTHON_ABI setting explicitly.

Migration (of python scripts) should not be a problem. If it worked with 3.14 as is, it works with freethreading enabled.

Compatibility is poor for a lot of modules, that clearly spell this out in their pyproject.toml or PKG-INFO:

"Programming Language :: Python :: Free Threading :: 1 - Unstable"

or something not quite production level support.
It's still experimental for 3rd party modules, despite being officially supported for the interpreter and built-in modules in 3.14.

This is the reason this was an RFC series.

Quoted 8 lines

Anyway, the reason I was experimenting with this was that with heavily threaded python code, an import storm can cause a crash in 3.12.11 or the manually updated 3.13.12 in Yocto 5.3 in importlib._bootstrap according to the backtrace.

I would like to see if Python 3.14 (with and without freethreading) fixes such crashes.

Definitely worth putting in the RFC as well, the motivation for the change matters!

Quoted 7 lines
Quoted 3 lines

The patch reader doesn't, so that all needs to be clearly supplied.

Maybe next time for v2 after I have a complete image successfully built with freethreading enabled.

Ack'd

Bruce

Quoted 65 lines
Quoted 12 lines

Bruce

FWIW, less than 30 recipes was spelling the latter out in openembedded-core and meta-openembedded combined.

> That's the type of information, along with the benefits of enabling > this would be information expected in a 0/N introductory patch. >

Quoted 6 lines

> Cheers,
>
> Bruce
>
>
> All of the changes included here were successfully build

tested,

Quoted 11 lines
Quoted 11 lines

except for the gdb-cross-canadian change. That one failed, but it's included for completeness.

Quite a few of python modules were build tested for native and target builds, including python3-pytorch from my meta-python-ai with reduced RDEPENDS because e.g. python3-onnx there failed to build with freethreading enabled.

All in all, this series at least allows to get things going. Though maybe not for Yocto 6.0, as it's quite late.

Quoted 8 lines
Quoted 8 lines
    Best regards,
    Zoltán Böszörményi

    -=-=-=-=-=-=-=-=-=-=-=-
    Links: You receive all messages sent to this group.
    View/Reply Online (#233874):

https://lists.openembedded.org/g/openembedded-core/message/233874
Mute This Topic:

https://lists.openembedded.org/mt/118499141/1050810

Quoted 5 lines
Quoted 1 line

Group Owner: [email protected]

<mailto:openembedded-core%[email protected]>
> <mailto:openembedded-core%[email protected]
<mailto:openembedded-core%[email protected]>>
> Unsubscribe:

https://lists.openembedded.org/g/openembedded-core/unsub

Quoted 7 lines
Quoted 7 lines
    [[email protected]]
    -=-=-=-=-=-=-=-=-=-=-=-

--
- Thou shalt not follow the NULL pointer, for chaos and madness

await thee at its end

Quoted 7 lines
Quoted 2 lines

- "Use the force Harry" - Gandalf, Star Trek II

--
- Thou shalt not follow the NULL pointer, for chaos and madness await

thee at its end

Quoted 2 lines

- "Use the force Harry" - Gandalf, Star Trek II

Signature · 3 lines
- Thou shalt not follow the NULL pointer, for chaos and madness await thee
at its end
- "Use the force Harry" - Gandalf, Star Trek II
#22 lore raw

2026. 03. 25. 15:17 keltezéssel, Bruce Ashfield írta:

Quoted 102 lines · Böszörményi Zoltán

2026. 03. 25. 14:17 keltezéssel, Bruce Ashfield írta:
>
>
> On Wed, Mar 25, 2026 at 9:13 AM Böszörményi Zoltán <[email protected]> wrote:
>
>     2026. 03. 25. 14:01 keltezéssel, Bruce Ashfield írta:
>     >
>     >
>     > On Wed, Mar 25, 2026 at 7:31 AM Zoltan Boszormenyi via lists.openembedded.org <http://lists.openembedded.org>
>     <http://lists.openembedded.org>
>     > <http://lists.openembedded.org> <[email protected]> wrote:
>     >
>     >     This patchset allows turning on Python freethreading by using >     >     a new DISTRO_FEATURES setting:
>     >
>     >     DISTRO_FEATURES += "python_freethreading"
>     >
>     >     and optionally:
>     >
>     >     DISTRO_FEATURES_NATIVE += "python_freethreading"
>     >
>     >     The change to python3-dir.bbclass is crucial, as all python module >     >     build helper classes rely on it.
>     >
>     >     I also added native variants of some settings because another >     >     series against meta-openembedded is needed to complete this one. >     >     In that, the change to the fontforge recipe relies on the native >     >     setting for setting BUILD_LDFLAGS.
>     >
>     >
>     > Looking at those other patches in the series, they don't seem to >     > be conditional on the distro feature.
>     >
>     > Are those changes valid for a build with and without freethreading ? >
>     Yes!
>
>     The main change is in python3-dir.bbclass, and the recipe tweaks >     are just adaptations to the fact that only ${PYTHON_DIR} is needed, >     not ${PYTHON_DIR}${PYTHON_ABI} from that point.
>
>     I was thinking that the introductory mail stressed this enough. >
>
> Obviously not :)

Obviously. :)

> I didn't get that impression at all from reading the introductory > message or looking at the patches themselves.
>
> You have the overview knowledge of the feature, why it is > useful, how it impacts existing recipes, how the migration would > work, what is the compatibility, etc.

I think the Python 3.14 release notes has enough details. See also https://www.phoronix.com/news/Python-3.14

The information needs to be distilled into your commit, not left to the patch reader to go off and read release notes. That's the point I'm trying to make.

With the two series I sent, only a single recipe is left (cmpi-bindings) in openembedded-core and meta-openembedded that uses
the PYTHON_ABI setting explicitly.

Migration (of python scripts) should not be a problem. If it worked with 3.14 as is, it works with freethreading enabled.

Compatibility is poor for a lot of modules, that clearly spell this out in their pyproject.toml or PKG-INFO:

     "Programming Language :: Python :: Free Threading :: 1 - Unstable"

or something not quite production level support.
It's still experimental for 3rd party modules, despite being officially supported for the interpreter and built-in modules in 3.14.

This is the reason this was an RFC series.

Anyway, the reason I was experimenting with this was that with heavily threaded python code, an import storm can cause a crash in 3.12.11 or the manually updated 3.13.12 in Yocto 5.3 in importlib._bootstrap according to the backtrace.

I would like to see if Python 3.14 (with and without freethreading) fixes such crashes.

Definitely worth putting in the RFC as well, the motivation for the change matters!

Quoted 6 lines
Quoted 2 lines

The patch reader doesn't, so that all needs to be clearly supplied.

Maybe next time for v2 after I have a complete image successfully built with freethreading enabled.

No such luck.

lldb in clang won't build because its swig source wants to play games with PyObject->ob_refcnt directly instead of using the proper macros. That struct member does not exist when freethreading is enabled. In fact, the whole struct _object (the PyObject implementation) is different when freethreading is enabled.

The llvm sources, even in the newly released 22.x and main do not have any reference to the Py_GIL_DISABLED macro, so as a whole, it does not support Python freethreading.

I can revisit this series a few year later when llvm adds the necessary support, which can be backported to older clang versions in my meta-clang-revival.

This concludes my little experiment.

That being said, I think the cleanup in the recipes is a good direction. PYTHON_DIR should include PYTHON_ABI in python3-dir.bbclass because build helper classes rely on it. Then PYTHON_DIR must be used consistently in recipes that need it, even if PYTHON_ABI stays empty.

Quoted 75 lines

Ack'd

Bruce

Quoted 29 lines

Bruce

     FWIW, less than 30 recipes was spelling the latter out in openembedded-core and meta-openembedded combined.

     > That's the type of information, along with the benefits of enabling > this would be information expected in a 0/N introductory patch. >
     > Cheers,
     >
     > Bruce
     >
     >
     >     All of the changes included here were successfully build tested, >     except for the gdb-cross-canadian change. That one failed, but >     it's included for completeness.
     >
     >     Quite a few of python modules were build tested for native and >     target builds, including python3-pytorch from my meta-python-ai >     with reduced RDEPENDS because e.g. python3-onnx there failed >     to build with freethreading enabled.
     >
     >     All in all, this series at least allows to get things going. >     Though maybe not for Yocto 6.0, as it's quite late. >
     >     Best regards,

Quoted 8 lines

     >     Zoltán Böszörményi
     >
     >     -=-=-=-=-=-=-=-=-=-=-=-
     >     Links: You receive all messages sent to this group.
     >     View/Reply Online (#233874):
     > https://lists.openembedded.org/g/openembedded-core/message/233874
     >     Mute This Topic: https://lists.openembedded.org/mt/118499141/1050810
     >     Group Owner: [email protected]

<mailto:openembedded-core%[email protected]>
>     <mailto:openembedded-core%[email protected] <mailto:openembedded-core%[email protected]>>
>     >     <mailto:openembedded-core%[email protected] <mailto:openembedded-core%[email protected]>
>     <mailto:openembedded-core%[email protected] <mailto:openembedded-core%[email protected]>>>
>     >     Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
>     >     [[email protected]]
>     >     -=-=-=-=-=-=-=-=-=-=-=-
>     >
>     >
>     >
>     > --
>     > - Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end
>     > - "Use the force Harry" - Gandalf, Star Trek II
>     >
>
>
>
> --
> - Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end > - "Use the force Harry" - Gandalf, Star Trek II
>

Signature · 2 lines
- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end
- "Use the force Harry" - Gandalf, Star Trek II

A message that is not in the archive (sent off-list or not yet received): [email protected]

#24 lore raw

2026. 03. 26. 6:29 keltezéssel, Zoltan Boszormenyi via lists.openembedded.org írta:

Quoted 126 lines

2026. 03. 25. 15:17 keltezéssel, Bruce Ashfield írta:

Quoted 102 lines · Böszörményi Zoltán

    2026. 03. 25. 14:17 keltezéssel, Bruce Ashfield írta:
    >
    >
    > On Wed, Mar 25, 2026 at 9:13 AM Böszörményi Zoltán <[email protected]> wrote:
    >
    >     2026. 03. 25. 14:01 keltezéssel, Bruce Ashfield írta:
    >     >
    >     >
    >     > On Wed, Mar 25, 2026 at 7:31 AM Zoltan Boszormenyi via lists.openembedded.org <http://lists.openembedded.org>
    >     <http://lists.openembedded.org>
    >     > <http://lists.openembedded.org> <[email protected]> wrote:
    >     >
    >     >     This patchset allows turning on Python freethreading by using >     >     a new DISTRO_FEATURES setting:
    >     >
    >     >     DISTRO_FEATURES += "python_freethreading"
    >     >
    >     >     and optionally:
    >     >
    >     >     DISTRO_FEATURES_NATIVE += "python_freethreading"
    >     >
    >     >     The change to python3-dir.bbclass is crucial, as all python module >     >     build helper classes rely on it.
    >     >
    >     >     I also added native variants of some settings because another >     >     series against meta-openembedded is needed to complete this one. >     >     In that, the change to the fontforge recipe relies on the native >     >     setting for setting BUILD_LDFLAGS.
    >     >
    >     >
    >     > Looking at those other patches in the series, they don't seem to >     > be conditional on the distro feature.
    >     >
    >     > Are those changes valid for a build with and without freethreading ? >
    >     Yes!
    >
    >     The main change is in python3-dir.bbclass, and the recipe tweaks >     are just adaptations to the fact that only ${PYTHON_DIR} is needed, >     not ${PYTHON_DIR}${PYTHON_ABI} from that point.
    >
    >     I was thinking that the introductory mail stressed this enough. >
    >
    > Obviously not :)

    Obviously. :)

    > I didn't get that impression at all from reading the introductory > message or looking at the patches themselves.
    >
    > You have the overview knowledge of the feature, why it is > useful, how it impacts existing recipes, how the migration would > work, what is the compatibility, etc.

    I think the Python 3.14 release notes has enough details. See also https://www.phoronix.com/news/Python-3.14

The information needs to be distilled into your commit, not left to the patch reader to go off and read release notes. That's the point I'm trying to make.

    With the two series I sent, only a single recipe is left (cmpi-bindings) in openembedded-core and meta-openembedded that uses
    the PYTHON_ABI setting explicitly.

    Migration (of python scripts) should not be a problem. If it worked with 3.14 as is, it works with freethreading enabled.

    Compatibility is poor for a lot of modules, that clearly spell this out in their pyproject.toml or PKG-INFO:

         "Programming Language :: Python :: Free Threading :: 1 - Unstable"

    or something not quite production level support.
    It's still experimental for 3rd party modules, despite being officially supported for the interpreter and built-in modules in 3.14.

    This is the reason this was an RFC series.

    Anyway, the reason I was experimenting with this was that with heavily threaded python code, an import storm can cause a crash in 3.12.11 or the manually updated 3.13.12 in Yocto 5.3 in importlib._bootstrap according to the backtrace.

    I would like to see if Python 3.14 (with and without freethreading) fixes such crashes.

Definitely worth putting in the RFC as well, the motivation for the change matters!

Quoted 6 lines

    > The patch reader doesn't, so that all needs to be clearly > supplied.

    Maybe next time for v2 after I have a complete image successfully built with freethreading enabled.

No such luck.

lldb in clang won't build because its swig source wants to play games with PyObject->ob_refcnt directly instead of using the proper macros. That struct member does not exist when freethreading is enabled. In fact, the whole struct _object (the PyObject implementation) is different when freethreading is enabled.

The llvm sources, even in the newly released 22.x and main do not have any reference to the Py_GIL_DISABLED macro, so as a whole, it does not support Python freethreading.

I can revisit this series a few year later when llvm adds the necessary support, which can be backported to older clang versions in my meta-clang-revival.

Quoted 6 lines

This concludes my little experiment.

That being said, I think the cleanup in the recipes is a good direction. PYTHON_DIR should include PYTHON_ABI in python3-dir.bbclass because build helper classes rely on it. Then PYTHON_DIR must be used consistently in recipes that need it, even if PYTHON_ABI stays empty.

Which means I can re-send the whole series as a cleanup or preparation series without the DISTRO_FEATURES enablement part in python3-dir.bbclass.

That one can stay RFC as a standalone patch, separate from the cleanup series.

I will re-test my build without enabling freethreading before I would do that.

Quoted 71 lines
Quoted 46 lines

Ack'd

Bruce

    >
    > Bruce
    >
    >
    >
    >     FWIW, less than 30 recipes was spelling the latter out in
    >     openembedded-core and meta-openembedded combined.
    >
    >     > That's the type of information, along with the benefits of enabling >     > this would be information expected in a 0/N introductory patch. >     >
    >     > Cheers,
    >     >
    >     > Bruce
    >     >
    >     >
    >     >     All of the changes included here were successfully build tested, >     >     except for the gdb-cross-canadian change. That one failed, but >     >     it's included for completeness.
    >     >
    >     >     Quite a few of python modules were build tested for native and >     >     target builds, including python3-pytorch from my meta-python-ai >     >     with reduced RDEPENDS because e.g. python3-onnx there failed >     >     to build with freethreading enabled.
    >     >
    >     >     All in all, this series at least allows to get things going. >     >     Though maybe not for Yocto 6.0, as it's quite late.
    >     >
    >     >     Best regards,
    >     >     Zoltán Böszörményi
    >     >
    >     >
    >     >
    >     >
    >     >
    >     > --
    >     > - Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end
    >     > - "Use the force Harry" - Gandalf, Star Trek II

Quoted 15 lines

    >     >
    >
    >
    >
    > --
    > - Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end > - "Use the force Harry" - Gandalf, Star Trek II
    >

Signature · 2 lines
- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end
- "Use the force Harry" - Gandalf, Star Trek II

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#233951): https://lists.openembedded.org/g/openembedded-core/message/233951
Mute This Topic: https://lists.openembedded.org/mt/118499141/3617728
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-