Enable Python free-threading
cover Enable Python free-threading
Enable Python free-threading and fix some obvious fallouts. See https://docs.python.org/3/howto/free-threading-python.html
Quote:
==================================================================
Python support for free threading
Starting with the 3.13 release, CPython has support for a build of Python called free threading where the global interpreter lock (GIL) is disabled. Free-threaded execution allows for full utilization of the available processing power by running threads in parallel on available CPU cores. While not all software will benefit from this automatically, programs designed with threading in mind will run faster on multi-core hardware.
Some third-party packages, in particular ones with an extension module, may not be ready for use in a free-threaded build, and will re-enable the GIL.
==================================================================
See also for the the post-release announcement on Phoronix and a benchmark:
https://www.phoronix.com/news/Python-3.14
https://www.phoronix.com/review/python-314-benchmarks
The reasons behind this series are:
* Python free-threading makes threaded Python scripts faster by not stalling each other while one of them is parsed.
* Even with the latest 3.13.12, highly threaded scripts may cause crashes in the Python interpreter with a backtrace ending similarly as below and Google search results pointing to a so called "import storm" that makes Python unstable.
Thread 0x00007f90ebfff6c0 (most recent call first):
File "<frozen importlib._bootstrap>", line 446 in cb
File "<frozen importlib._bootstrap>", line 1357 in _find_and_load File "<frozen importlib._bootstrap>", line 488 in _call_with_frames_removed File "<frozen importlib._bootstrap>", line 1415 in _handle_fromlist
...
I was interested to see whether Python 3.14 with free-threading enabled can fix it.
This series builds on the previous v2 series with the cover letter entitled "Use PYTHON_DIR consistently".
Unfortunately, this series by itself is not enough and a lot of recipes won't build:
* lldb in clang uses embedded Python code in C/C++ and uses the internals of the PyObject implementation, which changes when free-threading is enabled. As a result, lldb won't build. It may be possible to conditionally disable lldb.
* Many Python modules with binary extensions similary do not build against the free-threaded Python.
* Many Python modules, although build, do not work correctly. Usually they include thio, or something similarly not quite production level indicator line in their PKG-INFO or pyproject.toml
"Programming Language :: Python :: Free Threading :: 1 - Unstable"
In other words: DO NOT TRY THIS AT HOME.
RFC PATCH 1/4 xcb-proto: Support Python free-threading
libxcb won't find xcb-proto when Python free-threading is enabled.
This is caused by two problems:
* xcb-proto installs its python module into the ABI-less site-packages directory
* the pythondir= setting in xcb-proto.pc also points to the ABI-less site-packages directory
Move the installed site-packages directory to the correct location if needed, and fix xcb-proto.pc.
--- meta/recipes-graphics/xorg-proto/xcb-proto_1.17.0.bb | 9 +++++++++ 1 file changed, 9 insertions(+)
meta/recipes-graphics/xorg-proto/xcb-proto_1.17.0.bb +9 -0
@@ -19,6 +19,15 @@ SRC_URI[sha256sum] = "2c1bacd2110f4799f74de6ebb714b94cf6f80fb112316b1219480fd225 inherit autotools pkgconfig python3native +do_install:append () { + # Patch pythondir site-packages + sed -i 's:}/.*/site-packages:}/${PYTHON_DIR}/site-packages:' ${D}${libdir}/pkgconfig/xcb-proto.pc + + if [ ! -d ${D}${libdir}/${PYTHON_DIR} ]; then + mv ${D}${libdir}/python${PYTHON_BASEVERSION} ${D}${libdir}/${PYTHON_DIR} + fi +} + PACKAGES += "python-xcbgen" FILES:${PN} = ""
Signature · 1 line
2.53.0
RFC PATCH 2/4 rpm: Support Python freethreading
In reply to Zoltán BöszörményiAdded a patch to stop using the stable CPython API (of 3.7!) as it prevents building rpm against a Python build with freethreading enabled.
--- .../files/0001-rpm-Don-t-use-stable-API.patch | 37 +++++++++++++++++++ meta/recipes-devtools/rpm/rpm_4.20.1.bb | 1 + 2 files changed, 38 insertions(+) create mode 100644 meta/recipes-devtools/rpm/files/0001-rpm-Don-t-use-stable-API.patch
meta/recipes-devtools/rpm/files/0001-rpm-Don-t-use-stable-API.patch new file +37 -0
@@ -0,0 +1,37 @@ +From 2fb202730b49538bbe6d1439d419c18a89eb4c91 Mon Sep 17 00:00:00 2001 +From: =?UTF-8?q?Zolt=C3=A1n=20B=C3=B6sz=C3=B6rm=C3=A9nyi?= + <[email protected]> +Date: Wed, 25 Mar 2026 08:20:13 +0100 +Subject: [PATCH] rpm: Don't use stable API +MIME-Version: 1.0 +Content-Type: text/plain; charset=UTF-8 +Content-Transfer-Encoding: 8bit + +With Python freethreading enabled, these explicit errors are +triggered in the Python headers: + + #error "The limited API is not currently supported in the free-threaded build" + +Signed-off-by: Zoltán Böszörményi <[email protected]> +Upstream-Status: Inappropriate [python freethreading specific] +--- + python/CMakeLists.txt | 3 --- + 1 file changed, 3 deletions(-) + +diff --git a/python/CMakeLists.txt b/python/CMakeLists.txt +index acc003ad6..3f21205c8 100644 +--- a/python/CMakeLists.txt ++++ b/python/CMakeLists.txt +@@ -18,9 +18,6 @@ Python3_add_library(_rpm + spec-py.c spec-py.h + ) + +-# Select Python stable ABI +-target_compile_definitions(_rpm PRIVATE Py_LIMITED_API=0x03070000) +- + target_link_libraries(_rpm PRIVATE librpmio librpm librpmbuild librpmsign) + + install(TARGETS _rpm +-- +2.53.0 +
meta/recipes-devtools/rpm/rpm_4.20.1.bb +1 -0
@@ -39,6 +39,7 @@ SRC_URI = "git://github.com/rpm-software-management/rpm;branch=rpm-4.20.x;protoc file://0002-rpmio-rpmglob.c-avoid-using-GLOB_BRACE-if-undefined-.patch \ file://0001-CMakeLists.txt-set-libdir-to-CMAKE_INSTALL_FULL_LIBD.patch \ file://0001-CMakeLists.txt-Fix-checking-for-CFLAGS.patch \ + file://0001-rpm-Don-t-use-stable-API.patch \ " PE = "1"
Signature · 1 line
2.53.0
RFC PATCH 3/4 libcap-ng-python: Rename site-packages path if needed
In reply to Zoltán Böszörményilibcap-ng-python installs into the ABI-less ${libdir}/pythonX.Y directory, even if Python free-threading is enabled.
Move it to the correct location.
--- meta/recipes-support/libcap-ng/libcap-ng-python_0.9.1.bb | 3 +++ 1 file changed, 3 insertions(+)
meta/recipes-support/libcap-ng/libcap-ng-python_0.9.1.bb +3 -0
@@ -12,6 +12,9 @@ EXTRA_OECONF += "--with-python3" do_install() { oe_runmake 'DESTDIR=${D}' install -C ${B}/bindings/python3 + if [ ! -d ${D}${libdir}/${PYTHON_DIR} ]; then + mv ${D}${libdir}/python${PYTHON_BASEVERSION} ${D}${libdir}/${PYTHON_DIR} + fi } FILES:${PN} = "${libdir}/${PYTHON_DIR}"
Signature · 1 line
2.53.0
RFC PATCH 4/4 python3, python3-dir.bbclass: Support Python free-threading
In reply to Zoltán BöszörményiA new DISTRO_FEATURE[_NATIVE] = "python_freethreading" sets PYTHON_ABI[_NATIVE]. This enables PACKAGECONFIG[freethreading] for python3 and allows building Python modules.
Also added PYTHON_MAINVERSION_NATIVE and PYTHON_DIR_NATIVE in an anticipation that they may be different.
--- meta/classes-recipe/python3-dir.bbclass | 5 ++++- meta/recipes-devtools/python/python3_3.14.3.bb | 6 +++++- 2 files changed, 9 insertions(+), 2 deletions(-)
meta/classes-recipe/python3-dir.bbclass +4 -1
@@ -5,8 +5,11 @@ # PYTHON_BASEVERSION = "3.14" -PYTHON_ABI = "" +PYTHON_ABI = "${@bb.utils.contains('DISTRO_FEATURES', 'python_freethreading', 't', '', d)}" +PYTHON_ABI_NATIVE = "${@bb.utils.contains('DISTRO_FEATURES_NATIVE', 'python_freethreading', 't', '', d)}" PYTHON_MAINVERSION = "${PYTHON_BASEVERSION}${PYTHON_ABI}" +PYTHON_MAINVERSION_NATIVE = "${PYTHON_BASEVERSION}${PYTHON_ABI_NATIVE}" PYTHON_DIR = "python${PYTHON_MAINVERSION}" +PYTHON_DIR_NATIVE = "python${PYTHON_MAINVERSION_NATIVE}" PYTHON_PN = "python3" PYTHON_SITEPACKAGES_DIR = "${libdir}/${PYTHON_DIR}/site-packages"
meta/recipes-devtools/python/python3_3.14.3.bb +5 -1
@@ -109,7 +109,11 @@ CACHED_CONFIGUREVARS:append:libc-musl = "\ " # PGO currently causes builds to not be reproducible so disable by default, see YOCTO #13407 -PACKAGECONFIG ??= "editline gdbm ${@bb.utils.filter('DISTRO_FEATURES', 'lto', d)}" +# For freethreading, see the comment below. +PACKAGECONFIG ??= "editline gdbm \ + ${@bb.utils.filter('DISTRO_FEATURES', 'lto', d)} \ + ${@bb.utils.contains('DISTRO_FEATURES', 'python_freethreading', 'freethreading', '', d)} \ +" # Some LLVM/CLANG subprojects (e.g. lldb) and many Python modules # do not build when free-threading is enabled. Also, the support
Signature · 1 line
2.53.0