Skip to contentSkip to conversation

deprecation of distutils

PATCH In review Tim Orling 2 participants 25 messages Jan 11, 2022

lore
Share & tools
lore

Tim Orling

author
#0 lore raw

v2 cover deprecation of distutils

distutils is deprecated in Python 3.10 and will be removed in Python 3.12 (~October 2023).

This series inserts a deprecation warning into the distutils*.bbclasses and drops or replaces distutils usage wherever possible.

Instead of distutils3-base, we introduce setuptools3-base, which also incorporates distutils-common-base.

Remaining usage is in:
* lib/recipetool/create_buildsys_python.py until we are ready to factor it out (and add pyproject.toml and setup.cfg handling)
* oeqa selftests which are testing distutils3 functionality
* glib-2.0: upstream is still using distutils.version.LooseVersion
* gpgme: Still checks for distutils in m4/ax_python_devel.m4
* python3-numpy: upstream still has a distutils/ module
* python3-setuptools has a vendored _distutils

I might have missed something else.

No attempt has been made to look at layers outside of oe-core.

The intent is to let this percolate for a few weeks and then drop distutils*.bbclasses (and move them to meta-python, but with the warnings).

[YOCTO #14610]

Changes in v2:
- dropped the import of distutils.version into bb. in favor of bb.utils.vercmp_string_op()
- dropped some patches that have been addressed already
- update email
- fix typos
- drop already merged scripts/buildhistory-diff patch
The following changes since commit 16110cdca7211b0efc59464afbb253527c033cad:

oeqa/sstate: Fix allarch samesigs test (2022-01-11 15:48:11 +0000)

are available in the Git repository at:

git://git.yoctoproject.org/git/poky-contrib timo/deprecate_distutils_14610_v2
http://git.yoctoproject.org/cgit.cgi/git/poky-contrib/log/?h=timo/deprecate_distutils_14610_v2

Tim Orling (22):
classes/distutils-*: add warning of deprecation
setuptools3: refactor for no distutils bbclasses
python3-cython: DISTUTILS_* -> SETUPTOOLS_*
dnf: inherit setuptools3-base instead of distutils
libdnf: inherit setuptools3-base not distutils
python3-dbus: inherit setuptools3-base not distuils
gpgme: inherit setuptools-base not distuils
python3-pip: inherit setuptools3 not distuils3
systemtap: use setuptools-base not distutils
libcomps: inherit setuptools3-base not distutils
createrepo-c: inherit setuptools3-base not distutils
librepo: inherit setuptools3-base not distutils
btrfs-tools: use setuptools3-base not distutils
python3-pygobject: setuptools instead of distuils
bind: inherit setuptools3-base instead of distutils
python3-setuptools: do not depend on distutils gstreamer1.0-python: inherit setuptools3-base recipetool/create_buildsys_python.py: less distutils waf.bbclass: drop usage of distutils
dnf: drop python3-distutils dependency python3native.bbclass: distutils -> sysconfig rootfs_rpm.bbclass: distutils -> sysconfig module

 meta/classes/distutils-common-base.bbclass    |  3 +
 meta/classes/distutils3-base.bbclass          |  3 +
 meta/classes/distutils3.bbclass               |  4 ++
 meta/classes/python3native.bbclass            |  2 +-
 meta/classes/rootfs_rpm.bbclass               |  2 +-
 meta/classes/setuptools3-base.bbclass         | 31 +++++++++
 meta/classes/setuptools3.bbclass              | 66 ++++++++++++++++++-
 meta/classes/waf.bbclass                      |  3 +-
 .../recipes-connectivity/bind/bind_9.16.24.bb |  2 +-
 .../btrfs-tools/btrfs-tools_5.15.1.bb         |  2 +-
 .../createrepo-c/createrepo-c_0.18.0.bb       |  2 +-
 meta/recipes-devtools/dnf/dnf_4.10.0.bb       |  3 +-
 .../libcomps/libcomps_0.1.18.bb               |  2 +-
 meta/recipes-devtools/libdnf/libdnf_0.65.0.bb |  2 +-
 .../librepo/librepo_1.14.2.bb                 |  2 +-
 .../python/python3-cython_0.29.26.bb          |  2 +-
 .../python/python3-dbus_1.2.18.bb             |  2 +-
 .../python/python3-pip_21.3.1.bb              |  2 +-
 .../python/python3-pygobject_3.42.0.bb        |  2 +-
 .../python/python3-setuptools_59.5.0.bb       |  1 -
 .../recipes-kernel/systemtap/systemtap_git.bb |  2 +-
 .../gstreamer/gstreamer1.0-python_1.18.5.bb   |  2 +-
 meta/recipes-support/gpgme/gpgme_1.16.0.bb    |  2 +-
 .../lib/recipetool/create_buildsys_python.py  |  8 ++-
 24 files changed, 129 insertions(+), 23 deletions(-)
 create mode 100644 meta/classes/setuptools3-base.bbclass
Signature · 1 line
2.30.2
author
#1 lore raw

PATCH v2 1/22 classes/distutils-*: add warning of deprecation

distutils has been deprecated in Python 3.10 and will be removed in Python 3.12 (predicted release date October 2023).

Add a warning now that recommends using the roughly equivalent setuptools-* classes.

[YOCTO #14610]

---
 meta/classes/distutils-common-base.bbclass | 3 +++
 meta/classes/distutils3-base.bbclass       | 3 +++
 meta/classes/distutils3.bbclass            | 4 ++++
 3 files changed, 10 insertions(+)
meta/classes/distutils-common-base.bbclass +3 -0
index bc994f78de3..59c750a3cf7 100644
@@ -23,3 +23,6 @@ FILES:${PN}-dev += "\
   ${libdir}/pkgconfig \
   ${PYTHON_SITEPACKAGES_DIR}/*.la \
 "
+python __anonymous() {
+    bb.warn("distutils-common-base.bbclass is deprecated, please use setuptools3-base.bbclass instead")
+}
meta/classes/distutils3-base.bbclass +3 -0
index d41873e4869..850c535bb17 100644
@@ -4,3 +4,6 @@ RDEPENDS:${PN} += "${@['', '${PYTHON_PN}-core']['${CLASSOVERRIDE}' == 'class-tar
 
 inherit distutils-common-base python3native python3targetconfig
 
+python __anonymous() {
+    bb.warn("distutils3-base.bbclass is deprecated, please use setuptools3-base.bbclass instead")
+
meta/classes/distutils3.bbclass +4 -0
index be645d37bd0..a6d8e8763f3 100644
@@ -14,6 +14,10 @@ DISTUTILS_PYTHON:class-native = "nativepython3"
 
 DISTUTILS_SETUP_PATH ?= "${S}"
 
+python __anonymous() {
+    bb.warn("distutils3.bbclass is deprecated, please use setuptools3.bbclass instead")
+}
+
 distutils3_do_configure() {
     :
 }
Signature · 1 line
2.30.2
author
#2 lore raw

PATCH v2 2/22 setuptools3: refactor for no distutils bbclasses

In reply to Tim Orling

Add setuptools3-base.bbclass as a re-usable starting point similar to what used to be distutils-common-base.bbclass and disutils3-base.bbclass.

We no longer need to support python2, so no need for a setuptools-common-base.bbclass.

Refactor setuptools3.bbclass to use setuptools3-base.bbclass instead of the distulis*.bbclasses.

---
Changes in v2:
- Drop distutils_do_configure task dependency (copy paste error)

 meta/classes/setuptools3-base.bbclass | 31 +++++++++++++
 meta/classes/setuptools3.bbclass      | 66 ++++++++++++++++++++++++++-
 2 files changed, 96 insertions(+), 1 deletion(-)
 create mode 100644 meta/classes/setuptools3-base.bbclass
meta/classes/setuptools3-base.bbclass new file +31 -0
new file mode 100644
index 00000000000..5098ae9d64e
@@ -0,0 +1,31 @@
+DEPENDS:append:class-target = " ${PYTHON_PN}-native ${PYTHON_PN}"
+DEPENDS:append:class-nativesdk = " ${PYTHON_PN}-native ${PYTHON_PN}"
+RDEPENDS:${PN} += "${@['', '${PYTHON_PN}-core']['${CLASSOVERRIDE}' == 'class-target']}"
+
+export STAGING_INCDIR
+export STAGING_LIBDIR
+
+# LDSHARED is the ld *command* used to create shared library
+export LDSHARED  = "${CCLD} -shared"
+# LDXXSHARED is the ld *command* used to create shared library of C++
+# objects
+export LDCXXSHARED  = "${CXX} -shared"
+# CCSHARED are the C *flags* used to create objects to go into a shared
+# library (module)
+export CCSHARED  = "-fPIC -DPIC"
+# LINKFORSHARED are the flags passed to the $(CC) command that links
+# the python executable
+export LINKFORSHARED = "${SECURITY_CFLAGS} -Xlinker -export-dynamic"
+
+FILES:${PN} += "${libdir}/* ${libdir}/${PYTHON_DIR}/*"
+
+FILES:${PN}-staticdev += "\
+  ${PYTHON_SITEPACKAGES_DIR}/*.a \
+"
+FILES:${PN}-dev += "\
+  ${datadir}/pkgconfig \
+  ${libdir}/pkgconfig \
+  ${PYTHON_SITEPACKAGES_DIR}/*.la \
+"
+inherit python3native python3targetconfig
+
meta/classes/setuptools3.bbclass +65 -1

69 lines not on this page · raw message

Signature · 1 line
2.30.2
author
#4 lore raw

PATCH v2 4/22 dnf: inherit setuptools3-base instead of distutils

In reply to Tim Orling

[YOCTO #14610]

---
 meta/recipes-devtools/dnf/dnf_4.10.0.bb | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
meta/recipes-devtools/dnf/dnf_4.10.0.bb +1 -1
index cef2c03e4e2..6ff2a6b1e6f 100644
@@ -23,7 +23,7 @@ UPSTREAM_CHECK_GITTAGREGEX = "(?P<pver>\d+(\.\d+)+)"
 
 S = "${WORKDIR}/git"
 
-inherit cmake gettext bash-completion distutils3-base systemd
+inherit cmake gettext bash-completion setuptools3-base systemd
 
 DEPENDS += "libdnf librepo libcomps python3-iniparse"
 
Signature · 1 line
2.30.2
author
#5 lore raw

PATCH v2 5/22 libdnf: inherit setuptools3-base not distutils

In reply to Tim Orling

[YOCTO #14610]

---
 meta/recipes-devtools/libdnf/libdnf_0.65.0.bb | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
meta/recipes-devtools/libdnf/libdnf_0.65.0.bb +1 -1
index fbda7769c07..81da04c9ca1 100644
@@ -22,7 +22,7 @@ S = "${WORKDIR}/git"
 
 DEPENDS = "glib-2.0 libsolv libcheck librepo rpm gtk-doc libmodulemd json-c swig-native"
 
-inherit gtk-doc gobject-introspection cmake pkgconfig distutils3-base
+inherit gtk-doc gobject-introspection cmake pkgconfig setuptools3-base
 
 EXTRA_OECMAKE = " -DPYTHON_INSTALL_DIR=${PYTHON_SITEPACKAGES_DIR} -DWITH_MAN=OFF -DPYTHON_DESIRED=3 \
                   ${@bb.utils.contains('GI_DATA_ENABLED', 'True', '-DWITH_GIR=ON', '-DWITH_GIR=OFF', d)} \
Signature · 1 line
2.30.2
author
#3 lore raw

PATCH v2 3/22 python3-cython: DISTUTILS_* -> SETUPTOOLS_*

In reply to Tim Orling

Refactor to new SETUPTOOLS_* variables

[YOCTO #14610]

---
 meta/recipes-devtools/python/python3-cython_0.29.26.bb | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
meta/recipes-devtools/python/python3-cython_0.29.26.bb +1 -1
index b1d21bd15c0..26333cb2718 100644
@@ -8,7 +8,7 @@ RDEPENDS:${PN} += "\
 # running build_ext a second time during install fails, because Python
 # would then attempt to import cythonized modules built for the target
 # architecture.
-DISTUTILS_INSTALL_ARGS += "--skip-build"
+SETUPTOOLS_INSTALL_ARGS += "--skip-build"
 
 do_install:append() {
     # rename scripts that would conflict with the Python 2 build of Cython
Signature · 1 line
2.30.2
author
#7 lore raw

PATCH v2 7/22 gpgme: inherit setuptools-base not distuils

In reply to Tim Orling

What was distutils-common-base has been merged into setuptools3-base, since we no longer need to support python2 and python3.

[YOCTO #14610]

---
 meta/recipes-support/gpgme/gpgme_1.16.0.bb | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
meta/recipes-support/gpgme/gpgme_1.16.0.bb +1 -1
index 0ab30d93b9e..d31e1b3132e 100644
@@ -59,7 +59,7 @@ EXTRA_OECONF += '--enable-languages="${LANGUAGES}" \
                  --disable-lang-python-test \
 '
 
-inherit autotools texinfo binconfig-disabled pkgconfig distutils-common-base ${PYTHON_INHERIT} multilib_header
+inherit autotools texinfo binconfig-disabled pkgconfig setuptools3-base ${PYTHON_INHERIT} multilib_header
 
 export PKG_CONFIG='pkg-config'
 
Signature · 1 line
2.30.2
author
#8 lore raw

PATCH v2 8/22 python3-pip: inherit setuptools3 not distuils3

In reply to Tim Orling

[YOCTO #14610]

---
 meta/recipes-devtools/python/python3-pip_21.3.1.bb | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
meta/recipes-devtools/python/python3-pip_21.3.1.bb +1 -1
index 6ddb1d62966..c893135059b 100644
@@ -6,7 +6,7 @@ LIC_FILES_CHKSUM = "file://LICENSE.txt;md5=c4fa2b50f55649f43060fa04b0919b9b"
 
 DEPENDS += "python3 python3-setuptools-native"
 
-inherit pypi distutils3
+inherit pypi setuptools3
 
 SRC_URI += "file://0001-change-shebang-to-python3.patch"
 
Signature · 1 line
2.30.2
author
#6 lore raw

PATCH v2 6/22 python3-dbus: inherit setuptools3-base not distuils

In reply to Tim Orling

[YOCTO #14610]

---
 meta/recipes-devtools/python/python3-dbus_1.2.18.bb | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
meta/recipes-devtools/python/python3-dbus_1.2.18.bb +1 -1
index 9592fbf7169..c4687de13c2 100644
@@ -11,7 +11,7 @@ SRC_URI[sha256sum] = "92bdd1e68b45596c833307a5ff4b217ee6929a1502f5341bae28fd120a
 
 S = "${WORKDIR}/dbus-python-${PV}"
 
-inherit distutils3-base autotools pkgconfig
+inherit setuptools3-base autotools pkgconfig
 
 # documentation needs python3-sphinx, which is not in oe-core or meta-python for now
 # change to use PACKAGECONFIG when python3-sphinx is added to oe-core
Signature · 1 line
2.30.2
author
#9 lore raw

PATCH v2 9/22 systemtap: use setuptools-base not distutils

In reply to Tim Orling

[YOCTO #14610]

---
 meta/recipes-kernel/systemtap/systemtap_git.bb | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
meta/recipes-kernel/systemtap/systemtap_git.bb +1 -1
index d11d9b93a2d..04f7e03ab50 100644
@@ -31,7 +31,7 @@ PACKAGECONFIG[monitor] = "--enable-monitor,--disable-monitor,ncurses json-c"
 PACKAGECONFIG[python3-probes] = "--with-python3-probes,--without-python3-probes,python3-setuptools-native"
 
 inherit autotools gettext pkgconfig systemd
-inherit ${@bb.utils.contains('PACKAGECONFIG', 'python3-probes', 'distutils3-base', '', d)}
+inherit ${@bb.utils.contains('PACKAGECONFIG', 'python3-probes', 'setuptools3-base', '', d)}
 
 # exporter comes with python3-probes
 PACKAGES =+ "${PN}-exporter"
Signature · 1 line
2.30.2
author
#10 lore raw

PATCH v2 10/22 libcomps: inherit setuptools3-base not distutils

In reply to Tim Orling

[YOCTO #14610]

---
 meta/recipes-devtools/libcomps/libcomps_0.1.18.bb | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
meta/recipes-devtools/libcomps/libcomps_0.1.18.bb +1 -1
index ff6ecc76017..6603275d133 100644
@@ -12,7 +12,7 @@ SRCREV = "dee4ae37f7818709802de28c4d16fa823bd83ae2"
 
 S = "${WORKDIR}/git"
 
-inherit cmake distutils3-base
+inherit cmake setuptools3-base
 
 DEPENDS += "libxml2 expat libcheck"
 
Signature · 1 line
2.30.2
author
#11 lore raw

PATCH v2 11/22 createrepo-c: inherit setuptools3-base not distutils

In reply to Tim Orling

[YOCTO #14610]

---
 meta/recipes-devtools/createrepo-c/createrepo-c_0.18.0.bb | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
meta/recipes-devtools/createrepo-c/createrepo-c_0.18.0.bb +1 -1
index e249b767ffa..d5fa38d300d 100644
@@ -15,7 +15,7 @@ S = "${WORKDIR}/git"
 DEPENDS = "expat curl glib-2.0 libxml2 openssl bzip2 zlib file sqlite3 xz rpm"
 DEPENDS:append:class-native = " file-replacement-native"
 
-inherit cmake pkgconfig bash-completion distutils3-base
+inherit cmake pkgconfig bash-completion setuptools3-base
 
 EXTRA_OECMAKE = " -DPYTHON_INSTALL_DIR=${PYTHON_SITEPACKAGES_DIR} -DPYTHON_DESIRED=3 -DWITH_ZCHUNK=OFF -DENABLE_DRPM=OFF -DWITH_LIBMODULEMD=OFF"
 
Signature · 1 line
2.30.2
author
#12 lore raw

PATCH v2 12/22 librepo: inherit setuptools3-base not distutils

In reply to Tim Orling

[YOCTO #14610]

---
 meta/recipes-devtools/librepo/librepo_1.14.2.bb | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
meta/recipes-devtools/librepo/librepo_1.14.2.bb +1 -1
index 61a485cfe89..6cd8090b556 100644
@@ -16,7 +16,7 @@ S = "${WORKDIR}/git"
 
 DEPENDS = "curl glib-2.0 openssl attr gpgme libxml2"
 
-inherit cmake distutils3-base pkgconfig
+inherit cmake setuptools3-base pkgconfig
 
 EXTRA_OECMAKE = " \
     -DPYTHON_INSTALL_DIR=${PYTHON_SITEPACKAGES_DIR} \
Signature · 1 line
2.30.2
author
#13 lore raw

PATCH v2 13/22 btrfs-tools: use setuptools3-base not distutils

In reply to Tim Orling

[YOCTO #14610]

---
 meta/recipes-devtools/btrfs-tools/btrfs-tools_5.15.1.bb | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
meta/recipes-devtools/btrfs-tools/btrfs-tools_5.15.1.bb +1 -1
index 9f3c0e9abb0..3cf216724e0 100644
@@ -43,7 +43,7 @@ PACKAGECONFIG[crypto-libsodium] = "--with-crypto=libsodium,,libsodium"
 PACKAGECONFIG[crypto-libkcapi] = "--with-crypto=libkcapi,,libkcapi"
 
 inherit autotools-brokensep pkgconfig manpages
-inherit ${@bb.utils.contains('PACKAGECONFIG', 'python', 'distutils3-base', '', d)}
+inherit ${@bb.utils.contains('PACKAGECONFIG', 'python', 'setuptools3-base', '', d)}
 
 CLEANBROKEN = "1"
 
Signature · 1 line
2.30.2
author
#14 lore raw

PATCH v2 14/22 python3-pygobject: setuptools instead of distuils

In reply to Tim Orling

[YOCTO #14610]

---
 meta/recipes-devtools/python/python3-pygobject_3.42.0.bb | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
meta/recipes-devtools/python/python3-pygobject_3.42.0.bb +1 -1
index b07969c5efb..74318337bad 100644
@@ -8,7 +8,7 @@ LIC_FILES_CHKSUM = "file://COPYING;md5=a916467b91076e631dd8edb7424769c7"
 GNOMEBASEBUILDCLASS = "meson"
 GIR_MESON_OPTION = ""
 
-inherit gnomebase distutils3-base gobject-introspection upstream-version-is-even
+inherit gnomebase setuptools3-base gobject-introspection upstream-version-is-even
 
 DEPENDS += "python3 glib-2.0"
 
Signature · 1 line
2.30.2
author
#15 lore raw

PATCH v2 15/22 bind: inherit setuptools3-base instead of distutils

In reply to Tim Orling

[YOCTO #14610]

---
 meta/recipes-connectivity/bind/bind_9.16.24.bb | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
meta/recipes-connectivity/bind/bind_9.16.24.bb +1 -1
index c9e6f79dba8..ef5fd42ef85 100644
@@ -46,7 +46,7 @@ EXTRA_OECONF = " --with-libtool --disable-devpoll --disable-auto-validation --en
                "
 LDFLAGS:append = " -lz"
 
-inherit ${@bb.utils.contains('PACKAGECONFIG', 'python3', 'python3native distutils3-base', '', d)}
+inherit ${@bb.utils.contains('PACKAGECONFIG', 'python3', 'python3native setuptools3-base', '', d)}
 
 # dhcp needs .la so keep them
 REMOVE_LIBTOOL_LA = "0"
Signature · 1 line
2.30.2
author
#16 lore raw

PATCH v2 16/22 python3-setuptools: do not depend on distutils

In reply to Tim Orling

No longer depend on python3-distutils, current versions of setuptools vendor as _distutils.

[YOCTO #14610]

---
 meta/recipes-devtools/python/python3-setuptools_59.5.0.bb | 1 -
 1 file changed, 1 deletion(-)
meta/recipes-devtools/python/python3-setuptools_59.5.0.bb +0 -1
index 878fa084045..33ca928c3e5 100644
@@ -22,7 +22,6 @@ RDEPENDS:${PN} = "\
     ${PYTHON_PN}-compile \
     ${PYTHON_PN}-compression \
     ${PYTHON_PN}-ctypes \
-    ${PYTHON_PN}-distutils \
     ${PYTHON_PN}-email \
     ${PYTHON_PN}-html \
     ${PYTHON_PN}-json \
Signature · 1 line
2.30.2
author
#17 lore raw

PATCH v2 17/22 gstreamer1.0-python: inherit setuptools3-base

In reply to Tim Orling

Drop distutils3-base usage as it is now deprecated.

[YOCTO #14610]

---
 meta/recipes-multimedia/gstreamer/gstreamer1.0-python_1.18.5.bb | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
meta/recipes-multimedia/gstreamer/gstreamer1.0-python_1.18.5.bb +1 -1
index 045aea44099..1dd7d0d09a3 100644
@@ -23,4 +23,4 @@ EXTRA_OEMESON += "-Dlibpython-dir=${libdir}"
 REQUIRED_DISTRO_FEATURES = "gobject-introspection-data"
 GIR_MESON_OPTION = ""
 
-inherit meson pkgconfig distutils3-base upstream-version-is-even gobject-introspection features_check
+inherit meson pkgconfig setuptools3-base upstream-version-is-even gobject-introspection features_check
Signature · 1 line
2.30.2
author
#18 lore raw

PATCH v2 18/22 recipetool/create_buildsys_python.py: less distutils

In reply to Tim Orling

'distutils' is deprecated in Python 3.10 with removal in Python 3.12 (~October 2023). Replace 'distutils.command.build_py' with 'setuptools.command.build_py'.

To avoid an AttributeError, we call super().__init__() which provides the missing 'distribution' attribute. However, for some reason, __init__() in 'setuptools.command.build_py.build_py' class requires a 'dist' positional argument which must be a 'Distribution' instance. It is not clear why 'distutils.command.build_py.build_py' class does not require this.

There is still a check which decides to inherit setuptools3 vs distutils3 that will need to be refactored when we add pyproject.toml and setup.cfg support for more modern PEP 517 packaging.

Once distutils3.bbclass is dropped, any recipe inheriting distutils3 will throw a parsing error. The plan is to move distutils*.bbclasses to meta-python. However if meta-python is not in bblayers, the parsing error would still occur.

[YOCTO #14610]

---
 scripts/lib/recipetool/create_buildsys_python.py | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)
scripts/lib/recipetool/create_buildsys_python.py +6 -2
index 0b6b042ed17..1a150116872 100644
@@ -8,7 +8,7 @@
 import ast
 import codecs
 import collections
-import distutils.command.build_py
+import setuptools.command.build_py
 import email
 import imp
 import glob
@@ -459,9 +459,13 @@ class PythonRecipeHandler(RecipeHandler):
         else:
             package_dir = {}
 
-        class PackageDir(distutils.command.build_py.build_py):
+        dist = setuptools.Distribution()
+
+        class PackageDir(setuptools.command.build_py.build_py):
             def __init__(self, package_dir):
                 self.package_dir = package_dir
+                self.dist = dist
+                super().__init__(self.dist)
 
         pd = PackageDir(package_dir)
         to_scan = []
Signature · 1 line
2.30.2
author
#19 lore raw

PATCH v2 19/22 waf.bbclass: drop usage of distutils

In reply to Tim Orling

Refactor to use bb.utils.vercmp_string_op() instead of StrictVersion from distutils

[YOCTO #14610]

---
 meta/classes/waf.bbclass | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)
meta/classes/waf.bbclass +1 -2
index df0ab8d73ff..bc594d3c6b1 100644
@@ -43,14 +43,13 @@ BB_HASHBASE_WHITELIST += "WAFLOCK"
 
 python waf_preconfigure() {
     import subprocess
-    from distutils.version import StrictVersion
     subsrcdir = d.getVar('S')
     python = d.getVar('WAF_PYTHON')
     wafbin = os.path.join(subsrcdir, 'waf')
     try:
         result = subprocess.check_output([python, wafbin, '--version'], cwd=subsrcdir, stderr=subprocess.STDOUT)
         version = result.decode('utf-8').split()[1]
-        if StrictVersion(version) >= StrictVersion("1.8.7"):
+        if bb.utils.vercmp_string_op(version, "1.8.7", ">="):
             d.setVar("WAF_EXTRA_CONF", "--bindir=${bindir} --libdir=${libdir}")
     except subprocess.CalledProcessError as e:
         bb.warn("Unable to execute waf --version, exit code %d. Assuming waf version without bindir/libdir support." % e.returncode)
Signature · 1 line
2.30.2
author
#20 lore raw

PATCH v2 20/22 dnf: drop python3-distutils dependency

In reply to Tim Orling

Upstream dropped the now deprecated distutils:
18995d65 Remove distutils which are deprecated in python3.10 (RhBug:1950229)

Usage was replaced with sysconfig, which is in python3-core

[YOCTO #14610]

---
 meta/recipes-devtools/dnf/dnf_4.10.0.bb | 1 -
 1 file changed, 1 deletion(-)
meta/recipes-devtools/dnf/dnf_4.10.0.bb +0 -1
index 6ff2a6b1e6f..3fc24b132df 100644
@@ -38,7 +38,6 @@ RDEPENDS:${PN} += " \
   python3-netclient \
   python3-email \
   python3-threading \
-  python3-distutils \
   python3-logging \
   python3-fcntl \
   librepo \
Signature · 1 line
2.30.2
author
#21 lore raw

PATCH v2 21/22 python3native.bbclass: distutils -> sysconfig

In reply to Tim Orling

In the comment it says we patch distutils modules, but we now are patching sysconfig and not using distutils.

[YOCTO #14610]

---
 meta/classes/python3native.bbclass | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
meta/classes/python3native.bbclass +1 -1
index 13fbaa5f9c7..3783c0c47eb 100644
@@ -4,7 +4,7 @@ PYTHON="${STAGING_BINDIR_NATIVE}/python3-native/python3"
 EXTRANATIVEPATH += "python3-native"
 DEPENDS:append = " python3-native "
 
-# python-config and other scripts are using distutils modules
+# python-config and other scripts are using sysconfig modules
 # which we patch to access these variables
 export STAGING_INCDIR
 export STAGING_LIBDIR
Signature · 1 line
2.30.2
author
#22 lore raw

PATCH v2 22/22 rootfs_rpm.bbclass: distutils -> sysconfig module

In reply to Tim Orling

Change a comment to reference our custom changes to the sysconfig module, as dnf no longer uses distutils.

[YOCTO #14610]

---
 meta/classes/rootfs_rpm.bbclass | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
meta/classes/rootfs_rpm.bbclass +1 -1
index 3d8d784f792..bec4d63ed6c 100644
@@ -4,7 +4,7 @@
 
 ROOTFS_PKGMANAGE = "rpm dnf"
 
-# dnf is using our custom distutils, and so will fail without these
+# dnf is using our custom sysconfig module, and so will fail without these
 export STAGING_INCDIR
 export STAGING_LIBDIR
 
Signature · 1 line
2.30.2
#23 lore raw
In reply to Tim Orling

Thanks Tim, does this address the issue of building items that don't ship a setup.py anymore?

Alex

Quoted 113 lines · Tim Orling

distutils is deprecated in Python 3.10 and will be removed in Python 3.12 (~October 2023).

This series inserts a deprecation warning into the distutils*.bbclasses and drops or replaces distutils usage wherever possible.

Instead of distutils3-base, we introduce setuptools3-base, which also incorporates distutils-common-base.

Remaining usage is in:
* lib/recipetool/create_buildsys_python.py until we are ready to factor it out (and add pyproject.toml and setup.cfg handling)
* oeqa selftests which are testing distutils3 functionality
* glib-2.0: upstream is still using distutils.version.LooseVersion
* gpgme: Still checks for distutils in m4/ax_python_devel.m4
* python3-numpy: upstream still has a distutils/ module
* python3-setuptools has a vendored _distutils

I might have missed something else.

No attempt has been made to look at layers outside of oe-core.

The intent is to let this percolate for a few weeks and then drop distutils*.bbclasses (and move them to meta-python, but with the warnings).

[YOCTO #14610]

Changes in v2:
- dropped the import of distutils.version into bb. in favor of bb.utils.vercmp_string_op()
- dropped some patches that have been addressed already
- update email
- fix typos
- drop already merged scripts/buildhistory-diff patch
The following changes since commit 16110cdca7211b0efc59464afbb253527c033cad:

oeqa/sstate: Fix allarch samesigs test (2022-01-11 15:48:11 +0000)

are available in the Git repository at:

git://git.yoctoproject.org/git/poky-contrib
timo/deprecate_distutils_14610_v2

http://git.yoctoproject.org/cgit.cgi/git/poky-contrib/log/?h=timo/deprecate_distutils_14610_v2

Tim Orling (22):
classes/distutils-*: add warning of deprecation
setuptools3: refactor for no distutils bbclasses
python3-cython: DISTUTILS_* -> SETUPTOOLS_*
dnf: inherit setuptools3-base instead of distutils
libdnf: inherit setuptools3-base not distutils
python3-dbus: inherit setuptools3-base not distuils
gpgme: inherit setuptools-base not distuils
python3-pip: inherit setuptools3 not distuils3
systemtap: use setuptools-base not distutils
libcomps: inherit setuptools3-base not distutils
createrepo-c: inherit setuptools3-base not distutils
librepo: inherit setuptools3-base not distutils
btrfs-tools: use setuptools3-base not distutils
python3-pygobject: setuptools instead of distuils
bind: inherit setuptools3-base instead of distutils
python3-setuptools: do not depend on distutils gstreamer1.0-python: inherit setuptools3-base recipetool/create_buildsys_python.py: less distutils waf.bbclass: drop usage of distutils
dnf: drop python3-distutils dependency python3native.bbclass: distutils -> sysconfig rootfs_rpm.bbclass: distutils -> sysconfig module

 meta/classes/distutils-common-base.bbclass    |  3 +
 meta/classes/distutils3-base.bbclass          |  3 +
 meta/classes/distutils3.bbclass               |  4 ++
 meta/classes/python3native.bbclass            |  2 +-
 meta/classes/rootfs_rpm.bbclass               |  2 +-
 meta/classes/setuptools3-base.bbclass         | 31 +++++++++
 meta/classes/setuptools3.bbclass              | 66 ++++++++++++++++++-
 meta/classes/waf.bbclass                      |  3 +-
 .../recipes-connectivity/bind/bind_9.16.24.bb |  2 +-
 .../btrfs-tools/btrfs-tools_5.15.1.bb         |  2 +-
 .../createrepo-c/createrepo-c_0.18.0.bb       |  2 +-
 meta/recipes-devtools/dnf/dnf_4.10.0.bb       |  3 +-
 .../libcomps/libcomps_0.1.18.bb               |  2 +-
 meta/recipes-devtools/libdnf/libdnf_0.65.0.bb |  2 +-
 .../librepo/librepo_1.14.2.bb                 |  2 +-
 .../python/python3-cython_0.29.26.bb          |  2 +-
 .../python/python3-dbus_1.2.18.bb             |  2 +-
 .../python/python3-pip_21.3.1.bb              |  2 +-
 .../python/python3-pygobject_3.42.0.bb        |  2 +-
 .../python/python3-setuptools_59.5.0.bb       |  1 -
 .../recipes-kernel/systemtap/systemtap_git.bb |  2 +-
 .../gstreamer/gstreamer1.0-python_1.18.5.bb   |  2 +-
 meta/recipes-support/gpgme/gpgme_1.16.0.bb    |  2 +-
 .../lib/recipetool/create_buildsys_python.py  |  8 ++-
 24 files changed, 129 insertions(+), 23 deletions(-)
 create mode 100644 meta/classes/setuptools3-base.bbclass

--
2.30.2

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group. View/Reply Online (#160436):
https://lists.openembedded.org/g/openembedded-core/message/160436
Mute This Topic: https://lists.openembedded.org/mt/88356412/1686489
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [ [email protected]]
-=-=-=-=-=-=-=-=-=-=-=-

author
#24 lore raw
Quoted 4 lines · Alexander Kanavin

Thanks Tim, does this address the issue of building items that don't ship a setup.py anymore?

No. That falls under the PEP 517 work which is in a much rougher raw state.

The approach for that is expected to be supporting setup.cfg and pyproject.toml as first class citizens.

Quoted 117 lines

Alex

Quoted 113 lines · Tim Orling

distutils is deprecated in Python 3.10 and will be removed in Python 3.12 (~October 2023).

This series inserts a deprecation warning into the distutils*.bbclasses and drops or replaces distutils usage wherever possible.

Instead of distutils3-base, we introduce setuptools3-base, which also incorporates distutils-common-base.

Remaining usage is in:
* lib/recipetool/create_buildsys_python.py until we are ready to factor it out (and add pyproject.toml and setup.cfg handling)
* oeqa selftests which are testing distutils3 functionality
* glib-2.0: upstream is still using distutils.version.LooseVersion
* gpgme: Still checks for distutils in m4/ax_python_devel.m4
* python3-numpy: upstream still has a distutils/ module
* python3-setuptools has a vendored _distutils

I might have missed something else.

No attempt has been made to look at layers outside of oe-core.

The intent is to let this percolate for a few weeks and then drop distutils*.bbclasses (and move them to meta-python, but with the warnings).

[YOCTO #14610]

Changes in v2:
- dropped the import of distutils.version into bb. in favor of bb.utils.vercmp_string_op()
- dropped some patches that have been addressed already
- update email
- fix typos
- drop already merged scripts/buildhistory-diff patch
The following changes since commit 16110cdca7211b0efc59464afbb253527c033cad:

oeqa/sstate: Fix allarch samesigs test (2022-01-11 15:48:11 +0000)

are available in the Git repository at:

git://git.yoctoproject.org/git/poky-contrib
timo/deprecate_distutils_14610_v2

http://git.yoctoproject.org/cgit.cgi/git/poky-contrib/log/?h=timo/deprecate_distutils_14610_v2

Tim Orling (22):
classes/distutils-*: add warning of deprecation
setuptools3: refactor for no distutils bbclasses
python3-cython: DISTUTILS_* -> SETUPTOOLS_*
dnf: inherit setuptools3-base instead of distutils
libdnf: inherit setuptools3-base not distutils
python3-dbus: inherit setuptools3-base not distuils
gpgme: inherit setuptools-base not distuils
python3-pip: inherit setuptools3 not distuils3
systemtap: use setuptools-base not distutils
libcomps: inherit setuptools3-base not distutils
createrepo-c: inherit setuptools3-base not distutils
librepo: inherit setuptools3-base not distutils
btrfs-tools: use setuptools3-base not distutils
python3-pygobject: setuptools instead of distuils
bind: inherit setuptools3-base instead of distutils
python3-setuptools: do not depend on distutils gstreamer1.0-python: inherit setuptools3-base recipetool/create_buildsys_python.py: less distutils waf.bbclass: drop usage of distutils
dnf: drop python3-distutils dependency python3native.bbclass: distutils -> sysconfig rootfs_rpm.bbclass: distutils -> sysconfig module

 meta/classes/distutils-common-base.bbclass    |  3 +
 meta/classes/distutils3-base.bbclass          |  3 +
 meta/classes/distutils3.bbclass               |  4 ++
 meta/classes/python3native.bbclass            |  2 +-
 meta/classes/rootfs_rpm.bbclass               |  2 +-
 meta/classes/setuptools3-base.bbclass         | 31 +++++++++
 meta/classes/setuptools3.bbclass              | 66 ++++++++++++++++++-
 meta/classes/waf.bbclass                      |  3 +-
 .../recipes-connectivity/bind/bind_9.16.24.bb |  2 +-
 .../btrfs-tools/btrfs-tools_5.15.1.bb         |  2 +-
 .../createrepo-c/createrepo-c_0.18.0.bb       |  2 +-
 meta/recipes-devtools/dnf/dnf_4.10.0.bb       |  3 +-
 .../libcomps/libcomps_0.1.18.bb               |  2 +-
 meta/recipes-devtools/libdnf/libdnf_0.65.0.bb |  2 +-
 .../librepo/librepo_1.14.2.bb                 |  2 +-
 .../python/python3-cython_0.29.26.bb          |  2 +-
 .../python/python3-dbus_1.2.18.bb             |  2 +-
 .../python/python3-pip_21.3.1.bb              |  2 +-
 .../python/python3-pygobject_3.42.0.bb        |  2 +-
 .../python/python3-setuptools_59.5.0.bb       |  1 -
 .../recipes-kernel/systemtap/systemtap_git.bb |  2 +-
 .../gstreamer/gstreamer1.0-python_1.18.5.bb   |  2 +-
 meta/recipes-support/gpgme/gpgme_1.16.0.bb    |  2 +-
 .../lib/recipetool/create_buildsys_python.py  |  8 ++-
 24 files changed, 129 insertions(+), 23 deletions(-)
 create mode 100644 meta/classes/setuptools3-base.bbclass

--
2.30.2

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group. View/Reply Online (#160436):
https://lists.openembedded.org/g/openembedded-core/message/160436
Mute This Topic: https://lists.openembedded.org/mt/88356412/1686489
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [ [email protected]]
-=-=-=-=-=-=-=-=-=-=-=-