Python PEP-517: build wheels and installw with pip
cover Python PEP-517: build wheels and installw with pip
Upstream Python is moving to pyproject.toml rather than setup.py and wheels rather than eggs. The main place this is documented is in PEP-517[1]. Installing packages with setup.py install is deprecated.
We have a choice with the 'kirkstone' LTS release of introducing this series (and fixing any fall out during the stabilization period) or living with deprecated Python packaging for the duration of the LTS (at a minimum 2 years). This seems to be a compelling enough reason to bring in this series, even if it results in a slight slip of the 'feature freeze', which is officially tomorrow, Monday 21 Feb, 2022.
This series bootstraps python3-setuptools-native, python3-flit-core-native, python3-wheel-native and python3-pip-native, simply unzipping the wheel into PYTHON_SITEPACKAGES_DIR. This allows all other recipes to be installed with pip, as intended by upstream.
Three new classes are introduced:
(1) flit_core which builds PEP-517 compliant packages which use flit_core.buildapi as their build-backend.
(2) setuptools_build_meta which builds PEP-517 compliant packages which use setuptools_build_meta as their build-backend.
(3) pip_install_wheel, which simply installs wheels built by other classes.
The original setuptools3.bbclass do_compile is modified to 'setup.py bdist_wheel' (from python3-wheel-native) rather than the deprecated 'setup.py build' mechanism.
The original setuptools3.bbclass do_install is modified to use the pip_install_wheel.bbclass to install wheels with pip rather than the deprecated 'setup.py install' mechanism.
For pip install, we disable dependency checking (--no-deps) because it proves to be brittle in our environment without providing full wheel cache that seems like overhead that we do not need. We also disable fetching wheels from pypi (--no-index). This is no different than our normal workflow of requiring dependencies to be defined in a recipe.
Missing from this series is documentation of the new features, classes and variables (this will be done under separate cover).
Also desired and previously discussed in YP meetings is a helper function to warn that a recipe which currently inherits setuptools3 but has a pyproject.toml and defines a build-system.build-backend should instead use of the new PEP-517 classes. This function is still in progress.
During testing, recipes which use python3-setuptools-scm are failing to determine the version of the python package, For several recipes, the fix was to use pyproject.toml and PEP-517 build backends. To be sure, other recipes are still going to fail for the short term, until the root cause can be determined.
Another PEP-517 build-backend, poetry.core.masonry.api has been identified in recipes in meta-python, but not in oe-core. The python3-poetry-core recipe is WIP, but brings in a number of dependencies that are perhaps not worth it to bring into oe-core at this time. Rather we can support these recipes in meta-python?
[YOCTO #14638]
[1] https://www.python.org/dev/peps/pep-0517/
The following changes since commit 0539b563c6f41fef383429a13c28bf918231d404:
libhandy: Use upstream regex to check version of upgrade. (2022-02-20 16:45:25 +0000)
are available in the Git repository at:
git://git.yoctoproject.org/git/poky-contrib timo/bootstrap-wheels
http://git.yoctoproject.org/cgit.cgi/git/poky-contrib/log/?h=timo/bootstrap-wheels
Tim Orling (19):
python3-wheel: move 0.37.0 from meta-python
python3-flit-core: add recipe for 3.6.0
python3-flit-core: simplify
flit_core.bbclass: add helper for newer python packaging
python3-wheel: inherit flit_core pip_install_wheel.bbclass: add helper class setuptools_build_meta.bbclass: add helper class
python3-pip: inherit setuptools_build_meta
python3-attrs: inherit setuptools_build_meta
python3-git: inherit setuptools_build_meta
python3-pytest: inherit setuptools_build_meta
python3-setuptools-scm: inherit setuptools_build_meta
python3-zipp: inherit setuptools_build_meta
python3-iniconfig: inherit setuptools_build_meta
python3-py: inherit setuptools_build_meta
python3-pluggy: inherit setuptools_build_meta
python3-setuptools: inherit setuptools_base_meta setuptools3.bbclass: refactor for wheels
python3-more-itertools: set PIP_INSTALL_PACKAGE
meta/classes/flit_core.bbclass | 16 +++ meta/classes/pip_install_wheel.bbclass | 18 ++++ meta/classes/setuptools3.bbclass | 20 ++-- meta/classes/setuptools_build_meta.bbclass | 18 ++++ .../python/python3-attrs_21.4.0.bb | 2 +- .../python/python3-flit-core_3.6.0.bb | 41 +++++++ .../python/python3-git_3.1.26.bb | 2 +- .../python/python3-iniconfig_1.1.1.bb | 4 +- .../python/python3-more-itertools_8.12.0.bb | 2 + .../python/python3-pip_22.0.3.bb | 19 +++- .../python/python3-pluggy_1.0.0.bb | 2 +- .../python/python3-py_1.11.0.bb | 2 +- .../python/python3-pytest_7.0.1.bb | 4 +- .../python/python3-setuptools-scm_6.4.2.bb | 2 +- .../python/python3-setuptools_59.5.0.bb | 19 +++- ...roject.toml-from-flit-backend-branch.patch | 100 ++++++++++++++++++ .../python/python3-wheel_0.37.1.bb | 24 +++++ .../python/python3-zipp_3.7.0.bb | 2 +- 18 files changed, 269 insertions(+), 28 deletions(-) create mode 100644 meta/classes/flit_core.bbclass create mode 100644 meta/classes/pip_install_wheel.bbclass create mode 100644 meta/classes/setuptools_build_meta.bbclass create mode 100644 meta/recipes-devtools/python/python3-flit-core_3.6.0.bb create mode 100644 meta/recipes-devtools/python/python3-wheel/0001-Backport-pyproject.toml-from-flit-backend-branch.patch create mode 100644 meta/recipes-devtools/python/python3-wheel_0.37.1.bb
Signature · 1 line
2.30.2
PATCH 1/19 python3-wheel: move 0.37.0 from meta-python
This is one of the "new build tools" which are part of pypa (Python Packaging Authority) toolchain.
Wheels are the official delivery mechanism for Python packages, replacing the now deprecated Eggs (egg-info).
[YOCTO #14638]
--- meta/recipes-devtools/python/python3-wheel_0.37.1.bb | 12 ++++++++++++ 1 file changed, 12 insertions(+) create mode 100644 meta/recipes-devtools/python/python3-wheel_0.37.1.bb
meta/recipes-devtools/python/python3-wheel_0.37.1.bb new file +12 -0
@@ -0,0 +1,12 @@ +SUMMARY = "The official binary distribution format for Python " +HOMEPAGE = "https://github.com/pypa/wheel" +SECTION = "devel/python" +LICENSE = "MIT" +LIC_FILES_CHKSUM = "file://PKG-INFO;beginline=10;endline=10;md5=8227180126797a0148f94f483f3e1489" + +SRC_URI[sha256sum] = "e9a504e793efbca1b8e0e9cb979a249cf4a0a7b5b8c9e8b65a5e39d49529c1c4" + +inherit pypi setuptools3 + +BBCLASSEXTEND = "native" +
Signature · 1 line
2.30.2
PATCH 2/19 python3-flit-core: add recipe for 3.6.0
In reply to Tim OrlingThis is the core of one of the "new build system" tools, although not officially part of the pypa (Python Packaging Authority) repositories, it is an increasingly common build tool (e.g. typing_extensions and tomli) as declared in pyproject.toml for said packages.
This package provides a very simple bootstrapping method that builds the source tarball (build_sdist) and a wheel (build_wheel).
Wheels are the official vehicle for delivering Python packages now. Eggs (egg-info) are deprecated and will increasingly go away.
[YOCTO #14638]
--- .../python/python3-flit-core_3.6.0.bb | 35 +++++++++++++++++++ 1 file changed, 35 insertions(+) create mode 100644 meta/recipes-devtools/python/python3-flit-core_3.6.0.bb
meta/recipes-devtools/python/python3-flit-core_3.6.0.bb new file +35 -0
@@ -0,0 +1,35 @@ +LICENSE = "BSD-3-Clause" +LIC_FILES_CHKSUM = "file://LICENSE;md5=41eb78fa8a872983a882c694a8305f08" + +SRC_URI[sha256sum] = "b1464e006df4df4c8eeb37671c0e0ce66e1d04e4a36d91b702f180a25fde3c11" + +inherit python3native python3-dir pypi + +DEPENDS += "python3 python3-pip-native" + +# We need the full flit tarball +PYPI_PACKAGE = "flit" + +do_compile () { + nativepython3 flit_core/build_dists.py +} + +do_install () { + install -d ${D}${PYTHON_SITEPACKAGES_DIR} + PYTHONPATH=${D}${PYTHON_SITEPACKAGES_DIR} \ + nativepython3 -m pip install -vvvv --no-index --target ${D}${PYTHON_SITEPACKAGES_DIR} ./flit_core/dist/flit_core-${PV}-py3-none-any.whl +} + +FILES:${PN} += "\ + ${PYTHON_SITEPACKAGES_DIR}/flit_core/* \ + ${PYTHON_SITEPACKAGES_DIR}/flit_core-${PV}.dist-info/* \ +" + +PACKAGES =+ "${PN}-tests" + +FILES:${PN}-tests += "\ + ${PYTHON_SITEPACKAGES_DIR}/flit_core/tests/* \ +" + +BBCLASSEXTEND = "native nativesdk" +
Signature · 1 line
2.30.2
PATCH 3/19 python3-flit-core: simplify
In reply to Tim OrlingUpstream provides a build_dists.py which can be used to bootstrap building of the sdist and wheel.
Bootstrap -native by simply unzipping the wheel to PYTHON_SITEPACKAGES_DIR.
Use pip to install the wheel for target.
[YOCTO #14638]
--- .../python/python3-flit-core_3.6.0.bb | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-)
meta/recipes-devtools/python/python3-flit-core_3.6.0.bb +9 -3
@@ -3,9 +3,10 @@ LIC_FILES_CHKSUM = "file://LICENSE;md5=41eb78fa8a872983a882c694a8305f08" SRC_URI[sha256sum] = "b1464e006df4df4c8eeb37671c0e0ce66e1d04e4a36d91b702f180a25fde3c11" -inherit python3native python3-dir pypi +inherit python3native python3-dir pypi setuptools3-base -DEPENDS += "python3 python3-pip-native" +DEPENDS:append:class-target = " python3-pip-native" +DEPENDS:append:class-native = " unzip-native" # We need the full flit tarball PYPI_PACKAGE = "flit"
@@ -17,7 +18,12 @@ do_compile () { do_install () { install -d ${D}${PYTHON_SITEPACKAGES_DIR} PYTHONPATH=${D}${PYTHON_SITEPACKAGES_DIR} \ - nativepython3 -m pip install -vvvv --no-index --target ${D}${PYTHON_SITEPACKAGES_DIR} ./flit_core/dist/flit_core-${PV}-py3-none-any.whl + nativepython3 -m pip install -vvvv --no-deps --no-index --target ${D}${PYTHON_SITEPACKAGES_DIR} ./flit_core/dist/flit_core-${PV}-py3-none-any.whl +} + +do_install:class-native () { + install -d ${D}${PYTHON_SITEPACKAGES_DIR} + unzip -d ${D}${PYTHON_SITEPACKAGES_DIR} ./flit_core/dist/flit_core-${PV}-py3-none-any.whl } FILES:${PN} += "\
Signature · 1 line
2.30.2
PATCH 4/19 flit_core.bbclass: add helper for newer python packaging
In reply to Tim OrlingSome python packages now use pyproject.toml and declare flit_core.buildapi as the build engine
Use pip_install_wheel class to install.
[YOCTO #14638]
--- meta/classes/flit_core.bbclass | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) create mode 100644 meta/classes/flit_core.bbclass
meta/classes/flit_core.bbclass new file +16 -0
@@ -0,0 +1,16 @@ +inherit pip_install_wheel python3native python3-dir + +DEPENDS += "python3 python3-flit-core-native python3-pip-native" + +do_configure () { + mkdir -p ${S}/dist + cat > ${S}/build-it.py << EOF +from flit_core import buildapi +buildapi.build_wheel('./dist') +EOF +} + +do_compile () { + ${STAGING_BINDIR_NATIVE}/${PYTHON_PN}-native/${PYTHON_PN} ${S}/build-it.py +} +
Signature · 1 line
2.30.2
PATCH 5/19 python3-wheel: inherit flit_core
In reply to Tim OrlingBackport pyproject.toml from flit-backend branch. Inherit flit_core class to build
Inherit pip_install_wheel to install wheels for target
We need to bootstrap python3-wheel-native in order to have bdist_wheel available to python3-setuptools-native and the refactored setuptools3.bbclass. Simply unzip the wheel into PYTHON_SITEPACKAGES_DIR for class-native.
[YOCTO #14638]
--- ...roject.toml-from-flit-backend-branch.patch | 100 ++++++++++++++++++ .../python/python3-wheel_0.37.1.bb | 16 ++- 2 files changed, 114 insertions(+), 2 deletions(-) create mode 100644 meta/recipes-devtools/python/python3-wheel/0001-Backport-pyproject.toml-from-flit-backend-branch.patch
meta/recipes-devtools/python/python3-wheel/0001-Backport-pyproject.toml-from-flit-backend-branch.patch new file +100 -0
@@ -0,0 +1,100 @@ +From f00dd220346773bc088d403847ee7f06f2b4c30a Mon Sep 17 00:00:00 2001 +From: Tim Orling <[email protected]> +Date: Fri, 18 Feb 2022 11:09:16 -0800 +Subject: [PATCH] Backport pyproject.toml from flit-backend branch + +This allows us to bootstrap wheels and PEP-517 packaging. + +Upstream-Status: Backport from flit-backend branch +https://raw.githubusercontent.com/pypa/wheel/4f6ba78fede38a8d9e35a14e38377a121033afb3/pyproject.toml + +Signed-off-by: Tim Orling <[email protected]> +--- + pyproject.toml | 78 ++++++++++++++++++++++++++++++++++++++++++++++++++ + 1 file changed, 78 insertions(+) + create mode 100644 pyproject.toml + +diff --git a/pyproject.toml b/pyproject.toml +new file mode 100644 +index 0000000..749b8de +--- /dev/null ++++ b/pyproject.toml +@@ -0,0 +1,78 @@ ++[build-system] ++requires = ["flit_core >=3.2,<4"] ++build-backend = "flit_core.buildapi" ++ ++[project] ++name = "wheel" ++description = "A built-package format for Python" ++readme = "README.rst" ++classifiers = [ ++ "Development Status :: 5 - Production/Stable", ++ "Intended Audience :: Developers", ++ "Topic :: System :: Archiving :: Packaging", ++ "License :: OSI Approved :: MIT License", ++ "Programming Language :: Python", ++ "Programming Language :: Python :: 3 :: Only", ++ "Programming Language :: Python :: 3.7", ++ "Programming Language :: Python :: 3.8", ++ "Programming Language :: Python :: 3.9", ++ "Programming Language :: Python :: 3.10" ++] ++authors = [{name = "Daniel Holth", email = "[email protected]"}] ++maintainers = [{name = "Alex Grönholm", email = "[email protected]"}] ++keywords = ["wheel", "packaging"] ++license = {file = "LICENSE.txt"} ++requires-python = ">=3.7" ++dependencies = [ ++ "setuptools >= 45.2.0" ++] ++dynamic = ["version"] ++ ++[project.urls] ++Documentation = "https://wheel.readthedocs.io/" ++Changelog = "https://wheel.readthedocs.io/en/stable/news.html" ++"Issue Tracker" = "https://github.com/pypa/wheel/issues" ++ ++[project.scripts] ++wheel = "wheel.cli:main" ++ ++[project.entry-points."distutils.commands"] ++bdist_wheel = "wheel.bdist_wheel:bdist_wheel" ++ ++[project.optional-dependencies] ++test = [ ++ "pytest >= 3.0.0" ++] ++ ++[tool.flit.sdist] ++exclude = [ ++ ".cirrus.yml", ++ ".github/*", ++ ".gitignore", ++ ".pre-commit-config.yaml", ++ ".readthedocs.yml" ++] ++ ++[tool.black] ++target-version = ['py37'] ++extend-exclude = ''' ++^/src/wheel/vendored/ ++''' ++ ++[tool.isort] ++src_paths = ["src"] ++profile = "black" ++skip_gitignore = true ++ ++[tool.flake8] ++max-line-length = 88 ++ ++[tool.pytest.ini_options] ++testpaths = "tests" ++ ++[tool.coverage.run] ++source = ["wheel"] ++omit = ["*/vendored/*"] ++ ++[tool.coverage.report] ++show_missing = true
meta/recipes-devtools/python/python3-wheel_0.37.1.bb +14 -2
@@ -6,7 +6,19 @@ LIC_FILES_CHKSUM = "file://PKG-INFO;beginline=10;endline=10;md5=8227180126797a01 SRC_URI[sha256sum] = "e9a504e793efbca1b8e0e9cb979a249cf4a0a7b5b8c9e8b65a5e39d49529c1c4" -inherit pypi setuptools3 +inherit flit_core pypi setuptools3-base -BBCLASSEXTEND = "native" +SRC_URI += " file://0001-Backport-pyproject.toml-from-flit-backend-branch.patch" + +DEPENDS:remove:class-native = "python3-pip-native" + +do_install:class-native () { + # We need to bootstrap python3-wheel-native + install -d ${D}${PYTHON_SITEPACKAGES_DIR} + PYPA_WHEEL="${B}/dist/${PYPI_PACKAGE}-${PV}-*.whl" + unzip -d ${D}${PYTHON_SITEPACKAGES_DIR} ${PYPA_WHEEL} || \ + bbfatal_log "Failed to install" +} + +BBCLASSEXTEND = "native nativesdk"
Signature · 1 line
2.30.2
PATCH 6/19 pip_install_wheel.bbclass: add helper class
In reply to Tim OrlingProvide a helper class to use pip to install wheels built by either bdist_wheel or a PEP-517 backend.
By passings --no-deps and --no-index we avoid finicky dependency checking (pip expects wheels in its cache) and avoid trying to fetch wheels from pypi.org. This is basically the same behavior we have now, the dependencies should be declared in the recipe.
[YOCTO #14638]
--- meta/classes/pip_install_wheel.bbclass | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) create mode 100644 meta/classes/pip_install_wheel.bbclass
meta/classes/pip_install_wheel.bbclass new file +18 -0
@@ -0,0 +1,18 @@ +DEPENDS:append = " python3-pip-native" + +PIP_INSTALL_PACKAGE ?= "${PYPI_PACKAGE}" +PIP_INSTALL_DIST_PATH ?= "${B}/dist" +PYPA_WHEEL ??= "${PIP_INSTALL_DIST_PATH}/${PIP_INSTALL_PACKAGE}-${PV}-*.whl" + +pip_install_wheel_do_install:prepend () { + install -d ${D}${PYTHON_SITEPACKAGES_DIR} +} + +export PYPA_WHEEL + +pip_install_wheel_do_install () { + nativepython3 -m pip install -vvvv --no-deps --no-index --target ${D}${PYTHON_SITEPACKAGES_DIR} ${PYPA_WHEEL} || + bbfatal_log "Failed to pip install wheel. Check the logs." +} + +EXPORT_FUNCTIONS do_install
Signature · 1 line
2.30.2
PATCH 7/19 setuptools_build_meta.bbclass: add helper class
In reply to Tim OrlingThis class uses the PEP 517 compliant setuptools.buil_meta to build wheels.
[YOCTO #14638]
--- meta/classes/setuptools_build_meta.bbclass | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) create mode 100644 meta/classes/setuptools_build_meta.bbclass
meta/classes/setuptools_build_meta.bbclass new file +18 -0
@@ -0,0 +1,18 @@ +inherit pip_install_wheel setuptools3-base + +DEPENDS += "python3 python3-setuptools-native python3-wheel-native" + +setuptools_build_meta_do_configure () { + mkdir -p ${S}/dist + cat > ${S}/build-it.py << EOF +from setuptools import build_meta +wheel = build_meta.build_wheel('./dist') +print(wheel) +EOF +} + +setuptools_build_meta_do_compile () { + nativepython3 ${S}/build-it.py +} + +EXPORT_FUNCTIONS do_configure do_compile
Signature · 1 line
2.30.2
PATCH 8/19 python3-pip: inherit setuptools_build_meta
In reply to Tim OrlingUpstream provides a pyproject.toml which declares setuptools.build_meta as the backend for PEP-517 packaging.
We bootstrap the -native installation by simply unzipping the wheel to PYTHON_SITEPACKAGES_DIR, so that all other recipes can use pip to install wheels (as intended by upstream).
[YOCTO #14638]
--- .../python/python3-pip_22.0.3.bb | 19 +++++++++++++++++-- 1 file changed, 17 insertions(+), 2 deletions(-)
meta/recipes-devtools/python/python3-pip_22.0.3.bb +17 -2
@@ -4,17 +4,32 @@ SECTION = "devel/python" LICENSE = "MIT" LIC_FILES_CHKSUM = "file://LICENSE.txt;md5=63ec52baf95163b597008bb46db68030" +inherit pypi setuptools_build_meta + DEPENDS += "python3 python3-setuptools-native" -inherit pypi setuptools3 +# To avoid a dependency loop; we bootstrap -native +DEPENDS:remove:class-native = "python3-pip-native" +DEPENDS:append:class-native = " unzip-native" SRC_URI += "file://0001-change-shebang-to-python3.patch" SRC_URI[sha256sum] = "f29d589df8c8ab99c060e68ad294c4a9ed896624f6368c5349d70aa581b333d0" +PYPA_WHEEL ?= "${B}/dist/${PYPI_PACKAGE}-${PV}-*.whl" + +do_install:class-native() { + # Bootstrap to prevent dependency loop in python3-pip-native + install -d ${D}${PYTHON_SITEPACKAGES_DIR} + unzip -d ${D}${PYTHON_SITEPACKAGES_DIR} ${PYPA_WHEEL} || \ + bbfatal_log "Failed to unzip wheel: ${PYPA_WHEEL}. Check the logs." +} + do_install:append() { # Install as pip3 and leave pip2 as default - rm ${D}/${bindir}/pip + if [ -f ${D}/${bindir}/pip ]; then + rm ${D}/${bindir}/pip + fi } RDEPENDS:${PN} = "\
Signature · 1 line
2.30.2
PATCH 9/19 python3-attrs: inherit setuptools_build_meta
In reply to Tim OrlingUpstream provides a pyproject.toml which declares the setuptools.build_meta backend for PEP-517 packaging.
[YOCTO #14638]
--- meta/recipes-devtools/python/python3-attrs_21.4.0.bb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
meta/recipes-devtools/python/python3-attrs_21.4.0.bb +1 -1
@@ -5,7 +5,7 @@ LIC_FILES_CHKSUM = "file://LICENSE;md5=d4ab25949a73fe7d4fdee93bcbdbf8ff" SRC_URI[sha256sum] = "626ba8234211db98e869df76230a137c4c40a12d72445c45d5f5b716f076e2fd" -inherit pypi setuptools3 +inherit pypi setuptools_build_meta RDEPENDS:${PN}:class-target += " \ ${PYTHON_PN}-crypt \
Signature · 1 line
2.30.2
PATCH 10/19 python3-git: inherit setuptools_build_meta
In reply to Tim OrlingUpstream provides a pyproject.toml which declares the setuptools.build_meta backend for PEP-517 packaging.
[YOCTO #14638]
--- meta/recipes-devtools/python/python3-git_3.1.26.bb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
meta/recipes-devtools/python/python3-git_3.1.26.bb +1 -1
@@ -10,7 +10,7 @@ LIC_FILES_CHKSUM = "file://LICENSE;md5=8b8d26c37c1d5a04f9b0186edbebc183" PYPI_PACKAGE = "GitPython" -inherit pypi setuptools3 +inherit pypi setuptools_build_meta SRC_URI[sha256sum] = "fc8868f63a2e6d268fb25f481995ba185a85a66fcad126f039323ff6635669ee"
Signature · 1 line
2.30.2
PATCH 11/19 python3-pytest: inherit setuptools_build_meta
In reply to Tim OrlingUpstream provides a pyproject.toml which declares the setuptools.build_meta backend for PEP-517 packaging.
[YOCTO #14638]
--- meta/recipes-devtools/python/python3-pytest_7.0.1.bb | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
meta/recipes-devtools/python/python3-pytest_7.0.1.bb +2 -2
@@ -7,9 +7,9 @@ LIC_FILES_CHKSUM = "file://LICENSE;md5=bd27e41b6550fe0fc45356d1d81ee37c" SRC_URI[sha256sum] = "e30905a0c131d3d94b89624a1cc5afec3e0ba2fbdb151867d8e0ebd49850f171" -DEPENDS = "python3-setuptools-scm-native" +DEPENDS += "python3-setuptools-scm-native" -inherit update-alternatives pypi setuptools3 +inherit update-alternatives pypi setuptools_build_meta RDEPENDS:${PN}:class-target += " \ ${PYTHON_PN}-atomicwrites \
Signature · 1 line
2.30.2
PATCH 12/19 python3-setuptools-scm: inherit setuptools_build_meta
In reply to Tim OrlingUpstream provides a pyproject.toml which declares the setuptools.build_meta backend for PEP-517 packaging.
[YOCTO #14638]
--- meta/recipes-devtools/python/python3-setuptools-scm_6.4.2.bb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
meta/recipes-devtools/python/python3-setuptools-scm_6.4.2.bb +1 -1
@@ -7,7 +7,7 @@ LIC_FILES_CHKSUM = "file://PKG-INFO;beginline=8;endline=8;md5=8227180126797a0148 SRC_URI[sha256sum] = "6833ac65c6ed9711a4d5d2266f8024cfa07c533a0e55f4c12f6eff280a5a9e30" PYPI_PACKAGE = "setuptools_scm" -inherit pypi setuptools3 +inherit pypi setuptools_build_meta UPSTREAM_CHECK_REGEX = "setuptools_scm-(?P<pver>.*)\.tar"
Signature · 1 line
2.30.2
PATCH 13/19 python3-zipp: inherit setuptools_build_meta
In reply to Tim OrlingUpstream provides a pyproject.toml which declares the setuptools.build_meta backend for PEP-517 packaging.
[YOCTO #14638]
--- meta/recipes-devtools/python/python3-zipp_3.7.0.bb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
meta/recipes-devtools/python/python3-zipp_3.7.0.bb +1 -1
@@ -7,7 +7,7 @@ SRC_URI[sha256sum] = "9f50f446828eb9d45b267433fd3e9da8d801f614129124863f9c51ebce DEPENDS += "${PYTHON_PN}-setuptools-scm-native" -inherit pypi setuptools3 +inherit pypi setuptools_build_meta DEPENDS += "${PYTHON_PN}-toml-native"
Signature · 1 line
2.30.2
PATCH 14/19 python3-iniconfig: inherit setuptools_build_meta
In reply to Tim OrlingUpstream provides a pyproject.toml which decalres the setuptools.build_meta backend for PEP-517 packaging.
DEPENDS on python3-setuptools-scm-native (as declared in the pyproject.toml build-system.requires).
[YOCTO #14638]
--- meta/recipes-devtools/python/python3-iniconfig_1.1.1.bb | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
meta/recipes-devtools/python/python3-iniconfig_1.1.1.bb +3 -1
@@ -7,4 +7,6 @@ LIC_FILES_CHKSUM = "file://LICENSE;md5=a6bb0320b04a0a503f12f69fea479de9" SRC_URI[md5sum] = "0b7f3be87481211c183eae095bcea6f1" SRC_URI[sha256sum] = "bc3af051d7d14b2ee5ef9969666def0cd1a000e121eaea580d4a313df4b37f32" -inherit pypi setuptools3 +DEPENDS += "python3-setuptools-scm-native" + +inherit pypi setuptools_build_meta
Signature · 1 line
2.30.2
PATCH 15/19 python3-py: inherit setuptools_build_meta
In reply to Tim OrlingUpstreama provides a pyproject.toml which declares the setuptools.build_meta backend for PEP-517 packaging.
[YOCTO #14638]
--- meta/recipes-devtools/python/python3-py_1.11.0.bb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
meta/recipes-devtools/python/python3-py_1.11.0.bb +1 -1
@@ -7,7 +7,7 @@ SRC_URI[sha256sum] = "51c75c4126074b472f746a24399ad32f6053d1b34b68d2fa41e558e6f4 DEPENDS += "${PYTHON_PN}-setuptools-scm-native" -inherit pypi setuptools3 +inherit pypi setuptools_build_meta BBCLASSEXTEND = "native nativesdk"
Signature · 1 line
2.30.2
PATCH 16/19 python3-pluggy: inherit setuptools_build_meta
In reply to Tim OrlingUpstream provides a pyproject.toml which while it does not declare the setuptools.build_meta backend is compatible with it.
[YOCTO #14638]
--- meta/recipes-devtools/python/python3-pluggy_1.0.0.bb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
meta/recipes-devtools/python/python3-pluggy_1.0.0.bb +1 -1
@@ -10,7 +10,7 @@ RDEPENDS:${PN} += "${PYTHON_PN}-importlib-metadata \ ${PYTHON_PN}-more-itertools \ " -inherit pypi ptest setuptools3 +inherit pypi ptest setuptools_build_meta SRC_URI += " \ file://run-ptest \
Signature · 1 line
2.30.2
PATCH 17/19 python3-setuptools: inherit setuptools_base_meta
In reply to Tim OrlingUpstream provides a pyproject.toml which declares the setuptools.build_meta backend for PEP-517 packaging.
We need to bootstrap python3-setuptools-native, simply installing by unzipping the built wheel. This avoids a dependency loop.
[YOCTO #14638]
--- .../python/python3-setuptools_59.5.0.bb | 19 ++++++++++++++----- 1 file changed, 14 insertions(+), 5 deletions(-)
meta/recipes-devtools/python/python3-setuptools_59.5.0.bb +14 -5
@@ -4,7 +4,7 @@ SECTION = "devel/python" LICENSE = "MIT" LIC_FILES_CHKSUM = "file://LICENSE;beginline=1;endline=19;md5=7a7126e068206290f3fe9f8d6c713ea6" -inherit pypi setuptools3 +inherit pypi setuptools_build_meta SRC_URI:append:class-native = " file://0001-conditionally-do-not-fetch-code-by-easy_install.patch"
@@ -17,6 +17,19 @@ SRC_URI[sha256sum] = "d144f85102f999444d06f9c0e8c737fd0194f10f2f7e5fdb77573f6e2f DEPENDS += "${PYTHON_PN}" +# Avoid dependency loop; we bootstrap -native +DEPENDS:remove:class-native = "python3-pip-native python3-setuptools-native" +DEPENDS:append:class-native = " unzip-native" + +PYPA_WHEEL ?= "${B}/dist/${PYPI_PACKAGE}-${PV}-*.whl" + +do_install:class-native() { + # Bootstrap to prevent dependency loop in python3-pip-native + install -d ${D}${PYTHON_SITEPACKAGES_DIR} + unzip -d ${D}${PYTHON_SITEPACKAGES_DIR} ${PYPA_WHEEL} || \ + bbfatal_log "Failed to unzip wheel: ${PYPA_WHEEL}. Check the logs." +} + RDEPENDS:${PN} = "\ ${PYTHON_PN}-2to3 \ ${PYTHON_PN}-compile \
@@ -38,10 +51,6 @@ RDEPENDS:${PN} = "\ ${PYTHON_PN}-xml \ " -do_install:prepend() { - install -d ${D}${PYTHON_SITEPACKAGES_DIR} -} - BBCLASSEXTEND = "native nativesdk" # The pkg-resources module can be used by itself, without the package downloader
Signature · 1 line
2.30.2
PATCH 18/19 setuptools3.bbclass: refactor for wheels
In reply to Tim OrlingDepend on python3-wheel-native so that we can build with 'setup.py bdist_wheel'.
Use pip_install_wheel class to install the built wheels with pip, as intended by upstream Python.
[YOCTO #14638]
--- meta/classes/setuptools3.bbclass | 20 ++++++++------------ 1 file changed, 8 insertions(+), 12 deletions(-)
meta/classes/setuptools3.bbclass +8 -12
@@ -1,6 +1,7 @@ -inherit setuptools3-base +inherit setuptools3-base pip_install_wheel -B = "${WORKDIR}/build" +# bdist_wheel builds in ./dist +#B = "${WORKDIR}/build" SETUPTOOLS_BUILD_ARGS ?= "" SETUPTOOLS_INSTALL_ARGS ?= "--root=${D} \
@@ -23,20 +24,15 @@ setuptools3_do_compile() { STAGING_INCDIR=${STAGING_INCDIR} \ STAGING_LIBDIR=${STAGING_LIBDIR} \ ${STAGING_BINDIR_NATIVE}/${PYTHON_PN}-native/${PYTHON_PN} setup.py \ - build --build-base=${B} ${SETUPTOOLS_BUILD_ARGS} || \ - bbfatal_log "'${PYTHON_PN} setup.py build ${SETUPTOOLS_BUILD_ARGS}' execution failed." + bdist_wheel ${SETUPTOOLS_BUILD_ARGS} || \ + bbfatal_log "'${PYTHON_PN} setup.py bdist_wheel ${SETUPTOOLS_BUILD_ARGS}' execution failed." } setuptools3_do_compile[vardepsexclude] = "MACHINE" setuptools3_do_install() { cd ${SETUPTOOLS_SETUP_PATH} - install -d ${D}${PYTHON_SITEPACKAGES_DIR} - STAGING_INCDIR=${STAGING_INCDIR} \ - STAGING_LIBDIR=${STAGING_LIBDIR} \ - PYTHONPATH=${D}${PYTHON_SITEPACKAGES_DIR} \ - ${STAGING_BINDIR_NATIVE}/${PYTHON_PN}-native/${PYTHON_PN} setup.py \ - build --build-base=${B} install --skip-build ${SETUPTOOLS_INSTALL_ARGS} || \ - bbfatal_log "'${PYTHON_PN} setup.py install ${SETUPTOOLS_INSTALL_ARGS}' execution failed." + + pip_install_wheel_do_install # support filenames with *spaces* find ${D} -name "*.py" -exec grep -q ${D} {} \; \
@@ -64,5 +60,5 @@ setuptools3_do_install[vardepsexclude] = "MACHINE" EXPORT_FUNCTIONS do_configure do_compile do_install export LDSHARED="${CCLD} -shared" -DEPENDS += "python3-setuptools-native" +DEPENDS += "python3-setuptools-native python3-wheel-native"
Signature · 1 line
2.30.2
PATCH 19/19 python3-more-itertools: set PIP_INSTALL_PACKAGE
In reply to Tim OrlingDefine PIP_INSTALL_PACKAGE as more_itertools to match the name of the wheel.
[YOCTO #14638]
--- meta/recipes-devtools/python/python3-more-itertools_8.12.0.bb | 2 ++ 1 file changed, 2 insertions(+)
meta/recipes-devtools/python/python3-more-itertools_8.12.0.bb +2 -0
@@ -7,6 +7,8 @@ SRC_URI[sha256sum] = "7dc6ad46f05f545f900dd59e8dfb4e84a4827b97b3cfecb175ea0c7d24 inherit pypi setuptools3 ptest +PIP_INSTALL_PACKAGE = "more_itertools" + SRC_URI += " \ file://run-ptest \ "
Signature · 1 line
2.30.2
Hi,
Seems that python changes cause build fail https://swatbot.yoctoproject.org/collection/6991/
Regards,
Oleksiy
________________________________________
From: [email protected] <[email protected]> on behalf of Tim Orling <[email protected]>
Sent: Monday, February 21, 2022 03:27
To: [email protected]
Subject: [OE-core] [PATCH 00/19] Python PEP-517: build wheels and installw with pip
Upstream Python is moving to pyproject.toml rather than setup.py and wheels rather than eggs. The main place this is documented is in PEP-517[1]. Installing packages with setup.py install is deprecated.
We have a choice with the 'kirkstone' LTS release of introducing this series (and fixing any fall out during the stabilization period) or living with deprecated Python packaging for the duration of the LTS (at a minimum 2 years). This seems to be a compelling enough reason to bring in this series, even if it results in a slight slip of the 'feature freeze', which is officially tomorrow, Monday 21 Feb, 2022.
This series bootstraps python3-setuptools-native, python3-flit-core-native, python3-wheel-native and python3-pip-native, simply unzipping the wheel into PYTHON_SITEPACKAGES_DIR. This allows all other recipes to be installed with pip, as intended by upstream.
Three new classes are introduced:
(1) flit_core which builds PEP-517 compliant packages which use flit_core.buildapi as their build-backend.
(2) setuptools_build_meta which builds PEP-517 compliant packages which use setuptools_build_meta as their build-backend.
(3) pip_install_wheel, which simply installs wheels built by other classes.
The original setuptools3.bbclass do_compile is modified to 'setup.py bdist_wheel' (from python3-wheel-native) rather than the deprecated 'setup.py build' mechanism.
The original setuptools3.bbclass do_install is modified to use the pip_install_wheel.bbclass to install wheels with pip rather than the deprecated 'setup.py install' mechanism.
For pip install, we disable dependency checking (--no-deps) because it proves to be brittle in our environment without providing full wheel cache that seems like overhead that we do not need. We also disable fetching wheels from pypi (--no-index). This is no different than our normal workflow of requiring dependencies to be defined in a recipe.
Missing from this series is documentation of the new features, classes and variables (this will be done under separate cover).
Also desired and previously discussed in YP meetings is a helper function to warn that a recipe which currently inherits setuptools3 but has a pyproject.toml and defines a build-system.build-backend should instead use of the new PEP-517 classes. This function is still in progress.
During testing, recipes which use python3-setuptools-scm are failing to determine the version of the python package, For several recipes, the fix was to use pyproject.toml and PEP-517 build backends. To be sure, other recipes are still going to fail for the short term, until the root cause can be determined.
Another PEP-517 build-backend, poetry.core.masonry.api has been identified in recipes in meta-python, but not in oe-core. The python3-poetry-core recipe is WIP, but brings in a number of dependencies that are perhaps not worth it to bring into oe-core at this time. Rather we can support these recipes in meta-python?
[YOCTO #14638]
[1] https://www.python.org/dev/peps/pep-0517/
The following changes since commit 0539b563c6f41fef383429a13c28bf918231d404:
libhandy: Use upstream regex to check version of upgrade. (2022-02-20 16:45:25 +0000)
are available in the Git repository at:
git://git.yoctoproject.org/git/poky-contrib timo/bootstrap-wheels
http://git.yoctoproject.org/cgit.cgi/git/poky-contrib/log/?h=timo/bootstrap-wheels
Tim Orling (19):
python3-wheel: move 0.37.0 from meta-python
python3-flit-core: add recipe for 3.6.0
python3-flit-core: simplify
flit_core.bbclass: add helper for newer python packaging
python3-wheel: inherit flit_core pip_install_wheel.bbclass: add helper class setuptools_build_meta.bbclass: add helper class
python3-pip: inherit setuptools_build_meta
python3-attrs: inherit setuptools_build_meta
python3-git: inherit setuptools_build_meta
python3-pytest: inherit setuptools_build_meta
python3-setuptools-scm: inherit setuptools_build_meta
python3-zipp: inherit setuptools_build_meta
python3-iniconfig: inherit setuptools_build_meta
python3-py: inherit setuptools_build_meta
python3-pluggy: inherit setuptools_build_meta
python3-setuptools: inherit setuptools_base_meta setuptools3.bbclass: refactor for wheels
python3-more-itertools: set PIP_INSTALL_PACKAGE
meta/classes/flit_core.bbclass | 16 +++ meta/classes/pip_install_wheel.bbclass | 18 ++++ meta/classes/setuptools3.bbclass | 20 ++-- meta/classes/setuptools_build_meta.bbclass | 18 ++++ .../python/python3-attrs_21.4.0.bb | 2 +- .../python/python3-flit-core_3.6.0.bb | 41 +++++++ .../python/python3-git_3.1.26.bb | 2 +- .../python/python3-iniconfig_1.1.1.bb | 4 +- .../python/python3-more-itertools_8.12.0.bb | 2 + .../python/python3-pip_22.0.3.bb | 19 +++- .../python/python3-pluggy_1.0.0.bb | 2 +- .../python/python3-py_1.11.0.bb | 2 +- .../python/python3-pytest_7.0.1.bb | 4 +- .../python/python3-setuptools-scm_6.4.2.bb | 2 +- .../python/python3-setuptools_59.5.0.bb | 19 +++- ...roject.toml-from-flit-backend-branch.patch | 100 ++++++++++++++++++ .../python/python3-wheel_0.37.1.bb | 24 +++++ .../python/python3-zipp_3.7.0.bb | 2 +- 18 files changed, 269 insertions(+), 28 deletions(-) create mode 100644 meta/classes/flit_core.bbclass create mode 100644 meta/classes/pip_install_wheel.bbclass create mode 100644 meta/classes/setuptools_build_meta.bbclass create mode 100644 meta/recipes-devtools/python/python3-flit-core_3.6.0.bb create mode 100644 meta/recipes-devtools/python/python3-wheel/0001-Backport-pyproject.toml-from-flit-backend-branch.patch create mode 100644 meta/recipes-devtools/python/python3-wheel_0.37.1.bb
--
2.30.2
Quoted 3 lines · Oleksiy Obitotskyy via lists.openembedded.org
Hi,
Seems that python changes cause build fail https://swatbot.yoctoproject.org/collection/6991/
FWIW that was with the other patches in your branch too. I was curious to see how it looked but it isn't quite there yet :/
Cheers,
Richard