package/pkg-python: clean conflicting pep517 packages before install
PATCH v2 1/1 package/pkg-python: clean conflicting pep517 packages before install
The python installer package isn't able to overwrite packages that already exist, this causes problems when doing a rebuild or update without a full clean.
To fix this we can use functionality from importlib to identify and remove any conflicting python package files before installation.
Since we are adding this functionality to our pyinstaller.py script we must also use pyinstaller.py for host python packages to avoid hitting this issue there as well.
Fixes:
Traceback (most recent call last):
File "/home/buildroot/buildroot/support/scripts/pyinstaller.py", line 69, in <module>
main()
File "/home/buildroot/buildroot/support/scripts/pyinstaller.py", line 61, in main
install(
File "/home/buildroot/buildroot/output/host/lib/python3.10/site-packages/installer/_core.py", line 109, in install
record = destination.write_file( File "/home/buildroot/buildroot/output/host/lib/python3.10/site-packages/installer/destinations.py", line 207, in write_file
return self.write_to_fs(scheme, path_, stream, is_executable)
File "/home/buildroot/buildroot/output/host/lib/python3.10/site-packages/installer/destinations.py", line 167, in write_to_fs
raise FileExistsError(message)
FileExistsError: File already exists: /home/buildroot/buildroot/output/target/usr/lib/python3.10/site-packages/tinycss2/__init__.py
---
Changes v1 -> v2:
- remove unused os import
--- package/pkg-python.mk | 12 ++++++++++-- support/scripts/pyinstaller.py | 22 ++++++++++++++++++++++ 2 files changed, 32 insertions(+), 2 deletions(-)
package/pkg-python.mk +10 -2
@@ -139,6 +139,14 @@ PKG_PYTHON_PEP517_INSTALL_STAGING_OPTS = \ HOST_PKG_PYTHON_PEP517_ENV = \ $(HOST_PKG_PYTHON_ENV) +HOST_PKG_PYTHON_PEP517_INSTALL_OPTS = \ + --interpreter=/bin/python \ + --script-kind=posix \ + --purelib=$(HOST_DIR)/lib/python$(PYTHON3_VERSION_MAJOR)/site-packages \ + --headers=$(HOST_DIR)/include/python$(PYTHON3_VERSION_MAJOR) \ + --scripts=$(HOST_DIR)/bin \ + --data=$(HOST_DIR) + HOST_PKG_PYTHON_PEP517_BOOTSTRAP_INSTALL_OPTS = \ --installdir=$(HOST_DIR)/lib/python$(PYTHON3_VERSION_MAJOR)/site-packages
@@ -200,7 +208,7 @@ $(2)_BASE_INSTALL_STAGING_CMD = $(TOPDIR)/support/scripts/pyinstaller.py dist/* else $(2)_BASE_ENV = $$(HOST_PKG_PYTHON_PEP517_ENV) $(2)_BASE_BUILD_CMD = -m build -n -w -$(2)_BASE_INSTALL_CMD = -m installer dist/* +$(2)_BASE_INSTALL_CMD = $(TOPDIR)/support/scripts/pyinstaller.py dist/* $$(HOST_PKG_PYTHON_PEP517_INSTALL_OPTS) endif else ifeq ($$($(2)_SETUP_TYPE),flit-bootstrap) ifeq ($(4),target)
@@ -208,7 +216,7 @@ $$(error flit-bootstrap setup type only supported for host packages) else $(2)_BASE_ENV = $$(HOST_PKG_PYTHON_PEP517_ENV) $(2)_BASE_BUILD_CMD = -m flit_core.wheel -$(2)_BASE_INSTALL_CMD ?= -m installer dist/* +$(2)_BASE_INSTALL_CMD ?= $(TOPDIR)/support/scripts/pyinstaller.py dist/* $$(HOST_PKG_PYTHON_PEP517_INSTALL_OPTS) endif else $$(error "Invalid $(2)_SETUP_TYPE. Valid options are 'distutils', 'setuptools', 'pep517' or 'flit'.")
support/scripts/pyinstaller.py +22 -0
@@ -2,12 +2,33 @@ import argparse import glob +import pathlib + +from importlib.machinery import PathFinder +from importlib.metadata import DistributionFinder from installer import install +from installer._core import _process_WHEEL_file from installer.destinations import SchemeDictionaryDestination from installer.sources import WheelFile +def clean(source, destination): + scheme = _process_WHEEL_file(source) + scheme_path = destination.scheme_dict[scheme] + context = DistributionFinder.Context( + name=source.distribution, + path=[scheme_path], + ) + for path in PathFinder.find_distributions(context=context): + if not path.files: + continue + for file in path.files: + file_path = pathlib.Path(file.locate()) + if file_path.exists(): + file_path.unlink() + + def main(): """Entry point for CLI.""" ap = argparse.ArgumentParser("python pyinstaller.py")
@@ -58,6 +79,7 @@ def main(): ) with WheelFile.open(glob.glob(args.wheel_file)[0]) as source: + clean(source, destination) install( source=source, destination=destination,
Signature · 6 lines
2.34.1 _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot
James, All,
On 2022-08-22 00:02 -0600, James Hilliard spake thusly:
Quoted 1 line
The python installer package isn't able to overwrite packages that
s/packages/files of packges/
Quoted 9 lines
already exist, this causes problems when doing a rebuild or update without a full clean.
To fix this we can use functionality from importlib to identify and remove any conflicting python package files before installation.
Since we are adding this functionality to our
pyinstaller.pyscript we must also usepyinstaller.pyfor host python packages to avoid hitting this issue there as well.
Please split in two:
- first pach to use ur pyinstaller for host packages
- second patch to cleanup
[--SNIP--]
package/pkg-python.mk PKG_PYTHON_PEP517_INSTALL_STAGING_OPTS = \ · line 143
Quoted 10 lines
package/pkg-python.mk +2
-0@@ -139,6 +139,14 @@ PKG_PYTHON_PEP517_INSTALL_STAGING_OPTS = \ HOST_PKG_PYTHON_PEP517_ENV = \ $(HOST_PKG_PYTHON_ENV) +HOST_PKG_PYTHON_PEP517_INSTALL_OPTS = \ + --interpreter=/bin/python \
Why is this not $(HOST_DIR)/bin/python ?
There is no guarantee that there is a python interpreter on the build machine, and we anyway want to use our python, not the system one.
Regards,
Yann E. MORIN.
Signature · 10 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------' _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot
Quoted 7 lines · Yann E. MORIN
James, All,
On 2022-08-22 00:02 -0600, James Hilliard spake thusly:
Quoted 1 line
The python installer package isn't able to overwrite packages that
s/packages/files of packges/
Changed in v3
Quoted 7 lines
Quoted 7 lines
already exist, this causes problems when doing a rebuild or update without a full clean.
To fix this we can use functionality from importlib to identify and remove any conflicting python package files before installation.
Quoted 7 lines
Quoted 3 lines
Since we are adding this functionality to our
pyinstaller.pyscript we must also usepyinstaller.pyfor host python packages to avoid hitting this issue there as well.Please split in two:
- first pach to use ur pyinstaller for host packages
- second patch to cleanup
Split in v3:
https://patchwork.ozlabs.org/project/buildroot/patch/[email protected]/
https://patchwork.ozlabs.org/project/buildroot/patch/[email protected]/
Quoted 10 lines
[--SNIP--]
Quoted 8 lines
package/pkg-python.mk +0
-0@@ -139,6 +139,14 @@ PKG_PYTHON_PEP517_INSTALL_STAGING_OPTS = \ HOST_PKG_PYTHON_PEP517_ENV = \ $(HOST_PKG_PYTHON_ENV)
Quoted 4 lines
package/pkg-python.mk PKG_PYTHON_PEP517_INSTALL_STAGING_OPTS = \ · line 143
Quoted 2 lines
package/pkg-python.mk +2
-0+HOST_PKG_PYTHON_PEP517_INSTALL_OPTS = \ + --interpreter=/bin/python \Why is this not $(HOST_DIR)/bin/python ?
Fixed in v3:
https://patchwork.ozlabs.org/project/buildroot/patch/[email protected]/
Quoted 7 lines
There is no guarantee that there is a python interpreter on the build machine, and we anyway want to use our python, not the system one.
Regards,
Yann E. MORIN.
Quoted 7 lines
--
.-----------------.--------------------.------------------.--------------------.| Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. |'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot