package/pkg-python: use pyinstaller.py for host python packages
PATCH v3 1/2 package/pkg-python: use pyinstaller.py for host python packages
The python installer cli isn't able to overwrite files of packages that already exist, this causes problems when doing a rebuild or update without a full clean.
Since we need to add functionality to our pyinstaller.py script to fix this issue we must also use pyinstaller.py for host python packages.
---
Changes v2 -> v3:
- set interpreter to $(HOST_DIR)/bin/python
--- package/pkg-python.mk | 12 ++++++++++-- support/scripts/pyinstaller.py | 0 2 files changed, 10 insertions(+), 2 deletions(-) mode change 100755 => 100644 support/scripts/pyinstaller.py
package/pkg-python.mk +10 -2
@@ -139,6 +139,14 @@ PKG_PYTHON_PEP517_INSTALL_STAGING_OPTS = \ HOST_PKG_PYTHON_PEP517_ENV = \ $(HOST_PKG_PYTHON_ENV) +HOST_PKG_PYTHON_PEP517_INSTALL_OPTS = \ + --interpreter=$(HOST_DIR)/bin/python \ + --script-kind=posix \ + --purelib=$(HOST_DIR)/lib/python$(PYTHON3_VERSION_MAJOR)/site-packages \ + --headers=$(HOST_DIR)/include/python$(PYTHON3_VERSION_MAJOR) \ + --scripts=$(HOST_DIR)/bin \ + --data=$(HOST_DIR) + HOST_PKG_PYTHON_PEP517_BOOTSTRAP_INSTALL_OPTS = \ --installdir=$(HOST_DIR)/lib/python$(PYTHON3_VERSION_MAJOR)/site-packages
@@ -200,7 +208,7 @@ $(2)_BASE_INSTALL_STAGING_CMD = $(TOPDIR)/support/scripts/pyinstaller.py dist/* else $(2)_BASE_ENV = $$(HOST_PKG_PYTHON_PEP517_ENV) $(2)_BASE_BUILD_CMD = -m build -n -w -$(2)_BASE_INSTALL_CMD = -m installer dist/* +$(2)_BASE_INSTALL_CMD = $(TOPDIR)/support/scripts/pyinstaller.py dist/* $$(HOST_PKG_PYTHON_PEP517_INSTALL_OPTS) endif else ifeq ($$($(2)_SETUP_TYPE),flit-bootstrap) ifeq ($(4),target)
@@ -208,7 +216,7 @@ $$(error flit-bootstrap setup type only supported for host packages) else $(2)_BASE_ENV = $$(HOST_PKG_PYTHON_PEP517_ENV) $(2)_BASE_BUILD_CMD = -m flit_core.wheel -$(2)_BASE_INSTALL_CMD ?= -m installer dist/* +$(2)_BASE_INSTALL_CMD ?= $(TOPDIR)/support/scripts/pyinstaller.py dist/* $$(HOST_PKG_PYTHON_PEP517_INSTALL_OPTS) endif else $$(error "Invalid $(2)_SETUP_TYPE. Valid options are 'distutils', 'setuptools', 'pep517' or 'flit'.")
support/scripts/pyinstaller.py +0 -0
Signature · 6 lines
2.34.1 _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot
PATCH v3 2/2 package/pkg-python: clean conflicting pep517 packages before install
The python installer package isn't able to overwrite files of packges that already exist, this causes problems when doing a rebuild or update without a full clean.
To fix this we can use functionality from importlib to identify and remove any conflicting python package files before installation.
Fixes:
Traceback (most recent call last):
File "/home/buildroot/buildroot/support/scripts/pyinstaller.py", line 69, in <module>
main()
File "/home/buildroot/buildroot/support/scripts/pyinstaller.py", line 61, in main
install(
File "/home/buildroot/buildroot/output/host/lib/python3.10/site-packages/installer/_core.py", line 109, in install
record = destination.write_file( File "/home/buildroot/buildroot/output/host/lib/python3.10/site-packages/installer/destinations.py", line 207, in write_file
return self.write_to_fs(scheme, path_, stream, is_executable)
File "/home/buildroot/buildroot/output/host/lib/python3.10/site-packages/installer/destinations.py", line 167, in write_to_fs
raise FileExistsError(message)
FileExistsError: File already exists: /home/buildroot/buildroot/output/target/usr/lib/python3.10/site-packages/tinycss2/__init__.py
---
Changes v2 -> v3:
- split out host pyinstaller.py change
Changes v1 -> v2:
- remove unused os import
--- support/scripts/pyinstaller.py | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+)
support/scripts/pyinstaller.py +22 -0
@@ -2,12 +2,33 @@ import argparse import glob +import pathlib + +from importlib.machinery import PathFinder +from importlib.metadata import DistributionFinder from installer import install +from installer._core import _process_WHEEL_file from installer.destinations import SchemeDictionaryDestination from installer.sources import WheelFile +def clean(source, destination): + scheme = _process_WHEEL_file(source) + scheme_path = destination.scheme_dict[scheme] + context = DistributionFinder.Context( + name=source.distribution, + path=[scheme_path], + ) + for path in PathFinder.find_distributions(context=context): + if not path.files: + continue + for file in path.files: + file_path = pathlib.Path(file.locate()) + if file_path.exists(): + file_path.unlink() + + def main(): """Entry point for CLI.""" ap = argparse.ArgumentParser("python pyinstaller.py")
@@ -58,6 +79,7 @@ def main(): ) with WheelFile.open(glob.glob(args.wheel_file)[0]) as source: + clean(source, destination) install( source=source, destination=destination,
Signature · 6 lines
2.34.1 _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot
Yann E. MORIN
AppliedRe: [Buildroot] [PATCH v3 2/2] package/pkg-python: clean conflicting pep517 packages before install
James, All,
On 2022-08-22 01:54 -0600, James Hilliard spake thusly:
Quoted 37 lines
The python installer package isn't able to overwrite files of packges that already exist, this causes problems when doing a rebuild or update without a full clean.
To fix this we can use functionality from importlib to identify and remove any conflicting python package files before installation.
Fixes:
Traceback (most recent call last):
File "/home/buildroot/buildroot/support/scripts/pyinstaller.py", line 69, in <module>main() File "/home/buildroot/buildroot/support/scripts/pyinstaller.py", line 61, in main install( File "/home/buildroot/buildroot/output/host/lib/python3.10/site-packages/installer/_core.py", line 109, in install record = destination.write_file(File "
/home/buildroot/buildroot/output/host/lib/python3.10/site-packages/installer/destinations.py", line 207, inwrite_file
returnself.write_to_fs(scheme, path_, stream, is_executable)
File "/home/buildroot/buildroot/output/host/lib/python3.10/site-packages/installer/destinations.py", line 167, inwrite_to_fs
raise FileExistsError(message)
FileExistsError: File already exists:/home/buildroot/buildroot/output/target/usr/lib/python3.10/site-packages/tinycss2/__init__.pySigned-off-by James Hilliard <[email protected]>Tested-by Marcus Hoffmann <[email protected]>---
Changes v2 -> v3:
- split out hostpyinstaller.pychange
Changes v1 -> v2:
- remove unused os import--- support/scripts/pyinstaller.py | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+)support/scripts/pyinstaller.py +0
-0
support/scripts/pyinstaller.py line 11
Quoted 10 lines
support/scripts/pyinstaller.py +5
-0@@ -2,12 +2,33 @@ import argparse import glob +import pathlib + +from importlib.machinery import PathFinder +from importlib.metadata import DistributionFinder from installer import install +from installer._core import _process_WHEEL_file
I've extended the commit log to explain why we need to use private symbols.
support/scripts/pyinstaller.py line 24
Quoted 13 lines
support/scripts/pyinstaller.py +9
-0from installer.destinations import SchemeDictionaryDestination from installer.sources import WheelFile +def clean(source, destination): + scheme = _process_WHEEL_file(source) + scheme_path = destination.scheme_dict[scheme] + context = DistributionFinder.Context( + name=source.distribution, + path=[scheme_path], + ) + for path in PathFinder.find_distributions(context=context): + if not path.files:
I've changed this for an explicit test against None.
Applied to master, thanks.
Regards,
Yann E. MORIN.
Quoted 25 lines
support/scripts/pyinstaller.py +8
-0+ continue + for file in path.files: + file_path = pathlib.Path(file.locate()) + if file_path.exists(): + file_path.unlink() + + def main(): """Entry point for CLI.""" ap = argparse.ArgumentParser("python pyinstaller.py")@@ -58,6 +79,7 @@ def main(): ) with WheelFile.open(glob.glob(args.wheel_file)[0]) as source: + clean(source, destination) install( source=source, destination=destination,Signature · 6 lines
2.34.1 _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot
Signature · 10 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------' _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot
Yann E. MORIN
AppliedJames, All,
On 2022-08-22 01:54 -0600, James Hilliard spake thusly:
Quoted 9 lines
The python installer cli isn't able to overwrite files of packages that already exist, this causes problems when doing a rebuild or update without a full clean.
Since we need to add functionality to our
pyinstaller.pyscript to fix this issue we must also usepyinstaller.pyfor host python packages.Signed-off-by James Hilliard <[email protected]>Tested-by Marcus Hoffmann <[email protected]>
Applied to master, thanks.
Regards,
Yann E. MORIN.
Quoted 56 lines
---
Changes v2 -> v3:
- set interpreter to $(HOST_DIR)/bin/python--- package/pkg-python.mk | 12 ++++++++++-- support/scripts/pyinstaller.py | 0 2 files changed, 10 insertions(+), 2 deletions(-) mode change 100755 => 100644 support/scripts/pyinstaller.pypackage/pkg-python.mk +10
-2@@ -139,6 +139,14 @@ PKG_PYTHON_PEP517_INSTALL_STAGING_OPTS = \ HOST_PKG_PYTHON_PEP517_ENV = \ $(HOST_PKG_PYTHON_ENV) +HOST_PKG_PYTHON_PEP517_INSTALL_OPTS = \ + --interpreter=$(HOST_DIR)/bin/python \ + --script-kind=posix \ + --purelib=$(HOST_DIR)/lib/python$(PYTHON3_VERSION_MAJOR)/site-packages \ + --headers=$(HOST_DIR)/include/python$(PYTHON3_VERSION_MAJOR) \ + --scripts=$(HOST_DIR)/bin \ + --data=$(HOST_DIR) + HOST_PKG_PYTHON_PEP517_BOOTSTRAP_INSTALL_OPTS = \ --installdir=$(HOST_DIR)/lib/python$(PYTHON3_VERSION_MAJOR)/site-packages@@ -200,7 +208,7 @@ $(2)_BASE_INSTALL_STAGING_CMD = $(TOPDIR)/support/scripts/pyinstaller.py dist/* else $(2)_BASE_ENV = $$(HOST_PKG_PYTHON_PEP517_ENV) $(2)_BASE_BUILD_CMD = -m build -n -w -$(2)_BASE_INSTALL_CMD = -m installer dist/* +$(2)_BASE_INSTALL_CMD = $(TOPDIR)/support/scripts/pyinstaller.py dist/* $$(HOST_PKG_PYTHON_PEP517_INSTALL_OPTS) endif else ifeq ($$($(2)_SETUP_TYPE),flit-bootstrap) ifeq ($(4),target)@@ -208,7 +216,7 @@ $$(error flit-bootstrap setup type only supported for host packages) else $(2)_BASE_ENV = $$(HOST_PKG_PYTHON_PEP517_ENV) $(2)_BASE_BUILD_CMD = -m flit_core.wheel -$(2)_BASE_INSTALL_CMD ?= -m installer dist/* +$(2)_BASE_INSTALL_CMD ?= $(TOPDIR)/support/scripts/pyinstaller.py dist/* $$(HOST_PKG_PYTHON_PEP517_INSTALL_OPTS) endif else $$(error "Invalid $(2)_SETUP_TYPE. Valid options are 'distutils', 'setuptools', 'pep517' or 'flit'.")support/scripts/pyinstaller.py +0
-0Signature · 6 lines
2.34.1 _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot
Signature · 10 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------' _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot