Skip to contentSkip to conversation

package/pkg-python: clean conflicting pep517 packages before install

PATCH In review James Hilliard 2 participants 3 messages Aug 22, 2022

lore
Share & tools
lore
#0 lore raw

PATCH v2 1/1 package/pkg-python: clean conflicting pep517 packages before install

The python installer package isn't able to overwrite packages that already exist, this causes problems when doing a rebuild or update without a full clean.

To fix this we can use functionality from importlib to identify and remove any conflicting python package files before installation.

Since we are adding this functionality to our pyinstaller.py script we must also use pyinstaller.py for host python packages to avoid hitting this issue there as well.

Fixes:
Traceback (most recent call last):
File "/home/buildroot/buildroot/support/scripts/pyinstaller.py", line 69, in <module>

    main()
  File "/home/buildroot/buildroot/support/scripts/pyinstaller.py", line 61, in main
    install(
  File "/home/buildroot/buildroot/output/host/lib/python3.10/site-packages/installer/_core.py", line 109, in install
    record = destination.write_file(

File "/home/buildroot/buildroot/output/host/lib/python3.10/site-packages/installer/destinations.py", line 207, in write_file
return self.write_to_fs(scheme, path_, stream, is_executable)
File "/home/buildroot/buildroot/output/host/lib/python3.10/site-packages/installer/destinations.py", line 167, in write_to_fs
raise FileExistsError(message)
FileExistsError: File already exists: /home/buildroot/buildroot/output/target/usr/lib/python3.10/site-packages/tinycss2/__init__.py

---
Changes v1 -> v2:
- remove unused os import

---
 package/pkg-python.mk          | 12 ++++++++++--
 support/scripts/pyinstaller.py | 22 ++++++++++++++++++++++
 2 files changed, 32 insertions(+), 2 deletions(-)
package/pkg-python.mk +10 -2
index 5794e3a195..b147827821 100644
@@ -139,6 +139,14 @@ PKG_PYTHON_PEP517_INSTALL_STAGING_OPTS = \
 HOST_PKG_PYTHON_PEP517_ENV = \
 	$(HOST_PKG_PYTHON_ENV)
 
+HOST_PKG_PYTHON_PEP517_INSTALL_OPTS = \
+	--interpreter=/bin/python \
+	--script-kind=posix \
+	--purelib=$(HOST_DIR)/lib/python$(PYTHON3_VERSION_MAJOR)/site-packages \
+	--headers=$(HOST_DIR)/include/python$(PYTHON3_VERSION_MAJOR) \
+	--scripts=$(HOST_DIR)/bin \
+	--data=$(HOST_DIR)
+
 HOST_PKG_PYTHON_PEP517_BOOTSTRAP_INSTALL_OPTS = \
 	--installdir=$(HOST_DIR)/lib/python$(PYTHON3_VERSION_MAJOR)/site-packages
 
@@ -200,7 +208,7 @@ $(2)_BASE_INSTALL_STAGING_CMD = $(TOPDIR)/support/scripts/pyinstaller.py dist/*
 else
 $(2)_BASE_ENV = $$(HOST_PKG_PYTHON_PEP517_ENV)
 $(2)_BASE_BUILD_CMD = -m build -n -w
-$(2)_BASE_INSTALL_CMD = -m installer dist/*
+$(2)_BASE_INSTALL_CMD = $(TOPDIR)/support/scripts/pyinstaller.py dist/* $$(HOST_PKG_PYTHON_PEP517_INSTALL_OPTS)
 endif
 else ifeq ($$($(2)_SETUP_TYPE),flit-bootstrap)
 ifeq ($(4),target)
@@ -208,7 +216,7 @@ $$(error flit-bootstrap setup type only supported for host packages)
 else
 $(2)_BASE_ENV = $$(HOST_PKG_PYTHON_PEP517_ENV)
 $(2)_BASE_BUILD_CMD = -m flit_core.wheel
-$(2)_BASE_INSTALL_CMD ?= -m installer dist/*
+$(2)_BASE_INSTALL_CMD ?= $(TOPDIR)/support/scripts/pyinstaller.py dist/* $$(HOST_PKG_PYTHON_PEP517_INSTALL_OPTS)
 endif
 else
 $$(error "Invalid $(2)_SETUP_TYPE. Valid options are 'distutils', 'setuptools', 'pep517' or 'flit'.")
support/scripts/pyinstaller.py +22 -0
index 6dd9242327..d9b77cac19 100755
@@ -2,12 +2,33 @@
 
 import argparse
 import glob
+import pathlib
+
+from importlib.machinery import PathFinder
+from importlib.metadata import DistributionFinder
 
 from installer import install
+from installer._core import _process_WHEEL_file
 from installer.destinations import SchemeDictionaryDestination
 from installer.sources import WheelFile
 
 
+def clean(source, destination):
+    scheme = _process_WHEEL_file(source)
+    scheme_path = destination.scheme_dict[scheme]
+    context = DistributionFinder.Context(
+        name=source.distribution,
+        path=[scheme_path],
+    )
+    for path in PathFinder.find_distributions(context=context):
+        if not path.files:
+            continue
+        for file in path.files:
+            file_path = pathlib.Path(file.locate())
+            if file_path.exists():
+                file_path.unlink()
+
+
 def main():
     """Entry point for CLI."""
     ap = argparse.ArgumentParser("python pyinstaller.py")
@@ -58,6 +79,7 @@ def main():
     )
 
     with WheelFile.open(glob.glob(args.wheel_file)[0]) as source:
+        clean(source, destination)
         install(
             source=source,
             destination=destination,
Signature · 6 lines
2.34.1

_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
#1 lore raw

James, All,

On 2022-08-22 00:02 -0600, James Hilliard spake thusly:

Quoted 1 line

The python installer package isn't able to overwrite packages that

s/packages/files of packges/

Quoted 9 lines

already exist, this causes problems when doing a rebuild or update without a full clean.

To fix this we can use functionality from importlib to identify and remove any conflicting python package files before installation.

Since we are adding this functionality to our pyinstaller.py script we must also use pyinstaller.py for host python packages to avoid hitting this issue there as well.

Please split in two:
- first pach to use ur pyinstaller for host packages
- second patch to cleanup

[--SNIP--]

package/pkg-python.mk PKG_PYTHON_PEP517_INSTALL_STAGING_OPTS = \ · line 143

Quoted 10 lines
package/pkg-python.mk +2 -0
index 5794e3a195..b147827821 100644
@@ -139,6 +139,14 @@ PKG_PYTHON_PEP517_INSTALL_STAGING_OPTS = \
 HOST_PKG_PYTHON_PEP517_ENV = \
 	$(HOST_PKG_PYTHON_ENV)
 
+HOST_PKG_PYTHON_PEP517_INSTALL_OPTS = \
+	--interpreter=/bin/python \

Why is this not $(HOST_DIR)/bin/python ?

There is no guarantee that there is a python interpreter on the build machine, and we anyway want to use our python, not the system one.

Regards,
Yann E. MORIN.

Signature · 10 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
#2 lore raw
Quoted 7 lines · Yann E. MORIN

James, All,

On 2022-08-22 00:02 -0600, James Hilliard spake thusly:

Quoted 1 line

The python installer package isn't able to overwrite packages that

s/packages/files of packges/

Changed in v3

Quoted 7 lines
Quoted 7 lines

already exist, this causes problems when doing a rebuild or update without a full clean.

To fix this we can use functionality from importlib to identify and remove any conflicting python package files before installation.

Quoted 7 lines
Quoted 3 lines

Since we are adding this functionality to our pyinstaller.py script we must also use pyinstaller.py for host python packages to avoid hitting this issue there as well.

Please split in two:
- first pach to use ur pyinstaller for host packages
- second patch to cleanup

Split in v3:
https://patchwork.ozlabs.org/project/buildroot/patch/[email protected]/
https://patchwork.ozlabs.org/project/buildroot/patch/[email protected]/

Quoted 10 lines

[--SNIP--]

Quoted 8 lines
package/pkg-python.mk +0 -0
index 5794e3a195..b147827821 100644
@@ -139,6 +139,14 @@ PKG_PYTHON_PEP517_INSTALL_STAGING_OPTS = \
 HOST_PKG_PYTHON_PEP517_ENV = \
      $(HOST_PKG_PYTHON_ENV)
 
Quoted 4 lines

package/pkg-python.mk PKG_PYTHON_PEP517_INSTALL_STAGING_OPTS = \ · line 143

Quoted 2 lines
package/pkg-python.mk +2 -0
+HOST_PKG_PYTHON_PEP517_INSTALL_OPTS = \
+     --interpreter=/bin/python \

Why is this not $(HOST_DIR)/bin/python ?

Fixed in v3:
https://patchwork.ozlabs.org/project/buildroot/patch/[email protected]/

Quoted 7 lines

There is no guarantee that there is a python interpreter on the build machine, and we anyway want to use our python, not the system one.

Regards,
Yann E. MORIN.

Quoted 7 lines

--
.-----------------.--------------------.------------------.--------------------.

|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |

'------------------------------^-------^------------------^--------------------'

_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot