Skip to contentSkip to conversation

Support excluding .py files from removal

PATCH New Thomas Petazzoni 2 participants 7 messages Nov 29, 2019 – Dec 5, 2019

lore
Share & tools
lore
#0 lore raw

v2 cover Support excluding .py files from removal

Hello,

This is a v2 of the patch series fixing the OpenCV3 issue due to the removal of .py files when only .pyc files are kept on the target.

As reported by Peter, the v1 caused some check-package warnings due to the PYTHON_KEEP_PY_FILES/PYTHON3_KEEP_PY_FILES variables that were set in opencv3.mk: they have a prefix that doesn't match the package name.

So in the end, we go ahead and create a <pkg>_KEEP_PY_FILES variable, that gets collected by pkg-generic.mk into a global PYTHON_KEEP_PY_FILES, which in turn is observed by python.mk and python3.mk.

Best regards,

Thomas

Thomas Petazzoni (2):
package/{pkg-generic,python,python3}: add mechanism to exclude .py
files from removal
package/opencv3: ensure the python module works when

    BR2_PACKAGE_PYTHON{,3}_PYC_ONLY=y
 package/opencv3/opencv3.mk | 2 ++
 package/pkg-generic.mk     | 1 +
 package/python/python.mk   | 4 +++-
 package/python3/python3.mk | 4 +++-
 4 files changed, 9 insertions(+), 2 deletions(-)
Signature · 1 line
2.23.0
#1 lore raw

PATCH v2 1/2 package/{pkg-generic, python, python3}: add mechanism to exclude .py files from removal

When BR2_PACKAGE_PYTHON{,3}_PYC_ONLY=y, we force remove all .py files from the system, as they have all been byte-compiled into their .pyc variants.

However, it turns out that some packages (e.g: OpenCV) do some funky things with a few .py files: they pass them through Python's execfile() facility, which only works with .py files and not .pyc files. It is used by OpenCV for example to read two small configuration files.

In order to support such use cases, this commit introduces a very simple mechanism by which packages can exclude some path patterns from the .py removal: a per-package <pkg>_KEEP_PY_FILES variable that is collected into a global PYTHON_KEEP_PY_FILES variable, then used by the python/python3 target-finalize hooks.

This variable is intentionally not documented, this is really a hack that we ideally would like to see go away, and we'd rather not see its usage spread too much.

This is necessary to be able to fix bug #12171.

---
 package/pkg-generic.mk     | 1 +
 package/python/python.mk   | 4 +++-
 package/python3/python3.mk | 4 +++-
 3 files changed, 7 insertions(+), 2 deletions(-)
package/pkg-generic.mk +1 -0
index 7d6fa08418..0f93f3c58b 100644
@@ -1054,6 +1054,7 @@ PACKAGES_USERS += $$($(2)_USERS)$$(sep)
 endif
 TARGET_FINALIZE_HOOKS += $$($(2)_TARGET_FINALIZE_HOOKS)
 ROOTFS_PRE_CMD_HOOKS += $$($(2)_ROOTFS_PRE_CMD_HOOKS)
+PYTHON_KEEP_PY_FILES += $$($(2)_KEEP_PY_FILES)
 
 ifeq ($$($(2)_SITE_METHOD),svn)
 DL_TOOLS_DEPENDENCIES += svn
package/python/python.mk +3 -1
index 8c2b195c91..dda9898f03 100644
@@ -267,7 +267,9 @@ endif
 
 ifeq ($(BR2_PACKAGE_PYTHON_PYC_ONLY),y)
 define PYTHON_REMOVE_PY_FILES
-	find $(TARGET_DIR)/usr/lib/python$(PYTHON_VERSION_MAJOR) -name '*.py' -print0 | \
+	find $(TARGET_DIR)/usr/lib/python$(PYTHON_VERSION_MAJOR) -name '*.py' \
+		$(if $(PYTHON_KEEP_PY_FILES),-not \( $(call finddirclauses,$(TARGET_DIR),$(PYTHON_KEEP_PY_FILES)) \) ) \
+		-print0 | \
 		xargs -0 --no-run-if-empty rm -f
 endef
 PYTHON_TARGET_FINALIZE_HOOKS += PYTHON_REMOVE_PY_FILES
package/python3/python3.mk +3 -1
index d6fda5c274..d9084d126b 100644
@@ -285,7 +285,9 @@ endif
 
 ifeq ($(BR2_PACKAGE_PYTHON3_PYC_ONLY),y)
 define PYTHON3_REMOVE_PY_FILES
-	find $(TARGET_DIR)/usr/lib/python$(PYTHON3_VERSION_MAJOR) -name '*.py' -print0 | \
+	find $(TARGET_DIR)/usr/lib/python$(PYTHON3_VERSION_MAJOR) -name '*.py' \
+		$(if $(PYTHON_KEEP_PY_FILES),-not \( $(call finddirclauses,$(TARGET_DIR),$(PYTHON_KEEP_PY_FILES)) \) ) \
+		-print0 | \
 		xargs -0 --no-run-if-empty rm -f
 endef
 PYTHON3_TARGET_FINALIZE_HOOKS += PYTHON3_REMOVE_PY_FILES
Signature · 1 line
2.23.0
#2 lore raw

PATCH v2 1/2 package/{pkg-generic, python, python3}: add mechanism to exclude .py files from removal

Quoted 63 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

"Thomas" == Thomas Petazzoni <[email protected]> writes:

When BR2_PACKAGE_PYTHON{,3}_PYC_ONLY=y, we force remove all .py files from the system, as they have all been byte-compiled into their .pyc variants.

However, it turns out that some packages (e.g: OpenCV) do some funky things with a few .py files: they pass them through Python's execfile() facility, which only works with .py files and not .pyc files. It is used by OpenCV for example to read two small configuration files.

In order to support such use cases, this commit introduces a very simple mechanism by which packages can exclude some path patterns from the .py removal: a per-package <pkg>_KEEP_PY_FILES variable that is collected into a global PYTHON_KEEP_PY_FILES variable, then used by the python/python3 target-finalize hooks.

This variable is intentionally not documented, this is really a hack that we ideally would like to see go away, and we'd rather not see its usage spread too much.

This is necessary to be able to fix bug #12171.

---
 package/pkg-generic.mk     | 1 +
 package/python/python.mk   | 4 +++-
 package/python3/python3.mk | 4 +++-
 3 files changed, 7 insertions(+), 2 deletions(-)
package/pkg-generic.mk +1 -0
index 7d6fa08418..0f93f3c58b 100644
@@ -1054,6 +1054,7 @@ PACKAGES_USERS += $$($(2)_USERS)$$(sep)
 endif
 TARGET_FINALIZE_HOOKS += $$($(2)_TARGET_FINALIZE_HOOKS)
 ROOTFS_PRE_CMD_HOOKS += $$($(2)_ROOTFS_PRE_CMD_HOOKS)
+PYTHON_KEEP_PY_FILES += $$($(2)_KEEP_PY_FILES)
 
 ifeq ($$($(2)_SITE_METHOD),svn)
 DL_TOOLS_DEPENDENCIES += svn
package/python/python.mk +3 -1
index 8c2b195c91..dda9898f03 100644
@@ -267,7 +267,9 @@ endif
 
 ifeq ($(BR2_PACKAGE_PYTHON_PYC_ONLY),y)
 define PYTHON_REMOVE_PY_FILES
-	find $(TARGET_DIR)/usr/lib/python$(PYTHON_VERSION_MAJOR) -name '*.py' -print0 | \
+	find $(TARGET_DIR)/usr/lib/python$(PYTHON_VERSION_MAJOR) -name '*.py' \
+		$(if $(PYTHON_KEEP_PY_FILES),-not \( $(call finddirclauses,$(TARGET_DIR),$(PYTHON_KEEP_PY_FILES)) \) ) \
+		-print0 | \
 		xargs -0 --no-run-if-empty rm -f
 endef
 PYTHON_TARGET_FINALIZE_HOOKS += PYTHON_REMOVE_PY_FILES
package/python3/python3.mk +0 -0
index d6fda5c274..d9084d126b 100644

package/python3/python3.mk endif · line 289

Quoted 6 lines
package/python3/python3.mk +2 -1
@@ -285,7 +285,9 @@ endif
 
 ifeq ($(BR2_PACKAGE_PYTHON3_PYC_ONLY),y)
 define PYTHON3_REMOVE_PY_FILES
-	find $(TARGET_DIR)/usr/lib/python$(PYTHON3_VERSION_MAJOR) -name '*.py' -print0 | \
+	find $(TARGET_DIR)/usr/lib/python$(PYTHON3_VERSION_MAJOR) -name '*.py' \
+		$(if $(PYTHON_KEEP_PY_FILES),-not \( $(call finddirclauses,$(TARGET_DIR),$(PYTHON_KEEP_PY_FILES)) \) ) \

This unfortunately doesn't work as PYTHON_KEEP_PY_FILES will contain a bunch of white space, so without the opencv python module you end up with:

find /home/peko/source/buildroot/output-opencv/target/usr/lib/python3.8 -name '*.py' -not \( \) -print0 | xargs -0 --no-run-if-empty rm -f
find: invalid expression; empty parentheses are not allowed.

So I added a $(strip) around PYTHON_KEEP_PY_FILES and committed, thanks.

Signature · 1 line
Bye, Peter Korsgaard
#3 lore raw

PATCH v2 1/2 package/{pkg-generic, python, python3}: add mechanism to exclude .py files from removal

In reply to Thomas Petazzoni
Quoted 18 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

"Thomas" == Thomas Petazzoni <[email protected]> writes:

When BR2_PACKAGE_PYTHON{,3}_PYC_ONLY=y, we force remove all .py files from the system, as they have all been byte-compiled into their .pyc variants.

However, it turns out that some packages (e.g: OpenCV) do some funky things with a few .py files: they pass them through Python's execfile() facility, which only works with .py files and not .pyc files. It is used by OpenCV for example to read two small configuration files.

In order to support such use cases, this commit introduces a very simple mechanism by which packages can exclude some path patterns from the .py removal: a per-package <pkg>_KEEP_PY_FILES variable that is collected into a global PYTHON_KEEP_PY_FILES variable, then used by the python/python3 target-finalize hooks.

Quoted 7 lines

This variable is intentionally not documented, this is really a hack that we ideally would like to see go away, and we'd rather not see its usage spread too much.

This is necessary to be able to fix bug #12171.

Committed to 2019.02.x and 2019.08.x, thanks.

Signature · 1 line
Bye, Peter Korsgaard
#4 lore raw

PATCH v2 2/2 package/opencv3: ensure the python module works when BR2_PACKAGE_PYTHON{, 3}_PYC_ONLY=y

In reply to Thomas Petazzoni

The OpenCV Python module does a fairly strange thing to read a few configuration details: it uses Python's execfile() to execute two .py files and access a few variables. However, execfile() only works with .py files and not .pyc files.

When BR2_PACKAGE_PYTHON{,3}_PYC_ONLY=y, the .py files are all removed, causing the OpenCV Python module to not work:

File "usr/lib/python3.7/site-packages/cv2/__init__.py", line 89, in <module> File "usr/lib/python3.7/site-packages/cv2/__init__.py", line 58, in bootstrap File "usr/lib/python3.7/site-packages/cv2/__init__.py", line 56, in load_first_config
ImportError: OpenCV loader: missing configuration file: ['config.py']. Check OpenCV installation.

To fix this problem, this commit uses the newly introduced <pkg>_KEEP_PY_FILES mechanism, to ensure the important config*.py files are kept.

Fixes:

https://bugs.busybox.net/show_bug.cgi?id=12171

---
 package/opencv3/opencv3.mk | 2 ++
 1 file changed, 2 insertions(+)
package/opencv3/opencv3.mk +2 -0
index 83094ed78b..d38107f6e7 100644
@@ -328,6 +328,7 @@ OPENCV3_CONF_OPTS += \
 	-DPYTHON2_PACKAGES_PATH=/usr/lib/python$(PYTHON_VERSION_MAJOR)/site-packages \
 	-DPYTHON2_NUMPY_VERSION=$(PYTHON_NUMPY_VERSION)
 OPENCV3_DEPENDENCIES += python
+OPENCV3_KEEP_PY_FILES += usr/lib/python$(PYTHON_VERSION_MAJOR)/site-packages/cv2/config*.py
 else
 OPENCV3_CONF_OPTS += \
 	-DBUILD_opencv_python2=OFF \
@@ -339,6 +340,7 @@ OPENCV3_CONF_OPTS += \
 	-DPYTHON3_PACKAGES_PATH=/usr/lib/python$(PYTHON3_VERSION_MAJOR)/site-packages \
 	-DPYTHON3_NUMPY_VERSION=$(PYTHON_NUMPY_VERSION)
 OPENCV3_DEPENDENCIES += python3
+OPENCV3_KEEP_PY_FILES += usr/lib/python$(PYTHON3_VERSION_MAJOR)/site-packages/cv2/config*.py
 endif
 OPENCV3_CONF_ENV += $(PKG_PYTHON_DISTUTILS_ENV)
 OPENCV3_DEPENDENCIES += python-numpy
Signature · 1 line
2.23.0
#5 lore raw

PATCH v2 2/2 package/opencv3: ensure the python module works when BR2_PACKAGE_PYTHON{, 3}_PYC_ONLY=y

Quoted 19 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

"Thomas" == Thomas Petazzoni <[email protected]> writes:

The OpenCV Python module does a fairly strange thing to read a few configuration details: it uses Python's execfile() to execute two .py files and access a few variables. However, execfile() only works with .py files and not .pyc files.

When BR2_PACKAGE_PYTHON{,3}_PYC_ONLY=y, the .py files are all removed, causing the OpenCV Python module to not work:

File "usr/lib/python3.7/site-packages/cv2/__init__.py", line 89, in <module> File "usr/lib/python3.7/site-packages/cv2/__init__.py", line 58, in bootstrap File "usr/lib/python3.7/site-packages/cv2/__init__.py", line 56, in load_first_config
ImportError: OpenCV loader: missing configuration file: ['config.py']. Check OpenCV installation.

To fix this problem, this commit uses the newly introduced <pkg>_KEEP_PY_FILES mechanism, to ensure the important config*.py files are kept.

Quoted 5 lines

Fixes:

https://bugs.busybox.net/show_bug.cgi?id=12171

Committed, thanks.

Signature · 1 line
Bye, Peter Korsgaard
#6 lore raw

PATCH v2 2/2 package/opencv3: ensure the python module works when BR2_PACKAGE_PYTHON{, 3}_PYC_ONLY=y

In reply to Thomas Petazzoni
Quoted 19 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

"Thomas" == Thomas Petazzoni <[email protected]> writes:

The OpenCV Python module does a fairly strange thing to read a few configuration details: it uses Python's execfile() to execute two .py files and access a few variables. However, execfile() only works with .py files and not .pyc files.

When BR2_PACKAGE_PYTHON{,3}_PYC_ONLY=y, the .py files are all removed, causing the OpenCV Python module to not work:

File "usr/lib/python3.7/site-packages/cv2/__init__.py", line 89, in <module> File "usr/lib/python3.7/site-packages/cv2/__init__.py", line 58, in bootstrap File "usr/lib/python3.7/site-packages/cv2/__init__.py", line 56, in load_first_config
ImportError: OpenCV loader: missing configuration file: ['config.py']. Check OpenCV installation.

To fix this problem, this commit uses the newly introduced <pkg>_KEEP_PY_FILES mechanism, to ensure the important config*.py files are kept.

Quoted 5 lines

Fixes:

https://bugs.busybox.net/show_bug.cgi?id=12171

Committed to 2019.02.x and 2019.08.x, thanks.

Signature · 1 line
Bye, Peter Korsgaard