Support excluding .py files from removal
v2 cover Support excluding .py files from removal
Hello,
This is a v2 of the patch series fixing the OpenCV3 issue due to the removal of .py files when only .pyc files are kept on the target.
As reported by Peter, the v1 caused some check-package warnings due to the PYTHON_KEEP_PY_FILES/PYTHON3_KEEP_PY_FILES variables that were set in opencv3.mk: they have a prefix that doesn't match the package name.
So in the end, we go ahead and create a <pkg>_KEEP_PY_FILES variable, that gets collected by pkg-generic.mk into a global PYTHON_KEEP_PY_FILES, which in turn is observed by python.mk and python3.mk.
Best regards,
Thomas
Thomas Petazzoni (2):
package/{pkg-generic,python,python3}: add mechanism to exclude .py
files from removal
package/opencv3: ensure the python module works when
BR2_PACKAGE_PYTHON{,3}_PYC_ONLY=ypackage/opencv3/opencv3.mk | 2 ++ package/pkg-generic.mk | 1 + package/python/python.mk | 4 +++- package/python3/python3.mk | 4 +++- 4 files changed, 9 insertions(+), 2 deletions(-)
Signature · 1 line
2.23.0
PATCH v2 1/2 package/{pkg-generic, python, python3}: add mechanism to exclude .py files from removal
When BR2_PACKAGE_PYTHON{,3}_PYC_ONLY=y, we force remove all .py files from the system, as they have all been byte-compiled into their .pyc variants.
However, it turns out that some packages (e.g: OpenCV) do some funky things with a few .py files: they pass them through Python's execfile() facility, which only works with .py files and not .pyc files. It is used by OpenCV for example to read two small configuration files.
In order to support such use cases, this commit introduces a very simple mechanism by which packages can exclude some path patterns from the .py removal: a per-package <pkg>_KEEP_PY_FILES variable that is collected into a global PYTHON_KEEP_PY_FILES variable, then used by the python/python3 target-finalize hooks.
This variable is intentionally not documented, this is really a hack that we ideally would like to see go away, and we'd rather not see its usage spread too much.
This is necessary to be able to fix bug #12171.
--- package/pkg-generic.mk | 1 + package/python/python.mk | 4 +++- package/python3/python3.mk | 4 +++- 3 files changed, 7 insertions(+), 2 deletions(-)
package/pkg-generic.mk +1 -0
@@ -1054,6 +1054,7 @@ PACKAGES_USERS += $$($(2)_USERS)$$(sep) endif TARGET_FINALIZE_HOOKS += $$($(2)_TARGET_FINALIZE_HOOKS) ROOTFS_PRE_CMD_HOOKS += $$($(2)_ROOTFS_PRE_CMD_HOOKS) +PYTHON_KEEP_PY_FILES += $$($(2)_KEEP_PY_FILES) ifeq ($$($(2)_SITE_METHOD),svn) DL_TOOLS_DEPENDENCIES += svn
package/python/python.mk +3 -1
@@ -267,7 +267,9 @@ endif ifeq ($(BR2_PACKAGE_PYTHON_PYC_ONLY),y) define PYTHON_REMOVE_PY_FILES - find $(TARGET_DIR)/usr/lib/python$(PYTHON_VERSION_MAJOR) -name '*.py' -print0 | \ + find $(TARGET_DIR)/usr/lib/python$(PYTHON_VERSION_MAJOR) -name '*.py' \ + $(if $(PYTHON_KEEP_PY_FILES),-not \( $(call finddirclauses,$(TARGET_DIR),$(PYTHON_KEEP_PY_FILES)) \) ) \ + -print0 | \ xargs -0 --no-run-if-empty rm -f endef PYTHON_TARGET_FINALIZE_HOOKS += PYTHON_REMOVE_PY_FILES
package/python3/python3.mk +3 -1
@@ -285,7 +285,9 @@ endif ifeq ($(BR2_PACKAGE_PYTHON3_PYC_ONLY),y) define PYTHON3_REMOVE_PY_FILES - find $(TARGET_DIR)/usr/lib/python$(PYTHON3_VERSION_MAJOR) -name '*.py' -print0 | \ + find $(TARGET_DIR)/usr/lib/python$(PYTHON3_VERSION_MAJOR) -name '*.py' \ + $(if $(PYTHON_KEEP_PY_FILES),-not \( $(call finddirclauses,$(TARGET_DIR),$(PYTHON_KEEP_PY_FILES)) \) ) \ + -print0 | \ xargs -0 --no-run-if-empty rm -f endef PYTHON3_TARGET_FINALIZE_HOOKS += PYTHON3_REMOVE_PY_FILES
Signature · 1 line
2.23.0
PATCH v2 1/2 package/{pkg-generic, python, python3}: add mechanism to exclude .py files from removal
Quoted 63 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Thomas" == Thomas Petazzoni <[email protected]> writes:
When BR2_PACKAGE_PYTHON{,3}_PYC_ONLY=y, we force remove all .py files from the system, as they have all been byte-compiled into their .pyc variants.
However, it turns out that some packages (e.g: OpenCV) do some funky things with a few .py files: they pass them through Python's
execfile()facility, which only works with .py files and not .pyc files. It is used by OpenCV for example to read two small configuration files.In order to support such use cases, this commit introduces a very simple mechanism by which packages can exclude some path patterns from the .py removal: a per-package <pkg>_KEEP_PY_FILES variable that is collected into a global
PYTHON_KEEP_PY_FILESvariable, then used by the python/python3 target-finalize hooks.This variable is intentionally not documented, this is really a hack that we ideally would like to see go away, and we'd rather not see its usage spread too much.
This is necessary to be able to fix bug #12171.
Signed-off-by Thomas Petazzoni <[email protected]>--- package/pkg-generic.mk | 1 + package/python/python.mk | 4 +++- package/python3/python3.mk | 4 +++- 3 files changed, 7 insertions(+), 2 deletions(-)package/pkg-generic.mk +1
-0@@ -1054,6 +1054,7 @@ PACKAGES_USERS += $$($(2)_USERS)$$(sep) endif TARGET_FINALIZE_HOOKS += $$($(2)_TARGET_FINALIZE_HOOKS) ROOTFS_PRE_CMD_HOOKS += $$($(2)_ROOTFS_PRE_CMD_HOOKS) +PYTHON_KEEP_PY_FILES += $$($(2)_KEEP_PY_FILES) ifeq ($$($(2)_SITE_METHOD),svn) DL_TOOLS_DEPENDENCIES += svnpackage/python/python.mk +3
-1@@ -267,7 +267,9 @@ endif ifeq ($(BR2_PACKAGE_PYTHON_PYC_ONLY),y) define PYTHON_REMOVE_PY_FILES - find $(TARGET_DIR)/usr/lib/python$(PYTHON_VERSION_MAJOR) -name '*.py' -print0 | \ + find $(TARGET_DIR)/usr/lib/python$(PYTHON_VERSION_MAJOR) -name '*.py' \ + $(if $(PYTHON_KEEP_PY_FILES),-not \( $(call finddirclauses,$(TARGET_DIR),$(PYTHON_KEEP_PY_FILES)) \) ) \ + -print0 | \ xargs -0 --no-run-if-empty rm -f endef PYTHON_TARGET_FINALIZE_HOOKS += PYTHON_REMOVE_PY_FILESpackage/python3/python3.mk +0
-0
package/python3/python3.mk endif · line 289
Quoted 6 lines
package/python3/python3.mk +2
-1@@ -285,7 +285,9 @@ endif ifeq ($(BR2_PACKAGE_PYTHON3_PYC_ONLY),y) define PYTHON3_REMOVE_PY_FILES - find $(TARGET_DIR)/usr/lib/python$(PYTHON3_VERSION_MAJOR) -name '*.py' -print0 | \ + find $(TARGET_DIR)/usr/lib/python$(PYTHON3_VERSION_MAJOR) -name '*.py' \ + $(if $(PYTHON_KEEP_PY_FILES),-not \( $(call finddirclauses,$(TARGET_DIR),$(PYTHON_KEEP_PY_FILES)) \) ) \
This unfortunately doesn't work as PYTHON_KEEP_PY_FILES will contain a bunch of white space, so without the opencv python module you end up with:
find /home/peko/source/buildroot/output-opencv/target/usr/lib/python3.8 -name '*.py' -not \( \) -print0 | xargs -0 --no-run-if-empty rm -f
find: invalid expression; empty parentheses are not allowed.
So I added a $(strip) around PYTHON_KEEP_PY_FILES and committed, thanks.
Signature · 1 line
Bye, Peter Korsgaard
PATCH v2 1/2 package/{pkg-generic, python, python3}: add mechanism to exclude .py files from removal
In reply to Thomas PetazzoniQuoted 18 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Thomas" == Thomas Petazzoni <[email protected]> writes:
When BR2_PACKAGE_PYTHON{,3}_PYC_ONLY=y, we force remove all .py files from the system, as they have all been byte-compiled into their .pyc variants.
However, it turns out that some packages (e.g: OpenCV) do some funky things with a few .py files: they pass them through Python's
execfile()facility, which only works with .py files and not .pyc files. It is used by OpenCV for example to read two small configuration files.In order to support such use cases, this commit introduces a very simple mechanism by which packages can exclude some path patterns from the .py removal: a per-package <pkg>_KEEP_PY_FILES variable that is collected into a global
PYTHON_KEEP_PY_FILESvariable, then used by the python/python3 target-finalize hooks.
Quoted 7 lines
This variable is intentionally not documented, this is really a hack that we ideally would like to see go away, and we'd rather not see its usage spread too much.
This is necessary to be able to fix bug #12171.
Signed-off-by Thomas Petazzoni <[email protected]>
Committed to 2019.02.x and 2019.08.x, thanks.
Signature · 1 line
Bye, Peter Korsgaard
PATCH v2 2/2 package/opencv3: ensure the python module works when BR2_PACKAGE_PYTHON{, 3}_PYC_ONLY=y
In reply to Thomas PetazzoniThe OpenCV Python module does a fairly strange thing to read a few configuration details: it uses Python's execfile() to execute two .py files and access a few variables. However, execfile() only works with .py files and not .pyc files.
When BR2_PACKAGE_PYTHON{,3}_PYC_ONLY=y, the .py files are all removed, causing the OpenCV Python module to not work:
File "usr/lib/python3.7/site-packages/cv2/__init__.py", line 89, in <module> File "usr/lib/python3.7/site-packages/cv2/__init__.py", line 58, in bootstrap File "usr/lib/python3.7/site-packages/cv2/__init__.py", line 56, in load_first_config
ImportError: OpenCV loader: missing configuration file: ['config.py']. Check OpenCV installation.
To fix this problem, this commit uses the newly introduced <pkg>_KEEP_PY_FILES mechanism, to ensure the important config*.py files are kept.
Fixes:
https://bugs.busybox.net/show_bug.cgi?id=12171
--- package/opencv3/opencv3.mk | 2 ++ 1 file changed, 2 insertions(+)
package/opencv3/opencv3.mk +2 -0
@@ -328,6 +328,7 @@ OPENCV3_CONF_OPTS += \ -DPYTHON2_PACKAGES_PATH=/usr/lib/python$(PYTHON_VERSION_MAJOR)/site-packages \ -DPYTHON2_NUMPY_VERSION=$(PYTHON_NUMPY_VERSION) OPENCV3_DEPENDENCIES += python +OPENCV3_KEEP_PY_FILES += usr/lib/python$(PYTHON_VERSION_MAJOR)/site-packages/cv2/config*.py else OPENCV3_CONF_OPTS += \ -DBUILD_opencv_python2=OFF \
@@ -339,6 +340,7 @@ OPENCV3_CONF_OPTS += \ -DPYTHON3_PACKAGES_PATH=/usr/lib/python$(PYTHON3_VERSION_MAJOR)/site-packages \ -DPYTHON3_NUMPY_VERSION=$(PYTHON_NUMPY_VERSION) OPENCV3_DEPENDENCIES += python3 +OPENCV3_KEEP_PY_FILES += usr/lib/python$(PYTHON3_VERSION_MAJOR)/site-packages/cv2/config*.py endif OPENCV3_CONF_ENV += $(PKG_PYTHON_DISTUTILS_ENV) OPENCV3_DEPENDENCIES += python-numpy
Signature · 1 line
2.23.0
PATCH v2 2/2 package/opencv3: ensure the python module works when BR2_PACKAGE_PYTHON{, 3}_PYC_ONLY=y
Quoted 19 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Thomas" == Thomas Petazzoni <[email protected]> writes:
The OpenCV Python module does a fairly strange thing to read a few configuration details: it uses Python's
execfile()to execute two .py files and access a few variables. However,execfile()only works with .py files and not .pyc files.When BR2_PACKAGE_PYTHON{,3}_PYC_ONLY=y, the .py files are all removed, causing the OpenCV Python module to not work:
File "
usr/lib/python3.7/site-packages/cv2/__init__.py", line 89, in <module> File "usr/lib/python3.7/site-packages/cv2/__init__.py", line 58, in bootstrap File "usr/lib/python3.7/site-packages/cv2/__init__.py", line 56, inload_first_config
ImportError: OpenCV loader: missing configuration file: ['config.py']. Check OpenCV installation.To fix this problem, this commit uses the newly introduced <pkg>_KEEP_PY_FILES mechanism, to ensure the important config*.py files are kept.
Quoted 5 lines
Fixes:
https://bugs.busybox.net/show_bug.cgi?id=12171
Signed-off-by Thomas Petazzoni <[email protected]>
Committed, thanks.
Signature · 1 line
Bye, Peter Korsgaard
PATCH v2 2/2 package/opencv3: ensure the python module works when BR2_PACKAGE_PYTHON{, 3}_PYC_ONLY=y
In reply to Thomas PetazzoniQuoted 19 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Thomas" == Thomas Petazzoni <[email protected]> writes:
The OpenCV Python module does a fairly strange thing to read a few configuration details: it uses Python's
execfile()to execute two .py files and access a few variables. However,execfile()only works with .py files and not .pyc files.When BR2_PACKAGE_PYTHON{,3}_PYC_ONLY=y, the .py files are all removed, causing the OpenCV Python module to not work:
File "
usr/lib/python3.7/site-packages/cv2/__init__.py", line 89, in <module> File "usr/lib/python3.7/site-packages/cv2/__init__.py", line 58, in bootstrap File "usr/lib/python3.7/site-packages/cv2/__init__.py", line 56, inload_first_config
ImportError: OpenCV loader: missing configuration file: ['config.py']. Check OpenCV installation.To fix this problem, this commit uses the newly introduced <pkg>_KEEP_PY_FILES mechanism, to ensure the important config*.py files are kept.
Quoted 5 lines
Fixes:
https://bugs.busybox.net/show_bug.cgi?id=12171
Signed-off-by Thomas Petazzoni <[email protected]>
Committed to 2019.02.x and 2019.08.x, thanks.
Signature · 1 line
Bye, Peter Korsgaard