target-finalize: generate all *.pyc

PATCH New Samuel Martin 4 participants 9 messages Mar 9, 2015 – Aug 5, 2015 Kbuild

Share & tools

Samuel Martin

author

2015-03-09 16:24 UTC #0 lore raw

PATCH 1/2 target-finalize: generate all *.pyc

This patch generates all *.pyc files from *.py located in <target>/usr/python*/site-packages, before stripping the rootfs.

This prevents modules from packages that do not compile the *.py files from disappearing.

Reported-by Yegor Yefremov <[email protected]>

Cc Yegor Yefremov <[email protected]>

Signed-off-by Samuel Martin <[email protected]>

---
 Makefile | 5 +++++
 1 file changed, 5 insertions(+)

Makefile +5 -0

index af043a3..ec0b923 100644
@@ -564,6 +564,11 @@ $(TARGETS_ROOTFS): target-finalize
 
 target-finalize: $(TARGETS)
 	@$(call MESSAGE,"Finalizing target directory")
+ifneq ($(BR2_PACKAGE_PYTHON)$(BR2_PACKAGE_PYTHON3),)
+	PYTHONPATH="$(PYTHON_PATH)" \
+		$(HOST_DIR)/usr/bin/python -c "import compileall; \
+			compileall.compile_dir('$(TARGET_DIR)/usr/lib/python$(PYTHON_VERSION_MAJOR)/site-packages')"
+endif
 	$(foreach hook,$(TARGET_FINALIZE_HOOKS),$($(hook))$(sep))
 	rm -rf $(TARGET_DIR)/usr/include $(TARGET_DIR)/usr/share/aclocal \
 		$(TARGET_DIR)/usr/lib/pkgconfig $(TARGET_DIR)/usr/share/pkgconfig \

Signature · 1 line

2.1.0

Samuel Martin

author

2015-03-09 16:24 UTC #1 lore raw

PATCH 2/2 package/samba4: remove SAMBA4_BUILD_PYC_FILES hook

This is now handled by target-finalize.

Signed-off-by Samuel Martin <[email protected]>

---
 package/samba4/samba4.mk | 10 ----------
 1 file changed, 10 deletions(-)

package/samba4/samba4.mk +0 ~~-10~~

index 78d92ef..7b104fb 100644
@@ -122,16 +122,6 @@ define SAMBA4_INSTALL_TARGET_CMDS
 	$(TARGET_MAKE_ENV) $(MAKE) -C $(@D) DESTDIR=$(TARGET_DIR) install
 endef
 
-# Samba just installs .py files so the purge causes problems with some tools
-ifeq ($(BR2_PACKAGE_PYTHON_PYC_ONLY),y)
-define SAMBA4_BUILD_PYC_FILES
-	PYTHONPATH="$(PYTHON_PATH)" \
-		$(HOST_DIR)/usr/bin/python -c "import compileall; \
-		compileall.compile_dir('$(TARGET_DIR)/usr/lib/python$(PYTHON_VERSION_MAJOR)/site-packages/samba')"
-endef
-SAMBA4_POST_INSTALL_TARGET_HOOKS += SAMBA4_BUILD_PYC_FILES
-endif
-
 ifeq ($(BR2_PACKAGE_SAMBA4_AD_DC),)
 SAMBA4_CONF_OPTS += --without-ad-dc
 endif

Signature · 1 line

2.1.0

Thomas Petazzoni

2015-03-09 17:54 UTC #3 lore raw

PATCH 1/2 target-finalize: generate all *.pyc

In reply to Samuel Martin

Dear Samuel Martin,

Quoted 18 lines · Samuel Martin

This patch generates all *.pyc files from *.py located in <target>/usr/python*/site-packages, before stripping the rootfs.
This prevents modules from packages that do not compile the *.py files from disappearing.
Reported-by Yegor Yefremov <[email protected]>
Cc Yegor Yefremov <[email protected]>
Signed-off-by Samuel Martin <[email protected]>
---
 Makefile | 5 +++++
 1 file changed, 5 insertions(+)
Makefile +0 -0
index af043a3..ec0b923 100644

Makefile $(TARGETS_ROOTFS): target-finalize · line 571

Quoted 8 lines

Makefile +5 -0

@@ -564,6 +564,11 @@ $(TARGETS_ROOTFS): target-finalize
 
 target-finalize: $(TARGETS)
 	@$(call MESSAGE,"Finalizing target directory")
+ifneq ($(BR2_PACKAGE_PYTHON)$(BR2_PACKAGE_PYTHON3),)
+	PYTHONPATH="$(PYTHON_PATH)" \
+		$(HOST_DIR)/usr/bin/python -c "import compileall; \
+			compileall.compile_dir('$(TARGET_DIR)/usr/lib/python$(PYTHON_VERSION_MAJOR)/site-packages')"
+endif

Why not, but then don't we want to do something in the Python package infra, and in the Python 2/3 packages themselves to avoid spending time biulding the .pyc files?

Also, using TARGET_FINALIZE_HOOKS would be nicer. But there is the issue that the hook is the same between python and python3, so we wouldn't know where to put it.

Thomas

Signature · 3 lines

Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

Samuel Martin

author

2015-03-09 21:30 UTC #4 lore raw

PATCH 1/2 target-finalize: generate all *.pyc

Hi Thomas, all,

Quoted 22 lines · Thomas Petazzoni

Dear Samuel Martin,
Quoted 19 lines · Samuel Martin
This patch generates all *.pyc files from *.py located in <target>/usr/python*/site-packages, before stripping the rootfs.
This prevents modules from packages that do not compile the *.py files from disappearing.
Reported-by Yegor Yefremov <[email protected]>
Cc Yegor Yefremov <[email protected]>
Signed-off-by Samuel Martin <[email protected]>
---
 Makefile | 5 +++++
 1 file changed, 5 insertions(+)
Makefile +0 -0
index af043a3..ec0b923 100644
@@ -564,6 +564,11 @@ $(TARGETS_ROOTFS): target-finalize
 

Quoted 11 lines

Makefile $(TARGETS_ROOTFS): target-finalize · line 571
Quoted 7 lines
Makefile +5 -0
 target-finalize: $(TARGETS)
      @$(call MESSAGE,"Finalizing target directory")
+ifneq ($(BR2_PACKAGE_PYTHON)$(BR2_PACKAGE_PYTHON3),)
+     PYTHONPATH="$(PYTHON_PATH)" \
+             $(HOST_DIR)/usr/bin/python -c "import compileall; \
+                     compileall.compile_dir('$(TARGET_DIR)/usr/lib/python$(PYTHON_VERSION_MAJOR)/site-packages')"
+endif
Why not, but then don't we want to do something in the Python package infra, and in the Python 2/3 packages themselves to avoid spending time biulding the .pyc files?

Indeed, it'd be better to avoid wasting time building *.pyc during per-package build, though it does not seem really easy to do at first glance.
Well, after hacking around, it is not that hard to do it! :-) I cleanup my hacks and will submit them.

Quoted 4 lines

Also, using TARGET_FINALIZE_HOOKS would be nicer. But there is the issue that the hook is the same between python and python3, so we wouldn't know where to put it.

The hook is simple enough to be moved in pkg-python.mk as a helper, then just being added to the PYTHON{,3}_TARGET_FINALIZE_HOOKS.

However, if we decide to not go using a global TARGET_FINALIZE_HOOKS, but per-package target-install hook, the helper can be easily added to the packages that need it; the biggest remaining task being identifying and updating all packages not using the pkg-python infra but providing a python module.

Regards,

Signature · 1 line

Samuel

Thomas Petazzoni

2015-03-09 21:44 UTC #5 lore raw

PATCH 1/2 target-finalize: generate all *.pyc

Dear Samuel Martin,

Quoted 5 lines · Samuel Martin

Indeed, it'd be better to avoid wasting time building *.pyc during per-package build, though it does not seem really easy to do at first glance.
Well, after hacking around, it is not that hard to do it! :-) I cleanup my hacks and will submit them.

Ok, cool.

Quoted 2 lines

The hook is simple enough to be moved in pkg-python.mk as a helper, then just being added to the PYTHON{,3}_TARGET_FINALIZE_HOOKS.

Indeed.

Quoted 5 lines

However, if we decide to not go using a global TARGET_FINALIZE_HOOKS, but per-package target-install hook, the helper can be easily added to the packages that need it; the biggest remaining task being identifying and updating all packages not using the pkg-python infra but providing a python module.

Well, as it is currently implemented, the hook byte-compiles *all* .py files. So if you call this hook multiple times, you're going to byte-compile all .py files multiple times. Doesn't seem like the most efficient thing to do.

Thomas

Signature · 3 lines

Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

Arnout Vandecappelle

2015-03-09 21:42 UTC #6 lore raw

PATCH 1/2 target-finalize: generate all *.pyc

In reply to Thomas Petazzoni

Quoted 22 lines · Thomas Petazzoni

Dear Samuel Martin,
Quoted 19 lines · Samuel Martin
This patch generates all *.pyc files from *.py located in <target>/usr/python*/site-packages, before stripping the rootfs.
This prevents modules from packages that do not compile the *.py files from disappearing.
Reported-by Yegor Yefremov <[email protected]>
Cc Yegor Yefremov <[email protected]>
Signed-off-by Samuel Martin <[email protected]>
---
 Makefile | 5 +++++
 1 file changed, 5 insertions(+)
Makefile +0 -0
index af043a3..ec0b923 100644
@@ -564,6 +564,11 @@ $(TARGETS_ROOTFS): target-finalize
 

Quoted 3 lines

Makefile $(TARGETS_ROOTFS): target-finalize · line 567
Quoted 3 lines
Makefile +1 -0
 target-finalize: $(TARGETS)
 	@$(call MESSAGE,"Finalizing target directory")
+ifneq ($(BR2_PACKAGE_PYTHON)$(BR2_PACKAGE_PYTHON3),)

It should also be conditional on !BR2_PACKAGE_PYTHON_PY_ONLY

Quoted 3 lines

Makefile $(TARGETS_ROOTFS): target-finalize · line 570

Quoted 3 lines

Makefile +3 -0

+	PYTHONPATH="$(PYTHON_PATH)" \
+		$(HOST_DIR)/usr/bin/python -c "import compileall; \
+			compileall.compile_dir('$(TARGET_DIR)/usr/lib/python$(PYTHON_VERSION_MAJOR)/site-packages')"

I believe the canonical way to call compileall from the command line is using
the -m option:

	$(HOST_DIR)/usr/bin/python -m compileall \
		$(TARGET_DIR)/usr/lib/python$(PYTHON_VERSION_MAJOR)/site-packages

Quoted 5 lines

Makefile $(TARGETS_ROOTFS): target-finalize · line 571
Quoted 1 line
Makefile +1 -0
+endif
Why not, but then don't we want to do something in the Python package infra, and in the Python 2/3 packages themselves to avoid spending time biulding the .pyc files?

compileall will check timestamps to avoid recompiling unnecessarily. So you
just have the tree walk to consider, which I think is a minor issue.

Quoted 3 lines

Also, using TARGET_FINALIZE_HOOKS would be nicer. But there is the issue that the hook is the same between python and python3, so we wouldn't know where to put it.

In fact, we already have a PYTHON_FINALIZE_TARGET (and similar for PYTHON3)
that does the removal of the .py files, so the compileall should be added there. And that immediately handles the BR2_PACKAGE_PYTHON_PY_ONLY condition as well.

Regards,
Arnout

Signature · 6 lines

Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F

Thomas Petazzoni

2015-03-09 21:46 UTC #7 lore raw

PATCH 1/2 target-finalize: generate all *.pyc

Dear Arnout Vandecappelle,

Quoted 6 lines · Arnout Vandecappelle

Quoted 3 lines
Why not, but then don't we want to do something in the Python package infra, and in the Python 2/3 packages themselves to avoid spending time biulding the .pyc files?
compileall will check timestamps to avoid recompiling unnecessarily. So you
just have the tree walk to consider, which I think is a minor issue.

Ah, good to know. Then indeed it's useless to do any special trick:
let's just call compileall as a target finalize hook.

Thomas

Signature · 3 lines

Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

Yegor Yefremov

2015-08-05 06:26 UTC #8 lore raw

PATCH 1/2 target-finalize: generate all *.pyc

Quoted 8 lines · Thomas Petazzoni

Dear Arnout Vandecappelle,
Quoted 4 lines · Arnout Vandecappelle
Quoted 3 lines
Why not, but then don't we want to do something in the Python package infra, and in the Python 2/3 packages themselves to avoid spending time biulding the .pyc files?

Quoted 5 lines

Quoted 2 lines
compileall will check timestamps to avoid recompiling unnecessarily. So you
just have the tree walk to consider, which I think is a minor issue.
Ah, good to know. Then indeed it's useless to do any special trick:
let's just call compileall as a target finalize hook.

What's the current state of this issue? It would be great, if this could be fixed for the upcoming release.

Yegor