Skip to contentSkip to conversation

Python improvements: PEP 3147 fix and misc updates

PATCH New Thomas Petazzoni 2 participants 11 messages Sep 30, 2014 – Oct 5, 2014 Kbuild

lore
Share & tools
lore
#0 lore raw

cover Python improvements: PEP 3147 fix and misc updates

Hello,

The main purpose of this patch series is to fix the PEP 3147 handling with Python 3.

For those not familiar with PEP 3147, here is a quick explanation. In Python 2.x, before PEP 3147, the .py and corresponding byte-compiled .pyc were stored in the same directory. With PEP 3147, the .py and .pyc are stored in a separate location: the .py in the original place, and the .pyc in a __pycache__ subdirectory. In addition, the .pyc in __pycache__ will not be considered by Python if the corresponding .py is not available. This is a problem for Buildroot, as we try to save space, and by default remove all .py files and keep only .pyc files.

For the Python standard library itself, this issue is already handled by python3-004-old-stdlib-cache.patch. But for external modules, the issue is not handled, and basically, external modules are not found if only the .pyc file is on the target.

The first commit of this patch series is solving this problem by introducing an additional patch adjusting the distutils logic in Python 3 to install the .pyc file in their old, non-PEP3147 location. This solves both distutils and setuptools packages, because setuptools relies on distutils internally.

Note that this solution has my preference over the solution proposed by Philippe Proux in
http://patchwork.ozlabs.org/patch/373082/. Philippe's solution consist in fixing the problem entirely in target-finalize. But this means that after the package is installed, the .pyc files do not exist in their real location, and they are created only in target-finalize. This would prevent any per-package installed size analysis, as I've proposed through a few patches some time ago.

The other patches require less explanations:

- The second patch cleans up some useless __pycache__ directories that are created because of an incomplete isolution between the target build and the host build of Python.

- The third, fourth and fifth patches make version bumps/improvements in various Python packages.

Philippe, Christophe, if you could test this patch series, and see if it fixes the Python 3 problems you had with external modules, it would be great!

Thanks,

Thomas

Thomas Petazzoni (6):
python3: add distutils fix for PEP 3147 issue
python3: remove unneeded __pycache__ directories
python-pyparsing: bump to version 2.0.2, enable on Python 3
python-setuptools: bump to version 5.8
python-pyasn: requires the zlib Python module
python, python3: convert py/pyc removal to TARGET_FINALIZE_HOOKS

 Makefile                                           |  6 ---
 package/python-pyasn/Config.in                     |  2 +
 package/python-pyparsing/Config.in                 |  2 +-
 package/python-pyparsing/python-pyparsing.mk       |  2 +-
 .../python-setuptools-0001-add-executable.patch    | 55 ++++++++++++++++++++++
 .../python-setuptools-add-executable.patch         | 50 --------------------
 package/python-setuptools/python-setuptools.mk     |  6 +--
 package/python/python.mk                           | 14 ++++++
 .../python3/python3-016-distutils-no-pep3147.patch | 36 ++++++++++++++
 package/python3/python3.mk                         | 18 +++++++
 10 files changed, 130 insertions(+), 61 deletions(-)
 create mode 100644 package/python-setuptools/python-setuptools-0001-add-executable.patch
 delete mode 100644 package/python-setuptools/python-setuptools-add-executable.patch
 create mode 100644 package/python3/python3-016-distutils-no-pep3147.patch
Signature · 1 line
2.0.0
#1 lore raw

PATCH 1/6 python3: add distutils fix for PEP 3147 issue

Python 3 has a new standard for installing .pyc file, called PEP 3147. Unfortunately, this standard requires both the .py and .pyc files to be installed for a Python module to be found. This is quite annoying on space-constrained embedded systems, since the .py file is technically not required for execution.

For the Python standard library, our Python 3 package already contains a patch named python3-004-old-stdlib-cache.patch, which allows to disable the PEP 3147 installation.

But that lives the distutils/setuptools package an unsolved problem. This patch therefore adds a new patch to Python, which makes distutils package use the traditional installation path when byte compiling, rather than the PEP 3147 installation path. Since setuptools relies on distutils internally, it also fixes setuptools based packages.

---
 .../python3/python3-016-distutils-no-pep3147.patch | 36 ++++++++++++++++++++++
 1 file changed, 36 insertions(+)
 create mode 100644 package/python3/python3-016-distutils-no-pep3147.patch
package/python3/python3-016-distutils-no-pep3147.patch new file +36 -0
new file mode 100644
index 0000000..0fdfeb2
@@ -0,0 +1,36 @@
+Add distutils fix for PEP 3147 issue
+
+Python 3 has a new standard for installing .pyc file, called PEP
+3147. Unfortunately, this standard requires both the .py and .pyc
+files to be installed for a Python module to be found. This is quite
+annoying on space-constrained embedded systems, since the .py file is
+technically not required for execution.
+
+For the Python standard library, our Python 3 package already contains
+a patch named python3-004-old-stdlib-cache.patch, which allows to
+disable the PEP 3147 installation.
+
+But that lives the distutils/setuptools package an unsolved
+problem. This patch therefore adds a new patch to Python, which makes
+distutils package use the traditional installation path when byte
+compiling, rather than the PEP 3147 installation path. Since
+setuptools relies on distutils internally, it also fixes setuptools
+based packages.
+
+Signed-off-by: Thomas Petazzoni <[email protected]>
+
+Index: b/Lib/distutils/util.py
+===================================================================
+--- a/Lib/distutils/util.py
++++ b/Lib/distutils/util.py
+@@ -437,7 +437,9 @@
+             # Terminology from the py_compile module:
+             #   cfile - byte-compiled file
+             #   dfile - purported source filename (same as 'file' by default)
+-            if optimize >= 0:
++            if "_python_sysroot" in os.environ:
++                cfile = file + (__debug__ and "c" or "o")
++            elif optimize >= 0:
+                 cfile = importlib.util.cache_from_source(
+                     file, debug_override=not optimize)
+             else:
Signature · 1 line
2.0.0
#2 lore raw

PATCH 1/6 python3: add distutils fix for PEP 3147 issue

Hi Thomas,

Quoted 32 lines · Thomas Petazzoni

Python 3 has a new standard for installing .pyc file, called PEP 3147. Unfortunately, this standard requires both the .py and .pyc files to be installed for a Python module to be found. This is quite annoying on space-constrained embedded systems, since the .py file is technically not required for execution.

For the Python standard library, our Python 3 package already contains a patch named python3-004-old-stdlib-cache.patch, which allows to disable the PEP 3147 installation.

But that lives the distutils/setuptools package an unsolved problem. This patch therefore adds a new patch to Python, which makes distutils package use the traditional installation path when byte compiling, rather than the PEP 3147 installation path. Since setuptools relies on distutils internally, it also fixes setuptools based packages.

---
 .../python3/python3-016-distutils-no-pep3147.patch | 36 ++++++++++++++++++++++
 1 file changed, 36 insertions(+)
 create mode 100644 package/python3/python3-016-distutils-no-pep3147.patch
package/python3/python3-016-distutils-no-pep3147.patch new file +3 -0
new file mode 100644
index 0000000..0fdfeb2
@@ -0,0 +1,36 @@
+Add distutils fix for PEP 3147 issue
+
+Python 3 has a new standard for installing .pyc file, called PEP

package/python3/python3-016-distutils-no-pep3147.patch line 13

Quoted 10 lines
package/python3/python3-016-distutils-no-pep3147.patch new file +10 -0
+3147. Unfortunately, this standard requires both the .py and .pyc
+files to be installed for a Python module to be found. This is quite
+annoying on space-constrained embedded systems, since the .py file is
+technically not required for execution.
+
+For the Python standard library, our Python 3 package already contains
+a patch named python3-004-old-stdlib-cache.patch, which allows to
+disable the PEP 3147 installation.
+
+But that lives the distutils/setuptools package an unsolved
              ^
            leaves?

Otherwise, looks good to me.

Quoted 23 lines
package/python3/python3-016-distutils-no-pep3147.patch new file +23 -0
+problem. This patch therefore adds a new patch to Python, which makes
+distutils package use the traditional installation path when byte
+compiling, rather than the PEP 3147 installation path. Since
+setuptools relies on distutils internally, it also fixes setuptools
+based packages.
+
+Signed-off-by: Thomas Petazzoni <[email protected]>
+
+Index: b/Lib/distutils/util.py
+===================================================================
+--- a/Lib/distutils/util.py
++++ b/Lib/distutils/util.py
+@@ -437,7 +437,9 @@
+             # Terminology from the py_compile module:
+             #   cfile - byte-compiled file
+             #   dfile - purported source filename (same as 'file' by default)
+-            if optimize >= 0:
++            if "_python_sysroot" in os.environ:
++                cfile = file + (__debug__ and "c" or "o")
++            elif optimize >= 0:
+                 cfile = importlib.util.cache_from_source(
+                     file, debug_override=not optimize)
+             else:
Signature · 1 line
Christophe Vu-Brugier
#3 lore raw

PATCH 1/6 python3: add distutils fix for PEP 3147 issue

Dear Christophe Vu-Brugier,

Quoted 11 lines · Christophe Vu-Brugier

package/python3/python3-016-distutils-no-pep3147.patch line 13

Quoted 5 lines
package/python3/python3-016-distutils-no-pep3147.patch new file +5 -0
+For the Python standard library, our Python 3 package already contains
+a patch named python3-004-old-stdlib-cache.patch, which allows to
+disable the PEP 3147 installation.
+
+But that lives the distutils/setuptools package an unsolved
              ^
            leaves?

Otherwise, looks good to me.

Thanks for the review. Unless there are other comments, I'll leave to the person who will commit the patch to make the change to the commit log, since it's just a typo.

Thanks a lot!

Thomas

Signature · 3 lines
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com
#4 lore raw

PATCH 2/6 python3: remove unneeded __pycache__ directories

In reply to Thomas Petazzoni

The __pycache__ directories are only needed on PEP3147-enabled installations, which is not the choice Buildroot has made. However, during the build process of the target Python, the separation between the target and host Python is not entirely complete, and starting the host Python to byte-compile the target Python modules triggers the compilation of a bunch of __pycache__ directories by the host Python, which uses a PEP 3147 installation.

For now, simply get rid of those useless directories after the installation of Python is completed. In the long term, it would be good to achieve a better isolation between the host and target build to avoid this issue.

---
 package/python3/python3.mk | 4 ++++
 1 file changed, 4 insertions(+)
package/python3/python3.mk +4 -0
index 4941316..72c3bb1 100644
@@ -142,6 +142,10 @@ define PYTHON3_REMOVE_USELESS_FILES
 		-type f -not -name pyconfig.h -a -not -name Makefile` ; do \
 		rm -f $$i ; \
 	done
+	rm -rf $(TARGET_DIR)/usr/lib/python$(PYTHON3_VERSION_MAJOR)/__pycache__/
+	rm -rf $(TARGET_DIR)/usr/lib/python$(PYTHON3_VERSION_MAJOR)/lib-dynload/sysconfigdata/__pycache__
+	rm -rf $(TARGET_DIR)/usr/lib/python$(PYTHON3_VERSION_MAJOR)/collections/__pycache__
+	rm -rf $(TARGET_DIR)/usr/lib/python$(PYTHON3_VERSION_MAJOR)/importlib/__pycache__
 endef
 
 PYTHON3_POST_INSTALL_TARGET_HOOKS += PYTHON3_REMOVE_USELESS_FILES
Signature · 1 line
2.0.0
#5 lore raw

PATCH 3/6 python-pyparsing: bump to version 2.0.2, enable on Python 3

In reply to Thomas Petazzoni

This commit bumps the pyparsing package to version 2.0.2, and therefore enables it usage with Python 3.

---
 package/python-pyparsing/Config.in           | 2 +-
 package/python-pyparsing/python-pyparsing.mk | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)
package/python-pyparsing/Config.in +1 -1
index 49a474b..0408e5e 100644
@@ -1,6 +1,6 @@
 config BR2_PACKAGE_PYTHON_PYPARSING
 	bool "python-pyparsing"
-	depends on BR2_PACKAGE_PYTHON
+	depends on BR2_PACKAGE_PYTHON || BR2_PACKAGE_PYTHON3
 	help
 	  The pyparsing module is an alternative approach to creating and
 	  executing simple grammars, vs. the traditional lex/yacc approach,
package/python-pyparsing/python-pyparsing.mk +1 -1
index 07dda58..82a907b 100644
@@ -4,7 +4,7 @@
 #
 ################################################################################
 
-PYTHON_PYPARSING_VERSION         = 1.5.6
+PYTHON_PYPARSING_VERSION         = 2.0.2
 PYTHON_PYPARSING_SOURCE          = pyparsing-$(PYTHON_PYPARSING_VERSION).tar.gz
 PYTHON_PYPARSING_SITE            = http://downloads.sourceforge.net/project/pyparsing/pyparsing/pyparsing-$(PYTHON_PYPARSING_VERSION)
 PYTHON_PYPARSING_LICENSE         = MIT
Signature · 1 line
2.0.0
#6 lore raw

PATCH 3/6 python-pyparsing: bump to version 2.0.2, enable on Python 3

Hi Thomas,

Quoted 2 lines · Thomas Petazzoni

This commit bumps the pyparsing package to version 2.0.2, and therefore enables it usage with Python 3.

Tested with Python 3 on an x86_64 target:

/ # python
Python 3.4.1 (default, Oct 1 2014, 16:25:09)
[GCC 4.7.3] on linux
Type "help", "copyright", "credits" or "license" for more information.

Quoted 1 line
Quoted 1 line
Quoted 1 line

import pyparsing

Quoted 31 lines
---
 package/python-pyparsing/Config.in           | 2 +-
 package/python-pyparsing/python-pyparsing.mk | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)
package/python-pyparsing/Config.in +1 -1
index 49a474b..0408e5e 100644
@@ -1,6 +1,6 @@
 config BR2_PACKAGE_PYTHON_PYPARSING
 	bool "python-pyparsing"
-	depends on BR2_PACKAGE_PYTHON
+	depends on BR2_PACKAGE_PYTHON || BR2_PACKAGE_PYTHON3
 	help
 	  The pyparsing module is an alternative approach to creating and
 	  executing simple grammars, vs. the traditional lex/yacc approach,
package/python-pyparsing/python-pyparsing.mk +1 -1
index 07dda58..82a907b 100644
@@ -4,7 +4,7 @@
 #
 ################################################################################
 
-PYTHON_PYPARSING_VERSION         = 1.5.6
+PYTHON_PYPARSING_VERSION         = 2.0.2
 PYTHON_PYPARSING_SOURCE          = pyparsing-$(PYTHON_PYPARSING_VERSION).tar.gz
 PYTHON_PYPARSING_SITE            = http://downloads.sourceforge.net/project/pyparsing/pyparsing/pyparsing-$(PYTHON_PYPARSING_VERSION)
 PYTHON_PYPARSING_LICENSE         = MIT
Signature · 1 line
Christophe Vu-Brugier
#7 lore raw

PATCH 4/6 python-setuptools: bump to version 5.8

In reply to Thomas Petazzoni

A refresh of the existing patch is needed, and the patch is also renamed to use the correct patch file naming convention.

---
 .../python-setuptools-0001-add-executable.patch    | 55 ++++++++++++++++++++++
 .../python-setuptools-add-executable.patch         | 50 --------------------
 package/python-setuptools/python-setuptools.mk     |  6 +--
 3 files changed, 58 insertions(+), 53 deletions(-)
 create mode 100644 package/python-setuptools/python-setuptools-0001-add-executable.patch
 delete mode 100644 package/python-setuptools/python-setuptools-add-executable.patch
package/python-setuptools/python-setuptools-0001-add-executable.patch new file +55 -0
new file mode 100644
index 0000000..502d099
@@ -0,0 +1,55 @@
+Add a new --executable option to distribute so that we can
+force the shebang line in installed python scripts.
+
+[Thomas: refresh for setuptools 5.8.]
+
+Signed-off-by: Gustavo Zacarias <[email protected]>
+Signed-off-by: Thomas Petazzoni <[email protected]>
+
+Index: b/setuptools/command/install.py
+===================================================================
+--- a/setuptools/command/install.py
++++ b/setuptools/command/install.py
+@@ -16,6 +16,7 @@
+     """Use easy_install to install the package, w/dependencies"""
+ 
+     user_options = orig.install.user_options + [
++        ('executable=', 'e', "specify final destination interpreter path"),
+         ('old-and-unmanageable', None, "Try not to use this!"),
+         ('single-version-externally-managed', None,
+          "used by system package builders to create 'flat' eggs"),
+@@ -31,6 +32,7 @@
+ 
+     def initialize_options(self):
+         orig.install.initialize_options(self)
++        self.executable = None
+         self.old_and_unmanageable = None
+         self.single_version_externally_managed = None
+ 
+Index: b/setuptools/command/install_scripts.py
+===================================================================
+--- a/setuptools/command/install_scripts.py
++++ b/setuptools/command/install_scripts.py
+@@ -11,6 +11,13 @@
+     def initialize_options(self):
+         orig.install_scripts.initialize_options(self)
+         self.no_ep = False
++        self.executable = None
++
++    def finalize_options(self):
++        orig.install_scripts.finalize_options(self)
++        self.set_undefined_options('install',
++                ('executable','executable')
++        )
+ 
+     def run(self):
+         from setuptools.command.easy_install import get_script_args
+@@ -32,6 +39,8 @@
+         )
+         bs_cmd = self.get_finalized_command('build_scripts')
+         executable = getattr(bs_cmd, 'executable', sys_executable)
++        if self.executable is not None:
++            executable = self.executable
+         is_wininst = getattr(
+             self.get_finalized_command("bdist_wininst"), '_is_running', False
+         )
package/python-setuptools/python-setuptools-add-executable.patch deleted +0 -50

50 lines not on this page · raw message

package/python-setuptools/python-setuptools.mk +3 -3
index 54a11c4..fa90ed4 100644
@@ -4,9 +4,9 @@
 #
 ################################################################################
 
-PYTHON_SETUPTOOLS_VERSION = 2.1.2
-PYTHON_SETUPTOOLS_SOURCE  = setuptools-$(PYTHON_SETUPTOOLS_VERSION).tar.gz
-PYTHON_SETUPTOOLS_SITE    = http://pypi.python.org/packages/source/s/setuptools
+PYTHON_SETUPTOOLS_VERSION = 5.8
+PYTHON_SETUPTOOLS_SOURCE = setuptools-$(PYTHON_SETUPTOOLS_VERSION).tar.gz
+PYTHON_SETUPTOOLS_SITE = http://pypi.python.org/packages/source/s/setuptools
 PYTHON_SETUPTOOLS_LICENSE = Python Software Foundation or Zope Public License
 PYTHON_SETUPTOOLS_LICENSE_FILES = PKG-INFO
 PYTHON_SETUPTOOLS_SETUP_TYPE = setuptools
Signature · 1 line
2.0.0
#8 lore raw

PATCH 5/6 python-pyasn: requires the zlib Python module

In reply to Thomas Petazzoni

The python-pyasn module requires the zlib Python module (part of the Python standard installation, but optional in Buildroot), otherwise it fails to load at runtime.

---
 package/python-pyasn/Config.in | 2 ++
 1 file changed, 2 insertions(+)
package/python-pyasn/Config.in +2 -0
index eca53db..efdbbc5 100644
@@ -2,6 +2,8 @@ config BR2_PACKAGE_PYTHON_PYASN
 	bool "python-pyasn"
 	depends on BR2_PACKAGE_PYTHON || BR2_PACKAGE_PYTHON3
 	depends on BR2_INSTALL_LIBSTDCPP
+	select BR2_PACKAGE_PYTHON3_ZLIB if BR2_PACKAGE_PYTHON3
+	select BR2_PACKAGE_PYTHON_ZLIB if BR2_PACKAGE_PYTHON
 	help
 	  PyASN is a Python extension module that enables you to
 	  perform very fast IP address to Autonomous System Number
Signature · 1 line
2.0.0
#9 lore raw

PATCH 6/6 python, python3: convert py/pyc removal to TARGET_FINALIZE_HOOKS

In reply to Thomas Petazzoni

Since the removal of py/pyc files is Python-specific, this commit moves the logic removing those files to python.mk and python3.mk respectively.

---
 Makefile                   |  6 ------
 package/python/python.mk   | 14 ++++++++++++++
 package/python3/python3.mk | 14 ++++++++++++++
 3 files changed, 28 insertions(+), 6 deletions(-)
Makefile +0 -6
index 53f8ad5..6068e7f 100644
@@ -573,12 +573,6 @@ endif
 	rm -rf $(TARGET_DIR)/usr/doc $(TARGET_DIR)/usr/share/doc
 	rm -rf $(TARGET_DIR)/usr/share/gtk-doc
 	-rmdir $(TARGET_DIR)/usr/share 2>/dev/null
-ifeq ($(BR2_PACKAGE_PYTHON_PY_ONLY)$(BR2_PACKAGE_PYTHON3_PY_ONLY),y)
-	find $(TARGET_DIR)/usr/lib/ -name '*.pyc' -print0 | xargs -0 rm -f
-endif
-ifeq ($(BR2_PACKAGE_PYTHON_PYC_ONLY)$(BR2_PACKAGE_PYTHON3_PYC_ONLY),y)
-	find $(TARGET_DIR)/usr/lib/ -name '*.py' -print0 | xargs -0 rm -f
-endif
 	$(STRIP_FIND_CMD) | xargs $(STRIPCMD) 2>/dev/null || true
 	if test -d $(TARGET_DIR)/lib/modules; then \
 		find $(TARGET_DIR)/lib/modules -type f -name '*.ko' | \
package/python/python.mk +14 -0
index b237848..4558034 100644
@@ -208,3 +208,17 @@ PYTHON_PATH = $(TARGET_DIR)/usr/lib/python$(PYTHON_VERSION_MAJOR)/sysconfigdata/
 
 $(eval $(autotools-package))
 $(eval $(host-autotools-package))
+
+ifeq ($(BR2_PACKAGE_PYTHON_PYC_ONLY),y)
+define PYTHON_FINALIZE_TARGET
+	find $(TARGET_DIR)/usr/lib/python$(PYTHON_VERSION_MAJOR) -name '*.py' -print0 | xargs -0 rm -f
+endef
+endif
+
+ifeq ($(BR2_PACKAGE_PYTHON_PY_ONLY),y)
+define PYTHON_FINALIZE_TARGET
+	find $(TARGET_DIR)/usr/lib/python$(PYTHON_VERSION_MAJOR) -name '*.pyc' -print0 | xargs -0 rm -f
+endef
+endif
+
+TARGET_FINALIZE_HOOKS += PYTHON_FINALIZE_TARGET
package/python3/python3.mk +14 -0
index 72c3bb1..6161df2 100644
@@ -187,3 +187,17 @@ PYTHON3_PATH = $(TARGET_DIR)/usr/lib/python$(PYTHON3_VERSION_MAJOR)/sysconfigdat
 
 $(eval $(autotools-package))
 $(eval $(host-autotools-package))
+
+ifeq ($(BR2_PACKAGE_PYTHON3_PYC_ONLY),y)
+define PYTHON3_FINALIZE_TARGET
+	find $(TARGET_DIR)/usr/lib/python$(PYTHON3_VERSION_MAJOR) -name '*.py' -print0 | xargs -0 rm -f
+endef
+endif
+
+ifeq ($(BR2_PACKAGE_PYTHON3_PY_ONLY),y)
+define PYTHON3_FINALIZE_TARGET
+	find $(TARGET_DIR)/usr/lib/python$(PYTHON3_VERSION_MAJOR) -name '*.pyc' -print0 | xargs -0 rm -f
+endef
+endif
+
+TARGET_FINALIZE_HOOKS += PYTHON3_FINALIZE_TARGET
Signature · 1 line
2.0.0
#10 lore raw

cover Python improvements: PEP 3147 fix and misc updates

In reply to Thomas Petazzoni

Hello,

Quoted 7 lines · Thomas Petazzoni

Thomas Petazzoni (6):
python3: add distutils fix for PEP 3147 issue
python3: remove unneeded __pycache__ directories
python-pyparsing: bump to version 2.0.2, enable on Python 3
python-setuptools: bump to version 5.8
python-pyasn: requires the zlib Python module
python, python3: convert py/pyc removal to TARGET_FINALIZE_HOOKS

I've applied this patch series.

Thomas

Signature · 3 lines
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com