Python improvements: PEP 3147 fix and misc updates
cover Python improvements: PEP 3147 fix and misc updates
Hello,
The main purpose of this patch series is to fix the PEP 3147 handling with Python 3.
For those not familiar with PEP 3147, here is a quick explanation. In Python 2.x, before PEP 3147, the .py and corresponding byte-compiled .pyc were stored in the same directory. With PEP 3147, the .py and .pyc are stored in a separate location: the .py in the original place, and the .pyc in a __pycache__ subdirectory. In addition, the .pyc in __pycache__ will not be considered by Python if the corresponding .py is not available. This is a problem for Buildroot, as we try to save space, and by default remove all .py files and keep only .pyc files.
For the Python standard library itself, this issue is already handled by python3-004-old-stdlib-cache.patch. But for external modules, the issue is not handled, and basically, external modules are not found if only the .pyc file is on the target.
The first commit of this patch series is solving this problem by introducing an additional patch adjusting the distutils logic in Python 3 to install the .pyc file in their old, non-PEP3147 location. This solves both distutils and setuptools packages, because setuptools relies on distutils internally.
Note that this solution has my preference over the solution proposed by Philippe Proux in
http://patchwork.ozlabs.org/patch/373082/. Philippe's solution consist in fixing the problem entirely in target-finalize. But this means that after the package is installed, the .pyc files do not exist in their real location, and they are created only in target-finalize. This would prevent any per-package installed size analysis, as I've proposed through a few patches some time ago.
The other patches require less explanations:
- The second patch cleans up some useless __pycache__ directories that are created because of an incomplete isolution between the target build and the host build of Python.
- The third, fourth and fifth patches make version bumps/improvements in various Python packages.
Philippe, Christophe, if you could test this patch series, and see if it fixes the Python 3 problems you had with external modules, it would be great!
Thanks,
Thomas
Thomas Petazzoni (6):
python3: add distutils fix for PEP 3147 issue
python3: remove unneeded __pycache__ directories
python-pyparsing: bump to version 2.0.2, enable on Python 3
python-setuptools: bump to version 5.8
python-pyasn: requires the zlib Python module
python, python3: convert py/pyc removal to TARGET_FINALIZE_HOOKS
Makefile | 6 --- package/python-pyasn/Config.in | 2 + package/python-pyparsing/Config.in | 2 +- package/python-pyparsing/python-pyparsing.mk | 2 +- .../python-setuptools-0001-add-executable.patch | 55 ++++++++++++++++++++++ .../python-setuptools-add-executable.patch | 50 -------------------- package/python-setuptools/python-setuptools.mk | 6 +-- package/python/python.mk | 14 ++++++ .../python3/python3-016-distutils-no-pep3147.patch | 36 ++++++++++++++ package/python3/python3.mk | 18 +++++++ 10 files changed, 130 insertions(+), 61 deletions(-) create mode 100644 package/python-setuptools/python-setuptools-0001-add-executable.patch delete mode 100644 package/python-setuptools/python-setuptools-add-executable.patch create mode 100644 package/python3/python3-016-distutils-no-pep3147.patch
Signature · 1 line
2.0.0
PATCH 1/6 python3: add distutils fix for PEP 3147 issue
Python 3 has a new standard for installing .pyc file, called PEP 3147. Unfortunately, this standard requires both the .py and .pyc files to be installed for a Python module to be found. This is quite annoying on space-constrained embedded systems, since the .py file is technically not required for execution.
For the Python standard library, our Python 3 package already contains a patch named python3-004-old-stdlib-cache.patch, which allows to disable the PEP 3147 installation.
But that lives the distutils/setuptools package an unsolved problem. This patch therefore adds a new patch to Python, which makes distutils package use the traditional installation path when byte compiling, rather than the PEP 3147 installation path. Since setuptools relies on distutils internally, it also fixes setuptools based packages.
--- .../python3/python3-016-distutils-no-pep3147.patch | 36 ++++++++++++++++++++++ 1 file changed, 36 insertions(+) create mode 100644 package/python3/python3-016-distutils-no-pep3147.patch
package/python3/python3-016-distutils-no-pep3147.patch new file +36 -0
@@ -0,0 +1,36 @@ +Add distutils fix for PEP 3147 issue + +Python 3 has a new standard for installing .pyc file, called PEP +3147. Unfortunately, this standard requires both the .py and .pyc +files to be installed for a Python module to be found. This is quite +annoying on space-constrained embedded systems, since the .py file is +technically not required for execution. + +For the Python standard library, our Python 3 package already contains +a patch named python3-004-old-stdlib-cache.patch, which allows to +disable the PEP 3147 installation. + +But that lives the distutils/setuptools package an unsolved +problem. This patch therefore adds a new patch to Python, which makes +distutils package use the traditional installation path when byte +compiling, rather than the PEP 3147 installation path. Since +setuptools relies on distutils internally, it also fixes setuptools +based packages. + +Signed-off-by: Thomas Petazzoni <[email protected]> + +Index: b/Lib/distutils/util.py +=================================================================== +--- a/Lib/distutils/util.py ++++ b/Lib/distutils/util.py +@@ -437,7 +437,9 @@ + # Terminology from the py_compile module: + # cfile - byte-compiled file + # dfile - purported source filename (same as 'file' by default) +- if optimize >= 0: ++ if "_python_sysroot" in os.environ: ++ cfile = file + (__debug__ and "c" or "o") ++ elif optimize >= 0: + cfile = importlib.util.cache_from_source( + file, debug_override=not optimize) + else:
Signature · 1 line
2.0.0
PATCH 1/6 python3: add distutils fix for PEP 3147 issue
Hi Thomas,
Quoted 32 lines · Thomas Petazzoni
Python 3 has a new standard for installing .pyc file, called PEP 3147. Unfortunately, this standard requires both the .py and .pyc files to be installed for a Python module to be found. This is quite annoying on space-constrained embedded systems, since the .py file is technically not required for execution.
For the Python standard library, our Python 3 package already contains a patch named python3-004-old-stdlib-cache.patch, which allows to disable the PEP 3147 installation.
But that lives the distutils/setuptools package an unsolved problem. This patch therefore adds a new patch to Python, which makes distutils package use the traditional installation path when byte compiling, rather than the PEP 3147 installation path. Since setuptools relies on distutils internally, it also fixes setuptools based packages.
Signed-off-by Thomas Petazzoni <[email protected]>--- .../python3/python3-016-distutils-no-pep3147.patch | 36 ++++++++++++++++++++++ 1 file changed, 36 insertions(+) create mode 100644 package/python3/python3-016-distutils-no-pep3147.patchpackage/python3/python3-016-distutils-no-pep3147.patch new file +3
-0@@ -0,0 +1,36 @@ +Add distutils fix for PEP 3147 issue + +Python 3 has a new standard for installing .pyc file, called PEP
package/python3/python3-016-distutils-no-pep3147.patch line 13
Quoted 10 lines
package/python3/python3-016-distutils-no-pep3147.patch new file +10
-0+3147. Unfortunately, this standard requires both the .py and .pyc +files to be installed for a Python module to be found. This is quite +annoying on space-constrained embedded systems, since the .py file is +technically not required for execution. + +For the Python standard library, our Python 3 package already contains +a patch named python3-004-old-stdlib-cache.patch, which allows to +disable the PEP 3147 installation. + +But that lives the distutils/setuptools package an unsolved
^
leaves?Otherwise, looks good to me.
Quoted 23 lines
package/python3/python3-016-distutils-no-pep3147.patch new file +23
-0+problem. This patch therefore adds a new patch to Python, which makes +distutils package use the traditional installation path when byte +compiling, rather than the PEP 3147 installation path. Since +setuptools relies on distutils internally, it also fixes setuptools +based packages. + +Signed-off-by: Thomas Petazzoni <[email protected]> + +Index: b/Lib/distutils/util.py +=================================================================== +--- a/Lib/distutils/util.py ++++ b/Lib/distutils/util.py +@@ -437,7 +437,9 @@ + # Terminology from the py_compile module: + # cfile - byte-compiled file + # dfile - purported source filename (same as 'file' by default) +- if optimize >= 0: ++ if "_python_sysroot" in os.environ: ++ cfile = file + (__debug__ and "c" or "o") ++ elif optimize >= 0: + cfile = importlib.util.cache_from_source( + file, debug_override=not optimize) + else:
Signature · 1 line
Christophe Vu-Brugier
PATCH 1/6 python3: add distutils fix for PEP 3147 issue
Dear Christophe Vu-Brugier,
Quoted 11 lines · Christophe Vu-Brugier
package/python3/python3-016-distutils-no-pep3147.patch line 13
Quoted 5 lines
package/python3/python3-016-distutils-no-pep3147.patch new file +5
-0+For the Python standard library, our Python 3 package already contains +a patch named python3-004-old-stdlib-cache.patch, which allows to +disable the PEP 3147 installation. + +But that lives the distutils/setuptools package an unsolved^ leaves?Otherwise, looks good to me.
Reviewed-by Christophe Vu-Brugier <[email protected]>
Thanks for the review. Unless there are other comments, I'll leave to the person who will commit the patch to make the change to the commit log, since it's just a typo.
Thanks a lot!
Thomas
Signature · 3 lines
Thomas Petazzoni, CTO, Free Electrons Embedded Linux, Kernel and Android engineering http://free-electrons.com
PATCH 2/6 python3: remove unneeded __pycache__ directories
In reply to Thomas PetazzoniThe __pycache__ directories are only needed on PEP3147-enabled installations, which is not the choice Buildroot has made. However, during the build process of the target Python, the separation between the target and host Python is not entirely complete, and starting the host Python to byte-compile the target Python modules triggers the compilation of a bunch of __pycache__ directories by the host Python, which uses a PEP 3147 installation.
For now, simply get rid of those useless directories after the installation of Python is completed. In the long term, it would be good to achieve a better isolation between the host and target build to avoid this issue.
--- package/python3/python3.mk | 4 ++++ 1 file changed, 4 insertions(+)
package/python3/python3.mk +4 -0
@@ -142,6 +142,10 @@ define PYTHON3_REMOVE_USELESS_FILES -type f -not -name pyconfig.h -a -not -name Makefile` ; do \ rm -f $$i ; \ done + rm -rf $(TARGET_DIR)/usr/lib/python$(PYTHON3_VERSION_MAJOR)/__pycache__/ + rm -rf $(TARGET_DIR)/usr/lib/python$(PYTHON3_VERSION_MAJOR)/lib-dynload/sysconfigdata/__pycache__ + rm -rf $(TARGET_DIR)/usr/lib/python$(PYTHON3_VERSION_MAJOR)/collections/__pycache__ + rm -rf $(TARGET_DIR)/usr/lib/python$(PYTHON3_VERSION_MAJOR)/importlib/__pycache__ endef PYTHON3_POST_INSTALL_TARGET_HOOKS += PYTHON3_REMOVE_USELESS_FILES
Signature · 1 line
2.0.0
PATCH 3/6 python-pyparsing: bump to version 2.0.2, enable on Python 3
In reply to Thomas PetazzoniThis commit bumps the pyparsing package to version 2.0.2, and therefore enables it usage with Python 3.
--- package/python-pyparsing/Config.in | 2 +- package/python-pyparsing/python-pyparsing.mk | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-)
package/python-pyparsing/Config.in +1 -1
@@ -1,6 +1,6 @@ config BR2_PACKAGE_PYTHON_PYPARSING bool "python-pyparsing" - depends on BR2_PACKAGE_PYTHON + depends on BR2_PACKAGE_PYTHON || BR2_PACKAGE_PYTHON3 help The pyparsing module is an alternative approach to creating and executing simple grammars, vs. the traditional lex/yacc approach,
package/python-pyparsing/python-pyparsing.mk +1 -1
@@ -4,7 +4,7 @@ # ################################################################################ -PYTHON_PYPARSING_VERSION = 1.5.6 +PYTHON_PYPARSING_VERSION = 2.0.2 PYTHON_PYPARSING_SOURCE = pyparsing-$(PYTHON_PYPARSING_VERSION).tar.gz PYTHON_PYPARSING_SITE = http://downloads.sourceforge.net/project/pyparsing/pyparsing/pyparsing-$(PYTHON_PYPARSING_VERSION) PYTHON_PYPARSING_LICENSE = MIT
Signature · 1 line
2.0.0
PATCH 3/6 python-pyparsing: bump to version 2.0.2, enable on Python 3
Hi Thomas,
Quoted 2 lines · Thomas Petazzoni
This commit bumps the pyparsing package to version 2.0.2, and therefore enables it usage with Python 3.
Tested with Python 3 on an x86_64 target:
/ # python
Python 3.4.1 (default, Oct 1 2014, 16:25:09)
[GCC 4.7.3] on linux
Type "help", "copyright", "credits" or "license" for more information.
Quoted 1 line
Quoted 1 line
Quoted 1 line
import pyparsing
Quoted 31 lines
Signed-off-by Thomas Petazzoni <[email protected]>--- package/python-pyparsing/Config.in | 2 +- package/python-pyparsing/python-pyparsing.mk | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-)package/python-pyparsing/Config.in +1
-1@@ -1,6 +1,6 @@ config BR2_PACKAGE_PYTHON_PYPARSING bool "python-pyparsing" - depends on BR2_PACKAGE_PYTHON + depends on BR2_PACKAGE_PYTHON || BR2_PACKAGE_PYTHON3 help The pyparsing module is an alternative approach to creating and executing simple grammars, vs. the traditional lex/yacc approach,package/python-pyparsing/python-pyparsing.mk +1
-1@@ -4,7 +4,7 @@ # ################################################################################ -PYTHON_PYPARSING_VERSION = 1.5.6 +PYTHON_PYPARSING_VERSION = 2.0.2 PYTHON_PYPARSING_SOURCE = pyparsing-$(PYTHON_PYPARSING_VERSION).tar.gz PYTHON_PYPARSING_SITE = http://downloads.sourceforge.net/project/pyparsing/pyparsing/pyparsing-$(PYTHON_PYPARSING_VERSION) PYTHON_PYPARSING_LICENSE = MIT
Signature · 1 line
Christophe Vu-Brugier
PATCH 4/6 python-setuptools: bump to version 5.8
In reply to Thomas PetazzoniA refresh of the existing patch is needed, and the patch is also renamed to use the correct patch file naming convention.
--- .../python-setuptools-0001-add-executable.patch | 55 ++++++++++++++++++++++ .../python-setuptools-add-executable.patch | 50 -------------------- package/python-setuptools/python-setuptools.mk | 6 +-- 3 files changed, 58 insertions(+), 53 deletions(-) create mode 100644 package/python-setuptools/python-setuptools-0001-add-executable.patch delete mode 100644 package/python-setuptools/python-setuptools-add-executable.patch
package/python-setuptools/python-setuptools-0001-add-executable.patch new file +55 -0
@@ -0,0 +1,55 @@ +Add a new --executable option to distribute so that we can +force the shebang line in installed python scripts. + +[Thomas: refresh for setuptools 5.8.] + +Signed-off-by: Gustavo Zacarias <[email protected]> +Signed-off-by: Thomas Petazzoni <[email protected]> + +Index: b/setuptools/command/install.py +=================================================================== +--- a/setuptools/command/install.py ++++ b/setuptools/command/install.py +@@ -16,6 +16,7 @@ + """Use easy_install to install the package, w/dependencies""" + + user_options = orig.install.user_options + [ ++ ('executable=', 'e', "specify final destination interpreter path"), + ('old-and-unmanageable', None, "Try not to use this!"), + ('single-version-externally-managed', None, + "used by system package builders to create 'flat' eggs"), +@@ -31,6 +32,7 @@ + + def initialize_options(self): + orig.install.initialize_options(self) ++ self.executable = None + self.old_and_unmanageable = None + self.single_version_externally_managed = None + +Index: b/setuptools/command/install_scripts.py +=================================================================== +--- a/setuptools/command/install_scripts.py ++++ b/setuptools/command/install_scripts.py +@@ -11,6 +11,13 @@ + def initialize_options(self): + orig.install_scripts.initialize_options(self) + self.no_ep = False ++ self.executable = None ++ ++ def finalize_options(self): ++ orig.install_scripts.finalize_options(self) ++ self.set_undefined_options('install', ++ ('executable','executable') ++ ) + + def run(self): + from setuptools.command.easy_install import get_script_args +@@ -32,6 +39,8 @@ + ) + bs_cmd = self.get_finalized_command('build_scripts') + executable = getattr(bs_cmd, 'executable', sys_executable) ++ if self.executable is not None: ++ executable = self.executable + is_wininst = getattr( + self.get_finalized_command("bdist_wininst"), '_is_running', False + )
package/python-setuptools/python-setuptools-add-executable.patch deleted +0 -50
50 lines not on this page · raw message
package/python-setuptools/python-setuptools.mk +3 -3
@@ -4,9 +4,9 @@ # ################################################################################ -PYTHON_SETUPTOOLS_VERSION = 2.1.2 -PYTHON_SETUPTOOLS_SOURCE = setuptools-$(PYTHON_SETUPTOOLS_VERSION).tar.gz -PYTHON_SETUPTOOLS_SITE = http://pypi.python.org/packages/source/s/setuptools +PYTHON_SETUPTOOLS_VERSION = 5.8 +PYTHON_SETUPTOOLS_SOURCE = setuptools-$(PYTHON_SETUPTOOLS_VERSION).tar.gz +PYTHON_SETUPTOOLS_SITE = http://pypi.python.org/packages/source/s/setuptools PYTHON_SETUPTOOLS_LICENSE = Python Software Foundation or Zope Public License PYTHON_SETUPTOOLS_LICENSE_FILES = PKG-INFO PYTHON_SETUPTOOLS_SETUP_TYPE = setuptools
Signature · 1 line
2.0.0
PATCH 5/6 python-pyasn: requires the zlib Python module
In reply to Thomas PetazzoniThe python-pyasn module requires the zlib Python module (part of the Python standard installation, but optional in Buildroot), otherwise it fails to load at runtime.
--- package/python-pyasn/Config.in | 2 ++ 1 file changed, 2 insertions(+)
package/python-pyasn/Config.in +2 -0
@@ -2,6 +2,8 @@ config BR2_PACKAGE_PYTHON_PYASN bool "python-pyasn" depends on BR2_PACKAGE_PYTHON || BR2_PACKAGE_PYTHON3 depends on BR2_INSTALL_LIBSTDCPP + select BR2_PACKAGE_PYTHON3_ZLIB if BR2_PACKAGE_PYTHON3 + select BR2_PACKAGE_PYTHON_ZLIB if BR2_PACKAGE_PYTHON help PyASN is a Python extension module that enables you to perform very fast IP address to Autonomous System Number
Signature · 1 line
2.0.0
PATCH 6/6 python, python3: convert py/pyc removal to TARGET_FINALIZE_HOOKS
In reply to Thomas PetazzoniSince the removal of py/pyc files is Python-specific, this commit moves the logic removing those files to python.mk and python3.mk respectively.
--- Makefile | 6 ------ package/python/python.mk | 14 ++++++++++++++ package/python3/python3.mk | 14 ++++++++++++++ 3 files changed, 28 insertions(+), 6 deletions(-)
Makefile +0 -6
@@ -573,12 +573,6 @@ endif rm -rf $(TARGET_DIR)/usr/doc $(TARGET_DIR)/usr/share/doc rm -rf $(TARGET_DIR)/usr/share/gtk-doc -rmdir $(TARGET_DIR)/usr/share 2>/dev/null -ifeq ($(BR2_PACKAGE_PYTHON_PY_ONLY)$(BR2_PACKAGE_PYTHON3_PY_ONLY),y) - find $(TARGET_DIR)/usr/lib/ -name '*.pyc' -print0 | xargs -0 rm -f -endif -ifeq ($(BR2_PACKAGE_PYTHON_PYC_ONLY)$(BR2_PACKAGE_PYTHON3_PYC_ONLY),y) - find $(TARGET_DIR)/usr/lib/ -name '*.py' -print0 | xargs -0 rm -f -endif $(STRIP_FIND_CMD) | xargs $(STRIPCMD) 2>/dev/null || true if test -d $(TARGET_DIR)/lib/modules; then \ find $(TARGET_DIR)/lib/modules -type f -name '*.ko' | \
package/python/python.mk +14 -0
@@ -208,3 +208,17 @@ PYTHON_PATH = $(TARGET_DIR)/usr/lib/python$(PYTHON_VERSION_MAJOR)/sysconfigdata/ $(eval $(autotools-package)) $(eval $(host-autotools-package)) + +ifeq ($(BR2_PACKAGE_PYTHON_PYC_ONLY),y) +define PYTHON_FINALIZE_TARGET + find $(TARGET_DIR)/usr/lib/python$(PYTHON_VERSION_MAJOR) -name '*.py' -print0 | xargs -0 rm -f +endef +endif + +ifeq ($(BR2_PACKAGE_PYTHON_PY_ONLY),y) +define PYTHON_FINALIZE_TARGET + find $(TARGET_DIR)/usr/lib/python$(PYTHON_VERSION_MAJOR) -name '*.pyc' -print0 | xargs -0 rm -f +endef +endif + +TARGET_FINALIZE_HOOKS += PYTHON_FINALIZE_TARGET
package/python3/python3.mk +14 -0
@@ -187,3 +187,17 @@ PYTHON3_PATH = $(TARGET_DIR)/usr/lib/python$(PYTHON3_VERSION_MAJOR)/sysconfigdat $(eval $(autotools-package)) $(eval $(host-autotools-package)) + +ifeq ($(BR2_PACKAGE_PYTHON3_PYC_ONLY),y) +define PYTHON3_FINALIZE_TARGET + find $(TARGET_DIR)/usr/lib/python$(PYTHON3_VERSION_MAJOR) -name '*.py' -print0 | xargs -0 rm -f +endef +endif + +ifeq ($(BR2_PACKAGE_PYTHON3_PY_ONLY),y) +define PYTHON3_FINALIZE_TARGET + find $(TARGET_DIR)/usr/lib/python$(PYTHON3_VERSION_MAJOR) -name '*.pyc' -print0 | xargs -0 rm -f +endef +endif + +TARGET_FINALIZE_HOOKS += PYTHON3_FINALIZE_TARGET
Signature · 1 line
2.0.0
cover Python improvements: PEP 3147 fix and misc updates
In reply to Thomas PetazzoniHello,
Quoted 7 lines · Thomas Petazzoni
Thomas Petazzoni (6):
python3: add distutils fix for PEP 3147 issue
python3: remove unneeded__pycache__directories
python-pyparsing: bump to version 2.0.2, enable on Python 3
python-setuptools: bump to version 5.8
python-pyasn: requires the zlib Python module
python, python3: convert py/pyc removal toTARGET_FINALIZE_HOOKS
I've applied this patch series.
Thomas
Signature · 3 lines
Thomas Petazzoni, CTO, Free Electrons Embedded Linux, Kernel and Android engineering http://free-electrons.com