python3: add patch to fix PEP 3147 issue with automake built packages

PATCH New Christophe Vu-Brugier 2 participants 2 messages Nov 14, 2015 – Nov 17, 2015

Share & tools

Christophe Vu-Brugier

author

2015-11-14 15:54 UTC #0 lore raw

PATCH v3 1/1 python3: add patch to fix PEP 3147 issue with automake built packages

Packages built with automake use a py-compile helper to byte-compile Python source files. This script uses the "py_compile" module from the standard library. In turn, the compile() function in the "py_compile" module invokes the cache_from_source() function provided by importlib.

This commit adds a new patch named "020-importlib-no-pep3147.patch" that changes cache_from_source() and source_from_cache() in importlib to get rid of the "__pycache__" directory.

This commit fixes the following import error in kmod when the module is built for Python 3:

Quoted 1 line

Quoted 1 line
Quoted 1 line
from kmod import Kmod

Traceback (most recent call last):

    File "<stdin>", line 1, in <module>

ImportError: cannot import name 'Kmod'

Moreover, this commit removes two patches that are no longer necessary since modifying cache_from_source() and source_from_cache() disables PEP 3147 for the standard library and distutils / setuptools.

* 004-old-stdlib-cache.patch
* 016-distutils-no-pep3147.patch

Signed-off-by Christophe Vu-Brugier <[email protected]>

Signed-off-by Thomas Petazzoni <[email protected]>

---
Changes since v2:
* prevent importlib.h regeneration to fix an issue reported by Thomas Petazzoni

Changes since v1:
* remove patches 004 and 016 as suggested by Thomas Petazzoni

---
 package/python3/004-old-stdlib-cache.patch     | 78 --------------------------
 package/python3/005-pyc-pyo-conditional.patch  |  8 +--
 package/python3/016-distutils-no-pep3147.patch | 36 ------------
 package/python3/020-importlib-no-pep3147.patch | 69 +++++++++++++++++++++++
 package/python3/python3.mk                     | 10 ++++
 5 files changed, 83 insertions(+), 118 deletions(-)
 delete mode 100644 package/python3/004-old-stdlib-cache.patch
 delete mode 100644 package/python3/016-distutils-no-pep3147.patch
 create mode 100644 package/python3/020-importlib-no-pep3147.patch

package/python3/004-old-stdlib-cache.patch deleted +0 ~~-78~~

deleted file mode 100644
index 9ed7e59..0000000
@@ -1,78 +0,0 @@
-python3: Fix pyc-only related runtime exceptions
-
-Python3 changes the pyc lookup strategy, ignoring the
-__pycache__ directory if the .py file is missing. Change
-install location to enable use of .pyc without their parent .py
-
-See http://www.python.org/dev/peps/pep-3147
-
-Signed-off-by: Daniel Nelson <[email protected]>
-
-Index: b/configure.ac
-===================================================================
---- a/configure.ac
-+++ b/configure.ac
-@@ -352,6 +352,23 @@
- AC_SUBST(FRAMEWORKUNIXTOOLSPREFIX)
- AC_SUBST(FRAMEWORKINSTALLAPPSPREFIX)
- 
-+STDLIB_CACHE_FLAGS=
-+AC_MSG_CHECKING(for --enable-old-stdlib-cache)
-+AC_ARG_ENABLE(old-stdlib-cache,
-+    AS_HELP_STRING([--enable-old-stdlib-cache], [enable pre-pep3147 stdlib cache]),
-+[
-+    if test "$enableval" = "yes"
-+    then
-+        STDLIB_CACHE_FLAGS="-b"
-+    else
-+        STDLIB_CACHE_FLAGS="" 
-+    fi
-+],
-+[
-+    STDLIB_CACHE_FLAGS="" 
-+])
-+AC_SUBST(STDLIB_CACHE_FLAGS)
-+
- ##AC_ARG_WITH(dyld,
- ##            AS_HELP_STRING([--with-dyld],
- ##                           [Use (OpenStep|Rhapsody) dynamic linker]))
-Index: b/Makefile.pre.in
-===================================================================
---- a/Makefile.pre.in
-+++ b/Makefile.pre.in
-@@ -162,6 +162,9 @@
- # Options to enable prebinding (for fast startup prior to Mac OS X 10.3)
- OTHER_LIBTOOL_OPT=@OTHER_LIBTOOL_OPT@
- 
-+# Option to enable old-style precompiled stdlib
-+STDLIB_CACHE_FLAGS=@STDLIB_CACHE_FLAGS@
-+
- # Environment to run shared python without installed libraries
- RUNSHARED=       @RUNSHARED@
- 
-@@ -1247,21 +1250,21 @@
- 	fi
- 	-PYTHONPATH=$(DESTDIR)$(LIBDEST)  $(RUNSHARED) \
- 		$(PYTHON_FOR_BUILD) -Wi $(DESTDIR)$(LIBDEST)/compileall.py \
--		-d $(LIBDEST) -f \
-+		-d $(LIBDEST) -f $(STDLIB_CACHE_FLAGS) \
- 		-x 'bad_coding|badsyntax|site-packages|lib2to3/tests/data' \
- 		$(DESTDIR)$(LIBDEST)
- 	-PYTHONPATH=$(DESTDIR)$(LIBDEST) $(RUNSHARED) \
- 		$(PYTHON_FOR_BUILD) -Wi -O $(DESTDIR)$(LIBDEST)/compileall.py \
--		-d $(LIBDEST) -f \
-+		-d $(LIBDEST) -f $(STDLIB_CACHE_FLAGS) \
- 		-x 'bad_coding|badsyntax|site-packages|lib2to3/tests/data' \
- 		$(DESTDIR)$(LIBDEST)
- 	-PYTHONPATH=$(DESTDIR)$(LIBDEST)  $(RUNSHARED) \
- 		$(PYTHON_FOR_BUILD) -Wi $(DESTDIR)$(LIBDEST)/compileall.py \
--		-d $(LIBDEST)/site-packages -f \
-+		-d $(LIBDEST)/site-packages -f $(STDLIB_CACHE_FLAGS) \
- 		-x badsyntax $(DESTDIR)$(LIBDEST)/site-packages
- 	-PYTHONPATH=$(DESTDIR)$(LIBDEST) $(RUNSHARED) \
- 		$(PYTHON_FOR_BUILD) -Wi -O $(DESTDIR)$(LIBDEST)/compileall.py \
--		-d $(LIBDEST)/site-packages -f \
-+		-d $(LIBDEST)/site-packages -f $(STDLIB_CACHE_FLAGS) \
- 		-x badsyntax $(DESTDIR)$(LIBDEST)/site-packages
- 	-PYTHONPATH=$(DESTDIR)$(LIBDEST) $(RUNSHARED) \
- 		$(PYTHON_FOR_BUILD) -m lib2to3.pgen2.driver $(DESTDIR)$(LIBDEST)/lib2to3/Grammar.txt

package/python3/005-pyc-pyo-conditional.patch +4 -4

index c3b159d..0d60ff6 100644
@@ -9,27 +9,27 @@ Index: b/Makefile.pre.in
 +ifeq (@PYC_BUILD@,yes)
  	-PYTHONPATH=$(DESTDIR)$(LIBDEST)  $(RUNSHARED) \
  		$(PYTHON_FOR_BUILD) -Wi $(DESTDIR)$(LIBDEST)/compileall.py \
- 		-d $(LIBDEST) -f $(STDLIB_CACHE_FLAGS) \
+ 		-d $(LIBDEST) -f \
  		-x 'bad_coding|badsyntax|site-packages|lib2to3/tests/data' \
  		$(DESTDIR)$(LIBDEST)
 +endif
 +ifeq (@PYO_BUILD@,yes)
  	-PYTHONPATH=$(DESTDIR)$(LIBDEST) $(RUNSHARED) \
  		$(PYTHON_FOR_BUILD) -Wi -O $(DESTDIR)$(LIBDEST)/compileall.py \
- 		-d $(LIBDEST) -f $(STDLIB_CACHE_FLAGS) \
+ 		-d $(LIBDEST) -f \
  		-x 'bad_coding|badsyntax|site-packages|lib2to3/tests/data' \
  		$(DESTDIR)$(LIBDEST)
 +endif
 +ifeq (@PYC_BUILD@,yes)
  	-PYTHONPATH=$(DESTDIR)$(LIBDEST)  $(RUNSHARED) \
  		$(PYTHON_FOR_BUILD) -Wi $(DESTDIR)$(LIBDEST)/compileall.py \
- 		-d $(LIBDEST)/site-packages -f $(STDLIB_CACHE_FLAGS) \
+ 		-d $(LIBDEST)/site-packages -f \
  		-x badsyntax $(DESTDIR)$(LIBDEST)/site-packages
 +endif
 +ifeq (@PYO_BUILD@,yes)
  	-PYTHONPATH=$(DESTDIR)$(LIBDEST) $(RUNSHARED) \
  		$(PYTHON_FOR_BUILD) -Wi -O $(DESTDIR)$(LIBDEST)/compileall.py \
- 		-d $(LIBDEST)/site-packages -f $(STDLIB_CACHE_FLAGS) \
+ 		-d $(LIBDEST)/site-packages -f \
  		-x badsyntax $(DESTDIR)$(LIBDEST)/site-packages
 +endif
  	-PYTHONPATH=$(DESTDIR)$(LIBDEST) $(RUNSHARED) \

package/python3/016-distutils-no-pep3147.patch deleted +0 ~~-36~~

deleted file mode 100644
index 0fdfeb2..0000000
@@ -1,36 +0,0 @@
-Add distutils fix for PEP 3147 issue
-
-Python 3 has a new standard for installing .pyc file, called PEP
-3147. Unfortunately, this standard requires both the .py and .pyc
-files to be installed for a Python module to be found. This is quite
-annoying on space-constrained embedded systems, since the .py file is
-technically not required for execution.
-
-For the Python standard library, our Python 3 package already contains
-a patch named python3-004-old-stdlib-cache.patch, which allows to
-disable the PEP 3147 installation.
-
-But that lives the distutils/setuptools package an unsolved
-problem. This patch therefore adds a new patch to Python, which makes
-distutils package use the traditional installation path when byte
-compiling, rather than the PEP 3147 installation path. Since
-setuptools relies on distutils internally, it also fixes setuptools
-based packages.
-
-Signed-off-by: Thomas Petazzoni <[email protected]>
-
-Index: b/Lib/distutils/util.py
-===================================================================
---- a/Lib/distutils/util.py
-+++ b/Lib/distutils/util.py
-@@ -437,7 +437,9 @@
-             # Terminology from the py_compile module:
-             #   cfile - byte-compiled file
-             #   dfile - purported source filename (same as 'file' by default)
--            if optimize >= 0:
-+            if "_python_sysroot" in os.environ:
-+                cfile = file + (__debug__ and "c" or "o")
-+            elif optimize >= 0:
-                 cfile = importlib.util.cache_from_source(
-                     file, debug_override=not optimize)
-             else:

package/python3/020-importlib-no-pep3147.patch new file +69 -0

69 lines not on this page · raw message

package/python3/python3.mk +10 -0

index fd5709a..a37b791 100644
@@ -147,7 +147,17 @@ define PYTHON3_TOUCH_GRAMMAR_FILES
 	touch $(@D)/Include/graminit.h $(@D)/Python/graminit.c
 endef
 
+# This prevents the Python Makefile from regenerating the
+# Python/importlib.h header if Lib/importlib/_bootstrap.py has changed
+# because its generation is broken in a cross-compilation environment
+# and importlib.h is not used.
+
+define PYTHON3_TOUCH_IMPORTLIB_H
+	touch $(@D)/Python/importlib.h
+endef
+
 PYTHON3_POST_PATCH_HOOKS += PYTHON3_TOUCH_GRAMMAR_FILES
+PYTHON3_POST_PATCH_HOOKS += PYTHON3_TOUCH_IMPORTLIB_H
 
 #
 # Remove useless files. In the config/ directory, only the Makefile

Signature · 1 line

2.6.2

Thomas Petazzoni

Applied

2015-11-17 22:16 UTC #1 lore raw

PATCH v3 1/1 python3: add patch to fix PEP 3147 issue with automake built packages

Dear Christophe Vu-Brugier,

Quoted 24 lines · Christophe Vu-Brugier

Packages built with automake use a py-compile helper to byte-compile Python source files. This script uses the "py_compile" module from the standard library. In turn, the compile() function in the "py_compile" module invokes the cache_from_source() function provided by importlib.
This commit adds a new patch named "020-importlib-no-pep3147.patch" that changes cache_from_source() and source_from_cache() in importlib to get rid of the "__pycache__" directory.
This commit fixes the following import error in kmod when the module is built for Python 3:
Quoted 1 line
Quoted 1 line
Quoted 1 line
from kmod import Kmod
Traceback (most recent call last):
    File "<stdin>", line 1, in <module>
ImportError: cannot import name 'Kmod'
Moreover, this commit removes two patches that are no longer necessary since modifying cache_from_source() and source_from_cache() disables PEP 3147 for the standard library and distutils / setuptools.
* 004-old-stdlib-cache.patch
* 016-distutils-no-pep3147.patch

Quoted 7 lines

Signed-off-by Christophe Vu-Brugier <[email protected]>
Signed-off-by Thomas Petazzoni <[email protected]>
---
Changes since v2:
* prevent importlib.h regeneration to fix an issue reported by Thomas Petazzoni

Applied to next, thanks!

Thomas

Signature · 3 lines

Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com