python3: Several fixes, compilation of modules, runtime issues, and two new recipes
cover python3: Several fixes, compilation of modules, runtime issues, and two new recipes
Adds python3-pip recipe, upgrades python3-distribute to python3-setuptools to keep up to date with upstream, fixes runtime issues for these modules and for python3 in general.
The following changes since commit 62f3e62e0a2145d32fd75530b6f2aeba00dc1bc4:
python3: Upgrade from 3.4.2 to 3.4.3 (2015-05-12 10:44:53 +0000)
are available in the git repository at:
git://git.yoctoproject.org/poky-contrib hsalejandro/py3-pip-recipe
http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=hsalejandro/py3-pip-recipe
Alejandro Hernandez (5):
python3: fix pyconfig.h installation on target
python3: Fix distutils import error
python3-io: Fixes compilation of python3 ssl module
python3-distribute: Upgrade to python3-setuptools 15.2
python3-pip: Adds recipe for python3-pip
meta/classes/setuptools3.bbclass | 2 +-
.../python/python-3.4-manifest.inc | 2 +-
meta/recipes-devtools/python/python3-pip_6.1.1.bb | 40 ++++++++++++++++++++++
...ribute_0.6.32.bb => python3-setuptools_15.2.bb} | 40 ++++++++++++----------
.../python/python3/fix_ssl_include_dir.patch | 29 ++++++++++++++++
meta/recipes-devtools/python/python3_3.4.3.bb | 2 +-
scripts/contrib/python/generate-manifest-3.4.py | 2 +-
7 files changed, 95 insertions(+), 22 deletions(-)
create mode 100644 meta/recipes-devtools/python/python3-pip_6.1.1.bb
rename meta/recipes-devtools/python/{python3-distribute_0.6.32.bb => python3-setuptools_15.2.bb} (53%)
create mode 100644 meta/recipes-devtools/python/python3/fix_ssl_include_dir.patchSignature · 1 line
1.8.4.5
PATCH 1/5 python3: fix pyconfig.h installation on target
The file pyconfig.h wasnt being deployed on target, causing an error when importing some libraries that required it, this patch fixes python3 manifest to include this file and fix the issue.
[YOCTO #7764]
--- meta/recipes-devtools/python/python-3.4-manifest.inc | 2 +- scripts/contrib/python/generate-manifest-3.4.py | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-)
meta/recipes-devtools/python/python-3.4-manifest.inc +1 -1
@@ -35,7 +35,7 @@ FILES_${PN}-compression="${libdir}/python3.4/gzip.* ${libdir}/python3.4/zipfile. SUMMARY_${PN}-core="Python interpreter and core modules" RDEPENDS_${PN}-core="${PN}-lang ${PN}-re ${PN}-reprlib ${PN}-codecs ${PN}-io ${PN}-math" -FILES_${PN}-core="${libdir}/python3.4/__future__.* ${libdir}/python3.4/_abcoll.* ${libdir}/python3.4/abc.* ${libdir}/python3.4/copy.* ${libdir}/python3.4/copyreg.* ${libdir}/python3.4/ConfigParser.* ${libdir}/python3.4/genericpath.* ${libdir}/python3.4/getopt.* ${libdir}/python3.4/linecache.* ${libdir}/python3.4/new.* ${libdir}/python3.4/os.* ${libdir}/python3.4/posixpath.* ${libdir}/python3.4/struct.* ${libdir}/python3.4/warnings.* ${libdir}/python3.4/site.* ${libdir}/python3.4/stat.* ${libdir}/python3.4/UserDict.* ${libdir}/python3.4/UserList.* ${libdir}/python3.4/UserString.* ${libdir}/python3.4/lib-dynload/binascii.*.so ${libdir}/python3.4/lib-dynload/_struct.*.so ${libdir}/python3.4/lib-dynload/time.*.so ${libdir}/python3.4/lib-dynload/xreadlines.*.so ${libdir}/python3.4/types.* ${libdir}/python3.4/platform.* ${bindir}/python* ${libdir}/python3.4/_weakrefset.* ${libdir}/python3.4/sysconfig.* ${libdir}/python3.4/_sysconfigdata.* ${libdir}/python3.4/config/Makefile ${includedir}/python${PYTHON_MAJMIN}/pyconfig*.h ${libdir}/python${PYTHON_MAJMIN}/collections ${libdir}/python${PYTHON_MAJMIN}/_collections_abc.* ${libdir}/python${PYTHON_MAJMIN}/_sitebuiltins.* ${libdir}/python${PYTHON_MAJMIN}/sitecustomize.py " +FILES_${PN}-core="${libdir}/python3.4/__future__.* ${libdir}/python3.4/_abcoll.* ${libdir}/python3.4/abc.* ${libdir}/python3.4/copy.* ${libdir}/python3.4/copyreg.* ${libdir}/python3.4/ConfigParser.* ${libdir}/python3.4/genericpath.* ${libdir}/python3.4/getopt.* ${libdir}/python3.4/linecache.* ${libdir}/python3.4/new.* ${libdir}/python3.4/os.* ${libdir}/python3.4/posixpath.* ${libdir}/python3.4/struct.* ${libdir}/python3.4/warnings.* ${libdir}/python3.4/site.* ${libdir}/python3.4/stat.* ${libdir}/python3.4/UserDict.* ${libdir}/python3.4/UserList.* ${libdir}/python3.4/UserString.* ${libdir}/python3.4/lib-dynload/binascii.*.so ${libdir}/python3.4/lib-dynload/_struct.*.so ${libdir}/python3.4/lib-dynload/time.*.so ${libdir}/python3.4/lib-dynload/xreadlines.*.so ${libdir}/python3.4/types.* ${libdir}/python3.4/platform.* ${bindir}/python* ${libdir}/python3.4/_weakrefset.* ${libdir}/python3.4/sysconfig.* ${libdir}/python3.4/_sysconfigdata.* ${libdir}/python3.4/config/Makefile ${includedir}/python${PYTHON_BINABI}/pyconfig*.h ${libdir}/python${PYTHON_MAJMIN}/collections ${libdir}/python${PYTHON_MAJMIN}/_collections_abc.* ${libdir}/python${PYTHON_MAJMIN}/_sitebuiltins.* ${libdir}/python${PYTHON_MAJMIN}/sitecustomize.py " SUMMARY_${PN}-crypt="Python basic cryptographic and hashing support" RDEPENDS_${PN}-crypt="${PN}-core"
scripts/contrib/python/generate-manifest-3.4.py +1 -1
@@ -175,7 +175,7 @@ if __name__ == "__main__": "lib-dynload/binascii.*.so lib-dynload/_struct.*.so lib-dynload/time.*.so " + "lib-dynload/xreadlines.*.so types.* platform.* ${bindir}/python* " + "_weakrefset.* sysconfig.* _sysconfigdata.* config/Makefile " + - "${includedir}/python${PYTHON_MAJMIN}/pyconfig*.h " + + "${includedir}/python${PYTHON_BINABI}/pyconfig*.h " + "${libdir}/python${PYTHON_MAJMIN}/collections " + "${libdir}/python${PYTHON_MAJMIN}/_collections_abc.* " + "${libdir}/python${PYTHON_MAJMIN}/_sitebuiltins.* " +
Signature · 1 line
1.8.4.5
PATCH 2/5 python3: Fix distutils import error
In reply to Alejandro HernandezThe patch 12-distutils-prefix-is-inside-staging-area.patch is used to fix problems when trying to get some environmental variables but it is only necessary for python3-native, on python3-core it causes an error when importing distutils since env variables of the likes of HOST_SYS and such do not exist on target, this removes the patch from python3-core so distutils can be imported correctly.
[YOCTO #7766]
--- meta/recipes-devtools/python/python3_3.4.3.bb | 1 - 1 file changed, 1 deletion(-)
meta/recipes-devtools/python/python3_3.4.3.bb +0 -1
@@ -7,7 +7,6 @@ PYTHON_BINABI= "${PYTHON_MAJMIN}m" DISTRO_SRC_URI ?= "file://sitecustomize.py" DISTRO_SRC_URI_linuxstdbase = "" SRC_URI = "http://www.python.org/ftp/python/${PV}/Python-${PV}.tar.xz \ -file://12-distutils-prefix-is-inside-staging-area.patch \ file://python-config.patch \ file://000-cross-compile.patch \ file://020-dont-compile-python-files.patch \
Signature · 1 line
1.8.4.5
PATCH 3/5 python3-io: Fixes compilation of python3 ssl module
In reply to Alejandro HernandezDuring python3 compilation, the module ssl was being skippped due to the fact that the compilation script couldnt find the required files, this patch fixes setup.py so it looks for the files in the correct directory, hence fixing its compilation and installation.
[YOCTO #7768]
--- .../python/python3/fix_ssl_include_dir.patch | 29 ++++++++++++++++++++++ meta/recipes-devtools/python/python3_3.4.3.bb | 1 + 2 files changed, 30 insertions(+) create mode 100644 meta/recipes-devtools/python/python3/fix_ssl_include_dir.patch
meta/recipes-devtools/python/python3/fix_ssl_include_dir.patch new file +29 -0
@@ -0,0 +1,29 @@ +Index: Python-3.4.3/setup.py +=================================================================== +--- Python-3.4.3.orig/setup.py ++++ Python-3.4.3/setup.py +@@ -726,10 +726,9 @@ class PyBuildExt(build_ext): + exts.append( Extension('_socket', ['socketmodule.c'], + depends = ['socketmodule.h']) ) + # Detect SSL support for the socket module (via _ssl) +- search_for_ssl_incs_in = [ +- '/usr/local/ssl/include', +- '/usr/contrib/ssl/include/' +- ] ++ search_for_ssl_incs_in = [] ++ for dir in [os.getenv("STAGING_INCDIR")]: ++ search_for_ssl_incs_in.append(dir) + ssl_incs = find_file('openssl/ssl.h', inc_dirs, + search_for_ssl_incs_in + ) +@@ -739,9 +738,7 @@ class PyBuildExt(build_ext): + if krb5_h: + ssl_incs += krb5_h + ssl_libs = find_library_file(self.compiler, 'ssl',lib_dirs, +- ['/usr/local/ssl/lib', +- '/usr/contrib/ssl/lib/' +- ] ) ++ [os.getenv("STAGING_LIBDIR")]) + + if (ssl_incs is not None and + ssl_libs is not None):
meta/recipes-devtools/python/python3_3.4.3.bb +1 -0
@@ -36,6 +36,7 @@ SRC_URI += "\ file://python3-setup.py-no-host-headers-libs.patch \ file://sysconfig.py-add-_PYTHON_PROJECT_SRC.patch \ file://setup.py-check-cross_compiling-when-get-FLAGS.patch \ + file://fix_ssl_include_dir.patch \ " SRC_URI[md5sum] = "7d092d1bba6e17f0d9bd21b49e441dd5" SRC_URI[sha256sum] = "b5b3963533768d5fc325a4d7a6bd6f666726002d696f1d399ec06b043ea996b8"
Signature · 1 line
1.8.4.5
PATCH 4/5 python3-distribute: Upgrade to python3-setuptools 15.2
In reply to Alejandro Hernandezpython3-distribute was merged back to python3-setuptools in 2013, and it is no longer being maintained, this upgrade also provides functionality that will be needed for python3-pip.
[YOCTO #7763]
---
meta/classes/setuptools3.bbclass | 2 +-
...ribute_0.6.32.bb => python3-setuptools_15.2.bb} | 40 ++++++++++++----------
2 files changed, 23 insertions(+), 19 deletions(-)
rename meta/recipes-devtools/python/{python3-distribute_0.6.32.bb => python3-setuptools_15.2.bb} (53%)meta/classes/setuptools3.bbclass +1 -1
@@ -1,6 +1,6 @@ inherit distutils3 -DEPENDS += "python3-distribute-native" +DEPENDS += "python3-setuptools-native" DISTUTILS_INSTALL_ARGS = "--root=${D} \ --prefix=${prefix} \
meta/recipes-devtools/python/python3-setuptools_15.2.bb renamed from meta/recipes-devtools/python/python3-distribute_0.6.32.bb +22 -18
@@ -1,28 +1,21 @@ SUMMARY = "Download, build, install, upgrade, and uninstall Python packages" -HOMEPAGE = "http://packages.python.org/distribute" +HOMEPAGE = "http://packages.python.org/setuptools" SECTION = "devel/python" LICENSE = "PSF" -LIC_FILES_CHKSUM = "file://setup.py;beginline=234;endline=234;md5=26f6b02022b737126d3c88838782dddb" +LIC_FILES_CHKSUM = "file://setup.py;beginline=78;endline=78;md5=8a314270dd7a8dbca741775415f1716e" -SRCNAME = "distribute" -PR = "ml5" +SRCNAME = "setuptools" DEPENDS += "python3" DEPENDS_class-native += "python3-native" SRC_URI = " \ - http://pypi.python.org/packages/source/d/${SRCNAME}/${SRCNAME}-${PV}.tar.gz \ + http://pypi.python.org/packages/source/s/${SRCNAME}/${SRCNAME}-${PV}.tar.gz \ " -SRC_URI[md5sum] = "acb7a2da81e3612bfb1608abe4f0e568" -SRC_URI[sha256sum] = "8970cd1e148b5d1fea9430584aea66c45ea22d80e0933393ec49ebc388f718df" +SRC_URI[md5sum] = "a9028a9794fc7ae02320d32e2d7e12ee" +SRC_URI[sha256sum] = "381e78471fb0eff89c4b1a219e8739f48dd87c76ad2d3a790010ca3a62ee29a4" S = "${WORKDIR}/${SRCNAME}-${PV}" -# http://python-distribute.org/distribute_setup.py - -# force the selection of python3 -#PYTHON_BASEVERSION = "3.3" -#PYTHON_MAJMIN = "3.3" - inherit distutils3 DISTUTILS_INSTALL_ARGS += "--install-lib=${D}${libdir}/${PYTHON_DIR}/site-packages"
@@ -37,15 +30,26 @@ do_install_append() { rm ${D}${PYTHON_SITEPACKAGES_DIR}/setuptools.pth mv ${D}${bindir}/easy_install ${D}${bindir}/easy3_install echo "./${SRCNAME}-${PV}-py${PYTHON_BASEVERSION}.egg" > ${D}${PYTHON_SITEPACKAGES_DIR}/setuptools.pth - sed -i -e '1s|^#!.*python|#!/usr/bin/env python3|' \ - ${D}${PYTHON_SITEPACKAGES_DIR}/distribute-${PV}-py${PYTHON_BASEVERSION}.egg/setuptools/tests/test_resources.py } RDEPENDS_${PN} = "\ python3-distutils \ python3-compression \ " - -RPROVIDES_${PN} += "python3-setuptools" - +RDEPENDS_${PN}_class-target = "\ + python3-ctypes \ + python3-distutils \ + python3-email \ + python3-importlib \ + python3-numbers \ + python3-compression \ + python3-shell \ + python3-subprocess \ + python3-textutils \ + python3-pkgutil \ + python3-threading \ + python3-misc \ + python3-unittest \ + python3-xml \ +" BBCLASSEXTEND = "native"
Signature · 1 line
1.8.4.5
PATCH 5/5 python3-pip: Adds recipe for python3-pip
In reply to Alejandro HernandezPIP is installed by default since python 3.4, due to issues with the autobuilders, its default installation had to be disabled, this creates a recipe for it, it brings a lot of new dependencies so it is chosen not to be installed by default along with python3-core.
[YOCTO #7762]
--- meta/recipes-devtools/python/python3-pip_6.1.1.bb | 40 +++++++++++++++++++++++ 1 file changed, 40 insertions(+) create mode 100644 meta/recipes-devtools/python/python3-pip_6.1.1.bb
meta/recipes-devtools/python/python3-pip_6.1.1.bb new file +40 -0
@@ -0,0 +1,40 @@ +SUMMARY = "The PyPA recommended tool for installing Python packages" +sHOMEPAGEsss = "https://pypi.python.org/pypi/pip" +SECTION = "devel/python" +LICENSE = "MIT" +LIC_FILES_CHKSUM = "file://LICENSE.txt;md5=45665b53032c02b35e29ddab8e61fa91" + +SRCNAME = "pip" +DEPENDS += "python3" + +SRC_URI = " \ + http://pypi.python.org/packages/source/p/${SRCNAME}/${SRCNAME}-${PV}.tar.gz \ +" +SRC_URI[md5sum] = "6b19e0a934d982a5a4b798e957cb6d45" +SRC_URI[sha256sum] = "89f3b626d225e08e7f20d85044afa40f612eb3284484169813dc2d0631f2a556" + +S = "${WORKDIR}/${SRCNAME}-${PV}" + +inherit distutils3 + +DISTUTILS_INSTALL_ARGS += "--install-lib=${D}${libdir}/${PYTHON_DIR}/site-packages" + +do_install_prepend() { + install -d ${D}/${libdir}/${PYTHON_DIR}/site-packages +} + +# Use setuptools site.py instead, avoid shared state issue +do_install_append() { + rm ${D}/${libdir}/${PYTHON_DIR}/site-packages/site.py + rm ${D}/${libdir}/${PYTHON_DIR}/site-packages/__pycache__/site.cpython-34.pyc +} + +RDEPENDS_${PN} = "\ + python3-compile \ + python3-io \ + python3-json \ + python3-netserver \ + python3-setuptools \ + python3-unixadmin \ + python3-xmlrpc \ +"
Signature · 1 line
1.8.4.5
Re: [PATCH 3/5] python3-io: Fixes compilation of python3 ssl module
In reply to Alejandro HernandezQuoted 2 lines · Alejandro Hernandez
+++
b/meta/recipes-devtools/python/python3/fix_ssl_include_dir.patch
This patch needs an explanation, upstream-status, and signed-off-by before the actual diff.
Also, oh god, the upstream code is horrible!
Ross
Re: [PATCH 5/5] python3-pip: Adds recipe for python3-pip
In reply to Alejandro HernandezQuoted 4 lines · Alejandro Hernandez
PIP is installed by default since python 3.4, due to issues with the autobuilders, its default installation had to be disabled, this creates a recipe for it, it brings a lot of new dependencies so it is chosen not to be installed by default along with python3-core.
Missing dependencies?
ERROR: Function failed: do_compile (log file is located at /data/poky-master/tmp/work/corei7-64-poky-linux/python3-pip/6.1.1-r0/temp/log.do_compile.7121)
ERROR: Logfile of failure stored in:
/data/poky-master/tmp/work/corei7-64-poky-linux/python3-pip/6.1.1-r0/temp/log.do_compile.7121 Log data follows:
| DEBUG: Executing shell function do_compile | Traceback (most recent call last): | File "setup.py", line 6, in <module> | from setuptools import setup, find_packages | ImportError: No module named 'setuptools' | ERROR: python3 setup.py build_ext execution failed. | WARNING: exit code 1 from a shell command.
Ross
Re: [PATCH 4/5] python3-distribute: Upgrade to python3-setuptools 15.2
In reply to Alejandro HernandezQuoted 4 lines · Alejandro Hernandez
python3-distribute was merged back to python3-setuptools in 2013, and it is no longer being maintained, this upgrade also provides functionality that will be needed for python3-pip.
ERROR: The recipe python3-setuptools is trying to install files into a shared area when those files already exist. Those files and their manifest location are:
/data/poky-master/tmp/sysroots/intel-corei7-64/usr/lib/python3.4/site-packages/site.py Matched in manifest-intel-corei7-64-python3-distribute.populate_sysroot /data/poky-master/tmp/sysroots/intel-corei7-64/usr/lib/python3.4/site-packages/setuptools.pth Matched in manifest-intel-corei7-64-python3-distribute.populate_sysroot /data/poky-master/tmp/sysroots/intel-corei7-64/usr/lib/python3.4/site-packages/__pycache__/site.cpython-34.pyc Matched in manifest-intel-corei7-64-python3-distribute.populate_sysroot
Urgh, this again. We either live with this and tell people to delete their tmp/, or bump the tmp ABI and provide a migration path. Or improve bitbake to handle files moving between recipes in the sysroot and doing the right thing...
Ross
Re: [PATCH 3/5] python3-io: Fixes compilation of python3 ssl module
In reply to Burton, RossSorry about that, will send v2 in a bit. and yes I agree
Quoted 11 lines · Ross
On 21 May 2015 at 10:15, Alejandro Hernandez <[email protected] <mailto:[email protected]>> wrote:
+++
b/meta/recipes-devtools/python/python3/fix_ssl_include_dir.patchThis patch needs an explanation, upstream-status, and signed-off-by before the actual diff.
Also, oh god, the upstream code is horrible!
Ross
Re: [PATCH 5/5] python3-pip: Adds recipe for python3-pip
In reply to Burton, RossQuoted 24 lines · Ross
On 21 May 2015 at 10:15, Alejandro Hernandez <[email protected] <mailto:[email protected]>> wrote:
PIP is installed by default since python 3.4, due to issues with the autobuilders, its default installation had to be disabled, this creates
a recipe for it, it brings a lot of new dependencies so it is chosen not
to be installed by default along with python3-core.Missing dependencies?
ERROR: Function failed:
do_compile(log file is located at /data/poky-master/tmp/work/corei7-64-poky-linux/python3-pip/6.1.1-r0/temp/log.do_compile.7121)
ERROR: Logfile of failure stored in: /data/poky-master/tmp/work/corei7-64-poky-linux/python3-pip/6.1.1-r0/temp/log.do_compile.7121 Log data follows:| DEBUG: Executing shell function do_compile | Traceback (most recent call last): | File "setup.py", line 6, in <module> | from setuptools import setup, find_packages | ImportError: No module named 'setuptools' | ERROR: python3 setup.py build_ext execution failed. | WARNING: exit code 1 from a shell command.
Quoted 1 line
Ross
Hmm weird, this should not have happened, apparently there's a missing dependency to python3-setuptools, but I thought I had listed that
Re: [PATCH 4/5] python3-distribute: Upgrade to python3-setuptools 15.2
In reply to Burton, RossQuoted 16 lines · Ross
On 21 May 2015 at 10:15, Alejandro Hernandez <[email protected] <mailto:[email protected]>> wrote:
python3-distribute was merged back to python3-setuptools in 2013, and it is no longer being maintained, this upgrade also provides functionality that will be needed for python3-pip.
ERROR: The recipe python3-setuptools is trying to install files into a shared area when those files already exist. Those files and their manifest location are:
/data/poky-master/tmp/sysroots/intel-corei7-64/usr/lib/python3.4/site-packages/site.py
Matched in manifest-intel-corei7-64-python3-distribute.populate_sysroot /data/poky-master/tmp/sysroots/intel-corei7-64/usr/lib/python3.4/site-packages/setuptools.pth
Matched in manifest-intel-corei7-64-python3-distribute.populate_sysroot /data/poky-master/tmp/sysroots/intel-corei7-64/usr/lib/python3.4/site-packages/__pycache__/site.cpython-34.pyc
Matched in manifest-intel-corei7-64-python3-distribute.populate_sysroot
Quoted 5 lines
Urgh, this again. We either live with this and tell people to delete their tmp/, or bump the tmp ABI and provide a migration path. Or improve bitbake to handle files moving between recipes in the sysroot and doing the right thing...
Ross
I think this is because the package name is different, before it was python3-distribute and now its python3-setuptools but they do install the same files, so for this case in specific; a bitbake python3-distribtue -c cleanall should be done befooore merging? so the python3-distribute recipe actually exists or clean the /tmp manually.
Alejandro