Skip to contentSkip to conversation

distutils/setuptools:fix recipes may try to fetch code during do_compile

PATCH Superseded Hongxu Jia 4 participants 9 messages Jul 17, 2018 – Jul 27, 2018

lore
Share & tools
lore

Hongxu Jia

author
#0 lore raw

cover distutils/setuptools:fix recipes may try to fetch code during do_compile

The following changes since commit 05c32d2de1ee4681cc78cb107a158e9ab22c9619:

oeqa/selftest/wic: Split tests into two class groups (2018-07-16 13:21:35 +0100)

are available in the git repository at:

git://git.openembedded.org/openembedded-core-contrib hongxu/setuptools-fetch
http://cgit.openembedded.org/openembedded-core-contrib/log/?h=hongxu/setuptools-fetch

Hongxu Jia (2):
distutils/setuptools-native: do not try to fetch code during
do_compile
distutils3/setuptools3-native: do not try to fetch code during
do_compile

 meta/classes/distutils.bbclass                     |  2 ++
 meta/classes/distutils3.bbclass                    |  2 ++
 ...ionally-do-not-fetch-code-by-easy_install.patch | 35 ++++++++++++++++++++++
 .../python/python-setuptools_40.0.0.bb             |  2 ++
 ...ionally-do-not-fetch-code-by-easy_install.patch | 35 ++++++++++++++++++++++
 .../python/python3-setuptools_40.0.0.bb            |  2 ++
 6 files changed, 78 insertions(+)
 create mode 100644 meta/recipes-devtools/python/python-setuptools/0001-conditionally-do-not-fetch-code-by-easy_install.patch
 create mode 100644 meta/recipes-devtools/python/python3-setuptools/0001-conditionally-do-not-fetch-code-by-easy_install.patch
Signature · 1 line
2.8.1
author
#1 lore raw

PATCH 1/2 distutils/setuptools-native: do not try to fetch code during do_compile

If a python recipe is using setuptools and the setup_requires argument, where setuptools will use easy_install to fetch the module if it isn't present.

The build failed on a machine where a proxy was required, but succeeded on a machine which had direct access to the internet

Add var-NO_FETCH_BUILD, and set it in distutils_do_compile which does not allow to fetch code from internet during do_compile.

Example result:
...
ERROR: Do not try to fetch `pytest-runner1' for building. Please add its native recipe to DEPENDS. Traceback (most recent call last):
File "setup.py", line 56, in <module>
...

The improvement is flexible for test_requirements argument (used at `setup.py test'), where use easy_install also.

[YOCTO #12084]
Signed-off-by Hongxu Jia <[email protected]>
---
 meta/classes/distutils.bbclass                     |  2 ++
 ...ionally-do-not-fetch-code-by-easy_install.patch | 35 ++++++++++++++++++++++
 .../python/python-setuptools_40.0.0.bb             |  2 ++
 3 files changed, 39 insertions(+)
 create mode 100644 meta/recipes-devtools/python/python-setuptools/0001-conditionally-do-not-fetch-code-by-easy_install.patch
meta/classes/distutils.bbclass +2 -0
index 1930c35..872f4e3 100644
@@ -8,6 +8,8 @@ DISTUTILS_INSTALL_ARGS ?= "--prefix=${D}/${prefix} \
     --install-data=${D}/${datadir}"
 
 distutils_do_compile() {
+         NO_FETCH_BUILD=1 \
+         NO_FETCH_BUILD_HELP="Please add its native recipe to DEPENDS." \
          STAGING_INCDIR=${STAGING_INCDIR} \
          STAGING_LIBDIR=${STAGING_LIBDIR} \
          ${STAGING_BINDIR_NATIVE}/${PYTHON_PN}-native/${PYTHON_PN} setup.py build ${DISTUTILS_BUILD_ARGS} || \
meta/recipes-devtools/python/python-setuptools/0001-conditionally-do-not-fetch-code-by-easy_install.patch new file +35 -0
new file mode 100644
index 0000000..441af16
@@ -0,0 +1,35 @@
+From 019422d19c6401fa8f45b1311ecd31bb087c809d Mon Sep 17 00:00:00 2001
+From: Hongxu Jia <[email protected]>
+Date: Tue, 17 Jul 2018 10:13:38 +0800
+Subject: [PATCH] conditionally do not fetch code by easy_install
+
+If var-NO_FETCH_BUILD is set, do not allow to fetch code from
+internet by easy_install, and log the help message from
+var-NO_FETCH_BUILD_HELP
+
+Upstream-Status: Inappropriate [oe specific]
+
+Signed-off-by: Hongxu Jia <[email protected]>
+---
+ setuptools/command/easy_install.py | 5 +++++
+ 1 file changed, 5 insertions(+)
+
+diff --git a/setuptools/command/easy_install.py b/setuptools/command/easy_install.py
+index 85ee40f..7eb9d2a 100755
+--- a/setuptools/command/easy_install.py
++++ b/setuptools/command/easy_install.py
+@@ -637,6 +637,11 @@ class easy_install(Command):
+             os.path.exists(tmpdir) and rmtree(rmtree_safe(tmpdir))
+ 
+     def easy_install(self, spec, deps=False):
++        if os.environ.get('NO_FETCH_BUILD', None):
++            log.error("ERROR: Do not try to fetch `%s' for building. %s" %
++                             (spec,  os.environ.get('NO_FETCH_BUILD_HELP', '')))
++            return None
++
+         if not self.editable:
+             self.install_site_py()
+ 
+-- 
+2.7.4
+
meta/recipes-devtools/python/python-setuptools_40.0.0.bb +2 -0
index cf94404..9d9a093 100644
@@ -1,5 +1,7 @@
 require python-setuptools.inc
 
+SRC_URI_append_class-native = " file://0001-conditionally-do-not-fetch-code-by-easy_install.patch"
+
 PROVIDES = "python-distribute"
 
 inherit setuptools
Signature · 1 line
2.8.1
author
#2 lore raw

PATCH 2/2 distutils3/setuptools3-native: do not try to fetch code during do_compile

In reply to Hongxu Jia

If a python recipe is using setuptools and the setup_requires argument, where setuptools will use easy_install to fetch the module if it isn't present.

The build failed on a machine where a proxy was required, but succeeded on a machine which had direct access to the internet

Add var-NO_FETCH_BUILD, and set it in distutils_do_compile which does not allow to fetch code from internet during do_compile.

Example result:
...
ERROR: Do not try to fetch `pytest-runner1' for building. Please add its native recipe to DEPENDS. Traceback (most recent call last):
File "setup.py", line 56, in <module>
...

The improvement is flexible for test_requirements argument (used at `setup.py test'), where use easy_install also.

[YOCTO #12084]

Signed-off-by Hongxu Jia <[email protected]>
---
 meta/classes/distutils3.bbclass                    |  2 ++
 ...ionally-do-not-fetch-code-by-easy_install.patch | 35 ++++++++++++++++++++++
 .../python/python3-setuptools_40.0.0.bb            |  2 ++
 3 files changed, 39 insertions(+)
 create mode 100644 meta/recipes-devtools/python/python3-setuptools/0001-conditionally-do-not-fetch-code-by-easy_install.patch
meta/classes/distutils3.bbclass +2 -0
index d4b92a3..fea7368 100644
@@ -15,6 +15,8 @@ distutils3_do_configure() {
 }
 
 distutils3_do_compile() {
+        NO_FETCH_BUILD=1 \
+        NO_FETCH_BUILD_HELP="Please add its native recipe to DEPENDS." \
         STAGING_INCDIR=${STAGING_INCDIR} \
         STAGING_LIBDIR=${STAGING_LIBDIR} \
         ${STAGING_BINDIR_NATIVE}/${PYTHON_PN}-native/${PYTHON_PN} setup.py \
meta/recipes-devtools/python/python3-setuptools/0001-conditionally-do-not-fetch-code-by-easy_install.patch new file +35 -0
new file mode 100644
index 0000000..441af16
@@ -0,0 +1,35 @@
+From 019422d19c6401fa8f45b1311ecd31bb087c809d Mon Sep 17 00:00:00 2001
+From: Hongxu Jia <[email protected]>
+Date: Tue, 17 Jul 2018 10:13:38 +0800
+Subject: [PATCH] conditionally do not fetch code by easy_install
+
+If var-NO_FETCH_BUILD is set, do not allow to fetch code from
+internet by easy_install, and log the help message from
+var-NO_FETCH_BUILD_HELP
+
+Upstream-Status: Inappropriate [oe specific]
+
+Signed-off-by: Hongxu Jia <[email protected]>
+---
+ setuptools/command/easy_install.py | 5 +++++
+ 1 file changed, 5 insertions(+)
+
+diff --git a/setuptools/command/easy_install.py b/setuptools/command/easy_install.py
+index 85ee40f..7eb9d2a 100755
+--- a/setuptools/command/easy_install.py
++++ b/setuptools/command/easy_install.py
+@@ -637,6 +637,11 @@ class easy_install(Command):
+             os.path.exists(tmpdir) and rmtree(rmtree_safe(tmpdir))
+ 
+     def easy_install(self, spec, deps=False):
++        if os.environ.get('NO_FETCH_BUILD', None):
++            log.error("ERROR: Do not try to fetch `%s' for building. %s" %
++                             (spec,  os.environ.get('NO_FETCH_BUILD_HELP', '')))
++            return None
++
+         if not self.editable:
+             self.install_site_py()
+ 
+-- 
+2.7.4
+
meta/recipes-devtools/python/python3-setuptools_40.0.0.bb +2 -0
index 0dc1ed8..713a1b5 100644
@@ -1,6 +1,8 @@
 require python-setuptools.inc
 inherit setuptools3
 
+SRC_URI_append_class-native = " file://0001-conditionally-do-not-fetch-code-by-easy_install.patch"
+
 do_install_append() {
     mv ${D}${bindir}/easy_install ${D}${bindir}/easy3_install
 }
Signature · 1 line
2.8.1
#3 lore raw

Re: [PATCH 2/2] distutils3/setuptools3-native: do not try to fetch code during do_compile

Quoted 3 lines · Hongxu Jia

distutils3_do_compile() {

meta/classes/distutils3.bbclass +2 -0
+        NO_FETCH_BUILD=1 \
+        NO_FETCH_BUILD_HELP="Please add its native recipe to DEPENDS." \

...

Quoted 4 lines

+ def easy_install(self, spec, deps=False):
++ if os.environ.get('NO_FETCH_BUILD', None):
++ log.error("ERROR: Do not try to fetch `%s' for building. %s" %
++ (spec, os.environ.get('NO_FETCH_BUILD_HELP', '')))

Passing the help to the patch seems overkill considering we're patching in the message in the first place. Why not just put this in the patch itself?

Quoted 1 line

+SRC_URI_append_class-native = " file://0001-conditionally-do-not-fetch-code-by-easy_install.patch"

Do this in setuptools.inc and you won't need to patch both recipes.

Ross

author
#4 lore raw

PATCH v2 1/1 distutils/setuptools-native, distutils3/setuptools3-native: do not try to fetch code during do_compile

If a python recipe is using setuptools and the setup_requires argument, where setuptools will use easy_install to fetch the module if it isn't present.

The build failed on a machine where a proxy was required, but succeeded on a machine which had direct access to the internet

Add var-NO_FETCH_BUILD, and set it in distutils_do_compile which does not allow to fetch code from internet during do_compile.

Example result:
...
ERROR: Do not try to fetch `pytest-runner1' for building. Please add its native recipe to DEPENDS. Traceback (most recent call last):
File "setup.py", line 56, in <module>
...

The improvement is flexible for test_requirements argument (used at `setup.py test'), where use easy_install also.

[YOCTO #12084]
Signed-off-by Hongxu Jia <[email protected]>
---
 meta/classes/distutils.bbclass                     |  1 +
 meta/classes/distutils3.bbclass                    |  1 +
 ...ionally-do-not-fetch-code-by-easy_install.patch | 34 ++++++++++++++++++++++
 meta/recipes-devtools/python/python-setuptools.inc |  2 ++
 4 files changed, 38 insertions(+)
 create mode 100644 meta/recipes-devtools/python/files/0001-conditionally-do-not-fetch-code-by-easy_install.patch
meta/classes/distutils.bbclass +1 -0
index 1930c35..c2f7403 100644
@@ -8,6 +8,7 @@ DISTUTILS_INSTALL_ARGS ?= "--prefix=${D}/${prefix} \
     --install-data=${D}/${datadir}"
 
 distutils_do_compile() {
+         NO_FETCH_BUILD=1 \
          STAGING_INCDIR=${STAGING_INCDIR} \
          STAGING_LIBDIR=${STAGING_LIBDIR} \
          ${STAGING_BINDIR_NATIVE}/${PYTHON_PN}-native/${PYTHON_PN} setup.py build ${DISTUTILS_BUILD_ARGS} || \
meta/classes/distutils3.bbclass +1 -0
index d4b92a3..f8aca5e 100644
@@ -15,6 +15,7 @@ distutils3_do_configure() {
 }
 
 distutils3_do_compile() {
+        NO_FETCH_BUILD=1 \
         STAGING_INCDIR=${STAGING_INCDIR} \
         STAGING_LIBDIR=${STAGING_LIBDIR} \
         ${STAGING_BINDIR_NATIVE}/${PYTHON_PN}-native/${PYTHON_PN} setup.py \
meta/recipes-devtools/python/files/0001-conditionally-do-not-fetch-code-by-easy_install.patch new file +34 -0
new file mode 100644
index 0000000..4b17921
@@ -0,0 +1,34 @@
+From 019422d19c6401fa8f45b1311ecd31bb087c809d Mon Sep 17 00:00:00 2001
+From: Hongxu Jia <[email protected]>
+Date: Tue, 17 Jul 2018 10:13:38 +0800
+Subject: [PATCH] conditionally do not fetch code by easy_install
+
+If var-NO_FETCH_BUILD is set, do not allow to fetch code from
+internet by easy_install.
+
+Upstream-Status: Inappropriate [oe specific]
+
+Signed-off-by: Hongxu Jia <[email protected]>
+---
+ setuptools/command/easy_install.py | 5 +++++
+ 1 file changed, 5 insertions(+)
+
+diff --git a/setuptools/command/easy_install.py b/setuptools/command/easy_install.py
+index 85ee40f..7eb9d2a 100755
+--- a/setuptools/command/easy_install.py
++++ b/setuptools/command/easy_install.py
+@@ -637,6 +637,11 @@ class easy_install(Command):
+             os.path.exists(tmpdir) and rmtree(rmtree_safe(tmpdir))
+ 
+     def easy_install(self, spec, deps=False):
++        if os.environ.get('NO_FETCH_BUILD', None):
++            log.error("ERROR: Do not try to fetch `%s' for building. "
++                      "Please add its native recipe to DEPENDS." % spec)
++            return None
++
+         if not self.editable:
+             self.install_site_py()
+ 
+-- 
+2.7.4
+
meta/recipes-devtools/python/python-setuptools.inc +2 -0
index 25be446..24b3ac4 100644
@@ -9,6 +9,8 @@ PYPI_PACKAGE_EXT = "zip"
 
 inherit pypi
 
+SRC_URI_append_class-native = " file://0001-conditionally-do-not-fetch-code-by-easy_install.patch"
+
 SRC_URI[md5sum] = "260630ae1a64bafa39dcc53404d63829"
 SRC_URI[sha256sum] = "012adb8e25fbfd64c652e99e7bab58799a3aaf05d39ab38561f69190a909015f"
Signature · 1 line
2.7.4
#5 lore raw

Re: [PATCH V2] distutils/setuptools-native, distutils3/setuptools3-native: do not try to fetch code during do_compile

I like this change, some recipes in meta-oe seems to be broken now, is someone already working on a fix?

I'm seeing following 5 failures in my world builds:

python-stevedore-1.28.0-r0 do_compile and python3-stevedore-1.28.0-r0 do_compile
pkg_resources.DistributionNotFound: The 'pbr>=2.0.0' distribution was not found and is required by the application

python-dateutil-2.7.3-r0 do_compile and python3-dateutil-2.7.3-r0 do_compile pkg_resources.DistributionNotFound: The 'setuptools_scm' distribution was not found and is required by the application

python-daemon-2.1.2-r0 do_compile pkg_resources.DistributionNotFound: The 'docutils' distribution was not found and is required by the application

I've found a fix for python-dateutils from Derek:
https://patchwork.openembedded.org/patch/153002/

Quoted 132 lines · Hongxu Jia

If a python recipe is using setuptools and the setup_requires argument, where
setuptools will use easy_install to fetch the module if it isn't present.

The build failed on a machine where a proxy was required, but succeeded on a
machine which had direct access to the internet

Add var-NO_FETCH_BUILD, and set it in distutils_do_compile which does not allow to fetch code from internet during do_compile.

Example result:
...
ERROR: Do not try to fetch `pytest-runner1' for building. Please add its native recipe to DEPENDS.
Traceback (most recent call last):
File "setup.py", line 56, in <module>
...

The improvement is flexible for test_requirements argument (used at `setup.py test'), where use easy_install also.

[YOCTO #12084]
Signed-off-by Hongxu Jia <[email protected]>
---
 meta/classes/distutils.bbclass                     |  1 +
 meta/classes/distutils3.bbclass                    |  1 +
 ...ionally-do-not-fetch-code-by-easy_install.patch | 34
++++++++++++++++++++++
 meta/recipes-devtools/python/python-setuptools.inc |  2 ++
 4 files changed, 38 insertions(+)
 create mode 100644

meta/recipes-devtools/python/files/0001-conditionally-do-not-fetch-code-by-easy_install.patch

meta/classes/distutils.bbclass +1 -0
index 1930c35..c2f7403 100644
@@ -8,6 +8,7 @@ DISTUTILS_INSTALL_ARGS ?= "--prefix=${D}/${prefix} \
     --install-data=${D}/${datadir}"
 
 distutils_do_compile() {
+         NO_FETCH_BUILD=1 \
          STAGING_INCDIR=${STAGING_INCDIR} \
          STAGING_LIBDIR=${STAGING_LIBDIR} \
          ${STAGING_BINDIR_NATIVE}/${PYTHON_PN}-native/${PYTHON_PN}

setup.py build ${DISTUTILS_BUILD_ARGS} || \

meta/classes/distutils3.bbclass +1 -0
index d4b92a3..f8aca5e 100644
@@ -15,6 +15,7 @@ distutils3_do_configure() {
 }
 
 distutils3_do_compile() {
+        NO_FETCH_BUILD=1 \
         STAGING_INCDIR=${STAGING_INCDIR} \
         STAGING_LIBDIR=${STAGING_LIBDIR} \
         ${STAGING_BINDIR_NATIVE}/${PYTHON_PN}-native/${PYTHON_PN}

setup.py \
diff --git a/meta/recipes-devtools/python/files/0001-conditionally-do-not-fetch-code-by-easy_install.patch b/meta/recipes-devtools/python/files/0001-conditionally-do-not-fetch-code-by-easy_install.patch new file mode 100644
index 0000000..4b17921
--- /dev/null
+++
b/meta/recipes-devtools/python/files/0001-conditionally-do-not-fetch-code-by-easy_install.patch

meta/classes/distutils3.bbclass +16 -0
@@ -0,0 +1,34 @@
+From 019422d19c6401fa8f45b1311ecd31bb087c809d Mon Sep 17 00:00:00 2001
+From: Hongxu Jia <[email protected]>
+Date: Tue, 17 Jul 2018 10:13:38 +0800
+Subject: [PATCH] conditionally do not fetch code by easy_install
+
+If var-NO_FETCH_BUILD is set, do not allow to fetch code from
+internet by easy_install.
+
+Upstream-Status: Inappropriate [oe specific]
+
+Signed-off-by: Hongxu Jia <[email protected]>
+---
+ setuptools/command/easy_install.py | 5 +++++
+ 1 file changed, 5 insertions(+)
+
+diff --git a/setuptools/command/easy_install.py

b/setuptools/command/easy_install.py

meta/classes/distutils3.bbclass +18 -0
+index 85ee40f..7eb9d2a 100755
+--- a/setuptools/command/easy_install.py
++++ b/setuptools/command/easy_install.py
+@@ -637,6 +637,11 @@ class easy_install(Command):
+             os.path.exists(tmpdir) and rmtree(rmtree_safe(tmpdir))
+
+     def easy_install(self, spec, deps=False):
++        if os.environ.get('NO_FETCH_BUILD', None):
++            log.error("ERROR: Do not try to fetch `%s' for building. "
++                      "Please add its native recipe to DEPENDS." % spec)
++            return None
++
+         if not self.editable:
+             self.install_site_py()
+
+--
+2.7.4
+
meta/recipes-devtools/python/python-setuptools.inc +1 -0
index 25be446..24b3ac4 100644
@@ -9,6 +9,8 @@ PYPI_PACKAGE_EXT = "zip"
 
 inherit pypi
 
+SRC_URI_append_class-native = "

file://0001-conditionally-do-not-fetch-code-by-easy_install.patch" +
SRC_URI[md5sum] = "260630ae1a64bafa39dcc53404d63829" SRC_URI[sha256sum] =
"012adb8e25fbfd64c652e99e7bab58799a3aaf05d39ab38561f69190a909015f"

--
2.7.4

--
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-core

#6 lore raw

Re: [PATCH V2] distutils/setuptools-native, distutils3/setuptools3-native: do not try to fetch code during do_compile

Yes. I was just waiting for my master build to wrap up.

Quoted 158 lines · Martin Jansa

I like this change, some recipes in meta-oe seems to be broken now, is someone already working on a fix?

I'm seeing following 5 failures in my world builds:

python-stevedore-1.28.0-r0 do_compile and python3-stevedore-1.28.0-r0 do_compile
pkg_resources.DistributionNotFound: The 'pbr>=2.0.0' distribution was not found and is required by the application

python-dateutil-2.7.3-r0 do_compile and python3-dateutil-2.7.3-r0 do_compile
pkg_resources.DistributionNotFound: The 'setuptools_scm' distribution was not found and is required by the application

python-daemon-2.1.2-r0 do_compile pkg_resources.DistributionNotFound: The 'docutils' distribution was not found and is required by the application

I've found a fix for python-dateutils from Derek:
https://patchwork.openembedded.org/patch/153002/

Quoted 132 lines · Hongxu Jia

If a python recipe is using setuptools and the setup_requires argument, where
setuptools will use easy_install to fetch the module if it isn't present.

The build failed on a machine where a proxy was required, but succeeded on a
machine which had direct access to the internet

Add var-NO_FETCH_BUILD, and set it in distutils_do_compile which does not allow to fetch code from internet during do_compile.

Example result:
...
ERROR: Do not try to fetch `pytest-runner1' for building. Please add its native recipe to DEPENDS.
Traceback (most recent call last):
File "setup.py", line 56, in <module>
...

The improvement is flexible for test_requirements argument (used at `setup.py test'), where use easy_install also.

[YOCTO #12084]
Signed-off-by Hongxu Jia <[email protected]>
---
 meta/classes/distutils.bbclass                     |  1 +
 meta/classes/distutils3.bbclass                    |  1 +
 ...ionally-do-not-fetch-code-by-easy_install.patch | 34
++++++++++++++++++++++
 meta/recipes-devtools/python/python-setuptools.inc |  2 ++
 4 files changed, 38 insertions(+)
 create mode 100644

meta/recipes-devtools/python/files/0001-conditionally-do-not-fetch-code-by-easy_install.patch

meta/classes/distutils.bbclass +1 -0
index 1930c35..c2f7403 100644
@@ -8,6 +8,7 @@ DISTUTILS_INSTALL_ARGS ?= "--prefix=${D}/${prefix} \
     --install-data=${D}/${datadir}"
 
 distutils_do_compile() {
+         NO_FETCH_BUILD=1 \
          STAGING_INCDIR=${STAGING_INCDIR} \
          STAGING_LIBDIR=${STAGING_LIBDIR} \
          ${STAGING_BINDIR_NATIVE}/${PYTHON_PN}-native/${PYTHON_PN}

setup.py build ${DISTUTILS_BUILD_ARGS} || \

meta/classes/distutils3.bbclass +1 -0
index d4b92a3..f8aca5e 100644
@@ -15,6 +15,7 @@ distutils3_do_configure() {
 }
 
 distutils3_do_compile() {
+        NO_FETCH_BUILD=1 \
         STAGING_INCDIR=${STAGING_INCDIR} \
         STAGING_LIBDIR=${STAGING_LIBDIR} \
         ${STAGING_BINDIR_NATIVE}/${PYTHON_PN}-native/${PYTHON_PN}

setup.py \
diff --git a/meta/recipes-devtools/python/files/0001-conditionally-do-not-fetch-code-by-easy_install.patch b/meta/recipes-devtools/python/files/0001-conditionally-do-not-fetch-code-by-easy_install.patch new file mode 100644
index 0000000..4b17921
--- /dev/null
+++
b/meta/recipes-devtools/python/files/0001-conditionally-do-not-fetch-code-by-easy_install.patch

meta/classes/distutils3.bbclass +16 -0
@@ -0,0 +1,34 @@
+From 019422d19c6401fa8f45b1311ecd31bb087c809d Mon Sep 17 00:00:00 2001
+From: Hongxu Jia <[email protected]>
+Date: Tue, 17 Jul 2018 10:13:38 +0800
+Subject: [PATCH] conditionally do not fetch code by easy_install
+
+If var-NO_FETCH_BUILD is set, do not allow to fetch code from
+internet by easy_install.
+
+Upstream-Status: Inappropriate [oe specific]
+
+Signed-off-by: Hongxu Jia <[email protected]>
+---
+ setuptools/command/easy_install.py | 5 +++++
+ 1 file changed, 5 insertions(+)
+
+diff --git a/setuptools/command/easy_install.py

b/setuptools/command/easy_install.py

meta/classes/distutils3.bbclass +18 -0
+index 85ee40f..7eb9d2a 100755
+--- a/setuptools/command/easy_install.py
++++ b/setuptools/command/easy_install.py
+@@ -637,6 +637,11 @@ class easy_install(Command):
+             os.path.exists(tmpdir) and rmtree(rmtree_safe(tmpdir))
+
+     def easy_install(self, spec, deps=False):
++        if os.environ.get('NO_FETCH_BUILD', None):
++            log.error("ERROR: Do not try to fetch `%s' for building. "
++                      "Please add its native recipe to DEPENDS." % spec)
++            return None
++
+         if not self.editable:
+             self.install_site_py()
+
+--
+2.7.4
+
meta/recipes-devtools/python/python-setuptools.inc +1 -0
index 25be446..24b3ac4 100644
@@ -9,6 +9,8 @@ PYPI_PACKAGE_EXT = "zip"
 
 inherit pypi
 
+SRC_URI_append_class-native = "

file://0001-conditionally-do-not-fetch-code-by-easy_install.patch" +
SRC_URI[md5sum] = "260630ae1a64bafa39dcc53404d63829" SRC_URI[sha256sum] =
"012adb8e25fbfd64c652e99e7bab58799a3aaf05d39ab38561f69190a909015f"

--
2.7.4

--
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-core

#7 lore raw

Re: [PATCH V2] distutils/setuptools-native, distutils3/setuptools3-native: do not try to fetch code during do_compile

python-daemon - https://patchwork.openembedded.org/patch/153034/

python-stevedore - https://patchwork.openembedded.org/patch/153032/

I think that's everything though my world python build is still going. Thanks for pointing the failures out and let me know if there are others. I'm guessing this will ripple through the python heavy layers; I found a couple yesterday in meta-iot-cloud.

Quoted 164 lines · Derek Straka

Yes. I was just waiting for my master build to wrap up.

Quoted 160 lines · Martin Jansa

I like this change, some recipes in meta-oe seems to be broken now, is someone already working on a fix?

I'm seeing following 5 failures in my world builds:

python-stevedore-1.28.0-r0 do_compile and python3-stevedore-1.28.0-r0 do_compile
pkg_resources.DistributionNotFound: The 'pbr>=2.0.0' distribution was not found and is required by the application

python-dateutil-2.7.3-r0 do_compile and python3-dateutil-2.7.3-r0 do_compile
pkg_resources.DistributionNotFound: The 'setuptools_scm' distribution was not found and is required by the application

python-daemon-2.1.2-r0 do_compile pkg_resources.DistributionNotFound: The 'docutils' distribution was not found and is required by the application

I've found a fix for python-dateutils from Derek:
https://patchwork.openembedded.org/patch/153002/

Quoted 134 lines · Hongxu Jia

If a python recipe is using setuptools and the setup_requires argument, where
setuptools will use easy_install to fetch the module if it isn't present.

The build failed on a machine where a proxy was required, but succeeded on a
machine which had direct access to the internet

Add var-NO_FETCH_BUILD, and set it in distutils_do_compile which does not allow to fetch code from internet during do_compile.

Example result:
...
ERROR: Do not try to fetch `pytest-runner1' for building. Please add its native recipe to DEPENDS.
Traceback (most recent call last):
File "setup.py", line 56, in <module>
...

The improvement is flexible for test_requirements argument (used at `setup.py test'), where use easy_install also.

[YOCTO #12084]
Signed-off-by Hongxu Jia <[email protected]>
---
 meta/classes/distutils.bbclass                     |  1 +
 meta/classes/distutils3.bbclass                    |  1 +
 ...ionally-do-not-fetch-code-by-easy_install.patch | 34
++++++++++++++++++++++
 meta/recipes-devtools/python/python-setuptools.inc |  2 ++
 4 files changed, 38 insertions(+)
 create mode 100644 meta/recipes-devtools/python/

files/0001-conditionally-do-not-fetch-code-by-easy_install.patch

meta/classes/distutils. renamed from meta/classes/distutils.bbclass +0 -0

bbclass
index 1930c35..c2f7403 100644

meta/classes/distutils.bbclass +1 -0
@@ -8,6 +8,7 @@ DISTUTILS_INSTALL_ARGS ?= "--prefix=${D}/${prefix} \
     --install-data=${D}/${datadir}"
 
 distutils_do_compile() {
+         NO_FETCH_BUILD=1 \
          STAGING_INCDIR=${STAGING_INCDIR} \
          STAGING_LIBDIR=${STAGING_LIBDIR} \
          ${STAGING_BINDIR_NATIVE}/${PYTHON_PN}-native/${PYTHON_PN}

setup.py build ${DISTUTILS_BUILD_ARGS} || \

meta/classes/distutils3. renamed from meta/classes/distutils3.bbclass +0 -0

bbclass
index d4b92a3..f8aca5e 100644

meta/classes/distutils3.bbclass +1 -0
@@ -15,6 +15,7 @@ distutils3_do_configure() {
 }
 
 distutils3_do_compile() {
+        NO_FETCH_BUILD=1 \
         STAGING_INCDIR=${STAGING_INCDIR} \
         STAGING_LIBDIR=${STAGING_LIBDIR} \
         ${STAGING_BINDIR_NATIVE}/${PYTHON_PN}-native/${PYTHON_PN}

setup.py \

(quoted diff) +0 -0

conditionally-do-not-fetch-code-by-easy_install.patch b/meta/recipes-devtools/python/files/0001-conditionally-do-not-fetch- code-by-easy_install.patch
new file mode 100644
index 0000000..4b17921

meta/recipes-devtools/python/files/0001- new file +0 -0

conditionally-do-not-fetch-code-by-easy_install.patch

meta/recipes-devtools/python/files/0001- new file +16 -0
@@ -0,0 +1,34 @@
+From 019422d19c6401fa8f45b1311ecd31bb087c809d Mon Sep 17 00:00:00 2001
+From: Hongxu Jia <[email protected]>
+Date: Tue, 17 Jul 2018 10:13:38 +0800
+Subject: [PATCH] conditionally do not fetch code by easy_install
+
+If var-NO_FETCH_BUILD is set, do not allow to fetch code from
+internet by easy_install.
+
+Upstream-Status: Inappropriate [oe specific]
+
+Signed-off-by: Hongxu Jia <[email protected]>
+---
+ setuptools/command/easy_install.py | 5 +++++
+ 1 file changed, 5 insertions(+)
+
+diff --git a/setuptools/command/easy_install.py

b/setuptools/command/easy_install.py

meta/recipes-devtools/python/files/0001- new file +10 -0
+index 85ee40f..7eb9d2a 100755
+--- a/setuptools/command/easy_install.py
++++ b/setuptools/command/easy_install.py
+@@ -637,6 +637,11 @@ class easy_install(Command):
+             os.path.exists(tmpdir) and rmtree(rmtree_safe(tmpdir))
+
+     def easy_install(self, spec, deps=False):
++        if os.environ.get('NO_FETCH_BUILD', None):
++            log.error("ERROR: Do not try to fetch `%s' for building. "
++                      "Please add its native recipe to DEPENDS." %

spec)
++ return None
++

meta/recipes-devtools/python/files/0001- new file +6 -0
+         if not self.editable:
+             self.install_site_py()
+
+--
+2.7.4
+
meta/recipes-devtools/python/python-setuptools.inc +1 -0
index 25be446..24b3ac4 100644
@@ -9,6 +9,8 @@ PYPI_PACKAGE_EXT = "zip"
 
 inherit pypi
 
+SRC_URI_append_class-native = " file://0001-conditionally-do-

not-fetch-code-by-easy_install.patch"
+
SRC_URI[md5sum] = "260630ae1a64bafa39dcc53404d63829" SRC_URI[sha256sum] = "012adb8e25fbfd64c652e99e7bab58
799a3aaf05d39ab38561f69190a909015f"

--
2.7.4

--
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-core

#8 lore raw

Re: [PATCH V2] distutils/setuptools-native, distutils3/setuptools3-native: do not try to fetch code during do_compile

Thanks a lot.

Quoted 177 lines · Derek Straka

python-daemon - https://patchwork.openembedded.org/patch/153034/

python-stevedore - https://patchwork.openembedded.org/patch/153032/

I think that's everything though my world python build is still going. Thanks for pointing the failures out and let me know if there are others. I'm guessing this will ripple through the python heavy layers; I found a couple yesterday in meta-iot-cloud.

Quoted 165 lines · Derek Straka

Yes. I was just waiting for my master build to wrap up.

Quoted 160 lines · Martin Jansa

I like this change, some recipes in meta-oe seems to be broken now, is someone already working on a fix?

I'm seeing following 5 failures in my world builds:

python-stevedore-1.28.0-r0 do_compile and python3-stevedore-1.28.0-r0 do_compile
pkg_resources.DistributionNotFound: The 'pbr>=2.0.0' distribution was not found and is required by the application

python-dateutil-2.7.3-r0 do_compile and python3-dateutil-2.7.3-r0 do_compile
pkg_resources.DistributionNotFound: The 'setuptools_scm' distribution was not found and is required by the application

python-daemon-2.1.2-r0 do_compile pkg_resources.DistributionNotFound: The 'docutils' distribution was not found and is required by the application

I've found a fix for python-dateutils from Derek:
https://patchwork.openembedded.org/patch/153002/

Quoted 134 lines · Hongxu Jia

If a python recipe is using setuptools and the setup_requires argument, where
setuptools will use easy_install to fetch the module if it isn't present.

The build failed on a machine where a proxy was required, but succeeded on a
machine which had direct access to the internet

Add var-NO_FETCH_BUILD, and set it in distutils_do_compile which does not
allow to fetch code from internet during do_compile.

Example result:
...
ERROR: Do not try to fetch `pytest-runner1' for building. Please add its native recipe to DEPENDS.
Traceback (most recent call last):
File "setup.py", line 56, in <module>
...

The improvement is flexible for test_requirements argument (used at `setup.py test'), where use easy_install also.

[YOCTO #12084]
Signed-off-by Hongxu Jia <[email protected]>
---
 meta/classes/distutils.bbclass                     |  1 +
 meta/classes/distutils3.bbclass                    |  1 +
 ...ionally-do-not-fetch-code-by-easy_install.patch | 34
++++++++++++++++++++++
 meta/recipes-devtools/python/python-setuptools.inc |  2 ++
 4 files changed, 38 insertions(+)
 create mode 100644

meta/recipes-devtools/python/files/0001-conditionally-do-not-fetch-code-by-easy_install.patch

meta/classes/distutils.bbclass +1 -0
index 1930c35..c2f7403 100644
@@ -8,6 +8,7 @@ DISTUTILS_INSTALL_ARGS ?= "--prefix=${D}/${prefix} \
     --install-data=${D}/${datadir}"
 
 distutils_do_compile() {
+         NO_FETCH_BUILD=1 \
          STAGING_INCDIR=${STAGING_INCDIR} \
          STAGING_LIBDIR=${STAGING_LIBDIR} \
          ${STAGING_BINDIR_NATIVE}/${PYTHON_PN}-native/${PYTHON_PN}

setup.py build ${DISTUTILS_BUILD_ARGS} || \

meta/classes/distutils3.bbclass +1 -0
index d4b92a3..f8aca5e 100644
@@ -15,6 +15,7 @@ distutils3_do_configure() {
 }
 
 distutils3_do_compile() {
+        NO_FETCH_BUILD=1 \
         STAGING_INCDIR=${STAGING_INCDIR} \
         STAGING_LIBDIR=${STAGING_LIBDIR} \
         ${STAGING_BINDIR_NATIVE}/${PYTHON_PN}-native/${PYTHON_PN}

setup.py \
diff --git a/meta/recipes-devtools/python/files/0001-conditionally-do-not-fetch-code-by-easy_install.patch b/meta/recipes-devtools/python/files/0001-conditionally-do-not-fetch-code-by-easy_install.patch new file mode 100644
index 0000000..4b17921
--- /dev/null
+++
b/meta/recipes-devtools/python/files/0001-conditionally-do-not-fetch-code-by-easy_install.patch

meta/classes/distutils3.bbclass +16 -0
@@ -0,0 +1,34 @@
+From 019422d19c6401fa8f45b1311ecd31bb087c809d Mon Sep 17 00:00:00 2001
+From: Hongxu Jia <[email protected]>
+Date: Tue, 17 Jul 2018 10:13:38 +0800
+Subject: [PATCH] conditionally do not fetch code by easy_install
+
+If var-NO_FETCH_BUILD is set, do not allow to fetch code from
+internet by easy_install.
+
+Upstream-Status: Inappropriate [oe specific]
+
+Signed-off-by: Hongxu Jia <[email protected]>
+---
+ setuptools/command/easy_install.py | 5 +++++
+ 1 file changed, 5 insertions(+)
+
+diff --git a/setuptools/command/easy_install.py

b/setuptools/command/easy_install.py

meta/classes/distutils3.bbclass +10 -0
+index 85ee40f..7eb9d2a 100755
+--- a/setuptools/command/easy_install.py
++++ b/setuptools/command/easy_install.py
+@@ -637,6 +637,11 @@ class easy_install(Command):
+             os.path.exists(tmpdir) and rmtree(rmtree_safe(tmpdir))
+
+     def easy_install(self, spec, deps=False):
++        if os.environ.get('NO_FETCH_BUILD', None):
++            log.error("ERROR: Do not try to fetch `%s' for building. "
++                      "Please add its native recipe to DEPENDS." %

spec)
++ return None
++

meta/classes/distutils3.bbclass +6 -0
+         if not self.editable:
+             self.install_site_py()
+
+--
+2.7.4
+
meta/recipes-devtools/python/python-setuptools.inc +1 -0
index 25be446..24b3ac4 100644
@@ -9,6 +9,8 @@ PYPI_PACKAGE_EXT = "zip"
 
 inherit pypi
 
+SRC_URI_append_class-native = "

file://0001-conditionally-do-not-fetch-code-by-easy_install.patch" +
SRC_URI[md5sum] = "260630ae1a64bafa39dcc53404d63829" SRC_URI[sha256sum] =
"012adb8e25fbfd64c652e99e7bab58799a3aaf05d39ab38561f69190a909015f"

--
2.7.4

--
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-core