Skip to contentSkip to conversation

python*-setuptools: add separate packages for pkg_resources module

PATCH Superseded [email protected] 4 participants 11 messages May 16, 2019 – May 25, 2019

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 python*-setuptools: add separate packages for pkg_resources module

From: Luca Boccassi <[email protected]>

The pkg_resources Python module is useful by itself, for example for automatic loading of resources shipped in a Python package. Add separate packages for it, so that users can depend on them individually and avoid pulling in the entire setuptools, which include scripts to download other packages, which might not be desired on minimal images.

Other distributions like Debian and Ubuntu already split setuptools and pkg-resources in this way.

The setuptools packages now depend on the new pkg-resources packages, to avoid regressions for other packages that depend on them already.

---
 meta/recipes-devtools/python/python-setuptools.inc | 4 ++++
 1 file changed, 4 insertions(+)
meta/recipes-devtools/python/python-setuptools.inc +4 -0
index 357aa07086..ad00cbe0ef 100644
@@ -24,6 +24,7 @@ RDEPENDS_${PN} = "\
   ${PYTHON_PN}-html \
   ${PYTHON_PN}-netserver \
   ${PYTHON_PN}-numbers \
+  ${PYTHON_PN}-pkg-resources \
   ${PYTHON_PN}-pkgutil \
   ${PYTHON_PN}-plistlib \
   ${PYTHON_PN}-shell \
@@ -37,3 +38,6 @@ do_install_prepend() {
 }
 
 BBCLASSEXTEND = "native nativesdk"
+
+PACKAGES =+ "${PYTHON_PN}-pkg-resources "
+FILES_${PYTHON_PN}-pkg-resources = "${PYTHON_SITEPACKAGES_DIR}/pkg_resources/*"
Signature · 1 line
2.20.1
#1 lore raw
Quoted 41 lines · On Thu, 2019-05-16 at 22:09 +0100, [email protected] wrote:

From: Luca Boccassi <[email protected]>

The pkg_resources Python module is useful by itself, for example for automatic loading of resources shipped in a Python package. Add separate packages for it, so that users can depend on them individually and avoid pulling in the entire setuptools, which include scripts to download other packages, which might not be desired on minimal images.

Other distributions like Debian and Ubuntu already split setuptools and pkg-resources in this way.

The setuptools packages now depend on the new pkg-resources packages, to avoid regressions for other packages that depend on them already.

---
 meta/recipes-devtools/python/python-setuptools.inc | 4 ++++
 1 file changed, 4 insertions(+)
meta/recipes-devtools/python/python-setuptools.inc +4 -0
index 357aa07086..ad00cbe0ef 100644
@@ -24,6 +24,7 @@ RDEPENDS_${PN} = "\
   ${PYTHON_PN}-html \
   ${PYTHON_PN}-netserver \
   ${PYTHON_PN}-numbers \
+  ${PYTHON_PN}-pkg-resources \
   ${PYTHON_PN}-pkgutil \
   ${PYTHON_PN}-plistlib \
   ${PYTHON_PN}-shell \
@@ -37,3 +38,6 @@ do_install_prepend() {
 }
 
 BBCLASSEXTEND = "native nativesdk"
+
+PACKAGES =+ "${PYTHON_PN}-pkg-resources "
+FILES_${PYTHON_PN}-pkg-resources =
Quoted 1 line

"${PYTHON_SITEPACKAGES_DIR}/pkg_resources/*"

This breaks native builds:

https://autobuilder.yoctoproject.org/typhoon/#/builders/65/builds/614

(and many other similar failures)

Cheers,

Richard

#2 lore raw
Quoted 55 lines · Richard Purdie
Quoted 47 lines · On Thu, 2019-05-16 at 22:09 +0100, [email protected] wrote:

From: Luca Boccassi <
[email protected]

Quoted 1 line

The pkg_resources Python module is useful by itself, for example for
automatic loading of resources shipped in a Python package. Add separate packages for it, so that users can depend on them individually and avoid pulling in the entire setuptools, which include scripts to download other packages, which might not be desired on minimal images.

Other distributions like Debian and Ubuntu already split setuptools and pkg-resources in this way.

The setuptools packages now depend on the new pkg-resources packages,
to avoid regressions for other packages that depend on them already.

Signed-off-by: Luca Boccassi <
[email protected]

Quoted 1 line
---
 meta/recipes-devtools/python/python-setuptools.inc | 4 ++++
 1 file changed, 4 insertions(+)
meta/recipes-devtools/python/python-setuptools.inc +4 -0
index 357aa07086..ad00cbe0ef 100644
@@ -24,6 +24,7 @@ RDEPENDS_${PN} = "\
   ${PYTHON_PN}-html \
   ${PYTHON_PN}-netserver \
   ${PYTHON_PN}-numbers \
+  ${PYTHON_PN}-pkg-resources \
   ${PYTHON_PN}-pkgutil \
   ${PYTHON_PN}-plistlib \
   ${PYTHON_PN}-shell \
@@ -37,3 +38,6 @@ do_install_prepend() {
 }
 
 BBCLASSEXTEND = "native nativesdk"
+
+PACKAGES =+ "${PYTHON_PN}-pkg-resources "
+FILES_${PYTHON_PN}-pkg-resources =
Quoted 1 line

"${PYTHON_SITEPACKAGES_DIR}/pkg_resources/*"

This breaks native builds:

Quoted 8 lines

https://autobuilder.yoctoproject.org/typhoon/#/builders/65/builds/614

(and many other similar failures)

Cheers,

Richard

Hello Richard,

Thanks for the heads-up, any suggestion as to why the above patch
wouldn't be enough to build both native and target binary packages for $python-pkg-resources?

Signature · 2 lines
Kind regards,
Luca Boccassi
#3 lore raw

I think you need to reset RDEPENDS to an empty string for the native class. This is a long-standing problem, where bitbake is not able to fulfil RDEPENDS for native variants, if said RDEPENDS contains anything except recipe names.

Alex

Quoted 80 lines · Luca Boccassi
Quoted 55 lines · Richard Purdie
Quoted 47 lines · On Thu, 2019-05-16 at 22:09 +0100, [email protected] wrote:

From: Luca Boccassi <
[email protected]

Quoted 1 line

The pkg_resources Python module is useful by itself, for example for
automatic loading of resources shipped in a Python package. Add separate packages for it, so that users can depend on them individually and avoid pulling in the entire setuptools, which include scripts to download other packages, which might not be desired on minimal images.

Other distributions like Debian and Ubuntu already split setuptools and pkg-resources in this way.

The setuptools packages now depend on the new pkg-resources packages,
to avoid regressions for other packages that depend on them already.

Signed-off-by: Luca Boccassi <
[email protected]

Quoted 1 line
---
 meta/recipes-devtools/python/python-setuptools.inc | 4 ++++
 1 file changed, 4 insertions(+)
meta/recipes-devtools/python/python-setuptools.inc +4 -0
index 357aa07086..ad00cbe0ef 100644
@@ -24,6 +24,7 @@ RDEPENDS_${PN} = "\
   ${PYTHON_PN}-html \
   ${PYTHON_PN}-netserver \
   ${PYTHON_PN}-numbers \
+  ${PYTHON_PN}-pkg-resources \
   ${PYTHON_PN}-pkgutil \
   ${PYTHON_PN}-plistlib \
   ${PYTHON_PN}-shell \
@@ -37,3 +38,6 @@ do_install_prepend() {
 }
 
 BBCLASSEXTEND = "native nativesdk"
+
+PACKAGES =+ "${PYTHON_PN}-pkg-resources "
+FILES_${PYTHON_PN}-pkg-resources =
Quoted 1 line

"${PYTHON_SITEPACKAGES_DIR}/pkg_resources/*"

This breaks native builds:

Quoted 8 lines

https://autobuilder.yoctoproject.org/typhoon/#/builders/65/builds/614

(and many other similar failures)

Cheers,

Richard

Hello Richard,

Thanks for the heads-up, any suggestion as to why the above patch
wouldn't be enough to build both native and target binary packages for $python-pkg-resources?

--
Kind regards,
Luca Boccassi

--
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-core

#4 lore raw

PATCH v2 1/1 python*-setuptools: add separate packages for pkg_resources module

In reply to [email protected]

From: Luca Boccassi <[email protected]>

The pkg_resources Python module is useful by itself, for example for automatic loading of resources shipped in a Python package. Add separate packages for it, so that users can depend on them individually and avoid pulling in the entire setuptools, which include scripts to download other packages, which might not be desired on minimal images.

Other distributions like Debian and Ubuntu already split setuptools and pkg-resources in this way.

The setuptools packages now depend on the new pkg-resources packages, to avoid regressions for other packages that depend on them already.

---
v2: restrict new RDEPENDS to class-target. As advised by Alexander, bitbake
cannot resolve native rdeps that mention package names rather than recipe names.

 meta/recipes-devtools/python/python-setuptools.inc | 6 ++++++
 1 file changed, 6 insertions(+)
meta/recipes-devtools/python/python-setuptools.inc +6 -0
index 357aa07086..c5dcee7059 100644
@@ -37,3 +37,9 @@ do_install_prepend() {
 }
 
 BBCLASSEXTEND = "native nativesdk"
+
+PACKAGES =+ "${PYTHON_PN}-pkg-resources "
+FILES_${PYTHON_PN}-pkg-resources = "${PYTHON_SITEPACKAGES_DIR}/pkg_resources/*"
+# Due to a bitbake bug, the native class cannot have a dependency on something
+# that is not a recipe name.
+RDEPENDS_${PN}_append_class-target = " ${PYTHON_PN}-pkg-resources"
Signature · 1 line
2.20.1
#5 lore raw
Quoted 42 lines · On Tue, 2019-05-21 at 11:59 +0100, [email protected] wrote:

From: Luca Boccassi <[email protected]>

The pkg_resources Python module is useful by itself, for example for automatic loading of resources shipped in a Python package. Add separate packages for it, so that users can depend on them individually and avoid pulling in the entire setuptools, which include scripts to download other packages, which might not be desired on minimal images.

Other distributions like Debian and Ubuntu already split setuptools and pkg-resources in this way.

The setuptools packages now depend on the new pkg-resources packages, to avoid regressions for other packages that depend on them already.

---
v2: restrict new RDEPENDS to class-target. As advised by Alexander, bitbake
cannot resolve native rdeps that mention package names rather
than
recipe names.

 meta/recipes-devtools/python/python-setuptools.inc | 6 ++++++
 1 file changed, 6 insertions(+)
meta/recipes-devtools/python/python-setuptools.inc +3 -0
index 357aa07086..c5dcee7059 100644
@@ -37,3 +37,9 @@ do_install_prepend() {
 }
 
 BBCLASSEXTEND = "native nativesdk"
+
+PACKAGES =+ "${PYTHON_PN}-pkg-resources "
+FILES_${PYTHON_PN}-pkg-resources =

"${PYTHON_SITEPACKAGES_DIR}/pkg_resources/*"
+# Due to a bitbake bug, the native class cannot have a dependency on something

meta/recipes-devtools/python/python-setuptools.inc do_install_prepend() { · line 45

Quoted 2 lines
meta/recipes-devtools/python/python-setuptools.inc +2 -0
+# that is not a recipe name.
+RDEPENDS_${PN}_append_class-target = " ${PYTHON_PN}-pkg-resources"

Its not a bitbake bug as such but the way OE-Core decided to implement native recipes.

I'm guessing but the best thing to do here is probably add:

RPROVIDES_append_class-native = " ${PYTHON_PN}-pkg-resources-native"

which is slightly cleaner than the target change above (I'm left wondering if nativesdk works in that case).

Cheers,

Richard

#6 lore raw

PATCH v4 1/1 python*-setuptools: add separate packages for pkg_resources module

In reply to [email protected]

From: Luca Boccassi <[email protected]>

The pkg_resources Python module is useful by itself, for example for automatic loading of resources shipped in a Python package. Add separate packages for it, so that users can depend on them individually and avoid pulling in the entire setuptools, which include scripts to download other packages, which might not be desired on minimal images.

Other distributions like Debian and Ubuntu already split setuptools and pkg-resources in this way.

The setuptools packages now depend on the new pkg-resources packages, to avoid regressions for other packages that depend on them already.

---
v2: restrict new RDEPENDS to class-target. As advised by Alexander, bitbake
cannot resolve native rdeps that mention package names rather than recipe names.
v3: manually add RPROVIDES to the native class instead of restricting the
RDEPENDS to the target class as a better workaround. Also document why the package is being split.
v4: re-send to the correct thread, no changes.

 meta/recipes-devtools/python/python-setuptools.inc | 11 +++++++++++
 1 file changed, 11 insertions(+)
meta/recipes-devtools/python/python-setuptools.inc +11 -0
index 357aa07086..f49e078697 100644
@@ -37,3 +37,14 @@ do_install_prepend() {
 }
 
 BBCLASSEXTEND = "native nativesdk"
+
+# The pkg-resources module can be used by itself, without the package downloader
+# and easy_install. Ship it in a separate package so that it can be used by
+# minimal distributions.
+PACKAGES =+ "${PYTHON_PN}-pkg-resources "
+FILES_${PYTHON_PN}-pkg-resources = "${PYTHON_SITEPACKAGES_DIR}/pkg_resources/*"
+# Due to the way OE-Core implemented native recipes, the native class cannot
+# have a dependency on something that is not a recipe name. Work around that by
+# manually setting RPROVIDES.
+RDEPENDS_${PN}_append = " ${PYTHON_PN}-pkg-resources"
+RPROVIDES_append_class-native = " ${PYTHON_PN}-pkg-resources-native"
Signature · 1 line
2.20.1
#7 lore raw
Quoted 39 lines · On Tue, May 21, 2019 at 5:36 AM <[email protected]> wrote:

From: Luca Boccassi <[email protected]>

The pkg_resources Python module is useful by itself, for example for automatic loading of resources shipped in a Python package. Add separate packages for it, so that users can depend on them individually and avoid pulling in the entire setuptools, which include scripts to download other packages, which might not be desired on minimal images.

Other distributions like Debian and Ubuntu already split setuptools and pkg-resources in this way.

The setuptools packages now depend on the new pkg-resources packages, to avoid regressions for other packages that depend on them already.

---
v2: restrict new RDEPENDS to class-target. As advised by Alexander, bitbake
cannot resolve native rdeps that mention package names rather than recipe names.
v3: manually add RPROVIDES to the native class instead of restricting the
RDEPENDS to the target class as a better workaround. Also document why the package is being split.
v4: re-send to the correct thread, no changes.

 meta/recipes-devtools/python/python-setuptools.inc | 11 +++++++++++
 1 file changed, 11 insertions(+)
meta/recipes-devtools/python/python-setuptools.inc +1 -0
index 357aa07086..f49e078697 100644
@@ -37,3 +37,14 @@ do_install_prepend() {
 }
 
 BBCLASSEXTEND = "native nativesdk"
+

meta/recipes-devtools/python/python-setuptools.inc do_install_prepend() { · line 50

Quoted 10 lines
meta/recipes-devtools/python/python-setuptools.inc +10 -0
+# The pkg-resources module can be used by itself, without the package downloader
+# and easy_install. Ship it in a separate package so that it can be used by
+# minimal distributions.
+PACKAGES =+ "${PYTHON_PN}-pkg-resources "
+FILES_${PYTHON_PN}-pkg-resources = "${PYTHON_SITEPACKAGES_DIR}/pkg_resources/*"
+# Due to the way OE-Core implemented native recipes, the native class cannot
+# have a dependency on something that is not a recipe name. Work around that by
+# manually setting RPROVIDES.
+RDEPENDS_${PN}_append = " ${PYTHON_PN}-pkg-resources"
+RPROVIDES_append_class-native = " ${PYTHON_PN}-pkg-resources-native"

do we need to handle nativesdk case ?

Quoted 7 lines

2.20.1

--
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-core

#8 lore raw
Quoted 63 lines · Khem Raj

On Tue, May 21, 2019 at 5:36 AM <
[email protected]

Quoted 61 lines

wrote:
From: Luca Boccassi <
[email protected]

Quoted 1 line

The pkg_resources Python module is useful by itself, for example for
automatic loading of resources shipped in a Python package. Add separate packages for it, so that users can depend on them individually and avoid pulling in the entire setuptools, which include scripts to download other packages, which might not be desired on minimal images.

Other distributions like Debian and Ubuntu already split setuptools and pkg-resources in this way.

The setuptools packages now depend on the new pkg-resources packages,
to avoid regressions for other packages that depend on them already.

Signed-off-by: Luca Boccassi <
[email protected]

Quoted 1 line

---
v2: restrict new RDEPENDS to class-target. As advised by Alexander, bitbake
cannot resolve native rdeps that mention package names rather
than
recipe names.
v3: manually add RPROVIDES to the native class instead of restricting the
RDEPENDS to the target class as a better workaround. Also
document why
the package is being split.
v4: re-send to the correct thread, no changes.

 meta/recipes-devtools/python/python-setuptools.inc | 11
+++++++++++
 1 file changed, 11 insertions(+)
meta/recipes-devtools/python/python-setuptools.inc +2 -0
index 357aa07086..f49e078697 100644
@@ -37,3 +37,14 @@ do_install_prepend() {
 }
 
 BBCLASSEXTEND = "native nativesdk"
+
+# The pkg-resources module can be used by itself, without the

package downloader
+# and easy_install. Ship it in a separate package so that it can be used by

meta/recipes-devtools/python/python-setuptools.inc +3 -0
+# minimal distributions.
+PACKAGES =+ "${PYTHON_PN}-pkg-resources "
+FILES_${PYTHON_PN}-pkg-resources =

"${PYTHON_SITEPACKAGES_DIR}/pkg_resources/*"
+# Due to the way OE-Core implemented native recipes, the native class cannot

Quoted 8 lines
Quoted 6 lines

+# have a dependency on something that is not a recipe name. Work around that by

meta/recipes-devtools/python/python-setuptools.inc +3 -0
+# manually setting RPROVIDES.
+RDEPENDS_${PN}_append = " ${PYTHON_PN}-pkg-resources"
+RPROVIDES_append_class-native = " ${PYTHON_PN}-pkg-resources-

native"

do we need to handle nativesdk case ?

Hi,

The parsing step of "bitbake core-image-minimal -c populate_sdk" works, while without the append_class-native workaround it fails immediately. Is this enough or is there something else I should run to check?

Thanks!

Signature · 2 lines
Kind regards,
Luca Boccassi
#9 lore raw
Quoted 74 lines · Luca Boccassi
Quoted 63 lines · Khem Raj

On Tue, May 21, 2019 at 5:36 AM <
[email protected]

Quoted 61 lines

wrote:
From: Luca Boccassi <
[email protected]

Quoted 1 line

The pkg_resources Python module is useful by itself, for example for
automatic loading of resources shipped in a Python package. Add separate packages for it, so that users can depend on them individually and avoid pulling in the entire setuptools, which include scripts to download other packages, which might not be desired on minimal images.

Other distributions like Debian and Ubuntu already split setuptools and pkg-resources in this way.

The setuptools packages now depend on the new pkg-resources packages,
to avoid regressions for other packages that depend on them already.

Signed-off-by: Luca Boccassi <
[email protected]

Quoted 1 line

---
v2: restrict new RDEPENDS to class-target. As advised by Alexander, bitbake
cannot resolve native rdeps that mention package names rather
than
recipe names.
v3: manually add RPROVIDES to the native class instead of restricting the
RDEPENDS to the target class as a better workaround. Also
document why
the package is being split.
v4: re-send to the correct thread, no changes.

 meta/recipes-devtools/python/python-setuptools.inc | 11
+++++++++++
 1 file changed, 11 insertions(+)
meta/recipes-devtools/python/python-setuptools.inc +2 -0
index 357aa07086..f49e078697 100644
@@ -37,3 +37,14 @@ do_install_prepend() {
 }
 
 BBCLASSEXTEND = "native nativesdk"
+
+# The pkg-resources module can be used by itself, without the

package downloader
+# and easy_install. Ship it in a separate package so that it can be used by

meta/recipes-devtools/python/python-setuptools.inc +3 -0
+# minimal distributions.
+PACKAGES =+ "${PYTHON_PN}-pkg-resources "
+FILES_${PYTHON_PN}-pkg-resources =

"${PYTHON_SITEPACKAGES_DIR}/pkg_resources/*"
+# Due to the way OE-Core implemented native recipes, the native class cannot

Quoted 8 lines
Quoted 6 lines

+# have a dependency on something that is not a recipe name. Work around that by

meta/recipes-devtools/python/python-setuptools.inc +3 -0
+# manually setting RPROVIDES.
+RDEPENDS_${PN}_append = " ${PYTHON_PN}-pkg-resources"
+RPROVIDES_append_class-native = " ${PYTHON_PN}-pkg-resources-

native"

do we need to handle nativesdk case ?

Quoted 6 lines

Hi,

The parsing step of "bitbake core-image-minimal -c populate_sdk" works, while without the append_class-native workaround it fails immediately. Is this enough or is there something else I should run to check?

yeah usually to test nativesdk we need to do it with generated SDK

Quoted 6 lines

Thanks!

--
Kind regards,
Luca Boccassi

#10 lore raw
Quoted 86 lines · Khem Raj
Quoted 84 lines · Luca Boccassi
Quoted 71 lines · Khem Raj

On Tue, May 21, 2019 at 5:36 AM <
[email protected]

Quoted 69 lines

wrote:
From: Luca Boccassi <
[email protected]

Quoted 1 line

The pkg_resources Python module is useful by itself, for example
for
automatic loading of resources shipped in a Python package. Add separate packages for it, so that users can depend on them individually and avoid pulling in the entire setuptools, which include scripts to download other packages, which might not be desired on minimal images.

Other distributions like Debian and Ubuntu already split setuptools
and pkg-resources in this way.

The setuptools packages now depend on the new pkg-resources packages,
to avoid regressions for other packages that depend on them already.

Signed-off-by: Luca Boccassi <
[email protected]

Quoted 1 line

---
v2: restrict new RDEPENDS to class-target. As advised by Alexander,
bitbake
    cannot resolve native rdeps that mention package names rather
than
    recipe names.
v3: manually add RPROVIDES to the native class instead of restricting the
    RDEPENDS to the target class as a better workaround. Also document why
    the package is being split.
v4: re-send to the correct thread, no changes.

 meta/recipes-devtools/python/python-setuptools.inc | 11
+++++++++++

 1 file changed, 11 insertions(+)
meta/recipes-devtools/python/python-setuptools.inc +0 -0
index 357aa07086..f49e078697 100644
@@ -37,3 +37,14 @@ do_install_prepend() {
  }
 

 BBCLASSEXTEND = "native nativesdk"

meta/recipes-devtools/python/python-setuptools.inc +2 -0
+
+# The pkg-resources module can be used by itself, without the

package downloader
+# and easy_install. Ship it in a separate package so that it can
be used by

meta/recipes-devtools/python/python-setuptools.inc +3 -0
+# minimal distributions.
+PACKAGES =+ "${PYTHON_PN}-pkg-resources "
+FILES_${PYTHON_PN}-pkg-resources =

"${PYTHON_SITEPACKAGES_DIR}/pkg_resources/*"
+# Due to the way OE-Core implemented native recipes, the native
class cannot
+# have a dependency on something that is not a recipe name.

Quoted 8 lines
Quoted 6 lines

Work
around that by

meta/recipes-devtools/python/python-setuptools.inc +3 -0
+# manually setting RPROVIDES.
+RDEPENDS_${PN}_append = " ${PYTHON_PN}-pkg-resources"
+RPROVIDES_append_class-native = " ${PYTHON_PN}-pkg-resources-

native"

do we need to handle nativesdk case ?

Hi,

Quoted 8 lines
Quoted 6 lines

The parsing step of "bitbake core-image-minimal -c populate_sdk" works,
while without the append_class-native workaround it fails immediately.
Is this enough or is there something else I should run to check?

yeah usually to test nativesdk we need to do it with generated SDK

I do have a nativesdk target in the distro at $work, but I realise it might be different enough. Is there a simple config change/command to generate it from Poky?

Signature · 2 lines
Kind regards,
Luca Boccassi