Skip to contentSkip to conversation

providers.py: do not skip recipes providing 'virtual/x' in world builds

PATCH In review Alexander Kanavin 3 participants 7 messages Jan 1, 2022 – Jan 4, 2022 3 messages not in the archive

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 providers.py: do not skip recipes providing 'virtual/x' in world builds

This is very old code pre-dating 2005, and the reasons are probably lost in history.

Doing this skipping means that anything providing 'virtual/x' is excluded from populating packages to deploy/ in world builds, and thus, from reproducubility and packaging tests.

This issue has been largely masked until 'package_ipk/deb/rpm: Drop recursive do_build task dependencies' as previously recipes without virtual/x provides would pull in the dependent rpms instead of the world target doing that directly.

I have noticed the issue by observing that go-runtime is no longer reported as non-reproducible when it should be.

---
 bitbake/lib/bb/providers.py | 4 ----
 1 file changed, 4 deletions(-)
bitbake/lib/bb/providers.py +0 -4
index 8c1c31a5c5..3c4462fbdd 100644
@@ -429,10 +429,6 @@ def buildWorldTargetList(dataCache, task=None):
             terminal = False
 
         for p in dataCache.pn_provides[pn]:
-            if p.startswith('virtual/'):
-                logger.debug2("World build skipping %s due to %s provider starting with virtual/", f, p)
-                terminal = False
-                break
             for pf in dataCache.providers[p]:
                 if dataCache.pkg_fn[pf] != pn:
                     logger.debug2("World build skipping %s due to both us and %s providing %s", f, pf, p)
Signature · 1 line
2.20.1
#1 lore raw
Quoted 2 lines · Alexander Kanavin

This is very old code pre-dating 2005, and the reasons are probably lost in history.

They're not, I remember this.

Quoted 10 lines

Doing this skipping means that anything providing 'virtual/x' is excluded from populating packages to deploy/ in world builds, and thus, from reproducubility and packaging tests.

This issue has been largely masked until 'package_ipk/deb/rpm: Drop recursive do_build task dependencies' as previously recipes without virtual/x provides would pull in the dependent rpms instead of the world target doing that directly.

I have noticed the issue by observing that go-runtime is no longer reported as non-reproducible when it should be.

The principle is that virtual/XXX should only be built if something actually depends upon them as they're usually used to denote things which can be provided from multiple sources. I'd argue that the code is correct and wonder whether:

a) go-runtime should be providing a virtual
b) that OE-Core should have something depending on go-runtime for test purposes

Cheers,

Richard

Quoted 29 lines
---
 bitbake/lib/bb/providers.py | 4 ----
 1 file changed, 4 deletions(-)
bitbake/lib/bb/providers.py +0 -5
index 8c1c31a5c5..3c4462fbdd 100644
@@ -429,10 +429,6 @@ def buildWorldTargetList(dataCache, task=None):
             terminal = False
 
         for p in dataCache.pn_provides[pn]:
-            if p.startswith('virtual/'):
-                logger.debug2("World build skipping %s due to %s provider starting with virtual/", f, p)
-                terminal = False
-                break
             for pf in dataCache.providers[p]:
                 if dataCache.pkg_fn[pf] != pn:
                     logger.debug2("World build skipping %s due to both us and %s providing %s", f, pf, p)
-=-=-=-=-=-=-=-=-=-=-=-

Links: You receive all messages sent to this group.
View/Reply Online (#13209): https://lists.openembedded.org/g/bitbake-devel/message/13209
Mute This Topic: https://lists.openembedded.org/mt/88071309/1686473
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/bitbake-devel/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-

#2 lore raw

This seems like something that could be moved to oe-core rather than being in bitbake proper, set EXCLUDE_FROM_WORLD’s default value if providing virtual/, in which case this functionality could be easily disabled if needed as well.

--
Christopher “kergoth” Larson
[email protected], [email protected], [email protected] Founder - BitBake, OpenEmbedded, OpenZaurus
Senior Software Engineer, Mentor Graphics, a Siemens Business

Quoted 64 lines · On Jan 3, 2022, 3:38 PM -0700, Richard Purdie…
Quoted 2 lines · Alexander Kanavin

This is very old code pre-dating 2005, and the reasons are probably lost in history.

They're not, I remember this.

Quoted 10 lines

Doing this skipping means that anything providing 'virtual/x' is excluded from populating packages to deploy/ in world builds, and thus, from reproducubility and packaging tests.

This issue has been largely masked until 'package_ipk/deb/rpm: Drop recursive do_build task dependencies' as previously recipes without virtual/x provides would pull in the dependent rpms instead of the world target doing that directly.

I have noticed the issue by observing that go-runtime is no longer reported as non-reproducible when it should be.

The principle is that virtual/XXX should only be built if something actually depends upon them as they're usually used to denote things which can be provided from multiple sources. I'd argue that the code is correct and wonder whether:

a) go-runtime should be providing a virtual
b) that OE-Core should have something depending on go-runtime for test purposes

Cheers,

Richard

Quoted 13 lines

---
bitbake/lib/bb/providers.py | 4 ----

1 file changed, 4 deletions(-)
bitbake/lib/bb/providers.py +0 -0
index 8c1c31a5c5..3c4462fbdd 100644
@@ -429,10 +429,6 @@ def buildWorldTargetList(dataCache, task=None):

terminal = False

Quoted 10 lines

for p in dataCache.pn_provides[pn]:
- if p.startswith('virtual/'):
- logger.debug2("World build skipping %s due to %s provider starting with virtual/", f, p)
- terminal = False
- break
for pf in dataCache.providers[p]:
if dataCache.pkg_fn[pf] != pn:
logger.debug2("World build skipping %s due to both us and %s providing %s", f, pf, p)

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#13212): https://lists.openembedded.org/g/bitbake-devel/message/13212
Mute This Topic: https://lists.openembedded.org/mt/88071309/3617123
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/bitbake-devel/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-

#3 lore raw
In reply to Richard Purdie
Quoted 5 lines · Richard Purdie

The principle is that virtual/XXX should only be built if something actually
depends upon them as they're usually used to denote things which can be provided
from multiple sources.

I'm not sure I follow. The 'world' set is aiming to have the fullest amount of target items in it. If there are items which have the same 'virtual/x' provides in their recipes, why should the recipes be all excluded from the set? They're not going to clash in package deploy, or in sysroots (since sysroots became recipe-specific). Note that it's not 'virtual/x' being excluded, but the parent recipes themselves.

Quoted 6 lines

I'd argue that the code is correct and wonder whether:

a) go-runtime should be providing a virtual
b) that OE-Core should have something depending on go-runtime for test purposes

go-runtime is how I spotted the issue, but there are actually a fair bit more items with the same problem, some of them major.

DEBUG: collating packages for "world"
DEBUG: World build skipping /home/alex/development/poky/meta/recipes-graphics/libsdl2/libsdl2_2.0.18.bb due to virtual/libsdl2 provider starting with virtual/
DEBUG: World build skipping /home/alex/development/poky/meta/recipes-devtools/make/make_4.3.bb due to virtual/make provider starting with virtual/
DEBUG: World build skipping /home/alex/development/poky/meta/recipes-extended/libtirpc/libtirpc_1.3.2.bb due to virtual/librpc provider starting with virtual/
DEBUG: World build skipping /home/alex/development/poky/meta/recipes-devtools/go/go-runtime_1.17.5.bb due to virtual/x86_64-poky-linux-go-runtime provider starting with virtual/
DEBUG: World build skipping /home/alex/development/poky/meta/recipes-core/libxcrypt/libxcrypt_4.4.26.bb due to virtual/crypt provider starting with virtual/
DEBUG: World build skipping /home/alex/development/poky/meta/recipes-graphics/xorg-xserver/ xserver-xorg_21.1.1.bb due to virtual/xserver-xf86 provider starting with virtual/
DEBUG: World build skipping /home/alex/development/poky/meta/recipes-kernel/linux/linux-yocto_5.10.bb due to virtual/kernel provider starting with virtual/
DEBUG: World build skipping /home/alex/development/poky/meta/recipes-sato/rxvt-unicode/ rxvt-unicode_9.30.bb due to virtual/x-terminal-emulator provider starting with virtual/
DEBUG: World build skipping /home/alex/development/poky/meta/recipes-core/gettext/gettext_0.21.bb due to virtual/libintl provider starting with virtual/
DEBUG: World build skipping /home/alex/development/poky/meta/recipes-core/glibc/glibc-locale_2.34.bb due to virtual/libc-locale provider starting with virtual/
DEBUG: World build skipping /home/alex/development/poky/meta/recipes-multimedia/libomxil/ libomxil_0.9.3.bb due to virtual/libomxil provider starting with virtual/
DEBUG: World build skipping /home/alex/development/poky/meta/recipes-devtools/gcc/libgcc-initial_11.2.bb as task do_build doesn't exist
DEBUG: World build skipping /home/alex/development/poky/meta/recipes-core/glibc/glibc_2.34.bb due to virtual/libc provider starting with virtual/
DEBUG: World build skipping /home/alex/development/poky/meta/recipes-kernel/perf/perf.bb due to virtual/perf provider starting with virtual/
DEBUG: World build skipping /home/alex/development/poky/meta/recipes-bsp/u-boot/u-boot_2021.10.bb due to virtual/bootloader provider starting with virtual/
DEBUG: World build skipping /home/alex/development/poky/meta/recipes-graphics/mesa/mesa_21.3.1.bb due to virtual/libgl provider starting with virtual/
DEBUG: World build skipping /home/alex/development/poky/meta/recipes-kernel/linux/linux-yocto_5.15.bb due to virtual/kernel provider starting with virtual/
DEBUG: World build skipping /home/alex/development/poky/meta/recipes-devtools/dpkg/dpkg_1.21.1.bb due to virtual/update-alternatives provider starting with virtual/
DEBUG: World build skipping /home/alex/development/poky/meta/recipes-graphics/xorg-lib/libx11_1.7.3.1.bb due to virtual/libx11 provider starting with virtual/
DEBUG: World build skipping /home/alex/development/poky/meta/recipes-kernel/linux/linux-yocto_5.14.bb due to virtual/kernel provider starting with virtual/
DEBUG: World build skipping /home/alex/development/poky/meta/recipes-devtools/opkg-utils/ opkg-utils_0.5.0.bb due to virtual/update-alternatives provider starting with virtual/
DEBUG: World build skipping /home/alex/development/poky/meta/recipes-devtools/gcc/gcc-runtime_11.2.bb due to virtual/x86_64-poky-linux-compilerlibs provider starting with virtual/
DEBUG: World build skipping /home/alex/development/poky/meta/recipes-devtools/pseudo/pseudo_git.bb due to virtual/fakeroot provider starting with virtual/

Alex

A message that is not in the archive (sent off-list or not yet received): f80391bccd5070fed6d7d7196bba726459ff9146.camel@linuxfoundation.org

#5 lore raw
Quoted 5 lines · Richard Purdie

Building musl on a glibc configured system is pointless. Similarly, building
clang when you're targeting things with gcc is pointless too.

As mentioned, musl and clang and systemd are all excluded earlier by skipping them via base.bbclass. So this code only ends up excluding the sole remaining options (preferred alternatives set via PREFERRED_PROVIDER), which are both valid and useful in world builds.

Quoted 72 lines
Quoted 2 lines

DEBUG: collating packages for "world"
DEBUG: World build skipping

/home/alex/development/poky/meta/recipes-graphics/libsdl2/ libsdl2_2.0.18.bb due to virtual/libsdl2 provider starting with virtual/

Quoted 1 line

DEBUG: World build skipping

/home/alex/development/poky/meta/recipes-devtools/make/make_4.3.bb due to virtual/make provider starting with virtual/

Quoted 1 line

DEBUG: World build skipping

/home/alex/development/poky/meta/recipes-extended/libtirpc/ libtirpc_1.3.2.bb due to virtual/librpc provider starting with virtual/

Quoted 1 line

DEBUG: World build skipping

/home/alex/development/poky/meta/recipes-devtools/go/go-runtime_1.17.5.bb due to virtual/x86_64-poky-linux-go-runtime provider starting with virtual/

Quoted 1 line

DEBUG: World build skipping

/home/alex/development/poky/meta/recipes-core/libxcrypt/ libxcrypt_4.4.26.bb due to virtual/crypt provider starting with virtual/

Quoted 1 line

DEBUG: World build skipping

/home/alex/development/poky/meta/recipes-graphics/xorg-xserver/ xserver-xorg_21.1.1.bb due to virtual/xserver-xf86 provider starting with virtual/

Quoted 1 line

DEBUG: World build skipping

/home/alex/development/poky/meta/recipes-kernel/linux/linux-yocto_5.10.bb due to virtual/kernel provider starting with virtual/

Quoted 1 line

DEBUG: World build skipping

/home/alex/development/poky/meta/recipes-sato/rxvt-unicode/ rxvt-unicode_9.30.bb due to virtual/x-terminal-emulator provider starting with virtual/

Quoted 1 line

DEBUG: World build skipping

/home/alex/development/poky/meta/recipes-core/gettext/gettext_0.21.bb due to virtual/libintl provider starting with virtual/

Quoted 1 line

DEBUG: World build skipping

/home/alex/development/poky/meta/recipes-core/glibc/glibc-locale_2.34.bb due to virtual/libc-locale provider starting with virtual/

Quoted 1 line

DEBUG: World build skipping

/home/alex/development/poky/meta/recipes-multimedia/libomxil/ libomxil_0.9.3.bb due to virtual/libomxil provider starting with virtual/

Quoted 1 line

DEBUG: World build skipping

/home/alex/development/poky/meta/recipes-devtools/gcc/ libgcc-initial_11.2.bb as task do_build doesn't exist

Quoted 1 line

DEBUG: World build skipping

/home/alex/development/poky/meta/recipes-core/glibc/glibc_2.34.bb due to virtual/libc provider starting with virtual/

Quoted 1 line

DEBUG: World build skipping

/home/alex/development/poky/meta/recipes-kernel/perf/perf.bb due to virtual/perf provider starting with virtual/

Quoted 1 line

DEBUG: World build skipping

/home/alex/development/poky/meta/recipes-bsp/u-boot/u-boot_2021.10.bb due to virtual/bootloader provider starting with virtual/

Quoted 1 line

DEBUG: World build skipping

/home/alex/development/poky/meta/recipes-graphics/mesa/mesa_21.3.1.bb due to virtual/libgl provider starting with virtual/

Quoted 1 line

DEBUG: World build skipping

/home/alex/development/poky/meta/recipes-kernel/linux/linux-yocto_5.15.bb due to virtual/kernel provider starting with virtual/

Quoted 1 line

DEBUG: World build skipping

/home/alex/development/poky/meta/recipes-devtools/dpkg/dpkg_1.21.1.bb due to virtual/update-alternatives provider starting with virtual/

Quoted 1 line

DEBUG: World build skipping

/home/alex/development/poky/meta/recipes-graphics/xorg-lib/ libx11_1.7.3.1.bb due to virtual/libx11 provider starting with virtual/

Quoted 1 line

DEBUG: World build skipping

/home/alex/development/poky/meta/recipes-kernel/linux/linux-yocto_5.14.bb due to virtual/kernel provider starting with virtual/

Quoted 1 line

DEBUG: World build skipping

/home/alex/development/poky/meta/recipes-devtools/opkg-utils/ opkg-utils_0.5.0.bb due to virtual/update-alternatives provider starting with virtual/

Quoted 1 line

DEBUG: World build skipping

/home/alex/development/poky/meta/recipes-devtools/gcc/gcc-runtime_11.2.bb due to virtual/x86_64-poky-linux-compilerlibs provider starting with virtual/

Quoted 1 line

DEBUG: World build skipping

Quoted 9 lines

/home/alex/development/poky/meta/recipes-devtools/pseudo/pseudo_git.bb due to virtual/fakeroot provider starting with virtual/

Most of these would be have things depending on them and therefore would get
built anyway. If they don't, they're probably not "useful" in the world build
context so I still think the code is actually doing what was intended?

The issue here is not a build for purposes of populating another sysroot (without packaging for target), the issue is that there's no guarantee anything here would end up in package deploy and thus be tested for reproducibility. How should we solve this? I'd still want to drop the world skipping (given that it does not actually exclude non-useful alternatives). I would reluctantly agree to '-k universe' too in the repro test.

Alex

A message that is not in the archive (sent off-list or not yet received): [email protected]

#7 lore raw
Quoted 8 lines · Alexander Kanavin via lists.openembedded.org

Most of these would be have things depending on them and therefore would get

Quoted 4 lines

built anyway. If they don't, they're probably not "useful" in the world build
context so I still think the code is actually doing what was intended?

Quoted 7 lines

The issue here is not a build for purposes of populating another sysroot (without packaging for target), the issue is that there's no guarantee anything here would end up in package deploy and thus be tested for reproducibility. How should we solve this? I'd still want to drop the world skipping (given that it does not actually exclude non-useful alternatives). I would reluctantly agree to '-k universe' too in the repro test.

As a specific example, go-runtime is already part of the world builds because it's needed by go-helloworld - but only up to do_populate_sysroot. do_package and later simply do not run due to being excluded from world. What to do that doesn't require maintaining a list of such items manually?

Alex

A message that is not in the archive (sent off-list or not yet received): [email protected]

#9 lore raw

To follow this up, if there are indeed multiple providers, they are excluded even earlier in base.bbclass:
=====
provs = (d.getVar("PROVIDES") or "").split() multiwhitelist = (d.getVar("MULTI_PROVIDER_WHITELIST") or
"").split()
for p in provs:
if p.startswith("virtual/") and p not in multiwhitelist:
profprov = d.getVar("PREFERRED_PROVIDER_" + p) if profprov and pn != profprov:
raise bb.parse.SkipRecipe("PREFERRED_PROVIDER_%s
set to %s, not %s" % (p, profprov, pn))
=====

So the bitbake code only excludes the sole remaining option from world, which makes even less sense.

Alex

Quoted 116 lines · Alexander Kanavin via lists.openembedded.org
Quoted 5 lines · Richard Purdie

The principle is that virtual/XXX should only be built if something actually
depends upon them as they're usually used to denote things which can be provided
from multiple sources.

I'm not sure I follow. The 'world' set is aiming to have the fullest amount of target items in it. If there are items which have the same 'virtual/x' provides in their recipes, why should the recipes be all excluded from the set? They're not going to clash in package deploy, or in sysroots (since sysroots became recipe-specific). Note that it's not 'virtual/x' being excluded, but the parent recipes themselves.

Quoted 6 lines

I'd argue that the code is correct and wonder whether:

a) go-runtime should be providing a virtual
b) that OE-Core should have something depending on go-runtime for test purposes

go-runtime is how I spotted the issue, but there are actually a fair bit more items with the same problem, some of them major.

DEBUG: collating packages for "world"
DEBUG: World build skipping /home/alex/development/poky/meta/recipes-graphics/libsdl2/ libsdl2_2.0.18.bb due to virtual/libsdl2 provider starting with virtual/
DEBUG: World build skipping /home/alex/development/poky/meta/recipes-devtools/make/make_4.3.bb due to virtual/make provider starting with virtual/
DEBUG: World build skipping /home/alex/development/poky/meta/recipes-extended/libtirpc/ libtirpc_1.3.2.bb due to virtual/librpc provider starting with virtual/
DEBUG: World build skipping /home/alex/development/poky/meta/recipes-devtools/go/go-runtime_1.17.5.bb due to virtual/x86_64-poky-linux-go-runtime provider starting with virtual/
DEBUG: World build skipping /home/alex/development/poky/meta/recipes-core/libxcrypt/ libxcrypt_4.4.26.bb due to virtual/crypt provider starting with virtual/
DEBUG: World build skipping /home/alex/development/poky/meta/recipes-graphics/xorg-xserver/ xserver-xorg_21.1.1.bb due to virtual/xserver-xf86 provider starting with virtual/
DEBUG: World build skipping /home/alex/development/poky/meta/recipes-kernel/linux/linux-yocto_5.10.bb due to virtual/kernel provider starting with virtual/
DEBUG: World build skipping /home/alex/development/poky/meta/recipes-sato/rxvt-unicode/ rxvt-unicode_9.30.bb due to virtual/x-terminal-emulator provider starting with virtual/
DEBUG: World build skipping /home/alex/development/poky/meta/recipes-core/gettext/gettext_0.21.bb due to virtual/libintl provider starting with virtual/
DEBUG: World build skipping /home/alex/development/poky/meta/recipes-core/glibc/glibc-locale_2.34.bb due to virtual/libc-locale provider starting with virtual/
DEBUG: World build skipping /home/alex/development/poky/meta/recipes-multimedia/libomxil/ libomxil_0.9.3.bb due to virtual/libomxil provider starting with virtual/
DEBUG: World build skipping /home/alex/development/poky/meta/recipes-devtools/gcc/ libgcc-initial_11.2.bb as task do_build doesn't exist
DEBUG: World build skipping /home/alex/development/poky/meta/recipes-core/glibc/glibc_2.34.bb due to virtual/libc provider starting with virtual/
DEBUG: World build skipping /home/alex/development/poky/meta/recipes-kernel/perf/perf.bb due to virtual/perf provider starting with virtual/
DEBUG: World build skipping /home/alex/development/poky/meta/recipes-bsp/u-boot/u-boot_2021.10.bb due to virtual/bootloader provider starting with virtual/
DEBUG: World build skipping /home/alex/development/poky/meta/recipes-graphics/mesa/mesa_21.3.1.bb due to virtual/libgl provider starting with virtual/
DEBUG: World build skipping /home/alex/development/poky/meta/recipes-kernel/linux/linux-yocto_5.15.bb due to virtual/kernel provider starting with virtual/
DEBUG: World build skipping /home/alex/development/poky/meta/recipes-devtools/dpkg/dpkg_1.21.1.bb due to virtual/update-alternatives provider starting with virtual/
DEBUG: World build skipping /home/alex/development/poky/meta/recipes-graphics/xorg-lib/ libx11_1.7.3.1.bb due to virtual/libx11 provider starting with virtual/
DEBUG: World build skipping /home/alex/development/poky/meta/recipes-kernel/linux/linux-yocto_5.14.bb due to virtual/kernel provider starting with virtual/
DEBUG: World build skipping /home/alex/development/poky/meta/recipes-devtools/opkg-utils/ opkg-utils_0.5.0.bb due to virtual/update-alternatives provider starting with virtual/
DEBUG: World build skipping /home/alex/development/poky/meta/recipes-devtools/gcc/gcc-runtime_11.2.bb due to virtual/x86_64-poky-linux-compilerlibs provider starting with virtual/
DEBUG: World build skipping /home/alex/development/poky/meta/recipes-devtools/pseudo/pseudo_git.bb due to virtual/fakeroot provider starting with virtual/

Alex

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group. View/Reply Online (#13215):
https://lists.openembedded.org/g/bitbake-devel/message/13215
Mute This Topic: https://lists.openembedded.org/mt/88071309/1686489
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/bitbake-devel/unsub [ [email protected]]
-=-=-=-=-=-=-=-=-=-=-=-