providers.py: do not skip recipes providing 'virtual/x' in world builds
PATCH 1/1 providers.py: do not skip recipes providing 'virtual/x' in world builds
This is very old code pre-dating 2005, and the reasons are probably lost in history.
Doing this skipping means that anything providing 'virtual/x' is excluded from populating packages to deploy/ in world builds, and thus, from reproducubility and packaging tests.
This issue has been largely masked until 'package_ipk/deb/rpm: Drop recursive do_build task dependencies' as previously recipes without virtual/x provides would pull in the dependent rpms instead of the world target doing that directly.
I have noticed the issue by observing that go-runtime is no longer reported as non-reproducible when it should be.
--- bitbake/lib/bb/providers.py | 4 ---- 1 file changed, 4 deletions(-)
bitbake/lib/bb/providers.py +0 -4
@@ -429,10 +429,6 @@ def buildWorldTargetList(dataCache, task=None): terminal = False for p in dataCache.pn_provides[pn]: - if p.startswith('virtual/'): - logger.debug2("World build skipping %s due to %s provider starting with virtual/", f, p) - terminal = False - break for pf in dataCache.providers[p]: if dataCache.pkg_fn[pf] != pn: logger.debug2("World build skipping %s due to both us and %s providing %s", f, pf, p)
Signature · 1 line
2.20.1
Quoted 2 lines · Alexander Kanavin
This is very old code pre-dating 2005, and the reasons are probably lost in history.
They're not, I remember this.
Quoted 10 lines
Doing this skipping means that anything providing 'virtual/x' is excluded from populating packages to deploy/ in world builds, and thus, from reproducubility and packaging tests.
This issue has been largely masked until 'package_ipk/deb/rpm: Drop recursive
do_buildtask dependencies' as previously recipes without virtual/x provides would pull in the dependent rpms instead of the world target doing that directly.I have noticed the issue by observing that go-runtime is no longer reported as non-reproducible when it should be.
The principle is that virtual/XXX should only be built if something actually depends upon them as they're usually used to denote things which can be provided from multiple sources. I'd argue that the code is correct and wonder whether:
a) go-runtime should be providing a virtual
b) that OE-Core should have something depending on go-runtime for test purposes
Cheers,
Richard
Quoted 29 lines
Signed-off-by Alexander Kanavin <[email protected]>--- bitbake/lib/bb/providers.py | 4 ---- 1 file changed, 4 deletions(-)bitbake/lib/bb/providers.py +0
-5@@ -429,10 +429,6 @@ def buildWorldTargetList(dataCache, task=None): terminal = False for p in dataCache.pn_provides[pn]: - if p.startswith('virtual/'): - logger.debug2("World build skipping %s due to %s provider starting with virtual/", f, p) - terminal = False - break for pf in dataCache.providers[p]: if dataCache.pkg_fn[pf] != pn: logger.debug2("World build skipping %s due to both us and %s providing %s", f, pf, p) -=-=-=-=-=-=-=-=-=-=-=-Links: You receive all messages sent to this group.
View/Reply Online (#13209): https://lists.openembedded.org/g/bitbake-devel/message/13209
Mute This Topic: https://lists.openembedded.org/mt/88071309/1686473
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/bitbake-devel/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
This seems like something that could be moved to oe-core rather than being in bitbake proper, set EXCLUDE_FROM_WORLD’s default value if providing virtual/, in which case this functionality could be easily disabled if needed as well.
--
Christopher “kergoth” Larson
[email protected], [email protected], [email protected] Founder - BitBake, OpenEmbedded, OpenZaurus
Senior Software Engineer, Mentor Graphics, a Siemens Business
Quoted 64 lines · On Jan 3, 2022, 3:38 PM -0700, Richard Purdie…
Quoted 2 lines · Alexander Kanavin
This is very old code pre-dating 2005, and the reasons are probably lost in history.
They're not, I remember this.
Quoted 10 lines
Doing this skipping means that anything providing 'virtual/x' is excluded from populating packages to deploy/ in world builds, and thus, from reproducubility and packaging tests.
This issue has been largely masked until 'package_ipk/deb/rpm: Drop recursive
do_buildtask dependencies' as previously recipes without virtual/x provides would pull in the dependent rpms instead of the world target doing that directly.I have noticed the issue by observing that go-runtime is no longer reported as non-reproducible when it should be.
The principle is that virtual/XXX should only be built if something actually depends upon them as they're usually used to denote things which can be provided from multiple sources. I'd argue that the code is correct and wonder whether:
a) go-runtime should be providing a virtual
b) that OE-Core should have something depending on go-runtime for test purposesCheers,
Richard
Quoted 13 lines
Signed-off-by Alexander Kanavin <[email protected]>---
bitbake/lib/bb/providers.py| 4 ----1 file changed, 4 deletions(-)bitbake/lib/bb/providers.py +0
-0@@ -429,10 +429,6 @@ def buildWorldTargetList(dataCache, task=None):terminal = False
Quoted 10 lines
for p in dataCache.pn_provides[pn]:
- ifp.startswith('virtual/'):
-logger.debug2("World build skipping %s due to %s provider starting with virtual/", f, p)
- terminal = False
- break
for pf in dataCache.providers[p]:
if dataCache.pkg_fn[pf] != pn:logger.debug2("World build skipping %s due to both us and %s providing %s", f, pf, p)-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#13212): https://lists.openembedded.org/g/bitbake-devel/message/13212
Mute This Topic: https://lists.openembedded.org/mt/88071309/3617123
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/bitbake-devel/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
Quoted 5 lines · Richard Purdie
The principle is that virtual/XXX should only be built if something actually
depends upon them as they're usually used to denote things which can be provided
from multiple sources.
I'm not sure I follow. The 'world' set is aiming to have the fullest amount of target items in it. If there are items which have the same 'virtual/x' provides in their recipes, why should the recipes be all excluded from the set? They're not going to clash in package deploy, or in sysroots (since sysroots became recipe-specific). Note that it's not 'virtual/x' being excluded, but the parent recipes themselves.
Quoted 6 lines
I'd argue that the code is correct and wonder whether:
a) go-runtime should be providing a virtual
b) that OE-Core should have something depending on go-runtime for test purposes
go-runtime is how I spotted the issue, but there are actually a fair bit more items with the same problem, some of them major.
DEBUG: collating packages for "world"
DEBUG: World build skipping /home/alex/development/poky/meta/recipes-graphics/libsdl2/libsdl2_2.0.18.bb due to virtual/libsdl2 provider starting with virtual/
DEBUG: World build skipping /home/alex/development/poky/meta/recipes-devtools/make/make_4.3.bb due to virtual/make provider starting with virtual/
DEBUG: World build skipping /home/alex/development/poky/meta/recipes-extended/libtirpc/libtirpc_1.3.2.bb due to virtual/librpc provider starting with virtual/
DEBUG: World build skipping /home/alex/development/poky/meta/recipes-devtools/go/go-runtime_1.17.5.bb due to virtual/x86_64-poky-linux-go-runtime provider starting with virtual/
DEBUG: World build skipping /home/alex/development/poky/meta/recipes-core/libxcrypt/libxcrypt_4.4.26.bb due to virtual/crypt provider starting with virtual/
DEBUG: World build skipping /home/alex/development/poky/meta/recipes-graphics/xorg-xserver/ xserver-xorg_21.1.1.bb due to virtual/xserver-xf86 provider starting with virtual/
DEBUG: World build skipping /home/alex/development/poky/meta/recipes-kernel/linux/linux-yocto_5.10.bb due to virtual/kernel provider starting with virtual/
DEBUG: World build skipping /home/alex/development/poky/meta/recipes-sato/rxvt-unicode/ rxvt-unicode_9.30.bb due to virtual/x-terminal-emulator provider starting with virtual/
DEBUG: World build skipping /home/alex/development/poky/meta/recipes-core/gettext/gettext_0.21.bb due to virtual/libintl provider starting with virtual/
DEBUG: World build skipping /home/alex/development/poky/meta/recipes-core/glibc/glibc-locale_2.34.bb due to virtual/libc-locale provider starting with virtual/
DEBUG: World build skipping /home/alex/development/poky/meta/recipes-multimedia/libomxil/ libomxil_0.9.3.bb due to virtual/libomxil provider starting with virtual/
DEBUG: World build skipping /home/alex/development/poky/meta/recipes-devtools/gcc/libgcc-initial_11.2.bb as task do_build doesn't exist
DEBUG: World build skipping /home/alex/development/poky/meta/recipes-core/glibc/glibc_2.34.bb due to virtual/libc provider starting with virtual/
DEBUG: World build skipping /home/alex/development/poky/meta/recipes-kernel/perf/perf.bb due to virtual/perf provider starting with virtual/
DEBUG: World build skipping /home/alex/development/poky/meta/recipes-bsp/u-boot/u-boot_2021.10.bb due to virtual/bootloader provider starting with virtual/
DEBUG: World build skipping /home/alex/development/poky/meta/recipes-graphics/mesa/mesa_21.3.1.bb due to virtual/libgl provider starting with virtual/
DEBUG: World build skipping /home/alex/development/poky/meta/recipes-kernel/linux/linux-yocto_5.15.bb due to virtual/kernel provider starting with virtual/
DEBUG: World build skipping /home/alex/development/poky/meta/recipes-devtools/dpkg/dpkg_1.21.1.bb due to virtual/update-alternatives provider starting with virtual/
DEBUG: World build skipping /home/alex/development/poky/meta/recipes-graphics/xorg-lib/libx11_1.7.3.1.bb due to virtual/libx11 provider starting with virtual/
DEBUG: World build skipping /home/alex/development/poky/meta/recipes-kernel/linux/linux-yocto_5.14.bb due to virtual/kernel provider starting with virtual/
DEBUG: World build skipping /home/alex/development/poky/meta/recipes-devtools/opkg-utils/ opkg-utils_0.5.0.bb due to virtual/update-alternatives provider starting with virtual/
DEBUG: World build skipping /home/alex/development/poky/meta/recipes-devtools/gcc/gcc-runtime_11.2.bb due to virtual/x86_64-poky-linux-compilerlibs provider starting with virtual/
DEBUG: World build skipping /home/alex/development/poky/meta/recipes-devtools/pseudo/pseudo_git.bb due to virtual/fakeroot provider starting with virtual/
Alex
A message that is not in the archive (sent off-list or not yet received): f80391bccd5070fed6d7d7196bba726459ff9146.camel@linuxfoundation.org
Quoted 5 lines · Richard Purdie
Building musl on a glibc configured system is pointless. Similarly, building
clang when you're targeting things with gcc is pointless too.
As mentioned, musl and clang and systemd are all excluded earlier by skipping them via base.bbclass. So this code only ends up excluding the sole remaining options (preferred alternatives set via PREFERRED_PROVIDER), which are both valid and useful in world builds.
Quoted 72 lines
Quoted 2 lines
DEBUG: collating packages for "world"
DEBUG: World build skipping
/home/alex/development/poky/meta/recipes-graphics/libsdl2/libsdl2_2.0.18.bb due to virtual/libsdl2 provider starting with virtual/Quoted 1 line
DEBUG: World build skipping
/home/alex/development/poky/meta/recipes-devtools/make/make_4.3.bbdue to virtual/make provider starting with virtual/Quoted 1 line
DEBUG: World build skipping
/home/alex/development/poky/meta/recipes-extended/libtirpc/libtirpc_1.3.2.bb due to virtual/librpc provider starting with virtual/Quoted 1 line
DEBUG: World build skipping
/home/alex/development/poky/meta/recipes-devtools/go/go-runtime_1.17.5.bbdue to virtual/x86_64-poky-linux-go-runtime provider starting with virtual/Quoted 1 line
DEBUG: World build skipping
/home/alex/development/poky/meta/recipes-core/libxcrypt/libxcrypt_4.4.26.bb due to virtual/crypt provider starting with virtual/Quoted 1 line
DEBUG: World build skipping
/home/alex/development/poky/meta/recipes-graphics/xorg-xserver/xserver-xorg_21.1.1.bb due to virtual/xserver-xf86 provider starting with virtual/Quoted 1 line
DEBUG: World build skipping
/home/alex/development/poky/meta/recipes-kernel/linux/linux-yocto_5.10.bbdue to virtual/kernel provider starting with virtual/Quoted 1 line
DEBUG: World build skipping
/home/alex/development/poky/meta/recipes-sato/rxvt-unicode/rxvt-unicode_9.30.bb due to virtual/x-terminal-emulator provider starting with virtual/Quoted 1 line
DEBUG: World build skipping
/home/alex/development/poky/meta/recipes-core/gettext/gettext_0.21.bbdue to virtual/libintl provider starting with virtual/Quoted 1 line
DEBUG: World build skipping
/home/alex/development/poky/meta/recipes-core/glibc/glibc-locale_2.34.bbdue to virtual/libc-locale provider starting with virtual/Quoted 1 line
DEBUG: World build skipping
/home/alex/development/poky/meta/recipes-multimedia/libomxil/libomxil_0.9.3.bb due to virtual/libomxil provider starting with virtual/Quoted 1 line
DEBUG: World build skipping
/home/alex/development/poky/meta/recipes-devtools/gcc/libgcc-initial_11.2.bb as taskdo_builddoesn't existQuoted 1 line
DEBUG: World build skipping
/home/alex/development/poky/meta/recipes-core/glibc/glibc_2.34.bbdue to virtual/libc provider starting with virtual/Quoted 1 line
DEBUG: World build skipping
/home/alex/development/poky/meta/recipes-kernel/perf/perf.bbdue to virtual/perf provider starting with virtual/Quoted 1 line
DEBUG: World build skipping
/home/alex/development/poky/meta/recipes-bsp/u-boot/u-boot_2021.10.bbdue to virtual/bootloader provider starting with virtual/Quoted 1 line
DEBUG: World build skipping
/home/alex/development/poky/meta/recipes-graphics/mesa/mesa_21.3.1.bbdue to virtual/libgl provider starting with virtual/Quoted 1 line
DEBUG: World build skipping
/home/alex/development/poky/meta/recipes-kernel/linux/linux-yocto_5.15.bbdue to virtual/kernel provider starting with virtual/Quoted 1 line
DEBUG: World build skipping
/home/alex/development/poky/meta/recipes-devtools/dpkg/dpkg_1.21.1.bbdue to virtual/update-alternatives provider starting with virtual/Quoted 1 line
DEBUG: World build skipping
/home/alex/development/poky/meta/recipes-graphics/xorg-lib/libx11_1.7.3.1.bb due to virtual/libx11 provider starting with virtual/Quoted 1 line
DEBUG: World build skipping
/home/alex/development/poky/meta/recipes-kernel/linux/linux-yocto_5.14.bbdue to virtual/kernel provider starting with virtual/Quoted 1 line
DEBUG: World build skipping
/home/alex/development/poky/meta/recipes-devtools/opkg-utils/opkg-utils_0.5.0.bb due to virtual/update-alternatives provider starting with virtual/Quoted 1 line
DEBUG: World build skipping
/home/alex/development/poky/meta/recipes-devtools/gcc/gcc-runtime_11.2.bbdue to virtual/x86_64-poky-linux-compilerlibs provider starting with virtual/Quoted 1 line
DEBUG: World build skipping
Quoted 9 lines
/home/alex/development/poky/meta/recipes-devtools/pseudo/pseudo_git.bbdue to virtual/fakeroot provider starting with virtual/Most of these would be have things depending on them and therefore would get
built anyway. If they don't, they're probably not "useful" in the world build
context so I still think the code is actually doing what was intended?
The issue here is not a build for purposes of populating another sysroot (without packaging for target), the issue is that there's no guarantee anything here would end up in package deploy and thus be tested for reproducibility. How should we solve this? I'd still want to drop the world skipping (given that it does not actually exclude non-useful alternatives). I would reluctantly agree to '-k universe' too in the repro test.
Alex
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 8 lines · Alexander Kanavin via lists.openembedded.org
Most of these would be have things depending on them and therefore would get
Quoted 4 lines
built anyway. If they don't, they're probably not "useful" in the world build
context so I still think the code is actually doing what was intended?
Quoted 7 lines
The issue here is not a build for purposes of populating another sysroot (without packaging for target), the issue is that there's no guarantee anything here would end up in package deploy and thus be tested for reproducibility. How should we solve this? I'd still want to drop the world skipping (given that it does not actually exclude non-useful alternatives). I would reluctantly agree to '-k universe' too in the repro test.
As a specific example, go-runtime is already part of the world builds because it's needed by go-helloworld - but only up to do_populate_sysroot. do_package and later simply do not run due to being excluded from world. What to do that doesn't require maintaining a list of such items manually?
Alex
A message that is not in the archive (sent off-list or not yet received): [email protected]
To follow this up, if there are indeed multiple providers, they are excluded even earlier in base.bbclass:
=====
provs = (d.getVar("PROVIDES") or "").split() multiwhitelist = (d.getVar("MULTI_PROVIDER_WHITELIST") or
"").split()
for p in provs:
if p.startswith("virtual/") and p not in multiwhitelist:
profprov = d.getVar("PREFERRED_PROVIDER_" + p) if profprov and pn != profprov:
raise bb.parse.SkipRecipe("PREFERRED_PROVIDER_%s
set to %s, not %s" % (p, profprov, pn))
=====
So the bitbake code only excludes the sole remaining option from world, which makes even less sense.
Alex
Quoted 116 lines · Alexander Kanavin via lists.openembedded.org
Quoted 5 lines · Richard Purdie
The principle is that virtual/XXX should only be built if something actually
depends upon them as they're usually used to denote things which can be provided
from multiple sources.I'm not sure I follow. The 'world' set is aiming to have the fullest amount of target items in it. If there are items which have the same 'virtual/x' provides in their recipes, why should the recipes be all excluded from the set? They're not going to clash in package deploy, or in sysroots (since sysroots became recipe-specific). Note that it's not 'virtual/x' being excluded, but the parent recipes themselves.
Quoted 6 lines
I'd argue that the code is correct and wonder whether:
a) go-runtime should be providing a virtual
b) that OE-Core should have something depending on go-runtime for test purposesgo-runtime is how I spotted the issue, but there are actually a fair bit more items with the same problem, some of them major.
DEBUG: collating packages for "world"
DEBUG: World build skipping/home/alex/development/poky/meta/recipes-graphics/libsdl2/libsdl2_2.0.18.bb due to virtual/libsdl2 provider starting with virtual/
DEBUG: World build skipping/home/alex/development/poky/meta/recipes-devtools/make/make_4.3.bbdue to virtual/make provider starting with virtual/
DEBUG: World build skipping/home/alex/development/poky/meta/recipes-extended/libtirpc/libtirpc_1.3.2.bb due to virtual/librpc provider starting with virtual/
DEBUG: World build skipping/home/alex/development/poky/meta/recipes-devtools/go/go-runtime_1.17.5.bbdue to virtual/x86_64-poky-linux-go-runtime provider starting with virtual/
DEBUG: World build skipping/home/alex/development/poky/meta/recipes-core/libxcrypt/libxcrypt_4.4.26.bb due to virtual/crypt provider starting with virtual/
DEBUG: World build skipping/home/alex/development/poky/meta/recipes-graphics/xorg-xserver/xserver-xorg_21.1.1.bb due to virtual/xserver-xf86 provider starting with virtual/
DEBUG: World build skipping/home/alex/development/poky/meta/recipes-kernel/linux/linux-yocto_5.10.bbdue to virtual/kernel provider starting with virtual/
DEBUG: World build skipping/home/alex/development/poky/meta/recipes-sato/rxvt-unicode/rxvt-unicode_9.30.bb due to virtual/x-terminal-emulator provider starting with virtual/
DEBUG: World build skipping/home/alex/development/poky/meta/recipes-core/gettext/gettext_0.21.bbdue to virtual/libintl provider starting with virtual/
DEBUG: World build skipping/home/alex/development/poky/meta/recipes-core/glibc/glibc-locale_2.34.bbdue to virtual/libc-locale provider starting with virtual/
DEBUG: World build skipping/home/alex/development/poky/meta/recipes-multimedia/libomxil/libomxil_0.9.3.bb due to virtual/libomxil provider starting with virtual/
DEBUG: World build skipping/home/alex/development/poky/meta/recipes-devtools/gcc/libgcc-initial_11.2.bb as taskdo_builddoesn't exist
DEBUG: World build skipping/home/alex/development/poky/meta/recipes-core/glibc/glibc_2.34.bbdue to virtual/libc provider starting with virtual/
DEBUG: World build skipping/home/alex/development/poky/meta/recipes-kernel/perf/perf.bbdue to virtual/perf provider starting with virtual/
DEBUG: World build skipping/home/alex/development/poky/meta/recipes-bsp/u-boot/u-boot_2021.10.bbdue to virtual/bootloader provider starting with virtual/
DEBUG: World build skipping/home/alex/development/poky/meta/recipes-graphics/mesa/mesa_21.3.1.bbdue to virtual/libgl provider starting with virtual/
DEBUG: World build skipping/home/alex/development/poky/meta/recipes-kernel/linux/linux-yocto_5.15.bbdue to virtual/kernel provider starting with virtual/
DEBUG: World build skipping/home/alex/development/poky/meta/recipes-devtools/dpkg/dpkg_1.21.1.bbdue to virtual/update-alternatives provider starting with virtual/
DEBUG: World build skipping/home/alex/development/poky/meta/recipes-graphics/xorg-lib/libx11_1.7.3.1.bb due to virtual/libx11 provider starting with virtual/
DEBUG: World build skipping/home/alex/development/poky/meta/recipes-kernel/linux/linux-yocto_5.14.bbdue to virtual/kernel provider starting with virtual/
DEBUG: World build skipping/home/alex/development/poky/meta/recipes-devtools/opkg-utils/opkg-utils_0.5.0.bb due to virtual/update-alternatives provider starting with virtual/
DEBUG: World build skipping/home/alex/development/poky/meta/recipes-devtools/gcc/gcc-runtime_11.2.bbdue to virtual/x86_64-poky-linux-compilerlibs provider starting with virtual/
DEBUG: World build skipping/home/alex/development/poky/meta/recipes-devtools/pseudo/pseudo_git.bbdue to virtual/fakeroot provider starting with virtual/Alex
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group. View/Reply Online (#13215):
https://lists.openembedded.org/g/bitbake-devel/message/13215
Mute This Topic: https://lists.openembedded.org/mt/88071309/1686489
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/bitbake-devel/unsub [ [email protected]]
-=-=-=-=-=-=-=-=-=-=-=-