sstatesig.py: make it fatal error when sstate manifest isn't found
PATCH 1/1 sstatesig.py: make it fatal error when sstate manifest isn't found
* all known issues in this area were fixed, make it fatal that it cannot be overlooked if someone triggers this issue again
--- meta/lib/oe/sstatesig.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
meta/lib/oe/sstatesig.py +1 -1
@@ -453,7 +453,7 @@ def find_sstate_manifest(taskdata, taskdata2, taskname, d, multilibcache): manifest = d2.expand("${SSTATE_MANIFESTS}/manifest-%s-%s.%s" % (pkgarch, taskdata, taskname)) if os.path.exists(manifest): return manifest, d2 - bb.error("Manifest %s not found in %s (variant '%s')?" % (manifest, d2.expand(" ".join(pkgarchs)), variant)) + bb.fatal("Manifest %s not found in %s (variant '%s')?" % (manifest, d2.expand(" ".join(pkgarchs)), variant)) return None, d2 def OEOuthashBasic(path, sigfile, task, d):
Signature · 1 line
2.30.2
Quoted 13 lines · Martin Jansa
* all known issues in this area were fixed, make it fatal that it cannot be overlooked if someone triggers this issue again
Signed-off-by Martin Jansa <[email protected]>--- meta/lib/oe/sstatesig.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)meta/lib/oe/sstatesig.py +0
-0
meta/lib/oe/sstatesig.py def find_sstate_manifest(taskdata, taskdata2, taskname, d, multilibcache): · line 457
Quoted 6 lines
meta/lib/oe/sstatesig.py +1
-1@@ -453,7 +453,7 @@ def find_sstate_manifest(taskdata, taskdata2, taskname, d, multilibcache): manifest = d2.expand("${SSTATE_MANIFESTS}/manifest-%s-%s.%s" % (pkgarch, taskdata, taskname)) if os.path.exists(manifest): return manifest, d2 - bb.error("Manifest %s not found in %s (variant '%s')?" % (manifest, d2.expand(" ".join(pkgarchs)), variant)) + bb.fatal("Manifest %s not found in %s (variant '%s')?" % (manifest, d2.expand(" ".join(pkgarchs)), variant)) return None, d2
doesn't that make the return statement obsolete? also how can an error be overlooked? It does make the task triggering this issue fail, so how is a fatal better in this case (besides abnormally terminating the task)
Quoted 7 lines
meta/lib/oe/sstatesig.py +0
-0def OEOuthashBasic(path, sigfile, task, d):
Quoted 10 lines · Konrad Weihmann
Quoted 8 lines · Martin Jansa
* all known issues in this area were fixed, make it fatal that it cannot be overlooked if someone triggers this issue again
Signed-off-by Martin Jansa <[email protected]>---
meta/lib/oe/sstatesig.py | 2 +-1 file changed, 1 insertion(+), 1 deletion(-)
Quoted 15 lines
Quoted 11 lines
meta/lib/oe/sstatesig.py +0
-0@@ -453,7 +453,7 @@ def find_sstate_manifest(taskdata, taskdata2, taskname, d, multilibcache): manifest = d2.expand("${SSTATE_MANIFESTS}/manifest-%s-%s.%s" % (pkgarch, taskdata, taskname))if
os.path.exists(manifest):
return manifest, d2meta/lib/oe/sstatesig.py +1
-1- bb.error("Manifest %s not found in %s (variant '%s')?" % (manifest, d2.expand(" ".join(pkgarchs)), variant)) + bb.fatal("Manifest %s not found in %s (variant '%s')?" % (manifest, d2.expand(" ".join(pkgarchs)), variant))return None, d2
doesn't that make the return statement obsolete? also how can an error be overlooked? It does make the task triggering this issue fail, so how is a fatal better in this case (besides abnormally terminating the task)
This will fail the task rather than just setting the exit code so it should highlight there is a real problem to the user.
Cheers,
Richard
Quoted 22 lines · Konrad Weihmann
meta/lib/oe/sstatesig.py def find_sstate_manifest(taskdata, taskdata2,
Quoted 13 lines · Martin Jansa
* all known issues in this area were fixed, make it fatal that it cannot be overlooked if someone triggers this issue again
Signed-off-by Martin Jansa <[email protected]>--- meta/lib/oe/sstatesig.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)meta/lib/oe/sstatesig.py +0
-0@@ -453,7 +453,7 @@ def find_sstate_manifest(taskdata, taskdata2,taskname, d, multilibcache):
meta/lib/oe/sstatesig.py def find_sstate_manifest(taskdata, taskdata2, · line 453
Quoted 1 line
meta/lib/oe/sstatesig.py +0
-0manifest = d2.expand("${SSTATE_MANIFESTS}/manifest-%s-%s.%s" %(pkgarch, taskdata, taskname))
meta/lib/oe/sstatesig.py def find_sstate_manifest(taskdata, taskdata2, taskname, d, multilibcache):
Quoted 3 lines
meta/lib/oe/sstatesig.py +0
-1if os.path.exists(manifest): return manifest, d2 - bb.error("Manifest %s not found in %s (variant '%s')?" % (manifest,
Quoted 9 lines
d2.expand(" ".join(pkgarchs)), variant))meta/lib/oe/sstatesig.py def find_sstate_manifest(taskdata, taskdata2, · line 456
Quoted 1 line
meta/lib/oe/sstatesig.py +1
-0+ bb.fatal("Manifest %s not found in %s (variant '%s')?" % (manifest,
d2.expand(" ".join(pkgarchs)), variant))meta/lib/oe/sstatesig.py def find_sstate_manifest(taskdata, taskdata2, taskname, d, multilibcache): · line 457
Quoted 1 line
meta/lib/oe/sstatesig.py +0
-0return None, d2doesn't that make the return statement obsolete? also how can an error be overlooked? It does make the task triggering this issue fail, so how is a fatal better in this case (besides abnormally terminating the task)
bb.error doesn't make the task to fail, it only shows an error (which results in non-zero exit code), but lets depending tasks to continue and fail with other (often more misleading) errors, because their dependency is really missing in RSS
without this patch:
ERROR: adwaita-icon-theme-3.34.3-r0 do_prepare_recipe_sysroot: Manifest /OE/build/oe-core/tmp-glibc/sstate-control/manifest-x86_64_ubuntu-21.04-at-spi2-core-native.populate_sysroot not found in x86_64 x86_64_ubuntu-21.04 (variant '')?
ERROR: adwaita-icon-theme-3.34.3-r0 do_prepare_recipe_sysroot: Manifest /OE/build/oe-core/tmp-glibc/sstate-control/manifest-x86_64_ubuntu-21.04-at-spi2-core-native.populate_sysroot not found in x86_64 x86_64_ubuntu-21.04 (variant '')?
ERROR: adwaita-icon-theme-3.34.3-r0 do_configure: Manifest /OE/build/oe-core/tmp-glibc/sstate-control/manifest-x86_64_ubuntu-21.04-at-spi2-core-native.populate_sysroot not found in x86_64 x86_64_ubuntu-21.04 (variant '')?
ERROR: adwaita-icon-theme-3.34.3-r0 do_install: Manifest /OE/build/oe-core/tmp-glibc/sstate-control/manifest-x86_64_ubuntu-21.04-at-spi2-core-native.populate_sysroot not found in x86_64 x86_64_ubuntu-21.04 (variant '')?
ERROR: adwaita-icon-theme-3.34.3-r0 do_package: Manifest /OE/build/oe-core/tmp-glibc/sstate-control/manifest-x86_64_ubuntu-21.04-at-spi2-core-native.populate_sysroot not found in x86_64 x86_64_ubuntu-21.04 (variant '')?
ERROR: adwaita-icon-theme-3.34.3-r0 do_package_write_ipk: Manifest /OE/build/oe-core/tmp-glibc/sstate-control/manifest-x86_64_ubuntu-21.04-at-spi2-core-native.populate_sysroot not found in x86_64 x86_64_ubuntu-21.04 (variant '')?
NOTE: Tasks Summary: Attempted 2819 tasks of which 2803 didn't need to be
rerun and all succeeded.
Summary: There were 6 ERROR messages shown, returning a non-zero exit code.
with this patch:
ERROR: adwaita-icon-theme-3.34.3-r0 do_prepare_recipe_sysroot: Manifest /OE/build/oe-core/tmp-glibc/sstate-control/manifest-x86_64_ubuntu-21.04-at-spi2-core-native.populate_sysroot not found in x86_64 x86_64_ubuntu-21.04 (variant '')?
ERROR: Logfile of failure stored in:
/OE/build/oe-core/tmp-glibc/work/all-oe-linux/adwaita-icon-theme/3.34.3-r0/temp/log.do_prepare_recipe_sysroot.1638704
ERROR: Task (/OE/build/oe-core/openembedded-core/meta/recipes-gnome/gnome/adwaita-icon-theme_3.34.3.bb:do_prepare_recipe_sysroot) failed with exit code '1'
NOTE: Tasks Summary: Attempted 2805 tasks of which 2800 didn't need to be
rerun and 1 failed.
Summary: 1 task failed:
/OE/build/oe-core/openembedded-core/meta/recipes-gnome/gnome/adwaita-icon-theme_3.34.3.bb:do_prepare_recipe_sysroot
Summary: There was 1 ERROR message shown, returning a non-zero exit code.
Without this fix:
https://git.openembedded.org/openembedded-core/commit/?id=63da9a4f889c5b0e41bc8ec08abe0acea1546479
I was seeing various native dependencies missing and as you can see in the adwaita-icon-theme example, missing at-spi2-core-native isn't fatal for the build, but were adwaita-icon-theme packages and sstate built correctly? If you don't investigate all these ERROR messages and non-zero exit code (e.g. just because you have thousands of builds per day populating the same sstate mirror), you might miss some of these breakages. While using fatal prevents adwaita-icon-theme to even start building and producing possibly broken packages, images, sstate.
In some other cases e.g. qtwayland-native missing when building target qtwayland you will get do_compile failure complaining that the the binary from qtwayland-native is missing, but the root cause of this issue is written many lines above do_compile log. You can see some other examples in
https://git.openembedded.org/openembedded-core-contrib/commit/?h=jansa/master&id=817f1c14cec4d66504a5028908a4ebac147f3b52
and before changing this from bb.warn to bb.error in previous patch most of these issues were just "some random builddir breakage which was resolved by doing clean build from sstate".
And this happens in multi-machine builds with dunfell/gatesgarth/hardknott and only master is safe now. I plan to ask Steve to backport this and related fixes to dunfell after a bit more time in master:
https://git.openembedded.org/openembedded-core/commit/?id=0eb95212a7e6b6bdc1243094072dea432cb48f0f
https://git.openembedded.org/openembedded-core/commit/?id=63da9a4f889c5b0e41bc8ec08abe0acea1546479
https://git.openembedded.org/openembedded-core/commit/?id=95607a26854d873399d2b9d7e5fcbffc0cbdba4c
and then this one.
Regards,
Quoted 65 lines · Martin Jansa
On Wed, May 19, 2021 at 12:47 PM Konrad Weihmann <[email protected] <mailto:[email protected]>> wrote:
On 19.05.21 12:23, Martin Jansa wrote:
> * all known issues in this area were fixed, make it fatal that it > cannot be overlooked if someone triggers this issue again >
> Signed-off-by: Martin Jansa <[email protected]
<mailto:[email protected]>>
> ---
> meta/lib/oe/sstatesig.py | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --gita/meta/lib/oe/sstatesig.pyb/meta/lib/oe/sstatesig.py> index6cd6e11acc..e86a09b332100644
> ---a/meta/lib/oe/sstatesig.py
> +++b/meta/lib/oe/sstatesig.py
> @@ -453,7 +453,7 @@ def find_sstate_manifest(taskdata, taskdata2,
taskname, d, multilibcache):
> manifest =
d2.expand("${SSTATE_MANIFESTS}/manifest-%s-%s.%s" % (pkgarch, taskdata, taskname))
> ifos.path.exists(manifest):
> return manifest, d2
> - bb.error("Manifest %s not found in %s (variant '%s')?" %
(manifest,d2.expand(" ".join(pkgarchs)), variant))
> + bb.fatal("Manifest %s not found in %s (variant '%s')?" %
(manifest,d2.expand(" ".join(pkgarchs)), variant))
> return None, d2doesn't that make the return statement obsolete? also how can an error be overlooked? It does make the task triggering this issue fail, so how is a fatal better in this case (besides abnormally terminating the task)
bb.error doesn't make the task to fail, it only shows an error (which results in non-zero exit code), but lets depending tasks to continue and fail with other (often more misleading) errors, because their dependency is really missing in RSS
without this patch:
ERROR: adwaita-icon-theme-3.34.3-r0do_prepare_recipe_sysroot: Manifest /OE/build/oe-core/tmp-glibc/sstate-control/manifest-x86_64_ubuntu-21.04-at-spi2-core-native.populate_sysroot not found inx86_64x86_64_ubuntu-21.04 (variant '')?
ERROR: adwaita-icon-theme-3.34.3-r0do_prepare_recipe_sysroot: Manifest /OE/build/oe-core/tmp-glibc/sstate-control/manifest-x86_64_ubuntu-21.04-at-spi2-core-native.populate_sysroot not found inx86_64x86_64_ubuntu-21.04 (variant '')?
ERROR: adwaita-icon-theme-3.34.3-r0do_configure: Manifest /OE/build/oe-core/tmp-glibc/sstate-control/manifest-x86_64_ubuntu-21.04-at-spi2-core-native.populate_sysroot not found inx86_64x86_64_ubuntu-21.04 (variant '')?
ERROR: adwaita-icon-theme-3.34.3-r0do_install: Manifest /OE/build/oe-core/tmp-glibc/sstate-control/manifest-x86_64_ubuntu-21.04-at-spi2-core-native.populate_sysroot not found inx86_64x86_64_ubuntu-21.04 (variant '')?
ERROR: adwaita-icon-theme-3.34.3-r0do_package: Manifest /OE/build/oe-core/tmp-glibc/sstate-control/manifest-x86_64_ubuntu-21.04-at-spi2-core-native.populate_sysroot not found inx86_64x86_64_ubuntu-21.04 (variant '')?
ERROR: adwaita-icon-theme-3.34.3-r0do_package_write_ipk: Manifest /OE/build/oe-core/tmp-glibc/sstate-control/manifest-x86_64_ubuntu-21.04-at-spi2-core-native.populate_sysroot not found inx86_64x86_64_ubuntu-21.04 (variant '')?
NOTE: Tasks Summary: Attempted 2819 tasks of which 2803 didn't need to be rerun and all succeeded.
Summary: There were 6 ERROR messages shown, returning a non-zero exit code.with this patch:
ERROR: adwaita-icon-theme-3.34.3-r0do_prepare_recipe_sysroot: Manifest /OE/build/oe-core/tmp-glibc/sstate-control/manifest-x86_64_ubuntu-21.04-at-spi2-core-native.populate_sysroot not found inx86_64x86_64_ubuntu-21.04 (variant '')?
ERROR: Logfile of failure stored in: /OE/build/oe-core/tmp-glibc/work/all-oe-linux/adwaita-icon-theme/3.34.3-r0/temp/log.do_prepare_recipe_sysroot.1638704
ERROR: Task (/OE/build/oe-core/openembedded-core/meta/recipes-gnome/gnome/adwaita-icon-theme_3.34.3.bb:do_prepare_recipe_sysroot) failed with exit code '1'
NOTE: Tasks Summary: Attempted 2805 tasks of which 2800 didn't need to be rerun and 1 failed.
Summary: 1 task failed:/OE/build/oe-core/openembedded-core/meta/recipes-gnome/gnome/adwaita-icon-theme_3.34.3.bb:do_prepare_recipe_sysroot
Summary: There was 1 ERROR message shown, returning a non-zero exit code.Without this fix:
https://git.openembedded.org/openembedded-core/commit/?id=63da9a4f889c5b0e41bc8ec08abe0acea1546479 <https://git.openembedded.org/openembedded-core/commit/?id=63da9a4f889c5b0e41bc8ec08abe0acea1546479>I was seeing various native dependencies missing and as you can see in the adwaita-icon-theme example, missing at-spi2-core-native isn't fatal for the build, but were adwaita-icon-theme packages and sstate built correctly? If you don't investigate all these ERROR messages and non-zero exit code (e.g. just because you have thousands of builds per day populating the same sstate mirror), you might miss some of these breakages. While using fatal prevents adwaita-icon-theme to even start building and producing possibly broken packages, images, sstate.
Quoted 7 lines
In some other cases e.g. qtwayland-native missing when building target qtwayland you will get
do_compilefailure complaining that the the binary from qtwayland-native is missing, but the root cause of this issue is written many lines abovedo_compilelog. You can see some other examples in https://git.openembedded.org/openembedded-core-contrib/commit/?h=jansa/master&id=817f1c14cec4d66504a5028908a4ebac147f3b52 <https://git.openembedded.org/openembedded-core-contrib/commit/?h=jansa/master&id=817f1c14cec4d66504a5028908a4ebac147f3b52> and before changing this from bb.warn to bb.error in previous patch most of these issues were just "some random builddir breakage which was resolved by doing clean build from sstate".And this happens in multi-machine builds with dunfell/gatesgarth/hardknott and only master is safe now. I plan to ask Steve to backport this and related fixes to dunfell after a bit more time in master:
https://git.openembedded.org/openembedded-core/commit/?id=0eb95212a7e6b6bdc1243094072dea432cb48f0f <https://git.openembedded.org/openembedded-core/commit/?id=0eb95212a7e6b6bdc1243094072dea432cb48f0f>
https://git.openembedded.org/openembedded-core/commit/?id=63da9a4f889c5b0e41bc8ec08abe0acea1546479 <https://git.openembedded.org/openembedded-core/commit/?id=63da9a4f889c5b0e41bc8ec08abe0acea1546479>
https://git.openembedded.org/openembedded-core/commit/?id=95607a26854d873399d2b9d7e5fcbffc0cbdba4c <https://git.openembedded.org/openembedded-core/commit/?id=95607a26854d873399d2b9d7e5fcbffc0cbdba4c>
and then this one.
Thanks for the detailed explanation, if it reduces the noise (incl false positives) I'm all for it
Quoted 2 lines
Regards,
On Wed, May 19, 2021 at 6:46 PM Konrad Weihmann <[email protected]> wrote:
If I am not mistaken this breaks wide areas of meta-freescale + meta-qt5-extra. I have seen the manifest warning using meta-freescale for long time (and many licensed not found in do_rootfs) but now builds fail.
Most recipes failing are native recipes inheriting meta-qt5's cmake_qt5-bbcalss - have many of them in meta-qt5-extra.
To analyse I did:
[superandy@thanks-buchmuellers oe-core]$ bitbake -e extra-cmake-modules-native | grep ^DEPENDS DEPENDS="qtbase qtbase-native cmake-native qtbase-native qtbase-native cmake-native qttools-native python3-native python3-native ninja-native"
As soon as I remove (target) qtbase from DEPENDS, build is fine. Have the strong feeling that the dynamic-packagearch magic [1] is the culprit confusing a native recipe finding target recipe's manifest if that recipe is marked as SO-SPECIFIC - as qtbase [2] is.
Wrote here because I hope the audience is wider - will open an issue at meta-freescale tomorrow after zzz...
Cheers,
Andreas
[1] https://github.com/Freescale/meta-freescale/blob/master/classes/fsl-dynamic-packagearch.bbclass
[2] https://github.com/Freescale/meta-freescale/blob/733e1683ce8895a960c01a3ee1fce40118d82982/conf/machine/include/imx-base.inc#L246
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 16 lines · Andreas Müller via lists.openembedded.org
On Wed, May 19, 2021 at 6:46 PM Konrad Weihmann <[email protected]> wrote:
If I am not mistaken this breaks wide areas of meta-freescale + meta-qt5-extra. I have seen the manifest warning using meta-freescale for long time (and many licensed not found indo_rootfs) but now builds fail.
Most recipes failing are native recipes inheriting meta-qt5's cmake_qt5-bbcalss - have many of them in meta-qt5-extra.
To analyse I did:[superandy@thanks-buchmuellers oe-core]$ bitbake -e extra-cmake-modules-native | grep ^DEPENDS DEPENDS="qtbase qtbase-native cmake-native qtbase-native qtbase-native cmake-native qttools-native python3-native python3-native ninja-native"
Quoted 7 lines
As soon as I remove (target) qtbase from DEPENDS, build is fine. Have the strong feeling that the dynamic-packagearch magic [1] is the culprit confusing a native recipe finding target recipe's manifest if that recipe is marked as SO-SPECIFIC - as qtbase [2] is.
Wrote here because I hope the audience is wider - will open an issue at meta-freescale tomorrow after zzz...