Skip to contentSkip to conversation

sstatesig.py: show an error instead of warning when sstate manifest isn't found

RFC PATCH Martin Jansa 3 participants 5 messages Feb 23, 2021 – Mar 15, 2021 1 message not in the archive

lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): [email protected]

#1 lore raw
Quoted 15 lines · Martin Jansa via lists.openembedded.org

* not sure if there are some valid use-cases for missing manifest, but
recently I'm seeing increasing number of build failures where something
from native recipe is missing (seen it with pseudo, autoconf, nodejs
recently) and the only indication that something is wrong (before showing
sometimes misleading error like:
recipe-sysroot-native/usr/bin/node: No such file or directory
is this warning:
NOTE: Running task 7844 of 12431 (/meta-oe/meta-oe/recipes-devtools/nodejs/nodejs_12.20.2.bb:do_prepare_recipe_sysroot)
NOTE: recipe nodejs-12.20.2-r0: task do_prepare_recipe_sysroot: Started
WARNING: nodejs-12.20.2-r0 do_prepare_recipe_sysroot: Manifest /BUILD/sstate-control/manifest-x86_64_ubuntu-18.04-nodejs-native.populate_sysroot not found in x86_64 x86_64_ubuntu-18.04 (variant '')?
NOTE: Running task 7845 of 12431 (/meta-oe/meta-oe/recipes-devtools/nodejs/nodejs_12.20.2.bb:do_unpack)
NOTE: recipe nodejs-12.20.2-r0: task do_unpack: Started
WARNING: nodejs-12.20.2-r0 do_prepare_recipe_sysroot: Manifest /BUILD/sstate-control/manifest-x86_64_ubuntu-18.04-nodejs-native.populate_sysroot not found in x86_64 x86_64_ubuntu-18.04 (variant '')?
NOTE: recipe nodejs-12.20.2-r0: task do_prepare_recipe_sysroot: Succeeded

Quoted 6 lines

if I rebuild that native dependency, then it gets fixed and I don't see these failures in clean builds (as without sstate and with empty TMPDIR), only in incremental builds

* but if there isn't valid reason for missing manifest file, then I think it would be better to error early (or even bb.fatal())

Looks like this warning was first introduced in 2014:
https://git.openembedded.org/openembedded-core/commit/?id=79ea036de331bde65a88fb777647dc099ef05acf
then extended and updated for RSS in:
https://git.openembedded.org/openembedded-core/commit/?id=809746f56df4b91af014bf6a3f28997d6698ac78
then added for package management in:
https://git.openembedded.org/openembedded-core/commit/?id=85e72e129362db896b0d368077033e4a2e373cf9
and then unified last 2 to common function in:
https://git.openembedded.org/openembedded-core/commit/?id=68150bac7444f089f19c789e9f6602d59f605d7a

Quoted 25 lines
---
 meta/lib/oe/sstatesig.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
meta/lib/oe/sstatesig.py +1 -1
index 84999ee94d..6cd6e11acc 100644
@@ -453,7 +453,7 @@ def find_sstate_manifest(taskdata, taskdata2, taskname, d, multilibcache):
         manifest = d2.expand("${SSTATE_MANIFESTS}/manifest-%s-%s.%s" % (pkgarch, taskdata, taskname))
         if os.path.exists(manifest):
             return manifest, d2
-    bb.warn("Manifest %s not found in %s (variant '%s')?" % (manifest, d2.expand(" ".join(pkgarchs)), variant))
+    bb.error("Manifest %s not found in %s (variant '%s')?" % (manifest, d2.expand(" ".join(pkgarchs)), variant))
     return None, d2
 
 def OEOuthashBasic(path, sigfile, task, d):
Signature · 1 line
2.27.0
#2 lore raw
Quoted 23 lines · Martin Jansa
Quoted 15 lines · Martin Jansa via lists.openembedded.org

* not sure if there are some valid use-cases for missing manifest, but
  recently I'm seeing increasing number of build failures where something
  from native recipe is missing (seen it with pseudo, autoconf, nodejs
  recently) and the only indication that something is wrong (before showing
  sometimes misleading error like:
  recipe-sysroot-native/usr/bin/node: No such file or directory
  is this warning:
  NOTE: Running task 7844 of 12431 (/meta-oe/meta-oe/recipes-devtools/nodejs/nodejs_12.20.2.bb:do_prepare_recipe_sysroot)
  NOTE: recipe nodejs-12.20.2-r0: task do_prepare_recipe_sysroot: Started
  WARNING: nodejs-12.20.2-r0 do_prepare_recipe_sysroot: Manifest /BUILD/sstate-control/manifest-x86_64_ubuntu-18.04-nodejs-native.populate_sysroot not found in x86_64 x86_64_ubuntu-18.04 (variant '')?
  NOTE: Running task 7845 of 12431 (/meta-oe/meta-oe/recipes-devtools/nodejs/nodejs_12.20.2.bb:do_unpack)
  NOTE: recipe nodejs-12.20.2-r0: task do_unpack: Started
  WARNING: nodejs-12.20.2-r0 do_prepare_recipe_sysroot: Manifest /BUILD/sstate-control/manifest-x86_64_ubuntu-18.04-nodejs-native.populate_sysroot not found in x86_64 x86_64_ubuntu-18.04 (variant '')?
  NOTE: recipe nodejs-12.20.2-r0: task do_prepare_recipe_sysroot: Succeeded

Quoted 6 lines

  if I rebuild that native dependency, then it gets fixed and I don't see these failures in clean builds (as without sstate and with empty TMPDIR), only in incremental builds

* but if there isn't valid reason for missing manifest file, then I think it would be better to error early (or even bb.fatal())

Quoted 8 lines

Looks like this warning was first introduced in 2014:
https://git.openembedded.org/openembedded-core/commit/?id=79ea036de331bde65a88fb777647dc099ef05acf
then extended and updated for RSS in:
https://git.openembedded.org/openembedded-core/commit/?id=809746f56df4b91af014bf6a3f28997d6698ac78
then added for package management in:
https://git.openembedded.org/openembedded-core/commit/?id=85e72e129362db896b0d368077033e4a2e373cf9
and then unified last 2 to common function in:
https://git.openembedded.org/openembedded-core/commit/?id=68150bac7444f089f19c789e9f6602d59f605d7a

I merged this. There were cases where the warning showed in cases where there wasn't a problem but I think we've fixed all those now.

Cheers,

Richard

#3 lore raw
Quoted 33 lines · Richard Purdie
Quoted 23 lines · Martin Jansa
Quoted 15 lines · Martin Jansa via lists.openembedded.org

* not sure if there are some valid use-cases for missing manifest, but
  recently I'm seeing increasing number of build failures where something
  from native recipe is missing (seen it with pseudo, autoconf, nodejs
  recently) and the only indication that something is wrong (before showing
  sometimes misleading error like:
  recipe-sysroot-native/usr/bin/node: No such file or directory
  is this warning:
  NOTE: Running task 7844 of 12431 (/meta-oe/meta-oe/recipes-devtools/nodejs/nodejs_12.20.2.bb:do_prepare_recipe_sysroot)
  NOTE: recipe nodejs-12.20.2-r0: task do_prepare_recipe_sysroot: Started
  WARNING: nodejs-12.20.2-r0 do_prepare_recipe_sysroot: Manifest /BUILD/sstate-control/manifest-x86_64_ubuntu-18.04-nodejs-native.populate_sysroot not found in x86_64 x86_64_ubuntu-18.04 (variant '')?
  NOTE: Running task 7845 of 12431 (/meta-oe/meta-oe/recipes-devtools/nodejs/nodejs_12.20.2.bb:do_unpack)
  NOTE: recipe nodejs-12.20.2-r0: task do_unpack: Started
  WARNING: nodejs-12.20.2-r0 do_prepare_recipe_sysroot: Manifest /BUILD/sstate-control/manifest-x86_64_ubuntu-18.04-nodejs-native.populate_sysroot not found in x86_64 x86_64_ubuntu-18.04 (variant '')?
  NOTE: recipe nodejs-12.20.2-r0: task do_prepare_recipe_sysroot: Succeeded

Quoted 6 lines

  if I rebuild that native dependency, then it gets fixed and I don't see these failures in clean builds (as without sstate and with empty TMPDIR), only in incremental builds

* but if there isn't valid reason for missing manifest file, then I think it would be better to error early (or even bb.fatal())

Quoted 8 lines

Looks like this warning was first introduced in 2014:
https://git.openembedded.org/openembedded-core/commit/?id=79ea036de331bde65a88fb777647dc099ef05acf
then extended and updated for RSS in:
https://git.openembedded.org/openembedded-core/commit/?id=809746f56df4b91af014bf6a3f28997d6698ac78
then added for package management in:
https://git.openembedded.org/openembedded-core/commit/?id=85e72e129362db896b0d368077033e4a2e373cf9
and then unified last 2 to common function in:
https://git.openembedded.org/openembedded-core/commit/?id=68150bac7444f089f19c789e9f6602d59f605d7a

Quoted 2 lines

I merged this. There were cases where the warning showed in cases where there wasn't a problem but I think we've fixed all those now.

Richard, Martin,

Can you please point me to the right way of fixing this? I've been seeing this warning before and since it is now an error and backported to Dunfell, it breaks builds one in a while.

Signature · 1 line
Denys
#4 lore raw
Quoted 34 lines · Denys Dmytriyenko
Quoted 33 lines · Richard Purdie
Quoted 31 lines · Martin Jansa
Quoted 15 lines · Martin Jansa via lists.openembedded.org

* not sure if there are some valid use-cases for missing manifest, but
  recently I'm seeing increasing number of build failures where something
  from native recipe is missing (seen it with pseudo, autoconf, nodejs
  recently) and the only indication that something is wrong (before showing
  sometimes misleading error like:
  recipe-sysroot-native/usr/bin/node: No such file or directory
  is this warning:
  NOTE: Running task 7844 of 12431 (/meta-oe/meta-oe/recipes-devtools/nodejs/nodejs_12.20.2.bb:do_prepare_recipe_sysroot)
  NOTE: recipe nodejs-12.20.2-r0: task do_prepare_recipe_sysroot: Started
  WARNING: nodejs-12.20.2-r0 do_prepare_recipe_sysroot: Manifest /BUILD/sstate-control/manifest-x86_64_ubuntu-18.04-nodejs-native.populate_sysroot not found in x86_64 x86_64_ubuntu-18.04 (variant '')?
  NOTE: Running task 7845 of 12431 (/meta-oe/meta-oe/recipes-devtools/nodejs/nodejs_12.20.2.bb:do_unpack)
  NOTE: recipe nodejs-12.20.2-r0: task do_unpack: Started
  WARNING: nodejs-12.20.2-r0 do_prepare_recipe_sysroot: Manifest /BUILD/sstate-control/manifest-x86_64_ubuntu-18.04-nodejs-native.populate_sysroot not found in x86_64 x86_64_ubuntu-18.04 (variant '')?
  NOTE: recipe nodejs-12.20.2-r0: task do_prepare_recipe_sysroot: Succeeded

Quoted 6 lines

  if I rebuild that native dependency, then it gets fixed and I don't see these failures in clean builds (as without sstate and with empty TMPDIR), only in incremental builds

* but if there isn't valid reason for missing manifest file, then I think it would be better to error early (or even bb.fatal())

Looks like this warning was first introduced in 2014:
https://git.openembedded.org/openembedded-core/commit/?id=79ea036de331bde65a88fb777647dc099ef05acf
then extended and updated for RSS in:
https://git.openembedded.org/openembedded-core/commit/?id=809746f56df4b91af014bf6a3f28997d6698ac78
then added for package management in:
https://git.openembedded.org/openembedded-core/commit/?id=85e72e129362db896b0d368077033e4a2e373cf9
and then unified last 2 to common function in:
https://git.openembedded.org/openembedded-core/commit/?id=68150bac7444f089f19c789e9f6602d59f605d7a

Quoted 8 lines
Quoted 2 lines

I merged this. There were cases where the warning showed in cases where there wasn't a problem but I think we've fixed all those now.

Richard, Martin,

Can you please point me to the right way of fixing this? I've been seeing this warning before and since it is now an error and backported to Dunfell, it breaks builds one in a while.

Do you see this error triggered on initramfs manifest used by your kernel? See this thread for possible work arounds:
https://lists.openembedded.org/g/openembedded-core/topic/81249042#149271

but maybe you have some different corner case which will need different fix.

#5 lore raw
In reply to a message not in the archive
Quoted 15 lines · Martin Jansa via lists.openembedded.org

* not sure if there are some valid use-cases for missing manifest, but
recently I'm seeing increasing number of build failures where something
from native recipe is missing (seen it with pseudo, autoconf, nodejs
recently) and the only indication that something is wrong (before showing
sometimes misleading error like:
recipe-sysroot-native/usr/bin/node: No such file or directory
is this warning:
NOTE: Running task 7844 of 12431 (/meta-oe/meta-oe/recipes-devtools/nodejs/nodejs_12.20.2.bb:do_prepare_recipe_sysroot)
NOTE: recipe nodejs-12.20.2-r0: task do_prepare_recipe_sysroot: Started
WARNING: nodejs-12.20.2-r0 do_prepare_recipe_sysroot: Manifest /BUILD/sstate-control/manifest-x86_64_ubuntu-18.04-nodejs-native.populate_sysroot not found in x86_64 x86_64_ubuntu-18.04 (variant '')?
NOTE: Running task 7845 of 12431 (/meta-oe/meta-oe/recipes-devtools/nodejs/nodejs_12.20.2.bb:do_unpack)
NOTE: recipe nodejs-12.20.2-r0: task do_unpack: Started
WARNING: nodejs-12.20.2-r0 do_prepare_recipe_sysroot: Manifest /BUILD/sstate-control/manifest-x86_64_ubuntu-18.04-nodejs-native.populate_sysroot not found in x86_64 x86_64_ubuntu-18.04 (variant '')?
NOTE: recipe nodejs-12.20.2-r0: task do_prepare_recipe_sysroot: Succeeded

Quoted 6 lines

if I rebuild that native dependency, then it gets fixed and I don't see these failures in clean builds (as without sstate and with empty TMPDIR), only in incremental builds

* but if there isn't valid reason for missing manifest file, then I think it would be better to error early (or even bb.fatal())

FYI:

With bb.fatal() I'm using for my builds I got few more failures now
(which might have been silently ignored like nodejs-native in example
before), this time it was
image.do_prepare_recipe_sysroot:
ERROR: Manifest TOPDIR/BUILD/sstate-control/manifest-x86_64_ubuntu-18.04-bmap-tools-native.populate_sysroot not found in x86_64 x86_64_ubuntu-18.04 (variant '')?

linux-yocto.do_validate_branches
ERROR: Manifest TOPDIR/BUILD/sstate-control/manifest-x86_64_ubuntu-18.04-kern-tools-native.populate_sysroot not found in x86_64 x86_64_ubuntu-18.04 (variant '')?
u-boot.do_prepare_recipe_sysroot
ERROR: Manifest TOPDIR/BUILD/sstate-control/manifest-x86_64_ubuntu-18.04-kern-tools-native.populate_sysroot not found in x86_64 x86_64_ubuntu-18.04 (variant '')?
busybox.do_prepare_recipe_sysroot
NOTE: Direct dependencies are ['TOPDIR/oe-core/meta/recipes-devtools/gcc/gcc-cross_10.2.bb:do_populate_sysroot', 'TOPDIR/oe-core/meta/recipes-devtools/gcc/gcc-runtime_10.2.bb:do_populate_sysroot', 'TOPDIR/oe-core/meta/recipes-devtools/opkg-utils/opkg-utils_0.4.3.bb:do_populate_sysroot', 'TOPDIR/oe-core/meta/recipes-core/systemd/systemd-systemctl-native.bb:do_populate_sysroot', 'TOPDIR/oe-core/meta/recipes-kernel/kern-tools/kern-tools-native_git.bb:do_populate_sysroot', 'TOPDIR/oe-core/meta/recipes-core/glibc/glibc_2.33.bb:do_populate_sysroot', 'virtual:native:TOPDIR/oe-core/meta/recipes-devtools/icecc-create-env/icecc-create-env_0.1.bb:do_populate_sysroot', 'virtual:native:TOPDIR/oe-core/meta/recipes-devtools/pseudo/pseudo_git.bb:do_populate_sysroot', 'TOPDIR/oe-core/meta/recipes-core/libxcrypt/libxcrypt_4.4.18.bb:do_populate_sysroot']
ERROR: Manifest TOPDIR/BUILD/sstate-control/manifest-x86_64_ubuntu-18.04-kern-tools-native.populate_sysroot not found in x86_64 x86_64_ubuntu-18.04 (variant '')?

gstreamer1.0-plugins-base.do_prepare_recipe_sysroot
ERROR: gstreamer1.0-plugins-base-1.18.3-r0 do_prepare_recipe_sysroot: Manifest TOPDIR/BUILD/sstate-control/manifest-x86_64_x86_64-nativesdk-iso-codes.populate_sysroot

And the issue with kern-tools-native was reproduced in 3 different workspaces on different builders which don't share the same SSTATE_MIRROR.

I'll try to debug what happened in these cases, but it was again on jenkins where the workspace was already overwritten by some other builds and the evidence might be lost already and last time I've tried to reproduce this locally it never failed.

I'm pretty sure that previous successful build which left the TMPDIR as well as SSTATE_DIR populated was based on oe-core b27bff26a9 (+ some local modifications) and the current failing once is 69f8f3e213 (+ similar local modifications), and the git log shows that bmap-tools, kern-tools and iso-codes were updated in this:

$ git log --oneline b27bff26a9..69f8f3e213 | tee
69f8f3e213 bmap-tools: upgrade 3.5 -> 3.6
09cfa5ce1e spirv-tools: Replace strncpy with memcpy
f939624981 local.conf.sample.extended: sample value for SDK_PS1
81ed52f245 toolchain-scripts.bbclass: customize prompt string for SDKs
1d19b16846 sanity.bbclass: Add az fetcher to accepted URI protocols for mirrors sanity
7639ffed1a iso-codes: upgrade 4.5.0 -> 4.6.0
ad4e023bb3 help2man: upgrade 1.48.1 -> 1.48.2
a7d7c1412b sysklogd: upgrade 2.2.1 -> 2.2.2
d34cc88cb7 python3-zipp: upgrade 3.4.0 -> 3.4.1
27da7a57b0 python3-pygments: upgrade 2.8.0 -> 2.8.1
9f8d049dbb runqemu: modify novga to conform to documentation
9dea4cd2f9 runqemu: correct forcing of ttyS0
695c98b652 runqemu: add QB_GRAPHICS
7290b77348 meta/recipes-devtools: Add HOMEPAGE / DESCRIPTION
5ab087950c meta/recipes-graphics: Add HOMEPAGE / DESCRIPTION
ab7a945f6e meta/recipes-multimedia: Add HOMEPAGE / DESCRIPTION
693988ec42 meta/recipes-kernel: Add HOMEPAGE / DESCRIPTION
325db7fea1 meta/recipes-support: Add HOMEPAGE / DESCRIPTION
ffdce193f3 ffmpeg: upgrade 4.3.1 -> 4.3.2
07c09631d8 linux-yocto/qemuarmv5: fix configuration warning
7a48aad3d3 linux-yocto/5.10: update to v5.10.21
eea536b436 kern-tools: symbol-why fix and README update
820b552341 linux-yocto/5.4: update to v5.4.103
f9a5dc7eeb parselogs: Allow expected kernel messages for qemuppc64
be3a6b9791 inittab: Add getty launch on hvc0 for qemuppc64
13bc8fa9d7 formfactor: Add machine config for qemuppc64
3a741b0f39 linux-yocto: Enable powerpc64le QEMU
68275b25f0 qemuppc64: Add a QEMU machine definition for ppc64
20d9e13372 qemu: fix CVE-2021-20203
5610a5b4ef machine: Initial commit of qemuriscv32
8a40989f03 linux-yocto: Add qemuriscv32 as a compatible machine
22691df60a openssl: Enable building for RISC-V 32-bit
74e5c43802 buildstats.bbclass: improve timeout handling

At least I'm seeing this only with master, I'm using the same bb.fatal in dunfell and gatesgarth based builds and so far those triggered only the issue with initramfs discussed in the other thread.

It might be just being more lucky as there are fewer changes in dunfell/gatesgarth compared with master.

Cheers,

Quoted 25 lines
---
 meta/lib/oe/sstatesig.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
meta/lib/oe/sstatesig.py +1 -1
index 84999ee94d..6cd6e11acc 100644
@@ -453,7 +453,7 @@ def find_sstate_manifest(taskdata, taskdata2, taskname, d, multilibcache):
         manifest = d2.expand("${SSTATE_MANIFESTS}/manifest-%s-%s.%s" % (pkgarch, taskdata, taskname))
         if os.path.exists(manifest):
             return manifest, d2
-    bb.warn("Manifest %s not found in %s (variant '%s')?" % (manifest, d2.expand(" ".join(pkgarchs)), variant))
+    bb.error("Manifest %s not found in %s (variant '%s')?" % (manifest, d2.expand(" ".join(pkgarchs)), variant))
     return None, d2
 
 def OEOuthashBasic(path, sigfile, task, d):
Signature · 1 line
2.27.0