Skip to contentSkip to conversation

license manifest files effectively disappear when do_rootfs is not rerun

DISCUSSION Christian Thießen 3 participants 7 messages Oct 24, 2022 – Oct 25, 2022

lore
Share & tools
lore
#0 lore raw

(Cross-post: I first posted to https://lists.yoctoproject.org/g/yocto/message/58392, then realized that according to the poky readme this list is more appropriate. Sorry, beginner's mistake)

Dear list,
when building an image with a Yocto setup based on kirkstone, it seems I've come across an interesting corner case: The image built successfully, the build/tmp/deploy/licenses/${IMAGE}-${MACHINE} symlink got updated as expected, but that new directory contained only image_license.manifest. license.manifest and package.manifest were missing.
I've discovered that these latter files are generated by the image's do_rootfs task (ROOTFS_POSTPROCESS_COMMAND:prepend = "write_package_manifest; license_create_manifest; " in poky/meta/classes/license_image.bbclass). This task apparently did not need to be rerun in that particular build.
This behavior is at least a bit surprising to the user; bitbake claims to have run everything that needed to be run, completes successfully and then some files are just not there. It might be what happened here https://lists.yoctoproject.org/g/yocto/message/49600 too. For me, it broke a CI build that ran due to an environment change that didn't affect the image and then failed to deploy the manifest files. I could run bitbake with -C rootfs as a workaround, but that causes a warning and the problem could probably better be fixed within the oe-core classes. I'm not sure how though: Could the manifest generation be moved to a different point in the build, e.g. where image_license.manifest is generated? Or should build/tmp/deploy/licenses/${IMAGE}-${MACHINE} become a directory containing symlinks to files, instead of a symlink to a new directory, so that things that don't need to be updated can be retained? Maybe you have a completely different idea? I'll leave that up for discussion. Thank you,

Christian Thießen

#1 lore raw

Generally, you should not be deleting things from tmp/deploy (or tmp/ for that matter) - either delete all of tmp/ and rely on sstate or don't touch it at all, and just re-run the build.

It helps if you provide steps to reproduce this behavior.

Alex

Quoted 19 lines · Christian Thießen via lists.openembedded.org

(Cross-post: I first posted to https://lists.yoctoproject.org/g/yocto/message/58392, then realized that according to the poky readme this list is more appropriate. Sorry, beginner's mistake)

Dear list,
when building an image with a Yocto setup based on kirkstone, it seems I've come across an interesting corner case: The image built successfully, the build/tmp/deploy/licenses/${IMAGE}-${MACHINE} symlink got updated as expected, but that new directory contained only image_license.manifest. license.manifest and package.manifest were missing.
I've discovered that these latter files are generated by the image's do_rootfs task (ROOTFS_POSTPROCESS_COMMAND:prepend = "write_package_manifest; license_create_manifest; " in poky/meta/classes/license_image.bbclass). This task apparently did not need to be rerun in that particular build.
This behavior is at least a bit surprising to the user; bitbake claims to have run everything that needed to be run, completes successfully and then some files are just not there. It might be what happened here https://lists.yoctoproject.org/g/yocto/message/49600 too. For me, it broke a CI build that ran due to an environment change that didn't affect the image and then failed to deploy the manifest files. I could run bitbake with -C rootfs as a workaround, but that causes a warning and the problem could probably better be fixed within the oe-core classes. I'm not sure how though: Could the manifest generation be moved to a different point in the build, e.g. where image_license.manifest is generated? Or should build/tmp/deploy/licenses/${IMAGE}-${MACHINE} become a directory containing symlinks to files, instead of a symlink to a new directory, so that things that don't need to be updated can be retained? Maybe you have a completely different idea? I'll leave that up for discussion. Thank you,

Christian Thießen

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#172077): https://lists.openembedded.org/g/openembedded-core/message/172077
Mute This Topic: https://lists.openembedded.org/mt/94530086/1686489
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-

#2 lore raw

I'm not deleting anything.
Steps to reproduce would be:
Build just about any image for any machine, as long as it uses the standard classes like license_image.bbclass.
You'll get a build/tmp/deploy/licenses/${IMAGE}-${MACHINE} symlink pointing to build/tmp/deploy/licenses/${IMAGE}-${MACHINE}-<datetime> containing all the expected manifests. Run the same build again. Bitbake will rerun do_populate_lic_deploy, but not do_rootfs.
build/tmp/deploy/licenses/${IMAGE}-${MACHINE} now points to a new directory that contains only image_license.manifest.

Christian

#3 lore raw

If I run the same build again, no tasks will be executed, and accordingly nothing in tmp/deploy will change:

alex@Zen2:/srv/work/alex/poky/build-64$ bitbake core-image-minimal
...
NOTE: Tasks Summary: Attempted 2862 tasks of which 2862 didn't need to
be rerun and all succeeded.
alex@Zen2:/srv/work/alex/poky/build-64$ ls -l
tmp/deploy/licenses/core-image-minimal-qemux86-64/
total 16
-rw-r--r-- 1 alex alex  256 Oct 24 11:33 image_license.manifest
-rw-r--r-- 1 alex alex 6451 Oct 24 11:32 license.manifest
-rw-r--r-- 1 alex alex  958 Oct 24 11:32 package.manifest

So something happens between builds on your side, and there should be a reproducer for it.

Alex

Quoted 18 lines · Christian Thießen via lists.openembedded.org

I'm not deleting anything.
Steps to reproduce would be:
Build just about any image for any machine, as long as it uses the standard classes like license_image.bbclass.
You'll get a build/tmp/deploy/licenses/${IMAGE}-${MACHINE} symlink pointing to build/tmp/deploy/licenses/${IMAGE}-${MACHINE}-<datetime> containing all the expected manifests. Run the same build again. Bitbake will rerun do_populate_lic_deploy, but not do_rootfs.
build/tmp/deploy/licenses/${IMAGE}-${MACHINE} now points to a new directory that contains only image_license.manifest.

Christian

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#172079): https://lists.openembedded.org/g/openembedded-core/message/172079
Mute This Topic: https://lists.openembedded.org/mt/94530086/1686489
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-

#4 lore raw

Thanks, you're right; a normal rebuild doesn't execute any tasks. I've now found that this is related to rm_work: If you enable it with INHERIT+=" rm_work" in local.conf, bitbake will rerun three tasks on each rebuild: populate_lic_deploy, rm_work and build. After that you get the missing manifests. So another option for a workaround would be to exclude the image recipe from rm_work.

Christian

#5 lore raw

Hi Christian,

Quoted 3 lines · Christian Thießen via lists.openembedded.org

Thanks, you're right; a normal rebuild doesn't execute any tasks. I've now found that this is related to rm_work: If you enable it with INHERIT+=" rm_work" in local.conf, bitbake will rerun three tasks on each rebuild: populate_lic_deploy, rm_work and build. After that you get the missing manifests. So another option for a workaround would be to exclude the image recipe from rm_work.

Good job at identifying the culprit, would you mind sending us a patch and/or open a bug on our Bugzilla to keep track of that?

Cheers,
Quentin