license manifest files effectively disappear when do_rootfs is not rerun
(Cross-post: I first posted to https://lists.yoctoproject.org/g/yocto/message/58392, then realized that according to the poky readme this list is more appropriate. Sorry, beginner's mistake)
Dear list,
when building an image with a Yocto setup based on kirkstone, it seems I've come across an interesting corner case: The image built successfully, the build/tmp/deploy/licenses/${IMAGE}-${MACHINE} symlink got updated as expected, but that new directory contained only image_license.manifest. license.manifest and package.manifest were missing.
I've discovered that these latter files are generated by the image's do_rootfs task (ROOTFS_POSTPROCESS_COMMAND:prepend = "write_package_manifest; license_create_manifest; " in poky/meta/classes/license_image.bbclass). This task apparently did not need to be rerun in that particular build.
This behavior is at least a bit surprising to the user; bitbake claims to have run everything that needed to be run, completes successfully and then some files are just not there. It might be what happened here https://lists.yoctoproject.org/g/yocto/message/49600 too. For me, it broke a CI build that ran due to an environment change that didn't affect the image and then failed to deploy the manifest files. I could run bitbake with -C rootfs as a workaround, but that causes a warning and the problem could probably better be fixed within the oe-core classes. I'm not sure how though: Could the manifest generation be moved to a different point in the build, e.g. where image_license.manifest is generated? Or should build/tmp/deploy/licenses/${IMAGE}-${MACHINE} become a directory containing symlinks to files, instead of a symlink to a new directory, so that things that don't need to be updated can be retained? Maybe you have a completely different idea? I'll leave that up for discussion. Thank you,
Christian Thießen
Generally, you should not be deleting things from tmp/deploy (or tmp/ for that matter) - either delete all of tmp/ and rely on sstate or don't touch it at all, and just re-run the build.
It helps if you provide steps to reproduce this behavior.
Alex
Quoted 19 lines · Christian Thießen via lists.openembedded.org
(Cross-post: I first posted to https://lists.yoctoproject.org/g/yocto/message/58392, then realized that according to the poky readme this list is more appropriate. Sorry, beginner's mistake)
Dear list,
when building an image with a Yocto setup based on kirkstone, it seems I've come across an interesting corner case: The image built successfully, the build/tmp/deploy/licenses/${IMAGE}-${MACHINE} symlink got updated as expected, but that new directory contained only image_license.manifest. license.manifest and package.manifest were missing.
I've discovered that these latter files are generated by the image'sdo_rootfstask (ROOTFS_POSTPROCESS_COMMAND:prepend = "write_package_manifest;license_create_manifest; " in poky/meta/classes/license_image.bbclass). This task apparently did not need to be rerun in that particular build.
This behavior is at least a bit surprising to the user; bitbake claims to have run everything that needed to be run, completes successfully and then some files are just not there. It might be what happened here https://lists.yoctoproject.org/g/yocto/message/49600 too. For me, it broke a CI build that ran due to an environment change that didn't affect the image and then failed to deploy the manifest files. I could run bitbake with -C rootfs as a workaround, but that causes a warning and the problem could probably better be fixed within the oe-core classes. I'm not sure how though: Could the manifest generation be moved to a different point in the build, e.g. where image_license.manifest is generated? Or should build/tmp/deploy/licenses/${IMAGE}-${MACHINE} become a directory containing symlinks to files, instead of a symlink to a new directory, so that things that don't need to be updated can be retained? Maybe you have a completely different idea? I'll leave that up for discussion. Thank you,Christian Thießen
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#172077): https://lists.openembedded.org/g/openembedded-core/message/172077
Mute This Topic: https://lists.openembedded.org/mt/94530086/1686489
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
I'm not deleting anything.
Steps to reproduce would be:
Build just about any image for any machine, as long as it uses the standard classes like license_image.bbclass.
You'll get a build/tmp/deploy/licenses/${IMAGE}-${MACHINE} symlink pointing to build/tmp/deploy/licenses/${IMAGE}-${MACHINE}-<datetime> containing all the expected manifests. Run the same build again. Bitbake will rerun do_populate_lic_deploy, but not do_rootfs.
build/tmp/deploy/licenses/${IMAGE}-${MACHINE} now points to a new directory that contains only image_license.manifest.
Christian
If I run the same build again, no tasks will be executed, and accordingly nothing in tmp/deploy will change:
alex@Zen2:/srv/work/alex/poky/build-64$ bitbake core-image-minimal ... NOTE: Tasks Summary: Attempted 2862 tasks of which 2862 didn't need to be rerun and all succeeded.
alex@Zen2:/srv/work/alex/poky/build-64$ ls -l tmp/deploy/licenses/core-image-minimal-qemux86-64/ total 16 -rw-r--r-- 1 alex alex 256 Oct 24 11:33 image_license.manifest -rw-r--r-- 1 alex alex 6451 Oct 24 11:32 license.manifest -rw-r--r-- 1 alex alex 958 Oct 24 11:32 package.manifest
So something happens between builds on your side, and there should be a reproducer for it.
Alex
Quoted 18 lines · Christian Thießen via lists.openembedded.org
I'm not deleting anything.
Steps to reproduce would be:
Build just about any image for any machine, as long as it uses the standard classes like license_image.bbclass.
You'll get a build/tmp/deploy/licenses/${IMAGE}-${MACHINE} symlink pointing to build/tmp/deploy/licenses/${IMAGE}-${MACHINE}-<datetime> containing all the expected manifests. Run the same build again. Bitbake will rerundo_populate_lic_deploy, but notdo_rootfs.
build/tmp/deploy/licenses/${IMAGE}-${MACHINE} now points to a new directory that contains only image_license.manifest.Christian
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#172079): https://lists.openembedded.org/g/openembedded-core/message/172079
Mute This Topic: https://lists.openembedded.org/mt/94530086/1686489
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
Thanks, you're right; a normal rebuild doesn't execute any tasks. I've now found that this is related to rm_work: If you enable it with INHERIT+=" rm_work" in local.conf, bitbake will rerun three tasks on each rebuild: populate_lic_deploy, rm_work and build. After that you get the missing manifests. So another option for a workaround would be to exclude the image recipe from rm_work.
Christian
Hi Christian,
Quoted 3 lines · Christian Thießen via lists.openembedded.org
Thanks, you're right; a normal rebuild doesn't execute any tasks. I've now found that this is related to
rm_work: If you enable it with INHERIT+="rm_work" in local.conf, bitbake will rerun three tasks on each rebuild:populate_lic_deploy,rm_workand build. After that you get the missing manifests. So another option for a workaround would be to exclude the image recipe fromrm_work.
Good job at identifying the culprit, would you mind sending us a patch and/or open a bug on our Bugzilla to keep track of that?
Cheers,
Quentin
I don't feel familiar enough with the inner workings of oe-core to decide what a good (robust, maintainable) solution would be, but I've created a bug report with some suggestions:
https://bugzilla.yoctoproject.org/show_bug.cgi?id=14946