package.tgz files not touched in sstate-cache

We have a shared sstate-cache that is populated by us building nightly images for various branches using Jenkins. This cache is pruned periodically (with special care of the siginfo files since we don't have 80b3974081c4a8c604e23982a6db8fb32c616058 yet.)

Recently we went back to one of those branches that hadn't been changed for a while and were surprised to find that Bitbake wanted to build loads of packages from source.

It turned out that whilst the populate_sysroot.gz and package_ipk.tgz files were still in the sstate-cache, the package.tgz tarball was not. This meant that the whole package was built again in order to satisfy the build dependencies of other packages that had changed.

I'm suspicious that this behaviour was introduced by the optimisation in a0f584ac3d5a94dec121b684206ecd40c968f7fc so we hadn't seen it on older branches.

Is there a way to cause the package.tgz file to be touched during these nightly builds too? Or, is there a way to get hold of the complete set of sstate files so they can be touched all in one go?

Quoted 14 lines · Mike Crowe

We have a shared sstate-cache that is populated by us building nightly images for various branches using Jenkins. This cache is pruned periodically (with special care of the siginfo files since we don't have 80b3974081c4a8c604e23982a6db8fb32c616058 yet.)

Recently we went back to one of those branches that hadn't been changed for a while and were surprised to find that Bitbake wanted to build loads of packages from source.

It turned out that whilst the populate_sysroot.gz and package_ipk.tgz files were still in the sstate-cache, the package.tgz tarball was not. This meant that the whole package was built again in order to satisfy the build dependencies of other packages that had changed.

Quoted 7 lines

I'm suspicious that this behaviour was introduced by the optimisation in a0f584ac3d5a94dec121b684206ecd40c968f7fc so we hadn't seen it on older branches.

Is there a way to cause the package.tgz file to be touched during these nightly builds too? Or, is there a way to get hold of the complete set of sstate files so they can be touched all in one go?

The "right" answer here is to figure out what in your builds seems to be needing the package.tgz files.

The only time the system should need to pull the package tgzs from sstate is if package_qa or package_write_* were going to rerun. This doesn't happen often, if ever. I've sometimes wondered if we should in fact generate them. They are useful for switching package backends.

So the fact they're not getting touched doesn't surprise me. The more concerning thing is that its actually trying to use them.

We have found a number of bugs in this area in sstate and have some patches merged, I know there are a small number pending too. Whether we've found all the issues remains unknown.

For debugging this, you usually need the task graph info and the -DDDv output of a build, then investigate what was depending on the package sstate file.

Quoted 17 lines · Richard Purdie

Quoted 4 lines · Mike Crowe

It turned out that whilst the populate_sysroot.gz and package_ipk.tgz files were still in the sstate-cache, the package.tgz tarball was not. This meant that the whole package was built again in order to satisfy the build dependencies of other packages that had changed.

The "right" answer here is to figure out what in your builds seems to be needing the package.tgz files.

The only time the system should need to pull the package tgzs from sstate is if package_qa or package_write_* were going to rerun. This doesn't happen often, if ever. I've sometimes wondered if we should in fact generate them. They are useful for switching package backends.

So the fact they're not getting touched doesn't surprise me. The more concerning thing is that its actually trying to use them.

Quoted 3 lines

We have found a number of bugs in this area in sstate and have some patches merged, I know there are a small number pending too. Whether we've found all the issues remains unknown.

The branch we saw this on was using dizzy from about e848484989307ae6826ba0f5217f7702322181e3 so it's quite likely that we're lacking some more recent fixes.