Skip to contentSkip to conversation

Figuring out why sstate-cache is no longer used for a package

DISCUSSION Mike Looijmans 4 participants 4 messages Dec 10, 2013

lore
Share & tools
lore
#0 lore raw

I've been struggling with this for a few days.

We have a build server that build various images overnight. One of the packages in that image is "fpga-image", which takes more than an hour to build.

We have been sharing the the build server's sstate-cache via HTTP and this has worked excellently up until yesterday.

The current situation is that a client will grab everything from the buildserver's HTTP sstate-cache, potentially finishing a build from scratch in about five minutes. However, for some reason, the fpga-image does not fall into this category, and eache machine insists on re-building it from scratch. I've been trying to debug this, but the sstate-cache is on another machine. I tried copying part of the build server's sstate-cache onto my machine, but that only results in "bitbake-diffsigs -t fpga-image .." yielding "ERROR: No sigdata files found matching fpga-image .." so that apparently is a dead end.

How can I determine what is causing the system to think that it needs to rebuild this package?

And/Or how do tell the system that this package only depends on the SRCREV and MACHINE and nothing else? (I've already set INHIBIT_DEFAULT_DEPS = "1", is there more that I can do?)

--
Mike Looijmans - TOPIC Automation

#1 lore raw
Quoted 9 lines · Mike Looijmans

The current situation is that a client will grab everything from the buildserver's HTTP sstate-cache, potentially finishing a build from scratch in about five minutes. However, for some reason, the fpga-image does not fall into this category, and eache machine insists on re-building it from scratch. I've been trying to debug this, but the sstate-cache is on another machine. I tried copying part of the build server's sstate-cache onto my machine, but that only results in "bitbake-diffsigs -t fpga-image .." yielding "ERROR: No sigdata files found matching fpga-image .." so that apparently is a dead end.

The trick is to use bitbake -S to write stamp data for a particular image and copy that around. The script sstate-diff-machines can help here.

Note that there's been some sstate fallout from the recent
optimisations: it was discovered yesterday that sstate hashes using the "contains()" optimisation wasn't reliable so could produce different hashes on different machines. Try oe-core master from today and see if that's any better: it certainly is for me.

Ross

#2 lore raw
In reply to Mike Looijmans
Quoted 17 lines · Mike Looijmans

I've been struggling with this for a few days.

We have a build server that build various images overnight. One of the packages in that image is "fpga-image", which takes more than an hour to build.

We have been sharing the the build server's sstate-cache via HTTP and this has worked excellently up until yesterday.

The current situation is that a client will grab everything from the buildserver's HTTP sstate-cache, potentially finishing a build from scratch in about five minutes. However, for some reason, the fpga-image does not fall into this category, and eache machine insists on re-building it from scratch. I've been trying to debug this, but the sstate-cache is on another machine. I tried copying part of the build server's sstate-cache onto my machine, but that only results in "bitbake-diffsigs -t fpga-image .." yielding "ERROR: No sigdata files found matching fpga-image .." so that apparently is a dead end.

Quoted 2 lines

How can I determine what is causing the system to think that it needs to rebuild this package?

I use this script:
openembedded-core/scripts/sstate-diff-machines.sh

to create "backup" of sstate signatures and when some next build unexpectedly doesn't reuse some sstate packages I use the script again and compare new and old signatures to see why.

As bonus each build which populates out SSTATE_MIRROR also creates tarball with these signatures, so I can easily download one tarball and compare what and why won't be reused from sstate.

Signature · 1 line
Martin 'JaMa' Jansa     jabber: [email protected]
#3 lore raw
Quoted 33 lines · Martin Jansa
Quoted 4 lines · Mike Looijmans

I've been struggling with this for a few days.

We have a build server that build various images overnight. One of the packages in that image is "fpga-image", which takes more than an hour to

build.

Quoted 2 lines

We have been sharing the the build server's sstate-cache via HTTP and

this has

Quoted 4 lines

worked excellently up until yesterday.

The current situation is that a client will grab everything from the buildserver's HTTP sstate-cache, potentially finishing a build from

scratch in

Quoted 1 line

about five minutes. However, for some reason, the fpga-image does not

fall

Quoted 1 line

into this category, and eache machine insists on re-building it from

scratch.

Quoted 1 line

I've been trying to debug this, but the sstate-cache is on another

machine. I

Quoted 1 line

tried copying part of the build server's sstate-cache onto my machine,

but

Quoted 1 line

that only results in "bitbake-diffsigs -t fpga-image .." yielding

"ERROR: No

Quoted 1 line

sigdata files found matching fpga-image .." so that apparently is a dead

end.

Quoted 3 lines

How can I determine what is causing the system to think that it needs to rebuild this package?

I use this script:
openembedded-core/scripts/sstate-diff-machines.sh

Quoted 7 lines

to create "backup" of sstate signatures and when some next build unexpectedly doesn't reuse some sstate packages I use the script again and compare new and old signatures to see why.

As bonus each build which populates out SSTATE_MIRROR also creates tarball with these signatures, so I can easily download one tarball and compare what and why won't be reused from sstate.

You can also directly use bitbake-whatchanged -v <target>, with STAMPS_DIR set to the automated build’s stamps directory w/ sstates included.

Signature · 5 lines
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics