Skip to contentSkip to conversation

Little sstate reuse when using uninative and yocto sstate cache

DISCUSSION Jacob Kroon 2 participants 3 messages Oct 18, 2021

lore
Share & tools
lore
#0 lore raw

Hi,

I'm on latest dunfell and using uninative and have put the yocto servers in my SSTATE_MIRRORS:

my-distro.conf:
...

Quoted 3 lines

SSTATE_MIRRORS ?= "file://.* http://sstate.yoctoproject.org/3.1.11/PATH;downloadfilename=PATH"
require conf/distro/include/yocto-uninative.inc
INHERIT += "uninative"

...

but I thought I'd be getting more sstate reuse than I am:

Quoted 1 line

bitbake m4-native

...

Quoted 1 line

Sstate summary: Wanted 12 Found 1 Missed 11 Current 0 (8% match, 0% complete)

Is the above reasonable, or should I be able to pull m4-native directly from yocto servers sstate ? If so, is there a way to debug what differs between my local siginfos and the ones on the sstate server ?

Regards Jacob

#1 lore raw
Quoted 12 lines · Jacob Kroon

I'm on latest dunfell and using uninative and have put the yocto servers in my SSTATE_MIRRORS:

my-distro.conf:
...

Quoted 3 lines

SSTATE_MIRRORS ?= "file://.* http://sstate.yoctoproject.org/3.1.11/PATH;downloadfilename=PATH"
require conf/distro/include/yocto-uninative.inc
INHERIT += "uninative"

...

but I thought I'd be getting more sstate reuse than I am:

Quoted 7 lines
Quoted 1 line

bitbake m4-native

...

Quoted 1 line

Sstate summary: Wanted 12 Found 1 Missed 11 Current 0 (8% match, 0% complete)

Is the above reasonable, or should I be able to pull m4-native directly from yocto servers sstate ? If so, is there a way to debug what differs between my local siginfos and the ones on the sstate server ?

In dunfell hash equivalence is enabled and this unfortunately means the sstate doesn't work very well unless you also have the matching hash equivalence data.

We do now have a way to have a readonly hash equivalence server and we have made one available from the autobuilder so this data is shared. Unfortunately there isn't a way to configure dunfell to use one though :(. That patch is a feature but could be something we consider backporting.

That said, I did a lot of testing/work with master recently as sstate reuse with hash equivalence was not what we'd expected. We ended up making a lot of changes, including a bit of a rewrite of the server side of the hash equivalence. I mention this just to illustrate it isn't a simple problem and that fixing this in dunfell may be tricky and would need backporting of some invasive changes. I'm trying to at least get master/honister working right in the first instance.

Cheers,

Richard

#2 lore raw
Quoted 38 lines · Richard Purdie
Quoted 12 lines · Jacob Kroon

I'm on latest dunfell and using uninative and have put the yocto servers in my SSTATE_MIRRORS:

my-distro.conf:
...

Quoted 3 lines

SSTATE_MIRRORS ?= "file://.* http://sstate.yoctoproject.org/3.1.11/PATH;downloadfilename=PATH"
require conf/distro/include/yocto-uninative.inc
INHERIT += "uninative"

...

but I thought I'd be getting more sstate reuse than I am:

Quoted 7 lines
Quoted 1 line

bitbake m4-native

...

Quoted 1 line

Sstate summary: Wanted 12 Found 1 Missed 11 Current 0 (8% match, 0% complete)

Is the above reasonable, or should I be able to pull m4-native directly from yocto servers sstate ? If so, is there a way to debug what differs between my local siginfos and the ones on the sstate server ?

In dunfell hash equivalence is enabled and this unfortunately means the sstate doesn't work very well unless you also have the matching hash equivalence data.

We do now have a way to have a readonly hash equivalence server and we have made one available from the autobuilder so this data is shared. Unfortunately there isn't a way to configure dunfell to use one though :(. That patch is a feature but could be something we consider backporting.

That said, I did a lot of testing/work with master recently as sstate reuse with hash equivalence was not what we'd expected. We ended up making a lot of changes, including a bit of a rewrite of the server side of the hash equivalence. I mention this just to illustrate it isn't a simple problem and that fixing this in dunfell may be tricky and would need backporting of some invasive changes. I'm trying to at least get master/honister working right in the first instance.

Quoted 5 lines

Cheers,

Richard

Thanks for the detailed answer. I will focus on getting this to perform well with my master build then.

Regards
Jacob