Using SSTATE_MIRRORS with sstate subdirectories

I'm having trouble using SSTATE_MIRRORS as suggested at
https://wiki.yoctoproject.org/wiki/Enable_sstate_cache :

SSTATE_MIRRORS ?= "file://.* file:///private/sstate-cache/"

Since commit 5d45c7b025f6635b2232d7bf92b7c1aba350396b my sstate-cache files are stored in two-letter subdirectories (and 9360fc318cec5c873b17f59b817dd6312c81ee27 adds the LSB prefix too.) Although these components are present in the original URL they don't make it into the rewritten one:

DEBUG: SState using premirror of: file://.* file:///private/sstate-cache/
DEBUG: SState: Attempting to fetch file://Debian-testing/b0/sstate-quilt-native-x86_64-linux-0.60-r0-x86_64-2-b0869ac9e0e74960018909e56a6822a1_populate-sysroot.tgz
DEBUG: Searching for Debian-testing/b0/sstate-quilt-native-x86_64-linux-0.60-r0-x86_64-2-b0869ac9e0e74960018909e56a6822a1_populate-sysroot.tgz in paths:
DEBUG: Searching for Debian-testing/b0/sstate-quilt-native-x86_64-linux-0.60-r0-x86_64-2-b0869ac9e0e74960018909e56a6822a1_populate-sysroot.tgz in path: /vg0disk/mac/src/oe2/build-bcm7425/conf/.
DEBUG: Defaulting to /vg0disk/mac/src/oe2/build-bcm7425/sstate-cache/Debian-testing/b0/sstate-quilt-native-x86_64-linux-0.60-r0-x86_64-2-b0869ac9e0e74960018909e56a6822a1_populate-sysroot.tgz for Debian-testing/b0/sstate-quilt-native-x86_64-linux-0.60-r0-x86_64-2-b0869ac9e0e74960018909e56a6822a1_populate-sysroot.tgz
DEBUG: Testing URL file://Debian-testing/b0/sstate-quilt-native-x86_64-linux-0.60-r0-x86_64-2-b0869ac9e0e74960018909e56a6822a1_populate-sysroot.tgz
DEBUG: For url ['file', '', 'Debian-testing/b0/sstate-quilt-native-x86_64-linux-0.60-r0-x86_64-2-b0869ac9e0e74960018909e56a6822a1_populate-sysroot.tgz', '', '', {}] comparing ['file', '', '.*', '', '', {}] to ['file', '', '/private/sstate-cache/', '', '', {}]
DEBUG: For url file://Debian-testing/b0/sstate-quilt-native-x86_64-linux-0.60-r0-x86_64-2-b0869ac9e0e74960018909e56a6822a1_populate-sysroot.tgz returning file:///private/sstate-cache/sstate-quilt-native-x86_64-linux-0.60-r0-x86_64-2-b0869ac9e0e74960018909e56a6822a1_populate-sysroot.tgz
DEBUG: For url ['file', '', '/private/sstate-cache/sstate-quilt-native-x86_64-linux-0.60-r0-x86_64-2-b0869ac9e0e74960018909e56a6822a1_populate-sysroot.tgz', '', '', {}] comparing ['file', '', '.*', '', '', {}] to ['file', '', '/private/sstate-cache/', '', '', {}]
DEBUG: SState: Unsuccessful fetch test for file://Debian-testing/b0/sstate-quilt-native-x86_64-linux-0.60-r0-x86_64-2-b0869ac9e0e74960018909e56a6822a1_populate-sysroot.tgz

Inspired by 9360fc318cec5c873b17f59b817dd6312c81ee27 I tried:

SSTATE_MIRRORS ?= "file://(.*) file:///private/sstate-cache/\1"

But this just resulted in an infinite loop with progressively more private/sstate-cache/ components added to the path name.

I've had a poke at fetch2's uri_replace but it isn't obvious to me how to change this behaviour without risking breaking something else.

Any ideas?

Thanks.

Mike.

SSTATE_MIRRORS ?= "file://.* file:///private/sstate-cache/PATH"

Christopher Larson

Note: with master / danny, but not denzil or earlier.

Cheers,
Paul

Paul Eggleton
Intel Open Source Technology Centre

Thanks Christopher. That works.

I'm wary of attempting to update the wiki page for that reason.

Mike.

Thanks for your reply.

Although that works for the simple case, if I try and do something more adventurous like:

SSTATE_MIRRORS = "\
file://Debian-testing/.* file:///private/sstate-cache/Debian-6.0.6/PATH \n \
file://.* file:///private/sstate-cache/PATH \n \
"

Then I get paths like:

DEBUG: For url file://Debian-testing/8c/sstate-tar-replacement-native-x86_64-linux-1.26-r3-x86_64-2-8cc4342260b064ace38e0aa1acf2f618_populate-sysroot.tgz returning file:///private/sstate-cache/Debian-6.0.6/Debian-testing/8c/sstate-tar-replacement-native-x86_64-linux-1.26-r3-x86_64-2-8cc4342260b064ace38e0aa1acf2f618_populate-sysroot.tgz

so I really would like to be able to say:

"file://Debian-testing/(.*) file:///private/sstate-cache/Debian-6.0.6/\1"

Thanks.

Mike.

SSTATE_MIRRORS = "\
file://${LSBNATIVESTRING} file:///private/sstate-cache/Debian-6.0.6 \n \
file://.* file:///private/sstate-cache/PATH \n \
"

We do this at mentor with a class to simplify the setup. See
http://git.yoctoproject.org/cgit/cgit.cgi/meta-mentor/tree/classes/sstate-reuse.bbclass.
Then you'd do this:

INHERIT += "sstate-reuse" SSTATE_MIRROR_DISTROS = "Debian-testing" SSTATE_MIRROR_SITES = "file:///private/sstate-cache"

We do most of our automated builds on 32 bit and 64 bit centos 5.4 hosts, then those native/cross sstates can then be reused just about everywhere, since that distro is so old.

Christopher Larson

Hi Chris,

Thanks for your reply.

After correcting ${LSBNATIVESTRING} to ${NATIVELSBSTRING} that worked!

I'm still somewhat baffled as to why that one doesn't require PATH but the general one does but that no longer matters to me.

Thanks. That looks interesting. I'm trying to do something similar, except our main automated build machine is Debian stable and the "clients" are Debian stable, testing and Ubuntu recent. I'd made the same assumption that binaries compiledon Debian stable should run on all the others.

Thanks again.

Mike.

The file://.* in the other one dose a replacement of the full url contents, so you have to add back the directory component. The other one I showed you just does a search/replace on the leading component, leaving the rest alone.

Christopher Larson

Thanks again Chris. That makes sense now. I would now expect:

file:// file:///private/sstate-cache

to work just as well as:

file://.* file:///private/sstate-cache/PATH

I'll try that later.

Anyway, I ended up with:

SSTATE_MIRRORS = "\
file://Debian-[^/]* file:///private/sstate-cache/Debian-6 \n \
file://Ubuntu-[^/]* file:///private/sstate-cache/Debian-6 \n \
file://.* file:///private/sstate-cache/PATH \n \

along with a tweaked version of your lsb_distro_adjust hook from
http://git.yoctoproject.org/cgit/cgit.cgi/meta-mentor/tree/conf/distro/include/sstate.inc
to collapse Debian minor versions.

This seems to do exactly what I want.

Thanks to everyone who responded.

Mike.

Something like this might be what you are looking for:

SSTATE_MIRRORS = "file://.*/(.*)/(.*)
http://linux.freescale.net/yocto/sstate-cache/CentOS-5.8/\1/\2 \n"

This maps all native sstate-cache to my CentOS-5 box.

-M