sstate hash equivalence breaks rm_work
Hey all,
As part of the packagegroup-core-base-utils series I noticed that with rm_work enabled like I usually have, every build was starting over with rebuilding linux-libc-headers and going down from there. I finished bisecting this now and it comes down to:
commit d889acb4f8f06f09cece80fa12661725e6e5f037
Author: Joshua Watt <[email protected]> Date: Fri Jan 4 10:20:15 2019 -0600
sstate: Implement hash equivalence sstate
This can be reproduced easily with qemux86, core-image-minimal, oe-core and this in site.conf:
# Save more space
INHERIT += "rm_work"
# But keep this around for debuggingRM_WORK_EXCLUDE += ""
INHERIT += "buildstats"
INHERIT += "buildstats-summary"
(I doubt the last two matter, but I only noticed that was in my bisect just now).
Signature · 1 line
Tom
Quoted 6 lines · Tom Rini
Hey all,
As part of the packagegroup-core-base-utils series I noticed that with
rm_workenabled like I usually have, every build was starting over with rebuilding linux-libc-headers and going down from there. I finished bisecting this now and it comes down to:
Just for reference, are you trying to use hash equivalence? It sounds like not, but I would like to make sure.
Also, it there an AB build that uses rm_work that would have seen this?
Quoted 21 lines
commit
d889acb4f8f06f09cece80fa12661725e6e5f037Author: Joshua Watt <[email protected]> Date: Fri Jan 4 10:20:15 2019 -0600sstate: Implement hash equivalence sstate
This can be reproduced easily with qemux86, core-image-minimal, oe-core and this in site.conf:
# Save more space
INHERIT += "rm_work"
# But keep this around for debuggingRM_WORK_EXCLUDE+= ""
INHERIT += "buildstats"
INHERIT += "buildstats-summary"(I doubt the last two matter, but I only noticed that was in my bisect just now).
--
Tom
Quoted 11 lines · Joshua Watt
Quoted 6 lines · Tom Rini
Hey all,
As part of the packagegroup-core-base-utils series I noticed that with
rm_workenabled like I usually have, every build was starting over with rebuilding linux-libc-headers and going down from there. I finished bisecting this now and it comes down to:Just for reference, are you trying to use hash equivalence? It sounds like not, but I would like to make sure.
I am not.
Quoted 1 line
Also, it there an AB build that uses
rm_workthat would have seen this?
OK, so I see I forgot a step that was clear to me, but, uh, not in the email. What breaks is for example:
bitbake core-image-minimal && bitbake core-image-minimal That second build, rather than see "nothing to do" (or at least, almost nothing to do) says "Oh, I need to build linux-libc-headers from scratch" and possibly other stuff too, but that's the first thing I notice.
Signature · 1 line
Tom
Quoted 18 lines · Tom Rini
Quoted 11 lines · Joshua Watt
Quoted 6 lines · Tom Rini
Hey all,
As part of the packagegroup-core-base-utils series I noticed that with
rm_workenabled like I usually have, every build was starting over with rebuilding linux-libc-headers and going down from there. I finished bisecting this now and it comes down to:Just for reference, are you trying to use hash equivalence? It sounds like not, but I would like to make sure.
I am not.
Quoted 1 line
Also, it there an AB build that uses
rm_workthat would have seen this?
Quoted 7 lines
OK, so I see I forgot a step that was clear to me, but, uh, not in the email. What breaks is for example:
bitbake core-image-minimal && bitbake core-image-minimal That second build, rather than see "nothing to do" (or at least, almost nothing to do) says "Oh, I need to build linux-libc-headers from scratch" and possibly other stuff too, but that's the first thing I notice.
Ok. I've started a build to test it. Would you mind opening a Bugzilla for it and assigning it to me?
For posterity, the poky commit SHA is adc37721a86c
Quoted 3 lines
--
Tom
Quoted 28 lines · Joshua Watt
Quoted 18 lines · Tom Rini
Quoted 11 lines · Joshua Watt
Quoted 6 lines · Tom Rini
Hey all,
As part of the packagegroup-core-base-utils series I noticed that with
rm_workenabled like I usually have, every build was starting over with rebuilding linux-libc-headers and going down from there. I finished bisecting this now and it comes down to:Just for reference, are you trying to use hash equivalence? It sounds like not, but I would like to make sure.
I am not.
Quoted 1 line
Also, it there an AB build that uses
rm_workthat would have seen this?Quoted 7 lines
OK, so I see I forgot a step that was clear to me, but, uh, not in the email. What breaks is for example:
bitbake core-image-minimal && bitbake core-image-minimal That second build, rather than see "nothing to do" (or at least, almost nothing to do) says "Oh, I need to build linux-libc-headers from scratch" and possibly other stuff too, but that's the first thing I notice.
Quoted 4 lines
Ok. I've started a build to test it. Would you mind opening a Bugzilla for it and assigning it to me?
For posterity, the poky commit SHA is
adc37721a86c
Hmm, FWIW I just tried this on my setup with rm_work enabled and it worked as expected (e.g. did not rebuild unnecessarily).... would you be able to share more of your setup (local.conf, layers.conf, site.conf)?
Quoted 4 lines
Quoted 4 lines
--
Tom
Quoted 32 lines · Joshua Watt
Quoted 31 lines · Joshua Watt
Quoted 18 lines · Tom Rini
Quoted 11 lines · Joshua Watt
Quoted 6 lines · Tom Rini
Hey all,
As part of the packagegroup-core-base-utils series I noticed that with
rm_workenabled like I usually have, every build was starting over with rebuilding linux-libc-headers and going down from there. I finished bisecting this now and it comes down to:Just for reference, are you trying to use hash equivalence? It sounds like not, but I would like to make sure.
I am not.
Quoted 1 line
Also, it there an AB build that uses
rm_workthat would have seen this?Quoted 7 lines
OK, so I see I forgot a step that was clear to me, but, uh, not in the email. What breaks is for example:
bitbake core-image-minimal && bitbake core-image-minimal That second build, rather than see "nothing to do" (or at least, almost nothing to do) says "Oh, I need to build linux-libc-headers from scratch" and possibly other stuff too, but that's the first thing I notice.Ok. I've started a build to test it. Would you mind opening a Bugzilla for it and assigning it to me?
Quoted 6 lines
Quoted 1 line
For posterity, the poky commit SHA is
adc37721a86cHmm, FWIW I just tried this on my setup with
rm_workenabled and it worked as expected (e.g. did not rebuild unnecessarily).... would you be able to share more of your setup (local.conf, layers.conf, site.conf)?
On oe-core/master, local.conf / layers.conf are as setup. All of site.conf is:DL_DIR = "${HOME}/work/OE/downloads"SSTATE_DIR = "${HOME}/work/OE/sstate-cache-${DISTRO_VERSION}${TARGET_VENDOR}"
# Save more space
INHERIT += "rm_work"
# But keep this around for debuggingRM_WORK_EXCLUDE += ""
INHERIT += "buildstats"
INHERIT += "buildstats-summary"
And:
$ lsb_release -a No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 16.04.6 LTS Release: 16.04 Codename: xenial
Signature · 1 line
Tom
Quoted 1 line · Tom Rini
SSTATE_DIR= "${HOME}/work/OE/sstate-cache-${DISTRO_VERSION}${TARGET_VENDOR}"
FWIW there isn't much point in isolating sstate like that. You'll just rebuild stuff that would have been available.
Ross
There is, you can prune outdated duplicate sstate archives while still separately building multiple DISTRO_VERSIONs.
Quoted 14 lines · Ross
Quoted 1 line · Tom Rini
SSTATE_DIR="${HOME}/work/OE/sstate-cache-${DISTRO_VERSION}${TARGET_VENDOR}"
FWIW there isn't much point in isolating sstate like that. You'll just rebuild stuff that would have been available.
Ross
--
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-core
Quoted 1 line · Martin Jansa
There is, you can prune outdated duplicate sstate archives while still separately building multiple
DISTRO_VERSIONs.
We don't all have 4TB disks? :)
Ross
4TB of sstate is just for few weeks for me :).
Quoted 8 lines · Ross
Quoted 1 line · Martin Jansa
There is, you can prune outdated duplicate sstate archives while still
separately building multiple
DISTRO_VERSIONs.We don't all have 4TB disks? :)
Ross
Quoted 5 lines · Ross
Quoted 1 line · Tom Rini
SSTATE_DIR= "${HOME}/work/OE/sstate-cache-${DISTRO_VERSION}${TARGET_VENDOR}"FWIW there isn't much point in isolating sstate like that. You'll just rebuild stuff that would have been available.
Martin already chimed in, but this is my "copy this to every project" site.conf file. I do modify that base path a little bit for per project, But easily removable state is good and important, especially when chasing "recipe for proprietary thing behaves badly" bugs.
Signature · 1 line
Tom
Quoted 27 lines · Joshua Watt
Quoted 18 lines · Tom Rini
Quoted 11 lines · Joshua Watt
Quoted 6 lines · Tom Rini
Hey all,
As part of the packagegroup-core-base-utils series I noticed that with
rm_workenabled like I usually have, every build was starting over with rebuilding linux-libc-headers and going down from there. I finished bisecting this now and it comes down to:Just for reference, are you trying to use hash equivalence? It sounds like not, but I would like to make sure.
I am not.
Quoted 1 line
Also, it there an AB build that uses
rm_workthat would have seen this?Quoted 7 lines
OK, so I see I forgot a step that was clear to me, but, uh, not in the email. What breaks is for example:
bitbake core-image-minimal && bitbake core-image-minimal That second build, rather than see "nothing to do" (or at least, almost nothing to do) says "Oh, I need to build linux-libc-headers from scratch" and possibly other stuff too, but that's the first thing I notice.
Quoted 4 lines
Ok. I've started a build to test it. Would you mind opening a Bugzilla for it and assigning it to me?
For posterity, the poky commit SHA is
adc37721a86c
Ah, so "poky" got me thinking this morning. I was just on bitbake v1.40 (since I usually use thud, and that's the version for that). Moving to bitbake master/tip and I don't see the problem anymore.
Is it possible to have some logic within OE to check the version of bitbake and throw an error / warning?
Signature · 1 line
Tom
Quoted 8 lines · Tom Rini
Ah, so "poky" got me thinking this morning. I was just on bitbake v1.40
(since I usually use thud, and that's the version for that). Moving to
bitbake master/tip and I don't see the problem anymore.Is it possible to have some logic within OE to check the version of bitbake and throw an error / warning?
You mean like the BB_MIN_VERSION variable:
http://git.yoctoproject.org/cgit.cgi/poky/tree/meta/conf/sanity.conf#n6
?
:)
The changes were intended to be backwards compatible hence it wasn't bumped. Clearly something isn't playing as intended...
Cheers,
Richard
Quoted 12 lines · Richard Purdie
Quoted 8 lines · Tom Rini
Ah, so "poky" got me thinking this morning. I was just on bitbake v1.40
(since I usually use thud, and that's the version for that). Moving to
bitbake master/tip and I don't see the problem anymore.Is it possible to have some logic within OE to check the version of bitbake and throw an error / warning?
You mean like the
BB_MIN_VERSIONvariable:
Quoted 8 lines
http://git.yoctoproject.org/cgit.cgi/poky/tree/meta/conf/sanity.conf#n6
?
:)
The changes were intended to be backwards compatible hence it wasn't bumped. Clearly something isn't playing as intended...
Yes, that. I think we haven't fully used that well historically since I can only ever recall *kaboom* when using too old of a bitbake rather than a nice sane error message. Looking at
https://wiki.yoctoproject.org/wiki/Releases and then some quick git history, we also must have had that mechanism available when I last ran into something like that even for the fairly ancient jumping around I've had to do a time or three.
In fact, can we make BB_MIN_VERSION match the table found there? It doesn't look like we have resources / time to test things on both current and min versions, so we should I think have the min match what we say should be used.
Signature · 1 line
Tom
Quoted 13 lines · Tom Rini
Quoted 11 lines · Richard Purdie
You mean like the
BB_MIN_VERSIONvariable:http://git.yoctoproject.org/cgit.cgi/poky/tree/meta/conf/sanity.conf#n6
?
:)
The changes were intended to be backwards compatible hence it wasn't
bumped. Clearly something isn't playing as intended...
Quoted 7 lines
Yes, that. I think we haven't fully used that well historically since I can only ever recall *kaboom* when using too old of a bitbake rather than a nice sane error message. Looking at
https://wiki.yoctoproject.org/wiki/Releases and then some quick git history, we also must have had that mechanism available when I last ran into something like that even for the fairly ancient jumping around I've had to do a time or three.
That variable has been around for a long time. It does get incremented when we do nasty things which we know break things. If something sneaks in, its more problematic. We do try and avoid API breakage.
Quoted 4 lines
In fact, can we make
BB_MIN_VERSIONmatch the table found there? It doesn't look like we have resources / time to test things on both current and min versions, so we should I think have the min match what we say should be used.
The problem is more "test what?" on the min version. In this case its rm_work which triggers it, the bugs tend to be unusual corner cases which our test matrix, vast as it is might not catch. Case in hand, we don't test rm_work on the autobuilder.
I'm torn on forcing people to update, I've had a lot of complaints about being being forced when they didn't need to in the past. Its also true that we're not as backwards compatible as we could be, i.e. using a modern bitbake on old metadata isn't always guaranteed either due to the same kinds of problems :(.
By this line of argument, we'd bump the minor bitbake version for every bug fix and force people to it, just to "protect" them from themselves :/
Cheers,
Richard
Quoted 42 lines · Richard Purdie
Quoted 13 lines · Tom Rini
Quoted 11 lines · Richard Purdie
You mean like the
BB_MIN_VERSIONvariable:http://git.yoctoproject.org/cgit.cgi/poky/tree/meta/conf/sanity.conf#n6
?
:)
The changes were intended to be backwards compatible hence it wasn't
bumped. Clearly something isn't playing as intended...Quoted 7 lines
Yes, that. I think we haven't fully used that well historically since I can only ever recall *kaboom* when using too old of a bitbake rather than a nice sane error message. Looking at
https://wiki.yoctoproject.org/wiki/Releases and then some quick git history, we also must have had that mechanism available when I last ran into something like that even for the fairly ancient jumping around I've had to do a time or three.That variable has been around for a long time. It does get incremented when we do nasty things which we know break things. If something sneaks in, its more problematic. We do try and avoid API breakage.
Quoted 4 lines
In fact, can we make
BB_MIN_VERSIONmatch the table found there? It doesn't look like we have resources / time to test things on both current and min versions, so we should I think have the min match what we say should be used.The problem is more "test what?" on the min version. In this case its
rm_workwhich triggers it, the bugs tend to be unusual corner cases which our test matrix, vast as it is might not catch. Case in hand, we don't testrm_workon the autobuilder.I'm torn on forcing people to update, I've had a lot of complaints about being being forced when they didn't need to in the past. Its also true that we're not as backwards compatible as we could be, i.e. using a modern bitbake on old metadata isn't always guaranteed either due to the same kinds of problems :(.
Quoted 3 lines
By this line of argument, we'd bump the minor bitbake version for every bug fix and force people to it, just to "protect" them from themselves :/
So, what's worse, people hit a bug that's fixed in a newer version, or when people upgrade one set of branches they have to update another repository too?
Honestly, I could see the "but why do I have to update bitbake too?" complaint being more reasonable back in the early days after we moved to proper layers. Now that it's not just oe-core/poky but also meta-openembedded and meta-$BSP and maybe 2-5 other layers you also need to bump from branch to branch, adding bitbake to that mix seems a lot less of a big deal.
I thought about this and re-drafted something a few times now. I think perhaps a happy compromise would be to move the BB_MIN_VERSION check from a hard-failure to the same level of warning we do on SANITY_TESTED_DISTROS, and make it track the suggested version for a release. This will both help to catch bugs that have been fixed already but still let people that know what they want (or don't want) to do, do it.
Signature · 1 line
Tom
Quoted 12 lines · Richard Purdie
Quoted 8 lines · Tom Rini
Ah, so "poky" got me thinking this morning. I was just on bitbake v1.40
(since I usually use thud, and that's the version for that). Moving to
bitbake master/tip and I don't see the problem anymore.Is it possible to have some logic within OE to check the version of bitbake and throw an error / warning?
You mean like the
BB_MIN_VERSIONvariable:
Quoted 9 lines
http://git.yoctoproject.org/cgit.cgi/poky/tree/meta/conf/sanity.conf#n6
?
:)
The changes were intended to be backwards compatible hence it wasn't bumped. Clearly something isn't playing as intended...
Ah, older bitbake isn't going to set BB_UNIHASH, which I bet is what breaks it. I bet if we add:
BB_UNIHASH ?= "${BB_TASKHASH}"
somewhere (bitbake.conf?) it would fix it.
Quoted 5 lines
Cheers,
Richard