sstate_clean() overzealous?
The sstate_clean() function in sstate.bbclass is doing (inter alia):
stfile = d.getVar("STAMP", True) + ".do_" + ss['task']
[...]
oe.path.remove(stfile + ".*")
oe.path.remove(stfile + "_setscene" + ".*")which means that, for tasks which set the stamp-extra-info flag to ${MACHINE}, it ends up blowing away the stamps for all machines rather than just the current one. The net effect of this seems to be that there is no way to have the setscene stamps populated for more than one MACHINE at any time, and hence all those tasks get rerun every time you change MACHINE even if nothing else has been altered.
Is this behaviour deliberate? It is certainly a little bit annoying but I don't understand the internals of sstate well enough to judge what exactly that glob is trying to accomplish.
thanks
p.
Quoted 14 lines · Phil Blundell
The
sstate_clean()function in sstate.bbclass is doing (inter alia):stfile =
d.getVar("STAMP", True)+ ".do_" + ss['task'] [...]
oe.path.remove(stfile + ".*")oe.path.remove(stfile + "_setscene" + ".*")which means that, for tasks which set the stamp-extra-info flag to ${MACHINE}, it ends up blowing away the stamps for all machines rather than just the current one. The net effect of this seems to be that there is no way to have the setscene stamps populated for more than one MACHINE at any time, and hence all those tasks get rerun every time you change MACHINE even if nothing else has been altered.
Quoted 3 lines
Is this behaviour deliberate? It is certainly a little bit annoying but I don't understand the internals of sstate well enough to judge what exactly that glob is trying to accomplish.
I've noticed that changing the MACHINE will also invalidate the -native stuff as well.
-M
Quoted 15 lines · On 10/3/2011 2:11 PM, McClintock Matthew-B29882 wrote:
Quoted 14 lines · Phil Blundell
The
sstate_clean()function in sstate.bbclass is doing (inter alia):stfile = d.getVar("STAMP", True) + ".do_" + ss['task'] [...] oe.path.remove(stfile + ".*") oe.path.remove(stfile + "_setscene" + ".*")which means that, for tasks which set the stamp-extra-info flag to ${MACHINE}, it ends up blowing away the stamps for all machines rather than just the current one. The net effect of this seems to be that there is no way to have the setscene stamps populated for more than one MACHINE at any time, and hence all those tasks get rerun every time you change MACHINE even if nothing else has been altered.
Quoted 6 lines
Quoted 3 lines
Is this behaviour deliberate? It is certainly a little bit annoying but I don't understand the internals of sstate well enough to judge what exactly that glob is trying to accomplish.
I've noticed that changing the MACHINE will also invalidate the -native stuff as well.
hmmm so we can not share sstate parts for two machines in same tmpdir ?
Quoted 7 lines
-M
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
Quoted 1 line · Khem Raj
hmmm so we can not share sstate parts for two machines in same tmpdir ?
I'm not 100% sure I'm doing things properly but this is what I did:
$ bitbake pseudo-native $ cp sstate-cache/* ~/sstate-cache $ rm -rf tmp/ $ bitbake pseudo-native
Now sstate-cache is verified to be working.... then I changed machines and ran
$ cp ~/sstate-cache/* sstate-cache/ $ bitbake pseudo-native
bitbake starts rebuilding everything, and added -DDD I see SState cache is missing.
-M
Quoted 15 lines · On 10/03/2011 02:11 PM, McClintock Matthew-B29882 wrote:
Quoted 14 lines · Phil Blundell
The
sstate_clean()function in sstate.bbclass is doing (inter alia):stfile = d.getVar("STAMP", True) + ".do_" + ss['task'] [...] oe.path.remove(stfile + ".*") oe.path.remove(stfile + "_setscene" + ".*")which means that, for tasks which set the stamp-extra-info flag to ${MACHINE}, it ends up blowing away the stamps for all machines rather than just the current one. The net effect of this seems to be that there is no way to have the setscene stamps populated for more than one MACHINE at any time, and hence all those tasks get rerun every time you change MACHINE even if nothing else has been altered.
Quoted 7 lines
Quoted 3 lines
Is this behaviour deliberate? It is certainly a little bit annoying but I don't understand the internals of sstate well enough to judge what exactly that glob is trying to accomplish.
I've noticed that changing the MACHINE will also invalidate the -native stuff as well.
I am not sure that's correct, since we use sstate with the Autobuilder and it's shared across different hosts building different MACHINEs, we don't see the -natives getting build, do you have any examples?
Sau!
Quoted 7 lines
-M
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
Quoted 3 lines · Saul Wold
I am not sure that's correct, since we use sstate with the Autobuilder and it's shared across different hosts building different MACHINEs, we don't see the -natives getting build, do you have any examples?
I'm building with targets that are not upstream, so I can't comment on the upstream variant. There are different TUNES though. I'm actually currently in the process of looking at why sstate-cache is not being reused.
Can you point at the autobuilder stuff? Is it going from fsl-mpc8315e-rdb to and x86_64 target? Or between two x86_64 targets?
-M
Quoted 5 lines · Matthew McClintock
Quoted 3 lines · Saul Wold
I am not sure that's correct, since we use sstate with the Autobuilder and it's shared across different hosts building different MACHINEs, we don't see the -natives getting build, do you have any examples?
Quoted 7 lines
I'm building with targets that are not upstream, so I can't comment on the upstream variant. There are different TUNES though. I'm actually currently in the process of looking at why sstate-cache is not being reused.
Can you point at the autobuilder stuff? Is it going from fsl-mpc8315e-rdb to and
x86_64target? Or between twox86_64targets?
For example autoconf-native:
$ bitbake-diffsigs
../sstate-cache/sstate-autoconf-native-x86_64-linux-2.68-r2-x86_64-2-*siginfo
[mattsm@right build_p5020ds_release (master *$)]$ bitbake-diffsigs
../sstate-cache/sstate-autoconf-native-x86_64-linux-2.68-r2-x86_64-2-*siginfo
[snip]
basehash changed from bab02dfa833aca4810b0fae8c4ab48f9 to
e156f3fb86236bfc0262e6287bd73784
List of dependencies for variable baselib changed from
set(['BASELIB']) to set(['DEFAULTTUNE'])
Dependency on variable BASELIB was added
Dependency on Variable DEFAULTTUNE was removed
Variable baselib value changed from ${BASELIB} to
${@d.getVar('BASE_LIB_tune-' + (d.getVar('DEFAULTTUNE', True) or
'INVALID'), True) or 'lib'}
Hash for dependent task
virtual:native:/local/home/mattsm/git/poky/meta/recipes-devtools/autoconf/autoconf_2.68.bb.do_install
changed from e74616029727b6e61b00273425128f1d to
478e7900a2310f2669708e6909883865Back tracking this appears to be a multilib issue since I have that enabled on one of the -native packages:
../meta/conf/bitbake.conf:baselib = "${BASELIB}" ../meta/conf/multilib.conf:baselib = "${@d.getVar('BASE_LIB_tune-' + (d.getVar('DEFAULTTUNE', True) or 'INVALID'), True) or 'lib'}"
Which causes the other differences....
-M
Quoted 35 lines · On Mon, 2011-10-03 at 22:08 +0000, McClintock Matthew-B29882 wrote:
Quoted 5 lines · Matthew McClintock
Quoted 3 lines · Saul Wold
I am not sure that's correct, since we use sstate with the Autobuilder and it's shared across different hosts building different MACHINEs, we don't see the -natives getting build, do you have any examples?
Quoted 7 lines
I'm building with targets that are not upstream, so I can't comment on the upstream variant. There are different TUNES though. I'm actually currently in the process of looking at why sstate-cache is not being reused.
Can you point at the autobuilder stuff? Is it going from fsl-mpc8315e-rdb to and
x86_64target? Or between twox86_64targets?For example autoconf-native:
$ bitbake-diffsigs ../sstate-cache/sstate-autoconf-native-x86_64-linux-2.68-r2-x86_64-2-*siginfo [mattsm@right build_p5020ds_release (master *$)]$ bitbake-diffsigs ../sstate-cache/sstate-autoconf-native-x86_64-linux-2.68-r2-x86_64-2-*siginfo [snip] basehash changed from bab02dfa833aca4810b0fae8c4ab48f9 to e156f3fb86236bfc0262e6287bd73784 List of dependencies for variable baselib changed from set(['BASELIB']) to set(['DEFAULTTUNE']) Dependency on variable BASELIB was added Dependency on Variable DEFAULTTUNE was removed Variable baselib value changed from ${BASELIB} to ${@d.getVar('BASE_LIB_tune-' + (d.getVar('DEFAULTTUNE', True) or 'INVALID'), True) or 'lib'} Hash for dependent task virtual:native:/local/home/mattsm/git/poky/meta/recipes-devtools/autoconf/autoconf_2.68.bb.do_install changed from e74616029727b6e61b00273425128f1d to 478e7900a2310f2669708e6909883865
Quoted 7 lines
Back tracking this appears to be a multilib issue since I have that enabled on one of the -native packages:
../meta/conf/bitbake.conf:baselib = "${BASELIB}" ../meta/conf/multilib.conf:baselib = "${@d.getVar('BASE_LIB_tune-' + (
d.getVar('DEFAULTTUNE', True)or 'INVALID'), True) or 'lib'}"
This is bad and shouldn't be happening, particularly for the native packages. I did accidentally commit part of a work in progress in this commit:
http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=ea0b7440ec5a8663b66f56a19f0b38e803b523d3
I didn't revert it since I thought it was a good idea and fixed some problems I was seeing.
I suspect a baselib = "lib" in native.bbclass might be a good idea too...
Cheers,
Richard
Quoted 7 lines · Phil Blundell
The
sstate_clean()function in sstate.bbclass is doing (inter alia):stfile = d.getVar("STAMP", True) + ".do_" + ss['task'] [...] oe.path.remove(stfile + ".*") oe.path.remove(stfile + "_setscene" + ".*")
Quoted 8 lines
which means that, for tasks which set the stamp-extra-info flag to ${MACHINE}, it ends up blowing away the stamps for all machines rather than just the current one. The net effect of this seems to be that there is no way to have the setscene stamps populated for more than one MACHINE at any time, and hence all those tasks get rerun every time you change MACHINE even if nothing else has been altered.
Is this behaviour deliberate?
No, its not intended to be wiping out all the different machines.
Quoted 3 lines
It is certainly a little bit annoying but
I don't understand the internals of sstate well enough to judge what exactly that glob is trying to accomplish.
The trouble is we now support a variety of different stamp file formats depending on what settings you have enabled (and metadata you're using).
Clean can be called from two sources, do_clean, or just before a sstate controlled task is about to be run. In either case we want to clear out any previous "state" including any stamp files that might have represented this task since we're about to invalidate anything they did. This means wiping out "normal" stamps and any setscene variants of those stamps. We also want to remove any stamps representing other hashes for this task since there is no version installed.
If the BasicHash signature generator code is enabled this adds hashes to the stamp files. This turns the stamps into the form:
<stampfile>.<hash>
or in the case of machine specific tasks:
<stampfile>.<hash>.<machine>
I think we need to teach this clean function about stamp-extra-info and instead of stfile + ".*", remove stfile + ".*." + stamp-extra-info if stamp-extra-info is set. That would likely fix the problem you're seeing.
Cheers,
Richard
Quoted 4 lines · Richard Purdie
I think we need to teach this clean function about stamp-extra-info and instead of stfile + ".*", remove stfile + ".*." + stamp-extra-info if stamp-extra-info is set. That would likely fix the problem you're seeing.
Okay, thanks. I applied this change locally:
meta/classes/sstate.bbclass +7 -2
@@ -258,10 +258,15 @@ def sstate_clean(ss, d): bb.utils.unlockfile(lock) stfile = d.getVar("STAMP", True) + ".do_" + ss['task'] + extrainf = d.getVarFlag("do_" + ss['task'], 'stamp-extra-info') oe.path.remove(stfile) oe.path.remove(stfile + "_setscene") - oe.path.remove(stfile + ".*") - oe.path.remove(stfile + "_setscene" + ".*") + if extrainf: + oe.path.remove(stfile + ".*." + extrainf) + oe.path.remove(stfile + "_setscene" + ".*." + extrainf) + else: + oe.path.remove(stfile + ".*") + oe.path.remove(stfile + "_setscene" + ".*") CLEANFUNCS += "sstate_cleanall"
and it does indeed seem to resolve the problem I was seeing before.
p.
Quoted 14 lines · Phil Blundell
Quoted 4 lines · Richard Purdie
I think we need to teach this clean function about stamp-extra-info and instead of stfile + ".*", remove stfile + ".*." + stamp-extra-info if stamp-extra-info is set. That would likely fix the problem you're seeing.
Okay, thanks. I applied this change locally:
meta/classes/sstate.bbclass +0
-0@@ -258,10 +258,15 @@ def sstate_clean(ss, d): bb.utils.unlockfile(lock)
meta/classes/sstate.bbclass def sstate_clean(ss, d): · line 266
Quoted 10 lines
meta/classes/sstate.bbclass +4
-2stfile = d.getVar("STAMP", True) + ".do_" + ss['task'] + extrainf = d.getVarFlag("do_" + ss['task'], 'stamp-extra-info') oe.path.remove(stfile) oe.path.remove(stfile + "_setscene") - oe.path.remove(stfile + ".*") - oe.path.remove(stfile + "_setscene" + ".*") + if extrainf: + oe.path.remove(stfile + ".*." + extrainf) + oe.path.remove(stfile + "_setscene" + ".*." + extrainf)
Thinking further about this it may need to be ".*" instead of ".*.". I suspect with the above it wouldn't be wiping out some of the stamps it should be with OE-Core and the Basic siggen code. BasicHash would work though.
Cheers,
Richard