What's this
I just updated my Poky repo and did a rebuild and saw this "error"
(although it carried on with the build):
ERROR: base-files-3.0.14-r89 do_install: Taskhash mismatch ceb25febc6f11a5cc2b3c4a4b0b2c136 verses 44f498452dfce5875451bca66b230e31 for /local/poky-cutting-edge/meta/recipes-core/base-files/base-files_3.0.14.bb.do_install
ERROR: Taskhash mismatch ceb25febc6f11a5cc2b3c4a4b0b2c136 verses 44f498452dfce5875451bca66b230e31 for /local/poky-cutting-edge/meta/recipes-core/base-files/base-files_3.0.14.bb.do_install
What does this mean? Should I be worried?
meta = "master:04319c0340010c4e0f78cab252cfcb0ebe23ee84"
Signature · 4 lines
------------------------------------------------------------ Gary Thomas | Consulting for the MLB Associates | Embedded world ------------------------------------------------------------
Quoted 10 lines · Gary Thomas
I just updated my Poky repo and did a rebuild and saw this "error" (although it carried on with the build):
ERROR: base-files-3.0.14-r89do_install: Taskhash mismatchceb25febc6f11a5cc2b3c4a4b0b2c136verses44f498452dfce5875451bca66b230e31for /local/poky-cutting-edge/meta/recipes-core/base-files/base-files_3.0.14.bb.do_install
ERROR: Taskhash mismatchceb25febc6f11a5cc2b3c4a4b0b2c136verses44f498452dfce5875451bca66b230e31for /local/poky-cutting-edge/meta/recipes-core/base-files/base-files_3.0.14.bb.do_install
Quoted 2 lines
What does this mean? Should I be worried?
It means the hash calculated my the bitbake master was different to the hash calculated when the worker started up. This usually means that you're using something like ${TIME} in the recipe but not marking it appropriatly so the cache ignores it. Do you have a base-files bbappend that writes a timestamp?
Ross
Quoted 6 lines · Ross
It means the hash calculated my the bitbake master was different to the hash calculated when the worker started up. This usually means that you're using something like ${TIME} in the recipe but not marking it appropriatly so the cache ignores it. Do you have a base-files bbappend that writes a timestamp?
The always wise Joshua reminds me that if your DISTRO_VERSION contains ${DATETIME} then this happens. If you're doing this then you'll want to set [vardepsexclude] on DISTRO_VERSION to stop the DATETIME from getting into the cache (or not put the current date/time into the distro version).
Ross
Quoted 11 lines · Ross
On 5 June 2016 at 17:44, Gary Thomas <[email protected] <mailto:[email protected]>> wrote:
I just updated my Poky repo and did a rebuild and saw this "error"
(although it carried on with the build):
ERROR: base-files-3.0.14-r89do_install: Taskhash mismatchceb25febc6f11a5cc2b3c4a4b0b2c136verses44f498452dfce5875451bca66b230e31for /local/poky-cutting-edge/meta/recipes-core/base-files/base-files_3.0.14.bb.do_install
ERROR: Taskhash mismatchceb25febc6f11a5cc2b3c4a4b0b2c136verses44f498452dfce5875451bca66b230e31for /local/poky-cutting-edge/meta/recipes-core/base-files/base-files_3.0.14.bb.do_install
Quoted 6 lines
What does this mean? Should I be worried?
It means the hash calculated my the bitbake master was different to the hash calculated when the worker started up. This usually means that you're using something like ${TIME} in the recipe but not marking it appropriatly so the cache ignores it. Do you have a base-files bbappend that writes a timestamp?
Sorry, nothing like that in my layers. Note: this only happened once, immediately after a poky pull and the builds have been happy since. Whatever it was only happened that one time (not that that should be any less concerning...)
Signature · 4 lines
------------------------------------------------------------ Gary Thomas | Consulting for the MLB Associates | Embedded world ------------------------------------------------------------
Quoted 12 lines · Ross
On 7 June 2016 at 17:02, Burton, Ross <[email protected] <mailto:[email protected]>> wrote:
It means the hash calculated my the bitbake master was different to the hash calculated when the worker started up. This usually means that you're using something like ${TIME} in the recipe but not marking it appropriatly so the cache ignores it. Do you have a base-files bbappend that writes a timestamp?
The always wise Joshua reminds me that if your
DISTRO_VERSIONcontains ${DATETIME} then this happens. If you're doing this then you'll want to set [vardepsexclude] onDISTRO_VERSIONto stop the DATETIME from getting into the cache (or not put the current date/time into the distro version).
Ah, that's probably it. Exactly how do I set that exclusion?
Signature · 4 lines
------------------------------------------------------------ Gary Thomas | Consulting for the MLB Associates | Embedded world ------------------------------------------------------------
Quoted 8 lines · Ross
Quoted 6 lines · Ross
It means the hash calculated my the bitbake master was different to the hash calculated when the worker started up. This usually means that you're
using something like ${TIME} in the recipe but not marking it appropriatly so the cache ignores it. Do you have a base-files bbappend that writes a timestamp?
Quoted 4 lines
The always wise Joshua reminds me that if your
DISTRO_VERSIONcontains ${DATETIME} then this happens. If you're doing this then you'll want to set [vardepsexclude] onDISTRO_VERSIONto stop the DATETIME from getting into the cache (or not put the current date/time into the distro version).
I think we need to handle this situation better - if it's really worth producing an error about then it's worth producing an error message that people can actually understand, particularly as it's recently added validation.
Cheers,
Paul
Signature · 2 lines
Paul Eggleton Intel Open Source Technology Centre
Quoted 13 lines · Gary Thomas
Quoted 12 lines · Ross
On 7 June 2016 at 17:02, Burton, Ross <[email protected] <mailto:[email protected]>> wrote:
It means the hash calculated my the bitbake master was different to the hash calculated when the worker started up. This usually means that you're using something like ${TIME} in the recipe but not marking it appropriatly so the
cache ignores it. Do you have a base-files bbappend that writes a timestamp?
Quoted 8 lines
Quoted 6 lines
The always wise Joshua reminds me that if your
DISTRO_VERSIONcontains ${DATETIME} then this happens. If you're doing this then you'll want to set [vardepsexclude] onDISTRO_VERSIONto stop the DATETIME from getting into the cache (or not put the current date/time into the distro version).Ah, that's probably it. Exactly how do I set that exclusion?
Assuming DISTRO_VERSION includes DATETIME
DISTRO_VERSION[vardepsexclude] = "DATETIME"
see for example:
http://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/meta-poky/conf/dist
ro/poky.conf#n14
Regards,
Joshua
Quoted 23 lines · Paul Eggleton
Quoted 13 lines · Ross
Quoted 10 lines · Ross
It means the hash calculated my the bitbake master was different to the
hash calculated when the worker started up. This usually means that
you're
using something like ${TIME} in the recipe but not marking it appropriatly
so the cache ignores it. Do you have a base-files bbappend that writes a
timestamp?Quoted 8 lines
The always wise Joshua reminds me that if your
DISTRO_VERSIONcontains
${DATETIME} then this happens. If you're doing this then you'll want to
set [vardepsexclude] onDISTRO_VERSIONto stop the DATETIME from getting
into the cache (or not put the current date/time into the distro version).
Quoted 6 lines
I think we need to handle this situation better - if it's really worth
producing an error about then it's worth producing an error message that
people can actually understand, particularly as it's recently added validation.
It was silently running into problems due to this all along but not reporting it. Its now reporting it which is better than silently things behaving strangely.
Its very hard for bitbake to know why the hashes differ, it only knows the values afterwards and hence that they've changed, the information about how that hash was constructed is not present in any of bitbake's caches. That implies to have better messages we need to write out more data.
I did add a patch to make bitbake write out data to allow reconstruction of basehash (which is part of taskhash). Sadly the parsing performance was diabolical (10 times slower). I think that could perhaps be improved if the files don't require an atomic move during creation but I haven't had time to look further at it.
So whilst I do agree, what is the price people are willing to pay to have those better messages?
Cheers,
Richard
Quoted 16 lines · On Tue, 07 Jun 2016 23:20:26 Richard Purdie wrote:
Quoted 15 lines · Paul Eggleton
Quoted 13 lines · Ross
Quoted 5 lines · Ross
It means the hash calculated my the bitbake master was different to the hash calculated when the worker started up. This usually means that you're using something like ${TIME} in the recipe but not marking it appropriatly so the cache ignores it. Do you have a base-files bbappend that writes a timestamp?
The always wise Joshua reminds me that if your
DISTRO_VERSIONcontains ${DATETIME} then this happens. If you're doing this then you'll want to set [vardepsexclude] onDISTRO_VERSIONto stop the DATETIME from getting into the cache (or not put the current date/time into the distro version).
Quoted 8 lines
Quoted 4 lines
I think we need to handle this situation better - if it's really worth producing an error about then it's worth producing an error message that people can actually understand, particularly as it's recently added validation.
It was silently running into problems due to this all along but not reporting it. Its now reporting it which is better than silently things behaving strangely.
What are the actual consequences in a situation like this where we have something like ${DATETIME} in another variable referenced by the task?
I haven't looked at all of the code involved (and that's probably the root of my problem) but I don't quite understand how this is coming about. DATE and TIME are supposed to be determined at the start of the build and explicitly passed to the worker, so they don't change during the build. What am I missing?
Quoted 6 lines
Its very hard for bitbake to know why the hashes differ, it only knows the values afterwards and hence that they've changed, the information about how that hash was constructed is not present in any of bitbake's caches. That implies to have better messages we need to write out more data.
Quoted 8 lines
I did add a patch to make bitbake write out data to allow reconstruction of basehash (which is part of taskhash). Sadly the parsing performance was diabolical (10 times slower). I think that could perhaps be improved if the files don't require an atomic move during creation but I haven't had time to look further at it.
So whilst I do agree, what is the price people are willing to pay to have those better messages?
Clearly a 10x slowdown is unacceptable, hopefully we can find another way of dealing with this. I guess if we're able to do nothing else a brief explanation of what to look for (i.e. variables that might change with time) in the error message would be helpful, but I hope we can do better than that.
Cheers,
Paul
Signature · 2 lines
Paul Eggleton Intel Open Source Technology Centre
Quoted 51 lines · Mike Looijmans
Quoted 41 lines · Richard Purdie
Quoted 23 lines · Paul Eggleton
Quoted 13 lines · Ross
Quoted 10 lines · Ross
It means the hash calculated my the bitbake master was different to the
hash calculated when the worker started up. This usually means that
you're
using something like ${TIME} in the recipe but not marking it appropriatly
so the cache ignores it. Do you have a base-files bbappend that writes a
timestamp?Quoted 8 lines
The always wise Joshua reminds me that if your
DISTRO_VERSIONcontains
${DATETIME} then this happens. If you're doing this then you'll want to
set [vardepsexclude] onDISTRO_VERSIONto stop the DATETIME from getting
into the cache (or not put the current date/time into the distro version).Quoted 6 lines
I think we need to handle this situation better - if it's really worth
producing an error about then it's worth producing an error message that
people can actually understand, particularly as it's recently added validation.It was silently running into problems due to this all along but not reporting it. Its now reporting it which is better than silently things behaving strangely.
Its very hard for bitbake to know why the hashes differ, it only knows the values afterwards and hence that they've changed, the information about how that hash was constructed is not present in any of bitbake's caches. That implies to have better messages we need to write out more data.
Quoted 8 lines
I did add a patch to make bitbake write out data to allow reconstruction of basehash (which is part of taskhash). Sadly the parsing performance was diabolical (10 times slower). I think that could perhaps be improved if the files don't require an atomic move during creation but I haven't had time to look further at it.
So whilst I do agree, what is the price people are willing to pay to have those better messages?
Quoted 7 lines
I have a build machine that every so often runs into "basehash mismatch" problems (but never when you want it to, such as now), so getting some information on what might cause it would be "priceless". I'd happily let the machine run for 10 days straight if at the end I get a message that I can work with.
Which implies that any performance hit is acceptable, if there's a switch to enable and disable that extra diagnostic. Oh dear, just what we need, yet another switch...
I wonder if it could be something as simple as an NTP time update which happens periodically on my build machine? Since I've only ever seen this message _once_ after literally thousands of builds, it is a rare bird indeed!
Signature · 4 lines
------------------------------------------------------------ Gary Thomas | Consulting for the MLB Associates | Embedded world ------------------------------------------------------------
Quoted 31 lines · Paul Eggleton
Quoted 30 lines · On Tue, 07 Jun 2016 23:20:26 Richard Purdie wrote:
Quoted 28 lines · Paul Eggleton
Quoted 12 lines · Ross
Quoted 9 lines · Ross
It means the hash calculated my the bitbake master was different
to the hash calculated when the worker started up. This usually means
that you're using something like ${TIME} in the recipe but not marking
it appropriatly so the cache ignores it. Do you have a base -files
bbappend that writes a timestamp?Quoted 8 lines
The always wise Joshua reminds me that if your
DISTRO_VERSIONcontains ${DATETIME} then this happens. If you're doing this then
you'll want to set [vardepsexclude] onDISTRO_VERSIONto stop the
DATETIME from getting into the cache (or not put the current date/time
into the distro version).I think we need to handle this situation better - if it's really worth producing an error about then it's worth producing an error message
that people can actually understand, particularly as it's recently added
validation.
Quoted 8 lines
Quoted 4 lines
It was silently running into problems due to this all along but not reporting it. Its now reporting it which is better than silently things
behaving strangely.What are the actual consequences in a situation like this where we have something like ${DATETIME} in another variable referenced by the task?
A mismatch means the cached internal values bitbake is using on the cooker/server side don't match what the worker is actually doing. Its hard to say for sure exactly what impact that would have.
If for example its in a do_build task, it means that some invalid data may be written out to the do_build sigdata file and confuse later build analysis based on those files. You can usually tell such a file by the fact its named with one hash name but when you compute the value from the entries in the file, you get a different hash.
If its a more important task like do_install, it could mean something doesn't rerun when it should, or vice versa and the effect of a corrupt sidinfo/sigdata is more significant.
With the changes to error in these cases, we get to know about cases of mismatch and we went and fixed the majority of causes of these. I believe those mismatches were in part at fault for things like diffsigs not working as expected in some cases.
Quoted 5 lines
I haven't looked at all of the code involved (and that's probably the root of my problem) but I don't quite understand how this is coming about. DATE and TIME are supposed to be determined at the start of the build and explicitly passed to the worker, so they don't change during the build. What am I missing?
Bitbake computes the hashes at parse time and caches them. At run time, it then builds a new data store in the worker. These errors come about when the values in the cache from parsing don't match the runtime execution environment.
Quoted 20 lines
Quoted 16 lines
Its very hard for bitbake to know why the hashes differ, it only knows
the values afterwards and hence that they've changed, the information
about how that hash was constructed is not present in any of bitbake's
caches. That implies to have better messages we need to write out more
data.I did add a patch to make bitbake write out data to allow reconstruction of basehash (which is part of taskhash). Sadly the parsing performance was diabolical (10 times slower). I think that could perhaps be improved if the files don't require an atomic move during creation but I haven't had time to look further at it.
Quoted 3 lines
So whilst I do agree, what is the price people are willing to pay to
have those better messages?
Quoted 7 lines
Clearly a 10x slowdown is unacceptable, hopefully we can find another way of
dealing with this. I guess if we're able to do nothing else a brief explanation of what to look for (i.e. variables that might change with time)
in the error message would be helpful, but I hope we can do better than that.
So do I, its a time problem on my part to look into it. As mentioned above, I believe this problem was meaning diffsigs was unreliable and it was having other unknown impact on builds rebuilding when they shouldn't or vice versa. I believe those are important to fix. We need better sstate hash debugging/telemetry but exactly how to do this without performance impact remains to be seen. I'm continue to think on it (along with too many other issues) but am open to ideas.
Cheers,
Richard
I got that taskhash mismatch too and much more often than once per million. ;) I'm quite happy that you're discussing it here as I did not know what it was.
Re-running the task a second time in a row makes the warning disappearing.
Regards,
Quoted 65 lines · Gary Thomas
Quoted 51 lines · Mike Looijmans
Quoted 41 lines · Richard Purdie
Quoted 23 lines · Paul Eggleton
Quoted 13 lines · Ross
Quoted 10 lines · Ross
It means the hash calculated my the bitbake master was different to the
hash calculated when the worker started up. This usually means that
you're
using something like ${TIME} in the recipe but not marking it appropriatly
so the cache ignores it. Do you have a base-files bbappend that writes a
timestamp?Quoted 8 lines
The always wise Joshua reminds me that if your
DISTRO_VERSIONcontains
${DATETIME} then this happens. If you're doing this then you'll want to
set [vardepsexclude] onDISTRO_VERSIONto stop the DATETIME from getting
into the cache (or not put the current date/time into the distro version).Quoted 6 lines
I think we need to handle this situation better - if it's really worth
producing an error about then it's worth producing an error message that
people can actually understand, particularly as it's recently added validation.It was silently running into problems due to this all along but not reporting it. Its now reporting it which is better than silently things behaving strangely.
Its very hard for bitbake to know why the hashes differ, it only knows the values afterwards and hence that they've changed, the information about how that hash was constructed is not present in any of bitbake's caches. That implies to have better messages we need to write out more data.
Quoted 8 lines
I did add a patch to make bitbake write out data to allow reconstruction of basehash (which is part of taskhash). Sadly the parsing performance was diabolical (10 times slower). I think that could perhaps be improved if the files don't require an atomic move during creation but I haven't had time to look further at it.
So whilst I do agree, what is the price people are willing to pay to have those better messages?
Quoted 7 lines
I have a build machine that every so often runs into "basehash mismatch" problems (but never when you want it to, such as now), so getting some information on what might cause it would be "priceless". I'd happily let the machine run for 10 days straight if at the end I get a message that I can work with.
Which implies that any performance hit is acceptable, if there's a switch to enable and disable that extra diagnostic. Oh dear, just what we need, yet another switch...
I wonder if it could be something as simple as an NTP time update which happens periodically on my build machine? Since I've only ever seen this message _once_ after literally thousands of builds, it is a rare bird indeed!
Quoted 41 lines · Richard Purdie
Quoted 23 lines · Paul Eggleton
Quoted 13 lines · Ross
Quoted 10 lines · Ross
It means the hash calculated my the bitbake master was different to the
hash calculated when the worker started up. This usually means that
you're
using something like ${TIME} in the recipe but not marking it appropriatly
so the cache ignores it. Do you have a base-files bbappend that writes a
timestamp?Quoted 8 lines
The always wise Joshua reminds me that if your
DISTRO_VERSIONcontains
${DATETIME} then this happens. If you're doing this then you'll want to
set [vardepsexclude] onDISTRO_VERSIONto stop the DATETIME from getting
into the cache (or not put the current date/time into the distro version).Quoted 6 lines
I think we need to handle this situation better - if it's really worth
producing an error about then it's worth producing an error message that
people can actually understand, particularly as it's recently added validation.It was silently running into problems due to this all along but not reporting it. Its now reporting it which is better than silently things behaving strangely.
Its very hard for bitbake to know why the hashes differ, it only knows the values afterwards and hence that they've changed, the information about how that hash was constructed is not present in any of bitbake's caches. That implies to have better messages we need to write out more data.
Quoted 8 lines
I did add a patch to make bitbake write out data to allow reconstruction of basehash (which is part of taskhash). Sadly the parsing performance was diabolical (10 times slower). I think that could perhaps be improved if the files don't require an atomic move during creation but I haven't had time to look further at it.
So whilst I do agree, what is the price people are willing to pay to have those better messages?
I have a build machine that every so often runs into "basehash mismatch" problems (but never when you want it to, such as now), so getting some information on what might cause it would be "priceless". I'd happily let the machine run for 10 days straight if at the end I get a message that I can work with.
Which implies that any performance hit is acceptable, if there's a switch to enable and disable that extra diagnostic. Oh dear, just what we need, yet another switch...
M.
Kind regards,
Mike Looijmans
System Expert
TOPIC Embedded Products
Eindhovenseweg 32-C, NL-5683 KH Best
Postbus 440, NL-5680 AK Best
Telefoon: +31 (0) 499 33 69 79
E-mail: [email protected]
Website: www.topicproducts.com
Please consider the environment before printing this e-mail