sstate: remove corrupted artifacts from local mirror
Quoted 8 lines · Alexander Kanavin
Quoted 5 lines · Max
Corruptions are unavoidable part of our life, disks and network can inject failures due to unpredictable and unknown reasons like
https://www.sciencedirect.com/science/article/abs/pii/S0026271421003723.
Even multi-layer protection is not perfect as it depends on where the error is injected.I'm sorry but I am not convinced. You are the only one reporting the issue, so I don't think it's unavoidable.
The "only one reporting" does not mean the problem does not exist, it may be just not as big problem for others as for us. Similar problem exist for DRAM memory corruptions? Most of the people don't care about that but for some this is important problem, e.g. when you see 1 OS crash per 10 years it is not a big deal but if you own 10k servers you see 3 crashes per day. That is the scale factor that is important. Max talked about our scale already. Summarizing, the manual work is not a solution for us due to scale.
Quoted 7 lines
Quoted 3 lines
We proposed this patch as it is aligned with how we currently handle incorrect checksums:
https://github.com/yoctoproject/poky/commit/672c07de4a96eb67eaafba0873eced44ec9ae1a6.What the patch is doing is essentially -c cleansstate on a cache used by several consumers. This makes it a non-starter: you can't remove items from live sstate like that. Once an object is in the cache, it needs to be removed offline, or replaced atomically if it's corrupted.
I disagree, we can overwrite bad artifact. Yocto indirectly does that as it has to rebuild the package. This is "by design" behavior. And to be honest, there is no difference between (1) rebuilding the package every time and (2) overwriting sstate cache so any other build can reuse it. Is there any concern around uploading such freshly built artifact?
Quoted 4 lines
Before we decide how to handle corrupted items, it's perhaps better to consider first how they should be reported. Then users can decide what they want to do with that information without yocto forcing something on them. So please: how can I observe the issue?
This is random thing, we are not in control of e.g. DRAM bit flip error, they simply happen. To simulate the situation you can inject an error yourself by e.g., overwriting the random byte of the zstd file before it is uploaded.
Quoted 1 line
Alex
Regards,
Przemyslaw
Quoted 7 lines · Przemyslaw
The "only one reporting" does not mean the problem does not exist, it may be just not as big problem for others as for us. Similar problem exist for DRAM memory corruptions? Most of the people don't care about that but for some this is important problem, e.g. when you see 1 OS crash per 10 years it is not a big deal but if you own 10k servers you see 3 crashes per day. That is the scale factor that is important. Max talked about our scale already. Summarizing, the manual work is not a solution for us due to scale.
But can you say what your scale actually is? How many sstate objects are written into the shared cache per day? How often do you see corruptions?
Basically it helps if you introduce yourselves and your product first, as this is I think you first time interacting with the community?
Quoted 5 lines
I disagree, we can overwrite bad artifact. Yocto indirectly does that as it has to rebuild the package. This is "by design" behavior. And to be honest, there is no difference between (1) rebuilding the package every time and (2) overwriting sstate cache so any other build can reuse it. Is there any concern around uploading such freshly built artifact?
The concern is that your patch does not overwrite the artifact, rather it deletes it first, and recreates it later. This creates a time window where a cache object exists, and then it doesn't, and then it exists again. This will break builds running in parallel in all sorts of interesting ways, as sstate is not designed for objects disappearing after they've been checked and confirmed to exist by bitbake.
For example, when we do need to test cache deletions (for instance in oe-selftest), we make super-sure that this is done on a private small test cache that isn't shared with anything, as otherwise there have been notoriously strange failures in random places.
The other concern I expressed to Max: this auto-recovery sweeps the 'flaky hardware' problem under the rug, instead of being loud and clear about it. If someone had a perfectly working sstate (and many people do, including the yocto upstream), and then it started throwing random fails, they're not going to notice it. If someone had very rare corruptions and then the rate increased, they're not going to notice that either. Except when they start to wonder why builds seem to take longer and longer and longer.
Quoted 3 lines
This is random thing, we are not in control of e.g. DRAM bit flip error, they simply happen. To simulate the situation you can inject an error yourself by e.g., overwriting the random byte of the zstd file before it is uploaded.
Yes, I saw it. The key issue is this bit in sstate.bbclass before actually creating the sstate archive:
if sstate_pkg.exists():
touch(sstate_pkg)
return
If sstate item exists, then it will not be replaced, even if it's been determined to be corrupted earlier.
I don't know yet how to best handle this, but I would want to improve *reporting* of corrupt sstate before we can decide whether yocto can do something about it that doesn't make things worse than they are now.
Then you can take the report and run a script that deletes the offending items. This all can be automated, and doesn't have to be executed manually.
Alex