utils/genrandconfig: always use BR2_BACKUP_SITE in the autobuilders
PATCH 1/1 utils/genrandconfig: always use BR2_BACKUP_SITE in the autobuilders
In commit da5c25c9f91b17a3c00ff0b35164881f2d1aa425, we decided to generate 10% of configurations without BR2_BACKUP_SITE to test upstream locations and see if they fail.
In recent times, we have seen a significant increase of temporary failures of upstream locations, which cause build failures for which there is not much we can do. They cause noise in the build results and consume autobuilders CPU time for builds that don't lead to any useful conclusion.
So in the end let's revert back to using BR2_BACKUP_SITE in all configurations to avoid those build failures. If we want to detect really stale upstream locations, as suggested by Arnout, we can instrument the autobuilder runs to inject a special HTTP header and then look at the logs of sources.buildroot.net to see which files are grabbed by the autobuilders and determine if upstream locations are really dead and packages need to be fixed accordingly.
---
Note: this is a proposal. If accepted, it should be merged into all branches currently tested by autobuild.buildroot.org.
--- utils/genrandconfig | 5 ----- 1 file changed, 5 deletions(-)
utils/genrandconfig +0 -5
@@ -556,11 +556,6 @@ async def gen_config(args): configlines.append("BR2_REPRODUCIBLE=y\n") configlines.append("BR2_TARGET_ROOTFS_TAR=y\n") - # From time to time, ignore sources.buildroot.net to really fetch - # from upstream - if randint(0, 1) == 0: - configlines.append("""BR2_BACKUP_SITE=""\n""") - with open(configfile, "a") as configf: configf.writelines(configlines)
Signature · 6 lines
2.55.0 _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot
Hello Thomas,
In reply of:
Quoted 7 lines
So in the end let's revert back to using
BR2_BACKUP_SITEin all configurations to avoid those build failures. If we want to detect really stale upstream locations, as suggested by Arnout, we can instrument the autobuilder runs to inject a special HTTP header and then look at the logs of sources.buildroot.net to see which files are grabbed by the autobuilders and determine if upstream locations are really dead and packages need to be fixed accordingly.
How and how often are the 'source.buildroot.net' backups generated right now ? By reading this paragraph I'm thinking the upstream location check could be coupled to the generation of the backups instead.
Otherwise I think this change make sense.
PERALE Thomas
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
Quoted 1 line · Thomas Perale
How and how often are the 'source.buildroot.net' backups generated right now ?
Pretty much constantly:
3 */2 * * * flock -n /home/brmirror/br-mirror.lock /home/brmirror/bin/update-br-mirror.sh >/dev/null 2>&1 || true
Quoted 2 lines
By reading this paragraph I'm thinking the upstream location check could be coupled to the generation of the backups instead.
I'm not sure to understand what you mean. The "upstream location check" is done by the build machines, which are completely distinct from the machine hosting sources.b.o and doing the mirroring.
Thomas
Signature · 7 lines
Thomas Petazzoni, co-owner and CEO, Bootlin Embedded Linux and Kernel engineering and training https://bootlin.com _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot
In reply of:
Quoted 5 lines
Quoted 1 line
How and how often are the 'source.buildroot.net' backups generated right now ?
Pretty much constantly:
3 */2 * * * flock -n
/home/brmirror/br-mirror.lock/home/brmirror/bin/update-br-mirror.sh>/dev/null 2>&1 || true
Is the update-br-mirror.sh script available somewhere ?
Quoted 6 lines
Quoted 2 lines
By reading this paragraph I'm thinking the upstream location check could be coupled to the generation of the backups instead.
I'm not sure to understand what you mean. The "upstream location check" is done by the build machines, which are completely distinct from the machine hosting sources.b.o and doing the mirroring.
Yes that's what I meant. Since doing this verification by the autobuilders is an issue you could move the verification and publishing of the stale entries from the machine hosting the sources.
Since it already has the responsability to download all the entries, it seems like a perfect candidate to detect stale locations.
Unless 'update-br-mirror' script only trigger a the download if a new location is detected ?
What I'm afraid of with the HTTP header solution proposed is that it's gonna be burried in logs that would be quickly forgotten about. The current solution while generating a lot of false positive manage to catch some real stale locations publicly.
PERALE Thomas
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
Quoted 8 lines · Thomas Perale
Yes that's what I meant. Since doing this verification by the autobuilders is an issue you could move the verification and publishing of the stale entries from the machine hosting the sources.
Since it already has the responsability to download all the entries, it seems like a perfect candidate to detect stale locations.
Unless 'update-br-mirror' script only trigger a the download if a new location is detected ?
It does "make source" so it only triggers a download if the file is not already in the cache. And typically dead upstream servers are used to host dead projects, which don't see releases very often, and therefore we likely already have the tarball in our cache.
Quoted 4 lines
What I'm afraid of with the HTTP header solution proposed is that it's gonna be burried in logs that would be quickly forgotten about. The current solution while generating a lot of false positive manage to catch some real stale locations publicly.
True, but it's also generating a lot of noise.
To be honest, I am not 100% sure my proposal is the best solution, but if we're aiming to reduce the number of build failures, dropping false positives is important.
Thomas
Signature · 7 lines
Thomas Petazzoni, co-owner and CEO, Bootlin Embedded Linux and Kernel engineering and training https://bootlin.com _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot
Am 27.08.26 um 22:17 schrieb Thomas Petazzoni via buildroot:
Quoted 11 lines · Thomas Perale
Quoted 4 lines
What I'm afraid of with the HTTP header solution proposed is that it's gonna be burried in logs that would be quickly forgotten about. The current solution while generating a lot of false positive manage to catch some real stale locations publicly.
True, but it's also generating a lot of noise.
To be honest, I am not 100% sure my proposal is the best solution, but if we're aiming to reduce the number of build failures, dropping false positives is important.
I haven't looked into how the autobuilder reports are generated, would it be practical to separate download errors from build errors? That way download errors would still be visible, without messing up build statistics.
Best regards,
Fiona
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot