Skip to contentSkip to conversation

utils/genrandconfig: always use BR2_BACKUP_SITE in the autobuilders

PATCH In review Thomas Petazzoni via buildroot 1 participant 6 messages Aug 26 – Sep 3

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 utils/genrandconfig: always use BR2_BACKUP_SITE in the autobuilders

In commit da5c25c9f91b17a3c00ff0b35164881f2d1aa425, we decided to generate 10% of configurations without BR2_BACKUP_SITE to test upstream locations and see if they fail.

In recent times, we have seen a significant increase of temporary failures of upstream locations, which cause build failures for which there is not much we can do. They cause noise in the build results and consume autobuilders CPU time for builds that don't lead to any useful conclusion.

So in the end let's revert back to using BR2_BACKUP_SITE in all configurations to avoid those build failures. If we want to detect really stale upstream locations, as suggested by Arnout, we can instrument the autobuilder runs to inject a special HTTP header and then look at the logs of sources.buildroot.net to see which files are grabbed by the autobuilders and determine if upstream locations are really dead and packages need to be fixed accordingly.

---
Note: this is a proposal. If accepted, it should be merged into all branches currently tested by autobuild.buildroot.org.

---
 utils/genrandconfig | 5 -----
 1 file changed, 5 deletions(-)
utils/genrandconfig +0 -5
index 727c8193c4..fad3a20786 100755
@@ -556,11 +556,6 @@ async def gen_config(args):
         configlines.append("BR2_REPRODUCIBLE=y\n")
         configlines.append("BR2_TARGET_ROOTFS_TAR=y\n")
 
-    # From time to time, ignore sources.buildroot.net to really fetch
-    # from upstream
-    if randint(0, 1) == 0:
-        configlines.append("""BR2_BACKUP_SITE=""\n""")
-
     with open(configfile, "a") as configf:
         configf.writelines(configlines)
 
Signature · 6 lines
2.55.0

_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
#1 lore raw

Hello Thomas,

In reply of:

Quoted 7 lines

So in the end let's revert back to using BR2_BACKUP_SITE in all configurations to avoid those build failures. If we want to detect really stale upstream locations, as suggested by Arnout, we can instrument the autobuilder runs to inject a special HTTP header and then look at the logs of sources.buildroot.net to see which files are grabbed by the autobuilders and determine if upstream locations are really dead and packages need to be fixed accordingly.

How and how often are the 'source.buildroot.net' backups generated right now ? By reading this paragraph I'm thinking the upstream location check could be coupled to the generation of the backups instead.

Otherwise I think this change make sense.

PERALE Thomas
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot

#2 lore raw
Quoted 1 line · Thomas Perale

How and how often are the 'source.buildroot.net' backups generated right now ?

Pretty much constantly:

3 */2 * * * flock -n /home/brmirror/br-mirror.lock /home/brmirror/bin/update-br-mirror.sh >/dev/null 2>&1 || true

Quoted 2 lines

By reading this paragraph I'm thinking the upstream location check could be coupled to the generation of the backups instead.

I'm not sure to understand what you mean. The "upstream location check" is done by the build machines, which are completely distinct from the machine hosting sources.b.o and doing the mirroring.

Thomas

Signature · 7 lines
Thomas Petazzoni, co-owner and CEO, Bootlin
Embedded Linux and Kernel engineering and training
https://bootlin.com
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
#3 lore raw

In reply of:

Quoted 5 lines
Quoted 1 line

How and how often are the 'source.buildroot.net' backups generated right now ?

Pretty much constantly:

3 */2 * * * flock -n /home/brmirror/br-mirror.lock /home/brmirror/bin/update-br-mirror.sh >/dev/null 2>&1 || true

Is the update-br-mirror.sh script available somewhere ?

Quoted 6 lines
Quoted 2 lines

By reading this paragraph I'm thinking the upstream location check could be coupled to the generation of the backups instead.

I'm not sure to understand what you mean. The "upstream location check" is done by the build machines, which are completely distinct from the machine hosting sources.b.o and doing the mirroring.

Yes that's what I meant. Since doing this verification by the autobuilders is an issue you could move the verification and publishing of the stale entries from the machine hosting the sources.

Since it already has the responsability to download all the entries, it seems like a perfect candidate to detect stale locations.
Unless 'update-br-mirror' script only trigger a the download if a new location is detected ?

What I'm afraid of with the HTTP header solution proposed is that it's gonna be burried in logs that would be quickly forgotten about. The current solution while generating a lot of false positive manage to catch some real stale locations publicly.

PERALE Thomas
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot

#4 lore raw
Quoted 8 lines · Thomas Perale

Yes that's what I meant. Since doing this verification by the autobuilders is an issue you could move the verification and publishing of the stale entries from the machine hosting the sources.

Since it already has the responsability to download all the entries, it seems like a perfect candidate to detect stale locations.
Unless 'update-br-mirror' script only trigger a the download if a new location is detected ?

It does "make source" so it only triggers a download if the file is not already in the cache. And typically dead upstream servers are used to host dead projects, which don't see releases very often, and therefore we likely already have the tarball in our cache.

Quoted 4 lines

What I'm afraid of with the HTTP header solution proposed is that it's gonna be burried in logs that would be quickly forgotten about. The current solution while generating a lot of false positive manage to catch some real stale locations publicly.

True, but it's also generating a lot of noise.

To be honest, I am not 100% sure my proposal is the best solution, but if we're aiming to reduce the number of build failures, dropping false positives is important.

Thomas

Signature · 7 lines
Thomas Petazzoni, co-owner and CEO, Bootlin
Embedded Linux and Kernel engineering and training
https://bootlin.com
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
#5 lore raw

Am 27.08.26 um 22:17 schrieb Thomas Petazzoni via buildroot:

Quoted 11 lines · Thomas Perale
Quoted 4 lines

What I'm afraid of with the HTTP header solution proposed is that it's gonna be burried in logs that would be quickly forgotten about. The current solution while generating a lot of false positive manage to catch some real stale locations publicly.

True, but it's also generating a lot of noise.

To be honest, I am not 100% sure my proposal is the best solution, but if we're aiming to reduce the number of build failures, dropping false positives is important.

I haven't looked into how the autobuilder reports are generated, would it be practical to separate download errors from build errors? That way download errors would still be visible, without messing up build statistics.

Best regards,
Fiona

_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot