New: GIthub schema is broken
https://bugs.busybox.net/show_bug.cgi?id=14396
Bug ID: 14396
Summary: GIthub schema is broken
Product: buildroot
Version: 2021.08.2
Hardware: All
OS: Linux
Status: NEW
Severity: normal
Priority: P5
Component: Other
Assignee: [email protected]
Reporter: [email protected]
CC: [email protected]Target Milestone: ---
Github seems to have adopted a new schema for releases: instead of:
https://github.com/<user>/<project>/archive/<version>/<project>-<version>.tar.gz
it's now:
https://github.com/<user>/<project>/archive/refs/tags/<version>.tar.gz
which is then redirected to a location containing <project>-<version>.tar.gz. This effectively breaks at least jsmn and gtest, but probably all github packages.
Steps to reproduce:
1. Download a fresh Buildroot, e.g. the master branch.
2. make menuconfig and select gtest
3. In menuconfig: disable BR2_BACKUP_SITE
3. make source
This results in GTest download failing with error 404:
wget --passive-ftp -nd -t 3 -O '/home/user/projects/buildroot/output/build/.gtest-1.11.0.tar.gz.FcavV0/output' 'https://github.com/google/googletest/archive/release-1.11.0/gtest-1.11.0.tar.gz' --2021-11-26 18:12:27--
https://github.com/google/googletest/archive/release-1.11.0/gtest-1.11.0.tar.gz
Location:
https://codeload.github.com/google/googletest/tar.gz/release-1.11.0/gtest-1.11.0
[following]
--2021-11-26 18:12:27--
https://codeload.github.com/google/googletest/tar.gz/release-1.11.0/gtest-1.11.0
Connecting to proxy.elvees.com (proxy.elvees.com)|192.168.1.15|:3128... connected.
Proxy request sent, awaiting response... 404 Not Found
2021-11-26 18:12:28 ERROR 404: Not Found.
Signature · 6 lines
You are receiving this mail because: You are on the CC list for the bug. _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot
[Buildroot] [Bug 14396] GIthub schema is broken
https://bugs.busybox.net/show_bug.cgi?id=14396
--- Comment #1 from Jens Maus <[email protected]> --- I am seeing the same behaviour here, thus can reproduce it.
Signature · 6 lines
You are receiving this mail because: You are on the CC list for the bug. _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot
[Buildroot] [Bug 14396] GIthub schema is broken
In reply to [email protected]https://bugs.busybox.net/show_bug.cgi?id=14396
Yann E. MORIN <[email protected]> changed:
What |Removed |Added
----------------------------------------------------------------------------
CC| |[email protected] Assignee|[email protected] |[email protected] |.org | Status|NEW |ASSIGNED
Signature · 6 lines
You are receiving this mail because: You are on the CC list for the bug. _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot
[Buildroot] [Bug 14396] GIthub schema is broken
In reply to [email protected]https://bugs.busybox.net/show_bug.cgi?id=14396
--- Comment #2 from Olga <[email protected]> ---
To save maintainers some trouble, I've been brainstorming how to fix this issue.
The main problem that I see in this is that the value that should be assigned to <PACKAGE>_SOURCE and the actual name of the .tar.gz differ. The bluntest solution is to define <PACKAGE>_SOURCE for each package as something like release-$(PACKAGE_VERSION).tar.gz, but that means that all mirrors of those packages will fail and have to be rebuilt.
Instead, maybe $(call github,author,repo,version) should define a fake URI protocol called github, which allows us to bodge the download process via a new script located in support/download/github, saving the original <PACKAGE>_SOURCE definitions while allowing download from unusual URLs that github provides.
Signature · 6 lines
You are receiving this mail because: You are on the CC list for the bug. _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot
[Buildroot] [Bug 14396] GIthub schema is broken
In reply to [email protected]https://bugs.busybox.net/show_bug.cgi?id=14396
--- Comment #3 from Yann E. MORIN <[email protected]> --- Olga, All,
Thanks for the report.
Indeed, this is very broken: for tags, github.com returns 404, while for hashes, it returns 500. It sometimes also returns 502 or 504 in either case.
Quoted 2 lines
it's now:
https://github.com/<user>/<project>/archive/refs/tags/<version>.tar.gz
This unfortunately does not work either: it returns 500:
wget wget
https://github.com/google/googletest/refs/tags/release-1.11.0.tar.gz
[...]
HTTP request sent, awaiting response... 500 Internal Server Error
Another try a few minutes later yielded:
HTTP request sent, awaiting response... 502 Bad Gateway
Besides, the URLs makes it look that it would anyway only work for tag, not hashes.
Of course, this is not going to be easy to fix, and there does not seem like there is an obvious fix anyway...
My opinion is that we should stop using the forges' archive mechanisms, which have proved to be fluky in time, and use git for what it is and use git-clone to retrieve git-hosted code (or svn et al.). This will however raise two issues.
First, we have a lot (like a *lot*) of packages using the github macro; fixing them is quite easy with a bit of sed, and manual review/tweaks in corner cases. Second, this will trigger much bigger downloads, which can be a bit overwhelming
for large repositories. That latter issue is smoothened by the use of the local git cache, so this is only really a problem on the first download.
But right now, github.com seems to be having issues; it is very slow to return any result, and even the website is very un-responsive... I'll try to investigate
further tomorrow (GMT+1)...
Regards,
Yann E. MORIN.
Signature · 6 lines
You are receiving this mail because: You are on the CC list for the bug. _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot
[Buildroot] [Bug 14396] GIthub schema is broken
In reply to [email protected]https://bugs.busybox.net/show_bug.cgi?id=14396
--- Comment #4 from Olga <[email protected]> ---
(In reply to Yann E. MORIN from comment #3)
Quoted 9 lines
This unfortunately does not work either: it returns 500:
wget wget https://github.com/google/googletest/refs/tags/release-1.11.0.tar.gz
[...]
HTTP request sent, awaiting response... 500 Internal Server ErrorAnother try a few minutes later yielded:
HTTP request sent, awaiting response... 502 Bad Gateway
That was because you missed the "/archive/" part of the URL between <project> and /refs/tags/. With that part, it currently works.
Signature · 6 lines
You are receiving this mail because: You are on the CC list for the bug. _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot
[Buildroot] [Bug 14396] GIthub schema is broken
In reply to [email protected]https://bugs.busybox.net/show_bug.cgi?id=14396
--- Comment #5 from Yann E. MORIN <[email protected]> --- (In reply to Olga from comment #4)
Quoted 1 line
That was because you missed the "/archive/" part of the URL between <project> and /refs/tags/.
Indeed. And although this does work for tags, it does not for hashes.
It seems the old schema works for hashes, though. But given a FOO_VERSION, there is no systematic solution to automatically tell hashes apart from tags, so we don't have a solution to expand the github macro one way or the other.
Regards,
Yann E. MORIN.
Signature · 6 lines
You are receiving this mail because: You are on the CC list for the bug. _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot
[Buildroot] [Bug 14396] GIthub schema is broken
In reply to [email protected]https://bugs.busybox.net/show_bug.cgi?id=14396
--- Comment #6 from Jens Maus <[email protected]> --- (In reply to Yann E. MORIN from comment #3)
Quoted 3 lines
Second, this will trigger much bigger downloads, which can be a bit overwhelming for large repositories. That latter issue is smoothened by the use of the local git cache, so this is only really a problem on the first download.
Please keep in mind that there are a lot of CI driven build environments out there (like in my case) where no git cache environment is available, thus packages are always freshly downloaded. Therefore the direct archive download - rather than doing a full fledged git clone on each run - resulted in significantly smaller (i.e faster) downloads and thus faster CI build-times.
Signature · 6 lines
You are receiving this mail because: You are on the CC list for the bug. _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot
[Buildroot] [Bug 14396] GIthub schema is broken
In reply to [email protected]https://bugs.busybox.net/show_bug.cgi?id=14396
--- Comment #7 from Yann E. MORIN <[email protected]> ---
FTR: this is tracked in Github with:
https://github.com/github/feedback/discussions/8149
Signature · 6 lines
You are receiving this mail because: You are on the CC list for the bug. _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot
[Buildroot] [Bug 14396] GIthub schema is broken
In reply to [email protected]https://bugs.busybox.net/show_bug.cgi?id=14396
--- Comment #8 from Yann E. MORIN <[email protected]> --- (In reply to Jens Maus from comment #6)
Quoted 3 lines
there are a lot of CI driven build environments out there (like in my case) where no git cache environment is available, thus packages are always freshly downloaded
Yes, sure, this was the reason for the github helper macro in the first place.
But if Github changes their download scheme (that's not the first time), there is no way we can have our releases that are not broken at some point. For example, 2021.02.7 (latest LTS release) is now broken, 2021.08.2 (latest maintenance release) is now broken.
And if we have no way to detect whether we need the hash scheme, or the tag scheme, what is the alternate option?
(In reply to Olga from comment #2)
Quoted 2 lines
The bluntest solution is to define <PACKAGE>_SOURCE for each package as something like release-$(PACKAGE_VERSION).tar.gz,
For some packages, the tag is exactly the version string, for others a leading 'v' is added, others have a custom, non-standard prefix (like gtest, which has 'release-').
But such a mass fixup is not very nice... It needs carefull sed replacements and a lot of review time to catch corner cases... We should try and find another solution...
Quoted 2 lines
but that means that all mirrors of those packages will fail and have to be rebuilt.
sources.buildroot.org is automatically rebuilt, so that is not an issue for us. Others whi maintain their own mirror, will have to, well, just maintain it by opushing the new archives as needed. ;-)
Regards,
Yann E. MORIN.
Signature · 6 lines
You are receiving this mail because: You are on the CC list for the bug. _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot
[Buildroot] [Bug 14396] GIthub schema is broken
In reply to [email protected]https://bugs.busybox.net/show_bug.cgi?id=14396
Yann E. MORIN <[email protected]> changed:
What |Removed |Added
----------------------------------------------------------------------------
Resolution|--- |WONTFIX
Status|ASSIGNED |RESOLVED--- Comment #9 from Yann E. MORIN <[email protected]> --- Olga, All,
Github has reverted the change now:
https://github.com/github/feedback/discussions/8149#discussioncomment-1712006
As they said they will avoid such a regression in the future, we can (hopefully!) consider the issue fixed, I believe.
Thanks for the report! :-)
Regards,
Yann E. MORIN.
Signature · 6 lines
You are receiving this mail because: You are on the CC list for the bug. _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot