support/download: fix running on hosts with wget2 (branch yem/dl-curl)
v2 cover support/download: fix running on hosts with wget2 (branch yem/dl-curl)
Hello All!
Recently, Fedora 40 was released, which uses wget2 instead of the "original" wget (aka wget1). wget2 is almost a drop-in replacement for wget1, except it no longer supports FTP and WARC. While WARC was unused in Buildroot, FTP is still used by a few packges, some of which only available via FTP.
Switch to using curl for FTP downloads.
We could have switched to curl for http/s downlads too, but it is possible that existing packages (esp. in br2-external trees) use wget1 options, like we used to in the now-dropped amd-catalyst package for example. So we decided to only convert FTP downloads over to curl, and keep the rest unchanged, to minise the impact.
Changes v1 -> v2:
- drop --passive-ftp from BR2_WGET
Regards,
Yann E. MORIN.
----------------------------------------------------------------
Yann E. MORIN (2):
support/download: introduce curl backend for FTP transfers utils/genrandconfig: do not check certificates with curl
Config.in | 6 +++++- docs/manual/prerequisite.adoc | 1 + package/pkg-download.mk | 1 + package/pkg-generic.mk | 2 ++ support/download/curl | 45 +++++++++++++++++++++++++++++++++++++++++++ support/download/dl-wrapper | 1 + utils/genrandconfig | 3 ++- 7 files changed, 57 insertions(+), 2 deletions(-) create mode 100755 support/download/curl
--
.-----------------.--------------------.------------------.--------------------.
| Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
PATCH v2 1/2 support/download: introduce curl backend for FTP transfers
Recent versions of wget, starting with wget 2.0, aka wget2 thereafter, no longer support FTP (nor FTPS, aka FTP-over-SSL). wget2 is packaged in Fedora 40, recently released; F40 does not even have the old wget available in its repository anymore.
Introduce cURL as a download backend, that we use for FTP and FPTS protocols.
Note that the -q flag does not means being quiet; it means that a curlrc file should not be parsed. The long option is --disable, which meaning is not much more obivous than the short -q. It also has to be the first option on the command line.
Since we no longer use WGET to retrieve FTP-hosted files, we can drop the --passive-ftp option, which is what was causing wget2 to abort in error.
---
Notes: the pre-existing shellcheck errors in the dl-wrapper were not fixed; this is not urgent, so is left for another series.
---
Changes v1 ->v2;
- drop --passive-ftp for BR2_WGET
--- Config.in | 6 ++++- docs/manual/prerequisite.adoc | 1 + package/pkg-download.mk | 1 + package/pkg-generic.mk | 2 ++ support/download/curl | 45 +++++++++++++++++++++++++++++++++++ support/download/dl-wrapper | 1 + utils/genrandconfig | 2 +- 7 files changed, 56 insertions(+), 2 deletions(-) create mode 100755 support/download/curl
Config.in +5 -1
@@ -103,9 +103,13 @@ menu "Build options" menu "Commands" +config BR2_CURL + string "Curl command" + default "curl -q --ftp-pasv --retry 3" + config BR2_WGET string "Wget command" - default "wget --passive-ftp -nd -t 3" + default "wget -nd -t 3" config BR2_SVN string "Subversion (svn) command"
docs/manual/prerequisite.adoc +1 -0
@@ -75,6 +75,7 @@ packages using any of these methods, you will need to install the corresponding tool on the host system: + ** +bazaar+ +** +curl+ ** +cvs+ ** +git+ ** +mercurial+
package/pkg-download.mk +1 -0
@@ -8,6 +8,7 @@ ################################################################################ # Download method commands +export CURL := $(call qstrip,$(BR2_CURL)) export WGET := $(call qstrip,$(BR2_WGET)) export SVN := $(call qstrip,$(BR2_SVN)) export CVS := $(call qstrip,$(BR2_CVS))
package/pkg-generic.mk +2 -0
@@ -1253,6 +1253,8 @@ else ifeq ($$($(2)_SITE_METHOD),hg) DL_TOOLS_DEPENDENCIES += hg else ifeq ($$($(2)_SITE_METHOD),cvs) DL_TOOLS_DEPENDENCIES += cvs +else ifneq ($(filter ftp ftps,$$($(2)_SITE_METHOD)),) +DL_TOOLS_DEPENDENCIES += curl endif # SITE_METHOD # cargo/go vendoring (may) need git
support/download/curl new file +45 -0
45 lines not on this page · raw message
support/download/dl-wrapper +1 -0
@@ -91,6 +91,7 @@ main() { backend="${backend_urlencode%|*}" case "${backend}" in git|svn|cvs|bzr|file|scp|hg|sftp) ;; + ftp|ftps) backend="curl" ;; *) backend="wget" ;; esac uri=${uri#*+}
utils/genrandconfig +1 -1
@@ -688,7 +688,7 @@ async def gen_config(args): configlines += minimalf.readlines() # Allow hosts with old certificates to download over https - configlines.append("BR2_WGET=\"wget --passive-ftp -nd -t 3 --no-check-certificate\"\n") + configlines.append("BR2_WGET=\"wget -nd -t 3 --no-check-certificate\"\n") # Per-package folder if randint(0, 15) == 0:
Signature · 6 lines
2.45.1 _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot
Thomas Petazzoni via buildroot
AppliedRe: [Buildroot] [PATCH 1/2 v2] support/download: introduce curl backend for FTP transfers
Hello Yann,
Quoted 6 lines · On Sun, 2 Jun 2024 20:23:07 +0200 "Yann E. MORIN" <[email protected]>…
Config.in +0
-0@@ -103,9 +103,13 @@ menu "Build options"
Config.in menu "Build options" · line 112
Quoted 10 lines
Config.in +5
-1menu "Commands" +config BR2_CURL + string "Curl command" + default "curl -q --ftp-pasv --retry 3" + config BR2_WGET string "Wget command" - default "wget --passive-ftp -nd -t 3" + default "wget -nd -t 3"
This change was not really related to introducing curl support... and in fact Peter already applied 1a61c1d9b1919120883b689c3767925bd91d3028 doing such a change.
utils/genrandconfig async def gen_config(args): · line 694
Quoted 13 lines
utils/genrandconfig +1
-1@@ -688,7 +688,7 @@ async def gen_config(args): configlines += minimalf.readlines() # Allow hosts with old certificates to download over https - configlines.append("BR2_WGET=\"wget --passive-ftp -nd -t 3 --no-check-certificate\"\n") + configlines.append("BR2_WGET=\"wget -nd -t 3 --no-check-certificate\"\n") # Per-package folder if randint(0, 15) == 0:
Also, this was not directly related to the introduction of curl support, so I split this out into a separate commit.
Applied, thanks!
Thomas
Signature · 7 lines
Thomas Petazzoni, co-owner and CEO, Bootlin Embedded Linux and Kernel engineering and training https://bootlin.com _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot
PATCH v2 2/2 utils/genrandconfig: do not check certificates with curl
In reply to Yann E. MORINgenrandconfig is used in autobuilders, and some autobuilders are running on old distributions that are lacking the most recent CAs, causing build failures because package sources can't be retrieved.
Do for the curl backend what we already did a while back for the wget backend, with commit 0866a280e40a (utils/genrandconfig: use --no-check-certificate in wget by default); in curl, the equivalent would be --insecure, and applies to the ftps transport.
The integrity of the downloads are validated against our bundled hashes so there is no risk of corruption of the downloaded files. The only issue would be that an MITM could inspect the transaction, the same way as for the wget --no-check-certificate in 0866a280e40a, but this is not considered a high-level issue (we're anyway talking FTPS here, that's a legacy protocol that has other issues).
---
Note: this is totally untested, because FTPS is not widespread and no known package was available via FTPS. This patch can probably be dropped.
--- utils/genrandconfig | 1 + 1 file changed, 1 insertion(+)
utils/genrandconfig +1 -0
@@ -689,6 +689,7 @@ async def gen_config(args): # Allow hosts with old certificates to download over https configlines.append("BR2_WGET=\"wget -nd -t 3 --no-check-certificate\"\n") + configlines.append("BR2_CURL=\"curl --ftp-pasv --retry 3 --insecure\"\n") # Per-package folder if randint(0, 15) == 0:
Signature · 6 lines
2.45.1 _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot
Thomas Petazzoni via buildroot
AppliedRe: [Buildroot] [PATCH 2/2 v2] utils/genrandconfig: do not check certificates with curl
Quoted 16 lines · On Sun, 2 Jun 2024 20:23:08 +0200 "Yann E. MORIN" <[email protected]>…
genrandconfig is used in autobuilders, and some autobuilders are running on old distributions that are lacking the most recent CAs, causing build failures because package sources can't be retrieved.
Do for the curl backend what we already did a while back for the wget backend, with commit
0866a280e40a(utils/genrandconfig: use --no-check-certificate in wget by default); in curl, the equivalent would be --insecure, and applies to the ftps transport.The integrity of the downloads are validated against our bundled hashes so there is no risk of corruption of the downloaded files. The only issue would be that an MITM could inspect the transaction, the same way as for the wget --no-check-certificate in
0866a280e40a, but this is not considered a high-level issue (we're anyway talking FTPS here, that's a legacy protocol that has other issues).
Quoted 8 lines
Signed-off-by Yann E. MORIN <[email protected]>---
Note: this is totally untested, because FTPS is not widespread and no known package was available via FTPS. This patch can probably be dropped.--- utils/genrandconfig | 1 + 1 file changed, 1 insertion(+)
Applied to master, thanks.
Thomas
Signature · 7 lines
Thomas Petazzoni, CTO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot