Skip to contentSkip to conversation

support/download: fix running on hosts with wget2 (branch yem/dl-curl)

PATCH Applied Yann E. MORIN 2 participants 5 messages Jun 2, 2024 – Jul 12, 2024

lore
Share & tools
lore
#0 lore raw

v2 cover support/download: fix running on hosts with wget2 (branch yem/dl-curl)

Hello All!

Recently, Fedora 40 was released, which uses wget2 instead of the "original" wget (aka wget1). wget2 is almost a drop-in replacement for wget1, except it no longer supports FTP and WARC. While WARC was unused in Buildroot, FTP is still used by a few packges, some of which only available via FTP.

Switch to using curl for FTP downloads.

We could have switched to curl for http/s downlads too, but it is possible that existing packages (esp. in br2-external trees) use wget1 options, like we used to in the now-dropped amd-catalyst package for example. So we decided to only convert FTP downloads over to curl, and keep the rest unchanged, to minise the impact.

Changes v1 -> v2:
- drop --passive-ftp from BR2_WGET

Regards,
Yann E. MORIN.

----------------------------------------------------------------
Yann E. MORIN (2):
support/download: introduce curl backend for FTP transfers utils/genrandconfig: do not check certificates with curl

 Config.in                     |  6 +++++-
 docs/manual/prerequisite.adoc |  1 +
 package/pkg-download.mk       |  1 +
 package/pkg-generic.mk        |  2 ++
 support/download/curl         | 45 +++++++++++++++++++++++++++++++++++++++++++
 support/download/dl-wrapper   |  1 +
 utils/genrandconfig           |  3 ++-
 7 files changed, 57 insertions(+), 2 deletions(-)
 create mode 100755 support/download/curl

--
.-----------------.--------------------.------------------.--------------------.

|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |

'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot

#1 lore raw

PATCH v2 1/2 support/download: introduce curl backend for FTP transfers

Recent versions of wget, starting with wget 2.0, aka wget2 thereafter, no longer support FTP (nor FTPS, aka FTP-over-SSL). wget2 is packaged in Fedora 40, recently released; F40 does not even have the old wget available in its repository anymore.

Introduce cURL as a download backend, that we use for FTP and FPTS protocols.

Note that the -q flag does not means being quiet; it means that a curlrc file should not be parsed. The long option is --disable, which meaning is not much more obivous than the short -q. It also has to be the first option on the command line.

Since we no longer use WGET to retrieve FTP-hosted files, we can drop the --passive-ftp option, which is what was causing wget2 to abort in error.

---
Notes: the pre-existing shellcheck errors in the dl-wrapper were not fixed; this is not urgent, so is left for another series.

---
Changes v1 ->v2;
- drop --passive-ftp for BR2_WGET

---
 Config.in                     |  6 ++++-
 docs/manual/prerequisite.adoc |  1 +
 package/pkg-download.mk       |  1 +
 package/pkg-generic.mk        |  2 ++
 support/download/curl         | 45 +++++++++++++++++++++++++++++++++++
 support/download/dl-wrapper   |  1 +
 utils/genrandconfig           |  2 +-
 7 files changed, 56 insertions(+), 2 deletions(-)
 create mode 100755 support/download/curl
Config.in +5 -1
index b5a94325c4..e0257ba3e8 100644
@@ -103,9 +103,13 @@ menu "Build options"
 
 menu "Commands"
 
+config BR2_CURL
+	string "Curl command"
+	default "curl -q --ftp-pasv --retry 3"
+
 config BR2_WGET
 	string "Wget command"
-	default "wget --passive-ftp -nd -t 3"
+	default "wget -nd -t 3"
 
 config BR2_SVN
 	string "Subversion (svn) command"
docs/manual/prerequisite.adoc +1 -0
index 262a5153f5..846a7482ac 100644
@@ -75,6 +75,7 @@ packages using any of these methods, you will need to install the
 corresponding tool on the host system:
 +
 ** +bazaar+
+** +curl+
 ** +cvs+
 ** +git+
 ** +mercurial+
package/pkg-download.mk +1 -0
index 4be45c9d12..455443c164 100644
@@ -8,6 +8,7 @@
 ################################################################################
 
 # Download method commands
+export CURL := $(call qstrip,$(BR2_CURL))
 export WGET := $(call qstrip,$(BR2_WGET))
 export SVN := $(call qstrip,$(BR2_SVN))
 export CVS := $(call qstrip,$(BR2_CVS))
package/pkg-generic.mk +2 -0
index a2749320c3..e1c16b7343 100644
@@ -1253,6 +1253,8 @@ else ifeq ($$($(2)_SITE_METHOD),hg)
 DL_TOOLS_DEPENDENCIES += hg
 else ifeq ($$($(2)_SITE_METHOD),cvs)
 DL_TOOLS_DEPENDENCIES += cvs
+else ifneq ($(filter ftp ftps,$$($(2)_SITE_METHOD)),)
+DL_TOOLS_DEPENDENCIES += curl
 endif # SITE_METHOD
 
 # cargo/go vendoring (may) need git
support/download/curl new file +45 -0

45 lines not on this page · raw message

support/download/dl-wrapper +1 -0
index 35428faeef..069b2c1c21 100755
@@ -91,6 +91,7 @@ main() {
         backend="${backend_urlencode%|*}"
         case "${backend}" in
             git|svn|cvs|bzr|file|scp|hg|sftp) ;;
+            ftp|ftps) backend="curl" ;;
             *) backend="wget" ;;
         esac
         uri=${uri#*+}
utils/genrandconfig +1 -1
index b838dda34d..b7d277105f 100755
@@ -688,7 +688,7 @@ async def gen_config(args):
         configlines += minimalf.readlines()
 
     # Allow hosts with old certificates to download over https
-    configlines.append("BR2_WGET=\"wget --passive-ftp -nd -t 3 --no-check-certificate\"\n")
+    configlines.append("BR2_WGET=\"wget -nd -t 3 --no-check-certificate\"\n")
 
     # Per-package folder
     if randint(0, 15) == 0:
Signature · 6 lines
2.45.1

_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
#2 lore raw

Re: [Buildroot] [PATCH 1/2 v2] support/download: introduce curl backend for FTP transfers

Hello Yann,

Quoted 6 lines · On Sun, 2 Jun 2024 20:23:07 +0200 "Yann E. MORIN" <[email protected]>…
Config.in +0 -0
index b5a94325c4..e0257ba3e8 100644
@@ -103,9 +103,13 @@ menu "Build options"
 

Config.in menu "Build options" · line 112

Quoted 10 lines
Config.in +5 -1
 menu "Commands"
 
+config BR2_CURL
+	string "Curl command"
+	default "curl -q --ftp-pasv --retry 3"
+
 config BR2_WGET
 	string "Wget command"
-	default "wget --passive-ftp -nd -t 3"
+	default "wget -nd -t 3"

This change was not really related to introducing curl support... and in fact Peter already applied 1a61c1d9b1919120883b689c3767925bd91d3028 doing such a change.

utils/genrandconfig async def gen_config(args): · line 694

Quoted 13 lines
utils/genrandconfig +1 -1
index b838dda34d..b7d277105f 100755
@@ -688,7 +688,7 @@ async def gen_config(args):
         configlines += minimalf.readlines()
 
     # Allow hosts with old certificates to download over https
-    configlines.append("BR2_WGET=\"wget --passive-ftp -nd -t 3 --no-check-certificate\"\n")
+    configlines.append("BR2_WGET=\"wget -nd -t 3 --no-check-certificate\"\n")
 
     # Per-package folder
     if randint(0, 15) == 0:

Also, this was not directly related to the introduction of curl support, so I split this out into a separate commit.

Applied, thanks!

Thomas

Signature · 7 lines
Thomas Petazzoni, co-owner and CEO, Bootlin
Embedded Linux and Kernel engineering and training
https://bootlin.com
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
#3 lore raw

PATCH v2 2/2 utils/genrandconfig: do not check certificates with curl

In reply to Yann E. MORIN

genrandconfig is used in autobuilders, and some autobuilders are running on old distributions that are lacking the most recent CAs, causing build failures because package sources can't be retrieved.

Do for the curl backend what we already did a while back for the wget backend, with commit 0866a280e40a (utils/genrandconfig: use --no-check-certificate in wget by default); in curl, the equivalent would be --insecure, and applies to the ftps transport.

The integrity of the downloads are validated against our bundled hashes so there is no risk of corruption of the downloaded files. The only issue would be that an MITM could inspect the transaction, the same way as for the wget --no-check-certificate in 0866a280e40a, but this is not considered a high-level issue (we're anyway talking FTPS here, that's a legacy protocol that has other issues).

---
Note: this is totally untested, because FTPS is not widespread and no known package was available via FTPS. This patch can probably be dropped.

---
 utils/genrandconfig | 1 +
 1 file changed, 1 insertion(+)
utils/genrandconfig +1 -0
index b7d277105f..a7b012ba9b 100755
@@ -689,6 +689,7 @@ async def gen_config(args):
 
     # Allow hosts with old certificates to download over https
     configlines.append("BR2_WGET=\"wget -nd -t 3 --no-check-certificate\"\n")
+    configlines.append("BR2_CURL=\"curl --ftp-pasv --retry 3 --insecure\"\n")
 
     # Per-package folder
     if randint(0, 15) == 0:
Signature · 6 lines
2.45.1

_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
#4 lore raw

Re: [Buildroot] [PATCH 2/2 v2] utils/genrandconfig: do not check certificates with curl

Quoted 16 lines · On Sun, 2 Jun 2024 20:23:08 +0200 "Yann E. MORIN" <[email protected]>…

genrandconfig is used in autobuilders, and some autobuilders are running on old distributions that are lacking the most recent CAs, causing build failures because package sources can't be retrieved.

Do for the curl backend what we already did a while back for the wget backend, with commit 0866a280e40a (utils/genrandconfig: use --no-check-certificate in wget by default); in curl, the equivalent would be --insecure, and applies to the ftps transport.

The integrity of the downloads are validated against our bundled hashes so there is no risk of corruption of the downloaded files. The only issue would be that an MITM could inspect the transaction, the same way as for the wget --no-check-certificate in 0866a280e40a, but this is not considered a high-level issue (we're anyway talking FTPS here, that's a legacy protocol that has other issues).

Quoted 8 lines

---
Note: this is totally untested, because FTPS is not widespread and no known package was available via FTPS. This patch can probably be dropped.

---
 utils/genrandconfig | 1 +
 1 file changed, 1 insertion(+)

Applied to master, thanks.

Thomas

Signature · 7 lines
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot