Skip to contentSkip to conversation

download url broken if url include '+'

PATCH New jemy.zhang at gmail.com 2 participants 4 messages Nov 5, 2019 – Nov 8, 2019

lore
Share & tools
lore
#0 lore raw

cover download url broken if url include '+'

From: "Jemy Zhang" <[email protected]>

While downloading pacakges, the SITE_METHOD will be prefixed to the download url of the main package, but the extra download url would not be. If the SITE URL accidentally includes '+', url link before '+' would be parsed as SITE METHOD by dl-wrapper, and the url broken.

Take android-tools package for example:

```
ANDROID_TOOLS_SITE = https://launchpad.net/ubuntu/+archive/primary/+files
ANDROID_TOOLS_EXTRA_DOWNLOADS = android-tools_$(ANDROID_TOOLS_VERSION)-3ubuntu41.debian.tar.gz
```

while build the android-tools package, ALL_DOWNLOADS will be expended as:

```
for p in https+https://launchpad.net/ubuntu/+archive/primary/+files/android-tools_4.2.2+git20130218.orig.tar.xz   https://launchpad.ne
t/ubuntu/+archive/primary/+files/android-tools_4.2.2+git20130218-3ubuntu41.debian.tar.gz; do \
        if test ! -e /mnt/fileroot/jun.zhang/buildroot-git/buildroot/dl/android-tools/`basename $p` ; then \
                echo ">>> android-tools 4.2.2+git20130218 Downloading" ; \
                break ; \
        fi ; \
done
```

Notice that there's no 'https+' prefixed to the url, SITE_METHOD missing.

and while downloading package, url corrupted:

Quoted 1 line

--2019-11-05 14:10:29-- http://archive/primary/+files/android-tools_4.2.2+git20130218-3ubuntu41.debian.tar.gz

Jemy Zhang (1):
fix corrupted download url of extra packages

 package/pkg-generic.mk | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)
Signature · 1 line
2.22.0
#1 lore raw

PATCH 1/1 fix corrupted download url of extra packages

From: "Jemy Zhang" <[email protected]>

If the download url of extra packages contains '+', the contents before '+' will be parsed as SITE_METHOD, the original download url corrupted.

check with android-tools package.

---
 package/pkg-generic.mk | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)
package/pkg-generic.mk +3 -2
index 893faba22a..bb0ca939a3 100644
@@ -523,8 +523,9 @@ endif
 $(2)_ALL_DOWNLOADS = \
 	$$(if $$($(2)_SOURCE),$$($(2)_SITE_METHOD)+$$($(2)_SITE)/$$($(2)_SOURCE)) \
 	$$(foreach p,$$($(2)_PATCH) $$($(2)_EXTRA_DOWNLOADS),\
-		$$(if $$(findstring ://,$$(p)),$$(p),\
-			$$($(2)_SITE)/$$(p)))
+		$$(call getschemeplusuri,$$(call qstrip,\
+		  $$(if $$(findstring ://,$$(p)),$$(p),\
+			  $$($(2)_SITE)/$$(p))),urlencode))
 
 ifndef $(2)_SITE
  ifdef $(3)_SITE
Signature · 1 line
2.22.0
#2 lore raw

PATCH 1/1 fix corrupted download url of extra packages

Jemy, All,

On 2019-11-05 17:04 +0800, jemy.zhang at gmail.com spake thusly:

Quoted 6 lines

From: "Jemy Zhang" <[email protected]>

If the download url of extra packages contains '+', the contents before '+' will be parsed as SITE_METHOD, the original download url corrupted.

check with android-tools package.

Indeed this is broken. Thank you very much for noticing the issue.

Quoted 10 lines
---
 package/pkg-generic.mk | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)
package/pkg-generic.mk +0 -0
index 893faba22a..bb0ca939a3 100644

package/pkg-generic.mk endif · line 528

Quoted 8 lines
package/pkg-generic.mk +3 -2
@@ -523,8 +523,9 @@ endif
 $(2)_ALL_DOWNLOADS = \
 	$$(if $$($(2)_SOURCE),$$($(2)_SITE_METHOD)+$$($(2)_SITE)/$$($(2)_SOURCE)) \
 	$$(foreach p,$$($(2)_PATCH) $$($(2)_EXTRA_DOWNLOADS),\
-		$$(if $$(findstring ://,$$(p)),$$(p),\
-			$$($(2)_SITE)/$$(p)))
+		$$(call getschemeplusuri,$$(call qstrip,\
+		  $$(if $$(findstring ://,$$(p)),$$(p),\
+			  $$($(2)_SITE)/$$(p))),urlencode))

I am not certain that we want to unconditionally enforce urlencode on all additional downloads...

Instead, I think we should just replicate the site method for extra downloads that use the same site as the main download.

I've sent a patch that does just that:

    https://patchwork.ozlabs.org/patch/1192073/

Can you see if that fixes it for you (it did for me, but if you have an alternate test-case, that'd be nice to have feedback on it too).

Thanks again for the report! :-)

Regards,
Yann E. MORIN.

Quoted 9 lines
package/pkg-generic.mk +0 -0
 ifndef $(2)_SITE
  ifdef $(3)_SITE
Signature · 6 lines
2.22.0

_______________________________________________
buildroot mailing list
buildroot at busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
author
#3 lore raw

PATCH 1/1 fix corrupted download url of extra packages

Correct, if the extra download url is relative to the main, only SITE_METHOD prefix is needed. Thanks for your reviews.

Yann E. MORIN <[email protected]> ? 2019?11?9??? ??1:28???

Quoted 71 lines

Jemy, All,

On 2019-11-05 17:04 +0800, jemy.zhang at gmail.com spake thusly:

Quoted 6 lines

From: "Jemy Zhang" <[email protected]>

If the download url of extra packages contains '+', the contents before '+' will be parsed as SITE_METHOD, the original download url corrupted.

check with android-tools package.

Indeed this is broken. Thank you very much for noticing the issue.

package/pkg-generic.mk endif · line 524

Quoted 12 lines
---
 package/pkg-generic.mk | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)
package/pkg-generic.mk +0 -0
index 893faba22a..bb0ca939a3 100644
@@ -523,8 +523,9 @@ endif
 $(2)_ALL_DOWNLOADS = \
      $$(if

$$($(2)_SOURCE),$$($(2)_SITE_METHOD)+$$($(2)_SITE)/$$($(2)_SOURCE)) \

package/pkg-generic.mk endif · line 528

Quoted 6 lines
package/pkg-generic.mk +3 -2
      $$(foreach p,$$($(2)_PATCH) $$($(2)_EXTRA_DOWNLOADS),\
-             $$(if $$(findstring ://,$$(p)),$$(p),\
-                     $$($(2)_SITE)/$$(p)))
+             $$(call getschemeplusuri,$$(call qstrip,\
+               $$(if $$(findstring ://,$$(p)),$$(p),\
+                       $$($(2)_SITE)/$$(p))),urlencode))

I am not certain that we want to unconditionally enforce urlencode on all additional downloads...

Instead, I think we should just replicate the site method for extra downloads that use the same site as the main download.

I've sent a patch that does just that:

    https://patchwork.ozlabs.org/patch/1192073/

Can you see if that fixes it for you (it did for me, but if you have an alternate test-case, that'd be nice to have feedback on it too).

Thanks again for the report! :-)

Regards,
Yann E. MORIN.

Quoted 9 lines
package/pkg-generic.mk +0 -1
 ifndef $(2)_SITE
  ifdef $(3)_SITE
--

2.22.0

_______________________________________________
buildroot mailing list
buildroot at busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot

--

.-----------------.--------------------.------------------.--------------------.

|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics'
conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___
     |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is
no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v
 conspiracy.  |
Quoted 2 lines

'------------------------------^-------^------------------^--------------------'

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20191109/5535bf9a/attachment-0001.html>