Pull request for branch for-2012.05/wget-download-fix
The following changes since commit dca6e03eac0ec70bb01492e378c694d8dabcedfd:
kernel-headers: bump 3.0.x / 3.2.x stable versions (2012-03-01 14:07:29 +0100)
are available in the git repository at:
git://git.free-electrons.com/users/thomas-petazzoni/buildroot.git for-2012.05/wget-download-fix
Thomas Petazzoni (1):
package: fix WGET download method
package/Makefile.package.in | 8 ++++++-- 1 files changed, 6 insertions(+), 2 deletions(-)
Thanks,
Signature · 1 line
Thomas Petazzoni
PATCH 1/1 package: fix WGET download method
In cf2486bf317e4bbf88c801fb96183ba62be78cc8, we changed from using the -P option of wget (to set the output *directory*) to using the -O option (to set the output *file*). Unfortunately, wget -O has a strange behaviour: it creates an empty 0-byte file even if the download fails (for example when there is no network connection).
The problem is that then Buildroot thinks the download was successful and therefore goes on with extracting the tarball.
The following succession of events makes Buildroot think that the download has been sucessful:
* Buildroot calls the DOWNLOAD_WGET macro with the URL of the official site
* It tests if the file exists in the download directory, it doesn't exist.
* It calls wget. wget fails to download the file and returns an error code, but leaves an empty file with the correct name in the downloaded directory.
* Since the previously download failed, Buildroot tries another download from the Buildroot mirror (sources.buildroot.net)
* It tests if the file exists in the download directory... and it exists! So this second download returns with success, and Buildroot assumes the file has been downloaded properly.
This scenario brings us with the following result, where the download fails, but Buildroot continues its execution and tries to extract the tarball:
$ rm /opt/dl/glib-2.30.2.tar.bz2 rm: cannot remove `/opt/dl/glib-2.30.2.tar.bz2': No such file or directory $ rm -rf build/host-libglib2-2.30.2/ $ make make -C /home/thomas/projets/buildroot O=/opt/outputs/udisks/.
Quoted 1 line
Quoted 1 line
Quoted 1 line
host-libglib2 2.30.2 Downloading
--2012-03-03 12:06:25-- http://ftp.gnome.org/pub/gnome/sources/glib/2.30/glib-2.30.2.tar.bz2
Resolving ftp.gnome.org... failed: Name or service not known.
wget: unable to resolve host address `ftp.gnome.org'
Quoted 1 line
Quoted 1 line
Quoted 1 line
host-libglib2 2.30.2 Extracting
bzcat /opt/dl//glib-2.30.2.tar.bz2 | tar --strip-components=1 -C /opt/outputs/udisks/build/host-libglib2-2.30.2 -xf -
bzcat: Compressed file ends unexpectedly;
perhaps it is corrupted? *Possible* reason follows.
[...]
tar: This does not look like a tar archive
tar: Exiting with failure status due to previous errors
make[1]: *** [/opt/outputs/udisks/build/host-libglib2-2.30.2/.stamp_extracted] Error 2
make: *** [all] Error 2
$ ls -l /opt/dl/glib-2.30.2.tar.bz2 -rw-r--r-- 1 thomas thomas 0 Mar 3 12:12 /opt/dl/glib-2.30.2.tar.bz2
Therefore, this commit modifies DOWNLOAD_WGET so that it removes the downloaded file if wget returns with an error.
--- package/Makefile.package.in | 8 ++++++-- 1 files changed, 6 insertions(+), 2 deletions(-)
package/Makefile.package.in +6 -2
@@ -235,10 +235,14 @@ define SHOW_EXTERNAL_DEPS_HG echo $($(PKG)_SOURCE) endef - +# Download a file using wget. Only download the file if it doesn't +# already exist in the download directory. If the download fails, +# remove the file (because wget -O creates a 0-byte file even if the +# download fails). define DOWNLOAD_WGET test -e $(DL_DIR)/$(2) || \ - $(WGET) -O $(DL_DIR)/$(2) $(call qstrip,$(1))/$(2) + $(WGET) -O $(DL_DIR)/$(2) $(call qstrip,$(1))/$(2) || \ + (rm $(DL_DIR)/$(2) ; exit 1) endef define SOURCE_CHECK_WGET
Signature · 1 line
1.7.4.1
PATCH 1/1 package: fix WGET download method
Hi Thomas,
On Mon, Mar 05, 2012 at 10:31:05AM +0100, Thomas Petazzoni wrote:
[snip]
package/Makefile.package.in define SHOW_EXTERNAL_DEPS_HG · line 245
Quoted 9 lines
package/Makefile.package.in +6
-1+# Download a file using wget. Only download the file if it doesn't +# already exist in the download directory. If the download fails, +# remove the file (because wget -O creates a 0-byte file even if the +# download fails). define DOWNLOAD_WGET test -e $(DL_DIR)/$(2) || \ - $(WGET) -O $(DL_DIR)/$(2) $(call qstrip,$(1))/$(2) + $(WGET) -O $(DL_DIR)/$(2) $(call qstrip,$(1))/$(2) || \ + (rm $(DL_DIR)/$(2) ; exit 1)
'rm -f' may be better here, as wget might error out before creating any file, thus causing a spurious "No such file or directory" message.
baruch
Signature · 3 lines
http://baruch.siach.name/blog/ ~. .~ Tk Open Systems
=}------------------------------------------------ooO--U--Ooo------------{=
- baruch at tkos.co.il - tel: +972.2.679.5364, http://www.tkos.co.il -PATCH 1/1 package: fix WGET download method
Le Mon, 5 Mar 2012 11:42:29 +0200,
Baruch Siach <[email protected]> a ?crit :
Quoted 2 lines
'rm -f' may be better here, as wget might error out before creating any file, thus causing a spurious "No such file or directory" message.
Correct, will send a new version.
Thomas
Signature · 4 lines
Thomas Petazzoni, Free Electrons Kernel, drivers, real-time and embedded Linux development, consulting, training and support. http://free-electrons.com
PATCH 1/1 package: fix WGET download method
In reply to Thomas PetazzoniQuoted 73 lines · Thomas Petazzoni
In
cf2486bf317e4bbf88c801fb96183ba62be78cc8, we changed from using the -P option of wget (to set the output *directory*) to using the -O option (to set the output *file*). Unfortunately, wget -O has a strange behaviour: it creates an empty 0-byte file even if the download fails (for example when there is no network connection).The problem is that then Buildroot thinks the download was successful and therefore goes on with extracting the tarball.
The following succession of events makes Buildroot think that the download has been sucessful:
* Buildroot calls the
DOWNLOAD_WGETmacro with the URL of the official site* It tests if the file exists in the download directory, it doesn't exist.
* It calls wget. wget fails to download the file and returns an error code, but leaves an empty file with the correct name in the downloaded directory.
* Since the previously download failed, Buildroot tries another download from the Buildroot mirror (sources.buildroot.net)
* It tests if the file exists in the download directory... and it exists! So this second download returns with success, and Buildroot assumes the file has been downloaded properly.
This scenario brings us with the following result, where the download fails, but Buildroot continues its execution and tries to extract the tarball:
$ rm /opt/dl/glib-2.30.2.tar.bz2 rm: cannot remove `/opt/dl/glib-2.30.2.tar.bz2': No such file or directory $ rm -rf build/host-libglib2-2.30.2/ $ make make -C /home/thomas/projets/buildroot O=/opt/outputs/udisks/.Quoted 1 line
Quoted 1 line
Quoted 1 line
host-libglib2 2.30.2 Downloading
--2012-03-03 12:06:25-- http://ftp.gnome.org/pub/gnome/sources/glib/2.30/glib-2.30.2.tar.bz2
Resolving ftp.gnome.org... failed: Name or service not known.
wget: unable to resolve host address `ftp.gnome.org'Quoted 1 line
Quoted 1 line
Quoted 1 line
host-libglib2 2.30.2 Extracting
bzcat
/opt/dl//glib-2.30.2.tar.bz2| tar --strip-components=1 -C/opt/outputs/udisks/build/host-libglib2-2.30.2-xf -
bzcat: Compressed file ends unexpectedly;
perhaps it is corrupted? *Possible* reason follows.
[...]
tar: This does not look like a tar archive
tar: Exiting with failure status due to previous errors
make[1]: *** [/opt/outputs/udisks/build/host-libglib2-2.30.2/.stamp_extracted] Error 2
make: *** [all] Error 2$ ls -l /opt/dl/glib-2.30.2.tar.bz2 -rw-r--r-- 1 thomas thomas 0 Mar 3 12:12 /opt/dl/glib-2.30.2.tar.bz2Therefore, this commit modifies
DOWNLOAD_WGETso that it removes the downloaded file if wget returns with an error.Signed-off-by Thomas Petazzoni<[email protected]>--- package/Makefile.package.in | 8 ++++++-- 1 files changed, 6 insertions(+), 2 deletions(-)package/Makefile.package.in +2
-1@@ -235,10 +235,14 @@ define SHOW_EXTERNAL_DEPS_HG echo $($(PKG)_SOURCE) endef - +# Download a file using wget. Only download the file if it doesn't +# already exist in the download directory. If the download fails,
package/Makefile.package.in define SHOW_EXTERNAL_DEPS_HG · line 248
Quoted 10 lines
package/Makefile.package.in +4
-1+# remove the file (because wget -O creates a 0-byte file even if the +# download fails). define DOWNLOAD_WGET test -e $(DL_DIR)/$(2) || \ - $(WGET) -O $(DL_DIR)/$(2) $(call qstrip,$(1))/$(2) + $(WGET) -O $(DL_DIR)/$(2) $(call qstrip,$(1))/$(2) || \ + (rm $(DL_DIR)/$(2) ; exit 1) endef define SOURCE_CHECK_WGET
Luca