Fetcher error reporting
Richard,
Here is an example of the do_fetch/do_unpack failure that I believe is representative of what Dirk saw and what I have been seeing periodically.
I saw the following while building libproxy in poky-image-sdk-live on a Fedora 13 box behind the firewall. I saw identical failures with opkg on a different machine - also behind the firewall. Both SRC_URIs point to googlecode.
log.do_fetch:
-------------
NOTE: fetch http://libproxy.googlecode.com/files/libproxy-0.4.3.tar.gz
NOTE: libproxy-0.4.3: http://libproxy.googlecode.com/files/libproxy-0.4.3.tar.gz has no entry in conf/checksums.ini, not checking URI
log.do_unpack:
--------------
gzip: stdin: unexpected end of file
tar: Child returned status 1
tar: Exiting with failure status due to previous errors
NOTE: Unpacking /vol/1/dvhart/poky.git/build/downloads/libproxy-0.4.3.tar.gz to /vol/1/dvhart/poky.git/build/tmp/work/i586-poky-linux/libproxy-0.4.3-r2/
ERROR: Task failed:
With opkg, I tried the following which resulted in the same error.
$ bitbake -c clean; $ rm downloads/*opkg* $ bitbake opkg
In both cases, using wget to manually download the file and place it in the downloads directory allowed the build to continue.
Signature · 2 lines
Darren Hart Embedded Linux Kernel
Quoted 25 lines · Darren Hart
Here is an example of the do_fetch/do_unpack failure that I believe is representative of what Dirk saw and what I have been seeing periodically.
I saw the following while building libproxy in poky-image-sdk-live on a Fedora 13 box behind the firewall. I saw identical failures with opkg on a different machine - also behind the firewall. Both
SRC_URIspoint to googlecode.log.do_fetch:
-------------
NOTE: fetch http://libproxy.googlecode.com/files/libproxy-0.4.3.tar.gz
NOTE: libproxy-0.4.3:
http://libproxy.googlecode.com/files/libproxy-0.4.3.tar.gz has no entry in conf/checksums.ini, not checking URIlog.do_unpack:
--------------
gzip: stdin: unexpected end of file
tar: Child returned status 1
tar: Exiting with failure status due to previous errors
NOTE: Unpacking /vol/1/dvhart/poky.git/build/downloads/libproxy-0.4.3.tar.gz to /vol/1/dvhart/poky.git/build/tmp/work/i586-poky-linux/libproxy-0.4.3-r2/
ERROR: Task failed:
Quoted 8 lines
With opkg, I tried the following which resulted in the same error.
$ bitbake -c clean; $ rm downloads/*opkg* $ bitbake opkgIn both cases, using wget to manually download the file and place it in the downloads directory allowed the build to continue.
We've talked about this a lot over jabber. We don't have a reproducer. What we do see is its always the wget fetcher that hits this (for mirror urls in the case of git repos) and its failing creating a zero length file.
If we make the assumption that wget is creating the empty file and exiting with an error code and look at what bitbake would do, it would give the behaviour described in these bugs.
Why? If the original fetch fails, it falls back to the mirror code. That looks, sees a file on the disk and says "nothing to do".
Solution is therefore to make sure if the wget fetcher fails we wipe any file that may be present.
Is is possible the wget fetcher is creating a zero length file and no setting an error code but I really hope its not that broken.
So I've pushed this patch as a bandaid for the problem:
http://git.pokylinux.org/cgit/cgit.cgi/poky/commit/?id=c5fab99a6f979a4a0ce246c6395b35a3082aec0d
Cheers,
Richard
Quoted 50 lines · Richard Purdie
Quoted 25 lines · Darren Hart
Here is an example of the do_fetch/do_unpack failure that I believe is representative of what Dirk saw and what I have been seeing periodically.
I saw the following while building libproxy in poky-image-sdk-live on a Fedora 13 box behind the firewall. I saw identical failures with opkg on a different machine - also behind the firewall. Both
SRC_URIspoint to googlecode.log.do_fetch:
-------------
NOTE: fetch http://libproxy.googlecode.com/files/libproxy-0.4.3.tar.gz
NOTE: libproxy-0.4.3:
http://libproxy.googlecode.com/files/libproxy-0.4.3.tar.gz has no entry in conf/checksums.ini, not checking URIlog.do_unpack:
--------------
gzip: stdin: unexpected end of file
tar: Child returned status 1
tar: Exiting with failure status due to previous errors
NOTE: Unpacking /vol/1/dvhart/poky.git/build/downloads/libproxy-0.4.3.tar.gz to /vol/1/dvhart/poky.git/build/tmp/work/i586-poky-linux/libproxy-0.4.3-r2/
ERROR: Task failed:Quoted 8 lines
With opkg, I tried the following which resulted in the same error.
$ bitbake -c clean; $ rm downloads/*opkg* $ bitbake opkgIn both cases, using wget to manually download the file and place it in the downloads directory allowed the build to continue.
We've talked about this a lot over jabber. We don't have a reproducer. What we do see is its always the wget fetcher that hits this (for mirror urls in the case of git repos) and its failing creating a zero length file.
If we make the assumption that wget is creating the empty file and exiting with an error code and look at what bitbake would do, it would give the behaviour described in these bugs.
Why? If the original fetch fails, it falls back to the mirror code. That looks, sees a file on the disk and says "nothing to do".
Solution is therefore to make sure if the wget fetcher fails we wipe any file that may be present.
Quoted 7 lines
Is is possible the wget fetcher is creating a zero length file and no setting an error code but I really hope its not that broken.
So I've pushed this patch as a bandaid for the problem:
http://git.pokylinux.org/cgit/cgit.cgi/poky/commit/?id=c5fab99a6f979a4a0ce246c6395b35a3082aec0d
I've applied and am running another build. This looks sane to me. If I still run into this, I think the next step is to do a brute force zero-length test, and throw the FetchError and take this modified path from this patch in that case too.
wget will generated a 0 length file when using -O (which we confirmed we do not) and there are old bugs against it for creating 0 length files when a valid host fails to connect (without the -O option) but I wasn't able to reproduce that with a current wget.
Here's to hoping this fixes it.
Signature · 2 lines
Darren Hart Embedded Linux Kernel
Quoted 55 lines · Darren Hart
Quoted 54 lines · Richard Purdie
Quoted 26 lines · Darren Hart
Here is an example of the do_fetch/do_unpack failure that I believe is representative of what Dirk saw and what I have been seeing periodically.
I saw the following while building libproxy in poky-image-sdk-live on a Fedora 13 box behind the firewall. I saw identical failures with opkg on a different machine - also behind the firewall. Both
SRC_URIspoint to googlecode.log.do_fetch:
-------------
NOTE: fetch http://libproxy.googlecode.com/files/libproxy-0.4.3.tar.gz
NOTE: libproxy-0.4.3:
http://libproxy.googlecode.com/files/libproxy-0.4.3.tar.gz has no entry in conf/checksums.ini, not checking URIlog.do_unpack:
--------------
gzip: stdin: unexpected end of file
tar: Child returned status 1
tar: Exiting with failure status due to previous errors
NOTE: Unpacking /vol/1/dvhart/poky.git/build/downloads/libproxy-0.4.3.tar.gz to /vol/1/dvhart/poky.git/build/tmp/work/i586-poky-linux/libproxy-0.4.3-r2/
ERROR: Task failed:Quoted 8 lines
With opkg, I tried the following which resulted in the same error.
$ bitbake -c clean; $ rm downloads/*opkg* $ bitbake opkgIn both cases, using wget to manually download the file and place it in the downloads directory allowed the build to continue.
We've talked about this a lot over jabber. We don't have a reproducer. What we do see is its always the wget fetcher that hits this (for mirror urls in the case of git repos) and its failing creating a zero length file.
If we make the assumption that wget is creating the empty file and exiting with an error code and look at what bitbake would do, it would give the behaviour described in these bugs.
Why? If the original fetch fails, it falls back to the mirror code. That looks, sees a file on the disk and says "nothing to do".
Solution is therefore to make sure if the wget fetcher fails we wipe any file that may be present.
Is is possible the wget fetcher is creating a zero length file and no setting an error code but I really hope its not that broken.
Quoted 7 lines
Quoted 5 lines
So I've pushed this patch as a bandaid for the problem:
http://git.pokylinux.org/cgit/cgit.cgi/poky/commit/?id=c5fab99a6f979a4a0ce246c6395b35a3082aec0d
I've applied and am running another build.
One poky-image-sdk-live build completed from a clean slate from inside the firewall. I have another outside the firewall that will take a bit longer, but is well on its way without any errors. Looking good.
Signature · 2 lines
Darren Hart Embedded Linux Kernel