SRC checksum checking broken
I'm not sure how, but this revision broke the checking of checksums:
commit 3d69d9462d550ce4e00e14768cc616bc9ad7e8a5
Author: Richard Purdie <[email protected]> Date: Fri Mar 23 14:52:23 2012 +0000
bitbake/fetch2: Ensure mirror tarballs have .done stamps so we don't redownload and corrupt them
[YOCTO #2154]
(Bitbake rev: 521d6b14151d3bf934b9597557f7ae46e50a3d7a)
The attached log shows this failure - I've modified the busybox_1.19.4.bb recipe to have incorrect checksums. With revision 67c83a8, the failure is correctly identified. With revision 3d69d94, it is not.
The error happens with direct downloads (using DL_DIR) and also with a mirror setup (using own-mirrors).
I've reviewed the change and I don't see how/why this happens.
Signature · 4 lines
------------------------------------------------------------ Gary Thomas | Consulting for the MLB Associates | Embedded world ------------------------------------------------------------
[gthomas@titan poky_test]$ bitbake busybox -c patch
Loading cache: 100% |####################################################################################| ETA: 00:00:00 Loaded 1108 entries from dependency cache.
OE Build Configuration:
BB_VERSION = "1.15.1" TARGET_ARCH = "arm" TARGET_OS = "linux-gnueabi" MACHINE = "beagleboard" DISTRO = "poky" DISTRO_VERSION = "1.1+snapshot-20120410" TUNE_FEATURES = "armv7a vfp neon cortexa8" TARGET_FPU = "vfp-neon"
meta
meta-yocto = "(nobranch):67c83a8c0838c28cfbbb792daa83aa1f3c331302"
NOTE: Resolving any missing task queue dependencies
NOTE: Preparing runqueue
NOTE: Executing SetScene Tasks
NOTE: Executing RunQueue Tasks
NOTE: Running task 8 of 10 (ID: 2, /home/local/poky-master/meta/recipes-core/busybox/busybox_1.19.4.bb, do_fetch)
NOTE: package busybox-1.19.4-r0: task do_fetch: Started
WARNING: Failed to fetch URL http://www.busybox.net/downloads/busybox-1.19.4.tar.bz2;name=tarball
ERROR: Function failed: Fetcher failure for URL: 'http://www.busybox.net/downloads/busybox-1.19.4.tar.bz2;name=tarball'. Checksum mismatch!
File: '/home/local/poky_test/downloads/busybox-1.19.4.tar.bz2' has md5 checksum 9c0cae5a0379228e7b55e5b29528df8e when 9c0cae5a0379228e7b55e5b29528df8f was expected
File: '/home/local/poky_test/downloads/busybox-1.19.4.tar.bz2' has sha256 checksum 9b853406da61ffb59eb488495fe99cbb7fb3dd29a31307fcfa9cf070543710ee when 9b853406da61ffb59eb488495fe99cbb7fb3dd29a31307fcfa9cf070543710ef was expected
ERROR: Logfile of failure stored in: /home/local/poky_test/tmp/work/armv7a-vfp-neon-poky-linux-gnueabi/busybox-1.19.4-r0/temp/log.do_fetch.3868
Log data follows:
| DEBUG: Trying PREMIRRORS
| DEBUG: For url ['http', 'www.busybox.net', '/downloads/busybox-1.19.4.tar.bz2', '', '', {'name': 'tarball'}] comparing ['bzr', '.*', '/.*', '', '', {}] to ['http', 'downloads.yoctoproject.org', '/mirror/sources/', '', '', {}]
| DEBUG: For url ['http', 'www.busybox.net', '/downloads/busybox-1.19.4.tar.bz2', '', '', {'name': 'tarball'}] comparing ['cvs', '.*', '/.*', '', '', {}] to ['http', 'downloads.yoctoproject.org', '/mirror/sources/', '', '', {}]
| DEBUG: For url ['http', 'www.busybox.net', '/downloads/busybox-1.19.4.tar.bz2', '', '', {'name': 'tarball'}] comparing ['git', '.*', '/.*', '', '', {}] to ['http', 'downloads.yoctoproject.org', '/mirror/sources/', '', '', {}]
| DEBUG: For url ['http', 'www.busybox.net', '/downloads/busybox-1.19.4.tar.bz2', '', '', {'name': 'tarball'}] comparing ['hg', '.*', '/.*', '', '', {}] to ['http', 'downloads.yoctoproject.org', '/mirror/sources/', '', '', {}]
| DEBUG: For url ['http', 'www.busybox.net', '/downloads/busybox-1.19.4.tar.bz2', '', '', {'name': 'tarball'}] comparing ['osc', '.*', '/.*', '', '', {}] to ['http', 'downloads.yoctoproject.org', '/mirror/sources/', '', '', {}]
| DEBUG: For url ['http', 'www.busybox.net', '/downloads/busybox-1.19.4.tar.bz2', '', '', {'name': 'tarball'}] comparing ['p4', '.*', '/.*', '', '', {}] to ['http', 'downloads.yoctoproject.org', '/mirror/sources/', '', '', {}]
| DEBUG: For url ['http', 'www.busybox.net', '/downloads/busybox-1.19.4.tar.bz2', '', '', {'name': 'tarball'}] comparing ['svk', '.*', '/.*', '', '', {}] to ['http', 'downloads.yoctoproject.org', '/mirror/sources/', '', '', {}]
| DEBUG: For url ['http', 'www.busybox.net', '/downloads/busybox-1.19.4.tar.bz2', '', '', {'name': 'tarball'}] comparing ['svn', '.*', '/.*', '', '', {}] to ['http', 'downloads.yoctoproject.org', '/mirror/sources/', '', '', {}]
| DEBUG: Trying Upstream
| NOTE: fetch http://www.busybox.net/downloads/busybox-1.19.4.tar.bz2
| DEBUG: executing /usr/bin/env wget -t 5 -nv --passive-ftp --no-check-certificate -P /home/local/poky_test/downloads 'http://www.busybox.net/downloads/busybox-1.19.4.tar.bz2'
| DEBUG: Fetcher accessed the network with the command /usr/bin/env wget -t 5 -nv --passive-ftp --no-check-certificate -P /home/local/poky_test/downloads 'http://www.busybox.net/downloads/busybox-1.19.4.tar.bz2'
| DEBUG: Running export HOME="/home/gthomas"; export GIT_CONFIG="/home/local/poky_test/tmp/sysroots/i686-linux/etc/gitconfig"; export PATH="/home/local/poky_test/tmp/sysroots/i686-linux/usr/bin/armv7a-vfp-neon-poky-linux-gnueabi:/home/local/poky_test/tmp/sysroots/beagleboard/usr/bin/crossscripts:/home/local/poky_test/tmp/sysroots/i686-linux/usr/sbin:/home/local/poky_test/tmp/sysroots/i686-linux/usr/bin:/home/local/poky_test/tmp/sysroots/i686-linux/sbin:/home/local/poky_test/tmp/sysroots/i686-linux//bin:/home/local/poky-master/scripts:/home/local/poky-master/bitbake/bin/:/opt/amltd/bin:/usr/java/jdk1.6.0_10/bin:/home/gthomas/Android/android-sdk-linux_x86-1.1_r1/tools:/home/gthomas/bin:/usr/lib/qt-3.3/bin:/usr/kerberos/sbin:/usr/kerberos/bin:/usr/lib/ccache:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/local/poky-master/scripts"; /usr/bin/env wget -t 5 -nv --passive-ftp --no-check-certificate -P /home/local/poky_test/downloads 'http://www.busybox.net/downloads/busybox-1.19.4.tar.bz2'
| WARNING: Failed to fetch URL http://www.busybox.net/downloads/busybox-1.19.4.tar.bz2;name=tarball
| DEBUG: Fetcher failure for URL: 'http://www.busybox.net/downloads/busybox-1.19.4.tar.bz2;name=tarball'. Checksum mismatch!
| File: '/home/local/poky_test/downloads/busybox-1.19.4.tar.bz2' has md5 checksum 9c0cae5a0379228e7b55e5b29528df8e when 9c0cae5a0379228e7b55e5b29528df8f was expected
| File: '/home/local/poky_test/downloads/busybox-1.19.4.tar.bz2' has sha256 checksum 9b853406da61ffb59eb488495fe99cbb7fb3dd29a31307fcfa9cf070543710ee when 9b853406da61ffb59eb488495fe99cbb7fb3dd29a31307fcfa9cf070543710ef was expected
| DEBUG: Trying MIRRORS
| DEBUG: For url ['http', 'www.busybox.net', '/downloads/busybox-1.19.4.tar.bz2', '', '', {'name': 'tarball'}] comparing ['ftp', '.*', '/.*', '', '', {}] to ['http', 'downloads.yoctoproject.org', '/mirror/sources/', '', '', {}]
| DEBUG: For url ['http', 'www.busybox.net', '/downloads/busybox-1.19.4.tar.bz2', '', '', {'name': 'tarball'}] comparing ['http', '.*', '/.*', '', '', {}] to ['http', 'downloads.yoctoproject.org', '/mirror/sources/', '', '', {}]
| DEBUG: For url http://www.busybox.net/downloads/busybox-1.19.4.tar.bz2;name=tarball returning http://downloads.yoctoproject.org/mirror/sources/busybox-1.19.4.tar.bz2;name=tarball
| NOTE: fetch http://downloads.yoctoproject.org/mirror/sources/busybox-1.19.4.tar.bz2
| DEBUG: executing /usr/bin/env wget -t 5 -nv --passive-ftp --no-check-certificate -P /home/local/poky_test/downloads 'http://downloads.yoctoproject.org/mirror/sources/busybox-1.19.4.tar.bz2'
| DEBUG: Fetcher accessed the network with the command /usr/bin/env wget -t 5 -nv --passive-ftp --no-check-certificate -P /home/local/poky_test/downloads 'http://downloads.yoctoproject.org/mirror/sources/busybox-1.19.4.tar.bz2'
| DEBUG: Running export HOME="/home/gthomas"; export GIT_CONFIG="/home/local/poky_test/tmp/sysroots/i686-linux/etc/gitconfig"; export PATH="/home/local/poky_test/tmp/sysroots/i686-linux/usr/bin/armv7a-vfp-neon-poky-linux-gnueabi:/home/local/poky_test/tmp/sysroots/beagleboard/usr/bin/crossscripts:/home/local/poky_test/tmp/sysroots/i686-linux/usr/sbin:/home/local/poky_test/tmp/sysroots/i686-linux/usr/bin:/home/local/poky_test/tmp/sysroots/i686-linux/sbin:/home/local/poky_test/tmp/sysroots/i686-linux//bin:/home/local/poky-master/scripts:/home/local/poky-master/bitbake/bin/:/opt/amltd/bin:/usr/java/jdk1.6.0_10/bin:/home/gthomas/Android/android-sdk-linux_x86-1.1_r1/tools:/home/gthomas/bin:/usr/lib/qt-3.3/bin:/usr/kerberos/sbin:/usr/kerberos/bin:/usr/lib/ccache:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/local/poky-master/scripts"; /usr/bin/env wget -t 5 -nv --passive-ftp --no-check-certificate -P /home/local/poky_test/downloads 'http://downloads.yoctoproject.org/mirror/sources/busybox-1.19.4.tar.bz2'
| ERROR: Function failed: Fetcher failure for URL: 'http://www.busybox.net/downloads/busybox-1.19.4.tar.bz2;name=tarball'. Checksum mismatch!
| File: '/home/local/poky_test/downloads/busybox-1.19.4.tar.bz2' has md5 checksum 9c0cae5a0379228e7b55e5b29528df8e when 9c0cae5a0379228e7b55e5b29528df8f was expected
| File: '/home/local/poky_test/downloads/busybox-1.19.4.tar.bz2' has sha256 checksum 9b853406da61ffb59eb488495fe99cbb7fb3dd29a31307fcfa9cf070543710ee when 9b853406da61ffb59eb488495fe99cbb7fb3dd29a31307fcfa9cf070543710ef was expectedNOTE: package busybox-1.19.4-r0: task do_fetch: Failed
ERROR: Task 2 (/home/local/poky-master/meta/recipes-core/busybox/busybox_1.19.4.bb, do_fetch) failed with exit code '1'
NOTE: Tasks Summary: Attempted 8 tasks of which 7 didn't need to be rerun and 1 failed.
Summary: 1 task failed:
/home/local/poky-master/meta/recipes-core/busybox/busybox_1.19.4.bb, do_fetch
Summary: There was 1 WARNING message shown.
Summary: There was 1 ERROR message shown, returning a non-zero exit code.
[gthomas@titan poky_test]$ bitbake busybox -c patch
Loading cache: 100% |####################################################################################| ETA: 00:00:00 Loaded 1108 entries from dependency cache.
OE Build Configuration:
BB_VERSION = "1.15.1" TARGET_ARCH = "arm" TARGET_OS = "linux-gnueabi" MACHINE = "beagleboard" DISTRO = "poky" DISTRO_VERSION = "1.1+snapshot-20120410" TUNE_FEATURES = "armv7a vfp neon cortexa8" TARGET_FPU = "vfp-neon"
meta
meta-yocto = "(nobranch):3d69d9462d550ce4e00e14768cc616bc9ad7e8a5"
NOTE: Resolving any missing task queue dependencies
NOTE: Preparing runqueue
NOTE: Executing SetScene Tasks
NOTE: Executing RunQueue Tasks
NOTE: Running task 8 of 10 (ID: 2, /home/local/poky-master/meta/recipes-core/busybox/busybox_1.19.4.bb, do_fetch)
NOTE: package busybox-1.19.4-r0: task do_fetch: Started
WARNING: Failed to fetch URL http://www.busybox.net/downloads/busybox-1.19.4.tar.bz2;name=tarball
NOTE: package busybox-1.19.4-r0: task do_fetch: Succeeded
NOTE: Running task 9 of 10 (ID: 0, /home/local/poky-master/meta/recipes-core/busybox/busybox_1.19.4.bb, do_unpack)
NOTE: package busybox-1.19.4-r0: task do_unpack: Started
NOTE: package busybox-1.19.4-r0: task do_unpack: Succeeded
NOTE: Running task 10 of 10 (ID: 1, /home/local/poky-master/meta/recipes-core/busybox/busybox_1.19.4.bb, do_patch)
NOTE: package busybox-1.19.4-r0: task do_patch: Started
NOTE: package busybox-1.19.4-r0: task do_patch: Succeeded
NOTE: Tasks Summary: Attempted 10 tasks of which 7 didn't need to be rerun and all succeeded.
Summary: There was 1 WARNING message shown.
Quoted 14 lines · Gary Thomas
I'm not sure how, but this revision broke the checking of checksums:
commit
3d69d9462d550ce4e00e14768cc616bc9ad7e8a5Author: Richard Purdie <[email protected]> Date: Fri Mar 23 14:52:23 2012 +0000bitbake/fetch2: Ensure mirror tarballs have .done stamps so we don't redownload and corrupt them
[YOCTO #2154]
(Bitbake rev:
521d6b14151d3bf934b9597557f7ae46e50a3d7a)Signed-off-by Richard Purdie <[email protected]>
Quoted 8 lines
The attached log shows this failure - I've modified the busybox_1.19.4.bb recipe to have incorrect checksums. With revision 67c83a8, the failure is correctly identified. With revision 3d69d94, it is not.
The error happens with direct downloads (using
DL_DIR) and also with a mirror setup (using own-mirrors).I've reviewed the change and I don't see how/why this happens.
There was a patch just merged for something that sounds very similar. I was going to give you a link and realised something has gone wrong with the commit:
http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=a46aafb8b27bd86d046385a569deee6b4c9b8bb6
as its been squashed with a subsequent commit I had to resolve a conflict with.
Regardless, could you see whether that change solves your problem please?
Cheers,
Richard
Quoted 28 lines · Richard Purdie
Quoted 14 lines · Gary Thomas
I'm not sure how, but this revision broke the checking of checksums:
commit
3d69d9462d550ce4e00e14768cc616bc9ad7e8a5Author: Richard Purdie<[email protected]> Date: Fri Mar 23 14:52:23 2012 +0000bitbake/fetch2: Ensure mirror tarballs have .done stamps so we don't redownload and corrupt them
[YOCTO #2154]
(Bitbake rev:
521d6b14151d3bf934b9597557f7ae46e50a3d7a)Signed-off-by Richard Purdie<[email protected]>Quoted 8 lines
The attached log shows this failure - I've modified the busybox_1.19.4.bb recipe to have incorrect checksums. With revision 67c83a8, the failure is correctly identified. With revision 3d69d94, it is not.
The error happens with direct downloads (using
DL_DIR) and also with a mirror setup (using own-mirrors).I've reviewed the change and I don't see how/why this happens.
There was a patch just merged for something that sounds very similar. I was going to give you a link and realised something has gone wrong with the commit:
Quoted 7 lines
http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=a46aafb8b27bd86d046385a569deee6b4c9b8bb6
as its been squashed with a subsequent commit I had to resolve a conflict with.
Regardless, could you see whether that change solves your problem please?
No change, sorry.
Signature · 4 lines
------------------------------------------------------------ Gary Thomas | Consulting for the MLB Associates | Embedded world ------------------------------------------------------------
Quoted 31 lines · Gary Thomas
Quoted 30 lines · Richard Purdie
Quoted 14 lines · Gary Thomas
I'm not sure how, but this revision broke the checking of checksums:
commit
3d69d9462d550ce4e00e14768cc616bc9ad7e8a5
Author: Richard Purdie<[email protected]>
Date: Fri Mar 23 14:52:23 2012 +0000bitbake/fetch2: Ensure mirror tarballs have .done stamps so we don't redownload and corrupt them
[YOCTO #2154]
(Bitbake rev:
521d6b14151d3bf934b9597557f7ae46e50a3d7a)Signed-off-by Richard Purdie<[email protected]>Quoted 8 lines
The attached log shows this failure - I've modified the busybox_1.19.4.bb recipe to have incorrect checksums. With revision 67c83a8, the failure is correctly identified. With revision 3d69d94, it is not.
The error happens with direct downloads (using
DL_DIR) and also with a mirror setup (using own-mirrors).I've reviewed the change and I don't see how/why this happens.
There was a patch just merged for something that sounds very similar. I was going to give you a link and realised something has gone wrong with the commit:
http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=a46aafb8b27bd86d046385a569deee6b4c9b8bb6
Quoted 8 lines
Quoted 5 lines
as its been squashed with a subsequent commit I had to resolve a conflict with.
Regardless, could you see whether that change solves your problem please?
No change, sorry.
I think the problem is this part of the change:
bitbake/lib/bb/fetch2/__init__.py +1 -0
@@ -476,6 +476,7 @@ def try_mirrors(d, origud, mirrors, check = False): if not os.path.exists(ud.donestamp) or ud.method.need_update(newuri, ud, ld): ud.method.download(newuri, ud, ld) + open(ud.donestamp, 'w').close() if hasattr(ud.method,"build_mirror_data"): ud.method.build_mirror_data(newuri, ud, ld)
I think the call to open() that creates the .done stamp file should really be a call to the update_stamp() function. This is the only place checksums are being checked (as far as I can tell) and the fact that the .done file has already been created is getting in the way. I'd suggest a patch, but I don't understand enough of the context to figure out the parameters to call update_stamp() with [parameters named u, ud & d don't help me much]
Signature · 4 lines
------------------------------------------------------------ Gary Thomas | Consulting for the MLB Associates | Embedded world ------------------------------------------------------------
Quoted 54 lines · Gary Thomas
Quoted 31 lines · Gary Thomas
Quoted 30 lines · Richard Purdie
Quoted 14 lines · Gary Thomas
I'm not sure how, but this revision broke the checking of checksums:
commit
3d69d9462d550ce4e00e14768cc616bc9ad7e8a5
Author: Richard Purdie<[email protected]>
Date: Fri Mar 23 14:52:23 2012 +0000bitbake/fetch2: Ensure mirror tarballs have .done stamps so we don't redownload and corrupt them
[YOCTO #2154]
(Bitbake rev:
521d6b14151d3bf934b9597557f7ae46e50a3d7a)Signed-off-by Richard Purdie<[email protected]>Quoted 8 lines
The attached log shows this failure - I've modified the busybox_1.19.4.bb recipe to have incorrect checksums. With revision 67c83a8, the failure is correctly identified. With revision 3d69d94, it is not.
The error happens with direct downloads (using
DL_DIR) and also with a mirror setup (using own-mirrors).I've reviewed the change and I don't see how/why this happens.
There was a patch just merged for something that sounds very similar. I was going to give you a link and realised something has gone wrong with the commit:
http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=a46aafb8b27bd86d046385a569deee6b4c9b8bb6
Quoted 8 lines
Quoted 5 lines
as its been squashed with a subsequent commit I had to resolve a conflict with.
Regardless, could you see whether that change solves your problem please?
No change, sorry.
I think the problem is this part of the change:
bitbake/lib/bb/fetch2/__init__.py +0
-0@@ -476,6 +476,7 @@ def try_mirrors(d, origud, mirrors, check = False):if not
os.path.exists(ud.donestamp)orud.method.need_update(newuri, ud, ld):ud.method.download(newuri, ud, ld)
+open(ud.donestamp, 'w').close()
ifhasattr(ud.method,"build_mirror_data"):ud.method.build_mirror_data(newuri, ud, ld)
Quoted 6 lines
I think the call to
open()that creates the .done stamp file should really be a call to theupdate_stamp()function. This is the only place checksums are being checked (as far as I can tell) and the fact that the .done file has already been created is getting in the way. I'd suggest a patch, but I don't understand enough of the context to figure out the parameters to callupdate_stamp()with [parameters named u, ud & d don't help me much]
This is still broken, both in master (04d6aa1) and 1.2_M4.rc4 (4d9f4d6) Filed as https://bugzilla.yoctoproject.org/show_bug.cgi?id=2311
Signature · 4 lines
------------------------------------------------------------ Gary Thomas | Consulting for the MLB Associates | Embedded world ------------------------------------------------------------
Quoted 55 lines · Gary Thomas
Quoted 54 lines · Gary Thomas
Quoted 31 lines · Gary Thomas
Quoted 30 lines · Richard Purdie
Quoted 14 lines · Gary Thomas
I'm not sure how, but this revision broke the checking of checksums:
commit
3d69d9462d550ce4e00e14768cc616bc9ad7e8a5
Author: Richard Purdie<[email protected]>
Date: Fri Mar 23 14:52:23 2012 +0000bitbake/fetch2: Ensure mirror tarballs have .done stamps so we don't redownload and corrupt them
[YOCTO #2154]
(Bitbake rev:
521d6b14151d3bf934b9597557f7ae46e50a3d7a)Signed-off-by Richard Purdie<[email protected]>Quoted 8 lines
The attached log shows this failure - I've modified the busybox_1.19.4.bb recipe to have incorrect checksums. With revision 67c83a8, the failure is correctly identified. With revision 3d69d94, it is not.
The error happens with direct downloads (using
DL_DIR) and also with a mirror setup (using own-mirrors).I've reviewed the change and I don't see how/why this happens.
There was a patch just merged for something that sounds very similar. I was going to give you a link and realised something has gone wrong with the commit:
http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=a46aafb8b27bd86d046385a569deee6b4c9b8bb6
Quoted 8 lines
Quoted 5 lines
as its been squashed with a subsequent commit I had to resolve a conflict with.
Regardless, could you see whether that change solves your problem please?
No change, sorry.
I think the problem is this part of the change:
bitbake/lib/bb/fetch2/__init__.py +0
-0@@ -476,6 +476,7 @@ def try_mirrors(d, origud, mirrors, check = False):if not
os.path.exists(ud.donestamp)orud.method.need_update(newuri, ud, ld):ud.method.download(newuri, ud, ld)
+open(ud.donestamp, 'w').close()
ifhasattr(ud.method,"build_mirror_data"):ud.method.build_mirror_data(newuri, ud, ld)
Quoted 9 lines
Quoted 6 lines
I think the call to
open()that creates the .done stamp file should really be a call to theupdate_stamp()function. This is the only place checksums are being checked (as far as I can tell) and the fact that the .done file has already been created is getting in the way. I'd suggest a patch, but I don't understand enough of the context to figure out the parameters to callupdate_stamp()with [parameters named u, ud & d don't help me much]This is still broken, both in master (04d6aa1) and 1.2_M4.rc4 (4d9f4d6) Filed as https://bugzilla.yoctoproject.org/show_bug.cgi?id=2311
Does this help?:
bitbake/fetch2: Ensure directly mirrored files have their checksum validated
---
bitbake/lib/bb/fetch2/__init__.py +4 -1
@@ -477,7 +477,10 @@ def try_mirrors(d, origud, mirrors, check = False): if not os.path.exists(ud.donestamp) or ud.method.need_update(newuri, ud, ld): ud.method.download(newuri, ud, ld) if os.path.exists(ud.localpath): - open(ud.donestamp, 'w').close() + if ud.localpath == origud.localpath: + update_stamp(newuri, origud, ld) + else: + open(ud.donestamp, 'w').close() if hasattr(ud.method,"build_mirror_data"): ud.method.build_mirror_data(newuri, ud, ld)
Cheers,
Richard
Quoted 77 lines · Richard Purdie
Quoted 55 lines · Gary Thomas
Quoted 54 lines · Gary Thomas
Quoted 31 lines · Gary Thomas
Quoted 30 lines · Richard Purdie
Quoted 14 lines · Gary Thomas
I'm not sure how, but this revision broke the checking of checksums:
commit
3d69d9462d550ce4e00e14768cc616bc9ad7e8a5
Author: Richard Purdie<[email protected]>
Date: Fri Mar 23 14:52:23 2012 +0000bitbake/fetch2: Ensure mirror tarballs have .done stamps so we don't redownload and corrupt them
[YOCTO #2154]
(Bitbake rev:
521d6b14151d3bf934b9597557f7ae46e50a3d7a)Signed-off-by Richard Purdie<[email protected]>Quoted 8 lines
The attached log shows this failure - I've modified the busybox_1.19.4.bb recipe to have incorrect checksums. With revision 67c83a8, the failure is correctly identified. With revision 3d69d94, it is not.
The error happens with direct downloads (using
DL_DIR) and also with a mirror setup (using own-mirrors).I've reviewed the change and I don't see how/why this happens.
There was a patch just merged for something that sounds very similar. I was going to give you a link and realised something has gone wrong with the commit:
http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=a46aafb8b27bd86d046385a569deee6b4c9b8bb6
Quoted 8 lines
Quoted 5 lines
as its been squashed with a subsequent commit I had to resolve a conflict with.
Regardless, could you see whether that change solves your problem please?
No change, sorry.
I think the problem is this part of the change:
bitbake/lib/bb/fetch2/__init__.py +0
-0@@ -476,6 +476,7 @@ def try_mirrors(d, origud, mirrors, check = False):if not
os.path.exists(ud.donestamp)orud.method.need_update(newuri, ud, ld):ud.method.download(newuri, ud, ld)
+open(ud.donestamp, 'w').close()
ifhasattr(ud.method,"build_mirror_data"):ud.method.build_mirror_data(newuri, ud, ld)Quoted 9 lines
Quoted 6 lines
I think the call to
open()that creates the .done stamp file should really be a call to theupdate_stamp()function. This is the only place checksums are being checked (as far as I can tell) and the fact that the .done file has already been created is getting in the way. I'd suggest a patch, but I don't understand enough of the context to figure out the parameters to callupdate_stamp()with [parameters named u, ud& d don't help me much]This is still broken, both in master (04d6aa1) and 1.2_M4.rc4 (4d9f4d6) Filed as https://bugzilla.yoctoproject.org/show_bug.cgi?id=2311
Does this help?:
bitbake/fetch2: Ensure directly mirrored files have their checksum validated
Signed-off-by Richard Purdie<[email protected]>---
bitbake/lib/bb/fetch2/__init__.py +0
-0
bitbake/lib/bb/fetch2/__init__.py def try_mirrors(d, origud, mirrors, check = False): · line 485
Quoted 10 lines
bitbake/lib/bb/fetch2/__init__.py +4
-1@@ -477,7 +477,10 @@ def try_mirrors(d, origud, mirrors, check = False): if not os.path.exists(ud.donestamp) or ud.method.need_update(newuri, ud, ld): ud.method.download(newuri, ud, ld) if os.path.exists(ud.localpath): - open(ud.donestamp, 'w').close() + if ud.localpath == origud.localpath: + update_stamp(newuri, origud, ld) + else: + open(ud.donestamp, 'w').close() if hasattr(ud.method,"build_mirror_data"): ud.method.build_mirror_data(newuri, ud, ld)
It does fix the case when using DL_DIR, but own-mirrors is still broken.
Note: both used to work correctly.
Signature · 4 lines
------------------------------------------------------------ Gary Thomas | Consulting for the MLB Associates | Embedded world ------------------------------------------------------------
Quoted 83 lines · Gary Thomas
Quoted 82 lines · Richard Purdie
Quoted 55 lines · Gary Thomas
Quoted 54 lines · Gary Thomas
Quoted 31 lines · Gary Thomas
Quoted 30 lines · Richard Purdie
Quoted 14 lines · Gary Thomas
I'm not sure how, but this revision broke the checking of checksums:
commit
3d69d9462d550ce4e00e14768cc616bc9ad7e8a5
Author: Richard Purdie<[email protected]>
Date: Fri Mar 23 14:52:23 2012 +0000bitbake/fetch2: Ensure mirror tarballs have .done stamps so we don't redownload and corrupt them
[YOCTO #2154]
(Bitbake rev:
521d6b14151d3bf934b9597557f7ae46e50a3d7a)Signed-off-by Richard Purdie<[email protected]>Quoted 8 lines
The attached log shows this failure - I've modified the busybox_1.19.4.bb recipe to have incorrect checksums. With revision 67c83a8, the failure is correctly identified. With revision 3d69d94, it is not.
The error happens with direct downloads (using
DL_DIR) and also with a mirror setup (using own-mirrors).I've reviewed the change and I don't see how/why this happens.
There was a patch just merged for something that sounds very similar. I was going to give you a link and realised something has gone wrong with the commit:
http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=a46aafb8b27bd86d046385a569deee6b4c9b8bb6
Quoted 8 lines
Quoted 5 lines
as its been squashed with a subsequent commit I had to resolve a conflict with.
Regardless, could you see whether that change solves your problem please?
No change, sorry.
I think the problem is this part of the change:
bitbake/lib/bb/fetch2/__init__.py +0
-0@@ -476,6 +476,7 @@ def try_mirrors(d, origud, mirrors, check = False):if not
os.path.exists(ud.donestamp)orud.method.need_update(newuri, ud, ld):ud.method.download(newuri, ud, ld)
+open(ud.donestamp, 'w').close()
ifhasattr(ud.method,"build_mirror_data"):ud.method.build_mirror_data(newuri, ud, ld)Quoted 9 lines
Quoted 6 lines
I think the call to
open()that creates the .done stamp file should really be a call to theupdate_stamp()function. This is the only place checksums are being checked (as far as I can tell) and the fact that the .done file has already been created is getting in the way. I'd suggest a patch, but I don't understand enough of the context to figure out the parameters to callupdate_stamp()with [parameters named u, ud& d don't help me much]This is still broken, both in master (04d6aa1) and 1.2_M4.rc4 (4d9f4d6) Filed as https://bugzilla.yoctoproject.org/show_bug.cgi?id=2311
Does this help?:
bitbake/fetch2: Ensure directly mirrored files have their checksum validated
Signed-off-by Richard Purdie<[email protected]>---
bitbake/lib/bb/fetch2/__init__.py +1
-1@@ -477,7 +477,10 @@ def try_mirrors(d, origud, mirrors, check = False): if not os.path.exists(ud.donestamp) or ud.method.need_update(newuri, ud, ld): ud.method.download(newuri, ud, ld) if os.path.exists(ud.localpath): - open(ud.donestamp, 'w').close() + if ud.localpath == origud.localpath:
Quoted 8 lines
bitbake/lib/bb/fetch2/__init__.py def try_mirrors(d, origud, mirrors, check = False): · line 485
Quoted 5 lines
bitbake/lib/bb/fetch2/__init__.py +3
-0+ update_stamp(newuri, origud, ld) + else: + open(ud.donestamp, 'w').close() if hasattr(ud.method,"build_mirror_data"): ud.method.build_mirror_data(newuri, ud, ld)It does fix the case when using
DL_DIR, but own-mirrors is still broken.
Note: both used to work correctly.
Can you give more details of the configuration please? Is this using file:// urls?
For example in the bug you said you did "-c cleansstate" and I'd not expect that to recheck checksums, you'd need to use a "cleanall" for that.
Cheers,
Richard
Quoted 93 lines · Richard Purdie
Quoted 83 lines · Gary Thomas
Quoted 82 lines · Richard Purdie
Quoted 55 lines · Gary Thomas
Quoted 54 lines · Gary Thomas
Quoted 31 lines · Gary Thomas
Quoted 30 lines · Richard Purdie
Quoted 14 lines · Gary Thomas
I'm not sure how, but this revision broke the checking of checksums:
commit
3d69d9462d550ce4e00e14768cc616bc9ad7e8a5
Author: Richard Purdie<[email protected]>
Date: Fri Mar 23 14:52:23 2012 +0000bitbake/fetch2: Ensure mirror tarballs have .done stamps so we don't redownload and corrupt them
[YOCTO #2154]
(Bitbake rev:
521d6b14151d3bf934b9597557f7ae46e50a3d7a)Signed-off-by Richard Purdie<[email protected]>Quoted 8 lines
The attached log shows this failure - I've modified the busybox_1.19.4.bb recipe to have incorrect checksums. With revision 67c83a8, the failure is correctly identified. With revision 3d69d94, it is not.
The error happens with direct downloads (using
DL_DIR) and also with a mirror setup (using own-mirrors).I've reviewed the change and I don't see how/why this happens.
There was a patch just merged for something that sounds very similar. I was going to give you a link and realised something has gone wrong with the commit:
http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=a46aafb8b27bd86d046385a569deee6b4c9b8bb6
Quoted 8 lines
Quoted 5 lines
as its been squashed with a subsequent commit I had to resolve a conflict with.
Regardless, could you see whether that change solves your problem please?
No change, sorry.
I think the problem is this part of the change:
bitbake/lib/bb/fetch2/__init__.py +0
-0@@ -476,6 +476,7 @@ def try_mirrors(d, origud, mirrors, check = False):if not
os.path.exists(ud.donestamp)orud.method.need_update(newuri, ud, ld):ud.method.download(newuri, ud, ld)
+open(ud.donestamp, 'w').close()
ifhasattr(ud.method,"build_mirror_data"):ud.method.build_mirror_data(newuri, ud, ld)Quoted 9 lines
Quoted 6 lines
I think the call to
open()that creates the .done stamp file should really be a call to theupdate_stamp()function. This is the only place checksums are being checked (as far as I can tell) and the fact that the .done file has already been created is getting in the way. I'd suggest a patch, but I don't understand enough of the context to figure out the parameters to callupdate_stamp()with
[parameters named u, ud& d don't help me much]This is still broken, both in master (04d6aa1) and 1.2_M4.rc4 (4d9f4d6) Filed as https://bugzilla.yoctoproject.org/show_bug.cgi?id=2311
Does this help?:
bitbake/fetch2: Ensure directly mirrored files have their checksum validated
Signed-off-by Richard Purdie<[email protected]>---
bitbake/lib/bb/fetch2/__init__.py +1
-1@@ -477,7 +477,10 @@ def try_mirrors(d, origud, mirrors, check = False): if not os.path.exists(ud.donestamp) or ud.method.need_update(newuri, ud, ld): ud.method.download(newuri, ud, ld) if os.path.exists(ud.localpath): - open(ud.donestamp, 'w').close() + if ud.localpath == origud.localpath:Quoted 8 lines
bitbake/lib/bb/fetch2/__init__.py def try_mirrors(d, origud, mirrors, check = False): · line 485
Quoted 5 lines
bitbake/lib/bb/fetch2/__init__.py +3
-0+ update_stamp(newuri, origud, ld) + else: + open(ud.donestamp, 'w').close() if hasattr(ud.method,"build_mirror_data"): ud.method.build_mirror_data(newuri, ud, ld)It does fix the case when using
DL_DIR, but own-mirrors is still broken.
Note: both used to work correctly.
Quoted 2 lines
Can you give more details of the configuration please? Is this using file:// urls?
Here's what I use. I've not tried any other transport method:
SOURCE_MIRROR_URL ?= "file:///work/misc/Poky/sources/" INHERIT += "own-mirrors"
Quoted 4 lines
For example in the bug you said you did "-c cleansstate" and I'd not expect that to recheck checksums, you'd need to use a "cleanall" for that.
I manually removed the ${BUILD}/downloads/busybox* - tarballs and stamps to force it. I didn't think to use cleanall.
Signature · 4 lines
------------------------------------------------------------ Gary Thomas | Consulting for the MLB Associates | Embedded world ------------------------------------------------------------
Quoted 33 lines · Gary Thomas
Quoted 32 lines · Richard Purdie
Quoted 30 lines · Gary Thomas
Quoted 16 lines · Richard Purdie
Quoted 3 lines · Gary Thomas
On 2012-04-10 19:36, Gary Thomas wrote:
This is still broken, both in master (04d6aa1) and 1.2_M4.rc4 (4d9f4d6) Filed as https://bugzilla.yoctoproject.org/show_bug.cgi?id=2311Does this help?:
bitbake/fetch2: Ensure directly mirrored files have their checksum validated
Signed-off-by Richard Purdie<[email protected]>---
bitbake/lib/bb/fetch2/__init__.py +0
-0bitbake/lib/bb/fetch2/__init__.py def try_mirrors(d, origud, mirrors, check = False): · line 485
Quoted 10 lines
bitbake/lib/bb/fetch2/__init__.py +4
-1@@ -477,7 +477,10 @@ def try_mirrors(d, origud, mirrors, check = False): if not os.path.exists(ud.donestamp) or ud.method.need_update(newuri, ud, ld): ud.method.download(newuri, ud, ld) if os.path.exists(ud.localpath): - open(ud.donestamp, 'w').close() + if ud.localpath == origud.localpath: + update_stamp(newuri, origud, ld) + else: + open(ud.donestamp, 'w').close() if hasattr(ud.method,"build_mirror_data"): ud.method.build_mirror_data(newuri, ud, ld)It does fix the case when using
DL_DIR, but own-mirrors is still broken.
Note: both used to work correctly.
Quoted 6 lines
Quoted 2 lines
Can you give more details of the configuration please? Is this using file:// urls?
Here's what I use. I've not tried any other transport method:
SOURCE_MIRROR_URL ?= "file:///work/misc/Poky/sources/" INHERIT += "own-mirrors"
Part of me wants to argue that its not expected to check checksums for file:// urls, as indicated for example by the code:
def verify_checksum(u, ud, d):
[...]
if not ud.type in ["http", "https", "ftp", "ftps"]:
returnbut I can see a case for wanting to check the checksums I guess.
Perhaps this will work better?
bitbake/lib/bb/fetch2/__init__.py +2 -1
@@ -477,7 +477,6 @@ def try_mirrors(d, origud, mirrors, check = False): if not os.path.exists(ud.donestamp) or ud.method.need_update(newuri, ud, ld): ud.method.download(newuri, ud, ld) if os.path.exists(ud.localpath): - open(ud.donestamp, 'w').close() if hasattr(ud.method,"build_mirror_data"): ud.method.build_mirror_data(newuri, ud, ld)
@@ -492,12 +491,14 @@ def try_mirrors(d, origud, mirrors, check = False): dldir = ld.getVar("DL_DIR", True) if os.path.basename(ud.localpath) != os.path.basename(origud.localpath): dest = os.path.join(dldir, os.path.basename(ud.localpath)) + open(ud.donestamp, 'w').close() if not os.path.exists(dest): os.symlink(ud.localpath, dest) return None # Otherwise the result is a local file:// and we symlink to it if not os.path.exists(origud.localpath): os.symlink(ud.localpath, origud.localpath) + update_stamp(newuri, origud, ld) return ud.localpath except bb.fetch2.NetworkAccess:
Cheers,
Richard
Quoted 44 lines · Richard Purdie
Quoted 33 lines · Gary Thomas
Quoted 32 lines · Richard Purdie
Quoted 30 lines · Gary Thomas
Quoted 16 lines · Richard Purdie
Quoted 3 lines · Gary Thomas
On 2012-04-10 19:36, Gary Thomas wrote:
This is still broken, both in master (04d6aa1) and 1.2_M4.rc4 (4d9f4d6) Filed as https://bugzilla.yoctoproject.org/show_bug.cgi?id=2311Does this help?:
bitbake/fetch2: Ensure directly mirrored files have their checksum validated
Signed-off-by Richard Purdie<[email protected]>---
bitbake/lib/bb/fetch2/__init__.py +0
-0bitbake/lib/bb/fetch2/__init__.py def try_mirrors(d, origud, mirrors, check = False): · line 485
Quoted 10 lines
bitbake/lib/bb/fetch2/__init__.py +4
-1@@ -477,7 +477,10 @@ def try_mirrors(d, origud, mirrors, check = False): if not os.path.exists(ud.donestamp) or ud.method.need_update(newuri, ud, ld): ud.method.download(newuri, ud, ld) if os.path.exists(ud.localpath): - open(ud.donestamp, 'w').close() + if ud.localpath == origud.localpath: + update_stamp(newuri, origud, ld) + else: + open(ud.donestamp, 'w').close() if hasattr(ud.method,"build_mirror_data"): ud.method.build_mirror_data(newuri, ud, ld)It does fix the case when using
DL_DIR, but own-mirrors is still broken.
Note: both used to work correctly.Quoted 6 lines
Quoted 2 lines
Can you give more details of the configuration please? Is this using file:// urls?
Here's what I use. I've not tried any other transport method:
SOURCE_MIRROR_URL ?= "file:///work/misc/Poky/sources/" INHERIT += "own-mirrors"Part of me wants to argue that its not expected to check checksums for file:// urls, as indicated for example by the code:
Quoted 6 lines
def
verify_checksum(u, ud, d):[...] if not ud.type in ["http", "https", "ftp", "ftps"]: returnbut I can see a case for wanting to check the checksums I guess.
Well, it seems that the checksums are encouraged, if not required, these days. I was just making sure that they actually work & in my opinion, it shouldn't matter how you got the file, the checksums should still be checked and match.
Quoted 20 lines
Perhaps this will work better?
bitbake/lib/bb/fetch2/__init__.py +1
-1@@ -477,7 +477,6 @@ def try_mirrors(d, origud, mirrors, check = False): if not os.path.exists(ud.donestamp) or ud.method.need_update(newuri, ud, ld): ud.method.download(newuri, ud, ld) if os.path.exists(ud.localpath): - open(ud.donestamp, 'w').close() if hasattr(ud.method,"build_mirror_data"): ud.method.build_mirror_data(newuri, ud, ld)@@ -492,12 +491,14 @@ def try_mirrors(d, origud, mirrors, check = False): dldir = ld.getVar("DL_DIR", True) if os.path.basename(ud.localpath) != os.path.basename(origud.localpath): dest = os.path.join(dldir, os.path.basename(ud.localpath)) + open(ud.donestamp, 'w').close()
bitbake/lib/bb/fetch2/__init__.py def try_mirrors(d, origud, mirrors, check = False): · line 504
Quoted 10 lines
bitbake/lib/bb/fetch2/__init__.py +1
-0if not os.path.exists(dest): os.symlink(ud.localpath, dest) return None # Otherwise the result is a local file:// and we symlink to it if not os.path.exists(origud.localpath): os.symlink(ud.localpath, origud.localpath) + update_stamp(newuri, origud, ld) return ud.localpath except bb.fetch2.NetworkAccess:
Sadly, this is much worse. It even [somehow] reached "through" my mirror setup and deleted the file on the backing store (i.e. the file /work/misc/Poky/sources/busybox-1.19.4.tar.bz2 gets deleted when the checksum fails even though it's part of the own-mirror)
Signature · 4 lines
------------------------------------------------------------ Gary Thomas | Consulting for the MLB Associates | Embedded world ------------------------------------------------------------
Quoted 4 lines · Gary Thomas
On 2012-04-17 10:19, Richard Purdie wrote:
Well, it seems that the checksums are encouraged, if not required, these days. I was just making sure that they actually work & in my opinion, it shouldn't matter how you got the file, the checksums should still be checked and match.
You can argue this both ways. I'm leaning that direction although it has its risks.
Quoted 4 lines
Sadly, this is much worse. It even [somehow] reached "through" my mirror setup and deleted the file on the backing store (i.e. the file /work/misc/Poky/sources/busybox-1.19.4.tar.bz2 gets deleted when the checksum fails even though it's part of the own-mirror)
Was that the only issue? I can see how it managed to do that, I agree it shouldn't :/.
Cheers,
Richard
Quoted 9 lines · Richard Purdie
Quoted 4 lines · Gary Thomas
On 2012-04-17 10:19, Richard Purdie wrote:
Well, it seems that the checksums are encouraged, if not required, these days. I was just making sure that they actually work& in my opinion, it shouldn't matter how you got the file, the checksums should still be checked and match.You can argue this both ways. I'm leaning that direction although it has its risks.
Quoted 7 lines
Quoted 4 lines
Sadly, this is much worse. It even [somehow] reached "through" my mirror setup and deleted the file on the backing store (i.e. the file /work/misc/Poky/sources/busybox-1.19.4.tar.bz2 gets deleted when the checksum fails even though it's part of the own-mirror)
Was that the only issue? I can see how it managed to do that, I agree it shouldn't :/.
It sort of worked - at least it didn't pass silently - but the messages don't say why it failed. Here's what I get:
NOTE: package busybox-1.19.4-r2: task do_fetch: Started
WARNING: Failed to fetch URL http://www.busybox.net/downloads/busybox-1.19.4.tar.bz2;name=tarball
ERROR: Fetcher failure: Fetch command export HOME="/home/gthomas"; export GIT_CONFIG="/home/local/poky_test/tmp/sysroots/i686-linux/etc/gitconfig"; export PATH="/home/local/poky_test/tmp/sysroots/i686-linux/usr/bin/armv7a-vfp-neon-poky-linux-gnueabi:/home/local/poky_test/tmp/sysroots/beagleboard/usr/bin/crossscripts:/home/local/poky_test/tmp/sysroots/i686-linux/usr/sbin:/home/local/poky_test/tmp/sysroots/i686-linux/usr/bin:/home/local/poky_test/tmp/sysroots/i686-linux/sbin:/home/local/poky_test/tmp/sysroots/i686-linux//bin:/home/local/poky-master/scripts:/home/local/poky-master/bitbake/bin/:/opt/amltd/bin:/usr/java/jdk1.6.0_10/bin:/home/gthomas/Android/android-sdk-linux_x86-1.1_r1/tools:/home/gthomas/bin:/usr/lib/qt-3.3/bin:/usr/kerberos/sbin:/usr/kerberos/bin:/usr/lib/ccache:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/local/poky-master/scripts"; /usr/bin/env wget -t 5 -nv --passive-ftp --no-check-certificate -P /home/local/poky_test/downloads 'http://www.busybox.net/downloads/busybox-1.19.4.tar.bz2' could not be run:
None
ERROR: Function failed: Fetcher failure for URL: 'http://www.busybox.net/downloads/busybox-1.19.4.tar.bz2;name=tarball'. Unable to fetch URL from any source.
ERROR: Logfile of failure stored in: /home/local/poky_test/tmp/work/armv7a-vfp-neon-poky-linux-gnueabi/busybox-1.19.4-r2/temp/log.do_fetch.7907
NOTE: package busybox-1.19.4-r2: task do_fetch: Failed
ERROR: Task 2 (/home/local/poky-master/meta/recipes-core/busybox/busybox_1.19.4.bb, do_fetch) failed with exit code '1'
NOTE: Tasks Summary: Attempted 8 tasks of which 7 didn't need to be rerun and 1 failed.
No indication of checksum failures anywhere.
Signature · 4 lines
------------------------------------------------------------ Gary Thomas | Consulting for the MLB Associates | Embedded world ------------------------------------------------------------
Quoted 33 lines · Gary Thomas
Quoted 9 lines · Richard Purdie
Quoted 4 lines · Gary Thomas
On 2012-04-17 10:19, Richard Purdie wrote:
Well, it seems that the checksums are encouraged, if not required, these days. I was just making sure that they actually work& in my opinion, it shouldn't matter how you got the file, the checksums should still be checked and match.You can argue this both ways. I'm leaning that direction although it has its risks.
Quoted 7 lines
Quoted 4 lines
Sadly, this is much worse. It even [somehow] reached "through" my mirror setup and deleted the file on the backing store (i.e. the file /work/misc/Poky/sources/busybox-1.19.4.tar.bz2 gets deleted when the checksum fails even though it's part of the own-mirror)
Was that the only issue? I can see how it managed to do that, I agree it shouldn't :/.
It sort of worked - at least it didn't pass silently - but the messages don't say why it failed. Here's what I get:
NOTE: package busybox-1.19.4-r2: task
do_fetch: Started
WARNING: Failed to fetch URL http://www.busybox.net/downloads/busybox-1.19.4.tar.bz2;name=tarball
ERROR: Fetcher failure: Fetch command export HOME="/home/gthomas"; export GIT_CONFIG="/home/local/poky_test/tmp/sysroots/i686-linux/etc/gitconfig"; export PATH="/home/local/poky_test/tmp/sysroots/i686-linux/usr/bin/armv7a-vfp-neon-poky-linux-gnueabi:/home/local/poky_test/tmp/sysroots/beagleboard/usr/bin/crossscripts:/home/local/poky_test/tmp/sysroots/i686-linux/usr/sbin:/home/local/poky_test/tmp/sysroots/i686-linux/usr/bin:/home/local/poky_test/tmp/sysroots/i686-linux/sbin:/home/local/poky_test/tmp/sysroots/i686-linux//bin:/home/local/poky-master/scripts:/home/local/poky-master/bitbake/bin/:/opt/amltd/bin:/usr/java/jdk1.6.0_10/bin:/home/gthomas/Android/android-sdk-linux_x86-1.1_r1/tools:/home/gthomas/bin:/usr/lib/qt-3.3/bin:/usr/kerberos/sbin:/usr/kerberos/bin:/usr/lib/ccache:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/local/poky-master/scripts";/usr/bin/envwget -t 5 -nv --passive-ftp --no-check-certificate -P/home/local/poky_test/downloads'http://www.busybox.net/downloads/busybox-1.19.4.tar.bz2' could not be run:
None
ERROR: Function failed: Fetcher failure for URL: 'http://www.busybox.net/downloads/busybox-1.19.4.tar.bz2;name=tarball'. Unable to fetch URL from any source.
ERROR: Logfile of failure stored in:/home/local/poky_test/tmp/work/armv7a-vfp-neon-poky-linux-gnueabi/busybox-1.19.4-r2/temp/log.do_fetch.7907
NOTE: package busybox-1.19.4-r2: taskdo_fetch: Failed
ERROR: Task 2 (/home/local/poky-master/meta/recipes-core/busybox/busybox_1.19.4.bb,do_fetch) failed with exit code '1'
NOTE: Tasks Summary: Attempted 8 tasks of which 7 didn't need to be rerun and 1 failed.
Quoted 1 line
No indication of checksum failures anywhere.
Having gone over this several times, I think the original patch is correct but it highlights some other issues. The nastiest of those is the deletion of files. I think I've found a way of fixing that by correctly using fetcher clean functions (which are null operations for the local fetcher).
The remaining issue is then one of better output from the fetcher as there wasn't enough info in the above logs but that is really a separate problem.
I've sent out several fetcher patches to the bitbake list which will hopefully address some of these issues. I'm travelling at the moment and these were written without internet access so my testing is limited to the local file:// urls but I'm confident they move things in the right direction.
Cheers,
Richard
Quoted 46 lines · Richard Purdie
Quoted 33 lines · Gary Thomas
Quoted 9 lines · Richard Purdie
Quoted 4 lines · Gary Thomas
On 2012-04-17 10:19, Richard Purdie wrote:
Well, it seems that the checksums are encouraged, if not required, these days.
I was just making sure that they actually work& in my opinion, it shouldn't
matter how you got the file, the checksums should still be checked and match.You can argue this both ways. I'm leaning that direction although it has its risks.
Quoted 7 lines
Quoted 4 lines
Sadly, this is much worse. It even [somehow] reached "through" my mirror setup and deleted the file on the backing store (i.e. the file /work/misc/Poky/sources/busybox-1.19.4.tar.bz2 gets deleted when the checksum fails even though it's part of the own-mirror)
Was that the only issue? I can see how it managed to do that, I agree it shouldn't :/.
It sort of worked - at least it didn't pass silently - but the messages don't say why it failed. Here's what I get:
NOTE: package busybox-1.19.4-r2: task
do_fetch: Started
WARNING: Failed to fetch URL http://www.busybox.net/downloads/busybox-1.19.4.tar.bz2;name=tarball
ERROR: Fetcher failure: Fetch command export HOME="/home/gthomas"; export GIT_CONFIG="/home/local/poky_test/tmp/sysroots/i686-linux/etc/gitconfig"; export PATH="/home/local/poky_test/tmp/sysroots/i686-linux/usr/bin/armv7a-vfp-neon-poky-linux-gnueabi:/home/local/poky_test/tmp/sysroots/beagleboard/usr/bin/crossscripts:/home/local/poky_test/tmp/sysroots/i686-linux/usr/sbin:/home/local/poky_test/tmp/sysroots/i686-linux/usr/bin:/home/local/poky_test/tmp/sysroots/i686-linux/sbin:/home/local/poky_test/tmp/sysroots/i686-linux//bin:/home/local/poky-master/scripts:/home/local/poky-master/bitbake/bin/:/opt/amltd/bin:/usr/java/jdk1.6.0_10/bin:/home/gthomas/Android/android-sdk-linux_x86-1.1_r1/tools:/home/gthomas/bin:/usr/lib/qt-3.3/bin:/usr/kerberos/sbin:/usr/kerberos/bin:/usr/lib/ccache:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/local/poky-master/scripts";/usr/bin/envwget -t 5 -nv --passive-ftp --no-check-certificate -P/home/local/poky_test/downloads'http://www.busybox.net/downloads/busybox-1.19.4.tar.bz2' could not be run:
None
ERROR: Function failed: Fetcher failure for URL: 'http://www.busybox.net/downloads/busybox-1.19.4.tar.bz2;name=tarball'. Unable to fetch URL from any source.
ERROR: Logfile of failure stored in:/home/local/poky_test/tmp/work/armv7a-vfp-neon-poky-linux-gnueabi/busybox-1.19.4-r2/temp/log.do_fetch.7907
NOTE: package busybox-1.19.4-r2: taskdo_fetch: Failed
ERROR: Task 2 (/home/local/poky-master/meta/recipes-core/busybox/busybox_1.19.4.bb,do_fetch) failed with exit code '1'
NOTE: Tasks Summary: Attempted 8 tasks of which 7 didn't need to be rerun and 1 failed.Quoted 1 line
No indication of checksum failures anywhere.
Having gone over this several times, I think the original patch is correct but it highlights some other issues. The nastiest of those is the deletion of files. I think I've found a way of fixing that by correctly using fetcher clean functions (which are null operations for the local fetcher).
The remaining issue is then one of better output from the fetcher as there wasn't enough info in the above logs but that is really a separate problem.
Quoted 5 lines
I've sent out several fetcher patches to the bitbake list which will hopefully address some of these issues. I'm travelling at the moment and these were written without internet access so my testing is limited to the local file:// urls but I'm confident they move things in the right direction.
Is there something I should check wrt this problem? A bitbake URL/version?
Signature · 4 lines
------------------------------------------------------------ Gary Thomas | Consulting for the MLB Associates | Embedded world ------------------------------------------------------------
Quoted 11 lines · Gary Thomas
Quoted 10 lines · Richard Purdie
Having gone over this several times, I think the original patch is correct but it highlights some other issues. The nastiest of those is the deletion of files. I think I've found a way of fixing that by correctly using fetcher clean functions (which are null operations for the local fetcher).
The remaining issue is then one of better output from the fetcher as there wasn't enough info in the above logs but that is really a separate problem.
Quoted 7 lines
Quoted 5 lines
I've sent out several fetcher patches to the bitbake list which will hopefully address some of these issues. I'm travelling at the moment and these were written without internet access so my testing is limited to the local file:// urls but I'm confident they move things in the right direction.
Is there something I should check wrt this problem? A bitbake URL/version?
I've just checked in a number of patches to master (bitbake and poky) which should fix the various fetcher issues people have been seeing.
I'd appreciate it if you could test with that and see if this issue is now resolved.
Cheers,
Richard
Quoted 18 lines · Richard Purdie
Quoted 17 lines · Gary Thomas
Quoted 15 lines · Richard Purdie
Having gone over this several times, I think the original patch is correct but it highlights some other issues. The nastiest of those is the deletion of files. I think I've found a way of fixing that by correctly using fetcher clean functions (which are null operations for the local fetcher).
The remaining issue is then one of better output from the fetcher as there wasn't enough info in the above logs but that is really a separate problem.
I've sent out several fetcher patches to the bitbake list which will hopefully address some of these issues. I'm travelling at the moment and these were written without internet access so my testing is limited to the local file:// urls but I'm confident they move things in the right direction.
Quoted 7 lines
Quoted 1 line
Is there something I should check wrt this problem? A bitbake URL/version?
I've just checked in a number of patches to master (bitbake and poky) which should fix the various fetcher issues people have been seeing.
I'd appreciate it if you could test with that and see if this issue is now resolved.
Yes, SRC_URI checksum checking seems to be working again. Tested with "own-mirrors" and normal fetching.
Thanks
Signature · 4 lines
------------------------------------------------------------ Gary Thomas | Consulting for the MLB Associates | Embedded world ------------------------------------------------------------