Skip to contentSkip to conversation

SRC checksum checking broken

DISCUSSION Gary Thomas 2 participants 17 messages Apr 10, 2012 – May 21, 2012

lore
Share & tools
lore
#0 lore raw

I'm not sure how, but this revision broke the checking of checksums:

commit 3d69d9462d550ce4e00e14768cc616bc9ad7e8a5

Author: Richard Purdie <[email protected]>
Date:   Fri Mar 23 14:52:23 2012 +0000

bitbake/fetch2: Ensure mirror tarballs have .done stamps so we don't redownload and corrupt them

[YOCTO #2154]

(Bitbake rev: 521d6b14151d3bf934b9597557f7ae46e50a3d7a)

The attached log shows this failure - I've modified the busybox_1.19.4.bb recipe to have incorrect checksums. With revision 67c83a8, the failure is correctly identified. With revision 3d69d94, it is not.

The error happens with direct downloads (using DL_DIR) and also with a mirror setup (using own-mirrors).

I've reviewed the change and I don't see how/why this happens.

Signature · 4 lines
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------

[gthomas@titan poky_test]$ bitbake busybox -c patch
Loading cache: 100% |####################################################################################| ETA: 00:00:00 Loaded 1108 entries from dependency cache.

OE Build Configuration:

BB_VERSION        = "1.15.1"
TARGET_ARCH       = "arm"
TARGET_OS         = "linux-gnueabi"
MACHINE           = "beagleboard"
DISTRO            = "poky"
DISTRO_VERSION    = "1.1+snapshot-20120410"
TUNE_FEATURES     = "armv7a vfp neon cortexa8"
TARGET_FPU        = "vfp-neon"

meta
meta-yocto = "(nobranch):67c83a8c0838c28cfbbb792daa83aa1f3c331302"

NOTE: Resolving any missing task queue dependencies
NOTE: Preparing runqueue
NOTE: Executing SetScene Tasks
NOTE: Executing RunQueue Tasks
NOTE: Running task 8 of 10 (ID: 2, /home/local/poky-master/meta/recipes-core/busybox/busybox_1.19.4.bb, do_fetch)
NOTE: package busybox-1.19.4-r0: task do_fetch: Started
WARNING: Failed to fetch URL http://www.busybox.net/downloads/busybox-1.19.4.tar.bz2;name=tarball
ERROR: Function failed: Fetcher failure for URL: 'http://www.busybox.net/downloads/busybox-1.19.4.tar.bz2;name=tarball'. Checksum mismatch!
File: '/home/local/poky_test/downloads/busybox-1.19.4.tar.bz2' has md5 checksum 9c0cae5a0379228e7b55e5b29528df8e when 9c0cae5a0379228e7b55e5b29528df8f was expected
File: '/home/local/poky_test/downloads/busybox-1.19.4.tar.bz2' has sha256 checksum 9b853406da61ffb59eb488495fe99cbb7fb3dd29a31307fcfa9cf070543710ee when 9b853406da61ffb59eb488495fe99cbb7fb3dd29a31307fcfa9cf070543710ef was expected
ERROR: Logfile of failure stored in: /home/local/poky_test/tmp/work/armv7a-vfp-neon-poky-linux-gnueabi/busybox-1.19.4-r0/temp/log.do_fetch.3868
Log data follows:

| DEBUG: Trying PREMIRRORS
| DEBUG: For url ['http', 'www.busybox.net', '/downloads/busybox-1.19.4.tar.bz2', '', '', {'name': 'tarball'}] comparing ['bzr', '.*', '/.*', '', '', {}] to ['http', 'downloads.yoctoproject.org', '/mirror/sources/', '', '', {}]
| DEBUG: For url ['http', 'www.busybox.net', '/downloads/busybox-1.19.4.tar.bz2', '', '', {'name': 'tarball'}] comparing ['cvs', '.*', '/.*', '', '', {}] to ['http', 'downloads.yoctoproject.org', '/mirror/sources/', '', '', {}]
| DEBUG: For url ['http', 'www.busybox.net', '/downloads/busybox-1.19.4.tar.bz2', '', '', {'name': 'tarball'}] comparing ['git', '.*', '/.*', '', '', {}] to ['http', 'downloads.yoctoproject.org', '/mirror/sources/', '', '', {}]
| DEBUG: For url ['http', 'www.busybox.net', '/downloads/busybox-1.19.4.tar.bz2', '', '', {'name': 'tarball'}] comparing ['hg', '.*', '/.*', '', '', {}] to ['http', 'downloads.yoctoproject.org', '/mirror/sources/', '', '', {}]
| DEBUG: For url ['http', 'www.busybox.net', '/downloads/busybox-1.19.4.tar.bz2', '', '', {'name': 'tarball'}] comparing ['osc', '.*', '/.*', '', '', {}] to ['http', 'downloads.yoctoproject.org', '/mirror/sources/', '', '', {}]
| DEBUG: For url ['http', 'www.busybox.net', '/downloads/busybox-1.19.4.tar.bz2', '', '', {'name': 'tarball'}] comparing ['p4', '.*', '/.*', '', '', {}] to ['http', 'downloads.yoctoproject.org', '/mirror/sources/', '', '', {}]
| DEBUG: For url ['http', 'www.busybox.net', '/downloads/busybox-1.19.4.tar.bz2', '', '', {'name': 'tarball'}] comparing ['svk', '.*', '/.*', '', '', {}] to ['http', 'downloads.yoctoproject.org', '/mirror/sources/', '', '', {}]
| DEBUG: For url ['http', 'www.busybox.net', '/downloads/busybox-1.19.4.tar.bz2', '', '', {'name': 'tarball'}] comparing ['svn', '.*', '/.*', '', '', {}] to ['http', 'downloads.yoctoproject.org', '/mirror/sources/', '', '', {}]
| DEBUG: Trying Upstream
| NOTE: fetch http://www.busybox.net/downloads/busybox-1.19.4.tar.bz2
| DEBUG: executing /usr/bin/env wget -t 5 -nv --passive-ftp --no-check-certificate -P /home/local/poky_test/downloads 'http://www.busybox.net/downloads/busybox-1.19.4.tar.bz2'
| DEBUG: Fetcher accessed the network with the command /usr/bin/env wget -t 5 -nv --passive-ftp --no-check-certificate -P /home/local/poky_test/downloads 'http://www.busybox.net/downloads/busybox-1.19.4.tar.bz2'
| DEBUG: Running export HOME="/home/gthomas"; export GIT_CONFIG="/home/local/poky_test/tmp/sysroots/i686-linux/etc/gitconfig"; export PATH="/home/local/poky_test/tmp/sysroots/i686-linux/usr/bin/armv7a-vfp-neon-poky-linux-gnueabi:/home/local/poky_test/tmp/sysroots/beagleboard/usr/bin/crossscripts:/home/local/poky_test/tmp/sysroots/i686-linux/usr/sbin:/home/local/poky_test/tmp/sysroots/i686-linux/usr/bin:/home/local/poky_test/tmp/sysroots/i686-linux/sbin:/home/local/poky_test/tmp/sysroots/i686-linux//bin:/home/local/poky-master/scripts:/home/local/poky-master/bitbake/bin/:/opt/amltd/bin:/usr/java/jdk1.6.0_10/bin:/home/gthomas/Android/android-sdk-linux_x86-1.1_r1/tools:/home/gthomas/bin:/usr/lib/qt-3.3/bin:/usr/kerberos/sbin:/usr/kerberos/bin:/usr/lib/ccache:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/local/poky-master/scripts"; /usr/bin/env wget -t 5 -nv --passive-ftp --no-check-certificate -P /home/local/poky_test/downloads 'http://www.busybox.net/downloads/busybox-1.19.4.tar.bz2'
| WARNING: Failed to fetch URL http://www.busybox.net/downloads/busybox-1.19.4.tar.bz2;name=tarball
| DEBUG: Fetcher failure for URL: 'http://www.busybox.net/downloads/busybox-1.19.4.tar.bz2;name=tarball'. Checksum mismatch!
| File: '/home/local/poky_test/downloads/busybox-1.19.4.tar.bz2' has md5 checksum 9c0cae5a0379228e7b55e5b29528df8e when 9c0cae5a0379228e7b55e5b29528df8f was expected
| File: '/home/local/poky_test/downloads/busybox-1.19.4.tar.bz2' has sha256 checksum 9b853406da61ffb59eb488495fe99cbb7fb3dd29a31307fcfa9cf070543710ee when 9b853406da61ffb59eb488495fe99cbb7fb3dd29a31307fcfa9cf070543710ef was expected
| DEBUG: Trying MIRRORS
| DEBUG: For url ['http', 'www.busybox.net', '/downloads/busybox-1.19.4.tar.bz2', '', '', {'name': 'tarball'}] comparing ['ftp', '.*', '/.*', '', '', {}] to ['http', 'downloads.yoctoproject.org', '/mirror/sources/', '', '', {}]
| DEBUG: For url ['http', 'www.busybox.net', '/downloads/busybox-1.19.4.tar.bz2', '', '', {'name': 'tarball'}] comparing ['http', '.*', '/.*', '', '', {}] to ['http', 'downloads.yoctoproject.org', '/mirror/sources/', '', '', {}]
| DEBUG: For url http://www.busybox.net/downloads/busybox-1.19.4.tar.bz2;name=tarball returning http://downloads.yoctoproject.org/mirror/sources/busybox-1.19.4.tar.bz2;name=tarball
| NOTE: fetch http://downloads.yoctoproject.org/mirror/sources/busybox-1.19.4.tar.bz2
| DEBUG: executing /usr/bin/env wget -t 5 -nv --passive-ftp --no-check-certificate -P /home/local/poky_test/downloads 'http://downloads.yoctoproject.org/mirror/sources/busybox-1.19.4.tar.bz2'
| DEBUG: Fetcher accessed the network with the command /usr/bin/env wget -t 5 -nv --passive-ftp --no-check-certificate -P /home/local/poky_test/downloads 'http://downloads.yoctoproject.org/mirror/sources/busybox-1.19.4.tar.bz2'
| DEBUG: Running export HOME="/home/gthomas"; export GIT_CONFIG="/home/local/poky_test/tmp/sysroots/i686-linux/etc/gitconfig"; export PATH="/home/local/poky_test/tmp/sysroots/i686-linux/usr/bin/armv7a-vfp-neon-poky-linux-gnueabi:/home/local/poky_test/tmp/sysroots/beagleboard/usr/bin/crossscripts:/home/local/poky_test/tmp/sysroots/i686-linux/usr/sbin:/home/local/poky_test/tmp/sysroots/i686-linux/usr/bin:/home/local/poky_test/tmp/sysroots/i686-linux/sbin:/home/local/poky_test/tmp/sysroots/i686-linux//bin:/home/local/poky-master/scripts:/home/local/poky-master/bitbake/bin/:/opt/amltd/bin:/usr/java/jdk1.6.0_10/bin:/home/gthomas/Android/android-sdk-linux_x86-1.1_r1/tools:/home/gthomas/bin:/usr/lib/qt-3.3/bin:/usr/kerberos/sbin:/usr/kerberos/bin:/usr/lib/ccache:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/local/poky-master/scripts"; /usr/bin/env wget -t 5 -nv --passive-ftp --no-check-certificate -P /home/local/poky_test/downloads 'http://downloads.yoctoproject.org/mirror/sources/busybox-1.19.4.tar.bz2'
| ERROR: Function failed: Fetcher failure for URL: 'http://www.busybox.net/downloads/busybox-1.19.4.tar.bz2;name=tarball'. Checksum mismatch!
| File: '/home/local/poky_test/downloads/busybox-1.19.4.tar.bz2' has md5 checksum 9c0cae5a0379228e7b55e5b29528df8e when 9c0cae5a0379228e7b55e5b29528df8f was expected
| File: '/home/local/poky_test/downloads/busybox-1.19.4.tar.bz2' has sha256 checksum 9b853406da61ffb59eb488495fe99cbb7fb3dd29a31307fcfa9cf070543710ee when 9b853406da61ffb59eb488495fe99cbb7fb3dd29a31307fcfa9cf070543710ef was expected

NOTE: package busybox-1.19.4-r0: task do_fetch: Failed
ERROR: Task 2 (/home/local/poky-master/meta/recipes-core/busybox/busybox_1.19.4.bb, do_fetch) failed with exit code '1'
NOTE: Tasks Summary: Attempted 8 tasks of which 7 didn't need to be rerun and 1 failed.

Summary: 1 task failed:
/home/local/poky-master/meta/recipes-core/busybox/busybox_1.19.4.bb, do_fetch
Summary: There was 1 WARNING message shown.
Summary: There was 1 ERROR message shown, returning a non-zero exit code.

[gthomas@titan poky_test]$ bitbake busybox -c patch
Loading cache: 100% |####################################################################################| ETA: 00:00:00 Loaded 1108 entries from dependency cache.

OE Build Configuration:

BB_VERSION        = "1.15.1"
TARGET_ARCH       = "arm"
TARGET_OS         = "linux-gnueabi"
MACHINE           = "beagleboard"
DISTRO            = "poky"
DISTRO_VERSION    = "1.1+snapshot-20120410"
TUNE_FEATURES     = "armv7a vfp neon cortexa8"
TARGET_FPU        = "vfp-neon"

meta
meta-yocto = "(nobranch):3d69d9462d550ce4e00e14768cc616bc9ad7e8a5"

NOTE: Resolving any missing task queue dependencies
NOTE: Preparing runqueue
NOTE: Executing SetScene Tasks
NOTE: Executing RunQueue Tasks
NOTE: Running task 8 of 10 (ID: 2, /home/local/poky-master/meta/recipes-core/busybox/busybox_1.19.4.bb, do_fetch)
NOTE: package busybox-1.19.4-r0: task do_fetch: Started
WARNING: Failed to fetch URL http://www.busybox.net/downloads/busybox-1.19.4.tar.bz2;name=tarball
NOTE: package busybox-1.19.4-r0: task do_fetch: Succeeded
NOTE: Running task 9 of 10 (ID: 0, /home/local/poky-master/meta/recipes-core/busybox/busybox_1.19.4.bb, do_unpack)
NOTE: package busybox-1.19.4-r0: task do_unpack: Started
NOTE: package busybox-1.19.4-r0: task do_unpack: Succeeded
NOTE: Running task 10 of 10 (ID: 1, /home/local/poky-master/meta/recipes-core/busybox/busybox_1.19.4.bb, do_patch)
NOTE: package busybox-1.19.4-r0: task do_patch: Started
NOTE: package busybox-1.19.4-r0: task do_patch: Succeeded
NOTE: Tasks Summary: Attempted 10 tasks of which 7 didn't need to be rerun and all succeeded.

Summary: There was 1 WARNING message shown.

#1 lore raw
Quoted 14 lines · Gary Thomas

I'm not sure how, but this revision broke the checking of checksums:

commit 3d69d9462d550ce4e00e14768cc616bc9ad7e8a5

Author: Richard Purdie <[email protected]>
Date:   Fri Mar 23 14:52:23 2012 +0000

bitbake/fetch2: Ensure mirror tarballs have .done stamps so we don't redownload and corrupt them

[YOCTO #2154]

(Bitbake rev: 521d6b14151d3bf934b9597557f7ae46e50a3d7a)

Quoted 8 lines

The attached log shows this failure - I've modified the busybox_1.19.4.bb recipe to have incorrect checksums. With revision 67c83a8, the failure is correctly identified. With revision 3d69d94, it is not.

The error happens with direct downloads (using DL_DIR) and also with a mirror setup (using own-mirrors).

I've reviewed the change and I don't see how/why this happens.

There was a patch just merged for something that sounds very similar. I was going to give you a link and realised something has gone wrong with the commit:

http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=a46aafb8b27bd86d046385a569deee6b4c9b8bb6

as its been squashed with a subsequent commit I had to resolve a conflict with.

Regardless, could you see whether that change solves your problem please?

Cheers,

Richard

#2 lore raw
Quoted 28 lines · Richard Purdie
Quoted 14 lines · Gary Thomas

I'm not sure how, but this revision broke the checking of checksums:

commit 3d69d9462d550ce4e00e14768cc616bc9ad7e8a5

Author: Richard Purdie<[email protected]>
Date:   Fri Mar 23 14:52:23 2012 +0000

bitbake/fetch2: Ensure mirror tarballs have .done stamps so we don't redownload and corrupt them

[YOCTO #2154]

(Bitbake rev: 521d6b14151d3bf934b9597557f7ae46e50a3d7a)

Quoted 8 lines

The attached log shows this failure - I've modified the busybox_1.19.4.bb recipe to have incorrect checksums. With revision 67c83a8, the failure is correctly identified. With revision 3d69d94, it is not.

The error happens with direct downloads (using DL_DIR) and also with a mirror setup (using own-mirrors).

I've reviewed the change and I don't see how/why this happens.

There was a patch just merged for something that sounds very similar. I was going to give you a link and realised something has gone wrong with the commit:

Quoted 7 lines

http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=a46aafb8b27bd86d046385a569deee6b4c9b8bb6

as its been squashed with a subsequent commit I had to resolve a conflict with.

Regardless, could you see whether that change solves your problem please?

No change, sorry.

Signature · 4 lines
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------
#3 lore raw
Quoted 31 lines · Gary Thomas
Quoted 30 lines · Richard Purdie
Quoted 14 lines · Gary Thomas

I'm not sure how, but this revision broke the checking of checksums:

commit 3d69d9462d550ce4e00e14768cc616bc9ad7e8a5
Author: Richard Purdie<[email protected]>
Date: Fri Mar 23 14:52:23 2012 +0000

bitbake/fetch2: Ensure mirror tarballs have .done stamps so we don't redownload and corrupt them

[YOCTO #2154]

(Bitbake rev: 521d6b14151d3bf934b9597557f7ae46e50a3d7a)

Quoted 8 lines

The attached log shows this failure - I've modified the busybox_1.19.4.bb recipe to have incorrect checksums. With revision 67c83a8, the failure is correctly identified. With revision 3d69d94, it is not.

The error happens with direct downloads (using DL_DIR) and also with a mirror setup (using own-mirrors).

I've reviewed the change and I don't see how/why this happens.

There was a patch just merged for something that sounds very similar. I was going to give you a link and realised something has gone wrong with the commit:

http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=a46aafb8b27bd86d046385a569deee6b4c9b8bb6

Quoted 8 lines
Quoted 5 lines

as its been squashed with a subsequent commit I had to resolve a conflict with.

Regardless, could you see whether that change solves your problem please?

No change, sorry.

I think the problem is this part of the change:

bitbake/lib/bb/fetch2/__init__.py +1 -0
index 7524fa9..e5c4b34 100644
@@ -476,6 +476,7 @@ def try_mirrors(d, origud, mirrors, check = False):
 
             if not os.path.exists(ud.donestamp) or ud.method.need_update(newuri, ud, ld):
                 ud.method.download(newuri, ud, ld)
+                open(ud.donestamp, 'w').close()
                 if hasattr(ud.method,"build_mirror_data"):
                     ud.method.build_mirror_data(newuri, ud, ld)
 

I think the call to open() that creates the .done stamp file should really be a call to the update_stamp() function. This is the only place checksums are being checked (as far as I can tell) and the fact that the .done file has already been created is getting in the way. I'd suggest a patch, but I don't understand enough of the context to figure out the parameters to call update_stamp() with [parameters named u, ud & d don't help me much]

Signature · 4 lines
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------
#4 lore raw
Quoted 54 lines · Gary Thomas
Quoted 31 lines · Gary Thomas
Quoted 30 lines · Richard Purdie
Quoted 14 lines · Gary Thomas

I'm not sure how, but this revision broke the checking of checksums:

commit 3d69d9462d550ce4e00e14768cc616bc9ad7e8a5
Author: Richard Purdie<[email protected]>
Date: Fri Mar 23 14:52:23 2012 +0000

bitbake/fetch2: Ensure mirror tarballs have .done stamps so we don't redownload and corrupt them

[YOCTO #2154]

(Bitbake rev: 521d6b14151d3bf934b9597557f7ae46e50a3d7a)

Quoted 8 lines

The attached log shows this failure - I've modified the busybox_1.19.4.bb recipe to have incorrect checksums. With revision 67c83a8, the failure is correctly identified. With revision 3d69d94, it is not.

The error happens with direct downloads (using DL_DIR) and also with a mirror setup (using own-mirrors).

I've reviewed the change and I don't see how/why this happens.

There was a patch just merged for something that sounds very similar. I was going to give you a link and realised something has gone wrong with the commit:

http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=a46aafb8b27bd86d046385a569deee6b4c9b8bb6

Quoted 8 lines
Quoted 5 lines

as its been squashed with a subsequent commit I had to resolve a conflict with.

Regardless, could you see whether that change solves your problem please?

No change, sorry.

I think the problem is this part of the change:

bitbake/lib/bb/fetch2/__init__.py +0 -0
index 7524fa9..e5c4b34 100644
@@ -476,6 +476,7 @@ def try_mirrors(d, origud, mirrors, check = False):
 

if not os.path.exists(ud.donestamp) or ud.method.need_update(newuri, ud, ld):
ud.method.download(newuri, ud, ld)
+ open(ud.donestamp, 'w').close()
if hasattr(ud.method,"build_mirror_data"):
ud.method.build_mirror_data(newuri, ud, ld)

Quoted 6 lines

I think the call to open() that creates the .done stamp file should really be a call to the update_stamp() function. This is the only place checksums are being checked (as far as I can tell) and the fact that the .done file has already been created is getting in the way. I'd suggest a patch, but I don't understand enough of the context to figure out the parameters to call update_stamp() with [parameters named u, ud & d don't help me much]

This is still broken, both in master (04d6aa1) and 1.2_M4.rc4 (4d9f4d6) Filed as https://bugzilla.yoctoproject.org/show_bug.cgi?id=2311

Signature · 4 lines
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------
#5 lore raw
Quoted 55 lines · Gary Thomas
Quoted 54 lines · Gary Thomas
Quoted 31 lines · Gary Thomas
Quoted 30 lines · Richard Purdie
Quoted 14 lines · Gary Thomas

I'm not sure how, but this revision broke the checking of checksums:

commit 3d69d9462d550ce4e00e14768cc616bc9ad7e8a5
Author: Richard Purdie<[email protected]>
Date: Fri Mar 23 14:52:23 2012 +0000

bitbake/fetch2: Ensure mirror tarballs have .done stamps so we don't redownload and corrupt them

[YOCTO #2154]

(Bitbake rev: 521d6b14151d3bf934b9597557f7ae46e50a3d7a)

Quoted 8 lines

The attached log shows this failure - I've modified the busybox_1.19.4.bb recipe to have incorrect checksums. With revision 67c83a8, the failure is correctly identified. With revision 3d69d94, it is not.

The error happens with direct downloads (using DL_DIR) and also with a mirror setup (using own-mirrors).

I've reviewed the change and I don't see how/why this happens.

There was a patch just merged for something that sounds very similar. I was going to give you a link and realised something has gone wrong with the commit:

http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=a46aafb8b27bd86d046385a569deee6b4c9b8bb6

Quoted 8 lines
Quoted 5 lines

as its been squashed with a subsequent commit I had to resolve a conflict with.

Regardless, could you see whether that change solves your problem please?

No change, sorry.

I think the problem is this part of the change:

bitbake/lib/bb/fetch2/__init__.py +0 -0
index 7524fa9..e5c4b34 100644
@@ -476,6 +476,7 @@ def try_mirrors(d, origud, mirrors, check = False):
 

if not os.path.exists(ud.donestamp) or ud.method.need_update(newuri, ud, ld):
ud.method.download(newuri, ud, ld)
+ open(ud.donestamp, 'w').close()
if hasattr(ud.method,"build_mirror_data"):
ud.method.build_mirror_data(newuri, ud, ld)

Quoted 9 lines
Quoted 6 lines

I think the call to open() that creates the .done stamp file should really be a call to the update_stamp() function. This is the only place checksums are being checked (as far as I can tell) and the fact that the .done file has already been created is getting in the way. I'd suggest a patch, but I don't understand enough of the context to figure out the parameters to call update_stamp() with [parameters named u, ud & d don't help me much]

This is still broken, both in master (04d6aa1) and 1.2_M4.rc4 (4d9f4d6) Filed as https://bugzilla.yoctoproject.org/show_bug.cgi?id=2311

Does this help?:

bitbake/fetch2: Ensure directly mirrored files have their checksum validated

---

bitbake/lib/bb/fetch2/__init__.py +4 -1
index 414cc2b..e36d3ec 100644
@@ -477,7 +477,10 @@ def try_mirrors(d, origud, mirrors, check = False):
             if not os.path.exists(ud.donestamp) or ud.method.need_update(newuri, ud, ld):
                 ud.method.download(newuri, ud, ld)
                 if os.path.exists(ud.localpath):
-                    open(ud.donestamp, 'w').close()
+                    if ud.localpath == origud.localpath:
+                        update_stamp(newuri, origud, ld)
+                    else:
+                        open(ud.donestamp, 'w').close()
                     if hasattr(ud.method,"build_mirror_data"):
                         ud.method.build_mirror_data(newuri, ud, ld)
 

Cheers,

Richard

#6 lore raw
Quoted 77 lines · Richard Purdie
Quoted 55 lines · Gary Thomas
Quoted 54 lines · Gary Thomas
Quoted 31 lines · Gary Thomas
Quoted 30 lines · Richard Purdie
Quoted 14 lines · Gary Thomas

I'm not sure how, but this revision broke the checking of checksums:

commit 3d69d9462d550ce4e00e14768cc616bc9ad7e8a5
Author: Richard Purdie<[email protected]>
Date: Fri Mar 23 14:52:23 2012 +0000

bitbake/fetch2: Ensure mirror tarballs have .done stamps so we don't redownload and corrupt them

[YOCTO #2154]

(Bitbake rev: 521d6b14151d3bf934b9597557f7ae46e50a3d7a)

Quoted 8 lines

The attached log shows this failure - I've modified the busybox_1.19.4.bb recipe to have incorrect checksums. With revision 67c83a8, the failure is correctly identified. With revision 3d69d94, it is not.

The error happens with direct downloads (using DL_DIR) and also with a mirror setup (using own-mirrors).

I've reviewed the change and I don't see how/why this happens.

There was a patch just merged for something that sounds very similar. I was going to give you a link and realised something has gone wrong with the commit:

http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=a46aafb8b27bd86d046385a569deee6b4c9b8bb6

Quoted 8 lines
Quoted 5 lines

as its been squashed with a subsequent commit I had to resolve a conflict with.

Regardless, could you see whether that change solves your problem please?

No change, sorry.

I think the problem is this part of the change:

bitbake/lib/bb/fetch2/__init__.py +0 -0
index 7524fa9..e5c4b34 100644
@@ -476,6 +476,7 @@ def try_mirrors(d, origud, mirrors, check = False):
 

if not os.path.exists(ud.donestamp) or ud.method.need_update(newuri, ud, ld):
ud.method.download(newuri, ud, ld)
+ open(ud.donestamp, 'w').close()
if hasattr(ud.method,"build_mirror_data"):
ud.method.build_mirror_data(newuri, ud, ld)

Quoted 9 lines
Quoted 6 lines

I think the call to open() that creates the .done stamp file should really be a call to the update_stamp() function. This is the only place checksums are being checked (as far as I can tell) and the fact that the .done file has already been created is getting in the way. I'd suggest a patch, but I don't understand enough of the context to figure out the parameters to call update_stamp() with [parameters named u, ud& d don't help me much]

This is still broken, both in master (04d6aa1) and 1.2_M4.rc4 (4d9f4d6) Filed as https://bugzilla.yoctoproject.org/show_bug.cgi?id=2311

Does this help?:

bitbake/fetch2: Ensure directly mirrored files have their checksum validated

---

bitbake/lib/bb/fetch2/__init__.py +0 -0
index 414cc2b..e36d3ec 100644

bitbake/lib/bb/fetch2/__init__.py def try_mirrors(d, origud, mirrors, check = False): · line 485

Quoted 10 lines
bitbake/lib/bb/fetch2/__init__.py +4 -1
@@ -477,7 +477,10 @@ def try_mirrors(d, origud, mirrors, check = False):
              if not os.path.exists(ud.donestamp) or ud.method.need_update(newuri, ud, ld):
                  ud.method.download(newuri, ud, ld)
                  if os.path.exists(ud.localpath):
-                    open(ud.donestamp, 'w').close()
+                    if ud.localpath == origud.localpath:
+                        update_stamp(newuri, origud, ld)
+                    else:
+                        open(ud.donestamp, 'w').close()
                      if hasattr(ud.method,"build_mirror_data"):
                          ud.method.build_mirror_data(newuri, ud, ld)

It does fix the case when using DL_DIR, but own-mirrors is still broken.
Note: both used to work correctly.

Signature · 4 lines
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------
#7 lore raw
Quoted 83 lines · Gary Thomas
Quoted 82 lines · Richard Purdie
Quoted 55 lines · Gary Thomas
Quoted 54 lines · Gary Thomas
Quoted 31 lines · Gary Thomas
Quoted 30 lines · Richard Purdie
Quoted 14 lines · Gary Thomas

I'm not sure how, but this revision broke the checking of checksums:

commit 3d69d9462d550ce4e00e14768cc616bc9ad7e8a5
Author: Richard Purdie<[email protected]>
Date: Fri Mar 23 14:52:23 2012 +0000

bitbake/fetch2: Ensure mirror tarballs have .done stamps so we don't redownload and corrupt them

[YOCTO #2154]

(Bitbake rev: 521d6b14151d3bf934b9597557f7ae46e50a3d7a)

Quoted 8 lines

The attached log shows this failure - I've modified the busybox_1.19.4.bb recipe to have incorrect checksums. With revision 67c83a8, the failure is correctly identified. With revision 3d69d94, it is not.

The error happens with direct downloads (using DL_DIR) and also with a mirror setup (using own-mirrors).

I've reviewed the change and I don't see how/why this happens.

There was a patch just merged for something that sounds very similar. I was going to give you a link and realised something has gone wrong with the commit:

http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=a46aafb8b27bd86d046385a569deee6b4c9b8bb6

Quoted 8 lines
Quoted 5 lines

as its been squashed with a subsequent commit I had to resolve a conflict with.

Regardless, could you see whether that change solves your problem please?

No change, sorry.

I think the problem is this part of the change:

bitbake/lib/bb/fetch2/__init__.py +0 -0
index 7524fa9..e5c4b34 100644
@@ -476,6 +476,7 @@ def try_mirrors(d, origud, mirrors, check = False):
 

if not os.path.exists(ud.donestamp) or ud.method.need_update(newuri, ud, ld):
ud.method.download(newuri, ud, ld)
+ open(ud.donestamp, 'w').close()
if hasattr(ud.method,"build_mirror_data"):
ud.method.build_mirror_data(newuri, ud, ld)

Quoted 9 lines
Quoted 6 lines

I think the call to open() that creates the .done stamp file should really be a call to the update_stamp() function. This is the only place checksums are being checked (as far as I can tell) and the fact that the .done file has already been created is getting in the way. I'd suggest a patch, but I don't understand enough of the context to figure out the parameters to call update_stamp() with [parameters named u, ud& d don't help me much]

This is still broken, both in master (04d6aa1) and 1.2_M4.rc4 (4d9f4d6) Filed as https://bugzilla.yoctoproject.org/show_bug.cgi?id=2311

Does this help?:

bitbake/fetch2: Ensure directly mirrored files have their checksum validated

---

bitbake/lib/bb/fetch2/__init__.py +1 -1
index 414cc2b..e36d3ec 100644
@@ -477,7 +477,10 @@ def try_mirrors(d, origud, mirrors, check = False):
              if not os.path.exists(ud.donestamp) or ud.method.need_update(newuri, ud, ld):
                  ud.method.download(newuri, ud, ld)
                  if os.path.exists(ud.localpath):
-                    open(ud.donestamp, 'w').close()
+                    if ud.localpath == origud.localpath:
Quoted 8 lines

bitbake/lib/bb/fetch2/__init__.py def try_mirrors(d, origud, mirrors, check = False): · line 485

Quoted 5 lines
bitbake/lib/bb/fetch2/__init__.py +3 -0
+                        update_stamp(newuri, origud, ld)
+                    else:
+                        open(ud.donestamp, 'w').close()
                      if hasattr(ud.method,"build_mirror_data"):
                          ud.method.build_mirror_data(newuri, ud, ld)

It does fix the case when using DL_DIR, but own-mirrors is still broken.
Note: both used to work correctly.

Can you give more details of the configuration please? Is this using file:// urls?

For example in the bug you said you did "-c cleansstate" and I'd not expect that to recheck checksums, you'd need to use a "cleanall" for that.

Cheers,

Richard

#8 lore raw
Quoted 93 lines · Richard Purdie
Quoted 83 lines · Gary Thomas
Quoted 82 lines · Richard Purdie
Quoted 55 lines · Gary Thomas
Quoted 54 lines · Gary Thomas
Quoted 31 lines · Gary Thomas
Quoted 30 lines · Richard Purdie
Quoted 14 lines · Gary Thomas

I'm not sure how, but this revision broke the checking of checksums:

commit 3d69d9462d550ce4e00e14768cc616bc9ad7e8a5
Author: Richard Purdie<[email protected]>
Date: Fri Mar 23 14:52:23 2012 +0000

bitbake/fetch2: Ensure mirror tarballs have .done stamps so we don't redownload and corrupt them

[YOCTO #2154]

(Bitbake rev: 521d6b14151d3bf934b9597557f7ae46e50a3d7a)

Quoted 8 lines

The attached log shows this failure - I've modified the busybox_1.19.4.bb recipe to have incorrect checksums. With revision 67c83a8, the failure is correctly identified. With revision 3d69d94, it is not.

The error happens with direct downloads (using DL_DIR) and also with a mirror setup (using own-mirrors).

I've reviewed the change and I don't see how/why this happens.

There was a patch just merged for something that sounds very similar. I was going to give you a link and realised something has gone wrong with the commit:

http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=a46aafb8b27bd86d046385a569deee6b4c9b8bb6

Quoted 8 lines
Quoted 5 lines

as its been squashed with a subsequent commit I had to resolve a conflict with.

Regardless, could you see whether that change solves your problem please?

No change, sorry.

I think the problem is this part of the change:

bitbake/lib/bb/fetch2/__init__.py +0 -0
index 7524fa9..e5c4b34 100644
@@ -476,6 +476,7 @@ def try_mirrors(d, origud, mirrors, check = False):
 

if not os.path.exists(ud.donestamp) or ud.method.need_update(newuri, ud, ld):
ud.method.download(newuri, ud, ld)
+ open(ud.donestamp, 'w').close()
if hasattr(ud.method,"build_mirror_data"):
ud.method.build_mirror_data(newuri, ud, ld)

Quoted 9 lines
Quoted 6 lines

I think the call to open() that creates the .done stamp file should really be a call to the update_stamp() function. This is the only place checksums are being checked (as far as I can tell) and the fact that the .done file has already been created is getting in the way. I'd suggest a patch, but I don't understand enough of the context to figure out the parameters to call update_stamp() with
[parameters named u, ud& d don't help me much]

This is still broken, both in master (04d6aa1) and 1.2_M4.rc4 (4d9f4d6) Filed as https://bugzilla.yoctoproject.org/show_bug.cgi?id=2311

Does this help?:

bitbake/fetch2: Ensure directly mirrored files have their checksum validated

---

bitbake/lib/bb/fetch2/__init__.py +1 -1
index 414cc2b..e36d3ec 100644
@@ -477,7 +477,10 @@ def try_mirrors(d, origud, mirrors, check = False):
               if not os.path.exists(ud.donestamp) or ud.method.need_update(newuri, ud, ld):
                   ud.method.download(newuri, ud, ld)
                   if os.path.exists(ud.localpath):
-                    open(ud.donestamp, 'w').close()
+                    if ud.localpath == origud.localpath:
Quoted 8 lines

bitbake/lib/bb/fetch2/__init__.py def try_mirrors(d, origud, mirrors, check = False): · line 485

Quoted 5 lines
bitbake/lib/bb/fetch2/__init__.py +3 -0
+                        update_stamp(newuri, origud, ld)
+                    else:
+                        open(ud.donestamp, 'w').close()
                       if hasattr(ud.method,"build_mirror_data"):
                           ud.method.build_mirror_data(newuri, ud, ld)

It does fix the case when using DL_DIR, but own-mirrors is still broken.
Note: both used to work correctly.

Quoted 2 lines

Can you give more details of the configuration please? Is this using file:// urls?

Here's what I use. I've not tried any other transport method:
SOURCE_MIRROR_URL ?= "file:///work/misc/Poky/sources/" INHERIT += "own-mirrors"

Quoted 4 lines

For example in the bug you said you did "-c cleansstate" and I'd not expect that to recheck checksums, you'd need to use a "cleanall" for that.

I manually removed the ${BUILD}/downloads/busybox* - tarballs and stamps to force it. I didn't think to use cleanall.

Signature · 4 lines
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------
#9 lore raw
Quoted 33 lines · Gary Thomas
Quoted 32 lines · Richard Purdie
Quoted 30 lines · Gary Thomas
Quoted 16 lines · Richard Purdie
Quoted 3 lines · Gary Thomas

On 2012-04-10 19:36, Gary Thomas wrote:
This is still broken, both in master (04d6aa1) and 1.2_M4.rc4 (4d9f4d6) Filed as https://bugzilla.yoctoproject.org/show_bug.cgi?id=2311

Does this help?:

bitbake/fetch2: Ensure directly mirrored files have their checksum validated

---

bitbake/lib/bb/fetch2/__init__.py +0 -0
index 414cc2b..e36d3ec 100644

bitbake/lib/bb/fetch2/__init__.py def try_mirrors(d, origud, mirrors, check = False): · line 485

Quoted 10 lines
bitbake/lib/bb/fetch2/__init__.py +4 -1
@@ -477,7 +477,10 @@ def try_mirrors(d, origud, mirrors, check = False):
               if not os.path.exists(ud.donestamp) or ud.method.need_update(newuri, ud, ld):
                   ud.method.download(newuri, ud, ld)
                   if os.path.exists(ud.localpath):
-                    open(ud.donestamp, 'w').close()
+                    if ud.localpath == origud.localpath:
+                        update_stamp(newuri, origud, ld)
+                    else:
+                        open(ud.donestamp, 'w').close()
                       if hasattr(ud.method,"build_mirror_data"):
                           ud.method.build_mirror_data(newuri, ud, ld)

It does fix the case when using DL_DIR, but own-mirrors is still broken.
Note: both used to work correctly.

Quoted 6 lines
Quoted 2 lines

Can you give more details of the configuration please? Is this using file:// urls?

Here's what I use. I've not tried any other transport method:

   SOURCE_MIRROR_URL ?= "file:///work/misc/Poky/sources/"
   INHERIT += "own-mirrors"

Part of me wants to argue that its not expected to check checksums for file:// urls, as indicated for example by the code:

def verify_checksum(u, ud, d):

    [...]
    if not ud.type in ["http", "https", "ftp", "ftps"]:
        return

but I can see a case for wanting to check the checksums I guess.

Perhaps this will work better?

bitbake/lib/bb/fetch2/__init__.py +2 -1
index 414cc2b..329b5bc 100644
@@ -477,7 +477,6 @@ def try_mirrors(d, origud, mirrors, check = False):
             if not os.path.exists(ud.donestamp) or ud.method.need_update(newuri, ud, ld):
                 ud.method.download(newuri, ud, ld)
                 if os.path.exists(ud.localpath):
-                    open(ud.donestamp, 'w').close()
                     if hasattr(ud.method,"build_mirror_data"):
                         ud.method.build_mirror_data(newuri, ud, ld)
 
@@ -492,12 +491,14 @@ def try_mirrors(d, origud, mirrors, check = False):
             dldir = ld.getVar("DL_DIR", True)
             if os.path.basename(ud.localpath) != os.path.basename(origud.localpath):
                 dest = os.path.join(dldir, os.path.basename(ud.localpath))
+                open(ud.donestamp, 'w').close()
                 if not os.path.exists(dest):
                     os.symlink(ud.localpath, dest)
                 return None
             # Otherwise the result is a local file:// and we symlink to it
             if not os.path.exists(origud.localpath):
                  os.symlink(ud.localpath, origud.localpath)
+            update_stamp(newuri, origud, ld)
             return ud.localpath
 
         except bb.fetch2.NetworkAccess:

Cheers,

Richard

#10 lore raw
Quoted 44 lines · Richard Purdie
Quoted 33 lines · Gary Thomas
Quoted 32 lines · Richard Purdie
Quoted 30 lines · Gary Thomas
Quoted 16 lines · Richard Purdie
Quoted 3 lines · Gary Thomas

On 2012-04-10 19:36, Gary Thomas wrote:
This is still broken, both in master (04d6aa1) and 1.2_M4.rc4 (4d9f4d6) Filed as https://bugzilla.yoctoproject.org/show_bug.cgi?id=2311

Does this help?:

bitbake/fetch2: Ensure directly mirrored files have their checksum validated

---

bitbake/lib/bb/fetch2/__init__.py +0 -0
index 414cc2b..e36d3ec 100644

bitbake/lib/bb/fetch2/__init__.py def try_mirrors(d, origud, mirrors, check = False): · line 485

Quoted 10 lines
bitbake/lib/bb/fetch2/__init__.py +4 -1
@@ -477,7 +477,10 @@ def try_mirrors(d, origud, mirrors, check = False):
                if not os.path.exists(ud.donestamp) or ud.method.need_update(newuri, ud, ld):
                    ud.method.download(newuri, ud, ld)
                    if os.path.exists(ud.localpath):
-                    open(ud.donestamp, 'w').close()
+                    if ud.localpath == origud.localpath:
+                        update_stamp(newuri, origud, ld)
+                    else:
+                        open(ud.donestamp, 'w').close()
                        if hasattr(ud.method,"build_mirror_data"):
                            ud.method.build_mirror_data(newuri, ud, ld)

It does fix the case when using DL_DIR, but own-mirrors is still broken.
Note: both used to work correctly.

Quoted 6 lines
Quoted 2 lines

Can you give more details of the configuration please? Is this using file:// urls?

Here's what I use. I've not tried any other transport method:

    SOURCE_MIRROR_URL ?= "file:///work/misc/Poky/sources/"
    INHERIT += "own-mirrors"

Part of me wants to argue that its not expected to check checksums for file:// urls, as indicated for example by the code:

Quoted 6 lines

def verify_checksum(u, ud, d):

     [...]
     if not ud.type in ["http", "https", "ftp", "ftps"]:
         return

but I can see a case for wanting to check the checksums I guess.

Well, it seems that the checksums are encouraged, if not required, these days. I was just making sure that they actually work & in my opinion, it shouldn't matter how you got the file, the checksums should still be checked and match.

Quoted 20 lines

Perhaps this will work better?

bitbake/lib/bb/fetch2/__init__.py +1 -1
index 414cc2b..329b5bc 100644
@@ -477,7 +477,6 @@ def try_mirrors(d, origud, mirrors, check = False):
              if not os.path.exists(ud.donestamp) or ud.method.need_update(newuri, ud, ld):
                  ud.method.download(newuri, ud, ld)
                  if os.path.exists(ud.localpath):
-                    open(ud.donestamp, 'w').close()
                      if hasattr(ud.method,"build_mirror_data"):
                          ud.method.build_mirror_data(newuri, ud, ld)
 
@@ -492,12 +491,14 @@ def try_mirrors(d, origud, mirrors, check = False):
              dldir = ld.getVar("DL_DIR", True)
              if os.path.basename(ud.localpath) != os.path.basename(origud.localpath):
                  dest = os.path.join(dldir, os.path.basename(ud.localpath))
+                open(ud.donestamp, 'w').close()

bitbake/lib/bb/fetch2/__init__.py def try_mirrors(d, origud, mirrors, check = False): · line 504

Quoted 10 lines
bitbake/lib/bb/fetch2/__init__.py +1 -0
                  if not os.path.exists(dest):
                      os.symlink(ud.localpath, dest)
                  return None
              # Otherwise the result is a local file:// and we symlink to it
              if not os.path.exists(origud.localpath):
                   os.symlink(ud.localpath, origud.localpath)
+            update_stamp(newuri, origud, ld)
              return ud.localpath
 
          except bb.fetch2.NetworkAccess:

Sadly, this is much worse. It even [somehow] reached "through" my mirror setup and deleted the file on the backing store (i.e. the file /work/misc/Poky/sources/busybox-1.19.4.tar.bz2 gets deleted when the checksum fails even though it's part of the own-mirror)

Signature · 4 lines
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------
#11 lore raw
Quoted 4 lines · Gary Thomas

On 2012-04-17 10:19, Richard Purdie wrote:
Well, it seems that the checksums are encouraged, if not required, these days. I was just making sure that they actually work & in my opinion, it shouldn't matter how you got the file, the checksums should still be checked and match.

You can argue this both ways. I'm leaning that direction although it has its risks.

Quoted 4 lines

Sadly, this is much worse. It even [somehow] reached "through" my mirror setup and deleted the file on the backing store (i.e. the file /work/misc/Poky/sources/busybox-1.19.4.tar.bz2 gets deleted when the checksum fails even though it's part of the own-mirror)

Was that the only issue? I can see how it managed to do that, I agree it shouldn't :/.

Cheers,

Richard

#12 lore raw
Quoted 9 lines · Richard Purdie
Quoted 4 lines · Gary Thomas

On 2012-04-17 10:19, Richard Purdie wrote:
Well, it seems that the checksums are encouraged, if not required, these days. I was just making sure that they actually work& in my opinion, it shouldn't matter how you got the file, the checksums should still be checked and match.

You can argue this both ways. I'm leaning that direction although it has its risks.

Quoted 7 lines
Quoted 4 lines

Sadly, this is much worse. It even [somehow] reached "through" my mirror setup and deleted the file on the backing store (i.e. the file /work/misc/Poky/sources/busybox-1.19.4.tar.bz2 gets deleted when the checksum fails even though it's part of the own-mirror)

Was that the only issue? I can see how it managed to do that, I agree it shouldn't :/.

It sort of worked - at least it didn't pass silently - but the messages don't say why it failed. Here's what I get:

NOTE: package busybox-1.19.4-r2: task do_fetch: Started
WARNING: Failed to fetch URL http://www.busybox.net/downloads/busybox-1.19.4.tar.bz2;name=tarball
ERROR: Fetcher failure: Fetch command export HOME="/home/gthomas"; export GIT_CONFIG="/home/local/poky_test/tmp/sysroots/i686-linux/etc/gitconfig"; export PATH="/home/local/poky_test/tmp/sysroots/i686-linux/usr/bin/armv7a-vfp-neon-poky-linux-gnueabi:/home/local/poky_test/tmp/sysroots/beagleboard/usr/bin/crossscripts:/home/local/poky_test/tmp/sysroots/i686-linux/usr/sbin:/home/local/poky_test/tmp/sysroots/i686-linux/usr/bin:/home/local/poky_test/tmp/sysroots/i686-linux/sbin:/home/local/poky_test/tmp/sysroots/i686-linux//bin:/home/local/poky-master/scripts:/home/local/poky-master/bitbake/bin/:/opt/amltd/bin:/usr/java/jdk1.6.0_10/bin:/home/gthomas/Android/android-sdk-linux_x86-1.1_r1/tools:/home/gthomas/bin:/usr/lib/qt-3.3/bin:/usr/kerberos/sbin:/usr/kerberos/bin:/usr/lib/ccache:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/local/poky-master/scripts"; /usr/bin/env wget -t 5 -nv --passive-ftp --no-check-certificate -P /home/local/poky_test/downloads 'http://www.busybox.net/downloads/busybox-1.19.4.tar.bz2' could not be run:
None
ERROR: Function failed: Fetcher failure for URL: 'http://www.busybox.net/downloads/busybox-1.19.4.tar.bz2;name=tarball'. Unable to fetch URL from any source.
ERROR: Logfile of failure stored in: /home/local/poky_test/tmp/work/armv7a-vfp-neon-poky-linux-gnueabi/busybox-1.19.4-r2/temp/log.do_fetch.7907
NOTE: package busybox-1.19.4-r2: task do_fetch: Failed
ERROR: Task 2 (/home/local/poky-master/meta/recipes-core/busybox/busybox_1.19.4.bb, do_fetch) failed with exit code '1'
NOTE: Tasks Summary: Attempted 8 tasks of which 7 didn't need to be rerun and 1 failed.

No indication of checksum failures anywhere.

Signature · 4 lines
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------
#13 lore raw
Quoted 33 lines · Gary Thomas
Quoted 9 lines · Richard Purdie
Quoted 4 lines · Gary Thomas

On 2012-04-17 10:19, Richard Purdie wrote:
Well, it seems that the checksums are encouraged, if not required, these days. I was just making sure that they actually work& in my opinion, it shouldn't matter how you got the file, the checksums should still be checked and match.

You can argue this both ways. I'm leaning that direction although it has its risks.

Quoted 7 lines
Quoted 4 lines

Sadly, this is much worse. It even [somehow] reached "through" my mirror setup and deleted the file on the backing store (i.e. the file /work/misc/Poky/sources/busybox-1.19.4.tar.bz2 gets deleted when the checksum fails even though it's part of the own-mirror)

Was that the only issue? I can see how it managed to do that, I agree it shouldn't :/.

It sort of worked - at least it didn't pass silently - but the messages don't say why it failed. Here's what I get:

NOTE: package busybox-1.19.4-r2: task do_fetch: Started
WARNING: Failed to fetch URL http://www.busybox.net/downloads/busybox-1.19.4.tar.bz2;name=tarball
ERROR: Fetcher failure: Fetch command export HOME="/home/gthomas"; export GIT_CONFIG="/home/local/poky_test/tmp/sysroots/i686-linux/etc/gitconfig"; export PATH="/home/local/poky_test/tmp/sysroots/i686-linux/usr/bin/armv7a-vfp-neon-poky-linux-gnueabi:/home/local/poky_test/tmp/sysroots/beagleboard/usr/bin/crossscripts:/home/local/poky_test/tmp/sysroots/i686-linux/usr/sbin:/home/local/poky_test/tmp/sysroots/i686-linux/usr/bin:/home/local/poky_test/tmp/sysroots/i686-linux/sbin:/home/local/poky_test/tmp/sysroots/i686-linux//bin:/home/local/poky-master/scripts:/home/local/poky-master/bitbake/bin/:/opt/amltd/bin:/usr/java/jdk1.6.0_10/bin:/home/gthomas/Android/android-sdk-linux_x86-1.1_r1/tools:/home/gthomas/bin:/usr/lib/qt-3.3/bin:/usr/kerberos/sbin:/usr/kerberos/bin:/usr/lib/ccache:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/local/poky-master/scripts"; /usr/bin/env wget -t 5 -nv --passive-ftp --no-check-certificate -P /home/local/poky_test/downloads 'http://www.busybox.net/downloads/busybox-1.19.4.tar.bz2' could not be run:
None
ERROR: Function failed: Fetcher failure for URL: 'http://www.busybox.net/downloads/busybox-1.19.4.tar.bz2;name=tarball'. Unable to fetch URL from any source.
ERROR: Logfile of failure stored in: /home/local/poky_test/tmp/work/armv7a-vfp-neon-poky-linux-gnueabi/busybox-1.19.4-r2/temp/log.do_fetch.7907
NOTE: package busybox-1.19.4-r2: task do_fetch: Failed
ERROR: Task 2 (/home/local/poky-master/meta/recipes-core/busybox/busybox_1.19.4.bb, do_fetch) failed with exit code '1'
NOTE: Tasks Summary: Attempted 8 tasks of which 7 didn't need to be rerun and 1 failed.

Quoted 1 line

No indication of checksum failures anywhere.

Having gone over this several times, I think the original patch is correct but it highlights some other issues. The nastiest of those is the deletion of files. I think I've found a way of fixing that by correctly using fetcher clean functions (which are null operations for the local fetcher).

The remaining issue is then one of better output from the fetcher as there wasn't enough info in the above logs but that is really a separate problem.

I've sent out several fetcher patches to the bitbake list which will hopefully address some of these issues. I'm travelling at the moment and these were written without internet access so my testing is limited to the local file:// urls but I'm confident they move things in the right direction.

Cheers,

Richard

#14 lore raw
Quoted 46 lines · Richard Purdie
Quoted 33 lines · Gary Thomas
Quoted 9 lines · Richard Purdie
Quoted 4 lines · Gary Thomas

On 2012-04-17 10:19, Richard Purdie wrote:
Well, it seems that the checksums are encouraged, if not required, these days.
I was just making sure that they actually work& in my opinion, it shouldn't
matter how you got the file, the checksums should still be checked and match.

You can argue this both ways. I'm leaning that direction although it has its risks.

Quoted 7 lines
Quoted 4 lines

Sadly, this is much worse. It even [somehow] reached "through" my mirror setup and deleted the file on the backing store (i.e. the file /work/misc/Poky/sources/busybox-1.19.4.tar.bz2 gets deleted when the checksum fails even though it's part of the own-mirror)

Was that the only issue? I can see how it managed to do that, I agree it shouldn't :/.

It sort of worked - at least it didn't pass silently - but the messages don't say why it failed. Here's what I get:

NOTE: package busybox-1.19.4-r2: task do_fetch: Started
WARNING: Failed to fetch URL http://www.busybox.net/downloads/busybox-1.19.4.tar.bz2;name=tarball
ERROR: Fetcher failure: Fetch command export HOME="/home/gthomas"; export GIT_CONFIG="/home/local/poky_test/tmp/sysroots/i686-linux/etc/gitconfig"; export PATH="/home/local/poky_test/tmp/sysroots/i686-linux/usr/bin/armv7a-vfp-neon-poky-linux-gnueabi:/home/local/poky_test/tmp/sysroots/beagleboard/usr/bin/crossscripts:/home/local/poky_test/tmp/sysroots/i686-linux/usr/sbin:/home/local/poky_test/tmp/sysroots/i686-linux/usr/bin:/home/local/poky_test/tmp/sysroots/i686-linux/sbin:/home/local/poky_test/tmp/sysroots/i686-linux//bin:/home/local/poky-master/scripts:/home/local/poky-master/bitbake/bin/:/opt/amltd/bin:/usr/java/jdk1.6.0_10/bin:/home/gthomas/Android/android-sdk-linux_x86-1.1_r1/tools:/home/gthomas/bin:/usr/lib/qt-3.3/bin:/usr/kerberos/sbin:/usr/kerberos/bin:/usr/lib/ccache:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/local/poky-master/scripts"; /usr/bin/env wget -t 5 -nv --passive-ftp --no-check-certificate -P /home/local/poky_test/downloads 'http://www.busybox.net/downloads/busybox-1.19.4.tar.bz2' could not be run:
None
ERROR: Function failed: Fetcher failure for URL: 'http://www.busybox.net/downloads/busybox-1.19.4.tar.bz2;name=tarball'. Unable to fetch URL from any source.
ERROR: Logfile of failure stored in: /home/local/poky_test/tmp/work/armv7a-vfp-neon-poky-linux-gnueabi/busybox-1.19.4-r2/temp/log.do_fetch.7907
NOTE: package busybox-1.19.4-r2: task do_fetch: Failed
ERROR: Task 2 (/home/local/poky-master/meta/recipes-core/busybox/busybox_1.19.4.bb, do_fetch) failed with exit code '1'
NOTE: Tasks Summary: Attempted 8 tasks of which 7 didn't need to be rerun and 1 failed.

Quoted 1 line

No indication of checksum failures anywhere.

Having gone over this several times, I think the original patch is correct but it highlights some other issues. The nastiest of those is the deletion of files. I think I've found a way of fixing that by correctly using fetcher clean functions (which are null operations for the local fetcher).

The remaining issue is then one of better output from the fetcher as there wasn't enough info in the above logs but that is really a separate problem.

Quoted 5 lines

I've sent out several fetcher patches to the bitbake list which will hopefully address some of these issues. I'm travelling at the moment and these were written without internet access so my testing is limited to the local file:// urls but I'm confident they move things in the right direction.

Is there something I should check wrt this problem? A bitbake URL/version?

Signature · 4 lines
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------
#15 lore raw
Quoted 11 lines · Gary Thomas
Quoted 10 lines · Richard Purdie

Having gone over this several times, I think the original patch is correct but it highlights some other issues. The nastiest of those is the deletion of files. I think I've found a way of fixing that by correctly using fetcher clean functions (which are null operations for the local fetcher).

The remaining issue is then one of better output from the fetcher as there wasn't enough info in the above logs but that is really a separate problem.

Quoted 7 lines
Quoted 5 lines

I've sent out several fetcher patches to the bitbake list which will hopefully address some of these issues. I'm travelling at the moment and these were written without internet access so my testing is limited to the local file:// urls but I'm confident they move things in the right direction.

Is there something I should check wrt this problem? A bitbake URL/version?

I've just checked in a number of patches to master (bitbake and poky) which should fix the various fetcher issues people have been seeing.

I'd appreciate it if you could test with that and see if this issue is now resolved.

Cheers,

Richard

#16 lore raw
Quoted 18 lines · Richard Purdie
Quoted 17 lines · Gary Thomas
Quoted 15 lines · Richard Purdie

Having gone over this several times, I think the original patch is correct but it highlights some other issues. The nastiest of those is the deletion of files. I think I've found a way of fixing that by correctly using fetcher clean functions (which are null operations for the local fetcher).

The remaining issue is then one of better output from the fetcher as there wasn't enough info in the above logs but that is really a separate problem.

I've sent out several fetcher patches to the bitbake list which will hopefully address some of these issues. I'm travelling at the moment and these were written without internet access so my testing is limited to the local file:// urls but I'm confident they move things in the right direction.

Quoted 7 lines
Quoted 1 line

Is there something I should check wrt this problem? A bitbake URL/version?

I've just checked in a number of patches to master (bitbake and poky) which should fix the various fetcher issues people have been seeing.

I'd appreciate it if you could test with that and see if this issue is now resolved.

Yes, SRC_URI checksum checking seems to be working again. Tested with "own-mirrors" and normal fetching.

Thanks

Signature · 4 lines
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------