Skip to contentSkip to conversation

tslib keeps failing on checksum

DISCUSSION Mike Looijmans 3 participants 8 messages Nov 20, 2013 1 message not in the archive

lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): [email protected]

#1 lore raw
Quoted 22 lines · Mike Looijmans
Quoted 21 lines · Martin Jansa
Quoted 20 lines · Mike Looijmans

I get this error every time I try t build the current oe-core master:

ERROR: Checksum failure fetching
https://github.com/kergoth/tslib/releases/download/1.1/tslib-1.1.tar.xz;downloadfilename=tslib-1.1.tar.xz

ERROR: Function failed: Fetcher failure for URL:
'https://github.com/kergoth/tslib/releases/download/1.1/tslib-1.1.tar.xz;downloadfilename=tslib-1.1.tar.xz'.

Checksum mismatch!
File: '/home/mike/zynq-next/build/downloads/tslib-1.1.tar.xz' has md5 checksum d41d8cd98f00b204e9800998ecf8427e when 14771f8607b341bb4b297819d37e837d was expected

Now the weird thing is, that when I fetch the thing using wget, I get ablob with the right checksum:

# wget https://github.com/kergoth/tslib/releases/download/1.1/tslib-1.1.tar.xz
# md5sum tslib-1.1.tar.xz
14771f8607b341bb4b297819d37e837d  tslib-1.1.tar.xz
Quoted 8 lines
Quoted 4 lines
Quoted 2 lines

Is oe-core using some other flavour of wget or what else could cause this kind of fetch error?

Have you tried to unpack both tarballs and compare content in them?

Nope, I just assumed they're two somewhat different copies of the same software (e.g. one of them has a "downloaded from here" remark or so). Anything in particular I should look for?

After upgrading I tried again. The file as downloaded by bitbake is completely empty. Nothing in it. The md5 sum of this empty file is d41d8cd98f00b204e9800998ecf8427e indeed.

I'm now using bitbake 1.21.0 (current master) and OE rev 0eb947454e1c92467283e6f1adeca67c7c57698b to build with, with the above results still.

(I know I can just move my own file into the download directory and get on with it, but i'd rather actually solve this problem).

There's a direct connection to the internet, no proxy (just a router) that might have been caching things.

Any suggestions?

Mike.

Met vriendelijke groet / kind regards,

Mike Looijmans

TOPIC Embedded Systems
Eindhovenseweg 32-C, NL-5683 KH Best
Postbus 440, NL-5680 AK Best
Telefoon: (+31) – (0)499 - 33.69.79
Telefax: (+31) - (0)499 - 33.69.70
E-mail: [email protected]
Website: www.topic.nl

Dit e-mail bericht en de eventueel daarbij behorende bijlagen zijn uitsluitend bestemd voor de geadresseerde, zoals die blijkt uit het e-mail bericht en/of de bijlagen. Er kunnen gegevens met betrekking tot een derde instaan. Indien u als niet-geadresseerde dit bericht en de bijlagen ontvangt, terwijl u niet bevoegd of gemachtigd bent om dit bericht namens de geadresseerde te ontvangen, wordt u verzocht de afzender hierover direct te informeren en het e-mail bericht met de bijlagen te vernietigen. Ieder gebruik van de inhoud van het e-mail bericht, waaronder de daarbij behorende bijlagen, door een ander dan de geadresseerde is onrechtmatig jegens ons dan wel de eventueel in het e-mail bericht of de bijlagen voorkomende andere personen. TOPIC Embedded Systems is niet aansprakelijk voor enigerlei schade voortvloeiend uit het gebruik en/of acceptatie van dit e-mail bericht of de daarbij behorende bijlagen.

The contents of this message, as well as any enclosures, are addressed personally to, and thus solely intended for the addressee. They may contain information regarding a third party. A recipient who is neither the addressee, nor empowered to receive this message on behalf of the addressee, is kindly requested to immediately inform the sender of receipt, and to destroy the message and the enclosures. Any use of the contents of this message and/or the enclosures by any other person than the addressee or person who is empowered to receive this message, is illegal towards the sender and/or the aforementioned third party. TOPIC Embedded Systems is not liable for any damage as a result of the use and/or acceptance of this message and as well as any enclosures.

#2 lore raw
Quoted 23 lines · Mike Looijmans
Quoted 22 lines · Mike Looijmans
Quoted 21 lines · Martin Jansa
Quoted 20 lines · Mike Looijmans

I get this error every time I try t build the current oe-core master:

ERROR: Checksum failure fetching
https://github.com/kergoth/tslib/releases/download/1.1/tslib-1.1.tar.xz;downloadfilename=tslib-1.1.tar.xz

ERROR: Function failed: Fetcher failure for URL:
'https://github.com/kergoth/tslib/releases/download/1.1/tslib-1.1.tar.xz;downloadfilename=tslib-1.1.tar.xz'.

Checksum mismatch!
File: '/home/mike/zynq-next/build/downloads/tslib-1.1.tar.xz' has md5 checksum d41d8cd98f00b204e9800998ecf8427e when 14771f8607b341bb4b297819d37e837d was expected

Now the weird thing is, that when I fetch the thing using wget, I get ablob with the right checksum:

# wget https://github.com/kergoth/tslib/releases/download/1.1/tslib-1.1.tar.xz
# md5sum tslib-1.1.tar.xz
14771f8607b341bb4b297819d37e837d  tslib-1.1.tar.xz
Quoted 8 lines
Quoted 8 lines
Quoted 4 lines
Quoted 2 lines

Is oe-core using some other flavour of wget or what else could cause this kind of fetch error?

Have you tried to unpack both tarballs and compare content in them?

Nope, I just assumed they're two somewhat different copies of the same software (e.g. one of them has a "downloaded from here" remark or so). Anything in particular I should look for?

Any difference is important, sometimes you can find there HTML code from some proxy or server which generates HTML page instead of 404 or someone trying to sell you domain of long dead project etc

Quoted 7 lines

After upgrading I tried again. The file as downloaded by bitbake is completely empty. Nothing in it. The md5 sum of this empty file is d41d8cd98f00b204e9800998ecf8427e indeed.

I'm now using bitbake 1.21.0 (current master) and OE rev 0eb947454e1c92467283e6f1adeca67c7c57698b to build with, with the above results still.

OK, I was asking mostly because newer bitbake (1.19+ is new enough) renames file with bad checksum, moving corrupted download aside so it doesn't stand in way for new one.

Quoted 7 lines

(I know I can just move my own file into the download directory and get on with it, but i'd rather actually solve this problem).

There's a direct connection to the internet, no proxy (just a router) that might have been caching things.

Any suggestions?

Check log.do_fetch in WORKDIR/temp, I'm not sure if the error shows only the original URL or also possible (PRE)MIRROR url from which it could download that empty file.

You should see the exact URL in log.do_fetch.

You can also check for (PRE)MIRROR usage in bitbake -e tslib output

Signature · 1 line
Martin 'JaMa' Jansa     jabber: [email protected]
#3 lore raw
Quoted 55 lines · Martin Jansa
Quoted 23 lines · Mike Looijmans
Quoted 22 lines · Mike Looijmans
Quoted 21 lines · Martin Jansa
Quoted 20 lines · Mike Looijmans

I get this error every time I try t build the current oe-core master:

ERROR: Checksum failure fetching
https://github.com/kergoth/tslib/releases/download/1.1/tslib-1.1.tar.xz;downloadfilename=tslib-1.1.tar.xz

ERROR: Function failed: Fetcher failure for URL:
'https://github.com/kergoth/tslib/releases/download/1.1/tslib-1.1.tar.xz;downloadfilename=tslib-1.1.tar.xz'.

Checksum mismatch!
File: '/home/mike/zynq-next/build/downloads/tslib-1.1.tar.xz' has md5 checksum d41d8cd98f00b204e9800998ecf8427e when 14771f8607b341bb4b297819d37e837d was expected

Now the weird thing is, that when I fetch the thing using wget, I get ablob with the right checksum:

# wget https://github.com/kergoth/tslib/releases/download/1.1/tslib-1.1.tar.xz
# md5sum tslib-1.1.tar.xz
14771f8607b341bb4b297819d37e837d  tslib-1.1.tar.xz
Quoted 8 lines
Quoted 8 lines
Quoted 4 lines
Quoted 2 lines

Is oe-core using some other flavour of wget or what else could cause this kind of fetch error?

Have you tried to unpack both tarballs and compare content in them?

Nope, I just assumed they're two somewhat different copies of the same software (e.g. one of them has a "downloaded from here" remark or so). Anything in particular I should look for?

Any difference is important, sometimes you can find there HTML code from some proxy or server which generates HTML page instead of 404 or someone trying to sell you domain of long dead project etc

Quoted 7 lines

After upgrading I tried again. The file as downloaded by bitbake is completely empty. Nothing in it. The md5 sum of this empty file is d41d8cd98f00b204e9800998ecf8427e indeed.

I'm now using bitbake 1.21.0 (current master) and OE rev 0eb947454e1c92467283e6f1adeca67c7c57698b to build with, with the above results still.

OK, I was asking mostly because newer bitbake (1.19+ is new enough) renames file with bad checksum, moving corrupted download aside so it doesn't stand in way for new one.

Quoted 6 lines

(I know I can just move my own file into the download directory and get on with it, but i'd rather actually solve this problem).

There's a direct connection to the internet, no proxy (just a router) that might have been caching things.

Quoted 7 lines
Quoted 1 line

Any suggestions?

Check log.do_fetch in WORKDIR/temp, I'm not sure if the error shows only the original URL or also possible (PRE)MIRROR url from which it could download that empty file.

You should see the exact URL in log.do_fetch.

The problem seems to be that I am "my own mirror". There's a http server that serves sstate-cache and downloads to the rest of the company. And via an NFS mount, that dir is also linked to my local downloads directory. So bitbake ended up executing:

/usr/bin/env wget -t 2 -T 30 -nv --passive-ftp --no-check-certificate -O /home/mike/zynq-next/build/downloads/tslib-1.1.tar.xz -P /home/mike/zynq-next/build/downloads 'http://192.168.80.24/sources/tslib-1.1.tar.xz'

This would create an empty file "tslib-1.1.tar.xz" and the http server happily returns that empty file, and then bitbake thinks it all went well.

I removed the link to the NFS mount, and now it works.

Weird that only tslib suffers from this. Then again, it's probably just the only package that wasn't readily available.

Is this my mistake and don't do this again, or should there be some protection against fetching your own files? (for example, by NOT using the filename itself as a target, but download with a ".part" extension and then rename when done. This also helps when multible builds share the download dir).

Mike.

Met vriendelijke groet / kind regards,

Mike Looijmans

TOPIC Embedded Systems Eindhovenseweg 32-C, NL-5683 KH Best Postbus 440, NL-5680 AK Best
Telefoon: (+31) – (0)499 - 33.69.79
Telefax: (+31) - (0)499 - 33.69.70
E-mail: [email protected]
Website: www.topic.nl

Dit e-mail bericht en de eventueel daarbij behorende bijlagen zijn uitsluitend bestemd voor de geadresseerde, zoals die blijkt uit het e-mail bericht en/of de bijlagen. Er kunnen gegevens met betrekking tot een derde instaan. Indien u als niet-geadresseerde dit bericht en de bijlagen ontvangt, terwijl u niet bevoegd of gemachtigd bent om dit bericht namens de geadresseerde te ontvangen, wordt u verzocht de afzender hierover direct te informeren en het e-mail bericht met de bijlagen te vernietigen. Ieder gebruik van de inhoud van het e-mail bericht, waaronder de daarbij behorende bijlagen, door een ander dan de geadresseerde is onrechtmatig jegens ons dan wel de eventueel in het e-mail bericht of de bijlagen voorkomende andere personen. TOPIC Embedded Systems is niet aansprakelijk voor enigerlei schade voortvloeiend uit het gebruik en/of acceptatie van dit e-mail bericht of de daarbij behorende bijlagen.

The contents of this message, as well as any enclosures, are addressed personally to, and thus solely intended for the addressee. They may contain information regarding a third party. A recipient who is neither the addressee, nor empowered to receive this message on behalf of the addressee, is kindly requested to immediately inform the sender of receipt, and to destroy the message and the enclosures. Any use of the contents of this message and/or the enclosures by any other person than the addressee or person who is empowered to receive this message, is illegal towards the sender and/or the aforementioned third party. TOPIC Embedded Systems is not liable for any damage as a result of the use and/or acceptance of this message and as well as any enclosures.

#4 lore raw
Quoted 74 lines · Mike Looijmans
Quoted 55 lines · Martin Jansa
Quoted 23 lines · Mike Looijmans
Quoted 22 lines · Mike Looijmans
Quoted 21 lines · Martin Jansa
Quoted 20 lines · Mike Looijmans

I get this error every time I try t build the current oe-core master:

ERROR: Checksum failure fetching
https://github.com/kergoth/tslib/releases/download/1.1/tslib-1.1.tar.xz;downloadfilename=tslib-1.1.tar.xz

ERROR: Function failed: Fetcher failure for URL:
'https://github.com/kergoth/tslib/releases/download/1.1/tslib-1.1.tar.xz;downloadfilename=tslib-1.1.tar.xz'.

Checksum mismatch!
File: '/home/mike/zynq-next/build/downloads/tslib-1.1.tar.xz' has md5 checksum d41d8cd98f00b204e9800998ecf8427e when 14771f8607b341bb4b297819d37e837d was expected

Now the weird thing is, that when I fetch the thing using wget, I get ablob with the right checksum:

# wget https://github.com/kergoth/tslib/releases/download/1.1/tslib-1.1.tar.xz
# md5sum tslib-1.1.tar.xz
14771f8607b341bb4b297819d37e837d  tslib-1.1.tar.xz
Quoted 8 lines
Quoted 8 lines
Quoted 4 lines
Quoted 2 lines

Is oe-core using some other flavour of wget or what else could cause this kind of fetch error?

Have you tried to unpack both tarballs and compare content in them?

Nope, I just assumed they're two somewhat different copies of the same software (e.g. one of them has a "downloaded from here" remark or so). Anything in particular I should look for?

Any difference is important, sometimes you can find there HTML code from some proxy or server which generates HTML page instead of 404 or someone trying to sell you domain of long dead project etc

Quoted 7 lines

After upgrading I tried again. The file as downloaded by bitbake is completely empty. Nothing in it. The md5 sum of this empty file is d41d8cd98f00b204e9800998ecf8427e indeed.

I'm now using bitbake 1.21.0 (current master) and OE rev 0eb947454e1c92467283e6f1adeca67c7c57698b to build with, with the above results still.

OK, I was asking mostly because newer bitbake (1.19+ is new enough) renames file with bad checksum, moving corrupted download aside so it doesn't stand in way for new one.

Quoted 6 lines

(I know I can just move my own file into the download directory and get on with it, but i'd rather actually solve this problem).

There's a direct connection to the internet, no proxy (just a router) that might have been caching things.

Quoted 7 lines
Quoted 1 line

Any suggestions?

Check log.do_fetch in WORKDIR/temp, I'm not sure if the error shows only the original URL or also possible (PRE)MIRROR url from which it could download that empty file.

You should see the exact URL in log.do_fetch.

The problem seems to be that I am "my own mirror". There's a http server that serves sstate-cache and downloads to the rest of the company. And via an NFS mount, that dir is also linked to my local downloads directory. So bitbake ended up executing:

/usr/bin/env wget -t 2 -T 30 -nv --passive-ftp --no-check-certificate -O /home/mike/zynq-next/build/downloads/tslib-1.1.tar.xz -P /home/mike/zynq-next/build/downloads 'http://192.168.80.24/sources/tslib-1.1.tar.xz'

Quoted 4 lines

This would create an empty file "tslib-1.1.tar.xz" and the http server happily returns that empty file, and then bitbake thinks it all went well.

I removed the link to the NFS mount, and now it works.

If I understand you correctly your DL_DIR and (PRE)MIRROR point to physically the same directory, correct?

So there is no point to set "duplicate" (PRE)MIRROR on this builder and then it should work fine.

Quoted 26 lines

Weird that only tslib suffers from this. Then again, it's probably just the only package that wasn't readily available.

Is this my mistake and don't do this again, or should there be some protection against fetching your own files? (for example, by NOT using the filename itself as a target, but download with a ".part" extension and then rename when done. This also helps when multible builds share the download dir).

Mike.

Met vriendelijke groet / kind regards,

Mike Looijmans

TOPIC Embedded Systems
Eindhovenseweg 32-C, NL-5683 KH Best
Postbus 440, NL-5680 AK Best
Telefoon: (+31) – (0)499 - 33.69.79
Telefax: (+31) - (0)499 - 33.69.70
E-mail: [email protected]
Website: www.topic.nl

Dit e-mail bericht en de eventueel daarbij behorende bijlagen zijn uitsluitend bestemd voor de geadresseerde, zoals die blijkt uit het e-mail bericht en/of de bijlagen. Er kunnen gegevens met betrekking tot een derde instaan. Indien u als niet-geadresseerde dit bericht en de bijlagen ontvangt, terwijl u niet bevoegd of gemachtigd bent om dit bericht namens de geadresseerde te ontvangen, wordt u verzocht de afzender hierover direct te informeren en het e-mail bericht met de bijlagen te vernietigen. Ieder gebruik van de inhoud van het e-mail bericht, waaronder de daarbij behorende bijlagen, door een ander dan de geadresseerde is onrechtmatig jegens ons dan wel de eventueel in het e-mail bericht of de bijlagen voorkomende andere personen. TOPIC Embedded Systems is niet aansprakelijk voor enigerlei schade voortvloeiend uit het gebruik en/of acceptatie van dit e-mail bericht of de daarbij behorende bijlagen.

The contents of this message, as well as any enclosures, are addressed personally to, and thus solely intended for the addressee. They may contain information regarding a third party. A recipient who is neither the addressee, nor empowered to receive this message on behalf of the addressee, is kindly requested to immediately inform the sender of receipt, and to destroy the message and the enclosures. Any use of the contents of this message and/or the enclosures by any other person than the addressee or person who is empowered to receive this message, is illegal towards the sender and/or the aforementioned third party. TOPIC Embedded Systems is not liable for any damage as a result of the use and/or acceptance of this message and as well as any enclosures.

Signature · 1 line
Martin 'JaMa' Jansa     jabber: [email protected]
#5 lore raw
In reply to Mike Looijmans
Quoted 79 lines · Mike Looijmans
Quoted 55 lines · Martin Jansa
Quoted 23 lines · Mike Looijmans
Quoted 22 lines · Mike Looijmans
Quoted 21 lines · Martin Jansa
Quoted 20 lines · Mike Looijmans

I get this error every time I try t build the current oe-core master:

ERROR: Checksum failure fetching
https://github.com/kergoth/tslib/releases/download/1.1/tslib-1.1.tar.xz;downloadfilename=tslib-1.1.tar.xz

ERROR: Function failed: Fetcher failure for URL:
'https://github.com/kergoth/tslib/releases/download/1.1/tslib-1.1.tar.xz;downloadfilename=tslib-1.1.tar.xz'.

Checksum mismatch!
File: '/home/mike/zynq-next/build/downloads/tslib-1.1.tar.xz' has md5 checksum d41d8cd98f00b204e9800998ecf8427e when 14771f8607b341bb4b297819d37e837d was expected

Now the weird thing is, that when I fetch the thing using wget, I get ablob with the right checksum:

# wget https://github.com/kergoth/tslib/releases/download/1.1/tslib-1.1.tar.xz
# md5sum tslib-1.1.tar.xz
14771f8607b341bb4b297819d37e837d  tslib-1.1.tar.xz
Quoted 8 lines
Quoted 8 lines
Quoted 4 lines
Quoted 2 lines

Is oe-core using some other flavour of wget or what else could cause this kind of fetch error?

Have you tried to unpack both tarballs and compare content in them?

Nope, I just assumed they're two somewhat different copies of the same software (e.g. one of them has a "downloaded from here" remark or so). Anything in particular I should look for?

Any difference is important, sometimes you can find there HTML code from some proxy or server which generates HTML page instead of 404 or someone trying to sell you domain of long dead project etc

Quoted 7 lines

After upgrading I tried again. The file as downloaded by bitbake is completely empty. Nothing in it. The md5 sum of this empty file is d41d8cd98f00b204e9800998ecf8427e indeed.

I'm now using bitbake 1.21.0 (current master) and OE rev 0eb947454e1c92467283e6f1adeca67c7c57698b to build with, with the above results still.

OK, I was asking mostly because newer bitbake (1.19+ is new enough) renames file with bad checksum, moving corrupted download aside so it doesn't stand in way for new one.

Quoted 6 lines

(I know I can just move my own file into the download directory and get on with it, but i'd rather actually solve this problem).

There's a direct connection to the internet, no proxy (just a router) that might have been caching things.

Quoted 7 lines
Quoted 1 line

Any suggestions?

Check log.do_fetch in WORKDIR/temp, I'm not sure if the error shows only the original URL or also possible (PRE)MIRROR url from which it could download that empty file.

You should see the exact URL in log.do_fetch.

The problem seems to be that I am "my own mirror". There's a http server that serves sstate-cache and downloads to the rest of the company. And via an NFS mount, that dir is also linked to my local downloads directory. So bitbake ended up executing:

/usr/bin/env wget -t 2 -T 30 -nv --passive-ftp --no-check-certificate -O /home/mike/zynq-next/build/downloads/tslib-1.1.tar.xz -P /home/mike/zynq-next/build/downloads 'http://192.168.80.24/sources/tslib-1.1.tar.xz'

This would create an empty file "tslib-1.1.tar.xz" and the http server happily returns that empty file, and then bitbake thinks it all went well.

I removed the link to the NFS mount, and now it works.

Quoted 7 lines

Weird that only tslib suffers from this. Then again, it's probably just the only package that wasn't readily available.

Is this my mistake and don't do this again, or should there be some protection against fetching your own files? (for example, by NOT using the filename itself as a target, but download with a ".part" extension and then rename when done. This also helps when multible builds share the download dir).

We certainly don't support circular references like this. As you mention, we can work around some parts with the .part approach but for things like git repositories for example things are much harder to ensure they work.

I'm not even sure how we'd detect this kind of situation to be honest, I'm open to ideas though.

Sharing a "live" DL_DIR is probably a bad idea as files may be incomplete, etc. The checksums do at least help catch most of the problems.

Cheers,

Richard

#6 lore raw
Quoted 93 lines · Richard Purdie
Quoted 79 lines · Mike Looijmans
Quoted 55 lines · Martin Jansa
Quoted 23 lines · Mike Looijmans
Quoted 22 lines · Mike Looijmans
Quoted 21 lines · Martin Jansa
Quoted 20 lines · Mike Looijmans

I get this error every time I try t build the current oe-core master:

ERROR: Checksum failure fetching
https://github.com/kergoth/tslib/releases/download/1.1/tslib-1.1.tar.xz;downloadfilename=tslib-1.1.tar.xz

ERROR: Function failed: Fetcher failure for URL:
'https://github.com/kergoth/tslib/releases/download/1.1/tslib-1.1.tar.xz;downloadfilename=tslib-1.1.tar.xz'.

Checksum mismatch!
File: '/home/mike/zynq-next/build/downloads/tslib-1.1.tar.xz' has md5 checksum d41d8cd98f00b204e9800998ecf8427e when 14771f8607b341bb4b297819d37e837d was expected

Now the weird thing is, that when I fetch the thing using wget, I get ablob with the right checksum:

# wget https://github.com/kergoth/tslib/releases/download/1.1/tslib-1.1.tar.xz
# md5sum tslib-1.1.tar.xz
14771f8607b341bb4b297819d37e837d  tslib-1.1.tar.xz
Quoted 8 lines
Quoted 8 lines
Quoted 4 lines
Quoted 2 lines

Is oe-core using some other flavour of wget or what else could cause this kind of fetch error?

Have you tried to unpack both tarballs and compare content in them?

Nope, I just assumed they're two somewhat different copies of the same software (e.g. one of them has a "downloaded from here" remark or so). Anything in particular I should look for?

Any difference is important, sometimes you can find there HTML code from some proxy or server which generates HTML page instead of 404 or someone trying to sell you domain of long dead project etc

Quoted 7 lines

After upgrading I tried again. The file as downloaded by bitbake is completely empty. Nothing in it. The md5 sum of this empty file is d41d8cd98f00b204e9800998ecf8427e indeed.

I'm now using bitbake 1.21.0 (current master) and OE rev 0eb947454e1c92467283e6f1adeca67c7c57698b to build with, with the above results still.

OK, I was asking mostly because newer bitbake (1.19+ is new enough) renames file with bad checksum, moving corrupted download aside so it doesn't stand in way for new one.

Quoted 6 lines

(I know I can just move my own file into the download directory and get on with it, but i'd rather actually solve this problem).

There's a direct connection to the internet, no proxy (just a router) that might have been caching things.

Quoted 7 lines
Quoted 1 line

Any suggestions?

Check log.do_fetch in WORKDIR/temp, I'm not sure if the error shows only the original URL or also possible (PRE)MIRROR url from which it could download that empty file.

You should see the exact URL in log.do_fetch.

The problem seems to be that I am "my own mirror". There's a http server that serves sstate-cache and downloads to the rest of the company. And via an NFS mount, that dir is also linked to my local downloads directory. So bitbake ended up executing:

/usr/bin/env wget -t 2 -T 30 -nv --passive-ftp --no-check-certificate -O /home/mike/zynq-next/build/downloads/tslib-1.1.tar.xz -P /home/mike/zynq-next/build/downloads 'http://192.168.80.24/sources/tslib-1.1.tar.xz'

This would create an empty file "tslib-1.1.tar.xz" and the http server happily returns that empty file, and then bitbake thinks it all went well.

I removed the link to the NFS mount, and now it works.

Quoted 7 lines

Weird that only tslib suffers from this. Then again, it's probably just the only package that wasn't readily available.

Is this my mistake and don't do this again, or should there be some protection against fetching your own files? (for example, by NOT using the filename itself as a target, but download with a ".part" extension and then rename when done. This also helps when multible builds share the download dir).

We certainly don't support circular references like this. As you mention, we can work around some parts with the .part approach but for things like git repositories for example things are much harder to ensure they work.

Quoted 6 lines

I'm not even sure how we'd detect this kind of situation to be honest, I'm open to ideas though.

Sharing a "live" DL_DIR is probably a bad idea as files may be incomplete, etc. The checksums do at least help catch most of the problems.

Aren't .lock and .done stampfiles supposed to prevent downloading incomplete files when it's shared over NFS between multiple builders?

I haven't tried it in practise yet, but I was planing to convert all our jenkins builders to share common live DL_DIR/SSTATE_DIR, so that first builder to fetch or build something saves the work for other slaves.

I know that especially for sstate archives it could be too late when 2 builders are already in runqueue phase, but still it could be a bit sooner than waiting for whole build to finish and rsync DL_DIR/SSTATE_DIR after the build.

Cheers,

Signature · 1 line
Martin 'JaMa' Jansa     jabber: [email protected]
#7 lore raw
Quoted 66 lines · Martin Jansa
Quoted 65 lines · Richard Purdie
Quoted 48 lines · Mike Looijmans
Quoted 24 lines · Martin Jansa
Quoted 1 line · Mike Looijmans

On 11/20/2013 12:09 PM, Mike Looijmans wrote:

Any difference is important, sometimes you can find there HTML code from some proxy or server which generates HTML page instead of 404 or someone trying to sell you domain of long dead project etc

Quoted 7 lines

After upgrading I tried again. The file as downloaded by bitbake is completely empty. Nothing in it. The md5 sum of this empty file is d41d8cd98f00b204e9800998ecf8427e indeed.

I'm now using bitbake 1.21.0 (current master) and OE rev 0eb947454e1c92467283e6f1adeca67c7c57698b to build with, with the above results still.

OK, I was asking mostly because newer bitbake (1.19+ is new enough) renames file with bad checksum, moving corrupted download aside so it doesn't stand in way for new one.

Quoted 6 lines

(I know I can just move my own file into the download directory and get on with it, but i'd rather actually solve this problem).

There's a direct connection to the internet, no proxy (just a router) that might have been caching things.

Quoted 7 lines
Quoted 1 line

Any suggestions?

Check log.do_fetch in WORKDIR/temp, I'm not sure if the error shows only the original URL or also possible (PRE)MIRROR url from which it could download that empty file.

You should see the exact URL in log.do_fetch.

The problem seems to be that I am "my own mirror". There's a http server that serves sstate-cache and downloads to the rest of the company. And via an NFS mount, that dir is also linked to my local downloads directory. So bitbake ended up executing:

/usr/bin/env wget -t 2 -T 30 -nv --passive-ftp --no-check-certificate -O /home/mike/zynq-next/build/downloads/tslib-1.1.tar.xz -P /home/mike/zynq-next/build/downloads 'http://192.168.80.24/sources/tslib-1.1.tar.xz'

This would create an empty file "tslib-1.1.tar.xz" and the http server happily returns that empty file, and then bitbake thinks it all went well.

I removed the link to the NFS mount, and now it works.

Quoted 7 lines

Weird that only tslib suffers from this. Then again, it's probably just the only package that wasn't readily available.

Is this my mistake and don't do this again, or should there be some protection against fetching your own files? (for example, by NOT using the filename itself as a target, but download with a ".part" extension and then rename when done. This also helps when multible builds share the download dir).

We certainly don't support circular references like this. As you mention, we can work around some parts with the .part approach but for things like git repositories for example things are much harder to ensure they work.

I'm not even sure how we'd detect this kind of situation to be honest, I'm open to ideas though.

Quoted 6 lines
Quoted 3 lines

Sharing a "live" DL_DIR is probably a bad idea as files may be incomplete, etc. The checksums do at least help catch most of the problems.

Aren't .lock and .done stampfiles supposed to prevent downloading incomplete files when it's shared over NFS between multiple builders?

Yes, this was not an NFS case though. This was where DL_DIR was being served out over http from the same directory and that http address was listed in PREMIRRORS/MIRRORS. Locking doesn't work over http :)

Quoted 3 lines

I haven't tried it in practise yet, but I was planing to convert all our jenkins builders to share common live DL_DIR/SSTATE_DIR, so that first builder to fetch or build something saves the work for other slaves.

Right, we try and be safe over NFS so assuming your NFS locking works, this should work fine. We atomically move sstate files into place to ensure that is safe.

Quoted 4 lines

I know that especially for sstate archives it could be too late when 2 builders are already in runqueue phase, but still it could be a bit sooner than waiting for whole build to finish and rsync DL_DIR/SSTATE_DIR after the build.

Agreed and this is how the Yocto Project autobuilders are setup for that reason.

Cheers,

Richard

#8 lore raw
In reply to Mike Looijmans
Quoted 40 lines · Mike Looijmans
Quoted 22 lines · Mike Looijmans
Quoted 21 lines · Martin Jansa
Quoted 20 lines · Mike Looijmans

I get this error every time I try t build the current oe-core master:

ERROR: Checksum failure fetching
https://github.com/kergoth/tslib/releases/download/1.1/tslib-1.1.tar.xz;downloadfilename=tslib-1.1.tar.xz

ERROR: Function failed: Fetcher failure for URL:
'https://github.com/kergoth/tslib/releases/download/1.1/tslib-1.1.tar.xz;downloadfilename=tslib-1.1.tar.xz'.

Checksum mismatch!
File: '/home/mike/zynq-next/build/downloads/tslib-1.1.tar.xz' has md5 checksum d41d8cd98f00b204e9800998ecf8427e when 14771f8607b341bb4b297819d37e837d was expected

Now the weird thing is, that when I fetch the thing using wget, I get ablob with the right checksum:

# wget https://github.com/kergoth/tslib/releases/download/1.1/tslib-1.1.tar.xz
# md5sum tslib-1.1.tar.xz
14771f8607b341bb4b297819d37e837d  tslib-1.1.tar.xz
Quoted 8 lines
Quoted 4 lines
Quoted 2 lines

Is oe-core using some other flavour of wget or what else could cause this kind of fetch error?

Have you tried to unpack both tarballs and compare content in them?

Nope, I just assumed they're two somewhat different copies of the same software (e.g. one of them has a "downloaded from here" remark or so). Anything in particular I should look for?

After upgrading I tried again. The file as downloaded by bitbake is completely empty. Nothing in it. The md5 sum of this empty file is d41d8cd98f00b204e9800998ecf8427e indeed.

I'm now using bitbake 1.21.0 (current master) and OE rev 0eb947454e1c92467283e6f1adeca67c7c57698b to build with, with the above results still.

Quoted 7 lines

(I know I can just move my own file into the download directory and get on with it, but i'd rather actually solve this problem).

There's a direct connection to the internet, no proxy (just a router) that might have been caching things.

Any suggestions?

Can you share the tslib do_fetch logs (all of them). I'm interested to see what happened historically in the build...

Cheers,

Richard