Possible stale tags in the download directory
When I look at the "downloads" location I find a lot of files called.
"11_usagi_fix.patch.done"
but no "11_usagi_fix.patch" file in the directory.
If this is a indication of that a patch has been downloaded, what happens if you for some reason or other delete your OE-core directory and keep the downloads directory.
When you start from fresh, all the tags are then stale.
(Deleting the "downloads" directory seems a bad idea)
I think that if there should be tags in the "downloads" directory, they should only reflect things which are downloaded to the "downloads" directory, and nothing else.
As for the tags in the directory, I think a better approach is to download a file "tarball.tar.bz2" to a different filename first I.E: "tarball.tar.bz2.in-progress" and if the download completes then move the file to "tarball.tar.bz2".
Should remove a lot of clutter from the "downloads" directory.
Signature · 3 lines
Best Regards Ulf Samuelsson eMagii
Quoted 18 lines · Ulf Samuelsson
When I look at the "downloads" location I find a lot of files called.
"11_usagi_fix.patch.done"
but no "11_usagi_fix.patch" file in the directory.
If this is a indication of that a patch has been downloaded, what happens if you for some reason or other delete your OE-core directory and keep the downloads directory.
When you start from fresh, all the tags are then stale.
(Deleting the "downloads" directory seems a bad idea)
I think that if there should be tags in the "downloads" directory, they should only reflect things which are downloaded to the "downloads" directory, and nothing else.
Quoted 5 lines
As for the tags in the directory, I think a better approach is to download a file "tarball.tar.bz2" to a different filename first I.E: "tarball.tar.bz2.in-progress" and if the download completes then move the file to "tarball.tar.bz2".
Should remove a lot of clutter from the "downloads" directory.
What we've tried to do is simplify the fetcher code paths in fetch2. There were some many corner/special cases and different code paths it was near impossible to tell what was going on. One of the side effects is that local file:// urls do touch the done stamp in the same way as other downloads. The main reason for those files is now checksum tracking. If the done stamps were part of tmpdir, we'd keep checksumming the contents of the downloads at each build rather than once after the download. The way the implementation works, there is very little risk from stale stamps, it would just mean the checksum code wouldn't get triggered and that is likely unneeded for a local file:// url anyway.
So yes, its not ideal but looking at the code I'd be surprised if a build ever broke due to it.
Cheers,
Richard
Quoted 35 lines · Richard Purdie
Quoted 18 lines · Ulf Samuelsson
When I look at the "downloads" location I find a lot of files called.
"11_usagi_fix.patch.done"
but no "11_usagi_fix.patch" file in the directory.
If this is a indication of that a patch has been downloaded, what happens if you for some reason or other delete your OE-core directory and keep the downloads directory.
When you start from fresh, all the tags are then stale.
(Deleting the "downloads" directory seems a bad idea)
I think that if there should be tags in the "downloads" directory, they should only reflect things which are downloaded to the "downloads" directory, and nothing else.
Quoted 5 lines
As for the tags in the directory, I think a better approach is to download a file "tarball.tar.bz2" to a different filename first I.E: "tarball.tar.bz2.in-progress" and if the download completes then move the file to "tarball.tar.bz2".
Should remove a lot of clutter from the "downloads" directory.What we've tried to do is simplify the fetcher code paths in fetch2. There were some many corner/special cases and different code paths it was near impossible to tell what was going on. One of the side effects is that local file:// urls do touch the done stamp in the same way as other downloads. The main reason for those files is now checksum tracking. If the done stamps were part of tmpdir, we'd keep checksumming the contents of the downloads at each build rather than once after the download. The way the implementation works, there is very little risk from stale stamps, it would just mean the checksum code wouldn't get triggered and that is likely unneeded for a local file:// url anyway.
Quoted 3 lines
So yes, its not ideal but looking at the code I'd be surprised if a build ever broke due to it.
Maybe I am paranoid, but I think I remember seeing tarballs developing bit rot when moved between different machines.
Could of course clean out the downloads directory, but it would be nice if the stamps would be in a different directory. Then I just delete the directory when I move the tarballs.
I tend to use the "downloads" directory for other build systems as well, to avoid duplication, and having stamps in the directory could have real bad effects.
I agree that recomputing checksums for each build is not so good, but doing it once after moving to a new machine, or for each new user is acceptable (at least to me).
The stamps could be stored in a central location.
"~/.oe/downloads" ?
Con: If you have several repositories, maybe because you forgot to configure for your central repository, you may be in trouble.
Quoted 10 lines
Cheers,
Richard
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
Signature · 3 lines
Best Regards Ulf Samuelsson eMagii
Op 7 dec. 2011, om 12:27 heeft Ulf Samuelsson het volgende geschreven:
Quoted 36 lines · Richard Purdie
Quoted 35 lines · Ulf Samuelsson
Quoted 18 lines
When I look at the "downloads" location I find a lot of files called.
"11_usagi_fix.patch.done"
but no "11_usagi_fix.patch" file in the directory.
If this is a indication of that a patch has been downloaded, what happens if you for some reason or other delete your OE-core directory and keep the downloads directory.
When you start from fresh, all the tags are then stale.
(Deleting the "downloads" directory seems a bad idea)
I think that if there should be tags in the "downloads" directory, they should only reflect things which are downloaded to the "downloads" directory, and nothing else.
Quoted 5 lines
As for the tags in the directory, I think a better approach is to download a file "tarball.tar.bz2" to a different filename first I.E: "tarball.tar.bz2.in-progress" and if the download completes then move the file to "tarball.tar.bz2".
Should remove a lot of clutter from the "downloads" directory.What we've tried to do is simplify the fetcher code paths in fetch2. There were some many corner/special cases and different code paths it was near impossible to tell what was going on. One of the side effects is that local file:// urls do touch the done stamp in the same way as other downloads. The main reason for those files is now checksum tracking. If the done stamps were part of tmpdir, we'd keep checksumming the contents of the downloads at each build rather than once after the download. The way the implementation works, there is very little risk from stale stamps, it would just mean the checksum code wouldn't get triggered and that is likely unneeded for a local file:// url anyway.
Quoted 5 lines
Quoted 3 lines
So yes, its not ideal but looking at the code I'd be surprised if a build ever broke due to it.
Maybe I am paranoid, but I think I remember seeing tarballs developing bit rot when moved between different machines.
And that's why checksums are checked before do_unpack.
Quoted 38 lines · Koen Kooi
Op 7 dec. 2011, om 12:27 heeft Ulf Samuelsson het volgende geschreven:
Quoted 36 lines · Richard Purdie
Quoted 35 lines · Ulf Samuelsson
Quoted 18 lines
When I look at the "downloads" location I find a lot of files called.
"11_usagi_fix.patch.done"
but no "11_usagi_fix.patch" file in the directory.
If this is a indication of that a patch has been downloaded, what happens if you for some reason or other delete your OE-core directory and keep the downloads directory.
When you start from fresh, all the tags are then stale.
(Deleting the "downloads" directory seems a bad idea)
I think that if there should be tags in the "downloads" directory, they should only reflect things which are downloaded to the "downloads" directory, and nothing else.
Quoted 5 lines
As for the tags in the directory, I think a better approach is to download a file "tarball.tar.bz2" to a different filename first I.E: "tarball.tar.bz2.in-progress" and if the download completes then move the file to "tarball.tar.bz2".
Should remove a lot of clutter from the "downloads" directory.What we've tried to do is simplify the fetcher code paths in fetch2. There were some many corner/special cases and different code paths it was near impossible to tell what was going on. One of the side effects is that local file:// urls do touch the done stamp in the same way as other downloads. The main reason for those files is now checksum tracking. If the done stamps were part of tmpdir, we'd keep checksumming the contents of the downloads at each build rather than once after the download. The way the implementation works, there is very little risk from stale stamps, it would just mean the checksum code wouldn't get triggered and that is likely unneeded for a local file:// url anyway.
Quoted 6 lines
Quoted 5 lines
Quoted 3 lines
So yes, its not ideal but looking at the code I'd be surprised if a build ever broke due to it.
Maybe I am paranoid, but I think I remember seeing tarballs developing bit rot when moved between different machines.
And that's why checksums are checked before
do_unpack.
Richard says that if the *.done file is there, then the checksum is not calculated. Or that is at least how I interpret his comment.
If the check is always there, why the tag?
Quoted 5 lines
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
Signature · 3 lines
Best Regards Ulf Samuelsson eMagii
Quoted 5 lines · On Wednesday 07 December 2011 15:30:38 Ulf Samuelsson wrote:
Richard says that if the *.done file is there, then the checksum is not calculated.
Or that is at least how I interpret his comment.If the check is always there, why the tag?
That's not what my reading of the bitbake source says. bitbake/lib/bb/fetch2/__init__.py contains the only references to .done files and all it does is make sure the done file gets "touched" whenever a fetch for the associated file occurs; it does not otherwise check for its existence.
Cheers,
Paul
Signature · 2 lines
Paul Eggleton Intel Open Source Technology Centre
Quoted 11 lines · Paul Eggleton
Quoted 5 lines · On Wednesday 07 December 2011 15:30:38 Ulf Samuelsson wrote:
Richard says that if the *.done file is there, then the checksum is not calculated.
Or that is at least how I interpret his comment.If the check is always there, why the tag?
That's not what my reading of the bitbake source says.
bitbake/lib/bb/fetch2/__init__.pycontains the only references to .done files and all it does is make sure the done file gets "touched" whenever a fetch for the associated file occurs; it does not otherwise check for its existence.
It does two things, one is an activity marker useful for seeing whether files are in active use and the second is for the checksum calculation which only ever happens once after download due to the file (see verify_checksum() in that file).
Cheers,
Richard
Quoted 13 lines · Richard Purdie
Quoted 11 lines · Paul Eggleton
Quoted 5 lines · On Wednesday 07 December 2011 15:30:38 Ulf Samuelsson wrote:
Richard says that if the *.done file is there, then the checksum is not calculated.
Or that is at least how I interpret his comment.If the check is always there, why the tag?
That's not what my reading of the bitbake source says.
bitbake/lib/bb/fetch2/__init__.pycontains the only references to .done files and all it does is make sure the done file gets "touched" whenever a fetch for the associated file occurs; it does not otherwise check for its existence.
Quoted 4 lines
It does two things, one is an activity marker useful for seeing whether files are in active use and the second is for the checksum calculation which only ever happens once after download due to the file (see
verify_checksum()in that file).
when do we create .done files? if I am in middle of download of a tar file and I cancel it and restart it does not fetch the file again but tried to use the corrupt file.
Is it a big build time hit if checksums are verified all the time ?
Signature · 1 line
-Khem
Quoted 16 lines · Khem Raj
Quoted 7 lines · Richard Purdie
Quoted 6 lines · Paul Eggleton
Quoted 5 lines · On Wednesday 07 December 2011 15:30:38 Ulf Samuelsson wrote:
Richard says that if the *.done file is there, then the checksum is not calculated.
Or that is at least how I interpret his comment.If the check is always there, why the tag?
Quoted 8 lines
Quoted 4 lines
That's not what my reading of the bitbake source says.
bitbake/lib/bb/fetch2/__init__.pycontains the only references to .done files and all it does is make sure the done file gets "touched" whenever a fetch for the associated file occurs; it does not otherwise check for its existence.It does two things, one is an activity marker useful for seeing whether files are in active use and the second is for the checksum calculation which only ever happens once after download due to the file (see
verify_checksum()in that file).
Quoted 6 lines
when do we create .done files? if I am in middle of download of a tar file and I cancel it and restart it does not fetch the file again but tried to use the corrupt file.
Is it a big build time hit if checksums are verified all the time ?
I have seen the same problem.
Signature · 3 lines
Best Regards Ulf Samuelsson eMagii
Quoted 36 lines · Ulf Samuelsson
Quoted 35 lines · Richard Purdie
Quoted 18 lines · Ulf Samuelsson
When I look at the "downloads" location I find a lot of files called.
"11_usagi_fix.patch.done"
but no "11_usagi_fix.patch" file in the directory.
If this is a indication of that a patch has been downloaded, what happens if you for some reason or other delete your OE-core directory and keep the downloads directory.
When you start from fresh, all the tags are then stale.
(Deleting the "downloads" directory seems a bad idea)
I think that if there should be tags in the "downloads" directory, they should only reflect things which are downloaded to the "downloads" directory, and nothing else.
Quoted 5 lines
As for the tags in the directory, I think a better approach is to download a file "tarball.tar.bz2" to a different filename first I.E: "tarball.tar.bz2.in-progress" and if the download completes then move the file to "tarball.tar.bz2".
Should remove a lot of clutter from the "downloads" directory.What we've tried to do is simplify the fetcher code paths in fetch2. There were some many corner/special cases and different code paths it was near impossible to tell what was going on. One of the side effects is that local file:// urls do touch the done stamp in the same way as other downloads. The main reason for those files is now checksum tracking. If the done stamps were part of tmpdir, we'd keep checksumming the contents of the downloads at each build rather than once after the download. The way the implementation works, there is very little risk from stale stamps, it would just mean the checksum code wouldn't get triggered and that is likely unneeded for a local file:// url anyway.
Quoted 5 lines
Quoted 3 lines
So yes, its not ideal but looking at the code I'd be surprised if a build ever broke due to it.
Maybe I am paranoid, but I think I remember seeing tarballs developing bit rot when moved between different machines.
I don't understand what you mean here. If your process for copying files between machines corrupts files, you have bigger problems. You can just rm *.done at the same time if that is an issue.
Quoted 8 lines
Could of course clean out the downloads directory, but it would be nice if the stamps would be in a different directory. Then I just delete the directory when I move the tarballs.
I tend to use the "downloads" directory for other build systems as well, to avoid duplication, and having stamps in the directory could have real bad effects.
Quoted 7 lines
I agree that recomputing checksums for each build is not so good, but doing it once after moving to a new machine, or for each new user is acceptable (at least to me).
The stamps could be stored in a central location.
"~/.oe/downloads" ?
I believe the data the .done files represent is directly related to the files themselves and they should be kept together.
Nothing said here is convincing me there is a real world problem...
Cheers,
Richard