sharing DL_DIR?
Is it reasonable to expect to share DL_DIR between multiple builds?
That is, are downloads properly locked so that multiple concurrent downloads of the same file won't collide? And if so, are they NFS safe locks?
Or must each build download it's own copies of every component?
I ask because our, (Palm), branch has been, but it's a bit of a nuisance to pick a suitable locking mechanism that is both functional and performant.
* Flock/lockf aren't reliably supported in heterogeneous environments.
* The old lockfiles don't work over NFS. Lock directories apparently do, but trying to clean these up after interruptions or failures is a losing battle so if we do use these, we end up with orphan locks periodically.
* With NFS as a possibility, we can't assume any kernel local IPC mechanism, so sysV ipc is out.
* It seems like a pretty huge overhead to try to create any sort of zeroconf overhead.
Our solution so far has been to use NFS lock directories for multiple machine builds and flock/lockf for single machine builds. This mostly works, though it's not ideal, and it's significantly faster than downloading/copying all of the component source multiple times even over local mirrors.
--rich
I have a single downloads directory with symlinks for every target build I have to that directory. As long as I have the source already in there I don't fetch it again
Tom
Quoted 34 lines · Rich Pixley
Is it reasonable to expect to share
DL_DIRbetween multiple builds?That is, are downloads properly locked so that multiple concurrent downloads of the same file won't collide? And if so, are they NFS safe locks?
Or must each build download it's own copies of every component?
I ask because our, (Palm), branch has been, but it's a bit of a nuisance to pick a suitable locking mechanism that is both functional and performant.
- Flock/lockf aren't reliably supported in heterogeneous environments.
- The old lockfiles don't work over NFS. Lock directories apparently do, but trying to clean these up after interruptions or failures is a losing battle so if we do use these, we end up with orphan locks periodically.
- With NFS as a possibility, we can't assume any kernel local IPC mechanism, so sysV ipc is out.
- It seems like a pretty huge overhead to try to create any sort of zeroconf overhead.Our solution so far has been to use NFS lock directories for multiple machine builds and flock/lockf for single machine builds. This mostly works, though it's not ideal, and it's significantly faster than downloading/copying all of the component source multiple times even over local mirrors.
--rich_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
Quoted 2 lines · Rich Pixley
Is it reasonable to expect to share
DL_DIRbetween multiple builds?
Absolutely, We share DL_DIR via NFS across multiple machines and builds.
Quoted 4 lines
That is, are downloads properly locked so that multiple concurrent downloads of the same file won't collide? And if so, are they NFS safe locks?
Yes, as far as I know.
Quoted 2 lines
Or must each build download it's own copies of every component?
No need.
Quoted 13 lines
I ask because our, (Palm), branch has been, but it's a bit of a nuisance to pick a suitable locking mechanism that is both functional and performant.
* Flock/lockf aren't reliably supported in heterogeneous environments.
* The old lockfiles don't work over NFS. Lock directories apparently do, but trying to clean these up after interruptions or failures is a losing battle so if we do use these, we end up with orphan locks periodically.
* With NFS as a possibility, we can't assume any kernel local IPC mechanism, so sysV ipc is out.
* It seems like a pretty huge overhead to try to create any sort of zeroconf overhead.
Quoted 6 lines
Our solution so far has been to use NFS lock directories for multiple machine builds and flock/lockf for single machine builds. This mostly works, though it's not ideal, and it's significantly faster than downloading/copying all of the component source multiple times even over local mirrors.
I think we are using a very basic locking mechism that creates the lockfile and a donefile when complete. It seems to work well enough.
Sau!
Quoted 7 lines
--rich
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
I share it as well. I don't see locks being a big issue for most things, as the worst case is a re-download, not use of a corrupt or incomplete file, due to the completion stamps.
Signature · 1 line
Christopher Larson
Quoted 32 lines · Tom King
I have a single downloads directory with symlinks for every target build I have to that directory. As long as I have the source already in there I don't fetch it again
Tom
Quoted 12 lines · Rich Pixley
Is it reasonable to expect to share
DL_DIRbetween multiple builds?That is, are downloads properly locked so that multiple concurrent downloads of the same file won't collide? And if so, are they NFS safe locks?
Or must each build download it's own copies of every component?
I ask because our, (Palm), branch has been, but it's a bit of a nuisance to pick a suitable locking mechanism that is both functional and performant.
Flock/lockf aren't reliably supported in heterogeneous environments.
The old lockfiles don't work over NFS. Lock directories apparently do, but trying to clean these up after interruptions or failures is a losing battle so if we do use these, we end up with orphan locks periodically. With NFS as a possibility, we can't assume any kernel local IPC mechanism, so sysV ipc is out.
It seems like a pretty huge overhead to try to create any sort of zeroconf overhead.Quoted 8 lines
Our solution so far has been to use NFS lock directories for multiple machine builds and flock/lockf for single machine builds. This mostly works, though it's not ideal, and it's significantly faster than downloading/copying all of the component source multiple times even over local mirrors. --rich
_______________________________________________
Openembedded-core mailing list
[email protected] (mailto:[email protected])
http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core_______________________________________________
Openembedded-core mailing list
[email protected] (mailto:[email protected])
http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
What happens if two separate bitbake invocations both simultaneously attempt to download the same file?
--rich
Quoted 45 lines · Christopher Larson
I share it as well. I don't see locks being a big issue for most things, as the worst case is a re-download, not use of a corrupt or incomplete file, due to the completion stamps.
Signature · 1 line
Christopher LarsonQuoted 37 lines · Tom King
I have a single downloads directory with symlinks for every target build I have to that directory. As long as I have the source already in there I don't fetch it again
Tom
Quoted 18 lines · On Wed, Feb 22, 2012 at 1:33 PM, Rich Pixley <[email protected]…
Is it reasonable to expect to share
DL_DIRbetween multiple builds?That is, are downloads properly locked so that multiple concurrent downloads of the same file won't collide? And if so, are they NFS safe locks?
Or must each build download it's own copies of every component?
I ask because our, (Palm), branch has been, but it's a bit of a nuisance to pick a suitable locking mechanism that is both functional and performant.
* Flock/lockf aren't reliably supported in heterogeneous environments.
* The old lockfiles don't work over NFS. Lock directories apparently do, but trying to clean these up after interruptions or failures is a losing battle so if we do use these, we end up with orphan locks periodically.
* With NFS as a possibility, we can't assume any kernel local IPC mechanism, so sysV ipc is out.
* It seems like a pretty huge overhead to try to create any sort of zeroconf overhead.Quoted 9 lines
Our solution so far has been to use NFS lock directories for multiple machine builds and flock/lockf for single machine builds. This mostly works, though it's not ideal, and it's significantly faster than downloading/copying all of the component source multiple times even over local mirrors.
--rich
_______________________________________________
Openembedded-core mailing list
[email protected] <mailto:[email protected]>
http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core_______________________________________________
Openembedded-core mailing list
[email protected] <mailto:[email protected]>
http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
Excellent, thanks. That's the news I was hoping for.
--rich
Quoted 46 lines · Saul Wold
Quoted 2 lines · Rich Pixley
Is it reasonable to expect to share
DL_DIRbetween multiple builds?Absolutely, We share
DL_DIRvia NFS across multiple machines and builds.Quoted 4 lines
That is, are downloads properly locked so that multiple concurrent downloads of the same file won't collide? And if so, are they NFS safe locks?
Yes, as far as I know.
Quoted 2 lines
Or must each build download it's own copies of every component?
No need.
Quoted 13 lines
I ask because our, (Palm), branch has been, but it's a bit of a nuisance to pick a suitable locking mechanism that is both functional and performant.
* Flock/lockf aren't reliably supported in heterogeneous environments.
* The old lockfiles don't work over NFS. Lock directories apparently do, but trying to clean these up after interruptions or failures is a losing battle so if we do use these, we end up with orphan locks periodically.
* With NFS as a possibility, we can't assume any kernel local IPC mechanism, so sysV ipc is out.
* It seems like a pretty huge overhead to try to create any sort of zeroconf overhead.Quoted 6 lines
Our solution so far has been to use NFS lock directories for multiple machine builds and flock/lockf for single machine builds. This mostly works, though it's not ideal, and it's significantly faster than downloading/copying all of the component source multiple times even over local mirrors.
I think we are using a very basic locking mechism that creates the lockfile and a donefile when complete. It seems to work well enough.
Sau!
Quoted 7 lines
--rich
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
Quoted 2 lines · Rich Pixley
What happens if two separate bitbake invocations both simultaneously attempt to download the same file?
The first will download it, the second will wait for it to finish, then both will continue with the downloaded file.
Cheers,
Richard
Quoted 5 lines · Richard Purdie
Quoted 2 lines · Rich Pixley
What happens if two separate bitbake invocations both simultaneously attempt to download the same file?
The first will download it, the second will wait for it to finish, then both will continue with the downloaded file.
How does the second know that the first is in progress?
--rich
Quoted 7 lines · Rich Pixley
Quoted 5 lines · Richard Purdie
Quoted 2 lines · Rich Pixley
What happens if two separate bitbake invocations both simultaneously attempt to download the same file?
The first will download it, the second will wait for it to finish, then both will continue with the downloaded file.
How does the second know that the first is in progress?
It won't be able to take the lock and the done stamp file doesn't exist yet.
Cheers,
Richard
Quoted 10 lines · Richard Purdie
Quoted 7 lines · Rich Pixley
Quoted 5 lines · Richard Purdie
Quoted 2 lines · Rich Pixley
What happens if two separate bitbake invocations both simultaneously attempt to download the same file?
The first will download it, the second will wait for it to finish, then both will continue with the downloaded file.
How does the second know that the first is in progress?
It won't be able to take the lock and the done stamp file doesn't exist yet.
What lock?
I've just read through lib/bb/fetch2/__init__.py but I'm not seeing a lock. What am I missing?
--rich
Quoted 8 lines · Rich Pixley
Quoted 7 lines · Richard Purdie
Quoted 6 lines · Rich Pixley
Quoted 5 lines · Richard Purdie
Quoted 2 lines · Rich Pixley
What happens if two separate bitbake invocations both simultaneously attempt to download the same file?
The first will download it, the second will wait for it to finish, then both will continue with the downloaded file.
Quoted 7 lines
Quoted 3 lines
Quoted 1 line
How does the second know that the first is in progress?
It won't be able to take the lock and the done stamp file doesn't exist yet.
What lock?
I've just read through
lib/bb/fetch2/__init__.pybut I'm not seeing a lock. What am I missing?
Try searching that file for expressions like "lock" and "lockfile" ;-)
Cheers,
Richard