Per-package download folders and Git caching
Hello,
As was noted in Buildroot Hackathon day 3 highlights e-mail, the patch series changing the download infrastructure to support Git caching was merged. This brings a number of changes visible to the Buildroot user that are worth explaining.
Per-package download folders
============================
The first visible change is that your DL_DIR will no longer have all files stored flat in DL_DIR, but organized into sub-folders, one per package:
+ dl/
+ linux/
+ linux-v4.16.tar.bz2
+ busybox/
+ busybox-1.28.1.tar.bz2
+ busybox-1.28.2.tar.bz2
The benefit of such a new organization is that if two packages need to download a file with the same name, they won't conflict anymore.
So today, if you use the latest Buildroot master and start with an empty DL_DIR, everything will be stored in sub-folders, one per package.
However, it is likely that many users already have a DL_DIR with a number of files. What the new Buildroot will do is that it will first check if the requested file is in the per-package sub-folder. If it is not found there, it will then try to see if it is in the main DL_DIR, and if that's the case, it will create a hard link of this file into the per-package sub-folder. Therefore, you will see something like:
+ dl/
+ linux-v4.16.tar.bz2
+ linux/
+ linux-v4.16.tar.bz2
But the Linux tarball is not duplicated, it is only hard-linked.
This organization with per-package subfolders is also used when getting files from the BR2_PRIMARY_SITE or the BR2_BACKUP_SITE (sources.buildroot.net). We could therefore summarize the search logic of Buildroot as follows:
- Try to find the file in DL_DIR/<package>/. If found, we're good.
- If not found, try to find the file in DL_DIR/, and if found, create a hard link into DL_DIR/<package>/, and then we're good.
- If not found, go to BR2_PRIMARY_SITE/<package>/. If found, store the file in DL_DIR/<package>/, and we're good.
- If not found, go to BR2_PRIMARY_SITE/. If found, store the file in DL_DIR/<package>/, and we're good.
- If not found, go to the upstream location. If found, store the file in DL_DIR/<package>/ and we're good.
- If not found, go to BR2_BACKUP_SITE/<package>/. If found, store the file in DL_DIR/<package>/, and we're good.
- If not found, go to BR2_BACKUP_SITE/. If found, store the file in DL_DIR/<package>/ and we're good.
- If not found, well, bail out with an error.
It is worth mentioning that a few packages share the same source code (linux and linux-headers, gcc-initial and gcc-final, mesa3d and mesa3d-headers), and even if we have per-package download folders, we wanted to avoid downloading the tarballs for such packages twice, especially considering that gcc and linux are quite large. To solve this those packages have a special <pkg>_DL_SUBDIR variable, which they use to override the name of the download sub-folder. Hence, the tarballs for the gcc-initial and gcc-final packages are stored in DL_DIR/gcc/, because both of those packages define <pkg>_DL_SUBDIR = gcc.
Git caching
===========
Git caching is the primary and original motivation for this patch series. Before getting into this, let's summarize what Buildroot was doing to download source code from Git:
1. Clone the Git repository. Buildroot tries hard to download only what's needed using a "shallow clone" but depending on the type of reference (tag or full SHA1) and other parameters, a "shallow clone" is not always possible, in which case Buildroot fell back to a regular full clone.
2. Create a tarball out of the source code that has been checked out, without the .git/ metadata. This tarball is stored in DL_DIR/
3. The Git clone is completely removed.
The obvious drawback is that each time you bump the version of a package fetched from Git, you have to do a full clone of the upstream Git repository to re-create the new tarball. This is slow and inefficient, especially for large projects like the Linux kernel.
Therefore, what Buildroot does now is that it keeps in DL_DIR/<package>/git/ a Git clone. This Git clone is re-used whenever a new download from Git of the same package is done.
Let's say you start with an empty DL_DIR/linux/git/, and you start a build with a Linux kernel fetched from the official Linux Git repo from Linus Torvalds. Buildroot will clone it into DL_DIR/linux/git/, and use that to create a tarball in DL_DIR/linux/ containing the Linux source code at the version you specified. The next day, you build a different Buildroot configuration that uses a Linux kernel fetched from the RaspberryPi Github repository. Buildroot will see it already has a Git clone in DL_DIR/linux/git/, and it will simply fetch the missing Git objects from the RaspberryPi Github repository, and then generate the tarball.
This obviously increases the disk space being used, since we keep a clone instead of removing it after the download, but it greatly reduces the download time. If you think that your DL_DIR has grown too large, you can just remove whatever you want from there, including specifically the DL_DIR/<package>/git/. Buildroot will re-clone as needed.
It is worth mentioning that we continue to generate a tarball stored in DL_DIR/<package>/. One might think that we could just copy/rsync the code from DL_DIR/<package>/git/ into the package build directory, instead of creating a tarball, and re-extracting it later in the package build directory. However, there are a number of reasons why a tarball is still needed:
- Tarball is the mechanism we use to interact with the BR2_PRIMARY_SITE and BR2_BACKUP_SITE. BR2_PRIMARY_SITE and BR2_BACKUP_SITE don't backup a Git clone of the upstream project, but a tarball. Therefore, continuing to use tarballs make sense to keep this logic unchanged.
- Tarball is the unit on which we do the hash file verification. We certainly don't want to store a hash of all the source files of a project in the package .hash file.
Even if the SHA1 of a Git commit normally guarantees that we're really fetching what we think, and therefore would remove the need for a hash file, we still have the case of the BR2_PRIMARY_SITE and BR2_BACKUP_SITE that store tarballs, and we want to be sure those tarballs haven't been modified (maliciously or not).
- Tarballs are needed for legal-info. We could generate them only for legal-info, but because of the two other reasons above, we continue to generate tarballs from Git repositories.
Concurrency handling
====================
Until now, all downloads were made into a temporary folder, and only when the final tarball was ready it was moved to DL_DIR/. Thanks to the atomic property of "mv", this allowed multiple parallel Buildroot builds to access and populate a common DL_DIR/ without any problem.
However, with Git caching in place, the folder DL_DIR/<package>/git/ becomes shared, and can be used by multiple parallel Buildroot builds. This required some locking, with is achieved by using "flock". All download operations are now protected by a lock taken on the per-package download folder, i.e DL_DIR/<package>/.
This means that parallel Buildroot builds can continue to download files in parallel as long as they download files for different packages. If they download files for the same package at the same time, the lock will ensure that those operations will be serialized. Typically this means that if you start two Buildroot builds, and one starts cloning the Linux kernel, the other Buildroot build will be blocked if it tries to download the Linux kernel (even from HTTP), until the other Buildroot instance has finished cloning the Linux kernel. You will simply see the message ">>> linux 1.2.3 downloading" and nothing happening, until the lock gets released.
For now, the lock is taken for all download methods, even if some methods such as HTTP, do not require a lock. This is something that might be improved in the future (patches welcome!).
Conclusion
==========
Hopefully this e-mail has been useful to people who wanted to understand better the changes that we have brought to the download infrastructure recently. All credits for this work obviously go to Yann E. Morin, Maxime Hadjinlian, and Peter Seiderer.
Best regards,
Thomas
Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin (formerly Free Electrons) Embedded Linux and Kernel engineering https://bootlin.com
Hi Thomas,
Excellent write-up!
Quoted 7 lines · Thomas Petazzoni
However, it is likely that many users already have a
DL_DIRwith a number of files. What the new Buildroot will do is that it will first check if the requested file is in the per-package sub-folder. If it is not found there, it will then try to see if it is in the mainDL_DIR, and if that's the case, it will create a hard link of this file into the per-package sub-folder. Therefore, you will see something like:
Quoted 6 lines
+ dl/
+ linux-v4.16.tar.bz2
+ linux/
+ linux-v4.16.tar.bz2But the Linux tarball is not duplicated, it is only hard-linked.
I know a few users that put their download directory on a samba share or a
virtualbox/vmware share which has NTFS as underlying filesystem. Although NTFS does support hardlinks, I'm not sure that Samba and virtualbox/vmware do. Any idea what will happen in such a case?
Regards,
Arnout
Signature · 6 lines
Arnout Vandecappelle arnout at mind be Senior Embedded Software Architect +32-16-286500 Essensium/Mind http://www.mind.be G.Geenslaan 9, 3001 Leuven, Belgium BE 872 984 063 RPR Leuven LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle GPG fingerprint: 7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF
Quoted 4 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Arnout" == Arnout Vandecappelle <[email protected]> writes:
Hi Thomas,
Excellent write-up!
Yes, thanks!
Quoted 4 lines
I know a few users that put their download directory on a samba share or a
virtualbox/vmware share which has NTFS as underlying filesystem. Although NTFS does support hardlinks, I'm not sure that Samba and virtualbox/vmware do. Any idea what will happen in such a case?
Then presumably the ln call will fail and we download the tarball again in the new location?
Signature · 1 line
Bye, Peter Korsgaard
Peter, Arnout, All,
On 2018-04-03 15:17 +0200, Peter Korsgaard spake thusly:
Quoted 8 lines
Quoted 5 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Arnout" == Arnout Vandecappelle <[email protected]> writes:
I know a few users that put their download directory on a samba share or a
virtualbox/vmware share which has NTFS as underlying filesystem. Although NTFS does support hardlinks, I'm not sure that Samba and virtualbox/vmware do. Any idea what will happen in such a case?Then presumably the ln call will fail and we download the tarball again in the new location?
No, the script would fail, because the script runs with 'set -e'.
Should we do a cp if ln fails, then?
Regards,
Yann E. MORIN.
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------'
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Yann" == Yann E MORIN <[email protected]> writes:
Hi,
Quoted 9 lines
Quoted 7 lines
Quoted 4 lines
I know a few users that put their download directory on a samba share or a
virtualbox/vmware share which has NTFS as underlying filesystem. Although NTFS does support hardlinks, I'm not sure that Samba and virtualbox/vmware do. Any idea what will happen in such a case?Then presumably the ln call will fail and we download the tarball again in the new location?
No, the script would fail, because the script runs with 'set -e'.
Ahh yes :/
Quoted 1 line
Should we do a cp if ln fails, then?
That sounds like a sensible fallback, yes.
Signature · 1 line
Bye, Peter Korsgaard
Peter, All,
On 2018-04-03 18:49 +0200, Peter Korsgaard spake thusly:
Quoted 3 lines
Quoted 2 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Yann" == Yann E MORIN <[email protected]> writes:
Should we do a cp if ln fails, then?
That sounds like a sensible fallback, yes.
I was about to do so, but then we're back to a situation where we have to guarantee the atomicity.
This is currently possible, because the dl-wrapper now runs under flock. But we are going to optimise the locking with a more fine-grained solution soonish.
To be noted: the current code is not atomic-safe either, because we do have a TOCTTOU situation...
So, we'll have to again play tricks with atomicity...
We can't use flock again, because the destination directory is already locked, and there is not destination file to lock to start with...
So, maybe the full solution will in fact to really implement the more fine-grained solution... I'll see to it.
Regards,
Yann E. MORIN.
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------'
Quoted 6 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Yann" == Yann E MORIN <[email protected]> writes:
Peter, All,
On 2018-04-03 18:49 +0200, Peter Korsgaard spake thusly:Quoted 2 lines
Quoted 2 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Yann" == Yann E MORIN <[email protected]> writes:
Should we do a cp if ln fails, then?
Quoted 8 lines
Quoted 1 line
That sounds like a sensible fallback, yes.
I was about to do so, but then we're back to a situation where we have to guarantee the atomicity.
This is currently possible, because the dl-wrapper now runs under flock. But we are going to optimise the locking with a more fine-grained solution soonish.
Presumably we could copy to a temporary file and then rename like the dl-wrapper already does?
Signature · 1 line
Bye, Peter Korsgaard
Quoted 17 lines · Yann E. MORIN
Peter, All,
On 2018-04-03 18:49 +0200, Peter Korsgaard spake thusly:
Quoted 3 lines
Quoted 2 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Yann" == Yann E MORIN <[email protected]> writes:
Should we do a cp if ln fails, then?
That sounds like a sensible fallback, yes.
I was about to do so, but then we're back to a situation where we have to guarantee the atomicity.
This is currently possible, because the dl-wrapper now runs under flock. But we are going to optimise the locking with a more fine-grained solution soonish.
To be noted: the current code is not atomic-safe either, because we do have a TOCTTOU situation...
Quoted 7 lines
So, we'll have to again play tricks with atomicity...
We can't use flock again, because the destination directory is already locked, and there is not destination file to lock to start with...
So, maybe the full solution will in fact to really implement the more fine-grained solution... I'll see to it.
Note that for the specific case I mentioned (hardlinks may not work over samba
or VM shares), flock will fail as well...
Maybe we should just wait until someone complains so they can test whatever
solution we come up with.
Regards,
Arnout
Signature · 6 lines
Arnout Vandecappelle arnout at mind be Senior Embedded Software Architect +32-16-286500 Essensium/Mind http://www.mind.be G.Geenslaan 9, 3001 Leuven, Belgium BE 872 984 063 RPR Leuven LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle GPG fingerprint: 7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF
Arnout, All,
On 2018-04-03 22:44 +0200, Arnout Vandecappelle spake thusly:
Quoted 20 lines · Yann E. MORIN
Quoted 19 lines
Peter, All,
On 2018-04-03 18:49 +0200, Peter Korsgaard spake thusly:
Quoted 3 lines
Quoted 2 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Yann" == Yann E MORIN <[email protected]> writes:
Should we do a cp if ln fails, then?
That sounds like a sensible fallback, yes.
I was about to do so, but then we're back to a situation where we have to guarantee the atomicity.
This is currently possible, because the dl-wrapper now runs under flock. But we are going to optimise the locking with a more fine-grained solution soonish.
To be noted: the current code is not atomic-safe either, because we do have a TOCTTOU situation...
So, we'll have to again play tricks with atomicity...
Quoted 8 lines
Quoted 5 lines
We can't use flock again, because the destination directory is already locked, and there is not destination file to lock to start with...
So, maybe the full solution will in fact to really implement the more fine-grained solution... I'll see to it.
Note that for the specific case I mentioned (hardlinks may not work over samba
or VM shares), flock will fail as well...
Wait... So you're saying that we borked a working situation for a certain class of users, now? Damn... :-/
And why would flock fail on those (curious)?
Quoted 2 lines
Maybe we should just wait until someone complains so they can test whatever
solution we come up with.
In the end, yes, probably...
Regards,
Yann E. MORIN.
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------'
Quoted 29 lines · Yann E. MORIN
Arnout, All,
On 2018-04-03 22:44 +0200, Arnout Vandecappelle spake thusly:
Quoted 26 lines · Yann E. MORIN
Quoted 24 lines
Peter, All,
On 2018-04-03 18:49 +0200, Peter Korsgaard spake thusly:
Quoted 3 lines
Quoted 2 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Yann" == Yann E MORIN <[email protected]> writes:
Should we do a cp if ln fails, then?
That sounds like a sensible fallback, yes.
I was about to do so, but then we're back to a situation where we have to guarantee the atomicity.
This is currently possible, because the dl-wrapper now runs under flock. But we are going to optimise the locking with a more fine-grained solution soonish.
To be noted: the current code is not atomic-safe either, because we do have a TOCTTOU situation...
So, we'll have to again play tricks with atomicity...
We can't use flock again, because the destination directory is already locked, and there is not destination file to lock to start with...
So, maybe the full solution will in fact to really implement the more fine-grained solution... I'll see to it.
Quoted 7 lines
Quoted 2 lines
Note that for the specific case I mentioned (hardlinks may not work over samba
or VM shares), flock will fail as well...Wait... So you're saying that we borked a working situation for a certain class of users, now? Damn... :-/
And why would flock fail on those (curious)?
As I remember, flock(2) doesn't work on Samba (or NFS for that matter). I
couldn't find any real reference for that, however, just some unanswered SO question [1]. And since flock(1) uses flock(2)...
That said, Samba _does_ support fcntl(2), so I don't see why it wouldn't
support flock(2)...
Regards,
Arnout
Quoted 8 lines
Quoted 2 lines
Maybe we should just wait until someone complains so they can test whatever
solution we come up with.In the end, yes, probably...
Regards,
Yann E. MORIN.
Signature · 6 lines
Arnout Vandecappelle arnout at mind be Senior Embedded Software Architect +32-16-286500 Essensium/Mind http://www.mind.be G.Geenslaan 9, 3001 Leuven, Belgium BE 872 984 063 RPR Leuven LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle GPG fingerprint: 7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF