git fetcher: do not remove the tree after initial clone
PATCH 1/1 git fetcher: do not remove the tree after initial clone
With the current implementation of the 'git fetcher' the project git tree is cloned for every build (if the commit ID has changed, otherwise the .tar.gz would already exist). This appears to me quite inefficient since rebuilding with a more recent commit is a common use case.
With this patch I am changing how the 'git fetch algorithm' works as follows:
- if project <foobar> uses git, the tree will be cloned in dl/git/foobar
- the tree is cloned using 'git clone -n' to make sure that we can reliably update the tree later (--bare does not create the correct refspec, and --mirror would work but is doing more than necessary)
- the clone is done only the first time, e.g. when dl/git/foobar does not exist
- when making another build, if dl/git/foobar exists, the tree is not cloned, but instead we run 'git fetch --all --tags' to update the existing tree. that can save a lot of time, especially with large trees since only the delta is downloaded. '--all --tags' is required to make sure that all remotes commits are downloaded.
Note1: git fetch is actually called even the first time, right after the
clone, this is obviously not required, but it made the code much simpler in the makefile, and it does not hurt too much
Note2: similiar mechanism might be implemented for other SCM supported by
buildroot
--- package/Makefile.package.in | 10 +++++----- 1 files changed, 5 insertions(+), 5 deletions(-)
package/Makefile.package.in +5 -5
@@ -1,4 +1,4 @@ -################################################################################ +############################################################################### # Generic package infrastructure # # This file implements an infrastructure that eases development of
@@ -105,13 +105,13 @@ endif define DOWNLOAD_GIT test -e $(DL_DIR)/$($(PKG)_SOURCE) || \ - (pushd $(DL_DIR) > /dev/null && \ - $(GIT) clone --bare $($(PKG)_SITE) $($(PKG)_BASE_NAME) && \ - pushd $($(PKG)_BASE_NAME) > /dev/null && \ + (mkdir -p $(DL_DIR)/git && pushd $(DL_DIR)/git > /dev/null && \ + (test -d $($(PKG)_NAME) || $(GIT) clone -n $($(PKG)_SITE) $($(PKG)_NAME) ) && \ + pushd $($(PKG)_NAME) > /dev/null && \ + $(GIT) fetch --tags --all && \ $(GIT) archive --format=tar --prefix=$($(PKG)_BASE_NAME)/ $($(PKG)_DL_VERSION) | \ gzip -c > $(DL_DIR)/$($(PKG)_SOURCE) && \ popd > /dev/null && \ - rm -rf $($(PKG)_DL_DIR) && \ popd > /dev/null) endef
Signature · 1 line
1.7.5.4
PATCH 1/1 git fetcher: do not remove the tree after initial clone
Hi Nicolas,
Quoted 7 lines · Nicolas Dechesne
With the current implementation of the 'git fetcher' the project git tree is cloned for every build (if the commit ID has changed, otherwise the .tar.gz would already exist). This appears to me quite inefficient since rebuilding with a more recent commit is a common use case.
With this patch I am changing how the 'git fetch algorithm' works as follows:
Quoted 11 lines
- if project<foobar> uses git, the tree will be cloned in dl/git/foobar
- the tree is cloned using 'git clone -n' to make sure that we can reliably update the tree later (--bare does not create the correct refspec, and --mirror would work but is doing more than necessary)
- the clone is done only the first time, e.g. when dl/git/foobar does not exist
- when making another build, if dl/git/foobar exists, the tree is not cloned, but instead we run 'git fetch --all --tags' to update the existing tree. that can save a lot of time, especially with large trees since only the delta is downloaded. '--all --tags' is required to make sure that all remotes commits are downloaded.
The git downloader was the subject of a pretty long discussion around July 2010, and two different use cases were conceived with respect to git-hosted packages:
- integrate in BR an externally-developed package that is not released in tarballs, but only in a git repository (use BR for integration);
- integrate in BR a package that is internally under development, so that it can be easily tested inside the entire root fs (use BR for integration and development).
In the first case updates to the BR package would happen rarely, typically once per each new upstream release.
In the second case updates would be frequent, probably many per day, on the developer workstation.
Mine vision at the time was more close to the second case, as I do develop software that I test within buildroot. This is also your idea with this proposed patch.
In fact the implementation I submitted at that time was similar to yours in principle: clone once, fetch the following times.
Instead the choice for mainline BR was to privilege the first use case (use BR for integration).
I was not enthusiast of course, but I found that in my case it was not a big problem anyway since I clone from local servers, which makes clones relatively fast. This would be different if the repositories you want to clone are large or remote (or both!).
Using BR for development has not been completely neglected anyway. Recently, Thomas Petazzoni submitted a patch that would allow to use a local directory to build the package. This is much different from using git fetches, but is expected to serve the "use BR for development" use case effectively.
You might want to test it and comment to the author. Here it is:
http://lists.busybox.net/pipermail/buildroot/2011-July/044479.html
Luca
PATCH 1/1 git fetcher: do not remove the tree after initial clone
Le Mon, 29 Aug 2011 09:48:47 +0200,
Luca Ceresoli <[email protected]> a ?crit :
Quoted 7 lines
Using BR for development has not been completely neglected anyway. Recently, Thomas Petazzoni submitted a patch that would allow to use a local directory to build the package. This is much different from using git fetches, but is expected to serve the "use BR for development" use case effectively.
You might want to test it and comment to the author. Here it is:
http://lists.busybox.net/pipermail/buildroot/2011-July/044479.html
I still think that this local directory mechanism is a better solution.
I have a new version of my patch set, which I hope to submit soon, at least when the 2011.11 development process will open (by the end of the month, so should be in the next few days).
Regards,
Thomas
Signature · 4 lines
Thomas Petazzoni, Free Electrons Kernel, drivers, real-time and embedded Linux development, consulting, training and support. http://free-electrons.com
Dechesne
PATCH 1/1 git fetcher: do not remove the tree after initial clone
Lucas, Thomas,
Quoted 8 lines · Thomas Petazzoni
Quoted 7 lines
Using BR for development has not been completely neglected anyway. Recently, Thomas Petazzoni submitted a patch that would allow to use a local directory to build the package. This is much different from using git fetches, but is expected to serve the "use BR for development" use case effectively.
You might want to test it and comment to the author. Here it is:
http://lists.busybox.net/pipermail/buildroot/2011-July/044479.html
Quoted 6 lines
I still think that this local directory mechanism is a better solution.
I have a new version of my patch set, which I hope to submit soon, at least when the 2011.11 development process will open (by the end of the month, so should be in the next few days).
i like the 'local directory' method as well, really much. but i still believe that the 2 things are complementary..
i happen to use BR in a project using large git tree which are not close to me, so I care about git fetch vs git clone.
on top of that there are many users that regularly make rebuild.
the way i see what i propose is *just* an optimization of how BR will use git, not a new workflow. for those of us who use
BR for development, i fully agree that local method is a great addition.
fwiw, i am coming from OE which such a git clone/fetch optimization is already implemented.
any reason why we can't have both?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20110829/5426ef47/attachment.html>
PATCH 1/1 git fetcher: do not remove the tree after initial clone
Quoted 19 lines · Nicolas
Lucas, Thomas,
On Mon, Aug 29, 2011 at 2:23 PM, Thomas Petazzoni <[email protected] <mailto:[email protected]>> wrote:
> Using BR for development has not been completely neglected anyway. > Recently, Thomas Petazzoni submitted a patch that would allow to
use a
> local directory to build the package. This is much different from
using
> git fetches, but is expected to serve the "use BR for
development" use
> case effectively.
> You might want to test it and comment to the author. Here it is:
> http://lists.busybox.net/pipermail/buildroot/2011-July/044479.htmlI still think that this local directory mechanism is a better solution.
Quoted 7 lines
I have a new version of my patch set, which I hope to submit soon, at least when the 2011.11 development process will open (by the end of the month, so should be in the next few days).
i like the 'local directory' method as well, really much. but i still believe that the 2 things are complementary..
Thomas' local directory mechanism has an advantage: it works also for non-git (and even non-versioned) packages.
You could somehow use this method for your needs too: git clone in a local directory once, then git pull before each BR rebuild. Not as straightforward as you would like, but maybe a good compromise.
Quoted 7 lines
i happen to use BR in a project using large git tree which are not close to me, so I care about git fetch vs git clone.
on top of that there are many users that regularly make rebuild.the way i see what i propose is *just* an optimization of how BR will use git, not a new workflow.
Still it has the disadvantage of using a lot of disk space, and it needs to connect to the server at every build only to discover if there are new commits.
This would hurt people not having your needs.
If you would add an option to choose between the always-clone git downloader and the clone-once-fetch-many version, your work would become more interesting.
But this is only my opinion.
Luca
Dechesne
PATCH 1/1 git fetcher: do not remove the tree after initial clone
Quoted 9 lines · Luca Ceresoli
I have a new version of my patch set, which I hope to submit soon, at
Quoted 7 lines
least when the 2011.11 development process will open (by the end of the month, so should be in the next few days).
i like the 'local directory' method as well, really much. but i still believe that the 2 things are complementary..
Quoted 6 lines
Thomas' local directory mechanism has an advantage: it works also for non-git (and even non-versioned) packages.
You could somehow use this method for your needs too: git clone in a local directory once, then git pull before each BR rebuild. Not as straightforward as you would like, but maybe a good compromise.
ok. i am not planning to use this feature to improve the 'I am a developer' situation.
in this case the local method is definetly better.
my use case is:
- i am using BR for a large project and all our programs are hosted on many (internal) git servers
and these servers are far away (multisite company)
- many developers would often need to build a specific release (without making changes in the code), just
fetch new/updated recipes and rebuild
Quoted 7 lines
Quoted 7 lines
i happen to use BR in a project using large git tree which are not close to me, so I care about git fetch vs git clone.
on top of that there are many users that regularly make rebuild.
Quoted 8 lines
Quoted 3 lines
the way i see what i propose is *just* an optimization of how BR will use git, not a new workflow.
Still it has the disadvantage of using a lot of disk space, and it needs to connect to the server at every build only to discover if there are new commits.
agreed on disk space. since we don't remove the clone, we have to store them.
but on the 2nd point it would connect to server *only* if the recipe has changed and uses
a new commit. otherwise it wouldn't connect again since after a successful build the tarball would be
there already (and the extracted source code).
i could make one optimization. run git fetch only if the commit object I am looking for is not
there already.
Quoted 9 lines
This would hurt people not having your needs.
If you would add an option to choose between the always-clone git downloader and the clone-once-fetch-many version, your work would become more interesting.
But this is only my opinion.
is that true? i could make the update. i would just need to create two
variants of DOWNLOAD_GIT and
choose based on the config option...
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20110830/22d0a829/attachment.html>