Improved git download method performance.

PATCH New Theodore Witkamp 4 participants 4 messages Jan 6, 2012 – Jan 15, 2012

Share & tools

Theodore Witkamp

author

2012-01-06 22:56 UTC #0 lore raw

PATCH 1/1 Improved git download method performance.

Instead of cloning the a complete repository, git archive is used first. If git archive fails because the remote host does not support it, git clone is used with with --depth 1 so only minimal history is copied.

Also the $($(PKG)_SOURCE) default is changed to .tar instead of .tar.gz. This removes unneeded compression and decompression.

Relative file paths are now relative to $(TOPDIR) not $(DL_DIR).

---
 package/Makefile.package.in |   35 ++++++++++++++++++++---------------
 1 files changed, 20 insertions(+), 15 deletions(-)

package/Makefile.package.in +20 ~~-15~~

index e85eb15..14f1ba1 100644
@@ -146,14 +146,14 @@ domainseparator=$(if $(1),$(1),/)
 
 define DOWNLOAD_GIT
 	test -e $(DL_DIR)/$($(PKG)_SOURCE) || \
-	(pushd $(DL_DIR) > /dev/null && \
-	$(GIT) clone --bare $($(PKG)_SITE) $($(PKG)_BASE_NAME) && \
-	pushd $($(PKG)_BASE_NAME) > /dev/null && \
-	$(GIT) archive --format=tar --prefix=$($(PKG)_BASE_NAME)/ $($(PKG)_DL_VERSION) | \
-		gzip -c > $(DL_DIR)/$($(PKG)_SOURCE) && \
-	popd > /dev/null && \
-	rm -rf $($(PKG)_DL_DIR) && \
-	popd > /dev/null)
+	$(GIT) archive --format=tar --remote $($(PKG)_SITE) --prefix=$($(PKG)_BASE_NAME)/ $($(PKG)_DL_VERSION) \
+	> $(DL_DIR)/$($(PKG)_SOURCE) || \
+	( \
+	$(GIT) clone   -v --depth 1 --bare $($(PKG)_SITE) $($(PKG)_DL_DIR)   &&  \
+	$(GIT) archive --format=tar --remote $($(PKG)_DL_DIR) --prefix=$($(PKG)_BASE_NAME)/ $($(PKG)_DL_VERSION) \
+	> $(DL_DIR)/$($(PKG)_SOURCE) && \
+	rm -rf $($(PKG)_DL_DIR)  \
+	)
 endef
 
 # TODO: improve to check that the given PKG_DL_VERSION exists on the remote

@@ -504,13 +504,6 @@ ifneq ($$($(2)_OVERRIDE_SRCDIR),)
 $(2)_VERSION = custom
 endif
 
-ifndef $(2)_SOURCE
- ifdef $(3)_SOURCE
-  $(2)_SOURCE = $($(3)_SOURCE)
- else
-  $(2)_SOURCE			?= $$($(2)_BASE_NAME).tar.gz
- endif
-endif
 
 ifndef $(2)_PATCH
  ifdef $(3)_PATCH

@@ -536,6 +529,18 @@ ifndef $(2)_SITE_METHOD
  endif
 endif
 
+ifndef $(2)_SOURCE
+ ifdef $(3)_SOURCE
+  $(2)_SOURCE = $($(3)_SOURCE)
+ else
+  ifeq ($$($(2)_SITE_METHOD),git)
+   $(2)_SOURCE			?= $$($(2)_BASE_NAME).tar
+  else
+   $(2)_SOURCE			?= $$($(2)_BASE_NAME).tar.gz
+  endif
+ endif
+endif
+
 ifeq ($$($(2)_SITE_METHOD),local)
 ifeq ($$($(2)_OVERRIDE_SRCDIR),)
 $(2)_OVERRIDE_SRCDIR = $($(2)_SITE)

Signature · 1 line

1.7.4.1

Michael S. Zick

2012-01-06 23:40 UTC #1 lore raw

PATCH 1/1 Improved git download method performance.

Quoted 8 lines · Theodore Witkamp

Instead of cloning the a complete repository, git archive is used first. If git archive fails because the remote host does not support it, git clone is used with with --depth 1 so only minimal history is copied.
Also the $($(PKG)_SOURCE) default is changed to .tar instead of .tar.gz. This removes unneeded compression and decompression.

But some parts of the world charge per/byte still.

So maybe you want to offer a Wallet/CPU-Cycles preference or choice. ;-)

Mike

Quoted 65 lines

Relative file paths are now relative to $(TOPDIR) not $(DL_DIR).

---
 package/Makefile.package.in |   35 ++++++++++++++++++++---------------
 1 files changed, 20 insertions(+), 15 deletions(-)

package/Makefile.package.in +20 ~~-15~~

index e85eb15..14f1ba1 100644
@@ -146,14 +146,14 @@ domainseparator=$(if $(1),$(1),/)
 
 define DOWNLOAD_GIT
 	test -e $(DL_DIR)/$($(PKG)_SOURCE) || \
-	(pushd $(DL_DIR) > /dev/null && \
-	$(GIT) clone --bare $($(PKG)_SITE) $($(PKG)_BASE_NAME) && \
-	pushd $($(PKG)_BASE_NAME) > /dev/null && \
-	$(GIT) archive --format=tar --prefix=$($(PKG)_BASE_NAME)/ $($(PKG)_DL_VERSION) | \
-		gzip -c > $(DL_DIR)/$($(PKG)_SOURCE) && \
-	popd > /dev/null && \
-	rm -rf $($(PKG)_DL_DIR) && \
-	popd > /dev/null)
+	$(GIT) archive --format=tar --remote $($(PKG)_SITE) --prefix=$($(PKG)_BASE_NAME)/ $($(PKG)_DL_VERSION) \
+	> $(DL_DIR)/$($(PKG)_SOURCE) || \
+	( \
+	$(GIT) clone   -v --depth 1 --bare $($(PKG)_SITE) $($(PKG)_DL_DIR)   &&  \
+	$(GIT) archive --format=tar --remote $($(PKG)_DL_DIR) --prefix=$($(PKG)_BASE_NAME)/ $($(PKG)_DL_VERSION) \
+	> $(DL_DIR)/$($(PKG)_SOURCE) && \
+	rm -rf $($(PKG)_DL_DIR)  \
+	)
 endef
 
 # TODO: improve to check that the given PKG_DL_VERSION exists on the remote

@@ -504,13 +504,6 @@ ifneq ($$($(2)_OVERRIDE_SRCDIR),)
 $(2)_VERSION = custom
 endif
 
-ifndef $(2)_SOURCE
- ifdef $(3)_SOURCE
-  $(2)_SOURCE = $($(3)_SOURCE)
- else
-  $(2)_SOURCE			?= $$($(2)_BASE_NAME).tar.gz
- endif
-endif
 
 ifndef $(2)_PATCH
  ifdef $(3)_PATCH

@@ -536,6 +529,18 @@ ifndef $(2)_SITE_METHOD
  endif
 endif
 
+ifndef $(2)_SOURCE
+ ifdef $(3)_SOURCE
+  $(2)_SOURCE = $($(3)_SOURCE)
+ else
+  ifeq ($$($(2)_SITE_METHOD),git)
+   $(2)_SOURCE			?= $$($(2)_BASE_NAME).tar
+  else
+   $(2)_SOURCE			?= $$($(2)_BASE_NAME).tar.gz
+  endif
+ endif
+endif
+
 ifeq ($$($(2)_SITE_METHOD),local)
 ifeq ($$($(2)_OVERRIDE_SRCDIR),)
 $(2)_OVERRIDE_SRCDIR = $($(2)_SITE)

Thomas Petazzoni

2012-01-07 09:42 UTC #2 lore raw

PATCH 1/1 Improved git download method performance.

Le Fri, 6 Jan 2012 17:40:08 -0600, "Michael S. Zick" <[email protected]> a ?crit :

Quoted 4 lines

Quoted 2 lines
Also the $($(PKG)_SOURCE) default is changed to .tar instead of .tar.gz. This removes unneeded compression and decompression.
But some parts of the world charge per/byte still.

The move to .tar only changes the fact that tarball are saved uncompressed in the $(DL_DIR). It does not change anything to the amount of data transferred over the network, if I understand correctly. But it's true that the size difference of a Linux kernel tarball uncompressed and compressed is quite significant.

Regards,

Thomas

Signature · 4 lines

Thomas Petazzoni, Free Electrons
Kernel, drivers, real-time and embedded Linux
development, consulting, training and support.
http://free-electrons.com

Arnout Vandecappelle

2012-01-15 18:45 UTC #3 lore raw

PATCH 1/1 Improved git download method performance.

In reply to Theodore Witkamp

Reviewed-by Arnout Vandecappelle (Essensium/Mind) <[email protected]>

Quoted 3 lines · On Friday 06 January 2012 23:56:58 Theodore Witkamp wrote:

Instead of cloning the a complete repository, git archive is used first. If git archive fails because the remote host does not support it, git clone is used with with --depth 1 so only minimal history is copied.

Great idea! BTW, is there any upstream that actually supports git archive?

Quoted 2 lines

Also the $($(PKG)_SOURCE) default is changed to .tar instead of .tar.gz. This removes unneeded compression and decompression.

Bad idea. As noted elsewhere, this uses extra disk space (remember that
.tar.gz will stay in the dl directory). With gzip the cpu overhead is fairly small - unless you have an SSD and a slow CPU, disk access will probably dominate (since downloads/extracts are anyway never in parallel with anything else).

Quoted 1 line

Relative file paths are now relative to $(TOPDIR) not $(DL_DIR).

That's of course a better idea, but may be an incompatible change for some
people. So Peter, make sure it gets a big fat warning in the Changelog.

Quoted 18 lines

---
 package/Makefile.package.in |   35 ++++++++++++++++++++---------------
 1 files changed, 20 insertions(+), 15 deletions(-)

package/Makefile.package.in +0 -6

index e85eb15..14f1ba1 100644
@@ -146,14 +146,14 @@ domainseparator=$(if $(1),$(1),/)
 
 define DOWNLOAD_GIT
 	test -e $(DL_DIR)/$($(PKG)_SOURCE) || \
-	(pushd $(DL_DIR) > /dev/null && \
-	$(GIT) clone --bare $($(PKG)_SITE) $($(PKG)_BASE_NAME) && \
-	pushd $($(PKG)_BASE_NAME) > /dev/null && \
-	$(GIT) archive --format=tar --prefix=$($(PKG)_BASE_NAME)/ $($(PKG)_DL_VERSION) | \
-		gzip -c > $(DL_DIR)/$($(PKG)_SOURCE) && \
-	popd > /dev/null && \

package/Makefile.package.in domainseparator=$(if $(1),$(1),/) · line 156

Quoted 10 lines

package/Makefile.package.in +8 -2

-	rm -rf $($(PKG)_DL_DIR) && \
-	popd > /dev/null)
+	$(GIT) archive --format=tar --remote $($(PKG)_SITE) --prefix=$($(PKG)_BASE_NAME)/ $($(PKG)_DL_VERSION) \
+	> $(DL_DIR)/$($(PKG)_SOURCE) || \
+	( \
+	$(GIT) clone   -v --depth 1 --bare $($(PKG)_SITE) $($(PKG)_DL_DIR)   &&  \
+	$(GIT) archive --format=tar --remote $($(PKG)_DL_DIR) --prefix=$($(PKG)_BASE_NAME)/ $($(PKG)_DL_VERSION) \
+	> $(DL_DIR)/$($(PKG)_SOURCE) && \
+	rm -rf $($(PKG)_DL_DIR)  \
+	)

I would like to see the two git archive commands refactored into a single
make variable, but that's not a requirement for acceptance. Something like

define GIT_ARCHIVE_CMD

	$(GIT) archive --format=tar --remote $${remote} --prefix=$($(PKG)_BASE_NAME)/ $($(PKG)_DL_VERSION) | \
	  gzip -c > $(DL_DIR)/$($(PKG)_SOURCE)
endif
...
	remote=$($(PKG)_SITE); $(GIT_ARCHIVE_CMD) || \

...

Also please use some indentation in the shell scripts.

[snip, since the rest of the changes are unnecessary]

Regards,
Arnout

Signature · 9 lines

Arnout Vandecappelle                               arnout at mind be
Senior Embedded Software Architect                 +32-16-286540
Essensium/Mind                                     http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium                BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20120115/28471606/attachment-0001.html>