Skip to contentSkip to conversation

support/download/git: Prioritize remote archive

PATCH Superseded Benjamin Kamath 3 participants 9 messages Aug 17, 2016 – Feb 12, 2017

lore
Share & tools
lore
#0 lore raw

PATCH v2 1/1 support/download/git: Prioritize remote archive

From: Benjamin Kamath <[email protected]>

Attempt to do a remote archive since it shortcuts us past a few steps when available. Additionally. if the git server has uploadArchive.allowUnreachable set to true, then this method can also work on arbitrary sha1s, offering a huge speed advantage over a full clone.

---
Changes v1 -> v2:
- Skip remote archive for repositories with submodules (suggested by Yann)

 support/download/git | 11 +++++++++++
 1 file changed, 11 insertions(+)
support/download/git +11 -0
index 416cd1b..d060cbd 100755
@@ -36,6 +36,17 @@ _git() {
     eval ${GIT} "${@}"
 }
 
+# Try a remote archive, since it is as fast as a shallow clone and can give us
+# an archive directly. Also, if uploadArchive.allowUnreachable is set to true
+# on the remote, this will also work for arbitrary sha1s, and will offer a
+# considerable speedup over a full clone. This will not work for submodules.
+if [ ${recurse} -eq 0 ]; then
+    printf "Doing remote archive\n"
+    if _git archive --format=tar.gz --prefix=${basename}/ --remote=${repo} -o ${output} ${cset} 2>&1; then
+        exit 0
+    fi
+fi
+
 # Try a shallow clone, since it is faster than a full clone - but that only
 # works if the version is a ref (tag or branch). Before trying to do a shallow
 # clone we check if ${cset} is in the list provided by git ls-remote. If not
Signature · 1 line
2.7.4
#1 lore raw

PATCH v2 1/1 support/download/git: Prioritize remote archive

Benjmin, All,

On 2016-08-17 14:25 -0700, Benjamin Kamath spake thusly:

Quoted 6 lines

From: Benjamin Kamath <[email protected]>

Attempt to do a remote archive since it shortcuts us past a few steps when available. Additionally. if the git server has uploadArchive.allowUnreachable set to true, then this method can also work on arbitrary sha1s, offering a huge speed advantage over a full clone.

Do you know of a publicly-accessible server that allows remote archives?

I've tried with a github-hosted package, but githib won;t allow remote archives, it seems...

Regards,
Yann E. MORIN.

Quoted 37 lines

---
Changes v1 -> v2:
- Skip remote archive for repositories with submodules (suggested by Yann)

 support/download/git | 11 +++++++++++
 1 file changed, 11 insertions(+)
support/download/git +11 -0
index 416cd1b..d060cbd 100755
@@ -36,6 +36,17 @@ _git() {
     eval ${GIT} "${@}"
 }
 
+# Try a remote archive, since it is as fast as a shallow clone and can give us
+# an archive directly. Also, if uploadArchive.allowUnreachable is set to true
+# on the remote, this will also work for arbitrary sha1s, and will offer a
+# considerable speedup over a full clone. This will not work for submodules.
+if [ ${recurse} -eq 0 ]; then
+    printf "Doing remote archive\n"
+    if _git archive --format=tar.gz --prefix=${basename}/ --remote=${repo} -o ${output} ${cset} 2>&1; then
+        exit 0
+    fi
+fi
+
 # Try a shallow clone, since it is faster than a full clone - but that only
 # works if the version is a ref (tag or branch). Before trying to do a shallow
 # clone we check if ${cset} is in the list provided by git ls-remote. If not
Signature · 6 lines
2.7.4

_______________________________________________
buildroot mailing list
buildroot at busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
#2 lore raw

PATCH v2 1/1 support/download/git: Prioritize remote archive

Quoted 10 lines · Yann E. MORIN

Benjmin, All,

On 2016-08-17 14:25 -0700, Benjamin Kamath spake thusly:

Quoted 6 lines

From: Benjamin Kamath <[email protected]>

Attempt to do a remote archive since it shortcuts us past a few steps when available. Additionally. if the git server has uploadArchive.allowUnreachable set to true, then this method can also work on arbitrary sha1s, offering a huge speed advantage over a full clone.

Quoted 4 lines

Do you know of a publicly-accessible server that allows remote archives?

I've tried with a github-hosted package, but githib won;t allow remote archives, it seems...

I just tried quite a few and it doesn't seem to be frequently supported. gitlab-ce does seem to have it enabled by default:

git at gitlab.com:gitlab-org/gitlab-ce.git

Quoted 49 lines

Regards,
Yann E. MORIN.

Quoted 33 lines

---
Changes v1 -> v2:
- Skip remote archive for repositories with submodules (suggested by Yann)

 support/download/git | 11 +++++++++++
 1 file changed, 11 insertions(+)
support/download/git +11 -1
index 416cd1b..d060cbd 100755
@@ -36,6 +36,17 @@ _git() {
     eval ${GIT} "${@}"
 }
 
+# Try a remote archive, since it is as fast as a shallow clone and can give us
+# an archive directly. Also, if uploadArchive.allowUnreachable is set to true
+# on the remote, this will also work for arbitrary sha1s, and will offer a
+# considerable speedup over a full clone. This will not work for submodules.
+if [ ${recurse} -eq 0 ]; then
+    printf "Doing remote archive\n"
+    if _git archive --format=tar.gz --prefix=${basename}/ --remote=${repo} -o ${output} ${cset} 2>&1; then
+        exit 0
+    fi
+fi
+
 # Try a shallow clone, since it is faster than a full clone - but that only
 # works if the version is a ref (tag or branch). Before trying to do a shallow
 # clone we check if ${cset} is in the list provided by git ls-remote. If not
--

2.7.4

Quoted 4 lines

_______________________________________________
buildroot mailing list
buildroot at busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot

--
.-----------------.--------------------.------------------.--------------------.

|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |

'------------------------------^-------^------------------^--------------------'

#3 lore raw

PATCH v2 1/1 support/download/git: Prioritize remote archive

Benjamin, All,

On 2016-08-17 15:17 -0700, Benjamin Kamath spake thusly:

Quoted 11 lines · Yann E. MORIN
Quoted 10 lines

Benjmin, All,

On 2016-08-17 14:25 -0700, Benjamin Kamath spake thusly:

Quoted 6 lines

From: Benjamin Kamath <[email protected]>

Attempt to do a remote archive since it shortcuts us past a few steps when available. Additionally. if the git server has uploadArchive.allowUnreachable set to true, then this method can also work on arbitrary sha1s, offering a huge speed advantage over a full clone.

Quoted 8 lines
Quoted 4 lines

Do you know of a publicly-accessible server that allows remote archives?

I've tried with a github-hosted package, but githib won;t allow remote archives, it seems...

I just tried quite a few and it doesn't seem to be frequently supported. gitlab-ce does seem to have it enabled by default:

git at gitlab.com:gitlab-org/gitlab-ce.git

OK, it does work, as long as one uses the ssh protocol. It does not work via http:// or git://

Which means we have no package in Buildroot that can use this.

Do you have a need for that for an internal package?

Regards,
Yann E. MORIN.

Signature · 6 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
#4 lore raw

PATCH v2 1/1 support/download/git: Prioritize remote archive

Quoted 18 lines · Yann E. MORIN

Benjamin, All,

On 2016-08-17 15:17 -0700, Benjamin Kamath spake thusly:

Quoted 15 lines · Yann E. MORIN
Quoted 14 lines

Benjmin, All,

On 2016-08-17 14:25 -0700, Benjamin Kamath spake thusly:

Quoted 6 lines

From: Benjamin Kamath <[email protected]>

Attempt to do a remote archive since it shortcuts us past a few steps when available. Additionally. if the git server has uploadArchive.allowUnreachable set to true, then this method can also work on arbitrary sha1s, offering a huge speed advantage over a full clone.

Do you know of a publicly-accessible server that allows remote archives?

I've tried with a github-hosted package, but githib won;t allow remote archives, it seems...

Quoted 8 lines
Quoted 4 lines

I just tried quite a few and it doesn't seem to be frequently supported. gitlab-ce does seem to have it enabled by default:

git at gitlab.com:gitlab-org/gitlab-ce.git

OK, it does work, as long as one uses the ssh protocol. It does not work via http:// or git://

OK. I had not noticed that before, and as such that makes this patch much less useful.

Quoted 3 lines

Which means we have no package in Buildroot that can use this.

Do you have a need for that for an internal package?

Yes, but we can create a local patch for it. On dev-branches of our project we frequently have internal packages that point directly at sha1s. I mistakenly thought more servers would support remote archive. I suppose we should reject this patch.

Quoted 11 lines

Regards,
Yann E. MORIN.

--
.-----------------.--------------------.------------------.--------------------.

|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |

'------------------------------^-------^------------------^--------------------'

#5 lore raw

PATCH v2 1/1 support/download/git: Prioritize remote archive

Benjamin, All,

On 2016-08-18 02:26 -0700, Benjamin Kamath spake thusly:

Quoted 30 lines · Yann E. MORIN
Quoted 18 lines

Benjamin, All,

On 2016-08-17 15:17 -0700, Benjamin Kamath spake thusly:

Quoted 15 lines · Yann E. MORIN
Quoted 14 lines

Benjmin, All,

On 2016-08-17 14:25 -0700, Benjamin Kamath spake thusly:

Quoted 6 lines

From: Benjamin Kamath <[email protected]>

Attempt to do a remote archive since it shortcuts us past a few steps when available. Additionally. if the git server has uploadArchive.allowUnreachable set to true, then this method can also work on arbitrary sha1s, offering a huge speed advantage over a full clone.

Do you know of a publicly-accessible server that allows remote archives?

I've tried with a github-hosted package, but githib won;t allow remote archives, it seems...

Quoted 8 lines
Quoted 4 lines

I just tried quite a few and it doesn't seem to be frequently supported. gitlab-ce does seem to have it enabled by default:

git at gitlab.com:gitlab-org/gitlab-ce.git

OK, it does work, as long as one uses the ssh protocol. It does not work via http:// or git://

OK. I had not noticed that before, and as such that makes this patch much less useful.

Quoted 7 lines
Quoted 3 lines

Which means we have no package in Buildroot that can use this.

Do you have a need for that for an internal package?

Yes, but we can create a local patch for it. On dev-branches of our project we frequently have internal packages that point directly at sha1s. I mistakenly thought more servers would support remote archive. I suppose we should reject this patch.

I'm OK with adding this, as long as it does not break existing behaviour. From my little tests, it shows that it works.

Regards,
Yann E. MORIN.

Signature · 6 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
#6 lore raw

PATCH v2 1/1 support/download/git: Prioritize remote archive

In reply to Benjamin Kamath

Benjamin, All,

On 2016-08-17 14:25 -0700, Benjamin Kamath spake thusly:

Quoted 21 lines

From: Benjamin Kamath <[email protected]>

Attempt to do a remote archive since it shortcuts us past a few steps when available. Additionally. if the git server has uploadArchive.allowUnreachable set to true, then this method can also work on arbitrary sha1s, offering a huge speed advantage over a full clone.

---
Changes v1 -> v2:
- Skip remote archive for repositories with submodules (suggested by Yann)

 support/download/git | 11 +++++++++++
 1 file changed, 11 insertions(+)
support/download/git +0 -0
index 416cd1b..d060cbd 100755
@@ -36,6 +36,17 @@ _git() {
     eval ${GIT} "${@}"

support/download/git _git() { · line 46

Quoted 10 lines
support/download/git +8 -0
 }
 
+# Try a remote archive, since it is as fast as a shallow clone and can give us
+# an archive directly. Also, if uploadArchive.allowUnreachable is set to true
+# on the remote, this will also work for arbitrary sha1s, and will offer a
+# considerable speedup over a full clone. This will not work for submodules.
+if [ ${recurse} -eq 0 ]; then
+    printf "Doing remote archive\n"
+    if _git archive --format=tar.gz --prefix=${basename}/ --remote=${repo} -o ${output} ${cset} 2>&1; then
+        exit 0

I would have also done:

else
printf "Remote archive failed, falling back to cloning\n"

Like we do for the shallow clone.

Also, why do you redirect stderr? We don't do it in the clone commands. We only do it to fetch the special references (gitHub PRs of Gerrit changes) because it does not make sense to display an spurious message when the reference is in fact not special.

Regards,
Yann E. MORIN.

Quoted 13 lines
support/download/git +3 -0
+    fi
+fi
+
 # Try a shallow clone, since it is faster than a full clone - but that only
 # works if the version is a ref (tag or branch). Before trying to do a shallow
 # clone we check if ${cset} is in the list provided by git ls-remote. If not
Signature · 6 lines
2.7.4

_______________________________________________
buildroot mailing list
buildroot at busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
#7 lore raw

PATCH v3 1/1 support/download/git: Prioritize remote archive

In reply to Benjamin Kamath

Attempt to do a remote archive since it shortcuts us past a few steps when available. Additionally. if the git server has uploadArchive.allowUnreachable set to true, then this method can also work on arbitrary sha1s, offering a huge speed advantage over a full clone.

---
Changes v2 -> v3:
- Indicate to user if the remote archive fails (suggested by Yann)
- redirect all errors from git archive to /dev/null since this operation fails on most public git servers
Changes v1 -> v2:
- Skip remote archive for repositories with submodules (suggested by Yann)

 support/download/git | 16 ++++++++++++++++
 1 file changed, 16 insertions(+)
support/download/git +16 -0
index 416cd1b..ec49057 100755
@@ -36,6 +36,22 @@ _git() {
     eval ${GIT} "${@}"
 }
 
+# Try a remote archive, since it is as fast as a shallow clone and can give us
+# an archive directly. Also, if uploadArchive.allowUnreachable is set to true
+# on the remote, this will also work for arbitrary sha1s, and will offer a
+# considerable speedup over a full clone. This will not work for submodules.
+# Since this command fails on most public git servers, consign the output to 
+# oblivion but still trace as a warning.
+if [ ${recurse} -eq 0 ]; then
+    printf "Doing remote archive\n"
+    if _git archive --format=tar.gz --prefix=${basename}/ --remote=${repo} \
+        -o ${output} ${cset} >/dev/null 2>&1; then
+        exit 0
+    else
+        printf "Remote archive failed, falling back to cloning\n"
+    fi
+fi
+
 # Try a shallow clone, since it is faster than a full clone - but that only
 # works if the version is a ref (tag or branch). Before trying to do a shallow
 # clone we check if ${cset} is in the list provided by git ls-remote. If not
Signature · 1 line
2.7.4
#8 lore raw

PATCH v3 1/1 support/download/git: Prioritize remote archive

Hi Benjamin,

Quoted 6 lines · Benjamin Kamath

Attempt to do a remote archive since it shortcuts us past a few steps when available. Additionally. if the git server has uploadArchive.allowUnreachable set to true, then this method can also work on arbitrary sha1s, offering a huge speed advantage over a full clone.

After some discussion at the Buildroot developer meeting, we decided not to
accept this patch, for the following reasons.

* We want to keep the download/git script as simple as reasonably possible.

* Not many upstream sites support this option.

* We now have good support for shallow clones; in most cases, the shallow clone handling will give equal benefits.

So I've marked this patch as rejected in patchwork.

If you feel it is important to include this feature in Buildroot after all,
feel free to argue your case.

And finally, sorry that it took so long before we reacted to your patch.

Regards,
Arnout

Quoted 39 lines

---
Changes v2 -> v3:
- Indicate to user if the remote archive fails (suggested by Yann)
- redirect all errors from git archive to /dev/null since this operation fails on most public git servers
Changes v1 -> v2:
- Skip remote archive for repositories with submodules (suggested by Yann)

 support/download/git | 16 ++++++++++++++++
 1 file changed, 16 insertions(+)
support/download/git +16 -0
index 416cd1b..ec49057 100755
@@ -36,6 +36,22 @@ _git() {
     eval ${GIT} "${@}"
 }
 
+# Try a remote archive, since it is as fast as a shallow clone and can give us
+# an archive directly. Also, if uploadArchive.allowUnreachable is set to true
+# on the remote, this will also work for arbitrary sha1s, and will offer a
+# considerable speedup over a full clone. This will not work for submodules.
+# Since this command fails on most public git servers, consign the output to 
+# oblivion but still trace as a warning.
+if [ ${recurse} -eq 0 ]; then
+    printf "Doing remote archive\n"
+    if _git archive --format=tar.gz --prefix=${basename}/ --remote=${repo} \
+        -o ${output} ${cset} >/dev/null 2>&1; then
+        exit 0
+    else
+        printf "Remote archive failed, falling back to cloning\n"
+    fi
+fi
+
 # Try a shallow clone, since it is faster than a full clone - but that only
 # works if the version is a ref (tag or branch). Before trying to do a shallow
 # clone we check if ${cset} is in the list provided by git ls-remote. If not
Signature · 6 lines
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF