support/download/git: Prioritize remote archive
PATCH v2 1/1 support/download/git: Prioritize remote archive
From: Benjamin Kamath <[email protected]>
Attempt to do a remote archive since it shortcuts us past a few steps when available. Additionally. if the git server has uploadArchive.allowUnreachable set to true, then this method can also work on arbitrary sha1s, offering a huge speed advantage over a full clone.
---
Changes v1 -> v2:
- Skip remote archive for repositories with submodules (suggested by Yann)
support/download/git | 11 +++++++++++ 1 file changed, 11 insertions(+)
support/download/git +11 -0
@@ -36,6 +36,17 @@ _git() { eval ${GIT} "${@}" } +# Try a remote archive, since it is as fast as a shallow clone and can give us +# an archive directly. Also, if uploadArchive.allowUnreachable is set to true +# on the remote, this will also work for arbitrary sha1s, and will offer a +# considerable speedup over a full clone. This will not work for submodules. +if [ ${recurse} -eq 0 ]; then + printf "Doing remote archive\n" + if _git archive --format=tar.gz --prefix=${basename}/ --remote=${repo} -o ${output} ${cset} 2>&1; then + exit 0 + fi +fi + # Try a shallow clone, since it is faster than a full clone - but that only # works if the version is a ref (tag or branch). Before trying to do a shallow # clone we check if ${cset} is in the list provided by git ls-remote. If not
Signature · 1 line
2.7.4
PATCH v2 1/1 support/download/git: Prioritize remote archive
Benjmin, All,
On 2016-08-17 14:25 -0700, Benjamin Kamath spake thusly:
Quoted 6 lines
From: Benjamin Kamath <[email protected]>
Attempt to do a remote archive since it shortcuts us past a few steps when available. Additionally. if the git server has uploadArchive.allowUnreachable set to true, then this method can also work on arbitrary sha1s, offering a huge speed advantage over a full clone.
Do you know of a publicly-accessible server that allows remote archives?
I've tried with a github-hosted package, but githib won;t allow remote archives, it seems...
Regards,
Yann E. MORIN.
Quoted 37 lines
Signed-off-by Benjamin Kamath <[email protected]>---
Changes v1 -> v2:
- Skip remote archive for repositories with submodules (suggested by Yann)support/download/git | 11 +++++++++++ 1 file changed, 11 insertions(+)support/download/git +11
-0@@ -36,6 +36,17 @@ _git() { eval ${GIT} "${@}" } +# Try a remote archive, since it is as fast as a shallow clone and can give us +# an archive directly. Also, if uploadArchive.allowUnreachable is set to true +# on the remote, this will also work for arbitrary sha1s, and will offer a +# considerable speedup over a full clone. This will not work for submodules. +if [ ${recurse} -eq 0 ]; then + printf "Doing remote archive\n" + if _git archive --format=tar.gz --prefix=${basename}/ --remote=${repo} -o ${output} ${cset} 2>&1; then + exit 0 + fi +fi + # Try a shallow clone, since it is faster than a full clone - but that only # works if the version is a ref (tag or branch). Before trying to do a shallow # clone we check if ${cset} is in the list provided by git ls-remote. If notSignature · 6 lines
2.7.4 _______________________________________________ buildroot mailing list buildroot at busybox.net http://lists.busybox.net/mailman/listinfo/buildroot
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------'
PATCH v2 1/1 support/download/git: Prioritize remote archive
Quoted 10 lines · Yann E. MORIN
Benjmin, All,
On 2016-08-17 14:25 -0700, Benjamin Kamath spake thusly:
Quoted 6 lines
From: Benjamin Kamath <[email protected]>
Attempt to do a remote archive since it shortcuts us past a few steps when available. Additionally. if the git server has uploadArchive.allowUnreachable set to true, then this method can also work on arbitrary sha1s, offering a huge speed advantage over a full clone.
Quoted 4 lines
Do you know of a publicly-accessible server that allows remote archives?
I've tried with a github-hosted package, but githib won;t allow remote archives, it seems...
I just tried quite a few and it doesn't seem to be frequently supported. gitlab-ce does seem to have it enabled by default:
git at gitlab.com:gitlab-org/gitlab-ce.git
Quoted 49 lines
Regards,
Yann E. MORIN.Quoted 33 lines
Signed-off-by Benjamin Kamath <[email protected]>---
Changes v1 -> v2:
- Skip remote archive for repositories with submodules (suggested by Yann)support/download/git | 11 +++++++++++ 1 file changed, 11 insertions(+)support/download/git +11
-1@@ -36,6 +36,17 @@ _git() { eval ${GIT} "${@}" } +# Try a remote archive, since it is as fast as a shallow clone and can give us +# an archive directly. Also, if uploadArchive.allowUnreachable is set to true +# on the remote, this will also work for arbitrary sha1s, and will offer a +# considerable speedup over a full clone. This will not work for submodules. +if [ ${recurse} -eq 0 ]; then + printf "Doing remote archive\n" + if _git archive --format=tar.gz --prefix=${basename}/ --remote=${repo} -o ${output} ${cset} 2>&1; then + exit 0 + fi +fi + # Try a shallow clone, since it is faster than a full clone - but that only # works if the version is a ref (tag or branch). Before trying to do a shallow # clone we check if ${cset} is in the list provided by git ls-remote. If not --2.7.4
Quoted 4 lines
_______________________________________________
buildroot mailing list
buildroot at busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot--
.-----------------.--------------------.------------------.--------------------.| Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. |'------------------------------^-------^------------------^--------------------'
PATCH v2 1/1 support/download/git: Prioritize remote archive
Benjamin, All,
On 2016-08-17 15:17 -0700, Benjamin Kamath spake thusly:
Quoted 11 lines · Yann E. MORIN
Quoted 10 lines
Benjmin, All,
On 2016-08-17 14:25 -0700, Benjamin Kamath spake thusly:
Quoted 6 lines
From: Benjamin Kamath <[email protected]>
Attempt to do a remote archive since it shortcuts us past a few steps when available. Additionally. if the git server has uploadArchive.allowUnreachable set to true, then this method can also work on arbitrary sha1s, offering a huge speed advantage over a full clone.
Quoted 8 lines
Quoted 4 lines
Do you know of a publicly-accessible server that allows remote archives?
I've tried with a github-hosted package, but githib won;t allow remote archives, it seems...
I just tried quite a few and it doesn't seem to be frequently supported. gitlab-ce does seem to have it enabled by default:
git at gitlab.com:gitlab-org/gitlab-ce.git
OK, it does work, as long as one uses the ssh protocol. It does not work via http:// or git://
Which means we have no package in Buildroot that can use this.
Do you have a need for that for an internal package?
Regards,
Yann E. MORIN.
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------'
PATCH v2 1/1 support/download/git: Prioritize remote archive
Quoted 18 lines · Yann E. MORIN
Benjamin, All,
On 2016-08-17 15:17 -0700, Benjamin Kamath spake thusly:
Quoted 15 lines · Yann E. MORIN
Quoted 14 lines
Benjmin, All,
On 2016-08-17 14:25 -0700, Benjamin Kamath spake thusly:
Quoted 6 lines
From: Benjamin Kamath <[email protected]>
Attempt to do a remote archive since it shortcuts us past a few steps when available. Additionally. if the git server has uploadArchive.allowUnreachable set to true, then this method can also work on arbitrary sha1s, offering a huge speed advantage over a full clone.
Do you know of a publicly-accessible server that allows remote archives?
I've tried with a github-hosted package, but githib won;t allow remote archives, it seems...
Quoted 8 lines
Quoted 4 lines
I just tried quite a few and it doesn't seem to be frequently supported. gitlab-ce does seem to have it enabled by default:
git at gitlab.com:gitlab-org/gitlab-ce.git
OK, it does work, as long as one uses the ssh protocol. It does not work via http:// or
git://
OK. I had not noticed that before, and as such that makes this patch much less useful.
Quoted 3 lines
Which means we have no package in Buildroot that can use this.
Do you have a need for that for an internal package?
Yes, but we can create a local patch for it. On dev-branches of our project we frequently have internal packages that point directly at sha1s. I mistakenly thought more servers would support remote archive. I suppose we should reject this patch.
Quoted 11 lines
Regards,
Yann E. MORIN.--
.-----------------.--------------------.------------------.--------------------.| Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. |'------------------------------^-------^------------------^--------------------'
PATCH v2 1/1 support/download/git: Prioritize remote archive
Benjamin, All,
On 2016-08-18 02:26 -0700, Benjamin Kamath spake thusly:
Quoted 30 lines · Yann E. MORIN
Quoted 18 lines
Benjamin, All,
On 2016-08-17 15:17 -0700, Benjamin Kamath spake thusly:
Quoted 15 lines · Yann E. MORIN
Quoted 14 lines
Benjmin, All,
On 2016-08-17 14:25 -0700, Benjamin Kamath spake thusly:
Quoted 6 lines
From: Benjamin Kamath <[email protected]>
Attempt to do a remote archive since it shortcuts us past a few steps when available. Additionally. if the git server has uploadArchive.allowUnreachable set to true, then this method can also work on arbitrary sha1s, offering a huge speed advantage over a full clone.
Do you know of a publicly-accessible server that allows remote archives?
I've tried with a github-hosted package, but githib won;t allow remote archives, it seems...
Quoted 8 lines
Quoted 4 lines
I just tried quite a few and it doesn't seem to be frequently supported. gitlab-ce does seem to have it enabled by default:
git at gitlab.com:gitlab-org/gitlab-ce.git
OK, it does work, as long as one uses the ssh protocol. It does not work via http:// or
git://OK. I had not noticed that before, and as such that makes this patch much less useful.
Quoted 7 lines
Quoted 3 lines
Which means we have no package in Buildroot that can use this.
Do you have a need for that for an internal package?
Yes, but we can create a local patch for it. On dev-branches of our project we frequently have internal packages that point directly at sha1s. I mistakenly thought more servers would support remote archive. I suppose we should reject this patch.
I'm OK with adding this, as long as it does not break existing behaviour. From my little tests, it shows that it works.
Regards,
Yann E. MORIN.
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------'
PATCH v2 1/1 support/download/git: Prioritize remote archive
In reply to Benjamin KamathBenjamin, All,
On 2016-08-17 14:25 -0700, Benjamin Kamath spake thusly:
Quoted 21 lines
From: Benjamin Kamath <[email protected]>
Attempt to do a remote archive since it shortcuts us past a few steps when available. Additionally. if the git server has uploadArchive.allowUnreachable set to true, then this method can also work on arbitrary sha1s, offering a huge speed advantage over a full clone.
Signed-off-by Benjamin Kamath <[email protected]>---
Changes v1 -> v2:
- Skip remote archive for repositories with submodules (suggested by Yann)support/download/git | 11 +++++++++++ 1 file changed, 11 insertions(+)support/download/git +0
-0@@ -36,6 +36,17 @@ _git() { eval ${GIT} "${@}"
support/download/git _git() { · line 46
Quoted 10 lines
support/download/git +8
-0} +# Try a remote archive, since it is as fast as a shallow clone and can give us +# an archive directly. Also, if uploadArchive.allowUnreachable is set to true +# on the remote, this will also work for arbitrary sha1s, and will offer a +# considerable speedup over a full clone. This will not work for submodules. +if [ ${recurse} -eq 0 ]; then + printf "Doing remote archive\n" + if _git archive --format=tar.gz --prefix=${basename}/ --remote=${repo} -o ${output} ${cset} 2>&1; then + exit 0
I would have also done:
else
printf "Remote archive failed, falling back to cloning\n"
Like we do for the shallow clone.
Also, why do you redirect stderr? We don't do it in the clone commands. We only do it to fetch the special references (gitHub PRs of Gerrit changes) because it does not make sense to display an spurious message when the reference is in fact not special.
Regards,
Yann E. MORIN.
Quoted 13 lines
support/download/git +3
-0+ fi +fi + # Try a shallow clone, since it is faster than a full clone - but that only # works if the version is a ref (tag or branch). Before trying to do a shallow # clone we check if ${cset} is in the list provided by git ls-remote. If notSignature · 6 lines
2.7.4 _______________________________________________ buildroot mailing list buildroot at busybox.net http://lists.busybox.net/mailman/listinfo/buildroot
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------'
PATCH v3 1/1 support/download/git: Prioritize remote archive
In reply to Benjamin KamathAttempt to do a remote archive since it shortcuts us past a few steps when available. Additionally. if the git server has uploadArchive.allowUnreachable set to true, then this method can also work on arbitrary sha1s, offering a huge speed advantage over a full clone.
---
Changes v2 -> v3:
- Indicate to user if the remote archive fails (suggested by Yann)
- redirect all errors from git archive to /dev/null since this operation fails on most public git servers
Changes v1 -> v2:
- Skip remote archive for repositories with submodules (suggested by Yann)
support/download/git | 16 ++++++++++++++++ 1 file changed, 16 insertions(+)
support/download/git +16 -0
@@ -36,6 +36,22 @@ _git() { eval ${GIT} "${@}" } +# Try a remote archive, since it is as fast as a shallow clone and can give us +# an archive directly. Also, if uploadArchive.allowUnreachable is set to true +# on the remote, this will also work for arbitrary sha1s, and will offer a +# considerable speedup over a full clone. This will not work for submodules. +# Since this command fails on most public git servers, consign the output to +# oblivion but still trace as a warning. +if [ ${recurse} -eq 0 ]; then + printf "Doing remote archive\n" + if _git archive --format=tar.gz --prefix=${basename}/ --remote=${repo} \ + -o ${output} ${cset} >/dev/null 2>&1; then + exit 0 + else + printf "Remote archive failed, falling back to cloning\n" + fi +fi + # Try a shallow clone, since it is faster than a full clone - but that only # works if the version is a ref (tag or branch). Before trying to do a shallow # clone we check if ${cset} is in the list provided by git ls-remote. If not
Signature · 1 line
2.7.4
PATCH v3 1/1 support/download/git: Prioritize remote archive
Hi Benjamin,
Quoted 6 lines · Benjamin Kamath
Attempt to do a remote archive since it shortcuts us past a few steps when available. Additionally. if the git server has uploadArchive.allowUnreachable set to true, then this method can also work on arbitrary sha1s, offering a huge speed advantage over a full clone.
Signed-off-by Benjamin Kamath <[email protected]>
After some discussion at the Buildroot developer meeting, we decided not to
accept this patch, for the following reasons.
* We want to keep the download/git script as simple as reasonably possible.
* Not many upstream sites support this option.
* We now have good support for shallow clones; in most cases, the shallow clone handling will give equal benefits.
So I've marked this patch as rejected in patchwork.
If you feel it is important to include this feature in Buildroot after all,
feel free to argue your case.
And finally, sorry that it took so long before we reacted to your patch.
Regards,
Arnout
Quoted 39 lines
---
Changes v2 -> v3:
- Indicate to user if the remote archive fails (suggested by Yann)
- redirect all errors from git archive to/dev/nullsince this operation fails on most public git servers
Changes v1 -> v2:
- Skip remote archive for repositories with submodules (suggested by Yann)support/download/git | 16 ++++++++++++++++ 1 file changed, 16 insertions(+)support/download/git +16
-0@@ -36,6 +36,22 @@ _git() { eval ${GIT} "${@}" } +# Try a remote archive, since it is as fast as a shallow clone and can give us +# an archive directly. Also, if uploadArchive.allowUnreachable is set to true +# on the remote, this will also work for arbitrary sha1s, and will offer a +# considerable speedup over a full clone. This will not work for submodules. +# Since this command fails on most public git servers, consign the output to +# oblivion but still trace as a warning. +if [ ${recurse} -eq 0 ]; then + printf "Doing remote archive\n" + if _git archive --format=tar.gz --prefix=${basename}/ --remote=${repo} \ + -o ${output} ${cset} >/dev/null 2>&1; then + exit 0 + else + printf "Remote archive failed, falling back to cloning\n" + fi +fi + # Try a shallow clone, since it is faster than a full clone - but that only # works if the version is a ref (tag or branch). Before trying to do a shallow # clone we check if ${cset} is in the list provided by git ls-remote. If not
Signature · 6 lines
Arnout Vandecappelle arnout at mind be Senior Embedded Software Architect +32-16-286500 Essensium/Mind http://www.mind.be G.Geenslaan 9, 3001 Leuven, Belgium BE 872 984 063 RPR Leuven LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle GPG fingerprint: 7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF