fix some corner cases for download/git v1

=1.9.0: "fetch tags *in addition to* other stuff".

With old versions of git, when the shallow fetch was not enough, the download fails like this:
Fetching all references
Could not fetch special ref 'sha1'; assuming it is not special.
fatal: reference is not a tree: sha1

At git version 1.9.0, the semantics of the option -t for git fetch changed, see upstream commit [1]:
< 1.9.0: "fetch tags without also fetching other references";

Quoted 1 line

=1.9.0: "fetch tags *in addition to* other stuff".

Fall back to explicit refspec to support distros that use ancient versions of git.

support/download/git +0 -0

 fi
 
 # Try to get the special refs exposed by some forges (pull-requests for

Signature · 1 line

2.14.1

support/download/git if [ -n "$(_git ls-remote origin "'${cset}'" 2>&1)" ]; then · line 79

Quoted 2 lines

support/download/git +1 -1

-    _git fetch origin -t
+    _git fetch origin -u "'+refs/tags/*:refs/tags/*'" "'+refs/heads/*:refs/remotes/origin/*'"

Why don't we just run:

git fetch origin
git fetch origin -t

As I understand it:

ancient git | new git
--------------------------------------------------------------------

git fetch     | fetch all refs but tags | fetches all refs but tags
git fetch -t  | fetches only tags       | fetch all refs and tags

Right?

Quoted 5 lines

support/download/git if [ -n "$(_git ls-remote origin "'${cset}'" 2>&1)" ]; then · line 79

Quoted 2 lines

support/download/git +1 -1

-    _git fetch origin -t
+    _git fetch origin -u "'+refs/tags/*:refs/tags/*'" "'+refs/heads/*:refs/remotes/origin/*'"

Why don't we just run:
git fetch origin
git fetch origin -t

One extra connection to the server ...
but much more readable/maintainable!
I will do. I will change this patch to Changes Requested.

Quoted 5 lines

As I understand it:
ancient git | new git
--------------------------------------------------------------------

git fetch     | fetch all refs but tags | fetches all refs but tags
git fetch -t  | fetches only tags       | fetch all refs and tags

Nit: actually 'git fetch' does fetch some tags, but not necessarily all. Only tags in commits also reachable by branches are fetched.

On 2018-04-13 15:23 -0300, Ricardo Martincoski spake thusly:

Quoted 7 lines · Yann E. MORIN

Quoted 5 lines

support/download/git if [ -n "$(_git ls-remote origin "'${cset}'" 2>&1)" ]; then · line 79

Quoted 2 lines

support/download/git +1 -1

-    _git fetch origin -t
+    _git fetch origin -u "'+refs/tags/*:refs/tags/*'" "'+refs/heads/*:refs/remotes/origin/*'"

Why don't we just run:
git fetch origin
git fetch origin -t

Quoted 3 lines

One extra connection to the server ...
but much more readable/maintainable!
I will do. I will change this patch to Changes Requested.

Thanks. I don't care much about an extra round-trip to the server, especially if that makes the code easier to read.

OK, but overall, we would cover all cases with the two git-fetch, whether we use an old git or a newer one, and that's the important info.

In the case a previous operation in the git cache was abruptly interrupted the git cache can be in a dirty state (removed files, untracked files). This can lead to download failures during the checkout, like this:
error: The following untracked working tree files would be overwritten by checkout:
and/or this:
error: The following untracked working tree files would be removed by checkout:

If an untracked file, directory or submodule is present after the checkout, the tarball will have additional content, leading to a hash mismatch.

When performing the checkout, use -f to allow it to occur even if there are untracked files that would be overwritten or removed.

Use git clean to remove any untracked files before generating the tarball. Use the second -f to ensure the repo is pristine even if previous checked out ref contained a submodule that is not present in the ref just checked out.

Fixes:
http://autobuild.buildroot.net/results/b418394f679ad269d587753302c540036793334c
Possibly fixes:
http://autobuild.buildroot.net/results/9b584a624bb08c5ab6d977b93f4b92ed4bd1d40f

Signed-off-by Ricardo Martincoski <[email protected]>

Cc "Yann E. MORIN" <[email protected]>

---
I did not split this into 2 patches (the first one adding -f to checkout and the second one adding git reset) because only adding -f to checkout would make the git checkout command to succeed but the download would still fails when checking the hash if any untracked file was left. In this case the git cache also would still be dirty, so it is not really a fix to a dirty cache.

---
 support/download/git | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

support/download/git +2 -1

index 1b09e5c750..68b5d920a1 100755
@@ -92,7 +92,8 @@ fi
 
 # Checkout the required changeset, so that we can update the required
 # submodules.
-_git checkout -q "'${cset}'"
+_git checkout -f -q "'${cset}'"
+_git clean -ffdx
 
 # Get date of commit to generate a reproducible archive.
 # %cD is RFC2822, so it's fully qualified, with TZ and all.

Signature · 1 line

2.14.1

On 2018-04-12 06:28 -0300, Ricardo Martincoski spake thusly:

Quoted 7 lines

When performing the checkout, use -f to allow it to occur even if there are untracked files that would be overwritten or removed.

Use git clean to remove any untracked files before generating the tarball. Use the second -f to ensure the repo is pristine even if previous checked out ref contained a submodule that is not present in the ref just checked out.

I'm not very happy of those tricks... If the repos is broken, it may be in a state that repairing it is not even possible at all, in case a critical git data is missing/damaged.

If the repo is broken, either:

- we ditch the repository and restart frm scratch,

- or we just bail out and tell the user what hapened, so they can take action (e.g. remove the broken repo manually).

Quoted 8 lines

On 2018-04-12 06:28 -0300, Ricardo Martincoski spake thusly:

Quoted 7 lines

When performing the checkout, use -f to allow it to occur even if there are untracked files that would be overwritten or removed.

Use git clean to remove any untracked files before generating the tarball. Use the second -f to ensure the repo is pristine even if previous checked out ref contained a submodule that is not present in the ref just checked out.

[snip]

Quoted 4 lines

I'm not very happy of those tricks... If the repos is broken, it may be in a state that repairing it is not even possible at all, in case a critical git data is missing/damaged.

I created the series thinking only on git repos that are sane but have a dirty worktree. I should have used "dirty worktree" in the commit log.

Rethinking now, taking into account a broken repo, and also the fact that the default download directory is inside buildroot, this patch is a bit dangerous.

With a completely broken repo in the git cache, the commands called by the download script end up executed in any repo that eventually contains that file (parent \(of parent \)\+ directory), so:

$ cd buildroot
$ git init dl/package/git
$ cd dl/package/git
$ rm .git/HEAD # simulate a completely broken repo
$ git <command>
The git command would be executed in the buildroot tree.
A fetch to the wrong repo is bad, but not that bad.
A checkout would fail.
But checkout -f and git clean would succeed, potentially making the user to lose
changes.

I will change this patch to Changes Requested.

A way to avoid this, once the script entered the directory, is to force the git-dir:

$ git --git-dir=.git <command>
This command would fail for a too-much-broken repo.

This is important also if we test the repo with git fsck before ditch+restart, otherwise it would be possible to return OK based on the wrong repo.

Quoted 7 lines

If the repo is broken, either:

- we ditch the repository and restart frm scratch,

- or we just bail out and tell the user what hapened, so they can take action (e.g. remove the broken repo manually).

When the repo is broken (detectable by git fsck) I agree this is the best approach.

But I think a sane git repo (detectable by git fsck) that is in a dirty state (in the sense of git clean) is a different case.
It can happen for example if a previous checkout operation was abruptly interrupted during a git checkout. In this case it seems to me a bit extreme (but not incorrect of course) to remove the repo and restart from scratch.

For the case a file from the worktree is missing the current 'git checkout' succeeds and the problem shows up as a hash mismatch.
Using 'git checkout -f' the repo is recovered.
For the case a git object needed by the checkout is missing the current 'git checkout' succeeds and the problem shows up as a hash mismatch. Using 'git checkout -f' the checkout fails like this:
error: invalid object

The git clean makes sure, among other things, that the worktree is clean even in the case the previous checkout contained a submodule that is no longer present in the new checkout. Maybe there is another command that ensures this, but I don't know about yet.
Maybe I am missing something, but as I see, for a package with submodule, if a submodule is removed from the upstream project, it would trigger a ditch+restart too. Again, I think this is a bit extreme (more than git clean -ffdx) but not wrong.

In order to detect broken repos (and trigger the ditch) we could start the script by running fsck. Right?

Do you have a local patch for this ditch+restart? Or should I create one?

On 2018-04-13 15:28 -0300, Ricardo Martincoski spake thusly:

Quoted 6 lines · Yann E. MORIN

Quoted 1 line

On 2018-04-12 06:28 -0300, Ricardo Martincoski spake thusly:

A way to avoid this, once the script entered the directory, is to force the git-dir:

$ git --git-dir=.git <command>
This command would fail for a too-much-broken repo.

We recently reverted use of -C becasue it was not supported by old git versions. What is the oldest git version to support --git-dir? It seems it was introduced in v.1.4.2, dated 2006-08-12. Is that old enough that all the distros we care about have that version or later?

Quoted 2 lines

In order to detect broken repos (and trigger the ditch) we could start the script by running fsck. Right?

So, basically, we would do something along those lines:

    _git() {
        eval ${GIT} --git-dir=.git "${@}"
    }

    _git fsck || { rm -rf ${git_cache}; exec "${0}" "${@}"; exit 1; }
    _git fetch
    _git fetch -t
    _git checkout ${ref}
    _git clean -dX
    _git checkout -- .
    _git submodules update

Of course, that would require using appropriate options to fsck to bail out.

But what to do if any if the following actions fails? Should we simply exit, or should we clean up and clone again?

I can see shere that could go wrong: the ref does not exist, so the first checkout fails, so we ditch the repository, clone again, and checkout again fails...

My opinion, for what it's worth, is to clan only on the fsck. Any other failure should be left to the user to handle. Maybe with just a little message saying something like:

    If you are not sure how to solve this, remove ${cache_dir}.

Thoughts?

Quoted 2 lines

Of course, that would require using appropriate options to fsck to bail out.

Yes. Some interesting ones I listed below:
--no-dangling: AFAIK they cause no harm;
--no-reflogs: not sure;
--full: this is for the case someone is abusing the git cache with alternates,
should we care?

I find out that different git versions also use different sets of errors to return non-zero code. Not so different, don't worry.
Old versions return 0 for few errors (but the error: message is printed) git 1.8.3: return code 0 for a missing sha1 object pointed by a tag, printing:
error: refs/tags/tag_b does not point to a valid object!
git 2.14.1: return code 2 in the same case
error: refs/tags/tag_b: invalid sha1 pointer 1f95d47cc18a9ed4e1eab9b71fe2009c9555448d
BUT, as we always do a fetch before checkout, the fetch fixes it!
So again, we are good. I don't think it needs extra code.

Quoted 13 lines

But what to do if any if the following actions fails? Should we simply exit, or should we clean up and clone again?

I can see shere that could go wrong: the ref does not exist, so the first checkout fails, so we ditch the repository, clone again, and checkout again fails...

My opinion, for what it's worth, is to clan only on the fsck. Any other failure should be left to the user to handle. Maybe with just a little message saying something like:

    If you are not sure how to solve this, remove ${cache_dir}.

Quoted 1 line

Thoughts?

I agree, to clean only on the fsck is better.

The user-friendly message is not *needed* IMO.
But if you find an easy way to do that, it would be nice to have. Maybe in _git() but not in the fsck case.
Maybe a trap? Not sure.

On 2018-04-15 23:54 -0300, Ricardo Martincoski spake thusly:

Quoted 1 line · Yann E. MORIN

[--SNIP--]

Quoted 3 lines

Quoted 2 lines

Of course, that would require using appropriate options to fsck to bail out.

Quoted 5 lines

Yes. Some interesting ones I listed below:
--no-dangling: AFAIK they cause no harm;
--no-reflogs: not sure;
--full: this is for the case someone is abusing the git cache with alternates,
should we care?

So I played with git-fsck in quite a few setups, and it irremediably exits with a non-zero status when something is wrong.

However, on my machine, it takes about 8min to fsck the Linux git tree, which is a huge amount of time (much more than it takes to do the build of said kernel).

So, I am a bit reluctant at using git-fsck.

I'm trying to see if we can find a faster way to detect if the git tree is sane or not. After all, we only need a sanity check, not repairing. If it is not sane, we ditch it and reclone.

So, maybe just running "git status" or any other fast action should probably be enough.

Thoughts?

Quoted 9 lines

I find out that different git versions also use different sets of errors to return non-zero code. Not so different, don't worry.
Old versions return 0 for few errors (but the error: message is printed) git 1.8.3: return code 0 for a missing sha1 object pointed by a tag, printing:
error: refs/tags/tag_b does not point to a valid object!
git 2.14.1: return code 2 in the same case
error: refs/tags/tag_b: invalid sha1 pointer 1f95d47cc18a9ed4e1eab9b71fe2009c9555448d
BUT, as we always do a fetch before checkout, the fetch fixes it!
So again, we are good. I don't think it needs extra code.

Hmm... As long as the repo is not _broken_ and we can recover with the fetch / clean/ checkout actions, I'm OK.

Quoted 13 lines

Quoted 13 lines

But what to do if any if the following actions fails? Should we simply exit, or should we clean up and clone again?

I can see shere that could go wrong: the ref does not exist, so the first checkout fails, so we ditch the repository, clone again, and checkout again fails...

My opinion, for what it's worth, is to clan only on the fsck. Any other failure should be left to the user to handle. Maybe with just a little message saying something like:

    If you are not sure how to solve this, remove ${cache_dir}.

Quoted 8 lines

Quoted 1 line

Thoughts?

I agree, to clean only on the fsck is better.

The user-friendly message is not *needed* IMO.
But if you find an easy way to do that, it would be nice to have. Maybe in _git() but not in the fsck case.
Maybe a trap? Not sure.

Well, a trap is pretty simple, yes.

But let's leave that out for now, we already have enough complexity to handle...

Regards,
Yann E. MORIN.

Signature · 10 lines

.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
buildroot at busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot

Ricardo, All,To all autobuilder runners: could you create a tarball of the failed dtv-scan-tables git tree on yuour autobuilders, from each of your instances, please?

I'm not sure to understand what you reallly need, could you please clarify ?

Please! ;-)

On 2018-04-15 23:54 -0300, Ricardo Martincoski spake thusly:

Quoted 1 line · Yann E. MORIN

[--SNIP--]

Quoted 8 lines

Quoted 2 lines

Of course, that would require using appropriate options to fsck to bail out.

Yes. Some interesting ones I listed below:
--no-dangling: AFAIK they cause no harm;
--no-reflogs: not sure;
--full: this is for the case someone is abusing the git cache with alternates,
should we care?

So I played with git-fsck in quite a few setups, and it irremediably exits with a non-zero status when something is wrong.

However, on my machine, it takes about 8min to fsck the Linux git tree, which is a huge amount of time (much more than it takes to do the build of said kernel).

So, I am a bit reluctant at using git-fsck.

I'm trying to see if we can find a faster way to detect if the git tree is sane or not. After all, we only need a sanity check, not repairing. If it is not sane, we ditch it and reclone.

So, maybe just running "git status" or any other fast action should probably be enough.

Thoughts?

_git() {
eval ${GIT} --git-dir=.git "${@}"
}
_git init
_git remote
_git fetch
_git fetch -t

_git checkout -f ${ref}
if failed
if git log -1 ref
ditch and restart
else
bail out
_git clean -dX
_git submodules update

Hello,

Quoted 7 lines · Ricardo Martincoski

_git() {
eval ${GIT} --git-dir=.git "${@}"
}
_git init
_git remote
_git fetch
_git fetch -t

Quoted 8 lines

_git checkout -f ${ref}
if failed
if git log -1 ref
ditch and restart
else
bail out
_git clean -dX
_git submodules update

But let's take a step back...

A dirty worktree is not so hard to find in practice. It happens when certain git commands are abruptly interrupted. As Thomas wrote in another thread, his autobuilder instance is sometimes restarted with the script running. Therefore the abrupt interruption will eventually occur during a checkout, possibly causing a dirty worktree.
But the right commands (I believe they are checkout -f, git clean -ffdx) will fix it, assuming the repo is sane.

But a broken repo is usually associated either to hardware failures, especially those repos that cannot be recovered by 'git init' and/or 'git fetch', or to misuse, that from the git perspective is what the autobuilder script is currently doing by eventually removing random individual files from inside a .git dir.

And the implementation of ditch+restart seems to be more complicated than it seemed to me before.

So, back to your proposal for a broken repo:
either:
- ditch+restart
- bail out

I changed my mind and I think we should adopt bail out for now. It will keep the script simple, yet robust. ditch+restart can be implemented later if we want/need.

Thoughts?

    _git() {
        eval ${GIT} --git-dir=.git "${@}"

}
_git init
_git remote
_git fetch
_git fetch -t
_git checkout -f ${ref}
_git clean -dX
_git submodules update

A simple improvement to the script to recover few broken repo scenarios is to make git init unconditional. From its help:
"Running git init in an existing repository is safe. It will not overwrite things that are already there."
For example, running 'git init' in a repo without .git/HEAD fixes it.

Here is a squashed, no commit log, no comments on the code, hackish preview:
https://gitlab.com/RicardoMartincoski/buildroot/commit/f4e20bb62b761bb6c82a29acbd2f8d56d90e4e9c

On 17-04-18 09:04, Ricardo Martincoski wrote:

Quoted 2 lines

Here is a squashed, no commit log, no comments on the code, hackish preview:
https://gitlab.com/RicardoMartincoski/buildroot/commit/f4e20bb62b761bb6c82a29acbd2f8d56d90e4e9c

Looks good to me, with the following mods:

* use GIT_DIR
* add comment above git init explaining that it is safe

* explain why -ff is needed for git clean
* Removing the shallow fetch should be done in a separate patch (I'm not entirely convinced yet that it is needed).

Quoted 3 lines · Ricardo Martincoski

Quoted 2 lines

Here is a squashed, no commit log, no comments on the code, hackish preview:
https://gitlab.com/RicardoMartincoski/buildroot/commit/f4e20bb62b761bb6c82a29acbd2f8d56d90e4e9c

Quoted 5 lines

Looks good to me, with the following mods:

* use GIT_DIR
* add comment above git init explaining that it is safe
* explain why -ff is needed for git clean

Quoted 2 lines

* Removing the shallow fetch should be done in a separate patch (I'm not entirely convinced yet that it is needed).

These 2 are already covered by Yann's WIP series, with comments and proper commit logs:
https://git.buildroot.org/~ymorin/git/buildroot/log/?h=yem/git-robust

Ricardo, Arnout, All,

On 2018-04-17 05:56 -0300, Ricardo Martincoski spake thusly:

Quoted 1 line · Arnout Vandecappelle

Quoted 2 lines

Quoted 2 lines

* Removing the shallow fetch should be done in a separate patch (I'm not entirely convinced yet that it is needed).

IIRC, Ricardo explained the case in a previous message: there are cases where we may miss blobs in a tree.

But anyway, the current code only works by chance. Let's assume you want to use tag v4.17-rc1 from Linus' tree. The code would currently do something like (without an existing git cache)

    mkdir git
    git init git
    cd git
    git remote add origin https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
    git fetch origin --depth 1 v4.17-rc1
    if ! git fetch origin v4.17-rc1:v4.17-rc1 ; then
        echo "warning"
    fi
    git checkout v4.17-rc1

The if-block is initially there to fetch special refs created by things like gerrit or github PRs. If you do the above, you get the warning, i.e. the if-condition is false, i.e. fetching the tag that way fails with exit-code 1, and this message:

    error: cannot update the ref 'refs/heads/v4.17-rc1': Trying to write
    non-commit object a79c33a10ce2764c62fb8af6cbb571752d55c1c0 to branch
    refs/heads/v4.17-rc1
    From https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux
     ! [new tag]         v4.17-rc1  -> v4.17-rc1  (unable to update local ref)
     * [new tag]         v4.17-rc1  -> v4.17-rc1

But since it is in a if-ciondition, we just print a warning, and continue.

Then and the checkout succeeds.

But! If you didn't try to fetch the special refs, the checkout would have failed:

error: pathspec 'v4.17-rc1' did not match any file(s) known to git.

I.e. we have code that just happens to work by chance (or rather, by side-effects and luck).

So, even if the missing-blob problem could be solved another way, our shallow fetch is anyway already borked, but we were lucky so far not to fall for it...

I still think that this shalow-fetch dance is too risky, and that having a sane git cache helps so much more in the long run, so much so that we should just ditch the shallow fetch support.

With any version of git, after the git cache was once populated using a shallow fetch, a full fetch succeeds but doesn't download the commits behind the reference that was fetched with --depth 1. In this case the download fails like this:
Fetching all references
Could not fetch special ref 'sha1'; assuming it is not special.
fatal: reference is not a tree: sha1

One scenario in buildroot development that would trigger this sequence is for example when a package is bumped on master branch, some users create the git cache at this time, then the bump is reverted.

Another scenario is when giving maintenance to a product version that uses one version of buildroot using the same build farm that uses a newer version of buildroot.

Thanks a lot Ricardo for all this work on the download issues, much appreciated.

One question on this specific commit.

Quoted 8 lines · Ricardo Martincoski

With any version of git, after the git cache was once populated using a shallow fetch, a full fetch succeeds but doesn't download the commits behind the reference that was fetched with --depth 1. In this case the download fails like this:
Fetching all references
Could not fetch special ref 'sha1'; assuming it is not special.
fatal: reference is not a tree: sha1

Quoted 7 lines

One scenario in buildroot development that would trigger this sequence is for example when a package is bumped on master branch, some users create the git cache at this time, then the bump is reverted.

Another scenario is when giving maintenance to a product version that uses one version of buildroot using the same build farm that uses a newer version of buildroot.

In the light of this, wouldn't it be simpler to stop doing shallow clones at all ? A shallow clone did make a lot of sense back when we didn't cache the Git repositories. But now that we are caching the results, does it make sense to keep the complexity of the code to use shallow clones ?

Note that this is really an open question, there are possibly some convincing argument for us to keep a shallow clone strategy.

On 2018-04-12 13:28 +0200, Thomas Petazzoni spake thusly:

Quoted 14 lines

Quoted 7 lines

One scenario in buildroot development that would trigger this sequence is for example when a package is bumped on master branch, some users create the git cache at this time, then the bump is reverted.

Another scenario is when giving maintenance to a product version that uses one version of buildroot using the same build farm that uses a newer version of buildroot.

In the light of this, wouldn't it be simpler to stop doing shallow clones at all ? A shallow clone did make a lot of sense back when we didn't cache the Git repositories. But now that we are caching the results, does it make sense to keep the complexity of the code to use shallow clones ?

Quoted 2 lines

Note that this is really an open question, there are possibly some convincing argument for us to keep a shallow clone strategy.

Indeed, I don't think it makes sense anymore...

So, intead of fixing it, let's just do full clones now.

Quoted 16 lines

On 2018-04-12 13:28 +0200, Thomas Petazzoni spake thusly:

Quoted 14 lines

Quoted 7 lines

One scenario in buildroot development that would trigger this sequence is for example when a package is bumped on master branch, some users create the git cache at this time, then the bump is reverted.

Another scenario is when giving maintenance to a product version that uses one version of buildroot using the same build farm that uses a newer version of buildroot.

In the light of this, wouldn't it be simpler to stop doing shallow clones at all ? A shallow clone did make a lot of sense back when we didn't cache the Git repositories. But now that we are caching the results, does it make sense to keep the complexity of the code to use shallow clones ?

Quoted 6 lines

Quoted 2 lines

Note that this is really an open question, there are possibly some convincing argument for us to keep a shallow clone strategy.

Indeed, I don't think it makes sense anymore...

So, intead of fixing it, let's just do full clones now.

I agree it is less needed now than before.

The worst scenario (linux trees) is covered by:
- the use of tarballs from GitHub in the defconfigs, for newcomers who sometimes just want to use a defconfig, add few packages and use the image in a new board;
- in the long run, due to the git cache feature.

But I think people will miss this feature. At least I will.

In the case of someone always using tags from a linux repo, this person would never need to download the full repo if the shallow fetch remains.

When someone wants just to try a package, for example, a debug tool, which maybe will never be used again, this person would need to do a full clone, instead of potentially do a shallow fetch.

The repo for some packages can be located on slow servers (not necessarily a package in the tree, it can be on a br2-external).

And the least appealing argument: I was willing to recreate
http://patchwork.ozlabs.org/patch/702006/ on top of the new infra.

Quoted 6 lines

The worst scenario (linux trees) is covered by:
- the use of tarballs from GitHub in the defconfigs, for newcomers who sometimes just want to use a defconfig, add few packages and use the image in a new board;
- in the long run, due to the git cache feature.

Quoted 1 line

But I think people will miss this feature. At least I will.

Except for the linux git tree and a very few other packages we get from git, most git tree are reasonably sized, so that the overhead of doign a full clone is not much as comp[ared to the shallow clone.

And since it also makes our code much simpler and more maintaineable,

that's still a win for me...

Quoted 7 lines

In the case of someone always using tags from a linux repo, this person would never need to download the full repo if the shallow fetch remains.

When someone wants just to try a package, for example, a debug tool, which maybe will never be used again, this person would need to do a full clone, instead of potentially do a shallow fetch.

Quoted 5 lines

The repo for some packages can be located on slow servers (not necessarily a package in the tree, it can be on a br2-external).

And the least appealing argument: I was willing to recreate
http://patchwork.ozlabs.org/patch/702006/ on top of the new infra.

I'm still not convinced any more than I previously was. I.e. I still think we should go with full clone.