Git clone of a bundle fails, but works (somewhat) when run with strace

tl;dr:

- git bundle create without <git-rev-list-args> gives git rev-list help, then dies. Should point out missing <git-rev-list-args> instead.
- `git clone <bundle> <dir> gives "ERROR: Repository not found."
- strace ... git clone <bundle> <dir> (magically) appears to work but cannot checkout files b/c of nonexistent ref.
- Heisenbug? Race condition?
- Zaphod Beeblebrox has left the building, sulking.

Full description:

When I try to clone from a bundle created from a local repository, git clone <bundle> <dir> fails with: "ERROR: Repository not found. fatal: Could not read from remote repository." unless I run it with strace.

OS: Arch Linux (rolling release)
Git versions: 1.8.1.3 and git://github.com/git.git master@02339dd

Steps to reproduce:

$ # Clone the Linux kernel repository
$ git clone git://github.com/torvalds/linux.git
Cloning into 'linux'...
remote: Counting objects: 2841147, done.
remote: Compressing objects: 100% (670736/670736), done.
remote: Total 2841147 (delta 2308339), reused 2657487 (delta 2143012)
Receiving objects: 100% (2841147/2841147), 797.62 MiB | 2.59 MiB/s, done.
Resolving deltas: 100% (2308339/2308339), done.
Checking out files: 100% (41521/41521), done.
$ cd linux
$ git branch -av
* master                323a72d Merge git://git.kernel.org/pub/scm/linux/kernel/git/davem/net
  remotes/origin/HEAD   -> origin/master
  remotes/origin/master 323a72d Merge 
git://git.kernel.org/pub/scm/linux/kernel/git/davem/net

$ # Try to create a bundle
$ git bundle create ../linux.bundle
usage: git rev-list [OPTION] <commit-id>... [ -- paths... ]
  limiting output:
    --max-count=<n>
    --max-age=<epoch>
    --min-age=<epoch>
    --sparse
    --no-merges
    --min-parents=<n>
    --no-min-parents
    --max-parents=<n>
    --no-max-parents
    --remove-empty
    --all
    --branches
    --tags
    --remotes
    --stdin
    --quiet
  ordering output:
    --topo-order
    --date-order
    --reverse
  formatting output:
    --parents
    --children
    --objects | --objects-edge
    --unpacked
    --header | --pretty
    --abbrev=<n> | --no-abbrev
    --abbrev-commit
    --left-right
  special purpose:
    --bisect
    --bisect-vars
    --bisect-all
error: rev-list died
$ # IMHO the error should refer to the usage of `git bundle` with a proper basis, not `git rev-list`.
$ # Also nothing should die loudly because of a missing parameter.
$ git bundle create ../linux.bundle master
Counting objects: 2836191, done.
Delta compression using up to 2 threads.
Compressing objects: 100% (505627/505627), done.
Writing objects: 100% (2836191/2836191), 796.59 MiB | 16.23 MiB/s, done.
Total 2836191 (delta 2304454), reused 2834391 (delta 2303193)

$ # Try to clone a new repository from the bundle
$ cd ..
$ git clone linux.bundle linuxfrombundle
Cloning into 'linuxfrombundle'...
ERROR: Repository not found.
fatal: Could not read from remote repository.

Please make sure you have the correct access rights and the repository exists.

$ git clone linux.bundle -b master linuxfrombundle
Cloning into 'linuxfrombundle'...
ERROR: Repository not found.
fatal: Could not read from remote repository.

Please make sure you have the correct access rights and the repository exists.

# Try again using strace

$ # (Replace /dev/null with a filename if you really want to try and debug this, or if you just want to torture your hard drive ;) )
$ strace -o /dev/null git clone linux.bundle linuxfrombundle
Cloning into 'linuxfrombundle'...
Receiving objects: 100% (2836191/2836191), 796.59 MiB | 24.64 MiB/s, done.
Resolving deltas: 100% (2304454/2304454), done.
warning: remote HEAD refers to nonexistent ref, unable to checkout.

$ # Let's have a look at what we cloned
$ cd linuxfrombundle
$ ls
$ git branch -av
  remotes/origin/master 323a72d Merge 
git://git.kernel.org/pub/scm/linux/kernel/git/davem/net
$ git checkout master
Checking out files: 100% (41521/41521), done.
Branch master set up to track remote branch master from origin.
Already on 'master'
$ git branch -av
* master                323a72d Merge git://git.kernel.org/pub/scm/linux/kernel/git/davem/net
  remotes/origin/master 323a72d Merge 
git://git.kernel.org/pub/scm/linux/kernel/git/davem/net
$ # Where's my HEAD?

Kind regards,

Alain Kalker

For those who like to "save the trees" (source code or otherwise), here is a much simplified test case:

$ # Create test repository with a single commit in it
$ mkdir testrepo
$ cd testrepo
$ git init
$ echo Test > test.txt
$ git add test.txt
$ git commit -m "Add test.txt"

$ # Create bundle
$ git bundle create ../testrepo.bundle master

$ # Try to clone from bundle
$ cd ..
$ git clone testrepo.bundle testrepofrombundle

$ # Clone from bundle, wrapped with strace
$ strace -f -o /dev/null git clone testrepo.bundle testrepofrombundle

$ # Examine cloned repository
$ cd testrepofrombundle
$ ls
$ git branch -av
$ git checkout master
$ git branch -av
$ # Where's my HEAD?

After trying to bisect this using bisect start; bisect good v1.5.1; git bisect bad HEAD; git bisect run ..test.sh:

---test.sh---
#!/bin/sh

make clean
make || return 125
GIT=$(pwd)/git

cd /tmp
rm -rf testrepo
mkdir testrepo
cd testrepo
$GIT init
echo test > test.txt
$GIT add test.txt
$GIT commit -m "Add test.txt"
$GIT bundle create ../testrepo.bundle master || return 125 cd ..

rm -rf testrepofrombundle
$GIT clone testrepo.bundle testrepofrombundle || return 1
---
I was unable to find a bad revision.
After a lot more searching I found that I had git aliased to hub, a tool used to make Github actions easier.
Eliminating hub from the equation resolved most problems. The only ones remaining are the confusing error message from git bundle create and the "missing HEAD" (you can interpret that in different ways) ;-)

P.S. I hereby promise to _never_ _ever_ alias git to something else and then post a Git bug about that "something else" on this ML.

Sorry to have wasted your time,

Alain

Great.

I do not see any odd message from "bundle create" in the recipe above. Mine says:

$ git bundle create ../repo.bundle master
Counting objects: 3, done.
Writing objects: 100% (3/3), 209 bytes, done.
Total 3 (delta 0), reused 0 (delta 0)

What you _might_ be seeing is the fact that the invocation above is likely to be running two different versions of git under the hood. "git bundle" will invoke "git rev-list", and it will use the first git in your PATH, even if it is not $GIT. The proper way to test an un-installed version of git is to use $YOUR_GIT_BUILD/bin-wrappers/git, which will set up environment variables sufficient to make sure all sub-gits are from the same version. Sometimes mixing versions can have weird results (e.g., the new "git bundle" expects "rev-list" to have a particular option, but the older version does not have it).

Secondly, I do get the same warning about HEAD:

  $ git clone repo.bundle repofrombundle
  Cloning into 'repofrombundle'...
  Receiving objects: 100% (3/3), done.
  warning: remote HEAD refers to nonexistent ref, unable to checkout.

but that warning makes sense. You did not create a bundle that contains HEAD, therefore when we clone it, we do not know what to point HEAD to. You probably wanted "git bundle create ../repo.bundle --all" which includes both "master" and "HEAD".

It would be slightly more accurate to say "the remote HEAD does not exist", rather than "refers to nonexistent ref". It would perhaps be nicer still for "git clone" to make a guess about the correct HEAD when one is not present (especially in the single-branch case, it is easy to make the right guess).

Patches welcome. In the meantime, you can clone with "-b master" to tell it explicitly, or you can "git checkout master" inside the newly-cloned repository.

-Peff

Thanks for the very useful tip! I will try to remember to use this for any future testing.

That explains it, thanks! Maybe this could be added as an example to the documentation for git bundle? People looking at man git-bundle hoping to use it as one possible way to make a backup of a Git repository might not realize right away that --all is the way to specify all refs, like with git log.

Thanks again for your help, very welcome :-)

Alain

From: "Alain Kalker" <[email protected]>
Sent: Sunday, February 17, 2013 1:01 PM

[...]

I had posted a documentation patch back in Setemeber last year,
http://thread.gmane.org/gmane.comp.version-control.git/205887/focus=205897,
however Junio highlighted some additional concerns that I wasn't able to respond to at the time.

It may be worth resurrecting once the concerns have been addressed.

A saner thing to do, instead of explaining away the lack of HEAD as "the creator of the bundle did not bother to name it", might be to automatically add an artificial HEAD to the resulting bundle when the arguments given to specify the "range" do not have any negative ones (because by definition such a bundle is unsuitable for use with "git clone" [*1*]), and HEAD is not among the refs.

The heuristics to pick what to record as the artificial HEAD could vary, though. Without thinking things through...

* When only one positive ref is given, use it (sort of obvious);

* When two or more positive refs are given, and the current branch is one of them, use that;

* Otherwise, pick the "first" positive ref given from the command line.

perhaps?

[Footnote]

*1* Strictly speaking, this condition could be loosened, as long as cloning side uses an appropriate --depth, but I do not know such an "advanced use case" needs a hand-holding change to add a HEAD that was not asked by the user.

I'd like to revive this discussion and submit a patch, as I just spent significant time wondering why git clone failed. It's been a while, so I'll
summarize: when you make a git bundle without including HEAD explicitly, then clone from that bundle, Git throws a warning and leaves you with a broken HEAD.

I do not agree that the warning makes sense. It implies that HEAD exists but is invalid. In reality, no ref is referred to by HEAD in the first place. Furthermore, .git/HEAD in the clone is "autocorrected" to be refs/heads/master, so the error message is even more misleading.

It's like saying "Our CEO's guitar is actually an air guitar", then explaining where he stores his guitar, when I don't work in a company in the first place.

Seems sensible at first sight, though it seems orthogonal to the warning message.

Alright :) See below.

Cheers, Bram

From: Bram Geron <[email protected]>
Date: Wed, 28 May 2014 15:54:37 +0100
Subject: [PATCH] Clearer error message when cloning a bundle without a HEAD
ref.

---
 builtin/clone.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

index 9b3c04d..e3c1447 100644
@@ -623,7 +623,7 @@ static int checkout(void)
 
 	head = resolve_refdup("HEAD", sha1, 1, NULL);
 	if (!head) {
-		warning(_("remote HEAD refers to nonexistent ref, "
+		warning(_("no HEAD in remote or HEAD refers to nonexistent ref, "
 			  "unable to checkout.\n"));
 		return 0;
 	}

1.9.1

Thanks.

People around here tend to be quiet until they sufficiently have dug the issue themselves; unless the initial report grossly lack necessary level of details, you may not hear "does not reproduce for me" for some time, so some may still have been scratching their head, and your honestly following-up on your message will save time for them. Thanks again, and happy Gitting ;-)