Skip to contentSkip to conversation

repository corruption when pushing commits to a repository running 'git gc --prune='

DISCUSSION Jan Smets 3 participants 10 messages Jun 15, 2016 Documentation

lore
Share & tools
lore

Jan Smets

author
#0 lore raw

Hi

I've recently expired my reflog to prune loose objects. On a live, bare, repository I ran 'git gc --prune=now'

All clients ended up having problems, they would report:
error: refs/heads/master does not point to a valid object!
Running 'git log' on the bare repo gave : fatal: bad object HEAD

So I did a couple of tries reproducing the issue and with this little test I got a couple of interesting different/similar outcomes

A: mkdir -p /tmp/test; cd /tmp/test ; git init --bare bare ;
B: cd /tmp/test ; git clone bare 1 ; cd 1; touch test; git add test; while true; do date >> test ; git commit -m "$(date)" test ; git push || break; done
A: cd bare; git gc --prune=now (or run it in a loop)

1)

fatal: bad object 22f0351258fa0bb4cd28984b6473510957fbce69
fatal: bad object 22f0351258fa0bb4cd28984b6473510957fbce69 To /tmp/test/bare
! [remote rejected] master -> master (missing necessary objects)

2)

remote: error: unable to write sha1 filename objects/05/cdb51bb0ea3e229734a4b1bddd5ec70fbc65ed: No such file or directory
remote: fatal: failed to write object
error: unpack failed: unpack-objects abnormal exit
To /tmp/test/bare
! [remote rejected] master -> master (unpacker error)

3)

error: Ref refs/heads/master is at e992810e70949e797d33041bf6bc961c9fa4f3e5 but expected 0000000000000000000000000000000000000000
remote: error: Cannot lock ref 'refs/heads/master':
To /tmp/test/bare
! [remote rejected] master -> master (failed to update ref)

4)

remote: error: cannot lock ref 'refs/heads/master': Unable to create '/tmp/test/bare/refs/heads/master.lock': File exists. remote:
remote: If no other git process is currently running, this probably means a
remote: git process crashed in this repository earlier. Make sure no other git
remote: process is running and remove the file manually to continue.
To /tmp/test/bare
! [remote rejected] master -> master (failed to update ref)
error: failed to push some refs to '/tmp/test/bare'

And eventually running 'git gc --prune=now/all' on the bare repository could end with this message:

$ git gc --prune=all
error: refs/heads/master does not point to a valid object!
fatal: bad object refs/heads/master
error: failed to run repack
$ git log -1
fatal: bad object HEAD

This behaviour has been observed with version 2.4.4, 2.5.[12] and 2.6.1

Thank you

- Jan

#1 lore raw
Quoted 8 lines · Jan Smets

Hi

I've recently expired my reflog to prune loose objects. On a live, bare, repository I ran 'git gc --prune=now'

All clients ended up having problems, they would report:
error: refs/heads/master does not point to a valid object!
Running 'git log' on the bare repo gave : fatal: bad object HEAD

[...]

Quoted 4 lines

fatal: bad object 22f0351258fa0bb4cd28984b6473510957fbce69
fatal: bad object 22f0351258fa0bb4cd28984b6473510957fbce69 To /tmp/test/bare
! [remote rejected] master -> master (missing necessary objects)

I think this is the expected behavior. push will create new objects that are not referenced until the ref is updated (at the very end). prune can run concurrently since creating and deleting objects is done in a lockless way (only the ref update needs a lock).

Still, this is not the *documented* behavior, and an easy way to corrupt a repo should be very explicitly documented as very dangerous, and the precautions to take when using it should be explained clearly.

Signature · 2 lines
Matthieu Moy
http://www-verimag.imag.fr/~moy/
#2 lore raw
Quoted 8 lines · Matthieu Moy

I think this is the expected behavior. push will create new objects that are not referenced until the ref is updated (at the very end). prune can run concurrently since creating and deleting objects is done in a lockless way (only the ref update needs a lock).

Still, this is not the *documented* behavior, and an easy way to corrupt a repo should be very explicitly documented as very dangerous, and the precautions to take when using it should be explained clearly.

Yeah, I think this paragraph in the user-manual is the only thing that mentions it:

Anyway, once you are sure that you're not interested in any dangling state, you can just prune all unreachable objects:

------------------------------------------------
$ git prune
------------------------------------------------

and they'll be gone. (You should only run git prune on a quiescent repository--it's kind of like doing a filesystem fsck recovery: you don't want to do that while the filesystem is mounted.

Something along this line, perhaps?

 Documentation/git-gc.txt | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)
Documentation/git-gc.txt +5 -2
index 5223498..fa15104 100644
@@ -63,8 +63,11 @@ automatic consolidation of packs.
 --prune=<date>::
 	Prune loose objects older than date (default is 2 weeks ago,
 	overridable by the config variable `gc.pruneExpire`).
-	--prune=all prunes loose objects regardless of their age.
-	--prune is on by default.
+	--prune=all prunes loose objects regardless of their age (do
+	not use --prune=all unless you know exactly what you are doing.
+	Unless the repository is quiescent, you will lose newly created
+	objects that haven't been anchored with the refs and end up
+	corrupting your repository).  --prune is on by default.
 
 --no-prune::
 	Do not prune any loose objects.
#3 lore raw
Quoted 5 lines · Junio C Hamano
Documentation/git-gc.txt +0 -0
index 5223498..fa15104 100644

Documentation/git-gc.txt automatic consolidation of packs. · line 70

Quoted 10 lines
Documentation/git-gc.txt +5 -2
@@ -63,8 +63,11 @@ automatic consolidation of packs.
 --prune=<date>::
 	Prune loose objects older than date (default is 2 weeks ago,
 	overridable by the config variable `gc.pruneExpire`).
-	--prune=all prunes loose objects regardless of their age.
-	--prune is on by default.
+	--prune=all prunes loose objects regardless of their age (do
+	not use --prune=all unless you know exactly what you are doing.
+	Unless the repository is quiescent, you will lose newly created
+	objects that haven't been anchored with the refs and end up
+	corrupting your repository).  --prune is on by default.

Looks good to me. I think the same should be added to git-prune.txt under --expire.

Signature · 2 lines
Matthieu Moy
http://www-verimag.imag.fr/~moy/
#4 lore raw
Quoted 2 lines · Matthieu Moy

Looks good to me. I think the same should be added to git-prune.txt under --expire.

I thought about it, but decided against it, as the command is not even recommended to end users.

#5 lore raw
Quoted 7 lines · Junio C Hamano
Quoted 2 lines · Matthieu Moy

Looks good to me. I think the same should be added to git-prune.txt under --expire.

I thought about it, but decided against it, as the command is not even recommended to end users.

Even non-"end users" can shoot themselves in the foot. Someone writting a script using "git prune" may actually shoot in other people's foot ...

Signature · 2 lines
Matthieu Moy
http://www-verimag.imag.fr/~moy/
#6 lore raw
Quoted 4 lines · Matthieu Moy
Quoted 2 lines · Junio C Hamano

Matthieu Moy <[email protected]> writes:

Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

Looks good to me. I think the same should be added to git-prune.txt under --expire.

I thought about it, but decided against it, as the command is not even recommended to end users.

Even non-"end users" can shoot themselves in the foot. Someone writting a script using "git prune" may actually shoot in other people's foot ...

Sure, patches welcome.

To be honest, I felt that

--expire <time>::
Only expire loose objects older than <time>.

is sufficiently clear for these people.

#7 lore raw
In reply to Jan Smets
Quoted 6 lines · Jan Smets

2)

remote: error: unable to write sha1 filename objects/05/cdb51bb0ea3e229734a4b1bddd5ec70fbc65ed: No such file or directory
remote: fatal: failed to write object

If I read correctly, this happens when we move the temp file to its permanent position.

This looks like a race between deleting objects/05/ and creating objects/05/cdb51bb0ea3e229734a4b1bddd5ec70fbc65ed.

I don't understand how this is possible, since the temporary file and the final one are in the same directory, so deleting the directory should fail.

What am I missing?

Signature · 2 lines
Matthieu Moy
http://www-verimag.imag.fr/~moy/
#8 lore raw
Quoted 8 lines · Matthieu Moy

This looks like a race between deleting objects/05/ and creating objects/05/cdb51bb0ea3e229734a4b1bddd5ec70fbc65ed.

I don't understand how this is possible, since the temporary file and the final one are in the same directory, so deleting the directory should fail.

What am I missing?

1. tmp_obj_* gets created in objects/05/ by "git unpack-objects";

2. prune_cruft() comes, notices tmp_obj_* that is sufficiently old, decides to remove in "git prune".

3. prune_subdir() comes, notices objects/05/ is now empty, does

    rmdir().

4. "git unpack-objects" closes the file whose inode was originally

    associated with the path objects/05/tmp_obj_*;

5. "git unpack-objects" calls finalize_object_file() and tries to link(2) or rename(2) it to its final name. It does not find the source of the link or rename and says "oops, no such file".

#9 lore raw
Quoted 9 lines · Junio C Hamano
Quoted 7 lines · Matthieu Moy

This looks like a race between deleting objects/05/ and creating objects/05/cdb51bb0ea3e229734a4b1bddd5ec70fbc65ed.

I don't understand how this is possible, since the temporary file and the final one are in the same directory, so deleting the directory should fail.

Quoted 6 lines
Quoted 1 line

What am I missing?

1. tmp_obj_* gets created in objects/05/ by "git unpack-objects";

2. prune_cruft() comes, notices tmp_obj_* that is sufficiently old, decides to remove in "git prune".

Ah, OK, the real problem is the temporary file removal. Thanks.

Signature · 2 lines
Matthieu Moy
http://www-verimag.imag.fr/~moy/