git pull & git gc

Hello,

I have a local folder with the git-repository (so that its .git/config
contains ([remote "origin"]\n url = git://github.com/git/git.git\nfetch =
+refs/heads/*:refs/remotes/origin/* )

I do there "git pull".

Usually the output is
Already up to date

but since today it prints
Auto packing the repository in background for optimum performance. See "git help gc" for manual housekeeping.
Already up-to-date.

and starts in the background a "git gc --auto" process. This is all fine, however, when the "git gc" process finishes, and I do again "git pull" I get the same message, as above (git gc is again started).

Quoted 11 lines · Дилян Палаузов

Hello,

I have a local folder with the git-repository (so that its .git/config
contains ([remote "origin"]\n url = git://github.com/git/git.git\nfetch =
+refs/heads/*:refs/remotes/origin/* )

I do there "git pull".

Usually the output is
Already up to date

Quoted 8 lines

but since today it prints
Auto packing the repository in background for optimum performance. See "git help gc" for manual housekeeping.
Already up-to-date.

and starts in the background a "git gc --auto" process. This is all fine, however, when the "git gc" process finishes, and I do again "git pull" I get the same message, as above (git gc is again started).

So if you do "git gc --auto" now, does it exit immediately or go through the garbage collection process again (it'll print something)? What does "git count-objects -v" show?

Hello,

# git gc --auto
Auto packing the repository in background for optimum performance.
See "git help gc" for manual housekeeping.

and calls in the background:

25618     1  0 32451   884   1 14:20 ?        00:00:00 git gc --auto
25639 25618 51 49076 49428   0 14:20 ?        00:00:07 git prune --expire

2.weeks.ago

# git count-objects -v
count: 6039

size: 65464
in-pack: 185432
packs: 1

Quoted 6 lines · Дилян Палаузов

Hello,

# git gc --auto
Auto packing the repository in background for optimum performance.
See "git help gc" for manual housekeeping.

Quoted 8 lines

and calls in the background:

25618     1  0 32451   884   1 14:20 ?        00:00:00 git gc --auto
25639 25618 51 49076 49428   0 14:20 ?        00:00:07 git prune --expire

2.weeks.ago

# git count-objects -v
count: 6039

loose number threshold is 6700, unless you tweaked something. But there's a tweak, we'll come back to this.

Quoted 3 lines

size: 65464
in-pack: 185432
packs: 1

Pack threshold is 50, You only have one pack, good

OK back to the "count 6039" above. You have that many loose objects. But 'git gc' is lazier than 'git count-objects'. It assume a flat distribution, and only counts the number of objects in .git/objects/17 directory only, then extrapolate for the total number.

So can you see how many files you have in this directory .git/objects/17? That number, multiplied by 256, should be greater than 6700. If that's the case "git gc" laziness is the problem. If not, I made some mistake in analyzing this and we'll start again.

#ls .git/objects/17/* | wc -l
30

30 * 256 = 7 680 > 6 700

And now? Do I have to run git gc --aggressive ?

If not, I made some mistake in analyzing this and we'll start again.

Quoted 1 line · Duy Nguyen

If not, I made some mistake in analyzing this and we'll start again.

I did make one mistake, the first "gc" should have reduced the number of loose objects to zero. Why didn't it.? I'll come back to this tomorrow if nobody finds out first :)

Quoted 6 lines · Duy Nguyen

Quoted 1 line · Duy Nguyen

If not, I made some mistake in analyzing this and we'll start again.

I did make one mistake, the first "gc" should have reduced the number of loose objects to zero. Why didn't it.? I'll come back to this tomorrow if nobody finds out first :)

Most likely they are not referenced by anything but are younger than 2 weeks.

I saw a similar issue with automatic gc triggering after every operation when I did something equivalent to:

	git add <lots of files>
	git commit
	git reset --hard HEAD^

which creates a log of unreachable objects which are not old enough to be pruned.

Quoted 14 lines · John Keeping

Quoted 6 lines · Duy Nguyen

Quoted 1 line · Duy Nguyen

If not, I made some mistake in analyzing this and we'll start again.

I did make one mistake, the first "gc" should have reduced the number of loose objects to zero. Why didn't it.? I'll come back to this tomorrow if nobody finds out first :)

Most likely they are not referenced by anything but are younger than 2 weeks.

I saw a similar issue with automatic gc triggering after every operation when I did something equivalent to:

Quoted 6 lines

git add <lots of files>
git commit
git reset --hard HEAD^

which creates a log of unreachable objects which are not old enough to be pruned.

Yes, this is almost certainly the problem. Though to be pedantic, the command above will still have a reflog entry, so the objects will be reachable (and packed). But there are other variants that don't leave the objects reachable from even reflogs.

I don't know if there is an easy way around this. Auto-gc's object count is making the assumption that running the gc will reduce the number of objects, but obviously it does not always do so. We could do a more thorough check and find the number of actual packable and prune-able objects. The "prune-able" part of that is easy; just omit objects from the count that are newer than 2 weeks. But "packable" is expensive. You would have to compute reachability by walking from the tips. That can take tens of seconds on a large repo.

Or we could count/estimate the number of loose objects again after repack/prune. Then we can maybe have a way to prevent the next gc that we know will not improve the situation anyway. One option is pack unreachable objects in the second pack. This would stop the next gc, but that would screw prune up because st_mtime info is gone.. Maybe we just save a file to tell gc to ignore the number of loose objects until after a specific date.

Quoted 7 lines · Duy Nguyen

Or we could count/estimate the number of loose objects again after repack/prune. Then we can maybe have a way to prevent the next gc that we know will not improve the situation anyway. One option is pack unreachable objects in the second pack. This would stop the next gc, but that would screw prune up because st_mtime info is gone.. Maybe we just save a file to tell gc to ignore the number of loose objects until after a specific date.

I don't think packing the unreachables is a good plan. They just end up accumulating then, and they never expire, because we keep refreshing their mtime at each pack (unless you pack them once and then leave them to expire, but then you end up with a large number of packs).

Quoted 4 lines

I don't think packing the unreachables is a good plan. They just end up accumulating then, and they never expire, because we keep refreshing their mtime at each pack (unless you pack them once and then leave them to expire, but then you end up with a large number of packs).

Note, sometimes I wish unreachables were packed. Recently, I ended up in a situation where running gc created something like 3GB of data as per du, because I suddenly had something like 600K unreachable objects, each of them, as a loose object, taking at least 4K on disk. This made my .git take 5GB instead of 2GB. That surely didn't feel like garbage collection.

Quoted 11 lines · Mike Hommey

Quoted 4 lines

I don't think packing the unreachables is a good plan. They just end up accumulating then, and they never expire, because we keep refreshing their mtime at each pack (unless you pack them once and then leave them to expire, but then you end up with a large number of packs).

Note, sometimes I wish unreachables were packed. Recently, I ended up in a situation where running gc created something like 3GB of data as per du, because I suddenly had something like 600K unreachable objects, each of them, as a loose object, taking at least 4K on disk. This made my .git take 5GB instead of 2GB. That surely didn't feel like garbage collection.

That's definitely a thing that happens, but it is a bit of a corner case. It's unusual to have such a large number of unreferenced objects all at once.

I don't suppose you happen to remember the details, but would a lower expiration time (e.g., 1 day or 1 hour) have made all of those objects go away? Or were they really from some extremely recent event (of course, "event" here might just have been "I did a full repack right before rewriting history" which would freshen the mtimes on everything in the pack).

Certainly the "loosening" behavior for unreachable objects has corner cases like this, and they suck when you hit one. Leaving the objects packed would be better, but IMHO is not a viable alternative unless somebody comes up with a plan for segregating the "old" objects in a way that they actually expire eventually, and don't just keep getting repacked and freshened over and over.

Keeping a file that says "I ran gc at time T, and there were still N objects left over" is probably the best bet. When the next "gc --auto" runs, if T is recent enough, subtract N from the estimated number of objects. I'm not sure of the right value for "recent enough" there, though. If it is too far back, you will not gc when you could. If it is too close, then you will end up running gc repeatedly, waiting for those objects to leave the expiration window.

I guess leaving a bunch of loose objects around longer than necessary isn't the end of the world. It wastes space, but it does not actively make the rest of git slower (whereas having a large number of packs does impact performance). So you could probably make "recent enough" be "T < now - gc.pruneExpire / 4" or something. At most we would try to gc 4 times before dropping unreachable objects, and for the default period, that's only once every couple days.

Quoted 7 lines

I guess leaving a bunch of loose objects around longer than necessary isn't the end of the world. It wastes space, but it does not actively make the rest of git slower (whereas having a large number of packs does impact performance). So you could probably make "recent enough" be "T < now - gc.pruneExpire / 4" or something. At most we would try to gc 4 times before dropping unreachable objects, and for the default period, that's only once every couple days.

We could simply prune unreachables more aggressively, and it would solve this issue at the root cause, no?

Keeping a file that says "I ran gc at time T, and there were still N objects left over" is probably the best bet. When the next "gc --auto" runs, if T is recent enough, subtract N from the estimated number of objects. I'm not sure of the right value for "recent enough" there, though. If it is too far back, you will not gc when you could. If it is too close, then you will end up running gc repeatedly, waiting for those objects to leave the expiration window.

Quoted 7 lines · Jeff King

Keeping a file that says "I ran gc at time T, and there were still N objects left over" is probably the best bet. When the next "gc --auto" runs, if T is recent enough, subtract N from the estimated number of objects. I'm not sure of the right value for "recent enough" there, though. If it is too far back, you will not gc when you could. If it is too close, then you will end up running gc repeatedly, waiting for those objects to leave the expiration window.

And would not be hard to implement either. git-gc is already prepared to deal with stale gc.pid, which would stop git-gc for a day or so before it deletes gc.pid and starts anyway. All we need to do is check at the end of git-gc, if we know for sure the next 'gc --auto' is a waste, then leave gc.pid behind.

Quoted 9 lines · Duy Nguyen

Quoted 7 lines · Jeff King

Keeping a file that says "I ran gc at time T, and there were still N objects left over" is probably the best bet. When the next "gc --auto" runs, if T is recent enough, subtract N from the estimated number of objects. I'm not sure of the right value for "recent enough" there, though. If it is too far back, you will not gc when you could. If it is too close, then you will end up running gc repeatedly, waiting for those objects to leave the expiration window.

Quoted 5 lines

And would not be hard to implement either. git-gc is already prepared to deal with stale gc.pid, which would stop git-gc for a day or so before it deletes gc.pid and starts anyway. All we need to do is check at the end of git-gc, if we know for sure the next 'gc --auto' is a waste, then leave gc.pid behind.

That omits the "N objects left over" information. Which I think may be useful, because otherwise the rule is basically "don't do another gc at all for X time units". That's OK for most use, but it has its own corner cases.

E.g., imagine you are doing an SVN import that does an auto-gc check every 1000 commits. You have some unreferenced objects in your repository. After the first 1000 commits, we do a gc, and then say "wow, still a lot of cruft; let's block gc for a day". Five minutes later, after another 1000 commits, we run "gc --auto" again. It doesn't run because of the cruft-check, even though there are a _huge_ number of new packable objects.

If the blocker file tells us "7000 extra objects" and we see that there are 17,000 in the repo, then we know it's still worth doing the gc (i.e., we know we that we'll probably end up ignoring the 7000 cruft that didn't get cleaned up last time, but we also know that there are 10,000 new objects).

Quoted 4 lines

That omits the "N objects left over" information. Which I think may be useful, because otherwise the rule is basically "don't do another gc at all for X time units". That's OK for most use, but it has its own corner cases.

True. But saving "N objects left over" in a file also has a corner case. If the user "prune --expire=now" manually, the next 'gc --auto' still thinks we have that many leftovers and keeps delaying gc for some more time. Unless we make 'prune' (or any other commands that delete leftovers) to also delete this file. Yeah maybe saving this info in a file will work.

Quoted 6 lines · Duy Nguyen

Quoted 1 line · Duy Nguyen

If not, I made some mistake in analyzing this and we'll start again.

I did make one mistake, the first "gc" should have reduced the number of loose objects to zero. Why didn't it.? I'll come back to this tomorrow if nobody finds out first :)

Most likely they are not referenced by anything but are younger than 2 weeks.

I saw a similar issue with automatic gc triggering after every operation when I did something equivalent to:

git add <lots of files>
git commit
git reset --hard HEAD^

which creates a log of unreachable objects which are not old enough to be pruned.