Skip to contentSkip to conversation

"disabling bitmap writing, as some objects are not being packed"?

PATCH In review David Turner 6 participants 21 messages Dec 16, 2016 – Feb 9, 2017 Kbuild

lore
Share & tools
lore
#0 lore raw

I'm a bit confused by the message "disabling bitmap writing, as some objects are not being packed". I see it the my gc.log file on my git server.

1. Its presence in the gc.log file prevents future automatic garbage collection. This seems bad. I understand the desire to avoid making things worse if a past gc has run into issues. But this warning is non-fatal; the only consequence is that many operations get slower. But a lack of gc when there are too many packs causes that consequence too (often a much worse slowdown than would be caused by the missing bitmap).

So I wonder if it would be better for auto gc to grep gc.log for fatal errors (as opposed to warnings) and only skip running if any are found. Alternately, we could simply put warnings into gc.log.warning and reserve gc.log for fatal errors. I'm not sure which would be simpler.

2. I don't understand what would cause that message. That is, what bad thing am I doing that I should stop doing? I've briefly skimmed the code and commit message, but the answer isn't leaping out at me.

#1 lore raw
Quoted 8 lines · David Turner

1. Its presence in the gc.log file prevents future automatic garbage collection. This seems bad. I understand the desire to avoid making things worse if a past gc has run into issues. But this warning is non-fatal; the only consequence is that many operations get slower. But a lack of gc when there are too many packs causes that consequence too (often a much worse slowdown than would be caused by the missing bitmap).

Quoted 4 lines

So I wonder if it would be better for auto gc to grep gc.log for fatal errors (as opposed to warnings) and only skip running if any are found. Alternately, we could simply put warnings into gc.log.warning and reserve gc.log for fatal errors. I'm not sure which would be simpler.

Without thinking too hard on it, that seems like the appropriate solution to me, too.

Quoted 3 lines

2. I don't understand what would cause that message. That is, what bad thing am I doing that I should stop doing? I've briefly skimmed the code and commit message, but the answer isn't leaping out at me.

Do you have alternates and are using --local? Do you have .keep packs and have set repack.packKeptObjects to false?

There are other ways (e.g., an incremental repack), but I think those are the likely ones to get via "git gc".

-Peff

#2 lore raw
In reply to David Turner
Quoted 12 lines · David Turner

I'm a bit confused by the message "disabling bitmap writing, as some objects are not being packed". I see it the my gc.log file on my git server.

1. Its presence in the gc.log file prevents future automatic garbage collection. This seems bad. I understand the desire to avoid making things worse if a past gc has run into issues. But this warning is non-fatal; the only consequence is that many operations get slower. But a lack of gc when there are too many packs causes that consequence too (often a much worse slowdown than would be caused by the missing bitmap).

Quoted 4 lines

So I wonder if it would be better for auto gc to grep gc.log for fatal errors (as opposed to warnings) and only skip running if any are found. Alternately, we could simply put warnings into gc.log.warning and reserve gc.log for fatal errors. I'm not sure which would be simpler.

I am not sure if string matching is really a good idea, as I'd assume that these messages are eligible for i18n.

329e6e8794 ("gc: save log from daemonized gc --auto and print it next time", 2015-09-19) wanted to notice that auto-gc is not making progress and used the presense of error messages as a cue. In your case, I think the auto-gc _is_ making progress, reducing number of loose objects in the repository or consolidating many packfiles into one, and the message is only about the fact that packing is punting and not producing a bitmap as you asked, which is different from not making any progress. I do not think log vs warn is a good criteria to tell them apart, either.

In any case, as the error message asks the user to do, the user eventually wants to correct the root cause before removing the gc.log; I am not sure report_last_gc_error() is the place to correct this in the first place.

Quoted 3 lines

2. I don't understand what would cause that message. That is, what bad thing am I doing that I should stop doing? I've briefly skimmed the code and commit message, but the answer isn't leaping out at me.

Enabling bitmap generation for incremental packing that does not cram everything into a single pack is triggering it, I would presume. Perhaps we should ignore -b option in most of the cases and enable it only for "repack -a -d -f" codepath? Or detect that we are being run from "gc --auto" and automatically disable -b? I have a feeling that an approach along that line is closer to the real solution than tweaking report_last_gc_error() and trying to deduce if we are making any progress.

#3 lore raw
Quoted 4 lines · Junio C Hamano
Quoted 3 lines

2. I don't understand what would cause that message. That is, what bad thing am I doing that I should stop doing? I've briefly skimmed the code and commit message, but the answer isn't leaping out at me.

Quoted 8 lines

Enabling bitmap generation for incremental packing that does not cram everything into a single pack is triggering it, I would presume. Perhaps we should ignore -b option in most of the cases and enable it only for "repack -a -d -f" codepath? Or detect that we are being run from "gc --auto" and automatically disable -b? I have a feeling that an approach along that line is closer to the real solution than tweaking report_last_gc_error() and trying to deduce if we are making any progress.

Ah, indeed. I was thinking in my other response that "git gc" would always kick off an all-into-one repack. But "gc --auto" will not in certain cases. And yes, in those cases you definitely would want --no-write-bitmap-index. I think it would be reasonable for "git repack" to disable bitmap-writing automatically when not doing an all-into-one repack.

-Peff

#4 lore raw
Quoted 15 lines · Jeff King
Quoted 4 lines · Junio C Hamano
Quoted 3 lines

2. I don't understand what would cause that message. That is, what bad thing am I doing that I should stop doing? I've briefly skimmed the code and commit message, but the answer isn't leaping out at me.

Quoted 8 lines

Enabling bitmap generation for incremental packing that does not cram everything into a single pack is triggering it, I would presume. Perhaps we should ignore -b option in most of the cases and enable it only for "repack -a -d -f" codepath? Or detect that we are being run from "gc --auto" and automatically disable -b? I have a feeling that an approach along that line is closer to the real solution than tweaking report_last_gc_error() and trying to deduce if we are making any progress.

Quoted 6 lines

Ah, indeed. I was thinking in my other response that "git gc" would always kick off an all-into-one repack. But "gc --auto" will not in certain cases. And yes, in those cases you definitely would want --no-write-bitmap-index. I think it would be reasonable for "git repack" to disable bitmap-writing automatically when not doing an all-into-one repack.

I do not have alternates and am not using --local. Nor do I have .keep packs.

I would assume, based on the documentation, that auto gc would be doing an all-into-one repack:
"If the number of packs exceeds the value of gc.autopacklimit, then
existing packs (except those marked with a .keep file) are consolidated into a single pack by using the -A option of git repack."

I don't have any settings that limit the size of packs, either. And a manual git repack -a -d creates only a single pack. Its loneliness doesn't last long, because pretty soon a new pack is created by an incoming push.

Unless this just means that some objects are being kept loose (perhaps because they are unreferenced)?

#5 lore raw
Quoted 11 lines · David Turner

I would assume, based on the documentation, that auto gc would be doing an all-into-one repack:
"If the number of packs exceeds the value of gc.autopacklimit, then
existing packs (except those marked with a .keep file) are consolidated into a single pack by using the -A option of git repack."

I don't have any settings that limit the size of packs, either. And a manual git repack -a -d creates only a single pack. Its loneliness doesn't last long, because pretty soon a new pack is created by an incoming push.

The interesting code is in need_to_gc():

/*
* If there are too many loose objects, but not too many
* packs, we run "repack -d -l". If there are too many packs,
* we run "repack -A -d -l". Otherwise we tell the caller
* there is no need.
*/
if (too_many_packs())
add_repack_all_option();
else if (!too_many_loose_objects())
return 0;

So if you have (say) 10 packs and 10,000 objects, we'll incrementally pack those objects into a single new pack.

I never noticed this myself because we do not use auto-gc at GitHub at all. We only ever do a big all-into-one repack.

Quoted 2 lines

Unless this just means that some objects are being kept loose (perhaps because they are unreferenced)?

If they're unreferenced, they won't be part of the new pack. You might accumulate loose objects that are ejected from previous packs, which could trigger auto-gc to do an incremental pack (even though it wouldn't be productive, because they're unreferenced!). You may also get them from pushes (small pushes will be exploded into loose objects by default).

-Peff

#6 lore raw

PATCH 1/1 pack-objects: don't warn about bitmaps on incremental pack

When running git pack-objects --incremental, we do not expect to be able to write a bitmap; it is very likely that objects in the new pack will have references to objects outside of the pack. So we don't need to warn the user about it.

This warning was making its way into gc.log because auto-gc will do an incremental repack when there are too many loose objects but not too many packs. When the gc.log was present, future auto gc runs would refuse to run.

---
 builtin/pack-objects.c  |  3 ++-
 t/t5310-pack-bitmaps.sh | 12 ++++++++++++
 2 files changed, 14 insertions(+), 1 deletion(-)
builtin/pack-objects.c +2 -1
index 0fd52bd..96de213 100644
@@ -1083,7 +1083,8 @@ static int add_object_entry(const unsigned char *sha1, enum object_type type,
 	if (!want_object_in_pack(sha1, exclude, &found_pack, &found_offset)) {
 		/* The pack is missing an object, so it will not have closure */
 		if (write_bitmap_index) {
-			warning(_(no_closure_warning));
+			if (!incremental)
+				warning(_(no_closure_warning));
 			write_bitmap_index = 0;
 		}
 		return 0;
t/t5310-pack-bitmaps.sh +12 -0
index b4c7a6f..d81636e 100755
@@ -247,6 +247,18 @@ test_expect_success 'pack-objects respects --incremental' '
 	test_cmp 4.objects objects
 '
 
+test_expect_success 'incremental repack does not create bitmaps' '
+	test_commit 11 &&
+	ls .git/objects/pack/ | grep bitmap >existing_bitmaps &&
+	ls .git/objects/pack/ | grep -v bitmap >existing_packs &&
+	git repack -d 2>err &&
+	test_line_count = 0 err &&
+	ls .git/objects/pack/ | grep bitmap >output &&
+	ls .git/objects/pack/ | grep -v bitmap >post_packs &&
+	test_cmp existing_bitmaps output &&
+	! test_cmp existing_packs post_packs
+'
+
 test_expect_success 'pack with missing blob' '
 	rm $(objpath $blob) &&
 	git pack-objects --stdout --revs <revs >/dev/null
Signature · 1 line
2.8.0.rc4.22.g8ae061a
#7 lore raw

Re: [PATCH] pack-objects: don't warn about bitmaps on incremental pack

Quoted 10 lines · David Turner

When running git pack-objects --incremental, we do not expect to be able to write a bitmap; it is very likely that objects in the new pack will have references to objects outside of the pack. So we don't need to warn the user about it.
[...]

builtin/pack-objects.c +0 -0
index 0fd52bd..96de213 100644

builtin/pack-objects.c static int add_object_entry(const unsigned char *sha1, enum object_type type, · line 1090

Quoted 9 lines
builtin/pack-objects.c +2 -1
@@ -1083,7 +1083,8 @@ static int add_object_entry(const unsigned char *sha1, enum object_type type,
 	if (!want_object_in_pack(sha1, exclude, &found_pack, &found_offset)) {
 		/* The pack is missing an object, so it will not have closure */
 		if (write_bitmap_index) {
-			warning(_(no_closure_warning));
+			if (!incremental)
+				warning(_(no_closure_warning));
 			write_bitmap_index = 0;
 		}
 		return 0;

I agree that the user doesn't need to be warned about it when running "gc --auto", but I wonder if somebody invoking "pack-objects --incremental --write-bitmap-index" ought to be.

In other words, your patch is detecting at a low level that we've been given a nonsense combination of options, but should we perhaps stop passing nonsense in the first place?

Either at the repack level, with something like:

builtin/repack.c +3 -2
index 80dd06b4a2..6608a902b1 100644
@@ -231,8 +231,6 @@ int cmd_repack(int argc, const char **argv, const char *prefix)
 		argv_array_pushf(&cmd.args, "--no-reuse-delta");
 	if (no_reuse_object)
 		argv_array_pushf(&cmd.args, "--no-reuse-object");
-	if (write_bitmaps)
-		argv_array_push(&cmd.args, "--write-bitmap-index");
 
 	if (pack_everything & ALL_INTO_ONE) {
 		get_non_kept_pack_filenames(&existing_packs);
@@ -256,8 +254,11 @@ int cmd_repack(int argc, const char **argv, const char *prefix)
 	} else {
 		argv_array_push(&cmd.args, "--unpacked");
 		argv_array_push(&cmd.args, "--incremental");
+		write_bitmap_index = 0;
 	}
 
+	if (write_bitmaps)
+		argv_array_push(&cmd.args, "--write-bitmap-index");
 	if (local)
 		argv_array_push(&cmd.args,  "--local");
 	if (quiet)

Though that still means we do not warn on:

  git repack --write-bitmap-index

which is nonsense (it is asking for an incremental repack with bitmaps).

So maybe do it at the gc level, like:

builtin/gc.c +8 -1
index 069950d0b4..d3c978c765 100644
@@ -191,6 +191,11 @@ static void add_repack_all_option(void)
 	}
 }
 
+static void add_repack_incremental_option(void)
+{
+	argv_array_push(&repack, "--no-write-bitmap-index");
+}
+
 static int need_to_gc(void)
 {
 	/*
@@ -208,7 +213,9 @@ static int need_to_gc(void)
 	 */
 	if (too_many_packs())
 		add_repack_all_option();
-	else if (!too_many_loose_objects())
+	else if (too_many_loose_objects())
+		add_repack_incremental_option();
+	else
 		return 0;
 
 	if (run_hook_le(NULL, "pre-auto-gc", NULL))

-Peff

#8 lore raw

RE: [PATCH] pack-objects: don't warn about bitmaps on incremental pack

Quoted 16 lines
builtin/gc.c index 069950d0b4..d3c978c765 renamed from builtin/gc.c +0 -0

100644

builtin/gc.c +4 -0
@@ -191,6 +191,11 @@ static void add_repack_all_option(void)
 	}
 }
 
+static void add_repack_incremental_option(void)
+{
+	argv_array_push(&repack, "--no-write-bitmap-index"); }
+
 static int need_to_gc(void)
 {
 	/*

builtin/gc.c static int need_to_gc(void) · line 221

Quoted 10 lines
builtin/gc.c +3 -1
@@ -208,7 +213,9 @@ static int need_to_gc(void)
 	 */
 	if (too_many_packs())
 		add_repack_all_option();
-	else if (!too_many_loose_objects())
+	else if (too_many_loose_objects())
+		add_repack_incremental_option();
+	else
 		return 0;
 
 	if (run_hook_le(NULL, "pre-auto-gc", NULL))

Sure, that's fine.

#9 lore raw
In reply to Junio C Hamano
Quoted 14 lines · Junio C Hamano
Quoted 12 lines · David Turner

I'm a bit confused by the message "disabling bitmap writing, as some objects are not being packed". I see it the my gc.log file on my git server.

1. Its presence in the gc.log file prevents future automatic garbage collection. This seems bad. I understand the desire to avoid making things worse if a past gc has run into issues. But this warning is non-fatal; the only consequence is that many operations get slower. But a lack of gc when there are too many packs causes that consequence too (often a much worse slowdown than would be caused by the missing bitmap).

Quoted 7 lines
Quoted 4 lines

So I wonder if it would be better for auto gc to grep gc.log for fatal errors (as opposed to warnings) and only skip running if any are found. Alternately, we could simply put warnings into gc.log.warning and reserve gc.log for fatal errors. I'm not sure which would be simpler.

I am not sure if string matching is really a good idea, as I'd assume that these messages are eligible for i18n.

And we can't grep for fatal errors anyway. The problem that led to 329e6e8794 was this line

warning: There are too many unreachable loose objects; run 'git
prune' to remove them.

which is not fatal.

Quoted 6 lines

329e6e8794 ("gc: save log from daemonized gc --auto and print it next time", 2015-09-19) wanted to notice that auto-gc is not making progress and used the presense of error messages as a cue. In your case, I think the auto-gc _is_ making progress, reducing number of loose objects in the repository or consolidating many packfiles into one

Yeah the key point is making progress, and to reliably detect that we need some way for all the commands that git-gc executes to tell it about that, git-repack in this particular case but...

Quoted 14 lines

and the message is only about the fact that
packing is punting and not producing a bitmap as you asked, which is different from not making any progress. I do not think log vs warn is a good criteria to tell them apart, either.

In any case, as the error message asks the user to do, the user eventually wants to correct the root cause before removing the gc.log; I am not sure report_last_gc_error() is the place to correct this in the first place.

Quoted 3 lines

2. I don't understand what would cause that message. That is, what bad thing am I doing that I should stop doing? I've briefly skimmed the code and commit message, but the answer isn't leaping out at me.

Quoted 5 lines

Enabling bitmap generation for incremental packing that does not cram everything into a single pack is triggering it, I would presume. Perhaps we should ignore -b option in most of the cases and enable it only for "repack -a -d -f" codepath? Or detect that we are being run from "gc --auto" and automatically disable -b?

... since we have to change down in git-repack for that, perhaps doing this is better. We can pass --auto (or something) to repack to tell it about this special caller, so it only prints something to stderr in serious cases.

Or we detect cases where background gc'ing won't work well and always do it in foreground (e.g. when bitmap generation is enabled).

Quoted 3 lines

I have a feeling that an approach along that line is closer to the real solution than tweaking report_last_gc_error() and trying to deduce if we are making any progress.

Signature · 1 line
Duy
#10 lore raw
Quoted 7 lines · Duy Nguyen

And we can't grep for fatal errors anyway. The problem that led to 329e6e8794 was this line

warning: There are too many unreachable loose objects; run 'git
prune' to remove them.

which is not fatal.

So, speaking of that message, I noticed that our git servers were getting slow again and found that message in gc.log.

I propose to make auto gc not write that message either. Any objections?

#11 lore raw
Quoted 7 lines · David Turner
Quoted 6 lines · Duy Nguyen

And we can't grep for fatal errors anyway. The problem that led to 329e6e8794 was this line

warning: There are too many unreachable loose objects; run 'git
prune' to remove them.

Quoted 6 lines
Quoted 1 line

which is not fatal.

So, speaking of that message, I noticed that our git servers were getting slow again and found that message in gc.log.

I propose to make auto gc not write that message either. Any objections?

Does that really help? auto gc would run more often, but unreachable loose objects are still present and potentially make your servers slow? Should these servers run periodic and explicit gc/prune?

Signature · 1 line
Duy
#12 lore raw
Quoted 10 lines · Duy Nguyen
Quoted 9 lines · David Turner
Quoted 7 lines · Duy Nguyen

And we can't grep for fatal errors anyway. The problem that led to 329e6e8794 was this line

warning: There are too many unreachable loose objects; run 'git
prune' to remove them.

which is not fatal.

Quoted 8 lines
Quoted 4 lines

So, speaking of that message, I noticed that our git servers were getting slow again and found that message in gc.log.

I propose to make auto gc not write that message either. Any objections?

Does that really help? auto gc would run more often, but unreachable loose objects are still present and potentially make your servers slow? Should these servers run periodic and explicit gc/prune?

At least pack files wouldn't accumulate. This is the major cause of slowdown, since each pack file must be checked for each object.

(And, also, maybe those unreachable loose objects are too new to get gc'd, but if we retry next week, we'll gc them).

#13 lore raw
Quoted 20 lines · David Turner
Quoted 10 lines · Duy Nguyen
Quoted 9 lines · David Turner
Quoted 7 lines · Duy Nguyen

And we can't grep for fatal errors anyway. The problem that led to 329e6e8794 was this line

warning: There are too many unreachable loose objects; run 'git
prune' to remove them.

which is not fatal.

Quoted 8 lines
Quoted 4 lines

So, speaking of that message, I noticed that our git servers were getting slow again and found that message in gc.log.

I propose to make auto gc not write that message either. Any objections?

Does that really help? auto gc would run more often, but unreachable loose objects are still present and potentially make your servers slow? Should these servers run periodic and explicit gc/prune?

Quoted 5 lines

At least pack files wouldn't accumulate. This is the major cause of slowdown, since each pack file must be checked for each object.

(And, also, maybe those unreachable loose objects are too new to get gc'd, but if we retry next week, we'll gc them).

I was about to suggest a config option that lets you run auto gc unconditionally, which, I think, is better than suppressing the message. Then I found gc.autoDetach. If you set it to false globally, I think you'll get the behavior you want.

On second thought, perhaps gc.autoDetach should default to false if there's no tty, since its main point it to stop breaking interactive usage. That would make the server side happy (no tty there).

Signature · 1 line
Duy
#14 lore raw
Quoted 3 lines · Duy Nguyen

On second thought, perhaps gc.autoDetach should default to false if there's no tty, since its main point it to stop breaking interactive usage. That would make the server side happy (no tty there).

Sounds like an idea, but wouldn't that keep the end-user coming over the network waiting after accepting a push until the GC completes, I wonder. If an impatient user disconnects, would that end up killing an ongoing GC? etc.

#15 lore raw
Quoted 10 lines · Junio C Hamano
Quoted 3 lines · Duy Nguyen

On second thought, perhaps gc.autoDetach should default to false if there's no tty, since its main point it to stop breaking interactive usage. That would make the server side happy (no tty there).

Sounds like an idea, but wouldn't that keep the end-user coming over the network waiting after accepting a push until the GC completes, I wonder. If an impatient user disconnects, would that end up killing an ongoing GC? etc.

Regardless, it's impolite to keep the user waiting. So, I think we should just not write the "too many unreachable loose objects" message if auto-gc is on. Does that sound OK?

#16 lore raw
Quoted 7 lines · David Turner
Quoted 6 lines · Junio C Hamano
Quoted 3 lines · Duy Nguyen

On second thought, perhaps gc.autoDetach should default to false if there's no tty, since its main point it to stop breaking interactive usage. That would make the server side happy (no tty there).

Quoted 8 lines
Quoted 4 lines

Sounds like an idea, but wouldn't that keep the end-user coming over the network waiting after accepting a push until the GC completes, I wonder. If an impatient user disconnects, would that end up killing an ongoing GC? etc.

Regardless, it's impolite to keep the user waiting. So, I think we should just not write the "too many unreachable loose objects" message if auto-gc is on. Does that sound OK?

I thought the point of that message was to prevent auto-gc from kicking in over and over again due to objects that won't actually get pruned.

I wonder if you'd want to either bump the auto-gc object limit, or possibly reduce the gc.pruneExpire limit to keep this situation from coming up in the first place (or at least mitigating the amount of time it's the case).

-Peff

#17 lore raw
Quoted 18 lines · Jeff King
Quoted 7 lines · David Turner
Quoted 6 lines · Junio C Hamano
Quoted 3 lines · Duy Nguyen

On second thought, perhaps gc.autoDetach should default to false if there's no tty, since its main point it to stop breaking interactive usage. That would make the server side happy (no tty there).

Quoted 8 lines
Quoted 4 lines

Sounds like an idea, but wouldn't that keep the end-user coming over the network waiting after accepting a push until the GC completes, I wonder. If an impatient user disconnects, would that end up killing an ongoing GC? etc.

Regardless, it's impolite to keep the user waiting. So, I think we should just not write the "too many unreachable loose objects" message if auto-gc is on. Does that sound OK?

Quoted 7 lines

I thought the point of that message was to prevent auto-gc from kicking in over and over again due to objects that won't actually get pruned.

I wonder if you'd want to either bump the auto-gc object limit, or possibly reduce the gc.pruneExpire limit to keep this situation from coming up in the first place (or at least mitigating the amount of time it's the case).

Auto-gc might not succeed in pruning objects, but it will at least reduce the number of packs, which is super-important for performance.

I think the intent of automatic gc is to have a git repository be relatively low-maintenance from a server-operator perspective. (Side
note: it's fairly trivial for a user with push access to mess with the check simply by pushing a bunch of objects whose shas start with 17). It seems odd that git gets itself into a state where it refuses to do any maintenance just because at some point some piece of the maintenance didn't make progress.

Sure, I could change my configuration, but that doesn't help the other folks (e.g. https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=813084 ) who run into this.

I have three thoughts on this:

Idea 1: when gc --auto would issue this message, instead it could create a file named gc.too-much-garbage (instead of gc.log), with this message. If that file exists, and it is less than one day (?) old, then we don't attempt to do a full gc; instead we just run git repack -A -d. (If it's more than one day old, we just delete it and continue anyway).

Idea 2 : Like idea 1, but instead of repacking, just smash the existing packs together into one big pack. In other words, don't consider dangling objects, or recompute deltas. Twitter has a tool called "git combine-pack" that does this:
https://github.com/dturner-tw/git/blob/dturner/journal/builtin/combine-pack.c

That's less space-efficient than a true repack, but it's no worse than having the packs separate, and it's a win for read performance because there's no need to do a linear search over N packs to find an object.

Idea 3: As I suggested last time, separate fatal and non-fatal errors. If gc fails because of EIO or something, we probably don't want to touch the disk anymore. But here, the worst consequence is that we waste some processing power. And it's better to occasionally waste processing power in a non-interactive setting than it is to do so when a user will be blocked on it. So non-fatal warnings should go to gc.log, and fatal errors should go to gc.fatal. gc.log won't block gc from running. I think this is my preferred option.

#18 lore raw
Quoted 7 lines · David Turner
Quoted 4 lines

I wonder if you'd want to either bump the auto-gc object limit, or possibly reduce the gc.pruneExpire limit to keep this situation from coming up in the first place (or at least mitigating the amount of time it's the case).

Auto-gc might not succeed in pruning objects, but it will at least reduce the number of packs, which is super-important for performance.

Right, I mean to bump the loose-object limit but keep the gc.autoPackLimit at 50. If you couple that with setting transfer.unpackLimit, then each push creates a single pack, and you repack after 50 pushes.

You don't have to care about loose objects, because you know you only get them when a "gc" ejects loose objects (so they're not as efficient, but nothing actually accesses them; they just hang around until their mtime grace period is up).

Quoted 7 lines

I think the intent of automatic gc is to have a git repository be relatively low-maintenance from a server-operator perspective. (Side
note: it's fairly trivial for a user with push access to mess with the check simply by pushing a bunch of objects whose shas start with 17). It seems odd that git gets itself into a state where it refuses to do any maintenance just because at some point some piece of the maintenance didn't make progress.

In my experience, auto-gc has never been a low-maintenance operation on the server side (and I do think it was primarily designed with clients in mind).

At GitHub we disable it entirely, and do our own gc based on a throttled job queue (one reason to throttle is that repacking is memory and I/O intensive, so you really don't want to a bunch of repacks kicking off all at once). So basically any repo that gets pushed to goes on the queue, and then we pick the worst cases from the queue based on how badly they need packing[1].

I wish regular Git were more turn-key in that respect. Maybe it is for smaller sites, but we certainly didn't find it so. And I don't know that it's feasible to really share the solution. It's entangled with our database (to store last-pushed and last-maintenance values for repos) and our job scheduler.

[1] The "how bad" thing is a heuristic, and we found it's generally proportional to the number of bytes stored in objects _outside_ of the big "main" pack. So 2 big pushes may need maintenance more than 10 tiny pushes, because they have more objects (and our goal with maintenance isn't just saving disk space or avoiding the linear pack search, but having up-to-date bitmaps and good on-disk deltas to make serving fetches as cheap as possible).

Quoted 3 lines

Sure, I could change my configuration, but that doesn't help the other folks (e.g. https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=813084 ) who run into this.

Yeah, I'm certainly open to improving Git's defaults. If it's not clear from the above, I mostly just gave up for a site the size of GitHub. :)

Quoted 5 lines

Idea 1: when gc --auto would issue this message, instead it could create a file named gc.too-much-garbage (instead of gc.log), with this message. If that file exists, and it is less than one day (?) old, then we don't attempt to do a full gc; instead we just run git repack -A -d. (If it's more than one day old, we just delete it and continue anyway).

I kind of wonder if this should apply to _any_ error. I.e., just check the mtime of gc.log and forcibly remove it when it's older than a day. You never want to get into a state that will fail to resolve itself eventually. That might still happen (e.g., corrupt repo), but at the very least it won't be because Git is too dumb to try again.

Quoted 5 lines

Idea 2 : Like idea 1, but instead of repacking, just smash the existing packs together into one big pack. In other words, don't consider dangling objects, or recompute deltas. Twitter has a tool called "git combine-pack" that does this:
https://github.com/dturner-tw/git/blob/dturner/journal/builtin/combine-pack.c

We wrote something similar at GitHub, too, but we never ended up using it in production. We found that with a sane scheduler, it's not too big a deal to just do maintenance once in a while.

Also, our original problem was that repos which have gotten out of hand (say, 5000 packs) repacked _very_ slowly with a normal repack. So a "fast pack" followed by a real pack was a viable way out of that. In the end, I just made pack-objects handle this case better, and we don't need the fast-pack.

Quoted 3 lines

That's less space-efficient than a true repack, but it's no worse than having the packs separate, and it's a win for read performance because there's no need to do a linear search over N packs to find an object.

Over the long term you may end up with worse packs, because the true repack will drop some delta opportunities between objects in the same pack (reasoning that they weren't made into deltas last time, so it's not worth trying again). You'd probably need to use "-f" periodically.

This is all speculation, though. We never did it in production, so I was never able to measure the real impact over time.

Quoted 8 lines

Idea 3: As I suggested last time, separate fatal and non-fatal errors. If gc fails because of EIO or something, we probably don't want to touch the disk anymore. But here, the worst consequence is that we waste some processing power. And it's better to occasionally waste processing power in a non-interactive setting than it is to do so when a user will be blocked on it. So non-fatal warnings should go to gc.log, and fatal errors should go to gc.fatal. gc.log won't block gc from running. I think this is my preferred option.

This seems like your (1), except that it handles more than one type of non-fatal error. So I like it much better.

I'm still not sure if it's worth making the fatal/non-fatal distinction. Doing so is perhaps safer, but it does mean that somebody has to decide which errors are important enough to block a retry totally, and which are not. In theory, it would be safe to always _try_ and then the gc process can decide when something is broken and abort. And all you've wasted is some processing power each day.

-Peff

#19 lore raw
Quoted 3 lines · Jeff King

In my experience, auto-gc has never been a low-maintenance operation on the server side (and I do think it was primarily designed with clients in mind).

I do not think auto-gc was ever tweaked to help server usage, in its history since it was invented strictly to help end-users (mostly new ones).

Quoted 7 lines

At GitHub we disable it entirely, and do our own gc based on a throttled job queue ...
I wish regular Git were more turn-key in that respect. Maybe it is for smaller sites, but we certainly didn't find it so. And I don't know that it's feasible to really share the solution. It's entangled with our database (to store last-pushed and last-maintenance values for repos) and our job scheduler.

Thanks for sharing the insights from the trenches ;-)

Quoted 9 lines

Yeah, I'm certainly open to improving Git's defaults. If it's not clear from the above, I mostly just gave up for a site the size of GitHub. :)

Quoted 5 lines

Idea 1: when gc --auto would issue this message, instead it could create a file named gc.too-much-garbage (instead of gc.log), with this message. If that file exists, and it is less than one day (?) old, then we don't attempt to do a full gc; instead we just run git repack -A -d. (If it's more than one day old, we just delete it and continue anyway).

Quoted 5 lines

I kind of wonder if this should apply to _any_ error. I.e., just check the mtime of gc.log and forcibly remove it when it's older than a day. You never want to get into a state that will fail to resolve itself eventually. That might still happen (e.g., corrupt repo), but at the very least it won't be because Git is too dumb to try again.

;-)

Quoted 9 lines
Quoted 5 lines

Idea 2 : Like idea 1, but instead of repacking, just smash the existing packs together into one big pack. In other words, don't consider dangling objects, or recompute deltas. Twitter has a tool called "git combine-pack" that does this:
https://github.com/dturner-tw/git/blob/dturner/journal/builtin/combine-pack.c

We wrote something similar at GitHub, too, but we never ended up using it in production. We found that with a sane scheduler, it's not too big a deal to just do maintenance once in a while.

Thanks again for this. I've also been wondering about how effective a "concatenate packs without paying reachability penalty" would be.

Quoted 6 lines

I'm still not sure if it's worth making the fatal/non-fatal distinction. Doing so is perhaps safer, but it does mean that somebody has to decide which errors are important enough to block a retry totally, and which are not. In theory, it would be safe to always _try_ and then the gc process can decide when something is broken and abort. And all you've wasted is some processing power each day.

Yup, and somebody or something need to monitor so that repeated failures can be dealt with.

#20 lore raw
Quoted 6 lines · Junio C Hamano
Quoted 3 lines

We wrote something similar at GitHub, too, but we never ended up using it in production. We found that with a sane scheduler, it's not too big a deal to just do maintenance once in a while.

Thanks again for this. I've also been wondering about how effective a "concatenate packs without paying reachability penalty" would be.

For the sake of posterity, I'll include our patch at the end (sorry, not chunked into nice readable commits; that never existed in the first place).

Quoted 9 lines
Quoted 6 lines

I'm still not sure if it's worth making the fatal/non-fatal distinction. Doing so is perhaps safer, but it does mean that somebody has to decide which errors are important enough to block a retry totally, and which are not. In theory, it would be safe to always _try_ and then the gc process can decide when something is broken and abort. And all you've wasted is some processing power each day.

Yup, and somebody or something need to monitor so that repeated failures can be dealt with.

Yes. I think that part is probably outside the scope of Git. But if auto-gc leaves gc.log lying around, it would be easy to visit each repo and collect the various failures.

-- >8 --
This is the "pack-fast" patch, for reference. It applies on v2.6.5, though I had to do some wiggling due to a few of our other custom patches, so it's possible I introduced new bugs. It compiles, but I didn't actually re-test the result. I _think_ the original at least generated valid packs in all cases.

So I would certainly not recommend anybody run this. It's just a possible base to work off of if anybody's interested in the topic. I haven't looked at David's combine-packs at all to see if it is any less gross. :)

---
 Makefile            |   1 +
 builtin.h           |   1 +
 builtin/pack-fast.c | 618 +++++++++++++++++++++++++++++++++++
 cache.h             |   5 +
 git.c               |   1 +
 pack-bitmap-write.c | 167 +++++++++-
 pack-bitmap.c       |   2 +-
 pack-bitmap.h       |   8 +
 sha1_file.c         |   4 +-
 9 files changed, 792 insertions(+), 15 deletions(-)
Makefile +1 -0
index 37e2d9e18..524b185ec 100644
@@ -887,6 +887,7 @@ BUILTIN_OBJS += builtin/mv.o
 BUILTIN_OBJS += builtin/name-rev.o
 BUILTIN_OBJS += builtin/notes.o
 BUILTIN_OBJS += builtin/pack-objects.o
+BUILTIN_OBJS += builtin/pack-fast.o
 BUILTIN_OBJS += builtin/pack-redundant.o
 BUILTIN_OBJS += builtin/pack-refs.o
 BUILTIN_OBJS += builtin/patch-id.o
builtin.h +1 -0
index 79aaf0afe..df4e4d668 100644
@@ -95,6 +95,7 @@ extern int cmd_mv(int argc, const char **argv, const char *prefix);
 extern int cmd_name_rev(int argc, const char **argv, const char *prefix);
 extern int cmd_notes(int argc, const char **argv, const char *prefix);
 extern int cmd_pack_objects(int argc, const char **argv, const char *prefix);
+extern int cmd_pack_fast(int argc, const char **argv, const char *prefix);
 extern int cmd_pack_redundant(int argc, const char **argv, const char *prefix);
 extern int cmd_patch_id(int argc, const char **argv, const char *prefix);
 extern int cmd_prune(int argc, const char **argv, const char *prefix);
builtin/pack-fast.c new file +618 -0

618 lines not on this page · raw message

cache.h +5 -0
index 6f53962bf..1a13961bd 100644
@@ -1336,6 +1336,11 @@ extern void *unpack_entry(struct packed_git *, off_t, enum object_type *, unsign
 extern unsigned long unpack_object_header_buffer(const unsigned char *buf, unsigned long len, enum object_type *type, unsigned long *sizep);
 extern unsigned long get_size_from_delta(struct packed_git *, struct pack_window **, off_t);
 extern int unpack_object_header(struct packed_git *, struct pack_window **, off_t *, unsigned long *);
+extern off_t get_delta_base(struct packed_git *p,
+			    struct pack_window **w_curs,
+			    off_t *curpos,
+			    enum object_type type,
+			    off_t delta_obj_offset);
 
 /*
  * Iterate over the files in the loose-object parts of the object
git.c +1 -0
index 40f9df089..d81bd4469 100644
@@ -440,6 +440,7 @@ static struct cmd_struct commands[] = {
 	{ "name-rev", cmd_name_rev, RUN_SETUP },
 	{ "notes", cmd_notes, RUN_SETUP },
 	{ "pack-objects", cmd_pack_objects, RUN_SETUP },
+	{ "pack-fast", cmd_pack_fast, RUN_SETUP },
 	{ "pack-redundant", cmd_pack_redundant, RUN_SETUP },
 	{ "pack-refs", cmd_pack_refs, RUN_SETUP },
 	{ "patch-id", cmd_patch_id },
pack-bitmap-write.c +155 -12

189 lines not on this page · raw message

pack-bitmap.c +1 -1
index 637770af8..ee361fa6a 100644
@@ -250,7 +250,7 @@ static int load_bitmap_entries_v1(struct bitmap_index *index)
 	return 0;
 }
 
-static char *pack_bitmap_filename(struct packed_git *p)
+char *pack_bitmap_filename(struct packed_git *p)
 {
 	char *idx_name;
 	int len;
pack-bitmap.h +8 -0
index 0adcef77b..398523dbb 100644
@@ -34,6 +34,7 @@ typedef int (*show_reachable_fn)(
 	struct packed_git *found_pack,
 	off_t found_offset);
 
+char *pack_bitmap_filename(struct packed_git *p);
 int prepare_bitmap_git(void);
 void count_bitmap_commit_list(uint32_t *commits, uint32_t *trees, uint32_t *blobs, uint32_t *tags);
 void traverse_bitmap_commit_list(show_reachable_fn show_reachable);
@@ -53,5 +54,12 @@ void bitmap_writer_finish(struct pack_idx_entry **index,
 			  uint32_t index_nr,
 			  const char *filename,
 			  uint16_t options);
+void bitmap_rewrite_existing(
+	struct packed_git *p,
+	struct pack_idx_entry **index,
+	uint32_t index_nr,
+	off_t pack_offset,
+	const unsigned char *pack_sha1,
+	const char *filename);
 
 #endif
sha1_file.c +2 -2
index 72289696d..bcd447f16 100644
@@ -1821,7 +1821,7 @@ unsigned long get_size_from_delta(struct packed_git *p,
 	return get_delta_hdr_size(&data, delta_head+sizeof(delta_head));
 }
 
-static off_t get_delta_base(struct packed_git *p,
+off_t get_delta_base(struct packed_git *p,
 				    struct pack_window **w_curs,
 				    off_t *curpos,
 				    enum object_type type,
@@ -1936,7 +1936,7 @@ static int retry_bad_packed_offset(struct packed_git *p, off_t obj_offset)
 
 #define POI_STACK_PREALLOC 64
 
-static enum object_type packed_to_object_type(struct packed_git *p,
+enum object_type packed_to_object_type(struct packed_git *p,
 					      off_t obj_offset,
 					      enum object_type type,
 					      struct pack_window **w_curs,