git gc and kernel.org

We have run into two major problems with git on kernel.org. We have discussed workarounds, but I believe they are generally major problems which will affect other sites, and really need to be fixed upstream.

1) git receive-pack doesn't run git gc --auto.

This cause repositories which people thought were being autopacked to not be so. This can be done via a hook, but that is way too painful for people to do -- in fact, it's hard enough just to get people to enable the post-update hook for http fetching.

1b) as per the above, a way to make "git update-server-info" at post-update time a configurable option rather than needing to be done via the hook would be very nice, since a configuration option can be enabled sitewide.

2) When pushing to a repository, it apparently doesn't honor new objects in alternate repositories. This causes massive unnecessary duplication.

I believe these are major issues that need to be addressed. If we are *mistaken* on these, then we would appreciate being corrected.

I don't know how you create repos on kernel.org, but by default both git init and git clone uses /usr/share/git-core/templates for the new repo, for example I put a symbolic ref h -> HEAD there to save some typing. You can enable whatever hooks you want there, but they would only take effect for new repos.

We have run into two major problems with git on kernel.org. We have discussed workarounds, but I believe they are generally major problems which will affect other sites, and really need to be fixed upstream.

1) git receive-pack doesn't run git gc --auto.

This cause repositories which people thought were being autopacked to not be so. This can be done via a hook, but that is way too painful for people to do -- in fact, it's hard enough just to get people to enable the post-update hook for http fetching.

1b) as per the above, a way to make "git update-server-info" at post-update time a configurable option rather than needing to be done via the hook would be very nice, since a configuration option can be enabled sitewide.

2) When pushing to a repository, it apparently doesn't honor new objects in alternate repositories. This causes massive unnecessary duplication.

I believe these are major issues that need to be addressed. If we are *mistaken* on these, then we would appreciate being corrected.

At one point, update-server-info used to compute a lot more than what we currently compute and it made some sense to oppose against it on performance ground.

But these days it only lists the refs and packs and does nothing else; the performance impact should be immeasurable and it adds only two files to the repository. It cannot be a big deal, unless you oppose to http transport on a non-technical ground.

Quoted 4 lines · Junio C Hamano

At one point, update-server-info used to compute a lot more than what we currently compute and it made some sense to oppose against it on performance ground.

Quoted 4 lines

But these days it only lists the refs and packs and does nothing else; the performance impact should be immeasurable and it adds only two files to the repository. It cannot be a big deal, unless you oppose to http transport on a non-technical ground.

FYI, update-server-info takes ~0.7 seconds in a repo with one pack and ~50000 refs, so I guess it's acceptable to enable it by default, even in those kinds of repos.

Still... Hopefully this is going to become redundant information in the future with the eventual deployment of smart protocol over HTTP. So I think that as a config option being off by default this is a good compromize. Site administrators can turn it on by default in /etc/gitconfig.

builtin-receive-pack.c int cmd_receive_pack(int argc, const char **argv, const char *prefix) · line 691

Quoted 3 lines · Junio C Hamano

builtin-receive-pack.c +3 -0

+		if (auto_gc) {
+			const char *argv_gc_auto[] = { "gc", "--auto", NULL };
+			run_command_v_opt(argv_gc_auto, RUN_GIT_CMD);

Am I correct that this will produce progress output? If git-daemon runs receive-pack, then this output will go to the syslog. Do we care?

Quoted 7 lines · Johannes Sixt

builtin-receive-pack.c int cmd_receive_pack(int argc, const char **argv, const char *prefix) · line 691

Quoted 3 lines · Junio C Hamano

builtin-receive-pack.c +3 -0

+		if (auto_gc) {
+			const char *argv_gc_auto[] = { "gc", "--auto", NULL };
+			run_command_v_opt(argv_gc_auto, RUN_GIT_CMD);

Am I correct that this will produce progress output? If git-daemon runs receive-pack, then this output will go to the syslog. Do we care?

We do, and we don't want that. Thanks for spotting.

Would adding "--quiet" to the mix be enough?

Quoted 10 lines · Junio C Hamano

Quoted 6 lines · Johannes Sixt

builtin-receive-pack.c int cmd_receive_pack(int argc, const char **argv, const char *prefix) · line 691

Quoted 3 lines · Junio C Hamano

builtin-receive-pack.c +3 -0

+		if (auto_gc) {
+			const char *argv_gc_auto[] = { "gc", "--auto", NULL };
+			run_command_v_opt(argv_gc_auto, RUN_GIT_CMD);

Am I correct that this will produce progress output? If git-daemon runs receive-pack, then this output will go to the syslog. Do we care?

We do, and we don't want that. Thanks for spotting.

Quoted 1 line

Would adding "--quiet" to the mix be enough?

Actually I don't know. This originally came from "we _could_ add gc and update-server-info to hundreds of post-receive hooks, but any repository hosting site that holds many central repositories will exactly have the same issue, so why not do this internally", so it may make sense to do exactly the same thing as what we do to the output from hooks. What do we do to them now? stdout-to-stderr?

$ git gc --auto --quiet
Auto packing your repository for optimum performance. You may also
run "git gc" manually. See "git help gc" for more information.
[...]
[*] Skipping the hint to "run git gc manually" would even be good in
this case, because the hint pertains the remote repository, not the
one from which "git push" was issued.

I'm slightly in favor of adding "--quiet", because even with this option we see some output:

$ git gc --auto --quiet
Auto packing your repository for optimum performance. You may also
run "git gc" manually. See "git help gc" for more information.

A compromise would be to reduce this message to the first sentence if --quiet was given.

Quoted 5 lines · Nicolas Pitre

Still... Hopefully this is going to become redundant information in the future with the eventual deployment of smart protocol over HTTP. So I think that as a config option being off by default this is a good compromize. Site administrators can turn it on by default in /etc/gitconfig.

Ok, something like this?

Quoted 8 lines · Junio C Hamano

Quoted 5 lines · Nicolas Pitre

Still... Hopefully this is going to become redundant information in the future with the eventual deployment of smart protocol over HTTP. So I think that as a config option being off by default this is a good compromize. Site administrators can turn it on by default in /etc/gitconfig.

Quoted 1 line

Ok, something like this?

Looks fine to me, except...

Introduce two new configuration variables, receive.autogc (defaults to true) and receive.updateserverinfo (defaults to false). When these are set, receive-pack runs "gc --auto" and "update-server-info" respectively after it finishes receiving data from "git push" and updating refs.

Signed-off-by Junio C Hamano <[email protected]>

---

Quoted 10 lines · Nicolas Pitre

Quoted 8 lines · Junio C Hamano

Quoted 5 lines · Nicolas Pitre

Still... Hopefully this is going to become redundant information in the future with the eventual deployment of smart protocol over HTTP. So I think that as a config option being off by default this is a good compromize. Site administrators can turn it on by default in /etc/gitconfig.

Quoted 3 lines

Quoted 1 line

Ok, something like this?

Looks fine to me, except...

Everything you said sounded sensible. Thanks.

 Documentation/config.txt |    9 +++++++++
 builtin-receive-pack.c   |   18 ++++++++++++++++++
 2 files changed, 27 insertions(+), 0 deletions(-)

Documentation/config.txt +9 -0

index cd17814..ba6ed10 100644
@@ -1320,6 +1320,11 @@ rebase.stat::
 	Whether to show a diffstat of what changed upstream since the last
 	rebase. False by default.
 
+receive.autogc::
+	By default, git-receive-pack will run "git-gc --auto" after
+	receiving data from git-push and updating refs.  You can stop
+	it by setting this variable to false.
+
 receive.fsckObjects::
 	If it is set to true, git-receive-pack will check all received
 	objects. It will abort in the case of a malformed object or a

@@ -1355,6 +1360,10 @@ receive.denyNonFastForwards::
 	even if that push is forced. This configuration variable is
 	set when initializing a shared repository.
 
+receive.updateserverinfo::
+	If set to true, git-receive-pack will run git-update-server-info
+	after receiving data from git-push and updating refs.
+
 remote.<name>.url::
 	The URL of a remote repository.  See linkgit:git-fetch[1] or
 	linkgit:git-push[1].

builtin-receive-pack.c +18 -0

index b771fe9..92173ac 100644
@@ -28,6 +28,8 @@ static int transfer_unpack_limit = -1;
 static int unpack_limit = 100;
 static int report_status;
 static int prefer_ofs_delta = 1;
+static int auto_update_server_info;
+static int auto_gc = 1;
 static const char *head_name;
 static char *capabilities_to_send;
 

@@ -88,6 +90,16 @@ static int receive_pack_config(const char *var, const char *value, void *cb)
 		return 0;
 	}
 
+	if (strcmp(var, "receive.updateserverinfo") == 0) {
+		auto_update_server_info = git_config_bool(var, value);
+		return 0;
+	}
+
+	if (strcmp(var, "receive.autogc") == 0) {
+		auto_gc = git_config_bool(var, value);
+		return 0;
+	}
+
 	return git_default_config(var, value, cb);
 }
 

@@ -672,6 +684,12 @@ int cmd_receive_pack(int argc, const char **argv, const char *prefix)
 			report(unpack_status);
 		run_receive_hook(post_receive_hook);
 		run_update_post_hook(commands);
+		if (auto_gc) {
+			const char *argv_gc_auto[] = { "gc", "--auto", NULL };
+			run_command_v_opt(argv_gc_auto, RUN_GIT_CMD);
+		}
+		if (auto_update_server_info)
+			update_server_info(0);
 	}
 	return 0;
 }

Signature · 1 line

1.6.5.1.107.gba912

As to (1), I think it is reasonable to do some sort of "gc" in receive-pack, and while I recall that Linus was violently against the idea, I personally think it is reasonable to run update-server-info ourselves at the same time without needing the hook, even if the repository is never going to be served over dumb http [*1*]. At one point, update-server-info used to compute a lot more than what we currently compute and it made some sense to oppose against it on performance ground.

But these days it only lists the refs and packs and does nothing else; the performance impact should be immeasurable and it adds only two files to the repository. It cannot be a big deal, unless you oppose to http transport on a non-technical ground.