Skip to contentSkip to conversation

Fix size_mult option in fetch.writeCommitGraph

PATCH In review Derrick Stolee via GitGitGadget 4 participants 6 messages Jan 2, 2020 – Jan 6, 2020

lore
Share & tools
lore
#0 lore raw

cover Fix size_mult option in fetch.writeCommitGraph

I found this while doing some digging into fetch behavior and split commit graphs. I had been running fetch.writeCommitGraph=true on my local repos for a while and noticed that the commit-graph chains were much longer than expected.

The reason is silly, and the commit message includes all the details.

This behavior exists since v2.24.0, so I'm not sure if it makes the bar for v2.25.0 this late in the release cycle. At minimum, the change is very small and unlikely to cause more pain.

This is only a performance bug, and the effect is relatively small. A large list of commit-graph files slows down the commit lookup time as we need to perform a linear number of binary searches. This only affects finding the first commit(s) in a commit walk, as after that we can navigate quickly to the correct position using graph_pos. When a user runs gc (with gc.writeCommitGraph=true, on by default), the chain collapses to a single level, fixing the performance problem.

Thanks, -Stolee

Derrick Stolee (1):
fetch: set size_multiple in split_commit_graph_opts

 builtin/fetch.c | 4 +---
 commit-graph.c  | 4 +++-
 2 files changed, 4 insertions(+), 4 deletions(-)

base-commit: 99c33bed562b41de6ce9bd3fd561303d39645048
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-509%2Fderrickstolee%2Ffetch-write-commit-graph-split-v1
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-509/derrickstolee/fetch-write-commit-graph-split-v1
Pull-Request: https://github.com/gitgitgadget/git/pull/509

Signature · 1 line
gitgitgadget
#1 lore raw

PATCH 1/1 fetch: set size_multiple in split_commit_graph_opts

From: Derrick Stolee <[email protected]>

In 50f26bd ("fetch: add fetch.writeCommitGraph config setting", 2019-09-02), the fetch builtin added the capability to write a commit-graph using the "--split" feature. This feature creates multiple commit-graph files, and those can merge based on a set of "split options" including a size multiple. The default size multiple is 2, which intends to provide a log_2 N depth of the commit-graph chain where N is the number of commits.

However, I noticed during dogfooding that my commit-graph chains were becoming quite large when left only to builds by 'git fetch'. It turns out that in split_graph_merge_strategy(), we default the size_mult variable to 2 except we override it with the context's split_opts if they exist. In builtin/fetch.c, we create such a split_opts, but do not populate it with values.

This problem is due to two failures:

1. It is unclear that we can add the flag COMMIT_GRAPH_WRITE_SPLIT with a NULL split_opts.
2. If we have a non-NULL split_opts, then we override the default values even if a zero value is given.

Correct both of these issues. First, do not override size_mult when the options provide a zero value. Second, stop creating a split_opts in the fetch builtin.

---
 builtin/fetch.c | 4 +---
 commit-graph.c  | 4 +++-
 2 files changed, 4 insertions(+), 4 deletions(-)
builtin/fetch.c +1 -3
index f8765b385b..b4c6d921d0 100644
@@ -1866,15 +1866,13 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)
 	    (fetch_write_commit_graph < 0 &&
 	     the_repository->settings.fetch_write_commit_graph)) {
 		int commit_graph_flags = COMMIT_GRAPH_WRITE_SPLIT;
-		struct split_commit_graph_opts split_opts;
-		memset(&split_opts, 0, sizeof(struct split_commit_graph_opts));
 
 		if (progress)
 			commit_graph_flags |= COMMIT_GRAPH_WRITE_PROGRESS;
 
 		write_commit_graph_reachable(get_object_directory(),
 					     commit_graph_flags,
-					     &split_opts);
+					     NULL);
 	}
 
 	close_object_store(the_repository->objects);
commit-graph.c +3 -1
index e771394aff..b205e65ed1 100644
@@ -1542,7 +1542,9 @@ static void split_graph_merge_strategy(struct write_commit_graph_context *ctx)
 
 	if (ctx->split_opts) {
 		max_commits = ctx->split_opts->max_commits;
-		size_mult = ctx->split_opts->size_multiple;
+
+		if (ctx->split_opts->size_multiple)
+			size_mult = ctx->split_opts->size_multiple;
 	}
 
 	g = ctx->r->objects->commit_graph;
Signature · 1 line
gitgitgadget
#2 lore raw

Re: [PATCH 1/1] fetch: set size_multiple in split_commit_graph_opts

As I was writing this commit message, I changed the plan for how to fix this. Originally, I was going to set size_multiple = 2 in builtin/fetch.c, which is how the subject line was created. I forgot to change that to something more like:

"commit-graph: prefer default size_mult when given zero"

Quoted 19 lines · Derrick Stolee via GitGitGadget

From: Derrick Stolee <[email protected]>

In 50f26bd ("fetch: add fetch.writeCommitGraph config setting", 2019-09-02), the fetch builtin added the capability to write a commit-graph using the "--split" feature. This feature creates multiple commit-graph files, and those can merge based on a set of "split options" including a size multiple. The default size multiple is 2, which intends to provide a log_2 N depth of the commit-graph chain where N is the number of commits.

However, I noticed during dogfooding that my commit-graph chains were becoming quite large when left only to builds by 'git fetch'. It turns out that in split_graph_merge_strategy(), we default the size_mult variable to 2 except we override it with the context's split_opts if they exist. In builtin/fetch.c, we create such a split_opts, but do not populate it with values.

This problem is due to two failures:

Quoted 8 lines

1. It is unclear that we can add the flag COMMIT_GRAPH_WRITE_SPLIT with a NULL split_opts.
2. If we have a non-NULL split_opts, then we override the default values even if a zero value is given.

Correct both of these issues. First, do not override size_mult when the options provide a zero value. Second, stop creating a split_opts in the fetch builtin.

This is a correct description of the actual patch.

Thanks,
-Stolee

#3 lore raw

Re: [PATCH 1/1] fetch: set size_multiple in split_commit_graph_opts

In reply to Derrick Stolee via GitGitGadget
Quoted 10 lines · Derrick Stolee via GitGitGadget

This problem is due to two failures:

1. It is unclear that we can add the flag COMMIT_GRAPH_WRITE_SPLIT with a NULL split_opts.
2. If we have a non-NULL split_opts, then we override the default values even if a zero value is given.

Correct both of these issues. First, do not override size_mult when the options provide a zero value. Second, stop creating a split_opts in the fetch builtin.

OK, so there is the hardcoded default 2 in the code, and split_opts structure *can* override it, but 0 in the field of the structure is meant to signal "no, I do not have any value to override the default", not "I do want to set the multiple to 0"?

Makes sense. Will queue.

Thanks.

Quoted 42 lines
---
 builtin/fetch.c | 4 +---
 commit-graph.c  | 4 +++-
 2 files changed, 4 insertions(+), 4 deletions(-)
builtin/fetch.c +1 -3
index f8765b385b..b4c6d921d0 100644
@@ -1866,15 +1866,13 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)
 	    (fetch_write_commit_graph < 0 &&
 	     the_repository->settings.fetch_write_commit_graph)) {
 		int commit_graph_flags = COMMIT_GRAPH_WRITE_SPLIT;
-		struct split_commit_graph_opts split_opts;
-		memset(&split_opts, 0, sizeof(struct split_commit_graph_opts));
 
 		if (progress)
 			commit_graph_flags |= COMMIT_GRAPH_WRITE_PROGRESS;
 
 		write_commit_graph_reachable(get_object_directory(),
 					     commit_graph_flags,
-					     &split_opts);
+					     NULL);
 	}
 
 	close_object_store(the_repository->objects);
commit-graph.c +3 -1
index e771394aff..b205e65ed1 100644
@@ -1542,7 +1542,9 @@ static void split_graph_merge_strategy(struct write_commit_graph_context *ctx)
 
 	if (ctx->split_opts) {
 		max_commits = ctx->split_opts->max_commits;
-		size_mult = ctx->split_opts->size_multiple;
+
+		if (ctx->split_opts->size_multiple)
+			size_mult = ctx->split_opts->size_multiple;
 	}
 
 	g = ctx->r->objects->commit_graph;
#4 lore raw

Re: [PATCH 1/1] fetch: set size_multiple in split_commit_graph_opts

Quoted 9 lines · Junio C Hamano
Quoted 7 lines · Derrick Stolee via GitGitGadget

This problem is due to two failures:

1. It is unclear that we can add the flag COMMIT_GRAPH_WRITE_SPLIT with a NULL split_opts.
2. If we have a non-NULL split_opts, then we override the default values even if a zero value is given.

Quoted 8 lines
Quoted 3 lines

Correct both of these issues. First, do not override size_mult when the options provide a zero value. Second, stop creating a split_opts in the fetch builtin.

OK, so there is the hardcoded default 2 in the code, and split_opts structure *can* override it, but 0 in the field of the structure is meant to signal "no, I do not have any value to override the default", not "I do want to set the multiple to 0"?

Correct. The multiple 0 makes it so we never merge layers of the chain, and this was happening accidentally. A caller could still accomplish this by passing -1, but that is not recommended.

-Stolee

#5 lore raw

Re: [PATCH 1/1] fetch: set size_multiple in split_commit_graph_opts

In reply to Derrick Stolee via GitGitGadget
Quoted 19 lines · Derrick Stolee via GitGitGadget

From: Derrick Stolee <[email protected]>

In 50f26bd ("fetch: add fetch.writeCommitGraph config setting", 2019-09-02), the fetch builtin added the capability to write a commit-graph using the "--split" feature. This feature creates multiple commit-graph files, and those can merge based on a set of "split options" including a size multiple. The default size multiple is 2, which intends to provide a log_2 N depth of the commit-graph chain where N is the number of commits.

However, I noticed during dogfooding that my commit-graph chains were becoming quite large when left only to builds by 'git fetch'. It turns out that in split_graph_merge_strategy(), we default the size_mult variable to 2 except we override it with the context's split_opts if they exist. In builtin/fetch.c, we create such a split_opts, but do not populate it with values.

This problem is due to two failures:

Quoted 8 lines

1. It is unclear that we can add the flag COMMIT_GRAPH_WRITE_SPLIT with a NULL split_opts.
2. If we have a non-NULL split_opts, then we override the default values even if a zero value is given.

Correct both of these issues. First, do not override size_mult when the options provide a zero value. Second, stop creating a split_opts in the fetch builtin.

Thanks, the explanation and fix (both parts) look good to me, modulo the subject correction you already noted.

Quoted 3 lines
---
 builtin/fetch.c | 4 +---
 commit-graph.c  | 4 +++-

Is it worth covering this with a test?

I guess the non-fetch code paths for splitting already cover this pretty well, and this is just about managing to get the right number into the commit-graph code. So perhaps it isn't worth it.

-Peff