Fix size_mult option in fetch.writeCommitGraph
cover Fix size_mult option in fetch.writeCommitGraph
I found this while doing some digging into fetch behavior and split commit graphs. I had been running fetch.writeCommitGraph=true on my local repos for a while and noticed that the commit-graph chains were much longer than expected.
The reason is silly, and the commit message includes all the details.
This behavior exists since v2.24.0, so I'm not sure if it makes the bar for v2.25.0 this late in the release cycle. At minimum, the change is very small and unlikely to cause more pain.
This is only a performance bug, and the effect is relatively small. A large list of commit-graph files slows down the commit lookup time as we need to perform a linear number of binary searches. This only affects finding the first commit(s) in a commit walk, as after that we can navigate quickly to the correct position using graph_pos. When a user runs gc (with gc.writeCommitGraph=true, on by default), the chain collapses to a single level, fixing the performance problem.
Thanks, -Stolee
Derrick Stolee (1):
fetch: set size_multiple in split_commit_graph_opts
builtin/fetch.c | 4 +--- commit-graph.c | 4 +++- 2 files changed, 4 insertions(+), 4 deletions(-)
base-commit: 99c33bed562b41de6ce9bd3fd561303d39645048
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-509%2Fderrickstolee%2Ffetch-write-commit-graph-split-v1
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-509/derrickstolee/fetch-write-commit-graph-split-v1
Pull-Request: https://github.com/gitgitgadget/git/pull/509
Signature · 1 line
gitgitgadget
PATCH 1/1 fetch: set size_multiple in split_commit_graph_opts
From: Derrick Stolee <[email protected]>
In 50f26bd ("fetch: add fetch.writeCommitGraph config setting", 2019-09-02), the fetch builtin added the capability to write a commit-graph using the "--split" feature. This feature creates multiple commit-graph files, and those can merge based on a set of "split options" including a size multiple. The default size multiple is 2, which intends to provide a log_2 N depth of the commit-graph chain where N is the number of commits.
However, I noticed during dogfooding that my commit-graph chains were becoming quite large when left only to builds by 'git fetch'. It turns out that in split_graph_merge_strategy(), we default the size_mult variable to 2 except we override it with the context's split_opts if they exist. In builtin/fetch.c, we create such a split_opts, but do not populate it with values.
This problem is due to two failures:
1. It is unclear that we can add the flag COMMIT_GRAPH_WRITE_SPLIT with a NULL split_opts.
2. If we have a non-NULL split_opts, then we override the default values even if a zero value is given.
Correct both of these issues. First, do not override size_mult when the options provide a zero value. Second, stop creating a split_opts in the fetch builtin.
--- builtin/fetch.c | 4 +--- commit-graph.c | 4 +++- 2 files changed, 4 insertions(+), 4 deletions(-)
builtin/fetch.c +1 -3
@@ -1866,15 +1866,13 @@ int cmd_fetch(int argc, const char **argv, const char *prefix) (fetch_write_commit_graph < 0 && the_repository->settings.fetch_write_commit_graph)) {int commit_graph_flags =COMMIT_GRAPH_WRITE_SPLIT ; -struct split_commit_graph_opts split_opts; -memset (&split_opts,0 ,sizeof (struct split_commit_graph_opts ));if (progress) commit_graph_flags |=COMMIT_GRAPH_WRITE_PROGRESS ;write_commit_graph_reachable (get_object_directory (), commit_graph_flags, - &split_opts); +NULL ); }close_object_store (the_repository->objects);
commit-graph.c +3 -1
@@ -1542,7 +1542,9 @@ static void split_graph_merge_strategy(struct write_commit_graph_context *ctx)if (ctx->split_opts) { max_commits = ctx->split_opts->max_commits; - size_mult = ctx->split_opts->size_multiple; + +if (ctx->split_opts->size_multiple) + size_mult = ctx->split_opts->size_multiple; } g = ctx->r->objects->commit_graph;
Signature · 1 line
gitgitgadget
Re: [PATCH 1/1] fetch: set size_multiple in split_commit_graph_opts
As I was writing this commit message, I changed the plan for how to fix this. Originally, I was going to set size_multiple = 2 in builtin/fetch.c, which is how the subject line was created. I forgot to change that to something more like:
"commit-graph: prefer default size_mult when given zero"
Quoted 19 lines · Derrick Stolee via GitGitGadget
From: Derrick Stolee <[email protected]>
In 50f26bd ("fetch: add fetch.writeCommitGraph config setting", 2019-09-02), the fetch builtin added the capability to write a commit-graph using the "--split" feature. This feature creates multiple commit-graph files, and those can merge based on a set of "split options" including a size multiple. The default size multiple is 2, which intends to provide a
log_2N depth of the commit-graph chain where N is the number of commits.However, I noticed during dogfooding that my commit-graph chains were becoming quite large when left only to builds by 'git fetch'. It turns out that in
split_graph_merge_strategy(), we default thesize_multvariable to 2 except we override it with the context'ssplit_optsif they exist. Inbuiltin/fetch.c, we create such asplit_opts, but do not populate it with values.This problem is due to two failures:
Quoted 8 lines
1. It is unclear that we can add the flag
COMMIT_GRAPH_WRITE_SPLITwith a NULLsplit_opts.
2. If we have a non-NULLsplit_opts, then we override the default values even if a zero value is given.Correct both of these issues. First, do not override
size_multwhen the options provide a zero value. Second, stop creating asplit_optsin the fetch builtin.
This is a correct description of the actual patch.
Thanks,
-Stolee
Re: [PATCH 1/1] fetch: set size_multiple in split_commit_graph_opts
In reply to Derrick Stolee via GitGitGadgetQuoted 10 lines · Derrick Stolee via GitGitGadget
This problem is due to two failures:
1. It is unclear that we can add the flag
COMMIT_GRAPH_WRITE_SPLITwith a NULLsplit_opts.
2. If we have a non-NULLsplit_opts, then we override the default values even if a zero value is given.Correct both of these issues. First, do not override
size_multwhen the options provide a zero value. Second, stop creating asplit_optsin the fetch builtin.
OK, so there is the hardcoded default 2 in the code, and split_opts structure *can* override it, but 0 in the field of the structure is meant to signal "no, I do not have any value to override the default", not "I do want to set the multiple to 0"?
Makes sense. Will queue.
Thanks.
Quoted 42 lines
Signed-off-by Derrick Stolee <[email protected]>--- builtin/fetch.c | 4 +--- commit-graph.c | 4 +++- 2 files changed, 4 insertions(+), 4 deletions(-)builtin/fetch.c +1
-3@@ -1866,15 +1866,13 @@ int cmd_fetch(int argc, const char **argv, const char *prefix) (fetch_write_commit_graph < 0 && the_repository->settings.fetch_write_commit_graph)) {int commit_graph_flags =COMMIT_GRAPH_WRITE_SPLIT ; -struct split_commit_graph_opts split_opts; -memset (&split_opts,0 ,sizeof (struct split_commit_graph_opts ));if (progress) commit_graph_flags |=COMMIT_GRAPH_WRITE_PROGRESS ;write_commit_graph_reachable (get_object_directory (), commit_graph_flags, - &split_opts); +NULL ); }close_object_store (the_repository->objects);commit-graph.c +3
-1@@ -1542,7 +1542,9 @@ static void split_graph_merge_strategy(struct write_commit_graph_context *ctx)if (ctx->split_opts) { max_commits = ctx->split_opts->max_commits; - size_mult = ctx->split_opts->size_multiple; + +if (ctx->split_opts->size_multiple) + size_mult = ctx->split_opts->size_multiple; } g = ctx->r->objects->commit_graph;
Re: [PATCH 1/1] fetch: set size_multiple in split_commit_graph_opts
Quoted 9 lines · Junio C Hamano
Quoted 7 lines · Derrick Stolee via GitGitGadget
This problem is due to two failures:
1. It is unclear that we can add the flag
COMMIT_GRAPH_WRITE_SPLITwith a NULLsplit_opts.
2. If we have a non-NULLsplit_opts, then we override the default values even if a zero value is given.
Quoted 8 lines
Quoted 3 lines
Correct both of these issues. First, do not override
size_multwhen the options provide a zero value. Second, stop creating asplit_optsin the fetch builtin.OK, so there is the hardcoded default 2 in the code, and
split_optsstructure *can* override it, but 0 in the field of the structure is meant to signal "no, I do not have any value to override the default", not "I do want to set the multiple to 0"?
Correct. The multiple 0 makes it so we never merge layers of the chain, and this was happening accidentally. A caller could still accomplish this by passing -1, but that is not recommended.
-Stolee
Re: [PATCH 1/1] fetch: set size_multiple in split_commit_graph_opts
In reply to Derrick Stolee via GitGitGadgetQuoted 19 lines · Derrick Stolee via GitGitGadget
From: Derrick Stolee <[email protected]>
In 50f26bd ("fetch: add fetch.writeCommitGraph config setting", 2019-09-02), the fetch builtin added the capability to write a commit-graph using the "--split" feature. This feature creates multiple commit-graph files, and those can merge based on a set of "split options" including a size multiple. The default size multiple is 2, which intends to provide a
log_2N depth of the commit-graph chain where N is the number of commits.However, I noticed during dogfooding that my commit-graph chains were becoming quite large when left only to builds by 'git fetch'. It turns out that in
split_graph_merge_strategy(), we default thesize_multvariable to 2 except we override it with the context'ssplit_optsif they exist. Inbuiltin/fetch.c, we create such asplit_opts, but do not populate it with values.This problem is due to two failures:
Quoted 8 lines
1. It is unclear that we can add the flag
COMMIT_GRAPH_WRITE_SPLITwith a NULLsplit_opts.
2. If we have a non-NULLsplit_opts, then we override the default values even if a zero value is given.Correct both of these issues. First, do not override
size_multwhen the options provide a zero value. Second, stop creating asplit_optsin the fetch builtin.
Thanks, the explanation and fix (both parts) look good to me, modulo the subject correction you already noted.
Quoted 3 lines
--- builtin/fetch.c | 4 +--- commit-graph.c | 4 +++-
Is it worth covering this with a test?
I guess the non-fetch code paths for splitting already cover this pretty well, and this is just about managing to get the right number into the commit-graph code. So perhaps it isn't worth it.
-Peff