a few commit-graph improvements
cover a few commit-graph improvements
We've been playing with commit graphs at GitHub and found a few bits of low-hanging fruit (one liners -- it doesn't get any lower than that).
The first one is actually a resurrection of a patch from March:
https://public-inbox.org/git/[email protected]/
where the progress bar sometimes prints nonsense. There's some discussion in that thread about how we could sometimes show a real percentage instead of a counting-up progress meter. But given the number of corner cases discussed, and the fact that nothing has happened for 6 months, I think we should first make sure we're always doing the _correct_ thing, and then people can build a nicer meter on top if they want to.
The second is a fix for a small memory "leak", but it makes a big difference.
[1/2]: commit-graph: don't show progress percentages while expanding reachable commits [2/2]: commit-graph: turn off save_commit_buffer
builtin/commit-graph.c | 2 ++ commit-graph.c | 2 +- 2 files changed, 3 insertions(+), 1 deletion(-)
-Peff
PATCH 1/2 commit-graph: don't show progress percentages while expanding reachable commits
From: SZEDER Gábor <[email protected]>
Commit 49bbc57a57 (commit-graph write: emit a percentage for all progress, 2019-01-19) was a bit overeager when it added progress percentages to the "Expanding reachable commits in commit graph" phase as well, because most of the time the number of commits that phase has to iterate over is not known in advance and grows significantly, and, consequently, we end up with nonsensical numbers:
$ git commit-graph write --reachable Expanding reachable commits in commit graph: 138606% (824706/595), done. [...]
$ git rev-parse v5.0 | git commit-graph write --stdin-commits Expanding reachable commits in commit graph: 81264400% (812644/1), done. [...]
Even worse, because the percentage grows so quickly, the progress code outputs much more often than it should (because it ticks every second, or every 1%), slowing the whole process down. My time for "git commit-graph write --reachable" on linux.git went from 13.463s to 12.521s with this patch, ~7% savings.
Therefore, don't show progress percentages in the "Expanding reachable commits in commit graph" phase.
Note that the current code does sometimes do the right thing, if we picked up all commits initially (e.g., omitting "--reachable" in a fully-packed repository would get the correct count without any parent traversal). So it may be possible to come up with a way to tell when we could use a percentage here. But in the meantime, let's make sure we robustly avoid printing nonsense.
---
Compared to the original from:
https://public-inbox.org/git/[email protected]/
I rebased it to handle code movement, added in the timing data, and tried to summarize the discussion from the thread.
commit-graph.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
commit-graph.c +1 -1
@@ -1050,7 +1050,7 @@ static void close_reachable(struct write_commit_graph_context *ctx)if (ctx->report_progress) ctx->progress =start_delayed_progress (_ ("Expanding reachable commits in commit graph" ), - ctx->oids.nr); +0 );for (i =0 ; i < ctx->oids.nr; i++) {display_progress (ctx->progress, i +1 ); commit =lookup_commit (ctx->r, &ctx->oids.list[i]);
Signature · 1 line
2.23.0.474.gb1abd76f7a
Re: [PATCH 1/2] commit-graph: don't show progress percentages while expanding reachable commits
Quoted 17 lines · Jeff King
From: SZEDER Gábor <[email protected]>
Commit
49bbc57a57(commit-graph write: emit a percentage for all progress, 2019-01-19) was a bit overeager when it added progress percentages to the "Expanding reachable commits in commit graph" phase as well, because most of the time the number of commits that phase has to iterate over is not known in advance and grows significantly, and, consequently, we end up with nonsensical numbers:$ git commit-graph write --reachable Expanding reachable commits in commit graph: 138606% (824706/595), done. [...]$ git rev-parse v5.0 | git commit-graph write --stdin-commits Expanding reachable commits in commit graph: 81264400% (812644/1), done. [...]
Quoted 5 lines
Even worse, because the percentage grows so quickly, the progress code outputs much more often than it should (because it ticks every second, or every 1%), slowing the whole process down. My time for "git commit-graph write --reachable" on linux.git went from 13.463s to 12.521s with this patch, ~7% savings.
Oh, interesting.
Quoted 12 lines
Therefore, don't show progress percentages in the "Expanding reachable commits in commit graph" phase.
Note that the current code does sometimes do the right thing, if we picked up all commits initially (e.g., omitting "--reachable" in a fully-packed repository would get the correct count without any parent traversal). So it may be possible to come up with a way to tell when we could use a percentage here. But in the meantime, let's make sure we robustly avoid printing nonsense.
Signed-off-by SZEDER Gábor <[email protected]>Signed-off-by Jeff King <[email protected]>
Quoted 7 lines
---
Compared to the original from:https://public-inbox.org/git/[email protected]/
I rebased it to handle code movement, added in the timing data, and tried to summarize the discussion from the thread.
Thanks for resurrecting this patch and for the summary paragraph.
Re: [PATCH 1/2] commit-graph: don't show progress percentages while expanding reachable commits
Quoted 41 lines · SZEDER Gábor
Quoted 17 lines · Jeff King
From: SZEDER Gábor <[email protected]>
Commit
49bbc57a57(commit-graph write: emit a percentage for all progress, 2019-01-19) was a bit overeager when it added progress percentages to the "Expanding reachable commits in commit graph" phase as well, because most of the time the number of commits that phase has to iterate over is not known in advance and grows significantly, and, consequently, we end up with nonsensical numbers:$ git commit-graph write --reachable Expanding reachable commits in commit graph: 138606% (824706/595), done. [...]$ git rev-parse v5.0 | git commit-graph write --stdin-commits Expanding reachable commits in commit graph: 81264400% (812644/1), done. [...]Quoted 5 lines
Even worse, because the percentage grows so quickly, the progress code outputs much more often than it should (because it ticks every second, or every 1%), slowing the whole process down. My time for "git commit-graph write --reachable" on linux.git went from 13.463s to 12.521s with this patch, ~7% savings.
Oh, interesting.
Quoted 15 lines
Therefore, don't show progress percentages in the "Expanding reachable commits in commit graph" phase.
Note that the current code does sometimes do the right thing, if we picked up all commits initially (e.g., omitting "--reachable" in a fully-packed repository would get the correct count without any parent traversal). So it may be possible to come up with a way to tell when we could use a percentage here. But in the meantime, let's make sure we robustly avoid printing nonsense.
Signed-off-by SZEDER Gábor <[email protected]>Signed-off-by Jeff King <[email protected]>---
Compared to the original from:
Quoted 6 lines
Quoted 4 lines
https://public-inbox.org/git/[email protected]/
I rebased it to handle code movement, added in the timing data, and tried to summarize the discussion from the thread.
Thanks for resurrecting this patch and for the summary paragraph.
Thanks from me, as well. I noticed that we had achieved three billion percent progress on the repository that brought this to our attention, but didn't notice that you had already written these patches.
So, I am glad that they are getting the attribution that they deserve. Thanks again both.
Thanks,
Taylor
Re: [PATCH 1/2] commit-graph: don't show progress percentages while expanding reachable commits
In reply to Jeff KingI was going to make a bug-report about this funny behavior, but decided to search the list first.
Yeah, I smiled at
Expanding reachable commits in commit graph: 139276% (870481/625), done.
when I did the "git gc --prune=now" on the kernel, and apparently actually looked at the noise for the first time in a while.
So I don't mind the bug, but I think this should be fixed. Because almost 140 _thousand_ percent just isn't right.
So Ack on this patch. Looks like it is already in 'pu', but hasn't made it into next or master yet.
Linus
Quoted 6 lines · Jeff King
From: SZEDER Gábor <[email protected]>
Commit
49bbc57a57(commit-graph write: emit a percentage for all progress, 2019-01-19) was a bit overeager when it added progress percentages/ [...]
PATCH 2/2 commit-graph: turn off save_commit_buffer
In reply to Jeff KingThe commit-graph tool may read a lot of commits, but it only cares about parsing their metadata (parents, trees, etc) and doesn't ever show the messages to the user. And so it should not need save_commit_buffer, which is meant for holding onto the object data of parsed commits so that we can show them later. In fact, it's quite harmful to do so. According to massif, the max heap of "git commit-graph write --reachable" in linux.git before/after this patch (removing the commit graph file in between) goes from ~1.1GB to ~270MB.
Which isn't surprising, since the difference is about the sum of the uncompressed sizes of all commits in the repository, and this was equivalent to leaking them.
This obviously helps if you're under memory pressure, but even without it, things go faster. My before/after times for that command (without massif) went from 12.521s to 11.874s, a speedup of ~5%.
---
We didn't actually notice this on linux.git, but rather on a repository with 130 million commits (don't ask). With this patch, I was able to generate the commit-graph file with a peak heap of ~25GB, which is ~200 bytes per commit.
I'll bet we could do better with some effort, but obviously this case was just pathological. For most cases this should be cheaper than a normal repack (which probably spends that much memory on each object, not just commits).
builtin/commit-graph.c | 2 ++ 1 file changed, 2 insertions(+)
builtin/commit-graph.c +2 -0
@@ -251,6 +251,8 @@ int cmd_commit_graph(int argc, const char **argv, const char *prefix) builtin_commit_graph_usage, PARSE_OPT_STOP_AT_NON_OPTION); + save_commit_buffer =0 ; +if (argc >0 ) {if (!strcmp (argv[0 ],"read" ))return graph_read (argc, argv);
Signature · 1 line
2.23.0.474.gb1abd76f7a
Re: [PATCH 2/2] commit-graph: turn off save_commit_buffer
Quoted 37 lines · Jeff King
The commit-graph tool may read a lot of commits, but it only cares about parsing their metadata (parents, trees, etc) and doesn't ever show the messages to the user. And so it should not need
save_commit_buffer, which is meant for holding onto the object data of parsed commits so that we can show them later. In fact, it's quite harmful to do so. According to massif, the max heap of "git commit-graph write --reachable" in linux.git before/after this patch (removing the commit graph file in between) goes from ~1.1GB to ~270MB.Which isn't surprising, since the difference is about the sum of the uncompressed sizes of all commits in the repository, and this was equivalent to leaking them.
This obviously helps if you're under memory pressure, but even without it, things go faster. My before/after times for that command (without massif) went from 12.521s to 11.874s, a speedup of ~5%.
Signed-off-by Jeff King <[email protected]>---
We didn't actually notice this on linux.git, but rather on a repository with 130 million commits (don't ask). With this patch, I was able to generate the commit-graph file with a peak heap of ~25GB, which is ~200 bytes per commit.I'll bet we could do better with some effort, but obviously this case was just pathological. For most cases this should be cheaper than a normal repack (which probably spends that much memory on each object, not just commits).
builtin/commit-graph.c | 2 ++ 1 file changed, 2 insertions(+)builtin/commit-graph.c +0
-0
builtin/commit-graph.c int cmd_commit_graph(int argc, const char **argv, const char *prefix) · line 255
Quoted 5 lines
builtin/commit-graph.c +2
-0@@ -251,6 +251,8 @@ int cmd_commit_graph(int argc, const char **argv, const char *prefix) builtin_commit_graph_usage, PARSE_OPT_STOP_AT_NON_OPTION); + save_commit_buffer =0 ; +
This looks exactly right to me. We had discussed a little bit off-list about where you might place this line, but I think that the spot you picked is perfect: as late as possible.
Thankfully, the option parsing code here doesn't load any commits (though even if it did, I don't think that turning on/off 'save_commit_buffer' would really make much of a difference).
So, the patch here looks obviously correct, and I don't think it needs a test or anything like that... besides: what is there to test? :).
Quoted 5 lines
builtin/commit-graph.c +0
-1if (argc >0 ) {if (!strcmp (argv[0 ],"read" ))return graph_read (argc, argv); --2.23.0.474.gb1abd76f7a
Thanks,
Taylor
Re: [PATCH 2/2] commit-graph: turn off save_commit_buffer
Quoted 2 lines · Taylor Blau
So, the patch here looks obviously correct, and I don't think it needs a test or anything like that... besides: what is there to test? :).
There's no functional change, so as long as this has coverage in the regular suite (and I think it does), there's no point in adding a new functional test.
However, it may make sense to get some coverage of commit graphs in the perf suite, so that we could detect regressions there. I think that would make sense as a separate series, though, not attached to this particular fix. (It would also be cool if the perf suite could record peak memory usage, but that would take a fair bit of refactoring).
-Peff