Skip to contentSkip to conversation

experimental: default to fetch.writeCommitGraph=false

PATCH In review Jonathan Nieder 4 participants 7 messages Jul 7, 2020 – Jul 8, 2020 Documentation

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 experimental: default to fetch.writeCommitGraph=false

The fetch.writeCommitGraph feature makes fetches write out a commit graph file for the newly downloaded pack on fetch. This improves the performance of various commands that would perform a revision walk and eventually ought to be the default for everyone. To prepare for that future, it's enabled by default for users that set feature.experimental=true to experience such future defaults.

Alas, for --unshallow fetches from a shallow clone it runs into a
snag: by the time Git has fetched the new objects and is writing a commit graph, it has performed a revision walk and r->parsed_objects contains information about the shallow boundary from *before* the fetch. The commit graph writing code is careful to avoid writing a commit graph file in shallow repositories, but the new state is not shallow, and the result is that from that point on, commands like "git log" make use of a newly written commit graph file representing a fictional history with the old shallow boundary.

We could fix this by making the commit graph writing code more careful to avoid writing a commit graph that could have used any grafts or shallow state, but it is possible that there are other pieces of mutated state that fetch's commit graph writing code may be relying on. So disable it in the feature.experimental configuration.

Google developers have been running in this configuration (by setting fetch.writeCommitGraph=false in the system config) to work around this bug since it was discovered in April. Once the fix lands, we'll enable fetch.writeCommitGraph=true again to give it some early testing before rolling out to a wider audience.

In other words:

- this patch only affects behavior with feature.experimental=true

- it makes feature.experimental match the configuration Google has been using for the last few months, meaning it would leave users in a better tested state than without it

- this should improve testing for other features guarded by feature.experimental, by making feature.experimental safer to use

---
I realize this is late to send. That said, as described above, I think it's a good way to buy time by minimizing user exposure to fetch.writeCommitGraph=true until a fix for it is well cooked.

In other words, I'd like to see this patch in Git 2.28-rc0.

Thanks of all kinds welcome, as always. Previous discussion:
https://lore.kernel.org/git/[email protected]/

 Documentation/config/feature.txt | 8 --------
 Documentation/config/fetch.txt   | 3 +--
 repo-settings.c                  | 8 ++++----
 3 files changed, 5 insertions(+), 14 deletions(-)
Documentation/config/feature.txt +0 -8
index 28c33602d52..c0cbf2bb1cd 100644
@@ -15,14 +15,6 @@ feature.experimental::
 * `fetch.negotiationAlgorithm=skipping` may improve fetch negotiation times by
 skipping more commits at a time, reducing the number of round trips.
 +
-* `fetch.writeCommitGraph=true` writes a commit-graph after every `git fetch`
-command that downloads a pack-file from a remote. Using the `--split` option,
-most executions will create a very small commit-graph file on top of the
-existing commit-graph file(s). Occasionally, these files will merge and the
-write may take longer. Having an updated commit-graph file helps performance
-of many Git commands, including `git merge-base`, `git push -f`, and
-`git log --graph`.
-+
 * `protocol.version=2` speeds up fetches from repositories with many refs by
 allowing the client to specify which refs to list before the server lists
 them.
Documentation/config/fetch.txt +1 -2
index b1a9b1461d3..b20394038d1 100644
@@ -90,5 +90,4 @@ fetch.writeCommitGraph::
 	the existing commit-graph file(s). Occasionally, these files will
 	merge and the write may take longer. Having an updated commit-graph
 	file helps performance of many Git commands, including `git merge-base`,
-	`git push -f`, and `git log --graph`. Defaults to false, unless
-	`feature.experimental` is true.
+	`git push -f`, and `git log --graph`. Defaults to false.
repo-settings.c +4 -4
index dc6817daa95..0918408b344 100644
@@ -51,14 +51,14 @@ void prepare_repo_settings(struct repository *r)
 		UPDATE_DEFAULT_BOOL(r->settings.index_version, 4);
 		UPDATE_DEFAULT_BOOL(r->settings.core_untracked_cache, UNTRACKED_CACHE_WRITE);
 	}
+
 	if (!repo_config_get_bool(r, "fetch.writecommitgraph", &value))
 		r->settings.fetch_write_commit_graph = value;
-	if (!repo_config_get_bool(r, "feature.experimental", &value) && value) {
-		UPDATE_DEFAULT_BOOL(r->settings.fetch_negotiation_algorithm, FETCH_NEGOTIATION_SKIPPING);
-		UPDATE_DEFAULT_BOOL(r->settings.fetch_write_commit_graph, 1);
-	}
 	UPDATE_DEFAULT_BOOL(r->settings.fetch_write_commit_graph, 0);
 
+	if (!repo_config_get_bool(r, "feature.experimental", &value) && value)
+		UPDATE_DEFAULT_BOOL(r->settings.fetch_negotiation_algorithm, FETCH_NEGOTIATION_SKIPPING);
+
 	/* Hack for test programs like test-dump-untracked-cache */
 	if (ignore_untracked_cache_config)
 		r->settings.core_untracked_cache = UNTRACKED_CACHE_KEEP;
Signature · 1 line
2.27.0.383.g050319c2ae
#1 lore raw
Quoted 40 lines · Jonathan Nieder

The fetch.writeCommitGraph feature makes fetches write out a commit graph file for the newly downloaded pack on fetch. This improves the performance of various commands that would perform a revision walk and eventually ought to be the default for everyone. To prepare for that future, it's enabled by default for users that set feature.experimental=true to experience such future defaults.

Alas, for --unshallow fetches from a shallow clone it runs into a
snag: by the time Git has fetched the new objects and is writing a commit graph, it has performed a revision walk and r->parsed_objects contains information about the shallow boundary from *before* the fetch. The commit graph writing code is careful to avoid writing a commit graph file in shallow repositories, but the new state is not shallow, and the result is that from that point on, commands like "git log" make use of a newly written commit graph file representing a fictional history with the old shallow boundary.

We could fix this by making the commit graph writing code more careful to avoid writing a commit graph that could have used any grafts or shallow state, but it is possible that there are other pieces of mutated state that fetch's commit graph writing code may be relying on. So disable it in the feature.experimental configuration.

Google developers have been running in this configuration (by setting fetch.writeCommitGraph=false in the system config) to work around this bug since it was discovered in April. Once the fix lands, we'll enable fetch.writeCommitGraph=true again to give it some early testing before rolling out to a wider audience.

In other words:

- this patch only affects behavior with feature.experimental=true

- it makes feature.experimental match the configuration Google has been using for the last few months, meaning it would leave users in a better tested state than without it

- this should improve testing for other features guarded by feature.experimental, by making feature.experimental safer to use

Quoted 7 lines

---
I realize this is late to send. That said, as described above, I think it's a good way to buy time by minimizing user exposure to fetch.writeCommitGraph=true until a fix for it is well cooked.

While it would certainly be better to fix the commit_graph_compatible() method, I understand that that is a more complicated problem.

Quoted 1 line

In other words, I'd like to see this patch in Git 2.28-rc0.

This patch is 100% correct.

Normally, I would say "this is experimental, and a user can always disable fetch.writeCommitGraph manually if they are running into this." This is especially true because unshallowing a repo is (probably) a rare operation. At least, I expect that very few users actually do it, and those who do are expert users.

But, it is best to reduce user pain, even in rare cases like this.

Further, I hope to submit new maintenance tasks soon which can replace fetch.writeCommitGraph.

Thanks,
-Stolee

#2 lore raw
In reply to Jonathan Nieder
Quoted 4 lines · Jonathan Nieder

The fetch.writeCommitGraph feature makes fetches write out a commit

In other words:

Quoted 8 lines

- this patch only affects behavior with feature.experimental=true

- it makes feature.experimental match the configuration Google has been using for the last few months, meaning it would leave users in a better tested state than without it

- this should improve testing for other features guarded by feature.experimental, by making feature.experimental safer to use

In other words, fetch.writeCommitGraph in its current form is too broken to be recommended even for brave souls with "experimental" bit on.

I wonder if we perhaps wnat to add to the documentation for writeCommitGraph configuration that its use is currently not recommended in a shallow clone or something (I know it is not a problem just to use it with shallow but the breakage needs to involve unshallowing, but by definition those who do not use shallow would not hit the unshallowing bug, so...).

Quoted 9 lines

---
I realize this is late to send. That said, as described above, I think it's a good way to buy time by minimizing user exposure to fetch.writeCommitGraph=true until a fix for it is well cooked.

In other words, I'd like to see this patch in Git 2.28-rc0.

Yes, I do, too.

Thanks.

Quoted 34 lines
Documentation/config/fetch.txt +1 -2
index b1a9b1461d3..b20394038d1 100644
@@ -90,5 +90,4 @@ fetch.writeCommitGraph::
 	the existing commit-graph file(s). Occasionally, these files will
 	merge and the write may take longer. Having an updated commit-graph
 	file helps performance of many Git commands, including `git merge-base`,
-	`git push -f`, and `git log --graph`. Defaults to false, unless
-	`feature.experimental` is true.
+	`git push -f`, and `git log --graph`. Defaults to false.
repo-settings.c +4 -4
index dc6817daa95..0918408b344 100644
@@ -51,14 +51,14 @@ void prepare_repo_settings(struct repository *r)
 		UPDATE_DEFAULT_BOOL(r->settings.index_version, 4);
 		UPDATE_DEFAULT_BOOL(r->settings.core_untracked_cache, UNTRACKED_CACHE_WRITE);
 	}
+
 	if (!repo_config_get_bool(r, "fetch.writecommitgraph", &value))
 		r->settings.fetch_write_commit_graph = value;
-	if (!repo_config_get_bool(r, "feature.experimental", &value) && value) {
-		UPDATE_DEFAULT_BOOL(r->settings.fetch_negotiation_algorithm, FETCH_NEGOTIATION_SKIPPING);
-		UPDATE_DEFAULT_BOOL(r->settings.fetch_write_commit_graph, 1);
-	}
 	UPDATE_DEFAULT_BOOL(r->settings.fetch_write_commit_graph, 0);
 
+	if (!repo_config_get_bool(r, "feature.experimental", &value) && value)
+		UPDATE_DEFAULT_BOOL(r->settings.fetch_negotiation_algorithm, FETCH_NEGOTIATION_SKIPPING);
+
 	/* Hack for test programs like test-dump-untracked-cache */
 	if (ignore_untracked_cache_config)
 		r->settings.core_untracked_cache = UNTRACKED_CACHE_KEEP;
#3 lore raw

Hi Junio,

Quoted 20 lines · Junio C Hamano
Quoted 4 lines · Jonathan Nieder

The fetch.writeCommitGraph feature makes fetches write out a commit

In other words:

Quoted 8 lines

- this patch only affects behavior with feature.experimental=true

- it makes feature.experimental match the configuration Google has been using for the last few months, meaning it would leave users in a better tested state than without it

- this should improve testing for other features guarded by feature.experimental, by making feature.experimental safer to use

In other words, fetch.writeCommitGraph in its current form is too broken to be recommended even for brave souls with "experimental" bit on.

Quoted 6 lines

I wonder if we perhaps wnat to add to the documentation for writeCommitGraph configuration that its use is currently not recommended in a shallow clone or something (I know it is not a problem just to use it with shallow but the breakage needs to involve unshallowing, but by definition those who do not use shallow would not hit the unshallowing bug, so...).

I think this is a good direction if you don't want to take the patch I sent in [1] for v2.28.0. If you do, though, I don't think that this would be necessary.

Quoted 42 lines
Quoted 9 lines

---
I realize this is late to send. That said, as described above, I think it's a good way to buy time by minimizing user exposure to fetch.writeCommitGraph=true until a fix for it is well cooked.

In other words, I'd like to see this patch in Git 2.28-rc0.

Yes, I do, too.

Thanks.

Quoted 28 lines
Documentation/config/fetch.txt +1 -2
index b1a9b1461d3..b20394038d1 100644
@@ -90,5 +90,4 @@ fetch.writeCommitGraph::
 	the existing commit-graph file(s). Occasionally, these files will
 	merge and the write may take longer. Having an updated commit-graph
 	file helps performance of many Git commands, including `git merge-base`,
-	`git push -f`, and `git log --graph`. Defaults to false, unless
-	`feature.experimental` is true.
+	`git push -f`, and `git log --graph`. Defaults to false.
repo-settings.c +1 -4
index dc6817daa95..0918408b344 100644
@@ -51,14 +51,14 @@ void prepare_repo_settings(struct repository *r)
 		UPDATE_DEFAULT_BOOL(r->settings.index_version, 4);
 		UPDATE_DEFAULT_BOOL(r->settings.core_untracked_cache, UNTRACKED_CACHE_WRITE);
 	}
+
 	if (!repo_config_get_bool(r, "fetch.writecommitgraph", &value))
 		r->settings.fetch_write_commit_graph = value;
-	if (!repo_config_get_bool(r, "feature.experimental", &value) && value) {
-		UPDATE_DEFAULT_BOOL(r->settings.fetch_negotiation_algorithm, FETCH_NEGOTIATION_SKIPPING);
-		UPDATE_DEFAULT_BOOL(r->settings.fetch_write_commit_graph, 1);
-	}
 	UPDATE_DEFAULT_BOOL(r->settings.fetch_write_commit_graph, 0);
 
Quoted 6 lines

repo-settings.c void prepare_repo_settings(struct repository *r) · line 64

Quoted 6 lines
repo-settings.c +3 -0
+	if (!repo_config_get_bool(r, "feature.experimental", &value) && value)
+		UPDATE_DEFAULT_BOOL(r->settings.fetch_negotiation_algorithm, FETCH_NEGOTIATION_SKIPPING);
+
 	/* Hack for test programs like test-dump-untracked-cache */
 	if (ignore_untracked_cache_config)
 		r->settings.core_untracked_cache = UNTRACKED_CACHE_KEEP;

Thanks,
Taylor

[1]: https://lore.kernel.org/git/[email protected]/T/#t

#4 lore raw
Quoted 10 lines · Taylor Blau
Quoted 6 lines

I wonder if we perhaps wnat to add to the documentation for writeCommitGraph configuration that its use is currently not recommended in a shallow clone or something (I know it is not a problem just to use it with shallow but the breakage needs to involve unshallowing, but by definition those who do not use shallow would not hit the unshallowing bug, so...).

I think this is a good direction if you don't want to take the patch I sent in [1] for v2.28.0. If you do, though, I don't think that this would be necessary.

Good timing. I didn't know a "fix" was already being worked on ([1] is the patch from this morning, right? I haven't seen it except for its subject).

We could obviously do both excluding it from the usual experimental set and applying your fix, so that those who are really curious can help us make sure your fix would be all that is needed. Let's see what Jonathan says...

Thanks.

#5 lore raw
Quoted 9 lines · Junio C Hamano
Quoted 7 lines · Taylor Blau
Quoted 6 lines

I wonder if we perhaps wnat to add to the documentation for writeCommitGraph configuration that its use is currently not recommended in a shallow clone or something (I know it is not a problem just to use it with shallow but the breakage needs to involve unshallowing, but by definition those who do not use shallow would not hit the unshallowing bug, so...).

Quoted 7 lines
Quoted 3 lines

I think this is a good direction if you don't want to take the patch I sent in [1] for v2.28.0. If you do, though, I don't think that this would be necessary.

Good timing. I didn't know a "fix" was already being worked on ([1] is the patch from this morning, right? I haven't seen it except for its subject).

[1] is the fix. Jonathan wrote it a month or so ago, I just added a test on top. (Independently, I tested it with the reproduction in the original bug report, and it worked properly).

Quoted 4 lines

We could obviously do both excluding it from the usual experimental set and applying your fix, so that those who are really curious can help us make sure your fix would be all that is needed. Let's see what Jonathan says...

Either of those sound good to me.

Quoted 1 line

Thanks.

Thanks,
Taylor

#6 lore raw
In reply to Junio C Hamano
Quoted 4 lines · Junio C Hamano

We could obviously do both excluding it from the usual experimental set and applying your fix, so that those who are really curious can help us make sure your fix would be all that is needed. Let's see what Jonathan says...

Yes, that would be my preference.

That is, both:

* applying the fix to provide a good experience to users interested in fetch.writeCommitGraph

* disabling fetch.writeCommitGraph in the experimental set, since it has not had much production exposure yet. The experimental set is relatively young, so I want to ensure people's initial experiences with it are positive so that they stick with it if they're interested in experimental features (or in other words, I think there's still a place for features that are not yet proven enough to go in the experimental set).

Regardless of what is put in the experimental set, at $DAYJOB we will run with the fix applied and with fetch.writeCommitGraph=true starting next week. I'd encourage anyone else with a similarly controlled setup to try the same.

Thanks,
Jonathan