builtin-commit: re-read file index before launching editor

From: Samuel Yvon <[email protected]>

Changes made within a pre-commit hook are not reflected within the editor (for instance, with git commit --verbose) because the index is re-read after the editor has been used.

This has the consequence of not displaying the actual changes made by the pre-commit hook, but committing them anyways.

While it is often argued that the hook's purpose is not to automatically write content to the repository, it is acknowledged that using the pre-commit to apply mechanical fixes on top of the existing changes is a supported use case, along with verifying the content.

I think not seeing these mechanical fixes before commiting is incorrect. A developer might expect the commit to look a certain way but if the pre-commit goes wrong the invalid changes will go unnoticed until committed.

---
builtin-commit: Re-read file index before launching editor

Changes made within a pre-commit hook are not reflected within the editor (for instance, with git commit --verbose) because the index is re-read after the editor has been used.

This has the consequence of not displaying the actual changes made by the pre-commit hook, but committing them anyways.

While it is often argued that the hook's purpose is not to automatically write content to the repository, it is acknowledged that using the pre-commit to apply mechanical fixes on top of the existing changes is a supported use case, along with verifying the content.

I think not seeing these mechanical fixes before commiting is incorrect. A developer might expect the commit to look a certain way but if the pre-commit goes wrong the invalid changes will go unnoticed until committed.

I had a small exchange in the Google Group before submitting this Pr. Here is a link for cross-referencing:
https://groups.google.com/g/git-mentoring/c/FsP83I9mN6c

As a side note, I do not know who manages the Github Repo but the following description threw me off a little bit:

Git Source Code Mirror - This is a publish-only repository and all pull requests are ignored.

since after looking deeper it seems the PRs are not ignored.

Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-git-1127%2FSamuelYvon%2Fmaint-v1
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-git-1127/SamuelYvon/maint-v1
Pull-Request: https://github.com/git/git/pull/1127

 builtin/commit.c | 21 +++++++++++----------
 1 file changed, 11 insertions(+), 10 deletions(-)

index 7c9b1e7be3a..e75b11d1c60 100644
@@ -728,8 +728,17 @@ static int prepare_to_commit(const char *index_file, const char *prefix,
 	/* This checks and barfs if author is badly specified */
 	determine_author_info(author_ident);
 
-	if (!no_verify && run_commit_hook(use_editor, index_file, "pre-commit", NULL))
-		return 0;
+	if (!no_verify && find_hook("pre-commit")) {
+		if(run_commit_hook(use_editor, index_file, "pre-commit", NULL))
+			return 0;
+
+		/*
+		 * Re-read the index as pre-commit hook could have updated it,
+		 * and write it out as a tree.
+		 */
+		discard_cache();
+		read_cache_from(index_file);
+	}
 
 	if (squash_message) {
 		/*

@@ -1051,14 +1060,6 @@ static int prepare_to_commit(const char *index_file, const char *prefix,
 		return 0;
 	}
 
-	if (!no_verify && find_hook("pre-commit")) {
-		/*
-		 * Re-read the index as pre-commit hook could have updated it,
-		 * and write it out as a tree.  We must do this before we invoke
-		 * the editor and after we invoke run_status above.
-		 */
-		discard_cache();
-	}
 	read_cache_from(index_file);
 
 	if (update_main_cache_tree(0)) {

base-commit: 5fbd2fc5997dfa4d4593a862fe729b1e7a89bcf8

gitgitgadget

This hook logic is actively being changed these days, we're currently at batch 2/3 of 3/3 (or was it 4/4? I forget) to rewrite all these hook interfaces.

I haven't looked carefully at this, but I've got a patch that touches this exact logic here that'll be in-flight before too long:
https://lore.kernel.org/git/[email protected]/

Anyway, what you have here seems orthagonal, and it looks like these two could be combined, but on top of my linked patch we wouldn't be racy with the new behavior either.

But (and I've got to run) check out the commit message of that patch, perhaps it links to useful past discussions/commits.

I.e. a thing not covered in your commit message is a discussion of if the current behavior was explicitly desired by anyone, or if we just ended up with it by accident.

The code you're moving around has a comment which seems to suggest that what you want *is* the desired behavior, i.e. we re-read it before invoking the editor, so we should have the updated diff, but just don't?

Or maybe I'm misunderstanding it.

From: Samuel Yvon <[email protected]>

My understanding is that it was once the behaviour and has changed over time. I am saying this based on
https://lore.kernel.org/git/[email protected]/t/#u

Specifically,

Looking at ec84bd00 (git-commit: Refactor creation of log message., 2008-02-05), we can see that the editor is launched after the cache has been reset. The only part that troubles me is the line in the comment that specifies that "we must do this ... after we invoke run_status above". I have tested (with my limited knowledge of the internals of git) and it seems to be of no consequence of flushing before the call to run_status, but I might be missing something.

I think this is the case (based on the previously linked conversation).

*nod*, the implicit suggestion here being: Let's put more of that summary into the commit message. It helps when looking up/discovering older behavior.

The comment was first added in 2888605c649 (builtin-commit: fix partial-commit support, 2007-11-18), and quite suspicuous in timing is f5bbc3225c4 (Port git commit to C., 2007-11-08) where we moved from git-commit.sh.

It's a bit of a pain to build git that old, but my hunch is that perhaps this was tested with git-commit.sh, where the reading of the index would be another process.

Sorry! I will prepare an amended patch later. Exploring the linked commits has raised more questions on my end.

Exploring these commits, I fear I might have been confused; the editor was always launched after the reset (still today). The problem, as noted earlier, is the fact that the run_status is ran before the cache reset, which creates the outdated diff. I can't speculate to the intent of the author, but exploring the code makes this comment:

You seem to be quoting the thread over and over, but what you are quoting is somewhat different from the concluding part of what I said, which was:

If I have to guess, I think the reason is because pre-commit automation is expected to be some sort of mechanical change and not part of the actual work that the end-user produced, it would become easier to perform the "final review" of "what have I done so far---does everything make sense?" if such "extra" changes are excluded.

So, in short, it is not "undefined", but rather it seems to be a designed behaviour that we are seeing.

I do not personally mind if we change the philosophy but because it has been a longstanding designed behaviour, it may need a careful transition plan.

Apologies if you feel I misquoted you. I am confused by the comment in the original code w.r.t the location of the cache reset, which (from my understanding) is contradictory with the quoted conversation and so I focused on those parts.

Out of curiosity, what would that involve?

We need to make sure we do not break workflows of existing users who relies on the current behaviour.

As Git is widely used, it is very likely that some of the workflow element existing users use depend on the current behaviour.

Letting pre-commit do its thing and letting run_status() compute "commitable" without its effect may be something they depend on. It is the same for any other behaviour we may be tempted to modify. We would be breaking Git for them by changing.

So, not breaking by not changing behaviour, if we can do so, would be ideal. Introducing a configuration option and hide the new and different behaviour behind it, so that only the folks who agree to take responsibility of adjusting to the new behaviour, would be a good way to isolate the existing users from the change.

If we come up with a new behaviour that would make the world a better place if adopted by everybody, we might be tempted to eventually make it the _only_ behaviour available to everybody. I am not sure if it is the case here, but if it were, the transition plan would become even more involved. We'd need to start by warning existing users (i.e. the code must detect a case where pre-commit mucks with the index and having an extra "re-read index file" would have made a difference---and tell the user that they must adjust to the change of behaviour in the future), wait for a few years and then flip the behaviour while removing the warning, or something like that.

Hi Samuel,

I would be in favor of changing this description. How about:

Git Source Code Mirror - Pull Requests can be submitted via GitGitGadget

Thoughts?

Ciao,
Dscho

I think it might be somewhat obscure for newcomers (What is GitGitGadget?). I know Github supports links in project descriptions. Maybe

Git Source Code Mirror - Please read https://gitgitgadget.github.io/ before submitting Pull Requests

This would have helped me figure out if submitting a PR was gonna end up nowhere or not.

Presense of GGG does not change the fact that the repository is publish only. But "all pull requests are ignored" is not correct anymore.

Git Source Code Mirror - This is a publish-only repository but pull requests can be turned into patches to the mailing list via GitGitGadget. Please follow Documentation/SubmittingPatches procedure for any of your improvements.

I didn't double-check but I presume SubmittingPatches these days talk about GGG?

I don't think so. Documentation/MyFirstContribution.txt does as of 76644e3268 (documentation: add tutorial for first contribution, 2019-05-17), but the same treatment was not applied elsewhere.

I would not be sad to see a reference to GGG be made in SubmittingPatches.

Thanks,
Taylor

It does not, but it links to Documentation/MyFirstContribution, which does.

I think this is descriptive enough so that newcomers can find their way.

Hi,

Excellent. Junio had made the adjustment, and I merely added a link to GitGitGadget's high-level description.

Thank you, all,
Dscho

From: Samuel Yvon <[email protected]>

Changes made within a pre-commit hook are not reflected within the editor (for instance, with git commit --verbose) because the index is re-read after the status has been written.

This has the consequence of not displaying the actual changes made by the pre-commit hook, but committing them anyways.

While it is often argued that the hook's purpose is not to automatically write content to the repository, it is acknowledged that using the pre-commit to apply mechanical fixes on top of the existing changes is a supported use case, along with verifying the content.

I think not seeing these mechanical fixes before commiting is incorrect. A developer might expect the commit to look a certain way but if the pre-commit goes wrong the invalid changes will go unnoticed until committed.

The comment suggesting that the cache must be reset after run_status and before the editor being launched was added in ec84bd00, (git-commit: Refactor creation of log message., 2008-02-05). It is unclear why the run_status must be called *after* the cache reset. However, calling run_status after the cache reset does not update the status line to state of the current index in the case a pre-commit hook is ran and changes files in the staging area.

---
builtin-commit: re-read file index before run_status

Changes made within a pre-commit hook are not reflected within the editor (for instance, with git commit --verbose) because the index is re-read after the editor has been used.

This has the consequence of not displaying the actual changes made by the pre-commit hook, but committing them anyways.

While it is often argued that the hook's purpose is not to automatically write content to the repository, it is acknowledged that using the pre-commit to apply mechanical fixes on top of the existing changes is a supported use case, along with verifying the content.

I think not seeing these mechanical fixes before commiting is incorrect. A developer might expect the commit to look a certain way but if the pre-commit goes wrong the invalid changes will go unnoticed until committed.

I had a small exchange in the Google Group before submitting this Pr. Here is a link for cross-referencing:
https://groups.google.com/g/git-mentoring/c/FsP83I9mN6c

As a side note, I do not know who manages the Github Repo but the following description threw me off a little bit:

Git Source Code Mirror - This is a publish-only repository and all pull requests are ignored.

since after looking deeper it seems the PRs are not ignored.

Changes since v1:

* Edited the title to more accurately reflect the changes
* More details in commit messages

Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-git-1127%2FSamuelYvon%2Fmaint-v2
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-git-1127/SamuelYvon/maint-v2
Pull-Request: https://github.com/git/git/pull/1127

Range-diff vs v1:

1: 271bcf89d1e ! 1: f09244a0ba3 builtin-commit: re-read file index before launching editor
@@ Metadata
Author: Samuel Yvon <[email protected]>

## Commit message ##
- builtin-commit: re-read file index before launching editor
+ builtin-commit: re-read file index before run_status

Changes made within a pre-commit hook are not reflected within the editor (for instance, with git commit --verbose) because
- the index is re-read after the editor has been used.
+ the index is re-read after the status has been written.

This has the consequence of not displaying the actual changes made by the pre-commit hook, but committing them anyways.
@@ Commit message
pre-commit goes wrong the invalid changes will go unnoticed until committed.

+ The comment suggesting that the cache must be reset after run_status
+ and before the editor being launched was added in ec84bd00,
+ (git-commit: Refactor creation of log message., 2008-02-05). It is
+ unclear why the run_status must be called *after* the cache reset.
+ However, calling run_status after the cache reset does not update
+ the status line to state of the current index in the case a
+ pre-commit hook is ran and changes files in the staging area.
+

## builtin/commit.c ##

 builtin/commit.c | 21 +++++++++++----------
 1 file changed, 11 insertions(+), 10 deletions(-)

index 7c9b1e7be3a..e75b11d1c60 100644
@@ -728,8 +728,17 @@ static int prepare_to_commit(const char *index_file, const char *prefix,
 	/* This checks and barfs if author is badly specified */
 	determine_author_info(author_ident);
 
-	if (!no_verify && run_commit_hook(use_editor, index_file, "pre-commit", NULL))
-		return 0;
+	if (!no_verify && find_hook("pre-commit")) {
+		if(run_commit_hook(use_editor, index_file, "pre-commit", NULL))
+			return 0;
+
+		/*
+		 * Re-read the index as pre-commit hook could have updated it,
+		 * and write it out as a tree.
+		 */
+		discard_cache();
+		read_cache_from(index_file);
+	}
 
 	if (squash_message) {
 		/*

@@ -1051,14 +1060,6 @@ static int prepare_to_commit(const char *index_file, const char *prefix,
 		return 0;
 	}
 
-	if (!no_verify && find_hook("pre-commit")) {
-		/*
-		 * Re-read the index as pre-commit hook could have updated it,
-		 * and write it out as a tree.  We must do this before we invoke
-		 * the editor and after we invoke run_status above.
-		 */
-		discard_cache();
-	}
 	read_cache_from(index_file);
 
 	if (update_main_cache_tree(0)) {

base-commit: 5fbd2fc5997dfa4d4593a862fe729b1e7a89bcf8

gitgitgadget

An older thread cited earlier suspected that it is to reflect the changes given to "git commit" proper, excluding whatever pre-commit did, to the return value of run_status(), which becomes the value of committable variable to answer "do we have anything to commit?".

And moving the call would affect both the contents of the status buffer (i.e. the list of paths got changed starts including what pre-commit did) and the "committable" bit by counting such a change as a true change, avoiding the "no empty commit by default" check, in a consistent way, hopefully. I wonder if we have test to demonstrate that, and if there isn't perhaps we would want to add one.

And if this change also affects the "committable" assignment in a consistent way, it should probably want to be mentioned in this paragraph, too.

I am not convinced by the claim that there is no need for careful transition plans (yet), but I personally agree with the end state (with the above suggested tweaks, that is).

Thanks for working on the topic.

Apologies for the time I took to reply,

So, just to make sure I understand well, the concern is that an empty commit would trigger the commit routine, run the pre-commit hook, which may add files (thus making it an non-empty commit) and then push 100% automatic changes to a repo. I agree that this would be invalid behaviour and very odd, I will make sure no empty commit is allowed.

What do you mean by "commitable assignment"?

With the last message, I agree the safest option is probably to leave this configurable for now and off by default.

So here's the next steps that I intend to take to get this merged in:

- Add a test for empty commit (if non-existent) and ensure the behaviour is the same
- Add a config option (or maybe a switch?) for migration purposes, with the default being the current behaviour.

Assignment to the committable variable in the codepath you are looking at, was what I meant.