Skip to contentSkip to conversation

rebase: update branches in multi-part topic

PATCH Superseded Derrick Stolee via GitGitGadget 8 participants 144 messages Jun 3, 2022 – Jul 21, 2022 Documentation Page 2 of 2

lore
Share & tools
lore
#130 lore raw

PATCH v5 6/12 sequencer: add update-ref command

In reply to Derrick Stolee via GitGitGadget

From: Derrick Stolee <[email protected]>

Add the boilerplate for an "update-ref" command in the sequencer. This connects to the current no-op do_update_ref() which will be filled in after more connections are created.

The syntax in the todo list will be "update-ref <ref-name>" to signal that we should store the current commit as the value for updating <ref-name> at the end of the rebase.

---
 rebase-interactive.c |  3 +++
 sequencer.c          | 14 +++++++++++++-
 sequencer.h          |  1 +
 3 files changed, 17 insertions(+), 1 deletion(-)
rebase-interactive.c +3 -0
index 22394224faa..1ff07647af3 100644
@@ -57,6 +57,9 @@ void append_todo_help(int command_count,
 "        create a merge commit using the original merge commit's\n"
 "        message (or the oneline, if no original merge commit was\n"
 "        specified); use -c <commit> to reword the commit message\n"
+"u, update-ref <ref> = track a placeholder for the <ref> to be updated\n"
+"                      to this position in the new commits. The <ref> is\n"
+"                      updated at the end of the rebase\n"
 "\n"
 "These lines can be re-ordered; they are executed from top to bottom.\n");
 	unsigned edit_todo = !(shortrevisions && shortonto);
sequencer.c +13 -1
index 2711182e43f..0dc9c05c5bb 100644
@@ -1719,6 +1719,7 @@ static struct {
 	[TODO_LABEL] = { 'l', "label" },
 	[TODO_RESET] = { 't', "reset" },
 	[TODO_MERGE] = { 'm', "merge" },
+	[TODO_UPDATE_REF] = { 'u', "update-ref" },
 	[TODO_NOOP] = { 0,   "noop" },
 	[TODO_DROP] = { 'd', "drop" },
 	[TODO_COMMENT] = { 0,   NULL },
@@ -2480,7 +2481,7 @@ static int parse_insn_line(struct repository *r, struct todo_item *item,
 			     command_to_string(item->command));
 
 	if (item->command == TODO_EXEC || item->command == TODO_LABEL ||
-	    item->command == TODO_RESET) {
+	    item->command == TODO_RESET || item->command == TODO_UPDATE_REF) {
 		item->commit = NULL;
 		item->arg_offset = bol - buf;
 		item->arg_len = (int)(eol - bol);
@@ -4079,6 +4080,11 @@ leave_merge:
 	return ret;
 }
 
+static int do_update_ref(struct repository *r, const char *ref_name)
+{
+	return 0;
+}
+
 static int is_final_fixup(struct todo_list *todo_list)
 {
 	int i = todo_list->current;
@@ -4454,6 +4460,12 @@ static int pick_commits(struct repository *r,
 				return error_with_patch(r, item->commit,
 							arg, item->arg_len,
 							opts, res, 0);
+		} else if (item->command == TODO_UPDATE_REF) {
+			struct strbuf ref = STRBUF_INIT;
+			strbuf_add(&ref, arg, item->arg_len);
+			if ((res = do_update_ref(r, ref.buf)))
+				reschedule = 1;
+			strbuf_release(&ref);
 		} else if (!is_noop(item->command))
 			return error(_("unknown command %d"), item->command);
 
sequencer.h +1 -0
index 3ae541bb145..2cf5c1b9a38 100644
@@ -95,6 +95,7 @@ enum todo_command {
 	TODO_LABEL,
 	TODO_RESET,
 	TODO_MERGE,
+	TODO_UPDATE_REF,
 	/* commands that do nothing but are counted for reporting progress */
 	TODO_NOOP,
 	TODO_DROP,
Signature · 1 line
gitgitgadget
#131 lore raw

PATCH v5 7/12 rebase: add --update-refs option

In reply to Derrick Stolee via GitGitGadget

From: Derrick Stolee <[email protected]>

When working on a large feature, it can be helpful to break that feature into multiple smaller parts that become reviewed in sequence. During development or during review, a change to one part of the feature could affect multiple of these parts. An interactive rebase can help adjust the multi-part "story" of the branch.

However, if there are branches tracking the different parts of the feature, then rebasing the entire list of commits can create commits not reachable from those "sub branches". It can take a manual step to update those branches.

Add a new --update-refs option to 'git rebase -i' that adds 'update-ref <ref>' steps to the todo file whenever a commit that is being rebased is decorated with that <ref>. At the very end, the rebase process updates all of the listed refs to the values stored during the rebase operation.

Be sure to iterate after any squashing or fixups are placed. Update the branch only after those squashes and fixups are complete. This allows a --fixup commit at the tip of the feature to apply correctly to the sub branch, even if it is fixing up the most-recent commit in that part.

This change update the documentation and builtin to accept the --update-refs option as well as updating the todo file with the 'update-ref' commands. Tests are added to ensure that these todo commands are added in the correct locations.

This change does _not_ include the actual behavior of tracking the updated refs and writing the new ref values at the end of the rebase process. That is deferred to a later change.

---
 Documentation/git-rebase.txt  |   7 +++
 builtin/rebase.c              |   5 ++
 sequencer.c                   | 107 ++++++++++++++++++++++++++++++++++
 sequencer.h                   |   1 +
 t/t2407-worktree-heads.sh     |  22 +++++++
 t/t3404-rebase-interactive.sh |  74 +++++++++++++++++++++++
 6 files changed, 216 insertions(+)
Documentation/git-rebase.txt +7 -0
index 262fb01aec0..dfd847d6615 100644
@@ -609,6 +609,12 @@ provided. Otherwise an explicit `--no-reschedule-failed-exec` at the
 start would be overridden by the presence of
 `rebase.rescheduleFailedExec=true` configuration.
 
+--update-refs::
+--no-update-refs::
+	Automatically force-update any branches that point to commits that
+	are being rebased. Any branches that are checked out in a worktree
+	are not updated in this way.
+
 INCOMPATIBLE OPTIONS
 --------------------
 
@@ -632,6 +638,7 @@ are incompatible with the following options:
  * --empty=
  * --reapply-cherry-picks
  * --edit-todo
+ * --update-refs
  * --root when used in combination with --onto
 
 In addition, the following pairs of options are incompatible:
builtin/rebase.c +5 -0
index 7ab50cda2ad..fd64897132a 100644
@@ -102,6 +102,7 @@ struct rebase_options {
 	int reschedule_failed_exec;
 	int reapply_cherry_picks;
 	int fork_point;
+	int update_refs;
 };
 
 #define REBASE_OPTIONS_INIT {			  	\
@@ -298,6 +299,7 @@ static int do_interactive_rebase(struct rebase_options *opts, unsigned flags)
 		ret = complete_action(the_repository, &replay, flags,
 			shortrevisions, opts->onto_name, opts->onto,
 			&opts->orig_head, &commands, opts->autosquash,
+			opts->update_refs,
 			&todo_list);
 	}
 
@@ -1124,6 +1126,9 @@ int cmd_rebase(int argc, const char **argv, const char *prefix)
 		OPT_BOOL(0, "autosquash", &options.autosquash,
 			 N_("move commits that begin with "
 			    "squash!/fixup! under -i")),
+		OPT_BOOL(0, "update-refs", &options.update_refs,
+			 N_("update branches that point to commits "
+			    "that are being rebased")),
 		{ OPTION_STRING, 'S', "gpg-sign", &gpg_sign, N_("key-id"),
 			N_("GPG-sign commits"),
 			PARSE_OPT_OPTARG, NULL, (intptr_t) "" },
sequencer.c +107 -0

129 lines not on this page · raw message

sequencer.h +1 -0
index 2cf5c1b9a38..e671d7e0743 100644
@@ -167,6 +167,7 @@ int complete_action(struct repository *r, struct replay_opts *opts, unsigned fla
 		    const char *shortrevisions, const char *onto_name,
 		    struct commit *onto, const struct object_id *orig_head,
 		    struct string_list *commands, unsigned autosquash,
+		    unsigned update_refs,
 		    struct todo_list *todo_list);
 int todo_list_rearrange_squash(struct todo_list *todo_list);
 
t/t2407-worktree-heads.sh +22 -0
index 97f5c87f8c8..8a03f14df8d 100755
@@ -164,4 +164,26 @@ test_expect_success 'refuse to overwrite when in error states' '
 	done
 '
 
+. "$TEST_DIRECTORY"/lib-rebase.sh
+
+test_expect_success !SANITIZE_LEAK 'refuse to overwrite during rebase with --update-refs' '
+	git commit --fixup HEAD~2 --allow-empty &&
+	(
+		set_cat_todo_editor &&
+		test_must_fail git rebase -i --update-refs HEAD~3 >todo &&
+		! grep "update-refs" todo
+	) &&
+	git branch -f allow-update HEAD~2 &&
+	(
+		set_cat_todo_editor &&
+		test_must_fail git rebase -i --update-refs HEAD~3 >todo &&
+		grep "update-ref refs/heads/allow-update" todo
+	)
+'
+
+# This must be the last test in this file
+test_expect_success '$EDITOR and friends are unchanged' '
+	test_editor_unchanged
+'
+
 test_done
t/t3404-rebase-interactive.sh +74 -0

80 lines not on this page · raw message

Signature · 1 line
gitgitgadget
#132 lore raw

PATCH v5 8/12 rebase: update refs from 'update-ref' commands

In reply to Derrick Stolee via GitGitGadget

From: Derrick Stolee <[email protected]>

The previous change introduced the 'git rebase --update-refs' option which added 'update-ref <ref>' commands to the todo list of an interactive rebase.

Teach Git to record the HEAD position when reaching these 'update-ref' commands. The ref/before/after triple is stored in the $GIT_DIR/rebase-merge/update-refs file. A previous change parsed this file to avoid having other processes updating the refs in that file while the rebase is in progress.

Not only do we update the file when the sequencer reaches these 'update-ref' commands, we then update the refs themselves at the end of the rebase sequence. If the rebase is aborted before this final step, then the refs are not updated. The 'before' value is used to ensure that we do not accidentally obliterate a ref that was updated concurrently (say, by an older version of Git or a third-party tool).

Now that the 'git rebase --update-refs' command is implemented to write to the update-refs file, we can remove the fake construction of the update-refs file from a test in t2407-worktree-heads.sh.

---
 sequencer.c                   | 120 +++++++++++++++++++++++++++++++++-
 t/t2407-worktree-heads.sh     |  21 ++----
 t/t3404-rebase-interactive.sh |  17 +++++
 3 files changed, 140 insertions(+), 18 deletions(-)
sequencer.c +117 -3
index a4ee2799d6f..ce8c7d8cd3a 100644
@@ -36,6 +36,7 @@
 #include "rebase-interactive.h"
 #include "reset.h"
 #include "branch.h"
+#include "log-tree.h"
 
 #define GIT_REFLOG_ACTION "GIT_REFLOG_ACTION"
 
@@ -193,6 +194,21 @@ struct update_ref_record {
 	struct object_id after;
 };
 
+static struct update_ref_record *init_update_ref_record(const char *ref)
+{
+	struct update_ref_record *rec;
+
+	CALLOC_ARRAY(rec, 1);
+
+	oidcpy(&rec->before, null_oid());
+	oidcpy(&rec->after, null_oid());
+
+	/* This may fail, but that's fine, we will keep the null OID. */
+	read_ref(ref, &rec->before);
+
+	return rec;
+}
+
 static int git_sequencer_config(const char *k, const char *v, void *cb)
 {
 	struct replay_opts *opts = cb;
@@ -4081,11 +4097,97 @@ leave_merge:
 	return ret;
 }
 
-static int do_update_ref(struct repository *r, const char *ref_name)
+static int write_update_refs_state(struct string_list *refs_to_oids)
+{
+	int result = 0;
+	struct lock_file lock = LOCK_INIT;
+	FILE *fp = NULL;
+	struct string_list_item *item;
+	char *path;
+
+	if (!refs_to_oids->nr)
+		return 0;
+
+	path = rebase_path_update_refs(the_repository->gitdir);
+
+	if (safe_create_leading_directories(path)) {
+		result = error(_("unable to create leading directories of %s"),
+			       path);
+		goto cleanup;
+	}
+
+	if (hold_lock_file_for_update(&lock, path, 0) < 0) {
+		result = error(_("another 'rebase' process appears to be running; "
+				 "'%s.lock' already exists"),
+			       path);
+		goto cleanup;
+	}
+
+	fp = fdopen_lock_file(&lock, "w");
+	if (!fp) {
+		result = error_errno(_("could not open '%s' for writing"), path);
+		rollback_lock_file(&lock);
+		goto cleanup;
+	}
+
+	for_each_string_list_item(item, refs_to_oids) {
+		struct update_ref_record *rec = item->util;
+		fprintf(fp, "%s\n%s\n%s\n", item->string,
+			oid_to_hex(&rec->before), oid_to_hex(&rec->after));
+	}
+
+	result = commit_lock_file(&lock);
+
+cleanup:
+	free(path);
+	return result;
+}
+
+static int do_update_ref(struct repository *r, const char *refname)
 {
+	struct string_list_item *item;
+	struct string_list list = STRING_LIST_INIT_DUP;
+
+	if (sequencer_get_update_refs_state(r->gitdir, &list))
+		return -1;
+
+	for_each_string_list_item(item, &list) {
+		if (!strcmp(item->string, refname)) {
+			struct update_ref_record *rec = item->util;
+			if (read_ref("HEAD", &rec->after))
+				return -1;
+			break;
+		}
+	}
+
+	write_update_refs_state(&list);
+	string_list_clear(&list, 1);
 	return 0;
 }
 
+static int do_update_refs(struct repository *r)
+{
+	int res = 0;
+	struct string_list_item *item;
+	struct string_list refs_to_oids = STRING_LIST_INIT_DUP;
+	struct ref_store *refs = get_main_ref_store(r);
+
+	if ((res = sequencer_get_update_refs_state(r->gitdir, &refs_to_oids)))
+		return res;
+
+	for_each_string_list_item(item, &refs_to_oids) {
+		struct update_ref_record *rec = item->util;
+
+		res |= refs_update_ref(refs, "rewritten during rebase",
+				       item->string,
+				       &rec->after, &rec->before,
+				       0, UPDATE_REFS_MSG_ON_ERR);
+	}
+
+	string_list_clear(&refs_to_oids, 1);
+	return res;
+}
+
 static int is_final_fixup(struct todo_list *todo_list)
 {
 	int i = todo_list->current;
@@ -4601,6 +4703,9 @@ cleanup_head_ref:
 
 		strbuf_release(&buf);
 		strbuf_release(&head_ref);
+
+		if (do_update_refs(r))
+			return -1;
 	}
 
 	/*
@@ -5676,7 +5781,7 @@ static int add_decorations_to_list(const struct commit *commit,
 
 			sti = string_list_insert(&ctx->refs_to_oids,
 						 decoration->name);
-			sti->util = oiddup(the_hash_algo->null_oid);
+			sti->util = init_update_ref_record(decoration->name);
 		}
 
 		item->offset_in_buf = base_offset;
@@ -5696,7 +5801,7 @@ static int add_decorations_to_list(const struct commit *commit,
  */
 static int todo_list_add_update_ref_commands(struct todo_list *todo_list)
 {
-	int i;
+	int i, res;
 	static struct string_list decorate_refs_exclude = STRING_LIST_INIT_NODUP;
 	static struct string_list decorate_refs_exclude_config = STRING_LIST_INIT_NODUP;
 	static struct string_list decorate_refs_include = STRING_LIST_INIT_NODUP;
@@ -5732,7 +5837,16 @@ static int todo_list_add_update_ref_commands(struct todo_list *todo_list)
 		}
 	}
 
+	res = write_update_refs_state(&ctx.refs_to_oids);
+
 	string_list_clear(&ctx.refs_to_oids, 1);
+
+	if (res) {
+		/* we failed, so clean up the new list. */
+		free(ctx.items);
+		return res;
+	}
+
 	free(todo_list->items);
 	todo_list->items = ctx.items;
 	todo_list->nr = ctx.items_nr;
t/t2407-worktree-heads.sh +6 -15
index 8a03f14df8d..50815acd3e8 100755
@@ -81,25 +81,16 @@ test_expect_success !SANITIZE_LEAK 'refuse to overwrite: worktree in rebase (mer
 	grep "cannot force update the branch '\''wt-2'\'' checked out at.*wt-2" err
 '
 
-test_expect_success 'refuse to overwrite: worktree in rebase with --update-refs' '
-	test_when_finished rm -rf .git/worktrees/wt-3/rebase-merge &&
-
-	mkdir -p .git/worktrees/wt-3/rebase-merge &&
-	touch .git/worktrees/wt-3/rebase-merge/interactive &&
+test_expect_success !SANITIZE_LEAK 'refuse to overwrite: worktree in rebase with --update-refs' '
+	test_when_finished git -C wt-3 rebase --abort &&
 
-	cat >.git/worktrees/wt-3/rebase-merge/update-refs <<-EOF &&
-	refs/heads/fake-3
-	$(git rev-parse HEAD~1)
-	$(git rev-parse HEAD)
-	refs/heads/fake-4
-	$(git rev-parse HEAD)
-	$(git rev-parse HEAD)
-	EOF
+	git branch -f can-be-updated wt-3 &&
+	test_must_fail git -C wt-3 rebase --update-refs conflict-3 &&
 
 	for i in 3 4
 	do
-		test_must_fail git branch -f fake-$i HEAD 2>err &&
-		grep "cannot force update the branch '\''fake-$i'\'' checked out at.*wt-3" err ||
+		test_must_fail git branch -f can-be-updated HEAD 2>err &&
+		grep "cannot force update the branch '\''can-be-updated'\'' checked out at.*wt-3" err ||
 			return 1
 	done
 '
t/t3404-rebase-interactive.sh +17 -0
index 75c0b90fae1..2fffcf5d5fb 100755
@@ -1817,6 +1817,23 @@ test_expect_success '--update-refs adds commands with --rebase-merges' '
 	)
 '
 
+test_expect_success '--update-refs updates refs correctly' '
+	git checkout -B update-refs no-conflict-branch &&
+	git branch -f base HEAD~4 &&
+	git branch -f first HEAD~3 &&
+	git branch -f second HEAD~3 &&
+	git branch -f third HEAD~1 &&
+	test_commit extra2 fileX &&
+	git commit --amend --fixup=L &&
+
+	git rebase -i --autosquash --update-refs primary &&
+
+	test_cmp_rev HEAD~3 refs/heads/first &&
+	test_cmp_rev HEAD~3 refs/heads/second &&
+	test_cmp_rev HEAD~1 refs/heads/third &&
+	test_cmp_rev HEAD refs/heads/no-conflict-branch
+'
+
 # This must be the last test in this file
 test_expect_success '$EDITOR and friends are unchanged' '
 	test_editor_unchanged
Signature · 1 line
gitgitgadget
#133 lore raw

Re: [PATCH v5 08/12] rebase: update refs from 'update-ref' commands

Hi Stolee

sequencer.c leave_merge: · line 4152

Quoted 9 lines · Derrick Stolee via GitGitGadget

From: Derrick Stolee <[email protected]>

sequencer.c +6 -0
+static int do_update_ref(struct repository *r, const char *refname)
  {
+	struct string_list_item *item;
+	struct string_list list = STRING_LIST_INIT_DUP;
+
+	if (sequencer_get_update_refs_state(r->gitdir, &list))
+		return -1;

This is good, we now return an error if we cannot read the update-refs file.

sequencer.c leave_merge: · line 4163

Quoted 10 lines
sequencer.c +10 -0
+	for_each_string_list_item(item, &list) {
+		if (!strcmp(item->string, refname)) {
+			struct update_ref_record *rec = item->util;
+			if (read_ref("HEAD", &rec->after))
+				return -1;
+			break;
+		}
+	}
+
+	write_update_refs_state(&list);

The return value of write_update_refs_state() should be propagated to the caller.

sequencer.c leave_merge: · line 4166

Quoted 3 lines
sequencer.c +1 -0
+	string_list_clear(&list, 1);
  	return 0;
  }

Best Wishes

Phillip

#134 lore raw

PATCH v5 9/12 sequencer: rewrite update-refs as user edits todo list

In reply to Derrick Stolee via GitGitGadget

From: Derrick Stolee <[email protected]>

An interactive rebase provides opportunities for the user to edit the todo list. The --update-refs option initializes the list with some 'update-ref <ref>' steps, but the user could add these manually. Further, the user could add or remove these steps during pauses in the interactive rebase.

Add a new method, todo_list_filter_update_refs(), that scans a todo_list and compares it to the stored update-refs file. There are two actions that can happen at this point:

1. If a '<ref>/<before>/<after>' triple in the update-refs file does not have a matching 'update-ref <ref>' command in the todo-list _and_ the <after> value is the null OID, then remove that triple. Here, the user removed the 'update-ref <ref>' command before it was executed, since if it was executed then the <after> value would store the commit at that position.

2. If a 'update-ref <ref>' command in the todo-list does not have a matching '<ref>/<before>/<after>' triple in the update-refs file, then insert a new one. Store the <before> value to be the current OID pointed at by <ref>. This is handled inside of the init_update_ref_record() helper method.

We can test that this works by rewriting the todo-list several times in the course of a rebase. Check that each ref is locked or unlocked for updates after each todo-list update. We can also verify that the ref update fails if a concurrent process updates one of the refs after the rebase process records the "locked" ref location.

To help these tests, add a new 'set_replace_editor' helper that will replace the todo-list with an exact file.

---
 rebase-interactive.c          |   6 ++
 sequencer.c                   |  96 +++++++++++++++++++++++
 sequencer.h                   |  12 +++
 t/lib-rebase.sh               |  15 ++++
 t/t3404-rebase-interactive.sh | 139 ++++++++++++++++++++++++++++++++++
 5 files changed, 268 insertions(+)
rebase-interactive.c +6 -0
index 1ff07647af3..7407c593191 100644
@@ -146,6 +146,12 @@ int edit_todo_list(struct repository *r, struct todo_list *todo_list,
 		return -4;
 	}
 
+	/*
+	 * See if branches need to be added or removed from the update-refs
+	 * file based on the new todo list.
+	 */
+	todo_list_filter_update_refs(r, new_todo);
+
 	return 0;
 }
 
sequencer.c +96 -0

102 lines not on this page · raw message

sequencer.h +12 -0
index e671d7e0743..98f3bcc1658 100644
@@ -132,6 +132,18 @@ void todo_list_release(struct todo_list *todo_list);
 const char *todo_item_get_arg(struct todo_list *todo_list,
 			      struct todo_item *item);
 
+/*
+ * Parse the update-refs file for the current rebase, then remove the
+ * refs that do not appear in the todo_list (and have not had updated
+ * values stored) and add refs that are in the todo_list but not
+ * represented in the update-refs file.
+ *
+ * If there are changes to the update-refs list, then write the new state
+ * to disk.
+ */
+void todo_list_filter_update_refs(struct repository *r,
+				  struct todo_list *todo_list);
+
 /* Call this to setup defaults before parsing command line options */
 void sequencer_init_config(struct replay_opts *opts);
 int sequencer_pick_revisions(struct repository *repo,
t/lib-rebase.sh +15 -0
index ec6b9b107da..b57541356bd 100644
@@ -207,3 +207,18 @@ check_reworded_commits () {
 		>reword-log &&
 	test_cmp reword-expected reword-log
 }
+
+# usage: set_replace_editor <file>
+#
+# Replace the todo file with the exact contents of the given file.
+set_replace_editor () {
+	cat >script <<-\EOF &&
+	cat FILENAME >"$1"
+
+	echo 'rebase -i script after editing:'
+	cat "$1"
+	EOF
+
+	sed -e "s/FILENAME/$1/g" <script | write_script fake-editor.sh &&
+	test_set_editor "$(pwd)/fake-editor.sh"
+}
t/t3404-rebase-interactive.sh +139 -0

145 lines not on this page · raw message

Signature · 1 line
gitgitgadget
#135 lore raw

Re: [PATCH v5 09/12] sequencer: rewrite update-refs as user edits todo list

Hi Stolee

Quoted 76 lines · Derrick Stolee via GitGitGadget

From: Derrick Stolee <[email protected]>

sequencer.c +74 -0
+void todo_list_filter_update_refs(struct repository *r,
+				  struct todo_list *todo_list)
+{
+	int i;
+	int updated = 0;
+	struct string_list update_refs = STRING_LIST_INIT_DUP;
+
+	sequencer_get_update_refs_state(r->gitdir, &update_refs);
+
+	/*
+	 * For each item in the update_refs list, if it has no updated
+	 * value and does not appear in the todo_list, then remove it
+	 * from the update_refs list.
+	 */
+	for (i = 0; i < update_refs.nr; i++) {
+		int j;
+		int found = 0;
+		const char *ref = update_refs.items[i].string;
+		size_t reflen = strlen(ref);
+		struct update_ref_record *rec = update_refs.items[i].util;
+
+		/* OID already stored as updated. */
+		if (!is_null_oid(&rec->after))
+			continue;
+
+		for (j = 0; !found && j < todo_list->total_nr; j++) {
+			struct todo_item *item = &todo_list->items[j];
+			const char *arg = todo_list->buf.buf + item->arg_offset;
+
+			if (item->command != TODO_UPDATE_REF)
+				continue;
+
+			if (item->arg_len != reflen ||
+			    strncmp(arg, ref, reflen))
+				continue;
+
+			found = 1;
+		}
+
+		if (!found) {
+			free(update_refs.items[i].string);
+			free(update_refs.items[i].util);
+
+			update_refs.nr--;
+			MOVE_ARRAY(update_refs.items + i, update_refs.items + i + 1, update_refs.nr - i);
+
+			updated = 1;
+			i--;
+		}
+	}
+
+	/*
+	 * For each todo_item, check if its ref is in the update_refs list.
+	 * If not, then add it as an un-updated ref.
+	 */
+	for (i = 0; i < todo_list->total_nr; i++) {
+		struct todo_item *item = &todo_list->items[i];
+		const char *arg = todo_list->buf.buf + item->arg_offset;
+		int j, found = 0;
+
+		if (item->command != TODO_UPDATE_REF)
+			continue;
+
+		for (j = 0; !found && j < update_refs.nr; j++) {
+			const char *ref = update_refs.items[j].string;
+
+			found = strlen(ref) == item->arg_len &&
+				!strncmp(ref, arg, item->arg_len);
+		}
+
+		if (!found) {
+			struct string_list_item *inserted;
+			struct strbuf argref = STRBUF_INIT;
+

sequencer.c cleanup: · line 4238

Quoted 10 lines
sequencer.c +10 -0
+			strbuf_add(&argref, arg, item->arg_len);
+			inserted = string_list_insert(&update_refs, argref.buf);
+			inserted->util = init_update_ref_record(argref.buf);
+			strbuf_release(&argref);
+			updated = 1;
+		}
+	}
+
+	if (updated)
+		write_update_refs_state(&update_refs);

We should return an error if write_update_refs_state() fails

Best Wishes

Phillip

#136 lore raw

PATCH v5 10/12 rebase: add rebase.updateRefs config option

In reply to Derrick Stolee via GitGitGadget

From: Derrick Stolee <[email protected]>

The previous change added the --update-refs command-line option. For users who always want this mode, create the rebase.updateRefs config option which behaves the same way as rebase.autoSquash does with the --autosquash option.

---
 Documentation/config/rebase.txt |  3 +++
 Documentation/git-rebase.txt    |  3 +++
 builtin/rebase.c                |  5 +++++
 t/t3404-rebase-interactive.sh   | 14 ++++++++++++++
 4 files changed, 25 insertions(+)
Documentation/config/rebase.txt +3 -0
index 8c979cb20f2..f19bd0e0407 100644
@@ -21,6 +21,9 @@ rebase.autoStash::
 	`--autostash` options of linkgit:git-rebase[1].
 	Defaults to false.
 
+rebase.updateRefs::
+	If set to true enable `--update-refs` option by default.
+
 rebase.missingCommitsCheck::
 	If set to "warn", git rebase -i will print a warning if some
 	commits are removed (e.g. a line was deleted), however the
Documentation/git-rebase.txt +3 -0
index dfd847d6615..305255f1e15 100644
@@ -614,6 +614,9 @@ start would be overridden by the presence of
 	Automatically force-update any branches that point to commits that
 	are being rebased. Any branches that are checked out in a worktree
 	are not updated in this way.
++
+If the configuration variable `rebase.updateRefs` is set, then this option
+can be used to override and disable this setting.
 
 INCOMPATIBLE OPTIONS
 --------------------
builtin/rebase.c +5 -0
index fd64897132a..dbb91895687 100644
@@ -802,6 +802,11 @@ static int rebase_config(const char *var, const char *value, void *data)
 		return 0;
 	}
 
+	if (!strcmp(var, "rebase.updaterefs")) {
+		opts->update_refs = git_config_bool(var, value);
+		return 0;
+	}
+
 	if (!strcmp(var, "rebase.reschedulefailedexec")) {
 		opts->reschedule_failed_exec = git_config_bool(var, value);
 		return 0;
t/t3404-rebase-interactive.sh +14 -0
index bebf9b4def7..1a27bb0626d 100755
@@ -1772,6 +1772,12 @@ test_expect_success '--update-refs adds label and update-ref commands' '
 		EOF
 
 		test_must_fail git rebase -i --autosquash --update-refs primary >todo &&
+		test_cmp expect todo &&
+
+		test_must_fail git -c rebase.autosquash=true \
+				   -c rebase.updaterefs=true \
+				   rebase -i primary >todo &&
+
 		test_cmp expect todo
 	)
 '
@@ -1813,6 +1819,14 @@ test_expect_success '--update-refs adds commands with --rebase-merges' '
 				   --rebase-merges=rebase-cousins \
 				   --update-refs primary >todo &&
 
+		test_cmp expect todo &&
+
+		test_must_fail git -c rebase.autosquash=true \
+				   -c rebase.updaterefs=true \
+				   rebase -i \
+				   --rebase-merges=rebase-cousins \
+				   primary >todo &&
+
 		test_cmp expect todo
 	)
 '
Signature · 1 line
gitgitgadget
#137 lore raw

PATCH v5 11/12 sequencer: ignore HEAD ref under --update-refs

In reply to Derrick Stolee via GitGitGadget

From: Derrick Stolee <[email protected]>

When using the 'git rebase -i --update-refs' option, the todo list is populated with 'update-ref' commands for all tip refs in the history that is being rebased. Refs that are checked out by some worktree are instead added as a comment to warn the user that they will not be updated.

Until now, this included the HEAD ref, which is being updated by the rebase process itself, regardless of the --update-refs option. Remove the comment in this case by ignoring any decorations that match the HEAD ref.

---
 sequencer.c | 13 +++++++++++++
 1 file changed, 13 insertions(+)
sequencer.c +13 -0
index 67812c0294f..1602649332b 100644
@@ -5853,12 +5853,25 @@ static int add_decorations_to_list(const struct commit *commit,
 				   struct todo_add_branch_context *ctx)
 {
 	const struct name_decoration *decoration = get_name_decoration(&commit->object);
+	const char *head_ref = resolve_ref_unsafe("HEAD",
+						  RESOLVE_REF_READING,
+						  NULL,
+						  NULL);
 
 	while (decoration) {
 		struct todo_item *item;
 		const char *path;
 		size_t base_offset = ctx->buf->len;
 
+		/*
+		 * If the branch is the current HEAD, then it will be
+		 * updated by the default rebase behavior.
+		 */
+		if (head_ref && !strcmp(head_ref, decoration->name)) {
+			decoration = decoration->next;
+			continue;
+		}
+
 		ALLOC_GROW(ctx->items,
 			ctx->items_nr + 1,
 			ctx->items_alloc);
Signature · 1 line
gitgitgadget
#138 lore raw

PATCH v5 12/12 sequencer: notify user of --update-refs activity

In reply to Derrick Stolee via GitGitGadget

From: Derrick Stolee <[email protected]>

When the user runs 'git rebase -i --update-refs', the end message still says only

Successfully rebased and updated <HEAD-ref>.

Update the sequencer to collect the successful (and unsuccessful) ref updates due to the --update-refs option, so the end message now says

Successfully rebased and updated <HEAD-ref>. Updated the following refs with --update-refs:
refs/heads/first
refs/heads/third
Failed to update the following refs with --update-refs:
refs/heads/second

To test this output, we need to be very careful to format the expected error to drop the leading tab characters. Also, we need to be aware that the verbose output from 'git rebase' is writing progress lines which don't use traditional newlines but clear the line after every progress item is complete. When opening the error file in an editor, these lines are visible, but when looking at the diff in a terminal those lines disappear because of the characters that delete the previous characters. Use 'sed' to clear those progress lines and clear the tabs so we can get an exact match on our expected output.

---
 sequencer.c                   | 40 +++++++++++++++++++++++++++++------
 t/t3404-rebase-interactive.sh | 37 ++++++++++++++++++++++++++++----
 2 files changed, 67 insertions(+), 10 deletions(-)
sequencer.c +34 -6
index 1602649332b..96e1d58ede8 100644
@@ -4261,26 +4261,54 @@ static int do_update_ref(struct repository *r, const char *refname)
 	return 0;
 }
 
-static int do_update_refs(struct repository *r)
+static int do_update_refs(struct repository *r, int quiet)
 {
 	int res = 0;
 	struct string_list_item *item;
 	struct string_list refs_to_oids = STRING_LIST_INIT_DUP;
 	struct ref_store *refs = get_main_ref_store(r);
+	struct strbuf update_msg = STRBUF_INIT;
+	struct strbuf error_msg = STRBUF_INIT;
 
 	if ((res = sequencer_get_update_refs_state(r->gitdir, &refs_to_oids)))
 		return res;
 
 	for_each_string_list_item(item, &refs_to_oids) {
 		struct update_ref_record *rec = item->util;
+		int loop_res;
 
-		res |= refs_update_ref(refs, "rewritten during rebase",
-				       item->string,
-				       &rec->after, &rec->before,
-				       0, UPDATE_REFS_MSG_ON_ERR);
+		loop_res = refs_update_ref(refs, "rewritten during rebase",
+					   item->string,
+					   &rec->after, &rec->before,
+					   0, UPDATE_REFS_MSG_ON_ERR);
+		res |= loop_res;
+
+		if (quiet)
+			continue;
+
+		if (loop_res)
+			strbuf_addf(&error_msg, "\t%s\n", item->string);
+		else
+			strbuf_addf(&update_msg, "\t%s\n", item->string);
+	}
+
+	if (!quiet &&
+	    (update_msg.len || error_msg.len)) {
+		fprintf(stderr,
+			_("Updated the following refs with %s:\n%s"),
+			"--update-refs",
+			update_msg.buf);
+
+		if (res)
+			fprintf(stderr,
+				_("Failed to update the following refs with %s:\n%s"),
+				"--update-refs",
+				error_msg.buf);
 	}
 
 	string_list_clear(&refs_to_oids, 1);
+	strbuf_release(&update_msg);
+	strbuf_release(&error_msg);
 	return res;
 }
 
@@ -4800,7 +4828,7 @@ cleanup_head_ref:
 		strbuf_release(&buf);
 		strbuf_release(&head_ref);
 
-		if (do_update_refs(r))
+		if (do_update_refs(r, opts->quiet))
 			return -1;
 	}
 
t/t3404-rebase-interactive.sh +33 -4
index 1a27bb0626d..688b01e3eb6 100755
@@ -1840,12 +1840,26 @@ test_expect_success '--update-refs updates refs correctly' '
 	test_commit extra2 fileX &&
 	git commit --amend --fixup=L &&
 
-	git rebase -i --autosquash --update-refs primary &&
+	git rebase -i --autosquash --update-refs primary 2>err &&
 
 	test_cmp_rev HEAD~3 refs/heads/first &&
 	test_cmp_rev HEAD~3 refs/heads/second &&
 	test_cmp_rev HEAD~1 refs/heads/third &&
-	test_cmp_rev HEAD refs/heads/no-conflict-branch
+	test_cmp_rev HEAD refs/heads/no-conflict-branch &&
+
+	cat >expect <<-\EOF &&
+	Successfully rebased and updated refs/heads/update-refs.
+	Updated the following refs with --update-refs:
+		refs/heads/first
+		refs/heads/no-conflict-branch
+		refs/heads/second
+		refs/heads/third
+	EOF
+
+	# Clear "Rebasing (X/Y)" progress lines and drop leading tabs.
+	sed -e "s/Rebasing.*Successfully/Successfully/g" -e "s/^\t//g" \
+		<err >err.trimmed &&
+	test_cmp expect err.trimmed
 '
 
 test_expect_success 'respect user edits to update-ref steps' '
@@ -1983,8 +1997,23 @@ test_expect_success '--update-refs: check failed ref update' '
 	# the lock in the update-refs file.
 	git rev-parse third >.git/refs/heads/second &&
 
-	git rebase --continue 2>err &&
-	grep "update_ref failed for ref '\''refs/heads/second'\''" err
+	test_must_fail git rebase --continue 2>err &&
+	grep "update_ref failed for ref '\''refs/heads/second'\''" err &&
+
+	cat >expect <<-\EOF &&
+	Updated the following refs with --update-refs:
+		refs/heads/first
+		refs/heads/no-conflict-branch
+		refs/heads/third
+	Failed to update the following refs with --update-refs:
+		refs/heads/second
+	EOF
+
+	# Clear "Rebasing (X/Y)" progress lines and drop leading tabs.
+	tail -n 6 err >err.last &&
+	sed -e "s/Rebasing.*Successfully/Successfully/g" -e "s/^\t//g" \
+		<err.last >err.trimmed &&
+	test_cmp expect err.trimmed
 '
 
 # This must be the last test in this file
Signature · 1 line
gitgitgadget
#139 lore raw
In reply to Derrick Stolee via GitGitGadget
Quoted 80 lines · Derrick Stolee via GitGitGadget

This series is based on ds/branch-checked-out.

This is a feature I've wanted for quite a while. When working on the sparse index topic, I created a long RFC that actually broke into three topics for full review upstream. These topics were sequential, so any feedback on an earlier one required updates to the later ones. I would work on the full feature and use interactive rebase to update the full list of commits. However, I would need to update the branches pointing to those sub-topics.

This series adds a new --update-refs option to 'git rebase' (along with a rebase.updateRefs config option) that adds 'update-ref' commands into the TODO list. This is powered by the commit decoration machinery.

As an example, here is my in-progress bundle URI RFC split into subtopics as they appear during the TODO list of a git rebase -i --update-refs:

pick 2d966282ff3 docs: document bundle URI standard
pick 31396e9171a remote-curl: add 'get' capability
pick 54c6ab70f67 bundle-uri: create basic file-copy logic pick 96cb2e35af1 bundle-uri: add support for http(s):// and file:// pick 6adaf842684 fetch: add --bundle-uri option
pick 6c5840ed77e fetch: add 'refs/bundle/' to log.excludeDecoration update-ref refs/heads/bundle-redo/fetch

pick 1e3f6546632 clone: add --bundle-uri option
pick 9e4a6fe9b68 clone: --bundle-uri cannot be combined with --depth update-ref refs/heads/bundle-redo/clone

pick 5451cb6599c bundle-uri: create bundle_list struct and helpers pick 3029c3aca15 bundle-uri: create base key-value pair parsing pick a8b2de79ce8 bundle-uri: create "key=value" line parsing pick 92625a47673 bundle-uri: unit test "key=value" parsing pick a8616af4dc2 bundle-uri: limit recursion depth for bundle lists pick 9d6809a8d53 bundle-uri: parse bundle list in config format pick 287a732b54c bundle-uri: fetch a list of bundles update-ref refs/heads/bundle-redo/list

pick b09f8226185 protocol v2: add server-side "bundle-uri" skeleton pick 520204dcd1c bundle-uri client: add minimal NOOP client
pick 62e8b457b48 bundle-uri client: add "git ls-remote-bundle-uri" pick 00eae925043 bundle-uri: serve URI advertisement from bundle.* config pick 4277440a250 bundle-uri client: add boolean transfer.bundleURI setting pick caf4599a81d bundle-uri: allow relative URLs in bundle lists pick df255000b7e bundle-uri: download bundles from an advertised list pick d71beabf199 clone: unbundle the advertised bundles
pick c9578391976 t5601: basic bundle URI tests
# Ref refs/heads/bundle-redo/rfc-3 checked out at '/home/stolee/_git/git-bundles'

update-ref refs/heads/bundle-redo/advertise

Here is an outline of the series:

* Patch 1 updates some tests for branch_checked_out() to use 'git bisect' and 'git rebase' as black-boxes instead of manually editing files inside $GIT_DIR. (Thanks, Junio!)
* Patch 2 updates some tests for branch_checked_out() for the 'apply' backend.
* Patch 3 updates branch_checked_out() to parse the rebase-merge/update-refs file to block concurrent ref updates and checkouts on branches selected by --update-refs.
* Patch 4 updates the todo list documentation to remove some unnecessary dots in the 'merge' command. This makes it consistent with the 'fixup' command before we document the 'update-ref' command.
* Patch 5 updates the definition of todo_command_info to use enum values as array indices.
* Patches 6-8 implement the --update-refs logic itself.
* Patch 9 specifically updates the update-refs file every time the user edits the todo-list (Thanks Phillip!)
* Patch 10 adds the rebase.updateRefs config option similar to rebase.autoSquash.
* Patch 11 ignores the HEAD ref when creating the todo list instead of making a comment (Thanks Elijah!)
* Patch 12 adds messaging to the end of the rebase stating which refs were updated (Thanks Elijah!)

During review, we have identified some areas that would be good for #leftoverbits:

Quoted 7 lines

* Warn the user when they add an 'update-ref ' command but is checked out in another worktree.
* The checks in patch 9 are quadratic. They could be sped up using hashtables.
* Consider whether we should include an 'update-ref ' command for the HEAD ref, so that all refs are updated in the same way. This might help confused users.

Not necessarily so they are updated in the same way; the behind the scenes mechanism could perhaps still be different. Just so that if the user looks for the "list of things being updated" they don't get surprised that HEAD is missing.

Quoted 6 lines

* The error message for failed ref updates could include information on the commit IDs that would have been used. This can help the user fix the situation by updating the refs manually.
* Modify the --update-refs option from a boolean to an optionally-string-parameter that specifies refspecs for the 'update-ref' commands.

refspecs? Is that the term you really mean here?

Quoted 12 lines

Updates in v5
=============

* Rename 'wt_dir' to 'wt_git_dir' for clarity.
* The documented behavior around 'fixup!' and 'squash!' commits was incorrect, so update the commit message, documentation, and test to demonstrate the actual behavior.
* Use CALLOC_ARRAY() to be more idiomatic.
* Be much more careful about propagating errors.
* Commit message typo: "We an" to "We can"
* Remove unnecessary null OID check when writing refs, since those would already be removed by a previous step.

Thanks, I've read over the range-diff and these changes look good to me. One thing I'm curious about (sorry to bring this up so late):
"pick" commands come with the old commit hash. Perhaps the update-ref commands should too? (e.g. "update-ref refs/heads/topic from <OLDHASH>")

#140 lore raw
Quoted 13 lines · Elijah Newren

On Tue, Jul 19, 2022 at 11:33 AM Derrick Stolee via GitGitGadget

Quoted 10 lines

During review, we have identified some areas that would be good for #leftoverbits:

* Warn the user when they add an 'update-ref ' command but is checked out in another worktree.
* The checks in patch 9 are quadratic. They could be sped up using hashtables.
* Consider whether we should include an 'update-ref ' command for the HEAD ref, so that all refs are updated in the same way. This might help confused users.

Quoted 4 lines

Not necessarily so they are updated in the same way; the behind the scenes mechanism could perhaps still be different. Just so that if the user looks for the "list of things being updated" they don't get surprised that HEAD is missing.

Noted.

Quoted 8 lines
Quoted 6 lines

* The error message for failed ref updates could include information on the commit IDs that would have been used. This can help the user fix the situation by updating the refs manually.
* Modify the --update-refs option from a boolean to an optionally-string-parameter that specifies refspecs for the 'update-ref' commands.

refspecs? Is that the term you really mean here?

Probably "ref namespaces" or "ref prefixes" would be better.

Quoted 13 lines
Quoted 12 lines

Updates in v5
=============

* Rename 'wt_dir' to 'wt_git_dir' for clarity.
* The documented behavior around 'fixup!' and 'squash!' commits was incorrect, so update the commit message, documentation, and test to demonstrate the actual behavior.
* Use CALLOC_ARRAY() to be more idiomatic.
* Be much more careful about propagating errors.
* Commit message typo: "We an" to "We can"
* Remove unnecessary null OID check when writing refs, since those would already be removed by a previous step.

Quoted 5 lines

Thanks, I've read over the range-diff and these changes look good to me. One thing I'm curious about (sorry to bring this up so late):
"pick" commands come with the old commit hash. Perhaps the update-ref commands should too? (e.g. "update-ref refs/heads/topic from <OLDHASH>")

I don't personally see the value here other than to make it harder for someone to add new commands (and confusing when wanting to create a brand new ref). We could consider adding a comment in the future without any backwards compatibility issues:

update-ref refs/heads/my-ref # was 0123dead

Thanks,
-Stolee

#141 lore raw
Quoted 11 lines · Derrick Stolee
Quoted 4 lines · Elijah Newren

One thing I'm curious about (sorry to bring this up so late):
"pick" commands come with the old commit hash. Perhaps the update-ref commands should too? (e.g. "update-ref refs/heads/topic from <OLDHASH>")

I don't personally see the value here other than to make it harder for someone to add new commands (and confusing when wanting to create a brand new ref).

I apologize if my late query caused frustration. I was thinking it made more sense to ask now than later and was genuinely curious for thoughts on the idea, but perhaps it'd have been better if I just didn't bring it up. Sorry.

But, if you're curious why I was thinking about it...

There are four ways I can think of to handle the <oldvalue>: (A) include it in the "update-ref" instruction, (B) manually get the <oldvalue> at program invocation, (C) accept an increased race-window by not grabbing <oldvalue> until the line is parsed, or (D) ignore races and just don't provide <oldvalue> when updating refs. If one chooses (B), they can either (B1) pre-parse the entire todo list and look up the current values of any refs mentioned in an update-ref instruction, or (B2) record the value of all existing refs at invocation.

Rebase is already paying the cost for pre-parsing the entire todo list (i.e. B1). In fact, you _also_ preemptively locking the refs mentioned. So, in a sense, rebase is already covered. However:
* adding such a thing could potentially remove a small race window
(perhaps the user takes a while to edit the todo list, and the ref somehow gets updated before they even finish that step; being able to specify the oldref might help in such a case).
* existing commands (pick, merge, etc.) give the oldrefs (or labels
to them), so having update-ref also do so might provide some "feel good" consistency.

Perhaps those reasons are pretty weak for the rebase case, given you already do B1. However, git replay isn't going to do B1, and nor will it preemptively lock refs. I see git replay as a patch-based analogue to the snapshot-based tools of fast-export and fast-import. (fast-import doesn't pre-parse its entire input before starting to process the first commit, nor does it preemptively lock refs.) B2 is not appealing because some repositories have gargantuan numbers of refs. Also, the update-ref-during-edit-of-todo rationale is more important for git replay, because I want git replay to accept a filename containing a todo list in lieu of a range of revisions to replay -- perhaps users spent quite some time or programming effort generating their todo list and it took some time to generate/review. And that further undercuts both B1 and B2. So, between all those reasons, (A) is very appealing to me.

All that said, git replay already has other planned differences in its todo list (e.g. a "reset" directive must be the first non-comment directive, there's no separate "merge" directive, etc.), so if I have to add another one, it's no big deal.

Your point about specifying <oldref> being seen as friction for some users is well taken, and I'm thinking of making the "from <oldref>" part of the instruction optional (though documenting that users are accepting some race risk by switching from (A) to (C) when they omit it).

Quoted 4 lines

We could consider adding a comment in the future without any backwards compatibility issues:

update-ref refs/heads/my-ref # was 0123dead

Yes, this could certainly be added later (though none of my reasons for wanting <oldref> were geared around pointers for the end user, so this form wouldn't be helpful to me).

Also, if "from <oldref>" is optional, it could also be added later.

If you've read this far, let me just take a minute to again thank you for your work on this --update-refs option to rebase. It's some cool work. Also, thanks for listening to my random ideas and musings along the way. :-)

#142 lore raw
Quoted 9 lines · Elijah Newren
Quoted 8 lines · Derrick Stolee
Quoted 4 lines · Elijah Newren

One thing I'm curious about (sorry to bring this up so late):
"pick" commands come with the old commit hash. Perhaps the update-ref commands should too? (e.g. "update-ref refs/heads/topic from <OLDHASH>")

Quoted 5 lines
Quoted 3 lines

I don't personally see the value here other than to make it harder for someone to add new commands (and confusing when wanting to create a brand new ref).

I apologize if my late query caused frustration.

I intended this as "I don't understand, please enlighten me", but looking back now it definitely sounds more frustrated. Sorry about that.

Quoted 4 lines

I was thinking it
made more sense to ask now than later and was genuinely curious for thoughts on the idea, but perhaps it'd have been better if I just didn't bring it up. Sorry.

It's definitely the time to bring it up, because it will be harder to change it once it's shipped.

Quoted 39 lines

But, if you're curious why I was thinking about it...

There are four ways I can think of to handle the <oldvalue>: (A) include it in the "update-ref" instruction, (B) manually get the <oldvalue> at program invocation, (C) accept an increased race-window by not grabbing <oldvalue> until the line is parsed, or (D) ignore races and just don't provide <oldvalue> when updating refs. If one chooses (B), they can either (B1) pre-parse the entire todo list and look up the current values of any refs mentioned in an update-ref instruction, or (B2) record the value of all existing refs at invocation.

Rebase is already paying the cost for pre-parsing the entire todo list (i.e. B1). In fact, you _also_ preemptively locking the refs mentioned. So, in a sense, rebase is already covered. However:
* adding such a thing could potentially remove a small race window
(perhaps the user takes a while to edit the todo list, and the ref somehow gets updated before they even finish that step; being able to specify the oldref might help in such a case).
* existing commands (pick, merge, etc.) give the oldrefs (or labels
to them), so having update-ref also do so might provide some "feel good" consistency.

Perhaps those reasons are pretty weak for the rebase case, given you already do B1. However, git replay isn't going to do B1, and nor will it preemptively lock refs. I see git replay as a patch-based analogue to the snapshot-based tools of fast-export and fast-import. (fast-import doesn't pre-parse its entire input before starting to process the first commit, nor does it preemptively lock refs.) B2 is not appealing because some repositories have gargantuan numbers of refs. Also, the update-ref-during-edit-of-todo rationale is more important for git replay, because I want git replay to accept a filename containing a todo list in lieu of a range of revisions to replay -- perhaps users spent quite some time or programming effort generating their todo list and it took some time to generate/review. And that further undercuts both B1 and B2. So, between all those reasons, (A) is very appealing to me.

Quoted 4 lines

All that said, git replay already has other planned differences in its todo list (e.g. a "reset" directive must be the first non-comment directive, there's no separate "merge" directive, etc.), so if I have to add another one, it's no big deal.

I'm happy that you're working on 'git replay'. I'm excited to see how it works. Also, it would be helpful to keep these things in line as much as possible.

Quoted 9 lines

Your point about specifying <oldref> being seen as friction for some users is well taken, and I'm thinking of making the "from <oldref>" part of the instruction optional (though documenting that users are accepting some race risk by switching from (A) to (C) when they omit it).

Quoted 3 lines

We could consider adding a comment in the future without any backwards compatibility issues:

Quoted 7 lines
Quoted 1 line

update-ref refs/heads/my-ref # was 0123dead

Yes, this could certainly be added later (though none of my reasons for wanting <oldref> were geared around pointers for the end user, so this form wouldn't be helpful to me).

Also, if "from <oldref>" is optional, it could also be added later.

The optional value makes this something that we can add later, which I think is good flexibility. I was thinking about doing the comment thing as a follow-up (in addition to Phillip's error mode comments) but maybe I'll explore this direction instead.

Quoted 4 lines

If you've read this far, let me just take a minute to again thank you for your work on this --update-refs option to rebase. It's some cool work. Also, thanks for listening to my random ideas and musings along the way. :-)

I really appreciate the attention you've given it. It's already improved a lot due to your help.

Thanks,
-Stolee

#143 lore raw
In reply to Derrick Stolee via GitGitGadget

Hi Stolee

I'm afraid I've only had time to have a quick look at patches 8 & 9. The error propagation is looking better but I did notice a couple of calls to write_update_refs_state() where we don't propagate the return value.

Best Wishes

Phillip

Quoted 518 lines · Derrick Stolee via GitGitGadget

This series is based on ds/branch-checked-out.

This is a feature I've wanted for quite a while. When working on the sparse index topic, I created a long RFC that actually broke into three topics for full review upstream. These topics were sequential, so any feedback on an earlier one required updates to the later ones. I would work on the full feature and use interactive rebase to update the full list of commits. However, I would need to update the branches pointing to those sub-topics.

This series adds a new --update-refs option to 'git rebase' (along with a rebase.updateRefs config option) that adds 'update-ref' commands into the TODO list. This is powered by the commit decoration machinery.

As an example, here is my in-progress bundle URI RFC split into subtopics as they appear during the TODO list of a git rebase -i --update-refs:

pick 2d966282ff3 docs: document bundle URI standard
pick 31396e9171a remote-curl: add 'get' capability
pick 54c6ab70f67 bundle-uri: create basic file-copy logic pick 96cb2e35af1 bundle-uri: add support for http(s):// and file:// pick 6adaf842684 fetch: add --bundle-uri option
pick 6c5840ed77e fetch: add 'refs/bundle/' to log.excludeDecoration update-ref refs/heads/bundle-redo/fetch

pick 1e3f6546632 clone: add --bundle-uri option
pick 9e4a6fe9b68 clone: --bundle-uri cannot be combined with --depth update-ref refs/heads/bundle-redo/clone

pick 5451cb6599c bundle-uri: create bundle_list struct and helpers pick 3029c3aca15 bundle-uri: create base key-value pair parsing pick a8b2de79ce8 bundle-uri: create "key=value" line parsing pick 92625a47673 bundle-uri: unit test "key=value" parsing pick a8616af4dc2 bundle-uri: limit recursion depth for bundle lists pick 9d6809a8d53 bundle-uri: parse bundle list in config format pick 287a732b54c bundle-uri: fetch a list of bundles update-ref refs/heads/bundle-redo/list

pick b09f8226185 protocol v2: add server-side "bundle-uri" skeleton pick 520204dcd1c bundle-uri client: add minimal NOOP client
pick 62e8b457b48 bundle-uri client: add "git ls-remote-bundle-uri" pick 00eae925043 bundle-uri: serve URI advertisement from bundle.* config pick 4277440a250 bundle-uri client: add boolean transfer.bundleURI setting pick caf4599a81d bundle-uri: allow relative URLs in bundle lists pick df255000b7e bundle-uri: download bundles from an advertised list pick d71beabf199 clone: unbundle the advertised bundles
pick c9578391976 t5601: basic bundle URI tests
# Ref refs/heads/bundle-redo/rfc-3 checked out at '/home/stolee/_git/git-bundles'

update-ref refs/heads/bundle-redo/advertise

Here is an outline of the series:

* Patch 1 updates some tests for branch_checked_out() to use 'git bisect' and 'git rebase' as black-boxes instead of manually editing files inside $GIT_DIR. (Thanks, Junio!)
* Patch 2 updates some tests for branch_checked_out() for the 'apply' backend.
* Patch 3 updates branch_checked_out() to parse the rebase-merge/update-refs file to block concurrent ref updates and checkouts on branches selected by --update-refs.
* Patch 4 updates the todo list documentation to remove some unnecessary dots in the 'merge' command. This makes it consistent with the 'fixup' command before we document the 'update-ref' command.
* Patch 5 updates the definition of todo_command_info to use enum values as array indices.
* Patches 6-8 implement the --update-refs logic itself.
* Patch 9 specifically updates the update-refs file every time the user edits the todo-list (Thanks Phillip!)
* Patch 10 adds the rebase.updateRefs config option similar to rebase.autoSquash.
* Patch 11 ignores the HEAD ref when creating the todo list instead of making a comment (Thanks Elijah!)
* Patch 12 adds messaging to the end of the rebase stating which refs were updated (Thanks Elijah!)

During review, we have identified some areas that would be good for #leftoverbits:

* Warn the user when they add an 'update-ref ' command but is checked out in another worktree.
* The checks in patch 9 are quadratic. They could be sped up using hashtables.
* Consider whether we should include an 'update-ref ' command for the HEAD ref, so that all refs are updated in the same way. This might help confused users.
* The error message for failed ref updates could include information on the commit IDs that would have been used. This can help the user fix the situation by updating the refs manually.
* Modify the --update-refs option from a boolean to an optionally-string-parameter that specifies refspecs for the 'update-ref' commands.

Updates in v5
=============

* Rename 'wt_dir' to 'wt_git_dir' for clarity.
* The documented behavior around 'fixup!' and 'squash!' commits was incorrect, so update the commit message, documentation, and test to demonstrate the actual behavior.
* Use CALLOC_ARRAY() to be more idiomatic.
* Be much more careful about propagating errors.
* Commit message typo: "We an" to "We can"
* Remove unnecessary null OID check when writing refs, since those would already be removed by a previous step.

Updates in v4
=============

This version took longer than I'd hoped (I had less time to work on it than anticipated) but it also has some major updates. These major updates are direct responses to the significant review this series has received. Thank you!

* The update-refs file now stores "ref/before/after" triples (still separated by lines). This allows us to store the "before" OID of a ref in addition to the "after" that we will write to that ref at the end of the rebase. This allows us to do a "force-with-lease" update. The branch_checked_out() updates should prevent Git from updating those refs while under the rebase, but older versions and third-party tools don't have that protection.
* The update-refs file is updated with every update to the todo-list file. This allows for some advanced changes to the file, including removing, adding, and duplicating 'update-ref' commands.
* The message at the end of the rebase process now lists which refs were updated with the update-ref steps. This includes any ref updates that fail.
* The branch_checked_out() tests now use 'git bisect' and 'git rebase' as black-boxes instead of testing their internals directly.

Here are the more minor updates:

* Dropped an unnecessary stat() call.
* Updated commit messages to include extra details, based on confusion in last round.
* The HEAD branch no longer appears as a comment line in the initial todo list.
* The update-refs file is now written using a lockfile.
* Tests now use test_cmp_rev.
* A memory leak ('path' variable) is resolved.

Updates in v3
=============

* The branch_checked_out() API was extracted to its own topic and is now the ds/branch-checked-out branch. This series is now based on that one.
* The for_each_decoration() API was removed, since it became trivial once it did not take a commit directly.
* The branch_checked_out() tests did not verify the rebase-apply data (for the apply backend), so that is fixed.
* Instead of using the 'label' command and a final 'update-refs' command in the todo list, use a new 'update-ref ' command. This command updates the rebase-merge/update-refs file with the OID of HEAD at these steps. At the very end of the rebase sequence, those refs are updated to the stored OID values (assuming that they were not removed by the user, in which case we notice that the OID is the null OID and we do nothing).
* New tests are added.
* The todo-list comment documentation has some new formatting updates, but also includes a description of 'update-refs' in this version.

Updates in v2
=============

As recommended by the excellent feedback, I have removed the 'exec' commands in favor of the 'label' commands and a new 'update-refs' command at the very end. This way, there is only one step that updates all of the refs at the end instead of updating refs during the rebase. If a user runs 'git rebase --abort' in the middle, then their refs are still where they need to be.

Based on some of the discussion, it seemed like one way to do this would be to have an 'update-ref ' command that would take the place of these 'label' commands. However, this would require two things that make it a bit awkward:

1. We would need to replicate the storage of those positions during the rebase. 'label' already does this pretty well. I've added the "for-update-refs/" label to help here.
2. If we want to close out all of the refs as the rebase is finishing, then that "step" becomes invisible to the user (and a bit more complicated to insert). Thus, the 'update-refs' step performs this action. If the user wants to do things after that step, then they can do so by editing the TODO list.

Other updates:

* The 'keep_decorations' parameter was renamed to 'update_refs'.
* I added tests for --rebase-merges=rebase-cousins to show how these labels interact with other labels and merge commands.
* I changed the order of the insertion of these update-refs labels to be before the fixups are rearranged. This fixes a bug where the tip commit is a fixup! so its decorations are never inspected (and they would be in the wrong place even if they were). The fixup! commands are properly inserted between a pick and its following label command. Tests demonstrate this is correct.
* Numerous style choices are updated based on feedback.

Thank you for all of the detailed review and ideas in this space. I appreciate any more ideas that can make this feature as effective as it can be.

Thanks, -Stolee

Derrick Stolee (12):
t2407: test bisect and rebase as black-boxes
t2407: test branches currently using apply backend
branch: consider refs under 'update-refs'
rebase-interactive: update 'merge' description
sequencer: define array with enum values
sequencer: add update-ref command
rebase: add --update-refs option
rebase: update refs from 'update-ref' commands
sequencer: rewrite update-refs as user edits todo list
rebase: add rebase.updateRefs config option
sequencer: ignore HEAD ref under --update-refs
sequencer: notify user of --update-refs activity

  Documentation/config/rebase.txt |   3 +
  Documentation/git-rebase.txt    |  10 +
  branch.c                        |  13 +
  builtin/rebase.c                |  10 +
  rebase-interactive.c            |  15 +-
  sequencer.c                     | 474 +++++++++++++++++++++++++++++++-
  sequencer.h                     |  23 ++
  t/lib-rebase.sh                 |  15 +
  t/t2407-worktree-heads.sh       | 103 +++++--
  t/t3404-rebase-interactive.sh   | 273 ++++++++++++++++++
  10 files changed, 895 insertions(+), 44 deletions(-)

base-commit: 9bef0b1e6ec371e786c2fba3edcc06ad040a536c
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-1247%2Fderrickstolee%2Frebase-keep-decorations-v5
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1247/derrickstolee/rebase-keep-decorations-v5
Pull-Request: https://github.com/gitgitgadget/git/pull/1247

Range-diff vs v4:

   1:  9e53a27017a =  1:  9e53a27017a t2407: test bisect and rebase as black-boxes
   2:  540a3be256f =  2:  540a3be256f t2407: test branches currently using apply backend
   3:  bf301a054e3 !  3:  1089a0edb73 branch: consider refs under 'update-refs'
      @@ sequencer.c: static GIT_PATH_FUNC(rebase_path_squash_onto, "rebase-merge/squash-
       + * rebase_path_update_refs() returns the path to this file for a given
       + * worktree directory. For the current worktree, pass the_repository->gitdir.
       + */
      -+static char *rebase_path_update_refs(const char *wt_dir)
      ++static char *rebase_path_update_refs(const char *wt_git_dir)
       +{
      -+	return xstrfmt("%s/rebase-merge/update-refs", wt_dir);
      ++	return xstrfmt("%s/rebase-merge/update-refs", wt_git_dir);
       +}
       +
        /*
   4:  dec95681d2b =  4:  d1cce4f06aa rebase-interactive: update 'merge' description
   5:  b2c09600918 =  5:  4c086d477f0 sequencer: define array with enum values
   6:  fa7ecb718cf =  6:  7b3d6601960 sequencer: add update-ref command
   7:  3ec2cc922f9 !  7:  7efb55e4f14 rebase: add --update-refs option

@@ Commit message
--fixup commit at the tip of the feature to apply correctly to the sub branch, even if it is fixing up the most-recent commit in that part.

- One potential problem here is that refs decorating commits that are
- already marked as "fixup!" or "squash!" will not be included in this
- list. Generally, the reordering of the "fixup!" and "squash!" is likely
- to change the relative order of these refs, so it is not recommended.
- The workflow here is intended to allow these kinds of commits at the tip
- of the rebased branch while the other sub branches come along for the
- ride without intervention.
-
This change update the documentation and builtin to accept the --update-refs option as well as updating the todo file with the 'update-ref' commands. Tests are added to ensure that these todo
@@ Documentation/git-rebase.txt: provided. Otherwise an explicit `--no-reschedule-f
+--no-update-refs::
+ Automatically force-update any branches that point to commits that
+ are being rebased. Any branches that are checked out in a worktree
-+ or point to a squash! ... or fixup! ... commit are not updated
-+ in this way.
++ are not updated in this way.
+
INCOMPATIBLE OPTIONS
--------------------
@@ builtin/rebase.c: int cmd_rebase(int argc, const char **argv, const char *prefix
N_("move commits that begin with "
"squash!/fixup! under -i")),
+ OPT_BOOL(0, "update-refs", &options.update_refs,
-+ N_("update local refs that point to commits "
++ N_("update branches that point to commits "
+ "that are being rebased")),
{ OPTION_STRING, 'S', "gpg-sign", &gpg_sign, N_("key-id"),
N_("GPG-sign commits"),
@@ t/t3404-rebase-interactive.sh: test_expect_success 'ORIG_HEAD is updated correct
+ git branch -f second HEAD~3 &&
+ git branch -f third HEAD~1 &&
+ git commit --allow-empty --fixup=third &&
++ git branch -f is-not-reordered &&
++ git commit --allow-empty --fixup=HEAD~4 &&
+ git branch -f shared-tip &&
+ (
+ set_cat_todo_editor &&
+
+ cat >expect <<-EOF &&
+ pick $(git log -1 --format=%h J) J
++ fixup $(git log -1 --format=%h update-refs) fixup! J # empty
+ update-ref refs/heads/second
+ update-ref refs/heads/first
+ pick $(git log -1 --format=%h K) K
+ pick $(git log -1 --format=%h L) L
-+ fixup $(git log -1 --format=%h update-refs) fixup! L # empty
++ fixup $(git log -1 --format=%h is-not-reordered) fixup! L # empty
+ update-ref refs/heads/third
+ pick $(git log -1 --format=%h M) M
+ update-ref refs/heads/no-conflict-branch
++ update-ref refs/heads/is-not-reordered
+ update-ref refs/heads/shared-tip
+ EOF
+
8: fb5f64c5201 ! 8: e7a91bdffbd rebase: update refs from 'update-ref' commands

      @@ sequencer.c: struct update_ref_record {
        
       +static struct update_ref_record *init_update_ref_record(const char *ref)
       +{
      -+	struct update_ref_record *rec = xmalloc(sizeof(*rec));
      ++	struct update_ref_record *rec;
      ++
      ++	CALLOC_ARRAY(rec, 1);
       +
       +	oidcpy(&rec->before, null_oid());
       +	oidcpy(&rec->after, null_oid());
      @@ sequencer.c: leave_merge:
        
       -static int do_update_ref(struct repository *r, const char *ref_name)
       +static int write_update_refs_state(struct string_list *refs_to_oids)
      - {
      ++{
       +	int result = 0;
       +	struct lock_file lock = LOCK_INIT;
       +	FILE *fp = NULL;
      @@ sequencer.c: leave_merge:
       +}
       +
       +static int do_update_ref(struct repository *r, const char *refname)
      -+{
      + {
       +	struct string_list_item *item;
       +	struct string_list list = STRING_LIST_INIT_DUP;
       +
      -+	sequencer_get_update_refs_state(r->gitdir, &list);
      ++	if (sequencer_get_update_refs_state(r->gitdir, &list))
      ++		return -1;
       +
       +	for_each_string_list_item(item, &list) {
       +		if (!strcmp(item->string, refname)) {
       +			struct update_ref_record *rec = item->util;
      -+			read_ref("HEAD", &rec->after);
      ++			if (read_ref("HEAD", &rec->after))
      ++				return -1;
       +			break;
       +		}
       +	}
      @@ sequencer.c: leave_merge:
       +	struct string_list refs_to_oids = STRING_LIST_INIT_DUP;
       +	struct ref_store *refs = get_main_ref_store(r);
       +
      -+	sequencer_get_update_refs_state(r->gitdir, &refs_to_oids);
      ++	if ((res = sequencer_get_update_refs_state(r->gitdir, &refs_to_oids)))
      ++		return res;
       +
       +	for_each_string_list_item(item, &refs_to_oids) {
       +		struct update_ref_record *rec = item->util;
       +
      -+		if (oideq(&rec->after, the_hash_algo->null_oid)) {
      -+			/*
      -+			 * Ref was not updated. User may have deleted the
      -+			 * 'update-ref' step.
      -+			 */
      -+			continue;
      -+		}
      -+
       +		res |= refs_update_ref(refs, "rewritten during rebase",
       +				       item->string,
       +				       &rec->after, &rec->before,
      @@ sequencer.c: leave_merge:
        {
        	int i = todo_list->current;
       @@ sequencer.c: cleanup_head_ref:
      +
      + 		strbuf_release(&buf);
        		strbuf_release(&head_ref);
      ++
      ++		if (do_update_refs(r))
      ++			return -1;
        	}
        
      -+	do_update_refs(r);
      -+
        	/*
      - 	 * Sequence of picks finished successfully; cleanup by
      - 	 * removing the .git/sequencer directory
       @@ sequencer.c: static int add_decorations_to_list(const struct commit *commit,
        
        			sti = string_list_insert(&ctx->refs_to_oids,
      @@ sequencer.c: static int add_decorations_to_list(const struct commit *commit,
        		}
        
        		item->offset_in_buf = base_offset;
      +@@ sequencer.c: static int add_decorations_to_list(const struct commit *commit,
      +  */
      + static int todo_list_add_update_ref_commands(struct todo_list *todo_list)
      + {
      +-	int i;
      ++	int i, res;
      + 	static struct string_list decorate_refs_exclude = STRING_LIST_INIT_NODUP;
      + 	static struct string_list decorate_refs_exclude_config = STRING_LIST_INIT_NODUP;
      + 	static struct string_list decorate_refs_include = STRING_LIST_INIT_NODUP;
       @@ sequencer.c: static int todo_list_add_update_ref_commands(struct todo_list *todo_list)
        		}
        	}
        
      -+	write_update_refs_state(&ctx.refs_to_oids);
      ++	res = write_update_refs_state(&ctx.refs_to_oids);
       +
        	string_list_clear(&ctx.refs_to_oids, 1);
      ++
      ++	if (res) {
      ++		/* we failed, so clean up the new list. */
      ++		free(ctx.items);
      ++		return res;
      ++	}
      ++
        	free(todo_list->items);
        	todo_list->items = ctx.items;
      + 	todo_list->nr = ctx.items_nr;
       
        ## t/t2407-worktree-heads.sh ##
       @@ t/t2407-worktree-heads.sh: test_expect_success !SANITIZE_LEAK 'refuse to overwrite: worktree in rebase (mer

9: 29c7c76805a ! 9: 95e2bbcedb1 sequencer: rewrite update-refs as user edits todo list
@@ Commit message

We can test that this works by rewriting the todo-list several times in the course of a rebase. Check that each ref is locked or unlocked for
- updates after each todo-list update. We an also verify that the ref
+ updates after each todo-list update. We can also verify that the ref
update fails if a concurrent process updates one of the refs after the rebase process records the "locked" ref location.

10: c0022d07579 ! 10: a73b02568f3 rebase: add rebase.updateRefs config option
@@ Documentation/config/rebase.txt: rebase.autoStash::

## Documentation/git-rebase.txt ##
@@ Documentation/git-rebase.txt: start would be overridden by the presence of
+ Automatically force-update any branches that point to commits that
are being rebased. Any branches that are checked out in a worktree
- or point to a squash! ... or fixup! ... commit are not updated
- in this way.
+ are not updated in this way.
++
+If the configuration variable rebase.updateRefs is set, then this option +can be used to override and disable this setting.

  11:  d53b4ff2cee = 11:  2a6577974c7 sequencer: ignore HEAD ref under --update-refs
  12:  d5cd4b49e46 ! 12:  ec080ce1e90 sequencer: notify user of --update-refs activity

@@ sequencer.c: static int do_update_ref(struct repository *r, const char *refname)
+ struct strbuf update_msg = STRBUF_INIT;
+ struct strbuf error_msg = STRBUF_INIT;

- sequencer_get_update_refs_state(r->gitdir, &refs_to_oids);
+ if ((res = sequencer_get_update_refs_state(r->gitdir, &refs_to_oids)))
+ return res;

for_each_string_list_item(item, &refs_to_oids) {
struct update_ref_record *rec = item->util;
+ int loop_res;

- if (oideq(&rec->after, the_hash_algo->null_oid)) {
- /*
-@@ sequencer.c: static int do_update_refs(struct repository *r)
- continue;
- }
-
- res |= refs_update_ref(refs, "rewritten during rebase",
- item->string,
- &rec->after, &rec->before,
@@ sequencer.c: static int do_update_refs(struct repository *r)
}

@@ sequencer.c: cleanup_head_ref:
+ strbuf_release(&buf);
strbuf_release(&head_ref);
- }

-- do_update_refs(r);
-+ do_update_refs(r, opts->quiet);
+- if (do_update_refs(r))
++ if (do_update_refs(r, opts->quiet))
+ return -1;
+ }

- /*
- * Sequence of picks finished successfully; cleanup by

## t/t3404-rebase-interactive.sh ##
@@ t/t3404-rebase-interactive.sh: test_expect_success '--update-refs updates refs correctly' '
@@ t/t3404-rebase-interactive.sh: test_expect_success '--update-refs updates refs c

test_expect_success 'respect user edits to update-ref steps' '
@@ t/t3404-rebase-interactive.sh: test_expect_success '--update-refs: check failed ref update' '
+ # the lock in the update-refs file.
git rev-parse third >.git/refs/heads/second &&

- git rebase --continue 2>err &&
+- git rebase --continue 2>err &&
- grep "update_ref failed for ref '\''refs/heads/second'\''" err
++ test_must_fail git rebase --continue 2>err &&
+ grep "update_ref failed for ref '\''refs/heads/second'\''" err &&
+
+ cat >expect <<-\EOF &&