Skip to contentSkip to conversation

for-each-ref: introduce seeking functionality via '--skip-until'

PATCH Superseded Karthik Nayak 8 participants 102 messages Jul 1, 2025 – Jul 24, 2025 Documentation

lore
Share & tools
lore
#0 lore raw

cover for-each-ref: introduce seeking functionality via '--skip-until'

The git-for-each-ref(1) command is used to iterate over references present in a repository. In large repositories with millions of references, it would be optimal to paginate this output such that we can start iteration from a given reference. This would avoid having to iterate over all references from the beginning each time when paginating through results.

This series adds a '--skip-until' option in 'git-for-each-ref(1)'. When used, the reference iteration seeks to the first matching reference and iterates from there onward.

This enables efficient pagination workflows like:

    git for-each-ref --count=100
    git for-each-ref --count=100 --skip-until=refs/heads/branch-100
    git for-each-ref --count=100 --skip-until=refs/heads/branch-200

To add this functionality, we expose the ref_iterator outside the 'refs/' namespace and modify the ref_iterator_seek() to actually seek to a given reference and only set the prefix when the set_prefix field is set.

On the reftable and packed backend, the changes are simple. But since the files backend uses 'ref-cache' for reference handling, the changes there are a little more involved, since we need to setup the right levels and the indexing.

Initally I was also planning to cleanup all the refs_for_each...() functions in 'refs.h' by simply using the iterator, but this bloated the series. So I've left that for another day.

---
 Documentation/git-for-each-ref.adoc |   6 +-
 builtin/for-each-ref.c              |   5 +
 ref-filter.c                        |  57 ++++++++----
 ref-filter.h                        |   1 +
 refs.c                              |   2 +-
 refs.h                              | 153 ++++++++++++++++++++++++++++++
 refs/debug.c                        |   7 +-
 refs/files-backend.c                |   7 +-
 refs/iterator.c                     |  24 +++--
 refs/packed-backend.c               |  15 +--
 refs/ref-cache.c                    |  95 +++++++++++++++----
 refs/ref-cache.h                    |   7 --
 refs/refs-internal.h                | 152 ++----------------------------
 refs/reftable-backend.c             |  17 ++--
 t/t6302-for-each-ref-filter.sh      | 180 ++++++++++++++++++++++++++++++++++++
 15 files changed, 507 insertions(+), 221 deletions(-)

Karthik Nayak (4):
refs: expose ref_iterator via 'refs.h'
ref-cache: remove unused function 'find_ref_entry()'
refs: selectively set prefix in the seek functions
for-each-ref: introduce a '--skip-until' option

base-commit: cf6f63ea6bf35173e02e18bdc6a4ba41288acff9
change-id: 20250605-306-git-for-each-ref-pagination-0ba8a29ae646

Thanks
- Karthik

#1 lore raw

PATCH 1/4 refs: expose `ref_iterator` via 'refs.h'

The ref_iterator is an internal structure to the 'refs/' sub-directory, which allows iteration over refs. All reference iteration is built on top of these iterators.

External clients of the 'refs' subsystem use the various 'refs_for_each...()' functions to iterate over refs. However since these are wrapper functions, each combination of functionality requires a new wrapper function. This is not feasible as the functions pile up with the increase in requirements. Expose the internal reference iterator, so advanced users can mix and match options as needed.

---
 refs.h               | 148 +++++++++++++++++++++++++++++++++++++++++++++++++++
 refs/refs-internal.h | 145 +------------------------------------------------
 2 files changed, 150 insertions(+), 143 deletions(-)
refs.h +148 -0
index 46a6008e07..c05be6d0ac 100644
@@ -1190,4 +1190,152 @@ int repo_migrate_ref_storage_format(struct repository *repo,
 				    unsigned int flags,
 				    struct strbuf *err);
 
+/*
+ * Reference iterators
+ *
+ * A reference iterator encapsulates the state of an in-progress
+ * iteration over references. Create an instance of `struct
+ * ref_iterator` via one of the functions in this module.
+ *
+ * A freshly-created ref_iterator doesn't yet point at a reference. To
+ * advance the iterator, call ref_iterator_advance(). If successful,
+ * this sets the iterator's refname, oid, and flags fields to describe
+ * the next reference and returns ITER_OK. The data pointed at by
+ * refname and oid belong to the iterator; if you want to retain them
+ * after calling ref_iterator_advance() again or calling
+ * ref_iterator_free(), you must make a copy. When the iteration has
+ * been exhausted, ref_iterator_advance() releases any resources
+ * associated with the iteration, frees the ref_iterator object, and
+ * returns ITER_DONE. If you want to abort the iteration early, call
+ * ref_iterator_free(), which also frees the ref_iterator object and
+ * any associated resources. If there was an internal error advancing
+ * to the next entry, ref_iterator_advance() aborts the iteration,
+ * frees the ref_iterator, and returns ITER_ERROR.
+ *
+ * The reference currently being looked at can be peeled by calling
+ * ref_iterator_peel(). This function is often faster than peel_ref(),
+ * so it should be preferred when iterating over references.
+ *
+ * Putting it all together, a typical iteration looks like this:
+ *
+ *     int ok;
+ *     struct ref_iterator *iter = ...;
+ *
+ *     while ((ok = ref_iterator_advance(iter)) == ITER_OK) {
+ *             if (want_to_stop_iteration()) {
+ *                     ok = ITER_DONE;
+ *                     break;
+ *             }
+ *
+ *             // Access information about the current reference:
+ *             if (!(iter->flags & REF_ISSYMREF))
+ *                     printf("%s is %s\n", iter->refname, oid_to_hex(iter->oid));
+ *
+ *             // If you need to peel the reference:
+ *             ref_iterator_peel(iter, &oid);
+ *     }
+ *
+ *     if (ok != ITER_DONE)
+ *             handle_error();
+ *     ref_iterator_free(iter);
+ */
+struct ref_iterator;
+
+/*
+ * These flags are passed to refs_ref_iterator_begin() (and do_for_each_ref(),
+ * which feeds it).
+ */
+enum do_for_each_ref_flags {
+	/*
+	 * Include broken references in a do_for_each_ref*() iteration, which
+	 * would normally be omitted. This includes both refs that point to
+	 * missing objects (a true repository corruption), ones with illegal
+	 * names (which we prefer not to expose to callers), as well as
+	 * dangling symbolic refs (i.e., those that point to a non-existent
+	 * ref; this is not a corruption, but as they have no valid oid, we
+	 * omit them from normal iteration results).
+	 */
+	DO_FOR_EACH_INCLUDE_BROKEN = (1 << 0),
+
+	/*
+	 * Only include per-worktree refs in a do_for_each_ref*() iteration.
+	 * Normally this will be used with a files ref_store, since that's
+	 * where all reference backends will presumably store their
+	 * per-worktree refs.
+	 */
+	DO_FOR_EACH_PER_WORKTREE_ONLY = (1 << 1),
+
+	/*
+	 * Omit dangling symrefs from output; this only has an effect with
+	 * INCLUDE_BROKEN, since they are otherwise not included at all.
+	 */
+	DO_FOR_EACH_OMIT_DANGLING_SYMREFS = (1 << 2),
+
+	/*
+	 * Include root refs i.e. HEAD and pseudorefs along with the regular
+	 * refs.
+	 */
+	DO_FOR_EACH_INCLUDE_ROOT_REFS = (1 << 3),
+};
+
+/*
+ * Return an iterator that goes over each reference in `refs` for
+ * which the refname begins with prefix. If trim is non-zero, then
+ * trim that many characters off the beginning of each refname.
+ * The output is ordered by refname.
+ */
+struct ref_iterator *refs_ref_iterator_begin(
+		struct ref_store *refs,
+		const char *prefix, const char **exclude_patterns,
+		int trim, enum do_for_each_ref_flags flags);
+
+/*
+ * Advance the iterator to the first or next item and return ITER_OK.
+ * If the iteration is exhausted, free the resources associated with
+ * the ref_iterator and return ITER_DONE. On errors, free the iterator
+ * resources and return ITER_ERROR. It is a bug to use ref_iterator or
+ * call this function again after it has returned ITER_DONE or
+ * ITER_ERROR.
+ */
+int ref_iterator_advance(struct ref_iterator *ref_iterator);
+
+/*
+ * Seek the iterator to the first reference with the given prefix.
+ * The prefix is matched as a literal string, without regard for path
+ * separators. If prefix is NULL or the empty string, seek the iterator to the
+ * first reference again.
+ *
+ * This function is expected to behave as if a new ref iterator with the same
+ * prefix had been created, but allows reuse of iterators and thus may allow
+ * the backend to optimize. Parameters other than the prefix that have been
+ * passed when creating the iterator will remain unchanged.
+ *
+ * Returns 0 on success, a negative error code otherwise.
+ */
+int ref_iterator_seek(struct ref_iterator *ref_iterator,
+		      const char *prefix);
+
+/*
+ * If possible, peel the reference currently being viewed by the
+ * iterator. Return 0 on success.
+ */
+int ref_iterator_peel(struct ref_iterator *ref_iterator,
+		      struct object_id *peeled);
+
+/* Free the reference iterator and any associated resources. */
+void ref_iterator_free(struct ref_iterator *ref_iterator);
+
+/*
+ * The common backend for the for_each_*ref* functions. Call fn for
+ * each reference in iter. If the iterator itself ever returns
+ * ITER_ERROR, return -1. If fn ever returns a non-zero value, stop
+ * the iteration and return that value. Otherwise, return 0. In any
+ * case, free the iterator when done. This function is basically an
+ * adapter between the callback style of reference iteration and the
+ * iterator style.
+ */
+int do_for_each_ref_iterator(struct ref_iterator *iter,
+			     each_ref_fn fn, void *cb_data);
+
+
 #endif /* REFS_H */
refs/refs-internal.h +2 -143

169 lines not on this page · raw message

Signature · 1 line
2.49.0
#2 lore raw

PATCH 2/4 ref-cache: remove unused function 'find_ref_entry()'

In reply to Karthik Nayak

The 'find_ref_entry' function is no longer used, so remove it.

---
 refs/ref-cache.c | 14 --------------
 refs/ref-cache.h |  7 -------
 2 files changed, 21 deletions(-)
refs/ref-cache.c +0 -14
index c1f1bab1d5..8aaffa8c6b 100644
@@ -194,20 +194,6 @@ static struct ref_dir *find_containing_dir(struct ref_dir *dir,
 	return dir;
 }
 
-struct ref_entry *find_ref_entry(struct ref_dir *dir, const char *refname)
-{
-	int entry_index;
-	struct ref_entry *entry;
-	dir = find_containing_dir(dir, refname);
-	if (!dir)
-		return NULL;
-	entry_index = search_ref_dir(dir, refname, strlen(refname));
-	if (entry_index == -1)
-		return NULL;
-	entry = dir->entries[entry_index];
-	return (entry->flag & REF_DIR) ? NULL : entry;
-}
-
 /*
  * Emit a warning and return true iff ref1 and ref2 have the same name
  * and the same oid. Die if they have the same name but different
refs/ref-cache.h +0 -7
index 5f04e518c3..f635d2d824 100644
@@ -201,13 +201,6 @@ void free_ref_cache(struct ref_cache *cache);
  */
 void add_entry_to_dir(struct ref_dir *dir, struct ref_entry *entry);
 
-/*
- * Find the value entry with the given name in dir, sorting ref_dirs
- * and recursing into subdirectories as necessary.  If the name is not
- * found or it corresponds to a directory entry, return NULL.
- */
-struct ref_entry *find_ref_entry(struct ref_dir *dir, const char *refname);
-
 /*
  * Start iterating over references in `cache`. If `prefix` is
  * specified, only include references whose names start with that
Signature · 1 line
2.49.0
#3 lore raw

Re: [PATCH 2/4] ref-cache: remove unused function 'find_ref_entry()'

Quoted 1 line · Karthik Nayak

The 'find_ref_entry' function is no longer used, so remove it.

It seems that ba1c052f (ref_store: implement refs_peel_ref() generically, 2017-09-25) removed the last caller of it. This is long overdue ;-)

Quoted 49 lines
---
 refs/ref-cache.c | 14 --------------
 refs/ref-cache.h |  7 -------
 2 files changed, 21 deletions(-)
refs/ref-cache.c +0 -14
index c1f1bab1d5..8aaffa8c6b 100644
@@ -194,20 +194,6 @@ static struct ref_dir *find_containing_dir(struct ref_dir *dir,
 	return dir;
 }
 
-struct ref_entry *find_ref_entry(struct ref_dir *dir, const char *refname)
-{
-	int entry_index;
-	struct ref_entry *entry;
-	dir = find_containing_dir(dir, refname);
-	if (!dir)
-		return NULL;
-	entry_index = search_ref_dir(dir, refname, strlen(refname));
-	if (entry_index == -1)
-		return NULL;
-	entry = dir->entries[entry_index];
-	return (entry->flag & REF_DIR) ? NULL : entry;
-}
-
 /*
  * Emit a warning and return true iff ref1 and ref2 have the same name
  * and the same oid. Die if they have the same name but different
refs/ref-cache.h +0 -7
index 5f04e518c3..f635d2d824 100644
@@ -201,13 +201,6 @@ void free_ref_cache(struct ref_cache *cache);
  */
 void add_entry_to_dir(struct ref_dir *dir, struct ref_entry *entry);
 
-/*
- * Find the value entry with the given name in dir, sorting ref_dirs
- * and recursing into subdirectories as necessary.  If the name is not
- * found or it corresponds to a directory entry, return NULL.
- */
-struct ref_entry *find_ref_entry(struct ref_dir *dir, const char *refname);
-
 /*
  * Start iterating over references in `cache`. If `prefix` is
  * specified, only include references whose names start with that
#4 lore raw

PATCH 3/4 refs: selectively set prefix in the seek functions

In reply to Karthik Nayak

The ref iterator exposes a ref_iterator_seek() function. The name suggests that this would seek the iterator to a specific reference in some ways similar to how fseek() works for the filesystem.

However, the function actually sets the prefix for refs iteration. So further iteration would only yield references which match the particular prefix. This is a bit confusing.

Let's add a 'set_prefix' field to the function, which when set, will set the prefix for the iteration in-line with the existing behavior. But when the 'set_prefix' field is not set, the reference backends will simply seek to the specified reference without setting prefix. This allows users to start iteration from a specific reference.

In the packed and reftable backend, since references are available in a sorted list, the changes are simply setting the prefix if needed. The changes on the files-backend are a little more involved, since the files backend uses the 'ref-cache' mechanism. We move out the existing logic within cache_ref_iterator_seek() to cache_ref_iterator_set_prefix() which is called when set_prefix is set. We then parse the provided seek string and set the required levels and their indexes to ensure that seeking is possible.

---
 refs.c                  |  2 +-
 refs.h                  | 19 +++++++-----
 refs/debug.c            |  7 +++--
 refs/files-backend.c    |  7 +++--
 refs/iterator.c         | 24 +++++++++------
 refs/packed-backend.c   | 15 +++++----
 refs/ref-cache.c        | 81 ++++++++++++++++++++++++++++++++++++++++++++++---
 refs/refs-internal.h    |  7 +++--
 refs/reftable-backend.c | 17 ++++++-----
 9 files changed, 134 insertions(+), 45 deletions(-)
refs.c +1 -1
index dce5c49ca2..a4220d3537 100644
@@ -2669,7 +2669,7 @@ enum ref_transaction_error refs_verify_refnames_available(struct ref_store *refs
 			if (!iter) {
 				iter = refs_ref_iterator_begin(refs, dirname.buf, NULL, 0,
 							       DO_FOR_EACH_INCLUDE_BROKEN);
-			} else if (ref_iterator_seek(iter, dirname.buf) < 0) {
+			} else if (ref_iterator_seek(iter, dirname.buf, 1) < 0) {
 				goto cleanup;
 			}
 
refs.h +12 -7
index c05be6d0ac..c5e08db0ff 100644
@@ -1300,20 +1300,25 @@ struct ref_iterator *refs_ref_iterator_begin(
 int ref_iterator_advance(struct ref_iterator *ref_iterator);
 
 /*
- * Seek the iterator to the first reference with the given prefix.
- * The prefix is matched as a literal string, without regard for path
+ * Seek the iterator to the first reference matching the given seek string.
+ * The seek string is matched as a literal string, without regard for path
  * separators. If prefix is NULL or the empty string, seek the iterator to the
  * first reference again.
  *
- * This function is expected to behave as if a new ref iterator with the same
- * prefix had been created, but allows reuse of iterators and thus may allow
- * the backend to optimize. Parameters other than the prefix that have been
- * passed when creating the iterator will remain unchanged.
+ * When set_prefix is true, this function behaves as if a new ref iterator
+ * with the same prefix had been created, setting the prefix for subsequent
+ * iteration. When set_prefix is false, the iterator simply seeks to the
+ * specified reference without changing the existing prefix, allowing
+ * iteration to start from that specific reference.
+ *
+ * This function allows reuse of iterators and thus may allow the backend
+ * to optimize. Parameters other than the prefix that have been passed when
+ * creating the iterator will remain unchanged.
  *
  * Returns 0 on success, a negative error code otherwise.
  */
 int ref_iterator_seek(struct ref_iterator *ref_iterator,
-		      const char *prefix);
+		      const char *seek, int set_prefix);
 
 /*
  * If possible, peel the reference currently being viewed by the
refs/debug.c +4 -3
index 485e3079d7..7c04bcba10 100644
@@ -170,12 +170,13 @@ static int debug_ref_iterator_advance(struct ref_iterator *ref_iterator)
 }
 
 static int debug_ref_iterator_seek(struct ref_iterator *ref_iterator,
-				   const char *prefix)
+				   const char *seek, int set_prefix)
 {
 	struct debug_ref_iterator *diter =
 		(struct debug_ref_iterator *)ref_iterator;
-	int res = diter->iter->vtable->seek(diter->iter, prefix);
-	trace_printf_key(&trace_refs, "iterator_seek: %s: %d\n", prefix ? prefix : "", res);
+	int res = diter->iter->vtable->seek(diter->iter, seek, set_prefix);
+	trace_printf_key(&trace_refs, "iterator_seek: %s set_prefix: %d: %d\n",
+			 seek ? seek : "", set_prefix, res);
 	return res;
 }
 
refs/files-backend.c +4 -3
index bf6f89b1d1..827b15981c 100644
@@ -929,11 +929,11 @@ static int files_ref_iterator_advance(struct ref_iterator *ref_iterator)
 }
 
 static int files_ref_iterator_seek(struct ref_iterator *ref_iterator,
-				   const char *prefix)
+				   const char *seek, int set_prefix)
 {
 	struct files_ref_iterator *iter =
 		(struct files_ref_iterator *)ref_iterator;
-	return ref_iterator_seek(iter->iter0, prefix);
+	return ref_iterator_seek(iter->iter0, seek, set_prefix);
 }
 
 static int files_ref_iterator_peel(struct ref_iterator *ref_iterator,
@@ -2316,7 +2316,8 @@ static int files_reflog_iterator_advance(struct ref_iterator *ref_iterator)
 }
 
 static int files_reflog_iterator_seek(struct ref_iterator *ref_iterator UNUSED,
-				      const char *prefix UNUSED)
+				      const char *seek UNUSED,
+				      int set_prefix UNUSED)
 {
 	BUG("ref_iterator_seek() called for reflog_iterator");
 }
refs/iterator.c +14 -10
index 766d96e795..1f99045d40 100644
@@ -16,9 +16,9 @@ int ref_iterator_advance(struct ref_iterator *ref_iterator)
 }
 
 int ref_iterator_seek(struct ref_iterator *ref_iterator,
-		      const char *prefix)
+		      const char *seek, int set_prefix)
 {
-	return ref_iterator->vtable->seek(ref_iterator, prefix);
+	return ref_iterator->vtable->seek(ref_iterator, seek, set_prefix);
 }
 
 int ref_iterator_peel(struct ref_iterator *ref_iterator,
@@ -57,7 +57,8 @@ static int empty_ref_iterator_advance(struct ref_iterator *ref_iterator UNUSED)
 }
 
 static int empty_ref_iterator_seek(struct ref_iterator *ref_iterator UNUSED,
-				   const char *prefix UNUSED)
+				   const char *seek UNUSED,
+				   int set_prefix UNUSED)
 {
 	return 0;
 }
@@ -224,7 +225,7 @@ static int merge_ref_iterator_advance(struct ref_iterator *ref_iterator)
 }
 
 static int merge_ref_iterator_seek(struct ref_iterator *ref_iterator,
-				   const char *prefix)
+				   const char *seek, int set_prefix)
 {
 	struct merge_ref_iterator *iter =
 		(struct merge_ref_iterator *)ref_iterator;
@@ -234,11 +235,11 @@ static int merge_ref_iterator_seek(struct ref_iterator *ref_iterator,
 	iter->iter0 = iter->iter0_owned;
 	iter->iter1 = iter->iter1_owned;
 
-	ret = ref_iterator_seek(iter->iter0, prefix);
+	ret = ref_iterator_seek(iter->iter0, seek, set_prefix);
 	if (ret < 0)
 		return ret;
 
-	ret = ref_iterator_seek(iter->iter1, prefix);
+	ret = ref_iterator_seek(iter->iter1, seek, set_prefix);
 	if (ret < 0)
 		return ret;
 
@@ -407,13 +408,16 @@ static int prefix_ref_iterator_advance(struct ref_iterator *ref_iterator)
 }
 
 static int prefix_ref_iterator_seek(struct ref_iterator *ref_iterator,
-				    const char *prefix)
+				    const char *seek, int set_prefix)
 {
 	struct prefix_ref_iterator *iter =
 		(struct prefix_ref_iterator *)ref_iterator;
-	free(iter->prefix);
-	iter->prefix = xstrdup_or_null(prefix);
-	return ref_iterator_seek(iter->iter0, prefix);
+
+	if (set_prefix) {
+		free(iter->prefix);
+		iter->prefix = xstrdup_or_null(seek);
+	}
+	return ref_iterator_seek(iter->iter0, seek, set_prefix);
 }
 
 static int prefix_ref_iterator_peel(struct ref_iterator *ref_iterator,
refs/packed-backend.c +9 -6
index 7fd73a0e6d..dca886c5cc 100644
@@ -1004,19 +1004,22 @@ static int packed_ref_iterator_advance(struct ref_iterator *ref_iterator)
 }
 
 static int packed_ref_iterator_seek(struct ref_iterator *ref_iterator,
-				    const char *prefix)
+				    const char *seek, int set_prefix)
 {
 	struct packed_ref_iterator *iter =
 		(struct packed_ref_iterator *)ref_iterator;
 	const char *start;
 
-	if (prefix && *prefix)
-		start = find_reference_location(iter->snapshot, prefix, 0);
+	if (seek && *seek)
+		start = find_reference_location(iter->snapshot, seek, 0);
 	else
 		start = iter->snapshot->start;
 
-	free(iter->prefix);
-	iter->prefix = xstrdup_or_null(prefix);
+	if (set_prefix) {
+		free(iter->prefix);
+		iter->prefix = xstrdup_or_null(seek);
+	}
+
 	iter->pos = start;
 	iter->eof = iter->snapshot->eof;
 
@@ -1194,7 +1197,7 @@ static struct ref_iterator *packed_ref_iterator_begin(
 	iter->repo = ref_store->repo;
 	iter->flags = flags;
 
-	if (packed_ref_iterator_seek(&iter->base, prefix) < 0) {
+	if (packed_ref_iterator_seek(&iter->base, prefix, 1) < 0) {
 		ref_iterator_free(&iter->base);
 		return NULL;
 	}
refs/ref-cache.c +76 -5

100 lines not on this page · raw message

refs/refs-internal.h +4 -3
index 03f5df04d5..cee377696c 100644
@@ -353,11 +353,12 @@ void base_ref_iterator_init(struct ref_iterator *iter,
 typedef int ref_iterator_advance_fn(struct ref_iterator *ref_iterator);
 
 /*
- * Seek the iterator to the first reference matching the given prefix. Should
- * behave the same as if a new iterator was created with the same prefix.
+ * Seek the iterator to the first matching reference. If set_prefix is set,
+ * it would behave the same as if a new iterator was created with the same
+ * prefix.
  */
 typedef int ref_iterator_seek_fn(struct ref_iterator *ref_iterator,
-				 const char *prefix);
+				 const char *seek, int set_prefix);
 
 /*
  * Peels the current ref, returning 0 for success or -1 for failure.
refs/reftable-backend.c +10 -7
index 4c3817f4ec..81fb6a9028 100644
@@ -719,15 +719,17 @@ static int reftable_ref_iterator_advance(struct ref_iterator *ref_iterator)
 }
 
 static int reftable_ref_iterator_seek(struct ref_iterator *ref_iterator,
-				      const char *prefix)
+				      const char *seek, int set_prefix)
 {
 	struct reftable_ref_iterator *iter =
 		(struct reftable_ref_iterator *)ref_iterator;
 
-	free(iter->prefix);
-	iter->prefix = xstrdup_or_null(prefix);
-	iter->prefix_len = prefix ? strlen(prefix) : 0;
-	iter->err = reftable_iterator_seek_ref(&iter->iter, prefix);
+	if (set_prefix) {
+		free(iter->prefix);
+		iter->prefix = xstrdup_or_null(seek);
+		iter->prefix_len = seek ? strlen(seek) : 0;
+	}
+	iter->err = reftable_iterator_seek_ref(&iter->iter, seek);
 
 	return iter->err;
 }
@@ -839,7 +841,7 @@ static struct reftable_ref_iterator *ref_iterator_for_stack(struct reftable_ref_
 	if (ret)
 		goto done;
 
-	ret = reftable_ref_iterator_seek(&iter->base, prefix);
+	ret = reftable_ref_iterator_seek(&iter->base, prefix, 1);
 	if (ret)
 		goto done;
 
@@ -2042,7 +2044,8 @@ static int reftable_reflog_iterator_advance(struct ref_iterator *ref_iterator)
 }
 
 static int reftable_reflog_iterator_seek(struct ref_iterator *ref_iterator UNUSED,
-					 const char *prefix UNUSED)
+					 const char *seek UNUSED,
+					 int set_prefix UNUSED)
 {
 	BUG("reftable reflog iterator cannot be seeked");
 	return -1;
Signature · 1 line
2.49.0
#5 lore raw

Re: [PATCH 3/4] refs: selectively set prefix in the seek functions

Quoted 14 lines · Karthik Nayak

The ref iterator exposes a ref_iterator_seek() function. The name suggests that this would seek the iterator to a specific reference in some ways similar to how fseek() works for the filesystem.

However, the function actually sets the prefix for refs iteration. So further iteration would only yield references which match the particular prefix. This is a bit confusing.

Let's add a 'set_prefix' field to the function, which when set, will set the prefix for the iteration in-line with the existing behavior. But when the 'set_prefix' field is not set, the reference backends will simply seek to the specified reference without setting prefix. This allows users to start iteration from a specific reference.

Quoted 8 lines

In the packed and reftable backend, since references are available in a sorted list, the changes are simply setting the prefix if needed. The changes on the files-backend are a little more involved, since the files backend uses the 'ref-cache' mechanism. We move out the existing logic within cache_ref_iterator_seek() to cache_ref_iterator_set_prefix() which is called when set_prefix is set. We then parse the provided seek string and set the required levels and their indexes to ensure that seeking is possible.

That solution makes sense.

refs.c enum ref_transaction_error refs_verify_refnames_available(struct ref_store *refs · line 2675

Quoted 13 lines
refs.c +1 -1
index dce5c49ca2..a4220d3537 100644
@@ -2669,7 +2669,7 @@ enum ref_transaction_error refs_verify_refnames_available(struct ref_store *refs
 			if (!iter) {
 				iter = refs_ref_iterator_begin(refs, dirname.buf, NULL, 0,
 							       DO_FOR_EACH_INCLUDE_BROKEN);
-			} else if (ref_iterator_seek(iter, dirname.buf) < 0) {
+			} else if (ref_iterator_seek(iter, dirname.buf, 1) < 0) {
 				goto cleanup;
 			}
 

This is quite unreadable, as you have no idea what 1 could mean. Let's make this a unsigned flags variable instead so that we can provide meaningful names.

Quoted 14 lines
refs.h +2 -2
index c05be6d0ac..c5e08db0ff 100644
@@ -1300,20 +1300,25 @@ struct ref_iterator *refs_ref_iterator_begin(
 int ref_iterator_advance(struct ref_iterator *ref_iterator);
 
 /*
- * Seek the iterator to the first reference with the given prefix.
- * The prefix is matched as a literal string, without regard for path
+ * Seek the iterator to the first reference matching the given seek string.
+ * The seek string is matched as a literal string, without regard for path
  * separators. If prefix is NULL or the empty string, seek the iterator to the
  * first reference again.

refs.h struct ref_iterator *refs_ref_iterator_begin( · line 1312

Quoted 10 lines
refs.h +5 -4
  *
- * This function is expected to behave as if a new ref iterator with the same
- * prefix had been created, but allows reuse of iterators and thus may allow
- * the backend to optimize. Parameters other than the prefix that have been
- * passed when creating the iterator will remain unchanged.
+ * When set_prefix is true, this function behaves as if a new ref iterator
+ * with the same prefix had been created, setting the prefix for subsequent
+ * iteration. When set_prefix is false, the iterator simply seeks to the
+ * specified reference without changing the existing prefix, allowing
+ * iteration to start from that specific reference.

I think we should detangle this paragraph a bit.

This function is expected to behave as if a new ref iterator has been created, but allows reuse of it

#6 lore raw

Re: [PATCH 3/4] refs: selectively set prefix in the seek functions

Quoted 35 lines · Patrick Steinhardt
Quoted 14 lines · Karthik Nayak

The ref iterator exposes a ref_iterator_seek() function. The name suggests that this would seek the iterator to a specific reference in some ways similar to how fseek() works for the filesystem.

However, the function actually sets the prefix for refs iteration. So further iteration would only yield references which match the particular prefix. This is a bit confusing.

Let's add a 'set_prefix' field to the function, which when set, will set the prefix for the iteration in-line with the existing behavior. But when the 'set_prefix' field is not set, the reference backends will simply seek to the specified reference without setting prefix. This allows users to start iteration from a specific reference.

Quoted 8 lines

In the packed and reftable backend, since references are available in a sorted list, the changes are simply setting the prefix if needed. The changes on the files-backend are a little more involved, since the files backend uses the 'ref-cache' mechanism. We move out the existing logic within cache_ref_iterator_seek() to cache_ref_iterator_set_prefix() which is called when set_prefix is set. We then parse the provided seek string and set the required levels and their indexes to ensure that seeking is possible.

That solution makes sense.

Quoted 9 lines
refs.c +0 -1
index dce5c49ca2..a4220d3537 100644
@@ -2669,7 +2669,7 @@ enum ref_transaction_error refs_verify_refnames_available(struct ref_store *refs
 			if (!iter) {
 				iter = refs_ref_iterator_begin(refs, dirname.buf, NULL, 0,
 							       DO_FOR_EACH_INCLUDE_BROKEN);
-			} else if (ref_iterator_seek(iter, dirname.buf) < 0) {
Quoted 9 lines

refs.c enum ref_transaction_error refs_verify_refnames_available(struct ref_store *refs · line 2675

Quoted 4 lines
refs.c +1 -0
+			} else if (ref_iterator_seek(iter, dirname.buf, 1) < 0) {
 				goto cleanup;
 			}
 

This is quite unreadable, as you have no idea what 1 could mean. Let's make this a unsigned flags variable instead so that we can provide meaningful names.

Yeah, that would make a lot more sense. Will amend.

Quoted 27 lines
Quoted 14 lines
refs.h +2 -2
index c05be6d0ac..c5e08db0ff 100644
@@ -1300,20 +1300,25 @@ struct ref_iterator *refs_ref_iterator_begin(
 int ref_iterator_advance(struct ref_iterator *ref_iterator);
 
 /*
- * Seek the iterator to the first reference with the given prefix.
- * The prefix is matched as a literal string, without regard for path
+ * Seek the iterator to the first reference matching the given seek string.
+ * The seek string is matched as a literal string, without regard for path
  * separators. If prefix is NULL or the empty string, seek the iterator to the
  * first reference again.

refs.h struct ref_iterator *refs_ref_iterator_begin( · line 1312

Quoted 10 lines
refs.h +5 -4
  *
- * This function is expected to behave as if a new ref iterator with the same
- * prefix had been created, but allows reuse of iterators and thus may allow
- * the backend to optimize. Parameters other than the prefix that have been
- * passed when creating the iterator will remain unchanged.
+ * When set_prefix is true, this function behaves as if a new ref iterator
+ * with the same prefix had been created, setting the prefix for subsequent
+ * iteration. When set_prefix is false, the iterator simply seeks to the
+ * specified reference without changing the existing prefix, allowing
+ * iteration to start from that specific reference.

I think we should detangle this paragraph a bit.

Quoted 2 lines

This function is expected to behave as if a new ref iterator has been created, but allows reuse of it

Sure, let me add this in. Thanks!

#7 lore raw

PATCH 4/4 for-each-ref: introduce a '--skip-until' option

In reply to Karthik Nayak

The git-for-each-ref(1) command is used to iterate over references present in a repository. In large repositories with millions of references, it would be optimal to paginate this output such that we can start iteration from a given reference. This would avoid having to iterate over all references from the beginning each time when paginating through results.

The previous commit added 'seek' functionality to the reference backends. Utilize this and expose a '--skip-until' option in 'git-for-each-ref(1)'. When used, the reference iteration seeks to the first matching reference and iterates from there onward.

This enables efficient pagination workflows like:

    git for-each-ref --count=100
    git for-each-ref --count=100 --skip-until=refs/heads/branch-100
    git for-each-ref --count=100 --skip-until=refs/heads/branch-200
---
 Documentation/git-for-each-ref.adoc |   6 +-
 builtin/for-each-ref.c              |   5 +
 ref-filter.c                        |  57 ++++++++----
 ref-filter.h                        |   1 +
 t/t6302-for-each-ref-filter.sh      | 180 ++++++++++++++++++++++++++++++++++++
 5 files changed, 230 insertions(+), 19 deletions(-)
Documentation/git-for-each-ref.adoc +5 -1
index 5ef89fc0fe..4bf7c66b8c 100644
@@ -14,7 +14,7 @@ SYNOPSIS
 		   [--points-at=<object>]
 		   [--merged[=<object>]] [--no-merged[=<object>]]
 		   [--contains[=<object>]] [--no-contains[=<object>]]
-		   [--exclude=<pattern> ...]
+		   [--exclude=<pattern> ...] [--skip-until=<pattern>]
 
 DESCRIPTION
 -----------
@@ -108,6 +108,10 @@ TAB %(refname)`.
 --include-root-refs::
 	List root refs (HEAD and pseudorefs) apart from regular refs.
 
+--skip-until::
+    Skip references up to the specified pattern. Cannot be used with
+    general pattern matching.
+
 FIELD NAMES
 -----------
 
builtin/for-each-ref.c +5 -0
index 3d2207ec77..543013cd11 100644
@@ -13,6 +13,7 @@ static char const * const for_each_ref_usage[] = {
 	N_("git for-each-ref [--points-at <object>]"),
 	N_("git for-each-ref [--merged [<commit>]] [--no-merged [<commit>]]"),
 	N_("git for-each-ref [--contains [<commit>]] [--no-contains [<commit>]]"),
+	N_("git for-each-ref [--skip-until <pattern>]"),
 	NULL
 };
 
@@ -44,6 +45,7 @@ int cmd_for_each_ref(int argc,
 		OPT_GROUP(""),
 		OPT_INTEGER( 0 , "count", &format.array_opts.max_count, N_("show only <n> matched refs")),
 		OPT_STRING(  0 , "format", &format.format, N_("format"), N_("format to use for the output")),
+		OPT_STRING(  0 , "skip-until", &filter.seek, N_("skip-until"), N_("skip references until")),
 		OPT__COLOR(&format.use_color, N_("respect format colors")),
 		OPT_REF_FILTER_EXCLUDE(&filter),
 		OPT_REF_SORT(&sorting_options),
@@ -100,6 +102,9 @@ int cmd_for_each_ref(int argc,
 		filter.name_patterns = argv;
 	}
 
+	if (filter.seek && filter.name_patterns && filter.name_patterns[0])
+		die(_("cannot use --skip-until with patterns"));
+
 	if (include_root_refs)
 		flags |= FILTER_REFS_ROOT_REFS | FILTER_REFS_DETACHED_HEAD;
 
ref-filter.c +39 -18

102 lines not on this page · raw message

ref-filter.h +1 -0
index c98c4fbd4c..9e97c65bc2 100644
@@ -64,6 +64,7 @@ struct ref_array {
 
 struct ref_filter {
 	const char **name_patterns;
+	const char *seek;
 	struct strvec exclude;
 	struct oid_array points_at;
 	struct commit_list *with_commit;
t/t6302-for-each-ref-filter.sh +180 -0

184 lines not on this page · raw message

Signature · 1 line
2.49.0
#8 lore raw

Re: [PATCH 4/4] for-each-ref: introduce a '--skip-until' option

Quoted 14 lines · Karthik Nayak
Documentation/git-for-each-ref.adoc +1 -1
index 5ef89fc0fe..4bf7c66b8c 100644
@@ -14,7 +14,7 @@ SYNOPSIS
 		   [--points-at=<object>]
 		   [--merged[=<object>]] [--no-merged[=<object>]]
 		   [--contains[=<object>]] [--no-contains[=<object>]]
-		   [--exclude=<pattern> ...]
+		   [--exclude=<pattern> ...] [--skip-until=<pattern>]
 
 DESCRIPTION
 -----------

Documentation/git-for-each-ref.adoc TAB %(refname)`. · line 117

Quoted 10 lines
Documentation/git-for-each-ref.adoc +4 -0
@@ -108,6 +108,10 @@ TAB %(refname)`.
 --include-root-refs::
 	List root refs (HEAD and pseudorefs) apart from regular refs.
 
+--skip-until::
+    Skip references up to the specified pattern. Cannot be used with
+    general pattern matching.
+
 FIELD NAMES
 -----------
 

Is it "up to and including the specified pattern" or "up to but excluding the specified pattern"? It would help to make it very explicit whether the pattern itself would be yielded or not.

Quoted 20 lines
ref-filter.c +2 -4
index 7a274633cf..9d0255d5db 100644
@@ -2714,20 +2716,28 @@ static int for_each_fullref_in_pattern(struct ref_filter *filter,
 		 * so just return everything and let the caller
 		 * sort it out.
 		 */
-		return refs_for_each_fullref_in(get_main_ref_store(the_repository),
-						"", NULL, cb, cb_data);
+		goto non_prefix_iter;
 	}
 
 	if (!filter->name_patterns[0]) {
 		/* no patterns; we have to look at everything */
-		return refs_for_each_fullref_in(get_main_ref_store(the_repository),
-						 "", filter->exclude.v, cb, cb_data);
+		goto non_prefix_iter;
 	}
 

ref-filter.c static int for_each_fullref_in_pattern(struct ref_filter *filter, · line 2736

Quoted 10 lines
ref-filter.c +6 -0
 	return refs_for_each_fullref_in_prefixes(get_main_ref_store(the_repository),
 						 NULL, filter->name_patterns,
 						 filter->exclude.v,
 						 cb, cb_data);
+
+non_prefix_iter:
+	iter = refs_ref_iterator_begin(get_main_ref_store(the_repository), "",
+				       NULL, 0, flags);
+	if (filter->seek)
+		ret = ref_iterator_seek(iter, filter->seek, 0);

Hm, this interface is somewhat weird now, as we have a split in what the prefix-string meeks when creating the iterator and seeking it. I think we should align those two functions.

ref-filter.c static int do_filter_refs(struct ref_filter *filter, unsigned int type, each_ref · line 3212

Quoted 11 lines
ref-filter.c +4 -0
+	if (ret)
+		return ret;
+
+	return do_for_each_ref_iterator(iter, cb, cb_data);
 }
 
 /*
@@ -3200,6 +3210,8 @@ static int do_filter_refs(struct ref_filter *filter, unsigned int type, each_ref
 	if (!filter->kind)
 		die("filter_refs: invalid type");
 	else {

The if branch now needs to be updated to have curly braces, as well.

Patrick

#9 lore raw

Re: [PATCH 4/4] for-each-ref: introduce a '--skip-until' option

Quoted 18 lines · Patrick Steinhardt
Quoted 17 lines · Karthik Nayak
Documentation/git-for-each-ref.adoc +1 -1
index 5ef89fc0fe..4bf7c66b8c 100644
@@ -14,7 +14,7 @@ SYNOPSIS
 		   [--points-at=<object>]
 		   [--merged[=<object>]] [--no-merged[=<object>]]
 		   [--contains[=<object>]] [--no-contains[=<object>]]
-		   [--exclude=<pattern> ...]
+		   [--exclude=<pattern> ...] [--skip-until=<pattern>]
 
 DESCRIPTION
 -----------
@@ -108,6 +108,10 @@ TAB %(refname)`.
 --include-root-refs::
 	List root refs (HEAD and pseudorefs) apart from regular refs.
 
Quoted 12 lines

Documentation/git-for-each-ref.adoc TAB %(refname)`. · line 117

Quoted 7 lines
Documentation/git-for-each-ref.adoc +4 -0
+--skip-until::
+    Skip references up to the specified pattern. Cannot be used with
+    general pattern matching.
+
 FIELD NAMES
 -----------
 

Is it "up to and including the specified pattern" or "up to but excluding the specified pattern"? It would help to make it very explicit whether the pattern itself would be yielded or not.

It is "up to and including", will modify to make this more clearer.

Quoted 20 lines
Quoted 20 lines
ref-filter.c +2 -4
index 7a274633cf..9d0255d5db 100644
@@ -2714,20 +2716,28 @@ static int for_each_fullref_in_pattern(struct ref_filter *filter,
 		 * so just return everything and let the caller
 		 * sort it out.
 		 */
-		return refs_for_each_fullref_in(get_main_ref_store(the_repository),
-						"", NULL, cb, cb_data);
+		goto non_prefix_iter;
 	}
 
 	if (!filter->name_patterns[0]) {
 		/* no patterns; we have to look at everything */
-		return refs_for_each_fullref_in(get_main_ref_store(the_repository),
-						 "", filter->exclude.v, cb, cb_data);
+		goto non_prefix_iter;
 	}
 
Quoted 15 lines

ref-filter.c static int for_each_fullref_in_pattern(struct ref_filter *filter, · line 2736

Quoted 10 lines
ref-filter.c +6 -0
 	return refs_for_each_fullref_in_prefixes(get_main_ref_store(the_repository),
 						 NULL, filter->name_patterns,
 						 filter->exclude.v,
 						 cb, cb_data);
+
+non_prefix_iter:
+	iter = refs_ref_iterator_begin(get_main_ref_store(the_repository), "",
+				       NULL, 0, flags);
+	if (filter->seek)
+		ret = ref_iterator_seek(iter, filter->seek, 0);

Hm, this interface is somewhat weird now, as we have a split in what the prefix-string meeks when creating the iterator and seeking it. I think we should align those two functions.

The refs_ref_iterator_begin() takes in a prefix string, which sets the prefix.

The ref_iterator_seek() takes in a seek string, but a flag allows it also set the prefix.

I think this is okay since the naming matches what it does.

The alternate would be to refs_ref_iterator_begin() to also take in a seek string with a flag to also set the prefix. What do you think? I'm okay either ways.

Quoted 14 lines

ref-filter.c static int do_filter_refs(struct ref_filter *filter, unsigned int type, each_ref · line 3212

Quoted 11 lines
ref-filter.c +4 -0
+	if (ret)
+		return ret;
+
+	return do_for_each_ref_iterator(iter, cb, cb_data);
 }
 
 /*
@@ -3200,6 +3210,8 @@ static int do_filter_refs(struct ref_filter *filter, unsigned int type, each_ref
 	if (!filter->kind)
 		die("filter_refs: invalid type");
 	else {

The if branch now needs to be updated to have curly braces, as well.

Quoted 1 line

Patrick

Yes, will add.

Thanks for the review!

#10 lore raw

Re: [PATCH 4/4] for-each-ref: introduce a '--skip-until' option

Quoted 41 lines · Karthik Nayak
Quoted 21 lines · Patrick Steinhardt
Quoted 20 lines · Karthik Nayak
ref-filter.c +2 -4
index 7a274633cf..9d0255d5db 100644
@@ -2714,20 +2716,28 @@ static int for_each_fullref_in_pattern(struct ref_filter *filter,
 		 * so just return everything and let the caller
 		 * sort it out.
 		 */
-		return refs_for_each_fullref_in(get_main_ref_store(the_repository),
-						"", NULL, cb, cb_data);
+		goto non_prefix_iter;
 	}
 
 	if (!filter->name_patterns[0]) {
 		/* no patterns; we have to look at everything */
-		return refs_for_each_fullref_in(get_main_ref_store(the_repository),
-						 "", filter->exclude.v, cb, cb_data);
+		goto non_prefix_iter;
 	}
 
Quoted 15 lines

ref-filter.c static int for_each_fullref_in_pattern(struct ref_filter *filter, · line 2736

Quoted 10 lines
ref-filter.c +6 -0
 	return refs_for_each_fullref_in_prefixes(get_main_ref_store(the_repository),
 						 NULL, filter->name_patterns,
 						 filter->exclude.v,
 						 cb, cb_data);
+
+non_prefix_iter:
+	iter = refs_ref_iterator_begin(get_main_ref_store(the_repository), "",
+				       NULL, 0, flags);
+	if (filter->seek)
+		ret = ref_iterator_seek(iter, filter->seek, 0);

Hm, this interface is somewhat weird now, as we have a split in what the prefix-string meeks when creating the iterator and seeking it. I think we should align those two functions.

The refs_ref_iterator_begin() takes in a prefix string, which sets the prefix.

Quoted 8 lines

The ref_iterator_seek() takes in a seek string, but a flag allows it also set the prefix.

I think this is okay since the naming matches what it does.

The alternate would be to refs_ref_iterator_begin() to also take in a seek string with a flag to also set the prefix. What do you think? I'm okay either ways.

I just think that the interface is a bit confusing. It's weird that the needle that we're seeking for may or may not be used to update internal state, and that this is inconsistent with the similar fields that you pass to the iterator when creating it. So after seeking it sometimes acts like you have created a new iterator with the needle, sometimes it does not becaus we retain internal state. This kind of inconsistency invites mistakes.

Patrick

#11 lore raw
In reply to Karthik Nayak
Quoted 4 lines · Karthik Nayak

The git-for-each-ref(1) command is used to iterate over references present in a repository. In large repositories with millions of references, it would be optimal to paginate this output such that we can start iteration from a given reference.

I haven't looked at the patches, but should the end-user's mental model of the process be like this?

- We have a native order in which references are sorted and that is what "git for-each-ref" without "--sort" option gives them.

- They can use the "--skip-until" option to seek in the above order and start iterating in the middle.

- If they give "--sort", the set of refs to be shown would not change; skipping is done in the native order and then the remainder is given sorted.

Please make sure that the documentation is clear enough to avoid a misunderstanding that this feature would kick in after we grab all refs and sort them. If it worked that way, it would allow us to say "going from newer to older, but skipping the most recent ones that were touched within a week", which would have been nice, but that is not what we are doing with this feature---I think it is OK but we need to be clear about it in the documentation.

Quoted 3 lines

This series adds a '--skip-until' option in 'git-for-each-ref(1)'. When used, the reference iteration seeks to the first matching reference and iterates from there onward.

OK. Even the filesystem backed ones we internall sort after doing readdir() loop, so this is feasible. Nice.

Quoted 3 lines

Initally I was also planning to cleanup all the refs_for_each...() functions in 'refs.h' by simply using the iterator, but this bloated the series. So I've left that for another day.

OK.

#12 lore raw
Quoted 20 lines · Junio C Hamano
Quoted 4 lines · Karthik Nayak

The git-for-each-ref(1) command is used to iterate over references present in a repository. In large repositories with millions of references, it would be optimal to paginate this output such that we can start iteration from a given reference.

I haven't looked at the patches, but should the end-user's mental model of the process be like this?

- We have a native order in which references are sorted and that is what "git for-each-ref" without "--sort" option gives them.

- They can use the "--skip-until" option to seek in the above order and start iterating in the middle.

- If they give "--sort", the set of refs to be shown would not change; skipping is done in the native order and then the remainder is given sorted.

Quoted 8 lines

Please make sure that the documentation is clear enough to avoid a misunderstanding that this feature would kick in after we grab all refs and sort them. If it worked that way, it would allow us to say "going from newer to older, but skipping the most recent ones that were touched within a week", which would have been nice, but that is not what we are doing with this feature---I think it is OK but we need to be clear about it in the documentation.

I totally didn't consider '--sort'. I do agree that we should document that behavior if we tend to keep it as is. I wonder if it is worthwhile to even prevent the two from being used together. I find the whole "we skip before sorting" to be very confusing.

Quoted 7 lines
Quoted 3 lines

This series adds a '--skip-until' option in 'git-for-each-ref(1)'. When used, the reference iteration seeks to the first matching reference and iterates from there onward.

OK. Even the filesystem backed ones we internall sort after doing readdir() loop, so this is feasible. Nice.

Yup. We have 'sort_ref_dir()' to sort each directory parsed.

Quoted 5 lines
Quoted 3 lines

Initally I was also planning to cleanup all the refs_for_each...() functions in 'refs.h' by simply using the iterator, but this bloated the series. So I've left that for another day.

OK.

#13 lore raw
In reply to Karthik Nayak

Offtopic. After applying this topic, I asked clang-format if it wants to change anything.

    $ git clang-format --diff $(git merge-base HEAD master)

The result was disasterous. Can "clang-format --diff" mode be taught a bit more focused to avoid touching existing entries in the same array (in this case opts[] that has tons of options for the "git for-each-ref" command), when only one new entry was added, I wonder?

Also I am not impressed by the change it made to the code that is commented out (in refs.h).

Line wrapping it did to refs_ref_iterator_begin() is an improvement, but those to ref_iterator_seek() and do_for_each_ref_iterator() are unnecessary (both of these were more readble in the original).

Even though I found its output better for Toon's "last-modified" changes, I am not impressed by what clang-format suggested for this series.

builtin/for-each-ref.c +26 -17
index 543013cd11..39056557d4 100644
@@ -8,7 +8,7 @@
 #include "strbuf.h"
 #include "strvec.h"
 
-static char const * const for_each_ref_usage[] = {
+static char const *const for_each_ref_usage[] = {
 	N_("git for-each-ref [<options>] [<pattern>]"),
 	N_("git for-each-ref [--points-at <object>]"),
 	N_("git for-each-ref [--merged [<commit>]] [--no-merged [<commit>]]"),
@@ -33,32 +33,41 @@ int cmd_for_each_ref(int argc,
 	struct option opts[] = {
 		OPT_BIT('s', "shell", &format.quote_style,
 			N_("quote placeholders suitably for shells"), QUOTE_SHELL),
-		OPT_BIT('p', "perl",  &format.quote_style,
+		OPT_BIT('p', "perl", &format.quote_style,
 			N_("quote placeholders suitably for perl"), QUOTE_PERL),
-		OPT_BIT(0 , "python", &format.quote_style,
-			N_("quote placeholders suitably for python"), QUOTE_PYTHON),
-		OPT_BIT(0 , "tcl",  &format.quote_style,
+		OPT_BIT(0, "python", &format.quote_style,
+			N_("quote placeholders suitably for python"),
+			QUOTE_PYTHON),
+		OPT_BIT(0, "tcl", &format.quote_style,
 			N_("quote placeholders suitably for Tcl"), QUOTE_TCL),
-		OPT_BOOL(0, "omit-empty",  &format.array_opts.omit_empty,
-			N_("do not output a newline after empty formatted refs")),
+		OPT_BOOL(0, "omit-empty", &format.array_opts.omit_empty,
+			 N_("do not output a newline after empty formatted refs")),
 
 		OPT_GROUP(""),
-		OPT_INTEGER( 0 , "count", &format.array_opts.max_count, N_("show only <n> matched refs")),
-		OPT_STRING(  0 , "format", &format.format, N_("format"), N_("format to use for the output")),
-		OPT_STRING(  0 , "skip-until", &filter.seek, N_("skip-until"), N_("skip references until")),
+		OPT_INTEGER(0, "count", &format.array_opts.max_count,
+			    N_("show only <n> matched refs")),
+		OPT_STRING(0, "format", &format.format, N_("format"),
+			   N_("format to use for the output")),
+		OPT_STRING(0, "skip-until", &filter.seek, N_("skip-until"),
+			   N_("skip references until")),
 		OPT__COLOR(&format.use_color, N_("respect format colors")),
 		OPT_REF_FILTER_EXCLUDE(&filter),
 		OPT_REF_SORT(&sorting_options),
-		OPT_CALLBACK(0, "points-at", &filter.points_at,
-			     N_("object"), N_("print only refs which points at the given object"),
+		OPT_CALLBACK(0, "points-at", &filter.points_at, N_("object"),
+			     N_("print only refs which points at the given object"),
 			     parse_opt_object_name),
 		OPT_MERGED(&filter, N_("print only refs that are merged")),
 		OPT_NO_MERGED(&filter, N_("print only refs that are not merged")),
-		OPT_CONTAINS(&filter.with_commit, N_("print only refs which contain the commit")),
-		OPT_NO_CONTAINS(&filter.no_commit, N_("print only refs which don't contain the commit")),
-		OPT_BOOL(0, "ignore-case", &icase, N_("sorting and filtering are case insensitive")),
-		OPT_BOOL(0, "stdin", &from_stdin, N_("read reference patterns from stdin")),
-		OPT_BOOL(0, "include-root-refs", &include_root_refs, N_("also include HEAD ref and pseudorefs")),
+		OPT_CONTAINS(&filter.with_commit,
+			     N_("print only refs which contain the commit")),
+		OPT_NO_CONTAINS(&filter.no_commit,
+				N_("print only refs which don't contain the commit")),
+		OPT_BOOL(0, "ignore-case", &icase,
+			 N_("sorting and filtering are case insensitive")),
+		OPT_BOOL(0, "stdin", &from_stdin,
+			 N_("read reference patterns from stdin")),
+		OPT_BOOL(0, "include-root-refs", &include_root_refs,
+			 N_("also include HEAD ref and pseudorefs")),
 		OPT_END(),
 	};
 
refs.h +11 -10
index c5e08db0ff..518b17c748 100644
@@ -1229,7 +1229,8 @@ int repo_migrate_ref_storage_format(struct repository *repo,
  *
  *             // Access information about the current reference:
  *             if (!(iter->flags & REF_ISSYMREF))
- *                     printf("%s is %s\n", iter->refname, oid_to_hex(iter->oid));
+ *                     printf("%s is %s\n", iter->refname,
+ * oid_to_hex(iter->oid));
  *
  *             // If you need to peel the reference:
  *             ref_iterator_peel(iter, &oid);
@@ -1284,10 +1285,11 @@ enum do_for_each_ref_flags {
  * trim that many characters off the beginning of each refname.
  * The output is ordered by refname.
  */
-struct ref_iterator *refs_ref_iterator_begin(
-		struct ref_store *refs,
-		const char *prefix, const char **exclude_patterns,
-		int trim, enum do_for_each_ref_flags flags);
+struct ref_iterator *refs_ref_iterator_begin(struct ref_store *refs,
+					     const char *prefix,
+					     const char **exclude_patterns,
+					     int trim,
+					     enum do_for_each_ref_flags flags);
 
 /*
  * Advance the iterator to the first or next item and return ITER_OK.
@@ -1317,8 +1319,8 @@ int ref_iterator_advance(struct ref_iterator *ref_iterator);
  *
  * Returns 0 on success, a negative error code otherwise.
  */
-int ref_iterator_seek(struct ref_iterator *ref_iterator,
-		      const char *seek, int set_prefix);
+int ref_iterator_seek(struct ref_iterator *ref_iterator, const char *seek,
+		      int set_prefix);
 
 /*
  * If possible, peel the reference currently being viewed by the
@@ -1339,8 +1341,7 @@ void ref_iterator_free(struct ref_iterator *ref_iterator);
  * adapter between the callback style of reference iteration and the
  * iterator style.
  */
-int do_for_each_ref_iterator(struct ref_iterator *iter,
-			     each_ref_fn fn, void *cb_data);
-
+int do_for_each_ref_iterator(struct ref_iterator *iter, each_ref_fn fn,
+			     void *cb_data);
 
 #endif /* REFS_H */
refs/iterator.c +3 -4
index 1f99045d40..2b7f019c3e 100644
@@ -15,8 +15,8 @@ int ref_iterator_advance(struct ref_iterator *ref_iterator)
 	return ref_iterator->vtable->advance(ref_iterator);
 }
 
-int ref_iterator_seek(struct ref_iterator *ref_iterator,
-		      const char *seek, int set_prefix)
+int ref_iterator_seek(struct ref_iterator *ref_iterator, const char *seek,
+		      int set_prefix)
 {
 	return ref_iterator->vtable->seek(ref_iterator, seek, set_prefix);
 }
@@ -57,8 +57,7 @@ static int empty_ref_iterator_advance(struct ref_iterator *ref_iterator UNUSED)
 }
 
 static int empty_ref_iterator_seek(struct ref_iterator *ref_iterator UNUSED,
-				   const char *seek UNUSED,
-				   int set_prefix UNUSED)
+				   const char *seek UNUSED, int set_prefix UNUSED)
 {
 	return 0;
 }
refs/ref-cache.c +2 -1
index 656e6cd9ff..b812520dc7 100644
@@ -525,7 +525,8 @@ static int cache_ref_iterator_seek(struct ref_iterator *ref_iterator,
 				level->dir = dir;
 				level->index = -1;
 			} else {
-				/* reduce the index so the leaf node is iterated over */
+				/* reduce the index so the leaf node is iterated
+				 * over */
 				if (cmp <= 0 && !slash)
 					level->index = idx - 1;
 				/*
#14 lore raw
Quoted 11 lines · Junio C Hamano

Offtopic. After applying this topic, I asked clang-format if it wants to change anything.

    $ git clang-format --diff $(git merge-base HEAD master)

The result was disasterous. Can "clang-format --diff" mode be taught a bit more focused to avoid touching existing entries in the same array (in this case opts[] that has tons of options for the "git for-each-ref" command), when only one new entry was added, I wonder?

I couldn't find any way to do something like this.

Quoted 11 lines

Also I am not impressed by the change it made to the code that is commented out (in refs.h).

Line wrapping it did to refs_ref_iterator_begin() is an improvement, but those to ref_iterator_seek() and do_for_each_ref_iterator() are unnecessary (both of these were more readble in the original).

Even though I found its output better for Toon's "last-modified" changes, I am not impressed by what clang-format suggested for this series.

It indeed looks really bad, I had a go with the new changes from 'gitster/kn/clang-format-updates'. Which seems a lot better.

However, this does show a problem with using 'RemoveBracesLLVM', where it formats the following:

  if (...) {
     ...
     ...
  } else {
     ...
  }

to:

  if (...) {
     ...
     ...
  } else
     ...

Which isn't our style, I think we should completely drop this too, from my patch series. Let me go ahead and do that. I really want to strip out as many rules as possible to make the number of false positives 0 so we can actually start enforcing clang-format. Once we enforce it, we can slowly see what additional rules work well for us.

builtin/for-each-ref.c +6 -7
index 543013cd11..2ec96eff74 100644
@@ -8,7 +8,7 @@
 #include "strbuf.h"
 #include "strvec.h"
 
-static char const * const for_each_ref_usage[] = {
+static char const *const for_each_ref_usage[] = {
 	N_("git for-each-ref [<options>] [<pattern>]"),
 	N_("git for-each-ref [--points-at <object>]"),
 	N_("git for-each-ref [--merged [<commit>]] [--no-merged [<commit>]]"),
@@ -33,19 +33,19 @@ int cmd_for_each_ref(int argc,
 	struct option opts[] = {
 		OPT_BIT('s', "shell", &format.quote_style,
 			N_("quote placeholders suitably for shells"), QUOTE_SHELL),
-		OPT_BIT('p', "perl",  &format.quote_style,
+		OPT_BIT('p', "perl", &format.quote_style,
 			N_("quote placeholders suitably for perl"), QUOTE_PERL),
-		OPT_BIT(0 , "python", &format.quote_style,
+		OPT_BIT(0, "python", &format.quote_style,
 			N_("quote placeholders suitably for python"), QUOTE_PYTHON),
-		OPT_BIT(0 , "tcl",  &format.quote_style,
+		OPT_BIT(0, "tcl", &format.quote_style,
 			N_("quote placeholders suitably for Tcl"), QUOTE_TCL),
-		OPT_BOOL(0, "omit-empty",  &format.array_opts.omit_empty,
-			N_("do not output a newline after empty formatted refs")),
+		OPT_BOOL(0, "omit-empty", &format.array_opts.omit_empty,
+			 N_("do not output a newline after empty formatted refs")),
 
 		OPT_GROUP(""),
-		OPT_INTEGER( 0 , "count", &format.array_opts.max_count, N_("show

only <n> matched refs")),
- OPT_STRING( 0 , "format", &format.format, N_("format"), N_("format
to use for the output")),
- OPT_STRING( 0 , "skip-until", &filter.seek, N_("skip-until"),
N_("skip references until")),
+ OPT_INTEGER(0, "count", &format.array_opts.max_count, N_("show only
<n> matched refs")),
+ OPT_STRING(0, "format", &format.format, N_("format"), N_("format to
use for the output")),
+ OPT_STRING(0, "skip-until", &filter.seek, N_("skip-until"),
N_("skip references until")),
OPT__COLOR(&format.use_color, N_("respect format colors")), OPT_REF_FILTER_EXCLUDE(&filter), OPT_REF_SORT(&sorting_options),

refs.c +0 -0
index a4220d3537..d492e1b423 100644
@@ -2669,23 +2669,21 @@ enum ref_transaction_error

refs_verify_refnames_available(struct ref_store *refs
if (!iter) {
iter = refs_ref_iterator_begin(refs, dirname.buf, NULL, 0,
DO_FOR_EACH_INCLUDE_BROKEN);
- } else if (ref_iterator_seek(iter, dirname.buf, 1) < 0) {
- goto cleanup;
- }
+ else if (ref_iterator_seek(iter, dirname.buf, 1) < 0) goto cleanup;

- while ((ok = ref_iterator_advance(iter)) == ITER_OK) {
- if (skip &&
- string_list_has_string(skip, iter->refname))
- continue;
+ while ((ok = ref_iterator_advance(iter)) == ITER_OK) {
+ if (skip &&
+ string_list_has_string(skip, iter->refname))
+ continue;

- if (transaction && ref_transaction_maybe_set_rejected(
- transaction, *update_idx,
- REF_TRANSACTION_ERROR_NAME_CONFLICT))
- continue;
+ if (transaction && ref_transaction_maybe_set_rejected(
+ transaction, *update_idx,
+ REF_TRANSACTION_ERROR_NAME_CONFLICT))
+ continue;

- strbuf_addf(err, _("'%s' exists; cannot create '%s'"),
- iter->refname, refname);
- goto cleanup;
+ strbuf_addf(err, _("'%s' exists; cannot create '%s'"),
+ iter->refname, refname);
+ goto cleanup;
}

if (ok != ITER_DONE)

refs.h +3 -4
index c5e08db0ff..41fe96d688 100644
@@ -1285,9 +1285,9 @@ enum do_for_each_ref_flags {
  * The output is ordered by refname.
  */
 struct ref_iterator *refs_ref_iterator_begin(
-		struct ref_store *refs,
-		const char *prefix, const char **exclude_patterns,
-		int trim, enum do_for_each_ref_flags flags);
+	struct ref_store *refs,
+	const char *prefix, const char **exclude_patterns,
+	int trim, enum do_for_each_ref_flags flags);
 
 /*
  * Advance the iterator to the first or next item and return ITER_OK.
@@ -1342,5 +1342,4 @@ void ref_iterator_free(struct ref_iterator *ref_iterator);
 int do_for_each_ref_iterator(struct ref_iterator *iter,
 			     each_ref_fn fn, void *cb_data);
 
-
 #endif /* REFS_H */
#15 lore raw
Quoted 49 lines · Karthik Nayak
Quoted 11 lines · Junio C Hamano

Offtopic. After applying this topic, I asked clang-format if it wants to change anything.

    $ git clang-format --diff $(git merge-base HEAD master)

The result was disasterous. Can "clang-format --diff" mode be taught a bit more focused to avoid touching existing entries in the same array (in this case opts[] that has tons of options for the "git for-each-ref" command), when only one new entry was added, I wonder?

I couldn't find any way to do something like this.

Quoted 11 lines

Also I am not impressed by the change it made to the code that is commented out (in refs.h).

Line wrapping it did to refs_ref_iterator_begin() is an improvement, but those to ref_iterator_seek() and do_for_each_ref_iterator() are unnecessary (both of these were more readble in the original).

Even though I found its output better for Toon's "last-modified" changes, I am not impressed by what clang-format suggested for this series.

It indeed looks really bad, I had a go with the new changes from 'gitster/kn/clang-format-updates'. Which seems a lot better.

However, this does show a problem with using 'RemoveBracesLLVM', where it formats the following:

  if (...) {
     ...
     ...
  } else {
     ...
  }

to:

  if (...) {
     ...
     ...
  } else
     ...
Quoted 6 lines

Which isn't our style, I think we should completely drop this too, from my patch series. Let me go ahead and do that. I really want to strip out as many rules as possible to make the number of false positives 0 so we can actually start enforcing clang-format. Once we enforce it, we can slowly see what additional rules work well for us.

I did some more testing here, and it seems like this was because this particular instance was more like

   if (...) {
      ...
   } else {
      ...
   }

Where both the clauses had single line statement, but we only modified the 'else' part of the clause in this patch series, so clang-format, only suggested removing the braces from the 'else' clause.

So all is good here, I think we can go ahead with the 'gitster/kn/clang-format-updates' and merge it to 'next'. Sorry for being the false positive, I thought I missed testing a particular case and the series.

#16 lore raw
In reply to Karthik Nayak

Hi Karthik

Quoted 5 lines · Karthik Nayak

This enables efficient pagination workflows like:

     git for-each-ref --count=100
     git for-each-ref --count=100 --skip-until=refs/heads/branch-100
     git for-each-ref --count=100 --skip-until=refs/heads/branch-200

Doesn't that require you to know the name of the ref after the last one returned by the previous batch? If the use case here is pagination then being able to provide a numeric offset might be a better fit. For example

	git for-each-ref --count=100 --start=200

would show refs 200 to 300

Thanks

Phillip

#17 lore raw

Phillip Wood <[email protected]> writes:

Hello Phillip,

Quoted 4 lines

Hi Karthik

Quoted 1 line · Karthik Nayak
Quoted 9 lines
Quoted 4 lines

This enables efficient pagination workflows like:

     git for-each-ref --count=100
     git for-each-ref --count=100 --skip-until=refs/heads/branch-100
     git for-each-ref --count=100 --skip-until=refs/heads/branch-200

Doesn't that require you to know the name of the ref after the last one returned by the previous batch? If the use case here is pagination then being able to provide a numeric offset might be a better fit. For example

It does require that you know the last ref from the previous batch.

The reason for picking a reference offset is mostly for performance optimization. Our reference backends are built with prefix matching in mind, in short they do a binary search through the reference namespace to find the required prefix. By using a reference offset we can utilize this binary search mechanism to arrive at offset.

Using a count offset would require iteration to reach the desired offset (basically a O(N) operation). This wouldn't really matter in repositories with ~10^3 refs, but in larger repositories with around ~10^6 refs this starts to make a large difference.

Quoted 7 lines
	git for-each-ref --count=100 --start=200

would show refs 200 to 300

Thanks

Phillip

#18 lore raw
Quoted 26 lines · Karthik Nayak

Phillip Wood <[email protected]> writes:

Hello Phillip,

Quoted 4 lines

Hi Karthik

Quoted 1 line · Karthik Nayak
Quoted 9 lines
Quoted 4 lines

This enables efficient pagination workflows like:

     git for-each-ref --count=100
     git for-each-ref --count=100 --skip-until=refs/heads/branch-100
     git for-each-ref --count=100 --skip-until=refs/heads/branch-200

Doesn't that require you to know the name of the ref after the last one returned by the previous batch? If the use case here is pagination then being able to provide a numeric offset might be a better fit. For example

It does require that you know the last ref from the previous batch.

The reason for picking a reference offset is mostly for performance optimization. Our reference backends are built with prefix matching in mind, in short they do a binary search through the reference namespace to find the required prefix. By using a reference offset we can utilize this binary search mechanism to arrive at offset.

Quoted 4 lines

Using a count offset would require iteration to reach the desired offset (basically a O(N) operation). This wouldn't really matter in repositories with ~10^3 refs, but in larger repositories with around ~10^6 refs this starts to make a large difference.

Even more importantly though, a numeric offset would be invalidated by a concurrent write in case that write ends up inserting a ref in the range of commits you intend to skip now.

Patrick

#19 lore raw
Quoted 3 lines · Patrick Steinhardt

Even more importantly though, a numeric offset would be invalidated by a concurrent write in case that write ends up inserting a ref in the range of commits you intend to skip now.

That argument cuts both ways, no? You have shown up to some ref which you remember in the last cycle, and then while you are planning to formulate another query with --skip-until naming that ref, somebody removes that ref, then what happens? Or somebody inserts a new ref that sorts earlier than the ref you stopped at the last time.

#20 lore raw
Quoted 10 lines · Junio C Hamano
Quoted 3 lines · Patrick Steinhardt

Even more importantly though, a numeric offset would be invalidated by a concurrent write in case that write ends up inserting a ref in the range of commits you intend to skip now.

That argument cuts both ways, no? You have shown up to some ref which you remember in the last cycle, and then while you are planning to formulate another query with --skip-until naming that ref, somebody removes that ref, then what happens?

This ref was already yielded, and it wouldn't and shouldn't be yielded on the next page. This works as expected with the proposal, as --skip-until does not care whether the value itself actually exists.

Quoted 2 lines

Or somebody inserts a new ref that sorts earlier than the ref you stopped at the last time.

It wouldn't and shouldn't be shown. When I have already yielded all refs up to refs/heads/something, I don't expect to see any ref that sorts before refs/heads/something on the next page.

Patrick

#21 lore raw
Quoted 7 lines · Patrick Steinhardt
Quoted 6 lines · Junio C Hamano
Quoted 3 lines · Patrick Steinhardt

Even more importantly though, a numeric offset would be invalidated by a concurrent write in case that write ends up inserting a ref in the range of commits you intend to skip now.

Quoted 8 lines
Quoted 4 lines

That argument cuts both ways, no? You have shown up to some ref which you remember in the last cycle, and then while you are planning to formulate another query with --skip-until naming that ref, somebody removes that ref, then what happens?

This ref was already yielded, and it wouldn't and shouldn't be yielded on the next page. This works as expected with the proposal, as --skip-until does not care whether the value itself actually exists.

The current version of the series will include the reference provided to '--skip-until', if it exists. But your latter statement still holds, if the reference doesn't exist, it will still work by finding the next reference in the default sort order.

Quoted 7 lines
Quoted 2 lines

Or somebody inserts a new ref that sorts earlier than the ref you stopped at the last time.

It wouldn't and shouldn't be shown. When I have already yielded all refs up to refs/heads/something, I don't expect to see any ref that sorts before refs/heads/something on the next page.

Yeah, this was my thought too. Another way to think of this is that in a cursor based approach, a particular reference is guarateed never to occur again, even with modifications to the repository made between requests. However in a count based approach this doesn't stand.

Quoted 1 line

Patrick

#22 lore raw

v2 cover for-each-ref: introduce seeking functionality via '--skip-until'

In reply to Karthik Nayak

The git-for-each-ref(1) command is used to iterate over references present in a repository. In large repositories with millions of references, it would be optimal to paginate this output such that we can start iteration from a given reference. This would avoid having to iterate over all references from the beginning each time when paginating through results.

This series adds a '--skip-until' option in 'git-for-each-ref(1)'. When used, the reference iteration seeks to the first matching reference and iterates from there onward.

This enables efficient pagination workflows like:

    git for-each-ref --count=100
    git for-each-ref --count=100 --skip-until=refs/heads/branch-100
    git for-each-ref --count=100 --skip-until=refs/heads/branch-200

To add this functionality, we expose the ref_iterator outside the 'refs/' namespace and modify the ref_iterator_seek() to actually seek to a given reference and only set the prefix when the set_prefix field is set.

On the reftable and packed backend, the changes are simple. But since the files backend uses 'ref-cache' for reference handling, the changes there are a little more involved, since we need to setup the right levels and the indexing.

Initially I was also planning to cleanup all the refs_for_each...() functions in 'refs.h' by simply using the iterator, but this bloated the series. So I've left that for another day.

Changes in v2:
- Modify 'ref_iterator_seek()' to take in flags instead of a
'set_prefix' variable. This improves readability, where users would
use the 'REF_ITERATOR_SEEK_SET_PREFIX' instead of simply passing '1'.
- When the set prefix flag isn't usage, reset any previously set prefix.
This ensures that the internal prefix state is always reset whenever
we seek and unifies the behavior between 'ref_iterator_seek' and
'ref_iterator_begin'.
- Don't allow '--skip-until' to be run with '--sort', since the seeking
always takes place before any sorting and this can be confusing.
- Some styling fixes:
- Remove extra newline
- Skip braces around single lined if...else clause
- Add braces around 'if' clause
- Fix indentation
- Link to v1: https://lore.kernel.org/git/20250701-306-git-for-each-ref-pagination-v1-0-4f0ae7c0688f@gmail.com/

---
 Documentation/git-for-each-ref.adoc |   6 +-
 builtin/for-each-ref.c              |   8 ++
 ref-filter.c                        |  61 ++++++++----
 ref-filter.h                        |   1 +
 refs.c                              |   6 +-
 refs.h                              | 158 ++++++++++++++++++++++++++++++
 refs/debug.c                        |   7 +-
 refs/files-backend.c                |   7 +-
 refs/iterator.c                     |  26 ++---
 refs/packed-backend.c               |  17 ++--
 refs/ref-cache.c                    |  99 +++++++++++++++----
 refs/ref-cache.h                    |   7 --
 refs/refs-internal.h                | 152 ++---------------------------
 refs/reftable-backend.c             |  21 ++--
 t/t6302-for-each-ref-filter.sh      | 188 ++++++++++++++++++++++++++++++++++++
 15 files changed, 538 insertions(+), 226 deletions(-)

Karthik Nayak (4):
refs: expose ref_iterator via 'refs.h'
ref-cache: remove unused function 'find_ref_entry()'
refs: selectively set prefix in the seek functions
for-each-ref: introduce a '--skip-until' option

Range-diff versus v1:

1: a20ef2c841 ! 1: d19553c365 refs: expose ref_iterator via 'refs.h'
@@ refs.h: int repo_migrate_ref_storage_format(struct repository *repo,
+ * The output is ordered by refname.
+ */
+struct ref_iterator *refs_ref_iterator_begin(
-+ struct ref_store *refs,
-+ const char *prefix, const char **exclude_patterns,
-+ int trim, enum do_for_each_ref_flags flags);
++ struct ref_store *refs,
++ const char *prefix, const char **exclude_patterns,
++ int trim, enum do_for_each_ref_flags flags);
+
+/*
+ * Advance the iterator to the first or next item and return ITER_OK.
@@ refs.h: int repo_migrate_ref_storage_format(struct repository *repo,
+ */
+int do_for_each_ref_iterator(struct ref_iterator *iter,
+ each_ref_fn fn, void *cb_data);
-+
+
#endif /* REFS_H */

2:  96f3e6eb05 = 2:  1d3936132b ref-cache: remove unused function 'find_ref_entry()'
3:  fa19b53a37 ! 3:  aab2011494 refs: selectively set prefix in the seek functions
    @@ Commit message
         further iteration would only yield references which match the particular
         prefix. This is a bit confusing.
     
    -    Let's add a 'set_prefix' field to the function, which when set, will set
    -    the prefix for the iteration in-line with the existing behavior. But
    -    when the 'set_prefix' field is not set, the reference backends will
    -    simply seek to the specified reference without setting prefix. This
    -    allows users to start iteration from a specific reference.
    +    Let's add a 'flags' field to the function, which when set with the
    +    'REF_ITERATOR_SEEK_SET_PREFIX' flag, will set the prefix for the
    +    iteration in-line with the existing behavior. Otherwise, the reference
    +    backends will simply seek to the specified reference and clears any
    +    previously set prefix. This allows users to start iteration from a
    +    specific reference.
     
         In the packed and reftable backend, since references are available in a
         sorted list, the changes are simply setting the prefix if needed. The
         changes on the files-backend are a little more involved, since the files
         backend uses the 'ref-cache' mechanism. We move out the existing logic
         within `cache_ref_iterator_seek()` to `cache_ref_iterator_set_prefix()`
    -    which is called when `set_prefix` is set. We then parse the provided
    -    seek string and set the required levels and their indexes to ensure that
    -    seeking is possible.
    +    which is called when the 'REF_ITERATOR_SEEK_SET_PREFIX' flag is set. We
    +    then parse the provided seek string and set the required levels and
    +    their indexes to ensure that seeking is possible.
     
    +    Helped-by: Patrick Steinhardt <[email protected]>
         Signed-off-by: Karthik Nayak <[email protected]>
     
      ## refs.c ##
     @@ refs.c: enum ref_transaction_error refs_verify_refnames_available(struct ref_store *refs
    - 			if (!iter) {
    + 		if (!initial_transaction) {
    + 			int ok;
    + 
    +-			if (!iter) {
    ++			if (!iter)
      				iter = refs_ref_iterator_begin(refs, dirname.buf, NULL, 0,
      							       DO_FOR_EACH_INCLUDE_BROKEN);
     -			} else if (ref_iterator_seek(iter, dirname.buf) < 0) {
    -+			} else if (ref_iterator_seek(iter, dirname.buf, 1) < 0) {
    ++			else if (ref_iterator_seek(iter, dirname.buf,
    ++						   REF_ITERATOR_SEEK_SET_PREFIX) < 0)
      				goto cleanup;
    - 			}
    +-			}
      
    + 			while ((ok = ref_iterator_advance(iter)) == ITER_OK) {
    + 				if (skip &&
     
      ## refs.h ##
     @@ refs.h: struct ref_iterator *refs_ref_iterator_begin(
    +  */
      int ref_iterator_advance(struct ref_iterator *ref_iterator);
      
    ++enum ref_iterator_seek_flag {
    ++	/*
    ++	 * Also set the seek pattern as a prefix for iteration. This ensures
    ++	 * that only references which match the prefix are yielded.
    ++	 */
    ++	REF_ITERATOR_SEEK_SET_PREFIX = (1 << 0),
    ++};
    ++
      /*
     - * Seek the iterator to the first reference with the given prefix.
     - * The prefix is matched as a literal string, without regard for path
    +- * separators. If prefix is NULL or the empty string, seek the iterator to the
     + * Seek the iterator to the first reference matching the given seek string.
     + * The seek string is matched as a literal string, without regard for path
    -  * separators. If prefix is NULL or the empty string, seek the iterator to the
    ++ * separators. If seek is NULL or the empty string, seek the iterator to the
       * first reference again.
       *
     - * This function is expected to behave as if a new ref iterator with the same
     - * prefix had been created, but allows reuse of iterators and thus may allow
     - * the backend to optimize. Parameters other than the prefix that have been
     - * passed when creating the iterator will remain unchanged.
    -+ * When set_prefix is true, this function behaves as if a new ref iterator
    -+ * with the same prefix had been created, setting the prefix for subsequent
    -+ * iteration. When set_prefix is false, the iterator simply seeks to the
    -+ * specified reference without changing the existing prefix, allowing
    -+ * iteration to start from that specific reference.
    ++ * This function is expected to behave as if a new ref iterator has been
    ++ * created, but allows reuse of existing iterators for optimization.
     + *
    -+ * This function allows reuse of iterators and thus may allow the backend
    -+ * to optimize. Parameters other than the prefix that have been passed when
    -+ * creating the iterator will remain unchanged.
    ++ * When the REF_ITERATOR_SEEK_SET_PREFIX flag is set, the iterator's prefix is
    ++ * updated to match the seek string, affecting all subsequent iterations. If
    ++ * not, the iterator seeks to the specified reference and clears any previously
    ++ * set prefix.
       *
       * Returns 0 on success, a negative error code otherwise.
       */
    - int ref_iterator_seek(struct ref_iterator *ref_iterator,
    +-int ref_iterator_seek(struct ref_iterator *ref_iterator,
     -		      const char *prefix);
    -+		      const char *seek, int set_prefix);
    ++int ref_iterator_seek(struct ref_iterator *ref_iterator, const char *seek,
    ++		      unsigned int flags);
      
      /*
       * If possible, peel the reference currently being viewed by the
    @@ refs/debug.c: static int debug_ref_iterator_advance(struct ref_iterator *ref_ite
      
      static int debug_ref_iterator_seek(struct ref_iterator *ref_iterator,
     -				   const char *prefix)
    -+				   const char *seek, int set_prefix)
    ++				   const char *seek, unsigned int flags)
      {
      	struct debug_ref_iterator *diter =
      		(struct debug_ref_iterator *)ref_iterator;
     -	int res = diter->iter->vtable->seek(diter->iter, prefix);
     -	trace_printf_key(&trace_refs, "iterator_seek: %s: %d\n", prefix ? prefix : "", res);
    -+	int res = diter->iter->vtable->seek(diter->iter, seek, set_prefix);
    -+	trace_printf_key(&trace_refs, "iterator_seek: %s set_prefix: %d: %d\n",
    -+			 seek ? seek : "", set_prefix, res);
    ++	int res = diter->iter->vtable->seek(diter->iter, seek, flags);
    ++	trace_printf_key(&trace_refs, "iterator_seek: %s flags: %d: %d\n",
    ++			 seek ? seek : "", flags, res);
      	return res;
      }
      
    @@ refs/files-backend.c: static int files_ref_iterator_advance(struct ref_iterator
      
      static int files_ref_iterator_seek(struct ref_iterator *ref_iterator,
     -				   const char *prefix)
    -+				   const char *seek, int set_prefix)
    ++				   const char *seek, unsigned int flags)
      {
      	struct files_ref_iterator *iter =
      		(struct files_ref_iterator *)ref_iterator;
     -	return ref_iterator_seek(iter->iter0, prefix);
    -+	return ref_iterator_seek(iter->iter0, seek, set_prefix);
    ++	return ref_iterator_seek(iter->iter0, seek, flags);
      }
      
      static int files_ref_iterator_peel(struct ref_iterator *ref_iterator,
    @@ refs/files-backend.c: static int files_reflog_iterator_advance(struct ref_iterat
      static int files_reflog_iterator_seek(struct ref_iterator *ref_iterator UNUSED,
     -				      const char *prefix UNUSED)
     +				      const char *seek UNUSED,
    -+				      int set_prefix UNUSED)
    ++				      unsigned int flags UNUSED)
      {
      	BUG("ref_iterator_seek() called for reflog_iterator");
      }
     
      ## refs/iterator.c ##
     @@ refs/iterator.c: int ref_iterator_advance(struct ref_iterator *ref_iterator)
    + 	return ref_iterator->vtable->advance(ref_iterator);
      }
      
    - int ref_iterator_seek(struct ref_iterator *ref_iterator,
    +-int ref_iterator_seek(struct ref_iterator *ref_iterator,
     -		      const char *prefix)
    -+		      const char *seek, int set_prefix)
    ++int ref_iterator_seek(struct ref_iterator *ref_iterator, const char *seek,
    ++		      unsigned int flags)
      {
     -	return ref_iterator->vtable->seek(ref_iterator, prefix);
    -+	return ref_iterator->vtable->seek(ref_iterator, seek, set_prefix);
    ++	return ref_iterator->vtable->seek(ref_iterator, seek, flags);
      }
      
      int ref_iterator_peel(struct ref_iterator *ref_iterator,
    @@ refs/iterator.c: static int empty_ref_iterator_advance(struct ref_iterator *ref_
      static int empty_ref_iterator_seek(struct ref_iterator *ref_iterator UNUSED,
     -				   const char *prefix UNUSED)
     +				   const char *seek UNUSED,
    -+				   int set_prefix UNUSED)
    ++				   unsigned int flags UNUSED)
      {
      	return 0;
      }
    @@ refs/iterator.c: static int merge_ref_iterator_advance(struct ref_iterator *ref_
      
      static int merge_ref_iterator_seek(struct ref_iterator *ref_iterator,
     -				   const char *prefix)
    -+				   const char *seek, int set_prefix)
    ++				   const char *seek, unsigned int flags)
      {
      	struct merge_ref_iterator *iter =
      		(struct merge_ref_iterator *)ref_iterator;
    @@ refs/iterator.c: static int merge_ref_iterator_seek(struct ref_iterator *ref_ite
      	iter->iter1 = iter->iter1_owned;
      
     -	ret = ref_iterator_seek(iter->iter0, prefix);
    -+	ret = ref_iterator_seek(iter->iter0, seek, set_prefix);
    ++	ret = ref_iterator_seek(iter->iter0, seek, flags);
      	if (ret < 0)
      		return ret;
      
     -	ret = ref_iterator_seek(iter->iter1, prefix);
    -+	ret = ref_iterator_seek(iter->iter1, seek, set_prefix);
    ++	ret = ref_iterator_seek(iter->iter1, seek, flags);
      	if (ret < 0)
      		return ret;
      
    @@ refs/iterator.c: static int prefix_ref_iterator_advance(struct ref_iterator *ref
      
      static int prefix_ref_iterator_seek(struct ref_iterator *ref_iterator,
     -				    const char *prefix)
    -+				    const char *seek, int set_prefix)
    ++				    const char *seek, unsigned int flags)
      {
      	struct prefix_ref_iterator *iter =
      		(struct prefix_ref_iterator *)ref_iterator;
    @@ refs/iterator.c: static int prefix_ref_iterator_advance(struct ref_iterator *ref
     -	iter->prefix = xstrdup_or_null(prefix);
     -	return ref_iterator_seek(iter->iter0, prefix);
     +
    -+	if (set_prefix) {
    ++	if (flags & REF_ITERATOR_SEEK_SET_PREFIX) {
     +		free(iter->prefix);
     +		iter->prefix = xstrdup_or_null(seek);
     +	}
    -+	return ref_iterator_seek(iter->iter0, seek, set_prefix);
    ++	return ref_iterator_seek(iter->iter0, seek, flags);
      }
      
      static int prefix_ref_iterator_peel(struct ref_iterator *ref_iterator,
    @@ refs/packed-backend.c: static int packed_ref_iterator_advance(struct ref_iterato
      
      static int packed_ref_iterator_seek(struct ref_iterator *ref_iterator,
     -				    const char *prefix)
    -+				    const char *seek, int set_prefix)
    ++				    const char *seek, unsigned int flags)
      {
      	struct packed_ref_iterator *iter =
      		(struct packed_ref_iterator *)ref_iterator;
    @@ refs/packed-backend.c: static int packed_ref_iterator_advance(struct ref_iterato
      
     -	free(iter->prefix);
     -	iter->prefix = xstrdup_or_null(prefix);
    -+	if (set_prefix) {
    -+		free(iter->prefix);
    ++	/* Unset any previously set prefix */
    ++	FREE_AND_NULL(iter->prefix);
    ++
    ++	if (flags & REF_ITERATOR_SEEK_SET_PREFIX)
     +		iter->prefix = xstrdup_or_null(seek);
    -+	}
     +
      	iter->pos = start;
      	iter->eof = iter->snapshot->eof;
    @@ refs/packed-backend.c: static struct ref_iterator *packed_ref_iterator_begin(
      	iter->flags = flags;
      
     -	if (packed_ref_iterator_seek(&iter->base, prefix) < 0) {
    -+	if (packed_ref_iterator_seek(&iter->base, prefix, 1) < 0) {
    ++	if (packed_ref_iterator_seek(&iter->base, prefix,
    ++				     REF_ITERATOR_SEEK_SET_PREFIX) < 0) {
      		ref_iterator_free(&iter->base);
      		return NULL;
      	}
    @@ refs/ref-cache.c: static int cache_ref_iterator_seek(struct ref_iterator *ref_it
      }
      
     +static int cache_ref_iterator_seek(struct ref_iterator *ref_iterator,
    -+				   const char *seek, int set_prefix)
    ++				   const char *seek, unsigned int flags)
     +{
     +	struct cache_ref_iterator *iter =
     +		(struct cache_ref_iterator *)ref_iterator;
     +
    -+	if (set_prefix) {
    ++	if (flags & REF_ITERATOR_SEEK_SET_PREFIX) {
     +		return cache_ref_iterator_set_prefix(iter, seek);
     +	} else if (seek && *seek) {
     +		struct cache_ref_iterator_level *level;
    @@ refs/ref-cache.c: static int cache_ref_iterator_seek(struct ref_iterator *ref_it
     +		level->index = -1;
     +		level->dir = dir;
     +
    ++		/* Unset any previously set prefix */
    ++		FREE_AND_NULL(iter->prefix);
    ++
     +		/*
     +		 * Breakdown the provided seek path and assign the correct
     +		 * indexing to each level as needed.
    @@ refs/ref-cache.c: struct ref_iterator *cache_ref_iterator_begin(struct ref_cache
      	iter->prime_dir = prime_dir;
      
     -	if (cache_ref_iterator_seek(&iter->base, prefix) < 0) {
    -+	if (cache_ref_iterator_seek(&iter->base, prefix, 1) < 0) {
    ++	if (cache_ref_iterator_seek(&iter->base, prefix,
    ++				    REF_ITERATOR_SEEK_SET_PREFIX) < 0) {
      		ref_iterator_free(&iter->base);
      		return NULL;
      	}
    @@ refs/refs-internal.h: void base_ref_iterator_init(struct ref_iterator *iter,
       */
      typedef int ref_iterator_seek_fn(struct ref_iterator *ref_iterator,
     -				 const char *prefix);
    -+				 const char *seek, int set_prefix);
    ++				 const char *seek, unsigned int flags);
      
      /*
       * Peels the current ref, returning 0 for success or -1 for failure.
    @@ refs/reftable-backend.c: static int reftable_ref_iterator_advance(struct ref_ite
      
      static int reftable_ref_iterator_seek(struct ref_iterator *ref_iterator,
     -				      const char *prefix)
    -+				      const char *seek, int set_prefix)
    ++				      const char *seek, unsigned int flags)
      {
      	struct reftable_ref_iterator *iter =
      		(struct reftable_ref_iterator *)ref_iterator;
    @@ refs/reftable-backend.c: static int reftable_ref_iterator_advance(struct ref_ite
     -	iter->prefix = xstrdup_or_null(prefix);
     -	iter->prefix_len = prefix ? strlen(prefix) : 0;
     -	iter->err = reftable_iterator_seek_ref(&iter->iter, prefix);
    -+	if (set_prefix) {
    -+		free(iter->prefix);
    ++	/* Unset any previously set prefix */
    ++	FREE_AND_NULL(iter->prefix);
    ++	iter->prefix_len = 0;
    ++
    ++	if (flags & REF_ITERATOR_SEEK_SET_PREFIX) {
     +		iter->prefix = xstrdup_or_null(seek);
     +		iter->prefix_len = seek ? strlen(seek) : 0;
     +	}
    @@ refs/reftable-backend.c: static struct reftable_ref_iterator *ref_iterator_for_s
      		goto done;
      
     -	ret = reftable_ref_iterator_seek(&iter->base, prefix);
    -+	ret = reftable_ref_iterator_seek(&iter->base, prefix, 1);
    ++	ret = reftable_ref_iterator_seek(&iter->base, prefix,
    ++					 REF_ITERATOR_SEEK_SET_PREFIX);
      	if (ret)
      		goto done;
      
    @@ refs/reftable-backend.c: static int reftable_reflog_iterator_advance(struct ref_
      static int reftable_reflog_iterator_seek(struct ref_iterator *ref_iterator UNUSED,
     -					 const char *prefix UNUSED)
     +					 const char *seek UNUSED,
    -+					 int set_prefix UNUSED)
    ++					 unsigned int flags UNUSED)
      {
      	BUG("reftable reflog iterator cannot be seeked");
      	return -1;
4:  0cfc879a93 ! 4:  dc6ffa9002 for-each-ref: introduce a '--skip-until' option
    @@ Documentation/git-for-each-ref.adoc: TAB %(refname)`.
      	List root refs (HEAD and pseudorefs) apart from regular refs.
      
     +--skip-until::
    -+    Skip references up to the specified pattern. Cannot be used with
    -+    general pattern matching.
    ++    Skip references up to but excluding the specified pattern. Cannot be used
    ++    with general pattern matching or custom sort options.
     +
      FIELD NAMES
      -----------
    @@ builtin/for-each-ref.c: int cmd_for_each_ref(int argc,
      		OPT__COLOR(&format.use_color, N_("respect format colors")),
      		OPT_REF_FILTER_EXCLUDE(&filter),
      		OPT_REF_SORT(&sorting_options),
    +@@ builtin/for-each-ref.c: int cmd_for_each_ref(int argc,
    + 	if (verify_ref_format(&format))
    + 		usage_with_options(for_each_ref_usage, opts);
    + 
    ++	if (filter.seek && sorting_options.nr > 1)
    ++		die(_("cannot use --skip-until custom sort options"));
    ++
    + 	sorting = ref_sorting_options(&sorting_options);
    + 	ref_sorting_set_sort_flags_all(sorting, REF_SORTING_ICASE, icase);
    + 	filter.ignore_case = icase;
     @@ builtin/for-each-ref.c: int cmd_for_each_ref(int argc,
      		filter.name_patterns = argv;
      	}
    @@ ref-filter.c: static int for_each_fullref_in_pattern(struct ref_filter *filter,
      
      /*
     @@ ref-filter.c: static int do_filter_refs(struct ref_filter *filter, unsigned int type, each_ref
    - 	if (!filter->kind)
    + 	init_contains_cache(&filter->internal.no_contains_cache);
    + 
    + 	/*  Simple per-ref filtering */
    +-	if (!filter->kind)
    ++	if (!filter->kind) {
      		die("filter_refs: invalid type");
    - 	else {
    +-	else {
    ++	} else {
     +		const char *prefix = NULL;
     +
      		/*
    @@ t/t6302-for-each-ref-filter.sh: test_expect_success 'validate worktree atom' '
     +	cat >expect <<-\EOF &&
     +	fatal: cannot use --skip-until with patterns
     +	EOF
    -+	test_must_fail git for-each-ref --format="%(refname)" --skip-until=refs/odd/spot refs/tags  2>actual &&
    ++	test_must_fail git for-each-ref --format="%(refname)" --skip-until=refs/odd/spot refs/tags 2>actual &&
    ++	test_cmp expect actual
    ++'
    ++
    ++test_expect_success 'skip until used with custom sort order' '
    ++	cat >expect <<-\EOF &&
    ++	fatal: cannot use --skip-until custom sort options
    ++	EOF
    ++	test_must_fail git for-each-ref --format="%(refname)" --skip-until=refs/odd/spot --sort=author 2>actual &&
     +	test_cmp expect actual
     +'
     +

base-commit: cf6f63ea6bf35173e02e18bdc6a4ba41288acff9
change-id: 20250605-306-git-for-each-ref-pagination-0ba8a29ae646

Thanks
- Karthik

#23 lore raw

PATCH v2 1/4 refs: expose `ref_iterator` via 'refs.h'

The ref_iterator is an internal structure to the 'refs/' sub-directory, which allows iteration over refs. All reference iteration is built on top of these iterators.

External clients of the 'refs' subsystem use the various 'refs_for_each...()' functions to iterate over refs. However since these are wrapper functions, each combination of functionality requires a new wrapper function. This is not feasible as the functions pile up with the increase in requirements. Expose the internal reference iterator, so advanced users can mix and match options as needed.

---
 refs.h               | 147 +++++++++++++++++++++++++++++++++++++++++++++++++++
 refs/refs-internal.h | 145 +-------------------------------------------------
 2 files changed, 149 insertions(+), 143 deletions(-)
refs.h +147 -0
index 46a6008e07..7c21aaef3d 100644
@@ -1190,4 +1190,151 @@ int repo_migrate_ref_storage_format(struct repository *repo,
 				    unsigned int flags,
 				    struct strbuf *err);
 
+/*
+ * Reference iterators
+ *
+ * A reference iterator encapsulates the state of an in-progress
+ * iteration over references. Create an instance of `struct
+ * ref_iterator` via one of the functions in this module.
+ *
+ * A freshly-created ref_iterator doesn't yet point at a reference. To
+ * advance the iterator, call ref_iterator_advance(). If successful,
+ * this sets the iterator's refname, oid, and flags fields to describe
+ * the next reference and returns ITER_OK. The data pointed at by
+ * refname and oid belong to the iterator; if you want to retain them
+ * after calling ref_iterator_advance() again or calling
+ * ref_iterator_free(), you must make a copy. When the iteration has
+ * been exhausted, ref_iterator_advance() releases any resources
+ * associated with the iteration, frees the ref_iterator object, and
+ * returns ITER_DONE. If you want to abort the iteration early, call
+ * ref_iterator_free(), which also frees the ref_iterator object and
+ * any associated resources. If there was an internal error advancing
+ * to the next entry, ref_iterator_advance() aborts the iteration,
+ * frees the ref_iterator, and returns ITER_ERROR.
+ *
+ * The reference currently being looked at can be peeled by calling
+ * ref_iterator_peel(). This function is often faster than peel_ref(),
+ * so it should be preferred when iterating over references.
+ *
+ * Putting it all together, a typical iteration looks like this:
+ *
+ *     int ok;
+ *     struct ref_iterator *iter = ...;
+ *
+ *     while ((ok = ref_iterator_advance(iter)) == ITER_OK) {
+ *             if (want_to_stop_iteration()) {
+ *                     ok = ITER_DONE;
+ *                     break;
+ *             }
+ *
+ *             // Access information about the current reference:
+ *             if (!(iter->flags & REF_ISSYMREF))
+ *                     printf("%s is %s\n", iter->refname, oid_to_hex(iter->oid));
+ *
+ *             // If you need to peel the reference:
+ *             ref_iterator_peel(iter, &oid);
+ *     }
+ *
+ *     if (ok != ITER_DONE)
+ *             handle_error();
+ *     ref_iterator_free(iter);
+ */
+struct ref_iterator;
+
+/*
+ * These flags are passed to refs_ref_iterator_begin() (and do_for_each_ref(),
+ * which feeds it).
+ */
+enum do_for_each_ref_flags {
+	/*
+	 * Include broken references in a do_for_each_ref*() iteration, which
+	 * would normally be omitted. This includes both refs that point to
+	 * missing objects (a true repository corruption), ones with illegal
+	 * names (which we prefer not to expose to callers), as well as
+	 * dangling symbolic refs (i.e., those that point to a non-existent
+	 * ref; this is not a corruption, but as they have no valid oid, we
+	 * omit them from normal iteration results).
+	 */
+	DO_FOR_EACH_INCLUDE_BROKEN = (1 << 0),
+
+	/*
+	 * Only include per-worktree refs in a do_for_each_ref*() iteration.
+	 * Normally this will be used with a files ref_store, since that's
+	 * where all reference backends will presumably store their
+	 * per-worktree refs.
+	 */
+	DO_FOR_EACH_PER_WORKTREE_ONLY = (1 << 1),
+
+	/*
+	 * Omit dangling symrefs from output; this only has an effect with
+	 * INCLUDE_BROKEN, since they are otherwise not included at all.
+	 */
+	DO_FOR_EACH_OMIT_DANGLING_SYMREFS = (1 << 2),
+
+	/*
+	 * Include root refs i.e. HEAD and pseudorefs along with the regular
+	 * refs.
+	 */
+	DO_FOR_EACH_INCLUDE_ROOT_REFS = (1 << 3),
+};
+
+/*
+ * Return an iterator that goes over each reference in `refs` for
+ * which the refname begins with prefix. If trim is non-zero, then
+ * trim that many characters off the beginning of each refname.
+ * The output is ordered by refname.
+ */
+struct ref_iterator *refs_ref_iterator_begin(
+	struct ref_store *refs,
+	const char *prefix, const char **exclude_patterns,
+	int trim, enum do_for_each_ref_flags flags);
+
+/*
+ * Advance the iterator to the first or next item and return ITER_OK.
+ * If the iteration is exhausted, free the resources associated with
+ * the ref_iterator and return ITER_DONE. On errors, free the iterator
+ * resources and return ITER_ERROR. It is a bug to use ref_iterator or
+ * call this function again after it has returned ITER_DONE or
+ * ITER_ERROR.
+ */
+int ref_iterator_advance(struct ref_iterator *ref_iterator);
+
+/*
+ * Seek the iterator to the first reference with the given prefix.
+ * The prefix is matched as a literal string, without regard for path
+ * separators. If prefix is NULL or the empty string, seek the iterator to the
+ * first reference again.
+ *
+ * This function is expected to behave as if a new ref iterator with the same
+ * prefix had been created, but allows reuse of iterators and thus may allow
+ * the backend to optimize. Parameters other than the prefix that have been
+ * passed when creating the iterator will remain unchanged.
+ *
+ * Returns 0 on success, a negative error code otherwise.
+ */
+int ref_iterator_seek(struct ref_iterator *ref_iterator,
+		      const char *prefix);
+
+/*
+ * If possible, peel the reference currently being viewed by the
+ * iterator. Return 0 on success.
+ */
+int ref_iterator_peel(struct ref_iterator *ref_iterator,
+		      struct object_id *peeled);
+
+/* Free the reference iterator and any associated resources. */
+void ref_iterator_free(struct ref_iterator *ref_iterator);
+
+/*
+ * The common backend for the for_each_*ref* functions. Call fn for
+ * each reference in iter. If the iterator itself ever returns
+ * ITER_ERROR, return -1. If fn ever returns a non-zero value, stop
+ * the iteration and return that value. Otherwise, return 0. In any
+ * case, free the iterator when done. This function is basically an
+ * adapter between the callback style of reference iteration and the
+ * iterator style.
+ */
+int do_for_each_ref_iterator(struct ref_iterator *iter,
+			     each_ref_fn fn, void *cb_data);
+
 #endif /* REFS_H */
refs/refs-internal.h +2 -143

169 lines not on this page · raw message

Signature · 1 line
2.49.0
#24 lore raw

PATCH v2 2/4 ref-cache: remove unused function 'find_ref_entry()'

In reply to Karthik Nayak

The 'find_ref_entry' function is no longer used, so remove it.

---
 refs/ref-cache.c | 14 --------------
 refs/ref-cache.h |  7 -------
 2 files changed, 21 deletions(-)
refs/ref-cache.c +0 -14
index c1f1bab1d5..8aaffa8c6b 100644
@@ -194,20 +194,6 @@ static struct ref_dir *find_containing_dir(struct ref_dir *dir,
 	return dir;
 }
 
-struct ref_entry *find_ref_entry(struct ref_dir *dir, const char *refname)
-{
-	int entry_index;
-	struct ref_entry *entry;
-	dir = find_containing_dir(dir, refname);
-	if (!dir)
-		return NULL;
-	entry_index = search_ref_dir(dir, refname, strlen(refname));
-	if (entry_index == -1)
-		return NULL;
-	entry = dir->entries[entry_index];
-	return (entry->flag & REF_DIR) ? NULL : entry;
-}
-
 /*
  * Emit a warning and return true iff ref1 and ref2 have the same name
  * and the same oid. Die if they have the same name but different
refs/ref-cache.h +0 -7
index 5f04e518c3..f635d2d824 100644
@@ -201,13 +201,6 @@ void free_ref_cache(struct ref_cache *cache);
  */
 void add_entry_to_dir(struct ref_dir *dir, struct ref_entry *entry);
 
-/*
- * Find the value entry with the given name in dir, sorting ref_dirs
- * and recursing into subdirectories as necessary.  If the name is not
- * found or it corresponds to a directory entry, return NULL.
- */
-struct ref_entry *find_ref_entry(struct ref_dir *dir, const char *refname);
-
 /*
  * Start iterating over references in `cache`. If `prefix` is
  * specified, only include references whose names start with that
Signature · 1 line
2.49.0
#25 lore raw

PATCH v2 3/4 refs: selectively set prefix in the seek functions

In reply to Karthik Nayak

The ref iterator exposes a ref_iterator_seek() function. The name suggests that this would seek the iterator to a specific reference in some ways similar to how fseek() works for the filesystem.

However, the function actually sets the prefix for refs iteration. So further iteration would only yield references which match the particular prefix. This is a bit confusing.

Let's add a 'flags' field to the function, which when set with the 'REF_ITERATOR_SEEK_SET_PREFIX' flag, will set the prefix for the iteration in-line with the existing behavior. Otherwise, the reference backends will simply seek to the specified reference and clears any previously set prefix. This allows users to start iteration from a specific reference.

In the packed and reftable backend, since references are available in a sorted list, the changes are simply setting the prefix if needed. The changes on the files-backend are a little more involved, since the files backend uses the 'ref-cache' mechanism. We move out the existing logic within cache_ref_iterator_seek() to cache_ref_iterator_set_prefix() which is called when the 'REF_ITERATOR_SEEK_SET_PREFIX' flag is set. We then parse the provided seek string and set the required levels and their indexes to ensure that seeking is possible.

---
 refs.c                  |  6 ++--
 refs.h                  | 29 +++++++++++------
 refs/debug.c            |  7 ++--
 refs/files-backend.c    |  7 ++--
 refs/iterator.c         | 26 ++++++++-------
 refs/packed-backend.c   | 17 ++++++----
 refs/ref-cache.c        | 85 ++++++++++++++++++++++++++++++++++++++++++++++---
 refs/refs-internal.h    |  7 ++--
 refs/reftable-backend.c | 21 ++++++++----
 9 files changed, 155 insertions(+), 50 deletions(-)
refs.c +3 -3
index dce5c49ca2..243e6898b8 100644
@@ -2666,12 +2666,12 @@ enum ref_transaction_error refs_verify_refnames_available(struct ref_store *refs
 		if (!initial_transaction) {
 			int ok;
 
-			if (!iter) {
+			if (!iter)
 				iter = refs_ref_iterator_begin(refs, dirname.buf, NULL, 0,
 							       DO_FOR_EACH_INCLUDE_BROKEN);
-			} else if (ref_iterator_seek(iter, dirname.buf) < 0) {
+			else if (ref_iterator_seek(iter, dirname.buf,
+						   REF_ITERATOR_SEEK_SET_PREFIX) < 0)
 				goto cleanup;
-			}
 
 			while ((ok = ref_iterator_advance(iter)) == ITER_OK) {
 				if (skip &&
refs.h +20 -9
index 7c21aaef3d..7852ad36f3 100644
@@ -1299,21 +1299,32 @@ struct ref_iterator *refs_ref_iterator_begin(
  */
 int ref_iterator_advance(struct ref_iterator *ref_iterator);
 
+enum ref_iterator_seek_flag {
+	/*
+	 * Also set the seek pattern as a prefix for iteration. This ensures
+	 * that only references which match the prefix are yielded.
+	 */
+	REF_ITERATOR_SEEK_SET_PREFIX = (1 << 0),
+};
+
 /*
- * Seek the iterator to the first reference with the given prefix.
- * The prefix is matched as a literal string, without regard for path
- * separators. If prefix is NULL or the empty string, seek the iterator to the
+ * Seek the iterator to the first reference matching the given seek string.
+ * The seek string is matched as a literal string, without regard for path
+ * separators. If seek is NULL or the empty string, seek the iterator to the
  * first reference again.
  *
- * This function is expected to behave as if a new ref iterator with the same
- * prefix had been created, but allows reuse of iterators and thus may allow
- * the backend to optimize. Parameters other than the prefix that have been
- * passed when creating the iterator will remain unchanged.
+ * This function is expected to behave as if a new ref iterator has been
+ * created, but allows reuse of existing iterators for optimization.
+ *
+ * When the REF_ITERATOR_SEEK_SET_PREFIX flag is set, the iterator's prefix is
+ * updated to match the seek string, affecting all subsequent iterations. If
+ * not, the iterator seeks to the specified reference and clears any previously
+ * set prefix.
  *
  * Returns 0 on success, a negative error code otherwise.
  */
-int ref_iterator_seek(struct ref_iterator *ref_iterator,
-		      const char *prefix);
+int ref_iterator_seek(struct ref_iterator *ref_iterator, const char *seek,
+		      unsigned int flags);
 
 /*
  * If possible, peel the reference currently being viewed by the
refs/debug.c +4 -3
index 485e3079d7..2ed8cff2aa 100644
@@ -170,12 +170,13 @@ static int debug_ref_iterator_advance(struct ref_iterator *ref_iterator)
 }
 
 static int debug_ref_iterator_seek(struct ref_iterator *ref_iterator,
-				   const char *prefix)
+				   const char *seek, unsigned int flags)
 {
 	struct debug_ref_iterator *diter =
 		(struct debug_ref_iterator *)ref_iterator;
-	int res = diter->iter->vtable->seek(diter->iter, prefix);
-	trace_printf_key(&trace_refs, "iterator_seek: %s: %d\n", prefix ? prefix : "", res);
+	int res = diter->iter->vtable->seek(diter->iter, seek, flags);
+	trace_printf_key(&trace_refs, "iterator_seek: %s flags: %d: %d\n",
+			 seek ? seek : "", flags, res);
 	return res;
 }
 
refs/files-backend.c +4 -3
index bf6f89b1d1..0e63013319 100644
@@ -929,11 +929,11 @@ static int files_ref_iterator_advance(struct ref_iterator *ref_iterator)
 }
 
 static int files_ref_iterator_seek(struct ref_iterator *ref_iterator,
-				   const char *prefix)
+				   const char *seek, unsigned int flags)
 {
 	struct files_ref_iterator *iter =
 		(struct files_ref_iterator *)ref_iterator;
-	return ref_iterator_seek(iter->iter0, prefix);
+	return ref_iterator_seek(iter->iter0, seek, flags);
 }
 
 static int files_ref_iterator_peel(struct ref_iterator *ref_iterator,
@@ -2316,7 +2316,8 @@ static int files_reflog_iterator_advance(struct ref_iterator *ref_iterator)
 }
 
 static int files_reflog_iterator_seek(struct ref_iterator *ref_iterator UNUSED,
-				      const char *prefix UNUSED)
+				      const char *seek UNUSED,
+				      unsigned int flags UNUSED)
 {
 	BUG("ref_iterator_seek() called for reflog_iterator");
 }
refs/iterator.c +15 -11
index 766d96e795..f2364bd6e7 100644
@@ -15,10 +15,10 @@ int ref_iterator_advance(struct ref_iterator *ref_iterator)
 	return ref_iterator->vtable->advance(ref_iterator);
 }
 
-int ref_iterator_seek(struct ref_iterator *ref_iterator,
-		      const char *prefix)
+int ref_iterator_seek(struct ref_iterator *ref_iterator, const char *seek,
+		      unsigned int flags)
 {
-	return ref_iterator->vtable->seek(ref_iterator, prefix);
+	return ref_iterator->vtable->seek(ref_iterator, seek, flags);
 }
 
 int ref_iterator_peel(struct ref_iterator *ref_iterator,
@@ -57,7 +57,8 @@ static int empty_ref_iterator_advance(struct ref_iterator *ref_iterator UNUSED)
 }
 
 static int empty_ref_iterator_seek(struct ref_iterator *ref_iterator UNUSED,
-				   const char *prefix UNUSED)
+				   const char *seek UNUSED,
+				   unsigned int flags UNUSED)
 {
 	return 0;
 }
@@ -224,7 +225,7 @@ static int merge_ref_iterator_advance(struct ref_iterator *ref_iterator)
 }
 
 static int merge_ref_iterator_seek(struct ref_iterator *ref_iterator,
-				   const char *prefix)
+				   const char *seek, unsigned int flags)
 {
 	struct merge_ref_iterator *iter =
 		(struct merge_ref_iterator *)ref_iterator;
@@ -234,11 +235,11 @@ static int merge_ref_iterator_seek(struct ref_iterator *ref_iterator,
 	iter->iter0 = iter->iter0_owned;
 	iter->iter1 = iter->iter1_owned;
 
-	ret = ref_iterator_seek(iter->iter0, prefix);
+	ret = ref_iterator_seek(iter->iter0, seek, flags);
 	if (ret < 0)
 		return ret;
 
-	ret = ref_iterator_seek(iter->iter1, prefix);
+	ret = ref_iterator_seek(iter->iter1, seek, flags);
 	if (ret < 0)
 		return ret;
 
@@ -407,13 +408,16 @@ static int prefix_ref_iterator_advance(struct ref_iterator *ref_iterator)
 }
 
 static int prefix_ref_iterator_seek(struct ref_iterator *ref_iterator,
-				    const char *prefix)
+				    const char *seek, unsigned int flags)
 {
 	struct prefix_ref_iterator *iter =
 		(struct prefix_ref_iterator *)ref_iterator;
-	free(iter->prefix);
-	iter->prefix = xstrdup_or_null(prefix);
-	return ref_iterator_seek(iter->iter0, prefix);
+
+	if (flags & REF_ITERATOR_SEEK_SET_PREFIX) {
+		free(iter->prefix);
+		iter->prefix = xstrdup_or_null(seek);
+	}
+	return ref_iterator_seek(iter->iter0, seek, flags);
 }
 
 static int prefix_ref_iterator_peel(struct ref_iterator *ref_iterator,
refs/packed-backend.c +11 -6
index 7fd73a0e6d..11a363d246 100644
@@ -1004,19 +1004,23 @@ static int packed_ref_iterator_advance(struct ref_iterator *ref_iterator)
 }
 
 static int packed_ref_iterator_seek(struct ref_iterator *ref_iterator,
-				    const char *prefix)
+				    const char *seek, unsigned int flags)
 {
 	struct packed_ref_iterator *iter =
 		(struct packed_ref_iterator *)ref_iterator;
 	const char *start;
 
-	if (prefix && *prefix)
-		start = find_reference_location(iter->snapshot, prefix, 0);
+	if (seek && *seek)
+		start = find_reference_location(iter->snapshot, seek, 0);
 	else
 		start = iter->snapshot->start;
 
-	free(iter->prefix);
-	iter->prefix = xstrdup_or_null(prefix);
+	/* Unset any previously set prefix */
+	FREE_AND_NULL(iter->prefix);
+
+	if (flags & REF_ITERATOR_SEEK_SET_PREFIX)
+		iter->prefix = xstrdup_or_null(seek);
+
 	iter->pos = start;
 	iter->eof = iter->snapshot->eof;
 
@@ -1194,7 +1198,8 @@ static struct ref_iterator *packed_ref_iterator_begin(
 	iter->repo = ref_store->repo;
 	iter->flags = flags;
 
-	if (packed_ref_iterator_seek(&iter->base, prefix) < 0) {
+	if (packed_ref_iterator_seek(&iter->base, prefix,
+				     REF_ITERATOR_SEEK_SET_PREFIX) < 0) {
 		ref_iterator_free(&iter->base);
 		return NULL;
 	}
refs/ref-cache.c +80 -5

104 lines not on this page · raw message

refs/refs-internal.h +4 -3
index 03f5df04d5..6376a3b379 100644
@@ -353,11 +353,12 @@ void base_ref_iterator_init(struct ref_iterator *iter,
 typedef int ref_iterator_advance_fn(struct ref_iterator *ref_iterator);
 
 /*
- * Seek the iterator to the first reference matching the given prefix. Should
- * behave the same as if a new iterator was created with the same prefix.
+ * Seek the iterator to the first matching reference. If set_prefix is set,
+ * it would behave the same as if a new iterator was created with the same
+ * prefix.
  */
 typedef int ref_iterator_seek_fn(struct ref_iterator *ref_iterator,
-				 const char *prefix);
+				 const char *seek, unsigned int flags);
 
 /*
  * Peels the current ref, returning 0 for success or -1 for failure.
refs/reftable-backend.c +14 -7
index 4c3817f4ec..d627221b65 100644
@@ -719,15 +719,20 @@ static int reftable_ref_iterator_advance(struct ref_iterator *ref_iterator)
 }
 
 static int reftable_ref_iterator_seek(struct ref_iterator *ref_iterator,
-				      const char *prefix)
+				      const char *seek, unsigned int flags)
 {
 	struct reftable_ref_iterator *iter =
 		(struct reftable_ref_iterator *)ref_iterator;
 
-	free(iter->prefix);
-	iter->prefix = xstrdup_or_null(prefix);
-	iter->prefix_len = prefix ? strlen(prefix) : 0;
-	iter->err = reftable_iterator_seek_ref(&iter->iter, prefix);
+	/* Unset any previously set prefix */
+	FREE_AND_NULL(iter->prefix);
+	iter->prefix_len = 0;
+
+	if (flags & REF_ITERATOR_SEEK_SET_PREFIX) {
+		iter->prefix = xstrdup_or_null(seek);
+		iter->prefix_len = seek ? strlen(seek) : 0;
+	}
+	iter->err = reftable_iterator_seek_ref(&iter->iter, seek);
 
 	return iter->err;
 }
@@ -839,7 +844,8 @@ static struct reftable_ref_iterator *ref_iterator_for_stack(struct reftable_ref_
 	if (ret)
 		goto done;
 
-	ret = reftable_ref_iterator_seek(&iter->base, prefix);
+	ret = reftable_ref_iterator_seek(&iter->base, prefix,
+					 REF_ITERATOR_SEEK_SET_PREFIX);
 	if (ret)
 		goto done;
 
@@ -2042,7 +2048,8 @@ static int reftable_reflog_iterator_advance(struct ref_iterator *ref_iterator)
 }
 
 static int reftable_reflog_iterator_seek(struct ref_iterator *ref_iterator UNUSED,
-					 const char *prefix UNUSED)
+					 const char *seek UNUSED,
+					 unsigned int flags UNUSED)
 {
 	BUG("reftable reflog iterator cannot be seeked");
 	return -1;
Signature · 1 line
2.49.0
#26 lore raw

PATCH v2 4/4 for-each-ref: introduce a '--skip-until' option

In reply to Karthik Nayak

The git-for-each-ref(1) command is used to iterate over references present in a repository. In large repositories with millions of references, it would be optimal to paginate this output such that we can start iteration from a given reference. This would avoid having to iterate over all references from the beginning each time when paginating through results.

The previous commit added 'seek' functionality to the reference backends. Utilize this and expose a '--skip-until' option in 'git-for-each-ref(1)'. When used, the reference iteration seeks to the first matching reference and iterates from there onward.

This enables efficient pagination workflows like:

    git for-each-ref --count=100
    git for-each-ref --count=100 --skip-until=refs/heads/branch-100
    git for-each-ref --count=100 --skip-until=refs/heads/branch-200
---
 Documentation/git-for-each-ref.adoc |   6 +-
 builtin/for-each-ref.c              |   8 ++
 ref-filter.c                        |  61 ++++++++----
 ref-filter.h                        |   1 +
 t/t6302-for-each-ref-filter.sh      | 188 ++++++++++++++++++++++++++++++++++++
 5 files changed, 243 insertions(+), 21 deletions(-)
Documentation/git-for-each-ref.adoc +5 -1
index 5ef89fc0fe..e369fee9a1 100644
@@ -14,7 +14,7 @@ SYNOPSIS
 		   [--points-at=<object>]
 		   [--merged[=<object>]] [--no-merged[=<object>]]
 		   [--contains[=<object>]] [--no-contains[=<object>]]
-		   [--exclude=<pattern> ...]
+		   [--exclude=<pattern> ...] [--skip-until=<pattern>]
 
 DESCRIPTION
 -----------
@@ -108,6 +108,10 @@ TAB %(refname)`.
 --include-root-refs::
 	List root refs (HEAD and pseudorefs) apart from regular refs.
 
+--skip-until::
+    Skip references up to but excluding the specified pattern. Cannot be used
+    with general pattern matching or custom sort options.
+
 FIELD NAMES
 -----------
 
builtin/for-each-ref.c +8 -0
index 3d2207ec77..aee2e7489a 100644
@@ -13,6 +13,7 @@ static char const * const for_each_ref_usage[] = {
 	N_("git for-each-ref [--points-at <object>]"),
 	N_("git for-each-ref [--merged [<commit>]] [--no-merged [<commit>]]"),
 	N_("git for-each-ref [--contains [<commit>]] [--no-contains [<commit>]]"),
+	N_("git for-each-ref [--skip-until <pattern>]"),
 	NULL
 };
 
@@ -44,6 +45,7 @@ int cmd_for_each_ref(int argc,
 		OPT_GROUP(""),
 		OPT_INTEGER( 0 , "count", &format.array_opts.max_count, N_("show only <n> matched refs")),
 		OPT_STRING(  0 , "format", &format.format, N_("format"), N_("format to use for the output")),
+		OPT_STRING(  0 , "skip-until", &filter.seek, N_("skip-until"), N_("skip references until")),
 		OPT__COLOR(&format.use_color, N_("respect format colors")),
 		OPT_REF_FILTER_EXCLUDE(&filter),
 		OPT_REF_SORT(&sorting_options),
@@ -79,6 +81,9 @@ int cmd_for_each_ref(int argc,
 	if (verify_ref_format(&format))
 		usage_with_options(for_each_ref_usage, opts);
 
+	if (filter.seek && sorting_options.nr > 1)
+		die(_("cannot use --skip-until custom sort options"));
+
 	sorting = ref_sorting_options(&sorting_options);
 	ref_sorting_set_sort_flags_all(sorting, REF_SORTING_ICASE, icase);
 	filter.ignore_case = icase;
@@ -100,6 +105,9 @@ int cmd_for_each_ref(int argc,
 		filter.name_patterns = argv;
 	}
 
+	if (filter.seek && filter.name_patterns && filter.name_patterns[0])
+		die(_("cannot use --skip-until with patterns"));
+
 	if (include_root_refs)
 		flags |= FILTER_REFS_ROOT_REFS | FILTER_REFS_DETACHED_HEAD;
 
ref-filter.c +41 -20

107 lines not on this page · raw message

ref-filter.h +1 -0
index c98c4fbd4c..9e97c65bc2 100644
@@ -64,6 +64,7 @@ struct ref_array {
 
 struct ref_filter {
 	const char **name_patterns;
+	const char *seek;
 	struct strvec exclude;
 	struct oid_array points_at;
 	struct commit_list *with_commit;
t/t6302-for-each-ref-filter.sh +188 -0

192 lines not on this page · raw message

Signature · 1 line
2.49.0
#27 lore raw

Re: [PATCH v2 4/4] for-each-ref: introduce a '--skip-until' option

builtin/for-each-ref.c int cmd_for_each_ref(int argc, · line 85

Quoted 2 lines · Karthik Nayak
builtin/for-each-ref.c +2 -0
+	if (filter.seek && sorting_options.nr > 1)
+		die(_("cannot use --skip-until custom sort options"));

Missing "with" before "custom sort".

When I commented on the previous iteration about sorting, I didn't mean to suggest making them incompatible---it may have some use case to grab a batch out of the underlying refstore, sort refs in that batch, and then show them. But from usability's point of view, I tend to agree with this design decision. Such an unnatural batching and sorting operation is probably not worth supporting.

#28 lore raw

Re: [PATCH v2 4/4] for-each-ref: introduce a '--skip-until' option

Quoted 7 lines · Junio C Hamano

builtin/for-each-ref.c int cmd_for_each_ref(int argc, · line 85

Quoted 2 lines · Karthik Nayak
builtin/for-each-ref.c +2 -0
+	if (filter.seek && sorting_options.nr > 1)
+		die(_("cannot use --skip-until custom sort options"));

Missing "with" before "custom sort".

Thanks, will fix in the next version.

Quoted 6 lines

When I commented on the previous iteration about sorting, I didn't mean to suggest making them incompatible---it may have some use case to grab a batch out of the underlying refstore, sort refs in that batch, and then show them. But from usability's point of view, I tend to agree with this design decision. Such an unnatural batching and sorting operation is probably not worth supporting.

I think so too, there might be some usecase, when that usecase is more concrete we can probably re-visit this.

For now, it is confusing and it is just easier to not support them together.

#29 lore raw
In reply to Karthik Nayak
Quoted 3 lines · Karthik Nayak

This series adds a '--skip-until' option in 'git-for-each-ref(1)'. When used, the reference iteration seeks to the first matching reference and iterates from there onward.

I would have named the option --start-with. It has the advantage that it is clear whether the matched ref is included.

Signature · 3 lines
Andreas Schwab, [email protected]
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."
#30 lore raw
Quoted 9 lines · Andreas Schwab
Quoted 3 lines · Karthik Nayak

This series adds a '--skip-until' option in 'git-for-each-ref(1)'. When used, the reference iteration seeks to the first matching reference and iterates from there onward.

I would have named the option --start-with. It has the advantage that it is clear whether the matched ref is included.

We did discuss this internally, some other names we thought of:
--skip-to
--start-after
--start-from
--seek
--skip-before
--start-at

I think I was a bit against '--start-from' and '--start-at', because they imply that the reference provided must exist.

Consider the example

  $ git for-each-ref
  refs/heads/bar
  refs/heads/foo
  refs/heads/main
  $ git for-each-ref --seek=refs/heads/cat
  refs/heads/foo
  refs/heads/main

You can see that the reference doesn't have to exist. So implying that it should can be a bit confusing.

But I'm open to changing this, if we can conclude on any flag name...

Quoted 4 lines

--
Andreas Schwab, [email protected]
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510 2552 DF73 E780 A9DA AEC1 "And now for something completely different."

#31 lore raw
Quoted 7 lines · Karthik Nayak

Consider the example

  $ git for-each-ref
  refs/heads/bar
  refs/heads/foo
  refs/heads/main
Quoted 5 lines
  $ git for-each-ref --seek=refs/heads/cat
  refs/heads/foo
  refs/heads/main

You can see that the reference doesn't have to exist.

That is even more confusing. What is the first matching ref if none of them match? Doesn't that mean skipping _all_ refs?

Signature · 3 lines
Andreas Schwab, [email protected]
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."
#32 lore raw
Quoted 9 lines · Andreas Schwab
Quoted 7 lines · Karthik Nayak

Consider the example

  $ git for-each-ref
  refs/heads/bar
  refs/heads/foo
  refs/heads/main
Quoted 9 lines
Quoted 5 lines
  $ git for-each-ref --seek=refs/heads/cat
  refs/heads/foo
  refs/heads/main

You can see that the reference doesn't have to exist.

That is even more confusing. What is the first matching ref if none of them match? Doesn't that mean skipping _all_ refs?

Well the idea is it would seek to the offset where the reference would fit in.

This is to ensure that seeks to references which were deleted concurrently doesn't leave the client hanging with no results while paginating over all references.

Quoted 4 lines

--
Andreas Schwab, [email protected]
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510 2552 DF73 E780 A9DA AEC1 "And now for something completely different."

#33 lore raw
Quoted 21 lines · Karthik Nayak
Quoted 9 lines · Andreas Schwab
Quoted 7 lines · Karthik Nayak

Consider the example

  $ git for-each-ref
  refs/heads/bar
  refs/heads/foo
  refs/heads/main
Quoted 9 lines
Quoted 5 lines
  $ git for-each-ref --seek=refs/heads/cat
  refs/heads/foo
  refs/heads/main

You can see that the reference doesn't have to exist.

That is even more confusing. What is the first matching ref if none of them match? Doesn't that mean skipping _all_ refs?

Quoted 6 lines

Well the idea is it would seek to the offset where the reference would fit in.

This is to ensure that seeks to references which were deleted concurrently doesn't leave the client hanging with no results while paginating over all references.

Then don't call it a pattern. Pattern matching is a set operation, independent of sorting. What you really have is a marker that divides the sorted list in two parts according to how the marker sorts. And that makes --start-with more descriptive and less ambiguous.

Signature · 3 lines
Andreas Schwab, [email protected]
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."
#34 lore raw
Quoted 26 lines · Andreas Schwab
Quoted 24 lines · Karthik Nayak
Quoted 9 lines · Andreas Schwab
Quoted 7 lines · Karthik Nayak

Consider the example

  $ git for-each-ref
  refs/heads/bar
  refs/heads/foo
  refs/heads/main
Quoted 9 lines
Quoted 5 lines
  $ git for-each-ref --seek=refs/heads/cat
  refs/heads/foo
  refs/heads/main

You can see that the reference doesn't have to exist.

That is even more confusing. What is the first matching ref if none of them match? Doesn't that mean skipping _all_ refs?

Well the idea is it would seek to the offset where the reference would fit in.

Quoted 9 lines
Quoted 3 lines

This is to ensure that seeks to references which were deleted concurrently doesn't leave the client hanging with no results while paginating over all references.

Then don't call it a pattern. Pattern matching is a set operation, independent of sorting. What you really have is a marker that divides the sorted list in two parts according to how the marker sorts. And that makes --start-with more descriptive and less ambiguous.

Fair enough. I'll change the documentation and description in the next version.

Quoted 4 lines

--
Andreas Schwab, [email protected]
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510 2552 DF73 E780 A9DA AEC1 "And now for something completely different."

#35 lore raw
In reply to Karthik Nayak
Quoted 2 lines · Karthik Nayak

I think I was a bit against '--start-from' and '--start-at', because they imply that the reference provided must exist.

It also implies that if the reference does exist, that would be the first one that is shown. But I do not think you want that, as ...

Quoted 6 lines

Consider the example

  $ git for-each-ref
  refs/heads/bar
  refs/heads/foo
  refs/heads/main

... after a paging application starts from the beginning and showed a single page of some items, it knows the "last" one it showed. That last entry may have been refs/heads/bar. The application may not have seen the next entry (i.e. refs/heads/foo). So if it has to use '--start-at=refs/heads/bar', the first entry it gets from such a request may be for refs/heads/bar again. The application needs to remember the "last" one it showed and skip that, which is a bit awkward, isn't it?

Quoted 6 lines
  $ git for-each-ref --seek=refs/heads/cat
  refs/heads/foo
  refs/heads/main

You can see that the reference doesn't have to exist. So implying that it should can be a bit confusing.

For that reason, whatever verb you pick from seek or start or skip, it would be great if the option name also made it explicit that the named one, if exists, is not shown. Conceptually, it is "skip everything that sorts before the named item, including the named item itself" that such a paging application would want, wouldn't it?

Thanks.

#36 lore raw
Quoted 16 lines · Junio C Hamano
Quoted 2 lines · Karthik Nayak

I think I was a bit against '--start-from' and '--start-at', because they imply that the reference provided must exist.

It also implies that if the reference does exist, that would be the first one that is shown. But I do not think you want that, as ...

Quoted 6 lines

Consider the example

  $ git for-each-ref
  refs/heads/bar
  refs/heads/foo
  refs/heads/main
Quoted 9 lines

... after a paging application starts from the beginning and showed a single page of some items, it knows the "last" one it showed. That last entry may have been refs/heads/bar. The application may not have seen the next entry (i.e. refs/heads/foo). So if it has to use '--start-at=refs/heads/bar', the first entry it gets from such a request may be for refs/heads/bar again. The application needs to remember the "last" one it showed and skip that, which is a bit awkward, isn't it?

I do agree, I was modelling this after what would be the best approach within the Git codebase. But I think it would be nicer for the clients if we skip the provided reference.

Quoted 7 lines
Quoted 6 lines
  $ git for-each-ref --seek=refs/heads/cat
  refs/heads/foo
  refs/heads/main

You can see that the reference doesn't have to exist. So implying that it should can be a bit confusing.

Quoted 8 lines

For that reason, whatever verb you pick from seek or start or skip, it would be great if the option name also made it explicit that the named one, if exists, is not shown. Conceptually, it is "skip everything that sorts before the named item, including the named item itself" that such a paging application would want, wouldn't it?

Thanks.

With that I think '--start-after' sounds like the best option. I'll modify for the next version accordingly.

Thanks!

#37 lore raw
Quoted 5 lines · Karthik Nayak
Quoted 1 line · Junio C Hamano

Karthik Nayak <[email protected]> writes:

I do agree, I was modelling this after what would be the best approach within the Git codebase.

That was my fear when I asked about using a numeric offset. Patrick has made a principled argument for using a ref name rather than a numeric ofsset - I think you should build the motivation for this series around that and the documentation should explain the implications of references being added and deleted while paging them.

Quoted 2 lines

With that I think '--start-after' sounds like the best option. I'll modify for the next version accordingly.

That sounds like a good name

Thanks

Phillip

#38 lore raw
Quoted 4 lines · Phillip Wood
Quoted 3 lines · Karthik Nayak
Quoted 1 line · Junio C Hamano

Karthik Nayak <[email protected]> writes:

Quoted 9 lines
Quoted 2 lines

I do agree, I was modelling this after what would be the best approach within the Git codebase.

That was my fear when I asked about using a numeric offset. Patrick has made a principled argument for using a ref name rather than a numeric ofsset - I think you should build the motivation for this series around that and the documentation should explain the implications of references being added and deleted while paging them.

Yeah, I'll add something on those lines in the next version. I do appreciate these checks/questions.

Quoted 4 lines
Quoted 2 lines

With that I think '--start-after' sounds like the best option. I'll modify for the next version accordingly.

That sounds like a good name

Thanks!

Quoted 3 lines

Thanks

Phillip

#39 lore raw

v3 cover for-each-ref: introduce seeking functionality via '--start-after'

In reply to Karthik Nayak

The git-for-each-ref(1) command is used to iterate over references present in a repository. In large repositories with millions of references, it would be optimal to paginate this output such that we can start iteration from a given reference. This would avoid having to iterate over all references from the beginning each time when paginating through results.

This series adds a '--start-after' option in 'git-for-each-ref(1)'. When used, the reference iteration seeks to first reference following the marker alphabetically. When paging, it should be noted that references may be deleted, modified or added between invocations. Output will only yield those references which follow the marker lexicographically. If the marker does not exist, output begins from the first reference that would come after it alphabetically.

This enables efficient pagination workflows like:

    git for-each-ref --count=100
    git for-each-ref --count=100 --start-after=refs/heads/branch-100
    git for-each-ref --count=100 --start-after=refs/heads/branch-200

To add this functionality, we expose the ref_iterator outside the 'refs/' namespace and modify the ref_iterator_seek() to actually seek to a given reference and only set the prefix when the set_prefix field is set.

On the reftable and packed backend, the changes are simple. But since the files backend uses 'ref-cache' for reference handling, the changes there are a little more involved, since we need to setup the right levels and the indexing.

Initially I was also planning to cleanup all the refs_for_each...() functions in 'refs.h' by simply using the iterator, but this bloated the series. So I've left that for another day.

Changes in v3:
- Change the working of the command to exclude the marker provided. With
this rename the flag to '--start-after'.
- Extend the documentation to add a note about concurrent modifications
to the reference database.
- Link to v2: https://lore.kernel.org/r/20250704-306-git-for-each-ref-pagination-v2-0-bcde14acdd81@gmail.com

Changes in v2:
- Modify 'ref_iterator_seek()' to take in flags instead of a
'set_prefix' variable. This improves readability, where users would
use the 'REF_ITERATOR_SEEK_SET_PREFIX' instead of simply passing '1'.
- When the set prefix flag isn't usage, reset any previously set prefix.
This ensures that the internal prefix state is always reset whenever
we seek and unifies the behavior between 'ref_iterator_seek' and
'ref_iterator_begin'.
- Don't allow '--skip-until' to be run with '--sort', since the seeking
always takes place before any sorting and this can be confusing.
- Some styling fixes:
- Remove extra newline
- Skip braces around single lined if...else clause
- Add braces around 'if' clause
- Fix indentation
- Link to v1: https://lore.kernel.org/git/20250701-306-git-for-each-ref-pagination-v1-0-4f0ae7c0688f@gmail.com/

---
 Documentation/git-for-each-ref.adoc |  11 +-
 builtin/for-each-ref.c              |   8 ++
 ref-filter.c                        |  80 +++++++++++----
 ref-filter.h                        |   1 +
 refs.c                              |   6 +-
 refs.h                              | 158 +++++++++++++++++++++++++++++
 refs/debug.c                        |   7 +-
 refs/files-backend.c                |   7 +-
 refs/iterator.c                     |  26 +++--
 refs/packed-backend.c               |  17 ++--
 refs/ref-cache.c                    |  99 ++++++++++++++----
 refs/ref-cache.h                    |   7 --
 refs/refs-internal.h                | 152 ++--------------------------
 refs/reftable-backend.c             |  21 ++--
 t/t6302-for-each-ref-filter.sh      | 194 ++++++++++++++++++++++++++++++++++++
 15 files changed, 568 insertions(+), 226 deletions(-)

Karthik Nayak (4):
refs: expose ref_iterator via 'refs.h'
ref-cache: remove unused function 'find_ref_entry()'
refs: selectively set prefix in the seek functions
for-each-ref: introduce a '--start-after' option

Range-diff versus v2:

1:  c0ce873c35 = 1:  dbb03c2aa9 refs: expose `ref_iterator` via 'refs.h'
2:  2c50d1eba2 = 2:  fa5a0cb722 ref-cache: remove unused function 'find_ref_entry()'
3:  fae849749f = 3:  9940d390cc refs: selectively set prefix in the seek functions
4:  a0725a6647 ! 4:  ebe864095a for-each-ref: introduce a '--skip-until' option
    @@ Metadata
     Author: Karthik Nayak <[email protected]>
     
      ## Commit message ##
    -    for-each-ref: introduce a '--skip-until' option
    +    for-each-ref: introduce a '--start-after' option
     
         The `git-for-each-ref(1)` command is used to iterate over references
         present in a repository. In large repositories with millions of
    @@ Commit message
         through results.
     
         The previous commit added 'seek' functionality to the reference
    -    backends. Utilize this and expose a '--skip-until' option in
    +    backends. Utilize this and expose a '--start-after' option in
         'git-for-each-ref(1)'. When used, the reference iteration seeks to the
    -    first matching reference and iterates from there onward.
    +    lexicographically next reference and iterates from there onward.
     
         This enables efficient pagination workflows like:
             git for-each-ref --count=100
    -        git for-each-ref --count=100 --skip-until=refs/heads/branch-100
    -        git for-each-ref --count=100 --skip-until=refs/heads/branch-200
    +        git for-each-ref --count=100 --start-after=refs/heads/branch-100
    +        git for-each-ref --count=100 --start-after=refs/heads/branch-200
    +
    +    Since the reference iterators only allow seeking to a specified marker
    +    via the `ref_iterator_seek()`, we introduce a helper function
    +    `start_ref_iterator_after()`, which seeks to next reference by simply
    +    adding (char) 1 to the marker.
    +
    +    We must note that pagination always continues from the provided marker,
    +    as such any concurrent reference updates lexicographically behind the
    +    marker will not be output. Document the same.
     
         Signed-off-by: Karthik Nayak <[email protected]>
     
    @@ Documentation/git-for-each-ref.adoc: SYNOPSIS
      		   [--merged[=<object>]] [--no-merged[=<object>]]
      		   [--contains[=<object>]] [--no-contains[=<object>]]
     -		   [--exclude=<pattern> ...]
    -+		   [--exclude=<pattern> ...] [--skip-until=<pattern>]
    ++		   [--exclude=<pattern> ...] [--start-after=<marker>]
      
      DESCRIPTION
      -----------
    @@ Documentation/git-for-each-ref.adoc: TAB %(refname)`.
      --include-root-refs::
      	List root refs (HEAD and pseudorefs) apart from regular refs.
      
    -+--skip-until::
    -+    Skip references up to but excluding the specified pattern. Cannot be used
    -+    with general pattern matching or custom sort options.
    ++--start-after::
    ++    Allows paginating the output by skipping references up to and including the
    ++    specified marker. When paging, it should be noted that references may be
    ++    deleted, modified or added between invocations. Output will only yield those
    ++    references which follow the marker lexicographically. If the marker does not
    ++    exist, output begins from the first reference that would come after it
    ++    alphabetically. Cannot be used with general pattern matching or custom
    ++    sort options.
     +
      FIELD NAMES
      -----------
    @@ builtin/for-each-ref.c: static char const * const for_each_ref_usage[] = {
      	N_("git for-each-ref [--points-at <object>]"),
      	N_("git for-each-ref [--merged [<commit>]] [--no-merged [<commit>]]"),
      	N_("git for-each-ref [--contains [<commit>]] [--no-contains [<commit>]]"),
    -+	N_("git for-each-ref [--skip-until <pattern>]"),
    ++	N_("git for-each-ref [--start-after <marker>]"),
      	NULL
      };
      
    @@ builtin/for-each-ref.c: int cmd_for_each_ref(int argc,
      		OPT_GROUP(""),
      		OPT_INTEGER( 0 , "count", &format.array_opts.max_count, N_("show only <n> matched refs")),
      		OPT_STRING(  0 , "format", &format.format, N_("format"), N_("format to use for the output")),
    -+		OPT_STRING(  0 , "skip-until", &filter.seek, N_("skip-until"), N_("skip references until")),
    ++		OPT_STRING(  0 , "start-after", &filter.start_after, N_("start-start"), N_("start iteration after the provided marker")),
      		OPT__COLOR(&format.use_color, N_("respect format colors")),
      		OPT_REF_FILTER_EXCLUDE(&filter),
      		OPT_REF_SORT(&sorting_options),
    @@ builtin/for-each-ref.c: int cmd_for_each_ref(int argc,
      	if (verify_ref_format(&format))
      		usage_with_options(for_each_ref_usage, opts);
      
    -+	if (filter.seek && sorting_options.nr > 1)
    -+		die(_("cannot use --skip-until custom sort options"));
    ++	if (filter.start_after && sorting_options.nr > 1)
    ++		die(_("cannot use --start-after with custom sort options"));
     +
      	sorting = ref_sorting_options(&sorting_options);
      	ref_sorting_set_sort_flags_all(sorting, REF_SORTING_ICASE, icase);
    @@ builtin/for-each-ref.c: int cmd_for_each_ref(int argc,
      		filter.name_patterns = argv;
      	}
      
    -+	if (filter.seek && filter.name_patterns && filter.name_patterns[0])
    -+		die(_("cannot use --skip-until with patterns"));
    ++	if (filter.start_after && filter.name_patterns && filter.name_patterns[0])
    ++		die(_("cannot use --start-after with patterns"));
     +
      	if (include_root_refs)
      		flags |= FILTER_REFS_ROOT_REFS | FILTER_REFS_DETACHED_HEAD;
      
     
      ## ref-filter.c ##
    +@@ ref-filter.c: static int filter_exclude_match(struct ref_filter *filter, const char *refname)
    + 	return match_pattern(filter->exclude.v, refname, filter->ignore_case);
    + }
    + 
    ++/*
    ++ * We need to seek to the reference right after a given marker but excluding any
    ++ * matching references. So we seek to the lexicographically next reference.
    ++ */
    ++static int start_ref_iterator_after(struct ref_iterator *iter, const char *marker)
    ++{
    ++	struct strbuf sb = STRBUF_INIT;
    ++	int ret;
    ++
    ++	strbuf_addstr(&sb, marker);
    ++	strbuf_addch(&sb, 1);
    ++
    ++	ret = ref_iterator_seek(iter, sb.buf, 0);
    ++
    ++	strbuf_release(&sb);
    ++	return ret;
    ++}
    ++
    + /*
    +  * This is the same as for_each_fullref_in(), but it tries to iterate
    +  * only over the patterns we'll care about. Note that it _doesn't_ do a full
     @@ ref-filter.c: static int for_each_fullref_in_pattern(struct ref_filter *filter,
      				       each_ref_fn cb,
      				       void *cb_data)
    @@ ref-filter.c: static int for_each_fullref_in_pattern(struct ref_filter *filter,
     +non_prefix_iter:
     +	iter = refs_ref_iterator_begin(get_main_ref_store(the_repository), "",
     +				       NULL, 0, flags);
    -+	if (filter->seek)
    -+		ret = ref_iterator_seek(iter, filter->seek, 0);
    ++	if (filter->start_after)
    ++		ret = start_ref_iterator_after(iter, filter->start_after);
    ++
     +	if (ret)
     +		return ret;
     +
    @@ ref-filter.c: static int do_filter_refs(struct ref_filter *filter, unsigned int
     +			iter = refs_ref_iterator_begin(get_main_ref_store(the_repository),
     +						       "", NULL, 0, 0);
     +
    -+			if (filter->seek)
    -+				ret = ref_iterator_seek(iter, filter->seek, 0);
    ++			if (filter->start_after)
    ++				ret = start_ref_iterator_after(iter, filter->start_after);
     +			else if (prefix)
     +				ret = ref_iterator_seek(iter, prefix, 1);
     +
    @@ ref-filter.h: struct ref_array {
      
      struct ref_filter {
      	const char **name_patterns;
    -+	const char *seek;
    ++	const char *start_after;
      	struct strvec exclude;
      	struct oid_array points_at;
      	struct commit_list *with_commit;
    @@ t/t6302-for-each-ref-filter.sh: test_expect_success 'validate worktree atom' '
      	test_cmp expect actual
      '
      
    -+test_expect_success 'skip until with empty value' '
    ++test_expect_success 'start after with empty value' '
     +	cat >expect <<-\EOF &&
     +	refs/heads/main
     +	refs/heads/main_worktree
    @@ t/t6302-for-each-ref-filter.sh: test_expect_success 'validate worktree atom' '
     +	refs/tags/three
     +	refs/tags/two
     +	EOF
    -+	git for-each-ref --format="%(refname)" --skip-until="" >actual &&
    ++	git for-each-ref --format="%(refname)" --start-after="" >actual &&
     +	test_cmp expect actual
     +'
     +
    -+test_expect_success 'skip until to a specific reference' '
    ++test_expect_success 'start after a specific reference' '
     +	cat >expect <<-\EOF &&
    -+	refs/odd/spot
     +	refs/tags/annotated-tag
     +	refs/tags/doubly-annotated-tag
     +	refs/tags/doubly-signed-tag
    @@ t/t6302-for-each-ref-filter.sh: test_expect_success 'validate worktree atom' '
     +	refs/tags/three
     +	refs/tags/two
     +	EOF
    -+	git for-each-ref --format="%(refname)" --skip-until=refs/odd/spot >actual &&
    ++	git for-each-ref --format="%(refname)" --start-after=refs/odd/spot >actual &&
     +	test_cmp expect actual
     +'
     +
    -+test_expect_success 'skip until to a specific reference with partial match' '
    ++test_expect_success 'start after a specific reference with partial match' '
     +	cat >expect <<-\EOF &&
     +	refs/odd/spot
     +	refs/tags/annotated-tag
    @@ t/t6302-for-each-ref-filter.sh: test_expect_success 'validate worktree atom' '
     +	refs/tags/three
     +	refs/tags/two
     +	EOF
    -+	git for-each-ref --format="%(refname)" --skip-until=refs/odd/sp >actual &&
    ++	git for-each-ref --format="%(refname)" --start-after=refs/odd/sp >actual &&
     +	test_cmp expect actual
     +'
     +
    -+test_expect_success 'skip until just behind a specific reference' '
    ++test_expect_success 'start after, just behind a specific reference' '
     +	cat >expect <<-\EOF &&
     +	refs/odd/spot
     +	refs/tags/annotated-tag
    @@ t/t6302-for-each-ref-filter.sh: test_expect_success 'validate worktree atom' '
     +	refs/tags/three
     +	refs/tags/two
     +	EOF
    -+	git for-each-ref --format="%(refname)" --skip-until=refs/odd/parrot >actual &&
    ++	git for-each-ref --format="%(refname)" --start-after=refs/odd/parrot >actual &&
     +	test_cmp expect actual
     +'
     +
    -+test_expect_success 'skip until to specific directory' '
    ++test_expect_success 'start after with specific directory match' '
     +	cat >expect <<-\EOF &&
     +	refs/odd/spot
     +	refs/tags/annotated-tag
    @@ t/t6302-for-each-ref-filter.sh: test_expect_success 'validate worktree atom' '
     +	refs/tags/three
     +	refs/tags/two
     +	EOF
    -+	git for-each-ref --format="%(refname)" --skip-until=refs/odd >actual &&
    ++	git for-each-ref --format="%(refname)" --start-after=refs/odd >actual &&
     +	test_cmp expect actual
     +'
     +
    -+test_expect_success 'skip until to specific directory with trailing slash' '
    ++test_expect_success 'start after with specific directory and trailing slash' '
     +	cat >expect <<-\EOF &&
     +	refs/odd/spot
     +	refs/tags/annotated-tag
    @@ t/t6302-for-each-ref-filter.sh: test_expect_success 'validate worktree atom' '
     +	refs/tags/three
     +	refs/tags/two
     +	EOF
    -+	git for-each-ref --format="%(refname)" --skip-until=refs/lost >actual &&
    ++	git for-each-ref --format="%(refname)" --start-after=refs/lost >actual &&
     +	test_cmp expect actual
     +'
     +
    -+test_expect_success 'skip until just behind a specific directory' '
    ++test_expect_success 'start after, just behind a specific directory' '
     +	cat >expect <<-\EOF &&
     +	refs/odd/spot
     +	refs/tags/annotated-tag
    @@ t/t6302-for-each-ref-filter.sh: test_expect_success 'validate worktree atom' '
     +	refs/tags/three
     +	refs/tags/two
     +	EOF
    -+	git for-each-ref --format="%(refname)" --skip-until=refs/odd/ >actual &&
    ++	git for-each-ref --format="%(refname)" --start-after=refs/odd/ >actual &&
     +	test_cmp expect actual
     +'
     +
    -+test_expect_success 'skip until overflow specific reference length' '
    ++test_expect_success 'start after, overflow specific reference length' '
     +	cat >expect <<-\EOF &&
     +	refs/tags/annotated-tag
     +	refs/tags/doubly-annotated-tag
    @@ t/t6302-for-each-ref-filter.sh: test_expect_success 'validate worktree atom' '
     +	refs/tags/three
     +	refs/tags/two
     +	EOF
    -+	git for-each-ref --format="%(refname)" --skip-until=refs/odd/spotnew >actual &&
    ++	git for-each-ref --format="%(refname)" --start-after=refs/odd/spotnew >actual &&
     +	test_cmp expect actual
     +'
     +
    -+test_expect_success 'skip until overflow specific reference path' '
    ++test_expect_success 'start after, overflow specific reference path' '
     +	cat >expect <<-\EOF &&
     +	refs/tags/annotated-tag
     +	refs/tags/doubly-annotated-tag
    @@ t/t6302-for-each-ref-filter.sh: test_expect_success 'validate worktree atom' '
     +	refs/tags/three
     +	refs/tags/two
     +	EOF
    -+	git for-each-ref --format="%(refname)" --skip-until=refs/odd/spot/new >actual &&
    ++	git for-each-ref --format="%(refname)" --start-after=refs/odd/spot/new >actual &&
    ++	test_cmp expect actual
    ++'
    ++
    ++test_expect_success 'start after, last reference' '
    ++	cat >expect <<-\EOF &&
    ++	EOF
    ++	git for-each-ref --format="%(refname)" --start-after=refs/tags/two >actual &&
     +	test_cmp expect actual
     +'
     +
    -+test_expect_success 'skip until used with a pattern' '
    ++test_expect_success 'start after used with a pattern' '
     +	cat >expect <<-\EOF &&
    -+	fatal: cannot use --skip-until with patterns
    ++	fatal: cannot use --start-after with patterns
     +	EOF
    -+	test_must_fail git for-each-ref --format="%(refname)" --skip-until=refs/odd/spot refs/tags 2>actual &&
    ++	test_must_fail git for-each-ref --format="%(refname)" --start-after=refs/odd/spot refs/tags 2>actual &&
     +	test_cmp expect actual
     +'
     +
    -+test_expect_success 'skip until used with custom sort order' '
    ++test_expect_success 'start after used with custom sort order' '
     +	cat >expect <<-\EOF &&
    -+	fatal: cannot use --skip-until custom sort options
    ++	fatal: cannot use --start-after with custom sort options
     +	EOF
    -+	test_must_fail git for-each-ref --format="%(refname)" --skip-until=refs/odd/spot --sort=author 2>actual &&
    ++	test_must_fail git for-each-ref --format="%(refname)" --start-after=refs/odd/spot --sort=author 2>actual &&
     +	test_cmp expect actual
     +'
     +

base-commit: cf6f63ea6bf35173e02e18bdc6a4ba41288acff9
change-id: 20250605-306-git-for-each-ref-pagination-0ba8a29ae646

Thanks
- Karthik

#40 lore raw

PATCH v3 1/4 refs: expose `ref_iterator` via 'refs.h'

The ref_iterator is an internal structure to the 'refs/' sub-directory, which allows iteration over refs. All reference iteration is built on top of these iterators.

External clients of the 'refs' subsystem use the various 'refs_for_each...()' functions to iterate over refs. However since these are wrapper functions, each combination of functionality requires a new wrapper function. This is not feasible as the functions pile up with the increase in requirements. Expose the internal reference iterator, so advanced users can mix and match options as needed.

---
 refs.h               | 147 +++++++++++++++++++++++++++++++++++++++++++++++++++
 refs/refs-internal.h | 145 +-------------------------------------------------
 2 files changed, 149 insertions(+), 143 deletions(-)
refs.h +147 -0
index 46a6008e07..7c21aaef3d 100644
@@ -1190,4 +1190,151 @@ int repo_migrate_ref_storage_format(struct repository *repo,
 				    unsigned int flags,
 				    struct strbuf *err);
 
+/*
+ * Reference iterators
+ *
+ * A reference iterator encapsulates the state of an in-progress
+ * iteration over references. Create an instance of `struct
+ * ref_iterator` via one of the functions in this module.
+ *
+ * A freshly-created ref_iterator doesn't yet point at a reference. To
+ * advance the iterator, call ref_iterator_advance(). If successful,
+ * this sets the iterator's refname, oid, and flags fields to describe
+ * the next reference and returns ITER_OK. The data pointed at by
+ * refname and oid belong to the iterator; if you want to retain them
+ * after calling ref_iterator_advance() again or calling
+ * ref_iterator_free(), you must make a copy. When the iteration has
+ * been exhausted, ref_iterator_advance() releases any resources
+ * associated with the iteration, frees the ref_iterator object, and
+ * returns ITER_DONE. If you want to abort the iteration early, call
+ * ref_iterator_free(), which also frees the ref_iterator object and
+ * any associated resources. If there was an internal error advancing
+ * to the next entry, ref_iterator_advance() aborts the iteration,
+ * frees the ref_iterator, and returns ITER_ERROR.
+ *
+ * The reference currently being looked at can be peeled by calling
+ * ref_iterator_peel(). This function is often faster than peel_ref(),
+ * so it should be preferred when iterating over references.
+ *
+ * Putting it all together, a typical iteration looks like this:
+ *
+ *     int ok;
+ *     struct ref_iterator *iter = ...;
+ *
+ *     while ((ok = ref_iterator_advance(iter)) == ITER_OK) {
+ *             if (want_to_stop_iteration()) {
+ *                     ok = ITER_DONE;
+ *                     break;
+ *             }
+ *
+ *             // Access information about the current reference:
+ *             if (!(iter->flags & REF_ISSYMREF))
+ *                     printf("%s is %s\n", iter->refname, oid_to_hex(iter->oid));
+ *
+ *             // If you need to peel the reference:
+ *             ref_iterator_peel(iter, &oid);
+ *     }
+ *
+ *     if (ok != ITER_DONE)
+ *             handle_error();
+ *     ref_iterator_free(iter);
+ */
+struct ref_iterator;
+
+/*
+ * These flags are passed to refs_ref_iterator_begin() (and do_for_each_ref(),
+ * which feeds it).
+ */
+enum do_for_each_ref_flags {
+	/*
+	 * Include broken references in a do_for_each_ref*() iteration, which
+	 * would normally be omitted. This includes both refs that point to
+	 * missing objects (a true repository corruption), ones with illegal
+	 * names (which we prefer not to expose to callers), as well as
+	 * dangling symbolic refs (i.e., those that point to a non-existent
+	 * ref; this is not a corruption, but as they have no valid oid, we
+	 * omit them from normal iteration results).
+	 */
+	DO_FOR_EACH_INCLUDE_BROKEN = (1 << 0),
+
+	/*
+	 * Only include per-worktree refs in a do_for_each_ref*() iteration.
+	 * Normally this will be used with a files ref_store, since that's
+	 * where all reference backends will presumably store their
+	 * per-worktree refs.
+	 */
+	DO_FOR_EACH_PER_WORKTREE_ONLY = (1 << 1),
+
+	/*
+	 * Omit dangling symrefs from output; this only has an effect with
+	 * INCLUDE_BROKEN, since they are otherwise not included at all.
+	 */
+	DO_FOR_EACH_OMIT_DANGLING_SYMREFS = (1 << 2),
+
+	/*
+	 * Include root refs i.e. HEAD and pseudorefs along with the regular
+	 * refs.
+	 */
+	DO_FOR_EACH_INCLUDE_ROOT_REFS = (1 << 3),
+};
+
+/*
+ * Return an iterator that goes over each reference in `refs` for
+ * which the refname begins with prefix. If trim is non-zero, then
+ * trim that many characters off the beginning of each refname.
+ * The output is ordered by refname.
+ */
+struct ref_iterator *refs_ref_iterator_begin(
+	struct ref_store *refs,
+	const char *prefix, const char **exclude_patterns,
+	int trim, enum do_for_each_ref_flags flags);
+
+/*
+ * Advance the iterator to the first or next item and return ITER_OK.
+ * If the iteration is exhausted, free the resources associated with
+ * the ref_iterator and return ITER_DONE. On errors, free the iterator
+ * resources and return ITER_ERROR. It is a bug to use ref_iterator or
+ * call this function again after it has returned ITER_DONE or
+ * ITER_ERROR.
+ */
+int ref_iterator_advance(struct ref_iterator *ref_iterator);
+
+/*
+ * Seek the iterator to the first reference with the given prefix.
+ * The prefix is matched as a literal string, without regard for path
+ * separators. If prefix is NULL or the empty string, seek the iterator to the
+ * first reference again.
+ *
+ * This function is expected to behave as if a new ref iterator with the same
+ * prefix had been created, but allows reuse of iterators and thus may allow
+ * the backend to optimize. Parameters other than the prefix that have been
+ * passed when creating the iterator will remain unchanged.
+ *
+ * Returns 0 on success, a negative error code otherwise.
+ */
+int ref_iterator_seek(struct ref_iterator *ref_iterator,
+		      const char *prefix);
+
+/*
+ * If possible, peel the reference currently being viewed by the
+ * iterator. Return 0 on success.
+ */
+int ref_iterator_peel(struct ref_iterator *ref_iterator,
+		      struct object_id *peeled);
+
+/* Free the reference iterator and any associated resources. */
+void ref_iterator_free(struct ref_iterator *ref_iterator);
+
+/*
+ * The common backend for the for_each_*ref* functions. Call fn for
+ * each reference in iter. If the iterator itself ever returns
+ * ITER_ERROR, return -1. If fn ever returns a non-zero value, stop
+ * the iteration and return that value. Otherwise, return 0. In any
+ * case, free the iterator when done. This function is basically an
+ * adapter between the callback style of reference iteration and the
+ * iterator style.
+ */
+int do_for_each_ref_iterator(struct ref_iterator *iter,
+			     each_ref_fn fn, void *cb_data);
+
 #endif /* REFS_H */
refs/refs-internal.h +2 -143

169 lines not on this page · raw message

Signature · 1 line
2.49.0
#41 lore raw

PATCH v3 2/4 ref-cache: remove unused function 'find_ref_entry()'

In reply to Karthik Nayak

The 'find_ref_entry' function is no longer used, so remove it.

---
 refs/ref-cache.c | 14 --------------
 refs/ref-cache.h |  7 -------
 2 files changed, 21 deletions(-)
refs/ref-cache.c +0 -14
index c1f1bab1d5..8aaffa8c6b 100644
@@ -194,20 +194,6 @@ static struct ref_dir *find_containing_dir(struct ref_dir *dir,
 	return dir;
 }
 
-struct ref_entry *find_ref_entry(struct ref_dir *dir, const char *refname)
-{
-	int entry_index;
-	struct ref_entry *entry;
-	dir = find_containing_dir(dir, refname);
-	if (!dir)
-		return NULL;
-	entry_index = search_ref_dir(dir, refname, strlen(refname));
-	if (entry_index == -1)
-		return NULL;
-	entry = dir->entries[entry_index];
-	return (entry->flag & REF_DIR) ? NULL : entry;
-}
-
 /*
  * Emit a warning and return true iff ref1 and ref2 have the same name
  * and the same oid. Die if they have the same name but different
refs/ref-cache.h +0 -7
index 5f04e518c3..f635d2d824 100644
@@ -201,13 +201,6 @@ void free_ref_cache(struct ref_cache *cache);
  */
 void add_entry_to_dir(struct ref_dir *dir, struct ref_entry *entry);
 
-/*
- * Find the value entry with the given name in dir, sorting ref_dirs
- * and recursing into subdirectories as necessary.  If the name is not
- * found or it corresponds to a directory entry, return NULL.
- */
-struct ref_entry *find_ref_entry(struct ref_dir *dir, const char *refname);
-
 /*
  * Start iterating over references in `cache`. If `prefix` is
  * specified, only include references whose names start with that
Signature · 1 line
2.49.0
#42 lore raw

PATCH v3 3/4 refs: selectively set prefix in the seek functions

In reply to Karthik Nayak

The ref iterator exposes a ref_iterator_seek() function. The name suggests that this would seek the iterator to a specific reference in some ways similar to how fseek() works for the filesystem.

However, the function actually sets the prefix for refs iteration. So further iteration would only yield references which match the particular prefix. This is a bit confusing.

Let's add a 'flags' field to the function, which when set with the 'REF_ITERATOR_SEEK_SET_PREFIX' flag, will set the prefix for the iteration in-line with the existing behavior. Otherwise, the reference backends will simply seek to the specified reference and clears any previously set prefix. This allows users to start iteration from a specific reference.

In the packed and reftable backend, since references are available in a sorted list, the changes are simply setting the prefix if needed. The changes on the files-backend are a little more involved, since the files backend uses the 'ref-cache' mechanism. We move out the existing logic within cache_ref_iterator_seek() to cache_ref_iterator_set_prefix() which is called when the 'REF_ITERATOR_SEEK_SET_PREFIX' flag is set. We then parse the provided seek string and set the required levels and their indexes to ensure that seeking is possible.

---
 refs.c                  |  6 ++--
 refs.h                  | 29 +++++++++++------
 refs/debug.c            |  7 ++--
 refs/files-backend.c    |  7 ++--
 refs/iterator.c         | 26 ++++++++-------
 refs/packed-backend.c   | 17 ++++++----
 refs/ref-cache.c        | 85 ++++++++++++++++++++++++++++++++++++++++++++++---
 refs/refs-internal.h    |  7 ++--
 refs/reftable-backend.c | 21 ++++++++----
 9 files changed, 155 insertions(+), 50 deletions(-)
refs.c +3 -3
index dce5c49ca2..243e6898b8 100644
@@ -2666,12 +2666,12 @@ enum ref_transaction_error refs_verify_refnames_available(struct ref_store *refs
 		if (!initial_transaction) {
 			int ok;
 
-			if (!iter) {
+			if (!iter)
 				iter = refs_ref_iterator_begin(refs, dirname.buf, NULL, 0,
 							       DO_FOR_EACH_INCLUDE_BROKEN);
-			} else if (ref_iterator_seek(iter, dirname.buf) < 0) {
+			else if (ref_iterator_seek(iter, dirname.buf,
+						   REF_ITERATOR_SEEK_SET_PREFIX) < 0)
 				goto cleanup;
-			}
 
 			while ((ok = ref_iterator_advance(iter)) == ITER_OK) {
 				if (skip &&
refs.h +20 -9
index 7c21aaef3d..7852ad36f3 100644
@@ -1299,21 +1299,32 @@ struct ref_iterator *refs_ref_iterator_begin(
  */
 int ref_iterator_advance(struct ref_iterator *ref_iterator);
 
+enum ref_iterator_seek_flag {
+	/*
+	 * Also set the seek pattern as a prefix for iteration. This ensures
+	 * that only references which match the prefix are yielded.
+	 */
+	REF_ITERATOR_SEEK_SET_PREFIX = (1 << 0),
+};
+
 /*
- * Seek the iterator to the first reference with the given prefix.
- * The prefix is matched as a literal string, without regard for path
- * separators. If prefix is NULL or the empty string, seek the iterator to the
+ * Seek the iterator to the first reference matching the given seek string.
+ * The seek string is matched as a literal string, without regard for path
+ * separators. If seek is NULL or the empty string, seek the iterator to the
  * first reference again.
  *
- * This function is expected to behave as if a new ref iterator with the same
- * prefix had been created, but allows reuse of iterators and thus may allow
- * the backend to optimize. Parameters other than the prefix that have been
- * passed when creating the iterator will remain unchanged.
+ * This function is expected to behave as if a new ref iterator has been
+ * created, but allows reuse of existing iterators for optimization.
+ *
+ * When the REF_ITERATOR_SEEK_SET_PREFIX flag is set, the iterator's prefix is
+ * updated to match the seek string, affecting all subsequent iterations. If
+ * not, the iterator seeks to the specified reference and clears any previously
+ * set prefix.
  *
  * Returns 0 on success, a negative error code otherwise.
  */
-int ref_iterator_seek(struct ref_iterator *ref_iterator,
-		      const char *prefix);
+int ref_iterator_seek(struct ref_iterator *ref_iterator, const char *seek,
+		      unsigned int flags);
 
 /*
  * If possible, peel the reference currently being viewed by the
refs/debug.c +4 -3
index 485e3079d7..2ed8cff2aa 100644
@@ -170,12 +170,13 @@ static int debug_ref_iterator_advance(struct ref_iterator *ref_iterator)
 }
 
 static int debug_ref_iterator_seek(struct ref_iterator *ref_iterator,
-				   const char *prefix)
+				   const char *seek, unsigned int flags)
 {
 	struct debug_ref_iterator *diter =
 		(struct debug_ref_iterator *)ref_iterator;
-	int res = diter->iter->vtable->seek(diter->iter, prefix);
-	trace_printf_key(&trace_refs, "iterator_seek: %s: %d\n", prefix ? prefix : "", res);
+	int res = diter->iter->vtable->seek(diter->iter, seek, flags);
+	trace_printf_key(&trace_refs, "iterator_seek: %s flags: %d: %d\n",
+			 seek ? seek : "", flags, res);
 	return res;
 }
 
refs/files-backend.c +4 -3
index bf6f89b1d1..0e63013319 100644
@@ -929,11 +929,11 @@ static int files_ref_iterator_advance(struct ref_iterator *ref_iterator)
 }
 
 static int files_ref_iterator_seek(struct ref_iterator *ref_iterator,
-				   const char *prefix)
+				   const char *seek, unsigned int flags)
 {
 	struct files_ref_iterator *iter =
 		(struct files_ref_iterator *)ref_iterator;
-	return ref_iterator_seek(iter->iter0, prefix);
+	return ref_iterator_seek(iter->iter0, seek, flags);
 }
 
 static int files_ref_iterator_peel(struct ref_iterator *ref_iterator,
@@ -2316,7 +2316,8 @@ static int files_reflog_iterator_advance(struct ref_iterator *ref_iterator)
 }
 
 static int files_reflog_iterator_seek(struct ref_iterator *ref_iterator UNUSED,
-				      const char *prefix UNUSED)
+				      const char *seek UNUSED,
+				      unsigned int flags UNUSED)
 {
 	BUG("ref_iterator_seek() called for reflog_iterator");
 }
refs/iterator.c +15 -11
index 766d96e795..f2364bd6e7 100644
@@ -15,10 +15,10 @@ int ref_iterator_advance(struct ref_iterator *ref_iterator)
 	return ref_iterator->vtable->advance(ref_iterator);
 }
 
-int ref_iterator_seek(struct ref_iterator *ref_iterator,
-		      const char *prefix)
+int ref_iterator_seek(struct ref_iterator *ref_iterator, const char *seek,
+		      unsigned int flags)
 {
-	return ref_iterator->vtable->seek(ref_iterator, prefix);
+	return ref_iterator->vtable->seek(ref_iterator, seek, flags);
 }
 
 int ref_iterator_peel(struct ref_iterator *ref_iterator,
@@ -57,7 +57,8 @@ static int empty_ref_iterator_advance(struct ref_iterator *ref_iterator UNUSED)
 }
 
 static int empty_ref_iterator_seek(struct ref_iterator *ref_iterator UNUSED,
-				   const char *prefix UNUSED)
+				   const char *seek UNUSED,
+				   unsigned int flags UNUSED)
 {
 	return 0;
 }
@@ -224,7 +225,7 @@ static int merge_ref_iterator_advance(struct ref_iterator *ref_iterator)
 }
 
 static int merge_ref_iterator_seek(struct ref_iterator *ref_iterator,
-				   const char *prefix)
+				   const char *seek, unsigned int flags)
 {
 	struct merge_ref_iterator *iter =
 		(struct merge_ref_iterator *)ref_iterator;
@@ -234,11 +235,11 @@ static int merge_ref_iterator_seek(struct ref_iterator *ref_iterator,
 	iter->iter0 = iter->iter0_owned;
 	iter->iter1 = iter->iter1_owned;
 
-	ret = ref_iterator_seek(iter->iter0, prefix);
+	ret = ref_iterator_seek(iter->iter0, seek, flags);
 	if (ret < 0)
 		return ret;
 
-	ret = ref_iterator_seek(iter->iter1, prefix);
+	ret = ref_iterator_seek(iter->iter1, seek, flags);
 	if (ret < 0)
 		return ret;
 
@@ -407,13 +408,16 @@ static int prefix_ref_iterator_advance(struct ref_iterator *ref_iterator)
 }
 
 static int prefix_ref_iterator_seek(struct ref_iterator *ref_iterator,
-				    const char *prefix)
+				    const char *seek, unsigned int flags)
 {
 	struct prefix_ref_iterator *iter =
 		(struct prefix_ref_iterator *)ref_iterator;
-	free(iter->prefix);
-	iter->prefix = xstrdup_or_null(prefix);
-	return ref_iterator_seek(iter->iter0, prefix);
+
+	if (flags & REF_ITERATOR_SEEK_SET_PREFIX) {
+		free(iter->prefix);
+		iter->prefix = xstrdup_or_null(seek);
+	}
+	return ref_iterator_seek(iter->iter0, seek, flags);
 }
 
 static int prefix_ref_iterator_peel(struct ref_iterator *ref_iterator,
refs/packed-backend.c +11 -6
index 7fd73a0e6d..11a363d246 100644
@@ -1004,19 +1004,23 @@ static int packed_ref_iterator_advance(struct ref_iterator *ref_iterator)
 }
 
 static int packed_ref_iterator_seek(struct ref_iterator *ref_iterator,
-				    const char *prefix)
+				    const char *seek, unsigned int flags)
 {
 	struct packed_ref_iterator *iter =
 		(struct packed_ref_iterator *)ref_iterator;
 	const char *start;
 
-	if (prefix && *prefix)
-		start = find_reference_location(iter->snapshot, prefix, 0);
+	if (seek && *seek)
+		start = find_reference_location(iter->snapshot, seek, 0);
 	else
 		start = iter->snapshot->start;
 
-	free(iter->prefix);
-	iter->prefix = xstrdup_or_null(prefix);
+	/* Unset any previously set prefix */
+	FREE_AND_NULL(iter->prefix);
+
+	if (flags & REF_ITERATOR_SEEK_SET_PREFIX)
+		iter->prefix = xstrdup_or_null(seek);
+
 	iter->pos = start;
 	iter->eof = iter->snapshot->eof;
 
@@ -1194,7 +1198,8 @@ static struct ref_iterator *packed_ref_iterator_begin(
 	iter->repo = ref_store->repo;
 	iter->flags = flags;
 
-	if (packed_ref_iterator_seek(&iter->base, prefix) < 0) {
+	if (packed_ref_iterator_seek(&iter->base, prefix,
+				     REF_ITERATOR_SEEK_SET_PREFIX) < 0) {
 		ref_iterator_free(&iter->base);
 		return NULL;
 	}
refs/ref-cache.c +80 -5

104 lines not on this page · raw message

refs/refs-internal.h +4 -3
index 03f5df04d5..6376a3b379 100644
@@ -353,11 +353,12 @@ void base_ref_iterator_init(struct ref_iterator *iter,
 typedef int ref_iterator_advance_fn(struct ref_iterator *ref_iterator);
 
 /*
- * Seek the iterator to the first reference matching the given prefix. Should
- * behave the same as if a new iterator was created with the same prefix.
+ * Seek the iterator to the first matching reference. If set_prefix is set,
+ * it would behave the same as if a new iterator was created with the same
+ * prefix.
  */
 typedef int ref_iterator_seek_fn(struct ref_iterator *ref_iterator,
-				 const char *prefix);
+				 const char *seek, unsigned int flags);
 
 /*
  * Peels the current ref, returning 0 for success or -1 for failure.
refs/reftable-backend.c +14 -7
index 4c3817f4ec..d627221b65 100644
@@ -719,15 +719,20 @@ static int reftable_ref_iterator_advance(struct ref_iterator *ref_iterator)
 }
 
 static int reftable_ref_iterator_seek(struct ref_iterator *ref_iterator,
-				      const char *prefix)
+				      const char *seek, unsigned int flags)
 {
 	struct reftable_ref_iterator *iter =
 		(struct reftable_ref_iterator *)ref_iterator;
 
-	free(iter->prefix);
-	iter->prefix = xstrdup_or_null(prefix);
-	iter->prefix_len = prefix ? strlen(prefix) : 0;
-	iter->err = reftable_iterator_seek_ref(&iter->iter, prefix);
+	/* Unset any previously set prefix */
+	FREE_AND_NULL(iter->prefix);
+	iter->prefix_len = 0;
+
+	if (flags & REF_ITERATOR_SEEK_SET_PREFIX) {
+		iter->prefix = xstrdup_or_null(seek);
+		iter->prefix_len = seek ? strlen(seek) : 0;
+	}
+	iter->err = reftable_iterator_seek_ref(&iter->iter, seek);
 
 	return iter->err;
 }
@@ -839,7 +844,8 @@ static struct reftable_ref_iterator *ref_iterator_for_stack(struct reftable_ref_
 	if (ret)
 		goto done;
 
-	ret = reftable_ref_iterator_seek(&iter->base, prefix);
+	ret = reftable_ref_iterator_seek(&iter->base, prefix,
+					 REF_ITERATOR_SEEK_SET_PREFIX);
 	if (ret)
 		goto done;
 
@@ -2042,7 +2048,8 @@ static int reftable_reflog_iterator_advance(struct ref_iterator *ref_iterator)
 }
 
 static int reftable_reflog_iterator_seek(struct ref_iterator *ref_iterator UNUSED,
-					 const char *prefix UNUSED)
+					 const char *seek UNUSED,
+					 unsigned int flags UNUSED)
 {
 	BUG("reftable reflog iterator cannot be seeked");
 	return -1;
Signature · 1 line
2.49.0
#43 lore raw

Re: [PATCH v3 3/4] refs: selectively set prefix in the seek functions

Quoted 6 lines · Karthik Nayak
refs.h +0 -0
index 7c21aaef3d..7852ad36f3 100644
@@ -1299,21 +1299,32 @@ struct ref_iterator *refs_ref_iterator_begin(
  */

refs.h struct ref_iterator *refs_ref_iterator_begin( · line 1309

Quoted 10 lines
refs.h +8 -0
 int ref_iterator_advance(struct ref_iterator *ref_iterator);
 
+enum ref_iterator_seek_flag {
+	/*
+	 * Also set the seek pattern as a prefix for iteration. This ensures
+	 * that only references which match the prefix are yielded.
+	 */
+	REF_ITERATOR_SEEK_SET_PREFIX = (1 << 0),
+};
+

Nit: I think it's a tiny bit confusing that the documentation of this enum is split up across here and the doc of ref_iterator_seek(). I think it would be sensible to move the last paragraph of the function over here so that the whole behaviour of the enum is explained in a single place.

Quoted 17 lines
refs.h +7 -7
 /*
- * Seek the iterator to the first reference with the given prefix.
- * The prefix is matched as a literal string, without regard for path
- * separators. If prefix is NULL or the empty string, seek the iterator to the
+ * Seek the iterator to the first reference matching the given seek string.
+ * The seek string is matched as a literal string, without regard for path
+ * separators. If seek is NULL or the empty string, seek the iterator to the
  * first reference again.
  *
- * This function is expected to behave as if a new ref iterator with the same
- * prefix had been created, but allows reuse of iterators and thus may allow
- * the backend to optimize. Parameters other than the prefix that have been
- * passed when creating the iterator will remain unchanged.
+ * This function is expected to behave as if a new ref iterator has been
+ * created, but allows reuse of existing iterators for optimization.
+ *
+ * When the REF_ITERATOR_SEEK_SET_PREFIX flag is set, the iterator's prefix is

refs.h struct ref_iterator *refs_ref_iterator_begin( · line 1327

Quoted 10 lines
refs.h +5 -2
+ * updated to match the seek string, affecting all subsequent iterations. If
+ * not, the iterator seeks to the specified reference and clears any previously
+ * set prefix.
  *
  * Returns 0 on success, a negative error code otherwise.
  */
-int ref_iterator_seek(struct ref_iterator *ref_iterator,
-		      const char *prefix);
+int ref_iterator_seek(struct ref_iterator *ref_iterator, const char *seek,
+		      unsigned int flags);

Another tiny nit: instead of calling the variable seek we can just call it refname. That might give a bit more of a hint what you're actually seeking for.

But other than that I'm happy with the new behaviour, where we are now consistently either setting or resetting the prefix depending on whether or not the caller set the flag.

Quoted 22 lines
refs/iterator.c +7 -4
index 766d96e795..f2364bd6e7 100644
@@ -407,13 +408,16 @@ static int prefix_ref_iterator_advance(struct ref_iterator *ref_iterator)
 }
 
 static int prefix_ref_iterator_seek(struct ref_iterator *ref_iterator,
-				    const char *prefix)
+				    const char *seek, unsigned int flags)
 {
 	struct prefix_ref_iterator *iter =
 		(struct prefix_ref_iterator *)ref_iterator;
-	free(iter->prefix);
-	iter->prefix = xstrdup_or_null(prefix);
-	return ref_iterator_seek(iter->iter0, prefix);
+
+	if (flags & REF_ITERATOR_SEEK_SET_PREFIX) {
+		free(iter->prefix);
+		iter->prefix = xstrdup_or_null(seek);
+	}
+	return ref_iterator_se
#44 lore raw

Re: [PATCH v3 3/4] refs: selectively set prefix in the seek functions

Quoted 9 lines · Patrick Steinhardt
Quoted 8 lines · Karthik Nayak
refs.h +0 -0
index 7c21aaef3d..7852ad36f3 100644
@@ -1299,21 +1299,32 @@ struct ref_iterator *refs_ref_iterator_begin(
  */
 int ref_iterator_advance(struct ref_iterator *ref_iterator);
 
Quoted 15 lines

refs.h struct ref_iterator *refs_ref_iterator_begin( · line 1309

Quoted 8 lines
refs.h +8 -0
+enum ref_iterator_seek_flag {
+	/*
+	 * Also set the seek pattern as a prefix for iteration. This ensures
+	 * that only references which match the prefix are yielded.
+	 */
+	REF_ITERATOR_SEEK_SET_PREFIX = (1 << 0),
+};
+

Nit: I think it's a tiny bit confusing that the documentation of this enum is split up across here and the doc of ref_iterator_seek(). I think it would be sensible to move the last paragraph of the function over here so that the whole behaviour of the enum is explained in a single place.

Yeah I think that makes sense.

Quoted 23 lines
Quoted 23 lines
refs.h +10 -7
 /*
- * Seek the iterator to the first reference with the given prefix.
- * The prefix is matched as a literal string, without regard for path
- * separators. If prefix is NULL or the empty string, seek the iterator to the
+ * Seek the iterator to the first reference matching the given seek string.
+ * The seek string is matched as a literal string, without regard for path
+ * separators. If seek is NULL or the empty string, seek the iterator to the
  * first reference again.
  *
- * This function is expected to behave as if a new ref iterator with the same
- * prefix had been created, but allows reuse of iterators and thus may allow
- * the backend to optimize. Parameters other than the prefix that have been
- * passed when creating the iterator will remain unchanged.
+ * This function is expected to behave as if a new ref iterator has been
+ * created, but allows reuse of existing iterators for optimization.
+ *
+ * When the REF_ITERATOR_SEEK_SET_PREFIX flag is set, the iterator's prefix is
+ * updated to match the seek string, affecting all subsequent iterations. If
+ * not, the iterator seeks to the specified reference and clears any previously
+ * set prefix.
  *
  * Returns 0 on success, a negative error code otherwise.
  */
Quoted 9 lines

refs.h struct ref_iterator *refs_ref_iterator_begin( · line 1327

Quoted 4 lines
refs.h +2 -2
-int ref_iterator_seek(struct ref_iterator *ref_iterator,
-		      const char *prefix);
+int ref_iterator_seek(struct ref_iterator *ref_iterator, const char *seek,
+		      unsigned int flags);

Another tiny nit: instead of calling the variable seek we can just call it refname. That might give a bit more of a hint what you're actually seeking for.

Fair enough, let me change that.

Quoted 4 lines

But other than that I'm happy with the new behaviour, where we are now consistently either setting or resetting the prefix depending on whether or not the caller set the flag.

Thanks for the review!

Quoted 22 lines
Quoted 22 lines
refs/iterator.c +7 -4
index 766d96e795..f2364bd6e7 100644
@@ -407,13 +408,16 @@ static int prefix_ref_iterator_advance(struct ref_iterator *ref_iterator)
 }
 
 static int prefix_ref_iterator_seek(struct ref_iterator *ref_iterator,
-				    const char *prefix)
+				    const char *seek, unsigned int flags)
 {
 	struct prefix_ref_iterator *iter =
 		(struct prefix_ref_iterator *)ref_iterator;
-	free(iter->prefix);
-	iter->prefix = xstrdup_or_null(prefix);
-	return ref_iterator_seek(iter->iter0, prefix);
+
+	if (flags & REF_ITERATOR_SEEK_SET_PREFIX) {
+		free(iter->prefix);
+		iter->prefix = xstrdup_or_null(seek);
+	}
+	return ref_iterator_se
#45 lore raw

Re: [PATCH v3 3/4] refs: selectively set prefix in the seek functions

In reply to Patrick Steinhardt
Quoted 16 lines · Patrick Steinhardt

refs.h struct ref_iterator *refs_ref_iterator_begin( · line 1327

Quoted 11 lines

+ * When the REF_ITERATOR_SEEK_SET_PREFIX flag is set, the iterator's prefix is
+ * updated to match the seek string, affecting all subsequent iterations. If
+ * not, the iterator seeks to the specified reference and clears any previously
+ * set prefix.
*
* Returns 0 on success, a negative error code otherwise.
*/

refs.h +2 -2
-int ref_iterator_seek(struct ref_iterator *ref_iterator,
-		      const char *prefix);
+int ref_iterator_seek(struct ref_iterator *ref_iterator, const char *seek,
+		      unsigned int flags);

Another tiny nit: instead of calling the variable seek we can just call it refname. That might give a bit more of a hint what you're actually seeking for.

Quoted 3 lines

But other than that I'm happy with the new behaviour, where we are now consistently either setting or resetting the prefix depending on whether or not the caller set the flag.

I am not sure. The way the "prefix" is used, if I understand correctly, is

- it is set by iterator-begin, typically to the area to iterate over (e.g. "refs/heads/" for iterating over branches) in the for_each_ref_*() family of helpers, and internally we seek to that area (skipping anything that come strictly before "refs/heads/" for example).

- iterator-advance looks at it and decides we are done when the iterator points beyond that prefix

So if you are iterating inside "refs/heads/" hierarchy and seek to "refs/heads/m", don't you still want to stop when you step outside "refs/heads/" by keeping the original prefix, instead of unsetting the prefix to empty? A postimage of this patch for packed backend (picked at random) reads like this:

        static int packed_ref_iterator_seek(struct ref_iterator *ref_iterator,
                                            const char *seek, unsigned int flags)
        {
                struct packed_ref_iterator *iter =
                        (struct packed_ref_iterator *)ref_iterator;
                const char *start;

                if (seek && *seek)
                        start = find_reference_location(iter->snapshot, seek, 0);
                else
                        start = iter->snapshot->start;

                /* Unset any previously set prefix */
                FREE_AND_NULL(iter->prefix);

                if (flags & REF_ITERATOR_SEEK_SET_PREFIX)
                        iter->prefix = xstrdup_or_null(seek);

so after (true) seeking that does not have the SET_PREFIX flag on, wouldn't our iterator-advance run through the end since it no longer is aware of where to stop?

Thanks.

#46 lore raw

Re: [PATCH v3 3/4] refs: selectively set prefix in the seek functions

Quoted 24 lines · Junio C Hamano
Quoted 16 lines · Patrick Steinhardt

refs.h struct ref_iterator *refs_ref_iterator_begin( · line 1327

Quoted 11 lines

+ * When the REF_ITERATOR_SEEK_SET_PREFIX flag is set, the iterator's prefix is
+ * updated to match the seek string, affecting all subsequent iterations. If
+ * not, the iterator seeks to the specified reference and clears any previously
+ * set prefix.
*
* Returns 0 on success, a negative error code otherwise.
*/

refs.h +2 -2
-int ref_iterator_seek(struct ref_iterator *ref_iterator,
-		      const char *prefix);
+int ref_iterator_seek(struct ref_iterator *ref_iterator, const char *seek,
+		      unsigned int flags);

Another tiny nit: instead of calling the variable seek we can just call it refname. That might give a bit more of a hint what you're actually seeking for.

Quoted 3 lines

But other than that I'm happy with the new behaviour, where we are now consistently either setting or resetting the prefix depending on whether or not the caller set the flag.

I am not sure. The way the "prefix" is used, if I understand correctly, is

Quoted 9 lines

- it is set by iterator-begin, typically to the area to iterate over (e.g. "refs/heads/" for iterating over branches) in the for_each_ref_*() family of helpers, and internally we seek to that area (skipping anything that come strictly before "refs/heads/" for example).

- iterator-advance looks at it and decides we are done when the iterator points beyond that prefix

That's right.

Quoted 24 lines

So if you are iterating inside "refs/heads/" hierarchy and seek to "refs/heads/m", don't you still want to stop when you step outside "refs/heads/" by keeping the original prefix, instead of unsetting the prefix to empty? A postimage of this patch for packed backend (picked at random) reads like this:

        static int packed_ref_iterator_seek(struct ref_iterator *ref_iterator,
                                            const char *seek, unsigned int flags)
        {
                struct packed_ref_iterator *iter =
                        (struct packed_ref_iterator *)ref_iterator;
                const char *start;

                if (seek && *seek)
                        start = find_reference_location(iter->snapshot, seek, 0);
                else
                        start = iter->snapshot->start;

                /* Unset any previously set prefix */
                FREE_AND_NULL(iter->prefix);

                if (flags & REF_ITERATOR_SEEK_SET_PREFIX)
                        iter->prefix = xstrdup_or_null(seek);
Quoted 4 lines

so after (true) seeking that does not have the SET_PREFIX flag on, wouldn't our iterator-advance run through the end since it no longer is aware of where to stop?

That's also right and that is indeed the intention. We're trying to make the actions more intentional.

So if a user sets a 'prefix' for the iterator, all previous state of the iterator is reset. So, the same function for seeking an iterator should also have the same side-effect of resetting the previous state.

There could be a usecase where we add support for keeping the prefix, while also seeking the iterator. That would be an explicit change (perhaps with a corresponding flag?) that we'd have to build, add tests for and call out. Until then, we explicitly reset the state whenever a user calls 'ref_iterator_seek()', they can be sure that any previous state is reset.

Quoted 1 line

Thanks.

#47 lore raw

Re: [PATCH v3 3/4] refs: selectively set prefix in the seek functions

Quoted 11 lines · Karthik Nayak
Quoted 4 lines

so after (true) seeking that does not have the SET_PREFIX flag on, wouldn't our iterator-advance run through the end since it no longer is aware of where to stop?

That's also right and that is indeed the intention. We're trying to make the actions more intentional.

So if a user sets a 'prefix' for the iterator, all previous state of the iterator is reset. So, the same function for seeking an iterator should also have the same side-effect of resetting the previous state.

Perhaps we have different definition of "previous state" in mind? So let's imagine an iterator is walking over all branches (i.e. the prefix is set to refs/heads/, to allow it to stop once it steps outside refs/heads/ and moves over to refs/imerge). It starts iterating and I see branches whose name sorts early in alphabetical order. I tell it to seek to refs/heads/master and keep iterating.

Wouldn't it be a lot more natural if it still stops iterating after it finishes showing the last branch, iow, a ref in refs/heads/ hierarchy? In other words, I am not sure why ...

Quoted 2 lines

There could be a usecase where we add support for keeping the prefix, while also seeking the iterator. That would be an explicit change

... that is the optional and unimplemented feature, not the other way around. Is it just the ease of implementation?

#48 lore raw

Re: [PATCH v3 3/4] refs: selectively set prefix in the seek functions

Quoted 14 lines · Junio C Hamano
Quoted 11 lines · Karthik Nayak
Quoted 4 lines

so after (true) seeking that does not have the SET_PREFIX flag on, wouldn't our iterator-advance run through the end since it no longer is aware of where to stop?

That's also right and that is indeed the intention. We're trying to make the actions more intentional.

So if a user sets a 'prefix' for the iterator, all previous state of the iterator is reset. So, the same function for seeking an iterator should also have the same side-effect of resetting the previous state.

Quoted 7 lines

Perhaps we have different definition of "previous state" in mind? So let's imagine an iterator is walking over all branches (i.e. the prefix is set to refs/heads/, to allow it to stop once it steps outside refs/heads/ and moves over to refs/imerge). It starts iterating and I see branches whose name sorts early in alphabetical order. I tell it to seek to refs/heads/master and keep iterating.

I get what you're saying and indeed that would be natural. Let me draw another example to draw the contrast.

Let's say a user is iterating with a prefix set to 'refs/heads/', this would iterate over all the refs with that prefix. But mid-way the user realizes that they only care about 'refs/heads/feature/' prefix and they ask the iterator to set that as the prefix.

In such a situation, the iterator seeks to 'refs/heads/feature/' and will only yield references with that prefix. In short, the previous prefix state was reset.

So to avoid the two scenarios:

1. Only seek the iterator but maintain prefix
2. Seek and set new prefix, loosing old prefix

Where one resets the prefix while the other doesn't. We make it explicit and say, whenever 'ref_iterator_seek' is called, any set prefix is reset. I do see the other way around too, where prefix isn't treated as previous state.

What I was trying to argue for, was that, there could be a situation like what you mentioned, where a user might want to retain a prefix, this should be an explicit requirement which not implemented in this series. So as of this series, you cannot set a prefix and then seek and expect to retain the prefix.

Quoted 9 lines

Wouldn't it be a lot more natural if it still stops iterating after it finishes showing the last branch, iow, a ref in refs/heads/ hierarchy? In other words, I am not sure why ...

Quoted 2 lines

There could be a usecase where we add support for keeping the prefix, while also seeking the iterator. That would be an explicit change

... that is the optional and unimplemented feature, not the other way around. Is it just the ease of implementation?

This series did start out that way around, so ease of implementation isn't it. It was more of a side-effect of not clearing state. But I would be more comfortable if this wasn't a side-effect but rather a conscious choice with tests and adequate documentation.

#49 lore raw

Re: [PATCH v3 3/4] refs: selectively set prefix in the seek functions

Quoted 4 lines · Karthik Nayak

Let's say a user is iterating with a prefix set to 'refs/heads/', this would iterate over all the refs with that prefix. But mid-way the user realizes that they only care about 'refs/heads/feature/' prefix and they ask the iterator to set that as the prefix.

But is that user who changes their mind in the middle "seeking"?

It is more like "I am abandoning the enumeration I started earlier over refs/heads/, and I want a different enumeration over refs/heads/feature/, but because I know the implementation detail that abandoning an iterator and creating another is more expensive, let me reuse the one in use to repurpose it".

I wouldn't call it "seeking"; it sounds more like "resetting".

#50 lore raw

Re: [PATCH v3 3/4] refs: selectively set prefix in the seek functions

In reply to Karthik Nayak
Quoted 8 lines · Karthik Nayak

Let's say a user is iterating with a prefix set to 'refs/heads/', this would iterate over all the refs with that prefix. But mid-way the user realizes that they only care about 'refs/heads/feature/' prefix and they ask the iterator to set that as the prefix.

In such a situation, the iterator seeks to 'refs/heads/feature/' and will only yield references with that prefix. In short, the previous prefix state was reset.

Yes, even though I wouldn't call such an operation "seek", "Ah, I do not need the entire refs/heads/ walked, only refs/heads/feature/ is enough" is an operation mode that makes sense.

But not for paging, though.

If your web application is showing all branches, one pageful at a time, and the first page ended at refs/heads/feature/something and you ended up "seeking" to refs/heads/feature/ to start the second page, you do not want your second page to end when the iteration goes out of refs/heads/feature/ hierarchy, no?

It seems to me that the root cause of the confusion is because prefix, which is to let iteration finish way before the data runs out (instead finish when the iteration steps out of a given subhierarchy denoted by the prefix), is somehow abused as the current position of the cursor. Shouldn't they be two separate concepts? The cursor needs to fall within the prefix while the iterator is active, so they are not two totally independent things, but prefix is pretty much static while the cursor position is very dynamic.

Quoted 2 lines

This series did start out that way around, so ease of implementation isn't it. It was more of a side-effect of not clearing state.

I am even more worried about usability and correctness aspect of what was described here now. After seeking to refs/heads/feature/, do we continue to iterate and step out of refs/heads/feature/ hierarchy or can we cut off a particular page that started with a ref within refs/heads/feature/ subhierarchy when we exhaust refs in refs/heads/feature/ and have to wait for getting asked for the next page before we show refs/heads/gsomething that is outside refs/heads/feature/ and sorts after? The "I reset to iterate over refs/heads/feature/ because the entire refs/heads/ is not what I care about" example makes me worried about this.

Thanks.

#51 lore raw

Re: [PATCH v3 3/4] refs: selectively set prefix in the seek functions

Quoted 15 lines · Junio C Hamano
Quoted 8 lines · Karthik Nayak

Let's say a user is iterating with a prefix set to 'refs/heads/', this would iterate over all the refs with that prefix. But mid-way the user realizes that they only care about 'refs/heads/feature/' prefix and they ask the iterator to set that as the prefix.

In such a situation, the iterator seeks to 'refs/heads/feature/' and will only yield references with that prefix. In short, the previous prefix state was reset.

Yes, even though I wouldn't call such an operation "seek", "Ah, I do not need the entire refs/heads/ walked, only refs/heads/feature/ is enough" is an operation mode that makes sense.

Quoted 8 lines

But not for paging, though.

If your web application is showing all branches, one pageful at a time, and the first page ended at refs/heads/feature/something and you ended up "seeking" to refs/heads/feature/ to start the second page, you do not want your second page to end when the iteration goes out of refs/heads/feature/ hierarchy, no?

Yup and this (we show all references beyond the seek) is the current implementation. I was talking about the internal implementation of 'refs_iteration_seek()' which is the function used for seek and setting the prefix.

To clarify, this is the current implementation:

$ git for-each-ref
6f4b58c4968eb82277cf5b1cf8775117e5b83de0 commit	refs/heads/bar
6f4b58c4968eb82277cf5b1cf8775117e5b83de0 commit	refs/heads/feature/x
6f4b58c4968eb82277cf5b1cf8775117e5b83de0 commit	refs/heads/feature/y
6f4b58c4968eb82277cf5b1cf8775117e5b83de0 commit	refs/heads/foo
6f4b58c4968eb82277cf5b1cf8775117e5b83de0 commit	refs/heads/goo/x
6f4b58c4968eb82277cf5b1cf8775117e5b83de0 commit	refs/heads/goo/y
6f4b58c4968eb82277cf5b1cf8775117e5b83de0 commit	refs/heads/master
6f4b58c4968eb82277cf5b1cf8775117e5b83de0 commit	refs/tags/tagged/2
6f4b58c4968eb82277cf5b1cf8775117e5b83de0 commit	refs/tags/tagged/3
6f4b58c4968eb82277cf5b1cf8775117e5b83de0 commit	refs/tags/v1
6f4b58c4968eb82277cf5b1cf8775117e5b83de0 commit	refs/tags/v2
$ git for-each-ref --format="%(refname)" --start-after=refs/heads/goo
refs/heads/goo/x
refs/heads/goo/y
refs/heads/master
refs/tags/tagged/2
refs/tags/tagged/3
refs/tags/v1
refs/tags/v2
$ git for-each-ref --format="%(refname)" --start-after=refs/heads/master
refs/tags/tagged/2
refs/tags/tagged/3
refs/tags/v1
refs/tags/v2
$ git for-each-ref --format="%(refname)" --start-after=refs/heads/goo/x
refs/heads/goo/y
refs/heads/master
refs/tags/tagged/2
refs/tags/tagged/3
refs/tags/v1
refs/tags/v2
$ git for-each-ref --format="%(refname)" refs/heads/feature
refs/heads/feature/x
refs/heads/feature/y

You can see we list all references beyond the seek.

Quoted 10 lines

It seems to me that the root cause of the confusion is because prefix, which is to let iteration finish way before the data runs out (instead finish when the iteration steps out of a given subhierarchy denoted by the prefix), is somehow abused as the current position of the cursor. Shouldn't they be two separate concepts? The cursor needs to fall within the prefix while the iterator is active, so they are not two totally independent things, but prefix is pretty much static while the cursor position is very dynamic.

The prefix setup in 'ref_iteration_seek' does two things, let's consider
prefix: 'refs/heads/feature'
1. It sets the cursor to seek to 'refs/heads/feature'
2. It also sets the internal prefix matching to 'refs/heads/feature'

In Contrast seeking via 'ref_iteration_seek' only sets the cursor to 'refs/heads/feature'.

To make this simpler, we've changed 'ref_iteration_seek' to do:
1. seek the cursor to the requested reference
2. Set prefix if the REF_ITERATOR_SEEK_SET_PREFIX is set, and unset the prefix otherwise.

The state reset I was talking about in my previous emails refers to step #2 here, where when no 'REF_ITERATOR_SEEK_SET_PREFIX' is set, we remove any previous prefix set.

Quoted 14 lines
Quoted 2 lines

This series did start out that way around, so ease of implementation isn't it. It was more of a side-effect of not clearing state.

I am even more worried about usability and correctness aspect of what was described here now. After seeking to refs/heads/feature/, do we continue to iterate and step out of refs/heads/feature/ hierarchy or can we cut off a particular page that started with a ref within refs/heads/feature/ subhierarchy when we exhaust refs in refs/heads/feature/ and have to wait for getting asked for the next page before we show refs/heads/gsomething that is outside refs/heads/feature/ and sorts after? The "I reset to iterate over refs/heads/feature/ because the entire refs/heads/ is not what I care about" example makes me worried about this.

Quoted 1 line

Thanks.

I think we're crossing paths and talking different things. I hope the examples above clarify things. The current implementation doesn't support '--start-after' and prefix setting at the same time:

$ git for-each-ref --format="%(refname)"
--start-after=refs/heads/master refs/heads
fatal: cannot use --start-after with patterns

Happy to clarify if this doesn't make sense.

Thanks

#52 lore raw

Re: [PATCH v3 3/4] refs: selectively set prefix in the seek functions

Quoted 5 lines · Karthik Nayak
Quoted 5 lines

Yes, even though I wouldn't call such an operation "seek", "Ah, I do not need the entire refs/heads/ walked, only refs/heads/feature/ is enough" is an operation mode that makes sense.

But not for paging, though.

Actually, you are not using the new value for prefix; you are unsetting prefix to nothing, so you do not stop when you go over the boundary of refs/heads/feature/ hierarchy but will run through to the end, so my worry is unfounded.

Which is good.

But the application has to decide when to stop as "We obtained an iterator in order to list all branches in refs/heads/ and the last round we gave refs/heads/main out. Seek to it and continue the output" will no longer stop after making a callback with refs/heads/zz but will continue yielding non branch refs after that.

Quoted 6 lines

The current implementation doesn't
support '--start-after' and prefix setting at the same time:

$ git for-each-ref --format="%(refname)"
--start-after=refs/heads/master refs/heads
fatal: cannot use --start-after with patterns

Good---in that case, "unsetting prefix to nothing" does not make any difference and cannot introduce any confusing behaviour, as it has to run to the end with or without "--start-after" anyway.

Thanks.

#53 lore raw

PATCH v3 4/4 for-each-ref: introduce a '--start-after' option

In reply to Karthik Nayak

The git-for-each-ref(1) command is used to iterate over references present in a repository. In large repositories with millions of references, it would be optimal to paginate this output such that we can start iteration from a given reference. This would avoid having to iterate over all references from the beginning each time when paginating through results.

The previous commit added 'seek' functionality to the reference backends. Utilize this and expose a '--start-after' option in 'git-for-each-ref(1)'. When used, the reference iteration seeks to the lexicographically next reference and iterates from there onward.

This enables efficient pagination workflows like:

    git for-each-ref --count=100
    git for-each-ref --count=100 --start-after=refs/heads/branch-100
    git for-each-ref --count=100 --start-after=refs/heads/branch-200

Since the reference iterators only allow seeking to a specified marker via the ref_iterator_seek(), we introduce a helper function start_ref_iterator_after(), which seeks to next reference by simply adding (char) 1 to the marker.

We must note that pagination always continues from the provided marker, as such any concurrent reference updates lexicographically behind the marker will not be output. Document the same.

---
 Documentation/git-for-each-ref.adoc |  11 +-
 builtin/for-each-ref.c              |   8 ++
 ref-filter.c                        |  80 +++++++++++----
 ref-filter.h                        |   1 +
 t/t6302-for-each-ref-filter.sh      | 194 ++++++++++++++++++++++++++++++++++++
 5 files changed, 273 insertions(+), 21 deletions(-)
Documentation/git-for-each-ref.adoc +10 -1
index 5ef89fc0fe..e099d1ba7c 100644
@@ -14,7 +14,7 @@ SYNOPSIS
 		   [--points-at=<object>]
 		   [--merged[=<object>]] [--no-merged[=<object>]]
 		   [--contains[=<object>]] [--no-contains[=<object>]]
-		   [--exclude=<pattern> ...]
+		   [--exclude=<pattern> ...] [--start-after=<marker>]
 
 DESCRIPTION
 -----------
@@ -108,6 +108,15 @@ TAB %(refname)`.
 --include-root-refs::
 	List root refs (HEAD and pseudorefs) apart from regular refs.
 
+--start-after::
+    Allows paginating the output by skipping references up to and including the
+    specified marker. When paging, it should be noted that references may be
+    deleted, modified or added between invocations. Output will only yield those
+    references which follow the marker lexicographically. If the marker does not
+    exist, output begins from the first reference that would come after it
+    alphabetically. Cannot be used with general pattern matching or custom
+    sort options.
+
 FIELD NAMES
 -----------
 
builtin/for-each-ref.c +8 -0
index 3d2207ec77..3f21598046 100644
@@ -13,6 +13,7 @@ static char const * const for_each_ref_usage[] = {
 	N_("git for-each-ref [--points-at <object>]"),
 	N_("git for-each-ref [--merged [<commit>]] [--no-merged [<commit>]]"),
 	N_("git for-each-ref [--contains [<commit>]] [--no-contains [<commit>]]"),
+	N_("git for-each-ref [--start-after <marker>]"),
 	NULL
 };
 
@@ -44,6 +45,7 @@ int cmd_for_each_ref(int argc,
 		OPT_GROUP(""),
 		OPT_INTEGER( 0 , "count", &format.array_opts.max_count, N_("show only <n> matched refs")),
 		OPT_STRING(  0 , "format", &format.format, N_("format"), N_("format to use for the output")),
+		OPT_STRING(  0 , "start-after", &filter.start_after, N_("start-start"), N_("start iteration after the provided marker")),
 		OPT__COLOR(&format.use_color, N_("respect format colors")),
 		OPT_REF_FILTER_EXCLUDE(&filter),
 		OPT_REF_SORT(&sorting_options),
@@ -79,6 +81,9 @@ int cmd_for_each_ref(int argc,
 	if (verify_ref_format(&format))
 		usage_with_options(for_each_ref_usage, opts);
 
+	if (filter.start_after && sorting_options.nr > 1)
+		die(_("cannot use --start-after with custom sort options"));
+
 	sorting = ref_sorting_options(&sorting_options);
 	ref_sorting_set_sort_flags_all(sorting, REF_SORTING_ICASE, icase);
 	filter.ignore_case = icase;
@@ -100,6 +105,9 @@ int cmd_for_each_ref(int argc,
 		filter.name_patterns = argv;
 	}
 
+	if (filter.start_after && filter.name_patterns && filter.name_patterns[0])
+		die(_("cannot use --start-after with patterns"));
+
 	if (include_root_refs)
 		flags |= FILTER_REFS_ROOT_REFS | FILTER_REFS_DETACHED_HEAD;
 
ref-filter.c +60 -20

132 lines not on this page · raw message

ref-filter.h +1 -0
index c98c4fbd4c..f22ca94b49 100644
@@ -64,6 +64,7 @@ struct ref_array {
 
 struct ref_filter {
 	const char **name_patterns;
+	const char *start_after;
 	struct strvec exclude;
 	struct oid_array points_at;
 	struct commit_list *with_commit;
t/t6302-for-each-ref-filter.sh +194 -0

198 lines not on this page · raw message

Signature · 1 line
2.49.0
#54 lore raw

Re: [PATCH v3 4/4] for-each-ref: introduce a '--start-after' option

Quoted 12 lines · Karthik Nayak

The git-for-each-ref(1) command is used to iterate over references present in a repository. In large repositories with millions of references, it would be optimal to paginate this output such that we can start iteration from a given reference. This would avoid having to iterate over all references from the beginning each time when paginating through results.

The previous commit added 'seek' functionality to the reference backends. Utilize this and expose a '--start-after' option in 'git-for-each-ref(1)'. When used, the reference iteration seeks to the lexicographically next reference and iterates from there onward.

Quoted 4 lines

This enables efficient pagination workflows like:

    git for-each-ref --count=100
    git for-each-ref --count=100 --start-after=refs/heads/branch-100
    git for-each-ref --count=100 --start-after=refs/heads/branch-200

It is a bit hard to understand how this leads to "efficient pagination" unless the reader is told what the calling script does after the first call before making the second call. It remembers the last output from the Nth call and prepares the N+1th call by using that last output entry.

But that probably belongs to the end-user facing documentation, not in the log message.

Quoted 1 line

+--start-after::

--start-after=<marker>::

Documentation/git-for-each-ref.adoc TAB %(refname)`.

Quoted 6 lines
Documentation/git-for-each-ref.adoc +6 -0
+    Allows paginating the output by skipping references up to and including the
+    specified marker. When paging, it should be noted that references may be
+    deleted, modified or added between invocations. Output will only yield those
+    references which follow the marker lexicographically. If the marker does not
+    exist, output begins from the first reference that would come after it
+    alphabetically.

It is true that the first entry shown would be what would come immediately _after_ the given <marker>, whether the marker does or does not exist. So "If the marker does not exist, output begins..." -> "Output begins ..."

Other than that, looked pretty good to me.

Thanks, will queue.

#55 lore raw

Re: [PATCH v3 4/4] for-each-ref: introduce a '--start-after' option

Quoted 19 lines · Junio C Hamano
Quoted 12 lines · Karthik Nayak

The git-for-each-ref(1) command is used to iterate over references present in a repository. In large repositories with millions of references, it would be optimal to paginate this output such that we can start iteration from a given reference. This would avoid having to iterate over all references from the beginning each time when paginating through results.

The previous commit added 'seek' functionality to the reference backends. Utilize this and expose a '--start-after' option in 'git-for-each-ref(1)'. When used, the reference iteration seeks to the lexicographically next reference and iterates from there onward.

Quoted 4 lines

This enables efficient pagination workflows like:

    git for-each-ref --count=100
    git for-each-ref --count=100 --start-after=refs/heads/branch-100
    git for-each-ref --count=100 --start-after=refs/heads/branch-200
Quoted 9 lines

It is a bit hard to understand how this leads to "efficient pagination" unless the reader is told what the calling script does after the first call before making the second call. It remembers the last output from the Nth call and prepares the N+1th call by using that last output entry.

But that probably belongs to the end-user facing documentation, not in the log message.

I added a small line in the commit message to clarify this. I'm not sure this belongs in the user facing documentation. Mostly I see this in the commit message to explain the intention behind adding the flag. The documentation already mentions how the flag can be used, so we should be good there.

Quoted 4 lines
Quoted 1 line

+--start-after::

--start-after=<marker>::

Oops. Thanks

Quoted 7 lines

Documentation/git-for-each-ref.adoc TAB %(refname)`.

Quoted 6 lines
Documentation/git-for-each-ref.adoc +6 -0
+    Allows paginating the output by skipping references up to and including the
+    specified marker. When paging, it should be noted that references may be
+    deleted, modified or added between invocations. Output will only yield those
+    references which follow the marker lexicographically. If the marker does not
+    exist, output begins from the first reference that would come after it
+    alphabetically.
Quoted 7 lines

It is true that the first entry shown would be what would come immediately _after_ the given <marker>, whether the marker does or does not exist. So "If the marker does not exist, output begins..." -> "Output begins ..."

Other than that, looked pretty good to me.

Good point, will amend this.

Quoted 1 line

Thanks, will queue.

Thank you for the review. I'll add the changes locally and push a new version after a day or two.

#56 lore raw

v4 cover for-each-ref: introduce seeking functionality via '--start-after'

In reply to Karthik Nayak

The git-for-each-ref(1) command is used to iterate over references present in a repository. In large repositories with millions of references, it would be optimal to paginate this output such that we can start iteration from a given reference. This would avoid having to iterate over all references from the beginning each time when paginating through results.

This series adds a '--start-after' option in 'git-for-each-ref(1)'. When used, the reference iteration seeks to first reference following the marker alphabetically. When paging, it should be noted that references may be deleted, modified or added between invocations. Output will only yield those references which follow the marker lexicographically. If the marker does not exist, output begins from the first reference that would come after it alphabetically.

This enables efficient pagination workflows like:

    git for-each-ref --count=100
    git for-each-ref --count=100 --start-after=refs/heads/branch-100
    git for-each-ref --count=100 --start-after=refs/heads/branch-200

To add this functionality, we expose the ref_iterator outside the 'refs/' namespace and modify the ref_iterator_seek() to actually seek to a given reference and only set the prefix when the set_prefix field is set.

On the reftable and packed backend, the changes are simple. But since the files backend uses 'ref-cache' for reference handling, the changes there are a little more involved, since we need to setup the right levels and the indexing.

Initially I was also planning to cleanup all the refs_for_each...() functions in 'refs.h' by simply using the iterator, but this bloated the series. So I've left that for another day.

Changes in v4:
- Patch 3/4: Move around the documentation for the flag and rename the
seek variable to refname.
- Patch 4/4: Cleanup the commit message and also the documentation.
- Link to v3: https://lore.kernel.org/r/20250708-306-git-for-each-ref-pagination-v3-0-8cfba1080be4@gmail.com

Changes in v3:
- Change the working of the command to exclude the marker provided. With
this rename the flag to '--start-after'.
- Extend the documentation to add a note about concurrent modifications
to the reference database.
- Link to v2: https://lore.kernel.org/r/20250704-306-git-for-each-ref-pagination-v2-0-bcde14acdd81@gmail.com

Changes in v2:
- Modify 'ref_iterator_seek()' to take in flags instead of a
'set_prefix' variable. This improves readability, where users would
use the 'REF_ITERATOR_SEEK_SET_PREFIX' instead of simply passing '1'.
- When the set prefix flag isn't usage, reset any previously set prefix.
This ensures that the internal prefix state is always reset whenever
we seek and unifies the behavior between 'ref_iterator_seek' and
'ref_iterator_begin'.
- Don't allow '--skip-until' to be run with '--sort', since the seeking
always takes place before any sorting and this can be confusing.
- Some styling fixes:
- Remove extra newline
- Skip braces around single lined if...else clause
- Add braces around 'if' clause
- Fix indentation
- Link to v1: https://lore.kernel.org/git/20250701-306-git-for-each-ref-pagination-v1-0-4f0ae7c0688f@gmail.com/

---
 Documentation/git-for-each-ref.adoc |  10 +-
 builtin/for-each-ref.c              |   8 ++
 ref-filter.c                        |  80 +++++++++++----
 ref-filter.h                        |   1 +
 refs.c                              |   6 +-
 refs.h                              | 155 ++++++++++++++++++++++++++++
 refs/debug.c                        |   7 +-
 refs/files-backend.c                |   7 +-
 refs/iterator.c                     |  26 +++--
 refs/packed-backend.c               |  17 ++--
 refs/ref-cache.c                    |  99 ++++++++++++++----
 refs/ref-cache.h                    |   7 --
 refs/refs-internal.h                | 152 ++--------------------------
 refs/reftable-backend.c             |  21 ++--
 t/t6302-for-each-ref-filter.sh      | 194 ++++++++++++++++++++++++++++++++++++
 15 files changed, 564 insertions(+), 226 deletions(-)

Karthik Nayak (4):
refs: expose ref_iterator via 'refs.h'
ref-cache: remove unused function 'find_ref_entry()'
refs: selectively set prefix in the seek functions
for-each-ref: introduce a '--start-after' option

Range-diff versus v3:

1:  eed39162f5 = 1:  9e6ecff291 refs: expose `ref_iterator` via 'refs.h'
2:  b9db49d31b = 2:  22f5222e4f ref-cache: remove unused function 'find_ref_entry()'
3:  502e2696fd ! 3:  0e71d8ffd9 refs: selectively set prefix in the seek functions
    @@ refs.h: struct ref_iterator *refs_ref_iterator_begin(
      
     +enum ref_iterator_seek_flag {
     +	/*
    -+	 * Also set the seek pattern as a prefix for iteration. This ensures
    -+	 * that only references which match the prefix are yielded.
    ++	 * When the REF_ITERATOR_SEEK_SET_PREFIX flag is set, the iterator's prefix is
    ++	 * updated to match the provided string, affecting all subsequent iterations. If
    ++	 * not, the iterator seeks to the specified reference and clears any previously
    ++	 * set prefix.
     +	 */
     +	REF_ITERATOR_SEEK_SET_PREFIX = (1 << 0),
     +};
    @@ refs.h: struct ref_iterator *refs_ref_iterator_begin(
     - * passed when creating the iterator will remain unchanged.
     + * This function is expected to behave as if a new ref iterator has been
     + * created, but allows reuse of existing iterators for optimization.
    -+ *
    -+ * When the REF_ITERATOR_SEEK_SET_PREFIX flag is set, the iterator's prefix is
    -+ * updated to match the seek string, affecting all subsequent iterations. If
    -+ * not, the iterator seeks to the specified reference and clears any previously
    -+ * set prefix.
       *
       * Returns 0 on success, a negative error code otherwise.
       */
     -int ref_iterator_seek(struct ref_iterator *ref_iterator,
     -		      const char *prefix);
    -+int ref_iterator_seek(struct ref_iterator *ref_iterator, const char *seek,
    ++int ref_iterator_seek(struct ref_iterator *ref_iterator, const char *refname,
     +		      unsigned int flags);
      
      /*
    @@ refs/debug.c: static int debug_ref_iterator_advance(struct ref_iterator *ref_ite
      
      static int debug_ref_iterator_seek(struct ref_iterator *ref_iterator,
     -				   const char *prefix)
    -+				   const char *seek, unsigned int flags)
    ++				   const char *refname, unsigned int flags)
      {
      	struct debug_ref_iterator *diter =
      		(struct debug_ref_iterator *)ref_iterator;
     -	int res = diter->iter->vtable->seek(diter->iter, prefix);
     -	trace_printf_key(&trace_refs, "iterator_seek: %s: %d\n", prefix ? prefix : "", res);
    -+	int res = diter->iter->vtable->seek(diter->iter, seek, flags);
    ++	int res = diter->iter->vtable->seek(diter->iter, refname, flags);
     +	trace_printf_key(&trace_refs, "iterator_seek: %s flags: %d: %d\n",
    -+			 seek ? seek : "", flags, res);
    ++			 refname ? refname : "", flags, res);
      	return res;
      }
      
    @@ refs/files-backend.c: static int files_ref_iterator_advance(struct ref_iterator
      
      static int files_ref_iterator_seek(struct ref_iterator *ref_iterator,
     -				   const char *prefix)
    -+				   const char *seek, unsigned int flags)
    ++				   const char *refname, unsigned int flags)
      {
      	struct files_ref_iterator *iter =
      		(struct files_ref_iterator *)ref_iterator;
     -	return ref_iterator_seek(iter->iter0, prefix);
    -+	return ref_iterator_seek(iter->iter0, seek, flags);
    ++	return ref_iterator_seek(iter->iter0, refname, flags);
      }
      
      static int files_ref_iterator_peel(struct ref_iterator *ref_iterator,
    @@ refs/files-backend.c: static int files_reflog_iterator_advance(struct ref_iterat
      
      static int files_reflog_iterator_seek(struct ref_iterator *ref_iterator UNUSED,
     -				      const char *prefix UNUSED)
    -+				      const char *seek UNUSED,
    ++				      const char *refname UNUSED,
     +				      unsigned int flags UNUSED)
      {
      	BUG("ref_iterator_seek() called for reflog_iterator");
    @@ refs/iterator.c: int ref_iterator_advance(struct ref_iterator *ref_iterator)
      
     -int ref_iterator_seek(struct ref_iterator *ref_iterator,
     -		      const char *prefix)
    -+int ref_iterator_seek(struct ref_iterator *ref_iterator, const char *seek,
    ++int ref_iterator_seek(struct ref_iterator *ref_iterator, const char *refname,
     +		      unsigned int flags)
      {
     -	return ref_iterator->vtable->seek(ref_iterator, prefix);
    -+	return ref_iterator->vtable->seek(ref_iterator, seek, flags);
    ++	return ref_iterator->vtable->seek(ref_iterator, refname, flags);
      }
      
      int ref_iterator_peel(struct ref_iterator *ref_iterator,
    @@ refs/iterator.c: static int empty_ref_iterator_advance(struct ref_iterator *ref_
      
      static int empty_ref_iterator_seek(struct ref_iterator *ref_iterator UNUSED,
     -				   const char *prefix UNUSED)
    -+				   const char *seek UNUSED,
    ++				   const char *refname UNUSED,
     +				   unsigned int flags UNUSED)
      {
      	return 0;
    @@ refs/iterator.c: static int merge_ref_iterator_advance(struct ref_iterator *ref_
      
      static int merge_ref_iterator_seek(struct ref_iterator *ref_iterator,
     -				   const char *prefix)
    -+				   const char *seek, unsigned int flags)
    ++				   const char *refname, unsigned int flags)
      {
      	struct merge_ref_iterator *iter =
      		(struct merge_ref_iterator *)ref_iterator;
    @@ refs/iterator.c: static int merge_ref_iterator_seek(struct ref_iterator *ref_ite
      	iter->iter1 = iter->iter1_owned;
      
     -	ret = ref_iterator_seek(iter->iter0, prefix);
    -+	ret = ref_iterator_seek(iter->iter0, seek, flags);
    ++	ret = ref_iterator_seek(iter->iter0, refname, flags);
      	if (ret < 0)
      		return ret;
      
     -	ret = ref_iterator_seek(iter->iter1, prefix);
    -+	ret = ref_iterator_seek(iter->iter1, seek, flags);
    ++	ret = ref_iterator_seek(iter->iter1, refname, flags);
      	if (ret < 0)
      		return ret;
      
    @@ refs/iterator.c: static int prefix_ref_iterator_advance(struct ref_iterator *ref
      
      static int prefix_ref_iterator_seek(struct ref_iterator *ref_iterator,
     -				    const char *prefix)
    -+				    const char *seek, unsigned int flags)
    ++				    const char *refname, unsigned int flags)
      {
      	struct prefix_ref_iterator *iter =
      		(struct prefix_ref_iterator *)ref_iterator;
    @@ refs/iterator.c: static int prefix_ref_iterator_advance(struct ref_iterator *ref
     +
     +	if (flags & REF_ITERATOR_SEEK_SET_PREFIX) {
     +		free(iter->prefix);
    -+		iter->prefix = xstrdup_or_null(seek);
    ++		iter->prefix = xstrdup_or_null(refname);
     +	}
    -+	return ref_iterator_seek(iter->iter0, seek, flags);
    ++	return ref_iterator_seek(iter->iter0, refname, flags);
      }
      
      static int prefix_ref_iterator_peel(struct ref_iterator *ref_iterator,
    @@ refs/packed-backend.c: static int packed_ref_iterator_advance(struct ref_iterato
      
      static int packed_ref_iterator_seek(struct ref_iterator *ref_iterator,
     -				    const char *prefix)
    -+				    const char *seek, unsigned int flags)
    ++				    const char *refname, unsigned int flags)
      {
      	struct packed_ref_iterator *iter =
      		(struct packed_ref_iterator *)ref_iterator;
    @@ refs/packed-backend.c: static int packed_ref_iterator_advance(struct ref_iterato
      
     -	if (prefix && *prefix)
     -		start = find_reference_location(iter->snapshot, prefix, 0);
    -+	if (seek && *seek)
    -+		start = find_reference_location(iter->snapshot, seek, 0);
    ++	if (refname && *refname)
    ++		start = find_reference_location(iter->snapshot, refname, 0);
      	else
      		start = iter->snapshot->start;
      
    @@ refs/packed-backend.c: static int packed_ref_iterator_advance(struct ref_iterato
     +	FREE_AND_NULL(iter->prefix);
     +
     +	if (flags & REF_ITERATOR_SEEK_SET_PREFIX)
    -+		iter->prefix = xstrdup_or_null(seek);
    ++		iter->prefix = xstrdup_or_null(refname);
     +
      	iter->pos = start;
      	iter->eof = iter->snapshot->eof;
    @@ refs/ref-cache.c: static int cache_ref_iterator_seek(struct ref_iterator *ref_it
      }
      
     +static int cache_ref_iterator_seek(struct ref_iterator *ref_iterator,
    -+				   const char *seek, unsigned int flags)
    ++				   const char *refname, unsigned int flags)
     +{
     +	struct cache_ref_iterator *iter =
     +		(struct cache_ref_iterator *)ref_iterator;
     +
     +	if (flags & REF_ITERATOR_SEEK_SET_PREFIX) {
    -+		return cache_ref_iterator_set_prefix(iter, seek);
    -+	} else if (seek && *seek) {
    ++		return cache_ref_iterator_set_prefix(iter, refname);
    ++	} else if (refname && *refname) {
     +		struct cache_ref_iterator_level *level;
    -+		const char *slash = seek;
    ++		const char *slash = refname;
     +		struct ref_dir *dir;
     +
     +		dir = get_ref_dir(iter->cache->root);
     +
     +		if (iter->prime_dir)
    -+			prime_ref_dir(dir, seek);
    ++			prime_ref_dir(dir, refname);
     +
     +		iter->levels_nr = 1;
     +		level = &iter->levels[0];
    @@ refs/ref-cache.c: static int cache_ref_iterator_seek(struct ref_iterator *ref_it
     +			sort_ref_dir(dir);
     +
     +			slash = strchr(slash, '/');
    -+			len = slash ? slash - seek : (int)strlen(seek);
    ++			len = slash ? slash - refname : (int)strlen(refname);
     +
     +			for (idx = 0; idx < dir->nr; idx++) {
    -+				cmp = strncmp(seek, dir->entries[idx]->name, len);
    ++				cmp = strncmp(refname, dir->entries[idx]->name, len);
     +				if (cmp <= 0)
     +					break;
     +			}
    @@ refs/refs-internal.h: void base_ref_iterator_init(struct ref_iterator *iter,
       */
      typedef int ref_iterator_seek_fn(struct ref_iterator *ref_iterator,
     -				 const char *prefix);
    -+				 const char *seek, unsigned int flags);
    ++				 const char *refname, unsigned int flags);
      
      /*
       * Peels the current ref, returning 0 for success or -1 for failure.
    @@ refs/reftable-backend.c: static int reftable_ref_iterator_advance(struct ref_ite
      
      static int reftable_ref_iterator_seek(struct ref_iterator *ref_iterator,
     -				      const char *prefix)
    -+				      const char *seek, unsigned int flags)
    ++				      const char *refname, unsigned int flags)
      {
      	struct reftable_ref_iterator *iter =
      		(struct reftable_ref_iterator *)ref_iterator;
    @@ refs/reftable-backend.c: static int reftable_ref_iterator_advance(struct ref_ite
     +	iter->prefix_len = 0;
     +
     +	if (flags & REF_ITERATOR_SEEK_SET_PREFIX) {
    -+		iter->prefix = xstrdup_or_null(seek);
    -+		iter->prefix_len = seek ? strlen(seek) : 0;
    ++		iter->prefix = xstrdup_or_null(refname);
    ++		iter->prefix_len = refname ? strlen(refname) : 0;
     +	}
    -+	iter->err = reftable_iterator_seek_ref(&iter->iter, seek);
    ++	iter->err = reftable_iterator_seek_ref(&iter->iter, refname);
      
      	return iter->err;
      }
    @@ refs/reftable-backend.c: static int reftable_reflog_iterator_advance(struct ref_
      
      static int reftable_reflog_iterator_seek(struct ref_iterator *ref_iterator UNUSED,
     -					 const char *prefix UNUSED)
    -+					 const char *seek UNUSED,
    ++					 const char *refname UNUSED,
     +					 unsigned int flags UNUSED)
      {
      	BUG("reftable reflog iterator cannot be seeked");

4: a571579886 ! 4: e4e9dddd15 for-each-ref: introduce a '--start-after' option
@@ Commit message
'git-for-each-ref(1)'. When used, the reference iteration seeks to the lexicographically next reference and iterates from there onward.

- This enables efficient pagination workflows like:
+ This enables efficient pagination workflows, where the calling script
+ can remember the last provided reference and use that as the starting
+ point for the next set of references:
git for-each-ref --count=100
git for-each-ref --count=100 --start-after=refs/heads/branch-100 git for-each-ref --count=100 --start-after=refs/heads/branch-200
@@ Documentation/git-for-each-ref.adoc: TAB %(refname)`.
--include-root-refs::
List root refs (HEAD and pseudorefs) apart from regular refs.

-+--start-after::
++--start-after=<marker>::
+ Allows paginating the output by skipping references up to and including the
+ specified marker. When paging, it should be noted that references may be
+ deleted, modified or added between invocations. Output will only yield those
-+ references which follow the marker lexicographically. If the marker does not
-+ exist, output begins from the first reference that would come after it
-+ alphabetically. Cannot be used with general pattern matching or custom
-+ sort options.
++ references which follow the marker lexicographically. Output begins from the
++ first reference that would come after the marker alphabetically. Cannot be
++ used with general pattern matching or custom sort options.
+
FIELD NAMES
-----------

base-commit: cf6f63ea6bf35173e02e18bdc6a4ba41288acff9
change-id: 20250605-306-git-for-each-ref-pagination-0ba8a29ae646

Thanks
- Karthik

#57 lore raw

PATCH v4 1/4 refs: expose `ref_iterator` via 'refs.h'

The ref_iterator is an internal structure to the 'refs/' sub-directory, which allows iteration over refs. All reference iteration is built on top of these iterators.

External clients of the 'refs' subsystem use the various 'refs_for_each...()' functions to iterate over refs. However since these are wrapper functions, each combination of functionality requires a new wrapper function. This is not feasible as the functions pile up with the increase in requirements. Expose the internal reference iterator, so advanced users can mix and match options as needed.

---
 refs.h               | 147 +++++++++++++++++++++++++++++++++++++++++++++++++++
 refs/refs-internal.h | 145 +-------------------------------------------------
 2 files changed, 149 insertions(+), 143 deletions(-)
refs.h +147 -0
index 46a6008e07..7c21aaef3d 100644
@@ -1190,4 +1190,151 @@ int repo_migrate_ref_storage_format(struct repository *repo,
 				    unsigned int flags,
 				    struct strbuf *err);
 
+/*
+ * Reference iterators
+ *
+ * A reference iterator encapsulates the state of an in-progress
+ * iteration over references. Create an instance of `struct
+ * ref_iterator` via one of the functions in this module.
+ *
+ * A freshly-created ref_iterator doesn't yet point at a reference. To
+ * advance the iterator, call ref_iterator_advance(). If successful,
+ * this sets the iterator's refname, oid, and flags fields to describe
+ * the next reference and returns ITER_OK. The data pointed at by
+ * refname and oid belong to the iterator; if you want to retain them
+ * after calling ref_iterator_advance() again or calling
+ * ref_iterator_free(), you must make a copy. When the iteration has
+ * been exhausted, ref_iterator_advance() releases any resources
+ * associated with the iteration, frees the ref_iterator object, and
+ * returns ITER_DONE. If you want to abort the iteration early, call
+ * ref_iterator_free(), which also frees the ref_iterator object and
+ * any associated resources. If there was an internal error advancing
+ * to the next entry, ref_iterator_advance() aborts the iteration,
+ * frees the ref_iterator, and returns ITER_ERROR.
+ *
+ * The reference currently being looked at can be peeled by calling
+ * ref_iterator_peel(). This function is often faster than peel_ref(),
+ * so it should be preferred when iterating over references.
+ *
+ * Putting it all together, a typical iteration looks like this:
+ *
+ *     int ok;
+ *     struct ref_iterator *iter = ...;
+ *
+ *     while ((ok = ref_iterator_advance(iter)) == ITER_OK) {
+ *             if (want_to_stop_iteration()) {
+ *                     ok = ITER_DONE;
+ *                     break;
+ *             }
+ *
+ *             // Access information about the current reference:
+ *             if (!(iter->flags & REF_ISSYMREF))
+ *                     printf("%s is %s\n", iter->refname, oid_to_hex(iter->oid));
+ *
+ *             // If you need to peel the reference:
+ *             ref_iterator_peel(iter, &oid);
+ *     }
+ *
+ *     if (ok != ITER_DONE)
+ *             handle_error();
+ *     ref_iterator_free(iter);
+ */
+struct ref_iterator;
+
+/*
+ * These flags are passed to refs_ref_iterator_begin() (and do_for_each_ref(),
+ * which feeds it).
+ */
+enum do_for_each_ref_flags {
+	/*
+	 * Include broken references in a do_for_each_ref*() iteration, which
+	 * would normally be omitted. This includes both refs that point to
+	 * missing objects (a true repository corruption), ones with illegal
+	 * names (which we prefer not to expose to callers), as well as
+	 * dangling symbolic refs (i.e., those that point to a non-existent
+	 * ref; this is not a corruption, but as they have no valid oid, we
+	 * omit them from normal iteration results).
+	 */
+	DO_FOR_EACH_INCLUDE_BROKEN = (1 << 0),
+
+	/*
+	 * Only include per-worktree refs in a do_for_each_ref*() iteration.
+	 * Normally this will be used with a files ref_store, since that's
+	 * where all reference backends will presumably store their
+	 * per-worktree refs.
+	 */
+	DO_FOR_EACH_PER_WORKTREE_ONLY = (1 << 1),
+
+	/*
+	 * Omit dangling symrefs from output; this only has an effect with
+	 * INCLUDE_BROKEN, since they are otherwise not included at all.
+	 */
+	DO_FOR_EACH_OMIT_DANGLING_SYMREFS = (1 << 2),
+
+	/*
+	 * Include root refs i.e. HEAD and pseudorefs along with the regular
+	 * refs.
+	 */
+	DO_FOR_EACH_INCLUDE_ROOT_REFS = (1 << 3),
+};
+
+/*
+ * Return an iterator that goes over each reference in `refs` for
+ * which the refname begins with prefix. If trim is non-zero, then
+ * trim that many characters off the beginning of each refname.
+ * The output is ordered by refname.
+ */
+struct ref_iterator *refs_ref_iterator_begin(
+	struct ref_store *refs,
+	const char *prefix, const char **exclude_patterns,
+	int trim, enum do_for_each_ref_flags flags);
+
+/*
+ * Advance the iterator to the first or next item and return ITER_OK.
+ * If the iteration is exhausted, free the resources associated with
+ * the ref_iterator and return ITER_DONE. On errors, free the iterator
+ * resources and return ITER_ERROR. It is a bug to use ref_iterator or
+ * call this function again after it has returned ITER_DONE or
+ * ITER_ERROR.
+ */
+int ref_iterator_advance(struct ref_iterator *ref_iterator);
+
+/*
+ * Seek the iterator to the first reference with the given prefix.
+ * The prefix is matched as a literal string, without regard for path
+ * separators. If prefix is NULL or the empty string, seek the iterator to the
+ * first reference again.
+ *
+ * This function is expected to behave as if a new ref iterator with the same
+ * prefix had been created, but allows reuse of iterators and thus may allow
+ * the backend to optimize. Parameters other than the prefix that have been
+ * passed when creating the iterator will remain unchanged.
+ *
+ * Returns 0 on success, a negative error code otherwise.
+ */
+int ref_iterator_seek(struct ref_iterator *ref_iterator,
+		      const char *prefix);
+
+/*
+ * If possible, peel the reference currently being viewed by the
+ * iterator. Return 0 on success.
+ */
+int ref_iterator_peel(struct ref_iterator *ref_iterator,
+		      struct object_id *peeled);
+
+/* Free the reference iterator and any associated resources. */
+void ref_iterator_free(struct ref_iterator *ref_iterator);
+
+/*
+ * The common backend for the for_each_*ref* functions. Call fn for
+ * each reference in iter. If the iterator itself ever returns
+ * ITER_ERROR, return -1. If fn ever returns a non-zero value, stop
+ * the iteration and return that value. Otherwise, return 0. In any
+ * case, free the iterator when done. This function is basically an
+ * adapter between the callback style of reference iteration and the
+ * iterator style.
+ */
+int do_for_each_ref_iterator(struct ref_iterator *iter,
+			     each_ref_fn fn, void *cb_data);
+
 #endif /* REFS_H */
refs/refs-internal.h +2 -143

169 lines not on this page · raw message

Signature · 1 line
2.49.0
#58 lore raw

PATCH v4 2/4 ref-cache: remove unused function 'find_ref_entry()'

In reply to Karthik Nayak

The 'find_ref_entry' function is no longer used, so remove it.

---
 refs/ref-cache.c | 14 --------------
 refs/ref-cache.h |  7 -------
 2 files changed, 21 deletions(-)
refs/ref-cache.c +0 -14
index c1f1bab1d5..8aaffa8c6b 100644
@@ -194,20 +194,6 @@ static struct ref_dir *find_containing_dir(struct ref_dir *dir,
 	return dir;
 }
 
-struct ref_entry *find_ref_entry(struct ref_dir *dir, const char *refname)
-{
-	int entry_index;
-	struct ref_entry *entry;
-	dir = find_containing_dir(dir, refname);
-	if (!dir)
-		return NULL;
-	entry_index = search_ref_dir(dir, refname, strlen(refname));
-	if (entry_index == -1)
-		return NULL;
-	entry = dir->entries[entry_index];
-	return (entry->flag & REF_DIR) ? NULL : entry;
-}
-
 /*
  * Emit a warning and return true iff ref1 and ref2 have the same name
  * and the same oid. Die if they have the same name but different
refs/ref-cache.h +0 -7
index 5f04e518c3..f635d2d824 100644
@@ -201,13 +201,6 @@ void free_ref_cache(struct ref_cache *cache);
  */
 void add_entry_to_dir(struct ref_dir *dir, struct ref_entry *entry);
 
-/*
- * Find the value entry with the given name in dir, sorting ref_dirs
- * and recursing into subdirectories as necessary.  If the name is not
- * found or it corresponds to a directory entry, return NULL.
- */
-struct ref_entry *find_ref_entry(struct ref_dir *dir, const char *refname);
-
 /*
  * Start iterating over references in `cache`. If `prefix` is
  * specified, only include references whose names start with that
Signature · 1 line
2.49.0
#59 lore raw

PATCH v4 3/4 refs: selectively set prefix in the seek functions

In reply to Karthik Nayak

The ref iterator exposes a ref_iterator_seek() function. The name suggests that this would seek the iterator to a specific reference in some ways similar to how fseek() works for the filesystem.

However, the function actually sets the prefix for refs iteration. So further iteration would only yield references which match the particular prefix. This is a bit confusing.

Let's add a 'flags' field to the function, which when set with the 'REF_ITERATOR_SEEK_SET_PREFIX' flag, will set the prefix for the iteration in-line with the existing behavior. Otherwise, the reference backends will simply seek to the specified reference and clears any previously set prefix. This allows users to start iteration from a specific reference.

In the packed and reftable backend, since references are available in a sorted list, the changes are simply setting the prefix if needed. The changes on the files-backend are a little more involved, since the files backend uses the 'ref-cache' mechanism. We move out the existing logic within cache_ref_iterator_seek() to cache_ref_iterator_set_prefix() which is called when the 'REF_ITERATOR_SEEK_SET_PREFIX' flag is set. We then parse the provided seek string and set the required levels and their indexes to ensure that seeking is possible.

---
 refs.c                  |  6 ++--
 refs.h                  | 26 +++++++++------
 refs/debug.c            |  7 ++--
 refs/files-backend.c    |  7 ++--
 refs/iterator.c         | 26 ++++++++-------
 refs/packed-backend.c   | 17 ++++++----
 refs/ref-cache.c        | 85 ++++++++++++++++++++++++++++++++++++++++++++++---
 refs/refs-internal.h    |  7 ++--
 refs/reftable-backend.c | 21 ++++++++----
 9 files changed, 152 insertions(+), 50 deletions(-)
refs.c +3 -3
index dce5c49ca2..243e6898b8 100644
@@ -2666,12 +2666,12 @@ enum ref_transaction_error refs_verify_refnames_available(struct ref_store *refs
 		if (!initial_transaction) {
 			int ok;
 
-			if (!iter) {
+			if (!iter)
 				iter = refs_ref_iterator_begin(refs, dirname.buf, NULL, 0,
 							       DO_FOR_EACH_INCLUDE_BROKEN);
-			} else if (ref_iterator_seek(iter, dirname.buf) < 0) {
+			else if (ref_iterator_seek(iter, dirname.buf,
+						   REF_ITERATOR_SEEK_SET_PREFIX) < 0)
 				goto cleanup;
-			}
 
 			while ((ok = ref_iterator_advance(iter)) == ITER_OK) {
 				if (skip &&
refs.h +17 -9
index 7c21aaef3d..e6780a8848 100644
@@ -1299,21 +1299,29 @@ struct ref_iterator *refs_ref_iterator_begin(
  */
 int ref_iterator_advance(struct ref_iterator *ref_iterator);
 
+enum ref_iterator_seek_flag {
+	/*
+	 * When the REF_ITERATOR_SEEK_SET_PREFIX flag is set, the iterator's prefix is
+	 * updated to match the provided string, affecting all subsequent iterations. If
+	 * not, the iterator seeks to the specified reference and clears any previously
+	 * set prefix.
+	 */
+	REF_ITERATOR_SEEK_SET_PREFIX = (1 << 0),
+};
+
 /*
- * Seek the iterator to the first reference with the given prefix.
- * The prefix is matched as a literal string, without regard for path
- * separators. If prefix is NULL or the empty string, seek the iterator to the
+ * Seek the iterator to the first reference matching the given seek string.
+ * The seek string is matched as a literal string, without regard for path
+ * separators. If seek is NULL or the empty string, seek the iterator to the
  * first reference again.
  *
- * This function is expected to behave as if a new ref iterator with the same
- * prefix had been created, but allows reuse of iterators and thus may allow
- * the backend to optimize. Parameters other than the prefix that have been
- * passed when creating the iterator will remain unchanged.
+ * This function is expected to behave as if a new ref iterator has been
+ * created, but allows reuse of existing iterators for optimization.
  *
  * Returns 0 on success, a negative error code otherwise.
  */
-int ref_iterator_seek(struct ref_iterator *ref_iterator,
-		      const char *prefix);
+int ref_iterator_seek(struct ref_iterator *ref_iterator, const char *refname,
+		      unsigned int flags);
 
 /*
  * If possible, peel the reference currently being viewed by the
refs/debug.c +4 -3
index 485e3079d7..da300efaf3 100644
@@ -170,12 +170,13 @@ static int debug_ref_iterator_advance(struct ref_iterator *ref_iterator)
 }
 
 static int debug_ref_iterator_seek(struct ref_iterator *ref_iterator,
-				   const char *prefix)
+				   const char *refname, unsigned int flags)
 {
 	struct debug_ref_iterator *diter =
 		(struct debug_ref_iterator *)ref_iterator;
-	int res = diter->iter->vtable->seek(diter->iter, prefix);
-	trace_printf_key(&trace_refs, "iterator_seek: %s: %d\n", prefix ? prefix : "", res);
+	int res = diter->iter->vtable->seek(diter->iter, refname, flags);
+	trace_printf_key(&trace_refs, "iterator_seek: %s flags: %d: %d\n",
+			 refname ? refname : "", flags, res);
 	return res;
 }
 
refs/files-backend.c +4 -3
index bf6f89b1d1..8b282f2a60 100644
@@ -929,11 +929,11 @@ static int files_ref_iterator_advance(struct ref_iterator *ref_iterator)
 }
 
 static int files_ref_iterator_seek(struct ref_iterator *ref_iterator,
-				   const char *prefix)
+				   const char *refname, unsigned int flags)
 {
 	struct files_ref_iterator *iter =
 		(struct files_ref_iterator *)ref_iterator;
-	return ref_iterator_seek(iter->iter0, prefix);
+	return ref_iterator_seek(iter->iter0, refname, flags);
 }
 
 static int files_ref_iterator_peel(struct ref_iterator *ref_iterator,
@@ -2316,7 +2316,8 @@ static int files_reflog_iterator_advance(struct ref_iterator *ref_iterator)
 }
 
 static int files_reflog_iterator_seek(struct ref_iterator *ref_iterator UNUSED,
-				      const char *prefix UNUSED)
+				      const char *refname UNUSED,
+				      unsigned int flags UNUSED)
 {
 	BUG("ref_iterator_seek() called for reflog_iterator");
 }
refs/iterator.c +15 -11
index 766d96e795..17ef841d8a 100644
@@ -15,10 +15,10 @@ int ref_iterator_advance(struct ref_iterator *ref_iterator)
 	return ref_iterator->vtable->advance(ref_iterator);
 }
 
-int ref_iterator_seek(struct ref_iterator *ref_iterator,
-		      const char *prefix)
+int ref_iterator_seek(struct ref_iterator *ref_iterator, const char *refname,
+		      unsigned int flags)
 {
-	return ref_iterator->vtable->seek(ref_iterator, prefix);
+	return ref_iterator->vtable->seek(ref_iterator, refname, flags);
 }
 
 int ref_iterator_peel(struct ref_iterator *ref_iterator,
@@ -57,7 +57,8 @@ static int empty_ref_iterator_advance(struct ref_iterator *ref_iterator UNUSED)
 }
 
 static int empty_ref_iterator_seek(struct ref_iterator *ref_iterator UNUSED,
-				   const char *prefix UNUSED)
+				   const char *refname UNUSED,
+				   unsigned int flags UNUSED)
 {
 	return 0;
 }
@@ -224,7 +225,7 @@ static int merge_ref_iterator_advance(struct ref_iterator *ref_iterator)
 }
 
 static int merge_ref_iterator_seek(struct ref_iterator *ref_iterator,
-				   const char *prefix)
+				   const char *refname, unsigned int flags)
 {
 	struct merge_ref_iterator *iter =
 		(struct merge_ref_iterator *)ref_iterator;
@@ -234,11 +235,11 @@ static int merge_ref_iterator_seek(struct ref_iterator *ref_iterator,
 	iter->iter0 = iter->iter0_owned;
 	iter->iter1 = iter->iter1_owned;
 
-	ret = ref_iterator_seek(iter->iter0, prefix);
+	ret = ref_iterator_seek(iter->iter0, refname, flags);
 	if (ret < 0)
 		return ret;
 
-	ret = ref_iterator_seek(iter->iter1, prefix);
+	ret = ref_iterator_seek(iter->iter1, refname, flags);
 	if (ret < 0)
 		return ret;
 
@@ -407,13 +408,16 @@ static int prefix_ref_iterator_advance(struct ref_iterator *ref_iterator)
 }
 
 static int prefix_ref_iterator_seek(struct ref_iterator *ref_iterator,
-				    const char *prefix)
+				    const char *refname, unsigned int flags)
 {
 	struct prefix_ref_iterator *iter =
 		(struct prefix_ref_iterator *)ref_iterator;
-	free(iter->prefix);
-	iter->prefix = xstrdup_or_null(prefix);
-	return ref_iterator_seek(iter->iter0, prefix);
+
+	if (flags & REF_ITERATOR_SEEK_SET_PREFIX) {
+		free(iter->prefix);
+		iter->prefix = xstrdup_or_null(refname);
+	}
+	return ref_iterator_seek(iter->iter0, refname, flags);
 }
 
 static int prefix_ref_iterator_peel(struct ref_iterator *ref_iterator,
refs/packed-backend.c +11 -6
index 7fd73a0e6d..5fa4ae6655 100644
@@ -1004,19 +1004,23 @@ static int packed_ref_iterator_advance(struct ref_iterator *ref_iterator)
 }
 
 static int packed_ref_iterator_seek(struct ref_iterator *ref_iterator,
-				    const char *prefix)
+				    const char *refname, unsigned int flags)
 {
 	struct packed_ref_iterator *iter =
 		(struct packed_ref_iterator *)ref_iterator;
 	const char *start;
 
-	if (prefix && *prefix)
-		start = find_reference_location(iter->snapshot, prefix, 0);
+	if (refname && *refname)
+		start = find_reference_location(iter->snapshot, refname, 0);
 	else
 		start = iter->snapshot->start;
 
-	free(iter->prefix);
-	iter->prefix = xstrdup_or_null(prefix);
+	/* Unset any previously set prefix */
+	FREE_AND_NULL(iter->prefix);
+
+	if (flags & REF_ITERATOR_SEEK_SET_PREFIX)
+		iter->prefix = xstrdup_or_null(refname);
+
 	iter->pos = start;
 	iter->eof = iter->snapshot->eof;
 
@@ -1194,7 +1198,8 @@ static struct ref_iterator *packed_ref_iterator_begin(
 	iter->repo = ref_store->repo;
 	iter->flags = flags;
 
-	if (packed_ref_iterator_seek(&iter->base, prefix) < 0) {
+	if (packed_ref_iterator_seek(&iter->base, prefix,
+				     REF_ITERATOR_SEEK_SET_PREFIX) < 0) {
 		ref_iterator_free(&iter->base);
 		return NULL;
 	}
refs/ref-cache.c +80 -5

104 lines not on this page · raw message

refs/refs-internal.h +4 -3
index 03f5df04d5..90de7837f8 100644
@@ -353,11 +353,12 @@ void base_ref_iterator_init(struct ref_iterator *iter,
 typedef int ref_iterator_advance_fn(struct ref_iterator *ref_iterator);
 
 /*
- * Seek the iterator to the first reference matching the given prefix. Should
- * behave the same as if a new iterator was created with the same prefix.
+ * Seek the iterator to the first matching reference. If set_prefix is set,
+ * it would behave the same as if a new iterator was created with the same
+ * prefix.
  */
 typedef int ref_iterator_seek_fn(struct ref_iterator *ref_iterator,
-				 const char *prefix);
+				 const char *refname, unsigned int flags);
 
 /*
  * Peels the current ref, returning 0 for success or -1 for failure.
refs/reftable-backend.c +14 -7
index 4c3817f4ec..c3d48cc412 100644
@@ -719,15 +719,20 @@ static int reftable_ref_iterator_advance(struct ref_iterator *ref_iterator)
 }
 
 static int reftable_ref_iterator_seek(struct ref_iterator *ref_iterator,
-				      const char *prefix)
+				      const char *refname, unsigned int flags)
 {
 	struct reftable_ref_iterator *iter =
 		(struct reftable_ref_iterator *)ref_iterator;
 
-	free(iter->prefix);
-	iter->prefix = xstrdup_or_null(prefix);
-	iter->prefix_len = prefix ? strlen(prefix) : 0;
-	iter->err = reftable_iterator_seek_ref(&iter->iter, prefix);
+	/* Unset any previously set prefix */
+	FREE_AND_NULL(iter->prefix);
+	iter->prefix_len = 0;
+
+	if (flags & REF_ITERATOR_SEEK_SET_PREFIX) {
+		iter->prefix = xstrdup_or_null(refname);
+		iter->prefix_len = refname ? strlen(refname) : 0;
+	}
+	iter->err = reftable_iterator_seek_ref(&iter->iter, refname);
 
 	return iter->err;
 }
@@ -839,7 +844,8 @@ static struct reftable_ref_iterator *ref_iterator_for_stack(struct reftable_ref_
 	if (ret)
 		goto done;
 
-	ret = reftable_ref_iterator_seek(&iter->base, prefix);
+	ret = reftable_ref_iterator_seek(&iter->base, prefix,
+					 REF_ITERATOR_SEEK_SET_PREFIX);
 	if (ret)
 		goto done;
 
@@ -2042,7 +2048,8 @@ static int reftable_reflog_iterator_advance(struct ref_iterator *ref_iterator)
 }
 
 static int reftable_reflog_iterator_seek(struct ref_iterator *ref_iterator UNUSED,
-					 const char *prefix UNUSED)
+					 const char *refname UNUSED,
+					 unsigned int flags UNUSED)
 {
 	BUG("reftable reflog iterator cannot be seeked");
 	return -1;
Signature · 1 line
2.49.0
#60 lore raw

Re: [PATCH v4 3/4] refs: selectively set prefix in the seek functions

refs/refs-internal.h void base_ref_iterator_init(struct ref_iterator *iter, · line 356

Quoted 11 lines · Karthik Nayak
refs/refs-internal.h +1 -2
index 03f5df04d5..90de7837f8 100644
@@ -353,11 +353,12 @@ void base_ref_iterator_init(struct ref_iterator *iter,
 typedef int ref_iterator_advance_fn(struct ref_iterator *ref_iterator);
 
 /*
- * Seek the iterator to the first reference matching the given prefix. Should
- * behave the same as if a new iterator was created with the same prefix.
+ * Seek the iterator to the first matching reference. If set_prefix is set,

s/If set_prefix is set/If the REF_ITERATOR_SEEK_SET_PREFIX flag is set/

refs/refs-internal.h void base_ref_iterator_init(struct ref_iterator *iter, · line 358

Quoted 2 lines
refs/refs-internal.h +2 -0
+ * it would behave the same as if a new iterator was created with the same
+ * prefix.

Maybe: s/with the same prefix/at the same reference/

Quoted 4 lines
refs/refs-internal.h +1 -1
  */
 typedef int ref_iterator_seek_fn(struct ref_iterator *ref_iterator,
-                                const char *prefix);
+                                const char *refname, unsigned int flags);
#61 lore raw

Re: [PATCH v4 3/4] refs: selectively set prefix in the seek functions

Quoted 9 lines · Christian Couder
Quoted 7 lines · Karthik Nayak
refs/refs-internal.h +0 -0
index 03f5df04d5..90de7837f8 100644
@@ -353,11 +353,12 @@ void base_ref_iterator_init(struct ref_iterator *iter,
 typedef int ref_iterator_advance_fn(struct ref_iterator *ref_iterator);
 
Quoted 7 lines

refs/refs-internal.h void base_ref_iterator_init(struct ref_iterator *iter, · line 356

Quoted 4 lines
refs/refs-internal.h +1 -2
 /*
- * Seek the iterator to the first reference matching the given prefix. Should
- * behave the same as if a new iterator was created with the same prefix.
+ * Seek the iterator to the first matching reference. If set_prefix is set,

s/If set_prefix is set/If the REF_ITERATOR_SEEK_SET_PREFIX flag is set/

Will change, thanks.

Quoted 5 lines

refs/refs-internal.h void base_ref_iterator_init(struct ref_iterator *iter, · line 358

Quoted 2 lines
refs/refs-internal.h +2 -0
+ * it would behave the same as if a new iterator was created with the same
+ * prefix.

Maybe: s/with the same prefix/at the same reference/

Changed it to

If the REF_ITERATOR_SEEK_SET_PREFIX flag is set, it would behave the same as if a new iterator was created with the provided refname as prefix.

Quoted 4 lines
Quoted 4 lines
refs/refs-internal.h +1 -1
  */
 typedef int ref_iterator_seek_fn(struct ref_iterator *ref_iterator,
-                                const char *prefix);
+                                const char *refname, unsigned int flags);
#62 lore raw

PATCH v4 4/4 for-each-ref: introduce a '--start-after' option

In reply to Karthik Nayak

The git-for-each-ref(1) command is used to iterate over references present in a repository. In large repositories with millions of references, it would be optimal to paginate this output such that we can start iteration from a given reference. This would avoid having to iterate over all references from the beginning each time when paginating through results.

The previous commit added 'seek' functionality to the reference backends. Utilize this and expose a '--start-after' option in 'git-for-each-ref(1)'. When used, the reference iteration seeks to the lexicographically next reference and iterates from there onward.

This enables efficient pagination workflows, where the calling script can remember the last provided reference and use that as the starting point for the next set of references:

    git for-each-ref --count=100
    git for-each-ref --count=100 --start-after=refs/heads/branch-100
    git for-each-ref --count=100 --start-after=refs/heads/branch-200

Since the reference iterators only allow seeking to a specified marker via the ref_iterator_seek(), we introduce a helper function start_ref_iterator_after(), which seeks to next reference by simply adding (char) 1 to the marker.

We must note that pagination always continues from the provided marker, as such any concurrent reference updates lexicographically behind the marker will not be output. Document the same.

---
 Documentation/git-for-each-ref.adoc |  10 +-
 builtin/for-each-ref.c              |   8 ++
 ref-filter.c                        |  80 +++++++++++----
 ref-filter.h                        |   1 +
 t/t6302-for-each-ref-filter.sh      | 194 ++++++++++++++++++++++++++++++++++++
 5 files changed, 272 insertions(+), 21 deletions(-)
Documentation/git-for-each-ref.adoc +9 -1
index 5ef89fc0fe..ae61ba642a 100644
@@ -14,7 +14,7 @@ SYNOPSIS
 		   [--points-at=<object>]
 		   [--merged[=<object>]] [--no-merged[=<object>]]
 		   [--contains[=<object>]] [--no-contains[=<object>]]
-		   [--exclude=<pattern> ...]
+		   [--exclude=<pattern> ...] [--start-after=<marker>]
 
 DESCRIPTION
 -----------
@@ -108,6 +108,14 @@ TAB %(refname)`.
 --include-root-refs::
 	List root refs (HEAD and pseudorefs) apart from regular refs.
 
+--start-after=<marker>::
+    Allows paginating the output by skipping references up to and including the
+    specified marker. When paging, it should be noted that references may be
+    deleted, modified or added between invocations. Output will only yield those
+    references which follow the marker lexicographically. Output begins from the
+    first reference that would come after the marker alphabetically. Cannot be
+    used with general pattern matching or custom sort options.
+
 FIELD NAMES
 -----------
 
builtin/for-each-ref.c +8 -0
index 3d2207ec77..3f21598046 100644
@@ -13,6 +13,7 @@ static char const * const for_each_ref_usage[] = {
 	N_("git for-each-ref [--points-at <object>]"),
 	N_("git for-each-ref [--merged [<commit>]] [--no-merged [<commit>]]"),
 	N_("git for-each-ref [--contains [<commit>]] [--no-contains [<commit>]]"),
+	N_("git for-each-ref [--start-after <marker>]"),
 	NULL
 };
 
@@ -44,6 +45,7 @@ int cmd_for_each_ref(int argc,
 		OPT_GROUP(""),
 		OPT_INTEGER( 0 , "count", &format.array_opts.max_count, N_("show only <n> matched refs")),
 		OPT_STRING(  0 , "format", &format.format, N_("format"), N_("format to use for the output")),
+		OPT_STRING(  0 , "start-after", &filter.start_after, N_("start-start"), N_("start iteration after the provided marker")),
 		OPT__COLOR(&format.use_color, N_("respect format colors")),
 		OPT_REF_FILTER_EXCLUDE(&filter),
 		OPT_REF_SORT(&sorting_options),
@@ -79,6 +81,9 @@ int cmd_for_each_ref(int argc,
 	if (verify_ref_format(&format))
 		usage_with_options(for_each_ref_usage, opts);
 
+	if (filter.start_after && sorting_options.nr > 1)
+		die(_("cannot use --start-after with custom sort options"));
+
 	sorting = ref_sorting_options(&sorting_options);
 	ref_sorting_set_sort_flags_all(sorting, REF_SORTING_ICASE, icase);
 	filter.ignore_case = icase;
@@ -100,6 +105,9 @@ int cmd_for_each_ref(int argc,
 		filter.name_patterns = argv;
 	}
 
+	if (filter.start_after && filter.name_patterns && filter.name_patterns[0])
+		die(_("cannot use --start-after with patterns"));
+
 	if (include_root_refs)
 		flags |= FILTER_REFS_ROOT_REFS | FILTER_REFS_DETACHED_HEAD;
 
ref-filter.c +60 -20

132 lines not on this page · raw message

ref-filter.h +1 -0
index c98c4fbd4c..f22ca94b49 100644
@@ -64,6 +64,7 @@ struct ref_array {
 
 struct ref_filter {
 	const char **name_patterns;
+	const char *start_after;
 	struct strvec exclude;
 	struct oid_array points_at;
 	struct commit_list *with_commit;
t/t6302-for-each-ref-filter.sh +194 -0

198 lines not on this page · raw message

Signature · 1 line
2.49.0
#63 lore raw

Re: [PATCH v4 4/4] for-each-ref: introduce a '--start-after' option

Quoted 51 lines · Karthik Nayak

/*
* This is the same as for_each_fullref_in(), but it tries to iterate
* only over the patterns we'll care about. Note that it _doesn't_ do a full

ref-filter.c +10 -8
@@ -2692,10 +2710,13 @@ static int for_each_fullref_in_pattern(struct ref_filter *filter,
                                       each_ref_fn cb,
                                       void *cb_data)
 {
+       struct ref_iterator *iter;
+       int flags = 0, ret = 0;
+
        if (filter->kind & FILTER_REFS_ROOT_REFS) {
                /* In this case, we want to print all refs including root refs. */
-               return refs_for_each_include_root_refs(get_main_ref_store(the_repository),
-                                                      cb, cb_data);
+               flags |= DO_FOR_EACH_INCLUDE_ROOT_REFS;
+               goto non_prefix_iter;
        }
 
        if (!filter->match_as_path) {
@@ -2704,8 +2725,7 @@ static int for_each_fullref_in_pattern(struct ref_filter *filter,
                 * prefixes like "refs/heads/" etc. are stripped off,
                 * so we have to look at everything:
                 */
-               return refs_for_each_fullref_in(get_main_ref_store(the_repository),
-                                               "", NULL, cb, cb_data);
+               goto non_prefix_iter;
        }
 
        if (filter->ignore_case) {
@@ -2714,20 +2734,29 @@ static int for_each_fullref_in_pattern(struct ref_filter *filter,
                 * so just return everything and let the caller
                 * sort it out.
                 */
-               return refs_for_each_fullref_in(get_main_ref_store(the_repository),
-                                               "", NULL, cb, cb_data);
+               goto non_prefix_iter;
        }
 
        if (!filter->name_patterns[0]) {
                /* no patterns; we have to look at everything */
-               return refs_for_each_fullref_in(get_main_ref_store(the_repository),
-                                                "", filter->exclude.v, cb, cb_data);
+               goto non_prefix_iter;
        }
 
        return refs_for_each_fullref_in_prefixes(get_main_ref_store(the_repository),
                                                 NULL, filter->name_patterns,
                                                 filter->exclude.v,
                                                 cb, cb_data);
+
+non_prefix_iter:

ref-filter.c static int for_each_fullref_in_pattern(struct ref_filter *filter, · line 2760

Quoted 10 lines
ref-filter.c +9 -0
+       iter = refs_ref_iterator_begin(get_main_ref_store(the_repository), "",
+                                      NULL, 0, flags);
+       if (filter->start_after)
+               ret = start_ref_iterator_after(iter, filter->start_after);
+
+       if (ret)
+               return ret;
+
+       return do_for_each_ref_iterator(iter, cb, cb_data);
 }

Nit: I wonder if what is under the 'non_prefix_iter' label could be in a new function and instead of goto non_prefix_iter we could return the result of the new function.

ref-filter.c static int do_filter_refs(struct ref_filter *filter, unsigned int type, each_ref · line 3231

Quoted 10 lines
ref-filter.c +2 -2
 /*
@@ -3197,9 +3226,11 @@ static int do_filter_refs(struct ref_filter *filter, unsigned int type, each_ref
        init_contains_cache(&filter->internal.no_contains_cache);
 
        /*  Simple per-ref filtering */
-       if (!filter->kind)
+       if (!filter->kind) {
                die("filter_refs: invalid type");
-       else {
+       } else {

Nit: the else could be removed altogether here, but maybe that should be done in a preparatory patch.

Quoted 2 lines
ref-filter.c +2 -0
+               const char *prefix = NULL;
+

[...]

Quoted 6 lines
t/t6302-for-each-ref-filter.sh +6 -0
+test_expect_success 'start after with specific directory and trailing slash' '
+       cat >expect <<-\EOF &&
+       refs/odd/spot
+       refs/tags/annotated-tag
+       refs/tags/doubly-annotated-tag
+       refs/tags/doubly-signed-tag

t/t6302-for-each-ref-filter.sh test_expect_success 'validate worktree atom' ' · line 656

Quoted 10 lines
t/t6302-for-each-ref-filter.sh +10 -0
+       refs/tags/foo1.10
+       refs/tags/foo1.3
+       refs/tags/foo1.6
+       refs/tags/four
+       refs/tags/one
+       refs/tags/signed-tag
+       refs/tags/three
+       refs/tags/two
+       EOF
+       git for-each-ref --format="%(refname)" --start-after=refs/lost >actual &&

I don't see a trailing slash.

Quoted 9 lines
t/t6302-for-each-ref-filter.sh +9 -0
+       test_cmp expect actual
+'
+
+test_expect_success 'start after, just behind a specific directory' '
+       cat >expect <<-\EOF &&
+       refs/odd/spot
+       refs/tags/annotated-tag
+       refs/tags/doubly-annotated-tag
+       refs/tags/doubly-signed-tag

t/t6302-for-each-ref-filter.sh test_expect_success 'validate worktree atom' ' · line 675

Quoted 10 lines
t/t6302-for-each-ref-filter.sh +10 -0
+       refs/tags/foo1.10
+       refs/tags/foo1.3
+       refs/tags/foo1.6
+       refs/tags/four
+       refs/tags/one
+       refs/tags/signed-tag
+       refs/tags/three
+       refs/tags/two
+       EOF
+       git for-each-ref --format="%(refname)" --start-after=refs/odd/ >actual &&

Here there is a trailing slash though.

Quoted 2 lines

+ test_cmp expect actual
+'

#64 lore raw

Re: [PATCH v4 4/4] for-each-ref: introduce a '--start-after' option

Quoted 9 lines · Christian Couder

ref-filter.c static int do_filter_refs(struct ref_filter *filter, unsigned int type, each_ref · line 3231

Quoted 6 lines
        /*  Simple per-ref filtering */
ref-filter.c +2 -2
-       if (!filter->kind)
+       if (!filter->kind) {
                die("filter_refs: invalid type");
-       else {
+       } else {

Nit: the else could be removed altogether here, but maybe that should be done in a preparatory patch.

Good eyes. Thanks for carefully reading it over.

#65 lore raw

Re: [PATCH v4 4/4] for-each-ref: introduce a '--start-after' option

In reply to Christian Couder
Quoted 59 lines · Christian Couder
Quoted 57 lines · Karthik Nayak

/*
* This is the same as for_each_fullref_in(), but it tries to iterate
* only over the patterns we'll care about. Note that it _doesn't_ do a full

ref-filter.c +16 -8
@@ -2692,10 +2710,13 @@ static int for_each_fullref_in_pattern(struct ref_filter *filter,
                                       each_ref_fn cb,
                                       void *cb_data)
 {
+       struct ref_iterator *iter;
+       int flags = 0, ret = 0;
+
        if (filter->kind & FILTER_REFS_ROOT_REFS) {
                /* In this case, we want to print all refs including root refs. */
-               return refs_for_each_include_root_refs(get_main_ref_store(the_repository),
-                                                      cb, cb_data);
+               flags |= DO_FOR_EACH_INCLUDE_ROOT_REFS;
+               goto non_prefix_iter;
        }
 
        if (!filter->match_as_path) {
@@ -2704,8 +2725,7 @@ static int for_each_fullref_in_pattern(struct ref_filter *filter,
                 * prefixes like "refs/heads/" etc. are stripped off,
                 * so we have to look at everything:
                 */
-               return refs_for_each_fullref_in(get_main_ref_store(the_repository),
-                                               "", NULL, cb, cb_data);
+               goto non_prefix_iter;
        }
 
        if (filter->ignore_case) {
@@ -2714,20 +2734,29 @@ static int for_each_fullref_in_pattern(struct ref_filter *filter,
                 * so just return everything and let the caller
                 * sort it out.
                 */
-               return refs_for_each_fullref_in(get_main_ref_store(the_repository),
-                                               "", NULL, cb, cb_data);
+               goto non_prefix_iter;
        }
 
        if (!filter->name_patterns[0]) {
                /* no patterns; we have to look at everything */
-               return refs_for_each_fullref_in(get_main_ref_store(the_repository),
-                                                "", filter->exclude.v, cb, cb_data);
+               goto non_prefix_iter;
        }
 
        return refs_for_each_fullref_in_prefixes(get_main_ref_store(the_repository),
                                                 NULL, filter->name_patterns,
                                                 filter->exclude.v,
                                                 cb, cb_data);
+
+non_prefix_iter:
+       iter = refs_ref_iterator_begin(get_main_ref_store(the_repository), "",
+                                      NULL, 0, flags);
+       if (filter->start_after)
+               ret = start_ref_iterator_after(iter, filter->start_after);
+
+       if (ret)
Quoted 9 lines

ref-filter.c static int for_each_fullref_in_pattern(struct ref_filter *filter, · line 2760

Quoted 4 lines
ref-filter.c +3 -0
+               return ret;
+
+       return do_for_each_ref_iterator(iter, cb, cb_data);
 }

Nit: I wonder if what is under the 'non_prefix_iter' label could be in a new function and instead of goto non_prefix_iter we could return the result of the new function.

Yeah, that would work too. Let me do that and make it nicer! Thanks for the suggestion.

Quoted 4 lines
Quoted 4 lines
ref-filter.c +0 -0
 /*
@@ -3197,9 +3226,11 @@ static int do_filter_refs(struct ref_filter *filter, unsigned int type, each_ref
        init_contains_cache(&filter->internal.no_contains_cache);
 
Quoted 10 lines

ref-filter.c static int do_filter_refs(struct ref_filter *filter, unsigned int type, each_ref · line 3231

Quoted 6 lines
ref-filter.c +2 -2
        /*  Simple per-ref filtering */
-       if (!filter->kind)
+       if (!filter->kind) {
                die("filter_refs: invalid type");
-       else {
+       } else {

Nit: the else could be removed altogether here, but maybe that should be done in a preparatory patch.

Indeed, since I plan to re-roll with the changes you've suggested, I will add this in too.

Quoted 38 lines
Quoted 2 lines
ref-filter.c +2 -0
+               const char *prefix = NULL;
+

[...]

Quoted 6 lines
t/t6302-for-each-ref-filter.sh +6 -0
+test_expect_success 'start after with specific directory and trailing slash' '
+       cat >expect <<-\EOF &&
+       refs/odd/spot
+       refs/tags/annotated-tag
+       refs/tags/doubly-annotated-tag
+       refs/tags/doubly-signed-tag

t/t6302-for-each-ref-filter.sh test_expect_success 'validate worktree atom' ' · line 656

Quoted 10 lines
t/t6302-for-each-ref-filter.sh +10 -0
+       refs/tags/foo1.10
+       refs/tags/foo1.3
+       refs/tags/foo1.6
+       refs/tags/four
+       refs/tags/one
+       refs/tags/signed-tag
+       refs/tags/three
+       refs/tags/two
+       EOF
+       git for-each-ref --format="%(refname)" --start-after=refs/lost >actual &&

I don't see a trailing slash.

Quoted 13 lines
t/t6302-for-each-ref-filter.sh +13 -0
+       test_cmp expect actual
+'
+
+test_expect_success 'start after, just behind a specific directory' '
+       cat >expect <<-\EOF &&
+       refs/odd/spot
+       refs/tags/annotated-tag
+       refs/tags/doubly-annotated-tag
+       refs/tags/doubly-signed-tag
+       refs/tags/foo1.10
+       refs/tags/foo1.3
+       refs/tags/foo1.6
+       refs/tags/four
Quoted 8 lines

t/t6302-for-each-ref-filter.sh test_expect_success 'validate worktree atom' ' · line 675

Quoted 6 lines
t/t6302-for-each-ref-filter.sh +6 -0
+       refs/tags/one
+       refs/tags/signed-tag
+       refs/tags/three
+       refs/tags/two
+       EOF
+       git for-each-ref --format="%(refname)" --start-after=refs/odd/ >actual &&

Here there is a trailing slash though.

I think these tests would make more sense in the newer versions with the values swapped. Let me do that.

Thanks Christian for the thorough review.

#66 lore raw

Re: [PATCH v4 0/4] for-each-ref: introduce seeking functionality via '--start-after'

In reply to Karthik Nayak
Quoted 3 lines · Karthik Nayak

Initially I was also planning to cleanup all the refs_for_each...() functions in 'refs.h' by simply using the iterator, but this bloated the series. So I've left that for another day.

I wonder if there is a plan to add the '--start-after' option to git branch and git tag too?

Quoted 5 lines

Karthik Nayak (4):
refs: expose ref_iterator via 'refs.h'
ref-cache: remove unused function 'find_ref_entry()'
refs: selectively set prefix in the seek functions
for-each-ref: introduce a '--start-after' option

Except for the few small comments I left on the two last patches and one below, this looks good to me.

[ ... ]

Quoted 1 line

Range-diff versus v3:

[ ... ]

Quoted 8 lines
     +  struct cache_ref_iterator *iter =
     +          (struct cache_ref_iterator *)ref_iterator;
     +
     +  if (flags & REF_ITERATOR_SEEK_SET_PREFIX) {
    -+          return cache_ref_iterator_set_prefix(iter, seek);
    -+  } else if (seek && *seek) {
    ++          return cache_ref_iterator_set_prefix(iter, refname);
    ++  } else if (refname && *refname) {

Nit: the else here could be removed, but yeah it might be better to do that in a preparatory patch.

Quoted 4 lines
     +          struct cache_ref_iterator_level *level;
    -+          const char *slash = seek;
    ++          const char *slash = refname;
     +          struct ref_dir *dir;
#67 lore raw

Re: [PATCH v4 0/4] for-each-ref: introduce seeking functionality via '--start-after'

Quoted 8 lines · Christian Couder
Quoted 3 lines · Karthik Nayak

Initially I was also planning to cleanup all the refs_for_each...() functions in 'refs.h' by simply using the iterator, but this bloated the series. So I've left that for another day.

I wonder if there is a plan to add the '--start-after' option to git branch and git tag too?

Good question.

"git for-each-ref" is for scripters, "git branch/tag" are for humans. And humans do not page (outside "more/less").

So while it may be trivial to expose the feature to these Porcelain commands, it is not obvious that it is a good idea worth cluttering "git tag -h" output.

#68 lore raw

Re: [PATCH v4 0/4] for-each-ref: introduce seeking functionality via '--start-after'

Quoted 11 lines · Junio C Hamano
Quoted 8 lines · Christian Couder
Quoted 3 lines · Karthik Nayak

Initially I was also planning to cleanup all the refs_for_each...() functions in 'refs.h' by simply using the iterator, but this bloated the series. So I've left that for another day.

I wonder if there is a plan to add the '--start-after' option to git branch and git tag too?

Quoted 8 lines

Good question.

"git for-each-ref" is for scripters, "git branch/tag" are for humans. And humans do not page (outside "more/less").

So while it may be trivial to expose the feature to these Porcelain commands, it is not obvious that it is a good idea worth cluttering "git tag -h" output.

Agreed, since the functionality is built into 'ref-filter', it should be easy to expand to these commands. But I don't see the need for it.

Karthik

#69 lore raw

v5 cover for-each-ref: introduce seeking functionality via '--start-after'

In reply to Karthik Nayak

The git-for-each-ref(1) command is used to iterate over references present in a repository. In large repositories with millions of references, it would be optimal to paginate this output such that we can start iteration from a given reference. This would avoid having to iterate over all references from the beginning each time when paginating through results.

This series adds a '--start-after' option in 'git-for-each-ref(1)'. When used, the reference iteration seeks to first reference following the marker alphabetically. When paging, it should be noted that references may be deleted, modified or added between invocations. Output will only yield those references which follow the marker lexicographically. If the marker does not exist, output begins from the first reference that would come after it alphabetically.

This enables efficient pagination workflows like:

    git for-each-ref --count=100
    git for-each-ref --count=100 --start-after=refs/heads/branch-100
    git for-each-ref --count=100 --start-after=refs/heads/branch-200

To add this functionality, we expose the ref_iterator outside the 'refs/' namespace and modify the ref_iterator_seek() to actually seek to a given reference and only set the prefix when the set_prefix field is set.

On the reftable and packed backend, the changes are simple. But since the files backend uses 'ref-cache' for reference handling, the changes there are a little more involved, since we need to setup the right levels and the indexing.

Initially I was also planning to cleanup all the refs_for_each...() functions in 'refs.h' by simply using the iterator, but this bloated the series. So I've left that for another day.

Changes in v5:
- Changes to the comments to refer to the flag
'REF_ITERATOR_SEEK_SET_PREFIX' instead of a variable used in older
versions. Also other small grammar fixes.
- Added a commit to remove an unnecessary else clause.
- Move seeking functionality within for_each_fullref_in_pattern to its
own function.
- Fix incorrect naming in the tests.
- Link to v4: https://lore.kernel.org/r/20250711-306-git-for-each-ref-pagination-v4-0-ed3303ad5b89@gmail.com

Changes in v4:
- Patch 3/4: Move around the documentation for the flag and rename the
seek variable to refname.
- Patch 4/4: Cleanup the commit message and also the documentation.
- Link to v3: https://lore.kernel.org/r/20250708-306-git-for-each-ref-pagination-v3-0-8cfba1080be4@gmail.com

Changes in v3:
- Change the working of the command to exclude the marker provided. With
this rename the flag to '--start-after'.
- Extend the documentation to add a note about concurrent modifications
to the reference database.
- Link to v2: https://lore.kernel.org/r/20250704-306-git-for-each-ref-pagination-v2-0-bcde14acdd81@gmail.com

Changes in v2:
- Modify 'ref_iterator_seek()' to take in flags instead of a
'set_prefix' variable. This improves readability, where users would
use the 'REF_ITERATOR_SEEK_SET_PREFIX' instead of simply passing '1'.
- When the set prefix flag isn't usage, reset any previously set prefix.
This ensures that the internal prefix state is always reset whenever
we seek and unifies the behavior between 'ref_iterator_seek' and
'ref_iterator_begin'.
- Don't allow '--skip-until' to be run with '--sort', since the seeking
always takes place before any sorting and this can be confusing.
- Some styling fixes:
- Remove extra newline
- Skip braces around single lined if...else clause
- Add braces around 'if' clause
- Fix indentation
- Link to v1: https://lore.kernel.org/git/20250701-306-git-for-each-ref-pagination-v1-0-4f0ae7c0688f@gmail.com/

---
 Documentation/git-for-each-ref.adoc |  10 +-
 builtin/for-each-ref.c              |   8 ++
 ref-filter.c                        | 116 ++++++++++++++-------
 ref-filter.h                        |   1 +
 refs.c                              |   6 +-
 refs.h                              | 155 ++++++++++++++++++++++++++++
 refs/debug.c                        |   7 +-
 refs/files-backend.c                |   7 +-
 refs/iterator.c                     |  26 +++--
 refs/packed-backend.c               |  17 ++--
 refs/ref-cache.c                    |  99 ++++++++++++++----
 refs/ref-cache.h                    |   7 --
 refs/refs-internal.h                | 152 ++--------------------------
 refs/reftable-backend.c             |  21 ++--
 t/t6302-for-each-ref-filter.sh      | 194 ++++++++++++++++++++++++++++++++++++
 15 files changed, 583 insertions(+), 243 deletions(-)

Karthik Nayak (5):
refs: expose ref_iterator via 'refs.h'
ref-cache: remove unused function 'find_ref_entry()'
refs: selectively set prefix in the seek functions
ref-filter: remove unnecessary else clause
for-each-ref: introduce a '--start-after' option

Range-diff versus v4:

1:  dde167f421 = 1:  f9c9a7fdd9 refs: expose `ref_iterator` via 'refs.h'
2:  e392e93520 = 2:  83bee35517 ref-cache: remove unused function 'find_ref_entry()'
3:  711ffcac00 ! 3:  3b6019a1e7 refs: selectively set prefix in the seek functions
    @@ refs/refs-internal.h: void base_ref_iterator_init(struct ref_iterator *iter,
      /*
     - * Seek the iterator to the first reference matching the given prefix. Should
     - * behave the same as if a new iterator was created with the same prefix.
    -+ * Seek the iterator to the first matching reference. If set_prefix is set,
    -+ * it would behave the same as if a new iterator was created with the same
    -+ * prefix.
    ++ * Seek the iterator to the first matching reference. If the
    ++ * REF_ITERATOR_SEEK_SET_PREFIX flag is set, it would behave the same as if a
    ++ * new iterator was created with the provided refname as prefix.
       */
      typedef int ref_iterator_seek_fn(struct ref_iterator *ref_iterator,
     -				 const char *prefix);
-:  ---------- > 4:  3f89eeef26 ref-filter: remove unnecessary else clause
4:  3a0c89acbe ! 5:  7ee7d83cf0 for-each-ref: introduce a '--start-after' option
    @@ ref-filter.c: static int filter_exclude_match(struct ref_filter *filter, const c
     +	strbuf_release(&sb);
     +	return ret;
     +}
    ++
    ++static int for_each_fullref_with_seek(struct ref_filter *filter, each_ref_fn cb,
    ++				       void *cb_data, unsigned int flags)
    ++{
    ++	struct ref_iterator *iter;
    ++	int ret = 0;
    ++
    ++	iter = refs_ref_iterator_begin(get_main_ref_store(the_repository), "",
    ++				       NULL, 0, flags);
    ++	if (filter->start_after)
    ++		ret = start_ref_iterator_after(iter, filter->start_after);
    ++
    ++	if (ret)
    ++		return ret;
    ++
    ++	return do_for_each_ref_iterator(iter, cb, cb_data);
    ++}
     +
      /*
       * This is the same as for_each_fullref_in(), but it tries to iterate
       * only over the patterns we'll care about. Note that it _doesn't_ do a full
     @@ ref-filter.c: static int for_each_fullref_in_pattern(struct ref_filter *filter,
    - 				       each_ref_fn cb,
    - 				       void *cb_data)
      {
    -+	struct ref_iterator *iter;
    -+	int flags = 0, ret = 0;
    -+
      	if (filter->kind & FILTER_REFS_ROOT_REFS) {
      		/* In this case, we want to print all refs including root refs. */
     -		return refs_for_each_include_root_refs(get_main_ref_store(the_repository),
     -						       cb, cb_data);
    -+		flags |= DO_FOR_EACH_INCLUDE_ROOT_REFS;
    -+		goto non_prefix_iter;
    ++		return for_each_fullref_with_seek(filter, cb, cb_data,
    ++						  DO_FOR_EACH_INCLUDE_ROOT_REFS);
      	}
      
      	if (!filter->match_as_path) {
    @@ ref-filter.c: static int for_each_fullref_in_pattern(struct ref_filter *filter,
      		 */
     -		return refs_for_each_fullref_in(get_main_ref_store(the_repository),
     -						"", NULL, cb, cb_data);
    -+		goto non_prefix_iter;
    ++		return for_each_fullref_with_seek(filter, cb, cb_data, 0);
      	}
      
      	if (filter->ignore_case) {
    @@ ref-filter.c: static int for_each_fullref_in_pattern(struct ref_filter *filter,
      		 */
     -		return refs_for_each_fullref_in(get_main_ref_store(the_repository),
     -						"", NULL, cb, cb_data);
    -+		goto non_prefix_iter;
    ++		return for_each_fullref_with_seek(filter, cb, cb_data, 0);
      	}
      
      	if (!filter->name_patterns[0]) {
      		/* no patterns; we have to look at everything */
     -		return refs_for_each_fullref_in(get_main_ref_store(the_repository),
     -						 "", filter->exclude.v, cb, cb_data);
    -+		goto non_prefix_iter;
    ++		return for_each_fullref_with_seek(filter, cb, cb_data, 0);
      	}
      
      	return refs_for_each_fullref_in_prefixes(get_main_ref_store(the_repository),
    - 						 NULL, filter->name_patterns,
    - 						 filter->exclude.v,
    - 						 cb, cb_data);
    -+
    -+non_prefix_iter:
    -+	iter = refs_ref_iterator_begin(get_main_ref_store(the_repository), "",
    -+				       NULL, 0, flags);
    -+	if (filter->start_after)
    -+		ret = start_ref_iterator_after(iter, filter->start_after);
    -+
    -+	if (ret)
    -+		return ret;
    -+
    -+	return do_for_each_ref_iterator(iter, cb, cb_data);
    - }
    +@@ ref-filter.c: void filter_is_base(struct repository *r,
      
    - /*
    -@@ ref-filter.c: static int do_filter_refs(struct ref_filter *filter, unsigned int type, each_ref
    - 	init_contains_cache(&filter->internal.no_contains_cache);
    + static int do_filter_refs(struct ref_filter *filter, unsigned int type, each_ref_fn fn, void *cb_data)
    + {
    ++	const char *prefix = NULL;
    + 	int ret = 0;
      
    - 	/*  Simple per-ref filtering */
    --	if (!filter->kind)
    -+	if (!filter->kind) {
    - 		die("filter_refs: invalid type");
    --	else {
    -+	} else {
    -+		const char *prefix = NULL;
    -+
    - 		/*
    - 		 * For common cases where we need only branches or remotes or tags,
    - 		 * we only iterate through those refs. If a mix of refs is needed,
    + 	filter->kind = type & FILTER_REFS_KIND_MASK;
     @@ ref-filter.c: static int do_filter_refs(struct ref_filter *filter, unsigned int type, each_ref
    - 		 * of filter_ref_kind().
    - 		 */
    - 		if (filter->kind == FILTER_REFS_BRANCHES)
    --			ret = refs_for_each_fullref_in(get_main_ref_store(the_repository),
    --						       "refs/heads/", NULL,
    --						       fn, cb_data);
    -+			prefix = "refs/heads/";
    - 		else if (filter->kind == FILTER_REFS_REMOTES)
    --			ret = refs_for_each_fullref_in(get_main_ref_store(the_repository),
    --						       "refs/remotes/", NULL,
    --						       fn, cb_data);
    -+			prefix = "refs/remotes/";
    - 		else if (filter->kind == FILTER_REFS_TAGS)
    --			ret = refs_for_each_fullref_in(get_main_ref_store(the_repository),
    --						       "refs/tags/", NULL, fn,
    --						       cb_data);
    --		else if (filter->kind & FILTER_REFS_REGULAR)
    -+			prefix = "refs/tags/";
    + 	 * of filter_ref_kind().
    + 	 */
    + 	if (filter->kind == FILTER_REFS_BRANCHES)
    +-		ret = refs_for_each_fullref_in(get_main_ref_store(the_repository),
    +-					       "refs/heads/", NULL,
    +-					       fn, cb_data);
    ++		prefix = "refs/heads/";
    + 	else if (filter->kind == FILTER_REFS_REMOTES)
    +-		ret = refs_for_each_fullref_in(get_main_ref_store(the_repository),
    +-					       "refs/remotes/", NULL,
    +-					       fn, cb_data);
    ++		prefix = "refs/remotes/";
    + 	else if (filter->kind == FILTER_REFS_TAGS)
    +-		ret = refs_for_each_fullref_in(get_main_ref_store(the_repository),
    +-					       "refs/tags/", NULL, fn,
    +-					       cb_data);
    +-	else if (filter->kind & FILTER_REFS_REGULAR)
    ++		prefix = "refs/tags/";
     +
    -+		if (prefix) {
    -+			struct ref_iterator *iter;
    ++	if (prefix) {
    ++		struct ref_iterator *iter;
     +
    -+			iter = refs_ref_iterator_begin(get_main_ref_store(the_repository),
    -+						       "", NULL, 0, 0);
    ++		iter = refs_ref_iterator_begin(get_main_ref_store(the_repository),
    ++					       "", NULL, 0, 0);
     +
    -+			if (filter->start_after)
    -+				ret = start_ref_iterator_after(iter, filter->start_after);
    -+			else if (prefix)
    -+				ret = ref_iterator_seek(iter, prefix, 1);
    ++		if (filter->start_after)
    ++			ret = start_ref_iterator_after(iter, filter->start_after);
    ++		else if (prefix)
    ++			ret = ref_iterator_seek(iter, prefix, 1);
     +
    -+			if (!ret)
    -+				ret = do_for_each_ref_iterator(iter, fn, cb_data);
    -+		} else if (filter->kind & FILTER_REFS_REGULAR) {
    - 			ret = for_each_fullref_in_pattern(filter, fn, cb_data);
    -+		}
    ++		if (!ret)
    ++			ret = do_for_each_ref_iterator(iter, fn, cb_data);
    ++	} else if (filter->kind & FILTER_REFS_REGULAR) {
    + 		ret = for_each_fullref_in_pattern(filter, fn, cb_data);
    ++	}
      
    - 		/*
    - 		 * When printing all ref types, HEAD is already included,
    + 	/*
    + 	 * When printing all ref types, HEAD is already included,
     
      ## ref-filter.h ##
     @@ ref-filter.h: struct ref_array {
    @@ t/t6302-for-each-ref-filter.sh: test_expect_success 'validate worktree atom' '
     +	refs/tags/three
     +	refs/tags/two
     +	EOF
    -+	git for-each-ref --format="%(refname)" --start-after=refs/lost >actual &&
    ++	git for-each-ref --format="%(refname)" --start-after=refs/odd/ >actual &&
     +	test_cmp expect actual
     +'
     +
    @@ t/t6302-for-each-ref-filter.sh: test_expect_success 'validate worktree atom' '
     +	refs/tags/three
     +	refs/tags/two
     +	EOF
    -+	git for-each-ref --format="%(refname)" --start-after=refs/odd/ >actual &&
    ++	git for-each-ref --format="%(refname)" --start-after=refs/lost >actual &&
     +	test_cmp expect actual
     +'
     +

base-commit: cf6f63ea6bf35173e02e18bdc6a4ba41288acff9
change-id: 20250605-306-git-for-each-ref-pagination-0ba8a29ae646

Thanks
- Karthik

#70 lore raw

PATCH v5 1/5 refs: expose `ref_iterator` via 'refs.h'

The ref_iterator is an internal structure to the 'refs/' sub-directory, which allows iteration over refs. All reference iteration is built on top of these iterators.

External clients of the 'refs' subsystem use the various 'refs_for_each...()' functions to iterate over refs. However since these are wrapper functions, each combination of functionality requires a new wrapper function. This is not feasible as the functions pile up with the increase in requirements. Expose the internal reference iterator, so advanced users can mix and match options as needed.

---
 refs.h               | 147 +++++++++++++++++++++++++++++++++++++++++++++++++++
 refs/refs-internal.h | 145 +-------------------------------------------------
 2 files changed, 149 insertions(+), 143 deletions(-)
refs.h +147 -0
index 46a6008e07..7c21aaef3d 100644
@@ -1190,4 +1190,151 @@ int repo_migrate_ref_storage_format(struct repository *repo,
 				    unsigned int flags,
 				    struct strbuf *err);
 
+/*
+ * Reference iterators
+ *
+ * A reference iterator encapsulates the state of an in-progress
+ * iteration over references. Create an instance of `struct
+ * ref_iterator` via one of the functions in this module.
+ *
+ * A freshly-created ref_iterator doesn't yet point at a reference. To
+ * advance the iterator, call ref_iterator_advance(). If successful,
+ * this sets the iterator's refname, oid, and flags fields to describe
+ * the next reference and returns ITER_OK. The data pointed at by
+ * refname and oid belong to the iterator; if you want to retain them
+ * after calling ref_iterator_advance() again or calling
+ * ref_iterator_free(), you must make a copy. When the iteration has
+ * been exhausted, ref_iterator_advance() releases any resources
+ * associated with the iteration, frees the ref_iterator object, and
+ * returns ITER_DONE. If you want to abort the iteration early, call
+ * ref_iterator_free(), which also frees the ref_iterator object and
+ * any associated resources. If there was an internal error advancing
+ * to the next entry, ref_iterator_advance() aborts the iteration,
+ * frees the ref_iterator, and returns ITER_ERROR.
+ *
+ * The reference currently being looked at can be peeled by calling
+ * ref_iterator_peel(). This function is often faster than peel_ref(),
+ * so it should be preferred when iterating over references.
+ *
+ * Putting it all together, a typical iteration looks like this:
+ *
+ *     int ok;
+ *     struct ref_iterator *iter = ...;
+ *
+ *     while ((ok = ref_iterator_advance(iter)) == ITER_OK) {
+ *             if (want_to_stop_iteration()) {
+ *                     ok = ITER_DONE;
+ *                     break;
+ *             }
+ *
+ *             // Access information about the current reference:
+ *             if (!(iter->flags & REF_ISSYMREF))
+ *                     printf("%s is %s\n", iter->refname, oid_to_hex(iter->oid));
+ *
+ *             // If you need to peel the reference:
+ *             ref_iterator_peel(iter, &oid);
+ *     }
+ *
+ *     if (ok != ITER_DONE)
+ *             handle_error();
+ *     ref_iterator_free(iter);
+ */
+struct ref_iterator;
+
+/*
+ * These flags are passed to refs_ref_iterator_begin() (and do_for_each_ref(),
+ * which feeds it).
+ */
+enum do_for_each_ref_flags {
+	/*
+	 * Include broken references in a do_for_each_ref*() iteration, which
+	 * would normally be omitted. This includes both refs that point to
+	 * missing objects (a true repository corruption), ones with illegal
+	 * names (which we prefer not to expose to callers), as well as
+	 * dangling symbolic refs (i.e., those that point to a non-existent
+	 * ref; this is not a corruption, but as they have no valid oid, we
+	 * omit them from normal iteration results).
+	 */
+	DO_FOR_EACH_INCLUDE_BROKEN = (1 << 0),
+
+	/*
+	 * Only include per-worktree refs in a do_for_each_ref*() iteration.
+	 * Normally this will be used with a files ref_store, since that's
+	 * where all reference backends will presumably store their
+	 * per-worktree refs.
+	 */
+	DO_FOR_EACH_PER_WORKTREE_ONLY = (1 << 1),
+
+	/*
+	 * Omit dangling symrefs from output; this only has an effect with
+	 * INCLUDE_BROKEN, since they are otherwise not included at all.
+	 */
+	DO_FOR_EACH_OMIT_DANGLING_SYMREFS = (1 << 2),
+
+	/*
+	 * Include root refs i.e. HEAD and pseudorefs along with the regular
+	 * refs.
+	 */
+	DO_FOR_EACH_INCLUDE_ROOT_REFS = (1 << 3),
+};
+
+/*
+ * Return an iterator that goes over each reference in `refs` for
+ * which the refname begins with prefix. If trim is non-zero, then
+ * trim that many characters off the beginning of each refname.
+ * The output is ordered by refname.
+ */
+struct ref_iterator *refs_ref_iterator_begin(
+	struct ref_store *refs,
+	const char *prefix, const char **exclude_patterns,
+	int trim, enum do_for_each_ref_flags flags);
+
+/*
+ * Advance the iterator to the first or next item and return ITER_OK.
+ * If the iteration is exhausted, free the resources associated with
+ * the ref_iterator and return ITER_DONE. On errors, free the iterator
+ * resources and return ITER_ERROR. It is a bug to use ref_iterator or
+ * call this function again after it has returned ITER_DONE or
+ * ITER_ERROR.
+ */
+int ref_iterator_advance(struct ref_iterator *ref_iterator);
+
+/*
+ * Seek the iterator to the first reference with the given prefix.
+ * The prefix is matched as a literal string, without regard for path
+ * separators. If prefix is NULL or the empty string, seek the iterator to the
+ * first reference again.
+ *
+ * This function is expected to behave as if a new ref iterator with the same
+ * prefix had been created, but allows reuse of iterators and thus may allow
+ * the backend to optimize. Parameters other than the prefix that have been
+ * passed when creating the iterator will remain unchanged.
+ *
+ * Returns 0 on success, a negative error code otherwise.
+ */
+int ref_iterator_seek(struct ref_iterator *ref_iterator,
+		      const char *prefix);
+
+/*
+ * If possible, peel the reference currently being viewed by the
+ * iterator. Return 0 on success.
+ */
+int ref_iterator_peel(struct ref_iterator *ref_iterator,
+		      struct object_id *peeled);
+
+/* Free the reference iterator and any associated resources. */
+void ref_iterator_free(struct ref_iterator *ref_iterator);
+
+/*
+ * The common backend for the for_each_*ref* functions. Call fn for
+ * each reference in iter. If the iterator itself ever returns
+ * ITER_ERROR, return -1. If fn ever returns a non-zero value, stop
+ * the iteration and return that value. Otherwise, return 0. In any
+ * case, free the iterator when done. This function is basically an
+ * adapter between the callback style of reference iteration and the
+ * iterator style.
+ */
+int do_for_each_ref_iterator(struct ref_iterator *iter,
+			     each_ref_fn fn, void *cb_data);
+
 #endif /* REFS_H */
refs/refs-internal.h +2 -143

169 lines not on this page · raw message

Signature · 1 line
2.49.0
#71 lore raw

PATCH v5 2/5 ref-cache: remove unused function 'find_ref_entry()'

In reply to Karthik Nayak

The 'find_ref_entry' function is no longer used, so remove it.

---
 refs/ref-cache.c | 14 --------------
 refs/ref-cache.h |  7 -------
 2 files changed, 21 deletions(-)
refs/ref-cache.c +0 -14
index c1f1bab1d5..8aaffa8c6b 100644
@@ -194,20 +194,6 @@ static struct ref_dir *find_containing_dir(struct ref_dir *dir,
 	return dir;
 }
 
-struct ref_entry *find_ref_entry(struct ref_dir *dir, const char *refname)
-{
-	int entry_index;
-	struct ref_entry *entry;
-	dir = find_containing_dir(dir, refname);
-	if (!dir)
-		return NULL;
-	entry_index = search_ref_dir(dir, refname, strlen(refname));
-	if (entry_index == -1)
-		return NULL;
-	entry = dir->entries[entry_index];
-	return (entry->flag & REF_DIR) ? NULL : entry;
-}
-
 /*
  * Emit a warning and return true iff ref1 and ref2 have the same name
  * and the same oid. Die if they have the same name but different
refs/ref-cache.h +0 -7
index 5f04e518c3..f635d2d824 100644
@@ -201,13 +201,6 @@ void free_ref_cache(struct ref_cache *cache);
  */
 void add_entry_to_dir(struct ref_dir *dir, struct ref_entry *entry);
 
-/*
- * Find the value entry with the given name in dir, sorting ref_dirs
- * and recursing into subdirectories as necessary.  If the name is not
- * found or it corresponds to a directory entry, return NULL.
- */
-struct ref_entry *find_ref_entry(struct ref_dir *dir, const char *refname);
-
 /*
  * Start iterating over references in `cache`. If `prefix` is
  * specified, only include references whose names start with that
Signature · 1 line
2.49.0
#72 lore raw

Re: [PATCH v5 2/5] ref-cache: remove unused function 'find_ref_entry()'

Quoted 1 line · Karthik Nayak

The 'find_ref_entry' function is no longer used, so remove it.

If my spelunking is correct, ba1c052f (ref_store: implement refs_peel_ref() generically, 2017-09-25) is the commit that removed the last caller of it. Which may be worth noting here.

Quoted 49 lines
---
 refs/ref-cache.c | 14 --------------
 refs/ref-cache.h |  7 -------
 2 files changed, 21 deletions(-)
refs/ref-cache.c +0 -14
index c1f1bab1d5..8aaffa8c6b 100644
@@ -194,20 +194,6 @@ static struct ref_dir *find_containing_dir(struct ref_dir *dir,
 	return dir;
 }
 
-struct ref_entry *find_ref_entry(struct ref_dir *dir, const char *refname)
-{
-	int entry_index;
-	struct ref_entry *entry;
-	dir = find_containing_dir(dir, refname);
-	if (!dir)
-		return NULL;
-	entry_index = search_ref_dir(dir, refname, strlen(refname));
-	if (entry_index == -1)
-		return NULL;
-	entry = dir->entries[entry_index];
-	return (entry->flag & REF_DIR) ? NULL : entry;
-}
-
 /*
  * Emit a warning and return true iff ref1 and ref2 have the same name
  * and the same oid. Die if they have the same name but different
refs/ref-cache.h +0 -7
index 5f04e518c3..f635d2d824 100644
@@ -201,13 +201,6 @@ void free_ref_cache(struct ref_cache *cache);
  */
 void add_entry_to_dir(struct ref_dir *dir, struct ref_entry *entry);
 
-/*
- * Find the value entry with the given name in dir, sorting ref_dirs
- * and recursing into subdirectories as necessary.  If the name is not
- * found or it corresponds to a directory entry, return NULL.
- */
-struct ref_entry *find_ref_entry(struct ref_dir *dir, const char *refname);
-
 /*
  * Start iterating over references in `cache`. If `prefix` is
  * specified, only include references whose names start with that
#73 lore raw

Re: [PATCH v5 2/5] ref-cache: remove unused function 'find_ref_entry()'

Quoted 8 lines · Junio C Hamano
Quoted 1 line · Karthik Nayak

The 'find_ref_entry' function is no longer used, so remove it.

If my spelunking is correct, ba1c052f (ref_store: implement refs_peel_ref() generically, 2017-09-25) is the commit that removed the last caller of it. Which may be worth noting here.

Indeed, I could also verify this by running

$ git log -S find_ref_entry --oneline
2c90b85801 ref-cache: remove unused function 'find_ref_entry()'
ba1c052fa6 ref_store: implement `refs_peel_ref()` generically
9939b33d6a packed-backend: rip out some now-unused code
....

And looking at ba1c052fa6. I should've done this before. But thanks for the digging!

I plan to address a few comments on this version, but I also see that you've merged it to master. Should I raise followups for noteworthy changes or a new version?

Karthik

#74 lore raw

Re: [PATCH v5 2/5] ref-cache: remove unused function 'find_ref_entry()'

Quoted 19 lines · Karthik Nayak
Quoted 8 lines · Junio C Hamano
Quoted 1 line · Karthik Nayak

The 'find_ref_entry' function is no longer used, so remove it.

If my spelunking is correct, ba1c052f (ref_store: implement refs_peel_ref() generically, 2017-09-25) is the commit that removed the last caller of it. Which may be worth noting here.

Indeed, I could also verify this by running

$ git log -S find_ref_entry --oneline
2c90b85801 ref-cache: remove unused function 'find_ref_entry()'
ba1c052fa6 ref_store: implement `refs_peel_ref()` generically
9939b33d6a packed-backend: rip out some now-unused code
....
Quoted 6 lines

And looking at ba1c052fa6. I should've done this before. But thanks for the digging!

I plan to address a few comments on this version, but I also see that you've merged it to master. Should I raise followups for noteworthy changes or a new version?

You mean it is now in 'next'? Yes, please give incremental patches. Finding some more gotchas even after v5 is a sign that they are tricky enough that deserve separate explanation from the main part of the series. And I think the msan one Kyle and Peff found is also tricky enough that evaded reviewers' eyes.

Thanks.

#75 lore raw

PATCH v5 3/5 refs: selectively set prefix in the seek functions

In reply to Karthik Nayak

The ref iterator exposes a ref_iterator_seek() function. The name suggests that this would seek the iterator to a specific reference in some ways similar to how fseek() works for the filesystem.

However, the function actually sets the prefix for refs iteration. So further iteration would only yield references which match the particular prefix. This is a bit confusing.

Let's add a 'flags' field to the function, which when set with the 'REF_ITERATOR_SEEK_SET_PREFIX' flag, will set the prefix for the iteration in-line with the existing behavior. Otherwise, the reference backends will simply seek to the specified reference and clears any previously set prefix. This allows users to start iteration from a specific reference.

In the packed and reftable backend, since references are available in a sorted list, the changes are simply setting the prefix if needed. The changes on the files-backend are a little more involved, since the files backend uses the 'ref-cache' mechanism. We move out the existing logic within cache_ref_iterator_seek() to cache_ref_iterator_set_prefix() which is called when the 'REF_ITERATOR_SEEK_SET_PREFIX' flag is set. We then parse the provided seek string and set the required levels and their indexes to ensure that seeking is possible.

---
 refs.c                  |  6 ++--
 refs.h                  | 26 +++++++++------
 refs/debug.c            |  7 ++--
 refs/files-backend.c    |  7 ++--
 refs/iterator.c         | 26 ++++++++-------
 refs/packed-backend.c   | 17 ++++++----
 refs/ref-cache.c        | 85 ++++++++++++++++++++++++++++++++++++++++++++++---
 refs/refs-internal.h    |  7 ++--
 refs/reftable-backend.c | 21 ++++++++----
 9 files changed, 152 insertions(+), 50 deletions(-)
refs.c +3 -3
index dce5c49ca2..243e6898b8 100644
@@ -2666,12 +2666,12 @@ enum ref_transaction_error refs_verify_refnames_available(struct ref_store *refs
 		if (!initial_transaction) {
 			int ok;
 
-			if (!iter) {
+			if (!iter)
 				iter = refs_ref_iterator_begin(refs, dirname.buf, NULL, 0,
 							       DO_FOR_EACH_INCLUDE_BROKEN);
-			} else if (ref_iterator_seek(iter, dirname.buf) < 0) {
+			else if (ref_iterator_seek(iter, dirname.buf,
+						   REF_ITERATOR_SEEK_SET_PREFIX) < 0)
 				goto cleanup;
-			}
 
 			while ((ok = ref_iterator_advance(iter)) == ITER_OK) {
 				if (skip &&
refs.h +17 -9
index 7c21aaef3d..e6780a8848 100644
@@ -1299,21 +1299,29 @@ struct ref_iterator *refs_ref_iterator_begin(
  */
 int ref_iterator_advance(struct ref_iterator *ref_iterator);
 
+enum ref_iterator_seek_flag {
+	/*
+	 * When the REF_ITERATOR_SEEK_SET_PREFIX flag is set, the iterator's prefix is
+	 * updated to match the provided string, affecting all subsequent iterations. If
+	 * not, the iterator seeks to the specified reference and clears any previously
+	 * set prefix.
+	 */
+	REF_ITERATOR_SEEK_SET_PREFIX = (1 << 0),
+};
+
 /*
- * Seek the iterator to the first reference with the given prefix.
- * The prefix is matched as a literal string, without regard for path
- * separators. If prefix is NULL or the empty string, seek the iterator to the
+ * Seek the iterator to the first reference matching the given seek string.
+ * The seek string is matched as a literal string, without regard for path
+ * separators. If seek is NULL or the empty string, seek the iterator to the
  * first reference again.
  *
- * This function is expected to behave as if a new ref iterator with the same
- * prefix had been created, but allows reuse of iterators and thus may allow
- * the backend to optimize. Parameters other than the prefix that have been
- * passed when creating the iterator will remain unchanged.
+ * This function is expected to behave as if a new ref iterator has been
+ * created, but allows reuse of existing iterators for optimization.
  *
  * Returns 0 on success, a negative error code otherwise.
  */
-int ref_iterator_seek(struct ref_iterator *ref_iterator,
-		      const char *prefix);
+int ref_iterator_seek(struct ref_iterator *ref_iterator, const char *refname,
+		      unsigned int flags);
 
 /*
  * If possible, peel the reference currently being viewed by the
refs/debug.c +4 -3
index 485e3079d7..da300efaf3 100644
@@ -170,12 +170,13 @@ static int debug_ref_iterator_advance(struct ref_iterator *ref_iterator)
 }
 
 static int debug_ref_iterator_seek(struct ref_iterator *ref_iterator,
-				   const char *prefix)
+				   const char *refname, unsigned int flags)
 {
 	struct debug_ref_iterator *diter =
 		(struct debug_ref_iterator *)ref_iterator;
-	int res = diter->iter->vtable->seek(diter->iter, prefix);
-	trace_printf_key(&trace_refs, "iterator_seek: %s: %d\n", prefix ? prefix : "", res);
+	int res = diter->iter->vtable->seek(diter->iter, refname, flags);
+	trace_printf_key(&trace_refs, "iterator_seek: %s flags: %d: %d\n",
+			 refname ? refname : "", flags, res);
 	return res;
 }
 
refs/files-backend.c +4 -3
index bf6f89b1d1..8b282f2a60 100644
@@ -929,11 +929,11 @@ static int files_ref_iterator_advance(struct ref_iterator *ref_iterator)
 }
 
 static int files_ref_iterator_seek(struct ref_iterator *ref_iterator,
-				   const char *prefix)
+				   const char *refname, unsigned int flags)
 {
 	struct files_ref_iterator *iter =
 		(struct files_ref_iterator *)ref_iterator;
-	return ref_iterator_seek(iter->iter0, prefix);
+	return ref_iterator_seek(iter->iter0, refname, flags);
 }
 
 static int files_ref_iterator_peel(struct ref_iterator *ref_iterator,
@@ -2316,7 +2316,8 @@ static int files_reflog_iterator_advance(struct ref_iterator *ref_iterator)
 }
 
 static int files_reflog_iterator_seek(struct ref_iterator *ref_iterator UNUSED,
-				      const char *prefix UNUSED)
+				      const char *refname UNUSED,
+				      unsigned int flags UNUSED)
 {
 	BUG("ref_iterator_seek() called for reflog_iterator");
 }
refs/iterator.c +15 -11
index 766d96e795..17ef841d8a 100644
@@ -15,10 +15,10 @@ int ref_iterator_advance(struct ref_iterator *ref_iterator)
 	return ref_iterator->vtable->advance(ref_iterator);
 }
 
-int ref_iterator_seek(struct ref_iterator *ref_iterator,
-		      const char *prefix)
+int ref_iterator_seek(struct ref_iterator *ref_iterator, const char *refname,
+		      unsigned int flags)
 {
-	return ref_iterator->vtable->seek(ref_iterator, prefix);
+	return ref_iterator->vtable->seek(ref_iterator, refname, flags);
 }
 
 int ref_iterator_peel(struct ref_iterator *ref_iterator,
@@ -57,7 +57,8 @@ static int empty_ref_iterator_advance(struct ref_iterator *ref_iterator UNUSED)
 }
 
 static int empty_ref_iterator_seek(struct ref_iterator *ref_iterator UNUSED,
-				   const char *prefix UNUSED)
+				   const char *refname UNUSED,
+				   unsigned int flags UNUSED)
 {
 	return 0;
 }
@@ -224,7 +225,7 @@ static int merge_ref_iterator_advance(struct ref_iterator *ref_iterator)
 }
 
 static int merge_ref_iterator_seek(struct ref_iterator *ref_iterator,
-				   const char *prefix)
+				   const char *refname, unsigned int flags)
 {
 	struct merge_ref_iterator *iter =
 		(struct merge_ref_iterator *)ref_iterator;
@@ -234,11 +235,11 @@ static int merge_ref_iterator_seek(struct ref_iterator *ref_iterator,
 	iter->iter0 = iter->iter0_owned;
 	iter->iter1 = iter->iter1_owned;
 
-	ret = ref_iterator_seek(iter->iter0, prefix);
+	ret = ref_iterator_seek(iter->iter0, refname, flags);
 	if (ret < 0)
 		return ret;
 
-	ret = ref_iterator_seek(iter->iter1, prefix);
+	ret = ref_iterator_seek(iter->iter1, refname, flags);
 	if (ret < 0)
 		return ret;
 
@@ -407,13 +408,16 @@ static int prefix_ref_iterator_advance(struct ref_iterator *ref_iterator)
 }
 
 static int prefix_ref_iterator_seek(struct ref_iterator *ref_iterator,
-				    const char *prefix)
+				    const char *refname, unsigned int flags)
 {
 	struct prefix_ref_iterator *iter =
 		(struct prefix_ref_iterator *)ref_iterator;
-	free(iter->prefix);
-	iter->prefix = xstrdup_or_null(prefix);
-	return ref_iterator_seek(iter->iter0, prefix);
+
+	if (flags & REF_ITERATOR_SEEK_SET_PREFIX) {
+		free(iter->prefix);
+		iter->prefix = xstrdup_or_null(refname);
+	}
+	return ref_iterator_seek(iter->iter0, refname, flags);
 }
 
 static int prefix_ref_iterator_peel(struct ref_iterator *ref_iterator,
refs/packed-backend.c +11 -6
index 7fd73a0e6d..5fa4ae6655 100644
@@ -1004,19 +1004,23 @@ static int packed_ref_iterator_advance(struct ref_iterator *ref_iterator)
 }
 
 static int packed_ref_iterator_seek(struct ref_iterator *ref_iterator,
-				    const char *prefix)
+				    const char *refname, unsigned int flags)
 {
 	struct packed_ref_iterator *iter =
 		(struct packed_ref_iterator *)ref_iterator;
 	const char *start;
 
-	if (prefix && *prefix)
-		start = find_reference_location(iter->snapshot, prefix, 0);
+	if (refname && *refname)
+		start = find_reference_location(iter->snapshot, refname, 0);
 	else
 		start = iter->snapshot->start;
 
-	free(iter->prefix);
-	iter->prefix = xstrdup_or_null(prefix);
+	/* Unset any previously set prefix */
+	FREE_AND_NULL(iter->prefix);
+
+	if (flags & REF_ITERATOR_SEEK_SET_PREFIX)
+		iter->prefix = xstrdup_or_null(refname);
+
 	iter->pos = start;
 	iter->eof = iter->snapshot->eof;
 
@@ -1194,7 +1198,8 @@ static struct ref_iterator *packed_ref_iterator_begin(
 	iter->repo = ref_store->repo;
 	iter->flags = flags;
 
-	if (packed_ref_iterator_seek(&iter->base, prefix) < 0) {
+	if (packed_ref_iterator_seek(&iter->base, prefix,
+				     REF_ITERATOR_SEEK_SET_PREFIX) < 0) {
 		ref_iterator_free(&iter->base);
 		return NULL;
 	}
refs/ref-cache.c +80 -5

104 lines not on this page · raw message

refs/refs-internal.h +4 -3
index 03f5df04d5..40c1c0f93d 100644
@@ -353,11 +353,12 @@ void base_ref_iterator_init(struct ref_iterator *iter,
 typedef int ref_iterator_advance_fn(struct ref_iterator *ref_iterator);
 
 /*
- * Seek the iterator to the first reference matching the given prefix. Should
- * behave the same as if a new iterator was created with the same prefix.
+ * Seek the iterator to the first matching reference. If the
+ * REF_ITERATOR_SEEK_SET_PREFIX flag is set, it would behave the same as if a
+ * new iterator was created with the provided refname as prefix.
  */
 typedef int ref_iterator_seek_fn(struct ref_iterator *ref_iterator,
-				 const char *prefix);
+				 const char *refname, unsigned int flags);
 
 /*
  * Peels the current ref, returning 0 for success or -1 for failure.
refs/reftable-backend.c +14 -7
index 4c3817f4ec..c3d48cc412 100644
@@ -719,15 +719,20 @@ static int reftable_ref_iterator_advance(struct ref_iterator *ref_iterator)
 }
 
 static int reftable_ref_iterator_seek(struct ref_iterator *ref_iterator,
-				      const char *prefix)
+				      const char *refname, unsigned int flags)
 {
 	struct reftable_ref_iterator *iter =
 		(struct reftable_ref_iterator *)ref_iterator;
 
-	free(iter->prefix);
-	iter->prefix = xstrdup_or_null(prefix);
-	iter->prefix_len = prefix ? strlen(prefix) : 0;
-	iter->err = reftable_iterator_seek_ref(&iter->iter, prefix);
+	/* Unset any previously set prefix */
+	FREE_AND_NULL(iter->prefix);
+	iter->prefix_len = 0;
+
+	if (flags & REF_ITERATOR_SEEK_SET_PREFIX) {
+		iter->prefix = xstrdup_or_null(refname);
+		iter->prefix_len = refname ? strlen(refname) : 0;
+	}
+	iter->err = reftable_iterator_seek_ref(&iter->iter, refname);
 
 	return iter->err;
 }
@@ -839,7 +844,8 @@ static struct reftable_ref_iterator *ref_iterator_for_stack(struct reftable_ref_
 	if (ret)
 		goto done;
 
-	ret = reftable_ref_iterator_seek(&iter->base, prefix);
+	ret = reftable_ref_iterator_seek(&iter->base, prefix,
+					 REF_ITERATOR_SEEK_SET_PREFIX);
 	if (ret)
 		goto done;
 
@@ -2042,7 +2048,8 @@ static int reftable_reflog_iterator_advance(struct ref_iterator *ref_iterator)
 }
 
 static int reftable_reflog_iterator_seek(struct ref_iterator *ref_iterator UNUSED,
-					 const char *prefix UNUSED)
+					 const char *refname UNUSED,
+					 unsigned int flags UNUSED)
 {
 	BUG("reftable reflog iterator cannot be seeked");
 	return -1;
Signature · 1 line
2.49.0
#76 lore raw

Re: [PATCH v5 3/5] refs: selectively set prefix in the seek functions

Review

Quoted 11 lines · Karthik Nayak
(quoted diff) +10 -0
+static int cache_ref_iterator_seek(struct ref_iterator *ref_iterator,
+				   const char *refname, unsigned int flags)
[...]
+		do {
+			int len, idx;
+			int cmp = 0;
+
+			sort_ref_dir(dir);
+
+			slash = strchr(slash, '/');
+			len = slash ? slash - refname : (int)strlen(refname);

I was looking at this code due to a nearby thread and noticed this funny cast to int. I guess you added it to silence -Wsign-compare, but Why are we not using a size_t in the first place?

This kind of conversion can sometimes have security implications because a very large "refname" would cause "len" to become negative (i.e., if it's between 2GB and 4GB).

In this particular case it ends up cast back to a size_t via strncmp:

Review

Quoted 5 lines
(quoted diff) +5 -0
+			for (idx = 0; idx < dir->nr; idx++) {
+				cmp = strncmp(refname, dir->entries[idx]->name, len);
+				if (cmp <= 0)
+					break;
+			}

so we get the original value back. We'd still get truncation for a refname value over 4GB, which would presumably give us a slightly wrong answer. But I don't think we'd ever look outside the array.

Such sizes are probably unlikely if we are feeding filesystem paths. But we probably should not set a bad example, and just do;

refs/ref-cache.c +3 -2
index 1d95b56d40..3949d145e8 100644
@@ -498,13 +498,14 @@ static int cache_ref_iterator_seek(struct ref_iterator *ref_iterator,
 		 * indexing to each level as needed.
 		 */
 		do {
-			int len, idx;
+			size_t len;
+			int idx;
 			int cmp = 0;
 
 			sort_ref_dir(dir);
 
 			slash = strchr(slash, '/');
-			len = slash ? slash - refname : (int)strlen(refname);
+			len = slash ? slash - refname : strlen(refname);
 
 			for (idx = 0; idx < dir->nr; idx++) {
 				cmp = strncmp(refname, dir->entries[idx]->name, len);

-Peff

#77 lore raw

Re: [PATCH v5 3/5] refs: selectively set prefix in the seek functions

Quoted 18 lines · Jeff King

Review

Quoted 11 lines · Karthik Nayak
(quoted diff) +10 -0
+static int cache_ref_iterator_seek(struct ref_iterator *ref_iterator,
+				   const char *refname, unsigned int flags)
[...]
+		do {
+			int len, idx;
+			int cmp = 0;
+
+			sort_ref_dir(dir);
+
+			slash = strchr(slash, '/');
+			len = slash ? slash - refname : (int)strlen(refname);

I was looking at this code due to a nearby thread and noticed this funny cast to int. I guess you added it to silence -Wsign-compare, but Why are we not using a size_t in the first place?

That's an oversight from my side.

Quoted 4 lines

This kind of conversion can sometimes have security implications because a very large "refname" would cause "len" to become negative (i.e., if it's between 2GB and 4GB).

Indeed, I didn't think of that.

Quoted 12 lines

In this particular case it ends up cast back to a size_t via strncmp:

Review

Quoted 5 lines
(quoted diff) +5 -0
+			for (idx = 0; idx < dir->nr; idx++) {
+				cmp = strncmp(refname, dir->entries[idx]->name, len);
+				if (cmp <= 0)
+					break;
+			}

so we get the original value back. We'd still get truncation for a refname value over 4GB, which would presumably give us a slightly wrong answer. But I don't think we'd ever look outside the array.

Quoted 3 lines

Such sizes are probably unlikely if we are feeding filesystem paths. But we probably should not set a bad example, and just do;

Agreed.

Quoted 13 lines
refs/ref-cache.c +2 -1
index 1d95b56d40..3949d145e8 100644
@@ -498,13 +498,14 @@ static int cache_ref_iterator_seek(struct ref_iterator *ref_iterator,
 		 * indexing to each level as needed.
 		 */
 		do {
-			int len, idx;
+			size_t len;
+			int idx;
 			int cmp = 0;
 

refs/ref-cache.c static int cache_ref_iterator_seek(struct ref_iterator *ref_iterator,

Quoted 10 lines
refs/ref-cache.c +1 -2
 			sort_ref_dir(dir);
 
 			slash = strchr(slash, '/');
-			len = slash ? slash - refname : (int)strlen(refname);
+			len = slash ? slash - refname : strlen(refname);
 
 			for (idx = 0; idx < dir->nr; idx++) {
 				cmp = strncmp(refname, dir->entries[idx]->name, len);
 
-Peff

Thanks, I think we have to typecast slash - refname to size_t, but this is the right way to do it. Thanks for the review!

Karthik

#78 lore raw

Re: [PATCH v5 3/5] refs: selectively set prefix in the seek functions

Quoted 19 lines · Karthik Nayak
Quoted 19 lines
refs/ref-cache.c +3 -2
index 1d95b56d40..3949d145e8 100644
@@ -498,13 +498,14 @@ static int cache_ref_iterator_seek(struct ref_iterator *ref_iterator,
 		 * indexing to each level as needed.
 		 */
 		do {
-			int len, idx;
+			size_t len;
+			int idx;
 			int cmp = 0;
 
 			sort_ref_dir(dir);
 
 			slash = strchr(slash, '/');
-			len = slash ? slash - refname : (int)strlen(refname);
+			len = slash ? slash - refname : strlen(refname);
 
Quoted 7 lines

refs/ref-cache.c static int cache_ref_iterator_seek(struct ref_iterator *ref_iterator,

Quoted 4 lines
refs/ref-cache.c +0 -1
 			for (idx = 0; idx < dir->nr; idx++) {
 				cmp = strncmp(refname, dir->entries[idx]->name, len);
 
-Peff

Thanks, I think we have to typecast slash - refname to size_t, but this is the right way to do it. Thanks for the review!

Ah, yeah. I mistakenly test-compiled without DEVELOPER=1. ;)

I do think that cast is a lesser evil, though. It is a ptrdiff_t, but we know it is correctly unsigned because "slash > refname" via strchr. I wish there was a good way to use the type system to tell the compiler that.

-Peff

#79 lore raw

PATCH v5 4/5 ref-filter: remove unnecessary else clause

In reply to Karthik Nayak

In 'ref-filter.c', there is an 'else' clause within do_filter_refs(). This is unnecessary since the 'if' clause calls die(), which would exit the program. So let's remove the unnecessary 'else' clause. This improves readability since the indentation is also reduced and flow is simpler.

---
 ref-filter.c | 60 ++++++++++++++++++++++++++++++------------------------------
 1 file changed, 30 insertions(+), 30 deletions(-)
ref-filter.c +30 -30
index 7a274633cf..da663c7ac8 100644
@@ -3199,37 +3199,37 @@ static int do_filter_refs(struct ref_filter *filter, unsigned int type, each_ref
 	/*  Simple per-ref filtering */
 	if (!filter->kind)
 		die("filter_refs: invalid type");
-	else {
-		/*
-		 * For common cases where we need only branches or remotes or tags,
-		 * we only iterate through those refs. If a mix of refs is needed,
-		 * we iterate over all refs and filter out required refs with the help
-		 * of filter_ref_kind().
-		 */
-		if (filter->kind == FILTER_REFS_BRANCHES)
-			ret = refs_for_each_fullref_in(get_main_ref_store(the_repository),
-						       "refs/heads/", NULL,
-						       fn, cb_data);
-		else if (filter->kind == FILTER_REFS_REMOTES)
-			ret = refs_for_each_fullref_in(get_main_ref_store(the_repository),
-						       "refs/remotes/", NULL,
-						       fn, cb_data);
-		else if (filter->kind == FILTER_REFS_TAGS)
-			ret = refs_for_each_fullref_in(get_main_ref_store(the_repository),
-						       "refs/tags/", NULL, fn,
-						       cb_data);
-		else if (filter->kind & FILTER_REFS_REGULAR)
-			ret = for_each_fullref_in_pattern(filter, fn, cb_data);
 
-		/*
-		 * When printing all ref types, HEAD is already included,
-		 * so we don't want to print HEAD again.
-		 */
-		if (!ret && !(filter->kind & FILTER_REFS_ROOT_REFS) &&
-		    (filter->kind & FILTER_REFS_DETACHED_HEAD))
-			refs_head_ref(get_main_ref_store(the_repository), fn,
-				      cb_data);
-	}
+	/*
+	 * For common cases where we need only branches or remotes or tags,
+	 * we only iterate through those refs. If a mix of refs is needed,
+	 * we iterate over all refs and filter out required refs with the help
+	 * of filter_ref_kind().
+	 */
+	if (filter->kind == FILTER_REFS_BRANCHES)
+		ret = refs_for_each_fullref_in(get_main_ref_store(the_repository),
+					       "refs/heads/", NULL,
+					       fn, cb_data);
+	else if (filter->kind == FILTER_REFS_REMOTES)
+		ret = refs_for_each_fullref_in(get_main_ref_store(the_repository),
+					       "refs/remotes/", NULL,
+					       fn, cb_data);
+	else if (filter->kind == FILTER_REFS_TAGS)
+		ret = refs_for_each_fullref_in(get_main_ref_store(the_repository),
+					       "refs/tags/", NULL, fn,
+					       cb_data);
+	else if (filter->kind & FILTER_REFS_REGULAR)
+		ret = for_each_fullref_in_pattern(filter, fn, cb_data);
+
+	/*
+	 * When printing all ref types, HEAD is already included,
+	 * so we don't want to print HEAD again.
+	 */
+	if (!ret && !(filter->kind & FILTER_REFS_ROOT_REFS) &&
+	    (filter->kind & FILTER_REFS_DETACHED_HEAD))
+		refs_head_ref(get_main_ref_store(the_repository), fn,
+			      cb_data);
+
 
 	clear_contains_cache(&filter->internal.contains_cache);
 	clear_contains_cache(&filter->internal.no_contains_cache);
Signature · 1 line
2.49.0
#80 lore raw

PATCH v5 5/5 for-each-ref: introduce a '--start-after' option

In reply to Karthik Nayak

The git-for-each-ref(1) command is used to iterate over references present in a repository. In large repositories with millions of references, it would be optimal to paginate this output such that we can start iteration from a given reference. This would avoid having to iterate over all references from the beginning each time when paginating through results.

The previous commit added 'seek' functionality to the reference backends. Utilize this and expose a '--start-after' option in 'git-for-each-ref(1)'. When used, the reference iteration seeks to the lexicographically next reference and iterates from there onward.

This enables efficient pagination workflows, where the calling script can remember the last provided reference and use that as the starting point for the next set of references:

    git for-each-ref --count=100
    git for-each-ref --count=100 --start-after=refs/heads/branch-100
    git for-each-ref --count=100 --start-after=refs/heads/branch-200

Since the reference iterators only allow seeking to a specified marker via the ref_iterator_seek(), we introduce a helper function start_ref_iterator_after(), which seeks to next reference by simply adding (char) 1 to the marker.

We must note that pagination always continues from the provided marker, as such any concurrent reference updates lexicographically behind the marker will not be output. Document the same.

---
 Documentation/git-for-each-ref.adoc |  10 +-
 builtin/for-each-ref.c              |   8 ++
 ref-filter.c                        |  78 +++++++++++----
 ref-filter.h                        |   1 +
 t/t6302-for-each-ref-filter.sh      | 194 ++++++++++++++++++++++++++++++++++++
 5 files changed, 272 insertions(+), 19 deletions(-)
Documentation/git-for-each-ref.adoc +9 -1
index 5ef89fc0fe..ae61ba642a 100644
@@ -14,7 +14,7 @@ SYNOPSIS
 		   [--points-at=<object>]
 		   [--merged[=<object>]] [--no-merged[=<object>]]
 		   [--contains[=<object>]] [--no-contains[=<object>]]
-		   [--exclude=<pattern> ...]
+		   [--exclude=<pattern> ...] [--start-after=<marker>]
 
 DESCRIPTION
 -----------
@@ -108,6 +108,14 @@ TAB %(refname)`.
 --include-root-refs::
 	List root refs (HEAD and pseudorefs) apart from regular refs.
 
+--start-after=<marker>::
+    Allows paginating the output by skipping references up to and including the
+    specified marker. When paging, it should be noted that references may be
+    deleted, modified or added between invocations. Output will only yield those
+    references which follow the marker lexicographically. Output begins from the
+    first reference that would come after the marker alphabetically. Cannot be
+    used with general pattern matching or custom sort options.
+
 FIELD NAMES
 -----------
 
builtin/for-each-ref.c +8 -0
index 3d2207ec77..3f21598046 100644
@@ -13,6 +13,7 @@ static char const * const for_each_ref_usage[] = {
 	N_("git for-each-ref [--points-at <object>]"),
 	N_("git for-each-ref [--merged [<commit>]] [--no-merged [<commit>]]"),
 	N_("git for-each-ref [--contains [<commit>]] [--no-contains [<commit>]]"),
+	N_("git for-each-ref [--start-after <marker>]"),
 	NULL
 };
 
@@ -44,6 +45,7 @@ int cmd_for_each_ref(int argc,
 		OPT_GROUP(""),
 		OPT_INTEGER( 0 , "count", &format.array_opts.max_count, N_("show only <n> matched refs")),
 		OPT_STRING(  0 , "format", &format.format, N_("format"), N_("format to use for the output")),
+		OPT_STRING(  0 , "start-after", &filter.start_after, N_("start-start"), N_("start iteration after the provided marker")),
 		OPT__COLOR(&format.use_color, N_("respect format colors")),
 		OPT_REF_FILTER_EXCLUDE(&filter),
 		OPT_REF_SORT(&sorting_options),
@@ -79,6 +81,9 @@ int cmd_for_each_ref(int argc,
 	if (verify_ref_format(&format))
 		usage_with_options(for_each_ref_usage, opts);
 
+	if (filter.start_after && sorting_options.nr > 1)
+		die(_("cannot use --start-after with custom sort options"));
+
 	sorting = ref_sorting_options(&sorting_options);
 	ref_sorting_set_sort_flags_all(sorting, REF_SORTING_ICASE, icase);
 	filter.ignore_case = icase;
@@ -100,6 +105,9 @@ int cmd_for_each_ref(int argc,
 		filter.name_patterns = argv;
 	}
 
+	if (filter.start_after && filter.name_patterns && filter.name_patterns[0])
+		die(_("cannot use --start-after with patterns"));
+
 	if (include_root_refs)
 		flags |= FILTER_REFS_ROOT_REFS | FILTER_REFS_DETACHED_HEAD;
 
ref-filter.c +60 -18

121 lines not on this page · raw message

ref-filter.h +1 -0
index c98c4fbd4c..f22ca94b49 100644
@@ -64,6 +64,7 @@ struct ref_array {
 
 struct ref_filter {
 	const char **name_patterns;
+	const char *start_after;
 	struct strvec exclude;
 	struct oid_array points_at;
 	struct commit_list *with_commit;
t/t6302-for-each-ref-filter.sh +194 -0

198 lines not on this page · raw message

Signature · 1 line
2.49.0
#81 lore raw

Re: [PATCH v5 5/5] for-each-ref: introduce a '--start-after' option

Quoted 42 lines · Karthik Nayak

The git-for-each-ref(1) command is used to iterate over references present in a repository. In large repositories with millions of references, it would be optimal to paginate this output such that we can start iteration from a given reference. This would avoid having to iterate over all references from the beginning each time when paginating through results.

The previous commit added 'seek' functionality to the reference backends. Utilize this and expose a '--start-after' option in 'git-for-each-ref(1)'. When used, the reference iteration seeks to the lexicographically next reference and iterates from there onward.

This enables efficient pagination workflows, where the calling script can remember the last provided reference and use that as the starting point for the next set of references:

    git for-each-ref --count=100
    git for-each-ref --count=100 --start-after=refs/heads/branch-100
    git for-each-ref --count=100 --start-after=refs/heads/branch-200

Since the reference iterators only allow seeking to a specified marker via the ref_iterator_seek(), we introduce a helper function start_ref_iterator_after(), which seeks to next reference by simply adding (char) 1 to the marker.

We must note that pagination always continues from the provided marker, as such any concurrent reference updates lexicographically behind the marker will not be output. Document the same.

---
 Documentation/git-for-each-ref.adoc |  10 +-
 builtin/for-each-ref.c              |   8 ++
 ref-filter.c                        |  78 +++++++++++----
 ref-filter.h                        |   1 +
 t/t6302-for-each-ref-filter.sh      | 194 ++++++++++++++++++++++++++++++++++++
 5 files changed, 272 insertions(+), 19 deletions(-)
Documentation/git-for-each-ref.adoc +0 -0
index 5ef89fc0fe..ae61ba642a 100644

Documentation/git-for-each-ref.adoc SYNOPSIS · line 17

Quoted 5 lines
Documentation/git-for-each-ref.adoc +1 -1
@@ -14,7 +14,7 @@ SYNOPSIS
 		   [--points-at=<object>]
 		   [--merged[=<object>]] [--no-merged[=<object>]]
 		   [--contains[=<object>]] [--no-contains[=<object>]]
-		   [--exclude=<pattern> ...]
+		   [--exclude=<pattern> ...] [--start-after=<marker>]

Not a problem this patch introduces, but as I noticed it, let me leave a #leftoverbits comment here (it is OK to have a preliminary clean-up patch).

* "--exclude=<pattern>" should be enclosed inside a pair of (parentheses), just like the way how [(--sort=<key>)...] is shown.

* [--stdin | <pattern>...] should be moved to the end. There is no reason to require "--stdin" to be the end of dashed options, but the <pattern>... must be, as they are positional, not dashed.

Documentation/git-for-each-ref.adoc TAB %(refname)`. · line 117

Quoted 7 lines
Documentation/git-for-each-ref.adoc +7 -0
+--start-after=<marker>::
+    Allows paginating the output by skipping references up to and including the
+    specified marker. When paging, it should be noted that references may be
+    deleted, modified or added between invocations. Output will only yield those
+    references which follow the marker lexicographically. Output begins from the
+    first reference that would come after the marker alphabetically. Cannot be
+    used with general pattern matching or custom sort options.

It is unclear what "general" in "general pattern matching" refers to.

Cannot be used with --sort=<key> or --stdin options, or the _<pattern>_ argument(s) to limit the refs.

or something, perhaps? It is curious how --exclude=<pattern> interacts with the feature. Presumably the exclusion is done so late in the output phase that it does not have any effect? It does not have to be mentioned in this documentation if that is the case as it is a mere implementation detail.

Side note. The limitation that sorting and name_patterns cannot be used with the feature also comes from implementation (i.e. the name_patterns optimization will compete with this feature to take advantage of the "prefix" thing in an incompatible way), so while the reason does not have to be stated in the end-user facing documentation, the effect needs documenting.

Quoted 4 lines
Documentation/git-for-each-ref.adoc +0 -0
@@ -3189,6 +3221,7 @@ void filter_is_base(struct repository *r,
 
 static int do_filter_refs(struct ref_filter *filter, unsigned int type, each_ref_fn fn, void *cb_data)
 {

Documentation/git-for-each-ref.adoc void filter_is_base(struct repository *r,

Quoted 10 lines
Documentation/git-for-each-ref.adoc +9 -0
+	const char *prefix = NULL;
...
+
+	if (prefix) {
+		struct ref_iterator *iter;
+
+		iter = refs_ref_iterator_begin(get_main_ref_store(the_repository),
+					       "", NULL, 0, 0);
+
+		if (filter->start_after)

The start_after of the filter comes from "--start-after=<mark>". Can it be true with non-NULL prefix at this point? Unless you add support for the option to "git branch/tag", it would not happen, I guess.

More importantly, when you do add support to "git branch/tag", the code need to be updated to keep the original prefix while seeking the cursor to the specified <mark>, instead of clearing it.

Documentation/git-for-each-ref.adoc

Quoted 3 lines
Documentation/git-for-each-ref.adoc +3 -0
+			ret = start_ref_iterator_after(iter, filter->start_after);
+		else if (prefix)
+			ret = ref_iterator_seek(iter, prefix, 1);

We have "REF_ITERATOR_SEEK_SET_PREFIX" for that "1"?

#82 lore raw

Re: [PATCH v5 5/5] for-each-ref: introduce a '--start-after' option

Quoted 54 lines · Junio C Hamano
Quoted 42 lines · Karthik Nayak

The git-for-each-ref(1) command is used to iterate over references present in a repository. In large repositories with millions of references, it would be optimal to paginate this output such that we can start iteration from a given reference. This would avoid having to iterate over all references from the beginning each time when paginating through results.

The previous commit added 'seek' functionality to the reference backends. Utilize this and expose a '--start-after' option in 'git-for-each-ref(1)'. When used, the reference iteration seeks to the lexicographically next reference and iterates from there onward.

This enables efficient pagination workflows, where the calling script can remember the last provided reference and use that as the starting point for the next set of references:

    git for-each-ref --count=100
    git for-each-ref --count=100 --start-after=refs/heads/branch-100
    git for-each-ref --count=100 --start-after=refs/heads/branch-200

Since the reference iterators only allow seeking to a specified marker via the ref_iterator_seek(), we introduce a helper function start_ref_iterator_after(), which seeks to next reference by simply adding (char) 1 to the marker.

We must note that pagination always continues from the provided marker, as such any concurrent reference updates lexicographically behind the marker will not be output. Document the same.

---
 Documentation/git-for-each-ref.adoc |  10 +-
 builtin/for-each-ref.c              |   8 ++
 ref-filter.c                        |  78 +++++++++++----
 ref-filter.h                        |   1 +
 t/t6302-for-each-ref-filter.sh      | 194 ++++++++++++++++++++++++++++++++++++
 5 files changed, 272 insertions(+), 19 deletions(-)
Documentation/git-for-each-ref.adoc +0 -0
index 5ef89fc0fe..ae61ba642a 100644

Documentation/git-for-each-ref.adoc SYNOPSIS · line 17

Quoted 5 lines
Documentation/git-for-each-ref.adoc +1 -1
@@ -14,7 +14,7 @@ SYNOPSIS
 		   [--points-at=<object>]
 		   [--merged[=<object>]] [--no-merged[=<object>]]
 		   [--contains[=<object>]] [--no-contains[=<object>]]
-		   [--exclude=<pattern> ...]
+		   [--exclude=<pattern> ...] [--start-after=<marker>]

Not a problem this patch introduces, but as I noticed it, let me leave a #leftoverbits comment here (it is OK to have a preliminary clean-up patch).

Quoted 8 lines

* "--exclude=<pattern>" should be enclosed inside a pair of (parentheses), just like the way how [(--sort=<key>)...] is shown.

* [--stdin | <pattern>...] should be moved to the end. There is no reason to require "--stdin" to be the end of dashed options, but the <pattern>... must be, as they are positional, not dashed.

I'm sending in a series of these small fixes, so I'll add this in.

Quoted 8 lines

Documentation/git-for-each-ref.adoc TAB %(refname)`. · line 117

Quoted 7 lines
Documentation/git-for-each-ref.adoc +7 -0
+--start-after=<marker>::
+    Allows paginating the output by skipping references up to and including the
+    specified marker. When paging, it should be noted that references may be
+    deleted, modified or added between invocations. Output will only yield those
+    references which follow the marker lexicographically. Output begins from the
+    first reference that would come after the marker alphabetically. Cannot be
+    used with general pattern matching or custom sort options.
Quoted 6 lines

It is unclear what "general" in "general pattern matching" refers to.

Cannot be used with --sort=<key> or --stdin options, or the _<pattern>_ argument(s) to limit the refs.

This does read much better. I'll also add this.

Quoted 5 lines

or something, perhaps? It is curious how --exclude=<pattern> interacts with the feature. Presumably the exclusion is done so late in the output phase that it does not have any effect? It does not have to be mentioned in this documentation if that is the case as it is a mere implementation detail.

That is correct indeed, while this doesn't have to be documented, I think we can merit from a test. So I'll add that in.

Quoted 29 lines

Side note. The limitation that sorting and name_patterns cannot be used with the feature also comes from implementation (i.e. the name_patterns optimization will compete with this feature to take advantage of the "prefix" thing in an incompatible way), so while the reason does not have to be stated in the end-user facing documentation, the effect needs documenting.

Quoted 4 lines
Documentation/git-for-each-ref.adoc +0 -0
@@ -3189,6 +3221,7 @@ void filter_is_base(struct repository *r,
 
 static int do_filter_refs(struct ref_filter *filter, unsigned int type, each_ref_fn fn, void *cb_data)
 {

Documentation/git-for-each-ref.adoc void filter_is_base(struct repository *r,

Quoted 10 lines
Documentation/git-for-each-ref.adoc +9 -0
+	const char *prefix = NULL;
...
+
+	if (prefix) {
+		struct ref_iterator *iter;
+
+		iter = refs_ref_iterator_begin(get_main_ref_store(the_repository),
+					       "", NULL, 0, 0);
+
+		if (filter->start_after)

The start_after of the filter comes from "--start-after=<mark>". Can it be true with non-NULL prefix at this point? Unless you add support for the option to "git branch/tag", it would not happen, I guess.

Quoted 4 lines

More importantly, when you do add support to "git branch/tag", the code need to be updated to keep the original prefix while seeking the cursor to the specified <mark>, instead of clearing it.

Exactly, if we do add '<pattern>' and '--start-after' compatibility, we'll have to make that change.

Quoted 5 lines

Documentation/git-for-each-ref.adoc

Quoted 3 lines
Documentation/git-for-each-ref.adoc +3 -0
+			ret = start_ref_iterator_after(iter, filter->start_after);
+		else if (prefix)
+			ret = ref_iterator_seek(iter, prefix, 1);

We have "REF_ITERATOR_SEEK_SET_PREFIX" for that "1"?

Yup, also the 'if (prefix)' can be dropped too.

#83 lore raw

Re: [PATCH v5 0/5] for-each-ref: introduce seeking functionality via '--start-after'

In reply to Karthik Nayak
Quoted 9 lines · Karthik Nayak

Changes in v5:
- Changes to the comments to refer to the flag
'REF_ITERATOR_SEEK_SET_PREFIX' instead of a variable used in older
versions. Also other small grammar fixes.
- Added a commit to remove an unnecessary else clause.
- Move seeking functionality within for_each_fullref_in_pattern to its
own function.
- Fix incorrect naming in the tests.
- Link to v4: https://lore.kernel.org/r/20250711-306-git-for-each-ref-pagination-v4-0-ed3303ad5b89@gmail.com

The two refactoring differences relative to the previous round do make the result more plesant to eyes. Looking great.

Will replace. Thanks.

#84 lore raw

Re: [PATCH v5 0/5] for-each-ref: introduce seeking functionality via '--start-after'

There's something in this series that's triggering an msan warning in t/t6302-for-each-ref-filter:

==2147==WARNING: MemorySanitizer: use-of-uninitialized-value

    #0 0x562a64e923cc in cache_ref_iterator_advance refs/ref-cache.c:409:27
    #1 0x562a64e88dc0 in ref_iterator_advance refs/iterator.c:15:9
    #2 0x562a64e88dc0 in merge_ref_iterator_advance refs/iterator.c:164:13
    #3 0x562a64e8850b in ref_iterator_advance refs/iterator.c:15:9
    #4 0x562a64e85f5a in files_ref_iterator_advance refs/files-backend.c:902:15
    #5 0x562a64e88bb4 in ref_iterator_advance refs/iterator.c:15:9
    #6 0x562a64e88bb4 in do_for_each_ref_iterator refs/iterator.c:478:15
    #7 0x562a64e64593 in for_each_fullref_with_seek ref-filter.c:2718:9
    #8 0x562a64e5cfe8 in for_each_fullref_in_pattern ref-filter.c
    #9 0x562a64e5cfe8 in do_filter_refs ref-filter.c:3263:9
    #10 0x562a64e5d7fc in filter_and_format_refs ref-filter.c:3364:3
    #11 0x562a64af0235 in cmd_for_each_ref builtin/for-each-ref.c:115:2
    #12 0x562a64a3ebdc in run_builtin git.c:480:11
    #13 0x562a64a3d342 in handle_builtin git.c:746:9
    #14 0x562a64a3be33 in run_argv git.c:813:4
    #15 0x562a64a3be33 in cmd_main git.c:953:19
    #16 0x562a64c2f12f in main common-main.c:9:11

SUMMARY: MemorySanitizer: use-of-uninitialized-value refs/ref-cache.c:409:27 in cache_ref_iterator_advance

Unfortunately I can't provide great instructions for reproducing this locally, because it relies on our internal build stack (which uses blaze). Getting MemorySanitizer running can be quite annoying, though you might not have any issues if this test doesn't invoke any third party libraries (like zlib).

I need to sign off for the night soon, but if this isn't sufficient enough information to identify what's happening here, I can try to dig deeper tomorrow. This run was executed on an import of upstream commit 4ea3c74afd42a503b3e0d60e1fec33bc0431e7bc (Junio's merge of this series)

#85 lore raw

Re: [PATCH v5 0/5] for-each-ref: introduce seeking functionality via '--start-after'

Quoted 11 lines · Kyle Lippincott

Unfortunately I can't provide great instructions for reproducing this locally, because it relies on our internal build stack (which uses blaze). Getting MemorySanitizer running can be quite annoying, though you might not have any issues if this test doesn't invoke any third party libraries (like zlib).

I need to sign off for the night soon, but if this isn't sufficient enough information to identify what's happening here, I can try to dig deeper tomorrow. This run was executed on an import of upstream commit 4ea3c74afd42a503b3e0d60e1fec33bc0431e7bc (Junio's merge of this series)

valgrind can often find the same issues as MSan without as much headache to get it running (the downside is that it is _way_ slower). And indeed:

  git checkout 4ea3c74afd42a503b3e0d60e1fec33bc0431e7bc &&
  make &&
  (cd t && ./t6302-for-each-ref-filter.sh --valgrind-only=48)

yields:

==2177572== Conditional jump or move depends on uninitialised value(s)

  ==2177572==    at 0x3BC380: cache_ref_iterator_advance (ref-cache.c:409)
  ==2177572==    by 0x3B69D7: ref_iterator_advance (iterator.c:15)
  ==2177572==    by 0x3B6CC3: merge_ref_iterator_advance (iterator.c:179)
  ==2177572==    by 0x3B69D7: ref_iterator_advance (iterator.c:15)
  ==2177572==    by 0x3A9770: files_ref_iterator_advance (files-backend.c:902)
  ==2177572==    by 0x3B69D7: ref_iterator_advance (iterator.c:15)
  ==2177572==    by 0x3B7457: do_for_each_ref_iterator (iterator.c:478)
  ==2177572==    by 0x399B43: for_each_fullref_with_seek (ref-filter.c:2718)
  ==2177572==    by 0x399C09: for_each_fullref_in_pattern (ref-filter.c:2756)
  ==2177572==    by 0x39B031: do_filter_refs (ref-filter.c:3263)
  ==2177572==    by 0x39B2B7: filter_and_format_refs (ref-filter.c:3364)
  ==2177572==    by 0x18C1D2: cmd_for_each_ref (for-each-ref.c:115)
  ==2177572==  Uninitialised value was created by a heap allocation
  ==2177572==    at 0x484BDD0: realloc (vg_replace_malloc.c:1801)
  ==2177572==    by 0x44E941: xrealloc (wrapper.c:140)
  ==2177572==    by 0x3BCAD9: cache_ref_iterator_begin (ref-cache.c:580)
  ==2177572==    by 0x3A988A: files_ref_iterator_begin (files-backend.c:995)
  ==2177572==    by 0x3A295E: refs_ref_iterator_begin (refs.c:1776)
  ==2177572==    by 0x399AF6: for_each_fullref_with_seek (ref-filter.c:2710)
  ==2177572==    by 0x399C09: for_each_fullref_in_pattern (ref-filter.c:2756)
  ==2177572==    by 0x39B031: do_filter_refs (ref-filter.c:3263)
  ==2177572==    by 0x39B2B7: filter_and_format_refs (ref-filter.c:3364)
  ==2177572==    by 0x18C1D2: cmd_for_each_ref (for-each-ref.c:115)
  ==2177572==    by 0x128C90: run_builtin (git.c:480)
  ==2177572==    by 0x1290EB: handle_builtin (git.c:746)

Bisecting doesn't tell us much, though (the first commit that introduces the test shows the problem). I didn't dig further than that.

-Peff

#86 lore raw

Re: [PATCH v5 0/5] for-each-ref: introduce seeking functionality via '--start-after'

Quoted 51 lines · Jeff King
Quoted 11 lines · Kyle Lippincott

Unfortunately I can't provide great instructions for reproducing this locally, because it relies on our internal build stack (which uses blaze). Getting MemorySanitizer running can be quite annoying, though you might not have any issues if this test doesn't invoke any third party libraries (like zlib).

I need to sign off for the night soon, but if this isn't sufficient enough information to identify what's happening here, I can try to dig deeper tomorrow. This run was executed on an import of upstream commit 4ea3c74afd42a503b3e0d60e1fec33bc0431e7bc (Junio's merge of this series)

valgrind can often find the same issues as MSan without as much headache to get it running (the downside is that it is _way_ slower). And indeed:

  git checkout 4ea3c74afd42a503b3e0d60e1fec33bc0431e7bc &&
  make &&
  (cd t && ./t6302-for-each-ref-filter.sh --valgrind-only=48)

yields:

==2177572== Conditional jump or move depends on uninitialised value(s)

  ==2177572==    at 0x3BC380: cache_ref_iterator_advance (ref-cache.c:409)
  ==2177572==    by 0x3B69D7: ref_iterator_advance (iterator.c:15)
  ==2177572==    by 0x3B6CC3: merge_ref_iterator_advance (iterator.c:179)
  ==2177572==    by 0x3B69D7: ref_iterator_advance (iterator.c:15)
  ==2177572==    by 0x3A9770: files_ref_iterator_advance (files-backend.c:902)
  ==2177572==    by 0x3B69D7: ref_iterator_advance (iterator.c:15)
  ==2177572==    by 0x3B7457: do_for_each_ref_iterator (iterator.c:478)
  ==2177572==    by 0x399B43: for_each_fullref_with_seek (ref-filter.c:2718)
  ==2177572==    by 0x399C09: for_each_fullref_in_pattern (ref-filter.c:2756)
  ==2177572==    by 0x39B031: do_filter_refs (ref-filter.c:3263)
  ==2177572==    by 0x39B2B7: filter_and_format_refs (ref-filter.c:3364)
  ==2177572==    by 0x18C1D2: cmd_for_each_ref (for-each-ref.c:115)
  ==2177572==  Uninitialised value was created by a heap allocation
  ==2177572==    at 0x484BDD0: realloc (vg_replace_malloc.c:1801)
  ==2177572==    by 0x44E941: xrealloc (wrapper.c:140)
  ==2177572==    by 0x3BCAD9: cache_ref_iterator_begin (ref-cache.c:580)
  ==2177572==    by 0x3A988A: files_ref_iterator_begin (files-backend.c:995)
  ==2177572==    by 0x3A295E: refs_ref_iterator_begin (refs.c:1776)
  ==2177572==    by 0x399AF6: for_each_fullref_with_seek (ref-filter.c:2710)
  ==2177572==    by 0x399C09: for_each_fullref_in_pattern (ref-filter.c:2756)
  ==2177572==    by 0x39B031: do_filter_refs (ref-filter.c:3263)
  ==2177572==    by 0x39B2B7: filter_and_format_refs (ref-filter.c:3364)
  ==2177572==    by 0x18C1D2: cmd_for_each_ref (for-each-ref.c:115)
  ==2177572==    by 0x128C90: run_builtin (git.c:480)
  ==2177572==    by 0x1290EB: handle_builtin (git.c:746)
Quoted 4 lines

Bisecting doesn't tell us much, though (the first commit that introduces the test shows the problem). I didn't dig further than that.

-Peff

Thanks for that, that helped me a bit too as it provides more information than I was getting out of MemorySanitizer (I suspect MemorySanitizer was producing the information it just wasn't going to stderr or something, or maybe I was missing a flag to get it to report more). I'm not sure what the right fix would be; my guess is that the fix would be to modify the places where we set levels_nr and initialize the other fields in level to also set it to prefix_state (around lines 488 and 527 in ref-cache.c); and indeed setting the prefix_state to PREFIX_CONTAINS_DIR (the 0 value of the enum) makes the test pass even under valgrind. Unfortunately without a much more in-depth knowledge of the code and the enum values I can't definitively state that those are the correct values. I can say that setting it to PREFIX_WITHIN_DIR causes both additional valgrind failures and test failures even without valgrind, but setting it to PREFIX_EXCLUDES_DIR doesn't seem to be a problem. I also moved the if around like 409 into the following if, because that was the only time entry_prefix_state was used, I'd been thinking that maybe it needed the check for entry->flag & REF_DIR prior to referencing level->prefix_state, but that didn't resolve it on its own.

I don't mind if anyone else picks up this fix and runs with it, but I'm not comfortable sending this patch myself because I don't have enough knowledge of this are of the code to know if it's right, just that it fixes the issue we encountered, and I'm extremely overloaded right now and can't get that knowledge nor see the patch through to the end.

refs/ref-cache.c +0 -0
index 1d95b56d40..24feb33fcb 100644
@@ -391,7 +391,6 @@ static int cache_ref_iterator_advance(struct

ref_iterator *ref_iterator)

                        &iter->levels[iter->levels_nr - 1];
                struct ref_dir *dir = level->dir;
                struct ref_entry *entry;

- enum prefix_state entry_prefix_state;

                if (level->index == -1)
                        sort_ref_dir(dir);

@@ -406,16 +405,17 @@ static int cache_ref_iterator_advance(struct ref_iterator *ref_iterator)

                entry = dir->entries[level->index];

- if (level->prefix_state == PREFIX_WITHIN_DIR) {
- entry_prefix_state =
overlaps_prefix(entry->name, iter->prefix);
- if (entry_prefix_state == PREFIX_EXCLUDES_DIR ||
- (entry_prefix_state == PREFIX_WITHIN_DIR
&& !(entry->flag & REF_DIR)))
- continue;
- } else {
- entry_prefix_state = level->prefix_state;
- }
-

                if (entry->flag & REF_DIR) {

+ enum prefix_state entry_prefix_state;
+ if (level->prefix_state == PREFIX_WITHIN_DIR) {
+ entry_prefix_state =
overlaps_prefix(entry->name, iter->prefix);
+ if (entry_prefix_state == PREFIX_EXCLUDES_DIR ||
+ (entry_prefix_state ==
PREFIX_WITHIN_DIR && !(entry->flag & REF_DIR)))
+ continue;
+ } else {
+ entry_prefix_state = level->prefix_state;
+ }
+

                        /* push down a level */
                        ALLOC_GROW(iter->levels, iter->levels_nr + 1,
                                   iter->levels_alloc);
@@ -489,6 +489,7 @@ static int cache_ref_iterator_seek(struct
ref_iterator *ref_iterator,
                level = &iter->levels[0];
                level->index = -1;
                level->dir = dir;

+ level->prefix_state = PREFIX_EXCLUDES_DIR; //
FIXME: PROBABLY NOT CORRECT

                /* Unset any previously set prefix */
                FREE_AND_NULL(iter->prefix);
@@ -527,6 +528,7 @@ static int cache_ref_iterator_seek(struct
ref_iterator *ref_iterator,
                                level = &iter->levels[iter->levels_nr++];
                                level->dir = dir;
                                level->index = -1;
+                               level->prefix_state =
PREFIX_EXCLUDES_DIR;      // FIXME: PROBABLY NOT CORRECT
                        } else {
                                /* reduce the index so the leaf node
is iterated over */
                                if (cmp <= 0 && !slash)
#87 lore raw

Re: [PATCH v5 0/5] for-each-ref: introduce seeking functionality via '--start-after'

Quoted 55 lines · Kyle Lippincott
Quoted 54 lines · Jeff King
Quoted 11 lines · Kyle Lippincott

Unfortunately I can't provide great instructions for reproducing this locally, because it relies on our internal build stack (which uses blaze). Getting MemorySanitizer running can be quite annoying, though you might not have any issues if this test doesn't invoke any third party libraries (like zlib).

I need to sign off for the night soon, but if this isn't sufficient enough information to identify what's happening here, I can try to dig deeper tomorrow. This run was executed on an import of upstream commit 4ea3c74afd42a503b3e0d60e1fec33bc0431e7bc (Junio's merge of this series)

valgrind can often find the same issues as MSan without as much headache to get it running (the downside is that it is _way_ slower). And indeed:

  git checkout 4ea3c74afd42a503b3e0d60e1fec33bc0431e7bc &&
  make &&
  (cd t && ./t6302-for-each-ref-filter.sh --valgrind-only=48)

yields:

==2177572== Conditional jump or move depends on uninitialised value(s)

  ==2177572==    at 0x3BC380: cache_ref_iterator_advance (ref-cache.c:409)
  ==2177572==    by 0x3B69D7: ref_iterator_advance (iterator.c:15)
  ==2177572==    by 0x3B6CC3: merge_ref_iterator_advance (iterator.c:179)
  ==2177572==    by 0x3B69D7: ref_iterator_advance (iterator.c:15)
  ==2177572==    by 0x3A9770: files_ref_iterator_advance (files-backend.c:902)
  ==2177572==    by 0x3B69D7: ref_iterator_advance (iterator.c:15)
  ==2177572==    by 0x3B7457: do_for_each_ref_iterator (iterator.c:478)
  ==2177572==    by 0x399B43: for_each_fullref_with_seek (ref-filter.c:2718)
  ==2177572==    by 0x399C09: for_each_fullref_in_pattern (ref-filter.c:2756)
  ==2177572==    by 0x39B031: do_filter_refs (ref-filter.c:3263)
  ==2177572==    by 0x39B2B7: filter_and_format_refs (ref-filter.c:3364)
  ==2177572==    by 0x18C1D2: cmd_for_each_ref (for-each-ref.c:115)
  ==2177572==  Uninitialised value was created by a heap allocation
  ==2177572==    at 0x484BDD0: realloc (vg_replace_malloc.c:1801)
  ==2177572==    by 0x44E941: xrealloc (wrapper.c:140)
  ==2177572==    by 0x3BCAD9: cache_ref_iterator_begin (ref-cache.c:580)
  ==2177572==    by 0x3A988A: files_ref_iterator_begin (files-backend.c:995)
  ==2177572==    by 0x3A295E: refs_ref_iterator_begin (refs.c:1776)
  ==2177572==    by 0x399AF6: for_each_fullref_with_seek (ref-filter.c:2710)
  ==2177572==    by 0x399C09: for_each_fullref_in_pattern (ref-filter.c:2756)
  ==2177572==    by 0x39B031: do_filter_refs (ref-filter.c:3263)
  ==2177572==    by 0x39B2B7: filter_and_format_refs (ref-filter.c:3364)
  ==2177572==    by 0x18C1D2: cmd_for_each_ref (for-each-ref.c:115)
  ==2177572==    by 0x128C90: run_builtin (git.c:480)
  ==2177572==    by 0x1290EB: handle_builtin (git.c:746)

Bisecting doesn't tell us much, though (the first commit that introduces the test shows the problem). I didn't dig further than that.

Quoted 8 lines
Quoted 1 line

-Peff

Thanks for that, that helped me a bit too as it provides more information than I was getting out of MemorySanitizer (I suspect MemorySanitizer was producing the information it just wasn't going to stderr or something, or maybe I was missing a flag to get it to report more).

Thanks both for raising the issue. Thanks Jeff for also the valgrind instructions.

On a sidenote, was discussing this at work and Patrick also mentioned that we could try clang's MemorySanitizer. This seems to also be raising issues on master, so it was hard to find the exact output that valgrind was providing.

$ git checkout master
$ CC=clang meson setup --reconfigure memory_build . -Db_sanitize=memory
$ cd memory_build
$ meson test -i --test-args="-ix" t6302-for-each-ref-filter
...
==3275333==WARNING: MemorySanitizer: use-of-uninitialized-value
    #0 0x557bd886f4bb in git_mkstemps_mode ../wrapper.c:487:27
    #1 0x557bd886fb55 in git_mkstemp_mode ../wrapper.c:509:9
    #2 0x557bd8100d1a in create_tmpfile ../object-file.c:736:7
    #3 0x557bd80f1630 in start_loose_object_common ../object-file.c:781:7
    #4 0x557bd80f5203 in write_loose_object ../object-file.c:881:7
    #5 0x557bd80f4875 in write_object_file_flags ../object-file.c:1086:6
    #6 0x557bd80f9f65 in write_object_file ../object-file.h:181:9
    #7 0x557bd8101eb8 in index_mem ../object-file.c:1177:9
    #8 0x557bd80f8bd5 in index_core ../object-file.c:1247:10
    #9 0x557bd80f731d in index_fd ../object-file.c:1274:9
    #10 0x557bd80f95e4 in index_path ../object-file.c:1295:7
    #11 0x557bd831132d in add_to_index ../read-cache.c:771:7
    #12 0x557bd8313cb1 in add_file_to_index ../read-cache.c:804:9
    #13 0x557bd73f892c in add_files ../builtin/add.c:355:7
    #14 0x557bd73f4752 in cmd_add ../builtin/add.c:578:18
    #15 0x557bd7a38b6f in run_builtin ../git.c:480:11
    #16 0x557bd7a31d54 in handle_builtin ../git.c:746:9
    #17 0x557bd7a36644 in run_argv ../git.c:813:4
    #18 0x557bd7a30e09 in cmd_main ../git.c:953:19
    #19 0x557bd7a3ca01 in main ../common-main.c:9:11
    #20 0x7f7e3f02a4d7 in __libc_start_call_main
(/nix/store/g2jzxk3s7cnkhh8yq55l4fbvf639zy37-glibc-2.40-66/lib/libc.so.6+0x2a4d7)
(BuildId: f117ee0f586dfa828cbdd08e37393c8f04f6480a)
    #21 0x7f7e3f02a59a in __libc_start_main@GLIBC_2.2.5
(/nix/store/g2jzxk3s7cnkhh8yq55l4fbvf639zy37-glibc-2.40-66/lib/libc.so.6+0x2a59a)
(BuildId: f117ee0f586dfa828cbdd08e37393c8f04f6480a)
    #22 0x557bd7352b34 in _start (git+0x5db34)

Possibly something we need to look into cleaning up.

Quoted 16 lines

I'm not sure what the right fix would be; my guess is that the fix would be to modify the places where we set levels_nr and initialize the other fields in level to also set it to prefix_state (around lines 488 and 527 in ref-cache.c); and indeed setting the prefix_state to PREFIX_CONTAINS_DIR (the 0 value of the enum) makes the test pass even under valgrind. Unfortunately without a much more in-depth knowledge of the code and the enum values I can't definitively state that those are the correct values. I can say that setting it to PREFIX_WITHIN_DIR causes both additional valgrind failures and test failures even without valgrind, but setting it to PREFIX_EXCLUDES_DIR doesn't seem to be a problem. I also moved the if around like 409 into the following if, because that was the only time entry_prefix_state was used, I'd been thinking that maybe it needed the check for entry->flag & REF_DIR prior to referencing level->prefix_state, but that didn't resolve it on its own.

Quoted 7 lines

I don't mind if anyone else picks up this fix and runs with it, but I'm not comfortable sending this patch myself because I don't have enough knowledge of this are of the code to know if it's right, just that it fixes the issue we encountered, and I'm extremely overloaded right now and can't get that knowledge nor see the patch through to the end.

Thanks for taking a stab at this, your inference is correct. Let me clairfy some parts of it.

So the 'ref-cache' iteration logic is used to provide iteration over loose refs (which consists of directories and entries). Anytime we come across a directory, we add it to the level variable, which acts as stack, when all entries under the current level are yielded, we pop the stack to obtain the next level to iterate. This ensures we iterate over all directories recursively.

Before this series, the seek function was used to set the prefix for iteration, which meant we need to find the directory for matching the prefix and only iterate over that level and its subdirs. If the prefix provided was a directory like 'refs/heads/' then all refs under that would be yielded (PREFIX_CONTAINS_DIR). If the prefix was 'refs/heads/foo', then the level would be set to 'ref/heads/' with the PREFIX_WITHIN_DIR flag set since only some refs within the dir would match the prefix. Entries which didn't overlap the prefix are denoted by PREFIX_EXCLUDES_DIR.

This series allows the seek function to set the cursor without setting the prefix, which is a requirement for pagination. So there is no need to set 'prefix_state' for this functionality. Which is why I didn't set it, since the default value of '0' (PREFIX_CONTAINS_DIR) would be the correct setting for all dirs. This causes the issue.

So the only fix required would be

refs/ref-cache.c +0 -0
index 1d95b56d40..ceef3a2008 100644
@@ -527,6 +527,7 @@ static int cache_ref_iterator_seek(struct

ref_iterator *ref_iterator,
level = &iter->levels[iter->levels_nr++]; level->dir = dir;
level->index = -1;
+ level->prefix_state = PREFIX_CONTAINS_DIR;
} else {
/* reduce the index so the leaf node is iterated over */ if (cmp <= 0 && !slash)

The other location (Line 488), is not needed because that is the root directory and the 'prefix_state' for it is set in 'cache_ref_iterator_set_prefix()' when the iterator begins.

Quoted 67 lines
refs/ref-cache.c +0 -1
index 1d95b56d40..24feb33fcb 100644
@@ -391,7 +391,6 @@ static int cache_ref_iterator_advance(struct
 ref_iterator *ref_iterator)
                        &iter->levels[iter->levels_nr - 1];
                struct ref_dir *dir = level->dir;
                struct ref_entry *entry;
-               enum prefix_state entry_prefix_state;
 
                if (level->index == -1)
                        sort_ref_dir(dir);
refs/ref-cache.c +3 -9
@@ -406,16 +405,17 @@ static int cache_ref_iterator_advance(struct
 ref_iterator *ref_iterator)
 
                entry = dir->entries[level->index];
 
-               if (level->prefix_state == PREFIX_WITHIN_DIR) {
-                       entry_prefix_state =
 overlaps_prefix(entry->name, iter->prefix);
-                       if (entry_prefix_state == PREFIX_EXCLUDES_DIR ||
-                           (entry_prefix_state == PREFIX_WITHIN_DIR
 && !(entry->flag & REF_DIR)))
-                               continue;
-               } else {
-                       entry_prefix_state = level->prefix_state;
-               }
-
                if (entry->flag & REF_DIR) {
+                       enum prefix_state entry_prefix_state;
+                       if (level->prefix_state == PREFIX_WITHIN_DIR) {
+                               entry_prefix_state =

overlaps_prefix(entry->name, iter->prefix);

refs/ref-cache.c +2 -0
+                               if (entry_prefix_state == PREFIX_EXCLUDES_DIR ||
+                                   (entry_prefix_state ==

PREFIX_WITHIN_DIR && !(entry->flag & REF_DIR)))

refs/ref-cache.c +6 -0
+                                       continue;
+                       } else {
+                               entry_prefix_state = level->prefix_state;
+                       }
+
                        /* push down a level */
                        ALLOC_GROW(iter->levels, iter->levels_nr + 1,
                                   iter->levels_alloc);
@@ -489,6 +489,7 @@ static int cache_ref_iterator_seek(struct
 ref_iterator *ref_iterator,
                level = &iter->levels[0];
                level->index = -1;
                level->dir = dir;
+               level->prefix_state = PREFIX_EXCLUDES_DIR;      //

FIXME: PROBABLY NOT CORRECT

                /* Unset any previously set prefix */
                FREE_AND_NULL(iter->prefix);
refs/ref-cache.c +1 -0
@@ -527,6 +528,7 @@ static int cache_ref_iterator_seek(struct
 ref_iterator *ref_iterator,
                                level = &iter->levels[iter->levels_nr++];
                                level->dir = dir;
                                level->index = -1;
+                               level->prefix_state =

PREFIX_EXCLUDES_DIR; // FIXME: PROBABLY NOT CORRECT

                        } else {
                                /* reduce the index so the leaf node
is iterated over */
                                if (cmp <= 0 && !slash)
#88 lore raw

Re: [PATCH v5 0/5] for-each-ref: introduce seeking functionality via '--start-after'

Quoted 109 lines · Karthik Nayak
Quoted 55 lines · Kyle Lippincott
Quoted 54 lines · Jeff King
Quoted 11 lines · Kyle Lippincott

Unfortunately I can't provide great instructions for reproducing this locally, because it relies on our internal build stack (which uses blaze). Getting MemorySanitizer running can be quite annoying, though you might not have any issues if this test doesn't invoke any third party libraries (like zlib).

I need to sign off for the night soon, but if this isn't sufficient enough information to identify what's happening here, I can try to dig deeper tomorrow. This run was executed on an import of upstream commit 4ea3c74afd42a503b3e0d60e1fec33bc0431e7bc (Junio's merge of this series)

valgrind can often find the same issues as MSan without as much headache to get it running (the downside is that it is _way_ slower). And indeed:

  git checkout 4ea3c74afd42a503b3e0d60e1fec33bc0431e7bc &&
  make &&
  (cd t && ./t6302-for-each-ref-filter.sh --valgrind-only=48)

yields:

==2177572== Conditional jump or move depends on uninitialised value(s)

  ==2177572==    at 0x3BC380: cache_ref_iterator_advance (ref-cache.c:409)
  ==2177572==    by 0x3B69D7: ref_iterator_advance (iterator.c:15)
  ==2177572==    by 0x3B6CC3: merge_ref_iterator_advance (iterator.c:179)
  ==2177572==    by 0x3B69D7: ref_iterator_advance (iterator.c:15)
  ==2177572==    by 0x3A9770: files_ref_iterator_advance (files-backend.c:902)
  ==2177572==    by 0x3B69D7: ref_iterator_advance (iterator.c:15)
  ==2177572==    by 0x3B7457: do_for_each_ref_iterator (iterator.c:478)
  ==2177572==    by 0x399B43: for_each_fullref_with_seek (ref-filter.c:2718)
  ==2177572==    by 0x399C09: for_each_fullref_in_pattern (ref-filter.c:2756)
  ==2177572==    by 0x39B031: do_filter_refs (ref-filter.c:3263)
  ==2177572==    by 0x39B2B7: filter_and_format_refs (ref-filter.c:3364)
  ==2177572==    by 0x18C1D2: cmd_for_each_ref (for-each-ref.c:115)
  ==2177572==  Uninitialised value was created by a heap allocation
  ==2177572==    at 0x484BDD0: realloc (vg_replace_malloc.c:1801)
  ==2177572==    by 0x44E941: xrealloc (wrapper.c:140)
  ==2177572==    by 0x3BCAD9: cache_ref_iterator_begin (ref-cache.c:580)
  ==2177572==    by 0x3A988A: files_ref_iterator_begin (files-backend.c:995)
  ==2177572==    by 0x3A295E: refs_ref_iterator_begin (refs.c:1776)
  ==2177572==    by 0x399AF6: for_each_fullref_with_seek (ref-filter.c:2710)
  ==2177572==    by 0x399C09: for_each_fullref_in_pattern (ref-filter.c:2756)
  ==2177572==    by 0x39B031: do_filter_refs (ref-filter.c:3263)
  ==2177572==    by 0x39B2B7: filter_and_format_refs (ref-filter.c:3364)
  ==2177572==    by 0x18C1D2: cmd_for_each_ref (for-each-ref.c:115)
  ==2177572==    by 0x128C90: run_builtin (git.c:480)
  ==2177572==    by 0x1290EB: handle_builtin (git.c:746)

Bisecting doesn't tell us much, though (the first commit that introduces the test shows the problem). I didn't dig further than that.

Quoted 8 lines
Quoted 1 line

-Peff

Thanks for that, that helped me a bit too as it provides more information than I was getting out of MemorySanitizer (I suspect MemorySanitizer was producing the information it just wasn't going to stderr or something, or maybe I was missing a flag to get it to report more).

Thanks both for raising the issue. Thanks Jeff for also the valgrind instructions.

On a sidenote, was discussing this at work and Patrick also mentioned that we could try clang's MemorySanitizer. This seems to also be raising issues on master, so it was hard to find the exact output that valgrind was providing.

$ git checkout master
$ CC=clang meson setup --reconfigure memory_build . -Db_sanitize=memory
$ cd memory_build
$ meson test -i --test-args="-ix" t6302-for-each-ref-filter
...
==3275333==WARNING: MemorySanitizer: use-of-uninitialized-value
    #0 0x557bd886f4bb in git_mkstemps_mode ../wrapper.c:487:27
    #1 0x557bd886fb55 in git_mkstemp_mode ../wrapper.c:509:9
    #2 0x557bd8100d1a in create_tmpfile ../object-file.c:736:7
    #3 0x557bd80f1630 in start_loose_object_common ../object-file.c:781:7
    #4 0x557bd80f5203 in write_loose_object ../object-file.c:881:7
    #5 0x557bd80f4875 in write_object_file_flags ../object-file.c:1086:6
    #6 0x557bd80f9f65 in write_object_file ../object-file.h:181:9
    #7 0x557bd8101eb8 in index_mem ../object-file.c:1177:9
    #8 0x557bd80f8bd5 in index_core ../object-file.c:1247:10
    #9 0x557bd80f731d in index_fd ../object-file.c:1274:9
    #10 0x557bd80f95e4 in index_path ../object-file.c:1295:7
    #11 0x557bd831132d in add_to_index ../read-cache.c:771:7
    #12 0x557bd8313cb1 in add_file_to_index ../read-cache.c:804:9
    #13 0x557bd73f892c in add_files ../builtin/add.c:355:7
    #14 0x557bd73f4752 in cmd_add ../builtin/add.c:578:18
    #15 0x557bd7a38b6f in run_builtin ../git.c:480:11
    #16 0x557bd7a31d54 in handle_builtin ../git.c:746:9
    #17 0x557bd7a36644 in run_argv ../git.c:813:4
    #18 0x557bd7a30e09 in cmd_main ../git.c:953:19
    #19 0x557bd7a3ca01 in main ../common-main.c:9:11
    #20 0x7f7e3f02a4d7 in __libc_start_call_main
(/nix/store/g2jzxk3s7cnkhh8yq55l4fbvf639zy37-glibc-2.40-66/lib/libc.so.6+0x2a4d7)
(BuildId: f117ee0f586dfa828cbdd08e37393c8f04f6480a)
    #21 0x7f7e3f02a59a in __libc_start_main@GLIBC_2.2.5
(/nix/store/g2jzxk3s7cnkhh8yq55l4fbvf639zy37-glibc-2.40-66/lib/libc.so.6+0x2a59a)
(BuildId: f117ee0f586dfa828cbdd08e37393c8f04f6480a)
    #22 0x557bd7352b34 in _start (git+0x5db34)
Quoted 1 line

Possibly something we need to look into cleaning up.

I also saw those msan issues when trying make CFLAGS=-fsanitize=memory CC=clang, but not with Google's internal msan build. I don't know which variable in wrapper.c:487 it's complaining about - you'd think it'd be letters, but if it's v, then that potentially comes from OpenSSL or some other library, and that library would also need to be built with msan (which is why it's such a pain to get msan builds working - EVERY library needs to be built with memory sanitizer).

Quoted 139 lines
Quoted 17 lines

I'm not sure what the right fix would be; my guess is that the fix would be to modify the places where we set levels_nr and initialize the other fields in level to also set it to prefix_state (around lines 488 and 527 in ref-cache.c); and indeed setting the prefix_state to PREFIX_CONTAINS_DIR (the 0 value of the enum) makes the test pass even under valgrind. Unfortunately without a much more in-depth knowledge of the code and the enum values I can't definitively state that those are the correct values. I can say that setting it to PREFIX_WITHIN_DIR causes both additional valgrind failures and test failures even without valgrind, but setting it to PREFIX_EXCLUDES_DIR doesn't seem to be a problem. I also moved the if around like 409 into the following if, because that was the only time entry_prefix_state was used, I'd been thinking that maybe it needed the check for entry->flag & REF_DIR prior to referencing level->prefix_state, but that didn't resolve it on its own.

Quoted 7 lines

I don't mind if anyone else picks up this fix and runs with it, but I'm not comfortable sending this patch myself because I don't have enough knowledge of this are of the code to know if it's right, just that it fixes the issue we encountered, and I'm extremely overloaded right now and can't get that knowledge nor see the patch through to the end.

Thanks for taking a stab at this, your inference is correct. Let me clairfy some parts of it.

So the 'ref-cache' iteration logic is used to provide iteration over loose refs (which consists of directories and entries). Anytime we come across a directory, we add it to the level variable, which acts as stack, when all entries under the current level are yielded, we pop the stack to obtain the next level to iterate. This ensures we iterate over all directories recursively.

Before this series, the seek function was used to set the prefix for iteration, which meant we need to find the directory for matching the prefix and only iterate over that level and its subdirs. If the prefix provided was a directory like 'refs/heads/' then all refs under that would be yielded (PREFIX_CONTAINS_DIR). If the prefix was 'refs/heads/foo', then the level would be set to 'ref/heads/' with the PREFIX_WITHIN_DIR flag set since only some refs within the dir would match the prefix. Entries which didn't overlap the prefix are denoted by PREFIX_EXCLUDES_DIR.

This series allows the seek function to set the cursor without setting the prefix, which is a requirement for pagination. So there is no need to set 'prefix_state' for this functionality. Which is why I didn't set it, since the default value of '0' (PREFIX_CONTAINS_DIR) would be the correct setting for all dirs. This causes the issue.

So the only fix required would be

refs/ref-cache.c +1 -0
index 1d95b56d40..ceef3a2008 100644
@@ -527,6 +527,7 @@ static int cache_ref_iterator_seek(struct
 ref_iterator *ref_iterator,
                                level = &iter->levels[iter->levels_nr++];
                                level->dir = dir;
                                level->index = -1;
+                               level->prefix_state = PREFIX_CONTAINS_DIR;
                        } else {
                                /* reduce the index so the leaf node is iterated over */
                                if (cmp <= 0 && !slash)

The other location (Line 488), is not needed because that is the root directory and the 'prefix_state' for it is set in 'cache_ref_iterator_set_prefix()' when the iterator begins.

Quoted 67 lines
refs/ref-cache.c +0 -1
index 1d95b56d40..24feb33fcb 100644
@@ -391,7 +391,6 @@ static int cache_ref_iterator_advance(struct
 ref_iterator *ref_iterator)
                        &iter->levels[iter->levels_nr - 1];
                struct ref_dir *dir = level->dir;
                struct ref_entry *entry;
-               enum prefix_state entry_prefix_state;
 
                if (level->index == -1)
                        sort_ref_dir(dir);
refs/ref-cache.c +3 -9
@@ -406,16 +405,17 @@ static int cache_ref_iterator_advance(struct
 ref_iterator *ref_iterator)
 
                entry = dir->entries[level->index];
 
-               if (level->prefix_state == PREFIX_WITHIN_DIR) {
-                       entry_prefix_state =
 overlaps_prefix(entry->name, iter->prefix);
-                       if (entry_prefix_state == PREFIX_EXCLUDES_DIR ||
-                           (entry_prefix_state == PREFIX_WITHIN_DIR
 && !(entry->flag & REF_DIR)))
-                               continue;
-               } else {
-                       entry_prefix_state = level->prefix_state;
-               }
-
                if (entry->flag & REF_DIR) {
+                       enum prefix_state entry_prefix_state;
+                       if (level->prefix_state == PREFIX_WITHIN_DIR) {
+                               entry_prefix_state =

overlaps_prefix(entry->name, iter->prefix);

refs/ref-cache.c +2 -0
+                               if (entry_prefix_state == PREFIX_EXCLUDES_DIR ||
+                                   (entry_prefix_state ==

PREFIX_WITHIN_DIR && !(entry->flag & REF_DIR)))

refs/ref-cache.c +6 -0
+                                       continue;
+                       } else {
+                               entry_prefix_state = level->prefix_state;
+                       }
+
                        /* push down a level */
                        ALLOC_GROW(iter->levels, iter->levels_nr + 1,
                                   iter->levels_alloc);
@@ -489,6 +489,7 @@ static int cache_ref_iterator_seek(struct
 ref_iterator *ref_iterator,
                level = &iter->levels[0];
                level->index = -1;
                level->dir = dir;
+               level->prefix_state = PREFIX_EXCLUDES_DIR;      //

FIXME: PROBABLY NOT CORRECT

                /* Unset any previously set prefix */
                FREE_AND_NULL(iter->prefix);
refs/ref-cache.c +1 -0
@@ -527,6 +528,7 @@ static int cache_ref_iterator_seek(struct
 ref_iterator *ref_iterator,
                                level = &iter->levels[iter->levels_nr++];
                                level->dir = dir;
                                level->index = -1;
+                               level->prefix_state =

PREFIX_EXCLUDES_DIR; // FIXME: PROBABLY NOT CORRECT

                        } else {
                                /* reduce the index so the leaf node
is iterated over */
                                if (cmp <= 0 && !slash)
#89 lore raw

Re: [PATCH v5 0/5] for-each-ref: introduce seeking functionality via '--start-after'

Quoted 31 lines · Kyle Lippincott
Quoted 29 lines

==3275333==WARNING: MemorySanitizer: use-of-uninitialized-value

    #0 0x557bd886f4bb in git_mkstemps_mode ../wrapper.c:487:27
    #1 0x557bd886fb55 in git_mkstemp_mode ../wrapper.c:509:9
    #2 0x557bd8100d1a in create_tmpfile ../object-file.c:736:7
    #3 0x557bd80f1630 in start_loose_object_common ../object-file.c:781:7
    #4 0x557bd80f5203 in write_loose_object ../object-file.c:881:7
    #5 0x557bd80f4875 in write_object_file_flags ../object-file.c:1086:6
    #6 0x557bd80f9f65 in write_object_file ../object-file.h:181:9
    #7 0x557bd8101eb8 in index_mem ../object-file.c:1177:9
    #8 0x557bd80f8bd5 in index_core ../object-file.c:1247:10
    #9 0x557bd80f731d in index_fd ../object-file.c:1274:9
    #10 0x557bd80f95e4 in index_path ../object-file.c:1295:7
    #11 0x557bd831132d in add_to_index ../read-cache.c:771:7
    #12 0x557bd8313cb1 in add_file_to_index ../read-cache.c:804:9
    #13 0x557bd73f892c in add_files ../builtin/add.c:355:7
    #14 0x557bd73f4752 in cmd_add ../builtin/add.c:578:18
    #15 0x557bd7a38b6f in run_builtin ../git.c:480:11
    #16 0x557bd7a31d54 in handle_builtin ../git.c:746:9
    #17 0x557bd7a36644 in run_argv ../git.c:813:4
    #18 0x557bd7a30e09 in cmd_main ../git.c:953:19
    #19 0x557bd7a3ca01 in main ../common-main.c:9:11
    #20 0x7f7e3f02a4d7 in __libc_start_call_main
(/nix/store/g2jzxk3s7cnkhh8yq55l4fbvf639zy37-glibc-2.40-66/lib/libc.so.6+0x2a4d7)
(BuildId: f117ee0f586dfa828cbdd08e37393c8f04f6480a)
    #21 0x7f7e3f02a59a in __libc_start_main@GLIBC_2.2.5
(/nix/store/g2jzxk3s7cnkhh8yq55l4fbvf639zy37-glibc-2.40-66/lib/libc.so.6+0x2a59a)
(BuildId: f117ee0f586dfa828cbdd08e37393c8f04f6480a)
    #22 0x557bd7352b34 in _start (git+0x5db34)
Quoted 1 line

Possibly something we need to look into cleaning up.

Quoted 8 lines

I also saw those msan issues when trying make CFLAGS=-fsanitize=memory CC=clang, but not with Google's internal msan build. I don't know which variable in wrapper.c:487 it's complaining about - you'd think it'd be letters, but if it's v, then that potentially comes from OpenSSL or some other library, and that library would also need to be built with msan (which is why it's such a pain to get msan builds working - EVERY library needs to be built with memory sanitizer).

Yeah, presumably it is "v" from csprng_bytes(). If there are only a few such spots, we can manually "unpoison" memory coming from libraries. On my system, I didn't hit the case shown above but do have trouble with bytes coming back from zlib.

Applying this ancient patch:

https://lore.kernel.org/git/[email protected]/

and building with "make SANITIZE=memory CC=clang" let me run t6302 to completion, modulo the bug that started this thread (and which I confirmed goes away both with MSan and valgrind with the fix Karthik posted).

Probably:

wrapper.c +2 -0
index 2f00d2ac87..6a4c1c1c29 100644
@@ -482,6 +482,8 @@ int git_mkstemps_mode(char *pattern, int suffix_len, int mode)
 		if (csprng_bytes(&v, sizeof(v), 0) < 0)
 			return error_errno("unable to get random bytes for temporary file");
 
+		msan_unpoison(&v, sizeof(v));
+
 		/* Fill in the random bits. */
 		for (i = 0; i < num_x; i++) {
 			filename_template[i] = letters[v % num_letters];

on top of that would fix the problem you guys are seeing. I don't know if that path leads to insanity, though. Using MSan-enabled libraries is probably a better direction (should increase accuracy, and we don't have to carry these manual annotations around).

-Peff

#90 lore raw

Re: [PATCH v5 0/5] for-each-ref: introduce seeking functionality via '--start-after'

Quoted 17 lines · Jeff King

Probably:

wrapper.c +2 -0
index 2f00d2ac87..6a4c1c1c29 100644
@@ -482,6 +482,8 @@ int git_mkstemps_mode(char *pattern, int suffix_len, int mode)
 		if (csprng_bytes(&v, sizeof(v), 0) < 0)
 			return error_errno("unable to get random bytes for temporary file");
 
+		msan_unpoison(&v, sizeof(v));
+
 		/* Fill in the random bits. */
 		for (i = 0; i < num_x; i++) {
 			filename_template[i] = letters[v % num_letters];
Quoted 4 lines

on top of that would fix the problem you guys are seeing. I don't know if that path leads to insanity, though. Using MSan-enabled libraries is probably a better direction (should increase accuracy, and we don't have to carry these manual annotations around).

Hmm, probably insanity. Just for fun I tried to run the whole suite, but got this doozy:

Uninitialized bytes in fopen64 at offset 0 inside [0x7020000109c0, 25) ==2568195==WARNING: MemorySanitizer: use-of-uninitialized-value

      #0 0x7f90fe14fa46 in BIO_new_file (/lib/x86_64-linux-gnu/libcrypto.so.3+0x14fa46) (BuildId: 07a8321bad67632b52b47ad026125c79b7ebaab4)
      #1 0x7f90fe1a659c  (/lib/x86_64-linux-gnu/libcrypto.so.3+0x1a659c) (BuildId: 07a8321bad67632b52b47ad026125c79b7ebaab4)
      #2 0x7f90fe1a8453 in CONF_modules_load_file_ex (/lib/x86_64-linux-gnu/libcrypto.so.3+0x1a8453) (BuildId: 07a8321bad67632b52b47ad026125c79b7ebaab4)
      #3 0x7f90fe1a8807  (/lib/x86_64-linux-gnu/libcrypto.so.3+0x1a8807) (BuildId: 07a8321bad67632b52b47ad026125c79b7ebaab4)
      #4 0x7f90fe27274e  (/lib/x86_64-linux-gnu/libcrypto.so.3+0x27274e) (BuildId: 07a8321bad67632b52b47ad026125c79b7ebaab4)
      #5 0x7f90fea01bc6 in __pthread_once_slow nptl/pthread_once.c:116:7
      #6 0x7f90fea01c38 in __pthread_once nptl/pthread_once.c:143:12
      #7 0x7f90fe287f3c in CRYPTO_THREAD_run_once (/lib/x86_64-linux-gnu/libcrypto.so.3+0x287f3c) (BuildId: 07a8321bad67632b52b47ad026125c79b7ebaab4)
      #8 0x7f90fe272fd9 in OPENSSL_init_crypto (/lib/x86_64-linux-gnu/libcrypto.so.3+0x272fd9) (BuildId: 07a8321bad67632b52b47ad026125c79b7ebaab4)
      #9 0x7f90fe78a6d7 in OPENSSL_init_ssl (/lib/x86_64-linux-gnu/libssl.so.3+0x396d7) (BuildId: a0d77cb273378dec1d74a115ac1c9e40306e675d)
      #10 0x7f90fee2a503  (/lib/x86_64-linux-gnu/libcurl.so.4+0x9e503) (BuildId: 61ee7a8d1799c0e6c38a99b4d739e0c90391a05f)
      #11 0x7f90fedc2e42  (/lib/x86_64-linux-gnu/libcurl.so.4+0x36e42) (BuildId: 61ee7a8d1799c0e6c38a99b4d739e0c90391a05f)
      #12 0x7f90fedc33c9 in curl_global_init (/lib/x86_64-linux-gnu/libcurl.so.4+0x373c9) (BuildId: 61ee7a8d1799c0e6c38a99b4d739e0c90391a05f)
      #13 0x55fbbe211623 in http_init http.c:1347:6
      #14 0x55fbbe1f5b98 in cmd_main remote-curl.c:1583:2
      #15 0x55fbbe244571 in main common-main.c:9:11
      #16 0x7f90fe993ca7 in __libc_start_call_main csu/../sysdeps/nptl/libc_start_call_main.h:58:16
      #17 0x7f90fe993d64 in __libc_start_main csu/../csu/libc-start.c:360:3
      #18 0x55fbbe15bb60 in _start (git-remote-http+0x43b60) (BuildId: dfc63b9261f6d575776d30b4e048b235389a7b20)

SUMMARY: MemorySanitizer: use-of-uninitialized-value (/lib/x86_64-linux-gnu/libcrypto.so.3+0x14fa46) (BuildId: 07a8321bad67632b52b47ad026125c79b7ebaab4) in BIO_new_file

So MSan complaining about stuff deep within curl/openssl, and AFAICT not something we could influence or annotate as OK.

-Peff

#91 lore raw

Re: [PATCH v5 0/5] for-each-ref: introduce seeking functionality via '--start-after'

In reply to Jeff King
Quoted 73 lines · Jeff King
Quoted 31 lines · Kyle Lippincott
Quoted 29 lines

==3275333==WARNING: MemorySanitizer: use-of-uninitialized-value

    #0 0x557bd886f4bb in git_mkstemps_mode ../wrapper.c:487:27
    #1 0x557bd886fb55 in git_mkstemp_mode ../wrapper.c:509:9
    #2 0x557bd8100d1a in create_tmpfile ../object-file.c:736:7
    #3 0x557bd80f1630 in start_loose_object_common ../object-file.c:781:7
    #4 0x557bd80f5203 in write_loose_object ../object-file.c:881:7
    #5 0x557bd80f4875 in write_object_file_flags ../object-file.c:1086:6
    #6 0x557bd80f9f65 in write_object_file ../object-file.h:181:9
    #7 0x557bd8101eb8 in index_mem ../object-file.c:1177:9
    #8 0x557bd80f8bd5 in index_core ../object-file.c:1247:10
    #9 0x557bd80f731d in index_fd ../object-file.c:1274:9
    #10 0x557bd80f95e4 in index_path ../object-file.c:1295:7
    #11 0x557bd831132d in add_to_index ../read-cache.c:771:7
    #12 0x557bd8313cb1 in add_file_to_index ../read-cache.c:804:9
    #13 0x557bd73f892c in add_files ../builtin/add.c:355:7
    #14 0x557bd73f4752 in cmd_add ../builtin/add.c:578:18
    #15 0x557bd7a38b6f in run_builtin ../git.c:480:11
    #16 0x557bd7a31d54 in handle_builtin ../git.c:746:9
    #17 0x557bd7a36644 in run_argv ../git.c:813:4
    #18 0x557bd7a30e09 in cmd_main ../git.c:953:19
    #19 0x557bd7a3ca01 in main ../common-main.c:9:11
    #20 0x7f7e3f02a4d7 in __libc_start_call_main
(/nix/store/g2jzxk3s7cnkhh8yq55l4fbvf639zy37-glibc-2.40-66/lib/libc.so.6+0x2a4d7)
(BuildId: f117ee0f586dfa828cbdd08e37393c8f04f6480a)
    #21 0x7f7e3f02a59a in __libc_start_main@GLIBC_2.2.5
(/nix/store/g2jzxk3s7cnkhh8yq55l4fbvf639zy37-glibc-2.40-66/lib/libc.so.6+0x2a59a)
(BuildId: f117ee0f586dfa828cbdd08e37393c8f04f6480a)
    #22 0x557bd7352b34 in _start (git+0x5db34)
Quoted 1 line

Possibly something we need to look into cleaning up.

Quoted 8 lines

I also saw those msan issues when trying make CFLAGS=-fsanitize=memory CC=clang, but not with Google's internal msan build. I don't know which variable in wrapper.c:487 it's complaining about - you'd think it'd be letters, but if it's v, then that potentially comes from OpenSSL or some other library, and that library would also need to be built with msan (which is why it's such a pain to get msan builds working - EVERY library needs to be built with memory sanitizer).

Yeah, presumably it is "v" from csprng_bytes(). If there are only a few such spots, we can manually "unpoison" memory coming from libraries. On my system, I didn't hit the case shown above but do have trouble with bytes coming back from zlib.

Applying this ancient patch:

https://lore.kernel.org/git/[email protected]/

and building with "make SANITIZE=memory CC=clang" let me run t6302 to completion, modulo the bug that started this thread (and which I confirmed goes away both with MSan and valgrind with the fix Karthik posted).

Probably:

wrapper.c +2 -0
index 2f00d2ac87..6a4c1c1c29 100644
@@ -482,6 +482,8 @@ int git_mkstemps_mode(char *pattern, int suffix_len, int mode)
 		if (csprng_bytes(&v, sizeof(v), 0) < 0)
 			return error_errno("unable to get random bytes for temporary file");
 
+		msan_unpoison(&v, sizeof(v));
+
 		/* Fill in the random bits. */
 		for (i = 0; i < num_x; i++) {
 			filename_template[i] = letters[v % num_letters];
Quoted 5 lines

on top of that would fix the problem you guys are seeing. I don't know if that path leads to insanity, though. Using MSan-enabled libraries is probably a better direction (should increase accuracy, and we don't have to carry these manual annotations around).

I wonder if an alternate is to use '-fsanitize-ignorelist', since the MemorySanitizer is supposed to work with that too [1].

[1]: https://clang.llvm.org/docs/MemorySanitizer.html#ignorelist

Quoted 1 line

-Peff

#92 lore raw

Re: [PATCH v5 0/5] for-each-ref: introduce seeking functionality via '--start-after'

Quoted 4 lines · Karthik Nayak
Quoted 4 lines

Applying this ancient patch:

https://lore.kernel.org/git/[email protected]/

Quoted 8 lines
Quoted 4 lines

and building with "make SANITIZE=memory CC=clang" let me run t6302 to completion, modulo the bug that started this thread (and which I confirmed goes away both with MSan and valgrind with the fix Karthik posted).

[...]

I wonder if an alternate is to use '-fsanitize-ignorelist', since the MemorySanitizer is supposed to work with that too [1].

I think you could do that, but it isn't quite what we want: it is annotating the access of those (false-positive) "uninitialized" bytes. So you have to mark every spot that touches bytes that come from zlib, which in Git is a lot of places. And so the patch linked above was an attempt to silence all of those with a single line: marking the bytes coming out of zlib as OK.

-Peff

#93 lore raw

Re: [PATCH v5 0/5] for-each-ref: introduce seeking functionality via '--start-after'

Quoted 15 lines · Jeff King
Quoted 4 lines · Karthik Nayak
Quoted 4 lines

Applying this ancient patch:

https://lore.kernel.org/git/[email protected]/

Quoted 8 lines
Quoted 4 lines

and building with "make SANITIZE=memory CC=clang" let me run t6302 to completion, modulo the bug that started this thread (and which I confirmed goes away both with MSan and valgrind with the fix Karthik posted).

[...]

I wonder if an alternate is to use '-fsanitize-ignorelist', since the MemorySanitizer is supposed to work with that too [1].

Quoted 7 lines

I think you could do that, but it isn't quite what we want: it is annotating the access of those (false-positive) "uninitialized" bytes. So you have to mark every spot that touches bytes that come from zlib, which in Git is a lot of places. And so the patch linked above was an attempt to silence all of those with a single line: marking the bytes coming out of zlib as OK.

That makes sense, thanks for explaining.

Quoted 1 line

-Peff

#94 lore raw

Re: [PATCH v5 0/5] for-each-ref: introduce seeking functionality via '--start-after'

In reply to Karthik Nayak
Quoted 20 lines · Karthik Nayak

This series allows the seek function to set the cursor without setting the prefix, which is a requirement for pagination. So there is no need to set 'prefix_state' for this functionality. Which is why I didn't set it, since the default value of '0' (PREFIX_CONTAINS_DIR) would be the correct setting for all dirs. This causes the issue.

So the only fix required would be

refs/ref-cache.c +1 -0
index 1d95b56d40..ceef3a2008 100644
@@ -527,6 +527,7 @@ static int cache_ref_iterator_seek(struct
 ref_iterator *ref_iterator,
 				level = &iter->levels[iter->levels_nr++];
 				level->dir = dir;
 				level->index = -1;
+				level->prefix_state = PREFIX_CONTAINS_DIR;
 			} else {
 				/* reduce the index so the leaf node is iterated over */
Quoted 1 line

if (cmp <= 0 && !slash)

Yup, that is inside the code added by this series. It does look like it fixes the complaint from the checker.

Thanks.

#95 lore raw

PATCH 1/1 ref-iterator-seek: correctly initialize the prefix_state for a new level

In reply to Karthik Nayak

When cache_ref_iterator_seek() "jumps" to a middle of the sorted ref list, it forgets to set the .prefix_state member of the new (i.e. deeper) level it just initialized. This later causes cache_ref_iterator_advance() to look at this uninitialized member to base its decision on what to do next.

Kyle Lippincott [*] and Jeff King noticed this with MSAN and Valgrind, and Karthik Nayak as the original author located exactly where the missing initialization is.

[*] <CAO_smVg9TDakUnubepjPGmLyOzW6n8Z=MDbnZKvkwN2=kN2RRw@mail.gmail.com>

---
 refs/ref-cache.c | 1 +
 1 file changed, 1 insertion(+)

* I had this as "fixup!" on top of your topic for quite a while and forgot to ask you to send in an official fix. As Kyle's discovery was after the topic hit 'next' (understandable, as their internal edition of Git is based on 'next'), we need a separate fix on top.

To prepare for merging down the whole thing to 'master', I wrote the proposed log message to help expedite the process. Comments?

refs/ref-cache.c +1 -0
index 1d95b56d40..ceef3a2008 100644
@@ -527,6 +527,7 @@ static int cache_ref_iterator_seek(struct ref_iterator *ref_iterator,
 				level = &iter->levels[iter->levels_nr++];
 				level->dir = dir;
 				level->index = -1;
+				level->prefix_state = PREFIX_CONTAINS_DIR;
 			} else {
 				/* reduce the index so the leaf node is iterated over */
 				if (cmp <= 0 && !slash)
Signature · 1 line
2.50.1-521-gf11ee0bd80
#96 lore raw

Re: [PATCH] ref-iterator-seek: correctly initialize the prefix_state for a new level

Quoted 7 lines · Junio C Hamano

When cache_ref_iterator_seek() "jumps" to a middle of the sorted ref list, it forgets to set the .prefix_state member of the new (i.e. deeper) level it just initialized. This later causes cache_ref_iterator_advance() to look at this uninitialized member to base its decision on what to do next.

Quoted 7 lines

Kyle Lippincott [*] and Jeff King noticed this with MSAN and Valgrind, and Karthik Nayak as the original author located exactly where the missing initialization is.

[*] <CAO_smVg9TDakUnubepjPGmLyOzW6n8Z=MDbnZKvkwN2=kN2RRw@mail.gmail.com>

Looks good to me, thanks!

Quoted 28 lines
---
 refs/ref-cache.c | 1 +
 1 file changed, 1 insertion(+)

* I had this as "fixup!" on top of your topic for quite a while and forgot to ask you to send in an official fix. As Kyle's discovery was after the topic hit 'next' (understandable, as their internal edition of Git is based on 'next'), we need a separate fix on top.

To prepare for merging down the whole thing to 'master', I wrote the proposed log message to help expedite the process. Comments?

refs/ref-cache.c +1 -1
index 1d95b56d40..ceef3a2008 100644
@@ -527,6 +527,7 @@ static int cache_ref_iterator_seek(struct ref_iterator *ref_iterator,
                                level = &iter->levels[iter->levels_nr++];
                                level->dir = dir;
                                level->index = -1;
+                               level->prefix_state = PREFIX_CONTAINS_DIR;
                        } else {
                                /* reduce the index so the leaf node is iterated over */
                                if (cmp <= 0 && !slash)
--

2.50.1-521-gf11ee0bd80

#97 lore raw

Re: [PATCH] ref-iterator-seek: correctly initialize the prefix_state for a new level

In reply to Junio C Hamano
Quoted 9 lines · Junio C Hamano

When cache_ref_iterator_seek() "jumps" to a middle of the sorted ref list, it forgets to set the .prefix_state member of the new (i.e. deeper) level it just initialized. This later causes cache_ref_iterator_advance() to look at this uninitialized member to base its decision on what to do next.

Kyle Lippincott [*] and Jeff King noticed this with MSAN and Valgrind, and Karthik Nayak as the original author located exactly where the missing initialization is.

This explanation makes sense to me (from my admittedly rusty view of the ref iteration code). And certainly the patch looks right.

-Peff

#98 lore raw

Re: [PATCH] ref-iterator-seek: correctly initialize the prefix_state for a new level

In reply to Junio C Hamano
Quoted 6 lines · Junio C Hamano

When cache_ref_iterator_seek() "jumps" to a middle of the sorted ref list, it forgets to set the .prefix_state member of the new (i.e. deeper) level it just initialized. This later causes cache_ref_iterator_advance() to look at this uninitialized member to base its decision on what to do next.

I think the explanation is correct. For reference I had some more details in my local patch, but this is totally okay.

ref-cache: set prefix_state when seeking

In 090eb5336c (refs: selectively set prefix in the seek functions, 2025-07-15) we separated the seeking functionality of reference iterators from the functionality to set prefix to an iterator. This allows users of ref iterators to seek to a particular reference to provide pagination support.

The files-backend, uses the ref-cache iterator to iterate over loose refs. The iterator tracks directories and entries already processed via a stack of levels. Each level corresponds to a directory under the files backend. New levels are added to the stack, and when all entries from a level is yielded, the corresponding level is popped from the stack.

To accommodate seeking, we need to populate and traverse the levels to stop the requested seek marker at the appropriate level and its entry index. Each level also contains a 'prefix_state' which is used for prefix matching, this allows the iterator to skip levels/entries which don't match a prefix. The default value of 'prefix_state' is PREFIX_CONTAINS_DIR, which yields all entries within a level. When purely seeking without prefix matching, we want to yield all entries. The commit however, skips setting the value explicitly. This causes the MemorySanitizer to issue a 'use-of-uninitialized-value' error when running 't/t6302-for-each-ref-filter'.

Set the value explicitly to avoid to fix the issue.

Quoted 11 lines

Kyle Lippincott [*] and Jeff King noticed this with MSAN and Valgrind, and Karthik Nayak as the original author located exactly where the missing initialization is.

[*] <CAO_smVg9TDakUnubepjPGmLyOzW6n8Z=MDbnZKvkwN2=kN2RRw@mail.gmail.com>

---
 refs/ref-cache.c | 1 +
 1 file changed, 1 insertion(+)
Quoted 9 lines

* I had this as "fixup!" on top of your topic for quite a while and forgot to ask you to send in an official fix. As Kyle's discovery was after the topic hit 'next' (understandable, as their internal edition of Git is based on 'next'), we need a separate fix on top.

To prepare for merging down the whole thing to 'master', I wrote the proposed log message to help expedite the process. Comments?

I had a set of patches locally, I just didn't get around to sending it. Will send the others, omitting this. Thanks for doing it!

Quoted 13 lines
refs/ref-cache.c +1 -1
index 1d95b56d40..ceef3a2008 100644
@@ -527,6 +527,7 @@ static int cache_ref_iterator_seek(struct ref_iterator *ref_iterator,
 				level = &iter->levels[iter->levels_nr++];
 				level->dir = dir;
 				level->index = -1;
+				level->prefix_state = PREFIX_CONTAINS_DIR;
 			} else {
 				/* reduce the index so the leaf node is iterated over */
 				if (cmp <= 0 && !slash)
--
Quoted 1 line

2.50.1-521-gf11ee0bd80

The patch looks good.

#99 lore raw

Re: [PATCH] ref-iterator-seek: correctly initialize the prefix_state for a new level

Quoted 6 lines · Karthik Nayak
Quoted 6 lines

* I had this as "fixup!" on top of your topic for quite a while and forgot to ask you to send in an official fix. As Kyle's discovery was after the topic hit 'next' (understandable, as their internal edition of Git is based on 'next'), we need a separate fix on top.

Quoted 6 lines
Quoted 3 lines

To prepare for merging down the whole thing to 'master', I wrote the proposed log message to help expedite the process. Comments?

I had a set of patches locally, I just didn't get around to sending it. Will send the others, omitting this. Thanks for doing it!

I do not mind discarding what I sent out at all. I actually prefer if it came from you.

Thanks.

#100 lore raw

PATCH 1/1 ref-cache: set prefix_state when seeking

In 090eb5336c (refs: selectively set prefix in the seek functions, 2025-07-15) we separated the seeking functionality of reference iterators from the functionality to set prefix to an iterator. This allows users of ref iterators to seek to a particular reference to provide pagination support.

The files-backend, uses the ref-cache iterator to iterate over loose refs. The iterator tracks directories and entries already processed via a stack of levels. Each level corresponds to a directory under the files backend. New levels are added to the stack, and when all entries from a level is yielded, the corresponding level is popped from the stack.

To accommodate seeking, we need to populate and traverse the levels to stop the requested seek marker at the appropriate level and its entry index. Each level also contains a 'prefix_state' which is used for prefix matching, this allows the iterator to skip levels/entries which don't match a prefix. The default value of 'prefix_state' is PREFIX_CONTAINS_DIR, which yields all entries within a level. When purely seeking without prefix matching, we want to yield all entries. The commit however, skips setting the value explicitly. This causes the MemorySanitizer to issue a 'use-of-uninitialized-value' error when running 't/t6302-for-each-ref-filter'.

Set the value explicitly to avoid to fix the issue.

---

Here is my version of the same patch!

 refs/ref-cache.c | 1 +
 1 file changed, 1 insertion(+)
refs/ref-cache.c +1 -0
index 1d95b56d40..ceef3a2008 100644
@@ -527,6 +527,7 @@ static int cache_ref_iterator_seek(struct ref_iterator *ref_iterator,
 				level = &iter->levels[iter->levels_nr++];
 				level->dir = dir;
 				level->index = -1;
+				level->prefix_state = PREFIX_CONTAINS_DIR;
 			} else {
 				/* reduce the index so the leaf node is iterated over */
 				if (cmp <= 0 && !slash)
Signature · 1 line
2.49.0
#101 lore raw

Re: [PATCH] ref-cache: set prefix_state when seeking

Quoted 25 lines · Karthik Nayak

In 090eb5336c (refs: selectively set prefix in the seek functions, 2025-07-15) we separated the seeking functionality of reference iterators from the functionality to set prefix to an iterator. This allows users of ref iterators to seek to a particular reference to provide pagination support.

The files-backend, uses the ref-cache iterator to iterate over loose refs. The iterator tracks directories and entries already processed via a stack of levels. Each level corresponds to a directory under the files backend. New levels are added to the stack, and when all entries from a level is yielded, the corresponding level is popped from the stack.

To accommodate seeking, we need to populate and traverse the levels to stop the requested seek marker at the appropriate level and its entry index. Each level also contains a 'prefix_state' which is used for prefix matching, this allows the iterator to skip levels/entries which don't match a prefix. The default value of 'prefix_state' is PREFIX_CONTAINS_DIR, which yields all entries within a level. When purely seeking without prefix matching, we want to yield all entries. The commit however, skips setting the value explicitly. This causes the MemorySanitizer to issue a 'use-of-uninitialized-value' error when running 't/t6302-for-each-ref-filter'.

Set the value explicitly to avoid to fix the issue.

Quoted 7 lines

---

Here is my version of the same patch!

Thanks!

Quoted 16 lines
 refs/ref-cache.c | 1 +
 1 file changed, 1 insertion(+)
refs/ref-cache.c +1 -0
index 1d95b56d40..ceef3a2008 100644
@@ -527,6 +527,7 @@ static int cache_ref_iterator_seek(struct ref_iterator *ref_iterator,
 				level = &iter->levels[iter->levels_nr++];
 				level->dir = dir;
 				level->index = -1;
+				level->prefix_state = PREFIX_CONTAINS_DIR;
 			} else {
 				/* reduce the index so the leaf node is iterated over */
 				if (cmp <= 0 && !slash)