for-each-ref: introduce seeking functionality via '--skip-until'
cover for-each-ref: introduce seeking functionality via '--skip-until'
The git-for-each-ref(1) command is used to iterate over references present in a repository. In large repositories with millions of references, it would be optimal to paginate this output such that we can start iteration from a given reference. This would avoid having to iterate over all references from the beginning each time when paginating through results.
This series adds a '--skip-until' option in 'git-for-each-ref(1)'. When used, the reference iteration seeks to the first matching reference and iterates from there onward.
This enables efficient pagination workflows like:
git for-each-ref --count=100
git for-each-ref --count=100 --skip-until=refs/heads/branch-100
git for-each-ref --count=100 --skip-until=refs/heads/branch-200To add this functionality, we expose the ref_iterator outside the 'refs/' namespace and modify the ref_iterator_seek() to actually seek to a given reference and only set the prefix when the set_prefix field is set.
On the reftable and packed backend, the changes are simple. But since the files backend uses 'ref-cache' for reference handling, the changes there are a little more involved, since we need to setup the right levels and the indexing.
Initally I was also planning to cleanup all the refs_for_each...() functions in 'refs.h' by simply using the iterator, but this bloated the series. So I've left that for another day.
--- Documentation/git-for-each-ref.adoc | 6 +- builtin/for-each-ref.c | 5 + ref-filter.c | 57 ++++++++---- ref-filter.h | 1 + refs.c | 2 +- refs.h | 153 ++++++++++++++++++++++++++++++ refs/debug.c | 7 +- refs/files-backend.c | 7 +- refs/iterator.c | 24 +++-- refs/packed-backend.c | 15 +-- refs/ref-cache.c | 95 +++++++++++++++---- refs/ref-cache.h | 7 -- refs/refs-internal.h | 152 ++---------------------------- refs/reftable-backend.c | 17 ++-- t/t6302-for-each-ref-filter.sh | 180 ++++++++++++++++++++++++++++++++++++ 15 files changed, 507 insertions(+), 221 deletions(-)
Karthik Nayak (4):
refs: expose ref_iterator via 'refs.h'
ref-cache: remove unused function 'find_ref_entry()'
refs: selectively set prefix in the seek functions
for-each-ref: introduce a '--skip-until' option
base-commit: cf6f63ea6bf35173e02e18bdc6a4ba41288acff9
change-id: 20250605-306-git-for-each-ref-pagination-0ba8a29ae646
Thanks
- Karthik
PATCH 1/4 refs: expose `ref_iterator` via 'refs.h'
The ref_iterator is an internal structure to the 'refs/' sub-directory, which allows iteration over refs. All reference iteration is built on top of these iterators.
External clients of the 'refs' subsystem use the various 'refs_for_each...()' functions to iterate over refs. However since these are wrapper functions, each combination of functionality requires a new wrapper function. This is not feasible as the functions pile up with the increase in requirements. Expose the internal reference iterator, so advanced users can mix and match options as needed.
--- refs.h | 148 +++++++++++++++++++++++++++++++++++++++++++++++++++ refs/refs-internal.h | 145 +------------------------------------------------ 2 files changed, 150 insertions(+), 143 deletions(-)
refs.h +148 -0
@@ -1190,4 +1190,152 @@ int repo_migrate_ref_storage_format(struct repository *repo,unsigned int flags, struct strbuf *err); +/* +* Reference iterators +* +* A reference iterator encapsulates the state of an in-progress +* iteration over references. Create an instance of `struct +* ref_iterator` via one of the functions in this module. +* +* A freshly-created ref_iterator doesn't yet point at a reference. To +* advance the iterator, call ref_iterator_advance(). If successful, +* this sets the iterator's refname, oid, and flags fields to describe +* the next reference and returns ITER_OK. The data pointed at by +* refname and oid belong to the iterator; if you want to retain them +* after calling ref_iterator_advance() again or calling +* ref_iterator_free(), you must make a copy. When the iteration has +* been exhausted, ref_iterator_advance() releases any resources +* associated with the iteration, frees the ref_iterator object, and +* returns ITER_DONE. If you want to abort the iteration early, call +* ref_iterator_free(), which also frees the ref_iterator object and +* any associated resources. If there was an internal error advancing +* to the next entry, ref_iterator_advance() aborts the iteration, +* frees the ref_iterator, and returns ITER_ERROR. +* +* The reference currently being looked at can be peeled by calling +* ref_iterator_peel(). This function is often faster than peel_ref(), +* so it should be preferred when iterating over references. +* +* Putting it all together, a typical iteration looks like this: +* +* int ok; +* struct ref_iterator *iter = ...; +* +* while ((ok = ref_iterator_advance(iter)) == ITER_OK) { +* if (want_to_stop_iteration()) { +* ok = ITER_DONE; +* break; +* } +* +* // Access information about the current reference: +* if (!(iter->flags & REF_ISSYMREF)) +* printf("%s is %s\n", iter->refname, oid_to_hex(iter->oid)); +* +* // If you need to peel the reference: +* ref_iterator_peel(iter, &oid); +* } +* +* if (ok != ITER_DONE) +* handle_error(); +* ref_iterator_free(iter); +*/ +struct ref_iterator ; + +/* +* These flags are passed to refs_ref_iterator_begin() (and do_for_each_ref(), +* which feeds it). +*/ +enum do_for_each_ref_flags { +/* +* Include broken references in a do_for_each_ref*() iteration, which +* would normally be omitted. This includes both refs that point to +* missing objects (a true repository corruption), ones with illegal +* names (which we prefer not to expose to callers), as well as +* dangling symbolic refs (i.e., those that point to a non-existent +* ref; this is not a corruption, but as they have no valid oid, we +* omit them from normal iteration results). +*/ +DO_FOR_EACH_INCLUDE_BROKEN = (1 <<0 ), + +/* +* Only include per-worktree refs in a do_for_each_ref*() iteration. +* Normally this will be used with a files ref_store, since that's +* where all reference backends will presumably store their +* per-worktree refs. +*/ +DO_FOR_EACH_PER_WORKTREE_ONLY = (1 <<1 ), + +/* +* Omit dangling symrefs from output; this only has an effect with +* INCLUDE_BROKEN, since they are otherwise not included at all. +*/ +DO_FOR_EACH_OMIT_DANGLING_SYMREFS = (1 <<2 ), + +/* +* Include root refs i.e. HEAD and pseudorefs along with the regular +* refs. +*/ +DO_FOR_EACH_INCLUDE_ROOT_REFS = (1 <<3 ), +}; + +/* +* Return an iterator that goes over each reference in `refs` for +* which the refname begins with prefix. If trim is non-zero, then +* trim that many characters off the beginning of each refname. +* The output is ordered by refname. +*/ +struct ref_iterator *refs_ref_iterator_begin ( +struct ref_store *refs, +const char *prefix,const char **exclude_patterns, +int trim,enum do_for_each_ref_flags flags); + +/* +* Advance the iterator to the first or next item and return ITER_OK. +* If the iteration is exhausted, free the resources associated with +* the ref_iterator and return ITER_DONE. On errors, free the iterator +* resources and return ITER_ERROR. It is a bug to use ref_iterator or +* call this function again after it has returned ITER_DONE or +* ITER_ERROR. +*/ +int ref_iterator_advance (struct ref_iterator *ref_iterator); + +/* +* Seek the iterator to the first reference with the given prefix. +* The prefix is matched as a literal string, without regard for path +* separators. If prefix is NULL or the empty string, seek the iterator to the +* first reference again. +* +* This function is expected to behave as if a new ref iterator with the same +* prefix had been created, but allows reuse of iterators and thus may allow +* the backend to optimize. Parameters other than the prefix that have been +* passed when creating the iterator will remain unchanged. +* +* Returns 0 on success, a negative error code otherwise. +*/ +int ref_iterator_seek (struct ref_iterator *ref_iterator, +const char *prefix); + +/* +* If possible, peel the reference currently being viewed by the +* iterator. Return 0 on success. +*/ +int ref_iterator_peel (struct ref_iterator *ref_iterator, +struct object_id *peeled); + +/* Free the reference iterator and any associated resources. */ +void ref_iterator_free (struct ref_iterator *ref_iterator); + +/* +* The common backend for the for_each_*ref* functions. Call fn for +* each reference in iter. If the iterator itself ever returns +* ITER_ERROR, return -1. If fn ever returns a non-zero value, stop +* the iteration and return that value. Otherwise, return 0. In any +* case, free the iterator when done. This function is basically an +* adapter between the callback style of reference iteration and the +* iterator style. +*/ +int do_for_each_ref_iterator (struct ref_iterator *iter, +each_ref_fn fn,void *cb_data); + +#endif /* REFS_H */
refs/refs-internal.h +2 -143
169 lines not on this page · raw message
Signature · 1 line
2.49.0
PATCH 2/4 ref-cache: remove unused function 'find_ref_entry()'
In reply to Karthik NayakThe 'find_ref_entry' function is no longer used, so remove it.
--- refs/ref-cache.c | 14 -------------- refs/ref-cache.h | 7 ------- 2 files changed, 21 deletions(-)
refs/ref-cache.c +0 -14
@@ -194,20 +194,6 @@ static struct ref_dir *find_containing_dir(struct ref_dir *dir,return dir; } -struct ref_entry *find_ref_entry (struct ref_dir *dir,const char *refname) -{ -int entry_index; -struct ref_entry *entry; - dir =find_containing_dir (dir, refname); -if (!dir) -return NULL ; - entry_index =search_ref_dir (dir, refname,strlen (refname)); -if (entry_index ==-1 ) -return NULL ; - entry = dir->entries[entry_index]; -return (entry->flag &REF_DIR ) ?NULL : entry; -} -/* * Emit a warning and return true iff ref1 and ref2 have the same name * and the same oid. Die if they have the same name but different
refs/ref-cache.h +0 -7
@@ -201,13 +201,6 @@ void free_ref_cache(struct ref_cache *cache);*/ void add_entry_to_dir (struct ref_dir *dir,struct ref_entry *entry); -/* -* Find the value entry with the given name in dir, sorting ref_dirs -* and recursing into subdirectories as necessary. If the name is not -* found or it corresponds to a directory entry, return NULL. -*/ -struct ref_entry *find_ref_entry (struct ref_dir *dir,const char *refname); -/* * Start iterating over references in `cache`. If `prefix` is * specified, only include references whose names start with that
Signature · 1 line
2.49.0
Re: [PATCH 2/4] ref-cache: remove unused function 'find_ref_entry()'
Quoted 1 line · Karthik Nayak
The '
find_ref_entry' function is no longer used, so remove it.
It seems that ba1c052f (ref_store: implement refs_peel_ref() generically, 2017-09-25) removed the last caller of it. This is long overdue ;-)
Quoted 49 lines
Signed-off-by Karthik Nayak <[email protected]>--- refs/ref-cache.c | 14 -------------- refs/ref-cache.h | 7 ------- 2 files changed, 21 deletions(-)refs/ref-cache.c +0
-14@@ -194,20 +194,6 @@ static struct ref_dir *find_containing_dir(struct ref_dir *dir,return dir; } -struct ref_entry *find_ref_entry (struct ref_dir *dir,const char *refname) -{ -int entry_index; -struct ref_entry *entry; - dir =find_containing_dir (dir, refname); -if (!dir) -return NULL ; - entry_index =search_ref_dir (dir, refname,strlen (refname)); -if (entry_index ==-1 ) -return NULL ; - entry = dir->entries[entry_index]; -return (entry->flag &REF_DIR ) ?NULL : entry; -} -/* * Emit a warning and return true iff ref1 and ref2 have the same name * and the same oid. Die if they have the same name but different refs/ref-cache.h +0
-7@@ -201,13 +201,6 @@ void free_ref_cache(struct ref_cache *cache);*/ void add_entry_to_dir (struct ref_dir *dir,struct ref_entry *entry); -/* -* Find the value entry with the given name in dir, sorting ref_dirs -* and recursing into subdirectories as necessary. If the name is not -* found or it corresponds to a directory entry, return NULL. -*/ -struct ref_entry *find_ref_entry (struct ref_dir *dir,const char *refname); -/* * Start iterating over references in `cache`. If `prefix` is * specified, only include references whose names start with that
PATCH 3/4 refs: selectively set prefix in the seek functions
In reply to Karthik NayakThe ref iterator exposes a ref_iterator_seek() function. The name suggests that this would seek the iterator to a specific reference in some ways similar to how fseek() works for the filesystem.
However, the function actually sets the prefix for refs iteration. So further iteration would only yield references which match the particular prefix. This is a bit confusing.
Let's add a 'set_prefix' field to the function, which when set, will set the prefix for the iteration in-line with the existing behavior. But when the 'set_prefix' field is not set, the reference backends will simply seek to the specified reference without setting prefix. This allows users to start iteration from a specific reference.
In the packed and reftable backend, since references are available in a sorted list, the changes are simply setting the prefix if needed. The changes on the files-backend are a little more involved, since the files backend uses the 'ref-cache' mechanism. We move out the existing logic within cache_ref_iterator_seek() to cache_ref_iterator_set_prefix() which is called when set_prefix is set. We then parse the provided seek string and set the required levels and their indexes to ensure that seeking is possible.
--- refs.c | 2 +- refs.h | 19 +++++++----- refs/debug.c | 7 +++-- refs/files-backend.c | 7 +++-- refs/iterator.c | 24 +++++++++------ refs/packed-backend.c | 15 +++++---- refs/ref-cache.c | 81 ++++++++++++++++++++++++++++++++++++++++++++++--- refs/refs-internal.h | 7 +++-- refs/reftable-backend.c | 17 ++++++----- 9 files changed, 134 insertions(+), 45 deletions(-)
refs.c +1 -1
@@ -2669,7 +2669,7 @@ enum ref_transaction_error refs_verify_refnames_available(struct ref_store *refsif (!iter) { iter =refs_ref_iterator_begin (refs, dirname.buf,NULL ,0 ,DO_FOR_EACH_INCLUDE_BROKEN ); - }else if (ref_iterator_seek (iter, dirname.buf) <0 ) { + }else if (ref_iterator_seek (iter, dirname.buf,1 ) <0 ) { goto cleanup; }
refs.h +12 -7
@@ -1300,20 +1300,25 @@ struct ref_iterator *refs_ref_iterator_begin(int ref_iterator_advance (struct ref_iterator *ref_iterator);/* -* Seek the iterator to the first reference with the given prefix. -* The prefix is matched as a literal string, without regard for path +* Seek the iterator to the first reference matching the given seek string. +* The seek string is matched as a literal string, without regard for path * separators. If prefix is NULL or the empty string, seek the iterator to the * first reference again. * -* This function is expected to behave as if a new ref iterator with the same -* prefix had been created, but allows reuse of iterators and thus may allow -* the backend to optimize. Parameters other than the prefix that have been -* passed when creating the iterator will remain unchanged. +* When set_prefix is true, this function behaves as if a new ref iterator +* with the same prefix had been created, setting the prefix for subsequent +* iteration. When set_prefix is false, the iterator simply seeks to the +* specified reference without changing the existing prefix, allowing +* iteration to start from that specific reference. +* +* This function allows reuse of iterators and thus may allow the backend +* to optimize. Parameters other than the prefix that have been passed when +* creating the iterator will remain unchanged. * * Returns 0 on success, a negative error code otherwise. */ int ref_iterator_seek (struct ref_iterator *ref_iterator, -const char *prefix); +const char *seek,int set_prefix);/* * If possible, peel the reference currently being viewed by the
refs/debug.c +4 -3
@@ -170,12 +170,13 @@ static int debug_ref_iterator_advance(struct ref_iterator *ref_iterator) }static int debug_ref_iterator_seek (struct ref_iterator *ref_iterator, -const char *prefix) +const char *seek,int set_prefix) {struct debug_ref_iterator *diter = (struct debug_ref_iterator *)ref_iterator; -int res = diter->iter->vtable->seek (diter->iter, prefix); -trace_printf_key (&trace_refs,"iterator_seek: %s: %d\n" , prefix ? prefix :"" , res); +int res = diter->iter->vtable->seek (diter->iter, seek, set_prefix); +trace_printf_key (&trace_refs,"iterator_seek: %s set_prefix: %d: %d\n" , + seek ? seek :"" , set_prefix, res);return res; }
refs/files-backend.c +4 -3
@@ -929,11 +929,11 @@ static int files_ref_iterator_advance(struct ref_iterator *ref_iterator) }static int files_ref_iterator_seek (struct ref_iterator *ref_iterator, -const char *prefix) +const char *seek,int set_prefix) {struct files_ref_iterator *iter = (struct files_ref_iterator *)ref_iterator; -return ref_iterator_seek (iter->iter0, prefix); +return ref_iterator_seek (iter->iter0, seek, set_prefix); } static int files_ref_iterator_peel(struct ref_iterator *ref_iterator,
@@ -2316,7 +2316,8 @@ static int files_reflog_iterator_advance(struct ref_iterator *ref_iterator) } static int files_reflog_iterator_seek(struct ref_iterator *ref_iterator UNUSED, - const char *prefix UNUSED) + const char *seek UNUSED, + int set_prefix UNUSED) {BUG ("ref_iterator_seek() called for reflog_iterator" ); }
refs/iterator.c +14 -10
@@ -16,9 +16,9 @@ int ref_iterator_advance(struct ref_iterator *ref_iterator) }int ref_iterator_seek (struct ref_iterator *ref_iterator, -const char *prefix) +const char *seek,int set_prefix) { -return ref_iterator->vtable->seek (ref_iterator, prefix); +return ref_iterator->vtable->seek (ref_iterator, seek, set_prefix); } int ref_iterator_peel(struct ref_iterator *ref_iterator,
@@ -57,7 +57,8 @@ static int empty_ref_iterator_advance(struct ref_iterator *ref_iterator UNUSED) } static int empty_ref_iterator_seek(struct ref_iterator *ref_iterator UNUSED, - const char *prefix UNUSED) + const char *seek UNUSED, + int set_prefix UNUSED) {return 0 ; }
@@ -224,7 +225,7 @@ static int merge_ref_iterator_advance(struct ref_iterator *ref_iterator) }static int merge_ref_iterator_seek (struct ref_iterator *ref_iterator, -const char *prefix) +const char *seek,int set_prefix) {struct merge_ref_iterator *iter = (struct merge_ref_iterator *)ref_iterator;
@@ -234,11 +235,11 @@ static int merge_ref_iterator_seek(struct ref_iterator *ref_iterator, iter->iter0 = iter->iter0_owned; iter->iter1 = iter->iter1_owned; - ret =ref_iterator_seek (iter->iter0, prefix); + ret =ref_iterator_seek (iter->iter0, seek, set_prefix);if (ret <0 )return ret; - ret =ref_iterator_seek (iter->iter1, prefix); + ret =ref_iterator_seek (iter->iter1, seek, set_prefix);if (ret <0 )return ret;
@@ -407,13 +408,16 @@ static int prefix_ref_iterator_advance(struct ref_iterator *ref_iterator) }static int prefix_ref_iterator_seek (struct ref_iterator *ref_iterator, -const char *prefix) +const char *seek,int set_prefix) {struct prefix_ref_iterator *iter = (struct prefix_ref_iterator *)ref_iterator; -free (iter->prefix); - iter->prefix =xstrdup_or_null (prefix); -return ref_iterator_seek (iter->iter0, prefix); + +if (set_prefix) { +free (iter->prefix); + iter->prefix =xstrdup_or_null (seek); + } +return ref_iterator_seek (iter->iter0, seek, set_prefix); } static int prefix_ref_iterator_peel(struct ref_iterator *ref_iterator,
refs/packed-backend.c +9 -6
@@ -1004,19 +1004,22 @@ static int packed_ref_iterator_advance(struct ref_iterator *ref_iterator) }static int packed_ref_iterator_seek (struct ref_iterator *ref_iterator, -const char *prefix) +const char *seek,int set_prefix) {struct packed_ref_iterator *iter = (struct packed_ref_iterator *)ref_iterator;const char *start; -if (prefix && *prefix) - start =find_reference_location (iter->snapshot, prefix,0 ); +if (seek && *seek) + start =find_reference_location (iter->snapshot, seek,0 );else start = iter->snapshot->start; -free (iter->prefix); - iter->prefix =xstrdup_or_null (prefix); +if (set_prefix) { +free (iter->prefix); + iter->prefix =xstrdup_or_null (seek); + } + iter->pos = start; iter->eof = iter->snapshot->eof;
@@ -1194,7 +1197,7 @@ static struct ref_iterator *packed_ref_iterator_begin( iter->repo = ref_store->repo; iter->flags = flags; -if (packed_ref_iterator_seek (&iter->base, prefix) <0 ) { +if (packed_ref_iterator_seek (&iter->base, prefix,1 ) <0 ) {ref_iterator_free (&iter->base);return NULL ; }
refs/ref-cache.c +76 -5
100 lines not on this page · raw message
refs/refs-internal.h +4 -3
@@ -353,11 +353,12 @@ void base_ref_iterator_init(struct ref_iterator *iter,typedef int ref_iterator_advance_fn (struct ref_iterator *ref_iterator);/* -* Seek the iterator to the first reference matching the given prefix. Should -* behave the same as if a new iterator was created with the same prefix. +* Seek the iterator to the first matching reference. If set_prefix is set, +* it would behave the same as if a new iterator was created with the same +* prefix. */ typedef int ref_iterator_seek_fn (struct ref_iterator *ref_iterator, -const char *prefix); +const char *seek,int set_prefix);/* * Peels the current ref, returning 0 for success or -1 for failure.
refs/reftable-backend.c +10 -7
@@ -719,15 +719,17 @@ static int reftable_ref_iterator_advance(struct ref_iterator *ref_iterator) }static int reftable_ref_iterator_seek (struct ref_iterator *ref_iterator, -const char *prefix) +const char *seek,int set_prefix) {struct reftable_ref_iterator *iter = (struct reftable_ref_iterator *)ref_iterator; -free (iter->prefix); - iter->prefix =xstrdup_or_null (prefix); - iter->prefix_len = prefix ?strlen (prefix) :0 ; - iter->err =reftable_iterator_seek_ref (&iter->iter, prefix); +if (set_prefix) { +free (iter->prefix); + iter->prefix =xstrdup_or_null (seek); + iter->prefix_len = seek ?strlen (seek) :0 ; + } + iter->err =reftable_iterator_seek_ref (&iter->iter, seek);return iter->err; }
@@ -839,7 +841,7 @@ static struct reftable_ref_iterator *ref_iterator_for_stack(struct reftable_ref_if (ret) goto done; - ret =reftable_ref_iterator_seek (&iter->base, prefix); + ret =reftable_ref_iterator_seek (&iter->base, prefix,1 );if (ret) goto done;
@@ -2042,7 +2044,8 @@ static int reftable_reflog_iterator_advance(struct ref_iterator *ref_iterator) } static int reftable_reflog_iterator_seek(struct ref_iterator *ref_iterator UNUSED, - const char *prefix UNUSED) + const char *seek UNUSED, + int set_prefix UNUSED) {BUG ("reftable reflog iterator cannot be seeked" );return -1 ;
Signature · 1 line
2.49.0
Re: [PATCH 3/4] refs: selectively set prefix in the seek functions
Quoted 14 lines · Karthik Nayak
The ref iterator exposes a
ref_iterator_seek()function. The name suggests that this would seek the iterator to a specific reference in some ways similar to howfseek()works for the filesystem.However, the function actually sets the prefix for refs iteration. So further iteration would only yield references which match the particular prefix. This is a bit confusing.
Let's add a '
set_prefix' field to the function, which when set, will set the prefix for the iteration in-line with the existing behavior. But when the 'set_prefix' field is not set, the reference backends will simply seek to the specified reference without setting prefix. This allows users to start iteration from a specific reference.
Quoted 8 lines
In the packed and reftable backend, since references are available in a sorted list, the changes are simply setting the prefix if needed. The changes on the files-backend are a little more involved, since the files backend uses the 'ref-cache' mechanism. We move out the existing logic within
cache_ref_iterator_seek()tocache_ref_iterator_set_prefix()which is called whenset_prefixis set. We then parse the provided seek string and set the required levels and their indexes to ensure that seeking is possible.
That solution makes sense.
refs.c enum ref_transaction_error refs_verify_refnames_available(struct ref_store *refs · line 2675
Quoted 13 lines
refs.c +1
-1@@ -2669,7 +2669,7 @@ enum ref_transaction_error refs_verify_refnames_available(struct ref_store *refsif (!iter) { iter =refs_ref_iterator_begin (refs, dirname.buf,NULL ,0 ,DO_FOR_EACH_INCLUDE_BROKEN ); - }else if (ref_iterator_seek (iter, dirname.buf) <0 ) { + }else if (ref_iterator_seek (iter, dirname.buf,1 ) <0 ) { goto cleanup; }
This is quite unreadable, as you have no idea what 1 could mean. Let's make this a unsigned flags variable instead so that we can provide meaningful names.
Quoted 14 lines
refs.h +2
-2@@ -1300,20 +1300,25 @@ struct ref_iterator *refs_ref_iterator_begin(int ref_iterator_advance (struct ref_iterator *ref_iterator);/* -* Seek the iterator to the first reference with the given prefix. -* The prefix is matched as a literal string, without regard for path +* Seek the iterator to the first reference matching the given seek string. +* The seek string is matched as a literal string, without regard for path * separators. If prefix is NULL or the empty string, seek the iterator to the * first reference again.
refs.h struct ref_iterator *refs_ref_iterator_begin( · line 1312
Quoted 10 lines
refs.h +5
-4* -* This function is expected to behave as if a new ref iterator with the same -* prefix had been created, but allows reuse of iterators and thus may allow -* the backend to optimize. Parameters other than the prefix that have been -* passed when creating the iterator will remain unchanged. +* When set_prefix is true, this function behaves as if a new ref iterator +* with the same prefix had been created, setting the prefix for subsequent +* iteration. When set_prefix is false, the iterator simply seeks to the +* specified reference without changing the existing prefix, allowing +* iteration to start from that specific reference.
I think we should detangle this paragraph a bit.
This function is expected to behave as if a new ref iterator has been created, but allows reuse of it
Re: [PATCH 3/4] refs: selectively set prefix in the seek functions
Quoted 35 lines · Patrick Steinhardt
Quoted 14 lines · Karthik Nayak
The ref iterator exposes a
ref_iterator_seek()function. The name suggests that this would seek the iterator to a specific reference in some ways similar to howfseek()works for the filesystem.However, the function actually sets the prefix for refs iteration. So further iteration would only yield references which match the particular prefix. This is a bit confusing.
Let's add a '
set_prefix' field to the function, which when set, will set the prefix for the iteration in-line with the existing behavior. But when the 'set_prefix' field is not set, the reference backends will simply seek to the specified reference without setting prefix. This allows users to start iteration from a specific reference.Quoted 8 lines
In the packed and reftable backend, since references are available in a sorted list, the changes are simply setting the prefix if needed. The changes on the files-backend are a little more involved, since the files backend uses the 'ref-cache' mechanism. We move out the existing logic within
cache_ref_iterator_seek()tocache_ref_iterator_set_prefix()which is called whenset_prefixis set. We then parse the provided seek string and set the required levels and their indexes to ensure that seeking is possible.That solution makes sense.
Quoted 9 lines
refs.c +0
-1@@ -2669,7 +2669,7 @@ enum ref_transaction_error refs_verify_refnames_available(struct ref_store *refsif (!iter) { iter =refs_ref_iterator_begin (refs, dirname.buf,NULL ,0 ,DO_FOR_EACH_INCLUDE_BROKEN ); - }else if (ref_iterator_seek (iter, dirname.buf) <0 ) {
Quoted 9 lines
refs.c enum ref_transaction_error refs_verify_refnames_available(struct ref_store *refs · line 2675
Quoted 4 lines
refs.c +1
-0+ }else if (ref_iterator_seek (iter, dirname.buf,1 ) <0 ) { goto cleanup; }This is quite unreadable, as you have no idea what
1could mean. Let's make this aunsigned flagsvariable instead so that we can provide meaningful names.
Yeah, that would make a lot more sense. Will amend.
Quoted 27 lines
Quoted 14 lines
refs.h +2
-2@@ -1300,20 +1300,25 @@ struct ref_iterator *refs_ref_iterator_begin(int ref_iterator_advance (struct ref_iterator *ref_iterator);/* -* Seek the iterator to the first reference with the given prefix. -* The prefix is matched as a literal string, without regard for path +* Seek the iterator to the first reference matching the given seek string. +* The seek string is matched as a literal string, without regard for path * separators. If prefix is NULL or the empty string, seek the iterator to the * first reference again. refs.h struct ref_iterator *refs_ref_iterator_begin( · line 1312
Quoted 10 lines
refs.h +5
-4* -* This function is expected to behave as if a new ref iterator with the same -* prefix had been created, but allows reuse of iterators and thus may allow -* the backend to optimize. Parameters other than the prefix that have been -* passed when creating the iterator will remain unchanged. +* When set_prefix is true, this function behaves as if a new ref iterator +* with the same prefix had been created, setting the prefix for subsequent +* iteration. When set_prefix is false, the iterator simply seeks to the +* specified reference without changing the existing prefix, allowing +* iteration to start from that specific reference. I think we should detangle this paragraph a bit.
Quoted 2 lines
This function is expected to behave as if a new ref iterator has been created, but allows reuse of it
Sure, let me add this in. Thanks!
PATCH 4/4 for-each-ref: introduce a '--skip-until' option
In reply to Karthik NayakThe git-for-each-ref(1) command is used to iterate over references present in a repository. In large repositories with millions of references, it would be optimal to paginate this output such that we can start iteration from a given reference. This would avoid having to iterate over all references from the beginning each time when paginating through results.
The previous commit added 'seek' functionality to the reference backends. Utilize this and expose a '--skip-until' option in 'git-for-each-ref(1)'. When used, the reference iteration seeks to the first matching reference and iterates from there onward.
This enables efficient pagination workflows like:
git for-each-ref --count=100
git for-each-ref --count=100 --skip-until=refs/heads/branch-100
git for-each-ref --count=100 --skip-until=refs/heads/branch-200--- Documentation/git-for-each-ref.adoc | 6 +- builtin/for-each-ref.c | 5 + ref-filter.c | 57 ++++++++---- ref-filter.h | 1 + t/t6302-for-each-ref-filter.sh | 180 ++++++++++++++++++++++++++++++++++++ 5 files changed, 230 insertions(+), 19 deletions(-)
Documentation/git-for-each-ref.adoc +5 -1
@@ -14,7 +14,7 @@ SYNOPSIS [--points-at=<object>] [--merged[=<object>]] [--no-merged[=<object>]] [--contains[=<object>]] [--no-contains[=<object>]] - [--exclude=<pattern> ...] + [--exclude=<pattern> ...] [--skip-until=<pattern>] DESCRIPTION -----------
@@ -108,6 +108,10 @@ TAB %(refname)`. --include-root-refs:: List root refs (HEAD and pseudorefs) apart from regular refs. +--skip-until:: + Skip references up to the specified pattern. Cannot be used with + general pattern matching. + FIELD NAMES -----------
builtin/for-each-ref.c +5 -0
@@ -13,6 +13,7 @@ static char const * const for_each_ref_usage[] = {N_ ("git for-each-ref [--points-at <object>]" ),N_ ("git for-each-ref [--merged [<commit>]] [--no-merged [<commit>]]" ),N_ ("git for-each-ref [--contains [<commit>]] [--no-contains [<commit>]]" ), +N_ ("git for-each-ref [--skip-until <pattern>]" ), NULL };
@@ -44,6 +45,7 @@ int cmd_for_each_ref(int argc,OPT_GROUP ("" ),OPT_INTEGER (0 ,"count" , &format.array_opts.max_count,N_ ("show only <n> matched refs" )),OPT_STRING (0 ,"format" , &format.format,N_ ("format" ),N_ ("format to use for the output" )), +OPT_STRING (0 ,"skip-until" , &filter.seek,N_ ("skip-until" ),N_ ("skip references until" )),OPT__COLOR (&format.use_color,N_ ("respect format colors" )),OPT_REF_FILTER_EXCLUDE (&filter), OPT_REF_SORT(&sorting_options),
@@ -100,6 +102,9 @@ int cmd_for_each_ref(int argc, filter.name_patterns = argv; } +if (filter.seek && filter.name_patterns && filter.name_patterns[0 ]) +die (_ ("cannot use --skip-until with patterns" )); +if (include_root_refs) flags |=FILTER_REFS_ROOT_REFS |FILTER_REFS_DETACHED_HEAD ;
ref-filter.c +39 -18
102 lines not on this page · raw message
ref-filter.h +1 -0
@@ -64,6 +64,7 @@ struct ref_array {struct ref_filter {const char **name_patterns; +const char *seek;struct strvec exclude;struct oid_array points_at;struct commit_list *with_commit;
t/t6302-for-each-ref-filter.sh +180 -0
184 lines not on this page · raw message
Signature · 1 line
2.49.0
Re: [PATCH 4/4] for-each-ref: introduce a '--skip-until' option
Quoted 14 lines · Karthik Nayak
Documentation/git-for-each-ref.adoc +1
-1@@ -14,7 +14,7 @@ SYNOPSIS [--points-at=<object>] [--merged[=<object>]] [--no-merged[=<object>]] [--contains[=<object>]] [--no-contains[=<object>]] - [--exclude=<pattern> ...] + [--exclude=<pattern> ...] [--skip-until=<pattern>] DESCRIPTION -----------
Documentation/git-for-each-ref.adoc TAB %(refname)`. · line 117
Quoted 10 lines
Documentation/git-for-each-ref.adoc +4
-0@@ -108,6 +108,10 @@ TAB %(refname)`. --include-root-refs:: List root refs (HEAD and pseudorefs) apart from regular refs. +--skip-until:: + Skip references up to the specified pattern. Cannot be used with + general pattern matching. + FIELD NAMES -----------
Is it "up to and including the specified pattern" or "up to but excluding the specified pattern"? It would help to make it very explicit whether the pattern itself would be yielded or not.
Quoted 20 lines
ref-filter.c +2
-4@@ -2714,20 +2716,28 @@ static int for_each_fullref_in_pattern(struct ref_filter *filter,* so just return everything and let the caller * sort it out. */ -return refs_for_each_fullref_in (get_main_ref_store (the_repository), -"" ,NULL , cb, cb_data); + goto non_prefix_iter; }if (!filter->name_patterns[0 ]) {/* no patterns; we have to look at everything */ -return refs_for_each_fullref_in (get_main_ref_store (the_repository), -"" , filter->exclude.v, cb, cb_data); + goto non_prefix_iter; }
ref-filter.c static int for_each_fullref_in_pattern(struct ref_filter *filter, · line 2736
Quoted 10 lines
ref-filter.c +6
-0return refs_for_each_fullref_in_prefixes (get_main_ref_store (the_repository),NULL , filter->name_patterns, filter->exclude.v, cb, cb_data); + +non_prefix_iter: + iter =refs_ref_iterator_begin (get_main_ref_store (the_repository),"" , +NULL ,0 , flags); +if (filter->seek) + ret =ref_iterator_seek (iter, filter->seek,0 );
Hm, this interface is somewhat weird now, as we have a split in what the prefix-string meeks when creating the iterator and seeking it. I think we should align those two functions.
ref-filter.c static int do_filter_refs(struct ref_filter *filter, unsigned int type, each_ref · line 3212
Quoted 11 lines
ref-filter.c +4
-0+if (ret) +return ret; + +return do_for_each_ref_iterator (iter, cb, cb_data); }/* @@ -3200,6 +3210,8 @@ static int do_filter_refs(struct ref_filter *filter, unsigned int type, each_refif (!filter->kind)die ("filter_refs: invalid type" );else {
The if branch now needs to be updated to have curly braces, as well.
Patrick
Re: [PATCH 4/4] for-each-ref: introduce a '--skip-until' option
Quoted 18 lines · Patrick Steinhardt
Quoted 17 lines · Karthik Nayak
Documentation/git-for-each-ref.adoc +1
-1@@ -14,7 +14,7 @@ SYNOPSIS [--points-at=<object>] [--merged[=<object>]] [--no-merged[=<object>]] [--contains[=<object>]] [--no-contains[=<object>]] - [--exclude=<pattern> ...] + [--exclude=<pattern> ...] [--skip-until=<pattern>] DESCRIPTION -----------@@ -108,6 +108,10 @@ TAB %(refname)`. --include-root-refs:: List root refs (HEAD and pseudorefs) apart from regular refs.
Quoted 12 lines
Documentation/git-for-each-ref.adoc TAB %(refname)`. · line 117
Quoted 7 lines
Documentation/git-for-each-ref.adoc +4
-0+--skip-until:: + Skip references up to the specified pattern. Cannot be used with + general pattern matching. + FIELD NAMES -----------Is it "up to and including the specified pattern" or "up to but excluding the specified pattern"? It would help to make it very explicit whether the pattern itself would be yielded or not.
It is "up to and including", will modify to make this more clearer.
Quoted 20 lines
Quoted 20 lines
ref-filter.c +2
-4@@ -2714,20 +2716,28 @@ static int for_each_fullref_in_pattern(struct ref_filter *filter,* so just return everything and let the caller * sort it out. */ -return refs_for_each_fullref_in (get_main_ref_store (the_repository), -"" ,NULL , cb, cb_data); + goto non_prefix_iter; }if (!filter->name_patterns[0 ]) {/* no patterns; we have to look at everything */ -return refs_for_each_fullref_in (get_main_ref_store (the_repository), -"" , filter->exclude.v, cb, cb_data); + goto non_prefix_iter; }
Quoted 15 lines
ref-filter.c static int for_each_fullref_in_pattern(struct ref_filter *filter, · line 2736
Quoted 10 lines
ref-filter.c +6
-0return refs_for_each_fullref_in_prefixes (get_main_ref_store (the_repository),NULL , filter->name_patterns, filter->exclude.v, cb, cb_data); + +non_prefix_iter: + iter =refs_ref_iterator_begin (get_main_ref_store (the_repository),"" , +NULL ,0 , flags); +if (filter->seek) + ret =ref_iterator_seek (iter, filter->seek,0 );Hm, this interface is somewhat weird now, as we have a split in what the prefix-string meeks when creating the iterator and seeking it. I think we should align those two functions.
The refs_ref_iterator_begin() takes in a prefix string, which sets the prefix.
The ref_iterator_seek() takes in a seek string, but a flag allows it also set the prefix.
I think this is okay since the naming matches what it does.
The alternate would be to refs_ref_iterator_begin() to also take in a seek string with a flag to also set the prefix. What do you think? I'm okay either ways.
Quoted 14 lines
ref-filter.c static int do_filter_refs(struct ref_filter *filter, unsigned int type, each_ref · line 3212
Quoted 11 lines
ref-filter.c +4
-0+if (ret) +return ret; + +return do_for_each_ref_iterator (iter, cb, cb_data); }/* @@ -3200,6 +3210,8 @@ static int do_filter_refs(struct ref_filter *filter, unsigned int type, each_refif (!filter->kind)die ("filter_refs: invalid type" );else {The
ifbranch now needs to be updated to have curly braces, as well.
Quoted 1 line
Patrick
Yes, will add.
Thanks for the review!
Re: [PATCH 4/4] for-each-ref: introduce a '--skip-until' option
Quoted 41 lines · Karthik Nayak
Quoted 21 lines · Patrick Steinhardt
Quoted 20 lines · Karthik Nayak
ref-filter.c +2
-4@@ -2714,20 +2716,28 @@ static int for_each_fullref_in_pattern(struct ref_filter *filter,* so just return everything and let the caller * sort it out. */ -return refs_for_each_fullref_in (get_main_ref_store (the_repository), -"" ,NULL , cb, cb_data); + goto non_prefix_iter; }if (!filter->name_patterns[0 ]) {/* no patterns; we have to look at everything */ -return refs_for_each_fullref_in (get_main_ref_store (the_repository), -"" , filter->exclude.v, cb, cb_data); + goto non_prefix_iter; }Quoted 15 lines
ref-filter.c static int for_each_fullref_in_pattern(struct ref_filter *filter, · line 2736
Quoted 10 lines
ref-filter.c +6
-0return refs_for_each_fullref_in_prefixes (get_main_ref_store (the_repository),NULL , filter->name_patterns, filter->exclude.v, cb, cb_data); + +non_prefix_iter: + iter =refs_ref_iterator_begin (get_main_ref_store (the_repository),"" , +NULL ,0 , flags); +if (filter->seek) + ret =ref_iterator_seek (iter, filter->seek,0 );Hm, this interface is somewhat weird now, as we have a split in what the prefix-string meeks when creating the iterator and seeking it. I think we should align those two functions.
The
refs_ref_iterator_begin()takes in aprefixstring, which sets the prefix.
Quoted 8 lines
The
ref_iterator_seek()takes in aseekstring, but a flag allows it also set the prefix.I think this is okay since the naming matches what it does.
The alternate would be to
refs_ref_iterator_begin()to also take in aseekstring with a flag to also set the prefix. What do you think? I'm okay either ways.
I just think that the interface is a bit confusing. It's weird that the needle that we're seeking for may or may not be used to update internal state, and that this is inconsistent with the similar fields that you pass to the iterator when creating it. So after seeking it sometimes acts like you have created a new iterator with the needle, sometimes it does not becaus we retain internal state. This kind of inconsistency invites mistakes.
Patrick
Quoted 4 lines · Karthik Nayak
The
git-for-each-ref(1)command is used to iterate over references present in a repository. In large repositories with millions of references, it would be optimal to paginate this output such that we can start iteration from a given reference.
I haven't looked at the patches, but should the end-user's mental model of the process be like this?
- We have a native order in which references are sorted and that is what "git for-each-ref" without "--sort" option gives them.
- They can use the "--skip-until" option to seek in the above order and start iterating in the middle.
- If they give "--sort", the set of refs to be shown would not change; skipping is done in the native order and then the remainder is given sorted.
Please make sure that the documentation is clear enough to avoid a misunderstanding that this feature would kick in after we grab all refs and sort them. If it worked that way, it would allow us to say "going from newer to older, but skipping the most recent ones that were touched within a week", which would have been nice, but that is not what we are doing with this feature---I think it is OK but we need to be clear about it in the documentation.
Quoted 3 lines
This series adds a '--skip-until' option in 'git-for-each-ref(1)'. When used, the reference iteration seeks to the first matching reference and iterates from there onward.
OK. Even the filesystem backed ones we internall sort after doing readdir() loop, so this is feasible. Nice.
Quoted 3 lines
Initally I was also planning to cleanup all the
refs_for_each...()functions in 'refs.h' by simply using the iterator, but this bloated the series. So I've left that for another day.
OK.
Quoted 20 lines · Junio C Hamano
Quoted 4 lines · Karthik Nayak
The
git-for-each-ref(1)command is used to iterate over references present in a repository. In large repositories with millions of references, it would be optimal to paginate this output such that we can start iteration from a given reference.I haven't looked at the patches, but should the end-user's mental model of the process be like this?
- We have a native order in which references are sorted and that is what "git for-each-ref" without "--sort" option gives them.
- They can use the "--skip-until" option to seek in the above order and start iterating in the middle.
- If they give "--sort", the set of refs to be shown would not change; skipping is done in the native order and then the remainder is given sorted.
Quoted 8 lines
Please make sure that the documentation is clear enough to avoid a misunderstanding that this feature would kick in after we grab all refs and sort them. If it worked that way, it would allow us to say "going from newer to older, but skipping the most recent ones that were touched within a week", which would have been nice, but that is not what we are doing with this feature---I think it is OK but we need to be clear about it in the documentation.
I totally didn't consider '--sort'. I do agree that we should document that behavior if we tend to keep it as is. I wonder if it is worthwhile to even prevent the two from being used together. I find the whole "we skip before sorting" to be very confusing.
Quoted 7 lines
Quoted 3 lines
This series adds a '--skip-until' option in 'git-for-each-ref(1)'. When used, the reference iteration seeks to the first matching reference and iterates from there onward.
OK. Even the filesystem backed ones we internall sort after doing
readdir()loop, so this is feasible. Nice.
Yup. We have 'sort_ref_dir()' to sort each directory parsed.
Quoted 5 lines
Quoted 3 lines
Initally I was also planning to cleanup all the
refs_for_each...()functions in 'refs.h' by simply using the iterator, but this bloated the series. So I've left that for another day.OK.
Offtopic. After applying this topic, I asked clang-format if it wants to change anything.
$ git clang-format --diff $(git merge-base HEAD master)
The result was disasterous. Can "clang-format --diff" mode be taught a bit more focused to avoid touching existing entries in the same array (in this case opts[] that has tons of options for the "git for-each-ref" command), when only one new entry was added, I wonder?
Also I am not impressed by the change it made to the code that is commented out (in refs.h).
Line wrapping it did to refs_ref_iterator_begin() is an improvement, but those to ref_iterator_seek() and do_for_each_ref_iterator() are unnecessary (both of these were more readble in the original).
Even though I found its output better for Toon's "last-modified" changes, I am not impressed by what clang-format suggested for this series.
builtin/for-each-ref.c +26 -17
@@ -8,7 +8,7 @@#include "strbuf.h" #include "strvec.h" -static char const *const for_each_ref_usage[] = { +static char const *const for_each_ref_usage[] = {N_ ("git for-each-ref [<options>] [<pattern>]" ),N_ ("git for-each-ref [--points-at <object>]" ),N_ ("git for-each-ref [--merged [<commit>]] [--no-merged [<commit>]]" ),
@@ -33,32 +33,41 @@ int cmd_for_each_ref(int argc,struct option opts[] = {OPT_BIT ('s' ,"shell" , &format.quote_style,N_ ("quote placeholders suitably for shells" ),QUOTE_SHELL ), -OPT_BIT ('p' ,"perl" , &format.quote_style, +OPT_BIT ('p' ,"perl" , &format.quote_style,N_ ("quote placeholders suitably for perl" ),QUOTE_PERL ), -OPT_BIT (0 ,"python" , &format.quote_style, -N_ ("quote placeholders suitably for python" ),QUOTE_PYTHON ), -OPT_BIT (0 ,"tcl" , &format.quote_style, +OPT_BIT (0 ,"python" , &format.quote_style, +N_ ("quote placeholders suitably for python" ), +QUOTE_PYTHON ), +OPT_BIT (0 ,"tcl" , &format.quote_style,N_ ("quote placeholders suitably for Tcl" ),QUOTE_TCL ), -OPT_BOOL (0 ,"omit-empty" , &format.array_opts.omit_empty, -N_ ("do not output a newline after empty formatted refs" )), +OPT_BOOL (0 ,"omit-empty" , &format.array_opts.omit_empty, +N_ ("do not output a newline after empty formatted refs" )),OPT_GROUP ("" ), -OPT_INTEGER (0 ,"count" , &format.array_opts.max_count,N_ ("show only <n> matched refs" )), -OPT_STRING (0 ,"format" , &format.format,N_ ("format" ),N_ ("format to use for the output" )), -OPT_STRING (0 ,"skip-until" , &filter.seek,N_ ("skip-until" ),N_ ("skip references until" )), +OPT_INTEGER (0 ,"count" , &format.array_opts.max_count, +N_ ("show only <n> matched refs" )), +OPT_STRING (0 ,"format" , &format.format,N_ ("format" ), +N_ ("format to use for the output" )), +OPT_STRING (0 ,"skip-until" , &filter.seek,N_ ("skip-until" ), +N_ ("skip references until" )),OPT__COLOR (&format.use_color,N_ ("respect format colors" )),OPT_REF_FILTER_EXCLUDE (&filter),OPT_REF_SORT (&sorting_options), -OPT_CALLBACK (0 ,"points-at" , &filter.points_at, -N_ ("object" ),N_ ("print only refs which points at the given object" ), +OPT_CALLBACK (0 ,"points-at" , &filter.points_at,N_ ("object" ), +N_ ("print only refs which points at the given object" ), parse_opt_object_name),OPT_MERGED (&filter,N_ ("print only refs that are merged" )),OPT_NO_MERGED (&filter,N_ ("print only refs that are not merged" )), -OPT_CONTAINS (&filter.with_commit,N_ ("print only refs which contain the commit" )), -OPT_NO_CONTAINS (&filter.no_commit,N_ ("print only refs which don't contain the commit" )), -OPT_BOOL (0 ,"ignore-case" , &icase,N_ ("sorting and filtering are case insensitive" )), -OPT_BOOL (0 ,"stdin" , &from_stdin,N_ ("read reference patterns from stdin" )), -OPT_BOOL (0 ,"include-root-refs" , &include_root_refs,N_ ("also include HEAD ref and pseudorefs" )), +OPT_CONTAINS (&filter.with_commit, +N_ ("print only refs which contain the commit" )), +OPT_NO_CONTAINS (&filter.no_commit, +N_ ("print only refs which don't contain the commit" )), +OPT_BOOL (0 ,"ignore-case" , &icase, +N_ ("sorting and filtering are case insensitive" )), +OPT_BOOL (0 ,"stdin" , &from_stdin, +N_ ("read reference patterns from stdin" )), +OPT_BOOL (0 ,"include-root-refs" , &include_root_refs, +N_ ("also include HEAD ref and pseudorefs" )),OPT_END (), };
refs.h +11 -10
@@ -1229,7 +1229,8 @@ int repo_migrate_ref_storage_format(struct repository *repo,* * // Access information about the current reference: * if (!(iter->flags & REF_ISSYMREF)) -* printf("%s is %s\n", iter->refname, oid_to_hex(iter->oid)); +* printf("%s is %s\n", iter->refname, +* oid_to_hex(iter->oid)); * * // If you need to peel the reference: * ref_iterator_peel(iter, &oid);
@@ -1284,10 +1285,11 @@ enum do_for_each_ref_flags {* trim that many characters off the beginning of each refname. * The output is ordered by refname. */ -struct ref_iterator *refs_ref_iterator_begin ( -struct ref_store *refs, -const char *prefix,const char **exclude_patterns, -int trim,enum do_for_each_ref_flags flags); +struct ref_iterator *refs_ref_iterator_begin (struct ref_store *refs, +const char *prefix, +const char **exclude_patterns, +int trim, +enum do_for_each_ref_flags flags);/* * Advance the iterator to the first or next item and return ITER_OK.
@@ -1317,8 +1319,8 @@ int ref_iterator_advance(struct ref_iterator *ref_iterator);* * Returns 0 on success, a negative error code otherwise. */ -int ref_iterator_seek (struct ref_iterator *ref_iterator, -const char *seek,int set_prefix); +int ref_iterator_seek (struct ref_iterator *ref_iterator,const char *seek, +int set_prefix);/* * If possible, peel the reference currently being viewed by the
@@ -1339,8 +1341,7 @@ void ref_iterator_free(struct ref_iterator *ref_iterator);* adapter between the callback style of reference iteration and the * iterator style. */ -int do_for_each_ref_iterator (struct ref_iterator *iter, -each_ref_fn fn,void *cb_data); - +int do_for_each_ref_iterator (struct ref_iterator *iter,each_ref_fn fn, +void *cb_data);#endif /* REFS_H */
refs/iterator.c +3 -4
@@ -15,8 +15,8 @@ int ref_iterator_advance(struct ref_iterator *ref_iterator)return ref_iterator->vtable->advance (ref_iterator); } -int ref_iterator_seek (struct ref_iterator *ref_iterator, -const char *seek,int set_prefix) +int ref_iterator_seek (struct ref_iterator *ref_iterator,const char *seek, +int set_prefix) {return ref_iterator->vtable->seek (ref_iterator, seek, set_prefix); }
@@ -57,8 +57,7 @@ static int empty_ref_iterator_advance(struct ref_iterator *ref_iterator UNUSED) } static int empty_ref_iterator_seek(struct ref_iterator *ref_iterator UNUSED, - const char *seek UNUSED, - int set_prefix UNUSED) + const char *seek UNUSED, int set_prefix UNUSED) {return 0 ; }
refs/ref-cache.c +2 -1
@@ -525,7 +525,8 @@ static int cache_ref_iterator_seek(struct ref_iterator *ref_iterator, level->dir = dir; level->index =-1 ; }else { -/* reduce the index so the leaf node is iterated over */ +/* reduce the index so the leaf node is iterated +* over */ if (cmp <=0 && !slash) level->index = idx -1 ;/*
Quoted 11 lines · Junio C Hamano
Offtopic. After applying this topic, I asked clang-format if it wants to change anything.
$ git clang-format --diff $(git merge-base HEAD master)The result was disasterous. Can "clang-format --diff" mode be taught a bit more focused to avoid touching existing entries in the same array (in this case opts[] that has tons of options for the "git for-each-ref" command), when only one new entry was added, I wonder?
I couldn't find any way to do something like this.
Quoted 11 lines
Also I am not impressed by the change it made to the code that is commented out (in
refs.h).Line wrapping it did to
refs_ref_iterator_begin()is an improvement, but those toref_iterator_seek()anddo_for_each_ref_iterator()are unnecessary (both of these were more readble in the original).Even though I found its output better for Toon's "last-modified" changes, I am not impressed by what clang-format suggested for this series.
It indeed looks really bad, I had a go with the new changes from 'gitster/kn/clang-format-updates'. Which seems a lot better.
However, this does show a problem with using 'RemoveBracesLLVM', where it formats the following:
if (...) {
...
...
} else {
...
}to:
if (...) {
...
...
} else
...Which isn't our style, I think we should completely drop this too, from my patch series. Let me go ahead and do that. I really want to strip out as many rules as possible to make the number of false positives 0 so we can actually start enforcing clang-format. Once we enforce it, we can slowly see what additional rules work well for us.
builtin/for-each-ref.c +6 -7
@@ -8,7 +8,7 @@#include "strbuf.h" #include "strvec.h" -static char const *const for_each_ref_usage[] = { +static char const *const for_each_ref_usage[] = {N_ ("git for-each-ref [<options>] [<pattern>]" ),N_ ("git for-each-ref [--points-at <object>]" ),N_ ("git for-each-ref [--merged [<commit>]] [--no-merged [<commit>]]" ),
@@ -33,19 +33,19 @@ int cmd_for_each_ref(int argc,struct option opts[] = {OPT_BIT ('s' ,"shell" , &format.quote_style,N_ ("quote placeholders suitably for shells" ),QUOTE_SHELL ), -OPT_BIT ('p' ,"perl" , &format.quote_style, +OPT_BIT ('p' ,"perl" , &format.quote_style,N_ ("quote placeholders suitably for perl" ),QUOTE_PERL ), -OPT_BIT (0 ,"python" , &format.quote_style, +OPT_BIT (0 ,"python" , &format.quote_style,N_ ("quote placeholders suitably for python" ),QUOTE_PYTHON ), -OPT_BIT (0 ,"tcl" , &format.quote_style, +OPT_BIT (0 ,"tcl" , &format.quote_style,N_ ("quote placeholders suitably for Tcl" ),QUOTE_TCL ), -OPT_BOOL (0 ,"omit-empty" , &format.array_opts.omit_empty, -N_ ("do not output a newline after empty formatted refs" )), +OPT_BOOL (0 ,"omit-empty" , &format.array_opts.omit_empty, +N_ ("do not output a newline after empty formatted refs" )),OPT_GROUP ("" ), - OPT_INTEGER( 0 , "count", &format.array_opts.max_count, N_("show
only <n> matched refs")),
- OPT_STRING( 0 , "format", &format.format, N_("format"), N_("format
to use for the output")),
- OPT_STRING( 0 , "skip-until", &filter.seek, N_("skip-until"),N_("skip references until")),
+ OPT_INTEGER(0, "count", &format.array_opts.max_count, N_("show only
<n> matched refs")),
+ OPT_STRING(0, "format", &format.format, N_("format"), N_("format to
use for the output")),
+ OPT_STRING(0, "skip-until", &filter.seek, N_("skip-until"),N_("skip references until")),
OPT__COLOR(&format.use_color, N_("respect format colors")), OPT_REF_FILTER_EXCLUDE(&filter), OPT_REF_SORT(&sorting_options),
refs.c +0 -0
@@ -2669,23 +2669,21 @@ enum ref_transaction_error
refs_verify_refnames_available(struct ref_store *refs
if (!iter) {
iter = refs_ref_iterator_begin(refs, dirname.buf, NULL, 0,
DO_FOR_EACH_INCLUDE_BROKEN);
- } else if (ref_iterator_seek(iter, dirname.buf, 1) < 0) {
- goto cleanup;
- }
+ else if (ref_iterator_seek(iter, dirname.buf, 1) < 0) goto cleanup;
- while ((ok = ref_iterator_advance(iter)) == ITER_OK) {
- if (skip &&
- string_list_has_string(skip, iter->refname))
- continue;
+ while ((ok = ref_iterator_advance(iter)) == ITER_OK) {
+ if (skip &&
+ string_list_has_string(skip, iter->refname))
+ continue;
- if (transaction && ref_transaction_maybe_set_rejected(
- transaction, *update_idx,
- REF_TRANSACTION_ERROR_NAME_CONFLICT))
- continue;
+ if (transaction && ref_transaction_maybe_set_rejected(
+ transaction, *update_idx,
+ REF_TRANSACTION_ERROR_NAME_CONFLICT))
+ continue;
- strbuf_addf(err, _("'%s' exists; cannot create '%s'"),
- iter->refname, refname);
- goto cleanup;
+ strbuf_addf(err, _("'%s' exists; cannot create '%s'"),
+ iter->refname, refname);
+ goto cleanup;
}
if (ok != ITER_DONE)
refs.h +3 -4
@@ -1285,9 +1285,9 @@ enum do_for_each_ref_flags {* The output is ordered by refname. */ struct ref_iterator *refs_ref_iterator_begin ( -struct ref_store *refs, -const char *prefix,const char **exclude_patterns, -int trim,enum do_for_each_ref_flags flags); +struct ref_store *refs, +const char *prefix,const char **exclude_patterns, +int trim,enum do_for_each_ref_flags flags);/* * Advance the iterator to the first or next item and return ITER_OK.
@@ -1342,5 +1342,4 @@ void ref_iterator_free(struct ref_iterator *ref_iterator);int do_for_each_ref_iterator (struct ref_iterator *iter,each_ref_fn fn,void *cb_data); -#endif /* REFS_H */
Quoted 49 lines · Karthik Nayak
Quoted 11 lines · Junio C Hamano
Offtopic. After applying this topic, I asked clang-format if it wants to change anything.
$ git clang-format --diff $(git merge-base HEAD master)The result was disasterous. Can "clang-format --diff" mode be taught a bit more focused to avoid touching existing entries in the same array (in this case opts[] that has tons of options for the "git for-each-ref" command), when only one new entry was added, I wonder?
I couldn't find any way to do something like this.
Quoted 11 lines
Also I am not impressed by the change it made to the code that is commented out (in
refs.h).Line wrapping it did to
refs_ref_iterator_begin()is an improvement, but those toref_iterator_seek()anddo_for_each_ref_iterator()are unnecessary (both of these were more readble in the original).Even though I found its output better for Toon's "last-modified" changes, I am not impressed by what clang-format suggested for this series.
It indeed looks really bad, I had a go with the new changes from 'gitster/kn/clang-format-updates'. Which seems a lot better.
However, this does show a problem with using 'RemoveBracesLLVM', where it formats the following:
if (...) { ... ... } else { ... }to:
if (...) { ... ... } else ...
Quoted 6 lines
Which isn't our style, I think we should completely drop this too, from my patch series. Let me go ahead and do that. I really want to strip out as many rules as possible to make the number of false positives 0 so we can actually start enforcing clang-format. Once we enforce it, we can slowly see what additional rules work well for us.
I did some more testing here, and it seems like this was because this particular instance was more like
if (...) {
...
} else {
...
}Where both the clauses had single line statement, but we only modified the 'else' part of the clause in this patch series, so clang-format, only suggested removing the braces from the 'else' clause.
So all is good here, I think we can go ahead with the 'gitster/kn/clang-format-updates' and merge it to 'next'. Sorry for being the false positive, I thought I missed testing a particular case and the series.
Hi Karthik
Quoted 5 lines · Karthik Nayak
This enables efficient pagination workflows like:
git for-each-ref --count=100 git for-each-ref --count=100 --skip-until=refs/heads/branch-100 git for-each-ref --count=100 --skip-until=refs/heads/branch-200
Doesn't that require you to know the name of the ref after the last one returned by the previous batch? If the use case here is pagination then being able to provide a numeric offset might be a better fit. For example
git for-each-ref --count=100 --start=200
would show refs 200 to 300
Thanks
Phillip
Phillip Wood <[email protected]> writes:
Hello Phillip,
Quoted 4 lines
Hi Karthik
Quoted 1 line · Karthik Nayak
Quoted 9 lines
Quoted 4 lines
This enables efficient pagination workflows like:
git for-each-ref --count=100 git for-each-ref --count=100 --skip-until=refs/heads/branch-100 git for-each-ref --count=100 --skip-until=refs/heads/branch-200Doesn't that require you to know the name of the ref after the last one returned by the previous batch? If the use case here is pagination then being able to provide a numeric offset might be a better fit. For example
It does require that you know the last ref from the previous batch.
The reason for picking a reference offset is mostly for performance optimization. Our reference backends are built with prefix matching in mind, in short they do a binary search through the reference namespace to find the required prefix. By using a reference offset we can utilize this binary search mechanism to arrive at offset.
Using a count offset would require iteration to reach the desired offset (basically a O(N) operation). This wouldn't really matter in repositories with ~10^3 refs, but in larger repositories with around ~10^6 refs this starts to make a large difference.
Quoted 7 lines
git for-each-ref --count=100 --start=200would show refs 200 to 300
Thanks
Phillip
Quoted 26 lines · Karthik Nayak
Phillip Wood <[email protected]> writes:
Hello Phillip,
Quoted 4 lines
Hi Karthik
Quoted 1 line · Karthik Nayak
Quoted 9 lines
Quoted 4 lines
This enables efficient pagination workflows like:
git for-each-ref --count=100 git for-each-ref --count=100 --skip-until=refs/heads/branch-100 git for-each-ref --count=100 --skip-until=refs/heads/branch-200Doesn't that require you to know the name of the ref after the last one returned by the previous batch? If the use case here is pagination then being able to provide a numeric offset might be a better fit. For example
It does require that you know the last ref from the previous batch.
The reason for picking a reference offset is mostly for performance optimization. Our reference backends are built with prefix matching in mind, in short they do a binary search through the reference namespace to find the required prefix. By using a reference offset we can utilize this binary search mechanism to arrive at offset.
Quoted 4 lines
Using a count offset would require iteration to reach the desired offset (basically a O(N) operation). This wouldn't really matter in repositories with ~10^3 refs, but in larger repositories with around ~10^6 refs this starts to make a large difference.
Even more importantly though, a numeric offset would be invalidated by a concurrent write in case that write ends up inserting a ref in the range of commits you intend to skip now.
Patrick
Quoted 3 lines · Patrick Steinhardt
Even more importantly though, a numeric offset would be invalidated by a concurrent write in case that write ends up inserting a ref in the range of commits you intend to skip now.
That argument cuts both ways, no? You have shown up to some ref which you remember in the last cycle, and then while you are planning to formulate another query with --skip-until naming that ref, somebody removes that ref, then what happens? Or somebody inserts a new ref that sorts earlier than the ref you stopped at the last time.
Quoted 10 lines · Junio C Hamano
Quoted 3 lines · Patrick Steinhardt
Even more importantly though, a numeric offset would be invalidated by a concurrent write in case that write ends up inserting a ref in the range of commits you intend to skip now.
That argument cuts both ways, no? You have shown up to some ref which you remember in the last cycle, and then while you are planning to formulate another query with --skip-until naming that ref, somebody removes that ref, then what happens?
This ref was already yielded, and it wouldn't and shouldn't be yielded on the next page. This works as expected with the proposal, as --skip-until does not care whether the value itself actually exists.
Quoted 2 lines
Or somebody inserts a new ref that sorts earlier than the ref you stopped at the last time.
It wouldn't and shouldn't be shown. When I have already yielded all refs up to refs/heads/something, I don't expect to see any ref that sorts before refs/heads/something on the next page.
Patrick
Quoted 7 lines · Patrick Steinhardt
Quoted 6 lines · Junio C Hamano
Quoted 3 lines · Patrick Steinhardt
Even more importantly though, a numeric offset would be invalidated by a concurrent write in case that write ends up inserting a ref in the range of commits you intend to skip now.
Quoted 8 lines
Quoted 4 lines
That argument cuts both ways, no? You have shown up to some ref which you remember in the last cycle, and then while you are planning to formulate another query with --skip-until naming that ref, somebody removes that ref, then what happens?
This ref was already yielded, and it wouldn't and shouldn't be yielded on the next page. This works as expected with the proposal, as
--skip-untildoes not care whether the value itself actually exists.
The current version of the series will include the reference provided to '--skip-until', if it exists. But your latter statement still holds, if the reference doesn't exist, it will still work by finding the next reference in the default sort order.
Quoted 7 lines
Quoted 2 lines
Or somebody inserts a new ref that sorts earlier than the ref you stopped at the last time.
It wouldn't and shouldn't be shown. When I have already yielded all refs up to refs/heads/something, I don't expect to see any ref that sorts before refs/heads/something on the next page.
Yeah, this was my thought too. Another way to think of this is that in a cursor based approach, a particular reference is guarateed never to occur again, even with modifications to the repository made between requests. However in a count based approach this doesn't stand.
Quoted 1 line
Patrick
v2 cover for-each-ref: introduce seeking functionality via '--skip-until'
In reply to Karthik NayakThe git-for-each-ref(1) command is used to iterate over references present in a repository. In large repositories with millions of references, it would be optimal to paginate this output such that we can start iteration from a given reference. This would avoid having to iterate over all references from the beginning each time when paginating through results.
This series adds a '--skip-until' option in 'git-for-each-ref(1)'. When used, the reference iteration seeks to the first matching reference and iterates from there onward.
This enables efficient pagination workflows like:
git for-each-ref --count=100
git for-each-ref --count=100 --skip-until=refs/heads/branch-100
git for-each-ref --count=100 --skip-until=refs/heads/branch-200To add this functionality, we expose the ref_iterator outside the 'refs/' namespace and modify the ref_iterator_seek() to actually seek to a given reference and only set the prefix when the set_prefix field is set.
On the reftable and packed backend, the changes are simple. But since the files backend uses 'ref-cache' for reference handling, the changes there are a little more involved, since we need to setup the right levels and the indexing.
Initially I was also planning to cleanup all the refs_for_each...() functions in 'refs.h' by simply using the iterator, but this bloated the series. So I've left that for another day.
Changes in v2:
- Modify 'ref_iterator_seek()' to take in flags instead of a
'set_prefix' variable. This improves readability, where users would
use the 'REF_ITERATOR_SEEK_SET_PREFIX' instead of simply passing '1'.
- When the set prefix flag isn't usage, reset any previously set prefix.
This ensures that the internal prefix state is always reset whenever
we seek and unifies the behavior between 'ref_iterator_seek' and
'ref_iterator_begin'.
- Don't allow '--skip-until' to be run with '--sort', since the seeking
always takes place before any sorting and this can be confusing.
- Some styling fixes:
- Remove extra newline
- Skip braces around single lined if...else clause
- Add braces around 'if' clause
- Fix indentation
- Link to v1: https://lore.kernel.org/git/20250701-306-git-for-each-ref-pagination-v1-0-4f0ae7c0688f@gmail.com/
--- Documentation/git-for-each-ref.adoc | 6 +- builtin/for-each-ref.c | 8 ++ ref-filter.c | 61 ++++++++---- ref-filter.h | 1 + refs.c | 6 +- refs.h | 158 ++++++++++++++++++++++++++++++ refs/debug.c | 7 +- refs/files-backend.c | 7 +- refs/iterator.c | 26 ++--- refs/packed-backend.c | 17 ++-- refs/ref-cache.c | 99 +++++++++++++++---- refs/ref-cache.h | 7 -- refs/refs-internal.h | 152 ++--------------------------- refs/reftable-backend.c | 21 ++-- t/t6302-for-each-ref-filter.sh | 188 ++++++++++++++++++++++++++++++++++++ 15 files changed, 538 insertions(+), 226 deletions(-)
Karthik Nayak (4):
refs: expose ref_iterator via 'refs.h'
ref-cache: remove unused function 'find_ref_entry()'
refs: selectively set prefix in the seek functions
for-each-ref: introduce a '--skip-until' option
Range-diff versus v1:
1: a20ef2c841 ! 1: d19553c365 refs: expose ref_iterator via 'refs.h'
@@ refs.h: int repo_migrate_ref_storage_format(struct repository *repo,
+ * The output is ordered by refname.
+ */
+struct ref_iterator *refs_ref_iterator_begin(
-+ struct ref_store *refs,
-+ const char *prefix, const char **exclude_patterns,
-+ int trim, enum do_for_each_ref_flags flags);
++ struct ref_store *refs,
++ const char *prefix, const char **exclude_patterns,
++ int trim, enum do_for_each_ref_flags flags);
+
+/*
+ * Advance the iterator to the first or next item and return ITER_OK.
@@ refs.h: int repo_migrate_ref_storage_format(struct repository *repo,
+ */
+int do_for_each_ref_iterator(struct ref_iterator *iter,
+ each_ref_fn fn, void *cb_data);
-+
+
#endif /* REFS_H */
2: 96f3e6eb05 = 2: 1d3936132b ref-cache: remove unused function 'find_ref_entry()'
3: fa19b53a37 ! 3: aab2011494 refs: selectively set prefix in the seek functions
@@ Commit message
further iteration would only yield references which match the particular
prefix. This is a bit confusing.
- Let's add a 'set_prefix' field to the function, which when set, will set
- the prefix for the iteration in-line with the existing behavior. But
- when the 'set_prefix' field is not set, the reference backends will
- simply seek to the specified reference without setting prefix. This
- allows users to start iteration from a specific reference.
+ Let's add a 'flags' field to the function, which when set with the
+ 'REF_ITERATOR_SEEK_SET_PREFIX' flag, will set the prefix for the
+ iteration in-line with the existing behavior. Otherwise, the reference
+ backends will simply seek to the specified reference and clears any
+ previously set prefix. This allows users to start iteration from a
+ specific reference.
In the packed and reftable backend, since references are available in a
sorted list, the changes are simply setting the prefix if needed. The
changes on the files-backend are a little more involved, since the files
backend uses the 'ref-cache' mechanism. We move out the existing logic
within `cache_ref_iterator_seek()` to `cache_ref_iterator_set_prefix()`
- which is called when `set_prefix` is set. We then parse the provided
- seek string and set the required levels and their indexes to ensure that
- seeking is possible.
+ which is called when the 'REF_ITERATOR_SEEK_SET_PREFIX' flag is set. We
+ then parse the provided seek string and set the required levels and
+ their indexes to ensure that seeking is possible.
+ Helped-by: Patrick Steinhardt <[email protected]>
Signed-off-by: Karthik Nayak <[email protected]>
## refs.c ##
@@ refs.c: enum ref_transaction_error refs_verify_refnames_available(struct ref_store *refs
- if (!iter) {
+ if (!initial_transaction) {
+ int ok;
+
+- if (!iter) {
++ if (!iter)
iter = refs_ref_iterator_begin(refs, dirname.buf, NULL, 0,
DO_FOR_EACH_INCLUDE_BROKEN);
- } else if (ref_iterator_seek(iter, dirname.buf) < 0) {
-+ } else if (ref_iterator_seek(iter, dirname.buf, 1) < 0) {
++ else if (ref_iterator_seek(iter, dirname.buf,
++ REF_ITERATOR_SEEK_SET_PREFIX) < 0)
goto cleanup;
- }
+- }
+ while ((ok = ref_iterator_advance(iter)) == ITER_OK) {
+ if (skip &&
## refs.h ##
@@ refs.h: struct ref_iterator *refs_ref_iterator_begin(
+ */
int ref_iterator_advance(struct ref_iterator *ref_iterator);
++enum ref_iterator_seek_flag {
++ /*
++ * Also set the seek pattern as a prefix for iteration. This ensures
++ * that only references which match the prefix are yielded.
++ */
++ REF_ITERATOR_SEEK_SET_PREFIX = (1 << 0),
++};
++
/*
- * Seek the iterator to the first reference with the given prefix.
- * The prefix is matched as a literal string, without regard for path
+- * separators. If prefix is NULL or the empty string, seek the iterator to the
+ * Seek the iterator to the first reference matching the given seek string.
+ * The seek string is matched as a literal string, without regard for path
- * separators. If prefix is NULL or the empty string, seek the iterator to the
++ * separators. If seek is NULL or the empty string, seek the iterator to the
* first reference again.
*
- * This function is expected to behave as if a new ref iterator with the same
- * prefix had been created, but allows reuse of iterators and thus may allow
- * the backend to optimize. Parameters other than the prefix that have been
- * passed when creating the iterator will remain unchanged.
-+ * When set_prefix is true, this function behaves as if a new ref iterator
-+ * with the same prefix had been created, setting the prefix for subsequent
-+ * iteration. When set_prefix is false, the iterator simply seeks to the
-+ * specified reference without changing the existing prefix, allowing
-+ * iteration to start from that specific reference.
++ * This function is expected to behave as if a new ref iterator has been
++ * created, but allows reuse of existing iterators for optimization.
+ *
-+ * This function allows reuse of iterators and thus may allow the backend
-+ * to optimize. Parameters other than the prefix that have been passed when
-+ * creating the iterator will remain unchanged.
++ * When the REF_ITERATOR_SEEK_SET_PREFIX flag is set, the iterator's prefix is
++ * updated to match the seek string, affecting all subsequent iterations. If
++ * not, the iterator seeks to the specified reference and clears any previously
++ * set prefix.
*
* Returns 0 on success, a negative error code otherwise.
*/
- int ref_iterator_seek(struct ref_iterator *ref_iterator,
+-int ref_iterator_seek(struct ref_iterator *ref_iterator,
- const char *prefix);
-+ const char *seek, int set_prefix);
++int ref_iterator_seek(struct ref_iterator *ref_iterator, const char *seek,
++ unsigned int flags);
/*
* If possible, peel the reference currently being viewed by the
@@ refs/debug.c: static int debug_ref_iterator_advance(struct ref_iterator *ref_ite
static int debug_ref_iterator_seek(struct ref_iterator *ref_iterator,
- const char *prefix)
-+ const char *seek, int set_prefix)
++ const char *seek, unsigned int flags)
{
struct debug_ref_iterator *diter =
(struct debug_ref_iterator *)ref_iterator;
- int res = diter->iter->vtable->seek(diter->iter, prefix);
- trace_printf_key(&trace_refs, "iterator_seek: %s: %d\n", prefix ? prefix : "", res);
-+ int res = diter->iter->vtable->seek(diter->iter, seek, set_prefix);
-+ trace_printf_key(&trace_refs, "iterator_seek: %s set_prefix: %d: %d\n",
-+ seek ? seek : "", set_prefix, res);
++ int res = diter->iter->vtable->seek(diter->iter, seek, flags);
++ trace_printf_key(&trace_refs, "iterator_seek: %s flags: %d: %d\n",
++ seek ? seek : "", flags, res);
return res;
}
@@ refs/files-backend.c: static int files_ref_iterator_advance(struct ref_iterator
static int files_ref_iterator_seek(struct ref_iterator *ref_iterator,
- const char *prefix)
-+ const char *seek, int set_prefix)
++ const char *seek, unsigned int flags)
{
struct files_ref_iterator *iter =
(struct files_ref_iterator *)ref_iterator;
- return ref_iterator_seek(iter->iter0, prefix);
-+ return ref_iterator_seek(iter->iter0, seek, set_prefix);
++ return ref_iterator_seek(iter->iter0, seek, flags);
}
static int files_ref_iterator_peel(struct ref_iterator *ref_iterator,
@@ refs/files-backend.c: static int files_reflog_iterator_advance(struct ref_iterat
static int files_reflog_iterator_seek(struct ref_iterator *ref_iterator UNUSED,
- const char *prefix UNUSED)
+ const char *seek UNUSED,
-+ int set_prefix UNUSED)
++ unsigned int flags UNUSED)
{
BUG("ref_iterator_seek() called for reflog_iterator");
}
## refs/iterator.c ##
@@ refs/iterator.c: int ref_iterator_advance(struct ref_iterator *ref_iterator)
+ return ref_iterator->vtable->advance(ref_iterator);
}
- int ref_iterator_seek(struct ref_iterator *ref_iterator,
+-int ref_iterator_seek(struct ref_iterator *ref_iterator,
- const char *prefix)
-+ const char *seek, int set_prefix)
++int ref_iterator_seek(struct ref_iterator *ref_iterator, const char *seek,
++ unsigned int flags)
{
- return ref_iterator->vtable->seek(ref_iterator, prefix);
-+ return ref_iterator->vtable->seek(ref_iterator, seek, set_prefix);
++ return ref_iterator->vtable->seek(ref_iterator, seek, flags);
}
int ref_iterator_peel(struct ref_iterator *ref_iterator,
@@ refs/iterator.c: static int empty_ref_iterator_advance(struct ref_iterator *ref_
static int empty_ref_iterator_seek(struct ref_iterator *ref_iterator UNUSED,
- const char *prefix UNUSED)
+ const char *seek UNUSED,
-+ int set_prefix UNUSED)
++ unsigned int flags UNUSED)
{
return 0;
}
@@ refs/iterator.c: static int merge_ref_iterator_advance(struct ref_iterator *ref_
static int merge_ref_iterator_seek(struct ref_iterator *ref_iterator,
- const char *prefix)
-+ const char *seek, int set_prefix)
++ const char *seek, unsigned int flags)
{
struct merge_ref_iterator *iter =
(struct merge_ref_iterator *)ref_iterator;
@@ refs/iterator.c: static int merge_ref_iterator_seek(struct ref_iterator *ref_ite
iter->iter1 = iter->iter1_owned;
- ret = ref_iterator_seek(iter->iter0, prefix);
-+ ret = ref_iterator_seek(iter->iter0, seek, set_prefix);
++ ret = ref_iterator_seek(iter->iter0, seek, flags);
if (ret < 0)
return ret;
- ret = ref_iterator_seek(iter->iter1, prefix);
-+ ret = ref_iterator_seek(iter->iter1, seek, set_prefix);
++ ret = ref_iterator_seek(iter->iter1, seek, flags);
if (ret < 0)
return ret;
@@ refs/iterator.c: static int prefix_ref_iterator_advance(struct ref_iterator *ref
static int prefix_ref_iterator_seek(struct ref_iterator *ref_iterator,
- const char *prefix)
-+ const char *seek, int set_prefix)
++ const char *seek, unsigned int flags)
{
struct prefix_ref_iterator *iter =
(struct prefix_ref_iterator *)ref_iterator;
@@ refs/iterator.c: static int prefix_ref_iterator_advance(struct ref_iterator *ref
- iter->prefix = xstrdup_or_null(prefix);
- return ref_iterator_seek(iter->iter0, prefix);
+
-+ if (set_prefix) {
++ if (flags & REF_ITERATOR_SEEK_SET_PREFIX) {
+ free(iter->prefix);
+ iter->prefix = xstrdup_or_null(seek);
+ }
-+ return ref_iterator_seek(iter->iter0, seek, set_prefix);
++ return ref_iterator_seek(iter->iter0, seek, flags);
}
static int prefix_ref_iterator_peel(struct ref_iterator *ref_iterator,
@@ refs/packed-backend.c: static int packed_ref_iterator_advance(struct ref_iterato
static int packed_ref_iterator_seek(struct ref_iterator *ref_iterator,
- const char *prefix)
-+ const char *seek, int set_prefix)
++ const char *seek, unsigned int flags)
{
struct packed_ref_iterator *iter =
(struct packed_ref_iterator *)ref_iterator;
@@ refs/packed-backend.c: static int packed_ref_iterator_advance(struct ref_iterato
- free(iter->prefix);
- iter->prefix = xstrdup_or_null(prefix);
-+ if (set_prefix) {
-+ free(iter->prefix);
++ /* Unset any previously set prefix */
++ FREE_AND_NULL(iter->prefix);
++
++ if (flags & REF_ITERATOR_SEEK_SET_PREFIX)
+ iter->prefix = xstrdup_or_null(seek);
-+ }
+
iter->pos = start;
iter->eof = iter->snapshot->eof;
@@ refs/packed-backend.c: static struct ref_iterator *packed_ref_iterator_begin(
iter->flags = flags;
- if (packed_ref_iterator_seek(&iter->base, prefix) < 0) {
-+ if (packed_ref_iterator_seek(&iter->base, prefix, 1) < 0) {
++ if (packed_ref_iterator_seek(&iter->base, prefix,
++ REF_ITERATOR_SEEK_SET_PREFIX) < 0) {
ref_iterator_free(&iter->base);
return NULL;
}
@@ refs/ref-cache.c: static int cache_ref_iterator_seek(struct ref_iterator *ref_it
}
+static int cache_ref_iterator_seek(struct ref_iterator *ref_iterator,
-+ const char *seek, int set_prefix)
++ const char *seek, unsigned int flags)
+{
+ struct cache_ref_iterator *iter =
+ (struct cache_ref_iterator *)ref_iterator;
+
-+ if (set_prefix) {
++ if (flags & REF_ITERATOR_SEEK_SET_PREFIX) {
+ return cache_ref_iterator_set_prefix(iter, seek);
+ } else if (seek && *seek) {
+ struct cache_ref_iterator_level *level;
@@ refs/ref-cache.c: static int cache_ref_iterator_seek(struct ref_iterator *ref_it
+ level->index = -1;
+ level->dir = dir;
+
++ /* Unset any previously set prefix */
++ FREE_AND_NULL(iter->prefix);
++
+ /*
+ * Breakdown the provided seek path and assign the correct
+ * indexing to each level as needed.
@@ refs/ref-cache.c: struct ref_iterator *cache_ref_iterator_begin(struct ref_cache
iter->prime_dir = prime_dir;
- if (cache_ref_iterator_seek(&iter->base, prefix) < 0) {
-+ if (cache_ref_iterator_seek(&iter->base, prefix, 1) < 0) {
++ if (cache_ref_iterator_seek(&iter->base, prefix,
++ REF_ITERATOR_SEEK_SET_PREFIX) < 0) {
ref_iterator_free(&iter->base);
return NULL;
}
@@ refs/refs-internal.h: void base_ref_iterator_init(struct ref_iterator *iter,
*/
typedef int ref_iterator_seek_fn(struct ref_iterator *ref_iterator,
- const char *prefix);
-+ const char *seek, int set_prefix);
++ const char *seek, unsigned int flags);
/*
* Peels the current ref, returning 0 for success or -1 for failure.
@@ refs/reftable-backend.c: static int reftable_ref_iterator_advance(struct ref_ite
static int reftable_ref_iterator_seek(struct ref_iterator *ref_iterator,
- const char *prefix)
-+ const char *seek, int set_prefix)
++ const char *seek, unsigned int flags)
{
struct reftable_ref_iterator *iter =
(struct reftable_ref_iterator *)ref_iterator;
@@ refs/reftable-backend.c: static int reftable_ref_iterator_advance(struct ref_ite
- iter->prefix = xstrdup_or_null(prefix);
- iter->prefix_len = prefix ? strlen(prefix) : 0;
- iter->err = reftable_iterator_seek_ref(&iter->iter, prefix);
-+ if (set_prefix) {
-+ free(iter->prefix);
++ /* Unset any previously set prefix */
++ FREE_AND_NULL(iter->prefix);
++ iter->prefix_len = 0;
++
++ if (flags & REF_ITERATOR_SEEK_SET_PREFIX) {
+ iter->prefix = xstrdup_or_null(seek);
+ iter->prefix_len = seek ? strlen(seek) : 0;
+ }
@@ refs/reftable-backend.c: static struct reftable_ref_iterator *ref_iterator_for_s
goto done;
- ret = reftable_ref_iterator_seek(&iter->base, prefix);
-+ ret = reftable_ref_iterator_seek(&iter->base, prefix, 1);
++ ret = reftable_ref_iterator_seek(&iter->base, prefix,
++ REF_ITERATOR_SEEK_SET_PREFIX);
if (ret)
goto done;
@@ refs/reftable-backend.c: static int reftable_reflog_iterator_advance(struct ref_
static int reftable_reflog_iterator_seek(struct ref_iterator *ref_iterator UNUSED,
- const char *prefix UNUSED)
+ const char *seek UNUSED,
-+ int set_prefix UNUSED)
++ unsigned int flags UNUSED)
{
BUG("reftable reflog iterator cannot be seeked");
return -1;
4: 0cfc879a93 ! 4: dc6ffa9002 for-each-ref: introduce a '--skip-until' option
@@ Documentation/git-for-each-ref.adoc: TAB %(refname)`.
List root refs (HEAD and pseudorefs) apart from regular refs.
+--skip-until::
-+ Skip references up to the specified pattern. Cannot be used with
-+ general pattern matching.
++ Skip references up to but excluding the specified pattern. Cannot be used
++ with general pattern matching or custom sort options.
+
FIELD NAMES
-----------
@@ builtin/for-each-ref.c: int cmd_for_each_ref(int argc,
OPT__COLOR(&format.use_color, N_("respect format colors")),
OPT_REF_FILTER_EXCLUDE(&filter),
OPT_REF_SORT(&sorting_options),
+@@ builtin/for-each-ref.c: int cmd_for_each_ref(int argc,
+ if (verify_ref_format(&format))
+ usage_with_options(for_each_ref_usage, opts);
+
++ if (filter.seek && sorting_options.nr > 1)
++ die(_("cannot use --skip-until custom sort options"));
++
+ sorting = ref_sorting_options(&sorting_options);
+ ref_sorting_set_sort_flags_all(sorting, REF_SORTING_ICASE, icase);
+ filter.ignore_case = icase;
@@ builtin/for-each-ref.c: int cmd_for_each_ref(int argc,
filter.name_patterns = argv;
}
@@ ref-filter.c: static int for_each_fullref_in_pattern(struct ref_filter *filter,
/*
@@ ref-filter.c: static int do_filter_refs(struct ref_filter *filter, unsigned int type, each_ref
- if (!filter->kind)
+ init_contains_cache(&filter->internal.no_contains_cache);
+
+ /* Simple per-ref filtering */
+- if (!filter->kind)
++ if (!filter->kind) {
die("filter_refs: invalid type");
- else {
+- else {
++ } else {
+ const char *prefix = NULL;
+
/*
@@ t/t6302-for-each-ref-filter.sh: test_expect_success 'validate worktree atom' '
+ cat >expect <<-\EOF &&
+ fatal: cannot use --skip-until with patterns
+ EOF
-+ test_must_fail git for-each-ref --format="%(refname)" --skip-until=refs/odd/spot refs/tags 2>actual &&
++ test_must_fail git for-each-ref --format="%(refname)" --skip-until=refs/odd/spot refs/tags 2>actual &&
++ test_cmp expect actual
++'
++
++test_expect_success 'skip until used with custom sort order' '
++ cat >expect <<-\EOF &&
++ fatal: cannot use --skip-until custom sort options
++ EOF
++ test_must_fail git for-each-ref --format="%(refname)" --skip-until=refs/odd/spot --sort=author 2>actual &&
+ test_cmp expect actual
+'
+base-commit: cf6f63ea6bf35173e02e18bdc6a4ba41288acff9
change-id: 20250605-306-git-for-each-ref-pagination-0ba8a29ae646
Thanks
- Karthik
PATCH v2 1/4 refs: expose `ref_iterator` via 'refs.h'
The ref_iterator is an internal structure to the 'refs/' sub-directory, which allows iteration over refs. All reference iteration is built on top of these iterators.
External clients of the 'refs' subsystem use the various 'refs_for_each...()' functions to iterate over refs. However since these are wrapper functions, each combination of functionality requires a new wrapper function. This is not feasible as the functions pile up with the increase in requirements. Expose the internal reference iterator, so advanced users can mix and match options as needed.
--- refs.h | 147 +++++++++++++++++++++++++++++++++++++++++++++++++++ refs/refs-internal.h | 145 +------------------------------------------------- 2 files changed, 149 insertions(+), 143 deletions(-)
refs.h +147 -0
@@ -1190,4 +1190,151 @@ int repo_migrate_ref_storage_format(struct repository *repo,unsigned int flags, struct strbuf *err); +/* +* Reference iterators +* +* A reference iterator encapsulates the state of an in-progress +* iteration over references. Create an instance of `struct +* ref_iterator` via one of the functions in this module. +* +* A freshly-created ref_iterator doesn't yet point at a reference. To +* advance the iterator, call ref_iterator_advance(). If successful, +* this sets the iterator's refname, oid, and flags fields to describe +* the next reference and returns ITER_OK. The data pointed at by +* refname and oid belong to the iterator; if you want to retain them +* after calling ref_iterator_advance() again or calling +* ref_iterator_free(), you must make a copy. When the iteration has +* been exhausted, ref_iterator_advance() releases any resources +* associated with the iteration, frees the ref_iterator object, and +* returns ITER_DONE. If you want to abort the iteration early, call +* ref_iterator_free(), which also frees the ref_iterator object and +* any associated resources. If there was an internal error advancing +* to the next entry, ref_iterator_advance() aborts the iteration, +* frees the ref_iterator, and returns ITER_ERROR. +* +* The reference currently being looked at can be peeled by calling +* ref_iterator_peel(). This function is often faster than peel_ref(), +* so it should be preferred when iterating over references. +* +* Putting it all together, a typical iteration looks like this: +* +* int ok; +* struct ref_iterator *iter = ...; +* +* while ((ok = ref_iterator_advance(iter)) == ITER_OK) { +* if (want_to_stop_iteration()) { +* ok = ITER_DONE; +* break; +* } +* +* // Access information about the current reference: +* if (!(iter->flags & REF_ISSYMREF)) +* printf("%s is %s\n", iter->refname, oid_to_hex(iter->oid)); +* +* // If you need to peel the reference: +* ref_iterator_peel(iter, &oid); +* } +* +* if (ok != ITER_DONE) +* handle_error(); +* ref_iterator_free(iter); +*/ +struct ref_iterator ; + +/* +* These flags are passed to refs_ref_iterator_begin() (and do_for_each_ref(), +* which feeds it). +*/ +enum do_for_each_ref_flags { +/* +* Include broken references in a do_for_each_ref*() iteration, which +* would normally be omitted. This includes both refs that point to +* missing objects (a true repository corruption), ones with illegal +* names (which we prefer not to expose to callers), as well as +* dangling symbolic refs (i.e., those that point to a non-existent +* ref; this is not a corruption, but as they have no valid oid, we +* omit them from normal iteration results). +*/ +DO_FOR_EACH_INCLUDE_BROKEN = (1 <<0 ), + +/* +* Only include per-worktree refs in a do_for_each_ref*() iteration. +* Normally this will be used with a files ref_store, since that's +* where all reference backends will presumably store their +* per-worktree refs. +*/ +DO_FOR_EACH_PER_WORKTREE_ONLY = (1 <<1 ), + +/* +* Omit dangling symrefs from output; this only has an effect with +* INCLUDE_BROKEN, since they are otherwise not included at all. +*/ +DO_FOR_EACH_OMIT_DANGLING_SYMREFS = (1 <<2 ), + +/* +* Include root refs i.e. HEAD and pseudorefs along with the regular +* refs. +*/ +DO_FOR_EACH_INCLUDE_ROOT_REFS = (1 <<3 ), +}; + +/* +* Return an iterator that goes over each reference in `refs` for +* which the refname begins with prefix. If trim is non-zero, then +* trim that many characters off the beginning of each refname. +* The output is ordered by refname. +*/ +struct ref_iterator *refs_ref_iterator_begin ( +struct ref_store *refs, +const char *prefix,const char **exclude_patterns, +int trim,enum do_for_each_ref_flags flags); + +/* +* Advance the iterator to the first or next item and return ITER_OK. +* If the iteration is exhausted, free the resources associated with +* the ref_iterator and return ITER_DONE. On errors, free the iterator +* resources and return ITER_ERROR. It is a bug to use ref_iterator or +* call this function again after it has returned ITER_DONE or +* ITER_ERROR. +*/ +int ref_iterator_advance (struct ref_iterator *ref_iterator); + +/* +* Seek the iterator to the first reference with the given prefix. +* The prefix is matched as a literal string, without regard for path +* separators. If prefix is NULL or the empty string, seek the iterator to the +* first reference again. +* +* This function is expected to behave as if a new ref iterator with the same +* prefix had been created, but allows reuse of iterators and thus may allow +* the backend to optimize. Parameters other than the prefix that have been +* passed when creating the iterator will remain unchanged. +* +* Returns 0 on success, a negative error code otherwise. +*/ +int ref_iterator_seek (struct ref_iterator *ref_iterator, +const char *prefix); + +/* +* If possible, peel the reference currently being viewed by the +* iterator. Return 0 on success. +*/ +int ref_iterator_peel (struct ref_iterator *ref_iterator, +struct object_id *peeled); + +/* Free the reference iterator and any associated resources. */ +void ref_iterator_free (struct ref_iterator *ref_iterator); + +/* +* The common backend for the for_each_*ref* functions. Call fn for +* each reference in iter. If the iterator itself ever returns +* ITER_ERROR, return -1. If fn ever returns a non-zero value, stop +* the iteration and return that value. Otherwise, return 0. In any +* case, free the iterator when done. This function is basically an +* adapter between the callback style of reference iteration and the +* iterator style. +*/ +int do_for_each_ref_iterator (struct ref_iterator *iter, +each_ref_fn fn,void *cb_data); +#endif /* REFS_H */
refs/refs-internal.h +2 -143
169 lines not on this page · raw message
Signature · 1 line
2.49.0
PATCH v2 2/4 ref-cache: remove unused function 'find_ref_entry()'
In reply to Karthik NayakThe 'find_ref_entry' function is no longer used, so remove it.
--- refs/ref-cache.c | 14 -------------- refs/ref-cache.h | 7 ------- 2 files changed, 21 deletions(-)
refs/ref-cache.c +0 -14
@@ -194,20 +194,6 @@ static struct ref_dir *find_containing_dir(struct ref_dir *dir,return dir; } -struct ref_entry *find_ref_entry (struct ref_dir *dir,const char *refname) -{ -int entry_index; -struct ref_entry *entry; - dir =find_containing_dir (dir, refname); -if (!dir) -return NULL ; - entry_index =search_ref_dir (dir, refname,strlen (refname)); -if (entry_index ==-1 ) -return NULL ; - entry = dir->entries[entry_index]; -return (entry->flag &REF_DIR ) ?NULL : entry; -} -/* * Emit a warning and return true iff ref1 and ref2 have the same name * and the same oid. Die if they have the same name but different
refs/ref-cache.h +0 -7
@@ -201,13 +201,6 @@ void free_ref_cache(struct ref_cache *cache);*/ void add_entry_to_dir (struct ref_dir *dir,struct ref_entry *entry); -/* -* Find the value entry with the given name in dir, sorting ref_dirs -* and recursing into subdirectories as necessary. If the name is not -* found or it corresponds to a directory entry, return NULL. -*/ -struct ref_entry *find_ref_entry (struct ref_dir *dir,const char *refname); -/* * Start iterating over references in `cache`. If `prefix` is * specified, only include references whose names start with that
Signature · 1 line
2.49.0
PATCH v2 3/4 refs: selectively set prefix in the seek functions
In reply to Karthik NayakThe ref iterator exposes a ref_iterator_seek() function. The name suggests that this would seek the iterator to a specific reference in some ways similar to how fseek() works for the filesystem.
However, the function actually sets the prefix for refs iteration. So further iteration would only yield references which match the particular prefix. This is a bit confusing.
Let's add a 'flags' field to the function, which when set with the 'REF_ITERATOR_SEEK_SET_PREFIX' flag, will set the prefix for the iteration in-line with the existing behavior. Otherwise, the reference backends will simply seek to the specified reference and clears any previously set prefix. This allows users to start iteration from a specific reference.
In the packed and reftable backend, since references are available in a sorted list, the changes are simply setting the prefix if needed. The changes on the files-backend are a little more involved, since the files backend uses the 'ref-cache' mechanism. We move out the existing logic within cache_ref_iterator_seek() to cache_ref_iterator_set_prefix() which is called when the 'REF_ITERATOR_SEEK_SET_PREFIX' flag is set. We then parse the provided seek string and set the required levels and their indexes to ensure that seeking is possible.
--- refs.c | 6 ++-- refs.h | 29 +++++++++++------ refs/debug.c | 7 ++-- refs/files-backend.c | 7 ++-- refs/iterator.c | 26 ++++++++------- refs/packed-backend.c | 17 ++++++---- refs/ref-cache.c | 85 ++++++++++++++++++++++++++++++++++++++++++++++--- refs/refs-internal.h | 7 ++-- refs/reftable-backend.c | 21 ++++++++---- 9 files changed, 155 insertions(+), 50 deletions(-)
refs.c +3 -3
@@ -2666,12 +2666,12 @@ enum ref_transaction_error refs_verify_refnames_available(struct ref_store *refsif (!initial_transaction) {int ok; -if (!iter) { +if (!iter) iter =refs_ref_iterator_begin (refs, dirname.buf,NULL ,0 ,DO_FOR_EACH_INCLUDE_BROKEN ); - }else if (ref_iterator_seek (iter, dirname.buf) <0 ) { +else if (ref_iterator_seek (iter, dirname.buf, +REF_ITERATOR_SEEK_SET_PREFIX ) <0 ) goto cleanup; - }while ((ok =ref_iterator_advance (iter)) ==ITER_OK ) { if (skip &&
refs.h +20 -9
@@ -1299,21 +1299,32 @@ struct ref_iterator *refs_ref_iterator_begin(*/ int ref_iterator_advance (struct ref_iterator *ref_iterator); +enum ref_iterator_seek_flag { +/* +* Also set the seek pattern as a prefix for iteration. This ensures +* that only references which match the prefix are yielded. +*/ +REF_ITERATOR_SEEK_SET_PREFIX = (1 <<0 ), +}; +/* -* Seek the iterator to the first reference with the given prefix. -* The prefix is matched as a literal string, without regard for path -* separators. If prefix is NULL or the empty string, seek the iterator to the +* Seek the iterator to the first reference matching the given seek string. +* The seek string is matched as a literal string, without regard for path +* separators. If seek is NULL or the empty string, seek the iterator to the * first reference again. * -* This function is expected to behave as if a new ref iterator with the same -* prefix had been created, but allows reuse of iterators and thus may allow -* the backend to optimize. Parameters other than the prefix that have been -* passed when creating the iterator will remain unchanged. +* This function is expected to behave as if a new ref iterator has been +* created, but allows reuse of existing iterators for optimization. +* +* When the REF_ITERATOR_SEEK_SET_PREFIX flag is set, the iterator's prefix is +* updated to match the seek string, affecting all subsequent iterations. If +* not, the iterator seeks to the specified reference and clears any previously +* set prefix. * * Returns 0 on success, a negative error code otherwise. */ -int ref_iterator_seek (struct ref_iterator *ref_iterator, -const char *prefix); +int ref_iterator_seek (struct ref_iterator *ref_iterator,const char *seek, +unsigned int flags);/* * If possible, peel the reference currently being viewed by the
refs/debug.c +4 -3
@@ -170,12 +170,13 @@ static int debug_ref_iterator_advance(struct ref_iterator *ref_iterator) }static int debug_ref_iterator_seek (struct ref_iterator *ref_iterator, -const char *prefix) +const char *seek,unsigned int flags) {struct debug_ref_iterator *diter = (struct debug_ref_iterator *)ref_iterator; -int res = diter->iter->vtable->seek (diter->iter, prefix); -trace_printf_key (&trace_refs,"iterator_seek: %s: %d\n" , prefix ? prefix :"" , res); +int res = diter->iter->vtable->seek (diter->iter, seek, flags); +trace_printf_key (&trace_refs,"iterator_seek: %s flags: %d: %d\n" , + seek ? seek :"" , flags, res);return res; }
refs/files-backend.c +4 -3
@@ -929,11 +929,11 @@ static int files_ref_iterator_advance(struct ref_iterator *ref_iterator) }static int files_ref_iterator_seek (struct ref_iterator *ref_iterator, -const char *prefix) +const char *seek,unsigned int flags) {struct files_ref_iterator *iter = (struct files_ref_iterator *)ref_iterator; -return ref_iterator_seek (iter->iter0, prefix); +return ref_iterator_seek (iter->iter0, seek, flags); } static int files_ref_iterator_peel(struct ref_iterator *ref_iterator,
@@ -2316,7 +2316,8 @@ static int files_reflog_iterator_advance(struct ref_iterator *ref_iterator) } static int files_reflog_iterator_seek(struct ref_iterator *ref_iterator UNUSED, - const char *prefix UNUSED) + const char *seek UNUSED, + unsigned int flags UNUSED) {BUG ("ref_iterator_seek() called for reflog_iterator" ); }
refs/iterator.c +15 -11
@@ -15,10 +15,10 @@ int ref_iterator_advance(struct ref_iterator *ref_iterator)return ref_iterator->vtable->advance (ref_iterator); } -int ref_iterator_seek (struct ref_iterator *ref_iterator, -const char *prefix) +int ref_iterator_seek (struct ref_iterator *ref_iterator,const char *seek, +unsigned int flags) { -return ref_iterator->vtable->seek (ref_iterator, prefix); +return ref_iterator->vtable->seek (ref_iterator, seek, flags); } int ref_iterator_peel(struct ref_iterator *ref_iterator,
@@ -57,7 +57,8 @@ static int empty_ref_iterator_advance(struct ref_iterator *ref_iterator UNUSED) } static int empty_ref_iterator_seek(struct ref_iterator *ref_iterator UNUSED, - const char *prefix UNUSED) + const char *seek UNUSED, + unsigned int flags UNUSED) {return 0 ; }
@@ -224,7 +225,7 @@ static int merge_ref_iterator_advance(struct ref_iterator *ref_iterator) }static int merge_ref_iterator_seek (struct ref_iterator *ref_iterator, -const char *prefix) +const char *seek,unsigned int flags) {struct merge_ref_iterator *iter = (struct merge_ref_iterator *)ref_iterator;
@@ -234,11 +235,11 @@ static int merge_ref_iterator_seek(struct ref_iterator *ref_iterator, iter->iter0 = iter->iter0_owned; iter->iter1 = iter->iter1_owned; - ret =ref_iterator_seek (iter->iter0, prefix); + ret =ref_iterator_seek (iter->iter0, seek, flags);if (ret <0 )return ret; - ret =ref_iterator_seek (iter->iter1, prefix); + ret =ref_iterator_seek (iter->iter1, seek, flags);if (ret <0 )return ret;
@@ -407,13 +408,16 @@ static int prefix_ref_iterator_advance(struct ref_iterator *ref_iterator) }static int prefix_ref_iterator_seek (struct ref_iterator *ref_iterator, -const char *prefix) +const char *seek,unsigned int flags) {struct prefix_ref_iterator *iter = (struct prefix_ref_iterator *)ref_iterator; -free (iter->prefix); - iter->prefix =xstrdup_or_null (prefix); -return ref_iterator_seek (iter->iter0, prefix); + +if (flags &REF_ITERATOR_SEEK_SET_PREFIX ) { +free (iter->prefix); + iter->prefix =xstrdup_or_null (seek); + } +return ref_iterator_seek (iter->iter0, seek, flags); } static int prefix_ref_iterator_peel(struct ref_iterator *ref_iterator,
refs/packed-backend.c +11 -6
@@ -1004,19 +1004,23 @@ static int packed_ref_iterator_advance(struct ref_iterator *ref_iterator) }static int packed_ref_iterator_seek (struct ref_iterator *ref_iterator, -const char *prefix) +const char *seek,unsigned int flags) {struct packed_ref_iterator *iter = (struct packed_ref_iterator *)ref_iterator;const char *start; -if (prefix && *prefix) - start =find_reference_location (iter->snapshot, prefix,0 ); +if (seek && *seek) + start =find_reference_location (iter->snapshot, seek,0 );else start = iter->snapshot->start; -free (iter->prefix); - iter->prefix =xstrdup_or_null (prefix); +/* Unset any previously set prefix */ +FREE_AND_NULL (iter->prefix); + +if (flags &REF_ITERATOR_SEEK_SET_PREFIX ) + iter->prefix =xstrdup_or_null (seek); + iter->pos = start; iter->eof = iter->snapshot->eof;
@@ -1194,7 +1198,8 @@ static struct ref_iterator *packed_ref_iterator_begin( iter->repo = ref_store->repo; iter->flags = flags; -if (packed_ref_iterator_seek (&iter->base, prefix) <0 ) { +if (packed_ref_iterator_seek (&iter->base, prefix, +REF_ITERATOR_SEEK_SET_PREFIX ) <0 ) {ref_iterator_free (&iter->base);return NULL ; }
refs/ref-cache.c +80 -5
104 lines not on this page · raw message
refs/refs-internal.h +4 -3
@@ -353,11 +353,12 @@ void base_ref_iterator_init(struct ref_iterator *iter,typedef int ref_iterator_advance_fn (struct ref_iterator *ref_iterator);/* -* Seek the iterator to the first reference matching the given prefix. Should -* behave the same as if a new iterator was created with the same prefix. +* Seek the iterator to the first matching reference. If set_prefix is set, +* it would behave the same as if a new iterator was created with the same +* prefix. */ typedef int ref_iterator_seek_fn (struct ref_iterator *ref_iterator, -const char *prefix); +const char *seek,unsigned int flags);/* * Peels the current ref, returning 0 for success or -1 for failure.
refs/reftable-backend.c +14 -7
@@ -719,15 +719,20 @@ static int reftable_ref_iterator_advance(struct ref_iterator *ref_iterator) }static int reftable_ref_iterator_seek (struct ref_iterator *ref_iterator, -const char *prefix) +const char *seek,unsigned int flags) {struct reftable_ref_iterator *iter = (struct reftable_ref_iterator *)ref_iterator; -free (iter->prefix); - iter->prefix =xstrdup_or_null (prefix); - iter->prefix_len = prefix ?strlen (prefix) :0 ; - iter->err =reftable_iterator_seek_ref (&iter->iter, prefix); +/* Unset any previously set prefix */ +FREE_AND_NULL (iter->prefix); + iter->prefix_len =0 ; + +if (flags &REF_ITERATOR_SEEK_SET_PREFIX ) { + iter->prefix =xstrdup_or_null (seek); + iter->prefix_len = seek ?strlen (seek) :0 ; + } + iter->err =reftable_iterator_seek_ref (&iter->iter, seek);return iter->err; }
@@ -839,7 +844,8 @@ static struct reftable_ref_iterator *ref_iterator_for_stack(struct reftable_ref_if (ret) goto done; - ret =reftable_ref_iterator_seek (&iter->base, prefix); + ret =reftable_ref_iterator_seek (&iter->base, prefix, +REF_ITERATOR_SEEK_SET_PREFIX );if (ret) goto done;
@@ -2042,7 +2048,8 @@ static int reftable_reflog_iterator_advance(struct ref_iterator *ref_iterator) } static int reftable_reflog_iterator_seek(struct ref_iterator *ref_iterator UNUSED, - const char *prefix UNUSED) + const char *seek UNUSED, + unsigned int flags UNUSED) {BUG ("reftable reflog iterator cannot be seeked" );return -1 ;
Signature · 1 line
2.49.0
PATCH v2 4/4 for-each-ref: introduce a '--skip-until' option
In reply to Karthik NayakThe git-for-each-ref(1) command is used to iterate over references present in a repository. In large repositories with millions of references, it would be optimal to paginate this output such that we can start iteration from a given reference. This would avoid having to iterate over all references from the beginning each time when paginating through results.
The previous commit added 'seek' functionality to the reference backends. Utilize this and expose a '--skip-until' option in 'git-for-each-ref(1)'. When used, the reference iteration seeks to the first matching reference and iterates from there onward.
This enables efficient pagination workflows like:
git for-each-ref --count=100
git for-each-ref --count=100 --skip-until=refs/heads/branch-100
git for-each-ref --count=100 --skip-until=refs/heads/branch-200--- Documentation/git-for-each-ref.adoc | 6 +- builtin/for-each-ref.c | 8 ++ ref-filter.c | 61 ++++++++---- ref-filter.h | 1 + t/t6302-for-each-ref-filter.sh | 188 ++++++++++++++++++++++++++++++++++++ 5 files changed, 243 insertions(+), 21 deletions(-)
Documentation/git-for-each-ref.adoc +5 -1
@@ -14,7 +14,7 @@ SYNOPSIS [--points-at=<object>] [--merged[=<object>]] [--no-merged[=<object>]] [--contains[=<object>]] [--no-contains[=<object>]] - [--exclude=<pattern> ...] + [--exclude=<pattern> ...] [--skip-until=<pattern>] DESCRIPTION -----------
@@ -108,6 +108,10 @@ TAB %(refname)`. --include-root-refs:: List root refs (HEAD and pseudorefs) apart from regular refs. +--skip-until:: + Skip references up to but excluding the specified pattern. Cannot be used + with general pattern matching or custom sort options. + FIELD NAMES -----------
builtin/for-each-ref.c +8 -0
@@ -13,6 +13,7 @@ static char const * const for_each_ref_usage[] = {N_ ("git for-each-ref [--points-at <object>]" ),N_ ("git for-each-ref [--merged [<commit>]] [--no-merged [<commit>]]" ),N_ ("git for-each-ref [--contains [<commit>]] [--no-contains [<commit>]]" ), +N_ ("git for-each-ref [--skip-until <pattern>]" ), NULL };
@@ -44,6 +45,7 @@ int cmd_for_each_ref(int argc,OPT_GROUP ("" ),OPT_INTEGER (0 ,"count" , &format.array_opts.max_count,N_ ("show only <n> matched refs" )),OPT_STRING (0 ,"format" , &format.format,N_ ("format" ),N_ ("format to use for the output" )), +OPT_STRING (0 ,"skip-until" , &filter.seek,N_ ("skip-until" ),N_ ("skip references until" )),OPT__COLOR (&format.use_color,N_ ("respect format colors" )),OPT_REF_FILTER_EXCLUDE (&filter), OPT_REF_SORT(&sorting_options),
@@ -79,6 +81,9 @@ int cmd_for_each_ref(int argc,if (verify_ref_format (&format))usage_with_options (for_each_ref_usage, opts); +if (filter.seek && sorting_options.nr >1 ) +die (_ ("cannot use --skip-until custom sort options" )); + sorting =ref_sorting_options (&sorting_options);ref_sorting_set_sort_flags_all (sorting,REF_SORTING_ICASE , icase); filter.ignore_case = icase;
@@ -100,6 +105,9 @@ int cmd_for_each_ref(int argc, filter.name_patterns = argv; } +if (filter.seek && filter.name_patterns && filter.name_patterns[0 ]) +die (_ ("cannot use --skip-until with patterns" )); +if (include_root_refs) flags |=FILTER_REFS_ROOT_REFS |FILTER_REFS_DETACHED_HEAD ;
ref-filter.c +41 -20
107 lines not on this page · raw message
ref-filter.h +1 -0
@@ -64,6 +64,7 @@ struct ref_array {struct ref_filter {const char **name_patterns; +const char *seek;struct strvec exclude;struct oid_array points_at;struct commit_list *with_commit;
t/t6302-for-each-ref-filter.sh +188 -0
192 lines not on this page · raw message
Signature · 1 line
2.49.0
Re: [PATCH v2 4/4] for-each-ref: introduce a '--skip-until' option
builtin/for-each-ref.c int cmd_for_each_ref(int argc, · line 85
Quoted 2 lines · Karthik Nayak
builtin/for-each-ref.c +2
-0+if (filter.seek && sorting_options.nr >1 ) +die (_ ("cannot use --skip-until custom sort options" ));
Missing "with" before "custom sort".
When I commented on the previous iteration about sorting, I didn't mean to suggest making them incompatible---it may have some use case to grab a batch out of the underlying refstore, sort refs in that batch, and then show them. But from usability's point of view, I tend to agree with this design decision. Such an unnatural batching and sorting operation is probably not worth supporting.
Re: [PATCH v2 4/4] for-each-ref: introduce a '--skip-until' option
Quoted 7 lines · Junio C Hamano
builtin/for-each-ref.c int cmd_for_each_ref(int argc, · line 85
Quoted 2 lines · Karthik Nayak
builtin/for-each-ref.c +2
-0+if (filter.seek && sorting_options.nr >1 ) +die (_ ("cannot use --skip-until custom sort options" ));Missing "with" before "custom sort".
Thanks, will fix in the next version.
Quoted 6 lines
When I commented on the previous iteration about sorting, I didn't mean to suggest making them incompatible---it may have some use case to grab a batch out of the underlying refstore, sort refs in that batch, and then show them. But from usability's point of view, I tend to agree with this design decision. Such an unnatural batching and sorting operation is probably not worth supporting.
I think so too, there might be some usecase, when that usecase is more concrete we can probably re-visit this.
For now, it is confusing and it is just easier to not support them together.
Quoted 3 lines · Karthik Nayak
This series adds a '--skip-until' option in 'git-for-each-ref(1)'. When used, the reference iteration seeks to the first matching reference and iterates from there onward.
I would have named the option --start-with. It has the advantage that it is clear whether the matched ref is included.
Signature · 3 lines
Andreas Schwab, [email protected] GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510 2552 DF73 E780 A9DA AEC1 "And now for something completely different."
Quoted 9 lines · Andreas Schwab
Quoted 3 lines · Karthik Nayak
This series adds a '--skip-until' option in 'git-for-each-ref(1)'. When used, the reference iteration seeks to the first matching reference and iterates from there onward.
I would have named the option --start-with. It has the advantage that it is clear whether the matched ref is included.
We did discuss this internally, some other names we thought of:
--skip-to
--start-after
--start-from
--seek
--skip-before
--start-at
I think I was a bit against '--start-from' and '--start-at', because they imply that the reference provided must exist.
Consider the example
$ git for-each-ref refs/heads/bar refs/heads/foo refs/heads/main
$ git for-each-ref --seek=refs/heads/cat refs/heads/foo refs/heads/main
You can see that the reference doesn't have to exist. So implying that it should can be a bit confusing.
But I'm open to changing this, if we can conclude on any flag name...
Quoted 4 lines
--
Andreas Schwab, [email protected]
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510 2552 DF73 E780 A9DA AEC1 "And now for something completely different."
Quoted 7 lines · Karthik Nayak
Consider the example
$ git for-each-ref refs/heads/bar refs/heads/foo refs/heads/main
Quoted 5 lines
$ git for-each-ref --seek=refs/heads/cat refs/heads/foo refs/heads/mainYou can see that the reference doesn't have to exist.
That is even more confusing. What is the first matching ref if none of them match? Doesn't that mean skipping _all_ refs?
Signature · 3 lines
Andreas Schwab, [email protected] GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510 2552 DF73 E780 A9DA AEC1 "And now for something completely different."
Quoted 9 lines · Andreas Schwab
Quoted 7 lines · Karthik Nayak
Consider the example
$ git for-each-ref refs/heads/bar refs/heads/foo refs/heads/main
Quoted 9 lines
Quoted 5 lines
$ git for-each-ref --seek=refs/heads/cat refs/heads/foo refs/heads/mainYou can see that the reference doesn't have to exist.
That is even more confusing. What is the first matching ref if none of them match? Doesn't that mean skipping _all_ refs?
Well the idea is it would seek to the offset where the reference would fit in.
This is to ensure that seeks to references which were deleted concurrently doesn't leave the client hanging with no results while paginating over all references.
Quoted 4 lines
--
Andreas Schwab, [email protected]
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510 2552 DF73 E780 A9DA AEC1 "And now for something completely different."
Quoted 21 lines · Karthik Nayak
Quoted 9 lines · Andreas Schwab
Quoted 7 lines · Karthik Nayak
Consider the example
$ git for-each-ref refs/heads/bar refs/heads/foo refs/heads/mainQuoted 9 lines
Quoted 5 lines
$ git for-each-ref --seek=refs/heads/cat refs/heads/foo refs/heads/mainYou can see that the reference doesn't have to exist.
That is even more confusing. What is the first matching ref if none of them match? Doesn't that mean skipping _all_ refs?
Quoted 6 lines
Well the idea is it would seek to the offset where the reference would fit in.
This is to ensure that seeks to references which were deleted concurrently doesn't leave the client hanging with no results while paginating over all references.
Then don't call it a pattern. Pattern matching is a set operation, independent of sorting. What you really have is a marker that divides the sorted list in two parts according to how the marker sorts. And that makes --start-with more descriptive and less ambiguous.
Signature · 3 lines
Andreas Schwab, [email protected] GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510 2552 DF73 E780 A9DA AEC1 "And now for something completely different."
Quoted 26 lines · Andreas Schwab
Quoted 24 lines · Karthik Nayak
Quoted 9 lines · Andreas Schwab
Quoted 7 lines · Karthik Nayak
Consider the example
$ git for-each-ref refs/heads/bar refs/heads/foo refs/heads/mainQuoted 9 lines
Quoted 5 lines
$ git for-each-ref --seek=refs/heads/cat refs/heads/foo refs/heads/mainYou can see that the reference doesn't have to exist.
That is even more confusing. What is the first matching ref if none of them match? Doesn't that mean skipping _all_ refs?
Well the idea is it would seek to the offset where the reference would fit in.
Quoted 9 lines
Quoted 3 lines
This is to ensure that seeks to references which were deleted concurrently doesn't leave the client hanging with no results while paginating over all references.
Then don't call it a pattern. Pattern matching is a set operation, independent of sorting. What you really have is a marker that divides the sorted list in two parts according to how the marker sorts. And that makes --start-with more descriptive and less ambiguous.
Fair enough. I'll change the documentation and description in the next version.
Quoted 4 lines
--
Andreas Schwab, [email protected]
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510 2552 DF73 E780 A9DA AEC1 "And now for something completely different."
Quoted 2 lines · Karthik Nayak
I think I was a bit against '--start-from' and '--start-at', because they imply that the reference provided must exist.
It also implies that if the reference does exist, that would be the first one that is shown. But I do not think you want that, as ...
Quoted 6 lines
Consider the example
$ git for-each-ref refs/heads/bar refs/heads/foo refs/heads/main
... after a paging application starts from the beginning and showed a single page of some items, it knows the "last" one it showed. That last entry may have been refs/heads/bar. The application may not have seen the next entry (i.e. refs/heads/foo). So if it has to use '--start-at=refs/heads/bar', the first entry it gets from such a request may be for refs/heads/bar again. The application needs to remember the "last" one it showed and skip that, which is a bit awkward, isn't it?
Quoted 6 lines
$ git for-each-ref --seek=refs/heads/cat refs/heads/foo refs/heads/mainYou can see that the reference doesn't have to exist. So implying that it should can be a bit confusing.
For that reason, whatever verb you pick from seek or start or skip, it would be great if the option name also made it explicit that the named one, if exists, is not shown. Conceptually, it is "skip everything that sorts before the named item, including the named item itself" that such a paging application would want, wouldn't it?
Thanks.
Quoted 16 lines · Junio C Hamano
Quoted 2 lines · Karthik Nayak
I think I was a bit against '--start-from' and '--start-at', because they imply that the reference provided must exist.
It also implies that if the reference does exist, that would be the first one that is shown. But I do not think you want that, as ...
Quoted 6 lines
Consider the example
$ git for-each-ref refs/heads/bar refs/heads/foo refs/heads/main
Quoted 9 lines
... after a paging application starts from the beginning and showed a single page of some items, it knows the "last" one it showed. That last entry may have been refs/heads/bar. The application may not have seen the next entry (i.e. refs/heads/foo). So if it has to use '--start-at=refs/heads/bar', the first entry it gets from such a request may be for refs/heads/bar again. The application needs to remember the "last" one it showed and skip that, which is a bit awkward, isn't it?
I do agree, I was modelling this after what would be the best approach within the Git codebase. But I think it would be nicer for the clients if we skip the provided reference.
Quoted 7 lines
Quoted 6 lines
$ git for-each-ref --seek=refs/heads/cat refs/heads/foo refs/heads/mainYou can see that the reference doesn't have to exist. So implying that it should can be a bit confusing.
Quoted 8 lines
For that reason, whatever verb you pick from seek or start or skip, it would be great if the option name also made it explicit that the named one, if exists, is not shown. Conceptually, it is "skip everything that sorts before the named item, including the named item itself" that such a paging application would want, wouldn't it?
Thanks.
With that I think '--start-after' sounds like the best option. I'll modify for the next version accordingly.
Thanks!
Quoted 5 lines · Karthik Nayak
Quoted 1 line · Junio C Hamano
Karthik Nayak <[email protected]> writes:
I do agree, I was modelling this after what would be the best approach within the Git codebase.
That was my fear when I asked about using a numeric offset. Patrick has made a principled argument for using a ref name rather than a numeric ofsset - I think you should build the motivation for this series around that and the documentation should explain the implications of references being added and deleted while paging them.
Quoted 2 lines
With that I think '--start-after' sounds like the best option. I'll modify for the next version accordingly.
That sounds like a good name
Thanks
Phillip
Quoted 4 lines · Phillip Wood
Quoted 3 lines · Karthik Nayak
Quoted 1 line · Junio C Hamano
Karthik Nayak <[email protected]> writes:
Quoted 9 lines
Quoted 2 lines
I do agree, I was modelling this after what would be the best approach within the Git codebase.
That was my fear when I asked about using a numeric offset. Patrick has made a principled argument for using a ref name rather than a numeric ofsset - I think you should build the motivation for this series around that and the documentation should explain the implications of references being added and deleted while paging them.
Yeah, I'll add something on those lines in the next version. I do appreciate these checks/questions.
Quoted 4 lines
Quoted 2 lines
With that I think '--start-after' sounds like the best option. I'll modify for the next version accordingly.
That sounds like a good name
Thanks!
Quoted 3 lines
Thanks
Phillip
v3 cover for-each-ref: introduce seeking functionality via '--start-after'
In reply to Karthik NayakThe git-for-each-ref(1) command is used to iterate over references present in a repository. In large repositories with millions of references, it would be optimal to paginate this output such that we can start iteration from a given reference. This would avoid having to iterate over all references from the beginning each time when paginating through results.
This series adds a '--start-after' option in 'git-for-each-ref(1)'. When used, the reference iteration seeks to first reference following the marker alphabetically. When paging, it should be noted that references may be deleted, modified or added between invocations. Output will only yield those references which follow the marker lexicographically. If the marker does not exist, output begins from the first reference that would come after it alphabetically.
This enables efficient pagination workflows like:
git for-each-ref --count=100
git for-each-ref --count=100 --start-after=refs/heads/branch-100
git for-each-ref --count=100 --start-after=refs/heads/branch-200To add this functionality, we expose the ref_iterator outside the 'refs/' namespace and modify the ref_iterator_seek() to actually seek to a given reference and only set the prefix when the set_prefix field is set.
On the reftable and packed backend, the changes are simple. But since the files backend uses 'ref-cache' for reference handling, the changes there are a little more involved, since we need to setup the right levels and the indexing.
Initially I was also planning to cleanup all the refs_for_each...() functions in 'refs.h' by simply using the iterator, but this bloated the series. So I've left that for another day.
Changes in v3:
- Change the working of the command to exclude the marker provided. With
this rename the flag to '--start-after'.
- Extend the documentation to add a note about concurrent modifications
to the reference database.
- Link to v2: https://lore.kernel.org/r/20250704-306-git-for-each-ref-pagination-v2-0-bcde14acdd81@gmail.com
Changes in v2:
- Modify 'ref_iterator_seek()' to take in flags instead of a
'set_prefix' variable. This improves readability, where users would
use the 'REF_ITERATOR_SEEK_SET_PREFIX' instead of simply passing '1'.
- When the set prefix flag isn't usage, reset any previously set prefix.
This ensures that the internal prefix state is always reset whenever
we seek and unifies the behavior between 'ref_iterator_seek' and
'ref_iterator_begin'.
- Don't allow '--skip-until' to be run with '--sort', since the seeking
always takes place before any sorting and this can be confusing.
- Some styling fixes:
- Remove extra newline
- Skip braces around single lined if...else clause
- Add braces around 'if' clause
- Fix indentation
- Link to v1: https://lore.kernel.org/git/20250701-306-git-for-each-ref-pagination-v1-0-4f0ae7c0688f@gmail.com/
--- Documentation/git-for-each-ref.adoc | 11 +- builtin/for-each-ref.c | 8 ++ ref-filter.c | 80 +++++++++++---- ref-filter.h | 1 + refs.c | 6 +- refs.h | 158 +++++++++++++++++++++++++++++ refs/debug.c | 7 +- refs/files-backend.c | 7 +- refs/iterator.c | 26 +++-- refs/packed-backend.c | 17 ++-- refs/ref-cache.c | 99 ++++++++++++++---- refs/ref-cache.h | 7 -- refs/refs-internal.h | 152 ++-------------------------- refs/reftable-backend.c | 21 ++-- t/t6302-for-each-ref-filter.sh | 194 ++++++++++++++++++++++++++++++++++++ 15 files changed, 568 insertions(+), 226 deletions(-)
Karthik Nayak (4):
refs: expose ref_iterator via 'refs.h'
ref-cache: remove unused function 'find_ref_entry()'
refs: selectively set prefix in the seek functions
for-each-ref: introduce a '--start-after' option
Range-diff versus v2:
1: c0ce873c35 = 1: dbb03c2aa9 refs: expose `ref_iterator` via 'refs.h'
2: 2c50d1eba2 = 2: fa5a0cb722 ref-cache: remove unused function 'find_ref_entry()'
3: fae849749f = 3: 9940d390cc refs: selectively set prefix in the seek functions
4: a0725a6647 ! 4: ebe864095a for-each-ref: introduce a '--skip-until' option
@@ Metadata
Author: Karthik Nayak <[email protected]>
## Commit message ##
- for-each-ref: introduce a '--skip-until' option
+ for-each-ref: introduce a '--start-after' option
The `git-for-each-ref(1)` command is used to iterate over references
present in a repository. In large repositories with millions of
@@ Commit message
through results.
The previous commit added 'seek' functionality to the reference
- backends. Utilize this and expose a '--skip-until' option in
+ backends. Utilize this and expose a '--start-after' option in
'git-for-each-ref(1)'. When used, the reference iteration seeks to the
- first matching reference and iterates from there onward.
+ lexicographically next reference and iterates from there onward.
This enables efficient pagination workflows like:
git for-each-ref --count=100
- git for-each-ref --count=100 --skip-until=refs/heads/branch-100
- git for-each-ref --count=100 --skip-until=refs/heads/branch-200
+ git for-each-ref --count=100 --start-after=refs/heads/branch-100
+ git for-each-ref --count=100 --start-after=refs/heads/branch-200
+
+ Since the reference iterators only allow seeking to a specified marker
+ via the `ref_iterator_seek()`, we introduce a helper function
+ `start_ref_iterator_after()`, which seeks to next reference by simply
+ adding (char) 1 to the marker.
+
+ We must note that pagination always continues from the provided marker,
+ as such any concurrent reference updates lexicographically behind the
+ marker will not be output. Document the same.
Signed-off-by: Karthik Nayak <[email protected]>
@@ Documentation/git-for-each-ref.adoc: SYNOPSIS
[--merged[=<object>]] [--no-merged[=<object>]]
[--contains[=<object>]] [--no-contains[=<object>]]
- [--exclude=<pattern> ...]
-+ [--exclude=<pattern> ...] [--skip-until=<pattern>]
++ [--exclude=<pattern> ...] [--start-after=<marker>]
DESCRIPTION
-----------
@@ Documentation/git-for-each-ref.adoc: TAB %(refname)`.
--include-root-refs::
List root refs (HEAD and pseudorefs) apart from regular refs.
-+--skip-until::
-+ Skip references up to but excluding the specified pattern. Cannot be used
-+ with general pattern matching or custom sort options.
++--start-after::
++ Allows paginating the output by skipping references up to and including the
++ specified marker. When paging, it should be noted that references may be
++ deleted, modified or added between invocations. Output will only yield those
++ references which follow the marker lexicographically. If the marker does not
++ exist, output begins from the first reference that would come after it
++ alphabetically. Cannot be used with general pattern matching or custom
++ sort options.
+
FIELD NAMES
-----------
@@ builtin/for-each-ref.c: static char const * const for_each_ref_usage[] = {
N_("git for-each-ref [--points-at <object>]"),
N_("git for-each-ref [--merged [<commit>]] [--no-merged [<commit>]]"),
N_("git for-each-ref [--contains [<commit>]] [--no-contains [<commit>]]"),
-+ N_("git for-each-ref [--skip-until <pattern>]"),
++ N_("git for-each-ref [--start-after <marker>]"),
NULL
};
@@ builtin/for-each-ref.c: int cmd_for_each_ref(int argc,
OPT_GROUP(""),
OPT_INTEGER( 0 , "count", &format.array_opts.max_count, N_("show only <n> matched refs")),
OPT_STRING( 0 , "format", &format.format, N_("format"), N_("format to use for the output")),
-+ OPT_STRING( 0 , "skip-until", &filter.seek, N_("skip-until"), N_("skip references until")),
++ OPT_STRING( 0 , "start-after", &filter.start_after, N_("start-start"), N_("start iteration after the provided marker")),
OPT__COLOR(&format.use_color, N_("respect format colors")),
OPT_REF_FILTER_EXCLUDE(&filter),
OPT_REF_SORT(&sorting_options),
@@ builtin/for-each-ref.c: int cmd_for_each_ref(int argc,
if (verify_ref_format(&format))
usage_with_options(for_each_ref_usage, opts);
-+ if (filter.seek && sorting_options.nr > 1)
-+ die(_("cannot use --skip-until custom sort options"));
++ if (filter.start_after && sorting_options.nr > 1)
++ die(_("cannot use --start-after with custom sort options"));
+
sorting = ref_sorting_options(&sorting_options);
ref_sorting_set_sort_flags_all(sorting, REF_SORTING_ICASE, icase);
@@ builtin/for-each-ref.c: int cmd_for_each_ref(int argc,
filter.name_patterns = argv;
}
-+ if (filter.seek && filter.name_patterns && filter.name_patterns[0])
-+ die(_("cannot use --skip-until with patterns"));
++ if (filter.start_after && filter.name_patterns && filter.name_patterns[0])
++ die(_("cannot use --start-after with patterns"));
+
if (include_root_refs)
flags |= FILTER_REFS_ROOT_REFS | FILTER_REFS_DETACHED_HEAD;
## ref-filter.c ##
+@@ ref-filter.c: static int filter_exclude_match(struct ref_filter *filter, const char *refname)
+ return match_pattern(filter->exclude.v, refname, filter->ignore_case);
+ }
+
++/*
++ * We need to seek to the reference right after a given marker but excluding any
++ * matching references. So we seek to the lexicographically next reference.
++ */
++static int start_ref_iterator_after(struct ref_iterator *iter, const char *marker)
++{
++ struct strbuf sb = STRBUF_INIT;
++ int ret;
++
++ strbuf_addstr(&sb, marker);
++ strbuf_addch(&sb, 1);
++
++ ret = ref_iterator_seek(iter, sb.buf, 0);
++
++ strbuf_release(&sb);
++ return ret;
++}
++
+ /*
+ * This is the same as for_each_fullref_in(), but it tries to iterate
+ * only over the patterns we'll care about. Note that it _doesn't_ do a full
@@ ref-filter.c: static int for_each_fullref_in_pattern(struct ref_filter *filter,
each_ref_fn cb,
void *cb_data)
@@ ref-filter.c: static int for_each_fullref_in_pattern(struct ref_filter *filter,
+non_prefix_iter:
+ iter = refs_ref_iterator_begin(get_main_ref_store(the_repository), "",
+ NULL, 0, flags);
-+ if (filter->seek)
-+ ret = ref_iterator_seek(iter, filter->seek, 0);
++ if (filter->start_after)
++ ret = start_ref_iterator_after(iter, filter->start_after);
++
+ if (ret)
+ return ret;
+
@@ ref-filter.c: static int do_filter_refs(struct ref_filter *filter, unsigned int
+ iter = refs_ref_iterator_begin(get_main_ref_store(the_repository),
+ "", NULL, 0, 0);
+
-+ if (filter->seek)
-+ ret = ref_iterator_seek(iter, filter->seek, 0);
++ if (filter->start_after)
++ ret = start_ref_iterator_after(iter, filter->start_after);
+ else if (prefix)
+ ret = ref_iterator_seek(iter, prefix, 1);
+
@@ ref-filter.h: struct ref_array {
struct ref_filter {
const char **name_patterns;
-+ const char *seek;
++ const char *start_after;
struct strvec exclude;
struct oid_array points_at;
struct commit_list *with_commit;
@@ t/t6302-for-each-ref-filter.sh: test_expect_success 'validate worktree atom' '
test_cmp expect actual
'
-+test_expect_success 'skip until with empty value' '
++test_expect_success 'start after with empty value' '
+ cat >expect <<-\EOF &&
+ refs/heads/main
+ refs/heads/main_worktree
@@ t/t6302-for-each-ref-filter.sh: test_expect_success 'validate worktree atom' '
+ refs/tags/three
+ refs/tags/two
+ EOF
-+ git for-each-ref --format="%(refname)" --skip-until="" >actual &&
++ git for-each-ref --format="%(refname)" --start-after="" >actual &&
+ test_cmp expect actual
+'
+
-+test_expect_success 'skip until to a specific reference' '
++test_expect_success 'start after a specific reference' '
+ cat >expect <<-\EOF &&
-+ refs/odd/spot
+ refs/tags/annotated-tag
+ refs/tags/doubly-annotated-tag
+ refs/tags/doubly-signed-tag
@@ t/t6302-for-each-ref-filter.sh: test_expect_success 'validate worktree atom' '
+ refs/tags/three
+ refs/tags/two
+ EOF
-+ git for-each-ref --format="%(refname)" --skip-until=refs/odd/spot >actual &&
++ git for-each-ref --format="%(refname)" --start-after=refs/odd/spot >actual &&
+ test_cmp expect actual
+'
+
-+test_expect_success 'skip until to a specific reference with partial match' '
++test_expect_success 'start after a specific reference with partial match' '
+ cat >expect <<-\EOF &&
+ refs/odd/spot
+ refs/tags/annotated-tag
@@ t/t6302-for-each-ref-filter.sh: test_expect_success 'validate worktree atom' '
+ refs/tags/three
+ refs/tags/two
+ EOF
-+ git for-each-ref --format="%(refname)" --skip-until=refs/odd/sp >actual &&
++ git for-each-ref --format="%(refname)" --start-after=refs/odd/sp >actual &&
+ test_cmp expect actual
+'
+
-+test_expect_success 'skip until just behind a specific reference' '
++test_expect_success 'start after, just behind a specific reference' '
+ cat >expect <<-\EOF &&
+ refs/odd/spot
+ refs/tags/annotated-tag
@@ t/t6302-for-each-ref-filter.sh: test_expect_success 'validate worktree atom' '
+ refs/tags/three
+ refs/tags/two
+ EOF
-+ git for-each-ref --format="%(refname)" --skip-until=refs/odd/parrot >actual &&
++ git for-each-ref --format="%(refname)" --start-after=refs/odd/parrot >actual &&
+ test_cmp expect actual
+'
+
-+test_expect_success 'skip until to specific directory' '
++test_expect_success 'start after with specific directory match' '
+ cat >expect <<-\EOF &&
+ refs/odd/spot
+ refs/tags/annotated-tag
@@ t/t6302-for-each-ref-filter.sh: test_expect_success 'validate worktree atom' '
+ refs/tags/three
+ refs/tags/two
+ EOF
-+ git for-each-ref --format="%(refname)" --skip-until=refs/odd >actual &&
++ git for-each-ref --format="%(refname)" --start-after=refs/odd >actual &&
+ test_cmp expect actual
+'
+
-+test_expect_success 'skip until to specific directory with trailing slash' '
++test_expect_success 'start after with specific directory and trailing slash' '
+ cat >expect <<-\EOF &&
+ refs/odd/spot
+ refs/tags/annotated-tag
@@ t/t6302-for-each-ref-filter.sh: test_expect_success 'validate worktree atom' '
+ refs/tags/three
+ refs/tags/two
+ EOF
-+ git for-each-ref --format="%(refname)" --skip-until=refs/lost >actual &&
++ git for-each-ref --format="%(refname)" --start-after=refs/lost >actual &&
+ test_cmp expect actual
+'
+
-+test_expect_success 'skip until just behind a specific directory' '
++test_expect_success 'start after, just behind a specific directory' '
+ cat >expect <<-\EOF &&
+ refs/odd/spot
+ refs/tags/annotated-tag
@@ t/t6302-for-each-ref-filter.sh: test_expect_success 'validate worktree atom' '
+ refs/tags/three
+ refs/tags/two
+ EOF
-+ git for-each-ref --format="%(refname)" --skip-until=refs/odd/ >actual &&
++ git for-each-ref --format="%(refname)" --start-after=refs/odd/ >actual &&
+ test_cmp expect actual
+'
+
-+test_expect_success 'skip until overflow specific reference length' '
++test_expect_success 'start after, overflow specific reference length' '
+ cat >expect <<-\EOF &&
+ refs/tags/annotated-tag
+ refs/tags/doubly-annotated-tag
@@ t/t6302-for-each-ref-filter.sh: test_expect_success 'validate worktree atom' '
+ refs/tags/three
+ refs/tags/two
+ EOF
-+ git for-each-ref --format="%(refname)" --skip-until=refs/odd/spotnew >actual &&
++ git for-each-ref --format="%(refname)" --start-after=refs/odd/spotnew >actual &&
+ test_cmp expect actual
+'
+
-+test_expect_success 'skip until overflow specific reference path' '
++test_expect_success 'start after, overflow specific reference path' '
+ cat >expect <<-\EOF &&
+ refs/tags/annotated-tag
+ refs/tags/doubly-annotated-tag
@@ t/t6302-for-each-ref-filter.sh: test_expect_success 'validate worktree atom' '
+ refs/tags/three
+ refs/tags/two
+ EOF
-+ git for-each-ref --format="%(refname)" --skip-until=refs/odd/spot/new >actual &&
++ git for-each-ref --format="%(refname)" --start-after=refs/odd/spot/new >actual &&
++ test_cmp expect actual
++'
++
++test_expect_success 'start after, last reference' '
++ cat >expect <<-\EOF &&
++ EOF
++ git for-each-ref --format="%(refname)" --start-after=refs/tags/two >actual &&
+ test_cmp expect actual
+'
+
-+test_expect_success 'skip until used with a pattern' '
++test_expect_success 'start after used with a pattern' '
+ cat >expect <<-\EOF &&
-+ fatal: cannot use --skip-until with patterns
++ fatal: cannot use --start-after with patterns
+ EOF
-+ test_must_fail git for-each-ref --format="%(refname)" --skip-until=refs/odd/spot refs/tags 2>actual &&
++ test_must_fail git for-each-ref --format="%(refname)" --start-after=refs/odd/spot refs/tags 2>actual &&
+ test_cmp expect actual
+'
+
-+test_expect_success 'skip until used with custom sort order' '
++test_expect_success 'start after used with custom sort order' '
+ cat >expect <<-\EOF &&
-+ fatal: cannot use --skip-until custom sort options
++ fatal: cannot use --start-after with custom sort options
+ EOF
-+ test_must_fail git for-each-ref --format="%(refname)" --skip-until=refs/odd/spot --sort=author 2>actual &&
++ test_must_fail git for-each-ref --format="%(refname)" --start-after=refs/odd/spot --sort=author 2>actual &&
+ test_cmp expect actual
+'
+base-commit: cf6f63ea6bf35173e02e18bdc6a4ba41288acff9
change-id: 20250605-306-git-for-each-ref-pagination-0ba8a29ae646
Thanks
- Karthik
PATCH v3 1/4 refs: expose `ref_iterator` via 'refs.h'
The ref_iterator is an internal structure to the 'refs/' sub-directory, which allows iteration over refs. All reference iteration is built on top of these iterators.
External clients of the 'refs' subsystem use the various 'refs_for_each...()' functions to iterate over refs. However since these are wrapper functions, each combination of functionality requires a new wrapper function. This is not feasible as the functions pile up with the increase in requirements. Expose the internal reference iterator, so advanced users can mix and match options as needed.
--- refs.h | 147 +++++++++++++++++++++++++++++++++++++++++++++++++++ refs/refs-internal.h | 145 +------------------------------------------------- 2 files changed, 149 insertions(+), 143 deletions(-)
refs.h +147 -0
@@ -1190,4 +1190,151 @@ int repo_migrate_ref_storage_format(struct repository *repo,unsigned int flags, struct strbuf *err); +/* +* Reference iterators +* +* A reference iterator encapsulates the state of an in-progress +* iteration over references. Create an instance of `struct +* ref_iterator` via one of the functions in this module. +* +* A freshly-created ref_iterator doesn't yet point at a reference. To +* advance the iterator, call ref_iterator_advance(). If successful, +* this sets the iterator's refname, oid, and flags fields to describe +* the next reference and returns ITER_OK. The data pointed at by +* refname and oid belong to the iterator; if you want to retain them +* after calling ref_iterator_advance() again or calling +* ref_iterator_free(), you must make a copy. When the iteration has +* been exhausted, ref_iterator_advance() releases any resources +* associated with the iteration, frees the ref_iterator object, and +* returns ITER_DONE. If you want to abort the iteration early, call +* ref_iterator_free(), which also frees the ref_iterator object and +* any associated resources. If there was an internal error advancing +* to the next entry, ref_iterator_advance() aborts the iteration, +* frees the ref_iterator, and returns ITER_ERROR. +* +* The reference currently being looked at can be peeled by calling +* ref_iterator_peel(). This function is often faster than peel_ref(), +* so it should be preferred when iterating over references. +* +* Putting it all together, a typical iteration looks like this: +* +* int ok; +* struct ref_iterator *iter = ...; +* +* while ((ok = ref_iterator_advance(iter)) == ITER_OK) { +* if (want_to_stop_iteration()) { +* ok = ITER_DONE; +* break; +* } +* +* // Access information about the current reference: +* if (!(iter->flags & REF_ISSYMREF)) +* printf("%s is %s\n", iter->refname, oid_to_hex(iter->oid)); +* +* // If you need to peel the reference: +* ref_iterator_peel(iter, &oid); +* } +* +* if (ok != ITER_DONE) +* handle_error(); +* ref_iterator_free(iter); +*/ +struct ref_iterator ; + +/* +* These flags are passed to refs_ref_iterator_begin() (and do_for_each_ref(), +* which feeds it). +*/ +enum do_for_each_ref_flags { +/* +* Include broken references in a do_for_each_ref*() iteration, which +* would normally be omitted. This includes both refs that point to +* missing objects (a true repository corruption), ones with illegal +* names (which we prefer not to expose to callers), as well as +* dangling symbolic refs (i.e., those that point to a non-existent +* ref; this is not a corruption, but as they have no valid oid, we +* omit them from normal iteration results). +*/ +DO_FOR_EACH_INCLUDE_BROKEN = (1 <<0 ), + +/* +* Only include per-worktree refs in a do_for_each_ref*() iteration. +* Normally this will be used with a files ref_store, since that's +* where all reference backends will presumably store their +* per-worktree refs. +*/ +DO_FOR_EACH_PER_WORKTREE_ONLY = (1 <<1 ), + +/* +* Omit dangling symrefs from output; this only has an effect with +* INCLUDE_BROKEN, since they are otherwise not included at all. +*/ +DO_FOR_EACH_OMIT_DANGLING_SYMREFS = (1 <<2 ), + +/* +* Include root refs i.e. HEAD and pseudorefs along with the regular +* refs. +*/ +DO_FOR_EACH_INCLUDE_ROOT_REFS = (1 <<3 ), +}; + +/* +* Return an iterator that goes over each reference in `refs` for +* which the refname begins with prefix. If trim is non-zero, then +* trim that many characters off the beginning of each refname. +* The output is ordered by refname. +*/ +struct ref_iterator *refs_ref_iterator_begin ( +struct ref_store *refs, +const char *prefix,const char **exclude_patterns, +int trim,enum do_for_each_ref_flags flags); + +/* +* Advance the iterator to the first or next item and return ITER_OK. +* If the iteration is exhausted, free the resources associated with +* the ref_iterator and return ITER_DONE. On errors, free the iterator +* resources and return ITER_ERROR. It is a bug to use ref_iterator or +* call this function again after it has returned ITER_DONE or +* ITER_ERROR. +*/ +int ref_iterator_advance (struct ref_iterator *ref_iterator); + +/* +* Seek the iterator to the first reference with the given prefix. +* The prefix is matched as a literal string, without regard for path +* separators. If prefix is NULL or the empty string, seek the iterator to the +* first reference again. +* +* This function is expected to behave as if a new ref iterator with the same +* prefix had been created, but allows reuse of iterators and thus may allow +* the backend to optimize. Parameters other than the prefix that have been +* passed when creating the iterator will remain unchanged. +* +* Returns 0 on success, a negative error code otherwise. +*/ +int ref_iterator_seek (struct ref_iterator *ref_iterator, +const char *prefix); + +/* +* If possible, peel the reference currently being viewed by the +* iterator. Return 0 on success. +*/ +int ref_iterator_peel (struct ref_iterator *ref_iterator, +struct object_id *peeled); + +/* Free the reference iterator and any associated resources. */ +void ref_iterator_free (struct ref_iterator *ref_iterator); + +/* +* The common backend for the for_each_*ref* functions. Call fn for +* each reference in iter. If the iterator itself ever returns +* ITER_ERROR, return -1. If fn ever returns a non-zero value, stop +* the iteration and return that value. Otherwise, return 0. In any +* case, free the iterator when done. This function is basically an +* adapter between the callback style of reference iteration and the +* iterator style. +*/ +int do_for_each_ref_iterator (struct ref_iterator *iter, +each_ref_fn fn,void *cb_data); +#endif /* REFS_H */
refs/refs-internal.h +2 -143
169 lines not on this page · raw message
Signature · 1 line
2.49.0
PATCH v3 2/4 ref-cache: remove unused function 'find_ref_entry()'
In reply to Karthik NayakThe 'find_ref_entry' function is no longer used, so remove it.
--- refs/ref-cache.c | 14 -------------- refs/ref-cache.h | 7 ------- 2 files changed, 21 deletions(-)
refs/ref-cache.c +0 -14
@@ -194,20 +194,6 @@ static struct ref_dir *find_containing_dir(struct ref_dir *dir,return dir; } -struct ref_entry *find_ref_entry (struct ref_dir *dir,const char *refname) -{ -int entry_index; -struct ref_entry *entry; - dir =find_containing_dir (dir, refname); -if (!dir) -return NULL ; - entry_index =search_ref_dir (dir, refname,strlen (refname)); -if (entry_index ==-1 ) -return NULL ; - entry = dir->entries[entry_index]; -return (entry->flag &REF_DIR ) ?NULL : entry; -} -/* * Emit a warning and return true iff ref1 and ref2 have the same name * and the same oid. Die if they have the same name but different
refs/ref-cache.h +0 -7
@@ -201,13 +201,6 @@ void free_ref_cache(struct ref_cache *cache);*/ void add_entry_to_dir (struct ref_dir *dir,struct ref_entry *entry); -/* -* Find the value entry with the given name in dir, sorting ref_dirs -* and recursing into subdirectories as necessary. If the name is not -* found or it corresponds to a directory entry, return NULL. -*/ -struct ref_entry *find_ref_entry (struct ref_dir *dir,const char *refname); -/* * Start iterating over references in `cache`. If `prefix` is * specified, only include references whose names start with that
Signature · 1 line
2.49.0
PATCH v3 3/4 refs: selectively set prefix in the seek functions
In reply to Karthik NayakThe ref iterator exposes a ref_iterator_seek() function. The name suggests that this would seek the iterator to a specific reference in some ways similar to how fseek() works for the filesystem.
However, the function actually sets the prefix for refs iteration. So further iteration would only yield references which match the particular prefix. This is a bit confusing.
Let's add a 'flags' field to the function, which when set with the 'REF_ITERATOR_SEEK_SET_PREFIX' flag, will set the prefix for the iteration in-line with the existing behavior. Otherwise, the reference backends will simply seek to the specified reference and clears any previously set prefix. This allows users to start iteration from a specific reference.
In the packed and reftable backend, since references are available in a sorted list, the changes are simply setting the prefix if needed. The changes on the files-backend are a little more involved, since the files backend uses the 'ref-cache' mechanism. We move out the existing logic within cache_ref_iterator_seek() to cache_ref_iterator_set_prefix() which is called when the 'REF_ITERATOR_SEEK_SET_PREFIX' flag is set. We then parse the provided seek string and set the required levels and their indexes to ensure that seeking is possible.
--- refs.c | 6 ++-- refs.h | 29 +++++++++++------ refs/debug.c | 7 ++-- refs/files-backend.c | 7 ++-- refs/iterator.c | 26 ++++++++------- refs/packed-backend.c | 17 ++++++---- refs/ref-cache.c | 85 ++++++++++++++++++++++++++++++++++++++++++++++--- refs/refs-internal.h | 7 ++-- refs/reftable-backend.c | 21 ++++++++---- 9 files changed, 155 insertions(+), 50 deletions(-)
refs.c +3 -3
@@ -2666,12 +2666,12 @@ enum ref_transaction_error refs_verify_refnames_available(struct ref_store *refsif (!initial_transaction) {int ok; -if (!iter) { +if (!iter) iter =refs_ref_iterator_begin (refs, dirname.buf,NULL ,0 ,DO_FOR_EACH_INCLUDE_BROKEN ); - }else if (ref_iterator_seek (iter, dirname.buf) <0 ) { +else if (ref_iterator_seek (iter, dirname.buf, +REF_ITERATOR_SEEK_SET_PREFIX ) <0 ) goto cleanup; - }while ((ok =ref_iterator_advance (iter)) ==ITER_OK ) { if (skip &&
refs.h +20 -9
@@ -1299,21 +1299,32 @@ struct ref_iterator *refs_ref_iterator_begin(*/ int ref_iterator_advance (struct ref_iterator *ref_iterator); +enum ref_iterator_seek_flag { +/* +* Also set the seek pattern as a prefix for iteration. This ensures +* that only references which match the prefix are yielded. +*/ +REF_ITERATOR_SEEK_SET_PREFIX = (1 <<0 ), +}; +/* -* Seek the iterator to the first reference with the given prefix. -* The prefix is matched as a literal string, without regard for path -* separators. If prefix is NULL or the empty string, seek the iterator to the +* Seek the iterator to the first reference matching the given seek string. +* The seek string is matched as a literal string, without regard for path +* separators. If seek is NULL or the empty string, seek the iterator to the * first reference again. * -* This function is expected to behave as if a new ref iterator with the same -* prefix had been created, but allows reuse of iterators and thus may allow -* the backend to optimize. Parameters other than the prefix that have been -* passed when creating the iterator will remain unchanged. +* This function is expected to behave as if a new ref iterator has been +* created, but allows reuse of existing iterators for optimization. +* +* When the REF_ITERATOR_SEEK_SET_PREFIX flag is set, the iterator's prefix is +* updated to match the seek string, affecting all subsequent iterations. If +* not, the iterator seeks to the specified reference and clears any previously +* set prefix. * * Returns 0 on success, a negative error code otherwise. */ -int ref_iterator_seek (struct ref_iterator *ref_iterator, -const char *prefix); +int ref_iterator_seek (struct ref_iterator *ref_iterator,const char *seek, +unsigned int flags);/* * If possible, peel the reference currently being viewed by the
refs/debug.c +4 -3
@@ -170,12 +170,13 @@ static int debug_ref_iterator_advance(struct ref_iterator *ref_iterator) }static int debug_ref_iterator_seek (struct ref_iterator *ref_iterator, -const char *prefix) +const char *seek,unsigned int flags) {struct debug_ref_iterator *diter = (struct debug_ref_iterator *)ref_iterator; -int res = diter->iter->vtable->seek (diter->iter, prefix); -trace_printf_key (&trace_refs,"iterator_seek: %s: %d\n" , prefix ? prefix :"" , res); +int res = diter->iter->vtable->seek (diter->iter, seek, flags); +trace_printf_key (&trace_refs,"iterator_seek: %s flags: %d: %d\n" , + seek ? seek :"" , flags, res);return res; }
refs/files-backend.c +4 -3
@@ -929,11 +929,11 @@ static int files_ref_iterator_advance(struct ref_iterator *ref_iterator) }static int files_ref_iterator_seek (struct ref_iterator *ref_iterator, -const char *prefix) +const char *seek,unsigned int flags) {struct files_ref_iterator *iter = (struct files_ref_iterator *)ref_iterator; -return ref_iterator_seek (iter->iter0, prefix); +return ref_iterator_seek (iter->iter0, seek, flags); } static int files_ref_iterator_peel(struct ref_iterator *ref_iterator,
@@ -2316,7 +2316,8 @@ static int files_reflog_iterator_advance(struct ref_iterator *ref_iterator) } static int files_reflog_iterator_seek(struct ref_iterator *ref_iterator UNUSED, - const char *prefix UNUSED) + const char *seek UNUSED, + unsigned int flags UNUSED) {BUG ("ref_iterator_seek() called for reflog_iterator" ); }
refs/iterator.c +15 -11
@@ -15,10 +15,10 @@ int ref_iterator_advance(struct ref_iterator *ref_iterator)return ref_iterator->vtable->advance (ref_iterator); } -int ref_iterator_seek (struct ref_iterator *ref_iterator, -const char *prefix) +int ref_iterator_seek (struct ref_iterator *ref_iterator,const char *seek, +unsigned int flags) { -return ref_iterator->vtable->seek (ref_iterator, prefix); +return ref_iterator->vtable->seek (ref_iterator, seek, flags); } int ref_iterator_peel(struct ref_iterator *ref_iterator,
@@ -57,7 +57,8 @@ static int empty_ref_iterator_advance(struct ref_iterator *ref_iterator UNUSED) } static int empty_ref_iterator_seek(struct ref_iterator *ref_iterator UNUSED, - const char *prefix UNUSED) + const char *seek UNUSED, + unsigned int flags UNUSED) {return 0 ; }
@@ -224,7 +225,7 @@ static int merge_ref_iterator_advance(struct ref_iterator *ref_iterator) }static int merge_ref_iterator_seek (struct ref_iterator *ref_iterator, -const char *prefix) +const char *seek,unsigned int flags) {struct merge_ref_iterator *iter = (struct merge_ref_iterator *)ref_iterator;
@@ -234,11 +235,11 @@ static int merge_ref_iterator_seek(struct ref_iterator *ref_iterator, iter->iter0 = iter->iter0_owned; iter->iter1 = iter->iter1_owned; - ret =ref_iterator_seek (iter->iter0, prefix); + ret =ref_iterator_seek (iter->iter0, seek, flags);if (ret <0 )return ret; - ret =ref_iterator_seek (iter->iter1, prefix); + ret =ref_iterator_seek (iter->iter1, seek, flags);if (ret <0 )return ret;
@@ -407,13 +408,16 @@ static int prefix_ref_iterator_advance(struct ref_iterator *ref_iterator) }static int prefix_ref_iterator_seek (struct ref_iterator *ref_iterator, -const char *prefix) +const char *seek,unsigned int flags) {struct prefix_ref_iterator *iter = (struct prefix_ref_iterator *)ref_iterator; -free (iter->prefix); - iter->prefix =xstrdup_or_null (prefix); -return ref_iterator_seek (iter->iter0, prefix); + +if (flags &REF_ITERATOR_SEEK_SET_PREFIX ) { +free (iter->prefix); + iter->prefix =xstrdup_or_null (seek); + } +return ref_iterator_seek (iter->iter0, seek, flags); } static int prefix_ref_iterator_peel(struct ref_iterator *ref_iterator,
refs/packed-backend.c +11 -6
@@ -1004,19 +1004,23 @@ static int packed_ref_iterator_advance(struct ref_iterator *ref_iterator) }static int packed_ref_iterator_seek (struct ref_iterator *ref_iterator, -const char *prefix) +const char *seek,unsigned int flags) {struct packed_ref_iterator *iter = (struct packed_ref_iterator *)ref_iterator;const char *start; -if (prefix && *prefix) - start =find_reference_location (iter->snapshot, prefix,0 ); +if (seek && *seek) + start =find_reference_location (iter->snapshot, seek,0 );else start = iter->snapshot->start; -free (iter->prefix); - iter->prefix =xstrdup_or_null (prefix); +/* Unset any previously set prefix */ +FREE_AND_NULL (iter->prefix); + +if (flags &REF_ITERATOR_SEEK_SET_PREFIX ) + iter->prefix =xstrdup_or_null (seek); + iter->pos = start; iter->eof = iter->snapshot->eof;
@@ -1194,7 +1198,8 @@ static struct ref_iterator *packed_ref_iterator_begin( iter->repo = ref_store->repo; iter->flags = flags; -if (packed_ref_iterator_seek (&iter->base, prefix) <0 ) { +if (packed_ref_iterator_seek (&iter->base, prefix, +REF_ITERATOR_SEEK_SET_PREFIX ) <0 ) {ref_iterator_free (&iter->base);return NULL ; }
refs/ref-cache.c +80 -5
104 lines not on this page · raw message
refs/refs-internal.h +4 -3
@@ -353,11 +353,12 @@ void base_ref_iterator_init(struct ref_iterator *iter,typedef int ref_iterator_advance_fn (struct ref_iterator *ref_iterator);/* -* Seek the iterator to the first reference matching the given prefix. Should -* behave the same as if a new iterator was created with the same prefix. +* Seek the iterator to the first matching reference. If set_prefix is set, +* it would behave the same as if a new iterator was created with the same +* prefix. */ typedef int ref_iterator_seek_fn (struct ref_iterator *ref_iterator, -const char *prefix); +const char *seek,unsigned int flags);/* * Peels the current ref, returning 0 for success or -1 for failure.
refs/reftable-backend.c +14 -7
@@ -719,15 +719,20 @@ static int reftable_ref_iterator_advance(struct ref_iterator *ref_iterator) }static int reftable_ref_iterator_seek (struct ref_iterator *ref_iterator, -const char *prefix) +const char *seek,unsigned int flags) {struct reftable_ref_iterator *iter = (struct reftable_ref_iterator *)ref_iterator; -free (iter->prefix); - iter->prefix =xstrdup_or_null (prefix); - iter->prefix_len = prefix ?strlen (prefix) :0 ; - iter->err =reftable_iterator_seek_ref (&iter->iter, prefix); +/* Unset any previously set prefix */ +FREE_AND_NULL (iter->prefix); + iter->prefix_len =0 ; + +if (flags &REF_ITERATOR_SEEK_SET_PREFIX ) { + iter->prefix =xstrdup_or_null (seek); + iter->prefix_len = seek ?strlen (seek) :0 ; + } + iter->err =reftable_iterator_seek_ref (&iter->iter, seek);return iter->err; }
@@ -839,7 +844,8 @@ static struct reftable_ref_iterator *ref_iterator_for_stack(struct reftable_ref_if (ret) goto done; - ret =reftable_ref_iterator_seek (&iter->base, prefix); + ret =reftable_ref_iterator_seek (&iter->base, prefix, +REF_ITERATOR_SEEK_SET_PREFIX );if (ret) goto done;
@@ -2042,7 +2048,8 @@ static int reftable_reflog_iterator_advance(struct ref_iterator *ref_iterator) } static int reftable_reflog_iterator_seek(struct ref_iterator *ref_iterator UNUSED, - const char *prefix UNUSED) + const char *seek UNUSED, + unsigned int flags UNUSED) {BUG ("reftable reflog iterator cannot be seeked" );return -1 ;
Signature · 1 line
2.49.0
Re: [PATCH v3 3/4] refs: selectively set prefix in the seek functions
Quoted 6 lines · Karthik Nayak
refs.h +0
-0@@ -1299,21 +1299,32 @@ struct ref_iterator *refs_ref_iterator_begin(*/
refs.h struct ref_iterator *refs_ref_iterator_begin( · line 1309
Quoted 10 lines
refs.h +8
-0int ref_iterator_advance (struct ref_iterator *ref_iterator); +enum ref_iterator_seek_flag { +/* +* Also set the seek pattern as a prefix for iteration. This ensures +* that only references which match the prefix are yielded. +*/ +REF_ITERATOR_SEEK_SET_PREFIX = (1 <<0 ), +}; +
Nit: I think it's a tiny bit confusing that the documentation of this enum is split up across here and the doc of ref_iterator_seek(). I think it would be sensible to move the last paragraph of the function over here so that the whole behaviour of the enum is explained in a single place.
Quoted 17 lines
refs.h +7
-7/* -* Seek the iterator to the first reference with the given prefix. -* The prefix is matched as a literal string, without regard for path -* separators. If prefix is NULL or the empty string, seek the iterator to the +* Seek the iterator to the first reference matching the given seek string. +* The seek string is matched as a literal string, without regard for path +* separators. If seek is NULL or the empty string, seek the iterator to the * first reference again. * -* This function is expected to behave as if a new ref iterator with the same -* prefix had been created, but allows reuse of iterators and thus may allow -* the backend to optimize. Parameters other than the prefix that have been -* passed when creating the iterator will remain unchanged. +* This function is expected to behave as if a new ref iterator has been +* created, but allows reuse of existing iterators for optimization. +* +* When the REF_ITERATOR_SEEK_SET_PREFIX flag is set, the iterator's prefix is
refs.h struct ref_iterator *refs_ref_iterator_begin( · line 1327
Quoted 10 lines
refs.h +5
-2+* updated to match the seek string, affecting all subsequent iterations. If +* not, the iterator seeks to the specified reference and clears any previously +* set prefix. * * Returns 0 on success, a negative error code otherwise. */ -int ref_iterator_seek (struct ref_iterator *ref_iterator, -const char *prefix); +int ref_iterator_seek (struct ref_iterator *ref_iterator,const char *seek, +unsigned int flags);
Another tiny nit: instead of calling the variable seek we can just call it refname. That might give a bit more of a hint what you're actually seeking for.
But other than that I'm happy with the new behaviour, where we are now consistently either setting or resetting the prefix depending on whether or not the caller set the flag.
Quoted 22 lines
refs/iterator.c +7
-4@@ -407,13 +408,16 @@ static int prefix_ref_iterator_advance(struct ref_iterator *ref_iterator) }static int prefix_ref_iterator_seek (struct ref_iterator *ref_iterator, -const char *prefix) +const char *seek,unsigned int flags) {struct prefix_ref_iterator *iter = (struct prefix_ref_iterator *)ref_iterator; -free (iter->prefix); - iter->prefix =xstrdup_or_null (prefix); -return ref_iterator_seek (iter->iter0, prefix); + +if (flags &REF_ITERATOR_SEEK_SET_PREFIX ) { +free (iter->prefix); + iter->prefix =xstrdup_or_null (seek); + } +return ref_iterator_se
Re: [PATCH v3 3/4] refs: selectively set prefix in the seek functions
Quoted 9 lines · Patrick Steinhardt
Quoted 8 lines · Karthik Nayak
refs.h +0
-0@@ -1299,21 +1299,32 @@ struct ref_iterator *refs_ref_iterator_begin(*/ int ref_iterator_advance (struct ref_iterator *ref_iterator);
Quoted 15 lines
refs.h struct ref_iterator *refs_ref_iterator_begin( · line 1309
Quoted 8 lines
refs.h +8
-0+enum ref_iterator_seek_flag { +/* +* Also set the seek pattern as a prefix for iteration. This ensures +* that only references which match the prefix are yielded. +*/ +REF_ITERATOR_SEEK_SET_PREFIX = (1 <<0 ), +}; +Nit: I think it's a tiny bit confusing that the documentation of this enum is split up across here and the doc of
ref_iterator_seek(). I think it would be sensible to move the last paragraph of the function over here so that the whole behaviour of the enum is explained in a single place.
Yeah I think that makes sense.
Quoted 23 lines
Quoted 23 lines
refs.h +10
-7/* -* Seek the iterator to the first reference with the given prefix. -* The prefix is matched as a literal string, without regard for path -* separators. If prefix is NULL or the empty string, seek the iterator to the +* Seek the iterator to the first reference matching the given seek string. +* The seek string is matched as a literal string, without regard for path +* separators. If seek is NULL or the empty string, seek the iterator to the * first reference again. * -* This function is expected to behave as if a new ref iterator with the same -* prefix had been created, but allows reuse of iterators and thus may allow -* the backend to optimize. Parameters other than the prefix that have been -* passed when creating the iterator will remain unchanged. +* This function is expected to behave as if a new ref iterator has been +* created, but allows reuse of existing iterators for optimization. +* +* When the REF_ITERATOR_SEEK_SET_PREFIX flag is set, the iterator's prefix is +* updated to match the seek string, affecting all subsequent iterations. If +* not, the iterator seeks to the specified reference and clears any previously +* set prefix. * * Returns 0 on success, a negative error code otherwise. */
Quoted 9 lines
refs.h struct ref_iterator *refs_ref_iterator_begin( · line 1327
Quoted 4 lines
refs.h +2
-2-int ref_iterator_seek (struct ref_iterator *ref_iterator, -const char *prefix); +int ref_iterator_seek (struct ref_iterator *ref_iterator,const char *seek, +unsigned int flags);Another tiny nit: instead of calling the variable
seekwe can just call itrefname. That might give a bit more of a hint what you're actually seeking for.
Fair enough, let me change that.
Quoted 4 lines
But other than that I'm happy with the new behaviour, where we are now consistently either setting or resetting the prefix depending on whether or not the caller set the flag.
Thanks for the review!
Quoted 22 lines
Quoted 22 lines
refs/iterator.c +7
-4@@ -407,13 +408,16 @@ static int prefix_ref_iterator_advance(struct ref_iterator *ref_iterator) }static int prefix_ref_iterator_seek (struct ref_iterator *ref_iterator, -const char *prefix) +const char *seek,unsigned int flags) {struct prefix_ref_iterator *iter = (struct prefix_ref_iterator *)ref_iterator; -free (iter->prefix); - iter->prefix =xstrdup_or_null (prefix); -return ref_iterator_seek (iter->iter0, prefix); + +if (flags &REF_ITERATOR_SEEK_SET_PREFIX ) { +free (iter->prefix); + iter->prefix =xstrdup_or_null (seek); + } +return ref_iterator_se
Re: [PATCH v3 3/4] refs: selectively set prefix in the seek functions
In reply to Patrick SteinhardtQuoted 16 lines · Patrick Steinhardt
refs.h struct ref_iterator *refs_ref_iterator_begin( · line 1327
Quoted 11 lines
+ * When the
REF_ITERATOR_SEEK_SET_PREFIXflag is set, the iterator's prefix is
+ * updated to match the seek string, affecting all subsequent iterations. If
+ * not, the iterator seeks to the specified reference and clears any previously
+ * set prefix.
*
* Returns 0 on success, a negative error code otherwise.
*/refs.h +2
-2-int ref_iterator_seek (struct ref_iterator *ref_iterator, -const char *prefix); +int ref_iterator_seek (struct ref_iterator *ref_iterator,const char *seek, +unsigned int flags);Another tiny nit: instead of calling the variable
seekwe can just call itrefname. That might give a bit more of a hint what you're actually seeking for.
Quoted 3 lines
But other than that I'm happy with the new behaviour, where we are now consistently either setting or resetting the prefix depending on whether or not the caller set the flag.
I am not sure. The way the "prefix" is used, if I understand correctly, is
- it is set by iterator-begin, typically to the area to iterate over (e.g. "refs/heads/" for iterating over branches) in the for_each_ref_*() family of helpers, and internally we seek to that area (skipping anything that come strictly before "refs/heads/" for example).
- iterator-advance looks at it and decides we are done when the iterator points beyond that prefix
So if you are iterating inside "refs/heads/" hierarchy and seek to "refs/heads/m", don't you still want to stop when you step outside "refs/heads/" by keeping the original prefix, instead of unsetting the prefix to empty? A postimage of this patch for packed backend (picked at random) reads like this:
static int packed_ref_iterator_seek(struct ref_iterator *ref_iterator,
const char *seek, unsigned int flags)
{
struct packed_ref_iterator *iter =
(struct packed_ref_iterator *)ref_iterator;
const char *start;
if (seek && *seek)
start = find_reference_location(iter->snapshot, seek, 0);
else
start = iter->snapshot->start;
/* Unset any previously set prefix */
FREE_AND_NULL(iter->prefix);
if (flags & REF_ITERATOR_SEEK_SET_PREFIX)
iter->prefix = xstrdup_or_null(seek);so after (true) seeking that does not have the SET_PREFIX flag on, wouldn't our iterator-advance run through the end since it no longer is aware of where to stop?
Thanks.
Re: [PATCH v3 3/4] refs: selectively set prefix in the seek functions
Quoted 24 lines · Junio C Hamano
Quoted 16 lines · Patrick Steinhardt
refs.h struct ref_iterator *refs_ref_iterator_begin( · line 1327
Quoted 11 lines
+ * When the
REF_ITERATOR_SEEK_SET_PREFIXflag is set, the iterator's prefix is
+ * updated to match the seek string, affecting all subsequent iterations. If
+ * not, the iterator seeks to the specified reference and clears any previously
+ * set prefix.
*
* Returns 0 on success, a negative error code otherwise.
*/refs.h +2
-2-int ref_iterator_seek (struct ref_iterator *ref_iterator, -const char *prefix); +int ref_iterator_seek (struct ref_iterator *ref_iterator,const char *seek, +unsigned int flags);Another tiny nit: instead of calling the variable
seekwe can just call itrefname. That might give a bit more of a hint what you're actually seeking for.Quoted 3 lines
But other than that I'm happy with the new behaviour, where we are now consistently either setting or resetting the prefix depending on whether or not the caller set the flag.
I am not sure. The way the "prefix" is used, if I understand correctly, is
Quoted 9 lines
- it is set by iterator-begin, typically to the area to iterate over (e.g. "refs/heads/" for iterating over branches) in the for_each_ref_*() family of helpers, and internally we seek to that area (skipping anything that come strictly before "refs/heads/" for example).
- iterator-advance looks at it and decides we are done when the iterator points beyond that prefix
That's right.
Quoted 24 lines
So if you are iterating inside "refs/heads/" hierarchy and seek to "refs/heads/m", don't you still want to stop when you step outside "refs/heads/" by keeping the original prefix, instead of unsetting the prefix to empty? A postimage of this patch for packed backend (picked at random) reads like this:
static int packed_ref_iterator_seek(struct ref_iterator *ref_iterator, const char *seek, unsigned int flags) { struct packed_ref_iterator *iter = (struct packed_ref_iterator *)ref_iterator; const char *start; if (seek && *seek) start = find_reference_location(iter->snapshot, seek, 0); else start = iter->snapshot->start; /* Unset any previously set prefix */ FREE_AND_NULL(iter->prefix); if (flags & REF_ITERATOR_SEEK_SET_PREFIX) iter->prefix = xstrdup_or_null(seek);
Quoted 4 lines
so after (true) seeking that does not have the
SET_PREFIXflag on, wouldn't our iterator-advance run through the end since it no longer is aware of where to stop?
That's also right and that is indeed the intention. We're trying to make the actions more intentional.
So if a user sets a 'prefix' for the iterator, all previous state of the iterator is reset. So, the same function for seeking an iterator should also have the same side-effect of resetting the previous state.
There could be a usecase where we add support for keeping the prefix, while also seeking the iterator. That would be an explicit change (perhaps with a corresponding flag?) that we'd have to build, add tests for and call out. Until then, we explicitly reset the state whenever a user calls 'ref_iterator_seek()', they can be sure that any previous state is reset.
Quoted 1 line
Thanks.
Re: [PATCH v3 3/4] refs: selectively set prefix in the seek functions
Quoted 11 lines · Karthik Nayak
Quoted 4 lines
so after (true) seeking that does not have the
SET_PREFIXflag on, wouldn't our iterator-advance run through the end since it no longer is aware of where to stop?That's also right and that is indeed the intention. We're trying to make the actions more intentional.
So if a user sets a 'prefix' for the iterator, all previous state of the iterator is reset. So, the same function for seeking an iterator should also have the same side-effect of resetting the previous state.
Perhaps we have different definition of "previous state" in mind? So let's imagine an iterator is walking over all branches (i.e. the prefix is set to refs/heads/, to allow it to stop once it steps outside refs/heads/ and moves over to refs/imerge). It starts iterating and I see branches whose name sorts early in alphabetical order. I tell it to seek to refs/heads/master and keep iterating.
Wouldn't it be a lot more natural if it still stops iterating after it finishes showing the last branch, iow, a ref in refs/heads/ hierarchy? In other words, I am not sure why ...
Quoted 2 lines
There could be a usecase where we add support for keeping the prefix, while also seeking the iterator. That would be an explicit change
... that is the optional and unimplemented feature, not the other way around. Is it just the ease of implementation?
Re: [PATCH v3 3/4] refs: selectively set prefix in the seek functions
Quoted 14 lines · Junio C Hamano
Quoted 11 lines · Karthik Nayak
Quoted 4 lines
so after (true) seeking that does not have the
SET_PREFIXflag on, wouldn't our iterator-advance run through the end since it no longer is aware of where to stop?That's also right and that is indeed the intention. We're trying to make the actions more intentional.
So if a user sets a 'prefix' for the iterator, all previous state of the iterator is reset. So, the same function for seeking an iterator should also have the same side-effect of resetting the previous state.
Quoted 7 lines
Perhaps we have different definition of "previous state" in mind? So let's imagine an iterator is walking over all branches (i.e. the prefix is set to refs/heads/, to allow it to stop once it steps outside refs/heads/ and moves over to refs/imerge). It starts iterating and I see branches whose name sorts early in alphabetical order. I tell it to seek to refs/heads/master and keep iterating.
I get what you're saying and indeed that would be natural. Let me draw another example to draw the contrast.
Let's say a user is iterating with a prefix set to 'refs/heads/', this would iterate over all the refs with that prefix. But mid-way the user realizes that they only care about 'refs/heads/feature/' prefix and they ask the iterator to set that as the prefix.
In such a situation, the iterator seeks to 'refs/heads/feature/' and will only yield references with that prefix. In short, the previous prefix state was reset.
So to avoid the two scenarios:
1. Only seek the iterator but maintain prefix
2. Seek and set new prefix, loosing old prefix
Where one resets the prefix while the other doesn't. We make it explicit and say, whenever 'ref_iterator_seek' is called, any set prefix is reset. I do see the other way around too, where prefix isn't treated as previous state.
What I was trying to argue for, was that, there could be a situation like what you mentioned, where a user might want to retain a prefix, this should be an explicit requirement which not implemented in this series. So as of this series, you cannot set a prefix and then seek and expect to retain the prefix.
Quoted 9 lines
Wouldn't it be a lot more natural if it still stops iterating after it finishes showing the last branch, iow, a ref in refs/heads/ hierarchy? In other words, I am not sure why ...
Quoted 2 lines
There could be a usecase where we add support for keeping the prefix, while also seeking the iterator. That would be an explicit change
... that is the optional and unimplemented feature, not the other way around. Is it just the ease of implementation?
This series did start out that way around, so ease of implementation isn't it. It was more of a side-effect of not clearing state. But I would be more comfortable if this wasn't a side-effect but rather a conscious choice with tests and adequate documentation.
Re: [PATCH v3 3/4] refs: selectively set prefix in the seek functions
Quoted 4 lines · Karthik Nayak
Let's say a user is iterating with a prefix set to 'refs/heads/', this would iterate over all the refs with that prefix. But mid-way the user realizes that they only care about 'refs/heads/feature/' prefix and they ask the iterator to set that as the prefix.
But is that user who changes their mind in the middle "seeking"?
It is more like "I am abandoning the enumeration I started earlier over refs/heads/, and I want a different enumeration over refs/heads/feature/, but because I know the implementation detail that abandoning an iterator and creating another is more expensive, let me reuse the one in use to repurpose it".
I wouldn't call it "seeking"; it sounds more like "resetting".
Re: [PATCH v3 3/4] refs: selectively set prefix in the seek functions
In reply to Karthik NayakQuoted 8 lines · Karthik Nayak
Let's say a user is iterating with a prefix set to 'refs/heads/', this would iterate over all the refs with that prefix. But mid-way the user realizes that they only care about 'refs/heads/feature/' prefix and they ask the iterator to set that as the prefix.
In such a situation, the iterator seeks to 'refs/heads/feature/' and will only yield references with that prefix. In short, the previous prefix state was reset.
Yes, even though I wouldn't call such an operation "seek", "Ah, I do not need the entire refs/heads/ walked, only refs/heads/feature/ is enough" is an operation mode that makes sense.
But not for paging, though.
If your web application is showing all branches, one pageful at a time, and the first page ended at refs/heads/feature/something and you ended up "seeking" to refs/heads/feature/ to start the second page, you do not want your second page to end when the iteration goes out of refs/heads/feature/ hierarchy, no?
It seems to me that the root cause of the confusion is because prefix, which is to let iteration finish way before the data runs out (instead finish when the iteration steps out of a given subhierarchy denoted by the prefix), is somehow abused as the current position of the cursor. Shouldn't they be two separate concepts? The cursor needs to fall within the prefix while the iterator is active, so they are not two totally independent things, but prefix is pretty much static while the cursor position is very dynamic.
Quoted 2 lines
This series did start out that way around, so ease of implementation isn't it. It was more of a side-effect of not clearing state.
I am even more worried about usability and correctness aspect of what was described here now. After seeking to refs/heads/feature/, do we continue to iterate and step out of refs/heads/feature/ hierarchy or can we cut off a particular page that started with a ref within refs/heads/feature/ subhierarchy when we exhaust refs in refs/heads/feature/ and have to wait for getting asked for the next page before we show refs/heads/gsomething that is outside refs/heads/feature/ and sorts after? The "I reset to iterate over refs/heads/feature/ because the entire refs/heads/ is not what I care about" example makes me worried about this.
Thanks.
Re: [PATCH v3 3/4] refs: selectively set prefix in the seek functions
Quoted 15 lines · Junio C Hamano
Quoted 8 lines · Karthik Nayak
Let's say a user is iterating with a prefix set to 'refs/heads/', this would iterate over all the refs with that prefix. But mid-way the user realizes that they only care about 'refs/heads/feature/' prefix and they ask the iterator to set that as the prefix.
In such a situation, the iterator seeks to 'refs/heads/feature/' and will only yield references with that prefix. In short, the previous prefix state was reset.
Yes, even though I wouldn't call such an operation "seek", "Ah, I do not need the entire refs/heads/ walked, only refs/heads/feature/ is enough" is an operation mode that makes sense.
Quoted 8 lines
But not for paging, though.
If your web application is showing all branches, one pageful at a time, and the first page ended at refs/heads/feature/something and you ended up "seeking" to refs/heads/feature/ to start the second page, you do not want your second page to end when the iteration goes out of refs/heads/feature/ hierarchy, no?
Yup and this (we show all references beyond the seek) is the current implementation. I was talking about the internal implementation of 'refs_iteration_seek()' which is the function used for seek and setting the prefix.
To clarify, this is the current implementation:
$ git for-each-ref 6f4b58c4968eb82277cf5b1cf8775117e5b83de0 commit refs/heads/bar 6f4b58c4968eb82277cf5b1cf8775117e5b83de0 commit refs/heads/feature/x 6f4b58c4968eb82277cf5b1cf8775117e5b83de0 commit refs/heads/feature/y 6f4b58c4968eb82277cf5b1cf8775117e5b83de0 commit refs/heads/foo 6f4b58c4968eb82277cf5b1cf8775117e5b83de0 commit refs/heads/goo/x 6f4b58c4968eb82277cf5b1cf8775117e5b83de0 commit refs/heads/goo/y 6f4b58c4968eb82277cf5b1cf8775117e5b83de0 commit refs/heads/master 6f4b58c4968eb82277cf5b1cf8775117e5b83de0 commit refs/tags/tagged/2 6f4b58c4968eb82277cf5b1cf8775117e5b83de0 commit refs/tags/tagged/3 6f4b58c4968eb82277cf5b1cf8775117e5b83de0 commit refs/tags/v1 6f4b58c4968eb82277cf5b1cf8775117e5b83de0 commit refs/tags/v2
$ git for-each-ref --format="%(refname)" --start-after=refs/heads/goo refs/heads/goo/x refs/heads/goo/y refs/heads/master refs/tags/tagged/2 refs/tags/tagged/3 refs/tags/v1 refs/tags/v2
$ git for-each-ref --format="%(refname)" --start-after=refs/heads/master refs/tags/tagged/2 refs/tags/tagged/3 refs/tags/v1 refs/tags/v2
$ git for-each-ref --format="%(refname)" --start-after=refs/heads/goo/x refs/heads/goo/y refs/heads/master refs/tags/tagged/2 refs/tags/tagged/3 refs/tags/v1 refs/tags/v2
$ git for-each-ref --format="%(refname)" refs/heads/feature refs/heads/feature/x refs/heads/feature/y
You can see we list all references beyond the seek.
Quoted 10 lines
It seems to me that the root cause of the confusion is because prefix, which is to let iteration finish way before the data runs out (instead finish when the iteration steps out of a given subhierarchy denoted by the prefix), is somehow abused as the current position of the cursor. Shouldn't they be two separate concepts? The cursor needs to fall within the prefix while the iterator is active, so they are not two totally independent things, but prefix is pretty much static while the cursor position is very dynamic.
The prefix setup in 'ref_iteration_seek' does two things, let's consider
prefix: 'refs/heads/feature'
1. It sets the cursor to seek to 'refs/heads/feature'
2. It also sets the internal prefix matching to 'refs/heads/feature'
In Contrast seeking via 'ref_iteration_seek' only sets the cursor to 'refs/heads/feature'.
To make this simpler, we've changed 'ref_iteration_seek' to do:
1. seek the cursor to the requested reference
2. Set prefix if the REF_ITERATOR_SEEK_SET_PREFIX is set, and unset the prefix otherwise.
The state reset I was talking about in my previous emails refers to step #2 here, where when no 'REF_ITERATOR_SEEK_SET_PREFIX' is set, we remove any previous prefix set.
Quoted 14 lines
Quoted 2 lines
This series did start out that way around, so ease of implementation isn't it. It was more of a side-effect of not clearing state.
I am even more worried about usability and correctness aspect of what was described here now. After seeking to refs/heads/feature/, do we continue to iterate and step out of refs/heads/feature/ hierarchy or can we cut off a particular page that started with a ref within refs/heads/feature/ subhierarchy when we exhaust refs in refs/heads/feature/ and have to wait for getting asked for the next page before we show refs/heads/gsomething that is outside refs/heads/feature/ and sorts after? The "I reset to iterate over refs/heads/feature/ because the entire refs/heads/ is not what I care about" example makes me worried about this.
Quoted 1 line
Thanks.
I think we're crossing paths and talking different things. I hope the examples above clarify things. The current implementation doesn't support '--start-after' and prefix setting at the same time:
$ git for-each-ref --format="%(refname)" --start-after=refs/heads/master refs/heads fatal: cannot use --start-after with patterns
Happy to clarify if this doesn't make sense.
Thanks
Re: [PATCH v3 3/4] refs: selectively set prefix in the seek functions
Quoted 5 lines · Karthik Nayak
Quoted 5 lines
Yes, even though I wouldn't call such an operation "seek", "Ah, I do not need the entire refs/heads/ walked, only refs/heads/feature/ is enough" is an operation mode that makes sense.
But not for paging, though.
Actually, you are not using the new value for prefix; you are unsetting prefix to nothing, so you do not stop when you go over the boundary of refs/heads/feature/ hierarchy but will run through to the end, so my worry is unfounded.
Which is good.
But the application has to decide when to stop as "We obtained an iterator in order to list all branches in refs/heads/ and the last round we gave refs/heads/main out. Seek to it and continue the output" will no longer stop after making a callback with refs/heads/zz but will continue yielding non branch refs after that.
Quoted 6 lines
The current implementation doesn't
support '--start-after' and prefix setting at the same time:$ git for-each-ref --format="%(refname)" --start-after=refs/heads/master refs/heads fatal: cannot use --start-after with patterns
Good---in that case, "unsetting prefix to nothing" does not make any difference and cannot introduce any confusing behaviour, as it has to run to the end with or without "--start-after" anyway.
Thanks.
PATCH v3 4/4 for-each-ref: introduce a '--start-after' option
In reply to Karthik NayakThe git-for-each-ref(1) command is used to iterate over references present in a repository. In large repositories with millions of references, it would be optimal to paginate this output such that we can start iteration from a given reference. This would avoid having to iterate over all references from the beginning each time when paginating through results.
The previous commit added 'seek' functionality to the reference backends. Utilize this and expose a '--start-after' option in 'git-for-each-ref(1)'. When used, the reference iteration seeks to the lexicographically next reference and iterates from there onward.
This enables efficient pagination workflows like:
git for-each-ref --count=100
git for-each-ref --count=100 --start-after=refs/heads/branch-100
git for-each-ref --count=100 --start-after=refs/heads/branch-200Since the reference iterators only allow seeking to a specified marker via the ref_iterator_seek(), we introduce a helper function start_ref_iterator_after(), which seeks to next reference by simply adding (char) 1 to the marker.
We must note that pagination always continues from the provided marker, as such any concurrent reference updates lexicographically behind the marker will not be output. Document the same.
--- Documentation/git-for-each-ref.adoc | 11 +- builtin/for-each-ref.c | 8 ++ ref-filter.c | 80 +++++++++++---- ref-filter.h | 1 + t/t6302-for-each-ref-filter.sh | 194 ++++++++++++++++++++++++++++++++++++ 5 files changed, 273 insertions(+), 21 deletions(-)
Documentation/git-for-each-ref.adoc +10 -1
@@ -14,7 +14,7 @@ SYNOPSIS [--points-at=<object>] [--merged[=<object>]] [--no-merged[=<object>]] [--contains[=<object>]] [--no-contains[=<object>]] - [--exclude=<pattern> ...] + [--exclude=<pattern> ...] [--start-after=<marker>] DESCRIPTION -----------
@@ -108,6 +108,15 @@ TAB %(refname)`. --include-root-refs:: List root refs (HEAD and pseudorefs) apart from regular refs. +--start-after:: + Allows paginating the output by skipping references up to and including the + specified marker. When paging, it should be noted that references may be + deleted, modified or added between invocations. Output will only yield those + references which follow the marker lexicographically. If the marker does not + exist, output begins from the first reference that would come after it + alphabetically. Cannot be used with general pattern matching or custom + sort options. + FIELD NAMES -----------
builtin/for-each-ref.c +8 -0
@@ -13,6 +13,7 @@ static char const * const for_each_ref_usage[] = {N_ ("git for-each-ref [--points-at <object>]" ),N_ ("git for-each-ref [--merged [<commit>]] [--no-merged [<commit>]]" ),N_ ("git for-each-ref [--contains [<commit>]] [--no-contains [<commit>]]" ), +N_ ("git for-each-ref [--start-after <marker>]" ), NULL };
@@ -44,6 +45,7 @@ int cmd_for_each_ref(int argc,OPT_GROUP ("" ),OPT_INTEGER (0 ,"count" , &format.array_opts.max_count,N_ ("show only <n> matched refs" )),OPT_STRING (0 ,"format" , &format.format,N_ ("format" ),N_ ("format to use for the output" )), +OPT_STRING (0 ,"start-after" , &filter.start_after,N_ ("start-start" ),N_ ("start iteration after the provided marker" )),OPT__COLOR (&format.use_color,N_ ("respect format colors" )),OPT_REF_FILTER_EXCLUDE (&filter), OPT_REF_SORT(&sorting_options),
@@ -79,6 +81,9 @@ int cmd_for_each_ref(int argc,if (verify_ref_format (&format))usage_with_options (for_each_ref_usage, opts); +if (filter.start_after && sorting_options.nr >1 ) +die (_ ("cannot use --start-after with custom sort options" )); + sorting =ref_sorting_options (&sorting_options);ref_sorting_set_sort_flags_all (sorting,REF_SORTING_ICASE , icase); filter.ignore_case = icase;
@@ -100,6 +105,9 @@ int cmd_for_each_ref(int argc, filter.name_patterns = argv; } +if (filter.start_after && filter.name_patterns && filter.name_patterns[0 ]) +die (_ ("cannot use --start-after with patterns" )); +if (include_root_refs) flags |=FILTER_REFS_ROOT_REFS |FILTER_REFS_DETACHED_HEAD ;
ref-filter.c +60 -20
132 lines not on this page · raw message
ref-filter.h +1 -0
@@ -64,6 +64,7 @@ struct ref_array {struct ref_filter {const char **name_patterns; +const char *start_after;struct strvec exclude;struct oid_array points_at;struct commit_list *with_commit;
t/t6302-for-each-ref-filter.sh +194 -0
198 lines not on this page · raw message
Signature · 1 line
2.49.0
Re: [PATCH v3 4/4] for-each-ref: introduce a '--start-after' option
Quoted 12 lines · Karthik Nayak
The
git-for-each-ref(1)command is used to iterate over references present in a repository. In large repositories with millions of references, it would be optimal to paginate this output such that we can start iteration from a given reference. This would avoid having to iterate over all references from the beginning each time when paginating through results.The previous commit added 'seek' functionality to the reference backends. Utilize this and expose a '--start-after' option in 'git-for-each-ref(1)'. When used, the reference iteration seeks to the lexicographically next reference and iterates from there onward.
Quoted 4 lines
This enables efficient pagination workflows like:
git for-each-ref --count=100 git for-each-ref --count=100 --start-after=refs/heads/branch-100 git for-each-ref --count=100 --start-after=refs/heads/branch-200
It is a bit hard to understand how this leads to "efficient pagination" unless the reader is told what the calling script does after the first call before making the second call. It remembers the last output from the Nth call and prepares the N+1th call by using that last output entry.
But that probably belongs to the end-user facing documentation, not in the log message.
Quoted 1 line
+--start-after::
--start-after=<marker>::
Documentation/git-for-each-ref.adoc TAB %(refname)`.
Quoted 6 lines
Documentation/git-for-each-ref.adoc +6
-0+ Allows paginating the output by skipping references up to and including the + specified marker. When paging, it should be noted that references may be + deleted, modified or added between invocations. Output will only yield those + references which follow the marker lexicographically. If the marker does not + exist, output begins from the first reference that would come after it + alphabetically.
It is true that the first entry shown would be what would come immediately _after_ the given <marker>, whether the marker does or does not exist. So "If the marker does not exist, output begins..." -> "Output begins ..."
Other than that, looked pretty good to me.
Thanks, will queue.
Re: [PATCH v3 4/4] for-each-ref: introduce a '--start-after' option
Quoted 19 lines · Junio C Hamano
Quoted 12 lines · Karthik Nayak
The
git-for-each-ref(1)command is used to iterate over references present in a repository. In large repositories with millions of references, it would be optimal to paginate this output such that we can start iteration from a given reference. This would avoid having to iterate over all references from the beginning each time when paginating through results.The previous commit added 'seek' functionality to the reference backends. Utilize this and expose a '--start-after' option in 'git-for-each-ref(1)'. When used, the reference iteration seeks to the lexicographically next reference and iterates from there onward.
Quoted 4 lines
This enables efficient pagination workflows like:
git for-each-ref --count=100 git for-each-ref --count=100 --start-after=refs/heads/branch-100 git for-each-ref --count=100 --start-after=refs/heads/branch-200
Quoted 9 lines
It is a bit hard to understand how this leads to "efficient pagination" unless the reader is told what the calling script does after the first call before making the second call. It remembers the last output from the Nth call and prepares the N+1th call by using that last output entry.
But that probably belongs to the end-user facing documentation, not in the log message.
I added a small line in the commit message to clarify this. I'm not sure this belongs in the user facing documentation. Mostly I see this in the commit message to explain the intention behind adding the flag. The documentation already mentions how the flag can be used, so we should be good there.
Quoted 4 lines
Quoted 1 line
+--start-after::
--start-after=<marker>::
Oops. Thanks
Quoted 7 lines
Documentation/git-for-each-ref.adoc TAB %(refname)`.
Quoted 6 lines
Documentation/git-for-each-ref.adoc +6
-0+ Allows paginating the output by skipping references up to and including the + specified marker. When paging, it should be noted that references may be + deleted, modified or added between invocations. Output will only yield those + references which follow the marker lexicographically. If the marker does not + exist, output begins from the first reference that would come after it + alphabetically.
Quoted 7 lines
It is true that the first entry shown would be what would come immediately _after_ the given <marker>, whether the marker does or does not exist. So "If the marker does not exist, output begins..." -> "Output begins ..."
Other than that, looked pretty good to me.
Good point, will amend this.
Quoted 1 line
Thanks, will queue.
Thank you for the review. I'll add the changes locally and push a new version after a day or two.
v4 cover for-each-ref: introduce seeking functionality via '--start-after'
In reply to Karthik NayakThe git-for-each-ref(1) command is used to iterate over references present in a repository. In large repositories with millions of references, it would be optimal to paginate this output such that we can start iteration from a given reference. This would avoid having to iterate over all references from the beginning each time when paginating through results.
This series adds a '--start-after' option in 'git-for-each-ref(1)'. When used, the reference iteration seeks to first reference following the marker alphabetically. When paging, it should be noted that references may be deleted, modified or added between invocations. Output will only yield those references which follow the marker lexicographically. If the marker does not exist, output begins from the first reference that would come after it alphabetically.
This enables efficient pagination workflows like:
git for-each-ref --count=100
git for-each-ref --count=100 --start-after=refs/heads/branch-100
git for-each-ref --count=100 --start-after=refs/heads/branch-200To add this functionality, we expose the ref_iterator outside the 'refs/' namespace and modify the ref_iterator_seek() to actually seek to a given reference and only set the prefix when the set_prefix field is set.
On the reftable and packed backend, the changes are simple. But since the files backend uses 'ref-cache' for reference handling, the changes there are a little more involved, since we need to setup the right levels and the indexing.
Initially I was also planning to cleanup all the refs_for_each...() functions in 'refs.h' by simply using the iterator, but this bloated the series. So I've left that for another day.
Changes in v4:
- Patch 3/4: Move around the documentation for the flag and rename the
seek variable to refname.
- Patch 4/4: Cleanup the commit message and also the documentation.
- Link to v3: https://lore.kernel.org/r/20250708-306-git-for-each-ref-pagination-v3-0-8cfba1080be4@gmail.com
Changes in v3:
- Change the working of the command to exclude the marker provided. With
this rename the flag to '--start-after'.
- Extend the documentation to add a note about concurrent modifications
to the reference database.
- Link to v2: https://lore.kernel.org/r/20250704-306-git-for-each-ref-pagination-v2-0-bcde14acdd81@gmail.com
Changes in v2:
- Modify 'ref_iterator_seek()' to take in flags instead of a
'set_prefix' variable. This improves readability, where users would
use the 'REF_ITERATOR_SEEK_SET_PREFIX' instead of simply passing '1'.
- When the set prefix flag isn't usage, reset any previously set prefix.
This ensures that the internal prefix state is always reset whenever
we seek and unifies the behavior between 'ref_iterator_seek' and
'ref_iterator_begin'.
- Don't allow '--skip-until' to be run with '--sort', since the seeking
always takes place before any sorting and this can be confusing.
- Some styling fixes:
- Remove extra newline
- Skip braces around single lined if...else clause
- Add braces around 'if' clause
- Fix indentation
- Link to v1: https://lore.kernel.org/git/20250701-306-git-for-each-ref-pagination-v1-0-4f0ae7c0688f@gmail.com/
--- Documentation/git-for-each-ref.adoc | 10 +- builtin/for-each-ref.c | 8 ++ ref-filter.c | 80 +++++++++++---- ref-filter.h | 1 + refs.c | 6 +- refs.h | 155 ++++++++++++++++++++++++++++ refs/debug.c | 7 +- refs/files-backend.c | 7 +- refs/iterator.c | 26 +++-- refs/packed-backend.c | 17 ++-- refs/ref-cache.c | 99 ++++++++++++++---- refs/ref-cache.h | 7 -- refs/refs-internal.h | 152 ++-------------------------- refs/reftable-backend.c | 21 ++-- t/t6302-for-each-ref-filter.sh | 194 ++++++++++++++++++++++++++++++++++++ 15 files changed, 564 insertions(+), 226 deletions(-)
Karthik Nayak (4):
refs: expose ref_iterator via 'refs.h'
ref-cache: remove unused function 'find_ref_entry()'
refs: selectively set prefix in the seek functions
for-each-ref: introduce a '--start-after' option
Range-diff versus v3:
1: eed39162f5 = 1: 9e6ecff291 refs: expose `ref_iterator` via 'refs.h'
2: b9db49d31b = 2: 22f5222e4f ref-cache: remove unused function 'find_ref_entry()'
3: 502e2696fd ! 3: 0e71d8ffd9 refs: selectively set prefix in the seek functions
@@ refs.h: struct ref_iterator *refs_ref_iterator_begin(
+enum ref_iterator_seek_flag {
+ /*
-+ * Also set the seek pattern as a prefix for iteration. This ensures
-+ * that only references which match the prefix are yielded.
++ * When the REF_ITERATOR_SEEK_SET_PREFIX flag is set, the iterator's prefix is
++ * updated to match the provided string, affecting all subsequent iterations. If
++ * not, the iterator seeks to the specified reference and clears any previously
++ * set prefix.
+ */
+ REF_ITERATOR_SEEK_SET_PREFIX = (1 << 0),
+};
@@ refs.h: struct ref_iterator *refs_ref_iterator_begin(
- * passed when creating the iterator will remain unchanged.
+ * This function is expected to behave as if a new ref iterator has been
+ * created, but allows reuse of existing iterators for optimization.
-+ *
-+ * When the REF_ITERATOR_SEEK_SET_PREFIX flag is set, the iterator's prefix is
-+ * updated to match the seek string, affecting all subsequent iterations. If
-+ * not, the iterator seeks to the specified reference and clears any previously
-+ * set prefix.
*
* Returns 0 on success, a negative error code otherwise.
*/
-int ref_iterator_seek(struct ref_iterator *ref_iterator,
- const char *prefix);
-+int ref_iterator_seek(struct ref_iterator *ref_iterator, const char *seek,
++int ref_iterator_seek(struct ref_iterator *ref_iterator, const char *refname,
+ unsigned int flags);
/*
@@ refs/debug.c: static int debug_ref_iterator_advance(struct ref_iterator *ref_ite
static int debug_ref_iterator_seek(struct ref_iterator *ref_iterator,
- const char *prefix)
-+ const char *seek, unsigned int flags)
++ const char *refname, unsigned int flags)
{
struct debug_ref_iterator *diter =
(struct debug_ref_iterator *)ref_iterator;
- int res = diter->iter->vtable->seek(diter->iter, prefix);
- trace_printf_key(&trace_refs, "iterator_seek: %s: %d\n", prefix ? prefix : "", res);
-+ int res = diter->iter->vtable->seek(diter->iter, seek, flags);
++ int res = diter->iter->vtable->seek(diter->iter, refname, flags);
+ trace_printf_key(&trace_refs, "iterator_seek: %s flags: %d: %d\n",
-+ seek ? seek : "", flags, res);
++ refname ? refname : "", flags, res);
return res;
}
@@ refs/files-backend.c: static int files_ref_iterator_advance(struct ref_iterator
static int files_ref_iterator_seek(struct ref_iterator *ref_iterator,
- const char *prefix)
-+ const char *seek, unsigned int flags)
++ const char *refname, unsigned int flags)
{
struct files_ref_iterator *iter =
(struct files_ref_iterator *)ref_iterator;
- return ref_iterator_seek(iter->iter0, prefix);
-+ return ref_iterator_seek(iter->iter0, seek, flags);
++ return ref_iterator_seek(iter->iter0, refname, flags);
}
static int files_ref_iterator_peel(struct ref_iterator *ref_iterator,
@@ refs/files-backend.c: static int files_reflog_iterator_advance(struct ref_iterat
static int files_reflog_iterator_seek(struct ref_iterator *ref_iterator UNUSED,
- const char *prefix UNUSED)
-+ const char *seek UNUSED,
++ const char *refname UNUSED,
+ unsigned int flags UNUSED)
{
BUG("ref_iterator_seek() called for reflog_iterator");
@@ refs/iterator.c: int ref_iterator_advance(struct ref_iterator *ref_iterator)
-int ref_iterator_seek(struct ref_iterator *ref_iterator,
- const char *prefix)
-+int ref_iterator_seek(struct ref_iterator *ref_iterator, const char *seek,
++int ref_iterator_seek(struct ref_iterator *ref_iterator, const char *refname,
+ unsigned int flags)
{
- return ref_iterator->vtable->seek(ref_iterator, prefix);
-+ return ref_iterator->vtable->seek(ref_iterator, seek, flags);
++ return ref_iterator->vtable->seek(ref_iterator, refname, flags);
}
int ref_iterator_peel(struct ref_iterator *ref_iterator,
@@ refs/iterator.c: static int empty_ref_iterator_advance(struct ref_iterator *ref_
static int empty_ref_iterator_seek(struct ref_iterator *ref_iterator UNUSED,
- const char *prefix UNUSED)
-+ const char *seek UNUSED,
++ const char *refname UNUSED,
+ unsigned int flags UNUSED)
{
return 0;
@@ refs/iterator.c: static int merge_ref_iterator_advance(struct ref_iterator *ref_
static int merge_ref_iterator_seek(struct ref_iterator *ref_iterator,
- const char *prefix)
-+ const char *seek, unsigned int flags)
++ const char *refname, unsigned int flags)
{
struct merge_ref_iterator *iter =
(struct merge_ref_iterator *)ref_iterator;
@@ refs/iterator.c: static int merge_ref_iterator_seek(struct ref_iterator *ref_ite
iter->iter1 = iter->iter1_owned;
- ret = ref_iterator_seek(iter->iter0, prefix);
-+ ret = ref_iterator_seek(iter->iter0, seek, flags);
++ ret = ref_iterator_seek(iter->iter0, refname, flags);
if (ret < 0)
return ret;
- ret = ref_iterator_seek(iter->iter1, prefix);
-+ ret = ref_iterator_seek(iter->iter1, seek, flags);
++ ret = ref_iterator_seek(iter->iter1, refname, flags);
if (ret < 0)
return ret;
@@ refs/iterator.c: static int prefix_ref_iterator_advance(struct ref_iterator *ref
static int prefix_ref_iterator_seek(struct ref_iterator *ref_iterator,
- const char *prefix)
-+ const char *seek, unsigned int flags)
++ const char *refname, unsigned int flags)
{
struct prefix_ref_iterator *iter =
(struct prefix_ref_iterator *)ref_iterator;
@@ refs/iterator.c: static int prefix_ref_iterator_advance(struct ref_iterator *ref
+
+ if (flags & REF_ITERATOR_SEEK_SET_PREFIX) {
+ free(iter->prefix);
-+ iter->prefix = xstrdup_or_null(seek);
++ iter->prefix = xstrdup_or_null(refname);
+ }
-+ return ref_iterator_seek(iter->iter0, seek, flags);
++ return ref_iterator_seek(iter->iter0, refname, flags);
}
static int prefix_ref_iterator_peel(struct ref_iterator *ref_iterator,
@@ refs/packed-backend.c: static int packed_ref_iterator_advance(struct ref_iterato
static int packed_ref_iterator_seek(struct ref_iterator *ref_iterator,
- const char *prefix)
-+ const char *seek, unsigned int flags)
++ const char *refname, unsigned int flags)
{
struct packed_ref_iterator *iter =
(struct packed_ref_iterator *)ref_iterator;
@@ refs/packed-backend.c: static int packed_ref_iterator_advance(struct ref_iterato
- if (prefix && *prefix)
- start = find_reference_location(iter->snapshot, prefix, 0);
-+ if (seek && *seek)
-+ start = find_reference_location(iter->snapshot, seek, 0);
++ if (refname && *refname)
++ start = find_reference_location(iter->snapshot, refname, 0);
else
start = iter->snapshot->start;
@@ refs/packed-backend.c: static int packed_ref_iterator_advance(struct ref_iterato
+ FREE_AND_NULL(iter->prefix);
+
+ if (flags & REF_ITERATOR_SEEK_SET_PREFIX)
-+ iter->prefix = xstrdup_or_null(seek);
++ iter->prefix = xstrdup_or_null(refname);
+
iter->pos = start;
iter->eof = iter->snapshot->eof;
@@ refs/ref-cache.c: static int cache_ref_iterator_seek(struct ref_iterator *ref_it
}
+static int cache_ref_iterator_seek(struct ref_iterator *ref_iterator,
-+ const char *seek, unsigned int flags)
++ const char *refname, unsigned int flags)
+{
+ struct cache_ref_iterator *iter =
+ (struct cache_ref_iterator *)ref_iterator;
+
+ if (flags & REF_ITERATOR_SEEK_SET_PREFIX) {
-+ return cache_ref_iterator_set_prefix(iter, seek);
-+ } else if (seek && *seek) {
++ return cache_ref_iterator_set_prefix(iter, refname);
++ } else if (refname && *refname) {
+ struct cache_ref_iterator_level *level;
-+ const char *slash = seek;
++ const char *slash = refname;
+ struct ref_dir *dir;
+
+ dir = get_ref_dir(iter->cache->root);
+
+ if (iter->prime_dir)
-+ prime_ref_dir(dir, seek);
++ prime_ref_dir(dir, refname);
+
+ iter->levels_nr = 1;
+ level = &iter->levels[0];
@@ refs/ref-cache.c: static int cache_ref_iterator_seek(struct ref_iterator *ref_it
+ sort_ref_dir(dir);
+
+ slash = strchr(slash, '/');
-+ len = slash ? slash - seek : (int)strlen(seek);
++ len = slash ? slash - refname : (int)strlen(refname);
+
+ for (idx = 0; idx < dir->nr; idx++) {
-+ cmp = strncmp(seek, dir->entries[idx]->name, len);
++ cmp = strncmp(refname, dir->entries[idx]->name, len);
+ if (cmp <= 0)
+ break;
+ }
@@ refs/refs-internal.h: void base_ref_iterator_init(struct ref_iterator *iter,
*/
typedef int ref_iterator_seek_fn(struct ref_iterator *ref_iterator,
- const char *prefix);
-+ const char *seek, unsigned int flags);
++ const char *refname, unsigned int flags);
/*
* Peels the current ref, returning 0 for success or -1 for failure.
@@ refs/reftable-backend.c: static int reftable_ref_iterator_advance(struct ref_ite
static int reftable_ref_iterator_seek(struct ref_iterator *ref_iterator,
- const char *prefix)
-+ const char *seek, unsigned int flags)
++ const char *refname, unsigned int flags)
{
struct reftable_ref_iterator *iter =
(struct reftable_ref_iterator *)ref_iterator;
@@ refs/reftable-backend.c: static int reftable_ref_iterator_advance(struct ref_ite
+ iter->prefix_len = 0;
+
+ if (flags & REF_ITERATOR_SEEK_SET_PREFIX) {
-+ iter->prefix = xstrdup_or_null(seek);
-+ iter->prefix_len = seek ? strlen(seek) : 0;
++ iter->prefix = xstrdup_or_null(refname);
++ iter->prefix_len = refname ? strlen(refname) : 0;
+ }
-+ iter->err = reftable_iterator_seek_ref(&iter->iter, seek);
++ iter->err = reftable_iterator_seek_ref(&iter->iter, refname);
return iter->err;
}
@@ refs/reftable-backend.c: static int reftable_reflog_iterator_advance(struct ref_
static int reftable_reflog_iterator_seek(struct ref_iterator *ref_iterator UNUSED,
- const char *prefix UNUSED)
-+ const char *seek UNUSED,
++ const char *refname UNUSED,
+ unsigned int flags UNUSED)
{
BUG("reftable reflog iterator cannot be seeked");4: a571579886 ! 4: e4e9dddd15 for-each-ref: introduce a '--start-after' option
@@ Commit message
'git-for-each-ref(1)'. When used, the reference iteration seeks to the lexicographically next reference and iterates from there onward.
- This enables efficient pagination workflows like:
+ This enables efficient pagination workflows, where the calling script
+ can remember the last provided reference and use that as the starting
+ point for the next set of references:
git for-each-ref --count=100
git for-each-ref --count=100 --start-after=refs/heads/branch-100 git for-each-ref --count=100 --start-after=refs/heads/branch-200
@@ Documentation/git-for-each-ref.adoc: TAB %(refname)`.
--include-root-refs::
List root refs (HEAD and pseudorefs) apart from regular refs.
-+--start-after::
++--start-after=<marker>::
+ Allows paginating the output by skipping references up to and including the
+ specified marker. When paging, it should be noted that references may be
+ deleted, modified or added between invocations. Output will only yield those
-+ references which follow the marker lexicographically. If the marker does not
-+ exist, output begins from the first reference that would come after it
-+ alphabetically. Cannot be used with general pattern matching or custom
-+ sort options.
++ references which follow the marker lexicographically. Output begins from the
++ first reference that would come after the marker alphabetically. Cannot be
++ used with general pattern matching or custom sort options.
+
FIELD NAMES
-----------
base-commit: cf6f63ea6bf35173e02e18bdc6a4ba41288acff9
change-id: 20250605-306-git-for-each-ref-pagination-0ba8a29ae646
Thanks
- Karthik
PATCH v4 1/4 refs: expose `ref_iterator` via 'refs.h'
The ref_iterator is an internal structure to the 'refs/' sub-directory, which allows iteration over refs. All reference iteration is built on top of these iterators.
External clients of the 'refs' subsystem use the various 'refs_for_each...()' functions to iterate over refs. However since these are wrapper functions, each combination of functionality requires a new wrapper function. This is not feasible as the functions pile up with the increase in requirements. Expose the internal reference iterator, so advanced users can mix and match options as needed.
--- refs.h | 147 +++++++++++++++++++++++++++++++++++++++++++++++++++ refs/refs-internal.h | 145 +------------------------------------------------- 2 files changed, 149 insertions(+), 143 deletions(-)
refs.h +147 -0
@@ -1190,4 +1190,151 @@ int repo_migrate_ref_storage_format(struct repository *repo,unsigned int flags, struct strbuf *err); +/* +* Reference iterators +* +* A reference iterator encapsulates the state of an in-progress +* iteration over references. Create an instance of `struct +* ref_iterator` via one of the functions in this module. +* +* A freshly-created ref_iterator doesn't yet point at a reference. To +* advance the iterator, call ref_iterator_advance(). If successful, +* this sets the iterator's refname, oid, and flags fields to describe +* the next reference and returns ITER_OK. The data pointed at by +* refname and oid belong to the iterator; if you want to retain them +* after calling ref_iterator_advance() again or calling +* ref_iterator_free(), you must make a copy. When the iteration has +* been exhausted, ref_iterator_advance() releases any resources +* associated with the iteration, frees the ref_iterator object, and +* returns ITER_DONE. If you want to abort the iteration early, call +* ref_iterator_free(), which also frees the ref_iterator object and +* any associated resources. If there was an internal error advancing +* to the next entry, ref_iterator_advance() aborts the iteration, +* frees the ref_iterator, and returns ITER_ERROR. +* +* The reference currently being looked at can be peeled by calling +* ref_iterator_peel(). This function is often faster than peel_ref(), +* so it should be preferred when iterating over references. +* +* Putting it all together, a typical iteration looks like this: +* +* int ok; +* struct ref_iterator *iter = ...; +* +* while ((ok = ref_iterator_advance(iter)) == ITER_OK) { +* if (want_to_stop_iteration()) { +* ok = ITER_DONE; +* break; +* } +* +* // Access information about the current reference: +* if (!(iter->flags & REF_ISSYMREF)) +* printf("%s is %s\n", iter->refname, oid_to_hex(iter->oid)); +* +* // If you need to peel the reference: +* ref_iterator_peel(iter, &oid); +* } +* +* if (ok != ITER_DONE) +* handle_error(); +* ref_iterator_free(iter); +*/ +struct ref_iterator ; + +/* +* These flags are passed to refs_ref_iterator_begin() (and do_for_each_ref(), +* which feeds it). +*/ +enum do_for_each_ref_flags { +/* +* Include broken references in a do_for_each_ref*() iteration, which +* would normally be omitted. This includes both refs that point to +* missing objects (a true repository corruption), ones with illegal +* names (which we prefer not to expose to callers), as well as +* dangling symbolic refs (i.e., those that point to a non-existent +* ref; this is not a corruption, but as they have no valid oid, we +* omit them from normal iteration results). +*/ +DO_FOR_EACH_INCLUDE_BROKEN = (1 <<0 ), + +/* +* Only include per-worktree refs in a do_for_each_ref*() iteration. +* Normally this will be used with a files ref_store, since that's +* where all reference backends will presumably store their +* per-worktree refs. +*/ +DO_FOR_EACH_PER_WORKTREE_ONLY = (1 <<1 ), + +/* +* Omit dangling symrefs from output; this only has an effect with +* INCLUDE_BROKEN, since they are otherwise not included at all. +*/ +DO_FOR_EACH_OMIT_DANGLING_SYMREFS = (1 <<2 ), + +/* +* Include root refs i.e. HEAD and pseudorefs along with the regular +* refs. +*/ +DO_FOR_EACH_INCLUDE_ROOT_REFS = (1 <<3 ), +}; + +/* +* Return an iterator that goes over each reference in `refs` for +* which the refname begins with prefix. If trim is non-zero, then +* trim that many characters off the beginning of each refname. +* The output is ordered by refname. +*/ +struct ref_iterator *refs_ref_iterator_begin ( +struct ref_store *refs, +const char *prefix,const char **exclude_patterns, +int trim,enum do_for_each_ref_flags flags); + +/* +* Advance the iterator to the first or next item and return ITER_OK. +* If the iteration is exhausted, free the resources associated with +* the ref_iterator and return ITER_DONE. On errors, free the iterator +* resources and return ITER_ERROR. It is a bug to use ref_iterator or +* call this function again after it has returned ITER_DONE or +* ITER_ERROR. +*/ +int ref_iterator_advance (struct ref_iterator *ref_iterator); + +/* +* Seek the iterator to the first reference with the given prefix. +* The prefix is matched as a literal string, without regard for path +* separators. If prefix is NULL or the empty string, seek the iterator to the +* first reference again. +* +* This function is expected to behave as if a new ref iterator with the same +* prefix had been created, but allows reuse of iterators and thus may allow +* the backend to optimize. Parameters other than the prefix that have been +* passed when creating the iterator will remain unchanged. +* +* Returns 0 on success, a negative error code otherwise. +*/ +int ref_iterator_seek (struct ref_iterator *ref_iterator, +const char *prefix); + +/* +* If possible, peel the reference currently being viewed by the +* iterator. Return 0 on success. +*/ +int ref_iterator_peel (struct ref_iterator *ref_iterator, +struct object_id *peeled); + +/* Free the reference iterator and any associated resources. */ +void ref_iterator_free (struct ref_iterator *ref_iterator); + +/* +* The common backend for the for_each_*ref* functions. Call fn for +* each reference in iter. If the iterator itself ever returns +* ITER_ERROR, return -1. If fn ever returns a non-zero value, stop +* the iteration and return that value. Otherwise, return 0. In any +* case, free the iterator when done. This function is basically an +* adapter between the callback style of reference iteration and the +* iterator style. +*/ +int do_for_each_ref_iterator (struct ref_iterator *iter, +each_ref_fn fn,void *cb_data); +#endif /* REFS_H */
refs/refs-internal.h +2 -143
169 lines not on this page · raw message
Signature · 1 line
2.49.0
PATCH v4 2/4 ref-cache: remove unused function 'find_ref_entry()'
In reply to Karthik NayakThe 'find_ref_entry' function is no longer used, so remove it.
--- refs/ref-cache.c | 14 -------------- refs/ref-cache.h | 7 ------- 2 files changed, 21 deletions(-)
refs/ref-cache.c +0 -14
@@ -194,20 +194,6 @@ static struct ref_dir *find_containing_dir(struct ref_dir *dir,return dir; } -struct ref_entry *find_ref_entry (struct ref_dir *dir,const char *refname) -{ -int entry_index; -struct ref_entry *entry; - dir =find_containing_dir (dir, refname); -if (!dir) -return NULL ; - entry_index =search_ref_dir (dir, refname,strlen (refname)); -if (entry_index ==-1 ) -return NULL ; - entry = dir->entries[entry_index]; -return (entry->flag &REF_DIR ) ?NULL : entry; -} -/* * Emit a warning and return true iff ref1 and ref2 have the same name * and the same oid. Die if they have the same name but different
refs/ref-cache.h +0 -7
@@ -201,13 +201,6 @@ void free_ref_cache(struct ref_cache *cache);*/ void add_entry_to_dir (struct ref_dir *dir,struct ref_entry *entry); -/* -* Find the value entry with the given name in dir, sorting ref_dirs -* and recursing into subdirectories as necessary. If the name is not -* found or it corresponds to a directory entry, return NULL. -*/ -struct ref_entry *find_ref_entry (struct ref_dir *dir,const char *refname); -/* * Start iterating over references in `cache`. If `prefix` is * specified, only include references whose names start with that
Signature · 1 line
2.49.0
PATCH v4 3/4 refs: selectively set prefix in the seek functions
In reply to Karthik NayakThe ref iterator exposes a ref_iterator_seek() function. The name suggests that this would seek the iterator to a specific reference in some ways similar to how fseek() works for the filesystem.
However, the function actually sets the prefix for refs iteration. So further iteration would only yield references which match the particular prefix. This is a bit confusing.
Let's add a 'flags' field to the function, which when set with the 'REF_ITERATOR_SEEK_SET_PREFIX' flag, will set the prefix for the iteration in-line with the existing behavior. Otherwise, the reference backends will simply seek to the specified reference and clears any previously set prefix. This allows users to start iteration from a specific reference.
In the packed and reftable backend, since references are available in a sorted list, the changes are simply setting the prefix if needed. The changes on the files-backend are a little more involved, since the files backend uses the 'ref-cache' mechanism. We move out the existing logic within cache_ref_iterator_seek() to cache_ref_iterator_set_prefix() which is called when the 'REF_ITERATOR_SEEK_SET_PREFIX' flag is set. We then parse the provided seek string and set the required levels and their indexes to ensure that seeking is possible.
--- refs.c | 6 ++-- refs.h | 26 +++++++++------ refs/debug.c | 7 ++-- refs/files-backend.c | 7 ++-- refs/iterator.c | 26 ++++++++------- refs/packed-backend.c | 17 ++++++---- refs/ref-cache.c | 85 ++++++++++++++++++++++++++++++++++++++++++++++--- refs/refs-internal.h | 7 ++-- refs/reftable-backend.c | 21 ++++++++---- 9 files changed, 152 insertions(+), 50 deletions(-)
refs.c +3 -3
@@ -2666,12 +2666,12 @@ enum ref_transaction_error refs_verify_refnames_available(struct ref_store *refsif (!initial_transaction) {int ok; -if (!iter) { +if (!iter) iter =refs_ref_iterator_begin (refs, dirname.buf,NULL ,0 ,DO_FOR_EACH_INCLUDE_BROKEN ); - }else if (ref_iterator_seek (iter, dirname.buf) <0 ) { +else if (ref_iterator_seek (iter, dirname.buf, +REF_ITERATOR_SEEK_SET_PREFIX ) <0 ) goto cleanup; - }while ((ok =ref_iterator_advance (iter)) ==ITER_OK ) { if (skip &&
refs.h +17 -9
@@ -1299,21 +1299,29 @@ struct ref_iterator *refs_ref_iterator_begin(*/ int ref_iterator_advance (struct ref_iterator *ref_iterator); +enum ref_iterator_seek_flag { +/* +* When the REF_ITERATOR_SEEK_SET_PREFIX flag is set, the iterator's prefix is +* updated to match the provided string, affecting all subsequent iterations. If +* not, the iterator seeks to the specified reference and clears any previously +* set prefix. +*/ +REF_ITERATOR_SEEK_SET_PREFIX = (1 <<0 ), +}; +/* -* Seek the iterator to the first reference with the given prefix. -* The prefix is matched as a literal string, without regard for path -* separators. If prefix is NULL or the empty string, seek the iterator to the +* Seek the iterator to the first reference matching the given seek string. +* The seek string is matched as a literal string, without regard for path +* separators. If seek is NULL or the empty string, seek the iterator to the * first reference again. * -* This function is expected to behave as if a new ref iterator with the same -* prefix had been created, but allows reuse of iterators and thus may allow -* the backend to optimize. Parameters other than the prefix that have been -* passed when creating the iterator will remain unchanged. +* This function is expected to behave as if a new ref iterator has been +* created, but allows reuse of existing iterators for optimization. * * Returns 0 on success, a negative error code otherwise. */ -int ref_iterator_seek (struct ref_iterator *ref_iterator, -const char *prefix); +int ref_iterator_seek (struct ref_iterator *ref_iterator,const char *refname, +unsigned int flags);/* * If possible, peel the reference currently being viewed by the
refs/debug.c +4 -3
@@ -170,12 +170,13 @@ static int debug_ref_iterator_advance(struct ref_iterator *ref_iterator) }static int debug_ref_iterator_seek (struct ref_iterator *ref_iterator, -const char *prefix) +const char *refname,unsigned int flags) {struct debug_ref_iterator *diter = (struct debug_ref_iterator *)ref_iterator; -int res = diter->iter->vtable->seek (diter->iter, prefix); -trace_printf_key (&trace_refs,"iterator_seek: %s: %d\n" , prefix ? prefix :"" , res); +int res = diter->iter->vtable->seek (diter->iter, refname, flags); +trace_printf_key (&trace_refs,"iterator_seek: %s flags: %d: %d\n" , + refname ? refname :"" , flags, res);return res; }
refs/files-backend.c +4 -3
@@ -929,11 +929,11 @@ static int files_ref_iterator_advance(struct ref_iterator *ref_iterator) }static int files_ref_iterator_seek (struct ref_iterator *ref_iterator, -const char *prefix) +const char *refname,unsigned int flags) {struct files_ref_iterator *iter = (struct files_ref_iterator *)ref_iterator; -return ref_iterator_seek (iter->iter0, prefix); +return ref_iterator_seek (iter->iter0, refname, flags); } static int files_ref_iterator_peel(struct ref_iterator *ref_iterator,
@@ -2316,7 +2316,8 @@ static int files_reflog_iterator_advance(struct ref_iterator *ref_iterator) } static int files_reflog_iterator_seek(struct ref_iterator *ref_iterator UNUSED, - const char *prefix UNUSED) + const char *refname UNUSED, + unsigned int flags UNUSED) {BUG ("ref_iterator_seek() called for reflog_iterator" ); }
refs/iterator.c +15 -11
@@ -15,10 +15,10 @@ int ref_iterator_advance(struct ref_iterator *ref_iterator)return ref_iterator->vtable->advance (ref_iterator); } -int ref_iterator_seek (struct ref_iterator *ref_iterator, -const char *prefix) +int ref_iterator_seek (struct ref_iterator *ref_iterator,const char *refname, +unsigned int flags) { -return ref_iterator->vtable->seek (ref_iterator, prefix); +return ref_iterator->vtable->seek (ref_iterator, refname, flags); } int ref_iterator_peel(struct ref_iterator *ref_iterator,
@@ -57,7 +57,8 @@ static int empty_ref_iterator_advance(struct ref_iterator *ref_iterator UNUSED) } static int empty_ref_iterator_seek(struct ref_iterator *ref_iterator UNUSED, - const char *prefix UNUSED) + const char *refname UNUSED, + unsigned int flags UNUSED) {return 0 ; }
@@ -224,7 +225,7 @@ static int merge_ref_iterator_advance(struct ref_iterator *ref_iterator) }static int merge_ref_iterator_seek (struct ref_iterator *ref_iterator, -const char *prefix) +const char *refname,unsigned int flags) {struct merge_ref_iterator *iter = (struct merge_ref_iterator *)ref_iterator;
@@ -234,11 +235,11 @@ static int merge_ref_iterator_seek(struct ref_iterator *ref_iterator, iter->iter0 = iter->iter0_owned; iter->iter1 = iter->iter1_owned; - ret =ref_iterator_seek (iter->iter0, prefix); + ret =ref_iterator_seek (iter->iter0, refname, flags);if (ret <0 )return ret; - ret =ref_iterator_seek (iter->iter1, prefix); + ret =ref_iterator_seek (iter->iter1, refname, flags);if (ret <0 )return ret;
@@ -407,13 +408,16 @@ static int prefix_ref_iterator_advance(struct ref_iterator *ref_iterator) }static int prefix_ref_iterator_seek (struct ref_iterator *ref_iterator, -const char *prefix) +const char *refname,unsigned int flags) {struct prefix_ref_iterator *iter = (struct prefix_ref_iterator *)ref_iterator; -free (iter->prefix); - iter->prefix =xstrdup_or_null (prefix); -return ref_iterator_seek (iter->iter0, prefix); + +if (flags &REF_ITERATOR_SEEK_SET_PREFIX ) { +free (iter->prefix); + iter->prefix =xstrdup_or_null (refname); + } +return ref_iterator_seek (iter->iter0, refname, flags); } static int prefix_ref_iterator_peel(struct ref_iterator *ref_iterator,
refs/packed-backend.c +11 -6
@@ -1004,19 +1004,23 @@ static int packed_ref_iterator_advance(struct ref_iterator *ref_iterator) }static int packed_ref_iterator_seek (struct ref_iterator *ref_iterator, -const char *prefix) +const char *refname,unsigned int flags) {struct packed_ref_iterator *iter = (struct packed_ref_iterator *)ref_iterator;const char *start; -if (prefix && *prefix) - start =find_reference_location (iter->snapshot, prefix,0 ); +if (refname && *refname) + start =find_reference_location (iter->snapshot, refname,0 );else start = iter->snapshot->start; -free (iter->prefix); - iter->prefix =xstrdup_or_null (prefix); +/* Unset any previously set prefix */ +FREE_AND_NULL (iter->prefix); + +if (flags &REF_ITERATOR_SEEK_SET_PREFIX ) + iter->prefix =xstrdup_or_null (refname); + iter->pos = start; iter->eof = iter->snapshot->eof;
@@ -1194,7 +1198,8 @@ static struct ref_iterator *packed_ref_iterator_begin( iter->repo = ref_store->repo; iter->flags = flags; -if (packed_ref_iterator_seek (&iter->base, prefix) <0 ) { +if (packed_ref_iterator_seek (&iter->base, prefix, +REF_ITERATOR_SEEK_SET_PREFIX ) <0 ) {ref_iterator_free (&iter->base);return NULL ; }
refs/ref-cache.c +80 -5
104 lines not on this page · raw message
refs/refs-internal.h +4 -3
@@ -353,11 +353,12 @@ void base_ref_iterator_init(struct ref_iterator *iter,typedef int ref_iterator_advance_fn (struct ref_iterator *ref_iterator);/* -* Seek the iterator to the first reference matching the given prefix. Should -* behave the same as if a new iterator was created with the same prefix. +* Seek the iterator to the first matching reference. If set_prefix is set, +* it would behave the same as if a new iterator was created with the same +* prefix. */ typedef int ref_iterator_seek_fn (struct ref_iterator *ref_iterator, -const char *prefix); +const char *refname,unsigned int flags);/* * Peels the current ref, returning 0 for success or -1 for failure.
refs/reftable-backend.c +14 -7
@@ -719,15 +719,20 @@ static int reftable_ref_iterator_advance(struct ref_iterator *ref_iterator) }static int reftable_ref_iterator_seek (struct ref_iterator *ref_iterator, -const char *prefix) +const char *refname,unsigned int flags) {struct reftable_ref_iterator *iter = (struct reftable_ref_iterator *)ref_iterator; -free (iter->prefix); - iter->prefix =xstrdup_or_null (prefix); - iter->prefix_len = prefix ?strlen (prefix) :0 ; - iter->err =reftable_iterator_seek_ref (&iter->iter, prefix); +/* Unset any previously set prefix */ +FREE_AND_NULL (iter->prefix); + iter->prefix_len =0 ; + +if (flags &REF_ITERATOR_SEEK_SET_PREFIX ) { + iter->prefix =xstrdup_or_null (refname); + iter->prefix_len = refname ?strlen (refname) :0 ; + } + iter->err =reftable_iterator_seek_ref (&iter->iter, refname);return iter->err; }
@@ -839,7 +844,8 @@ static struct reftable_ref_iterator *ref_iterator_for_stack(struct reftable_ref_if (ret) goto done; - ret =reftable_ref_iterator_seek (&iter->base, prefix); + ret =reftable_ref_iterator_seek (&iter->base, prefix, +REF_ITERATOR_SEEK_SET_PREFIX );if (ret) goto done;
@@ -2042,7 +2048,8 @@ static int reftable_reflog_iterator_advance(struct ref_iterator *ref_iterator) } static int reftable_reflog_iterator_seek(struct ref_iterator *ref_iterator UNUSED, - const char *prefix UNUSED) + const char *refname UNUSED, + unsigned int flags UNUSED) {BUG ("reftable reflog iterator cannot be seeked" );return -1 ;
Signature · 1 line
2.49.0
Re: [PATCH v4 3/4] refs: selectively set prefix in the seek functions
refs/refs-internal.h void base_ref_iterator_init(struct ref_iterator *iter, · line 356
Quoted 11 lines · Karthik Nayak
refs/refs-internal.h +1
-2@@ -353,11 +353,12 @@ void base_ref_iterator_init(struct ref_iterator *iter,typedef int ref_iterator_advance_fn (struct ref_iterator *ref_iterator);/* -* Seek the iterator to the first reference matching the given prefix. Should -* behave the same as if a new iterator was created with the same prefix. +* Seek the iterator to the first matching reference. If set_prefix is set,
s/If set_prefix is set/If the REF_ITERATOR_SEEK_SET_PREFIX flag is set/
refs/refs-internal.h void base_ref_iterator_init(struct ref_iterator *iter, · line 358
Quoted 2 lines
refs/refs-internal.h +2
-0+* it would behave the same as if a new iterator was created with the same +* prefix.
Maybe: s/with the same prefix/at the same reference/
Quoted 4 lines
refs/refs-internal.h +1
-1*/ typedef int ref_iterator_seek_fn (struct ref_iterator *ref_iterator, -const char *prefix); +const char *refname,unsigned int flags);
Re: [PATCH v4 3/4] refs: selectively set prefix in the seek functions
Quoted 9 lines · Christian Couder
Quoted 7 lines · Karthik Nayak
refs/refs-internal.h +0
-0@@ -353,11 +353,12 @@ void base_ref_iterator_init(struct ref_iterator *iter,typedef int ref_iterator_advance_fn (struct ref_iterator *ref_iterator);
Quoted 7 lines
refs/refs-internal.h void base_ref_iterator_init(struct ref_iterator *iter, · line 356
Quoted 4 lines
refs/refs-internal.h +1
-2/* -* Seek the iterator to the first reference matching the given prefix. Should -* behave the same as if a new iterator was created with the same prefix. +* Seek the iterator to the first matching reference. If set_prefix is set, s/If
set_prefixis set/If theREF_ITERATOR_SEEK_SET_PREFIXflag is set/
Will change, thanks.
Quoted 5 lines
refs/refs-internal.h void base_ref_iterator_init(struct ref_iterator *iter, · line 358
Quoted 2 lines
refs/refs-internal.h +2
-0+* it would behave the same as if a new iterator was created with the same +* prefix. Maybe: s/with the same prefix/at the same reference/
Changed it to
If the REF_ITERATOR_SEEK_SET_PREFIX flag is set, it would behave the same as if a new iterator was created with the provided refname as prefix.
Quoted 4 lines
Quoted 4 lines
refs/refs-internal.h +1
-1*/ typedef int ref_iterator_seek_fn (struct ref_iterator *ref_iterator, -const char *prefix); +const char *refname,unsigned int flags);
PATCH v4 4/4 for-each-ref: introduce a '--start-after' option
In reply to Karthik NayakThe git-for-each-ref(1) command is used to iterate over references present in a repository. In large repositories with millions of references, it would be optimal to paginate this output such that we can start iteration from a given reference. This would avoid having to iterate over all references from the beginning each time when paginating through results.
The previous commit added 'seek' functionality to the reference backends. Utilize this and expose a '--start-after' option in 'git-for-each-ref(1)'. When used, the reference iteration seeks to the lexicographically next reference and iterates from there onward.
This enables efficient pagination workflows, where the calling script can remember the last provided reference and use that as the starting point for the next set of references:
git for-each-ref --count=100
git for-each-ref --count=100 --start-after=refs/heads/branch-100
git for-each-ref --count=100 --start-after=refs/heads/branch-200Since the reference iterators only allow seeking to a specified marker via the ref_iterator_seek(), we introduce a helper function start_ref_iterator_after(), which seeks to next reference by simply adding (char) 1 to the marker.
We must note that pagination always continues from the provided marker, as such any concurrent reference updates lexicographically behind the marker will not be output. Document the same.
--- Documentation/git-for-each-ref.adoc | 10 +- builtin/for-each-ref.c | 8 ++ ref-filter.c | 80 +++++++++++---- ref-filter.h | 1 + t/t6302-for-each-ref-filter.sh | 194 ++++++++++++++++++++++++++++++++++++ 5 files changed, 272 insertions(+), 21 deletions(-)
Documentation/git-for-each-ref.adoc +9 -1
@@ -14,7 +14,7 @@ SYNOPSIS [--points-at=<object>] [--merged[=<object>]] [--no-merged[=<object>]] [--contains[=<object>]] [--no-contains[=<object>]] - [--exclude=<pattern> ...] + [--exclude=<pattern> ...] [--start-after=<marker>] DESCRIPTION -----------
@@ -108,6 +108,14 @@ TAB %(refname)`. --include-root-refs:: List root refs (HEAD and pseudorefs) apart from regular refs. +--start-after=<marker>:: + Allows paginating the output by skipping references up to and including the + specified marker. When paging, it should be noted that references may be + deleted, modified or added between invocations. Output will only yield those + references which follow the marker lexicographically. Output begins from the + first reference that would come after the marker alphabetically. Cannot be + used with general pattern matching or custom sort options. + FIELD NAMES -----------
builtin/for-each-ref.c +8 -0
@@ -13,6 +13,7 @@ static char const * const for_each_ref_usage[] = {N_ ("git for-each-ref [--points-at <object>]" ),N_ ("git for-each-ref [--merged [<commit>]] [--no-merged [<commit>]]" ),N_ ("git for-each-ref [--contains [<commit>]] [--no-contains [<commit>]]" ), +N_ ("git for-each-ref [--start-after <marker>]" ), NULL };
@@ -44,6 +45,7 @@ int cmd_for_each_ref(int argc,OPT_GROUP ("" ),OPT_INTEGER (0 ,"count" , &format.array_opts.max_count,N_ ("show only <n> matched refs" )),OPT_STRING (0 ,"format" , &format.format,N_ ("format" ),N_ ("format to use for the output" )), +OPT_STRING (0 ,"start-after" , &filter.start_after,N_ ("start-start" ),N_ ("start iteration after the provided marker" )),OPT__COLOR (&format.use_color,N_ ("respect format colors" )),OPT_REF_FILTER_EXCLUDE (&filter), OPT_REF_SORT(&sorting_options),
@@ -79,6 +81,9 @@ int cmd_for_each_ref(int argc,if (verify_ref_format (&format))usage_with_options (for_each_ref_usage, opts); +if (filter.start_after && sorting_options.nr >1 ) +die (_ ("cannot use --start-after with custom sort options" )); + sorting =ref_sorting_options (&sorting_options);ref_sorting_set_sort_flags_all (sorting,REF_SORTING_ICASE , icase); filter.ignore_case = icase;
@@ -100,6 +105,9 @@ int cmd_for_each_ref(int argc, filter.name_patterns = argv; } +if (filter.start_after && filter.name_patterns && filter.name_patterns[0 ]) +die (_ ("cannot use --start-after with patterns" )); +if (include_root_refs) flags |=FILTER_REFS_ROOT_REFS |FILTER_REFS_DETACHED_HEAD ;
ref-filter.c +60 -20
132 lines not on this page · raw message
ref-filter.h +1 -0
@@ -64,6 +64,7 @@ struct ref_array {struct ref_filter {const char **name_patterns; +const char *start_after;struct strvec exclude;struct oid_array points_at;struct commit_list *with_commit;
t/t6302-for-each-ref-filter.sh +194 -0
198 lines not on this page · raw message
Signature · 1 line
2.49.0
Re: [PATCH v4 4/4] for-each-ref: introduce a '--start-after' option
Quoted 51 lines · Karthik Nayak
/*
* This is the same asfor_each_fullref_in(), but it tries to iterate
* only over the patterns we'll care about. Note that it _doesn't_ do a fullref-filter.c +10
-8@@ -2692,10 +2710,13 @@ static int for_each_fullref_in_pattern(struct ref_filter *filter,each_ref_fn cb, void *cb_data) { + struct ref_iterator *iter; +int flags =0 , ret =0 ; +if (filter->kind &FILTER_REFS_ROOT_REFS ) {/* In this case, we want to print all refs including root refs. */ -return refs_for_each_include_root_refs (get_main_ref_store (the_repository), - cb, cb_data); + flags |=DO_FOR_EACH_INCLUDE_ROOT_REFS ; + goto non_prefix_iter; }if (!filter->match_as_path) {@@ -2704,8 +2725,7 @@ static int for_each_fullref_in_pattern(struct ref_filter *filter,* prefixes like "refs/heads/" etc. are stripped off, * so we have to look at everything: */ -return refs_for_each_fullref_in (get_main_ref_store (the_repository), -"" ,NULL , cb, cb_data); + goto non_prefix_iter; }if (filter->ignore_case) {@@ -2714,20 +2734,29 @@ static int for_each_fullref_in_pattern(struct ref_filter *filter,* so just return everything and let the caller * sort it out. */ -return refs_for_each_fullref_in (get_main_ref_store (the_repository), -"" ,NULL , cb, cb_data); + goto non_prefix_iter; }if (!filter->name_patterns[0 ]) {/* no patterns; we have to look at everything */ -return refs_for_each_fullref_in (get_main_ref_store (the_repository), -"" , filter->exclude.v, cb, cb_data); + goto non_prefix_iter; }return refs_for_each_fullref_in_prefixes (get_main_ref_store (the_repository),NULL , filter->name_patterns, filter->exclude.v, cb, cb_data); + +non_prefix_iter:
ref-filter.c static int for_each_fullref_in_pattern(struct ref_filter *filter, · line 2760
Quoted 10 lines
ref-filter.c +9
-0+ iter =refs_ref_iterator_begin (get_main_ref_store (the_repository),"" , +NULL ,0 , flags); +if (filter->start_after) + ret =start_ref_iterator_after (iter, filter->start_after); + +if (ret) +return ret; + +return do_for_each_ref_iterator (iter, cb, cb_data); }
Nit: I wonder if what is under the 'non_prefix_iter' label could be in a new function and instead of goto non_prefix_iter we could return the result of the new function.
ref-filter.c static int do_filter_refs(struct ref_filter *filter, unsigned int type, each_ref · line 3231
Quoted 10 lines
ref-filter.c +2
-2/* @@ -3197,9 +3226,11 @@ static int do_filter_refs(struct ref_filter *filter, unsigned int type, each_refinit_contains_cache (&filter->internal.no_contains_cache);/* Simple per-ref filtering */ -if (!filter->kind) +if (!filter->kind) {die ("filter_refs: invalid type" ); -else { + }else {
Nit: the else could be removed altogether here, but maybe that should be done in a preparatory patch.
Quoted 2 lines
ref-filter.c +2
-0+const char *prefix =NULL ; +
[...]
Quoted 6 lines
t/t6302-for-each-ref-filter.sh +6
-0+test_expect_success 'start after with specific directory and trailing slash' ' + cat >expect <<-\EOF && + refs/odd/spot + refs/tags/annotated-tag + refs/tags/doubly-annotated-tag + refs/tags/doubly-signed-tag
t/t6302-for-each-ref-filter.sh test_expect_success 'validate worktree atom' ' · line 656
Quoted 10 lines
t/t6302-for-each-ref-filter.sh +10
-0+ refs/tags/foo1.10 + refs/tags/foo1.3 + refs/tags/foo1.6 + refs/tags/four + refs/tags/one + refs/tags/signed-tag + refs/tags/three + refs/tags/two + EOF + git for-each-ref --format="%(refname)" --start-after=refs/lost >actual &&
I don't see a trailing slash.
Quoted 9 lines
t/t6302-for-each-ref-filter.sh +9
-0+ test_cmp expect actual +' + +test_expect_success 'start after, just behind a specific directory' ' + cat >expect <<-\EOF && + refs/odd/spot + refs/tags/annotated-tag + refs/tags/doubly-annotated-tag + refs/tags/doubly-signed-tag
t/t6302-for-each-ref-filter.sh test_expect_success 'validate worktree atom' ' · line 675
Quoted 10 lines
t/t6302-for-each-ref-filter.sh +10
-0+ refs/tags/foo1.10 + refs/tags/foo1.3 + refs/tags/foo1.6 + refs/tags/four + refs/tags/one + refs/tags/signed-tag + refs/tags/three + refs/tags/two + EOF + git for-each-ref --format="%(refname)" --start-after=refs/odd/ >actual &&
Here there is a trailing slash though.
Quoted 2 lines
+
test_cmpexpect actual
+'
Re: [PATCH v4 4/4] for-each-ref: introduce a '--start-after' option
Quoted 9 lines · Christian Couder
ref-filter.c static int do_filter_refs(struct ref_filter *filter, unsigned int type, each_ref · line 3231
Quoted 6 lines
/* Simple per-ref filtering */ref-filter.c +2
-2-if (!filter->kind) +if (!filter->kind) {die ("filter_refs: invalid type" ); -else { + }else {Nit: the
elsecould be removed altogether here, but maybe that should be done in a preparatory patch.
Good eyes. Thanks for carefully reading it over.
Re: [PATCH v4 4/4] for-each-ref: introduce a '--start-after' option
In reply to Christian CouderQuoted 59 lines · Christian Couder
Quoted 57 lines · Karthik Nayak
/*
* This is the same asfor_each_fullref_in(), but it tries to iterate
* only over the patterns we'll care about. Note that it _doesn't_ do a fullref-filter.c +16
-8@@ -2692,10 +2710,13 @@ static int for_each_fullref_in_pattern(struct ref_filter *filter,each_ref_fn cb, void *cb_data) { + struct ref_iterator *iter; +int flags =0 , ret =0 ; +if (filter->kind &FILTER_REFS_ROOT_REFS ) {/* In this case, we want to print all refs including root refs. */ -return refs_for_each_include_root_refs (get_main_ref_store (the_repository), - cb, cb_data); + flags |=DO_FOR_EACH_INCLUDE_ROOT_REFS ; + goto non_prefix_iter; }if (!filter->match_as_path) {@@ -2704,8 +2725,7 @@ static int for_each_fullref_in_pattern(struct ref_filter *filter,* prefixes like "refs/heads/" etc. are stripped off, * so we have to look at everything: */ -return refs_for_each_fullref_in (get_main_ref_store (the_repository), -"" ,NULL , cb, cb_data); + goto non_prefix_iter; }if (filter->ignore_case) {@@ -2714,20 +2734,29 @@ static int for_each_fullref_in_pattern(struct ref_filter *filter,* so just return everything and let the caller * sort it out. */ -return refs_for_each_fullref_in (get_main_ref_store (the_repository), -"" ,NULL , cb, cb_data); + goto non_prefix_iter; }if (!filter->name_patterns[0 ]) {/* no patterns; we have to look at everything */ -return refs_for_each_fullref_in (get_main_ref_store (the_repository), -"" , filter->exclude.v, cb, cb_data); + goto non_prefix_iter; }return refs_for_each_fullref_in_prefixes (get_main_ref_store (the_repository),NULL , filter->name_patterns, filter->exclude.v, cb, cb_data); + +non_prefix_iter: + iter =refs_ref_iterator_begin (get_main_ref_store (the_repository),"" , +NULL ,0 , flags); +if (filter->start_after) + ret =start_ref_iterator_after (iter, filter->start_after); + +if (ret)
Quoted 9 lines
ref-filter.c static int for_each_fullref_in_pattern(struct ref_filter *filter, · line 2760
Quoted 4 lines
ref-filter.c +3
-0+return ret; + +return do_for_each_ref_iterator (iter, cb, cb_data); }Nit: I wonder if what is under the '
non_prefix_iter' label could be in a new function and instead ofgoto non_prefix_iterwe could return the result of the new function.
Yeah, that would work too. Let me do that and make it nicer! Thanks for the suggestion.
Quoted 4 lines
Quoted 4 lines
ref-filter.c +0
-0/* @@ -3197,9 +3226,11 @@ static int do_filter_refs(struct ref_filter *filter, unsigned int type, each_refinit_contains_cache (&filter->internal.no_contains_cache);
Quoted 10 lines
ref-filter.c static int do_filter_refs(struct ref_filter *filter, unsigned int type, each_ref · line 3231
Quoted 6 lines
ref-filter.c +2
-2/* Simple per-ref filtering */ -if (!filter->kind) +if (!filter->kind) {die ("filter_refs: invalid type" ); -else { + }else {Nit: the
elsecould be removed altogether here, but maybe that should be done in a preparatory patch.
Indeed, since I plan to re-roll with the changes you've suggested, I will add this in too.
Quoted 38 lines
Quoted 2 lines
ref-filter.c +2
-0+const char *prefix =NULL ; +[...]
Quoted 6 lines
t/t6302-for-each-ref-filter.sh +6
-0+test_expect_success 'start after with specific directory and trailing slash' ' + cat >expect <<-\EOF && + refs/odd/spot + refs/tags/annotated-tag + refs/tags/doubly-annotated-tag + refs/tags/doubly-signed-tagt/t6302-for-each-ref-filter.sh test_expect_success 'validate worktree atom' ' · line 656
Quoted 10 lines
t/t6302-for-each-ref-filter.sh +10
-0+ refs/tags/foo1.10 + refs/tags/foo1.3 + refs/tags/foo1.6 + refs/tags/four + refs/tags/one + refs/tags/signed-tag + refs/tags/three + refs/tags/two + EOF + git for-each-ref --format="%(refname)" --start-after=refs/lost >actual &&I don't see a trailing slash.
Quoted 13 lines
t/t6302-for-each-ref-filter.sh +13
-0+ test_cmp expect actual +' + +test_expect_success 'start after, just behind a specific directory' ' + cat >expect <<-\EOF && + refs/odd/spot + refs/tags/annotated-tag + refs/tags/doubly-annotated-tag + refs/tags/doubly-signed-tag + refs/tags/foo1.10 + refs/tags/foo1.3 + refs/tags/foo1.6 + refs/tags/four
Quoted 8 lines
t/t6302-for-each-ref-filter.sh test_expect_success 'validate worktree atom' ' · line 675
Quoted 6 lines
t/t6302-for-each-ref-filter.sh +6
-0+ refs/tags/one + refs/tags/signed-tag + refs/tags/three + refs/tags/two + EOF + git for-each-ref --format="%(refname)" --start-after=refs/odd/ >actual &&Here there is a trailing slash though.
I think these tests would make more sense in the newer versions with the values swapped. Let me do that.
Thanks Christian for the thorough review.
Re: [PATCH v4 0/4] for-each-ref: introduce seeking functionality via '--start-after'
In reply to Karthik NayakQuoted 3 lines · Karthik Nayak
Initially I was also planning to cleanup all the
refs_for_each...()functions in 'refs.h' by simply using the iterator, but this bloated the series. So I've left that for another day.
I wonder if there is a plan to add the '--start-after' option to git branch and git tag too?
Quoted 5 lines
Karthik Nayak (4):
refs: exposeref_iteratorvia 'refs.h'
ref-cache: remove unused function 'find_ref_entry()'
refs: selectively set prefix in the seek functions
for-each-ref: introduce a '--start-after' option
Except for the few small comments I left on the two last patches and one below, this looks good to me.
[ ... ]
Quoted 1 line
Range-diff versus v3:
[ ... ]
Quoted 8 lines
+ struct cache_ref_iterator *iter =
+ (struct cache_ref_iterator *)ref_iterator;
+
+ if (flags & REF_ITERATOR_SEEK_SET_PREFIX) {
-+ return cache_ref_iterator_set_prefix(iter, seek);
-+ } else if (seek && *seek) {
++ return cache_ref_iterator_set_prefix(iter, refname);
++ } else if (refname && *refname) {Nit: the else here could be removed, but yeah it might be better to do that in a preparatory patch.
Quoted 4 lines
+ struct cache_ref_iterator_level *level;
-+ const char *slash = seek;
++ const char *slash = refname;
+ struct ref_dir *dir;Re: [PATCH v4 0/4] for-each-ref: introduce seeking functionality via '--start-after'
Quoted 8 lines · Christian Couder
Quoted 3 lines · Karthik Nayak
Initially I was also planning to cleanup all the
refs_for_each...()functions in 'refs.h' by simply using the iterator, but this bloated the series. So I've left that for another day.I wonder if there is a plan to add the '--start-after' option to
git branchandgit tagtoo?
Good question.
"git for-each-ref" is for scripters, "git branch/tag" are for humans. And humans do not page (outside "more/less").
So while it may be trivial to expose the feature to these Porcelain commands, it is not obvious that it is a good idea worth cluttering "git tag -h" output.
Re: [PATCH v4 0/4] for-each-ref: introduce seeking functionality via '--start-after'
Quoted 11 lines · Junio C Hamano
Quoted 8 lines · Christian Couder
Quoted 3 lines · Karthik Nayak
Initially I was also planning to cleanup all the
refs_for_each...()functions in 'refs.h' by simply using the iterator, but this bloated the series. So I've left that for another day.I wonder if there is a plan to add the '--start-after' option to
git branchandgit tagtoo?
Quoted 8 lines
Good question.
"git for-each-ref" is for scripters, "git branch/tag" are for humans. And humans do not page (outside "more/less").
So while it may be trivial to expose the feature to these Porcelain commands, it is not obvious that it is a good idea worth cluttering "git tag -h" output.
Agreed, since the functionality is built into 'ref-filter', it should be easy to expand to these commands. But I don't see the need for it.
Karthik
v5 cover for-each-ref: introduce seeking functionality via '--start-after'
In reply to Karthik NayakThe git-for-each-ref(1) command is used to iterate over references present in a repository. In large repositories with millions of references, it would be optimal to paginate this output such that we can start iteration from a given reference. This would avoid having to iterate over all references from the beginning each time when paginating through results.
This series adds a '--start-after' option in 'git-for-each-ref(1)'. When used, the reference iteration seeks to first reference following the marker alphabetically. When paging, it should be noted that references may be deleted, modified or added between invocations. Output will only yield those references which follow the marker lexicographically. If the marker does not exist, output begins from the first reference that would come after it alphabetically.
This enables efficient pagination workflows like:
git for-each-ref --count=100
git for-each-ref --count=100 --start-after=refs/heads/branch-100
git for-each-ref --count=100 --start-after=refs/heads/branch-200To add this functionality, we expose the ref_iterator outside the 'refs/' namespace and modify the ref_iterator_seek() to actually seek to a given reference and only set the prefix when the set_prefix field is set.
On the reftable and packed backend, the changes are simple. But since the files backend uses 'ref-cache' for reference handling, the changes there are a little more involved, since we need to setup the right levels and the indexing.
Initially I was also planning to cleanup all the refs_for_each...() functions in 'refs.h' by simply using the iterator, but this bloated the series. So I've left that for another day.
Changes in v5:
- Changes to the comments to refer to the flag
'REF_ITERATOR_SEEK_SET_PREFIX' instead of a variable used in older
versions. Also other small grammar fixes.
- Added a commit to remove an unnecessary else clause.
- Move seeking functionality within for_each_fullref_in_pattern to its
own function.
- Fix incorrect naming in the tests.
- Link to v4: https://lore.kernel.org/r/20250711-306-git-for-each-ref-pagination-v4-0-ed3303ad5b89@gmail.com
Changes in v4:
- Patch 3/4: Move around the documentation for the flag and rename the
seek variable to refname.
- Patch 4/4: Cleanup the commit message and also the documentation.
- Link to v3: https://lore.kernel.org/r/20250708-306-git-for-each-ref-pagination-v3-0-8cfba1080be4@gmail.com
Changes in v3:
- Change the working of the command to exclude the marker provided. With
this rename the flag to '--start-after'.
- Extend the documentation to add a note about concurrent modifications
to the reference database.
- Link to v2: https://lore.kernel.org/r/20250704-306-git-for-each-ref-pagination-v2-0-bcde14acdd81@gmail.com
Changes in v2:
- Modify 'ref_iterator_seek()' to take in flags instead of a
'set_prefix' variable. This improves readability, where users would
use the 'REF_ITERATOR_SEEK_SET_PREFIX' instead of simply passing '1'.
- When the set prefix flag isn't usage, reset any previously set prefix.
This ensures that the internal prefix state is always reset whenever
we seek and unifies the behavior between 'ref_iterator_seek' and
'ref_iterator_begin'.
- Don't allow '--skip-until' to be run with '--sort', since the seeking
always takes place before any sorting and this can be confusing.
- Some styling fixes:
- Remove extra newline
- Skip braces around single lined if...else clause
- Add braces around 'if' clause
- Fix indentation
- Link to v1: https://lore.kernel.org/git/20250701-306-git-for-each-ref-pagination-v1-0-4f0ae7c0688f@gmail.com/
--- Documentation/git-for-each-ref.adoc | 10 +- builtin/for-each-ref.c | 8 ++ ref-filter.c | 116 ++++++++++++++------- ref-filter.h | 1 + refs.c | 6 +- refs.h | 155 ++++++++++++++++++++++++++++ refs/debug.c | 7 +- refs/files-backend.c | 7 +- refs/iterator.c | 26 +++-- refs/packed-backend.c | 17 ++-- refs/ref-cache.c | 99 ++++++++++++++---- refs/ref-cache.h | 7 -- refs/refs-internal.h | 152 ++-------------------------- refs/reftable-backend.c | 21 ++-- t/t6302-for-each-ref-filter.sh | 194 ++++++++++++++++++++++++++++++++++++ 15 files changed, 583 insertions(+), 243 deletions(-)
Karthik Nayak (5):
refs: expose ref_iterator via 'refs.h'
ref-cache: remove unused function 'find_ref_entry()'
refs: selectively set prefix in the seek functions
ref-filter: remove unnecessary else clause
for-each-ref: introduce a '--start-after' option
Range-diff versus v4:
1: dde167f421 = 1: f9c9a7fdd9 refs: expose `ref_iterator` via 'refs.h'
2: e392e93520 = 2: 83bee35517 ref-cache: remove unused function 'find_ref_entry()'
3: 711ffcac00 ! 3: 3b6019a1e7 refs: selectively set prefix in the seek functions
@@ refs/refs-internal.h: void base_ref_iterator_init(struct ref_iterator *iter,
/*
- * Seek the iterator to the first reference matching the given prefix. Should
- * behave the same as if a new iterator was created with the same prefix.
-+ * Seek the iterator to the first matching reference. If set_prefix is set,
-+ * it would behave the same as if a new iterator was created with the same
-+ * prefix.
++ * Seek the iterator to the first matching reference. If the
++ * REF_ITERATOR_SEEK_SET_PREFIX flag is set, it would behave the same as if a
++ * new iterator was created with the provided refname as prefix.
*/
typedef int ref_iterator_seek_fn(struct ref_iterator *ref_iterator,
- const char *prefix);
-: ---------- > 4: 3f89eeef26 ref-filter: remove unnecessary else clause
4: 3a0c89acbe ! 5: 7ee7d83cf0 for-each-ref: introduce a '--start-after' option
@@ ref-filter.c: static int filter_exclude_match(struct ref_filter *filter, const c
+ strbuf_release(&sb);
+ return ret;
+}
++
++static int for_each_fullref_with_seek(struct ref_filter *filter, each_ref_fn cb,
++ void *cb_data, unsigned int flags)
++{
++ struct ref_iterator *iter;
++ int ret = 0;
++
++ iter = refs_ref_iterator_begin(get_main_ref_store(the_repository), "",
++ NULL, 0, flags);
++ if (filter->start_after)
++ ret = start_ref_iterator_after(iter, filter->start_after);
++
++ if (ret)
++ return ret;
++
++ return do_for_each_ref_iterator(iter, cb, cb_data);
++}
+
/*
* This is the same as for_each_fullref_in(), but it tries to iterate
* only over the patterns we'll care about. Note that it _doesn't_ do a full
@@ ref-filter.c: static int for_each_fullref_in_pattern(struct ref_filter *filter,
- each_ref_fn cb,
- void *cb_data)
{
-+ struct ref_iterator *iter;
-+ int flags = 0, ret = 0;
-+
if (filter->kind & FILTER_REFS_ROOT_REFS) {
/* In this case, we want to print all refs including root refs. */
- return refs_for_each_include_root_refs(get_main_ref_store(the_repository),
- cb, cb_data);
-+ flags |= DO_FOR_EACH_INCLUDE_ROOT_REFS;
-+ goto non_prefix_iter;
++ return for_each_fullref_with_seek(filter, cb, cb_data,
++ DO_FOR_EACH_INCLUDE_ROOT_REFS);
}
if (!filter->match_as_path) {
@@ ref-filter.c: static int for_each_fullref_in_pattern(struct ref_filter *filter,
*/
- return refs_for_each_fullref_in(get_main_ref_store(the_repository),
- "", NULL, cb, cb_data);
-+ goto non_prefix_iter;
++ return for_each_fullref_with_seek(filter, cb, cb_data, 0);
}
if (filter->ignore_case) {
@@ ref-filter.c: static int for_each_fullref_in_pattern(struct ref_filter *filter,
*/
- return refs_for_each_fullref_in(get_main_ref_store(the_repository),
- "", NULL, cb, cb_data);
-+ goto non_prefix_iter;
++ return for_each_fullref_with_seek(filter, cb, cb_data, 0);
}
if (!filter->name_patterns[0]) {
/* no patterns; we have to look at everything */
- return refs_for_each_fullref_in(get_main_ref_store(the_repository),
- "", filter->exclude.v, cb, cb_data);
-+ goto non_prefix_iter;
++ return for_each_fullref_with_seek(filter, cb, cb_data, 0);
}
return refs_for_each_fullref_in_prefixes(get_main_ref_store(the_repository),
- NULL, filter->name_patterns,
- filter->exclude.v,
- cb, cb_data);
-+
-+non_prefix_iter:
-+ iter = refs_ref_iterator_begin(get_main_ref_store(the_repository), "",
-+ NULL, 0, flags);
-+ if (filter->start_after)
-+ ret = start_ref_iterator_after(iter, filter->start_after);
-+
-+ if (ret)
-+ return ret;
-+
-+ return do_for_each_ref_iterator(iter, cb, cb_data);
- }
+@@ ref-filter.c: void filter_is_base(struct repository *r,
- /*
-@@ ref-filter.c: static int do_filter_refs(struct ref_filter *filter, unsigned int type, each_ref
- init_contains_cache(&filter->internal.no_contains_cache);
+ static int do_filter_refs(struct ref_filter *filter, unsigned int type, each_ref_fn fn, void *cb_data)
+ {
++ const char *prefix = NULL;
+ int ret = 0;
- /* Simple per-ref filtering */
-- if (!filter->kind)
-+ if (!filter->kind) {
- die("filter_refs: invalid type");
-- else {
-+ } else {
-+ const char *prefix = NULL;
-+
- /*
- * For common cases where we need only branches or remotes or tags,
- * we only iterate through those refs. If a mix of refs is needed,
+ filter->kind = type & FILTER_REFS_KIND_MASK;
@@ ref-filter.c: static int do_filter_refs(struct ref_filter *filter, unsigned int type, each_ref
- * of filter_ref_kind().
- */
- if (filter->kind == FILTER_REFS_BRANCHES)
-- ret = refs_for_each_fullref_in(get_main_ref_store(the_repository),
-- "refs/heads/", NULL,
-- fn, cb_data);
-+ prefix = "refs/heads/";
- else if (filter->kind == FILTER_REFS_REMOTES)
-- ret = refs_for_each_fullref_in(get_main_ref_store(the_repository),
-- "refs/remotes/", NULL,
-- fn, cb_data);
-+ prefix = "refs/remotes/";
- else if (filter->kind == FILTER_REFS_TAGS)
-- ret = refs_for_each_fullref_in(get_main_ref_store(the_repository),
-- "refs/tags/", NULL, fn,
-- cb_data);
-- else if (filter->kind & FILTER_REFS_REGULAR)
-+ prefix = "refs/tags/";
+ * of filter_ref_kind().
+ */
+ if (filter->kind == FILTER_REFS_BRANCHES)
+- ret = refs_for_each_fullref_in(get_main_ref_store(the_repository),
+- "refs/heads/", NULL,
+- fn, cb_data);
++ prefix = "refs/heads/";
+ else if (filter->kind == FILTER_REFS_REMOTES)
+- ret = refs_for_each_fullref_in(get_main_ref_store(the_repository),
+- "refs/remotes/", NULL,
+- fn, cb_data);
++ prefix = "refs/remotes/";
+ else if (filter->kind == FILTER_REFS_TAGS)
+- ret = refs_for_each_fullref_in(get_main_ref_store(the_repository),
+- "refs/tags/", NULL, fn,
+- cb_data);
+- else if (filter->kind & FILTER_REFS_REGULAR)
++ prefix = "refs/tags/";
+
-+ if (prefix) {
-+ struct ref_iterator *iter;
++ if (prefix) {
++ struct ref_iterator *iter;
+
-+ iter = refs_ref_iterator_begin(get_main_ref_store(the_repository),
-+ "", NULL, 0, 0);
++ iter = refs_ref_iterator_begin(get_main_ref_store(the_repository),
++ "", NULL, 0, 0);
+
-+ if (filter->start_after)
-+ ret = start_ref_iterator_after(iter, filter->start_after);
-+ else if (prefix)
-+ ret = ref_iterator_seek(iter, prefix, 1);
++ if (filter->start_after)
++ ret = start_ref_iterator_after(iter, filter->start_after);
++ else if (prefix)
++ ret = ref_iterator_seek(iter, prefix, 1);
+
-+ if (!ret)
-+ ret = do_for_each_ref_iterator(iter, fn, cb_data);
-+ } else if (filter->kind & FILTER_REFS_REGULAR) {
- ret = for_each_fullref_in_pattern(filter, fn, cb_data);
-+ }
++ if (!ret)
++ ret = do_for_each_ref_iterator(iter, fn, cb_data);
++ } else if (filter->kind & FILTER_REFS_REGULAR) {
+ ret = for_each_fullref_in_pattern(filter, fn, cb_data);
++ }
- /*
- * When printing all ref types, HEAD is already included,
+ /*
+ * When printing all ref types, HEAD is already included,
## ref-filter.h ##
@@ ref-filter.h: struct ref_array {
@@ t/t6302-for-each-ref-filter.sh: test_expect_success 'validate worktree atom' '
+ refs/tags/three
+ refs/tags/two
+ EOF
-+ git for-each-ref --format="%(refname)" --start-after=refs/lost >actual &&
++ git for-each-ref --format="%(refname)" --start-after=refs/odd/ >actual &&
+ test_cmp expect actual
+'
+
@@ t/t6302-for-each-ref-filter.sh: test_expect_success 'validate worktree atom' '
+ refs/tags/three
+ refs/tags/two
+ EOF
-+ git for-each-ref --format="%(refname)" --start-after=refs/odd/ >actual &&
++ git for-each-ref --format="%(refname)" --start-after=refs/lost >actual &&
+ test_cmp expect actual
+'
+base-commit: cf6f63ea6bf35173e02e18bdc6a4ba41288acff9
change-id: 20250605-306-git-for-each-ref-pagination-0ba8a29ae646
Thanks
- Karthik
PATCH v5 1/5 refs: expose `ref_iterator` via 'refs.h'
The ref_iterator is an internal structure to the 'refs/' sub-directory, which allows iteration over refs. All reference iteration is built on top of these iterators.
External clients of the 'refs' subsystem use the various 'refs_for_each...()' functions to iterate over refs. However since these are wrapper functions, each combination of functionality requires a new wrapper function. This is not feasible as the functions pile up with the increase in requirements. Expose the internal reference iterator, so advanced users can mix and match options as needed.
--- refs.h | 147 +++++++++++++++++++++++++++++++++++++++++++++++++++ refs/refs-internal.h | 145 +------------------------------------------------- 2 files changed, 149 insertions(+), 143 deletions(-)
refs.h +147 -0
@@ -1190,4 +1190,151 @@ int repo_migrate_ref_storage_format(struct repository *repo,unsigned int flags, struct strbuf *err); +/* +* Reference iterators +* +* A reference iterator encapsulates the state of an in-progress +* iteration over references. Create an instance of `struct +* ref_iterator` via one of the functions in this module. +* +* A freshly-created ref_iterator doesn't yet point at a reference. To +* advance the iterator, call ref_iterator_advance(). If successful, +* this sets the iterator's refname, oid, and flags fields to describe +* the next reference and returns ITER_OK. The data pointed at by +* refname and oid belong to the iterator; if you want to retain them +* after calling ref_iterator_advance() again or calling +* ref_iterator_free(), you must make a copy. When the iteration has +* been exhausted, ref_iterator_advance() releases any resources +* associated with the iteration, frees the ref_iterator object, and +* returns ITER_DONE. If you want to abort the iteration early, call +* ref_iterator_free(), which also frees the ref_iterator object and +* any associated resources. If there was an internal error advancing +* to the next entry, ref_iterator_advance() aborts the iteration, +* frees the ref_iterator, and returns ITER_ERROR. +* +* The reference currently being looked at can be peeled by calling +* ref_iterator_peel(). This function is often faster than peel_ref(), +* so it should be preferred when iterating over references. +* +* Putting it all together, a typical iteration looks like this: +* +* int ok; +* struct ref_iterator *iter = ...; +* +* while ((ok = ref_iterator_advance(iter)) == ITER_OK) { +* if (want_to_stop_iteration()) { +* ok = ITER_DONE; +* break; +* } +* +* // Access information about the current reference: +* if (!(iter->flags & REF_ISSYMREF)) +* printf("%s is %s\n", iter->refname, oid_to_hex(iter->oid)); +* +* // If you need to peel the reference: +* ref_iterator_peel(iter, &oid); +* } +* +* if (ok != ITER_DONE) +* handle_error(); +* ref_iterator_free(iter); +*/ +struct ref_iterator ; + +/* +* These flags are passed to refs_ref_iterator_begin() (and do_for_each_ref(), +* which feeds it). +*/ +enum do_for_each_ref_flags { +/* +* Include broken references in a do_for_each_ref*() iteration, which +* would normally be omitted. This includes both refs that point to +* missing objects (a true repository corruption), ones with illegal +* names (which we prefer not to expose to callers), as well as +* dangling symbolic refs (i.e., those that point to a non-existent +* ref; this is not a corruption, but as they have no valid oid, we +* omit them from normal iteration results). +*/ +DO_FOR_EACH_INCLUDE_BROKEN = (1 <<0 ), + +/* +* Only include per-worktree refs in a do_for_each_ref*() iteration. +* Normally this will be used with a files ref_store, since that's +* where all reference backends will presumably store their +* per-worktree refs. +*/ +DO_FOR_EACH_PER_WORKTREE_ONLY = (1 <<1 ), + +/* +* Omit dangling symrefs from output; this only has an effect with +* INCLUDE_BROKEN, since they are otherwise not included at all. +*/ +DO_FOR_EACH_OMIT_DANGLING_SYMREFS = (1 <<2 ), + +/* +* Include root refs i.e. HEAD and pseudorefs along with the regular +* refs. +*/ +DO_FOR_EACH_INCLUDE_ROOT_REFS = (1 <<3 ), +}; + +/* +* Return an iterator that goes over each reference in `refs` for +* which the refname begins with prefix. If trim is non-zero, then +* trim that many characters off the beginning of each refname. +* The output is ordered by refname. +*/ +struct ref_iterator *refs_ref_iterator_begin ( +struct ref_store *refs, +const char *prefix,const char **exclude_patterns, +int trim,enum do_for_each_ref_flags flags); + +/* +* Advance the iterator to the first or next item and return ITER_OK. +* If the iteration is exhausted, free the resources associated with +* the ref_iterator and return ITER_DONE. On errors, free the iterator +* resources and return ITER_ERROR. It is a bug to use ref_iterator or +* call this function again after it has returned ITER_DONE or +* ITER_ERROR. +*/ +int ref_iterator_advance (struct ref_iterator *ref_iterator); + +/* +* Seek the iterator to the first reference with the given prefix. +* The prefix is matched as a literal string, without regard for path +* separators. If prefix is NULL or the empty string, seek the iterator to the +* first reference again. +* +* This function is expected to behave as if a new ref iterator with the same +* prefix had been created, but allows reuse of iterators and thus may allow +* the backend to optimize. Parameters other than the prefix that have been +* passed when creating the iterator will remain unchanged. +* +* Returns 0 on success, a negative error code otherwise. +*/ +int ref_iterator_seek (struct ref_iterator *ref_iterator, +const char *prefix); + +/* +* If possible, peel the reference currently being viewed by the +* iterator. Return 0 on success. +*/ +int ref_iterator_peel (struct ref_iterator *ref_iterator, +struct object_id *peeled); + +/* Free the reference iterator and any associated resources. */ +void ref_iterator_free (struct ref_iterator *ref_iterator); + +/* +* The common backend for the for_each_*ref* functions. Call fn for +* each reference in iter. If the iterator itself ever returns +* ITER_ERROR, return -1. If fn ever returns a non-zero value, stop +* the iteration and return that value. Otherwise, return 0. In any +* case, free the iterator when done. This function is basically an +* adapter between the callback style of reference iteration and the +* iterator style. +*/ +int do_for_each_ref_iterator (struct ref_iterator *iter, +each_ref_fn fn,void *cb_data); +#endif /* REFS_H */
refs/refs-internal.h +2 -143
169 lines not on this page · raw message
Signature · 1 line
2.49.0
PATCH v5 2/5 ref-cache: remove unused function 'find_ref_entry()'
In reply to Karthik NayakThe 'find_ref_entry' function is no longer used, so remove it.
--- refs/ref-cache.c | 14 -------------- refs/ref-cache.h | 7 ------- 2 files changed, 21 deletions(-)
refs/ref-cache.c +0 -14
@@ -194,20 +194,6 @@ static struct ref_dir *find_containing_dir(struct ref_dir *dir,return dir; } -struct ref_entry *find_ref_entry (struct ref_dir *dir,const char *refname) -{ -int entry_index; -struct ref_entry *entry; - dir =find_containing_dir (dir, refname); -if (!dir) -return NULL ; - entry_index =search_ref_dir (dir, refname,strlen (refname)); -if (entry_index ==-1 ) -return NULL ; - entry = dir->entries[entry_index]; -return (entry->flag &REF_DIR ) ?NULL : entry; -} -/* * Emit a warning and return true iff ref1 and ref2 have the same name * and the same oid. Die if they have the same name but different
refs/ref-cache.h +0 -7
@@ -201,13 +201,6 @@ void free_ref_cache(struct ref_cache *cache);*/ void add_entry_to_dir (struct ref_dir *dir,struct ref_entry *entry); -/* -* Find the value entry with the given name in dir, sorting ref_dirs -* and recursing into subdirectories as necessary. If the name is not -* found or it corresponds to a directory entry, return NULL. -*/ -struct ref_entry *find_ref_entry (struct ref_dir *dir,const char *refname); -/* * Start iterating over references in `cache`. If `prefix` is * specified, only include references whose names start with that
Signature · 1 line
2.49.0
Re: [PATCH v5 2/5] ref-cache: remove unused function 'find_ref_entry()'
Quoted 1 line · Karthik Nayak
The '
find_ref_entry' function is no longer used, so remove it.
If my spelunking is correct, ba1c052f (ref_store: implement refs_peel_ref() generically, 2017-09-25) is the commit that removed the last caller of it. Which may be worth noting here.
Quoted 49 lines
Signed-off-by Karthik Nayak <[email protected]>--- refs/ref-cache.c | 14 -------------- refs/ref-cache.h | 7 ------- 2 files changed, 21 deletions(-)refs/ref-cache.c +0
-14@@ -194,20 +194,6 @@ static struct ref_dir *find_containing_dir(struct ref_dir *dir,return dir; } -struct ref_entry *find_ref_entry (struct ref_dir *dir,const char *refname) -{ -int entry_index; -struct ref_entry *entry; - dir =find_containing_dir (dir, refname); -if (!dir) -return NULL ; - entry_index =search_ref_dir (dir, refname,strlen (refname)); -if (entry_index ==-1 ) -return NULL ; - entry = dir->entries[entry_index]; -return (entry->flag &REF_DIR ) ?NULL : entry; -} -/* * Emit a warning and return true iff ref1 and ref2 have the same name * and the same oid. Die if they have the same name but different refs/ref-cache.h +0
-7@@ -201,13 +201,6 @@ void free_ref_cache(struct ref_cache *cache);*/ void add_entry_to_dir (struct ref_dir *dir,struct ref_entry *entry); -/* -* Find the value entry with the given name in dir, sorting ref_dirs -* and recursing into subdirectories as necessary. If the name is not -* found or it corresponds to a directory entry, return NULL. -*/ -struct ref_entry *find_ref_entry (struct ref_dir *dir,const char *refname); -/* * Start iterating over references in `cache`. If `prefix` is * specified, only include references whose names start with that
Re: [PATCH v5 2/5] ref-cache: remove unused function 'find_ref_entry()'
Quoted 8 lines · Junio C Hamano
Quoted 1 line · Karthik Nayak
The '
find_ref_entry' function is no longer used, so remove it.If my spelunking is correct,
ba1c052f(ref_store: implementrefs_peel_ref()generically, 2017-09-25) is the commit that removed the last caller of it. Which may be worth noting here.
Indeed, I could also verify this by running
$ git log -S find_ref_entry --oneline 2c90b85801 ref-cache: remove unused function 'find_ref_entry()' ba1c052fa6 ref_store: implement `refs_peel_ref()` generically 9939b33d6a packed-backend: rip out some now-unused code ....
And looking at ba1c052fa6. I should've done this before. But thanks for the digging!
I plan to address a few comments on this version, but I also see that you've merged it to master. Should I raise followups for noteworthy changes or a new version?
Karthik
Re: [PATCH v5 2/5] ref-cache: remove unused function 'find_ref_entry()'
Quoted 19 lines · Karthik Nayak
Quoted 8 lines · Junio C Hamano
Quoted 1 line · Karthik Nayak
The '
find_ref_entry' function is no longer used, so remove it.If my spelunking is correct,
ba1c052f(ref_store: implementrefs_peel_ref()generically, 2017-09-25) is the commit that removed the last caller of it. Which may be worth noting here.Indeed, I could also verify this by running
$ git log -S find_ref_entry --oneline 2c90b85801 ref-cache: remove unused function 'find_ref_entry()' ba1c052fa6 ref_store: implement `refs_peel_ref()` generically 9939b33d6a packed-backend: rip out some now-unused code ....
Quoted 6 lines
And looking at
ba1c052fa6. I should've done this before. But thanks for the digging!I plan to address a few comments on this version, but I also see that you've merged it to master. Should I raise followups for noteworthy changes or a new version?
You mean it is now in 'next'? Yes, please give incremental patches. Finding some more gotchas even after v5 is a sign that they are tricky enough that deserve separate explanation from the main part of the series. And I think the msan one Kyle and Peff found is also tricky enough that evaded reviewers' eyes.
Thanks.
PATCH v5 3/5 refs: selectively set prefix in the seek functions
In reply to Karthik NayakThe ref iterator exposes a ref_iterator_seek() function. The name suggests that this would seek the iterator to a specific reference in some ways similar to how fseek() works for the filesystem.
However, the function actually sets the prefix for refs iteration. So further iteration would only yield references which match the particular prefix. This is a bit confusing.
Let's add a 'flags' field to the function, which when set with the 'REF_ITERATOR_SEEK_SET_PREFIX' flag, will set the prefix for the iteration in-line with the existing behavior. Otherwise, the reference backends will simply seek to the specified reference and clears any previously set prefix. This allows users to start iteration from a specific reference.
In the packed and reftable backend, since references are available in a sorted list, the changes are simply setting the prefix if needed. The changes on the files-backend are a little more involved, since the files backend uses the 'ref-cache' mechanism. We move out the existing logic within cache_ref_iterator_seek() to cache_ref_iterator_set_prefix() which is called when the 'REF_ITERATOR_SEEK_SET_PREFIX' flag is set. We then parse the provided seek string and set the required levels and their indexes to ensure that seeking is possible.
--- refs.c | 6 ++-- refs.h | 26 +++++++++------ refs/debug.c | 7 ++-- refs/files-backend.c | 7 ++-- refs/iterator.c | 26 ++++++++------- refs/packed-backend.c | 17 ++++++---- refs/ref-cache.c | 85 ++++++++++++++++++++++++++++++++++++++++++++++--- refs/refs-internal.h | 7 ++-- refs/reftable-backend.c | 21 ++++++++---- 9 files changed, 152 insertions(+), 50 deletions(-)
refs.c +3 -3
@@ -2666,12 +2666,12 @@ enum ref_transaction_error refs_verify_refnames_available(struct ref_store *refsif (!initial_transaction) {int ok; -if (!iter) { +if (!iter) iter =refs_ref_iterator_begin (refs, dirname.buf,NULL ,0 ,DO_FOR_EACH_INCLUDE_BROKEN ); - }else if (ref_iterator_seek (iter, dirname.buf) <0 ) { +else if (ref_iterator_seek (iter, dirname.buf, +REF_ITERATOR_SEEK_SET_PREFIX ) <0 ) goto cleanup; - }while ((ok =ref_iterator_advance (iter)) ==ITER_OK ) { if (skip &&
refs.h +17 -9
@@ -1299,21 +1299,29 @@ struct ref_iterator *refs_ref_iterator_begin(*/ int ref_iterator_advance (struct ref_iterator *ref_iterator); +enum ref_iterator_seek_flag { +/* +* When the REF_ITERATOR_SEEK_SET_PREFIX flag is set, the iterator's prefix is +* updated to match the provided string, affecting all subsequent iterations. If +* not, the iterator seeks to the specified reference and clears any previously +* set prefix. +*/ +REF_ITERATOR_SEEK_SET_PREFIX = (1 <<0 ), +}; +/* -* Seek the iterator to the first reference with the given prefix. -* The prefix is matched as a literal string, without regard for path -* separators. If prefix is NULL or the empty string, seek the iterator to the +* Seek the iterator to the first reference matching the given seek string. +* The seek string is matched as a literal string, without regard for path +* separators. If seek is NULL or the empty string, seek the iterator to the * first reference again. * -* This function is expected to behave as if a new ref iterator with the same -* prefix had been created, but allows reuse of iterators and thus may allow -* the backend to optimize. Parameters other than the prefix that have been -* passed when creating the iterator will remain unchanged. +* This function is expected to behave as if a new ref iterator has been +* created, but allows reuse of existing iterators for optimization. * * Returns 0 on success, a negative error code otherwise. */ -int ref_iterator_seek (struct ref_iterator *ref_iterator, -const char *prefix); +int ref_iterator_seek (struct ref_iterator *ref_iterator,const char *refname, +unsigned int flags);/* * If possible, peel the reference currently being viewed by the
refs/debug.c +4 -3
@@ -170,12 +170,13 @@ static int debug_ref_iterator_advance(struct ref_iterator *ref_iterator) }static int debug_ref_iterator_seek (struct ref_iterator *ref_iterator, -const char *prefix) +const char *refname,unsigned int flags) {struct debug_ref_iterator *diter = (struct debug_ref_iterator *)ref_iterator; -int res = diter->iter->vtable->seek (diter->iter, prefix); -trace_printf_key (&trace_refs,"iterator_seek: %s: %d\n" , prefix ? prefix :"" , res); +int res = diter->iter->vtable->seek (diter->iter, refname, flags); +trace_printf_key (&trace_refs,"iterator_seek: %s flags: %d: %d\n" , + refname ? refname :"" , flags, res);return res; }
refs/files-backend.c +4 -3
@@ -929,11 +929,11 @@ static int files_ref_iterator_advance(struct ref_iterator *ref_iterator) }static int files_ref_iterator_seek (struct ref_iterator *ref_iterator, -const char *prefix) +const char *refname,unsigned int flags) {struct files_ref_iterator *iter = (struct files_ref_iterator *)ref_iterator; -return ref_iterator_seek (iter->iter0, prefix); +return ref_iterator_seek (iter->iter0, refname, flags); } static int files_ref_iterator_peel(struct ref_iterator *ref_iterator,
@@ -2316,7 +2316,8 @@ static int files_reflog_iterator_advance(struct ref_iterator *ref_iterator) } static int files_reflog_iterator_seek(struct ref_iterator *ref_iterator UNUSED, - const char *prefix UNUSED) + const char *refname UNUSED, + unsigned int flags UNUSED) {BUG ("ref_iterator_seek() called for reflog_iterator" ); }
refs/iterator.c +15 -11
@@ -15,10 +15,10 @@ int ref_iterator_advance(struct ref_iterator *ref_iterator)return ref_iterator->vtable->advance (ref_iterator); } -int ref_iterator_seek (struct ref_iterator *ref_iterator, -const char *prefix) +int ref_iterator_seek (struct ref_iterator *ref_iterator,const char *refname, +unsigned int flags) { -return ref_iterator->vtable->seek (ref_iterator, prefix); +return ref_iterator->vtable->seek (ref_iterator, refname, flags); } int ref_iterator_peel(struct ref_iterator *ref_iterator,
@@ -57,7 +57,8 @@ static int empty_ref_iterator_advance(struct ref_iterator *ref_iterator UNUSED) } static int empty_ref_iterator_seek(struct ref_iterator *ref_iterator UNUSED, - const char *prefix UNUSED) + const char *refname UNUSED, + unsigned int flags UNUSED) {return 0 ; }
@@ -224,7 +225,7 @@ static int merge_ref_iterator_advance(struct ref_iterator *ref_iterator) }static int merge_ref_iterator_seek (struct ref_iterator *ref_iterator, -const char *prefix) +const char *refname,unsigned int flags) {struct merge_ref_iterator *iter = (struct merge_ref_iterator *)ref_iterator;
@@ -234,11 +235,11 @@ static int merge_ref_iterator_seek(struct ref_iterator *ref_iterator, iter->iter0 = iter->iter0_owned; iter->iter1 = iter->iter1_owned; - ret =ref_iterator_seek (iter->iter0, prefix); + ret =ref_iterator_seek (iter->iter0, refname, flags);if (ret <0 )return ret; - ret =ref_iterator_seek (iter->iter1, prefix); + ret =ref_iterator_seek (iter->iter1, refname, flags);if (ret <0 )return ret;
@@ -407,13 +408,16 @@ static int prefix_ref_iterator_advance(struct ref_iterator *ref_iterator) }static int prefix_ref_iterator_seek (struct ref_iterator *ref_iterator, -const char *prefix) +const char *refname,unsigned int flags) {struct prefix_ref_iterator *iter = (struct prefix_ref_iterator *)ref_iterator; -free (iter->prefix); - iter->prefix =xstrdup_or_null (prefix); -return ref_iterator_seek (iter->iter0, prefix); + +if (flags &REF_ITERATOR_SEEK_SET_PREFIX ) { +free (iter->prefix); + iter->prefix =xstrdup_or_null (refname); + } +return ref_iterator_seek (iter->iter0, refname, flags); } static int prefix_ref_iterator_peel(struct ref_iterator *ref_iterator,
refs/packed-backend.c +11 -6
@@ -1004,19 +1004,23 @@ static int packed_ref_iterator_advance(struct ref_iterator *ref_iterator) }static int packed_ref_iterator_seek (struct ref_iterator *ref_iterator, -const char *prefix) +const char *refname,unsigned int flags) {struct packed_ref_iterator *iter = (struct packed_ref_iterator *)ref_iterator;const char *start; -if (prefix && *prefix) - start =find_reference_location (iter->snapshot, prefix,0 ); +if (refname && *refname) + start =find_reference_location (iter->snapshot, refname,0 );else start = iter->snapshot->start; -free (iter->prefix); - iter->prefix =xstrdup_or_null (prefix); +/* Unset any previously set prefix */ +FREE_AND_NULL (iter->prefix); + +if (flags &REF_ITERATOR_SEEK_SET_PREFIX ) + iter->prefix =xstrdup_or_null (refname); + iter->pos = start; iter->eof = iter->snapshot->eof;
@@ -1194,7 +1198,8 @@ static struct ref_iterator *packed_ref_iterator_begin( iter->repo = ref_store->repo; iter->flags = flags; -if (packed_ref_iterator_seek (&iter->base, prefix) <0 ) { +if (packed_ref_iterator_seek (&iter->base, prefix, +REF_ITERATOR_SEEK_SET_PREFIX ) <0 ) {ref_iterator_free (&iter->base);return NULL ; }
refs/ref-cache.c +80 -5
104 lines not on this page · raw message
refs/refs-internal.h +4 -3
@@ -353,11 +353,12 @@ void base_ref_iterator_init(struct ref_iterator *iter,typedef int ref_iterator_advance_fn (struct ref_iterator *ref_iterator);/* -* Seek the iterator to the first reference matching the given prefix. Should -* behave the same as if a new iterator was created with the same prefix. +* Seek the iterator to the first matching reference. If the +* REF_ITERATOR_SEEK_SET_PREFIX flag is set, it would behave the same as if a +* new iterator was created with the provided refname as prefix. */ typedef int ref_iterator_seek_fn (struct ref_iterator *ref_iterator, -const char *prefix); +const char *refname,unsigned int flags);/* * Peels the current ref, returning 0 for success or -1 for failure.
refs/reftable-backend.c +14 -7
@@ -719,15 +719,20 @@ static int reftable_ref_iterator_advance(struct ref_iterator *ref_iterator) }static int reftable_ref_iterator_seek (struct ref_iterator *ref_iterator, -const char *prefix) +const char *refname,unsigned int flags) {struct reftable_ref_iterator *iter = (struct reftable_ref_iterator *)ref_iterator; -free (iter->prefix); - iter->prefix =xstrdup_or_null (prefix); - iter->prefix_len = prefix ?strlen (prefix) :0 ; - iter->err =reftable_iterator_seek_ref (&iter->iter, prefix); +/* Unset any previously set prefix */ +FREE_AND_NULL (iter->prefix); + iter->prefix_len =0 ; + +if (flags &REF_ITERATOR_SEEK_SET_PREFIX ) { + iter->prefix =xstrdup_or_null (refname); + iter->prefix_len = refname ?strlen (refname) :0 ; + } + iter->err =reftable_iterator_seek_ref (&iter->iter, refname);return iter->err; }
@@ -839,7 +844,8 @@ static struct reftable_ref_iterator *ref_iterator_for_stack(struct reftable_ref_if (ret) goto done; - ret =reftable_ref_iterator_seek (&iter->base, prefix); + ret =reftable_ref_iterator_seek (&iter->base, prefix, +REF_ITERATOR_SEEK_SET_PREFIX );if (ret) goto done;
@@ -2042,7 +2048,8 @@ static int reftable_reflog_iterator_advance(struct ref_iterator *ref_iterator) } static int reftable_reflog_iterator_seek(struct ref_iterator *ref_iterator UNUSED, - const char *prefix UNUSED) + const char *refname UNUSED, + unsigned int flags UNUSED) {BUG ("reftable reflog iterator cannot be seeked" );return -1 ;
Signature · 1 line
2.49.0
Re: [PATCH v5 3/5] refs: selectively set prefix in the seek functions
Review
Quoted 11 lines · Karthik Nayak
(quoted diff) +10
-0+static int cache_ref_iterator_seek(struct ref_iterator *ref_iterator, + const char *refname, unsigned int flags) + do { + int len, idx; + int cmp = 0; + + sort_ref_dir(dir); + + slash = strchr(slash, '/'); + len = slash ? slash - refname : (int)strlen(refname);
I was looking at this code due to a nearby thread and noticed this funny cast to int. I guess you added it to silence -Wsign-compare, but Why are we not using a size_t in the first place?
This kind of conversion can sometimes have security implications because a very large "refname" would cause "len" to become negative (i.e., if it's between 2GB and 4GB).
In this particular case it ends up cast back to a size_t via strncmp:
Review
Quoted 5 lines
(quoted diff) +5
-0+ for (idx = 0; idx < dir->nr; idx++) { + cmp = strncmp(refname, dir->entries[idx]->name, len); + if (cmp <= 0) + break; + }
so we get the original value back. We'd still get truncation for a refname value over 4GB, which would presumably give us a slightly wrong answer. But I don't think we'd ever look outside the array.
Such sizes are probably unlikely if we are feeding filesystem paths. But we probably should not set a bad example, and just do;
refs/ref-cache.c +3 -2
@@ -498,13 +498,14 @@ static int cache_ref_iterator_seek(struct ref_iterator *ref_iterator,* indexing to each level as needed. */ do { - int len, idx; + size_t len; + int idx; int cmp = 0; sort_ref_dir(dir); slash = strchr(slash, '/'); - len = slash ? slash - refname : (int)strlen(refname); + len = slash ? slash - refname : strlen(refname); for (idx = 0; idx < dir->nr; idx++) { cmp = strncmp(refname, dir->entries[idx]->name, len);
-Peff
Re: [PATCH v5 3/5] refs: selectively set prefix in the seek functions
Quoted 18 lines · Jeff King
Review
Quoted 11 lines · Karthik Nayak
(quoted diff) +10
-0+static int cache_ref_iterator_seek(struct ref_iterator *ref_iterator, + const char *refname, unsigned int flags) + do { + int len, idx; + int cmp = 0; + + sort_ref_dir(dir); + + slash = strchr(slash, '/'); + len = slash ? slash - refname : (int)strlen(refname);I was looking at this code due to a nearby thread and noticed this funny cast to int. I guess you added it to silence -Wsign-compare, but Why are we not using a
size_tin the first place?
That's an oversight from my side.
Quoted 4 lines
This kind of conversion can sometimes have security implications because a very large "refname" would cause "len" to become negative (i.e., if it's between 2GB and 4GB).
Indeed, I didn't think of that.
Quoted 12 lines
In this particular case it ends up cast back to a
size_tvia strncmp:Review
Quoted 5 lines
(quoted diff) +5
-0+ for (idx = 0; idx < dir->nr; idx++) { + cmp = strncmp(refname, dir->entries[idx]->name, len); + if (cmp <= 0) + break; + }so we get the original value back. We'd still get truncation for a refname value over 4GB, which would presumably give us a slightly wrong answer. But I don't think we'd ever look outside the array.
Quoted 3 lines
Such sizes are probably unlikely if we are feeding filesystem paths. But we probably should not set a bad example, and just do;
Agreed.
Quoted 13 lines
refs/ref-cache.c +2
-1@@ -498,13 +498,14 @@ static int cache_ref_iterator_seek(struct ref_iterator *ref_iterator,* indexing to each level as needed. */ do { - int len, idx; + size_t len; + int idx; int cmp = 0;
refs/ref-cache.c static int cache_ref_iterator_seek(struct ref_iterator *ref_iterator,
Quoted 10 lines
refs/ref-cache.c +1
-2sort_ref_dir (dir); slash =strchr (slash,'/' ); - len = slash ? slash - refname : (int )strlen (refname); + len = slash ? slash - refname :strlen (refname);for (idx =0 ; idx < dir->nr; idx++) { cmp =strncmp (refname, dir->entries[idx]->name, len); -Peff
Thanks, I think we have to typecast slash - refname to size_t, but this is the right way to do it. Thanks for the review!
Karthik
Re: [PATCH v5 3/5] refs: selectively set prefix in the seek functions
Quoted 19 lines · Karthik Nayak
Quoted 19 lines
refs/ref-cache.c +3
-2@@ -498,13 +498,14 @@ static int cache_ref_iterator_seek(struct ref_iterator *ref_iterator,* indexing to each level as needed. */ do { - int len, idx; + size_t len; + int idx; int cmp = 0; sort_ref_dir(dir); slash = strchr(slash, '/'); - len = slash ? slash - refname : (int)strlen(refname); + len = slash ? slash - refname : strlen(refname);
Quoted 7 lines
refs/ref-cache.c static int cache_ref_iterator_seek(struct ref_iterator *ref_iterator,
Quoted 4 lines
refs/ref-cache.c +0
-1for (idx =0 ; idx < dir->nr; idx++) { cmp =strncmp (refname, dir->entries[idx]->name, len); -PeffThanks, I think we have to typecast
slash - refnametosize_t, but this is the right way to do it. Thanks for the review!
Ah, yeah. I mistakenly test-compiled without DEVELOPER=1. ;)
I do think that cast is a lesser evil, though. It is a ptrdiff_t, but we know it is correctly unsigned because "slash > refname" via strchr. I wish there was a good way to use the type system to tell the compiler that.
-Peff
PATCH v5 4/5 ref-filter: remove unnecessary else clause
In reply to Karthik NayakIn 'ref-filter.c', there is an 'else' clause within do_filter_refs(). This is unnecessary since the 'if' clause calls die(), which would exit the program. So let's remove the unnecessary 'else' clause. This improves readability since the indentation is also reduced and flow is simpler.
--- ref-filter.c | 60 ++++++++++++++++++++++++++++++------------------------------ 1 file changed, 30 insertions(+), 30 deletions(-)
ref-filter.c +30 -30
@@ -3199,37 +3199,37 @@ static int do_filter_refs(struct ref_filter *filter, unsigned int type, each_ref/* Simple per-ref filtering */ if (!filter->kind)die ("filter_refs: invalid type" ); -else { -/* -* For common cases where we need only branches or remotes or tags, -* we only iterate through those refs. If a mix of refs is needed, -* we iterate over all refs and filter out required refs with the help -* of filter_ref_kind(). -*/ -if (filter->kind ==FILTER_REFS_BRANCHES ) - ret =refs_for_each_fullref_in (get_main_ref_store (the_repository), -"refs/heads/" ,NULL , - fn, cb_data); -else if (filter->kind ==FILTER_REFS_REMOTES ) - ret =refs_for_each_fullref_in (get_main_ref_store (the_repository), -"refs/remotes/" ,NULL , - fn, cb_data); -else if (filter->kind ==FILTER_REFS_TAGS ) - ret =refs_for_each_fullref_in (get_main_ref_store (the_repository), -"refs/tags/" ,NULL , fn, - cb_data); -else if (filter->kind &FILTER_REFS_REGULAR ) - ret =for_each_fullref_in_pattern (filter, fn, cb_data); -/* -* When printing all ref types, HEAD is already included, -* so we don't want to print HEAD again. -*/ -if (!ret && !(filter->kind &FILTER_REFS_ROOT_REFS ) && - (filter->kind &FILTER_REFS_DETACHED_HEAD )) -refs_head_ref (get_main_ref_store (the_repository), fn, - cb_data); - } +/* +* For common cases where we need only branches or remotes or tags, +* we only iterate through those refs. If a mix of refs is needed, +* we iterate over all refs and filter out required refs with the help +* of filter_ref_kind(). +*/ +if (filter->kind ==FILTER_REFS_BRANCHES ) + ret =refs_for_each_fullref_in (get_main_ref_store (the_repository), +"refs/heads/" ,NULL , + fn, cb_data); +else if (filter->kind ==FILTER_REFS_REMOTES ) + ret =refs_for_each_fullref_in (get_main_ref_store (the_repository), +"refs/remotes/" ,NULL , + fn, cb_data); +else if (filter->kind ==FILTER_REFS_TAGS ) + ret =refs_for_each_fullref_in (get_main_ref_store (the_repository), +"refs/tags/" ,NULL , fn, + cb_data); +else if (filter->kind &FILTER_REFS_REGULAR ) + ret =for_each_fullref_in_pattern (filter, fn, cb_data); + +/* +* When printing all ref types, HEAD is already included, +* so we don't want to print HEAD again. +*/ +if (!ret && !(filter->kind &FILTER_REFS_ROOT_REFS ) && + (filter->kind &FILTER_REFS_DETACHED_HEAD )) +refs_head_ref (get_main_ref_store (the_repository), fn, + cb_data); +clear_contains_cache (&filter->internal.contains_cache);clear_contains_cache (&filter->internal.no_contains_cache);
Signature · 1 line
2.49.0
PATCH v5 5/5 for-each-ref: introduce a '--start-after' option
In reply to Karthik NayakThe git-for-each-ref(1) command is used to iterate over references present in a repository. In large repositories with millions of references, it would be optimal to paginate this output such that we can start iteration from a given reference. This would avoid having to iterate over all references from the beginning each time when paginating through results.
The previous commit added 'seek' functionality to the reference backends. Utilize this and expose a '--start-after' option in 'git-for-each-ref(1)'. When used, the reference iteration seeks to the lexicographically next reference and iterates from there onward.
This enables efficient pagination workflows, where the calling script can remember the last provided reference and use that as the starting point for the next set of references:
git for-each-ref --count=100
git for-each-ref --count=100 --start-after=refs/heads/branch-100
git for-each-ref --count=100 --start-after=refs/heads/branch-200Since the reference iterators only allow seeking to a specified marker via the ref_iterator_seek(), we introduce a helper function start_ref_iterator_after(), which seeks to next reference by simply adding (char) 1 to the marker.
We must note that pagination always continues from the provided marker, as such any concurrent reference updates lexicographically behind the marker will not be output. Document the same.
--- Documentation/git-for-each-ref.adoc | 10 +- builtin/for-each-ref.c | 8 ++ ref-filter.c | 78 +++++++++++---- ref-filter.h | 1 + t/t6302-for-each-ref-filter.sh | 194 ++++++++++++++++++++++++++++++++++++ 5 files changed, 272 insertions(+), 19 deletions(-)
Documentation/git-for-each-ref.adoc +9 -1
@@ -14,7 +14,7 @@ SYNOPSIS [--points-at=<object>] [--merged[=<object>]] [--no-merged[=<object>]] [--contains[=<object>]] [--no-contains[=<object>]] - [--exclude=<pattern> ...] + [--exclude=<pattern> ...] [--start-after=<marker>] DESCRIPTION -----------
@@ -108,6 +108,14 @@ TAB %(refname)`. --include-root-refs:: List root refs (HEAD and pseudorefs) apart from regular refs. +--start-after=<marker>:: + Allows paginating the output by skipping references up to and including the + specified marker. When paging, it should be noted that references may be + deleted, modified or added between invocations. Output will only yield those + references which follow the marker lexicographically. Output begins from the + first reference that would come after the marker alphabetically. Cannot be + used with general pattern matching or custom sort options. + FIELD NAMES -----------
builtin/for-each-ref.c +8 -0
@@ -13,6 +13,7 @@ static char const * const for_each_ref_usage[] = {N_ ("git for-each-ref [--points-at <object>]" ),N_ ("git for-each-ref [--merged [<commit>]] [--no-merged [<commit>]]" ),N_ ("git for-each-ref [--contains [<commit>]] [--no-contains [<commit>]]" ), +N_ ("git for-each-ref [--start-after <marker>]" ), NULL };
@@ -44,6 +45,7 @@ int cmd_for_each_ref(int argc,OPT_GROUP ("" ),OPT_INTEGER (0 ,"count" , &format.array_opts.max_count,N_ ("show only <n> matched refs" )),OPT_STRING (0 ,"format" , &format.format,N_ ("format" ),N_ ("format to use for the output" )), +OPT_STRING (0 ,"start-after" , &filter.start_after,N_ ("start-start" ),N_ ("start iteration after the provided marker" )),OPT__COLOR (&format.use_color,N_ ("respect format colors" )),OPT_REF_FILTER_EXCLUDE (&filter), OPT_REF_SORT(&sorting_options),
@@ -79,6 +81,9 @@ int cmd_for_each_ref(int argc,if (verify_ref_format (&format))usage_with_options (for_each_ref_usage, opts); +if (filter.start_after && sorting_options.nr >1 ) +die (_ ("cannot use --start-after with custom sort options" )); + sorting =ref_sorting_options (&sorting_options);ref_sorting_set_sort_flags_all (sorting,REF_SORTING_ICASE , icase); filter.ignore_case = icase;
@@ -100,6 +105,9 @@ int cmd_for_each_ref(int argc, filter.name_patterns = argv; } +if (filter.start_after && filter.name_patterns && filter.name_patterns[0 ]) +die (_ ("cannot use --start-after with patterns" )); +if (include_root_refs) flags |=FILTER_REFS_ROOT_REFS |FILTER_REFS_DETACHED_HEAD ;
ref-filter.c +60 -18
121 lines not on this page · raw message
ref-filter.h +1 -0
@@ -64,6 +64,7 @@ struct ref_array {struct ref_filter {const char **name_patterns; +const char *start_after;struct strvec exclude;struct oid_array points_at;struct commit_list *with_commit;
t/t6302-for-each-ref-filter.sh +194 -0
198 lines not on this page · raw message
Signature · 1 line
2.49.0
Re: [PATCH v5 5/5] for-each-ref: introduce a '--start-after' option
Quoted 42 lines · Karthik Nayak
The
git-for-each-ref(1)command is used to iterate over references present in a repository. In large repositories with millions of references, it would be optimal to paginate this output such that we can start iteration from a given reference. This would avoid having to iterate over all references from the beginning each time when paginating through results.The previous commit added 'seek' functionality to the reference backends. Utilize this and expose a '--start-after' option in 'git-for-each-ref(1)'. When used, the reference iteration seeks to the lexicographically next reference and iterates from there onward.
This enables efficient pagination workflows, where the calling script can remember the last provided reference and use that as the starting point for the next set of references:
git for-each-ref --count=100 git for-each-ref --count=100 --start-after=refs/heads/branch-100 git for-each-ref --count=100 --start-after=refs/heads/branch-200Since the reference iterators only allow seeking to a specified marker via the
ref_iterator_seek(), we introduce a helper functionstart_ref_iterator_after(), which seeks to next reference by simply adding (char) 1 to the marker.We must note that pagination always continues from the provided marker, as such any concurrent reference updates lexicographically behind the marker will not be output. Document the same.
Signed-off-by Karthik Nayak <[email protected]>--- Documentation/git-for-each-ref.adoc | 10 +- builtin/for-each-ref.c | 8 ++ ref-filter.c | 78 +++++++++++---- ref-filter.h | 1 + t/t6302-for-each-ref-filter.sh | 194 ++++++++++++++++++++++++++++++++++++ 5 files changed, 272 insertions(+), 19 deletions(-)Documentation/git-for-each-ref.adoc +0
-0
Documentation/git-for-each-ref.adoc SYNOPSIS · line 17
Quoted 5 lines
Documentation/git-for-each-ref.adoc +1
-1@@ -14,7 +14,7 @@ SYNOPSIS [--points-at=<object>] [--merged[=<object>]] [--no-merged[=<object>]] [--contains[=<object>]] [--no-contains[=<object>]] - [--exclude=<pattern> ...] + [--exclude=<pattern> ...] [--start-after=<marker>]
Not a problem this patch introduces, but as I noticed it, let me leave a #leftoverbits comment here (it is OK to have a preliminary clean-up patch).
* "--exclude=<pattern>" should be enclosed inside a pair of (parentheses), just like the way how [(--sort=<key>)...] is shown.
* [--stdin | <pattern>...] should be moved to the end. There is no reason to require "--stdin" to be the end of dashed options, but the <pattern>... must be, as they are positional, not dashed.
Documentation/git-for-each-ref.adoc TAB %(refname)`. · line 117
Quoted 7 lines
Documentation/git-for-each-ref.adoc +7
-0+--start-after=<marker>:: + Allows paginating the output by skipping references up to and including the + specified marker. When paging, it should be noted that references may be + deleted, modified or added between invocations. Output will only yield those + references which follow the marker lexicographically. Output begins from the + first reference that would come after the marker alphabetically. Cannot be + used with general pattern matching or custom sort options.
It is unclear what "general" in "general pattern matching" refers to.
Cannot be used with --sort=<key> or --stdin options, or the _<pattern>_ argument(s) to limit the refs.
or something, perhaps? It is curious how --exclude=<pattern> interacts with the feature. Presumably the exclusion is done so late in the output phase that it does not have any effect? It does not have to be mentioned in this documentation if that is the case as it is a mere implementation detail.
Side note. The limitation that sorting and name_patterns cannot be used with the feature also comes from implementation (i.e. the name_patterns optimization will compete with this feature to take advantage of the "prefix" thing in an incompatible way), so while the reason does not have to be stated in the end-user facing documentation, the effect needs documenting.
Quoted 4 lines
Documentation/git-for-each-ref.adoc +0
-0@@ -3189,6 +3221,7 @@ void filter_is_base(struct repository *r, static int do_filter_refs(struct ref_filter *filter, unsigned int type, each_ref_fn fn, void *cb_data) {
Documentation/git-for-each-ref.adoc void filter_is_base(struct repository *r,
Quoted 10 lines
Documentation/git-for-each-ref.adoc +9
-0+ const char *prefix = NULL; + + if (prefix) { + struct ref_iterator *iter; + + iter = refs_ref_iterator_begin(get_main_ref_store(the_repository), + "", NULL, 0, 0); + + if (filter->start_after)
The start_after of the filter comes from "--start-after=<mark>". Can it be true with non-NULL prefix at this point? Unless you add support for the option to "git branch/tag", it would not happen, I guess.
More importantly, when you do add support to "git branch/tag", the code need to be updated to keep the original prefix while seeking the cursor to the specified <mark>, instead of clearing it.
Documentation/git-for-each-ref.adoc
Quoted 3 lines
Documentation/git-for-each-ref.adoc +3
-0+ ret = start_ref_iterator_after(iter, filter->start_after); + else if (prefix) + ret = ref_iterator_seek(iter, prefix, 1);
We have "REF_ITERATOR_SEEK_SET_PREFIX" for that "1"?
Re: [PATCH v5 5/5] for-each-ref: introduce a '--start-after' option
Quoted 54 lines · Junio C Hamano
Quoted 42 lines · Karthik Nayak
The
git-for-each-ref(1)command is used to iterate over references present in a repository. In large repositories with millions of references, it would be optimal to paginate this output such that we can start iteration from a given reference. This would avoid having to iterate over all references from the beginning each time when paginating through results.The previous commit added 'seek' functionality to the reference backends. Utilize this and expose a '--start-after' option in 'git-for-each-ref(1)'. When used, the reference iteration seeks to the lexicographically next reference and iterates from there onward.
This enables efficient pagination workflows, where the calling script can remember the last provided reference and use that as the starting point for the next set of references:
git for-each-ref --count=100 git for-each-ref --count=100 --start-after=refs/heads/branch-100 git for-each-ref --count=100 --start-after=refs/heads/branch-200Since the reference iterators only allow seeking to a specified marker via the
ref_iterator_seek(), we introduce a helper functionstart_ref_iterator_after(), which seeks to next reference by simply adding (char) 1 to the marker.We must note that pagination always continues from the provided marker, as such any concurrent reference updates lexicographically behind the marker will not be output. Document the same.
Signed-off-by Karthik Nayak <[email protected]>--- Documentation/git-for-each-ref.adoc | 10 +- builtin/for-each-ref.c | 8 ++ ref-filter.c | 78 +++++++++++---- ref-filter.h | 1 + t/t6302-for-each-ref-filter.sh | 194 ++++++++++++++++++++++++++++++++++++ 5 files changed, 272 insertions(+), 19 deletions(-)Documentation/git-for-each-ref.adoc +0
-0Documentation/git-for-each-ref.adoc SYNOPSIS · line 17
Quoted 5 lines
Documentation/git-for-each-ref.adoc +1
-1@@ -14,7 +14,7 @@ SYNOPSIS [--points-at=<object>] [--merged[=<object>]] [--no-merged[=<object>]] [--contains[=<object>]] [--no-contains[=<object>]] - [--exclude=<pattern> ...] + [--exclude=<pattern> ...] [--start-after=<marker>]Not a problem this patch introduces, but as I noticed it, let me leave a #leftoverbits comment here (it is OK to have a preliminary clean-up patch).
Quoted 8 lines
* "--exclude=<pattern>" should be enclosed inside a pair of (parentheses), just like the way how [(--sort=<key>)...] is shown.
* [--stdin | <pattern>...] should be moved to the end. There is no reason to require "--stdin" to be the end of dashed options, but the <pattern>... must be, as they are positional, not dashed.
I'm sending in a series of these small fixes, so I'll add this in.
Quoted 8 lines
Documentation/git-for-each-ref.adoc TAB %(refname)`. · line 117
Quoted 7 lines
Documentation/git-for-each-ref.adoc +7
-0+--start-after=<marker>:: + Allows paginating the output by skipping references up to and including the + specified marker. When paging, it should be noted that references may be + deleted, modified or added between invocations. Output will only yield those + references which follow the marker lexicographically. Output begins from the + first reference that would come after the marker alphabetically. Cannot be + used with general pattern matching or custom sort options.
Quoted 6 lines
It is unclear what "general" in "general pattern matching" refers to.
Cannot be used with
--sort=<key>or--stdinoptions, or the _<pattern>_ argument(s) to limit the refs.
This does read much better. I'll also add this.
Quoted 5 lines
or something, perhaps? It is curious how
--exclude=<pattern>interacts with the feature. Presumably the exclusion is done so late in the output phase that it does not have any effect? It does not have to be mentioned in this documentation if that is the case as it is a mere implementation detail.
That is correct indeed, while this doesn't have to be documented, I think we can merit from a test. So I'll add that in.
Quoted 29 lines
Side note. The limitation that sorting and
name_patternscannot be used with the feature also comes from implementation (i.e. thename_patternsoptimization will compete with this feature to take advantage of the "prefix" thing in an incompatible way), so while the reason does not have to be stated in the end-user facing documentation, the effect needs documenting.Quoted 4 lines
Documentation/git-for-each-ref.adoc +0
-0@@ -3189,6 +3221,7 @@ void filter_is_base(struct repository *r, static int do_filter_refs(struct ref_filter *filter, unsigned int type, each_ref_fn fn, void *cb_data) {Documentation/git-for-each-ref.adoc void filter_is_base(struct repository *r,
Quoted 10 lines
Documentation/git-for-each-ref.adoc +9
-0+ const char *prefix = NULL; + + if (prefix) { + struct ref_iterator *iter; + + iter = refs_ref_iterator_begin(get_main_ref_store(the_repository), + "", NULL, 0, 0); + + if (filter->start_after)The
start_afterof the filter comes from "--start-after=<mark>". Can it be true with non-NULL prefix at this point? Unless you add support for the option to "git branch/tag", it would not happen, I guess.
Quoted 4 lines
More importantly, when you do add support to "git branch/tag", the code need to be updated to keep the original prefix while seeking the cursor to the specified <mark>, instead of clearing it.
Exactly, if we do add '<pattern>' and '--start-after' compatibility, we'll have to make that change.
Quoted 5 lines
Documentation/git-for-each-ref.adoc
Quoted 3 lines
Documentation/git-for-each-ref.adoc +3
-0+ ret = start_ref_iterator_after(iter, filter->start_after); + else if (prefix) + ret = ref_iterator_seek(iter, prefix, 1);We have "
REF_ITERATOR_SEEK_SET_PREFIX" for that "1"?
Yup, also the 'if (prefix)' can be dropped too.
Re: [PATCH v5 0/5] for-each-ref: introduce seeking functionality via '--start-after'
In reply to Karthik NayakQuoted 9 lines · Karthik Nayak
Changes in v5:
- Changes to the comments to refer to the flag
'REF_ITERATOR_SEEK_SET_PREFIX' instead of a variable used in older
versions. Also other small grammar fixes.
- Added a commit to remove an unnecessary else clause.
- Move seeking functionality withinfor_each_fullref_in_patternto its
own function.
- Fix incorrect naming in the tests.
- Link to v4: https://lore.kernel.org/r/20250711-306-git-for-each-ref-pagination-v4-0-ed3303ad5b89@gmail.com
The two refactoring differences relative to the previous round do make the result more plesant to eyes. Looking great.
Will replace. Thanks.
Re: [PATCH v5 0/5] for-each-ref: introduce seeking functionality via '--start-after'
There's something in this series that's triggering an msan warning in t/t6302-for-each-ref-filter:
==2147==WARNING: MemorySanitizer: use-of-uninitialized-value
#0 0x562a64e923cc in cache_ref_iterator_advance refs/ref-cache.c:409:27
#1 0x562a64e88dc0 in ref_iterator_advance refs/iterator.c:15:9
#2 0x562a64e88dc0 in merge_ref_iterator_advance refs/iterator.c:164:13
#3 0x562a64e8850b in ref_iterator_advance refs/iterator.c:15:9
#4 0x562a64e85f5a in files_ref_iterator_advance refs/files-backend.c:902:15
#5 0x562a64e88bb4 in ref_iterator_advance refs/iterator.c:15:9
#6 0x562a64e88bb4 in do_for_each_ref_iterator refs/iterator.c:478:15
#7 0x562a64e64593 in for_each_fullref_with_seek ref-filter.c:2718:9
#8 0x562a64e5cfe8 in for_each_fullref_in_pattern ref-filter.c
#9 0x562a64e5cfe8 in do_filter_refs ref-filter.c:3263:9
#10 0x562a64e5d7fc in filter_and_format_refs ref-filter.c:3364:3
#11 0x562a64af0235 in cmd_for_each_ref builtin/for-each-ref.c:115:2
#12 0x562a64a3ebdc in run_builtin git.c:480:11
#13 0x562a64a3d342 in handle_builtin git.c:746:9
#14 0x562a64a3be33 in run_argv git.c:813:4
#15 0x562a64a3be33 in cmd_main git.c:953:19
#16 0x562a64c2f12f in main common-main.c:9:11SUMMARY: MemorySanitizer: use-of-uninitialized-value refs/ref-cache.c:409:27 in cache_ref_iterator_advance
Unfortunately I can't provide great instructions for reproducing this locally, because it relies on our internal build stack (which uses blaze). Getting MemorySanitizer running can be quite annoying, though you might not have any issues if this test doesn't invoke any third party libraries (like zlib).
I need to sign off for the night soon, but if this isn't sufficient enough information to identify what's happening here, I can try to dig deeper tomorrow. This run was executed on an import of upstream commit 4ea3c74afd42a503b3e0d60e1fec33bc0431e7bc (Junio's merge of this series)
Re: [PATCH v5 0/5] for-each-ref: introduce seeking functionality via '--start-after'
Quoted 11 lines · Kyle Lippincott
Unfortunately I can't provide great instructions for reproducing this locally, because it relies on our internal build stack (which uses blaze). Getting MemorySanitizer running can be quite annoying, though you might not have any issues if this test doesn't invoke any third party libraries (like zlib).
I need to sign off for the night soon, but if this isn't sufficient enough information to identify what's happening here, I can try to dig deeper tomorrow. This run was executed on an import of upstream commit
4ea3c74afd42a503b3e0d60e1fec33bc0431e7bc(Junio's merge of this series)
valgrind can often find the same issues as MSan without as much headache to get it running (the downside is that it is _way_ slower). And indeed:
git checkout 4ea3c74afd42a503b3e0d60e1fec33bc0431e7bc && make && (cd t && ./t6302-for-each-ref-filter.sh --valgrind-only=48)
yields:
==2177572== Conditional jump or move depends on uninitialised value(s)
==2177572== at 0x3BC380: cache_ref_iterator_advance (ref-cache.c:409) ==2177572== by 0x3B69D7: ref_iterator_advance (iterator.c:15) ==2177572== by 0x3B6CC3: merge_ref_iterator_advance (iterator.c:179) ==2177572== by 0x3B69D7: ref_iterator_advance (iterator.c:15) ==2177572== by 0x3A9770: files_ref_iterator_advance (files-backend.c:902) ==2177572== by 0x3B69D7: ref_iterator_advance (iterator.c:15) ==2177572== by 0x3B7457: do_for_each_ref_iterator (iterator.c:478) ==2177572== by 0x399B43: for_each_fullref_with_seek (ref-filter.c:2718) ==2177572== by 0x399C09: for_each_fullref_in_pattern (ref-filter.c:2756) ==2177572== by 0x39B031: do_filter_refs (ref-filter.c:3263) ==2177572== by 0x39B2B7: filter_and_format_refs (ref-filter.c:3364) ==2177572== by 0x18C1D2: cmd_for_each_ref (for-each-ref.c:115) ==2177572== Uninitialised value was created by a heap allocation ==2177572== at 0x484BDD0: realloc (vg_replace_malloc.c:1801) ==2177572== by 0x44E941: xrealloc (wrapper.c:140) ==2177572== by 0x3BCAD9: cache_ref_iterator_begin (ref-cache.c:580) ==2177572== by 0x3A988A: files_ref_iterator_begin (files-backend.c:995) ==2177572== by 0x3A295E: refs_ref_iterator_begin (refs.c:1776) ==2177572== by 0x399AF6: for_each_fullref_with_seek (ref-filter.c:2710) ==2177572== by 0x399C09: for_each_fullref_in_pattern (ref-filter.c:2756) ==2177572== by 0x39B031: do_filter_refs (ref-filter.c:3263) ==2177572== by 0x39B2B7: filter_and_format_refs (ref-filter.c:3364) ==2177572== by 0x18C1D2: cmd_for_each_ref (for-each-ref.c:115) ==2177572== by 0x128C90: run_builtin (git.c:480) ==2177572== by 0x1290EB: handle_builtin (git.c:746)
Bisecting doesn't tell us much, though (the first commit that introduces the test shows the problem). I didn't dig further than that.
-Peff
Re: [PATCH v5 0/5] for-each-ref: introduce seeking functionality via '--start-after'
Quoted 51 lines · Jeff King
Quoted 11 lines · Kyle Lippincott
Unfortunately I can't provide great instructions for reproducing this locally, because it relies on our internal build stack (which uses blaze). Getting MemorySanitizer running can be quite annoying, though you might not have any issues if this test doesn't invoke any third party libraries (like zlib).
I need to sign off for the night soon, but if this isn't sufficient enough information to identify what's happening here, I can try to dig deeper tomorrow. This run was executed on an import of upstream commit
4ea3c74afd42a503b3e0d60e1fec33bc0431e7bc(Junio's merge of this series)valgrind can often find the same issues as MSan without as much headache to get it running (the downside is that it is _way_ slower). And indeed:
git checkout 4ea3c74afd42a503b3e0d60e1fec33bc0431e7bc && make && (cd t && ./t6302-for-each-ref-filter.sh --valgrind-only=48)yields:
==2177572== Conditional jump or move depends on uninitialised value(s)
==2177572== at 0x3BC380: cache_ref_iterator_advance (ref-cache.c:409) ==2177572== by 0x3B69D7: ref_iterator_advance (iterator.c:15) ==2177572== by 0x3B6CC3: merge_ref_iterator_advance (iterator.c:179) ==2177572== by 0x3B69D7: ref_iterator_advance (iterator.c:15) ==2177572== by 0x3A9770: files_ref_iterator_advance (files-backend.c:902) ==2177572== by 0x3B69D7: ref_iterator_advance (iterator.c:15) ==2177572== by 0x3B7457: do_for_each_ref_iterator (iterator.c:478) ==2177572== by 0x399B43: for_each_fullref_with_seek (ref-filter.c:2718) ==2177572== by 0x399C09: for_each_fullref_in_pattern (ref-filter.c:2756) ==2177572== by 0x39B031: do_filter_refs (ref-filter.c:3263) ==2177572== by 0x39B2B7: filter_and_format_refs (ref-filter.c:3364) ==2177572== by 0x18C1D2: cmd_for_each_ref (for-each-ref.c:115) ==2177572== Uninitialised value was created by a heap allocation ==2177572== at 0x484BDD0: realloc (vg_replace_malloc.c:1801) ==2177572== by 0x44E941: xrealloc (wrapper.c:140) ==2177572== by 0x3BCAD9: cache_ref_iterator_begin (ref-cache.c:580) ==2177572== by 0x3A988A: files_ref_iterator_begin (files-backend.c:995) ==2177572== by 0x3A295E: refs_ref_iterator_begin (refs.c:1776) ==2177572== by 0x399AF6: for_each_fullref_with_seek (ref-filter.c:2710) ==2177572== by 0x399C09: for_each_fullref_in_pattern (ref-filter.c:2756) ==2177572== by 0x39B031: do_filter_refs (ref-filter.c:3263) ==2177572== by 0x39B2B7: filter_and_format_refs (ref-filter.c:3364) ==2177572== by 0x18C1D2: cmd_for_each_ref (for-each-ref.c:115) ==2177572== by 0x128C90: run_builtin (git.c:480) ==2177572== by 0x1290EB: handle_builtin (git.c:746)
Quoted 4 lines
Bisecting doesn't tell us much, though (the first commit that introduces the test shows the problem). I didn't dig further than that.
-Peff
Thanks for that, that helped me a bit too as it provides more information than I was getting out of MemorySanitizer (I suspect MemorySanitizer was producing the information it just wasn't going to stderr or something, or maybe I was missing a flag to get it to report more). I'm not sure what the right fix would be; my guess is that the fix would be to modify the places where we set levels_nr and initialize the other fields in level to also set it to prefix_state (around lines 488 and 527 in ref-cache.c); and indeed setting the prefix_state to PREFIX_CONTAINS_DIR (the 0 value of the enum) makes the test pass even under valgrind. Unfortunately without a much more in-depth knowledge of the code and the enum values I can't definitively state that those are the correct values. I can say that setting it to PREFIX_WITHIN_DIR causes both additional valgrind failures and test failures even without valgrind, but setting it to PREFIX_EXCLUDES_DIR doesn't seem to be a problem. I also moved the if around like 409 into the following if, because that was the only time entry_prefix_state was used, I'd been thinking that maybe it needed the check for entry->flag & REF_DIR prior to referencing level->prefix_state, but that didn't resolve it on its own.
I don't mind if anyone else picks up this fix and runs with it, but I'm not comfortable sending this patch myself because I don't have enough knowledge of this are of the code to know if it's right, just that it fixes the issue we encountered, and I'm extremely overloaded right now and can't get that knowledge nor see the patch through to the end.
refs/ref-cache.c +0 -0
@@ -391,7 +391,6 @@ static int cache_ref_iterator_advance(struct
ref_iterator *ref_iterator)
&iter->levels[iter->levels_nr - 1];
struct ref_dir *dir = level->dir;
struct ref_entry *entry;- enum prefix_state entry_prefix_state;
if (level->index == -1)
sort_ref_dir(dir);@@ -406,16 +405,17 @@ static int cache_ref_iterator_advance(struct ref_iterator *ref_iterator)
entry = dir->entries[level->index];
- if (level->prefix_state == PREFIX_WITHIN_DIR) {
- entry_prefix_state =overlaps_prefix(entry->name, iter->prefix);
- if (entry_prefix_state == PREFIX_EXCLUDES_DIR ||
- (entry_prefix_state == PREFIX_WITHIN_DIR
&& !(entry->flag & REF_DIR)))
- continue;
- } else {
- entry_prefix_state = level->prefix_state;
- }
-
if (entry->flag & REF_DIR) {+ enum prefix_state entry_prefix_state;
+ if (level->prefix_state == PREFIX_WITHIN_DIR) {
+ entry_prefix_state =overlaps_prefix(entry->name, iter->prefix);
+ if (entry_prefix_state == PREFIX_EXCLUDES_DIR ||
+ (entry_prefix_state ==PREFIX_WITHIN_DIR && !(entry->flag & REF_DIR)))
+ continue;
+ } else {
+ entry_prefix_state = level->prefix_state;
+ }
+
/* push down a level */
ALLOC_GROW(iter->levels, iter->levels_nr + 1,
iter->levels_alloc);
@@ -489,6 +489,7 @@ static int cache_ref_iterator_seek(struct
ref_iterator *ref_iterator,
level = &iter->levels[0];
level->index = -1;
level->dir = dir;+ level->prefix_state = PREFIX_EXCLUDES_DIR; //
FIXME: PROBABLY NOT CORRECT
/* Unset any previously set prefix */
FREE_AND_NULL(iter->prefix);
@@ -527,6 +528,7 @@ static int cache_ref_iterator_seek(struct
ref_iterator *ref_iterator,
level = &iter->levels[iter->levels_nr++];
level->dir = dir;
level->index = -1;
+ level->prefix_state =
PREFIX_EXCLUDES_DIR; // FIXME: PROBABLY NOT CORRECT
} else {
/* reduce the index so the leaf node
is iterated over */
if (cmp <= 0 && !slash)Re: [PATCH v5 0/5] for-each-ref: introduce seeking functionality via '--start-after'
Quoted 55 lines · Kyle Lippincott
Quoted 54 lines · Jeff King
Quoted 11 lines · Kyle Lippincott
Unfortunately I can't provide great instructions for reproducing this locally, because it relies on our internal build stack (which uses blaze). Getting MemorySanitizer running can be quite annoying, though you might not have any issues if this test doesn't invoke any third party libraries (like zlib).
I need to sign off for the night soon, but if this isn't sufficient enough information to identify what's happening here, I can try to dig deeper tomorrow. This run was executed on an import of upstream commit
4ea3c74afd42a503b3e0d60e1fec33bc0431e7bc(Junio's merge of this series)valgrind can often find the same issues as MSan without as much headache to get it running (the downside is that it is _way_ slower). And indeed:
git checkout 4ea3c74afd42a503b3e0d60e1fec33bc0431e7bc && make && (cd t && ./t6302-for-each-ref-filter.sh --valgrind-only=48)yields:
==2177572== Conditional jump or move depends on uninitialised value(s)
==2177572== at 0x3BC380: cache_ref_iterator_advance (ref-cache.c:409) ==2177572== by 0x3B69D7: ref_iterator_advance (iterator.c:15) ==2177572== by 0x3B6CC3: merge_ref_iterator_advance (iterator.c:179) ==2177572== by 0x3B69D7: ref_iterator_advance (iterator.c:15) ==2177572== by 0x3A9770: files_ref_iterator_advance (files-backend.c:902) ==2177572== by 0x3B69D7: ref_iterator_advance (iterator.c:15) ==2177572== by 0x3B7457: do_for_each_ref_iterator (iterator.c:478) ==2177572== by 0x399B43: for_each_fullref_with_seek (ref-filter.c:2718) ==2177572== by 0x399C09: for_each_fullref_in_pattern (ref-filter.c:2756) ==2177572== by 0x39B031: do_filter_refs (ref-filter.c:3263) ==2177572== by 0x39B2B7: filter_and_format_refs (ref-filter.c:3364) ==2177572== by 0x18C1D2: cmd_for_each_ref (for-each-ref.c:115) ==2177572== Uninitialised value was created by a heap allocation ==2177572== at 0x484BDD0: realloc (vg_replace_malloc.c:1801) ==2177572== by 0x44E941: xrealloc (wrapper.c:140) ==2177572== by 0x3BCAD9: cache_ref_iterator_begin (ref-cache.c:580) ==2177572== by 0x3A988A: files_ref_iterator_begin (files-backend.c:995) ==2177572== by 0x3A295E: refs_ref_iterator_begin (refs.c:1776) ==2177572== by 0x399AF6: for_each_fullref_with_seek (ref-filter.c:2710) ==2177572== by 0x399C09: for_each_fullref_in_pattern (ref-filter.c:2756) ==2177572== by 0x39B031: do_filter_refs (ref-filter.c:3263) ==2177572== by 0x39B2B7: filter_and_format_refs (ref-filter.c:3364) ==2177572== by 0x18C1D2: cmd_for_each_ref (for-each-ref.c:115) ==2177572== by 0x128C90: run_builtin (git.c:480) ==2177572== by 0x1290EB: handle_builtin (git.c:746)Bisecting doesn't tell us much, though (the first commit that introduces the test shows the problem). I didn't dig further than that.
Quoted 8 lines
Quoted 1 line
-Peff
Thanks for that, that helped me a bit too as it provides more information than I was getting out of MemorySanitizer (I suspect MemorySanitizer was producing the information it just wasn't going to stderr or something, or maybe I was missing a flag to get it to report more).
Thanks both for raising the issue. Thanks Jeff for also the valgrind instructions.
On a sidenote, was discussing this at work and Patrick also mentioned that we could try clang's MemorySanitizer. This seems to also be raising issues on master, so it was hard to find the exact output that valgrind was providing.
$ git checkout master
$ CC=clang meson setup --reconfigure memory_build . -Db_sanitize=memory
$ cd memory_build
$ meson test -i --test-args="-ix" t6302-for-each-ref-filter
...
==3275333==WARNING: MemorySanitizer: use-of-uninitialized-value
#0 0x557bd886f4bb in git_mkstemps_mode ../wrapper.c:487:27
#1 0x557bd886fb55 in git_mkstemp_mode ../wrapper.c:509:9
#2 0x557bd8100d1a in create_tmpfile ../object-file.c:736:7
#3 0x557bd80f1630 in start_loose_object_common ../object-file.c:781:7
#4 0x557bd80f5203 in write_loose_object ../object-file.c:881:7
#5 0x557bd80f4875 in write_object_file_flags ../object-file.c:1086:6
#6 0x557bd80f9f65 in write_object_file ../object-file.h:181:9
#7 0x557bd8101eb8 in index_mem ../object-file.c:1177:9
#8 0x557bd80f8bd5 in index_core ../object-file.c:1247:10
#9 0x557bd80f731d in index_fd ../object-file.c:1274:9
#10 0x557bd80f95e4 in index_path ../object-file.c:1295:7
#11 0x557bd831132d in add_to_index ../read-cache.c:771:7
#12 0x557bd8313cb1 in add_file_to_index ../read-cache.c:804:9
#13 0x557bd73f892c in add_files ../builtin/add.c:355:7
#14 0x557bd73f4752 in cmd_add ../builtin/add.c:578:18
#15 0x557bd7a38b6f in run_builtin ../git.c:480:11
#16 0x557bd7a31d54 in handle_builtin ../git.c:746:9
#17 0x557bd7a36644 in run_argv ../git.c:813:4
#18 0x557bd7a30e09 in cmd_main ../git.c:953:19
#19 0x557bd7a3ca01 in main ../common-main.c:9:11
#20 0x7f7e3f02a4d7 in __libc_start_call_main
(/nix/store/g2jzxk3s7cnkhh8yq55l4fbvf639zy37-glibc-2.40-66/lib/libc.so.6+0x2a4d7)
(BuildId: f117ee0f586dfa828cbdd08e37393c8f04f6480a)
#21 0x7f7e3f02a59a in __libc_start_main@GLIBC_2.2.5
(/nix/store/g2jzxk3s7cnkhh8yq55l4fbvf639zy37-glibc-2.40-66/lib/libc.so.6+0x2a59a)
(BuildId: f117ee0f586dfa828cbdd08e37393c8f04f6480a)
#22 0x557bd7352b34 in _start (git+0x5db34)Possibly something we need to look into cleaning up.
Quoted 16 lines
I'm not sure what the right fix would be; my guess is that the fix would be to modify the places where we set
levels_nrand initialize the other fields in level to also set it toprefix_state(around lines 488 and 527 inref-cache.c); and indeed setting theprefix_statetoPREFIX_CONTAINS_DIR(the 0 value of the enum) makes the test pass even under valgrind. Unfortunately without a much more in-depth knowledge of the code and the enum values I can't definitively state that those are the correct values. I can say that setting it toPREFIX_WITHIN_DIRcauses both additional valgrind failures and test failures even without valgrind, but setting it toPREFIX_EXCLUDES_DIRdoesn't seem to be a problem. I also moved theifaround like 409 into the following if, because that was the only timeentry_prefix_statewas used, I'd been thinking that maybe it needed the check forentry->flag&REF_DIRprior to referencinglevel->prefix_state, but that didn't resolve it on its own.
Quoted 7 lines
I don't mind if anyone else picks up this fix and runs with it, but I'm not comfortable sending this patch myself because I don't have enough knowledge of this are of the code to know if it's right, just that it fixes the issue we encountered, and I'm extremely overloaded right now and can't get that knowledge nor see the patch through to the end.
Thanks for taking a stab at this, your inference is correct. Let me clairfy some parts of it.
So the 'ref-cache' iteration logic is used to provide iteration over loose refs (which consists of directories and entries). Anytime we come across a directory, we add it to the level variable, which acts as stack, when all entries under the current level are yielded, we pop the stack to obtain the next level to iterate. This ensures we iterate over all directories recursively.
Before this series, the seek function was used to set the prefix for iteration, which meant we need to find the directory for matching the prefix and only iterate over that level and its subdirs. If the prefix provided was a directory like 'refs/heads/' then all refs under that would be yielded (PREFIX_CONTAINS_DIR). If the prefix was 'refs/heads/foo', then the level would be set to 'ref/heads/' with the PREFIX_WITHIN_DIR flag set since only some refs within the dir would match the prefix. Entries which didn't overlap the prefix are denoted by PREFIX_EXCLUDES_DIR.
This series allows the seek function to set the cursor without setting the prefix, which is a requirement for pagination. So there is no need to set 'prefix_state' for this functionality. Which is why I didn't set it, since the default value of '0' (PREFIX_CONTAINS_DIR) would be the correct setting for all dirs. This causes the issue.
So the only fix required would be
refs/ref-cache.c +0 -0
@@ -527,6 +527,7 @@ static int cache_ref_iterator_seek(struct
ref_iterator *ref_iterator,
level = &iter->levels[iter->levels_nr++]; level->dir = dir;
level->index = -1;
+ level->prefix_state = PREFIX_CONTAINS_DIR;
} else {
/* reduce the index so the leaf node is iterated over */ if (cmp <= 0 && !slash)
The other location (Line 488), is not needed because that is the root directory and the 'prefix_state' for it is set in 'cache_ref_iterator_set_prefix()' when the iterator begins.
Quoted 67 lines
refs/ref-cache.c +0
-1@@ -391,7 +391,6 @@ static int cache_ref_iterator_advance(struct ref_iterator *ref_iterator) &iter->levels[iter->levels_nr -1 ];struct ref_dir *dir = level->dir;struct ref_entry *entry; -enum prefix_state entry_prefix_state;if (level->index ==-1 )sort_ref_dir(dir);refs/ref-cache.c +3
-9@@ -406,16 +405,17 @@ static int cache_ref_iterator_advance(struct ref_iterator *ref_iterator) entry = dir->entries[level->index]; -if (level->prefix_state ==PREFIX_WITHIN_DIR ) { - entry_prefix_state =overlaps_prefix (entry->name, iter->prefix); -if (entry_prefix_state ==PREFIX_EXCLUDES_DIR || - (entry_prefix_state ==PREFIX_WITHIN_DIR && !(entry->flag & REF_DIR))) -continue ; - }else { - entry_prefix_state = level->prefix_state; - } -if (entry->flag &REF_DIR ) { +enum prefix_state entry_prefix_state; +if (level->prefix_state ==PREFIX_WITHIN_DIR ) { + entry_prefix_state =
overlaps_prefix(entry->name, iter->prefix);refs/ref-cache.c +2
-0+ if (entry_prefix_state == PREFIX_EXCLUDES_DIR || + (entry_prefix_state ==
PREFIX_WITHIN_DIR&& !(entry->flag &REF_DIR)))refs/ref-cache.c +6
-0+continue ; + }else { + entry_prefix_state = level->prefix_state; + } +/* push down a level */ ALLOC_GROW (iter->levels, iter->levels_nr +1 , iter->levels_alloc);@@ -489,6 +489,7 @@ static int cache_ref_iterator_seek(structref_iterator *ref_iterator, level = &iter->levels[0 ]; level->index =-1 ; level->dir = dir; + level->prefix_state =PREFIX_EXCLUDES_DIR ;// FIXME: PROBABLY NOT CORRECT
/* Unset any previously set prefix */ FREE_AND_NULL(iter->prefix);refs/ref-cache.c +1
-0@@ -527,6 +528,7 @@ static int cache_ref_iterator_seek(structref_iterator *ref_iterator, level = &iter->levels[iter->levels_nr++]; level->dir = dir; level->index =-1 ; + level->prefix_state =
PREFIX_EXCLUDES_DIR; // FIXME: PROBABLY NOT CORRECT} else { /* reduce the index so the leaf node is iterated over */ if (cmp <= 0 && !slash)
Re: [PATCH v5 0/5] for-each-ref: introduce seeking functionality via '--start-after'
Quoted 109 lines · Karthik Nayak
Quoted 55 lines · Kyle Lippincott
Quoted 54 lines · Jeff King
Quoted 11 lines · Kyle Lippincott
Unfortunately I can't provide great instructions for reproducing this locally, because it relies on our internal build stack (which uses blaze). Getting MemorySanitizer running can be quite annoying, though you might not have any issues if this test doesn't invoke any third party libraries (like zlib).
I need to sign off for the night soon, but if this isn't sufficient enough information to identify what's happening here, I can try to dig deeper tomorrow. This run was executed on an import of upstream commit
4ea3c74afd42a503b3e0d60e1fec33bc0431e7bc(Junio's merge of this series)valgrind can often find the same issues as MSan without as much headache to get it running (the downside is that it is _way_ slower). And indeed:
git checkout 4ea3c74afd42a503b3e0d60e1fec33bc0431e7bc && make && (cd t && ./t6302-for-each-ref-filter.sh --valgrind-only=48)yields:
==2177572== Conditional jump or move depends on uninitialised value(s)
==2177572== at 0x3BC380: cache_ref_iterator_advance (ref-cache.c:409) ==2177572== by 0x3B69D7: ref_iterator_advance (iterator.c:15) ==2177572== by 0x3B6CC3: merge_ref_iterator_advance (iterator.c:179) ==2177572== by 0x3B69D7: ref_iterator_advance (iterator.c:15) ==2177572== by 0x3A9770: files_ref_iterator_advance (files-backend.c:902) ==2177572== by 0x3B69D7: ref_iterator_advance (iterator.c:15) ==2177572== by 0x3B7457: do_for_each_ref_iterator (iterator.c:478) ==2177572== by 0x399B43: for_each_fullref_with_seek (ref-filter.c:2718) ==2177572== by 0x399C09: for_each_fullref_in_pattern (ref-filter.c:2756) ==2177572== by 0x39B031: do_filter_refs (ref-filter.c:3263) ==2177572== by 0x39B2B7: filter_and_format_refs (ref-filter.c:3364) ==2177572== by 0x18C1D2: cmd_for_each_ref (for-each-ref.c:115) ==2177572== Uninitialised value was created by a heap allocation ==2177572== at 0x484BDD0: realloc (vg_replace_malloc.c:1801) ==2177572== by 0x44E941: xrealloc (wrapper.c:140) ==2177572== by 0x3BCAD9: cache_ref_iterator_begin (ref-cache.c:580) ==2177572== by 0x3A988A: files_ref_iterator_begin (files-backend.c:995) ==2177572== by 0x3A295E: refs_ref_iterator_begin (refs.c:1776) ==2177572== by 0x399AF6: for_each_fullref_with_seek (ref-filter.c:2710) ==2177572== by 0x399C09: for_each_fullref_in_pattern (ref-filter.c:2756) ==2177572== by 0x39B031: do_filter_refs (ref-filter.c:3263) ==2177572== by 0x39B2B7: filter_and_format_refs (ref-filter.c:3364) ==2177572== by 0x18C1D2: cmd_for_each_ref (for-each-ref.c:115) ==2177572== by 0x128C90: run_builtin (git.c:480) ==2177572== by 0x1290EB: handle_builtin (git.c:746)Bisecting doesn't tell us much, though (the first commit that introduces the test shows the problem). I didn't dig further than that.
Quoted 8 lines
Quoted 1 line
-Peff
Thanks for that, that helped me a bit too as it provides more information than I was getting out of MemorySanitizer (I suspect MemorySanitizer was producing the information it just wasn't going to stderr or something, or maybe I was missing a flag to get it to report more).
Thanks both for raising the issue. Thanks Jeff for also the valgrind instructions.
On a sidenote, was discussing this at work and Patrick also mentioned that we could try clang's MemorySanitizer. This seems to also be raising issues on master, so it was hard to find the exact output that valgrind was providing.
$ git checkout master $ CC=clang meson setup --reconfigure memory_build . -Db_sanitize=memory $ cd memory_build $ meson test -i --test-args="-ix" t6302-for-each-ref-filter ... ==3275333==WARNING: MemorySanitizer: use-of-uninitialized-value #0 0x557bd886f4bb in git_mkstemps_mode ../wrapper.c:487:27 #1 0x557bd886fb55 in git_mkstemp_mode ../wrapper.c:509:9 #2 0x557bd8100d1a in create_tmpfile ../object-file.c:736:7 #3 0x557bd80f1630 in start_loose_object_common ../object-file.c:781:7 #4 0x557bd80f5203 in write_loose_object ../object-file.c:881:7 #5 0x557bd80f4875 in write_object_file_flags ../object-file.c:1086:6 #6 0x557bd80f9f65 in write_object_file ../object-file.h:181:9 #7 0x557bd8101eb8 in index_mem ../object-file.c:1177:9 #8 0x557bd80f8bd5 in index_core ../object-file.c:1247:10 #9 0x557bd80f731d in index_fd ../object-file.c:1274:9 #10 0x557bd80f95e4 in index_path ../object-file.c:1295:7 #11 0x557bd831132d in add_to_index ../read-cache.c:771:7 #12 0x557bd8313cb1 in add_file_to_index ../read-cache.c:804:9 #13 0x557bd73f892c in add_files ../builtin/add.c:355:7 #14 0x557bd73f4752 in cmd_add ../builtin/add.c:578:18 #15 0x557bd7a38b6f in run_builtin ../git.c:480:11 #16 0x557bd7a31d54 in handle_builtin ../git.c:746:9 #17 0x557bd7a36644 in run_argv ../git.c:813:4 #18 0x557bd7a30e09 in cmd_main ../git.c:953:19 #19 0x557bd7a3ca01 in main ../common-main.c:9:11 #20 0x7f7e3f02a4d7 in __libc_start_call_main (/nix/store/g2jzxk3s7cnkhh8yq55l4fbvf639zy37-glibc-2.40-66/lib/libc.so.6+0x2a4d7) (BuildId: f117ee0f586dfa828cbdd08e37393c8f04f6480a) #21 0x7f7e3f02a59a in __libc_start_main@GLIBC_2.2.5 (/nix/store/g2jzxk3s7cnkhh8yq55l4fbvf639zy37-glibc-2.40-66/lib/libc.so.6+0x2a59a) (BuildId: f117ee0f586dfa828cbdd08e37393c8f04f6480a) #22 0x557bd7352b34 in _start (git+0x5db34)
Quoted 1 line
Possibly something we need to look into cleaning up.
I also saw those msan issues when trying make CFLAGS=-fsanitize=memory CC=clang, but not with Google's internal msan build. I don't know which variable in wrapper.c:487 it's complaining about - you'd think it'd be letters, but if it's v, then that potentially comes from OpenSSL or some other library, and that library would also need to be built with msan (which is why it's such a pain to get msan builds working - EVERY library needs to be built with memory sanitizer).
Quoted 139 lines
Quoted 17 lines
I'm not sure what the right fix would be; my guess is that the fix would be to modify the places where we set
levels_nrand initialize the other fields in level to also set it toprefix_state(around lines 488 and 527 inref-cache.c); and indeed setting theprefix_statetoPREFIX_CONTAINS_DIR(the 0 value of the enum) makes the test pass even under valgrind. Unfortunately without a much more in-depth knowledge of the code and the enum values I can't definitively state that those are the correct values. I can say that setting it toPREFIX_WITHIN_DIRcauses both additional valgrind failures and test failures even without valgrind, but setting it toPREFIX_EXCLUDES_DIRdoesn't seem to be a problem. I also moved theifaround like 409 into the following if, because that was the only timeentry_prefix_statewas used, I'd been thinking that maybe it needed the check forentry->flag&REF_DIRprior to referencinglevel->prefix_state, but that didn't resolve it on its own.Quoted 7 lines
I don't mind if anyone else picks up this fix and runs with it, but I'm not comfortable sending this patch myself because I don't have enough knowledge of this are of the code to know if it's right, just that it fixes the issue we encountered, and I'm extremely overloaded right now and can't get that knowledge nor see the patch through to the end.
Thanks for taking a stab at this, your inference is correct. Let me clairfy some parts of it.
So the 'ref-cache' iteration logic is used to provide iteration over loose refs (which consists of directories and entries). Anytime we come across a directory, we add it to the level variable, which acts as stack, when all entries under the current level are yielded, we pop the stack to obtain the next level to iterate. This ensures we iterate over all directories recursively.
Before this series, the seek function was used to set the prefix for iteration, which meant we need to find the directory for matching the prefix and only iterate over that level and its subdirs. If the prefix provided was a directory like 'refs/heads/' then all refs under that would be yielded (
PREFIX_CONTAINS_DIR). If the prefix was 'refs/heads/foo', then the level would be set to 'ref/heads/' with thePREFIX_WITHIN_DIRflag set since only some refs within the dir would match the prefix. Entries which didn't overlap the prefix are denoted byPREFIX_EXCLUDES_DIR.This series allows the seek function to set the cursor without setting the prefix, which is a requirement for pagination. So there is no need to set '
prefix_state' for this functionality. Which is why I didn't set it, since the default value of '0' (PREFIX_CONTAINS_DIR) would be the correct setting for all dirs. This causes the issue.So the only fix required would be
refs/ref-cache.c +1
-0@@ -527,6 +527,7 @@ static int cache_ref_iterator_seek(structref_iterator *ref_iterator, level = &iter->levels[iter->levels_nr++]; level->dir = dir; level->index =-1 ; + level->prefix_state =PREFIX_CONTAINS_DIR ; }else {/* reduce the index so the leaf node is iterated over */ if (cmp <= 0 && !slash)The other location (Line 488), is not needed because that is the root directory and the '
prefix_state' for it is set in 'cache_ref_iterator_set_prefix()' when the iterator begins.Quoted 67 lines
refs/ref-cache.c +0
-1@@ -391,7 +391,6 @@ static int cache_ref_iterator_advance(struct ref_iterator *ref_iterator) &iter->levels[iter->levels_nr -1 ];struct ref_dir *dir = level->dir;struct ref_entry *entry; -enum prefix_state entry_prefix_state;if (level->index ==-1 )sort_ref_dir(dir);refs/ref-cache.c +3
-9@@ -406,16 +405,17 @@ static int cache_ref_iterator_advance(struct ref_iterator *ref_iterator) entry = dir->entries[level->index]; -if (level->prefix_state ==PREFIX_WITHIN_DIR ) { - entry_prefix_state =overlaps_prefix (entry->name, iter->prefix); -if (entry_prefix_state ==PREFIX_EXCLUDES_DIR || - (entry_prefix_state ==PREFIX_WITHIN_DIR && !(entry->flag & REF_DIR))) -continue ; - }else { - entry_prefix_state = level->prefix_state; - } -if (entry->flag &REF_DIR ) { +enum prefix_state entry_prefix_state; +if (level->prefix_state ==PREFIX_WITHIN_DIR ) { + entry_prefix_state =
overlaps_prefix(entry->name, iter->prefix);refs/ref-cache.c +2
-0+ if (entry_prefix_state == PREFIX_EXCLUDES_DIR || + (entry_prefix_state ==
PREFIX_WITHIN_DIR&& !(entry->flag &REF_DIR)))refs/ref-cache.c +6
-0+continue ; + }else { + entry_prefix_state = level->prefix_state; + } +/* push down a level */ ALLOC_GROW (iter->levels, iter->levels_nr +1 , iter->levels_alloc);@@ -489,6 +489,7 @@ static int cache_ref_iterator_seek(structref_iterator *ref_iterator, level = &iter->levels[0 ]; level->index =-1 ; level->dir = dir; + level->prefix_state =PREFIX_EXCLUDES_DIR ;// FIXME: PROBABLY NOT CORRECT
/* Unset any previously set prefix */ FREE_AND_NULL(iter->prefix);refs/ref-cache.c +1
-0@@ -527,6 +528,7 @@ static int cache_ref_iterator_seek(structref_iterator *ref_iterator, level = &iter->levels[iter->levels_nr++]; level->dir = dir; level->index =-1 ; + level->prefix_state =
PREFIX_EXCLUDES_DIR; // FIXME: PROBABLY NOT CORRECT} else { /* reduce the index so the leaf node is iterated over */ if (cmp <= 0 && !slash)
Re: [PATCH v5 0/5] for-each-ref: introduce seeking functionality via '--start-after'
Quoted 31 lines · Kyle Lippincott
Quoted 29 lines
==3275333==WARNING: MemorySanitizer: use-of-uninitialized-value
#0 0x557bd886f4bb in git_mkstemps_mode ../wrapper.c:487:27 #1 0x557bd886fb55 in git_mkstemp_mode ../wrapper.c:509:9 #2 0x557bd8100d1a in create_tmpfile ../object-file.c:736:7 #3 0x557bd80f1630 in start_loose_object_common ../object-file.c:781:7 #4 0x557bd80f5203 in write_loose_object ../object-file.c:881:7 #5 0x557bd80f4875 in write_object_file_flags ../object-file.c:1086:6 #6 0x557bd80f9f65 in write_object_file ../object-file.h:181:9 #7 0x557bd8101eb8 in index_mem ../object-file.c:1177:9 #8 0x557bd80f8bd5 in index_core ../object-file.c:1247:10 #9 0x557bd80f731d in index_fd ../object-file.c:1274:9 #10 0x557bd80f95e4 in index_path ../object-file.c:1295:7 #11 0x557bd831132d in add_to_index ../read-cache.c:771:7 #12 0x557bd8313cb1 in add_file_to_index ../read-cache.c:804:9 #13 0x557bd73f892c in add_files ../builtin/add.c:355:7 #14 0x557bd73f4752 in cmd_add ../builtin/add.c:578:18 #15 0x557bd7a38b6f in run_builtin ../git.c:480:11 #16 0x557bd7a31d54 in handle_builtin ../git.c:746:9 #17 0x557bd7a36644 in run_argv ../git.c:813:4 #18 0x557bd7a30e09 in cmd_main ../git.c:953:19 #19 0x557bd7a3ca01 in main ../common-main.c:9:11 #20 0x7f7e3f02a4d7 in __libc_start_call_main (/nix/store/g2jzxk3s7cnkhh8yq55l4fbvf639zy37-glibc-2.40-66/lib/libc.so.6+0x2a4d7) (BuildId: f117ee0f586dfa828cbdd08e37393c8f04f6480a) #21 0x7f7e3f02a59a in __libc_start_main@GLIBC_2.2.5 (/nix/store/g2jzxk3s7cnkhh8yq55l4fbvf639zy37-glibc-2.40-66/lib/libc.so.6+0x2a59a) (BuildId: f117ee0f586dfa828cbdd08e37393c8f04f6480a) #22 0x557bd7352b34 in _start (git+0x5db34)Quoted 1 line
Possibly something we need to look into cleaning up.
Quoted 8 lines
I also saw those msan issues when trying
make CFLAGS=-fsanitize=memory CC=clang, but not with Google's internal msan build. I don't know which variable in wrapper.c:487 it's complaining about - you'd think it'd beletters, but if it'sv, then that potentially comes from OpenSSL or some other library, and that library would also need to be built with msan (which is why it's such a pain to get msan builds working - EVERY library needs to be built with memory sanitizer).
Yeah, presumably it is "v" from csprng_bytes(). If there are only a few such spots, we can manually "unpoison" memory coming from libraries. On my system, I didn't hit the case shown above but do have trouble with bytes coming back from zlib.
Applying this ancient patch:
https://lore.kernel.org/git/[email protected]/
and building with "make SANITIZE=memory CC=clang" let me run t6302 to completion, modulo the bug that started this thread (and which I confirmed goes away both with MSan and valgrind with the fix Karthik posted).
Probably:
wrapper.c +2 -0
@@ -482,6 +482,8 @@ int git_mkstemps_mode(char *pattern, int suffix_len, int mode)if (csprng_bytes (&v,sizeof (v),0 ) <0 )return error_errno ("unable to get random bytes for temporary file" ); +msan_unpoison (&v,sizeof (v)); +/* Fill in the random bits. */ for (i =0 ; i < num_x; i++) { filename_template[i] = letters[v % num_letters];
on top of that would fix the problem you guys are seeing. I don't know if that path leads to insanity, though. Using MSan-enabled libraries is probably a better direction (should increase accuracy, and we don't have to carry these manual annotations around).
-Peff
Re: [PATCH v5 0/5] for-each-ref: introduce seeking functionality via '--start-after'
Quoted 17 lines · Jeff King
Probably:
wrapper.c +2
-0@@ -482,6 +482,8 @@ int git_mkstemps_mode(char *pattern, int suffix_len, int mode)if (csprng_bytes (&v,sizeof (v),0 ) <0 )return error_errno ("unable to get random bytes for temporary file" ); +msan_unpoison (&v,sizeof (v)); +/* Fill in the random bits. */ for (i =0 ; i < num_x; i++) { filename_template[i] = letters[v % num_letters];
Quoted 4 lines
on top of that would fix the problem you guys are seeing. I don't know if that path leads to insanity, though. Using MSan-enabled libraries is probably a better direction (should increase accuracy, and we don't have to carry these manual annotations around).
Hmm, probably insanity. Just for fun I tried to run the whole suite, but got this doozy:
Uninitialized bytes in fopen64 at offset 0 inside [0x7020000109c0, 25) ==2568195==WARNING: MemorySanitizer: use-of-uninitialized-value
#0 0x7f90fe14fa46 in BIO_new_file (/lib/x86_64-linux-gnu/libcrypto.so.3+0x14fa46) (BuildId: 07a8321bad67632b52b47ad026125c79b7ebaab4)
#1 0x7f90fe1a659c (/lib/x86_64-linux-gnu/libcrypto.so.3+0x1a659c) (BuildId: 07a8321bad67632b52b47ad026125c79b7ebaab4)
#2 0x7f90fe1a8453 in CONF_modules_load_file_ex (/lib/x86_64-linux-gnu/libcrypto.so.3+0x1a8453) (BuildId: 07a8321bad67632b52b47ad026125c79b7ebaab4)
#3 0x7f90fe1a8807 (/lib/x86_64-linux-gnu/libcrypto.so.3+0x1a8807) (BuildId: 07a8321bad67632b52b47ad026125c79b7ebaab4)
#4 0x7f90fe27274e (/lib/x86_64-linux-gnu/libcrypto.so.3+0x27274e) (BuildId: 07a8321bad67632b52b47ad026125c79b7ebaab4)
#5 0x7f90fea01bc6 in __pthread_once_slow nptl/pthread_once.c:116:7
#6 0x7f90fea01c38 in __pthread_once nptl/pthread_once.c:143:12
#7 0x7f90fe287f3c in CRYPTO_THREAD_run_once (/lib/x86_64-linux-gnu/libcrypto.so.3+0x287f3c) (BuildId: 07a8321bad67632b52b47ad026125c79b7ebaab4)
#8 0x7f90fe272fd9 in OPENSSL_init_crypto (/lib/x86_64-linux-gnu/libcrypto.so.3+0x272fd9) (BuildId: 07a8321bad67632b52b47ad026125c79b7ebaab4)
#9 0x7f90fe78a6d7 in OPENSSL_init_ssl (/lib/x86_64-linux-gnu/libssl.so.3+0x396d7) (BuildId: a0d77cb273378dec1d74a115ac1c9e40306e675d)
#10 0x7f90fee2a503 (/lib/x86_64-linux-gnu/libcurl.so.4+0x9e503) (BuildId: 61ee7a8d1799c0e6c38a99b4d739e0c90391a05f)
#11 0x7f90fedc2e42 (/lib/x86_64-linux-gnu/libcurl.so.4+0x36e42) (BuildId: 61ee7a8d1799c0e6c38a99b4d739e0c90391a05f)
#12 0x7f90fedc33c9 in curl_global_init (/lib/x86_64-linux-gnu/libcurl.so.4+0x373c9) (BuildId: 61ee7a8d1799c0e6c38a99b4d739e0c90391a05f)
#13 0x55fbbe211623 in http_init http.c:1347:6
#14 0x55fbbe1f5b98 in cmd_main remote-curl.c:1583:2
#15 0x55fbbe244571 in main common-main.c:9:11
#16 0x7f90fe993ca7 in __libc_start_call_main csu/../sysdeps/nptl/libc_start_call_main.h:58:16
#17 0x7f90fe993d64 in __libc_start_main csu/../csu/libc-start.c:360:3
#18 0x55fbbe15bb60 in _start (git-remote-http+0x43b60) (BuildId: dfc63b9261f6d575776d30b4e048b235389a7b20) SUMMARY: MemorySanitizer: use-of-uninitialized-value (/lib/x86_64-linux-gnu/libcrypto.so.3+0x14fa46) (BuildId: 07a8321bad67632b52b47ad026125c79b7ebaab4) in BIO_new_file
So MSan complaining about stuff deep within curl/openssl, and AFAICT not something we could influence or annotate as OK.
-Peff
Re: [PATCH v5 0/5] for-each-ref: introduce seeking functionality via '--start-after'
In reply to Jeff KingQuoted 73 lines · Jeff King
Quoted 31 lines · Kyle Lippincott
Quoted 29 lines
==3275333==WARNING: MemorySanitizer: use-of-uninitialized-value
#0 0x557bd886f4bb in git_mkstemps_mode ../wrapper.c:487:27 #1 0x557bd886fb55 in git_mkstemp_mode ../wrapper.c:509:9 #2 0x557bd8100d1a in create_tmpfile ../object-file.c:736:7 #3 0x557bd80f1630 in start_loose_object_common ../object-file.c:781:7 #4 0x557bd80f5203 in write_loose_object ../object-file.c:881:7 #5 0x557bd80f4875 in write_object_file_flags ../object-file.c:1086:6 #6 0x557bd80f9f65 in write_object_file ../object-file.h:181:9 #7 0x557bd8101eb8 in index_mem ../object-file.c:1177:9 #8 0x557bd80f8bd5 in index_core ../object-file.c:1247:10 #9 0x557bd80f731d in index_fd ../object-file.c:1274:9 #10 0x557bd80f95e4 in index_path ../object-file.c:1295:7 #11 0x557bd831132d in add_to_index ../read-cache.c:771:7 #12 0x557bd8313cb1 in add_file_to_index ../read-cache.c:804:9 #13 0x557bd73f892c in add_files ../builtin/add.c:355:7 #14 0x557bd73f4752 in cmd_add ../builtin/add.c:578:18 #15 0x557bd7a38b6f in run_builtin ../git.c:480:11 #16 0x557bd7a31d54 in handle_builtin ../git.c:746:9 #17 0x557bd7a36644 in run_argv ../git.c:813:4 #18 0x557bd7a30e09 in cmd_main ../git.c:953:19 #19 0x557bd7a3ca01 in main ../common-main.c:9:11 #20 0x7f7e3f02a4d7 in __libc_start_call_main (/nix/store/g2jzxk3s7cnkhh8yq55l4fbvf639zy37-glibc-2.40-66/lib/libc.so.6+0x2a4d7) (BuildId: f117ee0f586dfa828cbdd08e37393c8f04f6480a) #21 0x7f7e3f02a59a in __libc_start_main@GLIBC_2.2.5 (/nix/store/g2jzxk3s7cnkhh8yq55l4fbvf639zy37-glibc-2.40-66/lib/libc.so.6+0x2a59a) (BuildId: f117ee0f586dfa828cbdd08e37393c8f04f6480a) #22 0x557bd7352b34 in _start (git+0x5db34)Quoted 1 line
Possibly something we need to look into cleaning up.
Quoted 8 lines
I also saw those msan issues when trying
make CFLAGS=-fsanitize=memory CC=clang, but not with Google's internal msan build. I don't know which variable in wrapper.c:487 it's complaining about - you'd think it'd beletters, but if it'sv, then that potentially comes from OpenSSL or some other library, and that library would also need to be built with msan (which is why it's such a pain to get msan builds working - EVERY library needs to be built with memory sanitizer).Yeah, presumably it is "v" from
csprng_bytes(). If there are only a few such spots, we can manually "unpoison" memory coming from libraries. On my system, I didn't hit the case shown above but do have trouble with bytes coming back from zlib.Applying this ancient patch:
https://lore.kernel.org/git/[email protected]/
and building with "make SANITIZE=memory CC=clang" let me run t6302 to completion, modulo the bug that started this thread (and which I confirmed goes away both with MSan and valgrind with the fix Karthik posted).
Probably:
wrapper.c +2
-0@@ -482,6 +482,8 @@ int git_mkstemps_mode(char *pattern, int suffix_len, int mode)if (csprng_bytes (&v,sizeof (v),0 ) <0 )return error_errno ("unable to get random bytes for temporary file" ); +msan_unpoison (&v,sizeof (v)); +/* Fill in the random bits. */ for (i =0 ; i < num_x; i++) { filename_template[i] = letters[v % num_letters];
Quoted 5 lines
on top of that would fix the problem you guys are seeing. I don't know if that path leads to insanity, though. Using MSan-enabled libraries is probably a better direction (should increase accuracy, and we don't have to carry these manual annotations around).
I wonder if an alternate is to use '-fsanitize-ignorelist', since the MemorySanitizer is supposed to work with that too [1].
[1]: https://clang.llvm.org/docs/MemorySanitizer.html#ignorelist
Quoted 1 line
-Peff
Re: [PATCH v5 0/5] for-each-ref: introduce seeking functionality via '--start-after'
Quoted 4 lines · Karthik Nayak
Quoted 4 lines
Applying this ancient patch:
Quoted 8 lines
Quoted 4 lines
and building with "make SANITIZE=memory CC=clang" let me run t6302 to completion, modulo the bug that started this thread (and which I confirmed goes away both with MSan and valgrind with the fix Karthik posted).
[...]
I wonder if an alternate is to use '-fsanitize-ignorelist', since the MemorySanitizer is supposed to work with that too [1].
I think you could do that, but it isn't quite what we want: it is annotating the access of those (false-positive) "uninitialized" bytes. So you have to mark every spot that touches bytes that come from zlib, which in Git is a lot of places. And so the patch linked above was an attempt to silence all of those with a single line: marking the bytes coming out of zlib as OK.
-Peff
Re: [PATCH v5 0/5] for-each-ref: introduce seeking functionality via '--start-after'
Quoted 15 lines · Jeff King
Quoted 4 lines · Karthik Nayak
Quoted 4 lines
Applying this ancient patch:
Quoted 8 lines
Quoted 4 lines
and building with "make SANITIZE=memory CC=clang" let me run t6302 to completion, modulo the bug that started this thread (and which I confirmed goes away both with MSan and valgrind with the fix Karthik posted).
[...]
I wonder if an alternate is to use '-fsanitize-ignorelist', since the MemorySanitizer is supposed to work with that too [1].
Quoted 7 lines
I think you could do that, but it isn't quite what we want: it is annotating the access of those (false-positive) "uninitialized" bytes. So you have to mark every spot that touches bytes that come from zlib, which in Git is a lot of places. And so the patch linked above was an attempt to silence all of those with a single line: marking the bytes coming out of zlib as OK.
That makes sense, thanks for explaining.
Quoted 1 line
-Peff
Re: [PATCH v5 0/5] for-each-ref: introduce seeking functionality via '--start-after'
In reply to Karthik NayakQuoted 20 lines · Karthik Nayak
This series allows the seek function to set the cursor without setting the prefix, which is a requirement for pagination. So there is no need to set '
prefix_state' for this functionality. Which is why I didn't set it, since the default value of '0' (PREFIX_CONTAINS_DIR) would be the correct setting for all dirs. This causes the issue.So the only fix required would be
refs/ref-cache.c +1
-0@@ -527,6 +527,7 @@ static int cache_ref_iterator_seek(structref_iterator *ref_iterator, level = &iter->levels[iter->levels_nr++]; level->dir = dir; level->index =-1 ; + level->prefix_state =PREFIX_CONTAINS_DIR ; }else {/* reduce the index so the leaf node is iterated over */
Quoted 1 line
if (cmp <= 0 && !slash)
Yup, that is inside the code added by this series. It does look like it fixes the complaint from the checker.
Thanks.
PATCH 1/1 ref-iterator-seek: correctly initialize the prefix_state for a new level
In reply to Karthik NayakWhen cache_ref_iterator_seek() "jumps" to a middle of the sorted ref list, it forgets to set the .prefix_state member of the new (i.e. deeper) level it just initialized. This later causes cache_ref_iterator_advance() to look at this uninitialized member to base its decision on what to do next.
Kyle Lippincott [*] and Jeff King noticed this with MSAN and Valgrind, and Karthik Nayak as the original author located exactly where the missing initialization is.
[*] <CAO_smVg9TDakUnubepjPGmLyOzW6n8Z=MDbnZKvkwN2=kN2RRw@mail.gmail.com>
--- refs/ref-cache.c | 1 + 1 file changed, 1 insertion(+)
* I had this as "fixup!" on top of your topic for quite a while and forgot to ask you to send in an official fix. As Kyle's discovery was after the topic hit 'next' (understandable, as their internal edition of Git is based on 'next'), we need a separate fix on top.
To prepare for merging down the whole thing to 'master', I wrote the proposed log message to help expedite the process. Comments?
refs/ref-cache.c +1 -0
@@ -527,6 +527,7 @@ static int cache_ref_iterator_seek(struct ref_iterator *ref_iterator, level = &iter->levels[iter->levels_nr++]; level->dir = dir; level->index =-1 ; + level->prefix_state =PREFIX_CONTAINS_DIR ; }else {/* reduce the index so the leaf node is iterated over */ if (cmp <=0 && !slash)
Signature · 1 line
2.50.1-521-gf11ee0bd80
Re: [PATCH] ref-iterator-seek: correctly initialize the prefix_state for a new level
Quoted 7 lines · Junio C Hamano
When
cache_ref_iterator_seek()"jumps" to a middle of the sorted ref list, it forgets to set the .prefix_state member of the new (i.e. deeper) level it just initialized. This later causescache_ref_iterator_advance()to look at this uninitialized member to base its decision on what to do next.
Quoted 7 lines
Kyle Lippincott [*] and Jeff King noticed this with MSAN and Valgrind, and Karthik Nayak as the original author located exactly where the missing initialization is.
[*] <CAO_smVg9TDakUnubepjPGmLyOzW6n8Z=MDbnZKvkwN2=kN2RRw@mail.gmail.com>
Signed-off-by Junio C Hamano <[email protected]>
Looks good to me, thanks!
Quoted 28 lines
--- refs/ref-cache.c | 1 + 1 file changed, 1 insertion(+)* I had this as "fixup!" on top of your topic for quite a while and forgot to ask you to send in an official fix. As Kyle's discovery was after the topic hit 'next' (understandable, as their internal edition of Git is based on 'next'), we need a separate fix on top.
To prepare for merging down the whole thing to 'master', I wrote the proposed log message to help expedite the process. Comments?
refs/ref-cache.c +1
-1@@ -527,6 +527,7 @@ static int cache_ref_iterator_seek(struct ref_iterator *ref_iterator, level = &iter->levels[iter->levels_nr++]; level->dir = dir; level->index =-1 ; + level->prefix_state =PREFIX_CONTAINS_DIR ; }else {/* reduce the index so the leaf node is iterated over */ if (cmp <=0 && !slash) --2.50.1-521-gf11ee0bd80
Re: [PATCH] ref-iterator-seek: correctly initialize the prefix_state for a new level
In reply to Junio C HamanoQuoted 9 lines · Junio C Hamano
When
cache_ref_iterator_seek()"jumps" to a middle of the sorted ref list, it forgets to set the .prefix_state member of the new (i.e. deeper) level it just initialized. This later causescache_ref_iterator_advance()to look at this uninitialized member to base its decision on what to do next.Kyle Lippincott [*] and Jeff King noticed this with MSAN and Valgrind, and Karthik Nayak as the original author located exactly where the missing initialization is.
This explanation makes sense to me (from my admittedly rusty view of the ref iteration code). And certainly the patch looks right.
-Peff
Re: [PATCH] ref-iterator-seek: correctly initialize the prefix_state for a new level
In reply to Junio C HamanoQuoted 6 lines · Junio C Hamano
When
cache_ref_iterator_seek()"jumps" to a middle of the sorted ref list, it forgets to set the .prefix_state member of the new (i.e. deeper) level it just initialized. This later causescache_ref_iterator_advance()to look at this uninitialized member to base its decision on what to do next.
I think the explanation is correct. For reference I had some more details in my local patch, but this is totally okay.
ref-cache: set prefix_state when seeking
In 090eb5336c (refs: selectively set prefix in the seek functions, 2025-07-15) we separated the seeking functionality of reference iterators from the functionality to set prefix to an iterator. This allows users of ref iterators to seek to a particular reference to provide pagination support.
The files-backend, uses the ref-cache iterator to iterate over loose refs. The iterator tracks directories and entries already processed via a stack of levels. Each level corresponds to a directory under the files backend. New levels are added to the stack, and when all entries from a level is yielded, the corresponding level is popped from the stack.
To accommodate seeking, we need to populate and traverse the levels to stop the requested seek marker at the appropriate level and its entry index. Each level also contains a 'prefix_state' which is used for prefix matching, this allows the iterator to skip levels/entries which don't match a prefix. The default value of 'prefix_state' is PREFIX_CONTAINS_DIR, which yields all entries within a level. When purely seeking without prefix matching, we want to yield all entries. The commit however, skips setting the value explicitly. This causes the MemorySanitizer to issue a 'use-of-uninitialized-value' error when running 't/t6302-for-each-ref-filter'.
Set the value explicitly to avoid to fix the issue.
Quoted 11 lines
Kyle Lippincott [*] and Jeff King noticed this with MSAN and Valgrind, and Karthik Nayak as the original author located exactly where the missing initialization is.
[*] <CAO_smVg9TDakUnubepjPGmLyOzW6n8Z=MDbnZKvkwN2=kN2RRw@mail.gmail.com>
Signed-off-by Junio C Hamano <[email protected]>--- refs/ref-cache.c | 1 + 1 file changed, 1 insertion(+)
Quoted 9 lines
* I had this as "fixup!" on top of your topic for quite a while and forgot to ask you to send in an official fix. As Kyle's discovery was after the topic hit 'next' (understandable, as their internal edition of Git is based on 'next'), we need a separate fix on top.
To prepare for merging down the whole thing to 'master', I wrote the proposed log message to help expedite the process. Comments?
I had a set of patches locally, I just didn't get around to sending it. Will send the others, omitting this. Thanks for doing it!
Quoted 13 lines
refs/ref-cache.c +1
-1@@ -527,6 +527,7 @@ static int cache_ref_iterator_seek(struct ref_iterator *ref_iterator, level = &iter->levels[iter->levels_nr++]; level->dir = dir; level->index =-1 ; + level->prefix_state =PREFIX_CONTAINS_DIR ; }else {/* reduce the index so the leaf node is iterated over */ if (cmp <=0 && !slash) --
Quoted 1 line
2.50.1-521-gf11ee0bd80
The patch looks good.
Re: [PATCH] ref-iterator-seek: correctly initialize the prefix_state for a new level
Quoted 6 lines · Karthik Nayak
Quoted 6 lines
* I had this as "fixup!" on top of your topic for quite a while and forgot to ask you to send in an official fix. As Kyle's discovery was after the topic hit 'next' (understandable, as their internal edition of Git is based on 'next'), we need a separate fix on top.
Quoted 6 lines
Quoted 3 lines
To prepare for merging down the whole thing to 'master', I wrote the proposed log message to help expedite the process. Comments?
I had a set of patches locally, I just didn't get around to sending it. Will send the others, omitting this. Thanks for doing it!
I do not mind discarding what I sent out at all. I actually prefer if it came from you.
Thanks.
PATCH 1/1 ref-cache: set prefix_state when seeking
In 090eb5336c (refs: selectively set prefix in the seek functions, 2025-07-15) we separated the seeking functionality of reference iterators from the functionality to set prefix to an iterator. This allows users of ref iterators to seek to a particular reference to provide pagination support.
The files-backend, uses the ref-cache iterator to iterate over loose refs. The iterator tracks directories and entries already processed via a stack of levels. Each level corresponds to a directory under the files backend. New levels are added to the stack, and when all entries from a level is yielded, the corresponding level is popped from the stack.
To accommodate seeking, we need to populate and traverse the levels to stop the requested seek marker at the appropriate level and its entry index. Each level also contains a 'prefix_state' which is used for prefix matching, this allows the iterator to skip levels/entries which don't match a prefix. The default value of 'prefix_state' is PREFIX_CONTAINS_DIR, which yields all entries within a level. When purely seeking without prefix matching, we want to yield all entries. The commit however, skips setting the value explicitly. This causes the MemorySanitizer to issue a 'use-of-uninitialized-value' error when running 't/t6302-for-each-ref-filter'.
Set the value explicitly to avoid to fix the issue.
---
Here is my version of the same patch!
refs/ref-cache.c | 1 + 1 file changed, 1 insertion(+)
refs/ref-cache.c +1 -0
@@ -527,6 +527,7 @@ static int cache_ref_iterator_seek(struct ref_iterator *ref_iterator, level = &iter->levels[iter->levels_nr++]; level->dir = dir; level->index =-1 ; + level->prefix_state =PREFIX_CONTAINS_DIR ; }else {/* reduce the index so the leaf node is iterated over */ if (cmp <=0 && !slash)
Signature · 1 line
2.49.0
Re: [PATCH] ref-cache: set prefix_state when seeking
Quoted 25 lines · Karthik Nayak
In
090eb5336c(refs: selectively set prefix in the seek functions, 2025-07-15) we separated the seeking functionality of reference iterators from the functionality to set prefix to an iterator. This allows users of ref iterators to seek to a particular reference to provide pagination support.The files-backend, uses the ref-cache iterator to iterate over loose refs. The iterator tracks directories and entries already processed via a stack of levels. Each level corresponds to a directory under the files backend. New levels are added to the stack, and when all entries from a level is yielded, the corresponding level is popped from the stack.
To accommodate seeking, we need to populate and traverse the levels to stop the requested seek marker at the appropriate level and its entry index. Each level also contains a '
prefix_state' which is used for prefix matching, this allows the iterator to skip levels/entries which don't match a prefix. The default value of 'prefix_state' isPREFIX_CONTAINS_DIR, which yields all entries within a level. When purely seeking without prefix matching, we want to yield all entries. The commit however, skips setting the value explicitly. This causes the MemorySanitizer to issue a 'use-of-uninitialized-value' error when running 't/t6302-for-each-ref-filter'.Set the value explicitly to avoid to fix the issue.
Quoted 7 lines
Reported-by Kyle Lippincott <[email protected]>Helped-by Kyle Lippincott <[email protected]>Helped-by Jeff King <[email protected]>Signed-off-by Karthik Nayak <[email protected]>---
Here is my version of the same patch!
Thanks!
Quoted 16 lines
refs/ref-cache.c | 1 + 1 file changed, 1 insertion(+)refs/ref-cache.c +1
-0@@ -527,6 +527,7 @@ static int cache_ref_iterator_seek(struct ref_iterator *ref_iterator, level = &iter->levels[iter->levels_nr++]; level->dir = dir; level->index =-1 ; + level->prefix_state =PREFIX_CONTAINS_DIR ; }else {/* reduce the index so the leaf node is iterated over */ if (cmp <=0 && !slash)