Skip to contentSkip to conversation

refs: implement skip lists for packed backend

PATCH Superseded Taylor Blau 5 participants 149 messages May 8, 2023 – Jul 11, 2023 Documentation Page 1 of 2

lore
Share & tools
lore
#0 lore raw

cover refs: implement skip lists for packed backend

This series implements the concept of a skip list for the packed refs backend. In situations where the caller has many references they want to exclude, the packed-refs iterator maintains a list of (start, end) tuples of regions to jump over (i.e. if iter->pos == start, jump forward to end).

This series implements that concept, and uses it to power a couple of new things:

- git for-each-ref --exclude, which allows callers to specify prefixes they wish to discard.

In a synthetic example, the naive implementation improved runtime from ~820ms to enumerate all references and discard unwanted ones, down to ~106ms with --exclude. By the end of the series, this time drops to under ~5ms.

- git receive-pack uses the new skip-list machinery to avoid visiting references hidden via a hideRefs rule.

- git upload-pack has an analogous optimization as above (though it can't kick in quite as often for reasons described in that patch).

The series is laid out as follows:

- The first five patches are preparatory (various refs.c and ref-filter cleanup).
- The sixth patch provides a naive implementation of --exclude.
- The next four patches prepare for and implement skip lists in the packed-refs backend.
- The last five patches apply the optimization in upload-pack and

    `receive-pack`.

The series is ordered so that any prefix of sections list above could be queued independently. Thanks in advance for your review.

Jeff King (5):
refs.c: rename ref_filter
ref-filter.h: provide REF_FILTER_INIT
ref-filter: clear reachable list pointers after freeing
ref-filter: add ref_filter_clear() ref-filter.c: parameterize match functions over patterns

Taylor Blau (10):
builtin/for-each-ref.c: add --exclude option
refs: plumb exclude_patterns argument throughout refs/packed-backend.c: refactor find_reference_location() refs/packed-backend.c: implement skip lists to avoid excluded
pattern(s)
refs/packed-backend.c: add trace2 counters for skip list revision.h: store hidden refs in a strvec refs/packed-backend.c: ignore complicated hidden refs rules refs.h: let for_each_namespaced_ref() take excluded patterns upload-pack.c: avoid enumerating hidden refs where possible builtin/receive-pack.c: avoid enumerating hidden references

 Documentation/git-for-each-ref.txt |   6 +
 builtin/branch.c                   |   4 +-
 builtin/for-each-ref.c             |   8 +-
 builtin/receive-pack.c             |   7 +-
 builtin/tag.c                      |   4 +-
 http-backend.c                     |   2 +-
 ls-refs.c                          |   8 +-
 ref-filter.c                       |  61 ++++++--
 ref-filter.h                       |  12 ++
 refs.c                             |  61 ++++----
 refs.h                             |  15 +-
 refs/debug.c                       |   5 +-
 refs/files-backend.c               |   5 +-
 refs/packed-backend.c              | 222 ++++++++++++++++++++++++++---
 refs/refs-internal.h               |   7 +-
 revision.c                         |   4 +-
 revision.h                         |   5 +-
 t/helper/test-reach.c              |   2 +-
 t/helper/test-ref-store.c          |  10 ++
 t/t1419-exclude-refs.sh            | 131 +++++++++++++++++
 t/t6300-for-each-ref.sh            |  35 +++++
 trace2.h                           |   2 +
 trace2/tr2_ctr.c                   |   5 +
 upload-pack.c                      |  43 ++++--
 24 files changed, 559 insertions(+), 105 deletions(-)
 create mode 100755 t/t1419-exclude-refs.sh
Signature · 1 line
2.40.1.477.g956c797dfc
#1 lore raw

PATCH 1/15 refs.c: rename `ref_filter`

From: Jeff King <[email protected]>

The refs machinery has its own implementation of a ref_filter (used by for-each-ref), which is distinct from the ref-filler.h API (also used by for-each-ref, among other things).

Rename the one within refs.c to more clearly indicate its purpose.

---
 refs.c | 15 ++++++++-------
 1 file changed, 8 insertions(+), 7 deletions(-)
refs.c +8 -7
index d2a98e1c21..b9b77d2eff 100644
@@ -375,8 +375,8 @@ char *resolve_refdup(const char *refname, int resolve_flags,
 				   oid, flags);
 }
 
-/* The argument to filter_refs */
-struct ref_filter {
+/* The argument to for_each_filter_refs */
+struct for_each_ref_filter {
 	const char *pattern;
 	const char *prefix;
 	each_ref_fn *fn;
@@ -409,10 +409,11 @@ int ref_exists(const char *refname)
 	return refs_ref_exists(get_main_ref_store(the_repository), refname);
 }
 
-static int filter_refs(const char *refname, const struct object_id *oid,
-			   int flags, void *data)
+static int for_each_filter_refs(const char *refname,
+				const struct object_id *oid,
+				int flags, void *data)
 {
-	struct ref_filter *filter = (struct ref_filter *)data;
+	struct for_each_ref_filter *filter = data;
 
 	if (wildmatch(filter->pattern, refname, 0))
 		return 0;
@@ -569,7 +570,7 @@ int for_each_glob_ref_in(each_ref_fn fn, const char *pattern,
 	const char *prefix, void *cb_data)
 {
 	struct strbuf real_pattern = STRBUF_INIT;
-	struct ref_filter filter;
+	struct for_each_ref_filter filter;
 	int ret;
 
 	if (!prefix && !starts_with(pattern, "refs/"))
@@ -589,7 +590,7 @@ int for_each_glob_ref_in(each_ref_fn fn, const char *pattern,
 	filter.prefix = prefix;
 	filter.fn = fn;
 	filter.cb_data = cb_data;
-	ret = for_each_ref(filter_refs, &filter);
+	ret = for_each_ref(for_each_filter_refs, &filter);
 
 	strbuf_release(&real_pattern);
 	return ret;
Signature · 1 line
2.40.1.477.g956c797dfc
#3 lore raw

PATCH 3/15 ref-filter: clear reachable list pointers after freeing

In reply to Taylor Blau

From: Jeff King <[email protected]>

In reach_filter(), we pop all commits from the reachable lists, leaving them empty. But because we're operating on a list pointer that was passed by value, the original filter.reachable_from pointer is left dangling.

---
 ref-filter.c | 14 +++++++-------
 1 file changed, 7 insertions(+), 7 deletions(-)
ref-filter.c +7 -7
index 10aab14f03..b1d5022a51 100644
@@ -2416,13 +2416,13 @@ void ref_array_clear(struct ref_array *array)
 #define EXCLUDE_REACHED 0
 #define INCLUDE_REACHED 1
 static void reach_filter(struct ref_array *array,
-			 struct commit_list *check_reachable,
+			 struct commit_list **check_reachable,
 			 int include_reached)
 {
 	int i, old_nr;
 	struct commit **to_clear;
 
-	if (!check_reachable)
+	if (!*check_reachable)
 		return;
 
 	CALLOC_ARRAY(to_clear, array->nr);
@@ -2432,7 +2432,7 @@ static void reach_filter(struct ref_array *array,
 	}
 
 	tips_reachable_from_bases(the_repository,
-				  check_reachable,
+				  *check_reachable,
 				  to_clear, array->nr,
 				  UNINTERESTING);
 
@@ -2453,8 +2453,8 @@ static void reach_filter(struct ref_array *array,
 
 	clear_commit_marks_many(old_nr, to_clear, ALL_REV_FLAGS);
 
-	while (check_reachable) {
-		struct commit *merge_commit = pop_commit(&check_reachable);
+	while (*check_reachable) {
+		struct commit *merge_commit = pop_commit(check_reachable);
 		clear_commit_marks(merge_commit, ALL_REV_FLAGS);
 	}
 
@@ -2551,8 +2551,8 @@ int filter_refs(struct ref_array *array, struct ref_filter *filter, unsigned int
 	clear_contains_cache(&ref_cbdata.no_contains_cache);
 
 	/*  Filters that need revision walking */
-	reach_filter(array, filter->reachable_from, INCLUDE_REACHED);
-	reach_filter(array, filter->unreachable_from, EXCLUDE_REACHED);
+	reach_filter(array, &filter->reachable_from, INCLUDE_REACHED);
+	reach_filter(array, &filter->unreachable_from, EXCLUDE_REACHED);
 
 	save_commit_buffer = save_commit_buffer_orig;
 	return ret;
Signature · 1 line
2.40.1.477.g956c797dfc
#2 lore raw

PATCH 2/15 ref-filter.h: provide `REF_FILTER_INIT`

In reply to Taylor Blau

From: Jeff King <[email protected]>

Provide a sane initialization value for struct ref_filter, which in a subsequent patch will be used to initialize a new field.

In the meantime, fix a case in test-reach.c where its ref_filter is not even zero-initialized.

---
 builtin/branch.c       | 3 +--
 builtin/for-each-ref.c | 3 +--
 builtin/tag.c          | 3 +--
 ref-filter.h           | 3 +++
 t/helper/test-reach.c  | 2 +-
 5 files changed, 7 insertions(+), 7 deletions(-)
builtin/branch.c +1 -2
index 501c47657c..03bb8e414c 100644
@@ -662,7 +662,7 @@ int cmd_branch(int argc, const char **argv, const char *prefix)
 	int reflog = 0, quiet = 0, icase = 0, force = 0,
 	    recurse_submodules_explicit = 0;
 	enum branch_track track;
-	struct ref_filter filter;
+	struct ref_filter filter = REF_FILTER_INIT;
 	static struct ref_sorting *sorting;
 	struct string_list sorting_options = STRING_LIST_INIT_DUP;
 	struct ref_format format = REF_FORMAT_INIT;
@@ -720,7 +720,6 @@ int cmd_branch(int argc, const char **argv, const char *prefix)
 
 	setup_ref_filter_porcelain_msg();
 
-	memset(&filter, 0, sizeof(filter));
 	filter.kind = FILTER_REFS_BRANCHES;
 	filter.abbrev = -1;
 
builtin/for-each-ref.c +1 -2
index 695fc8f4a5..99ccb73518 100644
@@ -24,7 +24,7 @@ int cmd_for_each_ref(int argc, const char **argv, const char *prefix)
 	struct string_list sorting_options = STRING_LIST_INIT_DUP;
 	int maxcount = 0, icase = 0, omit_empty = 0;
 	struct ref_array array;
-	struct ref_filter filter;
+	struct ref_filter filter = REF_FILTER_INIT;
 	struct ref_format format = REF_FORMAT_INIT;
 	struct strbuf output = STRBUF_INIT;
 	struct strbuf err = STRBUF_INIT;
@@ -61,7 +61,6 @@ int cmd_for_each_ref(int argc, const char **argv, const char *prefix)
 	};
 
 	memset(&array, 0, sizeof(array));
-	memset(&filter, 0, sizeof(filter));
 
 	format.format = "%(objectname) %(objecttype)\t%(refname)";
 
builtin/tag.c +1 -2
index 1850a6a6fd..6b41bb7374 100644
@@ -443,7 +443,7 @@ int cmd_tag(int argc, const char **argv, const char *prefix)
 	struct msg_arg msg = { .buf = STRBUF_INIT };
 	struct ref_transaction *transaction;
 	struct strbuf err = STRBUF_INIT;
-	struct ref_filter filter;
+	struct ref_filter filter = REF_FILTER_INIT;
 	struct ref_sorting *sorting;
 	struct string_list sorting_options = STRING_LIST_INIT_DUP;
 	struct ref_format format = REF_FORMAT_INIT;
@@ -501,7 +501,6 @@ int cmd_tag(int argc, const char **argv, const char *prefix)
 	git_config(git_tag_config, &sorting_options);
 
 	memset(&opt, 0, sizeof(opt));
-	memset(&filter, 0, sizeof(filter));
 	filter.lines = -1;
 	opt.sign = -1;
 
ref-filter.h +3 -0
index 430701cfb7..a920f73b29 100644
@@ -92,6 +92,9 @@ struct ref_format {
 	struct string_list bases;
 };
 
+#define REF_FILTER_INIT { \
+	.points_at = OID_ARRAY_INIT, \
+}
 #define REF_FORMAT_INIT {             \
 	.use_color = -1,              \
 	.bases = STRING_LIST_INIT_DUP, \
t/helper/test-reach.c +1 -1
index 5b6f217441..ef58f10c2d 100644
@@ -139,7 +139,7 @@ int cmd__reach(int ac, const char **av)
 
 		printf("%s(X,_,_,0,0):%d\n", av[1], can_all_from_reach_with_flag(&X_obj, 2, 4, 0, 0));
 	} else if (!strcmp(av[1], "commit_contains")) {
-		struct ref_filter filter;
+		struct ref_filter filter = REF_FILTER_INIT;
 		struct contains_cache cache;
 		init_contains_cache(&cache);
 
Signature · 1 line
2.40.1.477.g956c797dfc
#4 lore raw

PATCH 4/15 ref-filter: add ref_filter_clear()

In reply to Taylor Blau

From: Jeff King <[email protected]>

We did not bother to clean up at all in branch/tag, and for-each-ref only hit a few elements. So this is probably cleaning up leaks, but I didn't check yet.

Note that the reachable_from and unreachable_from lists should be cleaned as they are used. So this is just covering any case where we might bail before running the reachability check.

---
 builtin/branch.c       |  1 +
 builtin/for-each-ref.c |  3 +--
 builtin/tag.c          |  1 +
 ref-filter.c           | 16 ++++++++++++++++
 ref-filter.h           |  3 +++
 5 files changed, 22 insertions(+), 2 deletions(-)
builtin/branch.c +1 -0
index 03bb8e414c..c201f0cb0b 100644
@@ -813,6 +813,7 @@ int cmd_branch(int argc, const char **argv, const char *prefix)
 		print_columns(&output, colopts, NULL);
 		string_list_clear(&output, 0);
 		ref_sorting_release(sorting);
+		ref_filter_clear(&filter);
 		return 0;
 	} else if (edit_description) {
 		const char *branch_name;
builtin/for-each-ref.c +1 -2
index 99ccb73518..c01fa6fefe 100644
@@ -120,8 +120,7 @@ int cmd_for_each_ref(int argc, const char **argv, const char *prefix)
 	strbuf_release(&err);
 	strbuf_release(&output);
 	ref_array_clear(&array);
-	free_commit_list(filter.with_commit);
-	free_commit_list(filter.no_commit);
+	ref_filter_clear(&filter);
 	ref_sorting_release(sorting);
 	strvec_clear(&vec);
 	return 0;
builtin/tag.c +1 -0
index 6b41bb7374..aab5e693fe 100644
@@ -645,6 +645,7 @@ int cmd_tag(int argc, const char **argv, const char *prefix)
 
 cleanup:
 	ref_sorting_release(sorting);
+	ref_filter_clear(&filter);
 	strbuf_release(&buf);
 	strbuf_release(&ref);
 	strbuf_release(&reflog_msg);
ref-filter.c +16 -0
index b1d5022a51..9ea92b9637 100644
@@ -2864,3 +2864,19 @@ int parse_opt_merge_filter(const struct option *opt, const char *arg, int unset)
 
 	return 0;
 }
+
+void ref_filter_init(struct ref_filter *filter)
+{
+	struct ref_filter blank = REF_FILTER_INIT;
+	memcpy(filter, &blank, sizeof(blank));
+}
+
+void ref_filter_clear(struct ref_filter *filter)
+{
+	oid_array_clear(&filter->points_at);
+	free_commit_list(filter->with_commit);
+	free_commit_list(filter->no_commit);
+	free_commit_list(filter->reachable_from);
+	free_commit_list(filter->unreachable_from);
+	ref_filter_init(filter);
+}
ref-filter.h +3 -0
index a920f73b29..160b807224 100644
@@ -170,4 +170,7 @@ void filter_ahead_behind(struct repository *r,
 			 struct ref_format *format,
 			 struct ref_array *array);
 
+void ref_filter_init(struct ref_filter *filter);
+void ref_filter_clear(struct ref_filter *filter);
+
 #endif /*  REF_FILTER_H  */
Signature · 1 line
2.40.1.477.g956c797dfc
#9 lore raw

PATCH 5/15 ref-filter.c: parameterize match functions over patterns

In reply to Taylor Blau

From: Jeff King <[email protected]>

match_pattern() and match_name_as_path() both take a struct ref_filter *, and then store a stack variable patterns pointing at filter->patterns.

The subsequent patch will add a new array of patterns to match over (the excluded patterns, via a new git for-each-ref --exclude option), treating the return value of these functions differently depending on which patterns are being used to match.

Tweak match_pattern() and match_name_as_path() to take an array of patterns to prepare for passing either in.

---
 ref-filter.c | 14 ++++++++------
 1 file changed, 8 insertions(+), 6 deletions(-)
ref-filter.c +8 -6
index 9ea92b9637..6c5eed144f 100644
@@ -2102,9 +2102,10 @@ static int get_ref_atom_value(struct ref_array_item *ref, int atom,
  * matches a pattern "refs/heads/mas") or a wildcard (e.g. the same ref
  * matches "refs/heads/mas*", too).
  */
-static int match_pattern(const struct ref_filter *filter, const char *refname)
+static int match_pattern(const struct ref_filter *filter,
+			 const char **patterns,
+			 const char *refname)
 {
-	const char **patterns = filter->name_patterns;
 	unsigned flags = 0;
 
 	if (filter->ignore_case)
@@ -2132,9 +2133,10 @@ static int match_pattern(const struct ref_filter *filter, const char *refname)
  * matches a pattern "refs/heads/" but not "refs/heads/m") or a
  * wildcard (e.g. the same ref matches "refs/heads/m*", too).
  */
-static int match_name_as_path(const struct ref_filter *filter, const char *refname)
+static int match_name_as_path(const struct ref_filter *filter,
+			      const char **pattern,
+			      const char *refname)
 {
-	const char **pattern = filter->name_patterns;
 	int namelen = strlen(refname);
 	unsigned flags = WM_PATHNAME;
 
@@ -2163,8 +2165,8 @@ static int filter_pattern_match(struct ref_filter *filter, const char *refname)
 	if (!*filter->name_patterns)
 		return 1; /* No pattern always matches */
 	if (filter->match_as_path)
-		return match_name_as_path(filter, refname);
-	return match_pattern(filter, refname);
+		return match_name_as_path(filter, filter->name_patterns, refname);
+	return match_pattern(filter, filter->name_patterns, refname);
 }
 
 /*
Signature · 1 line
2.40.1.477.g956c797dfc
#12 lore raw

PATCH 6/15 builtin/for-each-ref.c: add `--exclude` option

In reply to Taylor Blau

When using for-each-ref, it is sometimes convenient for the caller to be able to exclude certain parts of the references.

For example, if there are many refs/__hidden__/* references, the caller may want to emit all references *except* the hidden ones. Currently, the only way to do this is to post-process the output, like:

    $ git for-each-ref --format='%(refname)' | grep -v '^refs/hidden/'

Which is do-able, but requires processing a potentially large quantity of references.

Teach git for-each-ref a new --exclude=<pattern> option, which excludes references from the results if they match one or more excluded patterns.

This patch provides a naive implementation where the ref_filter still sees all references (including ones that it will discard) and is left to check whether each reference matches any excluded pattern(s) before emitting them.

By culling out references we know the caller doesn't care about, we can avoid allocating memory for their storage, as well as spending time sorting the output (among other things). Even the naive implementation provides a significant speed-up on a modified copy of linux.git (that has a hidden ref pointing at each commit):

    $ hyperfine \
      'git.compile for-each-ref --format="%(objectname) %(refname)" | grep -vE "[0-9a-f]{40} refs/pull/"' \
      'git.compile for-each-ref --format="%(objectname) %(refname)" --exclude refs/pull/'
    Benchmark 1: git.compile for-each-ref --format="%(objectname) %(refname)" | grep -vE "[0-9a-f]{40} refs/pull/"
      Time (mean ± σ):     820.1 ms ±   2.0 ms    [User: 703.7 ms, System: 152.0 ms]
      Range (min … max):   817.7 ms … 823.3 ms    10 runs
    Benchmark 2: git.compile for-each-ref --format="%(objectname) %(refname)" --exclude refs/pull/
      Time (mean ± σ):     106.6 ms ±   1.1 ms    [User: 99.4 ms, System: 7.1 ms]
      Range (min … max):   104.7 ms … 109.1 ms    27 runs

    Summary
      'git.compile for-each-ref --format="%(objectname) %(refname)" --exclude refs/pull/' ran
        7.69 ± 0.08 times faster than 'git.compile for-each-ref --format="%(objectname) %(refname)" | grep -vE "[0-9a-f]{40} refs/pull/"'

Subsequent patches will improve on this by avoiding visiting excluded sections of the packed-refs file in certain cases.

---
 Documentation/git-for-each-ref.txt |  6 +++++
 builtin/for-each-ref.c             |  2 ++
 ref-filter.c                       | 13 +++++++++++
 ref-filter.h                       |  6 +++++
 t/t6300-for-each-ref.sh            | 35 ++++++++++++++++++++++++++++++
 5 files changed, 62 insertions(+)
Documentation/git-for-each-ref.txt +6 -0
index 1e215d4e73..5743eb5def 100644
@@ -14,6 +14,7 @@ SYNOPSIS
 		   [--points-at=<object>]
 		   [--merged[=<object>]] [--no-merged[=<object>]]
 		   [--contains[=<object>]] [--no-contains[=<object>]]
+		   [--exclude=<pattern> ...]
 
 DESCRIPTION
 -----------
@@ -102,6 +103,11 @@ OPTIONS
 	Do not print a newline after formatted refs where the format expands
 	to the empty string.
 
+--exclude=<pattern>::
+	If one or more patterns are given, only refs which do not match
+	any excluded pattern(s) are shown. Matching is done using the
+	same rules as `<pattern>` above.
+
 FIELD NAMES
 -----------
 
builtin/for-each-ref.c +2 -0
index c01fa6fefe..449da61e11 100644
@@ -14,6 +14,7 @@ static char const * const for_each_ref_usage[] = {
 	N_("git for-each-ref [--points-at <object>]"),
 	N_("git for-each-ref [--merged [<commit>]] [--no-merged [<commit>]]"),
 	N_("git for-each-ref [--contains [<commit>]] [--no-contains [<commit>]]"),
+	N_("git for-each-ref [--exclude=<pattern> ...]"),
 	NULL
 };
 
@@ -47,6 +48,7 @@ int cmd_for_each_ref(int argc, const char **argv, const char *prefix)
 		OPT_INTEGER( 0 , "count", &maxcount, N_("show only <n> matched refs")),
 		OPT_STRING(  0 , "format", &format.format, N_("format"), N_("format to use for the output")),
 		OPT__COLOR(&format.use_color, N_("respect format colors")),
+		OPT_REF_FILTER_EXCLUDE(&filter),
 		OPT_REF_SORT(&sorting_options),
 		OPT_CALLBACK(0, "points-at", &filter.points_at,
 			     N_("object"), N_("print only refs which points at the given object"),
ref-filter.c +13 -0
index 6c5eed144f..93dc9b331f 100644
@@ -2169,6 +2169,15 @@ static int filter_pattern_match(struct ref_filter *filter, const char *refname)
 	return match_pattern(filter, filter->name_patterns, refname);
 }
 
+static int filter_exclude_match(struct ref_filter *filter, const char *refname)
+{
+	if (!filter->exclude.nr)
+		return 0;
+	if (filter->match_as_path)
+		return match_name_as_path(filter, filter->exclude.v, refname);
+	return match_pattern(filter, filter->exclude.v, refname);
+}
+
 /*
  * This is the same as for_each_fullref_in(), but it tries to iterate
  * only over the patterns we'll care about. Note that it _doesn't_ do a full
@@ -2336,6 +2345,9 @@ static int ref_filter_handler(const char *refname, const struct object_id *oid,
 	if (!filter_pattern_match(filter, refname))
 		return 0;
 
+	if (filter_exclude_match(filter, refname))
+		return 0;
+
 	if (filter->points_at.nr && !match_points_at(&filter->points_at, oid, refname))
 		return 0;
 
@@ -2875,6 +2887,7 @@ void ref_filter_init(struct ref_filter *filter)
 
 void ref_filter_clear(struct ref_filter *filter)
 {
+	strvec_clear(&filter->exclude);
 	oid_array_clear(&filter->points_at);
 	free_commit_list(filter->with_commit);
 	free_commit_list(filter->no_commit);
ref-filter.h +6 -0
index 160b807224..1524bc463a 100644
@@ -6,6 +6,7 @@
 #include "refs.h"
 #include "commit.h"
 #include "string-list.h"
+#include "strvec.h"
 
 /* Quoting styles */
 #define QUOTE_NONE 0
@@ -59,6 +60,7 @@ struct ref_array {
 
 struct ref_filter {
 	const char **name_patterns;
+	struct strvec exclude;
 	struct oid_array points_at;
 	struct commit_list *with_commit;
 	struct commit_list *no_commit;
@@ -94,6 +96,7 @@ struct ref_format {
 
 #define REF_FILTER_INIT { \
 	.points_at = OID_ARRAY_INIT, \
+	.exclude = STRVEC_INIT, \
 }
 #define REF_FORMAT_INIT {             \
 	.use_color = -1,              \
@@ -112,6 +115,9 @@ struct ref_format {
 #define OPT_REF_SORT(var) \
 	OPT_STRING_LIST(0, "sort", (var), \
 			N_("key"), N_("field name to sort on"))
+#define OPT_REF_FILTER_EXCLUDE(var) \
+	OPT_STRVEC(0, "exclude", &(var)->exclude, \
+		   N_("pattern"), N_("exclude refs which match pattern"))
 
 /*
  * API for filtering a set of refs. Based on the type of refs the user
t/t6300-for-each-ref.sh +35 -0
index 5c00607608..7e8d578522 100755
@@ -447,6 +447,41 @@ test_expect_success 'exercise glob patterns with prefixes' '
 	test_cmp expected actual
 '
 
+cat >expected <<\EOF
+refs/tags/bar
+refs/tags/baz
+refs/tags/testtag
+EOF
+
+test_expect_success 'exercise patterns with prefix exclusions' '
+	for tag in foo/one foo/two foo/three bar baz
+	do
+		git tag "$tag" || return 1
+	done &&
+	test_when_finished "git tag -d foo/one foo/two foo/three bar baz" &&
+	git for-each-ref --format="%(refname)" \
+		refs/tags/ --exclude=refs/tags/foo >actual &&
+	test_cmp expected actual
+'
+
+cat >expected <<\EOF
+refs/tags/bar
+refs/tags/baz
+refs/tags/foo/one
+refs/tags/testtag
+EOF
+
+test_expect_success 'exercise patterns with pattern exclusions' '
+	for tag in foo/one foo/two foo/three bar baz
+	do
+		git tag "$tag" || return 1
+	done &&
+	test_when_finished "git tag -d foo/one foo/two foo/three bar baz" &&
+	git for-each-ref --format="%(refname)" \
+		refs/tags/ --exclude="refs/tags/foo/t*" >actual &&
+	test_cmp expected actual
+'
+
 cat >expected <<\EOF
 'refs/heads/main'
 'refs/remotes/origin/main'
Signature · 1 line
2.40.1.477.g956c797dfc
#15 lore raw

PATCH 7/15 refs: plumb `exclude_patterns` argument throughout

In reply to Taylor Blau

The subsequent patch will want to access an optional excluded_patterns array within refs/packed-backend.c. To do so, the refs subsystem needs to be updated to pass this value across a number of different locations.

Prepare for a future patch by introducing this plumbing now, passing NULLs at top-level APIs in order to make that patch less noisy and more easily readable.

---
 ls-refs.c             |  2 +-
 ref-filter.c          |  5 +++--
 refs.c                | 32 +++++++++++++++++++-------------
 refs.h                |  8 +++++++-
 refs/debug.c          |  5 +++--
 refs/files-backend.c  |  5 +++--
 refs/packed-backend.c |  5 +++--
 refs/refs-internal.h  |  7 ++++---
 revision.c            |  2 +-
 9 files changed, 44 insertions(+), 27 deletions(-)
ls-refs.c +1 -1
index b9f3e08ec3..c7ad39611a 100644
@@ -192,7 +192,7 @@ int ls_refs(struct repository *r, struct packet_reader *request)
 		strvec_push(&data.prefixes, "");
 	refs_for_each_fullref_in_prefixes(get_main_ref_store(r),
 					  get_git_namespace(), data.prefixes.v,
-					  send_ref, &data);
+					  NULL, send_ref, &data);
 	packet_fflush(stdout);
 	strvec_clear(&data.prefixes);
 	strbuf_release(&data.buf);
ref-filter.c +3 -2
index 93dc9b331f..c8ced1104b 100644
@@ -2207,12 +2207,13 @@ static int for_each_fullref_in_pattern(struct ref_filter *filter,
 
 	if (!filter->name_patterns[0]) {
 		/* no patterns; we have to look at everything */
-		return for_each_fullref_in("", cb, cb_data);
+		return refs_for_each_fullref_in(get_main_ref_store(the_repository),
+						 "", NULL, cb, cb_data);
 	}
 
 	return refs_for_each_fullref_in_prefixes(get_main_ref_store(the_repository),
 						 NULL, filter->name_patterns,
-						 cb, cb_data);
+						 NULL, cb, cb_data);
 }
 
 /*
refs.c +19 -13
index b9b77d2eff..538bde644e 100644
@@ -1526,7 +1526,9 @@ int head_ref(each_ref_fn fn, void *cb_data)
 
 struct ref_iterator *refs_ref_iterator_begin(
 		struct ref_store *refs,
-		const char *prefix, int trim,
+		const char *prefix,
+		const char **exclude_patterns,
+		int trim,
 		enum do_for_each_ref_flags flags)
 {
 	struct ref_iterator *iter;
@@ -1542,8 +1544,7 @@ struct ref_iterator *refs_ref_iterator_begin(
 		}
 	}
 
-	iter = refs->be->iterator_begin(refs, prefix, flags);
-
+	iter = refs->be->iterator_begin(refs, prefix, exclude_patterns, flags);
 	/*
 	 * `iterator_begin()` already takes care of prefix, but we
 	 * might need to do some trimming:
@@ -1577,7 +1578,7 @@ static int do_for_each_repo_ref(struct repository *r, const char *prefix,
 	if (!refs)
 		return 0;
 
-	iter = refs_ref_iterator_begin(refs, prefix, trim, flags);
+	iter = refs_ref_iterator_begin(refs, prefix, NULL, trim, flags);
 
 	return do_for_each_repo_ref_iterator(r, iter, fn, cb_data);
 }
@@ -1599,6 +1600,7 @@ static int do_for_each_ref_helper(struct repository *r,
 }
 
 static int do_for_each_ref(struct ref_store *refs, const char *prefix,
+			   const char **exclude_patterns,
 			   each_ref_fn fn, int trim,
 			   enum do_for_each_ref_flags flags, void *cb_data)
 {
@@ -1608,7 +1610,8 @@ static int do_for_each_ref(struct ref_store *refs, const char *prefix,
 	if (!refs)
 		return 0;
 
-	iter = refs_ref_iterator_begin(refs, prefix, trim, flags);
+	iter = refs_ref_iterator_begin(refs, prefix, exclude_patterns, trim,
+				       flags);
 
 	return do_for_each_repo_ref_iterator(the_repository, iter,
 					do_for_each_ref_helper, &hp);
@@ -1616,7 +1619,7 @@ static int do_for_each_ref(struct ref_store *refs, const char *prefix,
 
 int refs_for_each_ref(struct ref_store *refs, each_ref_fn fn, void *cb_data)
 {
-	return do_for_each_ref(refs, "", fn, 0, 0, cb_data);
+	return do_for_each_ref(refs, "", NULL, fn, 0, 0, cb_data);
 }
 
 int for_each_ref(each_ref_fn fn, void *cb_data)
@@ -1627,7 +1630,7 @@ int for_each_ref(each_ref_fn fn, void *cb_data)
 int refs_for_each_ref_in(struct ref_store *refs, const char *prefix,
 			 each_ref_fn fn, void *cb_data)
 {
-	return do_for_each_ref(refs, prefix, fn, strlen(prefix), 0, cb_data);
+	return do_for_each_ref(refs, prefix, NULL, fn, strlen(prefix), 0, cb_data);
 }
 
 int for_each_ref_in(const char *prefix, each_ref_fn fn, void *cb_data)
@@ -1638,13 +1641,14 @@ int for_each_ref_in(const char *prefix, each_ref_fn fn, void *cb_data)
 int for_each_fullref_in(const char *prefix, each_ref_fn fn, void *cb_data)
 {
 	return do_for_each_ref(get_main_ref_store(the_repository),
-			       prefix, fn, 0, 0, cb_data);
+			       prefix, NULL, fn, 0, 0, cb_data);
 }
 
 int refs_for_each_fullref_in(struct ref_store *refs, const char *prefix,
+			     const char **exclude_patterns,
 			     each_ref_fn fn, void *cb_data)
 {
-	return do_for_each_ref(refs, prefix, fn, 0, 0, cb_data);
+	return do_for_each_ref(refs, prefix, exclude_patterns, fn, 0, 0, cb_data);
 }
 
 int for_each_replace_ref(struct repository *r, each_repo_ref_fn fn, void *cb_data)
@@ -1661,14 +1665,14 @@ int for_each_namespaced_ref(each_ref_fn fn, void *cb_data)
 	int ret;
 	strbuf_addf(&buf, "%srefs/", get_git_namespace());
 	ret = do_for_each_ref(get_main_ref_store(the_repository),
-			      buf.buf, fn, 0, 0, cb_data);
+			      buf.buf, NULL, fn, 0, 0, cb_data);
 	strbuf_release(&buf);
 	return ret;
 }
 
 int refs_for_each_rawref(struct ref_store *refs, each_ref_fn fn, void *cb_data)
 {
-	return do_for_each_ref(refs, "", fn, 0,
+	return do_for_each_ref(refs, "", NULL, fn, 0,
 			       DO_FOR_EACH_INCLUDE_BROKEN, cb_data);
 }
 
@@ -1738,6 +1742,7 @@ static void find_longest_prefixes(struct string_list *out,
 int refs_for_each_fullref_in_prefixes(struct ref_store *ref_store,
 				      const char *namespace,
 				      const char **patterns,
+				      const char **exclude_patterns,
 				      each_ref_fn fn, void *cb_data)
 {
 	struct string_list prefixes = STRING_LIST_INIT_DUP;
@@ -1753,7 +1758,8 @@ int refs_for_each_fullref_in_prefixes(struct ref_store *ref_store,
 
 	for_each_string_list_item(prefix, &prefixes) {
 		strbuf_addstr(&buf, prefix->string);
-		ret = refs_for_each_fullref_in(ref_store, buf.buf, fn, cb_data);
+		ret = refs_for_each_fullref_in(ref_store, buf.buf,
+					       exclude_patterns, fn, cb_data);
 		if (ret)
 			break;
 		strbuf_setlen(&buf, namespace_len);
@@ -2408,7 +2414,7 @@ int refs_verify_refname_available(struct ref_store *refs,
 	strbuf_addstr(&dirname, refname + dirname.len);
 	strbuf_addch(&dirname, '/');
 
-	iter = refs_ref_iterator_begin(refs, dirname.buf, 0,
+	iter = refs_ref_iterator_begin(refs, dirname.buf, NULL, 0,
 				       DO_FOR_EACH_INCLUDE_BROKEN);
 	while ((ok = ref_iterator_advance(iter)) == ITER_OK) {
 		if (skip &&
refs.h +7 -1
index 123cfa4424..d672d636cf 100644
@@ -338,6 +338,7 @@ int for_each_ref(each_ref_fn fn, void *cb_data);
 int for_each_ref_in(const char *prefix, each_ref_fn fn, void *cb_data);
 
 int refs_for_each_fullref_in(struct ref_store *refs, const char *prefix,
+			     const char **exclude_patterns,
 			     each_ref_fn fn, void *cb_data);
 int for_each_fullref_in(const char *prefix, each_ref_fn fn, void *cb_data);
 
@@ -345,10 +346,15 @@ int for_each_fullref_in(const char *prefix, each_ref_fn fn, void *cb_data);
  * iterate all refs in "patterns" by partitioning patterns into disjoint sets
  * and iterating the longest-common prefix of each set.
  *
+ * references matching any pattern in "exclude_patterns" are omitted from the
+ * result set on a best-effort basis.
+ *
  * callers should be prepared to ignore references that they did not ask for.
  */
 int refs_for_each_fullref_in_prefixes(struct ref_store *refs,
-				      const char *namespace, const char **patterns,
+				      const char *namespace,
+				      const char **patterns,
+				      const char **exclude_patterns,
 				      each_ref_fn fn, void *cb_data);
 
 /**
refs/debug.c +3 -2
index adc34c836f..8131133e99 100644
@@ -228,11 +228,12 @@ static struct ref_iterator_vtable debug_ref_iterator_vtable = {
 
 static struct ref_iterator *
 debug_ref_iterator_begin(struct ref_store *ref_store, const char *prefix,
-			 unsigned int flags)
+			 const char **exclude_patterns, unsigned int flags)
 {
 	struct debug_ref_store *drefs = (struct debug_ref_store *)ref_store;
 	struct ref_iterator *res =
-		drefs->refs->be->iterator_begin(drefs->refs, prefix, flags);
+		drefs->refs->be->iterator_begin(drefs->refs, prefix,
+						exclude_patterns, flags);
 	struct debug_ref_iterator *diter = xcalloc(1, sizeof(*diter));
 	base_ref_iterator_init(&diter->base, &debug_ref_iterator_vtable, 1);
 	diter->iter = res;
refs/files-backend.c +3 -2
index d0581ee41a..5fae864334 100644
@@ -827,7 +827,8 @@ static struct ref_iterator_vtable files_ref_iterator_vtable = {
 
 static struct ref_iterator *files_ref_iterator_begin(
 		struct ref_store *ref_store,
-		const char *prefix, unsigned int flags)
+		const char *prefix, const char **exclude_patterns,
+		unsigned int flags)
 {
 	struct files_ref_store *refs;
 	struct ref_iterator *loose_iter, *packed_iter, *overlay_iter;
@@ -872,7 +873,7 @@ static struct ref_iterator *files_ref_iterator_begin(
 	 * the packed and loose references.
 	 */
 	packed_iter = refs_ref_iterator_begin(
-			refs->packed_ref_store, prefix, 0,
+			refs->packed_ref_store, prefix, exclude_patterns, 0,
 			DO_FOR_EACH_INCLUDE_BROKEN);
 
 	overlay_iter = overlay_ref_iterator_begin(loose_iter, packed_iter);
refs/packed-backend.c +3 -2
index 34c0c4e20f..e54e78e540 100644
@@ -923,7 +923,8 @@ static struct ref_iterator_vtable packed_ref_iterator_vtable = {
 
 static struct ref_iterator *packed_ref_iterator_begin(
 		struct ref_store *ref_store,
-		const char *prefix, unsigned int flags)
+		const char *prefix, const char **exclude_patterns,
+		unsigned int flags)
 {
 	struct packed_ref_store *refs;
 	struct snapshot *snapshot;
@@ -1148,7 +1149,7 @@ static int write_with_updates(struct packed_ref_store *refs,
 	 * list of refs is exhausted, set iter to NULL. When the list
 	 * of updates is exhausted, leave i set to updates->nr.
 	 */
-	iter = packed_ref_iterator_begin(&refs->base, "",
+	iter = packed_ref_iterator_begin(&refs->base, "", NULL,
 					 DO_FOR_EACH_INCLUDE_BROKEN);
 	if ((ok = ref_iterator_advance(iter)) != ITER_OK)
 		iter = NULL;
refs/refs-internal.h +4 -3
index a85d113123..28a11b9d61 100644
@@ -367,8 +367,8 @@ int is_empty_ref_iterator(struct ref_iterator *ref_iterator);
  */
 struct ref_iterator *refs_ref_iterator_begin(
 		struct ref_store *refs,
-		const char *prefix, int trim,
-		enum do_for_each_ref_flags flags);
+		const char *prefix, const char **exclude_patterns,
+		int trim, enum do_for_each_ref_flags flags);
 
 /*
  * A callback function used to instruct merge_ref_iterator how to
@@ -570,7 +570,8 @@ typedef int copy_ref_fn(struct ref_store *ref_store,
  */
 typedef struct ref_iterator *ref_iterator_begin_fn(
 		struct ref_store *ref_store,
-		const char *prefix, unsigned int flags);
+		const char *prefix, const char **exclude_patterns,
+		unsigned int flags);
 
 /* reflog functions */
 
revision.c +1 -1
index b33cc1d106..89953592f9 100644
@@ -2670,7 +2670,7 @@ static int for_each_bisect_ref(struct ref_store *refs, each_ref_fn fn,
 	struct strbuf bisect_refs = STRBUF_INIT;
 	int status;
 	strbuf_addf(&bisect_refs, "refs/bisect/%s", term);
-	status = refs_for_each_fullref_in(refs, bisect_refs.buf, fn, cb_data);
+	status = refs_for_each_fullref_in(refs, bisect_refs.buf, NULL, fn, cb_data);
 	strbuf_release(&bisect_refs);
 	return status;
 }
Signature · 1 line
2.40.1.477.g956c797dfc
#18 lore raw

PATCH 8/15 refs/packed-backend.c: refactor `find_reference_location()`

In reply to Taylor Blau

The function find_reference_location() is used to perform a binary search-like function over the contents of a repository's $GIT_DIR/packed-refs file.

The search it implements is unlike a standard binary search in that the records it searches over are not of a fixed width, so the comparison must locate the end of a record before comparing it.

Extract the core routine of find_reference_location() in order to implement a function in the following patch which will find the first location in the packed-refs file that *doesn't* match the given pattern.

The behavior of find_reference_location() is unchanged.

---
 refs/packed-backend.c | 46 +++++++++++++++++++++++++------------------
 1 file changed, 27 insertions(+), 19 deletions(-)
refs/packed-backend.c +27 -19
index e54e78e540..98f96bf3ee 100644
@@ -302,7 +302,8 @@ static int cmp_packed_ref_records(const void *v1, const void *v2)
  * Compare a snapshot record at `rec` to the specified NUL-terminated
  * refname.
  */
-static int cmp_record_to_refname(const char *rec, const char *refname)
+static int cmp_record_to_refname(const char *rec, const char *refname,
+				 int start)
 {
 	const char *r1 = rec + the_hash_algo->hexsz + 1;
 	const char *r2 = refname;
@@ -311,7 +312,7 @@ static int cmp_record_to_refname(const char *rec, const char *refname)
 		if (*r1 == '\n')
 			return *r2 ? -1 : 0;
 		if (!*r2)
-			return 1;
+			return start ? 1 : -1;
 		if (*r1 != *r2)
 			return (unsigned char)*r1 < (unsigned char)*r2 ? -1 : +1;
 		r1++;
@@ -526,22 +527,9 @@ static int load_contents(struct snapshot *snapshot)
 	return 1;
 }
 
-/*
- * Find the place in `snapshot->buf` where the start of the record for
- * `refname` starts. If `mustexist` is true and the reference doesn't
- * exist, then return NULL. If `mustexist` is false and the reference
- * doesn't exist, then return the point where that reference would be
- * inserted, or `snapshot->eof` (which might be NULL) if it would be
- * inserted at the end of the file. In the latter mode, `refname`
- * doesn't have to be a proper reference name; for example, one could
- * search for "refs/replace/" to find the start of any replace
- * references.
- *
- * The record is sought using a binary search, so `snapshot->buf` must
- * be sorted.
- */
-static const char *find_reference_location(struct snapshot *snapshot,
-					   const char *refname, int mustexist)
+static const char *find_reference_location_1(struct snapshot *snapshot,
+					     const char *refname, int mustexist,
+					     int start)
 {
 	/*
 	 * This is not *quite* a garden-variety binary search, because
@@ -571,7 +559,7 @@ static const char *find_reference_location(struct snapshot *snapshot,
 
 		mid = lo + (hi - lo) / 2;
 		rec = find_start_of_record(lo, mid);
-		cmp = cmp_record_to_refname(rec, refname);
+		cmp = cmp_record_to_refname(rec, refname, start);
 		if (cmp < 0) {
 			lo = find_end_of_record(mid, hi);
 		} else if (cmp > 0) {
@@ -587,6 +575,26 @@ static const char *find_reference_location(struct snapshot *snapshot,
 		return lo;
 }
 
+/*
+ * Find the place in `snapshot->buf` where the start of the record for
+ * `refname` starts. If `mustexist` is true and the reference doesn't
+ * exist, then return NULL. If `mustexist` is false and the reference
+ * doesn't exist, then return the point where that reference would be
+ * inserted, or `snapshot->eof` (which might be NULL) if it would be
+ * inserted at the end of the file. In the latter mode, `refname`
+ * doesn't have to be a proper reference name; for example, one could
+ * search for "refs/replace/" to find the start of any replace
+ * references.
+ *
+ * The record is sought using a binary search, so `snapshot->buf` must
+ * be sorted.
+ */
+static const char *find_reference_location(struct snapshot *snapshot,
+					   const char *refname, int mustexist)
+{
+	return find_reference_location_1(snapshot, refname, mustexist, 1);
+}
+
 /*
  * Create a newly-allocated `snapshot` of the `packed-refs` file in
  * its current state and return it. The return value will already have
Signature · 1 line
2.40.1.477.g956c797dfc
#21 lore raw

PATCH 9/15 refs/packed-backend.c: implement skip lists to avoid excluded pattern(s)

In reply to Taylor Blau

When iterating through the packed-refs file in order to answer a query like:

    $ git for-each-ref --exclude=refs/__hidden__

it would be useful to avoid walking over all of the entries in refs/__hidden__/* when possible, since we know that the ref-filter code is going to throw them away anyways.

In certain circumstances, doing so is possible. The algorithm for doing so is as follows:

- For each excluded pattern, find the first record that matches it, and the first pattern that *doesn't* match it (i.e. the location you'd next want to consider when excluding that pattern).

- Sort the patterns by their starting location within the packed-refs file.

- Construct a skip list of regions by combining adjacent and overlapping regions from the previous step.

- When iterating through the packed-refs file, if iter->pos is ever contained in one of the regions from the previous steps, advance iter->pos past the end of that region, and continue enumeration.

Note that this optimization is only possible when none of the excluded pattern(s) have special meta-characters in them. To see why this is the case, consider the exclusion pattern "refs/foo[a]". In general, in order to find the location of the first record that matches this pattern, we could only consider up to the first meta-character, "refs/foo". But this doesn't work, since the excluded region we'd come up with would include "refs/foobar", even though it is not excluded.

There are a few other gotchas worth considering. First, note that the skip list is sorted, so once we skip past a region, we can avoid considering it (or any regions preceding it) again. The member skip_pos is used to track the first next-possible region to jump through.

Second, note that the exclusion list is best-effort, since we do not handle loose references, and because of the meta-character issue above.

In repositories with a large number of hidden references, the speed-up can be significant. Tests here are done with a copy of linux.git with a reference "refs/pull/N" pointing at every commit, as in:

    $ git rev-list HEAD | awk '{ print "create refs/pull/" NR " " $0 }' |
        git update-ref --stdin
    $ git pack-refs --all

, it is significantly faster to have for-each-ref skip over the excluded references, as opposed to filtering them out after the fact:

    $ hyperfine \
      'git for-each-ref --format="%(objectname) %(refname)" | grep -vE "^[0-9a-f]{40} refs/pull/"' \
      'git.compile for-each-ref --format="%(objectname) %(refname)" --exclude="refs/pull"'
    Benchmark 1: git for-each-ref --format="%(objectname) %(refname)" | grep -vE "^[0-9a-f]{40} refs/pull/"
      Time (mean ± σ):     802.7 ms ±   2.1 ms    [User: 691.6 ms, System: 147.0 ms]
      Range (min … max):   800.0 ms … 807.7 ms    10 runs
    Benchmark 2: git.compile for-each-ref --format="%(objectname) %(refname)" --exclude="refs/pull"
      Time (mean ± σ):       4.7 ms ±   0.3 ms    [User: 0.7 ms, System: 4.0 ms]
      Range (min … max):     4.3 ms …   6.7 ms    422 runs

    Summary
      'git.compile for-each-ref --format="%(objectname) %(refname)" --exclude="refs/pull"' ran
      172.03 ± 9.60 times faster than 'git for-each-ref --format="%(objectname) %(refname)" | grep -vE "^[0-9a-f]{40} refs/pull/"'

Using the skip list is fairly straightforward (see the changes to refs/packed-backend.c::next_record()), but constructing the list is not. To ensure that the construction is correct, add a new suite of tests in t1419 covering various corner cases (overlapping regions, partially overlapping regions, adjacent regions, etc.).

---
 ref-filter.c              |   5 +-
 refs/packed-backend.c     | 150 +++++++++++++++++++++++++++++++++++++-
 t/helper/test-ref-store.c |  10 +++
 t/t1419-exclude-refs.sh   | 101 +++++++++++++++++++++++++
 4 files changed, 263 insertions(+), 3 deletions(-)
 create mode 100755 t/t1419-exclude-refs.sh
ref-filter.c +3 -2
index c8ced1104b..56ebd332fa 100644
@@ -2208,12 +2208,13 @@ static int for_each_fullref_in_pattern(struct ref_filter *filter,
 	if (!filter->name_patterns[0]) {
 		/* no patterns; we have to look at everything */
 		return refs_for_each_fullref_in(get_main_ref_store(the_repository),
-						 "", NULL, cb, cb_data);
+						 "", filter->exclude.v, cb, cb_data);
 	}
 
 	return refs_for_each_fullref_in_prefixes(get_main_ref_store(the_repository),
 						 NULL, filter->name_patterns,
-						 NULL, cb, cb_data);
+						 filter->exclude.v,
+						 cb, cb_data);
 }
 
 /*
refs/packed-backend.c +149 -1

193 lines not on this page · raw message

t/helper/test-ref-store.c +10 -0
index 6d8f844e9c..2bff003f7c 100644
@@ -175,6 +175,15 @@ static int cmd_for_each_ref(struct ref_store *refs, const char **argv)
 	return refs_for_each_ref_in(refs, prefix, each_ref, NULL);
 }
 
+static int cmd_for_each_ref__exclude(struct ref_store *refs, const char **argv)
+{
+	const char *prefix = notnull(*argv++, "prefix");
+	const char **exclude_patterns = argv;
+
+	return refs_for_each_fullref_in(refs, prefix, exclude_patterns, each_ref,
+					NULL);
+}
+
 static int cmd_resolve_ref(struct ref_store *refs, const char **argv)
 {
 	struct object_id oid = *null_oid();
@@ -307,6 +316,7 @@ static struct command commands[] = {
 	{ "delete-refs", cmd_delete_refs },
 	{ "rename-ref", cmd_rename_ref },
 	{ "for-each-ref", cmd_for_each_ref },
+	{ "for-each-ref--exclude", cmd_for_each_ref__exclude },
 	{ "resolve-ref", cmd_resolve_ref },
 	{ "verify-ref", cmd_verify_ref },
 	{ "for-each-reflog", cmd_for_each_reflog },
t/t1419-exclude-refs.sh new file +101 -0

101 lines not on this page · raw message

Signature · 1 line
2.40.1.477.g956c797dfc
#30 lore raw

PATCH 10/15 refs/packed-backend.c: add trace2 counters for skip list

In reply to Taylor Blau

The previous commit added low-level tests to ensure that the packed-refs iterator did not enumerate excluded sections of the refspace.

However, there was no guarantee that these sections weren't being visited, only that they were being suppressed from the output. To harden these tests, add a trace2 counter which tracks the number of regions skipped by the packed-refs iterator, and assert on its value.

---
 refs/packed-backend.c   |  2 ++
 t/t1419-exclude-refs.sh | 54 ++++++++++++++++++++++++++++-------------
 trace2.h                |  2 ++
 trace2/tr2_ctr.c        |  5 ++++
 4 files changed, 46 insertions(+), 17 deletions(-)
refs/packed-backend.c +2 -0
index 137a4233f6..ddfa9add14 100644
@@ -11,6 +11,7 @@
 #include "../chdir-notify.h"
 #include "../wrapper.h"
 #include "../write-or-die.h"
+#include "../trace2.h"
 
 enum mmap_strategy {
 	/*
@@ -844,6 +845,7 @@ static int next_record(struct packed_ref_iterator *iter)
 		iter->skip_pos++;
 		if (iter->pos < curr->end) {
 			iter->pos = curr->end;
+			trace2_counter_add(TRACE2_COUNTER_ID_PACKED_REFS_SKIPS, 1);
 			break;
 		}
 	}
t/t1419-exclude-refs.sh +37 -17

116 lines not on this page · raw message

trace2.h +2 -0
index 4ced30c0db..6a116f60a9 100644
@@ -551,6 +551,8 @@ enum trace2_counter_id {
 	TRACE2_COUNTER_ID_TEST1 = 0, /* emits summary event only */
 	TRACE2_COUNTER_ID_TEST2,     /* emits summary and thread events */
 
+	TRACE2_COUNTER_ID_PACKED_REFS_SKIPS, /* counts number of skips */
+
 	/* Add additional counter definitions before here. */
 	TRACE2_NUMBER_OF_COUNTERS
 };
trace2/tr2_ctr.c +5 -0
index b342d3b1a3..f7efbc7646 100644
@@ -27,6 +27,11 @@ static struct tr2_counter_metadata tr2_counter_metadata[TRACE2_NUMBER_OF_COUNTER
 		.name = "test2",
 		.want_per_thread_events = 1,
 	},
+	[TRACE2_COUNTER_ID_PACKED_REFS_SKIPS] = {
+		.category = "packed-refs",
+		.name = "skips_made",
+		.want_per_thread_events = 0,
+	},
 
 	/* Add additional metadata before here. */
 };
Signature · 1 line
2.40.1.477.g956c797dfc
#32 lore raw

PATCH 12/15 refs/packed-backend.c: ignore complicated hidden refs rules

In reply to Taylor Blau

In subsequent commits, we'll teach receive-pack and upload-pack to use the new skip-list feature in the packed-refs iterator by ignoring references which are mentioned via its respective hideRefs lists.

However, the packed-ref skip lists cannot handle un-hiding rules (that begin with '!'), or namespace comparisons (that begin with '^'). Detect and avoid these cases by falling back to the normal enumeration without a skip list when such patterns exist.

---
 refs/packed-backend.c   | 19 +++++++++++++++++++
 t/t1419-exclude-refs.sh | 10 ++++++++++
 2 files changed, 29 insertions(+)
refs/packed-backend.c +19 -0
index ddfa9add14..7f09201f35 100644
@@ -1013,6 +1013,25 @@ static void populate_excluded_skip_list(struct packed_ref_iterator *iter,
 	if (!excluded_patterns)
 		return;
 
+	for (pattern = excluded_patterns; *pattern; pattern++) {
+		/*
+		 * We also can't feed any excludes from hidden refs
+		 * config sections, since later rules may override
+		 * previous ones. For example, with rules "refs/foo" and
+		 * "!refs/foo/bar", we should show "refs/foo/bar" (and
+		 * everything underneath it), but the earlier exclusion
+		 * would cause us to skip all of "refs/foo". We likewise
+		 * don't implement the namespace stripping required for
+		 * '^' rules.
+		 *
+		 * Both are possible to do, but complicated, so avoid
+		 * populating the skip list at all if we see either of
+		 * these patterns.
+		 */
+		if (**pattern == '!' || **pattern == '^')
+			return;
+	}
+
 	for (pattern = excluded_patterns; *pattern; pattern++) {
 		struct skip_list_entry *e;
 
t/t1419-exclude-refs.sh +10 -0
index 051b5a54ce..026e4414cd 100755
@@ -118,4 +118,14 @@ test_expect_success 'for_each_ref__exclude(refs/heads/ba*)' '
 	assert_no_skips
 '
 
+test_expect_success 'for_each_ref__exclude(refs/heads/foo, !refs/heads/foo/1)' '
+	# discards complex hidden ref rules
+	for_each_ref__exclude refs/heads refs/heads/foo "!refs/heads/foo/1" \
+		>actual 2>perf &&
+	for_each_ref >expect &&
+
+	test_cmp expect actual &&
+	assert_no_skips
+'
+
 test_done
Signature · 1 line
2.40.1.477.g956c797dfc
#31 lore raw

PATCH 11/15 revision.h: store hidden refs in a `strvec`

In reply to Taylor Blau

In subsequent commits, it will be convenient to have a 'const char **' of hidden refs (matching transfer.hiderefs, uploadpack.hideRefs, etc.), instead of a string_list.

Convert spots throughout the tree that store the list of hidden refs from a string_list to a strvec.

Note that in parse_hide_refs_config() there is an ugly const-cast used to avoid an extra copy of each value before trimming any trailing slash characters. This could instead be written as:

    ref = xstrdup(value);
    len = strlen(ref);
    while (len && ref[len - 1] == '/')
            ref[--len] = '\0';
    strvec_push(hide_refs, ref);
    free(ref);

but the double-copy (once when calling xstrdup(), and another via strvec_push()) is wasteful.

---
 builtin/receive-pack.c |  4 ++--
 ls-refs.c              |  6 +++---
 refs.c                 | 11 ++++++-----
 refs.h                 |  4 ++--
 revision.c             |  2 +-
 revision.h             |  5 +++--
 upload-pack.c          | 10 +++++-----
 7 files changed, 22 insertions(+), 20 deletions(-)
builtin/receive-pack.c +2 -2
index d22180435c..064df74715 100644
@@ -89,7 +89,7 @@ static struct object_id push_cert_oid;
 static struct signature_check sigcheck;
 static const char *push_cert_nonce;
 static const char *cert_nonce_seed;
-static struct string_list hidden_refs = STRING_LIST_INIT_DUP;
+static struct strvec hidden_refs = STRVEC_INIT;
 
 static const char *NONCE_UNSOLICITED = "UNSOLICITED";
 static const char *NONCE_BAD = "BAD";
@@ -2618,7 +2618,7 @@ int cmd_receive_pack(int argc, const char **argv, const char *prefix)
 		packet_flush(1);
 	oid_array_clear(&shallow);
 	oid_array_clear(&ref);
-	string_list_clear(&hidden_refs, 0);
+	strvec_clear(&hidden_refs);
 	free((void *)push_cert_nonce);
 	return 0;
 }
ls-refs.c +3 -3
index c7ad39611a..d3d7e13e5a 100644
@@ -71,7 +71,7 @@ struct ls_refs_data {
 	unsigned symrefs;
 	struct strvec prefixes;
 	struct strbuf buf;
-	struct string_list hidden_refs;
+	struct strvec hidden_refs;
 	unsigned unborn : 1;
 };
 
@@ -154,7 +154,7 @@ int ls_refs(struct repository *r, struct packet_reader *request)
 	memset(&data, 0, sizeof(data));
 	strvec_init(&data.prefixes);
 	strbuf_init(&data.buf, 0);
-	string_list_init_dup(&data.hidden_refs);
+	strvec_init(&data.hidden_refs);
 
 	git_config(ls_refs_config, &data);
 
@@ -196,7 +196,7 @@ int ls_refs(struct repository *r, struct packet_reader *request)
 	packet_fflush(stdout);
 	strvec_clear(&data.prefixes);
 	strbuf_release(&data.buf);
-	string_list_clear(&data.hidden_refs, 0);
+	strvec_clear(&data.hidden_refs);
 	return 0;
 }
 
refs.c +6 -5
index 538bde644e..ec4d5b9101 100644
@@ -1427,7 +1427,7 @@ char *shorten_unambiguous_ref(const char *refname, int strict)
 }
 
 int parse_hide_refs_config(const char *var, const char *value, const char *section,
-			   struct string_list *hide_refs)
+			   struct strvec *hide_refs)
 {
 	const char *key;
 	if (!strcmp("transfer.hiderefs", var) ||
@@ -1438,22 +1438,23 @@ int parse_hide_refs_config(const char *var, const char *value, const char *secti
 
 		if (!value)
 			return config_error_nonbool(var);
-		ref = xstrdup(value);
+
+		/* drop const to remove trailing '/' characters */
+		ref = (char *)strvec_push(hide_refs, value);
 		len = strlen(ref);
 		while (len && ref[len - 1] == '/')
 			ref[--len] = '\0';
-		string_list_append_nodup(hide_refs, ref);
 	}
 	return 0;
 }
 
 int ref_is_hidden(const char *refname, const char *refname_full,
-		  const struct string_list *hide_refs)
+		  const struct strvec *hide_refs)
 {
 	int i;
 
 	for (i = hide_refs->nr - 1; i >= 0; i--) {
-		const char *match = hide_refs->items[i].string;
+		const char *match = hide_refs->v[i];
 		const char *subject;
 		int neg = 0;
 		const char *p;
refs.h +2 -2
index d672d636cf..a7751a1fc9 100644
@@ -810,7 +810,7 @@ int update_ref(const char *msg, const char *refname,
 	       unsigned int flags, enum action_on_err onerr);
 
 int parse_hide_refs_config(const char *var, const char *value, const char *,
-			   struct string_list *);
+			   struct strvec *);
 
 /*
  * Check whether a ref is hidden. If no namespace is set, both the first and
@@ -820,7 +820,7 @@ int parse_hide_refs_config(const char *var, const char *value, const char *,
  * the ref is outside that namespace, the first parameter is NULL. The second
  * parameter always points to the full ref name.
  */
-int ref_is_hidden(const char *, const char *, const struct string_list *);
+int ref_is_hidden(const char *, const char *, const struct strvec *);
 
 /* Is this a per-worktree ref living in the refs/ namespace? */
 int is_per_worktree_ref(const char *refname);
revision.c +1 -1
index 89953592f9..7c9367a266 100644
@@ -1558,7 +1558,7 @@ void init_ref_exclusions(struct ref_exclusions *exclusions)
 void clear_ref_exclusions(struct ref_exclusions *exclusions)
 {
 	string_list_clear(&exclusions->excluded_refs, 0);
-	string_list_clear(&exclusions->hidden_refs, 0);
+	strvec_clear(&exclusions->hidden_refs);
 	exclusions->hidden_refs_configured = 0;
 }
 
revision.h +3 -2
index e8f6de9684..30b5b5919d 100644
@@ -9,6 +9,7 @@
 #include "commit-slab-decl.h"
 #include "ident.h"
 #include "list-objects-filter-options.h"
+#include "strvec.h"
 
 /**
  * The revision walking API offers functions to build a list of revisions
@@ -94,7 +95,7 @@ struct ref_exclusions {
 	 * Hidden refs is a list of patterns that is to be hidden via
 	 * `ref_is_hidden()`.
 	 */
-	struct string_list hidden_refs;
+	struct strvec hidden_refs;
 
 	/*
 	 * Indicates whether hidden refs have been configured. This is to
@@ -109,7 +110,7 @@ struct ref_exclusions {
  */
 #define REF_EXCLUSIONS_INIT { \
 	.excluded_refs = STRING_LIST_INIT_DUP, \
-	.hidden_refs = STRING_LIST_INIT_DUP, \
+	.hidden_refs = STRVEC_INIT, \
 }
 
 struct oidset;
upload-pack.c +5 -5
index 08633dc121..d77d58bdde 100644
@@ -69,7 +69,7 @@ struct upload_pack_data {
 	struct object_array have_obj;
 	struct oid_array haves;					/* v2 only */
 	struct string_list wanted_refs;				/* v2 only */
-	struct string_list hidden_refs;
+	struct strvec hidden_refs;
 
 	struct object_array shallows;
 	struct string_list deepen_not;
@@ -126,7 +126,7 @@ static void upload_pack_data_init(struct upload_pack_data *data)
 {
 	struct string_list symref = STRING_LIST_INIT_DUP;
 	struct string_list wanted_refs = STRING_LIST_INIT_DUP;
-	struct string_list hidden_refs = STRING_LIST_INIT_DUP;
+	struct strvec hidden_refs = STRVEC_INIT;
 	struct object_array want_obj = OBJECT_ARRAY_INIT;
 	struct object_array have_obj = OBJECT_ARRAY_INIT;
 	struct oid_array haves = OID_ARRAY_INIT;
@@ -161,7 +161,7 @@ static void upload_pack_data_clear(struct upload_pack_data *data)
 {
 	string_list_clear(&data->symref, 1);
 	string_list_clear(&data->wanted_refs, 1);
-	string_list_clear(&data->hidden_refs, 0);
+	strvec_clear(&data->hidden_refs);
 	object_array_clear(&data->want_obj);
 	object_array_clear(&data->have_obj);
 	oid_array_clear(&data->haves);
@@ -1169,7 +1169,7 @@ static void receive_needs(struct upload_pack_data *data,
 
 /* return non-zero if the ref is hidden, otherwise 0 */
 static int mark_our_ref(const char *refname, const char *refname_full,
-			const struct object_id *oid, const struct string_list *hidden_refs)
+			const struct object_id *oid, const struct strvec *hidden_refs)
 {
 	struct object *o = lookup_unknown_object(the_repository, oid);
 
@@ -1453,7 +1453,7 @@ static int parse_want(struct packet_writer *writer, const char *line,
 
 static int parse_want_ref(struct packet_writer *writer, const char *line,
 			  struct string_list *wanted_refs,
-			  struct string_list *hidden_refs,
+			  struct strvec *hidden_refs,
 			  struct object_array *want_obj)
 {
 	const char *refname_nons;
Signature · 1 line
2.40.1.477.g956c797dfc
#33 lore raw

PATCH 13/15 refs.h: let `for_each_namespaced_ref()` take excluded patterns

In reply to Taylor Blau

The following commit will want to call for_each_namespaced_ref() with a list of excluded patterns.

We could introduce a variant of that function, say, for_each_namespaced_ref_exclude() which takes the extra parameter, and reimplement the original function in terms of that. But all but one caller (in http-backend.c) will supply the new parameter, so add the new parameter to for_each_namespaced_ref() itself instead of introducing a new function.

For now, supply NULL for the list of excluded patterns at all callers to avoid changing behavior, which we will do in the subsequent commit.

---
 http-backend.c | 2 +-
 refs.c         | 5 +++--
 refs.h         | 3 ++-
 upload-pack.c  | 6 +++---
 4 files changed, 9 insertions(+), 7 deletions(-)
http-backend.c +1 -1
index ac146d85c5..ad500683c8 100644
@@ -559,7 +559,7 @@ static void get_info_refs(struct strbuf *hdr, char *arg UNUSED)
 
 	} else {
 		select_getanyfile(hdr);
-		for_each_namespaced_ref(show_text_ref, &buf);
+		for_each_namespaced_ref(NULL, show_text_ref, &buf);
 		send_strbuf(hdr, "text/plain", &buf);
 	}
 	strbuf_release(&buf);
refs.c +3 -2
index ec4d5b9101..95a7db9563 100644
@@ -1660,13 +1660,14 @@ int for_each_replace_ref(struct repository *r, each_repo_ref_fn fn, void *cb_dat
 				    DO_FOR_EACH_INCLUDE_BROKEN, cb_data);
 }
 
-int for_each_namespaced_ref(each_ref_fn fn, void *cb_data)
+int for_each_namespaced_ref(const char **exclude_patterns,
+			    each_ref_fn fn, void *cb_data)
 {
 	struct strbuf buf = STRBUF_INIT;
 	int ret;
 	strbuf_addf(&buf, "%srefs/", get_git_namespace());
 	ret = do_for_each_ref(get_main_ref_store(the_repository),
-			      buf.buf, NULL, fn, 0, 0, cb_data);
+			      buf.buf, exclude_patterns, fn, 0, 0, cb_data);
 	strbuf_release(&buf);
 	return ret;
 }
refs.h +2 -1
index a7751a1fc9..f23626beca 100644
@@ -372,7 +372,8 @@ int for_each_glob_ref_in(each_ref_fn fn, const char *pattern,
 			 const char *prefix, void *cb_data);
 
 int head_ref_namespaced(each_ref_fn fn, void *cb_data);
-int for_each_namespaced_ref(each_ref_fn fn, void *cb_data);
+int for_each_namespaced_ref(const char **exclude_patterns,
+			    each_ref_fn fn, void *cb_data);
 
 /* can be used to learn about broken ref and symref */
 int refs_for_each_rawref(struct ref_store *refs, each_ref_fn fn, void *cb_data);
upload-pack.c +3 -3
index d77d58bdde..7c646ea5bd 100644
@@ -854,7 +854,7 @@ static void deepen(struct upload_pack_data *data, int depth)
 		 * marked with OUR_REF.
 		 */
 		head_ref_namespaced(check_ref, data);
-		for_each_namespaced_ref(check_ref, data);
+		for_each_namespaced_ref(NULL, check_ref, data);
 
 		get_reachable_list(data, &reachable_shallows);
 		result = get_shallow_commits(&reachable_shallows,
@@ -1378,7 +1378,7 @@ void upload_pack(const int advertise_refs, const int stateless_rpc,
 		if (advertise_refs)
 			data.no_done = 1;
 		head_ref_namespaced(send_ref, &data);
-		for_each_namespaced_ref(send_ref, &data);
+		for_each_namespaced_ref(NULL, send_ref, &data);
 		/*
 		 * fflush stdout before calling advertise_shallow_grafts because send_ref
 		 * uses stdio.
@@ -1388,7 +1388,7 @@ void upload_pack(const int advertise_refs, const int stateless_rpc,
 		packet_flush(1);
 	} else {
 		head_ref_namespaced(check_ref, &data);
-		for_each_namespaced_ref(check_ref, &data);
+		for_each_namespaced_ref(NULL, check_ref, &data);
 	}
 
 	if (!advertise_refs) {
Signature · 1 line
2.40.1.477.g956c797dfc
#34 lore raw

PATCH 14/15 upload-pack.c: avoid enumerating hidden refs where possible

In reply to Taylor Blau

In a similar fashion as a previous commit, teach upload-pack to avoid enumerating hidden references where possible.

Note, however, that there are certain cases where cannot avoid enumerating even hidden references, in particular when either of:

- uploadpack.allowTipSHA1InWant, or
- uploadpack.allowReachableSHA1InWant

are set, corresponding to ALLOW_TIP_SHA1 and ALLOW_REACHABLE_SHA1, respectively.

When either of these bits are set, upload-pack's is_our_ref() function needs to consider the HIDDEN_REF bit of the referent's object flags. So we must visit all references, including the hidden ones, in order to mark their referents with the HIDDEN_REF bit.

When neither ALLOW_TIP_SHA1 nor ALLOW_REACHABLE_SHA1 are set, the is_our_ref() function considers only the OUR_REF bit, and not the HIDDEN_REF one. OUR_REF is applied via mark_our_ref(), and only to objects at the tips of non-hidden references, so we do not need to visit hidden references in this case.

When neither of those bits are set, upload-pack can potentially avoid enumerating a large number of references. In the same example as a previous commit (linux.git with one hidden reference per commit, "refs/pull/N"):

    $ printf 0000 >in
    $ hyperfine --warmup=1 \
      'git -c transfer.hideRefs=refs/pull upload-pack . <in' \
      'git.compile -c transfer.hideRefs=refs/pull -c uploadpack.allowTipSHA1InWant upload-pack . <in' \
      'git.compile -c transfer.hideRefs=refs/pull upload-pack . <in'
    Benchmark 1: git -c transfer.hideRefs=refs/pull upload-pack . <in
      Time (mean ± σ):     406.9 ms ±   1.1 ms    [User: 357.3 ms, System: 49.5 ms]
      Range (min … max):   405.7 ms … 409.2 ms    10 runs

Benchmark 2: git.compile -c transfer.hideRefs=refs/pull -c uploadpack.allowTipSHA1InWant upload-pack . <in
Time (mean ± σ): 406.5 ms ± 1.3 ms [User: 356.5 ms, System: 49.9 ms]
Range (min … max): 404.6 ms … 408.8 ms 10 runs

Benchmark 3: git.compile -c transfer.hideRefs=refs/pull upload-pack . <in
Time (mean ± σ): 4.7 ms ± 0.2 ms [User: 0.7 ms, System: 3.9 ms]
Range (min … max): 4.3 ms … 6.1 ms 472 runs

Summary
'git.compile -c transfer.hideRefs=refs/pull upload-pack . <in' ran
86.62 ± 4.33 times faster than 'git.compile -c transfer.hideRefs=refs/pull -c uploadpack.allowTipSHA1InWant upload-pack . <in' 86.70 ± 4.33 times faster than 'git -c transfer.hideRefs=refs/pull upload-pack . <in'

As above, we must visit every reference when uploadPack.allowTipSHA1InWant is set. But when it is unset, we can visit far fewer references.

---
 upload-pack.c | 33 +++++++++++++++++++++++++++------
 1 file changed, 27 insertions(+), 6 deletions(-)
upload-pack.c +27 -6
index 7c646ea5bd..0162fffce0 100644
@@ -601,11 +601,32 @@ static int get_common_commits(struct upload_pack_data *data,
 	}
 }
 
+static int allow_hidden_refs(enum allow_uor allow_uor)
+{
+	return allow_uor & (ALLOW_TIP_SHA1 | ALLOW_REACHABLE_SHA1);
+}
+
+static void for_each_namespaced_ref_1(each_ref_fn fn,
+				      struct upload_pack_data *data)
+{
+	/*
+	 * If `data->allow_uor` allows updating hidden refs, we need to
+	 * mark all references (including hidden ones), to check in
+	 * `is_our_ref()` below.
+	 *
+	 * Otherwise, we only care about whether each reference's object
+	 * has the OUR_REF bit set or not, so do not need to visit
+	 * hidden references.
+	 */
+	if (allow_hidden_refs(data->allow_uor))
+		for_each_namespaced_ref(NULL, fn, data);
+	else
+		for_each_namespaced_ref(data->hidden_refs.v, fn, data);
+}
+
 static int is_our_ref(struct object *o, enum allow_uor allow_uor)
 {
-	int allow_hidden_ref = (allow_uor &
-				(ALLOW_TIP_SHA1 | ALLOW_REACHABLE_SHA1));
-	return o->flags & ((allow_hidden_ref ? HIDDEN_REF : 0) | OUR_REF);
+	return o->flags & ((allow_hidden_refs(allow_uor) ? HIDDEN_REF : 0) | OUR_REF);
 }
 
 /*
@@ -854,7 +875,7 @@ static void deepen(struct upload_pack_data *data, int depth)
 		 * marked with OUR_REF.
 		 */
 		head_ref_namespaced(check_ref, data);
-		for_each_namespaced_ref(NULL, check_ref, data);
+		for_each_namespaced_ref_1(check_ref, data);
 
 		get_reachable_list(data, &reachable_shallows);
 		result = get_shallow_commits(&reachable_shallows,
@@ -1378,7 +1399,7 @@ void upload_pack(const int advertise_refs, const int stateless_rpc,
 		if (advertise_refs)
 			data.no_done = 1;
 		head_ref_namespaced(send_ref, &data);
-		for_each_namespaced_ref(NULL, send_ref, &data);
+		for_each_namespaced_ref_1(send_ref, &data);
 		/*
 		 * fflush stdout before calling advertise_shallow_grafts because send_ref
 		 * uses stdio.
@@ -1388,7 +1409,7 @@ void upload_pack(const int advertise_refs, const int stateless_rpc,
 		packet_flush(1);
 	} else {
 		head_ref_namespaced(check_ref, &data);
-		for_each_namespaced_ref(NULL, check_ref, &data);
+		for_each_namespaced_ref_1(check_ref, &data);
 	}
 
 	if (!advertise_refs) {
Signature · 1 line
2.40.1.477.g956c797dfc
#37 lore raw

PATCH 15/15 builtin/receive-pack.c: avoid enumerating hidden references

In reply to Taylor Blau

Now that refs_for_each_fullref_in() has the ability to avoid enumerating references matching certain pattern(s), use that to avoid visiting hidden refs when constructing the ref advertisement via receive-pack.

Note that since this exclusion is best-effort, we still need show_ref_cb() to check whether or not each reference is hidden or not before including it in the advertisement.

As was the case when applying this same optimization to upload-pack, receive-pack's reference advertisement phase can proceed much quicker by avoiding enumerating references that will not be part of the advertisement.

(Below, we're still using linux.git with one hidden refs/pull/N ref per commit):

    $ hyperfine -L v ,.compile 'git{v} -c transfer.hideRefs=refs/pull receive-pack --advertise-refs .git'
    Benchmark 1: git -c transfer.hideRefs=refs/pull receive-pack --advertise-refs .git
      Time (mean ± σ):      89.1 ms ±   1.7 ms    [User: 82.0 ms, System: 7.0 ms]
      Range (min … max):    87.7 ms …  95.5 ms    31 runs
    Benchmark 2: git.compile -c transfer.hideRefs=refs/pull receive-pack --advertise-refs .git
      Time (mean ± σ):       4.5 ms ±   0.2 ms    [User: 0.5 ms, System: 3.9 ms]
      Range (min … max):     4.1 ms …   5.6 ms    508 runs

    Summary
      'git.compile -c transfer.hideRefs=refs/pull receive-pack --advertise-refs .git' ran
       20.00 ± 1.05 times faster than 'git -c transfer.hideRefs=refs/pull receive-pack --advertise-refs .git'
---
 builtin/receive-pack.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
builtin/receive-pack.c +2 -1
index 064df74715..b954bcf802 100644
@@ -336,7 +336,8 @@ static void write_head_info(void)
 {
 	static struct oidset seen = OIDSET_INIT;
 
-	for_each_ref(show_ref_cb, &seen);
+	refs_for_each_fullref_in(get_main_ref_store(the_repository), "",
+				 hidden_refs.v, show_ref_cb, &seen);
 	for_each_alternate_ref(show_one_alternate_ref, &seen);
 	oidset_clear(&seen);
 	if (!sent_capabilities)
Signature · 1 line
2.40.1.477.g956c797dfc
#5 lore raw

Re: [PATCH 04/15] ref-filter: add ref_filter_clear()

In reply to Taylor Blau
Quoted 10 lines · Taylor Blau

From: Jeff King <[email protected]>

We did not bother to clean up at all in branch/tag, and for-each-ref only hit a few elements. So this is probably cleaning up leaks, but I didn't check yet.

Note that the reachable_from and unreachable_from lists should be cleaned as they are used. So this is just covering any case where we might bail before running the reachability check.
---

Not signed-off?

ref-filter.c int parse_opt_merge_filter(const struct option *opt, const char *arg, int unset) · line 2882

Quoted 9 lines
ref-filter.c +9 -0
+void ref_filter_clear(struct ref_filter *filter)
+{
+	oid_array_clear(&filter->points_at);
+	free_commit_list(filter->with_commit);
+	free_commit_list(filter->no_commit);
+	free_commit_list(filter->reachable_from);
+	free_commit_list(filter->unreachable_from);
+	ref_filter_init(filter);
+}

And the previous step matters here---otherwise we will end up walking two commit lists whose elements have all been popped in reach_filter().

Makes sense.

ref-filter.c int parse_opt_merge_filter(const struct option *opt, const char *arg, int unset) · line 2872

Quoted 5 lines
ref-filter.c +5 -0
+void ref_filter_init(struct ref_filter *filter)
+{
+	struct ref_filter blank = REF_FILTER_INIT;
+	memcpy(filter, &blank, sizeof(blank));
+}

I wonder if structure assignment "*filter = blank" is easier to see but I think we've seen this "_INIT; memcpy()" dance before.

#6 lore raw

Re: [PATCH 04/15] ref-filter: add ref_filter_clear()

Quoted 8 lines · Junio C Hamano
Quoted 6 lines · Taylor Blau

From: Jeff King <[email protected]>

We did not bother to clean up at all in branch/tag, and for-each-ref only hit a few elements. So this is probably cleaning up leaks, but I didn't check yet.

Quoted 6 lines
Quoted 4 lines

Note that the reachable_from and unreachable_from lists should be cleaned as they are used. So this is just covering any case where we might bail before running the reachability check.
---

Not signed-off?

Oops. Sorry about that, this should be:

Quoted 2 lines

I wonder if structure assignment "*filter = blank" is easier to see but I think we've seen this "_INIT; memcpy()" dance before.

Yeah, this matches how many other _init() functions work (just looking through the output of git grep -B3 memcpy | grep -A3 _init).

Thanks,
Taylor

#10 lore raw

Re: [PATCH 05/15] ref-filter.c: parameterize match functions over patterns

In reply to Taylor Blau
Quoted 12 lines · Taylor Blau
ref-filter.c +3 -2
-static int match_pattern(const struct ref_filter *filter, const char *refname)
+static int match_pattern(const struct ref_filter *filter,
+			 const char **patterns,
+			 const char *refname)
 {
-	const char **patterns = filter->name_patterns;
 	unsigned flags = 0;
 
 	if (filter->ignore_case)
@@ -2132,9 +2133,10 @@ static int match_pattern(const struct ref_filter *filter, const char *refname)
  * matches a pattern "refs/heads/" but not "refs/heads/m") or a
  * wildcard (e.g. the same ref matches "refs/heads/m*", too).

ref-filter.c static int match_pattern(const struct ref_filter *filter, const char *refname) · line 2142

Quoted 10 lines
ref-filter.c +3 -2
  */
-static int match_name_as_path(const struct ref_filter *filter, const char *refname)
+static int match_name_as_path(const struct ref_filter *filter,
+			      const char **pattern,
+			      const char *refname)
 {
-	const char **pattern = filter->name_patterns;
 	int namelen = strlen(refname);
 	unsigned flags = WM_PATHNAME;
 

These hint that we'd eventually lose .name_patterns member from the structure so that we can pass pattern array that is not necessarily tied to any instance of a filter?

ref-filter.c static int filter_pattern_match(struct ref_filter *filter, const char *refname) · line 2169

Quoted 8 lines
ref-filter.c +2 -2
@@ -2163,8 +2165,8 @@ static int filter_pattern_match(struct ref_filter *filter, const char *refname)
 	if (!*filter->name_patterns)
 		return 1; /* No pattern always matches */
 	if (filter->match_as_path)
-		return match_name_as_path(filter, refname);
-	return match_pattern(filter, refname);
+		return match_name_as_path(filter, filter->name_patterns, refname);
+	return match_pattern(filter, filter->name_patterns, refname);

And we are not there yet, so we hoist the use of .name_patterns member one level up to the only caller?

Without seeing how it evolves, we can tell this does not make anything break, but we cannot tell how this helps anything (yet).

Let's read on.

#13 lore raw

Re: [PATCH 06/15] builtin/for-each-ref.c: add `--exclude` option

In reply to Taylor Blau

builtin/for-each-ref.c static char const * const for_each_ref_usage[] = { · line 19

Quoted 11 lines · Taylor Blau
builtin/for-each-ref.c +1 -0
index c01fa6fefe..449da61e11 100644
@@ -14,6 +14,7 @@ static char const * const for_each_ref_usage[] = {
 	N_("git for-each-ref [--points-at <object>]"),
 	N_("git for-each-ref [--merged [<commit>]] [--no-merged [<commit>]]"),
 	N_("git for-each-ref [--contains [<commit>]] [--no-contains [<commit>]]"),
+	N_("git for-each-ref [--exclude=<pattern> ...]"),
 	NULL
 };

I think the original is already wrong, but the easiest thing we can do in order to avoid making it worse is to drop this hunk, as the existing usage is this:

static char const * const for_each_ref_usage[] = {
	N_("git for-each-ref [<options>] [<pattern>]"),
	N_("git for-each-ref [--points-at <object>]"),
	N_("git for-each-ref [--merged [<commit>]] [--no-merged [<commit>]]"),
	N_("git for-each-ref [--contains [<commit>]] [--no-contains [<commit>]]"),
	NULL
};

and this series merely adds a new "--exclude=<pattern>" as one of the "<options>".

As we can see from the fact that for example

 $ git for-each-ref --no-merged next refs/heads/\?\?/\*

works just fine, exactly the same thing can be said about the other --points-at/--merged/--no-merged/--contains/--no-contains options.

The SYNOPSIS section of the manual page is fine.

Quoted 6 lines
ref-filter.c +0 -0
index 6c5eed144f..93dc9b331f 100644
@@ -2169,6 +2169,15 @@ static int filter_pattern_match(struct ref_filter *filter, const char *refname)
 	return match_pattern(filter, filter->name_patterns, refname);

ref-filter.c static int filter_pattern_match(struct ref_filter *filter, const char *refname) · line 2179

Quoted 10 lines
ref-filter.c +8 -0
 }
 
+static int filter_exclude_match(struct ref_filter *filter, const char *refname)
+{
+	if (!filter->exclude.nr)
+		return 0;
+	if (filter->match_as_path)
+		return match_name_as_path(filter, filter->exclude.v, refname);
+	return match_pattern(filter, filter->exclude.v, refname);
+}

Earlier I made a comment about .name_patterns member becoming unnecessary, but I think what should need to happen is instead match_pattern() and match_name_as_path() to lose the "filter" parameter, and take a boolean "ignore_case" instead.

ref-filter.c static int filter_pattern_match(struct ref_filter *filter, const char *refname) · line 2182

Quoted 3 lines
ref-filter.c +1 -0
 struct ref_filter {
 	const char **name_patterns;
+	struct strvec exclude;

At some point after the dust settles, we may want to tweak name_patterns so that these two appear to complement each other more clearly, e.g. use "struct strvec include" to replace "name_patterns" or something. But that is an unrelated tangent.

In any case, there wasn't anything surprising or unexpected in the code. Looking good.

Quoted 32 lines
t/t6300-for-each-ref.sh +24 -0
index 5c00607608..7e8d578522 100755
@@ -447,6 +447,41 @@ test_expect_success 'exercise glob patterns with prefixes' '
 	test_cmp expected actual
 '
 
+cat >expected <<\EOF
+refs/tags/bar
+refs/tags/baz
+refs/tags/testtag
+EOF
+
+test_expect_success 'exercise patterns with prefix exclusions' '
+	for tag in foo/one foo/two foo/three bar baz
+	do
+		git tag "$tag" || return 1
+	done &&
+	test_when_finished "git tag -d foo/one foo/two foo/three bar baz" &&
+	git for-each-ref --format="%(refname)" \
+		refs/tags/ --exclude=refs/tags/foo >actual &&
+	test_cmp expected actual
+'
+
+cat >expected <<\EOF
+refs/tags/bar
+refs/tags/baz
+refs/tags/foo/one
+refs/tags/testtag
+EOF
+

t/t6300-for-each-ref.sh test_expect_success 'exercise glob patterns with prefixes' ' · line 483

Quoted 10 lines
t/t6300-for-each-ref.sh +10 -0
+test_expect_success 'exercise patterns with pattern exclusions' '
+	for tag in foo/one foo/two foo/three bar baz
+	do
+		git tag "$tag" || return 1
+	done &&
+	test_when_finished "git tag -d foo/one foo/two foo/three bar baz" &&
+	git for-each-ref --format="%(refname)" \
+		refs/tags/ --exclude="refs/tags/foo/t*" >actual &&
+	test_cmp expected actual
+'

These are doing as Romans do, so I won't comment on the outdated pattern of preparing the expectation outside the test script. After the dust settles, somebody needs to go in and clean it up.

Thanks.

#19 lore raw

Re: [PATCH 08/15] refs/packed-backend.c: refactor `find_reference_location()`

In reply to Taylor Blau
Quoted 8 lines · Taylor Blau

The function find_reference_location() is used to perform a binary search-like function over the contents of a repository's $GIT_DIR/packed-refs file.

The search it implements is unlike a standard binary search in that the records it searches over are not of a fixed width, so the comparison must locate the end of a record before comparing it.

Quoted 6 lines

Extract the core routine of find_reference_location() in order to implement a function in the following patch which will find the first location in the packed-refs file that *doesn't* match the given pattern.

The behavior of find_reference_location() is unchanged.

The splitting out of this step is rather unfortunate in that the extra parameter "start" given to cmp_record_to_refname(), together with a rather curious returning to "-1" when the parameter is false, are not justified at all by looking at the caller, because the only caller of the function, find_reference_location_1() always passes "start" it got from its caller without modifying, and the sole caller of that passes "1" to "start". IOW, the behaviour of cmp_record_to_refname() when "start" is false can be any arbitrary garbage (it could even be BUG("")) and the claim "the behaviour is unchanged" will still be true.

But that does not help the readers all that much.

So, yes I can agree that this does not introduce any new bug, it is a mysterious no-op, and why we want to pass different values in "start" in future steps in order to achieve what is not explained and leaves the readers frustrated.

I'll stop here for now and come back to the rest laster.

Thanks.

#22 lore raw

Re: [PATCH 09/15] refs/packed-backend.c: implement skip lists to avoid excluded pattern(s)

In reply to Taylor Blau

I haven't read through any of this closely (don't have time for it now) but want to mention one thing here:

Quoted 2 lines · Taylor Blau

- Construct a skip list of regions by combining adjacent and overlapping regions from the previous step.

You might want to add a note to the code that there is no relationship here to the skip list data structure (see
https://en.wikipedia.org/wiki/Skip_list).

Chris

#7 lore raw

Re: [PATCH 04/15] ref-filter: add ref_filter_clear()

In reply to Taylor Blau
Quoted 5 lines · Taylor Blau

From: Jeff King <[email protected]>

We did not bother to clean up at all in branch/tag, and for-each-ref only hit a few elements. So this is probably cleaning up leaks, but I didn't check yet.

Nit: it sounds like there still is the intent to check whether this does fix leaks. How about updating the commit message to either not mention the intent or perform the check?

Patrick

#16 lore raw

Re: [PATCH 07/15] refs: plumb `exclude_patterns` argument throughout

In reply to Taylor Blau
Quoted 7 lines · Taylor Blau

The subsequent patch will want to access an optional excluded_patterns array within refs/packed-backend.c. To do so, the refs subsystem needs to be updated to pass this value across a number of different locations.

Prepare for a future patch by introducing this plumbing now, passing NULLs at top-level APIs in order to make that patch less noisy and more easily readable.

It might be worth mentioning in the commit message that the exclude patterns are supposed to be best-effort. In other words, any caller would still need to do manual filtering if I understand correctly. And while this is indeed documented via refs_for_each_fullref_in_prefixes it is quite easy to miss this important little detail.

Patrick

#24 lore raw

Re: [PATCH 09/15] refs/packed-backend.c: implement skip lists to avoid excluded pattern(s)

In reply to Taylor Blau
Quoted 27 lines · Taylor Blau

When iterating through the packed-refs file in order to answer a query like:

    $ git for-each-ref --exclude=refs/__hidden__

it would be useful to avoid walking over all of the entries in refs/__hidden__/* when possible, since we know that the ref-filter code is going to throw them away anyways.

In certain circumstances, doing so is possible. The algorithm for doing so is as follows:

- For each excluded pattern, find the first record that matches it, and the first pattern that *doesn't* match it (i.e. the location you'd next want to consider when excluding that pattern).

- Sort the patterns by their starting location within the packed-refs file.

- Construct a skip list of regions by combining adjacent and overlapping regions from the previous step.

- When iterating through the packed-refs file, if iter->pos is ever contained in one of the regions from the previous steps, advance iter->pos past the end of that region, and continue enumeration.

Quoted 7 lines

Note that this optimization is only possible when none of the excluded pattern(s) have special meta-characters in them. To see why this is the case, consider the exclusion pattern "refs/foo[a]". In general, in order to find the location of the first record that matches this pattern, we could only consider up to the first meta-character, "refs/foo". But this doesn't work, since the excluded region we'd come up with would include "refs/foobar", even though it is not excluded.

Is this generally true though? A naive implementation would iterate through all references and find the first reference that matches the exclusion regular exepression. From thereon we continue to iterate until we find the first entry that doesn't match. This may cause us to end up with a suboptimal skip list, but the skip list would still be valid.

As I said, this implementation would be naive as we're now forced to iterate through all references starting at the beginning. I assume that your implementation will instead use a binary search to locate the first entry that matches the exclusion pattern and the last pattern. But the way this paragraph is formulated makes it sound like this is a general fact, even though it is a fact that derives from the implementation.

I of course don't propose to change the algorithm here, but instead to clarify where this restriction actually comes from and why the tradeoff makes sense.

[snip]

Quoted 52 lines
refs/packed-backend.c +48 -0
@@ -929,6 +972,108 @@ static struct ref_iterator_vtable packed_ref_iterator_vtable = {
 	.abort = packed_ref_iterator_abort
 };
 
+static int skip_list_entry_cmp(const void *va, const void *vb)
+{
+	const struct skip_list_entry *a = va;
+	const struct skip_list_entry *b = vb;
+
+	if (a->start < b->start)
+		return -1;
+	if (a->start > b->start)
+		return 1;
+	return 0;
+}
+
+static int has_glob_special(const char *str)
+{
+	const char *p;
+	for (p = str; *p; p++) {
+		if (is_glob_special(*p))
+			return 1;
+	}
+	return 0;
+}
+
+static const char *ptr_max(const char *x, const char *y)
+{
+	if (x > y)
+		return x;
+	return y;
+}
+
+static void populate_excluded_skip_list(struct packed_ref_iterator *iter,
+					struct snapshot *snapshot,
+					const char **excluded_patterns)
+{
+	size_t i, j;
+	const char **pattern;
+
+	if (!excluded_patterns)
+		return;
+
+	for (pattern = excluded_patterns; *pattern; pattern++) {
+		struct skip_list_entry *e;
+
+		/*
+		 * We can't feed any excludes with globs in them to the
+		 * refs machinery.  It only understands prefix matching.
+		 * We likewise can't even feed the string leading up to
+		 * the first meta-character, as something like "foo[a]"
+		 * should not exclude "foobar" (but the prefix "foo"

refs/packed-backend.c static struct ref_iterator_vtable packed_ref_iterator_vtable = { · line 1032

Quoted 10 lines
refs/packed-backend.c +10 -0
+		 * would match that and mark it for exclusion).
+		 */
+		if (has_glob_special(*pattern))
+			continue;
+
+		ALLOC_GROW(iter->skip, iter->skip_nr + 1, iter->skip_alloc);
+
+		e = &iter->skip[iter->skip_nr++];
+		e->start = find_reference_location(snapshot, *pattern, 0);
+		e->end = find_reference_location_end(snapshot, *pattern, 0);

One thing that makes this hard to reason about is that we don't explicitly handle the case where the pattern doesn't match at all. So you require a bunch of knowledge about what exactly the functions find_reference_location() and find_reference_location_end() do in that case in order to determine whether we will end up doing the right thing.

Explicitly handling this would give us some benefits:

- It makes the code more obvious.

- We can avoid an extra skip list entry for every non-matching pattern.

- We wouldn't have to perform binary search over the snapshot twice.

Might be I'm missing something though.

Quoted 29 lines
refs/packed-backend.c +29 -0
+	}
+
+	if (!iter->skip_nr) {
+		/*
+		 * Every entry in exclude_patterns has a meta-character,
+		 * nothing to do here.
+		 */
+		return;
+	}
+
+	QSORT(iter->skip, iter->skip_nr, skip_list_entry_cmp);
+
+	/*
+	 * As an optimization, merge adjacent entries in the skip list
+	 * to jump forwards as far as possible when entering a skipped
+	 * region.
+	 *
+	 * For example, if we have two skipped regions:
+	 *
+	 *	[[A, B], [B, C]]
+	 *
+	 * we want to combine that into a single entry jumping from A to
+	 * C.
+	 */
+	for (i = 1, j = 1; i < iter->skip_nr; i++) {
+		struct skip_list_entry *ours = &iter->skip[i];
+		struct skip_list_entry *prev = &iter->skip[i - 1];
+
+		if (ours->start == ours->end) {

refs/packed-backend.c static struct ref_iterator_vtable packed_ref_iterator_vtable = { · line 1071

Quoted 10 lines
refs/packed-backend.c +10 -0
+			/* ignore empty regions (no matching entries) */
+			continue;
+		} else if (prev->end >= ours->start) {
+			/* overlapping regions extend the previous one */
+			prev->end = ptr_max(prev->end, ours->end);
+		} else {
+			/* otherwise, insert a new region */
+			iter->skip[j++] = *ours;
+		}
+	}

Mh. Does this do the right thing in case we have multiple consecutive overlapping skip list entries? We always end up adjusting the immediate predecessor as we use i - 1 to find prev. Shouldn't we instead start with j = 0 and assign prev = &iter->skip[j]?

[snip]

Quoted 39 lines
t/t1419-exclude-refs.sh new file +33 -0
new file mode 100755
index 0000000000..da5265a5a8
@@ -0,0 +1,101 @@
+#!/bin/sh
+
+test_description='test exclude_patterns functionality in main ref store'
+
+GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main
+export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME
+
+TEST_PASSES_SANITIZE_LEAK=true
+. ./test-lib.sh
+
+for_each_ref__exclude () {
+	test-tool ref-store main for-each-ref--exclude "$@" >actual.raw
+	cut -d ' ' -f 2 actual.raw
+}
+
+for_each_ref () {
+	git for-each-ref --format='%(refname)' "$@"
+}
+
+test_expect_success 'setup' '
+	test_commit --no-tag base &&
+	base="$(git rev-parse HEAD)" &&
+
+	for name in foo bar baz quux
+	do
+		for i in 1 2 3
+		do
+			echo "create refs/heads/$name/$i $base" || return 1
+		done || return 1
+	done >in &&
+	echo "delete refs/heads/main" >>in &&
+
+	git update-ref --stdin <in &&

t/t1419-exclude-refs.sh line 43

Quoted 10 lines
t/t1419-exclude-refs.sh new file +10 -0
+	git pack-refs --all
+'
+
+test_expect_success 'for_each_ref__exclude(refs/heads/foo/)' '
+	# region in middle
+	for_each_ref__exclude refs/heads refs/heads/foo >actual &&
+	for_each_ref refs/heads/bar refs/heads/baz refs/heads/quux >expect &&
+
+	test_cmp expect actual
+'

Nit: it might be a bit more readable if we put the comment into the test description instead of having an opaque description that mentions ref names that don't have much of a meaning without reading the test itself.

Patrick

#35 lore raw

Re: [PATCH 14/15] upload-pack.c: avoid enumerating hidden refs where possible

In reply to Taylor Blau

On Mon, May 08, 2023 at 06:00:26PM -0400, Taylor Blau wrote:
[snip]

Quoted 5 lines
upload-pack.c +0 -0
index 7c646ea5bd..0162fffce0 100644

upload-pack.c static int get_common_commits(struct upload_pack_data *data, · line 610

Quoted 10 lines
upload-pack.c +7 -0
@@ -601,11 +601,32 @@ static int get_common_commits(struct upload_pack_data *data,
 	}
 }
 
+static int allow_hidden_refs(enum allow_uor allow_uor)
+{
+	return allow_uor & (ALLOW_TIP_SHA1 | ALLOW_REACHABLE_SHA1);
+}
+
+static void for_each_namespaced_ref_1(each_ref_fn fn,
+				      struct upload_pack_data *data)

I know it's common practice in the Git project, but personally I tend to fight with functions that have a _1 suffix. You simply cannot tell what the difference is to the non-suffixed variant without checking its declaration.

for_each_namespaced_ref_with_optional_hidden_refs() is definitely a mouthful though, and I can't really think of something better either.

upload-pack.c static int get_common_commits(struct upload_pack_data *data, · line 615

Quoted 5 lines
upload-pack.c +5 -0
+{
+	/*
+	 * If `data->allow_uor` allows updating hidden refs, we need to
+	 * mark all references (including hidden ones), to check in
+	 * `is_our_ref()` below.

Doesn't this influence whether somebody can _fetch_ objects pointed to by the hidden refs instead of _updating_ them?

Patrick

#8 lore raw

Re: [PATCH 04/15] ref-filter: add ref_filter_clear()

In reply to Patrick Steinhardt
Quoted 10 lines · Patrick Steinhardt
Quoted 5 lines · Taylor Blau

From: Jeff King <[email protected]>

We did not bother to clean up at all in branch/tag, and for-each-ref only hit a few elements. So this is probably cleaning up leaks, but I didn't check yet.

Nit: it sounds like there still is the intent to check whether this does fix leaks. How about updating the commit message to either not mention the intent or perform the check?

Oops. Thanks for the reminder. Peff sent this patch to me while we were working on this topic together, and I forgot to come back and actually perform this check.

Luckily, it helps out quite a bit:

   t/t0041-usage.sh                          |  1 +
   t/t2016-checkout-patch.sh                 |  1 +
   t/t3402-rebase-merge.sh                   |  1 +
   t/t3407-rebase-abort.sh                   |  1 +
   t/t4058-diff-duplicates.sh                |  2 ++
   t/t5407-post-rewrite-hook.sh              |  1 +
   t/t5811-proto-disable-git.sh              |  2 ++
   t/t6001-rev-list-graft.sh                 |  1 +
   t/t7008-filter-branch-null-sha1.sh        |  1 +
   t/t7408-submodule-reference.sh            |  2 ++
   t/t9502-gitweb-standalone-parse-output.sh |  1 +

11 new leak-free tests! I'll take it ;-).

Thanks,
Taylor

#11 lore raw

Re: [PATCH 05/15] ref-filter.c: parameterize match functions over patterns

In reply to Junio C Hamano
Quoted 9 lines · Junio C Hamano
Quoted 9 lines
ref-filter.c +3 -1
@@ -2132,9 +2133,10 @@ static int match_pattern(const struct ref_filter *filter, const char *refname)
  * matches a pattern "refs/heads/" but not "refs/heads/m") or a
  * wildcard (e.g. the same ref matches "refs/heads/m*", too).
  */
-static int match_name_as_path(const struct ref_filter *filter, const char *refname)
+static int match_name_as_path(const struct ref_filter *filter,
+			      const char **pattern,
+			      const char *refname)
 {
Quoted 8 lines

ref-filter.c static int match_pattern(const struct ref_filter *filter, const char *refname) · line 2142

Quoted 4 lines
ref-filter.c +0 -1
-	const char **pattern = filter->name_patterns;
 	int namelen = strlen(refname);
 	unsigned flags = WM_PATHNAME;
 

These hint that we'd eventually lose .name_patterns member from the structure so that we can pass pattern array that is not necessarily tied to any instance of a filter?

Right. We'll pass in the excluded patterns in the next commit.

Quoted 5 lines

And we are not there yet, so we hoist the use of .name_patterns member one level up to the only caller?

Without seeing how it evolves, we can tell this does not make anything break, but we cannot tell how this helps anything (yet).

Hmm. I tried to allude to this in the patch message with the paragraph:

The subsequent patch will add a new array of patterns to match over (the excluded patterns, via a new git for-each-ref --exclude option), treating the return value of these functions differently depending on which patterns are being used to match.

Tweak match_pattern() and match_name_as_path() to take an array of patterns to prepare for passing either in.

...but I'm happy to add detail or clarify it if you think that these paragraphs could use it.

Thanks,
Taylor

#14 lore raw

Re: [PATCH 06/15] builtin/for-each-ref.c: add `--exclude` option

In reply to Junio C Hamano
Quoted 29 lines · Junio C Hamano

builtin/for-each-ref.c static char const * const for_each_ref_usage[] = { · line 19

Quoted 11 lines · Taylor Blau
builtin/for-each-ref.c +1 -0
index c01fa6fefe..449da61e11 100644
@@ -14,6 +14,7 @@ static char const * const for_each_ref_usage[] = {
 	N_("git for-each-ref [--points-at <object>]"),
 	N_("git for-each-ref [--merged [<commit>]] [--no-merged [<commit>]]"),
 	N_("git for-each-ref [--contains [<commit>]] [--no-contains [<commit>]]"),
+	N_("git for-each-ref [--exclude=<pattern> ...]"),
 	NULL
 };

I think the original is already wrong, but the easiest thing we can do in order to avoid making it worse is to drop this hunk, as the existing usage is this:

static char const * const for_each_ref_usage[] = {
	N_("git for-each-ref [<options>] [<pattern>]"),
	N_("git for-each-ref [--points-at <object>]"),
	N_("git for-each-ref [--merged [<commit>]] [--no-merged [<commit>]]"),
	N_("git for-each-ref [--contains [<commit>]] [--no-contains [<commit>]]"),
	NULL
};

and this series merely adds a new "--exclude=<pattern>" as one of the "<options>".

Quoted 8 lines

As we can see from the fact that for example

 $ git for-each-ref --no-merged next refs/heads/\?\?/\*

works just fine, exactly the same thing can be said about the other --points-at/--merged/--no-merged/--contains/--no-contains options.

The SYNOPSIS section of the manual page is fine.

Good point, will tweak; thanks.

Quoted 4 lines
Quoted 4 lines
builtin/for-each-ref.c +0 -0
@@ -2169,6 +2169,15 @@ static int filter_pattern_match(struct ref_filter *filter, const char *refname)
 	return match_pattern(filter, filter->name_patterns, refname);
 }
 
Quoted 13 lines

builtin/for-each-ref.c static int filter_pattern_match(struct ref_filter *filter, const char *refname) · line 2179

Quoted 8 lines
builtin/for-each-ref.c +8 -0
+static int filter_exclude_match(struct ref_filter *filter, const char *refname)
+{
+	if (!filter->exclude.nr)
+		return 0;
+	if (filter->match_as_path)
+		return match_name_as_path(filter, filter->exclude.v, refname);
+	return match_pattern(filter, filter->exclude.v, refname);
+}

Earlier I made a comment about .name_patterns member becoming unnecessary, but I think what should need to happen is instead match_pattern() and match_name_as_path() to lose the "filter" parameter, and take a boolean "ignore_case" instead.

Agreed.

Quoted 13 lines
Quoted 13 lines
builtin/for-each-ref.c +13 -0
+cat >expected <<\EOF
+refs/tags/bar
+refs/tags/baz
+refs/tags/foo/one
+refs/tags/testtag
+EOF
+
+test_expect_success 'exercise patterns with pattern exclusions' '
+	for tag in foo/one foo/two foo/three bar baz
+	do
+		git tag "$tag" || return 1
+	done &&
+	test_when_finished "git tag -d foo/one foo/two foo/three bar baz" &&
Quoted 8 lines

builtin/for-each-ref.c

Quoted 4 lines
builtin/for-each-ref.c +4 -0
+	git for-each-ref --format="%(refname)" \
+		refs/tags/ --exclude="refs/tags/foo/t*" >actual &&
+	test_cmp expected actual
+'

These are doing as Romans do, so I won't comment on the outdated pattern of preparing the expectation outside the test script. After the dust settles, somebody needs to go in and clean it up.

Yeah, I figured that this series was already getting pretty long, but that it would be expedient to propagate forward this pattern. But it should be cleaned up. Let's tag it with #leftoverbits accordingly.

Thanks,
Taylor

#17 lore raw

Re: [PATCH 07/15] refs: plumb `exclude_patterns` argument throughout

In reply to Patrick Steinhardt
Quoted 9 lines · Patrick Steinhardt
Quoted 7 lines · Taylor Blau

The subsequent patch will want to access an optional excluded_patterns array within refs/packed-backend.c. To do so, the refs subsystem needs to be updated to pass this value across a number of different locations.

Prepare for a future patch by introducing this plumbing now, passing NULLs at top-level APIs in order to make that patch less noisy and more easily readable.

Quoted 5 lines

It might be worth mentioning in the commit message that the exclude patterns are supposed to be best-effort. In other words, any caller would still need to do manual filtering if I understand correctly. And while this is indeed documented via refs_for_each_fullref_in_prefixes it is quite easy to miss this important little detail.

Good suggestion, I updated this patch message accordingly.

Thanks,
Taylor

#20 lore raw

Re: [PATCH 08/15] refs/packed-backend.c: refactor `find_reference_location()`

In reply to Junio C Hamano
Quoted 4 lines · Junio C Hamano

So, yes I can agree that this does not introduce any new bug, it is a mysterious no-op, and why we want to pass different values in "start" in future steps in order to achieve what is not explained and leaves the readers frustrated.

Re-reading this patch again with your review in mind, I agree that the split is poorly placed.

I modified this patch to just extract the core routine behind a helper function and avoided adding the "start" parameter here.

Thanks,
Taylor

#23 lore raw

Re: [PATCH 09/15] refs/packed-backend.c: implement skip lists to avoid excluded pattern(s)

In reply to Chris Torek
Quoted 7 lines · Chris Torek
Quoted 2 lines · Taylor Blau

- Construct a skip list of regions by combining adjacent and overlapping regions from the previous step.

You might want to add a note to the code that there is no relationship here to the skip list data structure (see
https://en.wikipedia.org/wiki/Skip_list).

Good suggestion, thanks. I picked the name skip list for this concept since we're skipping over excluded regions of the packed-refs file, but you're right it has no relation to the skip list data structure.

Will note in the patch message.

Thanks,
Taylor

#25 lore raw

Re: [PATCH 09/15] refs/packed-backend.c: implement skip lists to avoid excluded pattern(s)

In reply to Patrick Steinhardt
Quoted 23 lines · Patrick Steinhardt
Quoted 7 lines · Taylor Blau

Note that this optimization is only possible when none of the excluded pattern(s) have special meta-characters in them. To see why this is the case, consider the exclusion pattern "refs/foo[a]". In general, in order to find the location of the first record that matches this pattern, we could only consider up to the first meta-character, "refs/foo". But this doesn't work, since the excluded region we'd come up with would include "refs/foobar", even though it is not excluded.

Is this generally true though? A naive implementation would iterate through all references and find the first reference that matches the exclusion regular exepression. From thereon we continue to iterate until we find the first entry that doesn't match. This may cause us to end up with a suboptimal skip list, but the skip list would still be valid.

As I said, this implementation would be naive as we're now forced to iterate through all references starting at the beginning. I assume that your implementation will instead use a binary search to locate the first entry that matches the exclusion pattern and the last pattern. But the way this paragraph is formulated makes it sound like this is a general fact, even though it is a fact that derives from the implementation.

Quoted 3 lines

I of course don't propose to change the algorithm here, but instead to clarify where this restriction actually comes from and why the tradeoff makes sense.

In the example you include, it's possible. But consider something like:

    $ git for-each-ref --exclude='refs/foo[ac]'

The region that matches that expression ("refs/fooa", "refs/fooc" and everything underneath them) does not have to appear as a continuous single region in the packed-refs file. If you have, say, "refs/foobar", that will appear between the two regions you want to exclude.

So I think you *might* be able to do it in general, but at the very least it would involve splitting each character class and finding the start and end of any region(s) that it matches.

Even so, you'd have to try and match each entry as you determine the width of the excluded region, at which point you're at par with enumerating them anyway and having the caller discard any entries it doesn't want.

Quoted 29 lines
Quoted 9 lines
(quoted diff) +9 -0
+	for (pattern = excluded_patterns; *pattern; pattern++) {
+		struct skip_list_entry *e;
+
+		/*
+		 * We can't feed any excludes with globs in them to the
+		 * refs machinery.  It only understands prefix matching.
+		 * We likewise can't even feed the string leading up to
+		 * the first meta-character, as something like "foo[a]"
+		 * should not exclude "foobar" (but the prefix "foo"

Review

Quoted 10 lines
(quoted diff) +10 -0
+		 * would match that and mark it for exclusion).
+		 */
+		if (has_glob_special(*pattern))
+			continue;
+
+		ALLOC_GROW(iter->skip, iter->skip_nr + 1, iter->skip_alloc);
+
+		e = &iter->skip[iter->skip_nr++];
+		e->start = find_reference_location(snapshot, *pattern, 0);
+		e->end = find_reference_location_end(snapshot, *pattern, 0);

One thing that makes this hard to reason about is that we don't explicitly handle the case where the pattern doesn't match at all. So you require a bunch of knowledge about what exactly the functions find_reference_location() and find_reference_location_end() do in that case in order to determine whether we will end up doing the right thing.

Explicitly handling this would give us some benefits:

Quoted 8 lines

- It makes the code more obvious.

- We can avoid an extra skip list entry for every non-matching pattern.

- We wouldn't have to perform binary search over the snapshot twice.

Might be I'm missing something though.

We handle it implicitly via the mustexist parameter that both find_reference_location() and find_reference_location_end() take, which returns the position that you would find a matching entry if one does not already exist.

You're right that it would save you a second binary search, but that is likely to be a vanishingly small cost compared to the actual traversal, disk I/O, etc.

I could imagine modifying the signature of both of those functions to be something like:

int find_reference_location(struct snapshot *, const char *pattern,
int mustexist, char *out)

Which would propagate the result through out, and the return value would be whether or not it found a matching entry. But that only really makes sense when mustexist is set to 0, and it adds some verbosity throughout.

I think that it's totally possible to avoid the second search, but I'm not sure that the cost of additional complexity and verbosity is worth the benefit of avoiding one binary search (which will likely be resident, anyway).

Note though that if e->start == e->end here we will discard the empty skip region below.

Quoted 12 lines
Quoted 12 lines
(quoted diff) +12 -0
+	for (i = 1, j = 1; i < iter->skip_nr; i++) {
+		struct skip_list_entry *ours = &iter->skip[i];
+		struct skip_list_entry *prev = &iter->skip[i - 1];
+
+		if (ours->start == ours->end) {
+			/* ignore empty regions (no matching entries) */
+			continue;
+		} else if (prev->end >= ours->start) {
+			/* overlapping regions extend the previous one */
+			prev->end = ptr_max(prev->end, ours->end);
+		} else {
+			/* otherwise, insert a new region */
Quoted 8 lines

Review

Quoted 3 lines
(quoted diff) +3 -0
+			iter->skip[j++] = *ours;
+		}
+	}

Mh. Does this do the right thing in case we have multiple consecutive overlapping skip list entries? We always end up adjusting the immediate predecessor as we use i - 1 to find prev. Shouldn't we instead start with j = 0 and assign prev = &iter->skip[j]?

Good catch. I think applying this on top should do the trick:

--- 8< ---

refs/packed-backend.c +3 -4
index 137a4233f6..3b1337267a 100644
@@ -1054,9 +1054,9 @@ static void populate_excluded_skip_list(struct packed_ref_iterator *iter,
 	 * we want to combine that into a single entry jumping from A to
 	 * C.
 	 */
-	for (i = 1, j = 1; i < iter->skip_nr; i++) {
+	for (i = 0, j = 0; i < iter->skip_nr; i++) {
 		struct skip_list_entry *ours = &iter->skip[i];
-		struct skip_list_entry *prev = &iter->skip[i - 1];
+		struct skip_list_entry *prev = &iter->skip[j];
 
 		if (ours->start == ours->end) {
 			/* ignore empty regions (no matching entries) */
@@ -1066,7 +1066,7 @@ static void populate_excluded_skip_list(struct packed_ref_iterator *iter,
 			prev->end = ptr_max(prev->end, ours->end);
 		} else {
 			/* otherwise, insert a new region */
-			iter->skip[j++] = *ours;
+			iter->skip[++j] = *ours;
 		}
 	}
--- >8 ---
Quoted 11 lines

Review

Quoted 7 lines
(quoted diff) +7 -0
+test_expect_success 'for_each_ref__exclude(refs/heads/foo/)' '
+	# region in middle
+	for_each_ref__exclude refs/heads refs/heads/foo >actual &&
+	for_each_ref refs/heads/bar refs/heads/baz refs/heads/quux >expect &&
+
+	test_cmp expect actual
+'

Nit: it might be a bit more readable if we put the comment into the test description instead of having an opaque description that mentions ref names that don't have much of a meaning without reading the test itself.

Fair enough ;-).

Thanks,
Taylor

#26 lore raw

Re: [PATCH 09/15] refs/packed-backend.c: implement skip lists to avoid excluded pattern(s)

Quoted 20 lines · Taylor Blau

Good catch. I think applying this on top should do the trick:

--- 8< ---

refs/packed-backend.c +2 -2
index 137a4233f6..3b1337267a 100644
@@ -1054,9 +1054,9 @@ static void populate_excluded_skip_list(struct packed_ref_iterator *iter,
 	 * we want to combine that into a single entry jumping from A to
 	 * C.
 	 */
-	for (i = 1, j = 1; i < iter->skip_nr; i++) {
+	for (i = 0, j = 0; i < iter->skip_nr; i++) {
 		struct skip_list_entry *ours = &iter->skip[i];
-		struct skip_list_entry *prev = &iter->skip[i - 1];
+		struct skip_list_entry *prev = &iter->skip[j];
 
 		if (ours->start == ours->end) {
 			/* ignore empty regions (no matching entries) */

refs/packed-backend.c static void populate_excluded_skip_list(struct packed_ref_iterator *iter, · line 1072

Quoted 8 lines
refs/packed-backend.c +1 -2
@@ -1066,7 +1066,7 @@ static void populate_excluded_skip_list(struct packed_ref_iterator *iter,
 			prev->end = ptr_max(prev->end, ours->end);
 		} else {
 			/* otherwise, insert a new region */
-			iter->skip[j++] = *ours;
+			iter->skip[++j] = *ours;
 		}
 	}
--- >8 ---

Oops, this is wrong. It should be something like this instead:

--- 8< ---

refs/packed-backend.c +7 -4
index 137a4233f6..574f32d67f 100644
@@ -1007,6 +1007,7 @@ static void populate_excluded_skip_list(struct packed_ref_iterator *iter,
 {
 	size_t i, j;
 	const char **pattern;
+	struct skip_list_entry *last_disjoint;
 
 	if (!excluded_patterns)
 		return;
@@ -1054,19 +1055,22 @@ static void populate_excluded_skip_list(struct packed_ref_iterator *iter,
 	 * we want to combine that into a single entry jumping from A to
 	 * C.
 	 */
+	last_disjoint = iter->skip;
+
 	for (i = 1, j = 1; i < iter->skip_nr; i++) {
 		struct skip_list_entry *ours = &iter->skip[i];
-		struct skip_list_entry *prev = &iter->skip[i - 1];
 
 		if (ours->start == ours->end) {
 			/* ignore empty regions (no matching entries) */
 			continue;
-		} else if (prev->end >= ours->start) {
+		} else if (ours->start <= last_disjoint->end) {
 			/* overlapping regions extend the previous one */
-			prev->end = ptr_max(prev->end, ours->end);
+			last_disjoint->end = ptr_max(last_disjoint->end, ours->end);
 		} else {
 			/* otherwise, insert a new region */
 			iter->skip[j++] = *ours;
+			last_disjoint = ours;
+
 		}
 	}
--- >8 ---

Thanks,
Taylor

#36 lore raw

Re: [PATCH 14/15] upload-pack.c: avoid enumerating hidden refs where possible

In reply to Patrick Steinhardt
Quoted 18 lines · Patrick Steinhardt

upload-pack.c static int get_common_commits(struct upload_pack_data *data, · line 610

Quoted 11 lines · Taylor Blau
upload-pack.c +7 -0
@@ -601,11 +601,32 @@ static int get_common_commits(struct upload_pack_data *data,
 	}
 }
 
+static int allow_hidden_refs(enum allow_uor allow_uor)
+{
+	return allow_uor & (ALLOW_TIP_SHA1 | ALLOW_REACHABLE_SHA1);
+}
+
+static void for_each_namespaced_ref_1(each_ref_fn fn,
+				      struct upload_pack_data *data)

I know it's common practice in the Git project, but personally I tend to fight with functions that have a _1 suffix. You simply cannot tell what the difference is to the non-suffixed variant without checking its declaration.

Quoted 2 lines

for_each_namespaced_ref_with_optional_hidden_refs() is definitely a mouthful though, and I can't really think of something better either.

Yeah, I know. It's not my favorite convention, either, but I also couldn't come up with anything shorter ;-).

Quoted 8 lines

upload-pack.c static int get_common_commits(struct upload_pack_data *data, · line 615

Quoted 5 lines
upload-pack.c +5 -0
+{
+	/*
+	 * If `data->allow_uor` allows updating hidden refs, we need to
+	 * mark all references (including hidden ones), to check in
+	 * `is_our_ref()` below.

Doesn't this influence whether somebody can _fetch_ objects pointed to by the hidden refs instead of _updating_ them?

Oops, yes. Thanks for catching my obvious typo.

Thanks,
Taylor

#28 lore raw

Re: [PATCH 09/15] refs/packed-backend.c: implement skip lists to avoid excluded pattern(s)

In reply to Taylor Blau
Quoted 9 lines · Taylor Blau

When iterating through the packed-refs file in order to answer a query like:

    $ git for-each-ref --exclude=refs/__hidden__

it would be useful to avoid walking over all of the entries in refs/__hidden__/* when possible, since we know that the ref-filter code is going to throw them away anyways.

Quoted 6 lines

In certain circumstances, doing so is possible. The algorithm for doing so is as follows:

- For each excluded pattern, find the first record that matches it, and the first pattern that *doesn't* match it (i.e. the location you'd next want to consider when excluding that pattern).

Do we find "record" and then "pattern", or is the latter a misspelt "record"? I will assume it is the latter while reading the rest. Is the latter "the record that does not match the pattern, whose record number is the smallest but yet larger than the first record that matches the pattern?" That is we assume that the refs in the packed refs file are sorted and can be partitioned into three by each pattern: (1) refs before the first matching record---they do not match the pattern. (2) refs at or after the first matching record that match. (3) refs after (2) that do not match.

I am not sure if that would work. What if the pattern were refs/tags/[ad-z]* and there are packed tags refs/tags/{a,b,c,d,e}? The pattern would partition the refs into [a](matches), [b,c](does not match), [d,e](matches).

Perhaps I am grossly misunderstanding what the above explanation says.

Quoted 2 lines

- Sort the patterns by their starting location within the packed-refs file.

So the idea is that the patterns are sorted by the first record they match, and after sorting these patterns, refs that are between the beginning of the list of refs and the first record associated with the first pattern will not match _any_ pattern in that list?

Quoted 2 lines

- Construct a skip list of regions by combining adjacent and overlapping regions from the previous step.

"skip list of regions" -> "list of regions to skip", I guess.

Quoted 7 lines

- When iterating through the packed-refs file, if iter->pos is ever contained in one of the regions from the previous steps, advance iter->pos past the end of that region, and continue enumeration.

Note that this optimization is only possible when none of the excluded pattern(s) have special meta-characters in them.

Ah, so this is called "patterns" but it deals with literal patterns without globs? Then I'll stop worrying about the refs/tags/[ad-z] counter-example.

Quoted 6 lines

To see why this is the
case, consider the exclusion pattern "refs/foo[a]". In general, in order to find the location of the first record that matches this pattern, we could only consider up to the first meta-character, "refs/foo". But this doesn't work, since the excluded region we'd come up with would include "refs/foobar", even though it is not excluded.

OK.

Quoted 5 lines

Using the skip list is fairly straightforward (see the changes to refs/packed-backend.c::next_record()), but constructing the list is not. To ensure that the construction is correct, add a new suite of tests in t1419 covering various corner cases (overlapping regions, partially overlapping regions, adjacent regions, etc.).

Sounds good. Does this actually use the skip list data structure, or do they happen to share the same two words in their names, but otherwise have nothing common with each other? If the latter, we may want to revise the explanation, data type names, and variable names, to avoid confusion, as Chris pointed out earlier.

Quoted 13 lines
refs/packed-backend.c +13 -0
+static void populate_excluded_skip_list(struct packed_ref_iterator *iter,
+					struct snapshot *snapshot,
+					const char **excluded_patterns)
+{
+	size_t i, j;
+	const char **pattern;
+
+	if (!excluded_patterns)
+		return;
+
+	for (pattern = excluded_patterns; *pattern; pattern++) {
+		struct skip_list_entry *e;
+

refs/packed-backend.c static struct ref_iterator_vtable packed_ref_iterator_vtable = { · line 1026

Quoted 10 lines
refs/packed-backend.c +10 -0
+		/*
+		 * We can't feed any excludes with globs in them to the
+		 * refs machinery.  It only understands prefix matching.
+		 * We likewise can't even feed the string leading up to
+		 * the first meta-character, as something like "foo[a]"
+		 * should not exclude "foobar" (but the prefix "foo"
+		 * would match that and mark it for exclusion).
+		 */
+		if (has_glob_special(*pattern))
+			continue;

Hmph. I would have expected that a set of patterns with any one pattern with glob would invalidate the whole skip optimization, but it is nice if we can salvage such a set and still optimize, if only for literal patterns. Interesting.

Ah, that is because we are dealing with ranges that cannot possibly match. With the mention of "first record that matches" etc. in the earlier descriptoin, I somehow misled myself that we are dealing with ranges that have interesting records. So, a pattern with glob does not contribute any range to be skipped, but that is OK.

refs/packed-backend.c static struct ref_iterator_vtable packed_ref_iterator_vtable = { · line 1032

Quoted 5 lines
refs/packed-backend.c +5 -0
+		ALLOC_GROW(iter->skip, iter->skip_nr + 1, iter->skip_alloc);
+
+		e = &iter->skip[iter->skip_nr++];
+		e->start = find_reference_location(snapshot, *pattern, 0);
+		e->end = find_reference_location_end(snapshot, *pattern, 0);

So, iter->skip[] array has one range per pattern at most, but some patterns may not contribute any range to the list.

Quoted 10 lines
refs/packed-backend.c +10 -0
+	}
+
+	if (!iter->skip_nr) {
+		/*
+		 * Every entry in exclude_patterns has a meta-character,
+		 * nothing to do here.
+		 */
+		return;
+	}
+

refs/packed-backend.c static struct ref_iterator_vtable packed_ref_iterator_vtable = { · line 1052

Quoted 10 lines
refs/packed-backend.c +10 -0
+	QSORT(iter->skip, iter->skip_nr, skip_list_entry_cmp);
+
+	/*
+	 * As an optimization, merge adjacent entries in the skip list
+	 * to jump forwards as far as possible when entering a skipped
+	 * region.
+	 *
+	 * For example, if we have two skipped regions:
+	 *
+	 *	[[A, B], [B, C]]

I am confused. The first pattern may never match records in [A..B] range, and the second pattern may never match records in [B..C] range, but what does it mean to combine these two ranges?

Quoted 8 lines
refs/packed-backend.c +6 -0
+	 * we want to combine that into a single entry jumping from A to
+	 * C.
+	 */
+	for (i = 1, j = 1; i < iter->skip_nr; i++) {
+		struct skip_list_entry *ours = &iter->skip[i];
+		struct skip_list_entry *prev = &iter->skip[i - 1];
 
 

refs/packed-backend.c static struct ref_iterator_vtable packed_ref_iterator_vtable = { · line 1070

Quoted 10 lines
refs/packed-backend.c +10 -0
+		if (ours->start == ours->end) {
+			/* ignore empty regions (no matching entries) */
+			continue;
+		} else if (prev->end >= ours->start) {
+			/* overlapping regions extend the previous one */
+			prev->end = ptr_max(prev->end, ours->end);
+		} else {
+			/* otherwise, insert a new region */
+			iter->skip[j++] = *ours;
+		}

None of the {braces} seem needed, but OK.

Quoted 5 lines
refs/packed-backend.c +5 -0
+	}
+
+	iter->skip_nr = j;
+	iter->skip_pos = 0;
+}
#29 lore raw

Re: [PATCH 09/15] refs/packed-backend.c: implement skip lists to avoid excluded pattern(s)

Quoted 28 lines · Junio C Hamano
Quoted 9 lines · Taylor Blau

When iterating through the packed-refs file in order to answer a query like:

    $ git for-each-ref --exclude=refs/__hidden__

it would be useful to avoid walking over all of the entries in refs/__hidden__/* when possible, since we know that the ref-filter code is going to throw them away anyways.

Quoted 6 lines

In certain circumstances, doing so is possible. The algorithm for doing so is as follows:

- For each excluded pattern, find the first record that matches it, and the first pattern that *doesn't* match it (i.e. the location you'd next want to consider when excluding that pattern).

Do we find "record" and then "pattern", or is the latter a misspelt "record"? I will assume it is the latter while reading the rest. Is the latter "the record that does not match the pattern, whose record number is the smallest but yet larger than the first record that matches the pattern?" That is we assume that the refs in the packed refs file are sorted and can be partitioned into three by each pattern: (1) refs before the first matching record---they do not match the pattern. (2) refs at or after the first matching record that match. (3) refs after (2) that do not match.

Quoted 6 lines

I am not sure if that would work. What if the pattern were refs/tags/[ad-z]* and there are packed tags refs/tags/{a,b,c,d,e}? The pattern would partition the refs into [a](matches), [b,c](does not match), [d,e](matches).

Perhaps I am grossly misunderstanding what the above explanation says.

Sorry, you're right to assume the latter. This should read "..., find the record that matches it, and the first record that *doesn't* match."

In the example you give, we would ignore that whole pattern and enumerate everything (including entries that match 'refs/tags/[ad-z]*'), and the caller is expected to discard them.

Quoted 7 lines
Quoted 2 lines

- Sort the patterns by their starting location within the packed-refs file.

So the idea is that the patterns are sorted by the first record they match, and after sorting these patterns, refs that are between the beginning of the list of refs and the first record associated with the first pattern will not match _any_ pattern in that list?

My patch could use some clarification here, since it is much easier to treat the patterns as unsorted, and then sort the beginning of the range that they match.

Quoted 4 lines
Quoted 2 lines

- Construct a skip list of regions by combining adjacent and overlapping regions from the previous step.

"skip list of regions" -> "list of regions to skip", I guess.

Thanks, will update.

Quoted 11 lines
Quoted 5 lines

Using the skip list is fairly straightforward (see the changes to refs/packed-backend.c::next_record()), but constructing the list is not. To ensure that the construction is correct, add a new suite of tests in t1419 covering various corner cases (overlapping regions, partially overlapping regions, adjacent regions, etc.).

Sounds good. Does this actually use the skip list data structure, or do they happen to share the same two words in their names, but otherwise have nothing common with each other? If the latter, we may want to revise the explanation, data type names, and variable names, to avoid confusion, as Chris pointed out earlier.

They have nothing to do with each other ;-). I made a note in this patch in the revised version I'll send in the next day or two to note the distinction. But I'm fine with renaming the whole concept to "jump list" or something like that if you prefer.

Quoted 19 lines

refs/packed-backend.c static struct ref_iterator_vtable packed_ref_iterator_vtable = { · line 1026

Quoted 13 lines
refs/packed-backend.c +13 -0
+	for (pattern = excluded_patterns; *pattern; pattern++) {
+		struct skip_list_entry *e;
+
+		/*
+		 * We can't feed any excludes with globs in them to the
+		 * refs machinery.  It only understands prefix matching.
+		 * We likewise can't even feed the string leading up to
+		 * the first meta-character, as something like "foo[a]"
+		 * should not exclude "foobar" (but the prefix "foo"
+		 * would match that and mark it for exclusion).
+		 */
+		if (has_glob_special(*pattern))
+			continue;

Hmph. I would have expected that a set of patterns with any one pattern with glob would invalidate the whole skip optimization, but it is nice if we can salvage such a set and still optimize, if only for literal patterns. Interesting.

Quoted 5 lines

Ah, that is because we are dealing with ranges that cannot possibly match. With the mention of "first record that matches" etc. in the earlier descriptoin, I somehow misled myself that we are dealing with ranges that have interesting records. So, a pattern with glob does not contribute any range to be skipped, but that is OK.

Exactly right. This is all "best effort" anyway, since there are some patterns that we cannot construct a skip list entry for in the general case. So it's fine if we enumerate some references that match one or more of the excluded patterns, because the caller is expected to drop those results themselves.

Quoted 4 lines
Quoted 4 lines
refs/packed-backend.c +4 -0
+	/*
+	 * As an optimization, merge adjacent entries in the skip list
+	 * to jump forwards as far as possible when entering a skipped
+	 * region.
Quoted 8 lines

refs/packed-backend.c static struct ref_iterator_vtable packed_ref_iterator_vtable = { · line 1052

Quoted 4 lines
refs/packed-backend.c +4 -0
+	 *
+	 * For example, if we have two skipped regions:
+	 *
+	 *	[[A, B], [B, C]]

I am confused. The first pattern may never match records in [A..B] range, and the second pattern may never match records in [B..C] range, but what does it mean to combine these two ranges?

The patterns would match all records in their respective regions, but since they are excluded we want to jump over those references instead of iterating (and then discarding them later).

So if we have a jump from A->B, and another from B->C, it would be fine to perform two jumps to get from A to C. But we can detect this case by combining adjacent/overlapping regions.

Quoted 4 lines
Quoted 4 lines
refs/packed-backend.c +4 -0
+		if (ours->start == ours->end) {
+			/* ignore empty regions (no matching entries) */
+			continue;
+		} else if (prev->end >= ours->start) {
Quoted 8 lines

refs/packed-backend.c static struct ref_iterator_vtable packed_ref_iterator_vtable = { · line 1070

Quoted 6 lines
refs/packed-backend.c +6 -0
+			/* overlapping regions extend the previous one */
+			prev->end = ptr_max(prev->end, ours->end);
+		} else {
+			/* otherwise, insert a new region */
+			iter->skip[j++] = *ours;
+		}

None of the {braces} seem needed, but OK.

Yeah. I added the braces here intentionally to match the CodingGuidelines, which state that this case is an exception.

Thanks,
Taylor

#27 lore raw

Re: [PATCH 09/15] refs/packed-backend.c: implement skip lists to avoid excluded pattern(s)

In reply to Taylor Blau
Quoted 37 lines · Taylor Blau
Quoted 23 lines · Patrick Steinhardt
Quoted 7 lines · Taylor Blau

Note that this optimization is only possible when none of the excluded pattern(s) have special meta-characters in them. To see why this is the case, consider the exclusion pattern "refs/foo[a]". In general, in order to find the location of the first record that matches this pattern, we could only consider up to the first meta-character, "refs/foo". But this doesn't work, since the excluded region we'd come up with would include "refs/foobar", even though it is not excluded.

Is this generally true though? A naive implementation would iterate through all references and find the first reference that matches the exclusion regular exepression. From thereon we continue to iterate until we find the first entry that doesn't match. This may cause us to end up with a suboptimal skip list, but the skip list would still be valid.

As I said, this implementation would be naive as we're now forced to iterate through all references starting at the beginning. I assume that your implementation will instead use a binary search to locate the first entry that matches the exclusion pattern and the last pattern. But the way this paragraph is formulated makes it sound like this is a general fact, even though it is a fact that derives from the implementation.

Quoted 3 lines

I of course don't propose to change the algorithm here, but instead to clarify where this restriction actually comes from and why the tradeoff makes sense.

In the example you include, it's possible. But consider something like:

    $ git for-each-ref --exclude='refs/foo[ac]'

The region that matches that expression ("refs/fooa", "refs/fooc" and everything underneath them) does not have to appear as a continuous single region in the packed-refs file. If you have, say, "refs/foobar", that will appear between the two regions you want to exclude.

Quoted 8 lines

So I think you *might* be able to do it in general, but at the very least it would involve splitting each character class and finding the start and end of any region(s) that it matches.

Even so, you'd have to try and match each entry as you determine the width of the excluded region, at which point you're at par with enumerating them anyway and having the caller discard any entries it doesn't want.

Alternatively you could also do this on a best-effort basis and only find the first matching region. But anyway, as said: I'm fine with the limitations but think that we should document better where they come from. The current commit message sounds like the limitation is of general nature even though it is in fact a conciously-chosen tradeoff that allows us to make the implementation more efficient for most cases.

Patrick

#38 lore raw

v2 cover refs: implement jump lists for packed backend

In reply to Taylor Blau

Here is a reroll of my series to implement jump (née skip) lists for the packed refs backend.

Not a ton has changed since last time, but some notable things that have changed include:

- Renaming "skip lists" to "jump lists" to clarify that this implementation does not use the skip list data structure.
- Patch reorganization, splitting out find_reference_location_end() more sensibly, rewording patch messages, etc.
- Addresses feedback from Junio and Patrick Steinhardt's helpful reviews.

As usual, a range-diff is included below for convenience.

Given that we are expecting -rc0 today, we should aim to not let review of this topic direct our attention away from testing the release candidates. We can get more serious about it on the other side of 2.41.

Thanks in advance for another look.

Jeff King (5):
refs.c: rename ref_filter
ref-filter.h: provide REF_FILTER_INIT
ref-filter: clear reachable list pointers after freeing
ref-filter: add ref_filter_clear() ref-filter.c: parameterize match functions over patterns

Taylor Blau (11):
builtin/for-each-ref.c: add --exclude option
refs: plumb exclude_patterns argument throughout refs/packed-backend.c: refactor find_reference_location() refs/packed-backend.c: implement jump lists to avoid excluded
pattern(s)
refs/packed-backend.c: add trace2 counters for jump list revision.h: store hidden refs in a strvec refs/packed-backend.c: ignore complicated hidden refs rules refs.h: let for_each_namespaced_ref() take excluded patterns builtin/receive-pack.c: avoid enumerating hidden references upload-pack.c: avoid enumerating hidden refs where possible ls-refs.c: avoid enumerating hidden refs where possible

 Documentation/git-for-each-ref.txt |   6 +
 builtin/branch.c                   |   4 +-
 builtin/for-each-ref.c             |   7 +-
 builtin/receive-pack.c             |   7 +-
 builtin/tag.c                      |   4 +-
 http-backend.c                     |   2 +-
 ls-refs.c                          |   8 +-
 ref-filter.c                       |  63 ++++++--
 ref-filter.h                       |  12 ++
 refs.c                             |  61 ++++----
 refs.h                             |  15 +-
 refs/debug.c                       |   5 +-
 refs/files-backend.c               |   5 +-
 refs/packed-backend.c              | 226 ++++++++++++++++++++++++++---
 refs/refs-internal.h               |   7 +-
 revision.c                         |   4 +-
 revision.h                         |   5 +-
 t/helper/test-reach.c              |   2 +-
 t/helper/test-ref-store.c          |  10 ++
 t/t0041-usage.sh                   |   1 +
 t/t1419-exclude-refs.sh            | 131 +++++++++++++++++
 t/t3402-rebase-merge.sh            |   1 +
 t/t6300-for-each-ref.sh            |  35 +++++
 trace2.h                           |   2 +
 trace2/tr2_ctr.c                   |   5 +
 upload-pack.c                      |  43 ++++--
 26 files changed, 565 insertions(+), 106 deletions(-)
 create mode 100755 t/t1419-exclude-refs.sh

Range-diff against v1:

 1:  2225f79941 =  1:  6cac42e70e refs.c: rename `ref_filter`
 2:  ea1c7834db =  2:  8dda7db738 ref-filter.h: provide `REF_FILTER_INIT`
 3:  7fe8623f60 =  3:  bf21df783d ref-filter: clear reachable list pointers after freeing
 4:  c804ba3620 !  4:  85ecb70957 ref-filter: add ref_filter_clear()

@@ Metadata
Author: Jeff King <[email protected]>

## Commit message ##
- ref-filter: add ref_filter_clear()
+ ref-filter: add ref_filter_clear()

- We did not bother to clean up at all in branch/tag, and for-each-ref
- only hit a few elements. So this is probably cleaning up leaks, but I
- didn't check yet.
+ We did not bother to clean up at all in git branch or git tag, and
+ git for-each-ref only cleans up a couple of members.

- Note that the reachable_from and unreachable_from lists should be
+ Add and call ref_filter_clear() when cleaning up a `struct
+ ref_filter`. Running this patch (without any test changes) indicates a
+ couple of now leak-free tests. This was found by running:
+
+ $ make SANITIZE=leak
+ $ make -C t GIT_TEST_PASSING_SANITIZE_LEAK=check GIT_TEST_OPTS=--immediate
+
+ (Note that the reachable_from and unreachable_from lists should be
cleaned as they are used. So this is just covering any case where we
- might bail before running the reachability check.
+ might bail before running the reachability check.)
+
+ Signed-off-by: Jeff King <[email protected]>
+ Signed-off-by: Taylor Blau <[email protected]>

## builtin/branch.c ##
@@ builtin/branch.c: int cmd_branch(int argc, const char **argv, const char *prefix)
@@ ref-filter.h: void filter_ahead_behind(struct repository *r,
+void ref_filter_clear(struct ref_filter *filter);
+
#endif /* REF_FILTER_H */
+
+ ## t/t0041-usage.sh ##
+@@ t/t0041-usage.sh: test_description='Test commands behavior when given invalid argument value'
+ GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main
+ export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME
+
++TEST_PASSES_SANITIZE_LEAK=true
+ . ./test-lib.sh
+
+ test_expect_success 'setup ' '
+
+ ## t/t3402-rebase-merge.sh ##
+@@ t/t3402-rebase-merge.sh: test_description='git rebase --merge test'
+ GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main
+ export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME
+
++TEST_PASSES_SANITIZE_LEAK=true
+ . ./test-lib.sh
+
+ T="A quick brown fox
5: c54000f5f5 ! 5: 385890b459 ref-filter.c: parameterize match functions over patterns
@@ Commit message
Tweak match_pattern() and match_name_as_path() to take an array of patterns to prepare for passing either in.

+ Once we start passing either in, match_pattern() will have little to
+ do with a particular struct ref_filter * instance. To clarify this,
+ drop it from the argument list, and replace it with the only bit of the
+ ref_filter that we care about (filter->ignore_case).
+

@@ ref-filter.c: static int get_ref_atom_value(struct ref_array_item *ref, int atom
* matches "refs/heads/mas*", too).
*/
-static int match_pattern(const struct ref_filter *filter, const char *refname)
-+static int match_pattern(const struct ref_filter *filter,
-+ const char **patterns,
-+ const char *refname)
++static int match_pattern(const char **patterns, const char *refname,
++ const int ignore_case)
{
- const char **patterns = filter->name_patterns;
unsigned flags = 0;

- if (filter->ignore_case)
+- if (filter->ignore_case)
++ if (ignore_case)
+ flags |= WM_CASEFOLD;
+
+ /*
@@ ref-filter.c: static int match_pattern(const struct ref_filter *filter, const char *refname)
* matches a pattern "refs/heads/" but not "refs/heads/m") or a
* wildcard (e.g. the same ref matches "refs/heads/m*", too).
@@ ref-filter.c: static int filter_pattern_match(struct ref_filter *filter, const c
- return match_name_as_path(filter, refname);
- return match_pattern(filter, refname);
+ return match_name_as_path(filter, filter->name_patterns, refname);
-+ return match_pattern(filter, filter->name_patterns, refname);
++ return match_pattern(filter->name_patterns, refname,
++ filter->ignore_case);
}

/*
6: ea5c0ddc10 ! 6: 1a3371a0a7 builtin/for-each-ref.c: add --exclude option

    @@ Documentation/git-for-each-ref.txt: OPTIONS
      
     
      ## builtin/for-each-ref.c ##
    -@@ builtin/for-each-ref.c: static char const * const for_each_ref_usage[] = {
    - 	N_("git for-each-ref [--points-at <object>]"),
    - 	N_("git for-each-ref [--merged [<commit>]] [--no-merged [<commit>]]"),
    - 	N_("git for-each-ref [--contains [<commit>]] [--no-contains [<commit>]]"),
    -+	N_("git for-each-ref [--exclude=<pattern> ...]"),
    - 	NULL
    - };
    - 
     @@ builtin/for-each-ref.c: int cmd_for_each_ref(int argc, const char **argv, const char *prefix)
      		OPT_INTEGER( 0 , "count", &maxcount, N_("show only <n> matched refs")),
      		OPT_STRING(  0 , "format", &format.format, N_("format"), N_("format to use for the output")),
    @@ builtin/for-each-ref.c: int cmd_for_each_ref(int argc, const char **argv, const
     
      ## ref-filter.c ##
     @@ ref-filter.c: static int filter_pattern_match(struct ref_filter *filter, const char *refname)
    - 	return match_pattern(filter, filter->name_patterns, refname);
    + 			     filter->ignore_case);
      }
      
     +static int filter_exclude_match(struct ref_filter *filter, const char *refname)
    @@ ref-filter.c: static int filter_pattern_match(struct ref_filter *filter, const c
     +		return 0;
     +	if (filter->match_as_path)
     +		return match_name_as_path(filter, filter->exclude.v, refname);
    -+	return match_pattern(filter, filter->exclude.v, refname);
    ++	return match_pattern(filter->exclude.v, refname, filter->ignore_case);
     +}
     +
      /*

7: f437cd83e2 ! 7: aa05549b6e refs: plumb exclude_patterns argument throughout
@@ Commit message
refs: plumb exclude_patterns argument throughout

The subsequent patch will want to access an optional excluded_patterns
- array within refs/packed-backend.c. To do so, the refs subsystem needs
- to be updated to pass this value across a number of different locations.
+ array within refs/packed-backend.c that will cull out certain
+ references matching any of the given patterns on a best-effort basis.
+
+ To do so, the refs subsystem needs to be updated to pass this value
+ across a number of different locations.

Prepare for a future patch by introducing this plumbing now, passing NULLs at top-level APIs in order to make that patch less noisy and more
8: 836a5665b7 ! 8: 6002c568b5 refs/packed-backend.c: refactor find_reference_location()
@@ Commit message

## refs/packed-backend.c ##
-@@ refs/packed-backend.c: static int cmp_packed_ref_records(const void *v1, const void *v2)
- * Compare a snapshot record at rec to the specified NUL-terminated
- * refname.
- */
--static int cmp_record_to_refname(const char *rec, const char *refname)
-+static int cmp_record_to_refname(const char *rec, const char *refname,
-+ int start)
- {
- const char *r1 = rec + the_hash_algo->hexsz + 1;
- const char *r2 = refname;
-@@ refs/packed-backend.c: static int cmp_record_to_refname(const char *rec, const char *refname)
- if (*r1 == '\n')
- return *r2 ? -1 : 0;
- if (!*r2)
-- return 1;
-+ return start ? 1 : -1;
- if (*r1 != *r2)
- return (unsigned char)*r1 < (unsigned char)*r2 ? -1 : +1;
- r1++;
@@ refs/packed-backend.c: static int load_contents(struct snapshot *snapshot)
return 1;
}
@@ refs/packed-backend.c: static int load_contents(struct snapshot *snapshot)
-static const char *find_reference_location(struct snapshot *snapshot,
- const char *refname, int mustexist)
+static const char *find_reference_location_1(struct snapshot *snapshot,
-+ const char *refname, int mustexist,
-+ int start)
++ const char *refname, int mustexist)
{
/*
* This is not *quite* a garden-variety binary search, because
@@ refs/packed-backend.c: static const char *find_reference_location(struct snapshot *snapshot,
-
- mid = lo + (hi - lo) / 2;
- rec = find_start_of_record(lo, mid);
-- cmp = cmp_record_to_refname(rec, refname);
-+ cmp = cmp_record_to_refname(rec, refname, start);
- if (cmp < 0) {
- lo = find_end_of_record(mid, hi);
- } else if (cmp > 0) {
-@@ refs/packed-backend.c: static const char *find_reference_location(struct snapshot *snapshot,
return lo;
}

@@ refs/packed-backend.c: static const char *find_reference_location(struct snapsho
+static const char *find_reference_location(struct snapshot *snapshot,
+ const char *refname, int mustexist)
+{
-+ return find_reference_location_1(snapshot, refname, mustexist, 1);
++ return find_reference_location_1(snapshot, refname, mustexist);
+}
+
/*
9: a39d1107c1 ! 9: 8c78f49a8d refs/packed-backend.c: implement skip lists to avoid excluded pattern(s)

    @@ Metadata
     Author: Taylor Blau <[email protected]>
     
      ## Commit message ##
    -    refs/packed-backend.c: implement skip lists to avoid excluded pattern(s)
    +    refs/packed-backend.c: implement jump lists to avoid excluded pattern(s)
     
         When iterating through the `packed-refs` file in order to answer a query
         like:
    @@ Commit message
         so is as follows:
     
           - For each excluded pattern, find the first record that matches it,
    -        and the first pattern that *doesn't* match it (i.e. the location
    +        and the first record that *doesn't* match it (i.e. the location
             you'd next want to consider when excluding that pattern).
     
    -      - Sort the patterns by their starting location within the
    -        `packed-refs` file.
    +      - Sort the set of excluded regions from the previous step in ascending
    +        order of the first location within the `packed-refs` file that
    +        matches.
     
    -      - Construct a skip list of regions by combining adjacent and
    -        overlapping regions from the previous step.
    +      - Clean up the results from the previous step: discard empty regions,
    +        and combine adjacent regions.
     
    -      - When iterating through the `packed-refs` file, if `iter->pos` is
    -        ever contained in one of the regions from the previous steps,
    -        advance `iter->pos` past the end of that region, and continue
    -        enumeration.
    +    Then when iterating through the `packed-refs` file, if `iter->pos` is
    +    ever contained in one of the regions from the previous steps, advance
    +    `iter->pos` past the end of that region, and continue enumeration.
     
    -    Note that this optimization is only possible when none of the excluded
    -    pattern(s) have special meta-characters in them. To see why this is the
    -    case, consider the exclusion pattern "refs/foo[a]". In general, in order
    -    to find the location of the first record that matches this pattern, we
    -    could only consider up to the first meta-character, "refs/foo". But this
    -    doesn't work, since the excluded region we'd come up with would include
    -    "refs/foobar", even though it is not excluded.
    +    Note that we only perform this optimization when none of the excluded
    +    pattern(s) have special meta-characters in them. For a pattern like
    +    "refs/foo[ac]", the excluded regions ("refs/fooa", "refs/fooc", and
    +    everything underneath them) are not connected. A future implementation
    +    that handles this case may split the character class (pretending as if
    +    two patterns were excluded: "refs/fooa", and "refs/fooc").
     
         There are a few other gotchas worth considering. First, note that the
    -    skip list is sorted, so once we skip past a region, we can avoid
    +    jump list is sorted, so once we jump past a region, we can avoid
         considering it (or any regions preceding it) again. The member
    -    `skip_pos` is used to track the first next-possible region to jump
    +    `jump_pos` is used to track the first next-possible region to jump
         through.
     
         Second, note that the exclusion list is best-effort, since we do not
    @@ Commit message
                 git update-ref --stdin
             $ git pack-refs --all
     
    -    , it is significantly faster to have `for-each-ref` skip over the
    +    , it is significantly faster to have `for-each-ref` jump over the
         excluded references, as opposed to filtering them out after the fact:
     
             $ hyperfine \
    @@ Commit message
               'git.compile for-each-ref --format="%(objectname) %(refname)" --exclude="refs/pull"' ran
               172.03 ± 9.60 times faster than 'git for-each-ref --format="%(objectname) %(refname)" | grep -vE "^[0-9a-f]{40} refs/pull/"'
     
    -    Using the skip list is fairly straightforward (see the changes to
    +    Using the jump list is fairly straightforward (see the changes to
         `refs/packed-backend.c::next_record()`), but constructing the list is
         not. To ensure that the construction is correct, add a new suite of
         tests in t1419 covering various corner cases (overlapping regions,
    @@ ref-filter.c: static int for_each_fullref_in_pattern(struct ref_filter *filter,
      /*
     
      ## refs/packed-backend.c ##
    -@@ refs/packed-backend.c: static const char *find_reference_location(struct snapshot *snapshot,
    - 	return find_reference_location_1(snapshot, refname, mustexist, 1);
    +@@ refs/packed-backend.c: static int cmp_packed_ref_records(const void *v1, const void *v2)
    +  * Compare a snapshot record at `rec` to the specified NUL-terminated
    +  * refname.
    +  */
    +-static int cmp_record_to_refname(const char *rec, const char *refname)
    ++static int cmp_record_to_refname(const char *rec, const char *refname,
    ++				 int start)
    + {
    + 	const char *r1 = rec + the_hash_algo->hexsz + 1;
    + 	const char *r2 = refname;
    +@@ refs/packed-backend.c: static int cmp_record_to_refname(const char *rec, const char *refname)
    + 		if (*r1 == '\n')
    + 			return *r2 ? -1 : 0;
    + 		if (!*r2)
    +-			return 1;
    ++			return start ? 1 : -1;
    + 		if (*r1 != *r2)
    + 			return (unsigned char)*r1 < (unsigned char)*r2 ? -1 : +1;
    + 		r1++;
    +@@ refs/packed-backend.c: static int load_contents(struct snapshot *snapshot)
      }
      
    + static const char *find_reference_location_1(struct snapshot *snapshot,
    +-					     const char *refname, int mustexist)
    ++					     const char *refname, int mustexist,
    ++					     int start)
    + {
    + 	/*
    + 	 * This is not *quite* a garden-variety binary search, because
    +@@ refs/packed-backend.c: static const char *find_reference_location_1(struct snapshot *snapshot,
    + 
    + 		mid = lo + (hi - lo) / 2;
    + 		rec = find_start_of_record(lo, mid);
    +-		cmp = cmp_record_to_refname(rec, refname);
    ++		cmp = cmp_record_to_refname(rec, refname, start);
    + 		if (cmp < 0) {
    + 			lo = find_end_of_record(mid, hi);
    + 		} else if (cmp > 0) {
    +@@ refs/packed-backend.c: static const char *find_reference_location_1(struct snapshot *snapshot,
    + static const char *find_reference_location(struct snapshot *snapshot,
    + 					   const char *refname, int mustexist)
    + {
    +-	return find_reference_location_1(snapshot, refname, mustexist);
    ++	return find_reference_location_1(snapshot, refname, mustexist, 1);
    ++}
    ++
     +/*
     + * Find the place in `snapshot->buf` after the end of the record for
     + * `refname`. In other words, find the location of first thing *after*
    @@ refs/packed-backend.c: static const char *find_reference_location(struct snapsho
     +					       int mustexist)
     +{
     +	return find_reference_location_1(snapshot, refname, mustexist, 0);
    -+}
    -+
    + }
    + 
      /*
    -  * Create a newly-allocated `snapshot` of the `packed-refs` file in
    -  * its current state and return it. The return value will already have
     @@ refs/packed-backend.c: struct packed_ref_iterator {
      	/* The end of the part of the buffer that will be iterated over: */
      	const char *eof;
      
    -+	struct skip_list_entry {
    ++	struct jump_list_entry {
     +		const char *start;
     +		const char *end;
    -+	} *skip;
    -+	size_t skip_nr, skip_alloc;
    -+	size_t skip_pos;
    ++	} *jump;
    ++	size_t jump_nr, jump_alloc;
    ++	size_t jump_pos;
     +
      	/* Scratch space for current values: */
      	struct object_id oid, peeled;
    @@ refs/packed-backend.c: struct packed_ref_iterator {
     +	 * Note that each skipped region is considered at most once,
     +	 * since they are ordered based on their starting position.
     +	 */
    -+	while (iter->skip_pos < iter->skip_nr) {
    -+		struct skip_list_entry *curr = &iter->skip[iter->skip_pos];
    ++	while (iter->jump_pos < iter->jump_nr) {
    ++		struct jump_list_entry *curr = &iter->jump[iter->jump_pos];
     +		if (iter->pos < curr->start)
     +			break; /* not to the next jump yet */
     +
    -+		iter->skip_pos++;
    ++		iter->jump_pos++;
     +		if (iter->pos < curr->end) {
     +			iter->pos = curr->end;
     +			break;
    @@ refs/packed-backend.c: static int packed_ref_iterator_abort(struct ref_iterator
      	int ok = ITER_DONE;
      
      	strbuf_release(&iter->refname_buf);
    -+	free(iter->skip);
    ++	free(iter->jump);
      	release_snapshot(iter->snapshot);
      	base_ref_iterator_free(ref_iterator);
      	return ok;
    @@ refs/packed-backend.c: static struct ref_iterator_vtable packed_ref_iterator_vta
      	.abort = packed_ref_iterator_abort
      };
      
    -+static int skip_list_entry_cmp(const void *va, const void *vb)
    ++static int jump_list_entry_cmp(const void *va, const void *vb)
     +{
    -+	const struct skip_list_entry *a = va;
    -+	const struct skip_list_entry *b = vb;
    ++	const struct jump_list_entry *a = va;
    ++	const struct jump_list_entry *b = vb;
     +
     +	if (a->start < b->start)
     +		return -1;
    @@ refs/packed-backend.c: static struct ref_iterator_vtable packed_ref_iterator_vta
     +	return y;
     +}
     +
    -+static void populate_excluded_skip_list(struct packed_ref_iterator *iter,
    ++static void populate_excluded_jump_list(struct packed_ref_iterator *iter,
     +					struct snapshot *snapshot,
     +					const char **excluded_patterns)
     +{
     +	size_t i, j;
     +	const char **pattern;
    ++	struct jump_list_entry *last_disjoint;
     +
     +	if (!excluded_patterns)
     +		return;
     +
     +	for (pattern = excluded_patterns; *pattern; pattern++) {
    -+		struct skip_list_entry *e;
    ++		struct jump_list_entry *e;
     +
     +		/*
     +		 * We can't feed any excludes with globs in them to the
    @@ refs/packed-backend.c: static struct ref_iterator_vtable packed_ref_iterator_vta
     +		if (has_glob_special(*pattern))
     +			continue;
     +
    -+		ALLOC_GROW(iter->skip, iter->skip_nr + 1, iter->skip_alloc);
    ++		ALLOC_GROW(iter->jump, iter->jump_nr + 1, iter->jump_alloc);
     +
    -+		e = &iter->skip[iter->skip_nr++];
    ++		e = &iter->jump[iter->jump_nr++];
     +		e->start = find_reference_location(snapshot, *pattern, 0);
     +		e->end = find_reference_location_end(snapshot, *pattern, 0);
     +	}
     +
    -+	if (!iter->skip_nr) {
    ++	if (!iter->jump_nr) {
     +		/*
     +		 * Every entry in exclude_patterns has a meta-character,
     +		 * nothing to do here.
    @@ refs/packed-backend.c: static struct ref_iterator_vtable packed_ref_iterator_vta
     +		return;
     +	}
     +
    -+	QSORT(iter->skip, iter->skip_nr, skip_list_entry_cmp);
    ++	QSORT(iter->jump, iter->jump_nr, jump_list_entry_cmp);
     +
     +	/*
    -+	 * As an optimization, merge adjacent entries in the skip list
    ++	 * As an optimization, merge adjacent entries in the jump list
     +	 * to jump forwards as far as possible when entering a skipped
     +	 * region.
     +	 *
    @@ refs/packed-backend.c: static struct ref_iterator_vtable packed_ref_iterator_vta
     +	 * we want to combine that into a single entry jumping from A to
     +	 * C.
     +	 */
    -+	for (i = 1, j = 1; i < iter->skip_nr; i++) {
    -+		struct skip_list_entry *ours = &iter->skip[i];
    -+		struct skip_list_entry *prev = &iter->skip[i - 1];
    ++	last_disjoint = iter->jump;
    ++
    ++	for (i = 1, j = 1; i < iter->jump_nr; i++) {
    ++		struct jump_list_entry *ours = &iter->jump[i];
     +
     +		if (ours->start == ours->end) {
     +			/* ignore empty regions (no matching entries) */
     +			continue;
    -+		} else if (prev->end >= ours->start) {
    ++		} else if (ours->start <= last_disjoint->end) {
     +			/* overlapping regions extend the previous one */
    -+			prev->end = ptr_max(prev->end, ours->end);
    ++			last_disjoint->end = ptr_max(last_disjoint->end, ours->end);
     +		} else {
     +			/* otherwise, insert a new region */
    -+			iter->skip[j++] = *ours;
    ++			iter->jump[j++] = *ours;
    ++			last_disjoint = ours;
    ++
     +		}
     +	}
     +
    -+	iter->skip_nr = j;
    -+	iter->skip_pos = 0;
    ++	iter->jump_nr = j;
    ++	iter->jump_pos = 0;
     +}
     +
      static struct ref_iterator *packed_ref_iterator_begin(
    @@ refs/packed-backend.c: static struct ref_iterator *packed_ref_iterator_begin(
      	base_ref_iterator_init(ref_iterator, &packed_ref_iterator_vtable, 1);
      
     +	if (exclude_patterns)
    -+		populate_excluded_skip_list(iter, snapshot, exclude_patterns);
    ++		populate_excluded_jump_list(iter, snapshot, exclude_patterns);
     +
      	iter->snapshot = snapshot;
      	acquire_snapshot(snapshot);
    @@ t/t1419-exclude-refs.sh (new)
     +	git pack-refs --all
     +'
     +
    -+test_expect_success 'for_each_ref__exclude(refs/heads/foo/)' '
    -+	# region in middle
    ++test_expect_success 'excluded region in middle' '
     +	for_each_ref__exclude refs/heads refs/heads/foo >actual &&
     +	for_each_ref refs/heads/bar refs/heads/baz refs/heads/quux >expect &&
     +
     +	test_cmp expect actual
     +'
     +
    -+test_expect_success 'for_each_ref__exclude(refs/heads/bar/)' '
    -+	# region at beginning
    ++test_expect_success 'excluded region at beginning' '
     +	for_each_ref__exclude refs/heads refs/heads/bar >actual &&
     +	for_each_ref refs/heads/baz refs/heads/foo refs/heads/quux >expect &&
     +
     +	test_cmp expect actual
     +'
     +
    -+test_expect_success 'for_each_ref__exclude(refs/heads/quux/)' '
    -+	# region at end
    ++test_expect_success 'excluded region at end' '
     +	for_each_ref__exclude refs/heads refs/heads/quux >actual &&
     +	for_each_ref refs/heads/foo refs/heads/bar refs/heads/baz >expect &&
     +
     +	test_cmp expect actual
     +'
     +
    -+test_expect_success 'for_each_ref__exclude(refs/heads/bar/, refs/heads/quux/)' '
    -+	# disjoint regions
    ++test_expect_success 'disjoint excluded regions' '
     +	for_each_ref__exclude refs/heads refs/heads/bar refs/heads/quux >actual &&
     +	for_each_ref refs/heads/baz refs/heads/foo >expect &&
     +
     +	test_cmp expect actual
     +'
     +
    -+test_expect_success 'for_each_ref__exclude(refs/heads/bar/, refs/heads/baz/)' '
    -+	# adjacent, non-overlapping regions
    ++test_expect_success 'adjacent, non-overlapping excluded regions' '
     +	for_each_ref__exclude refs/heads refs/heads/bar refs/heads/baz >actual &&
     +	for_each_ref refs/heads/foo refs/heads/quux >expect &&
     +
     +	test_cmp expect actual
     +'
     +
    -+test_expect_success 'for_each_ref__exclude(refs/heads/ba refs/heads/baz/)' '
    -+	# overlapping region
    ++test_expect_success 'overlapping excluded regions' '
     +	for_each_ref__exclude refs/heads refs/heads/ba refs/heads/baz >actual &&
     +	for_each_ref refs/heads/foo refs/heads/quux >expect &&
     +
     +	test_cmp expect actual
     +'
     +
    -+test_expect_success 'for_each_ref__exclude(refs/heads/does/not/exist)' '
    -+	# empty region
    ++test_expect_success 'several overlapping excluded regions' '
    ++	for_each_ref__exclude refs/heads \
    ++		refs/heads/bar refs/heads/baz refs/heads/foo >actual &&
    ++	for_each_ref refs/heads/quux >expect &&
    ++
    ++	test_cmp expect actual
    ++'
    ++
    ++test_expect_success 'non-matching excluded section' '
     +	for_each_ref__exclude refs/heads refs/heads/does/not/exist >actual &&
     +	for_each_ref >expect &&
     +
     +	test_cmp expect actual
     +'
     +
    -+test_expect_success 'for_each_ref__exclude(refs/heads/ba*)' '
    -+	# discards meta-characters
    ++test_expect_success 'meta-characters are discarded' '
     +	for_each_ref__exclude refs/heads "refs/heads/ba*" >actual &&
     +	for_each_ref >expect &&
     +
10:  5698c2794f ! 10:  5059f5dd42 refs/packed-backend.c: add trace2 counters for skip list
    @@ Metadata
     Author: Taylor Blau <[email protected]>
     
      ## Commit message ##
    -    refs/packed-backend.c: add trace2 counters for skip list
    +    refs/packed-backend.c: add trace2 counters for jump list
     
         The previous commit added low-level tests to ensure that the packed-refs
         iterator did not enumerate excluded sections of the refspace.
    @@ refs/packed-backend.c
      enum mmap_strategy {
      	/*
     @@ refs/packed-backend.c: static int next_record(struct packed_ref_iterator *iter)
    - 		iter->skip_pos++;
    + 		iter->jump_pos++;
      		if (iter->pos < curr->end) {
      			iter->pos = curr->end;
    -+			trace2_counter_add(TRACE2_COUNTER_ID_PACKED_REFS_SKIPS, 1);
    ++			trace2_counter_add(TRACE2_COUNTER_ID_PACKED_REFS_JUMPS, 1);
      			break;
      		}
      	}
    @@ t/t1419-exclude-refs.sh: for_each_ref () {
      	git for-each-ref --format='%(refname)' "$@"
      }
      
    -+assert_skips () {
    ++assert_jumps () {
     +	local nr="$1"
     +	local trace="$2"
     +
    -+	grep -q "name:skips_made value:$nr" $trace
    ++	grep -q "name:jumps_made value:$nr" $trace
     +}
     +
    -+assert_no_skips () {
    -+	! assert_skips ".*" "$1"
    ++assert_no_jumps () {
    ++	! assert_jumps ".*" "$1"
     +}
     +
      test_expect_success 'setup' '
      	test_commit --no-tag base &&
      	base="$(git rev-parse HEAD)" &&
     @@ t/t1419-exclude-refs.sh: test_expect_success 'setup' '
    + '
      
    - test_expect_success 'for_each_ref__exclude(refs/heads/foo/)' '
    - 	# region in middle
    + test_expect_success 'excluded region in middle' '
     -	for_each_ref__exclude refs/heads refs/heads/foo >actual &&
     +	for_each_ref__exclude refs/heads refs/heads/foo >actual 2>perf &&
      	for_each_ref refs/heads/bar refs/heads/baz refs/heads/quux >expect &&
      
     -	test_cmp expect actual
     +	test_cmp expect actual &&
    -+	assert_skips 1 perf
    ++	assert_jumps 1 perf
      '
      
    - test_expect_success 'for_each_ref__exclude(refs/heads/bar/)' '
    - 	# region at beginning
    + test_expect_success 'excluded region at beginning' '
     -	for_each_ref__exclude refs/heads refs/heads/bar >actual &&
     +	for_each_ref__exclude refs/heads refs/heads/bar >actual 2>perf &&
      	for_each_ref refs/heads/baz refs/heads/foo refs/heads/quux >expect &&
      
     -	test_cmp expect actual
     +	test_cmp expect actual &&
    -+	assert_skips 1 perf
    ++	assert_jumps 1 perf
      '
      
    - test_expect_success 'for_each_ref__exclude(refs/heads/quux/)' '
    - 	# region at end
    + test_expect_success 'excluded region at end' '
     -	for_each_ref__exclude refs/heads refs/heads/quux >actual &&
     +	for_each_ref__exclude refs/heads refs/heads/quux >actual 2>perf &&
      	for_each_ref refs/heads/foo refs/heads/bar refs/heads/baz >expect &&
      
     -	test_cmp expect actual
     +	test_cmp expect actual &&
    -+	assert_skips 1 perf
    ++	assert_jumps 1 perf
      '
      
    - test_expect_success 'for_each_ref__exclude(refs/heads/bar/, refs/heads/quux/)' '
    - 	# disjoint regions
    + test_expect_success 'disjoint excluded regions' '
     -	for_each_ref__exclude refs/heads refs/heads/bar refs/heads/quux >actual &&
     +	for_each_ref__exclude refs/heads refs/heads/bar refs/heads/quux >actual 2>perf &&
      	for_each_ref refs/heads/baz refs/heads/foo >expect &&
      
     -	test_cmp expect actual
     +	test_cmp expect actual &&
    -+	assert_skips 2 perf
    ++	assert_jumps 2 perf
      '
      
    - test_expect_success 'for_each_ref__exclude(refs/heads/bar/, refs/heads/baz/)' '
    - 	# adjacent, non-overlapping regions
    + test_expect_success 'adjacent, non-overlapping excluded regions' '
     -	for_each_ref__exclude refs/heads refs/heads/bar refs/heads/baz >actual &&
     +	for_each_ref__exclude refs/heads refs/heads/bar refs/heads/baz >actual 2>perf &&
      	for_each_ref refs/heads/foo refs/heads/quux >expect &&
      
     -	test_cmp expect actual
     +	test_cmp expect actual &&
    -+	assert_skips 1 perf
    ++	assert_jumps 1 perf
      '
      
    - test_expect_success 'for_each_ref__exclude(refs/heads/ba refs/heads/baz/)' '
    - 	# overlapping region
    + test_expect_success 'overlapping excluded regions' '
     -	for_each_ref__exclude refs/heads refs/heads/ba refs/heads/baz >actual &&
     +	for_each_ref__exclude refs/heads refs/heads/ba refs/heads/baz >actual 2>perf &&
      	for_each_ref refs/heads/foo refs/heads/quux >expect &&
      
     -	test_cmp expect actual
     +	test_cmp expect actual &&
    -+	assert_skips 1 perf
    ++	assert_jumps 1 perf
      '
      
    - test_expect_success 'for_each_ref__exclude(refs/heads/does/not/exist)' '
    - 	# empty region
    + test_expect_success 'several overlapping excluded regions' '
    + 	for_each_ref__exclude refs/heads \
    +-		refs/heads/bar refs/heads/baz refs/heads/foo >actual &&
    ++		refs/heads/bar refs/heads/baz refs/heads/foo >actual 2>perf &&
    + 	for_each_ref refs/heads/quux >expect &&
    + 
    +-	test_cmp expect actual
    ++	test_cmp expect actual &&
    ++	assert_jumps 1 perf
    + '
    + 
    + test_expect_success 'non-matching excluded section' '
     -	for_each_ref__exclude refs/heads refs/heads/does/not/exist >actual &&
     +	for_each_ref__exclude refs/heads refs/heads/does/not/exist >actual 2>perf &&
      	for_each_ref >expect &&
      
     -	test_cmp expect actual
     +	test_cmp expect actual &&
    -+	assert_no_skips
    ++	assert_no_jumps
      '
      
    - test_expect_success 'for_each_ref__exclude(refs/heads/ba*)' '
    - 	# discards meta-characters
    + test_expect_success 'meta-characters are discarded' '
     -	for_each_ref__exclude refs/heads "refs/heads/ba*" >actual &&
     +	for_each_ref__exclude refs/heads "refs/heads/ba*" >actual 2>perf &&
      	for_each_ref >expect &&
      
     -	test_cmp expect actual
     +	test_cmp expect actual &&
    -+	assert_no_skips
    ++	assert_no_jumps
      '
      
      test_done
    @@ trace2.h: enum trace2_counter_id {
      	TRACE2_COUNTER_ID_TEST1 = 0, /* emits summary event only */
      	TRACE2_COUNTER_ID_TEST2,     /* emits summary and thread events */
      
    -+	TRACE2_COUNTER_ID_PACKED_REFS_SKIPS, /* counts number of skips */
    ++	TRACE2_COUNTER_ID_PACKED_REFS_JUMPS, /* counts number of jumps */
     +
      	/* Add additional counter definitions before here. */
      	TRACE2_NUMBER_OF_COUNTERS
    @@ trace2/tr2_ctr.c: static struct tr2_counter_metadata tr2_counter_metadata[TRACE2
      		.name = "test2",
      		.want_per_thread_events = 1,
      	},
    -+	[TRACE2_COUNTER_ID_PACKED_REFS_SKIPS] = {
    ++	[TRACE2_COUNTER_ID_PACKED_REFS_JUMPS] = {
     +		.category = "packed-refs",
    -+		.name = "skips_made",
    ++		.name = "jumps_made",
     +		.want_per_thread_events = 0,
     +	},

11: 5b9814ad8c ! 11: f765b50a84 revision.h: store hidden refs in a strvec

    @@ revision.c: void init_ref_exclusions(struct ref_exclusions *exclusions)
     
      ## revision.h ##
     @@
    - #include "commit-slab-decl.h"
    + #include "decorate.h"
      #include "ident.h"
      #include "list-objects-filter-options.h"
     +#include "strvec.h"
12:  dd5b34185c ! 12:  254bcc4361 refs/packed-backend.c: ignore complicated hidden refs rules
    @@ Commit message
         Signed-off-by: Taylor Blau <[email protected]>
     
      ## refs/packed-backend.c ##
    -@@ refs/packed-backend.c: static void populate_excluded_skip_list(struct packed_ref_iterator *iter,
    +@@ refs/packed-backend.c: static void populate_excluded_jump_list(struct packed_ref_iterator *iter,
      	if (!excluded_patterns)
      		return;
      
    @@ refs/packed-backend.c: static void populate_excluded_skip_list(struct packed_ref
     +		 * '^' rules.
     +		 *
     +		 * Both are possible to do, but complicated, so avoid
    -+		 * populating the skip list at all if we see either of
    ++		 * populating the jump list at all if we see either of
     +		 * these patterns.
     +		 */
     +		if (**pattern == '!' || **pattern == '^')
    @@ refs/packed-backend.c: static void populate_excluded_skip_list(struct packed_ref
     +	}
     +
      	for (pattern = excluded_patterns; *pattern; pattern++) {
    - 		struct skip_list_entry *e;
    + 		struct jump_list_entry *e;
      
     
      ## t/t1419-exclude-refs.sh ##
    -@@ t/t1419-exclude-refs.sh: test_expect_success 'for_each_ref__exclude(refs/heads/ba*)' '
    - 	assert_no_skips
    +@@ t/t1419-exclude-refs.sh: test_expect_success 'meta-characters are discarded' '
    + 	assert_no_jumps
      '
      
    -+test_expect_success 'for_each_ref__exclude(refs/heads/foo, !refs/heads/foo/1)' '
    -+	# discards complex hidden ref rules
    ++test_expect_success 'complex hidden ref rules are discarded' '
     +	for_each_ref__exclude refs/heads refs/heads/foo "!refs/heads/foo/1" \
     +		>actual 2>perf &&
     +	for_each_ref >expect &&
     +
     +	test_cmp expect actual &&
    -+	assert_no_skips
    ++	assert_no_jumps
     +'
     +
      test_done
13:  c65b3dea81 = 13:  50e7df7dc0 refs.h: let `for_each_namespaced_ref()` take excluded patterns
15:  7d3383083d = 14:  f6a3a5a6ba builtin/receive-pack.c: avoid enumerating hidden references
14:  44bbf85e73 ! 15:  2331fa7a4d upload-pack.c: avoid enumerating hidden refs where possible
    @@ upload-pack.c: static int get_common_commits(struct upload_pack_data *data,
     +				      struct upload_pack_data *data)
     +{
     +	/*
    -+	 * If `data->allow_uor` allows updating hidden refs, we need to
    ++	 * If `data->allow_uor` allows fetching hidden refs, we need to
     +	 * mark all references (including hidden ones), to check in
     +	 * `is_our_ref()` below.
     +	 *

-: ---------- > 16: 2c6b89d64a ls-refs.c: avoid enumerating hidden refs where possible

Signature · 1 line
2.40.1.572.g5c4ab523ef
#39 lore raw

PATCH v2 1/16 refs.c: rename `ref_filter`

From: Jeff King <[email protected]>

The refs machinery has its own implementation of a ref_filter (used by for-each-ref), which is distinct from the ref-filler.h API (also used by for-each-ref, among other things).

Rename the one within refs.c to more clearly indicate its purpose.

---
 refs.c | 15 ++++++++-------
 1 file changed, 8 insertions(+), 7 deletions(-)
refs.c +8 -7
index d2a98e1c21..b9b77d2eff 100644
@@ -375,8 +375,8 @@ char *resolve_refdup(const char *refname, int resolve_flags,
 				   oid, flags);
 }
 
-/* The argument to filter_refs */
-struct ref_filter {
+/* The argument to for_each_filter_refs */
+struct for_each_ref_filter {
 	const char *pattern;
 	const char *prefix;
 	each_ref_fn *fn;
@@ -409,10 +409,11 @@ int ref_exists(const char *refname)
 	return refs_ref_exists(get_main_ref_store(the_repository), refname);
 }
 
-static int filter_refs(const char *refname, const struct object_id *oid,
-			   int flags, void *data)
+static int for_each_filter_refs(const char *refname,
+				const struct object_id *oid,
+				int flags, void *data)
 {
-	struct ref_filter *filter = (struct ref_filter *)data;
+	struct for_each_ref_filter *filter = data;
 
 	if (wildmatch(filter->pattern, refname, 0))
 		return 0;
@@ -569,7 +570,7 @@ int for_each_glob_ref_in(each_ref_fn fn, const char *pattern,
 	const char *prefix, void *cb_data)
 {
 	struct strbuf real_pattern = STRBUF_INIT;
-	struct ref_filter filter;
+	struct for_each_ref_filter filter;
 	int ret;
 
 	if (!prefix && !starts_with(pattern, "refs/"))
@@ -589,7 +590,7 @@ int for_each_glob_ref_in(each_ref_fn fn, const char *pattern,
 	filter.prefix = prefix;
 	filter.fn = fn;
 	filter.cb_data = cb_data;
-	ret = for_each_ref(filter_refs, &filter);
+	ret = for_each_ref(for_each_filter_refs, &filter);
 
 	strbuf_release(&real_pattern);
 	return ret;
Signature · 1 line
2.40.1.572.g5c4ab523ef
#40 lore raw

PATCH v2 2/16 ref-filter.h: provide `REF_FILTER_INIT`

In reply to Taylor Blau

From: Jeff King <[email protected]>

Provide a sane initialization value for struct ref_filter, which in a subsequent patch will be used to initialize a new field.

In the meantime, fix a case in test-reach.c where its ref_filter is not even zero-initialized.

---
 builtin/branch.c       | 3 +--
 builtin/for-each-ref.c | 3 +--
 builtin/tag.c          | 3 +--
 ref-filter.h           | 3 +++
 t/helper/test-reach.c  | 2 +-
 5 files changed, 7 insertions(+), 7 deletions(-)
builtin/branch.c +1 -2
index 501c47657c..03bb8e414c 100644
@@ -662,7 +662,7 @@ int cmd_branch(int argc, const char **argv, const char *prefix)
 	int reflog = 0, quiet = 0, icase = 0, force = 0,
 	    recurse_submodules_explicit = 0;
 	enum branch_track track;
-	struct ref_filter filter;
+	struct ref_filter filter = REF_FILTER_INIT;
 	static struct ref_sorting *sorting;
 	struct string_list sorting_options = STRING_LIST_INIT_DUP;
 	struct ref_format format = REF_FORMAT_INIT;
@@ -720,7 +720,6 @@ int cmd_branch(int argc, const char **argv, const char *prefix)
 
 	setup_ref_filter_porcelain_msg();
 
-	memset(&filter, 0, sizeof(filter));
 	filter.kind = FILTER_REFS_BRANCHES;
 	filter.abbrev = -1;
 
builtin/for-each-ref.c +1 -2
index 695fc8f4a5..99ccb73518 100644
@@ -24,7 +24,7 @@ int cmd_for_each_ref(int argc, const char **argv, const char *prefix)
 	struct string_list sorting_options = STRING_LIST_INIT_DUP;
 	int maxcount = 0, icase = 0, omit_empty = 0;
 	struct ref_array array;
-	struct ref_filter filter;
+	struct ref_filter filter = REF_FILTER_INIT;
 	struct ref_format format = REF_FORMAT_INIT;
 	struct strbuf output = STRBUF_INIT;
 	struct strbuf err = STRBUF_INIT;
@@ -61,7 +61,6 @@ int cmd_for_each_ref(int argc, const char **argv, const char *prefix)
 	};
 
 	memset(&array, 0, sizeof(array));
-	memset(&filter, 0, sizeof(filter));
 
 	format.format = "%(objectname) %(objecttype)\t%(refname)";
 
builtin/tag.c +1 -2
index 1850a6a6fd..6b41bb7374 100644
@@ -443,7 +443,7 @@ int cmd_tag(int argc, const char **argv, const char *prefix)
 	struct msg_arg msg = { .buf = STRBUF_INIT };
 	struct ref_transaction *transaction;
 	struct strbuf err = STRBUF_INIT;
-	struct ref_filter filter;
+	struct ref_filter filter = REF_FILTER_INIT;
 	struct ref_sorting *sorting;
 	struct string_list sorting_options = STRING_LIST_INIT_DUP;
 	struct ref_format format = REF_FORMAT_INIT;
@@ -501,7 +501,6 @@ int cmd_tag(int argc, const char **argv, const char *prefix)
 	git_config(git_tag_config, &sorting_options);
 
 	memset(&opt, 0, sizeof(opt));
-	memset(&filter, 0, sizeof(filter));
 	filter.lines = -1;
 	opt.sign = -1;
 
ref-filter.h +3 -0
index 430701cfb7..a920f73b29 100644
@@ -92,6 +92,9 @@ struct ref_format {
 	struct string_list bases;
 };
 
+#define REF_FILTER_INIT { \
+	.points_at = OID_ARRAY_INIT, \
+}
 #define REF_FORMAT_INIT {             \
 	.use_color = -1,              \
 	.bases = STRING_LIST_INIT_DUP, \
t/helper/test-reach.c +1 -1
index 5b6f217441..ef58f10c2d 100644
@@ -139,7 +139,7 @@ int cmd__reach(int ac, const char **av)
 
 		printf("%s(X,_,_,0,0):%d\n", av[1], can_all_from_reach_with_flag(&X_obj, 2, 4, 0, 0));
 	} else if (!strcmp(av[1], "commit_contains")) {
-		struct ref_filter filter;
+		struct ref_filter filter = REF_FILTER_INIT;
 		struct contains_cache cache;
 		init_contains_cache(&cache);
 
Signature · 1 line
2.40.1.572.g5c4ab523ef
#41 lore raw

PATCH v2 3/16 ref-filter: clear reachable list pointers after freeing

In reply to Taylor Blau

From: Jeff King <[email protected]>

In reach_filter(), we pop all commits from the reachable lists, leaving them empty. But because we're operating on a list pointer that was passed by value, the original filter.reachable_from pointer is left dangling.

---
 ref-filter.c | 14 +++++++-------
 1 file changed, 7 insertions(+), 7 deletions(-)
ref-filter.c +7 -7
index 4991cd4f7a..048d277cbf 100644
@@ -2418,13 +2418,13 @@ void ref_array_clear(struct ref_array *array)
 #define EXCLUDE_REACHED 0
 #define INCLUDE_REACHED 1
 static void reach_filter(struct ref_array *array,
-			 struct commit_list *check_reachable,
+			 struct commit_list **check_reachable,
 			 int include_reached)
 {
 	int i, old_nr;
 	struct commit **to_clear;
 
-	if (!check_reachable)
+	if (!*check_reachable)
 		return;
 
 	CALLOC_ARRAY(to_clear, array->nr);
@@ -2434,7 +2434,7 @@ static void reach_filter(struct ref_array *array,
 	}
 
 	tips_reachable_from_bases(the_repository,
-				  check_reachable,
+				  *check_reachable,
 				  to_clear, array->nr,
 				  UNINTERESTING);
 
@@ -2455,8 +2455,8 @@ static void reach_filter(struct ref_array *array,
 
 	clear_commit_marks_many(old_nr, to_clear, ALL_REV_FLAGS);
 
-	while (check_reachable) {
-		struct commit *merge_commit = pop_commit(&check_reachable);
+	while (*check_reachable) {
+		struct commit *merge_commit = pop_commit(check_reachable);
 		clear_commit_marks(merge_commit, ALL_REV_FLAGS);
 	}
 
@@ -2553,8 +2553,8 @@ int filter_refs(struct ref_array *array, struct ref_filter *filter, unsigned int
 	clear_contains_cache(&ref_cbdata.no_contains_cache);
 
 	/*  Filters that need revision walking */
-	reach_filter(array, filter->reachable_from, INCLUDE_REACHED);
-	reach_filter(array, filter->unreachable_from, EXCLUDE_REACHED);
+	reach_filter(array, &filter->reachable_from, INCLUDE_REACHED);
+	reach_filter(array, &filter->unreachable_from, EXCLUDE_REACHED);
 
 	save_commit_buffer = save_commit_buffer_orig;
 	return ret;
Signature · 1 line
2.40.1.572.g5c4ab523ef
#42 lore raw

PATCH v2 4/16 ref-filter: add `ref_filter_clear()`

In reply to Taylor Blau

From: Jeff King <[email protected]>

We did not bother to clean up at all in git branch or git tag, and git for-each-ref only cleans up a couple of members.

Add and call ref_filter_clear() when cleaning up a struct ref_filter. Running this patch (without any test changes) indicates a couple of now leak-free tests. This was found by running:

    $ make SANITIZE=leak
    $ make -C t GIT_TEST_PASSING_SANITIZE_LEAK=check GIT_TEST_OPTS=--immediate

(Note that the reachable_from and unreachable_from lists should be cleaned as they are used. So this is just covering any case where we might bail before running the reachability check.)

---
 builtin/branch.c        |  1 +
 builtin/for-each-ref.c  |  3 +--
 builtin/tag.c           |  1 +
 ref-filter.c            | 16 ++++++++++++++++
 ref-filter.h            |  3 +++
 t/t0041-usage.sh        |  1 +
 t/t3402-rebase-merge.sh |  1 +
 7 files changed, 24 insertions(+), 2 deletions(-)
builtin/branch.c +1 -0
index 03bb8e414c..c201f0cb0b 100644
@@ -813,6 +813,7 @@ int cmd_branch(int argc, const char **argv, const char *prefix)
 		print_columns(&output, colopts, NULL);
 		string_list_clear(&output, 0);
 		ref_sorting_release(sorting);
+		ref_filter_clear(&filter);
 		return 0;
 	} else if (edit_description) {
 		const char *branch_name;
builtin/for-each-ref.c +1 -2
index 99ccb73518..c01fa6fefe 100644
@@ -120,8 +120,7 @@ int cmd_for_each_ref(int argc, const char **argv, const char *prefix)
 	strbuf_release(&err);
 	strbuf_release(&output);
 	ref_array_clear(&array);
-	free_commit_list(filter.with_commit);
-	free_commit_list(filter.no_commit);
+	ref_filter_clear(&filter);
 	ref_sorting_release(sorting);
 	strvec_clear(&vec);
 	return 0;
builtin/tag.c +1 -0
index 6b41bb7374..aab5e693fe 100644
@@ -645,6 +645,7 @@ int cmd_tag(int argc, const char **argv, const char *prefix)
 
 cleanup:
 	ref_sorting_release(sorting);
+	ref_filter_clear(&filter);
 	strbuf_release(&buf);
 	strbuf_release(&ref);
 	strbuf_release(&reflog_msg);
ref-filter.c +16 -0
index 048d277cbf..d32f426898 100644
@@ -2866,3 +2866,19 @@ int parse_opt_merge_filter(const struct option *opt, const char *arg, int unset)
 
 	return 0;
 }
+
+void ref_filter_init(struct ref_filter *filter)
+{
+	struct ref_filter blank = REF_FILTER_INIT;
+	memcpy(filter, &blank, sizeof(blank));
+}
+
+void ref_filter_clear(struct ref_filter *filter)
+{
+	oid_array_clear(&filter->points_at);
+	free_commit_list(filter->with_commit);
+	free_commit_list(filter->no_commit);
+	free_commit_list(filter->reachable_from);
+	free_commit_list(filter->unreachable_from);
+	ref_filter_init(filter);
+}
ref-filter.h +3 -0
index a920f73b29..160b807224 100644
@@ -170,4 +170,7 @@ void filter_ahead_behind(struct repository *r,
 			 struct ref_format *format,
 			 struct ref_array *array);
 
+void ref_filter_init(struct ref_filter *filter);
+void ref_filter_clear(struct ref_filter *filter);
+
 #endif /*  REF_FILTER_H  */
t/t0041-usage.sh +1 -0
index c4fc34eb18..9ea974b0c6 100755
@@ -5,6 +5,7 @@ test_description='Test commands behavior when given invalid argument value'
 GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main
 export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME
 
+TEST_PASSES_SANITIZE_LEAK=true
 . ./test-lib.sh
 
 test_expect_success 'setup ' '
t/t3402-rebase-merge.sh +1 -0
index 79b0640c00..e9e03ca4b5 100755
@@ -8,6 +8,7 @@ test_description='git rebase --merge test'
 GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main
 export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME
 
+TEST_PASSES_SANITIZE_LEAK=true
 . ./test-lib.sh
 
 T="A quick brown fox
Signature · 1 line
2.40.1.572.g5c4ab523ef
#43 lore raw

PATCH v2 5/16 ref-filter.c: parameterize match functions over patterns

In reply to Taylor Blau

From: Jeff King <[email protected]>

match_pattern() and match_name_as_path() both take a struct ref_filter *, and then store a stack variable patterns pointing at filter->patterns.

The subsequent patch will add a new array of patterns to match over (the excluded patterns, via a new git for-each-ref --exclude option), treating the return value of these functions differently depending on which patterns are being used to match.

Tweak match_pattern() and match_name_as_path() to take an array of patterns to prepare for passing either in.

Once we start passing either in, match_pattern() will have little to do with a particular struct ref_filter * instance. To clarify this, drop it from the argument list, and replace it with the only bit of the ref_filter that we care about (filter->ignore_case).

---
 ref-filter.c | 16 +++++++++-------
 1 file changed, 9 insertions(+), 7 deletions(-)
ref-filter.c +9 -7
index d32f426898..6d91c7cb0d 100644
@@ -2104,12 +2104,12 @@ static int get_ref_atom_value(struct ref_array_item *ref, int atom,
  * matches a pattern "refs/heads/mas") or a wildcard (e.g. the same ref
  * matches "refs/heads/mas*", too).
  */
-static int match_pattern(const struct ref_filter *filter, const char *refname)
+static int match_pattern(const char **patterns, const char *refname,
+			 const int ignore_case)
 {
-	const char **patterns = filter->name_patterns;
 	unsigned flags = 0;
 
-	if (filter->ignore_case)
+	if (ignore_case)
 		flags |= WM_CASEFOLD;
 
 	/*
@@ -2134,9 +2134,10 @@ static int match_pattern(const struct ref_filter *filter, const char *refname)
  * matches a pattern "refs/heads/" but not "refs/heads/m") or a
  * wildcard (e.g. the same ref matches "refs/heads/m*", too).
  */
-static int match_name_as_path(const struct ref_filter *filter, const char *refname)
+static int match_name_as_path(const struct ref_filter *filter,
+			      const char **pattern,
+			      const char *refname)
 {
-	const char **pattern = filter->name_patterns;
 	int namelen = strlen(refname);
 	unsigned flags = WM_PATHNAME;
 
@@ -2165,8 +2166,9 @@ static int filter_pattern_match(struct ref_filter *filter, const char *refname)
 	if (!*filter->name_patterns)
 		return 1; /* No pattern always matches */
 	if (filter->match_as_path)
-		return match_name_as_path(filter, refname);
-	return match_pattern(filter, refname);
+		return match_name_as_path(filter, filter->name_patterns, refname);
+	return match_pattern(filter->name_patterns, refname,
+			     filter->ignore_case);
 }
 
 /*
Signature · 1 line
2.40.1.572.g5c4ab523ef
#44 lore raw

PATCH v2 6/16 builtin/for-each-ref.c: add `--exclude` option

In reply to Taylor Blau

When using for-each-ref, it is sometimes convenient for the caller to be able to exclude certain parts of the references.

For example, if there are many refs/__hidden__/* references, the caller may want to emit all references *except* the hidden ones. Currently, the only way to do this is to post-process the output, like:

    $ git for-each-ref --format='%(refname)' | grep -v '^refs/hidden/'

Which is do-able, but requires processing a potentially large quantity of references.

Teach git for-each-ref a new --exclude=<pattern> option, which excludes references from the results if they match one or more excluded patterns.

This patch provides a naive implementation where the ref_filter still sees all references (including ones that it will discard) and is left to check whether each reference matches any excluded pattern(s) before emitting them.

By culling out references we know the caller doesn't care about, we can avoid allocating memory for their storage, as well as spending time sorting the output (among other things). Even the naive implementation provides a significant speed-up on a modified copy of linux.git (that has a hidden ref pointing at each commit):

    $ hyperfine \
      'git.compile for-each-ref --format="%(objectname) %(refname)" | grep -vE "[0-9a-f]{40} refs/pull/"' \
      'git.compile for-each-ref --format="%(objectname) %(refname)" --exclude refs/pull/'
    Benchmark 1: git.compile for-each-ref --format="%(objectname) %(refname)" | grep -vE "[0-9a-f]{40} refs/pull/"
      Time (mean ± σ):     820.1 ms ±   2.0 ms    [User: 703.7 ms, System: 152.0 ms]
      Range (min … max):   817.7 ms … 823.3 ms    10 runs
    Benchmark 2: git.compile for-each-ref --format="%(objectname) %(refname)" --exclude refs/pull/
      Time (mean ± σ):     106.6 ms ±   1.1 ms    [User: 99.4 ms, System: 7.1 ms]
      Range (min … max):   104.7 ms … 109.1 ms    27 runs

    Summary
      'git.compile for-each-ref --format="%(objectname) %(refname)" --exclude refs/pull/' ran
        7.69 ± 0.08 times faster than 'git.compile for-each-ref --format="%(objectname) %(refname)" | grep -vE "[0-9a-f]{40} refs/pull/"'

Subsequent patches will improve on this by avoiding visiting excluded sections of the packed-refs file in certain cases.

---
 Documentation/git-for-each-ref.txt |  6 +++++
 builtin/for-each-ref.c             |  1 +
 ref-filter.c                       | 13 +++++++++++
 ref-filter.h                       |  6 +++++
 t/t6300-for-each-ref.sh            | 35 ++++++++++++++++++++++++++++++
 5 files changed, 61 insertions(+)
Documentation/git-for-each-ref.txt +6 -0
index 1e215d4e73..5743eb5def 100644
@@ -14,6 +14,7 @@ SYNOPSIS
 		   [--points-at=<object>]
 		   [--merged[=<object>]] [--no-merged[=<object>]]
 		   [--contains[=<object>]] [--no-contains[=<object>]]
+		   [--exclude=<pattern> ...]
 
 DESCRIPTION
 -----------
@@ -102,6 +103,11 @@ OPTIONS
 	Do not print a newline after formatted refs where the format expands
 	to the empty string.
 
+--exclude=<pattern>::
+	If one or more patterns are given, only refs which do not match
+	any excluded pattern(s) are shown. Matching is done using the
+	same rules as `<pattern>` above.
+
 FIELD NAMES
 -----------
 
builtin/for-each-ref.c +1 -0
index c01fa6fefe..3384987428 100644
@@ -47,6 +47,7 @@ int cmd_for_each_ref(int argc, const char **argv, const char *prefix)
 		OPT_INTEGER( 0 , "count", &maxcount, N_("show only <n> matched refs")),
 		OPT_STRING(  0 , "format", &format.format, N_("format"), N_("format to use for the output")),
 		OPT__COLOR(&format.use_color, N_("respect format colors")),
+		OPT_REF_FILTER_EXCLUDE(&filter),
 		OPT_REF_SORT(&sorting_options),
 		OPT_CALLBACK(0, "points-at", &filter.points_at,
 			     N_("object"), N_("print only refs which points at the given object"),
ref-filter.c +13 -0
index 6d91c7cb0d..d44418efb7 100644
@@ -2171,6 +2171,15 @@ static int filter_pattern_match(struct ref_filter *filter, const char *refname)
 			     filter->ignore_case);
 }
 
+static int filter_exclude_match(struct ref_filter *filter, const char *refname)
+{
+	if (!filter->exclude.nr)
+		return 0;
+	if (filter->match_as_path)
+		return match_name_as_path(filter, filter->exclude.v, refname);
+	return match_pattern(filter->exclude.v, refname, filter->ignore_case);
+}
+
 /*
  * This is the same as for_each_fullref_in(), but it tries to iterate
  * only over the patterns we'll care about. Note that it _doesn't_ do a full
@@ -2338,6 +2347,9 @@ static int ref_filter_handler(const char *refname, const struct object_id *oid,
 	if (!filter_pattern_match(filter, refname))
 		return 0;
 
+	if (filter_exclude_match(filter, refname))
+		return 0;
+
 	if (filter->points_at.nr && !match_points_at(&filter->points_at, oid, refname))
 		return 0;
 
@@ -2877,6 +2889,7 @@ void ref_filter_init(struct ref_filter *filter)
 
 void ref_filter_clear(struct ref_filter *filter)
 {
+	strvec_clear(&filter->exclude);
 	oid_array_clear(&filter->points_at);
 	free_commit_list(filter->with_commit);
 	free_commit_list(filter->no_commit);
ref-filter.h +6 -0
index 160b807224..1524bc463a 100644
@@ -6,6 +6,7 @@
 #include "refs.h"
 #include "commit.h"
 #include "string-list.h"
+#include "strvec.h"
 
 /* Quoting styles */
 #define QUOTE_NONE 0
@@ -59,6 +60,7 @@ struct ref_array {
 
 struct ref_filter {
 	const char **name_patterns;
+	struct strvec exclude;
 	struct oid_array points_at;
 	struct commit_list *with_commit;
 	struct commit_list *no_commit;
@@ -94,6 +96,7 @@ struct ref_format {
 
 #define REF_FILTER_INIT { \
 	.points_at = OID_ARRAY_INIT, \
+	.exclude = STRVEC_INIT, \
 }
 #define REF_FORMAT_INIT {             \
 	.use_color = -1,              \
@@ -112,6 +115,9 @@ struct ref_format {
 #define OPT_REF_SORT(var) \
 	OPT_STRING_LIST(0, "sort", (var), \
 			N_("key"), N_("field name to sort on"))
+#define OPT_REF_FILTER_EXCLUDE(var) \
+	OPT_STRVEC(0, "exclude", &(var)->exclude, \
+		   N_("pattern"), N_("exclude refs which match pattern"))
 
 /*
  * API for filtering a set of refs. Based on the type of refs the user
t/t6300-for-each-ref.sh +35 -0
index 5c00607608..7e8d578522 100755
@@ -447,6 +447,41 @@ test_expect_success 'exercise glob patterns with prefixes' '
 	test_cmp expected actual
 '
 
+cat >expected <<\EOF
+refs/tags/bar
+refs/tags/baz
+refs/tags/testtag
+EOF
+
+test_expect_success 'exercise patterns with prefix exclusions' '
+	for tag in foo/one foo/two foo/three bar baz
+	do
+		git tag "$tag" || return 1
+	done &&
+	test_when_finished "git tag -d foo/one foo/two foo/three bar baz" &&
+	git for-each-ref --format="%(refname)" \
+		refs/tags/ --exclude=refs/tags/foo >actual &&
+	test_cmp expected actual
+'
+
+cat >expected <<\EOF
+refs/tags/bar
+refs/tags/baz
+refs/tags/foo/one
+refs/tags/testtag
+EOF
+
+test_expect_success 'exercise patterns with pattern exclusions' '
+	for tag in foo/one foo/two foo/three bar baz
+	do
+		git tag "$tag" || return 1
+	done &&
+	test_when_finished "git tag -d foo/one foo/two foo/three bar baz" &&
+	git for-each-ref --format="%(refname)" \
+		refs/tags/ --exclude="refs/tags/foo/t*" >actual &&
+	test_cmp expected actual
+'
+
 cat >expected <<\EOF
 'refs/heads/main'
 'refs/remotes/origin/main'
Signature · 1 line
2.40.1.572.g5c4ab523ef
#45 lore raw

PATCH v2 7/16 refs: plumb `exclude_patterns` argument throughout

In reply to Taylor Blau

The subsequent patch will want to access an optional excluded_patterns array within refs/packed-backend.c that will cull out certain references matching any of the given patterns on a best-effort basis.

To do so, the refs subsystem needs to be updated to pass this value across a number of different locations.

Prepare for a future patch by introducing this plumbing now, passing NULLs at top-level APIs in order to make that patch less noisy and more easily readable.

---
 ls-refs.c             |  2 +-
 ref-filter.c          |  5 +++--
 refs.c                | 32 +++++++++++++++++++-------------
 refs.h                |  8 +++++++-
 refs/debug.c          |  5 +++--
 refs/files-backend.c  |  5 +++--
 refs/packed-backend.c |  5 +++--
 refs/refs-internal.h  |  7 ++++---
 revision.c            |  2 +-
 9 files changed, 44 insertions(+), 27 deletions(-)
ls-refs.c +1 -1
index f385938b64..6f490b2d9c 100644
@@ -193,7 +193,7 @@ int ls_refs(struct repository *r, struct packet_reader *request)
 		strvec_push(&data.prefixes, "");
 	refs_for_each_fullref_in_prefixes(get_main_ref_store(r),
 					  get_git_namespace(), data.prefixes.v,
-					  send_ref, &data);
+					  NULL, send_ref, &data);
 	packet_fflush(stdout);
 	strvec_clear(&data.prefixes);
 	strbuf_release(&data.buf);
ref-filter.c +3 -2
index d44418efb7..717c3c4bcf 100644
@@ -2209,12 +2209,13 @@ static int for_each_fullref_in_pattern(struct ref_filter *filter,
 
 	if (!filter->name_patterns[0]) {
 		/* no patterns; we have to look at everything */
-		return for_each_fullref_in("", cb, cb_data);
+		return refs_for_each_fullref_in(get_main_ref_store(the_repository),
+						 "", NULL, cb, cb_data);
 	}
 
 	return refs_for_each_fullref_in_prefixes(get_main_ref_store(the_repository),
 						 NULL, filter->name_patterns,
-						 cb, cb_data);
+						 NULL, cb, cb_data);
 }
 
 /*
refs.c +19 -13
index b9b77d2eff..538bde644e 100644
@@ -1526,7 +1526,9 @@ int head_ref(each_ref_fn fn, void *cb_data)
 
 struct ref_iterator *refs_ref_iterator_begin(
 		struct ref_store *refs,
-		const char *prefix, int trim,
+		const char *prefix,
+		const char **exclude_patterns,
+		int trim,
 		enum do_for_each_ref_flags flags)
 {
 	struct ref_iterator *iter;
@@ -1542,8 +1544,7 @@ struct ref_iterator *refs_ref_iterator_begin(
 		}
 	}
 
-	iter = refs->be->iterator_begin(refs, prefix, flags);
-
+	iter = refs->be->iterator_begin(refs, prefix, exclude_patterns, flags);
 	/*
 	 * `iterator_begin()` already takes care of prefix, but we
 	 * might need to do some trimming:
@@ -1577,7 +1578,7 @@ static int do_for_each_repo_ref(struct repository *r, const char *prefix,
 	if (!refs)
 		return 0;
 
-	iter = refs_ref_iterator_begin(refs, prefix, trim, flags);
+	iter = refs_ref_iterator_begin(refs, prefix, NULL, trim, flags);
 
 	return do_for_each_repo_ref_iterator(r, iter, fn, cb_data);
 }
@@ -1599,6 +1600,7 @@ static int do_for_each_ref_helper(struct repository *r,
 }
 
 static int do_for_each_ref(struct ref_store *refs, const char *prefix,
+			   const char **exclude_patterns,
 			   each_ref_fn fn, int trim,
 			   enum do_for_each_ref_flags flags, void *cb_data)
 {
@@ -1608,7 +1610,8 @@ static int do_for_each_ref(struct ref_store *refs, const char *prefix,
 	if (!refs)
 		return 0;
 
-	iter = refs_ref_iterator_begin(refs, prefix, trim, flags);
+	iter = refs_ref_iterator_begin(refs, prefix, exclude_patterns, trim,
+				       flags);
 
 	return do_for_each_repo_ref_iterator(the_repository, iter,
 					do_for_each_ref_helper, &hp);
@@ -1616,7 +1619,7 @@ static int do_for_each_ref(struct ref_store *refs, const char *prefix,
 
 int refs_for_each_ref(struct ref_store *refs, each_ref_fn fn, void *cb_data)
 {
-	return do_for_each_ref(refs, "", fn, 0, 0, cb_data);
+	return do_for_each_ref(refs, "", NULL, fn, 0, 0, cb_data);
 }
 
 int for_each_ref(each_ref_fn fn, void *cb_data)
@@ -1627,7 +1630,7 @@ int for_each_ref(each_ref_fn fn, void *cb_data)
 int refs_for_each_ref_in(struct ref_store *refs, const char *prefix,
 			 each_ref_fn fn, void *cb_data)
 {
-	return do_for_each_ref(refs, prefix, fn, strlen(prefix), 0, cb_data);
+	return do_for_each_ref(refs, prefix, NULL, fn, strlen(prefix), 0, cb_data);
 }
 
 int for_each_ref_in(const char *prefix, each_ref_fn fn, void *cb_data)
@@ -1638,13 +1641,14 @@ int for_each_ref_in(const char *prefix, each_ref_fn fn, void *cb_data)
 int for_each_fullref_in(const char *prefix, each_ref_fn fn, void *cb_data)
 {
 	return do_for_each_ref(get_main_ref_store(the_repository),
-			       prefix, fn, 0, 0, cb_data);
+			       prefix, NULL, fn, 0, 0, cb_data);
 }
 
 int refs_for_each_fullref_in(struct ref_store *refs, const char *prefix,
+			     const char **exclude_patterns,
 			     each_ref_fn fn, void *cb_data)
 {
-	return do_for_each_ref(refs, prefix, fn, 0, 0, cb_data);
+	return do_for_each_ref(refs, prefix, exclude_patterns, fn, 0, 0, cb_data);
 }
 
 int for_each_replace_ref(struct repository *r, each_repo_ref_fn fn, void *cb_data)
@@ -1661,14 +1665,14 @@ int for_each_namespaced_ref(each_ref_fn fn, void *cb_data)
 	int ret;
 	strbuf_addf(&buf, "%srefs/", get_git_namespace());
 	ret = do_for_each_ref(get_main_ref_store(the_repository),
-			      buf.buf, fn, 0, 0, cb_data);
+			      buf.buf, NULL, fn, 0, 0, cb_data);
 	strbuf_release(&buf);
 	return ret;
 }
 
 int refs_for_each_rawref(struct ref_store *refs, each_ref_fn fn, void *cb_data)
 {
-	return do_for_each_ref(refs, "", fn, 0,
+	return do_for_each_ref(refs, "", NULL, fn, 0,
 			       DO_FOR_EACH_INCLUDE_BROKEN, cb_data);
 }
 
@@ -1738,6 +1742,7 @@ static void find_longest_prefixes(struct string_list *out,
 int refs_for_each_fullref_in_prefixes(struct ref_store *ref_store,
 				      const char *namespace,
 				      const char **patterns,
+				      const char **exclude_patterns,
 				      each_ref_fn fn, void *cb_data)
 {
 	struct string_list prefixes = STRING_LIST_INIT_DUP;
@@ -1753,7 +1758,8 @@ int refs_for_each_fullref_in_prefixes(struct ref_store *ref_store,
 
 	for_each_string_list_item(prefix, &prefixes) {
 		strbuf_addstr(&buf, prefix->string);
-		ret = refs_for_each_fullref_in(ref_store, buf.buf, fn, cb_data);
+		ret = refs_for_each_fullref_in(ref_store, buf.buf,
+					       exclude_patterns, fn, cb_data);
 		if (ret)
 			break;
 		strbuf_setlen(&buf, namespace_len);
@@ -2408,7 +2414,7 @@ int refs_verify_refname_available(struct ref_store *refs,
 	strbuf_addstr(&dirname, refname + dirname.len);
 	strbuf_addch(&dirname, '/');
 
-	iter = refs_ref_iterator_begin(refs, dirname.buf, 0,
+	iter = refs_ref_iterator_begin(refs, dirname.buf, NULL, 0,
 				       DO_FOR_EACH_INCLUDE_BROKEN);
 	while ((ok = ref_iterator_advance(iter)) == ITER_OK) {
 		if (skip &&
refs.h +7 -1
index 123cfa4424..d672d636cf 100644
@@ -338,6 +338,7 @@ int for_each_ref(each_ref_fn fn, void *cb_data);
 int for_each_ref_in(const char *prefix, each_ref_fn fn, void *cb_data);
 
 int refs_for_each_fullref_in(struct ref_store *refs, const char *prefix,
+			     const char **exclude_patterns,
 			     each_ref_fn fn, void *cb_data);
 int for_each_fullref_in(const char *prefix, each_ref_fn fn, void *cb_data);
 
@@ -345,10 +346,15 @@ int for_each_fullref_in(const char *prefix, each_ref_fn fn, void *cb_data);
  * iterate all refs in "patterns" by partitioning patterns into disjoint sets
  * and iterating the longest-common prefix of each set.
  *
+ * references matching any pattern in "exclude_patterns" are omitted from the
+ * result set on a best-effort basis.
+ *
  * callers should be prepared to ignore references that they did not ask for.
  */
 int refs_for_each_fullref_in_prefixes(struct ref_store *refs,
-				      const char *namespace, const char **patterns,
+				      const char *namespace,
+				      const char **patterns,
+				      const char **exclude_patterns,
 				      each_ref_fn fn, void *cb_data);
 
 /**
refs/debug.c +3 -2
index 6f11e6de46..328f894177 100644
@@ -229,11 +229,12 @@ static struct ref_iterator_vtable debug_ref_iterator_vtable = {
 
 static struct ref_iterator *
 debug_ref_iterator_begin(struct ref_store *ref_store, const char *prefix,
-			 unsigned int flags)
+			 const char **exclude_patterns, unsigned int flags)
 {
 	struct debug_ref_store *drefs = (struct debug_ref_store *)ref_store;
 	struct ref_iterator *res =
-		drefs->refs->be->iterator_begin(drefs->refs, prefix, flags);
+		drefs->refs->be->iterator_begin(drefs->refs, prefix,
+						exclude_patterns, flags);
 	struct debug_ref_iterator *diter = xcalloc(1, sizeof(*diter));
 	base_ref_iterator_init(&diter->base, &debug_ref_iterator_vtable, 1);
 	diter->iter = res;
refs/files-backend.c +3 -2
index bca7b851c5..3bc3c57c05 100644
@@ -829,7 +829,8 @@ static struct ref_iterator_vtable files_ref_iterator_vtable = {
 
 static struct ref_iterator *files_ref_iterator_begin(
 		struct ref_store *ref_store,
-		const char *prefix, unsigned int flags)
+		const char *prefix, const char **exclude_patterns,
+		unsigned int flags)
 {
 	struct files_ref_store *refs;
 	struct ref_iterator *loose_iter, *packed_iter, *overlay_iter;
@@ -874,7 +875,7 @@ static struct ref_iterator *files_ref_iterator_begin(
 	 * the packed and loose references.
 	 */
 	packed_iter = refs_ref_iterator_begin(
-			refs->packed_ref_store, prefix, 0,
+			refs->packed_ref_store, prefix, exclude_patterns, 0,
 			DO_FOR_EACH_INCLUDE_BROKEN);
 
 	overlay_iter = overlay_ref_iterator_begin(loose_iter, packed_iter);
refs/packed-backend.c +3 -2
index 5b412a133b..176bd3905b 100644
@@ -924,7 +924,8 @@ static struct ref_iterator_vtable packed_ref_iterator_vtable = {
 
 static struct ref_iterator *packed_ref_iterator_begin(
 		struct ref_store *ref_store,
-		const char *prefix, unsigned int flags)
+		const char *prefix, const char **exclude_patterns,
+		unsigned int flags)
 {
 	struct packed_ref_store *refs;
 	struct snapshot *snapshot;
@@ -1149,7 +1150,7 @@ static int write_with_updates(struct packed_ref_store *refs,
 	 * list of refs is exhausted, set iter to NULL. When the list
 	 * of updates is exhausted, leave i set to updates->nr.
 	 */
-	iter = packed_ref_iterator_begin(&refs->base, "",
+	iter = packed_ref_iterator_begin(&refs->base, "", NULL,
 					 DO_FOR_EACH_INCLUDE_BROKEN);
 	if ((ok = ref_iterator_advance(iter)) != ITER_OK)
 		iter = NULL;
refs/refs-internal.h +4 -3
index a85d113123..28a11b9d61 100644
@@ -367,8 +367,8 @@ int is_empty_ref_iterator(struct ref_iterator *ref_iterator);
  */
 struct ref_iterator *refs_ref_iterator_begin(
 		struct ref_store *refs,
-		const char *prefix, int trim,
-		enum do_for_each_ref_flags flags);
+		const char *prefix, const char **exclude_patterns,
+		int trim, enum do_for_each_ref_flags flags);
 
 /*
  * A callback function used to instruct merge_ref_iterator how to
@@ -570,7 +570,8 @@ typedef int copy_ref_fn(struct ref_store *ref_store,
  */
 typedef struct ref_iterator *ref_iterator_begin_fn(
 		struct ref_store *ref_store,
-		const char *prefix, unsigned int flags);
+		const char *prefix, const char **exclude_patterns,
+		unsigned int flags);
 
 /* reflog functions */
 
revision.c +1 -1
index b33cc1d106..89953592f9 100644
@@ -2670,7 +2670,7 @@ static int for_each_bisect_ref(struct ref_store *refs, each_ref_fn fn,
 	struct strbuf bisect_refs = STRBUF_INIT;
 	int status;
 	strbuf_addf(&bisect_refs, "refs/bisect/%s", term);
-	status = refs_for_each_fullref_in(refs, bisect_refs.buf, fn, cb_data);
+	status = refs_for_each_fullref_in(refs, bisect_refs.buf, NULL, fn, cb_data);
 	strbuf_release(&bisect_refs);
 	return status;
 }
Signature · 1 line
2.40.1.572.g5c4ab523ef
#46 lore raw

PATCH v2 8/16 refs/packed-backend.c: refactor `find_reference_location()`

In reply to Taylor Blau

The function find_reference_location() is used to perform a binary search-like function over the contents of a repository's $GIT_DIR/packed-refs file.

The search it implements is unlike a standard binary search in that the records it searches over are not of a fixed width, so the comparison must locate the end of a record before comparing it.

Extract the core routine of find_reference_location() in order to implement a function in the following patch which will find the first location in the packed-refs file that *doesn't* match the given pattern.

The behavior of find_reference_location() is unchanged.

---
 refs/packed-backend.c | 38 ++++++++++++++++++++++----------------
 1 file changed, 22 insertions(+), 16 deletions(-)
refs/packed-backend.c +22 -16
index 176bd3905b..33639f73e1 100644
@@ -527,22 +527,8 @@ static int load_contents(struct snapshot *snapshot)
 	return 1;
 }
 
-/*
- * Find the place in `snapshot->buf` where the start of the record for
- * `refname` starts. If `mustexist` is true and the reference doesn't
- * exist, then return NULL. If `mustexist` is false and the reference
- * doesn't exist, then return the point where that reference would be
- * inserted, or `snapshot->eof` (which might be NULL) if it would be
- * inserted at the end of the file. In the latter mode, `refname`
- * doesn't have to be a proper reference name; for example, one could
- * search for "refs/replace/" to find the start of any replace
- * references.
- *
- * The record is sought using a binary search, so `snapshot->buf` must
- * be sorted.
- */
-static const char *find_reference_location(struct snapshot *snapshot,
-					   const char *refname, int mustexist)
+static const char *find_reference_location_1(struct snapshot *snapshot,
+					     const char *refname, int mustexist)
 {
 	/*
 	 * This is not *quite* a garden-variety binary search, because
@@ -588,6 +574,26 @@ static const char *find_reference_location(struct snapshot *snapshot,
 		return lo;
 }
 
+/*
+ * Find the place in `snapshot->buf` where the start of the record for
+ * `refname` starts. If `mustexist` is true and the reference doesn't
+ * exist, then return NULL. If `mustexist` is false and the reference
+ * doesn't exist, then return the point where that reference would be
+ * inserted, or `snapshot->eof` (which might be NULL) if it would be
+ * inserted at the end of the file. In the latter mode, `refname`
+ * doesn't have to be a proper reference name; for example, one could
+ * search for "refs/replace/" to find the start of any replace
+ * references.
+ *
+ * The record is sought using a binary search, so `snapshot->buf` must
+ * be sorted.
+ */
+static const char *find_reference_location(struct snapshot *snapshot,
+					   const char *refname, int mustexist)
+{
+	return find_reference_location_1(snapshot, refname, mustexist);
+}
+
 /*
  * Create a newly-allocated `snapshot` of the `packed-refs` file in
  * its current state and return it. The return value will already have
Signature · 1 line
2.40.1.572.g5c4ab523ef
#47 lore raw

PATCH v2 9/16 refs/packed-backend.c: implement jump lists to avoid excluded pattern(s)

In reply to Taylor Blau

When iterating through the packed-refs file in order to answer a query like:

    $ git for-each-ref --exclude=refs/__hidden__

it would be useful to avoid walking over all of the entries in refs/__hidden__/* when possible, since we know that the ref-filter code is going to throw them away anyways.

In certain circumstances, doing so is possible. The algorithm for doing so is as follows:

- For each excluded pattern, find the first record that matches it, and the first record that *doesn't* match it (i.e. the location you'd next want to consider when excluding that pattern).

- Sort the set of excluded regions from the previous step in ascending order of the first location within the packed-refs file that matches.

- Clean up the results from the previous step: discard empty regions, and combine adjacent regions.

Then when iterating through the packed-refs file, if iter->pos is ever contained in one of the regions from the previous steps, advance iter->pos past the end of that region, and continue enumeration.

Note that we only perform this optimization when none of the excluded pattern(s) have special meta-characters in them. For a pattern like "refs/foo[ac]", the excluded regions ("refs/fooa", "refs/fooc", and everything underneath them) are not connected. A future implementation that handles this case may split the character class (pretending as if two patterns were excluded: "refs/fooa", and "refs/fooc").

There are a few other gotchas worth considering. First, note that the jump list is sorted, so once we jump past a region, we can avoid considering it (or any regions preceding it) again. The member jump_pos is used to track the first next-possible region to jump through.

Second, note that the exclusion list is best-effort, since we do not handle loose references, and because of the meta-character issue above.

In repositories with a large number of hidden references, the speed-up can be significant. Tests here are done with a copy of linux.git with a reference "refs/pull/N" pointing at every commit, as in:

    $ git rev-list HEAD | awk '{ print "create refs/pull/" NR " " $0 }' |
        git update-ref --stdin
    $ git pack-refs --all

, it is significantly faster to have for-each-ref jump over the excluded references, as opposed to filtering them out after the fact:

    $ hyperfine \
      'git for-each-ref --format="%(objectname) %(refname)" | grep -vE "^[0-9a-f]{40} refs/pull/"' \
      'git.compile for-each-ref --format="%(objectname) %(refname)" --exclude="refs/pull"'
    Benchmark 1: git for-each-ref --format="%(objectname) %(refname)" | grep -vE "^[0-9a-f]{40} refs/pull/"
      Time (mean ± σ):     802.7 ms ±   2.1 ms    [User: 691.6 ms, System: 147.0 ms]
      Range (min … max):   800.0 ms … 807.7 ms    10 runs
    Benchmark 2: git.compile for-each-ref --format="%(objectname) %(refname)" --exclude="refs/pull"
      Time (mean ± σ):       4.7 ms ±   0.3 ms    [User: 0.7 ms, System: 4.0 ms]
      Range (min … max):     4.3 ms …   6.7 ms    422 runs

    Summary
      'git.compile for-each-ref --format="%(objectname) %(refname)" --exclude="refs/pull"' ran
      172.03 ± 9.60 times faster than 'git for-each-ref --format="%(objectname) %(refname)" | grep -vE "^[0-9a-f]{40} refs/pull/"'

Using the jump list is fairly straightforward (see the changes to refs/packed-backend.c::next_record()), but constructing the list is not. To ensure that the construction is correct, add a new suite of tests in t1419 covering various corner cases (overlapping regions, partially overlapping regions, adjacent regions, etc.).

---
 ref-filter.c              |   5 +-
 refs/packed-backend.c     | 166 ++++++++++++++++++++++++++++++++++++--
 t/helper/test-ref-store.c |  10 +++
 t/t1419-exclude-refs.sh   | 101 +++++++++++++++++++++++
 4 files changed, 274 insertions(+), 8 deletions(-)
 create mode 100755 t/t1419-exclude-refs.sh
ref-filter.c +3 -2
index 717c3c4bcf..ddc7f5204f 100644
@@ -2210,12 +2210,13 @@ static int for_each_fullref_in_pattern(struct ref_filter *filter,
 	if (!filter->name_patterns[0]) {
 		/* no patterns; we have to look at everything */
 		return refs_for_each_fullref_in(get_main_ref_store(the_repository),
-						 "", NULL, cb, cb_data);
+						 "", filter->exclude.v, cb, cb_data);
 	}
 
 	return refs_for_each_fullref_in_prefixes(get_main_ref_store(the_repository),
 						 NULL, filter->name_patterns,
-						 NULL, cb, cb_data);
+						 filter->exclude.v,
+						 cb, cb_data);
 }
 
 /*
refs/packed-backend.c +160 -6

233 lines not on this page · raw message

t/helper/test-ref-store.c +10 -0
index 6d8f844e9c..2bff003f7c 100644
@@ -175,6 +175,15 @@ static int cmd_for_each_ref(struct ref_store *refs, const char **argv)
 	return refs_for_each_ref_in(refs, prefix, each_ref, NULL);
 }
 
+static int cmd_for_each_ref__exclude(struct ref_store *refs, const char **argv)
+{
+	const char *prefix = notnull(*argv++, "prefix");
+	const char **exclude_patterns = argv;
+
+	return refs_for_each_fullref_in(refs, prefix, exclude_patterns, each_ref,
+					NULL);
+}
+
 static int cmd_resolve_ref(struct ref_store *refs, const char **argv)
 {
 	struct object_id oid = *null_oid();
@@ -307,6 +316,7 @@ static struct command commands[] = {
 	{ "delete-refs", cmd_delete_refs },
 	{ "rename-ref", cmd_rename_ref },
 	{ "for-each-ref", cmd_for_each_ref },
+	{ "for-each-ref--exclude", cmd_for_each_ref__exclude },
 	{ "resolve-ref", cmd_resolve_ref },
 	{ "verify-ref", cmd_verify_ref },
 	{ "for-each-reflog", cmd_for_each_reflog },
t/t1419-exclude-refs.sh new file +101 -0

101 lines not on this page · raw message

Signature · 1 line
2.40.1.572.g5c4ab523ef
#50 lore raw

PATCH v2 10/16 refs/packed-backend.c: add trace2 counters for jump list

In reply to Taylor Blau

The previous commit added low-level tests to ensure that the packed-refs iterator did not enumerate excluded sections of the refspace.

However, there was no guarantee that these sections weren't being visited, only that they were being suppressed from the output. To harden these tests, add a trace2 counter which tracks the number of regions skipped by the packed-refs iterator, and assert on its value.

---
 refs/packed-backend.c   |  2 ++
 t/t1419-exclude-refs.sh | 59 ++++++++++++++++++++++++++++-------------
 trace2.h                |  2 ++
 trace2/tr2_ctr.c        |  5 ++++
 4 files changed, 49 insertions(+), 19 deletions(-)
refs/packed-backend.c +2 -0
index 67327e579c..7ba9fa2bb8 100644
@@ -12,6 +12,7 @@
 #include "../chdir-notify.h"
 #include "../wrapper.h"
 #include "../write-or-die.h"
+#include "../trace2.h"
 
 enum mmap_strategy {
 	/*
@@ -845,6 +846,7 @@ static int next_record(struct packed_ref_iterator *iter)
 		iter->jump_pos++;
 		if (iter->pos < curr->end) {
 			iter->pos = curr->end;
+			trace2_counter_add(TRACE2_COUNTER_ID_PACKED_REFS_JUMPS, 1);
 			break;
 		}
 	}
t/t1419-exclude-refs.sh +40 -19

120 lines not on this page · raw message

trace2.h +2 -0
index 4ced30c0db..9452e291f5 100644
@@ -551,6 +551,8 @@ enum trace2_counter_id {
 	TRACE2_COUNTER_ID_TEST1 = 0, /* emits summary event only */
 	TRACE2_COUNTER_ID_TEST2,     /* emits summary and thread events */
 
+	TRACE2_COUNTER_ID_PACKED_REFS_JUMPS, /* counts number of jumps */
+
 	/* Add additional counter definitions before here. */
 	TRACE2_NUMBER_OF_COUNTERS
 };
trace2/tr2_ctr.c +5 -0
index b342d3b1a3..50570d0165 100644
@@ -27,6 +27,11 @@ static struct tr2_counter_metadata tr2_counter_metadata[TRACE2_NUMBER_OF_COUNTER
 		.name = "test2",
 		.want_per_thread_events = 1,
 	},
+	[TRACE2_COUNTER_ID_PACKED_REFS_JUMPS] = {
+		.category = "packed-refs",
+		.name = "jumps_made",
+		.want_per_thread_events = 0,
+	},
 
 	/* Add additional metadata before here. */
 };
Signature · 1 line
2.40.1.572.g5c4ab523ef
#51 lore raw

PATCH v2 11/16 revision.h: store hidden refs in a `strvec`

In reply to Taylor Blau

In subsequent commits, it will be convenient to have a 'const char **' of hidden refs (matching transfer.hiderefs, uploadpack.hideRefs, etc.), instead of a string_list.

Convert spots throughout the tree that store the list of hidden refs from a string_list to a strvec.

Note that in parse_hide_refs_config() there is an ugly const-cast used to avoid an extra copy of each value before trimming any trailing slash characters. This could instead be written as:

    ref = xstrdup(value);
    len = strlen(ref);
    while (len && ref[len - 1] == '/')
            ref[--len] = '\0';
    strvec_push(hide_refs, ref);
    free(ref);

but the double-copy (once when calling xstrdup(), and another via strvec_push()) is wasteful.

---
 builtin/receive-pack.c |  4 ++--
 ls-refs.c              |  6 +++---
 refs.c                 | 11 ++++++-----
 refs.h                 |  4 ++--
 revision.c             |  2 +-
 revision.h             |  5 +++--
 upload-pack.c          | 10 +++++-----
 7 files changed, 22 insertions(+), 20 deletions(-)
builtin/receive-pack.c +2 -2
index 1a31a58367..1a8472eddc 100644
@@ -90,7 +90,7 @@ static struct object_id push_cert_oid;
 static struct signature_check sigcheck;
 static const char *push_cert_nonce;
 static const char *cert_nonce_seed;
-static struct string_list hidden_refs = STRING_LIST_INIT_DUP;
+static struct strvec hidden_refs = STRVEC_INIT;
 
 static const char *NONCE_UNSOLICITED = "UNSOLICITED";
 static const char *NONCE_BAD = "BAD";
@@ -2619,7 +2619,7 @@ int cmd_receive_pack(int argc, const char **argv, const char *prefix)
 		packet_flush(1);
 	oid_array_clear(&shallow);
 	oid_array_clear(&ref);
-	string_list_clear(&hidden_refs, 0);
+	strvec_clear(&hidden_refs);
 	free((void *)push_cert_nonce);
 	return 0;
 }
ls-refs.c +3 -3
index 6f490b2d9c..8c3181d051 100644
@@ -72,7 +72,7 @@ struct ls_refs_data {
 	unsigned symrefs;
 	struct strvec prefixes;
 	struct strbuf buf;
-	struct string_list hidden_refs;
+	struct strvec hidden_refs;
 	unsigned unborn : 1;
 };
 
@@ -155,7 +155,7 @@ int ls_refs(struct repository *r, struct packet_reader *request)
 	memset(&data, 0, sizeof(data));
 	strvec_init(&data.prefixes);
 	strbuf_init(&data.buf, 0);
-	string_list_init_dup(&data.hidden_refs);
+	strvec_init(&data.hidden_refs);
 
 	git_config(ls_refs_config, &data);
 
@@ -197,7 +197,7 @@ int ls_refs(struct repository *r, struct packet_reader *request)
 	packet_fflush(stdout);
 	strvec_clear(&data.prefixes);
 	strbuf_release(&data.buf);
-	string_list_clear(&data.hidden_refs, 0);
+	strvec_clear(&data.hidden_refs);
 	return 0;
 }
 
refs.c +6 -5
index 538bde644e..ec4d5b9101 100644
@@ -1427,7 +1427,7 @@ char *shorten_unambiguous_ref(const char *refname, int strict)
 }
 
 int parse_hide_refs_config(const char *var, const char *value, const char *section,
-			   struct string_list *hide_refs)
+			   struct strvec *hide_refs)
 {
 	const char *key;
 	if (!strcmp("transfer.hiderefs", var) ||
@@ -1438,22 +1438,23 @@ int parse_hide_refs_config(const char *var, const char *value, const char *secti
 
 		if (!value)
 			return config_error_nonbool(var);
-		ref = xstrdup(value);
+
+		/* drop const to remove trailing '/' characters */
+		ref = (char *)strvec_push(hide_refs, value);
 		len = strlen(ref);
 		while (len && ref[len - 1] == '/')
 			ref[--len] = '\0';
-		string_list_append_nodup(hide_refs, ref);
 	}
 	return 0;
 }
 
 int ref_is_hidden(const char *refname, const char *refname_full,
-		  const struct string_list *hide_refs)
+		  const struct strvec *hide_refs)
 {
 	int i;
 
 	for (i = hide_refs->nr - 1; i >= 0; i--) {
-		const char *match = hide_refs->items[i].string;
+		const char *match = hide_refs->v[i];
 		const char *subject;
 		int neg = 0;
 		const char *p;
refs.h +2 -2
index d672d636cf..a7751a1fc9 100644
@@ -810,7 +810,7 @@ int update_ref(const char *msg, const char *refname,
 	       unsigned int flags, enum action_on_err onerr);
 
 int parse_hide_refs_config(const char *var, const char *value, const char *,
-			   struct string_list *);
+			   struct strvec *);
 
 /*
  * Check whether a ref is hidden. If no namespace is set, both the first and
@@ -820,7 +820,7 @@ int parse_hide_refs_config(const char *var, const char *value, const char *,
  * the ref is outside that namespace, the first parameter is NULL. The second
  * parameter always points to the full ref name.
  */
-int ref_is_hidden(const char *, const char *, const struct string_list *);
+int ref_is_hidden(const char *, const char *, const struct strvec *);
 
 /* Is this a per-worktree ref living in the refs/ namespace? */
 int is_per_worktree_ref(const char *refname);
revision.c +1 -1
index 89953592f9..7c9367a266 100644
@@ -1558,7 +1558,7 @@ void init_ref_exclusions(struct ref_exclusions *exclusions)
 void clear_ref_exclusions(struct ref_exclusions *exclusions)
 {
 	string_list_clear(&exclusions->excluded_refs, 0);
-	string_list_clear(&exclusions->hidden_refs, 0);
+	strvec_clear(&exclusions->hidden_refs);
 	exclusions->hidden_refs_configured = 0;
 }
 
revision.h +3 -2
index 31828748dc..94f035fa22 100644
@@ -10,6 +10,7 @@
 #include "decorate.h"
 #include "ident.h"
 #include "list-objects-filter-options.h"
+#include "strvec.h"
 
 /**
  * The revision walking API offers functions to build a list of revisions
@@ -95,7 +96,7 @@ struct ref_exclusions {
 	 * Hidden refs is a list of patterns that is to be hidden via
 	 * `ref_is_hidden()`.
 	 */
-	struct string_list hidden_refs;
+	struct strvec hidden_refs;
 
 	/*
 	 * Indicates whether hidden refs have been configured. This is to
@@ -110,7 +111,7 @@ struct ref_exclusions {
  */
 #define REF_EXCLUSIONS_INIT { \
 	.excluded_refs = STRING_LIST_INIT_DUP, \
-	.hidden_refs = STRING_LIST_INIT_DUP, \
+	.hidden_refs = STRVEC_INIT, \
 }
 
 struct oidset;
upload-pack.c +5 -5
index 08633dc121..d77d58bdde 100644
@@ -69,7 +69,7 @@ struct upload_pack_data {
 	struct object_array have_obj;
 	struct oid_array haves;					/* v2 only */
 	struct string_list wanted_refs;				/* v2 only */
-	struct string_list hidden_refs;
+	struct strvec hidden_refs;
 
 	struct object_array shallows;
 	struct string_list deepen_not;
@@ -126,7 +126,7 @@ static void upload_pack_data_init(struct upload_pack_data *data)
 {
 	struct string_list symref = STRING_LIST_INIT_DUP;
 	struct string_list wanted_refs = STRING_LIST_INIT_DUP;
-	struct string_list hidden_refs = STRING_LIST_INIT_DUP;
+	struct strvec hidden_refs = STRVEC_INIT;
 	struct object_array want_obj = OBJECT_ARRAY_INIT;
 	struct object_array have_obj = OBJECT_ARRAY_INIT;
 	struct oid_array haves = OID_ARRAY_INIT;
@@ -161,7 +161,7 @@ static void upload_pack_data_clear(struct upload_pack_data *data)
 {
 	string_list_clear(&data->symref, 1);
 	string_list_clear(&data->wanted_refs, 1);
-	string_list_clear(&data->hidden_refs, 0);
+	strvec_clear(&data->hidden_refs);
 	object_array_clear(&data->want_obj);
 	object_array_clear(&data->have_obj);
 	oid_array_clear(&data->haves);
@@ -1169,7 +1169,7 @@ static void receive_needs(struct upload_pack_data *data,
 
 /* return non-zero if the ref is hidden, otherwise 0 */
 static int mark_our_ref(const char *refname, const char *refname_full,
-			const struct object_id *oid, const struct string_list *hidden_refs)
+			const struct object_id *oid, const struct strvec *hidden_refs)
 {
 	struct object *o = lookup_unknown_object(the_repository, oid);
 
@@ -1453,7 +1453,7 @@ static int parse_want(struct packet_writer *writer, const char *line,
 
 static int parse_want_ref(struct packet_writer *writer, const char *line,
 			  struct string_list *wanted_refs,
-			  struct string_list *hidden_refs,
+			  struct strvec *hidden_refs,
 			  struct object_array *want_obj)
 {
 	const char *refname_nons;
Signature · 1 line
2.40.1.572.g5c4ab523ef
#54 lore raw

PATCH v2 12/16 refs/packed-backend.c: ignore complicated hidden refs rules

In reply to Taylor Blau

In subsequent commits, we'll teach receive-pack and upload-pack to use the new skip-list feature in the packed-refs iterator by ignoring references which are mentioned via its respective hideRefs lists.

However, the packed-ref skip lists cannot handle un-hiding rules (that begin with '!'), or namespace comparisons (that begin with '^'). Detect and avoid these cases by falling back to the normal enumeration without a skip list when such patterns exist.

---
 refs/packed-backend.c   | 19 +++++++++++++++++++
 t/t1419-exclude-refs.sh |  9 +++++++++
 2 files changed, 28 insertions(+)
refs/packed-backend.c +19 -0
index 7ba9fa2bb8..9ea6c07866 100644
@@ -1015,6 +1015,25 @@ static void populate_excluded_jump_list(struct packed_ref_iterator *iter,
 	if (!excluded_patterns)
 		return;
 
+	for (pattern = excluded_patterns; *pattern; pattern++) {
+		/*
+		 * We also can't feed any excludes from hidden refs
+		 * config sections, since later rules may override
+		 * previous ones. For example, with rules "refs/foo" and
+		 * "!refs/foo/bar", we should show "refs/foo/bar" (and
+		 * everything underneath it), but the earlier exclusion
+		 * would cause us to skip all of "refs/foo". We likewise
+		 * don't implement the namespace stripping required for
+		 * '^' rules.
+		 *
+		 * Both are possible to do, but complicated, so avoid
+		 * populating the jump list at all if we see either of
+		 * these patterns.
+		 */
+		if (**pattern == '!' || **pattern == '^')
+			return;
+	}
+
 	for (pattern = excluded_patterns; *pattern; pattern++) {
 		struct jump_list_entry *e;
 
t/t1419-exclude-refs.sh +9 -0
index 350a7d2587..0e91e2f399 100755
@@ -119,4 +119,13 @@ test_expect_success 'meta-characters are discarded' '
 	assert_no_jumps
 '
 
+test_expect_success 'complex hidden ref rules are discarded' '
+	for_each_ref__exclude refs/heads refs/heads/foo "!refs/heads/foo/1" \
+		>actual 2>perf &&
+	for_each_ref >expect &&
+
+	test_cmp expect actual &&
+	assert_no_jumps
+'
+
 test_done
Signature · 1 line
2.40.1.572.g5c4ab523ef
#55 lore raw

PATCH v2 13/16 refs.h: let `for_each_namespaced_ref()` take excluded patterns

In reply to Taylor Blau

The following commit will want to call for_each_namespaced_ref() with a list of excluded patterns.

We could introduce a variant of that function, say, for_each_namespaced_ref_exclude() which takes the extra parameter, and reimplement the original function in terms of that. But all but one caller (in http-backend.c) will supply the new parameter, so add the new parameter to for_each_namespaced_ref() itself instead of introducing a new function.

For now, supply NULL for the list of excluded patterns at all callers to avoid changing behavior, which we will do in the subsequent commit.

---
 http-backend.c | 2 +-
 refs.c         | 5 +++--
 refs.h         | 3 ++-
 upload-pack.c  | 6 +++---
 4 files changed, 9 insertions(+), 7 deletions(-)
http-backend.c +1 -1
index ac146d85c5..ad500683c8 100644
@@ -559,7 +559,7 @@ static void get_info_refs(struct strbuf *hdr, char *arg UNUSED)
 
 	} else {
 		select_getanyfile(hdr);
-		for_each_namespaced_ref(show_text_ref, &buf);
+		for_each_namespaced_ref(NULL, show_text_ref, &buf);
 		send_strbuf(hdr, "text/plain", &buf);
 	}
 	strbuf_release(&buf);
refs.c +3 -2
index ec4d5b9101..95a7db9563 100644
@@ -1660,13 +1660,14 @@ int for_each_replace_ref(struct repository *r, each_repo_ref_fn fn, void *cb_dat
 				    DO_FOR_EACH_INCLUDE_BROKEN, cb_data);
 }
 
-int for_each_namespaced_ref(each_ref_fn fn, void *cb_data)
+int for_each_namespaced_ref(const char **exclude_patterns,
+			    each_ref_fn fn, void *cb_data)
 {
 	struct strbuf buf = STRBUF_INIT;
 	int ret;
 	strbuf_addf(&buf, "%srefs/", get_git_namespace());
 	ret = do_for_each_ref(get_main_ref_store(the_repository),
-			      buf.buf, NULL, fn, 0, 0, cb_data);
+			      buf.buf, exclude_patterns, fn, 0, 0, cb_data);
 	strbuf_release(&buf);
 	return ret;
 }
refs.h +2 -1
index a7751a1fc9..f23626beca 100644
@@ -372,7 +372,8 @@ int for_each_glob_ref_in(each_ref_fn fn, const char *pattern,
 			 const char *prefix, void *cb_data);
 
 int head_ref_namespaced(each_ref_fn fn, void *cb_data);
-int for_each_namespaced_ref(each_ref_fn fn, void *cb_data);
+int for_each_namespaced_ref(const char **exclude_patterns,
+			    each_ref_fn fn, void *cb_data);
 
 /* can be used to learn about broken ref and symref */
 int refs_for_each_rawref(struct ref_store *refs, each_ref_fn fn, void *cb_data);
upload-pack.c +3 -3
index d77d58bdde..7c646ea5bd 100644
@@ -854,7 +854,7 @@ static void deepen(struct upload_pack_data *data, int depth)
 		 * marked with OUR_REF.
 		 */
 		head_ref_namespaced(check_ref, data);
-		for_each_namespaced_ref(check_ref, data);
+		for_each_namespaced_ref(NULL, check_ref, data);
 
 		get_reachable_list(data, &reachable_shallows);
 		result = get_shallow_commits(&reachable_shallows,
@@ -1378,7 +1378,7 @@ void upload_pack(const int advertise_refs, const int stateless_rpc,
 		if (advertise_refs)
 			data.no_done = 1;
 		head_ref_namespaced(send_ref, &data);
-		for_each_namespaced_ref(send_ref, &data);
+		for_each_namespaced_ref(NULL, send_ref, &data);
 		/*
 		 * fflush stdout before calling advertise_shallow_grafts because send_ref
 		 * uses stdio.
@@ -1388,7 +1388,7 @@ void upload_pack(const int advertise_refs, const int stateless_rpc,
 		packet_flush(1);
 	} else {
 		head_ref_namespaced(check_ref, &data);
-		for_each_namespaced_ref(check_ref, &data);
+		for_each_namespaced_ref(NULL, check_ref, &data);
 	}
 
 	if (!advertise_refs) {
Signature · 1 line
2.40.1.572.g5c4ab523ef
#58 lore raw

PATCH v2 14/16 builtin/receive-pack.c: avoid enumerating hidden references

In reply to Taylor Blau

Now that refs_for_each_fullref_in() has the ability to avoid enumerating references matching certain pattern(s), use that to avoid visiting hidden refs when constructing the ref advertisement via receive-pack.

Note that since this exclusion is best-effort, we still need show_ref_cb() to check whether or not each reference is hidden or not before including it in the advertisement.

As was the case when applying this same optimization to upload-pack, receive-pack's reference advertisement phase can proceed much quicker by avoiding enumerating references that will not be part of the advertisement.

(Below, we're still using linux.git with one hidden refs/pull/N ref per commit):

    $ hyperfine -L v ,.compile 'git{v} -c transfer.hideRefs=refs/pull receive-pack --advertise-refs .git'
    Benchmark 1: git -c transfer.hideRefs=refs/pull receive-pack --advertise-refs .git
      Time (mean ± σ):      89.1 ms ±   1.7 ms    [User: 82.0 ms, System: 7.0 ms]
      Range (min … max):    87.7 ms …  95.5 ms    31 runs
    Benchmark 2: git.compile -c transfer.hideRefs=refs/pull receive-pack --advertise-refs .git
      Time (mean ± σ):       4.5 ms ±   0.2 ms    [User: 0.5 ms, System: 3.9 ms]
      Range (min … max):     4.1 ms …   5.6 ms    508 runs

    Summary
      'git.compile -c transfer.hideRefs=refs/pull receive-pack --advertise-refs .git' ran
       20.00 ± 1.05 times faster than 'git -c transfer.hideRefs=refs/pull receive-pack --advertise-refs .git'
---
 builtin/receive-pack.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
builtin/receive-pack.c +2 -1
index 1a8472eddc..bd5bcc375f 100644
@@ -337,7 +337,8 @@ static void write_head_info(void)
 {
 	static struct oidset seen = OIDSET_INIT;
 
-	for_each_ref(show_ref_cb, &seen);
+	refs_for_each_fullref_in(get_main_ref_store(the_repository), "",
+				 hidden_refs.v, show_ref_cb, &seen);
 	for_each_alternate_ref(show_one_alternate_ref, &seen);
 	oidset_clear(&seen);
 	if (!sent_capabilities)
Signature · 1 line
2.40.1.572.g5c4ab523ef
#59 lore raw

PATCH v2 15/16 upload-pack.c: avoid enumerating hidden refs where possible

In reply to Taylor Blau

In a similar fashion as a previous commit, teach upload-pack to avoid enumerating hidden references where possible.

Note, however, that there are certain cases where cannot avoid enumerating even hidden references, in particular when either of:

- uploadpack.allowTipSHA1InWant, or
- uploadpack.allowReachableSHA1InWant

are set, corresponding to ALLOW_TIP_SHA1 and ALLOW_REACHABLE_SHA1, respectively.

When either of these bits are set, upload-pack's is_our_ref() function needs to consider the HIDDEN_REF bit of the referent's object flags. So we must visit all references, including the hidden ones, in order to mark their referents with the HIDDEN_REF bit.

When neither ALLOW_TIP_SHA1 nor ALLOW_REACHABLE_SHA1 are set, the is_our_ref() function considers only the OUR_REF bit, and not the HIDDEN_REF one. OUR_REF is applied via mark_our_ref(), and only to objects at the tips of non-hidden references, so we do not need to visit hidden references in this case.

When neither of those bits are set, upload-pack can potentially avoid enumerating a large number of references. In the same example as a previous commit (linux.git with one hidden reference per commit, "refs/pull/N"):

    $ printf 0000 >in
    $ hyperfine --warmup=1 \
      'git -c transfer.hideRefs=refs/pull upload-pack . <in' \
      'git.compile -c transfer.hideRefs=refs/pull -c uploadpack.allowTipSHA1InWant upload-pack . <in' \
      'git.compile -c transfer.hideRefs=refs/pull upload-pack . <in'
    Benchmark 1: git -c transfer.hideRefs=refs/pull upload-pack . <in
      Time (mean ± σ):     406.9 ms ±   1.1 ms    [User: 357.3 ms, System: 49.5 ms]
      Range (min … max):   405.7 ms … 409.2 ms    10 runs

Benchmark 2: git.compile -c transfer.hideRefs=refs/pull -c uploadpack.allowTipSHA1InWant upload-pack . <in
Time (mean ± σ): 406.5 ms ± 1.3 ms [User: 356.5 ms, System: 49.9 ms]
Range (min … max): 404.6 ms … 408.8 ms 10 runs

Benchmark 3: git.compile -c transfer.hideRefs=refs/pull upload-pack . <in
Time (mean ± σ): 4.7 ms ± 0.2 ms [User: 0.7 ms, System: 3.9 ms]
Range (min … max): 4.3 ms … 6.1 ms 472 runs

Summary
'git.compile -c transfer.hideRefs=refs/pull upload-pack . <in' ran
86.62 ± 4.33 times faster than 'git.compile -c transfer.hideRefs=refs/pull -c uploadpack.allowTipSHA1InWant upload-pack . <in' 86.70 ± 4.33 times faster than 'git -c transfer.hideRefs=refs/pull upload-pack . <in'

As above, we must visit every reference when uploadPack.allowTipSHA1InWant is set. But when it is unset, we can visit far fewer references.

---
 upload-pack.c | 33 +++++++++++++++++++++++++++------
 1 file changed, 27 insertions(+), 6 deletions(-)
upload-pack.c +27 -6
index 7c646ea5bd..6fa667bf25 100644
@@ -601,11 +601,32 @@ static int get_common_commits(struct upload_pack_data *data,
 	}
 }
 
+static int allow_hidden_refs(enum allow_uor allow_uor)
+{
+	return allow_uor & (ALLOW_TIP_SHA1 | ALLOW_REACHABLE_SHA1);
+}
+
+static void for_each_namespaced_ref_1(each_ref_fn fn,
+				      struct upload_pack_data *data)
+{
+	/*
+	 * If `data->allow_uor` allows fetching hidden refs, we need to
+	 * mark all references (including hidden ones), to check in
+	 * `is_our_ref()` below.
+	 *
+	 * Otherwise, we only care about whether each reference's object
+	 * has the OUR_REF bit set or not, so do not need to visit
+	 * hidden references.
+	 */
+	if (allow_hidden_refs(data->allow_uor))
+		for_each_namespaced_ref(NULL, fn, data);
+	else
+		for_each_namespaced_ref(data->hidden_refs.v, fn, data);
+}
+
 static int is_our_ref(struct object *o, enum allow_uor allow_uor)
 {
-	int allow_hidden_ref = (allow_uor &
-				(ALLOW_TIP_SHA1 | ALLOW_REACHABLE_SHA1));
-	return o->flags & ((allow_hidden_ref ? HIDDEN_REF : 0) | OUR_REF);
+	return o->flags & ((allow_hidden_refs(allow_uor) ? HIDDEN_REF : 0) | OUR_REF);
 }
 
 /*
@@ -854,7 +875,7 @@ static void deepen(struct upload_pack_data *data, int depth)
 		 * marked with OUR_REF.
 		 */
 		head_ref_namespaced(check_ref, data);
-		for_each_namespaced_ref(NULL, check_ref, data);
+		for_each_namespaced_ref_1(check_ref, data);
 
 		get_reachable_list(data, &reachable_shallows);
 		result = get_shallow_commits(&reachable_shallows,
@@ -1378,7 +1399,7 @@ void upload_pack(const int advertise_refs, const int stateless_rpc,
 		if (advertise_refs)
 			data.no_done = 1;
 		head_ref_namespaced(send_ref, &data);
-		for_each_namespaced_ref(NULL, send_ref, &data);
+		for_each_namespaced_ref_1(send_ref, &data);
 		/*
 		 * fflush stdout before calling advertise_shallow_grafts because send_ref
 		 * uses stdio.
@@ -1388,7 +1409,7 @@ void upload_pack(const int advertise_refs, const int stateless_rpc,
 		packet_flush(1);
 	} else {
 		head_ref_namespaced(check_ref, &data);
-		for_each_namespaced_ref(NULL, check_ref, &data);
+		for_each_namespaced_ref_1(check_ref, &data);
 	}
 
 	if (!advertise_refs) {
Signature · 1 line
2.40.1.572.g5c4ab523ef
#60 lore raw

PATCH v2 16/16 ls-refs.c: avoid enumerating hidden refs where possible

In reply to Taylor Blau

In a similar fashion as in previous commits, teach ls-refs to avoid enumerating hidden references where possible.

As before, this is linux.git with one hidden reference per commit.

    $ hyperfine -L v ,.compile 'git{v} -c protocol.version=2 ls-remote .'
    Benchmark 1: git -c protocol.version=2 ls-remote .
      Time (mean ± σ):      89.8 ms ±   0.6 ms    [User: 84.3 ms, System: 5.7 ms]
      Range (min … max):    88.8 ms …  91.3 ms    32 runs

Benchmark 2: git.compile -c protocol.version=2 ls-remote .
Time (mean ± σ): 6.5 ms ± 0.1 ms [User: 2.4 ms, System: 4.3 ms]
Range (min … max): 6.2 ms … 8.3 ms 397 runs

Summary
'git.compile -c protocol.version=2 ls-remote .' ran
13.85 ± 0.33 times faster than 'git -c protocol.version=2 ls-remote .'

---
 ls-refs.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
ls-refs.c +1 -1
index 8c3181d051..c9a723ba89 100644
@@ -193,7 +193,7 @@ int ls_refs(struct repository *r, struct packet_reader *request)
 		strvec_push(&data.prefixes, "");
 	refs_for_each_fullref_in_prefixes(get_main_ref_store(r),
 					  get_git_namespace(), data.prefixes.v,
-					  NULL, send_ref, &data);
+					  data.hidden_refs.v, send_ref, &data);
 	packet_fflush(stdout);
 	strvec_clear(&data.prefixes);
 	strbuf_release(&data.buf);
Signature · 1 line
2.40.1.572.g5c4ab523ef
#52 lore raw

Re: [PATCH v2 11/16] revision.h: store hidden refs in a `strvec`

In reply to Taylor Blau
Quoted 18 lines · Taylor Blau

In subsequent commits, it will be convenient to have a 'const char **' of hidden refs (matching transfer.hiderefs, uploadpack.hideRefs, etc.), instead of a string_list.

Convert spots throughout the tree that store the list of hidden refs from a string_list to a strvec.

Note that in parse_hide_refs_config() there is an ugly const-cast used to avoid an extra copy of each value before trimming any trailing slash characters. This could instead be written as:

    ref = xstrdup(value);
    len = strlen(ref);
    while (len && ref[len - 1] == '/')
            ref[--len] = '\0';
    strvec_push(hide_refs, ref);
    free(ref);
Quoted 2 lines

but the double-copy (once when calling xstrdup(), and another via strvec_push()) is wasteful.

I guess the proper way to fix this would be to introduce something like a strvec_push_nodup() function that takes ownership. And in fact this helper exists already, but it's declared as static. So we could get around the ugly cast with a simple change to expose the helper function.

Patrick

#48 lore raw

Re: [PATCH v2 09/16] refs/packed-backend.c: implement jump lists to avoid excluded pattern(s)

In reply to Taylor Blau
Quoted 173 lines · Taylor Blau

When iterating through the packed-refs file in order to answer a query like:

    $ git for-each-ref --exclude=refs/__hidden__

it would be useful to avoid walking over all of the entries in refs/__hidden__/* when possible, since we know that the ref-filter code is going to throw them away anyways.

In certain circumstances, doing so is possible. The algorithm for doing so is as follows:

- For each excluded pattern, find the first record that matches it, and the first record that *doesn't* match it (i.e. the location you'd next want to consider when excluding that pattern).

- Sort the set of excluded regions from the previous step in ascending order of the first location within the packed-refs file that matches.

- Clean up the results from the previous step: discard empty regions, and combine adjacent regions.

Then when iterating through the packed-refs file, if iter->pos is ever contained in one of the regions from the previous steps, advance iter->pos past the end of that region, and continue enumeration.

Note that we only perform this optimization when none of the excluded pattern(s) have special meta-characters in them. For a pattern like "refs/foo[ac]", the excluded regions ("refs/fooa", "refs/fooc", and everything underneath them) are not connected. A future implementation that handles this case may split the character class (pretending as if two patterns were excluded: "refs/fooa", and "refs/fooc").

There are a few other gotchas worth considering. First, note that the jump list is sorted, so once we jump past a region, we can avoid considering it (or any regions preceding it) again. The member jump_pos is used to track the first next-possible region to jump through.

Second, note that the exclusion list is best-effort, since we do not handle loose references, and because of the meta-character issue above.

In repositories with a large number of hidden references, the speed-up can be significant. Tests here are done with a copy of linux.git with a reference "refs/pull/N" pointing at every commit, as in:

    $ git rev-list HEAD | awk '{ print "create refs/pull/" NR " " $0 }' |
        git update-ref --stdin
    $ git pack-refs --all

, it is significantly faster to have for-each-ref jump over the excluded references, as opposed to filtering them out after the fact:

    $ hyperfine \
      'git for-each-ref --format="%(objectname) %(refname)" | grep -vE "^[0-9a-f]{40} refs/pull/"' \
      'git.compile for-each-ref --format="%(objectname) %(refname)" --exclude="refs/pull"'
    Benchmark 1: git for-each-ref --format="%(objectname) %(refname)" | grep -vE "^[0-9a-f]{40} refs/pull/"
      Time (mean ± σ):     802.7 ms ±   2.1 ms    [User: 691.6 ms, System: 147.0 ms]
      Range (min … max):   800.0 ms … 807.7 ms    10 runs
    Benchmark 2: git.compile for-each-ref --format="%(objectname) %(refname)" --exclude="refs/pull"
      Time (mean ± σ):       4.7 ms ±   0.3 ms    [User: 0.7 ms, System: 4.0 ms]
      Range (min … max):     4.3 ms …   6.7 ms    422 runs

    Summary
      'git.compile for-each-ref --format="%(objectname) %(refname)" --exclude="refs/pull"' ran
      172.03 ± 9.60 times faster than 'git for-each-ref --format="%(objectname) %(refname)" | grep -vE "^[0-9a-f]{40} refs/pull/"'

Using the jump list is fairly straightforward (see the changes to refs/packed-backend.c::next_record()), but constructing the list is not. To ensure that the construction is correct, add a new suite of tests in t1419 covering various corner cases (overlapping regions, partially overlapping regions, adjacent regions, etc.).

---
 ref-filter.c              |   5 +-
 refs/packed-backend.c     | 166 ++++++++++++++++++++++++++++++++++++--
 t/helper/test-ref-store.c |  10 +++
 t/t1419-exclude-refs.sh   | 101 +++++++++++++++++++++++
 4 files changed, 274 insertions(+), 8 deletions(-)
 create mode 100755 t/t1419-exclude-refs.sh
ref-filter.c +3 -2
index 717c3c4bcf..ddc7f5204f 100644
@@ -2210,12 +2210,13 @@ static int for_each_fullref_in_pattern(struct ref_filter *filter,
 	if (!filter->name_patterns[0]) {
 		/* no patterns; we have to look at everything */
 		return refs_for_each_fullref_in(get_main_ref_store(the_repository),
-						 "", NULL, cb, cb_data);
+						 "", filter->exclude.v, cb, cb_data);
 	}
 
 	return refs_for_each_fullref_in_prefixes(get_main_ref_store(the_repository),
 						 NULL, filter->name_patterns,
-						 NULL, cb, cb_data);
+						 filter->exclude.v,
+						 cb, cb_data);
 }
 
 /*
refs/packed-backend.c +22 -5
index 33639f73e1..67327e579c 100644
@@ -303,7 +303,8 @@ static int cmp_packed_ref_records(const void *v1, const void *v2)
  * Compare a snapshot record at `rec` to the specified NUL-terminated
  * refname.
  */
-static int cmp_record_to_refname(const char *rec, const char *refname)
+static int cmp_record_to_refname(const char *rec, const char *refname,
+				 int start)
 {
 	const char *r1 = rec + the_hash_algo->hexsz + 1;
 	const char *r2 = refname;
@@ -312,7 +313,7 @@ static int cmp_record_to_refname(const char *rec, const char *refname)
 		if (*r1 == '\n')
 			return *r2 ? -1 : 0;
 		if (!*r2)
-			return 1;
+			return start ? 1 : -1;
 		if (*r1 != *r2)
 			return (unsigned char)*r1 < (unsigned char)*r2 ? -1 : +1;
 		r1++;
@@ -528,7 +529,8 @@ static int load_contents(struct snapshot *snapshot)
 }
 
 static const char *find_reference_location_1(struct snapshot *snapshot,
-					     const char *refname, int mustexist)
+					     const char *refname, int mustexist,
+					     int start)
 {
 	/*
 	 * This is not *quite* a garden-variety binary search, because
@@ -558,7 +560,7 @@ static const char *find_reference_location_1(struct snapshot *snapshot,
 
 		mid = lo + (hi - lo) / 2;
 		rec = find_start_of_record(lo, mid);
-		cmp = cmp_record_to_refname(rec, refname);
+		cmp = cmp_record_to_refname(rec, refname, start);
 		if (cmp < 0) {
 			lo = find_end_of_record(mid, hi);
 		} else if (cmp > 0) {
@@ -591,7 +593,22 @@ static const char *find_reference_location_1(struct snapshot *snapshot,
 static const char *find_reference_location(struct snapshot *snapshot,
 					   const char *refname, int mustexist)
 {
-	return find_reference_location_1(snapshot, refname, mustexist);
+	return find_reference_location_1(snapshot, refname, mustexist, 1);
+}
+
+/*
+ * Find the place in `snapshot->buf` after the end of the record for
+ * `refname`. In other words, find the location of first thing *after*
+ * `refname`.
+ *
+ * Other semantics are identical to the ones in
+ * `find_reference_location()`.
+ */
+static const char *find_reference_location_end(struct snapshot *snapshot,
+					       const char *refname,
+					       int mustexist)
+{
+	return find_reference_location_1(snapshot, refname, mustexist, 0);
 }
 
 /*

refs/packed-backend.c struct packed_ref_iterator { · line 811

Quoted 10 lines
refs/packed-backend.c +6 -0
@@ -785,6 +802,13 @@ struct packed_ref_iterator {
 	/* The end of the part of the buffer that will be iterated over: */
 	const char *eof;
 
+	struct jump_list_entry {
+		const char *start;
+		const char *end;
+	} *jump;
+	size_t jump_nr, jump_alloc;
+	size_t jump_pos;
 

Nit: I had some trouble with jump_pos given that it sounds so similar to iter->pos, and thus you tend to think that they both apply to the position in the packed-refs file. jump_curr or jump_idx might help to avoid this confusion.

Quoted 100 lines
refs/packed-backend.c +71 -1
 	/* Scratch space for current values: */
 	struct object_id oid, peeled;
 	struct strbuf refname_buf;
@@ -802,14 +826,34 @@ struct packed_ref_iterator {
  */
 static int next_record(struct packed_ref_iterator *iter)
 {
-	const char *p = iter->pos, *eol;
+	const char *p, *eol;
 
 	strbuf_reset(&iter->refname_buf);
 
+	/*
+	 * If iter->pos is contained within a skipped region, jump past
+	 * it.
+	 *
+	 * Note that each skipped region is considered at most once,
+	 * since they are ordered based on their starting position.
+	 */
+	while (iter->jump_pos < iter->jump_nr) {
+		struct jump_list_entry *curr = &iter->jump[iter->jump_pos];
+		if (iter->pos < curr->start)
+			break; /* not to the next jump yet */
+
+		iter->jump_pos++;
+		if (iter->pos < curr->end) {
+			iter->pos = curr->end;
+			break;
+		}
+	}
+
 	if (iter->pos == iter->eof)
 		return ITER_DONE;
 
 	iter->base.flags = REF_ISPACKED;
+	p = iter->pos;
 
 	if (iter->eof - p < the_hash_algo->hexsz + 2 ||
 	    parse_oid_hex(p, &iter->oid, &p) ||
@@ -917,6 +961,7 @@ static int packed_ref_iterator_abort(struct ref_iterator *ref_iterator)
 	int ok = ITER_DONE;
 
 	strbuf_release(&iter->refname_buf);
+	free(iter->jump);
 	release_snapshot(iter->snapshot);
 	base_ref_iterator_free(ref_iterator);
 	return ok;
@@ -928,6 +973,112 @@ static struct ref_iterator_vtable packed_ref_iterator_vtable = {
 	.abort = packed_ref_iterator_abort
 };
 
+static int jump_list_entry_cmp(const void *va, const void *vb)
+{
+	const struct jump_list_entry *a = va;
+	const struct jump_list_entry *b = vb;
+
+	if (a->start < b->start)
+		return -1;
+	if (a->start > b->start)
+		return 1;
+	return 0;
+}
+
+static int has_glob_special(const char *str)
+{
+	const char *p;
+	for (p = str; *p; p++) {
+		if (is_glob_special(*p))
+			return 1;
+	}
+	return 0;
+}
+
+static const char *ptr_max(const char *x, const char *y)
+{
+	if (x > y)
+		return x;
+	return y;
+}
+
+static void populate_excluded_jump_list(struct packed_ref_iterator *iter,
+					struct snapshot *snapshot,
+					const char **excluded_patterns)
+{
+	size_t i, j;
+	const char **pattern;
+	struct jump_list_entry *last_disjoint;
+
+	if (!excluded_patterns)
+		return;
+
+	for (pattern = excluded_patterns; *pattern; pattern++) {
+		struct jump_list_entry *e;
+
+		/*
+		 * We can't feed any excludes with globs in them to the
+		 * refs machinery.  It only understands prefix matching.
+		 * We likewise can't even feed the string leading up to
+		 * the first meta-character, as something like "foo[a]"
+		 * should not exclude "foobar" (but the prefix "foo"

refs/packed-backend.c static struct ref_iterator_vtable packed_ref_iterator_vtable = { · line 1034

Quoted 10 lines
refs/packed-backend.c +10 -0
+		 * would match that and mark it for exclusion).
+		 */
+		if (has_glob_special(*pattern))
+			continue;
+
+		ALLOC_GROW(iter->jump, iter->jump_nr + 1, iter->jump_alloc);
+
+		e = &iter->jump[iter->jump_nr++];
+		e->start = find_reference_location(snapshot, *pattern, 0);
+		e->end = find_reference_location_end(snapshot, *pattern, 0);

Nit: we could detect the non-matching case here already, which would allow us to skip an allocation. It's probably pre-mature optimization though, so please feel free to ignore.

Quoted 15 lines
refs/packed-backend.c +15 -0
+	}
+
+	if (!iter->jump_nr) {
+		/*
+		 * Every entry in exclude_patterns has a meta-character,
+		 * nothing to do here.
+		 */
+		return;
+	}
+
+	QSORT(iter->jump, iter->jump_nr, jump_list_entry_cmp);
+
+	/*
+	 * As an optimization, merge adjacent entries in the jump list
+	 * to jump forwards as far as possible when entering a skipped

refs/packed-backend.c static struct ref_iterator_vtable packed_ref_iterator_vtable = { · line 1059

Quoted 10 lines
refs/packed-backend.c +10 -0
+	 * region.
+	 *
+	 * For example, if we have two skipped regions:
+	 *
+	 *	[[A, B], [B, C]]
+	 *
+	 * we want to combine that into a single entry jumping from A to
+	 * C.
+	 */
+	last_disjoint = iter->jump;

Nit: if we initialized j = 0, then last_disjoint would always be equal to iter->jump[j]. We could then declare the variable inside of the loop to make it a bit easier to understand.

Patrick

Quoted 172 lines
refs/packed-backend.c +24 -0
+	for (i = 1, j = 1; i < iter->jump_nr; i++) {
+		struct jump_list_entry *ours = &iter->jump[i];
+
+		if (ours->start == ours->end) {
+			/* ignore empty regions (no matching entries) */
+			continue;
+		} else if (ours->start <= last_disjoint->end) {
+			/* overlapping regions extend the previous one */
+			last_disjoint->end = ptr_max(last_disjoint->end, ours->end);
+		} else {
+			/* otherwise, insert a new region */
+			iter->jump[j++] = *ours;
+			last_disjoint = ours;
+
+		}
+	}
+
+	iter->jump_nr = j;
+	iter->jump_pos = 0;
+}
+
 static struct ref_iterator *packed_ref_iterator_begin(
 		struct ref_store *ref_store,
 		const char *prefix, const char **exclude_patterns,
@@ -963,6 +1114,9 @@ static struct ref_iterator *packed_ref_iterator_begin(
 	ref_iterator = &iter->base;
 	base_ref_iterator_init(ref_iterator, &packed_ref_iterator_vtable, 1);
 
+	if (exclude_patterns)
+		populate_excluded_jump_list(iter, snapshot, exclude_patterns);
+
 	iter->snapshot = snapshot;
 	acquire_snapshot(snapshot);
 
t/helper/test-ref-store.c +10 -0
index 6d8f844e9c..2bff003f7c 100644
@@ -175,6 +175,15 @@ static int cmd_for_each_ref(struct ref_store *refs, const char **argv)
 	return refs_for_each_ref_in(refs, prefix, each_ref, NULL);
 }
 
+static int cmd_for_each_ref__exclude(struct ref_store *refs, const char **argv)
+{
+	const char *prefix = notnull(*argv++, "prefix");
+	const char **exclude_patterns = argv;
+
+	return refs_for_each_fullref_in(refs, prefix, exclude_patterns, each_ref,
+					NULL);
+}
+
 static int cmd_resolve_ref(struct ref_store *refs, const char **argv)
 {
 	struct object_id oid = *null_oid();
@@ -307,6 +316,7 @@ static struct command commands[] = {
 	{ "delete-refs", cmd_delete_refs },
 	{ "rename-ref", cmd_rename_ref },
 	{ "for-each-ref", cmd_for_each_ref },
+	{ "for-each-ref--exclude", cmd_for_each_ref__exclude },
 	{ "resolve-ref", cmd_resolve_ref },
 	{ "verify-ref", cmd_verify_ref },
 	{ "for-each-reflog", cmd_for_each_reflog },
t/t1419-exclude-refs.sh new file +101 -0
new file mode 100755
index 0000000000..bc534c8ea1
@@ -0,0 +1,101 @@
+#!/bin/sh
+
+test_description='test exclude_patterns functionality in main ref store'
+
+GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main
+export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME
+
+TEST_PASSES_SANITIZE_LEAK=true
+. ./test-lib.sh
+
+for_each_ref__exclude () {
+	test-tool ref-store main for-each-ref--exclude "$@" >actual.raw
+	cut -d ' ' -f 2 actual.raw
+}
+
+for_each_ref () {
+	git for-each-ref --format='%(refname)' "$@"
+}
+
+test_expect_success 'setup' '
+	test_commit --no-tag base &&
+	base="$(git rev-parse HEAD)" &&
+
+	for name in foo bar baz quux
+	do
+		for i in 1 2 3
+		do
+			echo "create refs/heads/$name/$i $base" || return 1
+		done || return 1
+	done >in &&
+	echo "delete refs/heads/main" >>in &&
+
+	git update-ref --stdin <in &&
+	git pack-refs --all
+'
+
+test_expect_success 'excluded region in middle' '
+	for_each_ref__exclude refs/heads refs/heads/foo >actual &&
+	for_each_ref refs/heads/bar refs/heads/baz refs/heads/quux >expect &&
+
+	test_cmp expect actual
+'
+
+test_expect_success 'excluded region at beginning' '
+	for_each_ref__exclude refs/heads refs/heads/bar >actual &&
+	for_each_ref refs/heads/baz refs/heads/foo refs/heads/quux >expect &&
+
+	test_cmp expect actual
+'
+
+test_expect_success 'excluded region at end' '
+	for_each_ref__exclude refs/heads refs/heads/quux >actual &&
+	for_each_ref refs/heads/foo refs/heads/bar refs/heads/baz >expect &&
+
+	test_cmp expect actual
+'
+
+test_expect_success 'disjoint excluded regions' '
+	for_each_ref__exclude refs/heads refs/heads/bar refs/heads/quux >actual &&
+	for_each_ref refs/heads/baz refs/heads/foo >expect &&
+
+	test_cmp expect actual
+'
+
+test_expect_success 'adjacent, non-overlapping excluded regions' '
+	for_each_ref__exclude refs/heads refs/heads/bar refs/heads/baz >actual &&
+	for_each_ref refs/heads/foo refs/heads/quux >expect &&
+
+	test_cmp expect actual
+'
+
+test_expect_success 'overlapping excluded regions' '
+	for_each_ref__exclude refs/heads refs/heads/ba refs/heads/baz >actual &&
+	for_each_ref refs/heads/foo refs/heads/quux >expect &&
+
+	test_cmp expect actual
+'
+
+test_expect_success 'several overlapping excluded regions' '
+	for_each_ref__exclude refs/heads \
+		refs/heads/bar refs/heads/baz refs/heads/foo >actual &&
+	for_each_ref refs/heads/quux >expect &&
+
+	test_cmp expect actual
+'
+
+test_expect_success 'non-matching excluded section' '
+	for_each_ref__exclude refs/heads refs/heads/does/not/exist >actual &&
+	for_each_ref >expect &&
+
+	test_cmp expect actual
+'
+
+test_expect_success 'meta-characters are discarded' '
+	for_each_ref__exclude refs/heads "refs/heads/ba*" >actual &&
+	for_each_ref >expect &&
+
+	test_cmp expect actual
+'
+
+test_done
Signature · 1 line
2.40.1.572.g5c4ab523ef
#56 lore raw

Re: [PATCH v2 13/16] refs.h: let `for_each_namespaced_ref()` take excluded patterns

In reply to Taylor Blau
Quoted 2 lines · Taylor Blau

The following commit will want to call for_each_namespaced_ref() with a list of excluded patterns.

This statement isn't quite true as the following commit touches git-receive-pack(1), which doesn't use for_each_namespaced_ref_().

Patrick

#61 lore raw

Re: [PATCH v2 00/16] refs: implement jump lists for packed backend

In reply to Taylor Blau
Quoted 13 lines · Taylor Blau

Here is a reroll of my series to implement jump (née skip) lists for the packed refs backend.

Not a ton has changed since last time, but some notable things that have changed include:

- Renaming "skip lists" to "jump lists" to clarify that this implementation does not use the skip list data structure.
- Patch reorganization, splitting out find_reference_location_end() more sensibly, rewording patch messages, etc.
- Addresses feedback from Junio and Patrick Steinhardt's helpful reviews.

Quoted 7 lines

As usual, a range-diff is included below for convenience.

Given that we are expecting -rc0 today, we should aim to not let review of this topic direct our attention away from testing the release candidates. We can get more serious about it on the other side of 2.41.

Thanks in advance for another look.

I didn't have many comments in this round. Personally though I'd split up this patch series into two in order to land the individual parts faster, where the first part introduces git for-each-ref --exclude and the second part introduces the jump list for the packed-refs backend.

Each of these have merit on their own, and especially the first part should require less discussion. Furthermore, by splitting it up the review becomes easier to manage as 16 patches does require quite a long attention span to handle.

Anyway, this is just a suggestion from my sided, please feel free to ignore.

Patrick

#63 lore raw

v3 cover refs: implement jump lists for packed backend

In reply to Taylor Blau

Here is a reroll of my series to implement jump (née skip) lists for the packed refs backend, based on top of the current 'master'.

Nothing substantive has changed since the last version, where review had stabilized. This version just resolves a couple of merge conflicts with 633390bd08 (Merge branch 'bc/clone-empty-repo-via-protocol-v0', 2023-05-19).

As usual, a range-diff is included below for convenience.

Thanks in advance for a hopefully final look ;-).

Jeff King (5):
refs.c: rename ref_filter
ref-filter.h: provide REF_FILTER_INIT
ref-filter: clear reachable list pointers after freeing
ref-filter: add ref_filter_clear() ref-filter.c: parameterize match functions over patterns

Taylor Blau (11):
builtin/for-each-ref.c: add --exclude option
refs: plumb exclude_patterns argument throughout refs/packed-backend.c: refactor find_reference_location() refs/packed-backend.c: implement jump lists to avoid excluded
pattern(s)
refs/packed-backend.c: add trace2 counters for jump list revision.h: store hidden refs in a strvec refs/packed-backend.c: ignore complicated hidden refs rules refs.h: let for_each_namespaced_ref() take excluded patterns builtin/receive-pack.c: avoid enumerating hidden references upload-pack.c: avoid enumerating hidden refs where possible ls-refs.c: avoid enumerating hidden refs where possible

 Documentation/git-for-each-ref.txt |   6 +
 builtin/branch.c                   |   4 +-
 builtin/for-each-ref.c             |   7 +-
 builtin/receive-pack.c             |   7 +-
 builtin/tag.c                      |   4 +-
 http-backend.c                     |   2 +-
 ls-refs.c                          |   8 +-
 ref-filter.c                       |  63 ++++++--
 ref-filter.h                       |  12 ++
 refs.c                             |  61 ++++----
 refs.h                             |  15 +-
 refs/debug.c                       |   5 +-
 refs/files-backend.c               |   5 +-
 refs/packed-backend.c              | 226 ++++++++++++++++++++++++++---
 refs/refs-internal.h               |   7 +-
 revision.c                         |   4 +-
 revision.h                         |   5 +-
 t/helper/test-reach.c              |   2 +-
 t/helper/test-ref-store.c          |  10 ++
 t/t0041-usage.sh                   |   1 +
 t/t1419-exclude-refs.sh            | 131 +++++++++++++++++
 t/t3402-rebase-merge.sh            |   1 +
 t/t6300-for-each-ref.sh            |  35 +++++
 trace2.h                           |   2 +
 trace2/tr2_ctr.c                   |   5 +
 upload-pack.c                      |  43 ++++--
 26 files changed, 565 insertions(+), 106 deletions(-)
 create mode 100755 t/t1419-exclude-refs.sh

Range-diff against v2:

 1:  6cac42e70e =  1:  afac948f04 refs.c: rename `ref_filter`
 2:  8dda7db738 =  2:  b9336e3b77 ref-filter.h: provide `REF_FILTER_INIT`
 3:  bf21df783d =  3:  fc28b5caaa ref-filter: clear reachable list pointers after freeing
 4:  85ecb70957 =  4:  bc5356fe4b ref-filter: add `ref_filter_clear()`
 5:  385890b459 =  5:  1988ca4c0a ref-filter.c: parameterize match functions over patterns
 6:  1a3371a0a7 =  6:  60d85aa4ad builtin/for-each-ref.c: add `--exclude` option
 7:  aa05549b6e =  7:  c4fe9a1893 refs: plumb `exclude_patterns` argument throughout
 8:  6002c568b5 =  8:  9cab5e0699 refs/packed-backend.c: refactor `find_reference_location()`
 9:  8c78f49a8d =  9:  8066858bf5 refs/packed-backend.c: implement jump lists to avoid excluded pattern(s)
10:  5059f5dd42 = 10:  3c045076a9 refs/packed-backend.c: add trace2 counters for jump list
11:  f765b50a84 = 11:  0ff542eaef revision.h: store hidden refs in a `strvec`
12:  254bcc4361 = 12:  ca006b2c3f refs/packed-backend.c: ignore complicated hidden refs rules
13:  50e7df7dc0 ! 13:  cae703a425 refs.h: let `for_each_namespaced_ref()` take excluded patterns
    @@ upload-pack.c: void upload_pack(const int advertise_refs, const int stateless_rp
      		head_ref_namespaced(send_ref, &data);
     -		for_each_namespaced_ref(send_ref, &data);
     +		for_each_namespaced_ref(NULL, send_ref, &data);
    - 		/*
    - 		 * fflush stdout before calling advertise_shallow_grafts because send_ref
    - 		 * uses stdio.
    + 		if (!data.sent_capabilities) {
    + 			const char *refname = "capabilities^{}";
    + 			write_v0_ref(&data, refname, refname, null_oid());
     @@ upload-pack.c: void upload_pack(const int advertise_refs, const int stateless_rpc,
      		packet_flush(1);
      	} else {
14:  f6a3a5a6ba = 14:  1db10b76ea builtin/receive-pack.c: avoid enumerating hidden references
15:  2331fa7a4d ! 15:  014243ebe6 upload-pack.c: avoid enumerating hidden refs where possible

@@ upload-pack.c: void upload_pack(const int advertise_refs, const int stateless_rp
head_ref_namespaced(send_ref, &data);
- for_each_namespaced_ref(NULL, send_ref, &data);
+ for_each_namespaced_ref_1(send_ref, &data);
- /*
- * fflush stdout before calling advertise_shallow_grafts because send_ref
- * uses stdio.
+ if (!data.sent_capabilities) {
+ const char *refname = "capabilities^{}";
+ write_v0_ref(&data, refname, refname, null_oid());
@@ upload-pack.c: void upload_pack(const int advertise_refs, const int stateless_rpc,
packet_flush(1);
} else {
16: 2c6b89d64a = 16: e02fe93379 ls-refs.c: avoid enumerating hidden refs where possible

Signature · 1 line
2.41.0.16.g26cd413590
#64 lore raw

PATCH v3 1/16 refs.c: rename `ref_filter`

From: Jeff King <[email protected]>

The refs machinery has its own implementation of a ref_filter (used by for-each-ref), which is distinct from the ref-filler.h API (also used by for-each-ref, among other things).

Rename the one within refs.c to more clearly indicate its purpose.

---
 refs.c | 15 ++++++++-------
 1 file changed, 8 insertions(+), 7 deletions(-)
refs.c +8 -7
index d2a98e1c21..b9b77d2eff 100644
@@ -375,8 +375,8 @@ char *resolve_refdup(const char *refname, int resolve_flags,
 				   oid, flags);
 }
 
-/* The argument to filter_refs */
-struct ref_filter {
+/* The argument to for_each_filter_refs */
+struct for_each_ref_filter {
 	const char *pattern;
 	const char *prefix;
 	each_ref_fn *fn;
@@ -409,10 +409,11 @@ int ref_exists(const char *refname)
 	return refs_ref_exists(get_main_ref_store(the_repository), refname);
 }
 
-static int filter_refs(const char *refname, const struct object_id *oid,
-			   int flags, void *data)
+static int for_each_filter_refs(const char *refname,
+				const struct object_id *oid,
+				int flags, void *data)
 {
-	struct ref_filter *filter = (struct ref_filter *)data;
+	struct for_each_ref_filter *filter = data;
 
 	if (wildmatch(filter->pattern, refname, 0))
 		return 0;
@@ -569,7 +570,7 @@ int for_each_glob_ref_in(each_ref_fn fn, const char *pattern,
 	const char *prefix, void *cb_data)
 {
 	struct strbuf real_pattern = STRBUF_INIT;
-	struct ref_filter filter;
+	struct for_each_ref_filter filter;
 	int ret;
 
 	if (!prefix && !starts_with(pattern, "refs/"))
@@ -589,7 +590,7 @@ int for_each_glob_ref_in(each_ref_fn fn, const char *pattern,
 	filter.prefix = prefix;
 	filter.fn = fn;
 	filter.cb_data = cb_data;
-	ret = for_each_ref(filter_refs, &filter);
+	ret = for_each_ref(for_each_filter_refs, &filter);
 
 	strbuf_release(&real_pattern);
 	return ret;
Signature · 1 line
2.41.0.16.g26cd413590
#66 lore raw

PATCH v3 2/16 ref-filter.h: provide `REF_FILTER_INIT`

In reply to Taylor Blau

From: Jeff King <[email protected]>

Provide a sane initialization value for struct ref_filter, which in a subsequent patch will be used to initialize a new field.

In the meantime, fix a case in test-reach.c where its ref_filter is not even zero-initialized.

---
 builtin/branch.c       | 3 +--
 builtin/for-each-ref.c | 3 +--
 builtin/tag.c          | 3 +--
 ref-filter.h           | 3 +++
 t/helper/test-reach.c  | 2 +-
 5 files changed, 7 insertions(+), 7 deletions(-)
builtin/branch.c +1 -2
index e6c2655af6..7891dec361 100644
@@ -707,7 +707,7 @@ int cmd_branch(int argc, const char **argv, const char *prefix)
 	int reflog = 0, quiet = 0, icase = 0, force = 0,
 	    recurse_submodules_explicit = 0;
 	enum branch_track track;
-	struct ref_filter filter;
+	struct ref_filter filter = REF_FILTER_INIT;
 	static struct ref_sorting *sorting;
 	struct string_list sorting_options = STRING_LIST_INIT_DUP;
 	struct ref_format format = REF_FORMAT_INIT;
@@ -765,7 +765,6 @@ int cmd_branch(int argc, const char **argv, const char *prefix)
 
 	setup_ref_filter_porcelain_msg();
 
-	memset(&filter, 0, sizeof(filter));
 	filter.kind = FILTER_REFS_BRANCHES;
 	filter.abbrev = -1;
 
builtin/for-each-ref.c +1 -2
index 695fc8f4a5..99ccb73518 100644
@@ -24,7 +24,7 @@ int cmd_for_each_ref(int argc, const char **argv, const char *prefix)
 	struct string_list sorting_options = STRING_LIST_INIT_DUP;
 	int maxcount = 0, icase = 0, omit_empty = 0;
 	struct ref_array array;
-	struct ref_filter filter;
+	struct ref_filter filter = REF_FILTER_INIT;
 	struct ref_format format = REF_FORMAT_INIT;
 	struct strbuf output = STRBUF_INIT;
 	struct strbuf err = STRBUF_INIT;
@@ -61,7 +61,6 @@ int cmd_for_each_ref(int argc, const char **argv, const char *prefix)
 	};
 
 	memset(&array, 0, sizeof(array));
-	memset(&filter, 0, sizeof(filter));
 
 	format.format = "%(objectname) %(objecttype)\t%(refname)";
 
builtin/tag.c +1 -2
index 1850a6a6fd..6b41bb7374 100644
@@ -443,7 +443,7 @@ int cmd_tag(int argc, const char **argv, const char *prefix)
 	struct msg_arg msg = { .buf = STRBUF_INIT };
 	struct ref_transaction *transaction;
 	struct strbuf err = STRBUF_INIT;
-	struct ref_filter filter;
+	struct ref_filter filter = REF_FILTER_INIT;
 	struct ref_sorting *sorting;
 	struct string_list sorting_options = STRING_LIST_INIT_DUP;
 	struct ref_format format = REF_FORMAT_INIT;
@@ -501,7 +501,6 @@ int cmd_tag(int argc, const char **argv, const char *prefix)
 	git_config(git_tag_config, &sorting_options);
 
 	memset(&opt, 0, sizeof(opt));
-	memset(&filter, 0, sizeof(filter));
 	filter.lines = -1;
 	opt.sign = -1;
 
ref-filter.h +3 -0
index 430701cfb7..a920f73b29 100644
@@ -92,6 +92,9 @@ struct ref_format {
 	struct string_list bases;
 };
 
+#define REF_FILTER_INIT { \
+	.points_at = OID_ARRAY_INIT, \
+}
 #define REF_FORMAT_INIT {             \
 	.use_color = -1,              \
 	.bases = STRING_LIST_INIT_DUP, \
t/helper/test-reach.c +1 -1
index 5b6f217441..ef58f10c2d 100644
@@ -139,7 +139,7 @@ int cmd__reach(int ac, const char **av)
 
 		printf("%s(X,_,_,0,0):%d\n", av[1], can_all_from_reach_with_flag(&X_obj, 2, 4, 0, 0));
 	} else if (!strcmp(av[1], "commit_contains")) {
-		struct ref_filter filter;
+		struct ref_filter filter = REF_FILTER_INIT;
 		struct contains_cache cache;
 		init_contains_cache(&cache);
 
Signature · 1 line
2.41.0.16.g26cd413590
#67 lore raw

PATCH v3 3/16 ref-filter: clear reachable list pointers after freeing

In reply to Taylor Blau

From: Jeff King <[email protected]>

In reach_filter(), we pop all commits from the reachable lists, leaving them empty. But because we're operating on a list pointer that was passed by value, the original filter.reachable_from pointer is left dangling.

---
 ref-filter.c | 14 +++++++-------
 1 file changed, 7 insertions(+), 7 deletions(-)
ref-filter.c +7 -7
index 4991cd4f7a..048d277cbf 100644
@@ -2418,13 +2418,13 @@ void ref_array_clear(struct ref_array *array)
 #define EXCLUDE_REACHED 0
 #define INCLUDE_REACHED 1
 static void reach_filter(struct ref_array *array,
-			 struct commit_list *check_reachable,
+			 struct commit_list **check_reachable,
 			 int include_reached)
 {
 	int i, old_nr;
 	struct commit **to_clear;
 
-	if (!check_reachable)
+	if (!*check_reachable)
 		return;
 
 	CALLOC_ARRAY(to_clear, array->nr);
@@ -2434,7 +2434,7 @@ static void reach_filter(struct ref_array *array,
 	}
 
 	tips_reachable_from_bases(the_repository,
-				  check_reachable,
+				  *check_reachable,
 				  to_clear, array->nr,
 				  UNINTERESTING);
 
@@ -2455,8 +2455,8 @@ static void reach_filter(struct ref_array *array,
 
 	clear_commit_marks_many(old_nr, to_clear, ALL_REV_FLAGS);
 
-	while (check_reachable) {
-		struct commit *merge_commit = pop_commit(&check_reachable);
+	while (*check_reachable) {
+		struct commit *merge_commit = pop_commit(check_reachable);
 		clear_commit_marks(merge_commit, ALL_REV_FLAGS);
 	}
 
@@ -2553,8 +2553,8 @@ int filter_refs(struct ref_array *array, struct ref_filter *filter, unsigned int
 	clear_contains_cache(&ref_cbdata.no_contains_cache);
 
 	/*  Filters that need revision walking */
-	reach_filter(array, filter->reachable_from, INCLUDE_REACHED);
-	reach_filter(array, filter->unreachable_from, EXCLUDE_REACHED);
+	reach_filter(array, &filter->reachable_from, INCLUDE_REACHED);
+	reach_filter(array, &filter->unreachable_from, EXCLUDE_REACHED);
 
 	save_commit_buffer = save_commit_buffer_orig;
 	return ret;
Signature · 1 line
2.41.0.16.g26cd413590
#68 lore raw

PATCH v3 4/16 ref-filter: add `ref_filter_clear()`

In reply to Taylor Blau

From: Jeff King <[email protected]>

We did not bother to clean up at all in git branch or git tag, and git for-each-ref only cleans up a couple of members.

Add and call ref_filter_clear() when cleaning up a struct ref_filter. Running this patch (without any test changes) indicates a couple of now leak-free tests. This was found by running:

    $ make SANITIZE=leak
    $ make -C t GIT_TEST_PASSING_SANITIZE_LEAK=check GIT_TEST_OPTS=--immediate

(Note that the reachable_from and unreachable_from lists should be cleaned as they are used. So this is just covering any case where we might bail before running the reachability check.)

---
 builtin/branch.c        |  1 +
 builtin/for-each-ref.c  |  3 +--
 builtin/tag.c           |  1 +
 ref-filter.c            | 16 ++++++++++++++++
 ref-filter.h            |  3 +++
 t/t0041-usage.sh        |  1 +
 t/t3402-rebase-merge.sh |  1 +
 7 files changed, 24 insertions(+), 2 deletions(-)
builtin/branch.c +1 -0
index 7891dec361..07ee874617 100644
@@ -858,6 +858,7 @@ int cmd_branch(int argc, const char **argv, const char *prefix)
 		print_columns(&output, colopts, NULL);
 		string_list_clear(&output, 0);
 		ref_sorting_release(sorting);
+		ref_filter_clear(&filter);
 		return 0;
 	} else if (edit_description) {
 		const char *branch_name;
builtin/for-each-ref.c +1 -2
index 99ccb73518..c01fa6fefe 100644
@@ -120,8 +120,7 @@ int cmd_for_each_ref(int argc, const char **argv, const char *prefix)
 	strbuf_release(&err);
 	strbuf_release(&output);
 	ref_array_clear(&array);
-	free_commit_list(filter.with_commit);
-	free_commit_list(filter.no_commit);
+	ref_filter_clear(&filter);
 	ref_sorting_release(sorting);
 	strvec_clear(&vec);
 	return 0;
builtin/tag.c +1 -0
index 6b41bb7374..aab5e693fe 100644
@@ -645,6 +645,7 @@ int cmd_tag(int argc, const char **argv, const char *prefix)
 
 cleanup:
 	ref_sorting_release(sorting);
+	ref_filter_clear(&filter);
 	strbuf_release(&buf);
 	strbuf_release(&ref);
 	strbuf_release(&reflog_msg);
ref-filter.c +16 -0
index 048d277cbf..d32f426898 100644
@@ -2866,3 +2866,19 @@ int parse_opt_merge_filter(const struct option *opt, const char *arg, int unset)
 
 	return 0;
 }
+
+void ref_filter_init(struct ref_filter *filter)
+{
+	struct ref_filter blank = REF_FILTER_INIT;
+	memcpy(filter, &blank, sizeof(blank));
+}
+
+void ref_filter_clear(struct ref_filter *filter)
+{
+	oid_array_clear(&filter->points_at);
+	free_commit_list(filter->with_commit);
+	free_commit_list(filter->no_commit);
+	free_commit_list(filter->reachable_from);
+	free_commit_list(filter->unreachable_from);
+	ref_filter_init(filter);
+}
ref-filter.h +3 -0
index a920f73b29..160b807224 100644
@@ -170,4 +170,7 @@ void filter_ahead_behind(struct repository *r,
 			 struct ref_format *format,
 			 struct ref_array *array);
 
+void ref_filter_init(struct ref_filter *filter);
+void ref_filter_clear(struct ref_filter *filter);
+
 #endif /*  REF_FILTER_H  */
t/t0041-usage.sh +1 -0
index c4fc34eb18..9ea974b0c6 100755
@@ -5,6 +5,7 @@ test_description='Test commands behavior when given invalid argument value'
 GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main
 export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME
 
+TEST_PASSES_SANITIZE_LEAK=true
 . ./test-lib.sh
 
 test_expect_success 'setup ' '
t/t3402-rebase-merge.sh +1 -0
index 79b0640c00..e9e03ca4b5 100755
@@ -8,6 +8,7 @@ test_description='git rebase --merge test'
 GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main
 export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME
 
+TEST_PASSES_SANITIZE_LEAK=true
 . ./test-lib.sh
 
 T="A quick brown fox
Signature · 1 line
2.41.0.16.g26cd413590
#69 lore raw

PATCH v3 5/16 ref-filter.c: parameterize match functions over patterns

In reply to Taylor Blau

From: Jeff King <[email protected]>

match_pattern() and match_name_as_path() both take a struct ref_filter *, and then store a stack variable patterns pointing at filter->patterns.

The subsequent patch will add a new array of patterns to match over (the excluded patterns, via a new git for-each-ref --exclude option), treating the return value of these functions differently depending on which patterns are being used to match.

Tweak match_pattern() and match_name_as_path() to take an array of patterns to prepare for passing either in.

Once we start passing either in, match_pattern() will have little to do with a particular struct ref_filter * instance. To clarify this, drop it from the argument list, and replace it with the only bit of the ref_filter that we care about (filter->ignore_case).

---
 ref-filter.c | 16 +++++++++-------
 1 file changed, 9 insertions(+), 7 deletions(-)
ref-filter.c +9 -7
index d32f426898..6d91c7cb0d 100644
@@ -2104,12 +2104,12 @@ static int get_ref_atom_value(struct ref_array_item *ref, int atom,
  * matches a pattern "refs/heads/mas") or a wildcard (e.g. the same ref
  * matches "refs/heads/mas*", too).
  */
-static int match_pattern(const struct ref_filter *filter, const char *refname)
+static int match_pattern(const char **patterns, const char *refname,
+			 const int ignore_case)
 {
-	const char **patterns = filter->name_patterns;
 	unsigned flags = 0;
 
-	if (filter->ignore_case)
+	if (ignore_case)
 		flags |= WM_CASEFOLD;
 
 	/*
@@ -2134,9 +2134,10 @@ static int match_pattern(const struct ref_filter *filter, const char *refname)
  * matches a pattern "refs/heads/" but not "refs/heads/m") or a
  * wildcard (e.g. the same ref matches "refs/heads/m*", too).
  */
-static int match_name_as_path(const struct ref_filter *filter, const char *refname)
+static int match_name_as_path(const struct ref_filter *filter,
+			      const char **pattern,
+			      const char *refname)
 {
-	const char **pattern = filter->name_patterns;
 	int namelen = strlen(refname);
 	unsigned flags = WM_PATHNAME;
 
@@ -2165,8 +2166,9 @@ static int filter_pattern_match(struct ref_filter *filter, const char *refname)
 	if (!*filter->name_patterns)
 		return 1; /* No pattern always matches */
 	if (filter->match_as_path)
-		return match_name_as_path(filter, refname);
-	return match_pattern(filter, refname);
+		return match_name_as_path(filter, filter->name_patterns, refname);
+	return match_pattern(filter->name_patterns, refname,
+			     filter->ignore_case);
 }
 
 /*
Signature · 1 line
2.41.0.16.g26cd413590
#71 lore raw

PATCH v3 6/16 builtin/for-each-ref.c: add `--exclude` option

In reply to Taylor Blau

When using for-each-ref, it is sometimes convenient for the caller to be able to exclude certain parts of the references.

For example, if there are many refs/__hidden__/* references, the caller may want to emit all references *except* the hidden ones. Currently, the only way to do this is to post-process the output, like:

    $ git for-each-ref --format='%(refname)' | grep -v '^refs/hidden/'

Which is do-able, but requires processing a potentially large quantity of references.

Teach git for-each-ref a new --exclude=<pattern> option, which excludes references from the results if they match one or more excluded patterns.

This patch provides a naive implementation where the ref_filter still sees all references (including ones that it will discard) and is left to check whether each reference matches any excluded pattern(s) before emitting them.

By culling out references we know the caller doesn't care about, we can avoid allocating memory for their storage, as well as spending time sorting the output (among other things). Even the naive implementation provides a significant speed-up on a modified copy of linux.git (that has a hidden ref pointing at each commit):

    $ hyperfine \
      'git.compile for-each-ref --format="%(objectname) %(refname)" | grep -vE "[0-9a-f]{40} refs/pull/"' \
      'git.compile for-each-ref --format="%(objectname) %(refname)" --exclude refs/pull/'
    Benchmark 1: git.compile for-each-ref --format="%(objectname) %(refname)" | grep -vE "[0-9a-f]{40} refs/pull/"
      Time (mean ± σ):     820.1 ms ±   2.0 ms    [User: 703.7 ms, System: 152.0 ms]
      Range (min … max):   817.7 ms … 823.3 ms    10 runs
    Benchmark 2: git.compile for-each-ref --format="%(objectname) %(refname)" --exclude refs/pull/
      Time (mean ± σ):     106.6 ms ±   1.1 ms    [User: 99.4 ms, System: 7.1 ms]
      Range (min … max):   104.7 ms … 109.1 ms    27 runs

    Summary
      'git.compile for-each-ref --format="%(objectname) %(refname)" --exclude refs/pull/' ran
        7.69 ± 0.08 times faster than 'git.compile for-each-ref --format="%(objectname) %(refname)" | grep -vE "[0-9a-f]{40} refs/pull/"'

Subsequent patches will improve on this by avoiding visiting excluded sections of the packed-refs file in certain cases.

---
 Documentation/git-for-each-ref.txt |  6 +++++
 builtin/for-each-ref.c             |  1 +
 ref-filter.c                       | 13 +++++++++++
 ref-filter.h                       |  6 +++++
 t/t6300-for-each-ref.sh            | 35 ++++++++++++++++++++++++++++++
 5 files changed, 61 insertions(+)
Documentation/git-for-each-ref.txt +6 -0
index 1e215d4e73..5743eb5def 100644
@@ -14,6 +14,7 @@ SYNOPSIS
 		   [--points-at=<object>]
 		   [--merged[=<object>]] [--no-merged[=<object>]]
 		   [--contains[=<object>]] [--no-contains[=<object>]]
+		   [--exclude=<pattern> ...]
 
 DESCRIPTION
 -----------
@@ -102,6 +103,11 @@ OPTIONS
 	Do not print a newline after formatted refs where the format expands
 	to the empty string.
 
+--exclude=<pattern>::
+	If one or more patterns are given, only refs which do not match
+	any excluded pattern(s) are shown. Matching is done using the
+	same rules as `<pattern>` above.
+
 FIELD NAMES
 -----------
 
builtin/for-each-ref.c +1 -0
index c01fa6fefe..3384987428 100644
@@ -47,6 +47,7 @@ int cmd_for_each_ref(int argc, const char **argv, const char *prefix)
 		OPT_INTEGER( 0 , "count", &maxcount, N_("show only <n> matched refs")),
 		OPT_STRING(  0 , "format", &format.format, N_("format"), N_("format to use for the output")),
 		OPT__COLOR(&format.use_color, N_("respect format colors")),
+		OPT_REF_FILTER_EXCLUDE(&filter),
 		OPT_REF_SORT(&sorting_options),
 		OPT_CALLBACK(0, "points-at", &filter.points_at,
 			     N_("object"), N_("print only refs which points at the given object"),
ref-filter.c +13 -0
index 6d91c7cb0d..d44418efb7 100644
@@ -2171,6 +2171,15 @@ static int filter_pattern_match(struct ref_filter *filter, const char *refname)
 			     filter->ignore_case);
 }
 
+static int filter_exclude_match(struct ref_filter *filter, const char *refname)
+{
+	if (!filter->exclude.nr)
+		return 0;
+	if (filter->match_as_path)
+		return match_name_as_path(filter, filter->exclude.v, refname);
+	return match_pattern(filter->exclude.v, refname, filter->ignore_case);
+}
+
 /*
  * This is the same as for_each_fullref_in(), but it tries to iterate
  * only over the patterns we'll care about. Note that it _doesn't_ do a full
@@ -2338,6 +2347,9 @@ static int ref_filter_handler(const char *refname, const struct object_id *oid,
 	if (!filter_pattern_match(filter, refname))
 		return 0;
 
+	if (filter_exclude_match(filter, refname))
+		return 0;
+
 	if (filter->points_at.nr && !match_points_at(&filter->points_at, oid, refname))
 		return 0;
 
@@ -2877,6 +2889,7 @@ void ref_filter_init(struct ref_filter *filter)
 
 void ref_filter_clear(struct ref_filter *filter)
 {
+	strvec_clear(&filter->exclude);
 	oid_array_clear(&filter->points_at);
 	free_commit_list(filter->with_commit);
 	free_commit_list(filter->no_commit);
ref-filter.h +6 -0
index 160b807224..1524bc463a 100644
@@ -6,6 +6,7 @@
 #include "refs.h"
 #include "commit.h"
 #include "string-list.h"
+#include "strvec.h"
 
 /* Quoting styles */
 #define QUOTE_NONE 0
@@ -59,6 +60,7 @@ struct ref_array {
 
 struct ref_filter {
 	const char **name_patterns;
+	struct strvec exclude;
 	struct oid_array points_at;
 	struct commit_list *with_commit;
 	struct commit_list *no_commit;
@@ -94,6 +96,7 @@ struct ref_format {
 
 #define REF_FILTER_INIT { \
 	.points_at = OID_ARRAY_INIT, \
+	.exclude = STRVEC_INIT, \
 }
 #define REF_FORMAT_INIT {             \
 	.use_color = -1,              \
@@ -112,6 +115,9 @@ struct ref_format {
 #define OPT_REF_SORT(var) \
 	OPT_STRING_LIST(0, "sort", (var), \
 			N_("key"), N_("field name to sort on"))
+#define OPT_REF_FILTER_EXCLUDE(var) \
+	OPT_STRVEC(0, "exclude", &(var)->exclude, \
+		   N_("pattern"), N_("exclude refs which match pattern"))
 
 /*
  * API for filtering a set of refs. Based on the type of refs the user
t/t6300-for-each-ref.sh +35 -0
index 5c00607608..7e8d578522 100755
@@ -447,6 +447,41 @@ test_expect_success 'exercise glob patterns with prefixes' '
 	test_cmp expected actual
 '
 
+cat >expected <<\EOF
+refs/tags/bar
+refs/tags/baz
+refs/tags/testtag
+EOF
+
+test_expect_success 'exercise patterns with prefix exclusions' '
+	for tag in foo/one foo/two foo/three bar baz
+	do
+		git tag "$tag" || return 1
+	done &&
+	test_when_finished "git tag -d foo/one foo/two foo/three bar baz" &&
+	git for-each-ref --format="%(refname)" \
+		refs/tags/ --exclude=refs/tags/foo >actual &&
+	test_cmp expected actual
+'
+
+cat >expected <<\EOF
+refs/tags/bar
+refs/tags/baz
+refs/tags/foo/one
+refs/tags/testtag
+EOF
+
+test_expect_success 'exercise patterns with pattern exclusions' '
+	for tag in foo/one foo/two foo/three bar baz
+	do
+		git tag "$tag" || return 1
+	done &&
+	test_when_finished "git tag -d foo/one foo/two foo/three bar baz" &&
+	git for-each-ref --format="%(refname)" \
+		refs/tags/ --exclude="refs/tags/foo/t*" >actual &&
+	test_cmp expected actual
+'
+
 cat >expected <<\EOF
 'refs/heads/main'
 'refs/remotes/origin/main'
Signature · 1 line
2.41.0.16.g26cd413590
#72 lore raw

PATCH v3 7/16 refs: plumb `exclude_patterns` argument throughout

In reply to Taylor Blau

The subsequent patch will want to access an optional excluded_patterns array within refs/packed-backend.c that will cull out certain references matching any of the given patterns on a best-effort basis.

To do so, the refs subsystem needs to be updated to pass this value across a number of different locations.

Prepare for a future patch by introducing this plumbing now, passing NULLs at top-level APIs in order to make that patch less noisy and more easily readable.

---
 ls-refs.c             |  2 +-
 ref-filter.c          |  5 +++--
 refs.c                | 32 +++++++++++++++++++-------------
 refs.h                |  8 +++++++-
 refs/debug.c          |  5 +++--
 refs/files-backend.c  |  5 +++--
 refs/packed-backend.c |  5 +++--
 refs/refs-internal.h  |  7 ++++---
 revision.c            |  2 +-
 9 files changed, 44 insertions(+), 27 deletions(-)
ls-refs.c +1 -1
index f385938b64..6f490b2d9c 100644
@@ -193,7 +193,7 @@ int ls_refs(struct repository *r, struct packet_reader *request)
 		strvec_push(&data.prefixes, "");
 	refs_for_each_fullref_in_prefixes(get_main_ref_store(r),
 					  get_git_namespace(), data.prefixes.v,
-					  send_ref, &data);
+					  NULL, send_ref, &data);
 	packet_fflush(stdout);
 	strvec_clear(&data.prefixes);
 	strbuf_release(&data.buf);
ref-filter.c +3 -2
index d44418efb7..717c3c4bcf 100644
@@ -2209,12 +2209,13 @@ static int for_each_fullref_in_pattern(struct ref_filter *filter,
 
 	if (!filter->name_patterns[0]) {
 		/* no patterns; we have to look at everything */
-		return for_each_fullref_in("", cb, cb_data);
+		return refs_for_each_fullref_in(get_main_ref_store(the_repository),
+						 "", NULL, cb, cb_data);
 	}
 
 	return refs_for_each_fullref_in_prefixes(get_main_ref_store(the_repository),
 						 NULL, filter->name_patterns,
-						 cb, cb_data);
+						 NULL, cb, cb_data);
 }
 
 /*
refs.c +19 -13
index b9b77d2eff..538bde644e 100644
@@ -1526,7 +1526,9 @@ int head_ref(each_ref_fn fn, void *cb_data)
 
 struct ref_iterator *refs_ref_iterator_begin(
 		struct ref_store *refs,
-		const char *prefix, int trim,
+		const char *prefix,
+		const char **exclude_patterns,
+		int trim,
 		enum do_for_each_ref_flags flags)
 {
 	struct ref_iterator *iter;
@@ -1542,8 +1544,7 @@ struct ref_iterator *refs_ref_iterator_begin(
 		}
 	}
 
-	iter = refs->be->iterator_begin(refs, prefix, flags);
-
+	iter = refs->be->iterator_begin(refs, prefix, exclude_patterns, flags);
 	/*
 	 * `iterator_begin()` already takes care of prefix, but we
 	 * might need to do some trimming:
@@ -1577,7 +1578,7 @@ static int do_for_each_repo_ref(struct repository *r, const char *prefix,
 	if (!refs)
 		return 0;
 
-	iter = refs_ref_iterator_begin(refs, prefix, trim, flags);
+	iter = refs_ref_iterator_begin(refs, prefix, NULL, trim, flags);
 
 	return do_for_each_repo_ref_iterator(r, iter, fn, cb_data);
 }
@@ -1599,6 +1600,7 @@ static int do_for_each_ref_helper(struct repository *r,
 }
 
 static int do_for_each_ref(struct ref_store *refs, const char *prefix,
+			   const char **exclude_patterns,
 			   each_ref_fn fn, int trim,
 			   enum do_for_each_ref_flags flags, void *cb_data)
 {
@@ -1608,7 +1610,8 @@ static int do_for_each_ref(struct ref_store *refs, const char *prefix,
 	if (!refs)
 		return 0;
 
-	iter = refs_ref_iterator_begin(refs, prefix, trim, flags);
+	iter = refs_ref_iterator_begin(refs, prefix, exclude_patterns, trim,
+				       flags);
 
 	return do_for_each_repo_ref_iterator(the_repository, iter,
 					do_for_each_ref_helper, &hp);
@@ -1616,7 +1619,7 @@ static int do_for_each_ref(struct ref_store *refs, const char *prefix,
 
 int refs_for_each_ref(struct ref_store *refs, each_ref_fn fn, void *cb_data)
 {
-	return do_for_each_ref(refs, "", fn, 0, 0, cb_data);
+	return do_for_each_ref(refs, "", NULL, fn, 0, 0, cb_data);
 }
 
 int for_each_ref(each_ref_fn fn, void *cb_data)
@@ -1627,7 +1630,7 @@ int for_each_ref(each_ref_fn fn, void *cb_data)
 int refs_for_each_ref_in(struct ref_store *refs, const char *prefix,
 			 each_ref_fn fn, void *cb_data)
 {
-	return do_for_each_ref(refs, prefix, fn, strlen(prefix), 0, cb_data);
+	return do_for_each_ref(refs, prefix, NULL, fn, strlen(prefix), 0, cb_data);
 }
 
 int for_each_ref_in(const char *prefix, each_ref_fn fn, void *cb_data)
@@ -1638,13 +1641,14 @@ int for_each_ref_in(const char *prefix, each_ref_fn fn, void *cb_data)
 int for_each_fullref_in(const char *prefix, each_ref_fn fn, void *cb_data)
 {
 	return do_for_each_ref(get_main_ref_store(the_repository),
-			       prefix, fn, 0, 0, cb_data);
+			       prefix, NULL, fn, 0, 0, cb_data);
 }
 
 int refs_for_each_fullref_in(struct ref_store *refs, const char *prefix,
+			     const char **exclude_patterns,
 			     each_ref_fn fn, void *cb_data)
 {
-	return do_for_each_ref(refs, prefix, fn, 0, 0, cb_data);
+	return do_for_each_ref(refs, prefix, exclude_patterns, fn, 0, 0, cb_data);
 }
 
 int for_each_replace_ref(struct repository *r, each_repo_ref_fn fn, void *cb_data)
@@ -1661,14 +1665,14 @@ int for_each_namespaced_ref(each_ref_fn fn, void *cb_data)
 	int ret;
 	strbuf_addf(&buf, "%srefs/", get_git_namespace());
 	ret = do_for_each_ref(get_main_ref_store(the_repository),
-			      buf.buf, fn, 0, 0, cb_data);
+			      buf.buf, NULL, fn, 0, 0, cb_data);
 	strbuf_release(&buf);
 	return ret;
 }
 
 int refs_for_each_rawref(struct ref_store *refs, each_ref_fn fn, void *cb_data)
 {
-	return do_for_each_ref(refs, "", fn, 0,
+	return do_for_each_ref(refs, "", NULL, fn, 0,
 			       DO_FOR_EACH_INCLUDE_BROKEN, cb_data);
 }
 
@@ -1738,6 +1742,7 @@ static void find_longest_prefixes(struct string_list *out,
 int refs_for_each_fullref_in_prefixes(struct ref_store *ref_store,
 				      const char *namespace,
 				      const char **patterns,
+				      const char **exclude_patterns,
 				      each_ref_fn fn, void *cb_data)
 {
 	struct string_list prefixes = STRING_LIST_INIT_DUP;
@@ -1753,7 +1758,8 @@ int refs_for_each_fullref_in_prefixes(struct ref_store *ref_store,
 
 	for_each_string_list_item(prefix, &prefixes) {
 		strbuf_addstr(&buf, prefix->string);
-		ret = refs_for_each_fullref_in(ref_store, buf.buf, fn, cb_data);
+		ret = refs_for_each_fullref_in(ref_store, buf.buf,
+					       exclude_patterns, fn, cb_data);
 		if (ret)
 			break;
 		strbuf_setlen(&buf, namespace_len);
@@ -2408,7 +2414,7 @@ int refs_verify_refname_available(struct ref_store *refs,
 	strbuf_addstr(&dirname, refname + dirname.len);
 	strbuf_addch(&dirname, '/');
 
-	iter = refs_ref_iterator_begin(refs, dirname.buf, 0,
+	iter = refs_ref_iterator_begin(refs, dirname.buf, NULL, 0,
 				       DO_FOR_EACH_INCLUDE_BROKEN);
 	while ((ok = ref_iterator_advance(iter)) == ITER_OK) {
 		if (skip &&
refs.h +7 -1
index 123cfa4424..d672d636cf 100644
@@ -338,6 +338,7 @@ int for_each_ref(each_ref_fn fn, void *cb_data);
 int for_each_ref_in(const char *prefix, each_ref_fn fn, void *cb_data);
 
 int refs_for_each_fullref_in(struct ref_store *refs, const char *prefix,
+			     const char **exclude_patterns,
 			     each_ref_fn fn, void *cb_data);
 int for_each_fullref_in(const char *prefix, each_ref_fn fn, void *cb_data);
 
@@ -345,10 +346,15 @@ int for_each_fullref_in(const char *prefix, each_ref_fn fn, void *cb_data);
  * iterate all refs in "patterns" by partitioning patterns into disjoint sets
  * and iterating the longest-common prefix of each set.
  *
+ * references matching any pattern in "exclude_patterns" are omitted from the
+ * result set on a best-effort basis.
+ *
  * callers should be prepared to ignore references that they did not ask for.
  */
 int refs_for_each_fullref_in_prefixes(struct ref_store *refs,
-				      const char *namespace, const char **patterns,
+				      const char *namespace,
+				      const char **patterns,
+				      const char **exclude_patterns,
 				      each_ref_fn fn, void *cb_data);
 
 /**
refs/debug.c +3 -2
index 6f11e6de46..328f894177 100644
@@ -229,11 +229,12 @@ static struct ref_iterator_vtable debug_ref_iterator_vtable = {
 
 static struct ref_iterator *
 debug_ref_iterator_begin(struct ref_store *ref_store, const char *prefix,
-			 unsigned int flags)
+			 const char **exclude_patterns, unsigned int flags)
 {
 	struct debug_ref_store *drefs = (struct debug_ref_store *)ref_store;
 	struct ref_iterator *res =
-		drefs->refs->be->iterator_begin(drefs->refs, prefix, flags);
+		drefs->refs->be->iterator_begin(drefs->refs, prefix,
+						exclude_patterns, flags);
 	struct debug_ref_iterator *diter = xcalloc(1, sizeof(*diter));
 	base_ref_iterator_init(&diter->base, &debug_ref_iterator_vtable, 1);
 	diter->iter = res;
refs/files-backend.c +3 -2
index bca7b851c5..3bc3c57c05 100644
@@ -829,7 +829,8 @@ static struct ref_iterator_vtable files_ref_iterator_vtable = {
 
 static struct ref_iterator *files_ref_iterator_begin(
 		struct ref_store *ref_store,
-		const char *prefix, unsigned int flags)
+		const char *prefix, const char **exclude_patterns,
+		unsigned int flags)
 {
 	struct files_ref_store *refs;
 	struct ref_iterator *loose_iter, *packed_iter, *overlay_iter;
@@ -874,7 +875,7 @@ static struct ref_iterator *files_ref_iterator_begin(
 	 * the packed and loose references.
 	 */
 	packed_iter = refs_ref_iterator_begin(
-			refs->packed_ref_store, prefix, 0,
+			refs->packed_ref_store, prefix, exclude_patterns, 0,
 			DO_FOR_EACH_INCLUDE_BROKEN);
 
 	overlay_iter = overlay_ref_iterator_begin(loose_iter, packed_iter);
refs/packed-backend.c +3 -2
index 5b412a133b..176bd3905b 100644
@@ -924,7 +924,8 @@ static struct ref_iterator_vtable packed_ref_iterator_vtable = {
 
 static struct ref_iterator *packed_ref_iterator_begin(
 		struct ref_store *ref_store,
-		const char *prefix, unsigned int flags)
+		const char *prefix, const char **exclude_patterns,
+		unsigned int flags)
 {
 	struct packed_ref_store *refs;
 	struct snapshot *snapshot;
@@ -1149,7 +1150,7 @@ static int write_with_updates(struct packed_ref_store *refs,
 	 * list of refs is exhausted, set iter to NULL. When the list
 	 * of updates is exhausted, leave i set to updates->nr.
 	 */
-	iter = packed_ref_iterator_begin(&refs->base, "",
+	iter = packed_ref_iterator_begin(&refs->base, "", NULL,
 					 DO_FOR_EACH_INCLUDE_BROKEN);
 	if ((ok = ref_iterator_advance(iter)) != ITER_OK)
 		iter = NULL;
refs/refs-internal.h +4 -3
index a85d113123..28a11b9d61 100644
@@ -367,8 +367,8 @@ int is_empty_ref_iterator(struct ref_iterator *ref_iterator);
  */
 struct ref_iterator *refs_ref_iterator_begin(
 		struct ref_store *refs,
-		const char *prefix, int trim,
-		enum do_for_each_ref_flags flags);
+		const char *prefix, const char **exclude_patterns,
+		int trim, enum do_for_each_ref_flags flags);
 
 /*
  * A callback function used to instruct merge_ref_iterator how to
@@ -570,7 +570,8 @@ typedef int copy_ref_fn(struct ref_store *ref_store,
  */
 typedef struct ref_iterator *ref_iterator_begin_fn(
 		struct ref_store *ref_store,
-		const char *prefix, unsigned int flags);
+		const char *prefix, const char **exclude_patterns,
+		unsigned int flags);
 
 /* reflog functions */
 
revision.c +1 -1
index b33cc1d106..89953592f9 100644
@@ -2670,7 +2670,7 @@ static int for_each_bisect_ref(struct ref_store *refs, each_ref_fn fn,
 	struct strbuf bisect_refs = STRBUF_INIT;
 	int status;
 	strbuf_addf(&bisect_refs, "refs/bisect/%s", term);
-	status = refs_for_each_fullref_in(refs, bisect_refs.buf, fn, cb_data);
+	status = refs_for_each_fullref_in(refs, bisect_refs.buf, NULL, fn, cb_data);
 	strbuf_release(&bisect_refs);
 	return status;
 }
Signature · 1 line
2.41.0.16.g26cd413590
#75 lore raw

PATCH v3 8/16 refs/packed-backend.c: refactor `find_reference_location()`

In reply to Taylor Blau

The function find_reference_location() is used to perform a binary search-like function over the contents of a repository's $GIT_DIR/packed-refs file.

The search it implements is unlike a standard binary search in that the records it searches over are not of a fixed width, so the comparison must locate the end of a record before comparing it.

Extract the core routine of find_reference_location() in order to implement a function in the following patch which will find the first location in the packed-refs file that *doesn't* match the given pattern.

The behavior of find_reference_location() is unchanged.

---
 refs/packed-backend.c | 38 ++++++++++++++++++++++----------------
 1 file changed, 22 insertions(+), 16 deletions(-)
refs/packed-backend.c +22 -16
index 176bd3905b..33639f73e1 100644
@@ -527,22 +527,8 @@ static int load_contents(struct snapshot *snapshot)
 	return 1;
 }
 
-/*
- * Find the place in `snapshot->buf` where the start of the record for
- * `refname` starts. If `mustexist` is true and the reference doesn't
- * exist, then return NULL. If `mustexist` is false and the reference
- * doesn't exist, then return the point where that reference would be
- * inserted, or `snapshot->eof` (which might be NULL) if it would be
- * inserted at the end of the file. In the latter mode, `refname`
- * doesn't have to be a proper reference name; for example, one could
- * search for "refs/replace/" to find the start of any replace
- * references.
- *
- * The record is sought using a binary search, so `snapshot->buf` must
- * be sorted.
- */
-static const char *find_reference_location(struct snapshot *snapshot,
-					   const char *refname, int mustexist)
+static const char *find_reference_location_1(struct snapshot *snapshot,
+					     const char *refname, int mustexist)
 {
 	/*
 	 * This is not *quite* a garden-variety binary search, because
@@ -588,6 +574,26 @@ static const char *find_reference_location(struct snapshot *snapshot,
 		return lo;
 }
 
+/*
+ * Find the place in `snapshot->buf` where the start of the record for
+ * `refname` starts. If `mustexist` is true and the reference doesn't
+ * exist, then return NULL. If `mustexist` is false and the reference
+ * doesn't exist, then return the point where that reference would be
+ * inserted, or `snapshot->eof` (which might be NULL) if it would be
+ * inserted at the end of the file. In the latter mode, `refname`
+ * doesn't have to be a proper reference name; for example, one could
+ * search for "refs/replace/" to find the start of any replace
+ * references.
+ *
+ * The record is sought using a binary search, so `snapshot->buf` must
+ * be sorted.
+ */
+static const char *find_reference_location(struct snapshot *snapshot,
+					   const char *refname, int mustexist)
+{
+	return find_reference_location_1(snapshot, refname, mustexist);
+}
+
 /*
  * Create a newly-allocated `snapshot` of the `packed-refs` file in
  * its current state and return it. The return value will already have
Signature · 1 line
2.41.0.16.g26cd413590
#76 lore raw

PATCH v3 9/16 refs/packed-backend.c: implement jump lists to avoid excluded pattern(s)

In reply to Taylor Blau

When iterating through the packed-refs file in order to answer a query like:

    $ git for-each-ref --exclude=refs/__hidden__

it would be useful to avoid walking over all of the entries in refs/__hidden__/* when possible, since we know that the ref-filter code is going to throw them away anyways.

In certain circumstances, doing so is possible. The algorithm for doing so is as follows:

- For each excluded pattern, find the first record that matches it, and the first record that *doesn't* match it (i.e. the location you'd next want to consider when excluding that pattern).

- Sort the set of excluded regions from the previous step in ascending order of the first location within the packed-refs file that matches.

- Clean up the results from the previous step: discard empty regions, and combine adjacent regions.

Then when iterating through the packed-refs file, if iter->pos is ever contained in one of the regions from the previous steps, advance iter->pos past the end of that region, and continue enumeration.

Note that we only perform this optimization when none of the excluded pattern(s) have special meta-characters in them. For a pattern like "refs/foo[ac]", the excluded regions ("refs/fooa", "refs/fooc", and everything underneath them) are not connected. A future implementation that handles this case may split the character class (pretending as if two patterns were excluded: "refs/fooa", and "refs/fooc").

There are a few other gotchas worth considering. First, note that the jump list is sorted, so once we jump past a region, we can avoid considering it (or any regions preceding it) again. The member jump_pos is used to track the first next-possible region to jump through.

Second, note that the exclusion list is best-effort, since we do not handle loose references, and because of the meta-character issue above.

In repositories with a large number of hidden references, the speed-up can be significant. Tests here are done with a copy of linux.git with a reference "refs/pull/N" pointing at every commit, as in:

    $ git rev-list HEAD | awk '{ print "create refs/pull/" NR " " $0 }' |
        git update-ref --stdin
    $ git pack-refs --all

, it is significantly faster to have for-each-ref jump over the excluded references, as opposed to filtering them out after the fact:

    $ hyperfine \
      'git for-each-ref --format="%(objectname) %(refname)" | grep -vE "^[0-9a-f]{40} refs/pull/"' \
      'git.compile for-each-ref --format="%(objectname) %(refname)" --exclude="refs/pull"'
    Benchmark 1: git for-each-ref --format="%(objectname) %(refname)" | grep -vE "^[0-9a-f]{40} refs/pull/"
      Time (mean ± σ):     802.7 ms ±   2.1 ms    [User: 691.6 ms, System: 147.0 ms]
      Range (min … max):   800.0 ms … 807.7 ms    10 runs
    Benchmark 2: git.compile for-each-ref --format="%(objectname) %(refname)" --exclude="refs/pull"
      Time (mean ± σ):       4.7 ms ±   0.3 ms    [User: 0.7 ms, System: 4.0 ms]
      Range (min … max):     4.3 ms …   6.7 ms    422 runs

    Summary
      'git.compile for-each-ref --format="%(objectname) %(refname)" --exclude="refs/pull"' ran
      172.03 ± 9.60 times faster than 'git for-each-ref --format="%(objectname) %(refname)" | grep -vE "^[0-9a-f]{40} refs/pull/"'

Using the jump list is fairly straightforward (see the changes to refs/packed-backend.c::next_record()), but constructing the list is not. To ensure that the construction is correct, add a new suite of tests in t1419 covering various corner cases (overlapping regions, partially overlapping regions, adjacent regions, etc.).

---
 ref-filter.c              |   5 +-
 refs/packed-backend.c     | 166 ++++++++++++++++++++++++++++++++++++--
 t/helper/test-ref-store.c |  10 +++
 t/t1419-exclude-refs.sh   | 101 +++++++++++++++++++++++
 4 files changed, 274 insertions(+), 8 deletions(-)
 create mode 100755 t/t1419-exclude-refs.sh
ref-filter.c +3 -2
index 717c3c4bcf..ddc7f5204f 100644
@@ -2210,12 +2210,13 @@ static int for_each_fullref_in_pattern(struct ref_filter *filter,
 	if (!filter->name_patterns[0]) {
 		/* no patterns; we have to look at everything */
 		return refs_for_each_fullref_in(get_main_ref_store(the_repository),
-						 "", NULL, cb, cb_data);
+						 "", filter->exclude.v, cb, cb_data);
 	}
 
 	return refs_for_each_fullref_in_prefixes(get_main_ref_store(the_repository),
 						 NULL, filter->name_patterns,
-						 NULL, cb, cb_data);
+						 filter->exclude.v,
+						 cb, cb_data);
 }
 
 /*
refs/packed-backend.c +160 -6

233 lines not on this page · raw message

t/helper/test-ref-store.c +10 -0
index 6d8f844e9c..2bff003f7c 100644
@@ -175,6 +175,15 @@ static int cmd_for_each_ref(struct ref_store *refs, const char **argv)
 	return refs_for_each_ref_in(refs, prefix, each_ref, NULL);
 }
 
+static int cmd_for_each_ref__exclude(struct ref_store *refs, const char **argv)
+{
+	const char *prefix = notnull(*argv++, "prefix");
+	const char **exclude_patterns = argv;
+
+	return refs_for_each_fullref_in(refs, prefix, exclude_patterns, each_ref,
+					NULL);
+}
+
 static int cmd_resolve_ref(struct ref_store *refs, const char **argv)
 {
 	struct object_id oid = *null_oid();
@@ -307,6 +316,7 @@ static struct command commands[] = {
 	{ "delete-refs", cmd_delete_refs },
 	{ "rename-ref", cmd_rename_ref },
 	{ "for-each-ref", cmd_for_each_ref },
+	{ "for-each-ref--exclude", cmd_for_each_ref__exclude },
 	{ "resolve-ref", cmd_resolve_ref },
 	{ "verify-ref", cmd_verify_ref },
 	{ "for-each-reflog", cmd_for_each_reflog },
t/t1419-exclude-refs.sh new file +101 -0

101 lines not on this page · raw message

Signature · 1 line
2.41.0.16.g26cd413590
#80 lore raw

PATCH v3 10/16 refs/packed-backend.c: add trace2 counters for jump list

In reply to Taylor Blau

The previous commit added low-level tests to ensure that the packed-refs iterator did not enumerate excluded sections of the refspace.

However, there was no guarantee that these sections weren't being visited, only that they were being suppressed from the output. To harden these tests, add a trace2 counter which tracks the number of regions skipped by the packed-refs iterator, and assert on its value.

---
 refs/packed-backend.c   |  2 ++
 t/t1419-exclude-refs.sh | 59 ++++++++++++++++++++++++++++-------------
 trace2.h                |  2 ++
 trace2/tr2_ctr.c        |  5 ++++
 4 files changed, 49 insertions(+), 19 deletions(-)
refs/packed-backend.c +2 -0
index 67327e579c..7ba9fa2bb8 100644
@@ -12,6 +12,7 @@
 #include "../chdir-notify.h"
 #include "../wrapper.h"
 #include "../write-or-die.h"
+#include "../trace2.h"
 
 enum mmap_strategy {
 	/*
@@ -845,6 +846,7 @@ static int next_record(struct packed_ref_iterator *iter)
 		iter->jump_pos++;
 		if (iter->pos < curr->end) {
 			iter->pos = curr->end;
+			trace2_counter_add(TRACE2_COUNTER_ID_PACKED_REFS_JUMPS, 1);
 			break;
 		}
 	}
t/t1419-exclude-refs.sh +40 -19

120 lines not on this page · raw message

trace2.h +2 -0
index 4ced30c0db..9452e291f5 100644
@@ -551,6 +551,8 @@ enum trace2_counter_id {
 	TRACE2_COUNTER_ID_TEST1 = 0, /* emits summary event only */
 	TRACE2_COUNTER_ID_TEST2,     /* emits summary and thread events */
 
+	TRACE2_COUNTER_ID_PACKED_REFS_JUMPS, /* counts number of jumps */
+
 	/* Add additional counter definitions before here. */
 	TRACE2_NUMBER_OF_COUNTERS
 };
trace2/tr2_ctr.c +5 -0
index b342d3b1a3..50570d0165 100644
@@ -27,6 +27,11 @@ static struct tr2_counter_metadata tr2_counter_metadata[TRACE2_NUMBER_OF_COUNTER
 		.name = "test2",
 		.want_per_thread_events = 1,
 	},
+	[TRACE2_COUNTER_ID_PACKED_REFS_JUMPS] = {
+		.category = "packed-refs",
+		.name = "jumps_made",
+		.want_per_thread_events = 0,
+	},
 
 	/* Add additional metadata before here. */
 };
Signature · 1 line
2.41.0.16.g26cd413590
#83 lore raw

PATCH v3 11/16 revision.h: store hidden refs in a `strvec`

In reply to Taylor Blau

In subsequent commits, it will be convenient to have a 'const char **' of hidden refs (matching transfer.hiderefs, uploadpack.hideRefs, etc.), instead of a string_list.

Convert spots throughout the tree that store the list of hidden refs from a string_list to a strvec.

Note that in parse_hide_refs_config() there is an ugly const-cast used to avoid an extra copy of each value before trimming any trailing slash characters. This could instead be written as:

    ref = xstrdup(value);
    len = strlen(ref);
    while (len && ref[len - 1] == '/')
            ref[--len] = '\0';
    strvec_push(hide_refs, ref);
    free(ref);

but the double-copy (once when calling xstrdup(), and another via strvec_push()) is wasteful.

---
 builtin/receive-pack.c |  4 ++--
 ls-refs.c              |  6 +++---
 refs.c                 | 11 ++++++-----
 refs.h                 |  4 ++--
 revision.c             |  2 +-
 revision.h             |  5 +++--
 upload-pack.c          | 10 +++++-----
 7 files changed, 22 insertions(+), 20 deletions(-)
builtin/receive-pack.c +2 -2
index 1a31a58367..1a8472eddc 100644
@@ -90,7 +90,7 @@ static struct object_id push_cert_oid;
 static struct signature_check sigcheck;
 static const char *push_cert_nonce;
 static const char *cert_nonce_seed;
-static struct string_list hidden_refs = STRING_LIST_INIT_DUP;
+static struct strvec hidden_refs = STRVEC_INIT;
 
 static const char *NONCE_UNSOLICITED = "UNSOLICITED";
 static const char *NONCE_BAD = "BAD";
@@ -2619,7 +2619,7 @@ int cmd_receive_pack(int argc, const char **argv, const char *prefix)
 		packet_flush(1);
 	oid_array_clear(&shallow);
 	oid_array_clear(&ref);
-	string_list_clear(&hidden_refs, 0);
+	strvec_clear(&hidden_refs);
 	free((void *)push_cert_nonce);
 	return 0;
 }
ls-refs.c +3 -3
index 6f490b2d9c..8c3181d051 100644
@@ -72,7 +72,7 @@ struct ls_refs_data {
 	unsigned symrefs;
 	struct strvec prefixes;
 	struct strbuf buf;
-	struct string_list hidden_refs;
+	struct strvec hidden_refs;
 	unsigned unborn : 1;
 };
 
@@ -155,7 +155,7 @@ int ls_refs(struct repository *r, struct packet_reader *request)
 	memset(&data, 0, sizeof(data));
 	strvec_init(&data.prefixes);
 	strbuf_init(&data.buf, 0);
-	string_list_init_dup(&data.hidden_refs);
+	strvec_init(&data.hidden_refs);
 
 	git_config(ls_refs_config, &data);
 
@@ -197,7 +197,7 @@ int ls_refs(struct repository *r, struct packet_reader *request)
 	packet_fflush(stdout);
 	strvec_clear(&data.prefixes);
 	strbuf_release(&data.buf);
-	string_list_clear(&data.hidden_refs, 0);
+	strvec_clear(&data.hidden_refs);
 	return 0;
 }
 
refs.c +6 -5
index 538bde644e..ec4d5b9101 100644
@@ -1427,7 +1427,7 @@ char *shorten_unambiguous_ref(const char *refname, int strict)
 }
 
 int parse_hide_refs_config(const char *var, const char *value, const char *section,
-			   struct string_list *hide_refs)
+			   struct strvec *hide_refs)
 {
 	const char *key;
 	if (!strcmp("transfer.hiderefs", var) ||
@@ -1438,22 +1438,23 @@ int parse_hide_refs_config(const char *var, const char *value, const char *secti
 
 		if (!value)
 			return config_error_nonbool(var);
-		ref = xstrdup(value);
+
+		/* drop const to remove trailing '/' characters */
+		ref = (char *)strvec_push(hide_refs, value);
 		len = strlen(ref);
 		while (len && ref[len - 1] == '/')
 			ref[--len] = '\0';
-		string_list_append_nodup(hide_refs, ref);
 	}
 	return 0;
 }
 
 int ref_is_hidden(const char *refname, const char *refname_full,
-		  const struct string_list *hide_refs)
+		  const struct strvec *hide_refs)
 {
 	int i;
 
 	for (i = hide_refs->nr - 1; i >= 0; i--) {
-		const char *match = hide_refs->items[i].string;
+		const char *match = hide_refs->v[i];
 		const char *subject;
 		int neg = 0;
 		const char *p;
refs.h +2 -2
index d672d636cf..a7751a1fc9 100644
@@ -810,7 +810,7 @@ int update_ref(const char *msg, const char *refname,
 	       unsigned int flags, enum action_on_err onerr);
 
 int parse_hide_refs_config(const char *var, const char *value, const char *,
-			   struct string_list *);
+			   struct strvec *);
 
 /*
  * Check whether a ref is hidden. If no namespace is set, both the first and
@@ -820,7 +820,7 @@ int parse_hide_refs_config(const char *var, const char *value, const char *,
  * the ref is outside that namespace, the first parameter is NULL. The second
  * parameter always points to the full ref name.
  */
-int ref_is_hidden(const char *, const char *, const struct string_list *);
+int ref_is_hidden(const char *, const char *, const struct strvec *);
 
 /* Is this a per-worktree ref living in the refs/ namespace? */
 int is_per_worktree_ref(const char *refname);
revision.c +1 -1
index 89953592f9..7c9367a266 100644
@@ -1558,7 +1558,7 @@ void init_ref_exclusions(struct ref_exclusions *exclusions)
 void clear_ref_exclusions(struct ref_exclusions *exclusions)
 {
 	string_list_clear(&exclusions->excluded_refs, 0);
-	string_list_clear(&exclusions->hidden_refs, 0);
+	strvec_clear(&exclusions->hidden_refs);
 	exclusions->hidden_refs_configured = 0;
 }
 
revision.h +3 -2
index 31828748dc..94f035fa22 100644
@@ -10,6 +10,7 @@
 #include "decorate.h"
 #include "ident.h"
 #include "list-objects-filter-options.h"
+#include "strvec.h"
 
 /**
  * The revision walking API offers functions to build a list of revisions
@@ -95,7 +96,7 @@ struct ref_exclusions {
 	 * Hidden refs is a list of patterns that is to be hidden via
 	 * `ref_is_hidden()`.
 	 */
-	struct string_list hidden_refs;
+	struct strvec hidden_refs;
 
 	/*
 	 * Indicates whether hidden refs have been configured. This is to
@@ -110,7 +111,7 @@ struct ref_exclusions {
  */
 #define REF_EXCLUSIONS_INIT { \
 	.excluded_refs = STRING_LIST_INIT_DUP, \
-	.hidden_refs = STRING_LIST_INIT_DUP, \
+	.hidden_refs = STRVEC_INIT, \
 }
 
 struct oidset;
upload-pack.c +5 -5
index d3312006a3..1a213ed775 100644
@@ -69,7 +69,7 @@ struct upload_pack_data {
 	struct object_array have_obj;
 	struct oid_array haves;					/* v2 only */
 	struct string_list wanted_refs;				/* v2 only */
-	struct string_list hidden_refs;
+	struct strvec hidden_refs;
 
 	struct object_array shallows;
 	struct string_list deepen_not;
@@ -127,7 +127,7 @@ static void upload_pack_data_init(struct upload_pack_data *data)
 {
 	struct string_list symref = STRING_LIST_INIT_DUP;
 	struct string_list wanted_refs = STRING_LIST_INIT_DUP;
-	struct string_list hidden_refs = STRING_LIST_INIT_DUP;
+	struct strvec hidden_refs = STRVEC_INIT;
 	struct object_array want_obj = OBJECT_ARRAY_INIT;
 	struct object_array have_obj = OBJECT_ARRAY_INIT;
 	struct oid_array haves = OID_ARRAY_INIT;
@@ -162,7 +162,7 @@ static void upload_pack_data_clear(struct upload_pack_data *data)
 {
 	string_list_clear(&data->symref, 1);
 	string_list_clear(&data->wanted_refs, 1);
-	string_list_clear(&data->hidden_refs, 0);
+	strvec_clear(&data->hidden_refs);
 	object_array_clear(&data->want_obj);
 	object_array_clear(&data->have_obj);
 	oid_array_clear(&data->haves);
@@ -1170,7 +1170,7 @@ static void receive_needs(struct upload_pack_data *data,
 
 /* return non-zero if the ref is hidden, otherwise 0 */
 static int mark_our_ref(const char *refname, const char *refname_full,
-			const struct object_id *oid, const struct string_list *hidden_refs)
+			const struct object_id *oid, const struct strvec *hidden_refs)
 {
 	struct object *o = lookup_unknown_object(the_repository, oid);
 
@@ -1465,7 +1465,7 @@ static int parse_want(struct packet_writer *writer, const char *line,
 
 static int parse_want_ref(struct packet_writer *writer, const char *line,
 			  struct string_list *wanted_refs,
-			  struct string_list *hidden_refs,
+			  struct strvec *hidden_refs,
 			  struct object_array *want_obj)
 {
 	const char *refname_nons;
Signature · 1 line
2.41.0.16.g26cd413590
#84 lore raw

PATCH v3 12/16 refs/packed-backend.c: ignore complicated hidden refs rules

In reply to Taylor Blau

In subsequent commits, we'll teach receive-pack and upload-pack to use the new skip-list feature in the packed-refs iterator by ignoring references which are mentioned via its respective hideRefs lists.

However, the packed-ref skip lists cannot handle un-hiding rules (that begin with '!'), or namespace comparisons (that begin with '^'). Detect and avoid these cases by falling back to the normal enumeration without a skip list when such patterns exist.

---
 refs/packed-backend.c   | 19 +++++++++++++++++++
 t/t1419-exclude-refs.sh |  9 +++++++++
 2 files changed, 28 insertions(+)
refs/packed-backend.c +19 -0
index 7ba9fa2bb8..9ea6c07866 100644
@@ -1015,6 +1015,25 @@ static void populate_excluded_jump_list(struct packed_ref_iterator *iter,
 	if (!excluded_patterns)
 		return;
 
+	for (pattern = excluded_patterns; *pattern; pattern++) {
+		/*
+		 * We also can't feed any excludes from hidden refs
+		 * config sections, since later rules may override
+		 * previous ones. For example, with rules "refs/foo" and
+		 * "!refs/foo/bar", we should show "refs/foo/bar" (and
+		 * everything underneath it), but the earlier exclusion
+		 * would cause us to skip all of "refs/foo". We likewise
+		 * don't implement the namespace stripping required for
+		 * '^' rules.
+		 *
+		 * Both are possible to do, but complicated, so avoid
+		 * populating the jump list at all if we see either of
+		 * these patterns.
+		 */
+		if (**pattern == '!' || **pattern == '^')
+			return;
+	}
+
 	for (pattern = excluded_patterns; *pattern; pattern++) {
 		struct jump_list_entry *e;
 
t/t1419-exclude-refs.sh +9 -0
index 350a7d2587..0e91e2f399 100755
@@ -119,4 +119,13 @@ test_expect_success 'meta-characters are discarded' '
 	assert_no_jumps
 '
 
+test_expect_success 'complex hidden ref rules are discarded' '
+	for_each_ref__exclude refs/heads refs/heads/foo "!refs/heads/foo/1" \
+		>actual 2>perf &&
+	for_each_ref >expect &&
+
+	test_cmp expect actual &&
+	assert_no_jumps
+'
+
 test_done
Signature · 1 line
2.41.0.16.g26cd413590
#86 lore raw

PATCH v3 13/16 refs.h: let `for_each_namespaced_ref()` take excluded patterns

In reply to Taylor Blau

The following commit will want to call for_each_namespaced_ref() with a list of excluded patterns.

We could introduce a variant of that function, say, for_each_namespaced_ref_exclude() which takes the extra parameter, and reimplement the original function in terms of that. But all but one caller (in http-backend.c) will supply the new parameter, so add the new parameter to for_each_namespaced_ref() itself instead of introducing a new function.

For now, supply NULL for the list of excluded patterns at all callers to avoid changing behavior, which we will do in the subsequent commit.

---
 http-backend.c | 2 +-
 refs.c         | 5 +++--
 refs.h         | 3 ++-
 upload-pack.c  | 6 +++---
 4 files changed, 9 insertions(+), 7 deletions(-)
http-backend.c +1 -1
index ac146d85c5..ad500683c8 100644
@@ -559,7 +559,7 @@ static void get_info_refs(struct strbuf *hdr, char *arg UNUSED)
 
 	} else {
 		select_getanyfile(hdr);
-		for_each_namespaced_ref(show_text_ref, &buf);
+		for_each_namespaced_ref(NULL, show_text_ref, &buf);
 		send_strbuf(hdr, "text/plain", &buf);
 	}
 	strbuf_release(&buf);
refs.c +3 -2
index ec4d5b9101..95a7db9563 100644
@@ -1660,13 +1660,14 @@ int for_each_replace_ref(struct repository *r, each_repo_ref_fn fn, void *cb_dat
 				    DO_FOR_EACH_INCLUDE_BROKEN, cb_data);
 }
 
-int for_each_namespaced_ref(each_ref_fn fn, void *cb_data)
+int for_each_namespaced_ref(const char **exclude_patterns,
+			    each_ref_fn fn, void *cb_data)
 {
 	struct strbuf buf = STRBUF_INIT;
 	int ret;
 	strbuf_addf(&buf, "%srefs/", get_git_namespace());
 	ret = do_for_each_ref(get_main_ref_store(the_repository),
-			      buf.buf, NULL, fn, 0, 0, cb_data);
+			      buf.buf, exclude_patterns, fn, 0, 0, cb_data);
 	strbuf_release(&buf);
 	return ret;
 }
refs.h +2 -1
index a7751a1fc9..f23626beca 100644
@@ -372,7 +372,8 @@ int for_each_glob_ref_in(each_ref_fn fn, const char *pattern,
 			 const char *prefix, void *cb_data);
 
 int head_ref_namespaced(each_ref_fn fn, void *cb_data);
-int for_each_namespaced_ref(each_ref_fn fn, void *cb_data);
+int for_each_namespaced_ref(const char **exclude_patterns,
+			    each_ref_fn fn, void *cb_data);
 
 /* can be used to learn about broken ref and symref */
 int refs_for_each_rawref(struct ref_store *refs, each_ref_fn fn, void *cb_data);
upload-pack.c +3 -3
index 1a213ed775..99d216938c 100644
@@ -855,7 +855,7 @@ static void deepen(struct upload_pack_data *data, int depth)
 		 * marked with OUR_REF.
 		 */
 		head_ref_namespaced(check_ref, data);
-		for_each_namespaced_ref(check_ref, data);
+		for_each_namespaced_ref(NULL, check_ref, data);
 
 		get_reachable_list(data, &reachable_shallows);
 		result = get_shallow_commits(&reachable_shallows,
@@ -1386,7 +1386,7 @@ void upload_pack(const int advertise_refs, const int stateless_rpc,
 		if (advertise_refs)
 			data.no_done = 1;
 		head_ref_namespaced(send_ref, &data);
-		for_each_namespaced_ref(send_ref, &data);
+		for_each_namespaced_ref(NULL, send_ref, &data);
 		if (!data.sent_capabilities) {
 			const char *refname = "capabilities^{}";
 			write_v0_ref(&data, refname, refname, null_oid());
@@ -1400,7 +1400,7 @@ void upload_pack(const int advertise_refs, const int stateless_rpc,
 		packet_flush(1);
 	} else {
 		head_ref_namespaced(check_ref, &data);
-		for_each_namespaced_ref(check_ref, &data);
+		for_each_namespaced_ref(NULL, check_ref, &data);
 	}
 
 	if (!advertise_refs) {
Signature · 1 line
2.41.0.16.g26cd413590
#87 lore raw

PATCH v3 14/16 builtin/receive-pack.c: avoid enumerating hidden references

In reply to Taylor Blau

Now that refs_for_each_fullref_in() has the ability to avoid enumerating references matching certain pattern(s), use that to avoid visiting hidden refs when constructing the ref advertisement via receive-pack.

Note that since this exclusion is best-effort, we still need show_ref_cb() to check whether or not each reference is hidden or not before including it in the advertisement.

As was the case when applying this same optimization to upload-pack, receive-pack's reference advertisement phase can proceed much quicker by avoiding enumerating references that will not be part of the advertisement.

(Below, we're still using linux.git with one hidden refs/pull/N ref per commit):

    $ hyperfine -L v ,.compile 'git{v} -c transfer.hideRefs=refs/pull receive-pack --advertise-refs .git'
    Benchmark 1: git -c transfer.hideRefs=refs/pull receive-pack --advertise-refs .git
      Time (mean ± σ):      89.1 ms ±   1.7 ms    [User: 82.0 ms, System: 7.0 ms]
      Range (min … max):    87.7 ms …  95.5 ms    31 runs
    Benchmark 2: git.compile -c transfer.hideRefs=refs/pull receive-pack --advertise-refs .git
      Time (mean ± σ):       4.5 ms ±   0.2 ms    [User: 0.5 ms, System: 3.9 ms]
      Range (min … max):     4.1 ms …   5.6 ms    508 runs

    Summary
      'git.compile -c transfer.hideRefs=refs/pull receive-pack --advertise-refs .git' ran
       20.00 ± 1.05 times faster than 'git -c transfer.hideRefs=refs/pull receive-pack --advertise-refs .git'
---
 builtin/receive-pack.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
builtin/receive-pack.c +2 -1
index 1a8472eddc..bd5bcc375f 100644
@@ -337,7 +337,8 @@ static void write_head_info(void)
 {
 	static struct oidset seen = OIDSET_INIT;
 
-	for_each_ref(show_ref_cb, &seen);
+	refs_for_each_fullref_in(get_main_ref_store(the_repository), "",
+				 hidden_refs.v, show_ref_cb, &seen);
 	for_each_alternate_ref(show_one_alternate_ref, &seen);
 	oidset_clear(&seen);
 	if (!sent_capabilities)
Signature · 1 line
2.41.0.16.g26cd413590
#88 lore raw

PATCH v3 15/16 upload-pack.c: avoid enumerating hidden refs where possible

In reply to Taylor Blau

In a similar fashion as a previous commit, teach upload-pack to avoid enumerating hidden references where possible.

Note, however, that there are certain cases where cannot avoid enumerating even hidden references, in particular when either of:

- uploadpack.allowTipSHA1InWant, or
- uploadpack.allowReachableSHA1InWant

are set, corresponding to ALLOW_TIP_SHA1 and ALLOW_REACHABLE_SHA1, respectively.

When either of these bits are set, upload-pack's is_our_ref() function needs to consider the HIDDEN_REF bit of the referent's object flags. So we must visit all references, including the hidden ones, in order to mark their referents with the HIDDEN_REF bit.

When neither ALLOW_TIP_SHA1 nor ALLOW_REACHABLE_SHA1 are set, the is_our_ref() function considers only the OUR_REF bit, and not the HIDDEN_REF one. OUR_REF is applied via mark_our_ref(), and only to objects at the tips of non-hidden references, so we do not need to visit hidden references in this case.

When neither of those bits are set, upload-pack can potentially avoid enumerating a large number of references. In the same example as a previous commit (linux.git with one hidden reference per commit, "refs/pull/N"):

    $ printf 0000 >in
    $ hyperfine --warmup=1 \
      'git -c transfer.hideRefs=refs/pull upload-pack . <in' \
      'git.compile -c transfer.hideRefs=refs/pull -c uploadpack.allowTipSHA1InWant upload-pack . <in' \
      'git.compile -c transfer.hideRefs=refs/pull upload-pack . <in'
    Benchmark 1: git -c transfer.hideRefs=refs/pull upload-pack . <in
      Time (mean ± σ):     406.9 ms ±   1.1 ms    [User: 357.3 ms, System: 49.5 ms]
      Range (min … max):   405.7 ms … 409.2 ms    10 runs

Benchmark 2: git.compile -c transfer.hideRefs=refs/pull -c uploadpack.allowTipSHA1InWant upload-pack . <in
Time (mean ± σ): 406.5 ms ± 1.3 ms [User: 356.5 ms, System: 49.9 ms]
Range (min … max): 404.6 ms … 408.8 ms 10 runs

Benchmark 3: git.compile -c transfer.hideRefs=refs/pull upload-pack . <in
Time (mean ± σ): 4.7 ms ± 0.2 ms [User: 0.7 ms, System: 3.9 ms]
Range (min … max): 4.3 ms … 6.1 ms 472 runs

Summary
'git.compile -c transfer.hideRefs=refs/pull upload-pack . <in' ran
86.62 ± 4.33 times faster than 'git.compile -c transfer.hideRefs=refs/pull -c uploadpack.allowTipSHA1InWant upload-pack . <in' 86.70 ± 4.33 times faster than 'git -c transfer.hideRefs=refs/pull upload-pack . <in'

As above, we must visit every reference when uploadPack.allowTipSHA1InWant is set. But when it is unset, we can visit far fewer references.

---
 upload-pack.c | 33 +++++++++++++++++++++++++++------
 1 file changed, 27 insertions(+), 6 deletions(-)
upload-pack.c +27 -6
index 99d216938c..366a101d8d 100644
@@ -602,11 +602,32 @@ static int get_common_commits(struct upload_pack_data *data,
 	}
 }
 
+static int allow_hidden_refs(enum allow_uor allow_uor)
+{
+	return allow_uor & (ALLOW_TIP_SHA1 | ALLOW_REACHABLE_SHA1);
+}
+
+static void for_each_namespaced_ref_1(each_ref_fn fn,
+				      struct upload_pack_data *data)
+{
+	/*
+	 * If `data->allow_uor` allows fetching hidden refs, we need to
+	 * mark all references (including hidden ones), to check in
+	 * `is_our_ref()` below.
+	 *
+	 * Otherwise, we only care about whether each reference's object
+	 * has the OUR_REF bit set or not, so do not need to visit
+	 * hidden references.
+	 */
+	if (allow_hidden_refs(data->allow_uor))
+		for_each_namespaced_ref(NULL, fn, data);
+	else
+		for_each_namespaced_ref(data->hidden_refs.v, fn, data);
+}
+
 static int is_our_ref(struct object *o, enum allow_uor allow_uor)
 {
-	int allow_hidden_ref = (allow_uor &
-				(ALLOW_TIP_SHA1 | ALLOW_REACHABLE_SHA1));
-	return o->flags & ((allow_hidden_ref ? HIDDEN_REF : 0) | OUR_REF);
+	return o->flags & ((allow_hidden_refs(allow_uor) ? HIDDEN_REF : 0) | OUR_REF);
 }
 
 /*
@@ -855,7 +876,7 @@ static void deepen(struct upload_pack_data *data, int depth)
 		 * marked with OUR_REF.
 		 */
 		head_ref_namespaced(check_ref, data);
-		for_each_namespaced_ref(NULL, check_ref, data);
+		for_each_namespaced_ref_1(check_ref, data);
 
 		get_reachable_list(data, &reachable_shallows);
 		result = get_shallow_commits(&reachable_shallows,
@@ -1386,7 +1407,7 @@ void upload_pack(const int advertise_refs, const int stateless_rpc,
 		if (advertise_refs)
 			data.no_done = 1;
 		head_ref_namespaced(send_ref, &data);
-		for_each_namespaced_ref(NULL, send_ref, &data);
+		for_each_namespaced_ref_1(send_ref, &data);
 		if (!data.sent_capabilities) {
 			const char *refname = "capabilities^{}";
 			write_v0_ref(&data, refname, refname, null_oid());
@@ -1400,7 +1421,7 @@ void upload_pack(const int advertise_refs, const int stateless_rpc,
 		packet_flush(1);
 	} else {
 		head_ref_namespaced(check_ref, &data);
-		for_each_namespaced_ref(NULL, check_ref, &data);
+		for_each_namespaced_ref_1(check_ref, &data);
 	}
 
 	if (!advertise_refs) {
Signature · 1 line
2.41.0.16.g26cd413590
#89 lore raw

PATCH v3 16/16 ls-refs.c: avoid enumerating hidden refs where possible

In reply to Taylor Blau

In a similar fashion as in previous commits, teach ls-refs to avoid enumerating hidden references where possible.

As before, this is linux.git with one hidden reference per commit.

    $ hyperfine -L v ,.compile 'git{v} -c protocol.version=2 ls-remote .'
    Benchmark 1: git -c protocol.version=2 ls-remote .
      Time (mean ± σ):      89.8 ms ±   0.6 ms    [User: 84.3 ms, System: 5.7 ms]
      Range (min … max):    88.8 ms …  91.3 ms    32 runs

Benchmark 2: git.compile -c protocol.version=2 ls-remote .
Time (mean ± σ): 6.5 ms ± 0.1 ms [User: 2.4 ms, System: 4.3 ms]
Range (min … max): 6.2 ms … 8.3 ms 397 runs

Summary
'git.compile -c protocol.version=2 ls-remote .' ran
13.85 ± 0.33 times faster than 'git -c protocol.version=2 ls-remote .'

---
 ls-refs.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
ls-refs.c +1 -1
index 8c3181d051..c9a723ba89 100644
@@ -193,7 +193,7 @@ int ls_refs(struct repository *r, struct packet_reader *request)
 		strvec_push(&data.prefixes, "");
 	refs_for_each_fullref_in_prefixes(get_main_ref_store(r),
 					  get_git_namespace(), data.prefixes.v,
-					  NULL, send_ref, &data);
+					  data.hidden_refs.v, send_ref, &data);
 	packet_fflush(stdout);
 	strvec_clear(&data.prefixes);
 	strbuf_release(&data.buf);
Signature · 1 line
2.41.0.16.g26cd413590
#90 lore raw

Re: [PATCH v3 00/16] refs: implement jump lists for packed backend

In reply to Taylor Blau
Quoted 2 lines · Taylor Blau

Here is a reroll of my series to implement jump (née skip) lists for the packed refs backend, based on top of the current 'master'.

Hmph. I kind of liked Patrick's suggestion to split this into two series to make it easier for the earlier half to graduate faster, but perhaps it didn't make much difference? I certainly did not get the impression that "review had stabilized". During my review of the initial round, for example, I lost steam in the middle because it was simply too long a series and didn't have a chance to give the remainder a proper review. I do not know about others.

#65 lore raw

Re: [PATCH v3 01/16] refs.c: rename `ref_filter`

In reply to Taylor Blau

refs.c int ref_exists(const char *refname) · line 416

Quoted 8 lines · Taylor Blau
refs.c +4 -3
-static int filter_refs(const char *refname, const struct object_id *oid,
-			   int flags, void *data)
+static int for_each_filter_refs(const char *refname,
+				const struct object_id *oid,
+				int flags, void *data)
 {
-	struct ref_filter *filter = (struct ref_filter *)data;
+	struct for_each_ref_filter *filter = data;

Nice to see that a trivial and obvious clean-up like this is silently included ;-)

#70 lore raw

Re: [PATCH v3 05/16] ref-filter.c: parameterize match functions over patterns

In reply to Taylor Blau
Quoted 4 lines · Taylor Blau

Once we start passing either in, match_pattern() will have little to do with a particular struct ref_filter * instance. To clarify this, drop it from the argument list, and replace it with the only bit of the ref_filter that we care about (filter->ignore_case).

Makes sense.

#73 lore raw

Re: [PATCH v3 07/16] refs: plumb `exclude_patterns` argument throughout

In reply to Taylor Blau

ls-refs.c int ls_refs(struct repository *r, struct packet_reader *request) · line 196

Quoted 10 lines · Taylor Blau
ls-refs.c +1 -1
index f385938b64..6f490b2d9c 100644
@@ -193,7 +193,7 @@ int ls_refs(struct repository *r, struct packet_reader *request)
 		strvec_push(&data.prefixes, "");
 	refs_for_each_fullref_in_prefixes(get_main_ref_store(r),
 					  get_git_namespace(), data.prefixes.v,
-					  send_ref, &data);
+					  NULL, send_ref, &data);

OK.

ref-filter.c static int for_each_fullref_in_pattern(struct ref_filter *filter, · line 2214

Quoted 12 lines
ref-filter.c +2 -1
index d44418efb7..717c3c4bcf 100644
@@ -2209,12 +2209,13 @@ static int for_each_fullref_in_pattern(struct ref_filter *filter,
 
 	if (!filter->name_patterns[0]) {
 		/* no patterns; we have to look at everything */
-		return for_each_fullref_in("", cb, cb_data);
+		return refs_for_each_fullref_in(get_main_ref_store(the_repository),
+						 "", NULL, cb, cb_data);
 	}

Is this merely "while at it", or was there a reason why refs_* variant must be used here? It is curious that we do not teach the exclude_patterns to some functions like for_each_fullref_in() while adding exclude_patterns to others, making the API surface uneven.

#77 lore raw

Re: [PATCH v3 09/16] refs/packed-backend.c: implement jump lists to avoid excluded pattern(s)

In reply to Taylor Blau
Quoted 16 lines · Taylor Blau

When iterating through the packed-refs file in order to answer a query like:

    $ git for-each-ref --exclude=refs/__hidden__

it would be useful to avoid walking over all of the entries in refs/__hidden__/* when possible, since we know that the ref-filter code is going to throw them away anyways.

In certain circumstances, doing so is possible. The algorithm for doing so is as follows:

- For each excluded pattern, find the first record that matches it, and the first record that *doesn't* match it (i.e. the location you'd next want to consider when excluding that pattern).

Quoted 6 lines

- Sort the set of excluded regions from the previous step in ascending order of the first location within the packed-refs file that matches.

- Clean up the results from the previous step: discard empty regions, and combine adjacent regions.

Say something like

The resulting list of regions that would never contain refs that are not excluded is called the "jump list".

here, probably. Otherwise the first reference to "jump list" we see later feels a bit too abrupt.

Quoted 10 lines

Then when iterating through the packed-refs file, if iter->pos is ever contained in one of the regions from the previous steps, advance iter->pos past the end of that region, and continue enumeration.

Note that we only perform this optimization when none of the excluded pattern(s) have special meta-characters in them. For a pattern like "refs/foo[ac]", the excluded regions ("refs/fooa", "refs/fooc", and everything underneath them) are not connected. A future implementation that handles this case may split the character class (pretending as if two patterns were excluded: "refs/fooa", and "refs/fooc").

Makes sense.

Quoted 8 lines

There are a few other gotchas worth considering. First, note that the jump list is sorted, so once we jump past a region, we can avoid considering it (or any regions preceding it) again. The member jump_pos is used to track the first next-possible region to jump through.

Second, note that the exclusion list is best-effort, since we do not handle loose references, and because of the meta-character issue above.

I found this a bit misleading; a natural reading of "exclusion list" is that the phrase refers to the list of exclude patterns given from the command line, and users would be upset if the processing of it is best effort.

I think what you meant to say was that optimization to avoid full scan using the jump list does not aim for perfection, and entries that are not skipped using the jump list may still be excluded by the exclude patterns.

Quoted 1 line

In repositories with a large number of hidden references, the speed-up

"hidden" -> "excluded". Your final objective to implement the feature to exclude refs using patterns and optimize it using the jump list data may be to implement "hidden references", but that hasn't be talked about in the above. All we have been hearing was that we are optimizing the walk over packed-refs using exclude patterns.

Quoted 6 lines

can be significant. Tests here are done with a copy of linux.git with a reference "refs/pull/N" pointing at every commit, as in:
...

Quoted 7 lines
---
 ref-filter.c              |   5 +-
 refs/packed-backend.c     | 166 ++++++++++++++++++++++++++++++++++++--
 t/helper/test-ref-store.c |  10 +++
 t/t1419-exclude-refs.sh   | 101 +++++++++++++++++++++++
 4 files changed, 274 insertions(+), 8 deletions(-)
 create mode 100755 t/t1419-exclude-refs.sh

Nice.

Quoted 31 lines
refs/packed-backend.c +16 -1
@@ -785,6 +802,13 @@ struct packed_ref_iterator {
 	/* The end of the part of the buffer that will be iterated over: */
 	const char *eof;
 
+	struct jump_list_entry {
+		const char *start;
+		const char *end;
+	} *jump;
+	size_t jump_nr, jump_alloc;
+	size_t jump_pos;
+
 	/* Scratch space for current values: */
 	struct object_id oid, peeled;
 	struct strbuf refname_buf;
@@ -802,14 +826,34 @@ struct packed_ref_iterator {
  */
 static int next_record(struct packed_ref_iterator *iter)
 {
-	const char *p = iter->pos, *eol;
+	const char *p, *eol;
 
 	strbuf_reset(&iter->refname_buf);
 
+	/*
+	 * If iter->pos is contained within a skipped region, jump past
+	 * it.
+	 *
+	 * Note that each skipped region is considered at most once,
+	 * since they are ordered based on their starting position.
+	 */
+	while (iter->jump_pos < iter->jump_nr) {

refs/packed-backend.c struct packed_ref_iterator { · line 850

Quoted 10 lines
refs/packed-backend.c +10 -0
+		struct jump_list_entry *curr = &iter->jump[iter->jump_pos];
+		if (iter->pos < curr->start)
+			break; /* not to the next jump yet */
+
+		iter->jump_pos++;
+		if (iter->pos < curr->end) {
+			iter->pos = curr->end;
+			break;
+		}
+	}

Quite straight-forward.

refs/packed-backend.c static struct ref_iterator_vtable packed_ref_iterator_vtable = { · line 1003

Quoted 6 lines
refs/packed-backend.c +6 -0
+static const char *ptr_max(const char *x, const char *y)
+{
+	if (x > y)
+		return x;
+	return y;
+}

Hopefully the compiler would inline the function without being told.

These pointers point into the same mmapped region of contiguous memory that holds the contents of the packed-refs file, so comparison between them is always defined. Good.

I wondered if

	return (x > y) ? x : y;

is easier to read, simply because it treats both cases more equally (in other words, as written, (x>y) appears more "special"), but that is minor.

Quoted 14 lines
refs/packed-backend.c +14 -0
+static void populate_excluded_jump_list(struct packed_ref_iterator *iter,
+					struct snapshot *snapshot,
+					const char **excluded_patterns)
+{
+	size_t i, j;
+	const char **pattern;
+	struct jump_list_entry *last_disjoint;
+
+	if (!excluded_patterns)
+		return;
+
+	for (pattern = excluded_patterns; *pattern; pattern++) {
+		struct jump_list_entry *e;
+

refs/packed-backend.c static struct ref_iterator_vtable packed_ref_iterator_vtable = { · line 1028

Quoted 10 lines
refs/packed-backend.c +10 -0
+		/*
+		 * We can't feed any excludes with globs in them to the
+		 * refs machinery.  It only understands prefix matching.
+		 * We likewise can't even feed the string leading up to
+		 * the first meta-character, as something like "foo[a]"
+		 * should not exclude "foobar" (but the prefix "foo"
+		 * would match that and mark it for exclusion).
+		 */
+		if (has_glob_special(*pattern))
+			continue;

OK. When we have a "literal" exclude pattern and another "glob" exclude pattern, we can afford to ignore the "glob" one when building the jump list and include only the "literal" one, because the jump list is used only to skip over entries that obviously can never be in the result, *and* --exclude are additive (i.e. being on jump list because of the "literal" pattern is a reason enough to be excluded from the result; matching or not matching the other patterns does not affect the fate of the ref that got excluded due to matching the "literal" pattern).

Makes sense.

Quoted 4 lines
refs/packed-backend.c +4 -0
+		ALLOC_GROW(iter->jump, iter->jump_nr + 1, iter->jump_alloc);
+
+		e = &iter->jump[iter->jump_nr++];
+		e->start = find_reference_location(snapshot, *pattern, 0);

refs/packed-backend.c static struct ref_iterator_vtable packed_ref_iterator_vtable = { · line 1043

Quoted 10 lines
refs/packed-backend.c +10 -0
+		e->end = find_reference_location_end(snapshot, *pattern, 0);
+	}
+
+	if (!iter->jump_nr) {
+		/*
+		 * Every entry in exclude_patterns has a meta-character,
+		 * nothing to do here.
+		 */
+		return;
+	}

OK.

Thanks.

#81 lore raw

Re: [PATCH v3 10/16] refs/packed-backend.c: add trace2 counters for jump list

In reply to Taylor Blau

t/t1419-exclude-refs.sh for_each_ref () { · line 26

Quoted 6 lines · Taylor Blau
t/t1419-exclude-refs.sh +6 -0
+assert_jumps () {
+	local nr="$1"
+	local trace="$2"
+
+	grep -q "name:jumps_made value:$nr" $trace
+}

You may expect 1 hit and the test will pass if trace had 10 or 12 hits.

#85 lore raw

Re: [PATCH v3 12/16] refs/packed-backend.c: ignore complicated hidden refs rules

In reply to Taylor Blau
Quoted 2 lines · Taylor Blau

In subsequent commits, we'll teach receive-pack and upload-pack to use the new skip-list feature in the packed-refs iterator by ignoring

"skip" -> "jump".

refs/packed-backend.c static void populate_excluded_jump_list(struct packed_ref_iterator *iter,

Quoted 13 lines

references which are mentioned via its respective hideRefs lists.

However, the packed-ref skip lists cannot handle un-hiding rules (that begin with '!'), or namespace comparisons (that begin with '^'). Detect and avoid these cases by falling back to the normal enumeration without a skip list when such patterns exist.

refs/packed-backend.c +6 -0
+		 * We also can't feed any excludes from hidden refs
+		 * config sections, since later rules may override
+		 * previous ones. For example, with rules "refs/foo" and
+		 * "!refs/foo/bar", we should show "refs/foo/bar" (and
+		 * everything underneath it), but the earlier exclusion
+		 * would cause us to skip all of "refs/foo".

Good observation. The presence of !refs/foo/bar in hide list forbids us from adding refs/foo to the jump list, and it is the simplest to disable the whole jump list business when we have such a feature in use.

Makes sense. Thanks.

#74 lore raw

Re: [PATCH v3 07/16] refs: plumb `exclude_patterns` argument throughout

In reply to Junio C Hamano
Quoted 6 lines · Junio C Hamano
Quoted 6 lines
ref-filter.c +0 -0
index d44418efb7..717c3c4bcf 100644
@@ -2209,12 +2209,13 @@ static int for_each_fullref_in_pattern(struct ref_filter *filter,
 
Quoted 9 lines

ref-filter.c static int for_each_fullref_in_pattern(struct ref_filter *filter, · line 2214

Quoted 6 lines
ref-filter.c +2 -1
 	if (!filter->name_patterns[0]) {
 		/* no patterns; we have to look at everything */
-		return for_each_fullref_in("", cb, cb_data);
+		return refs_for_each_fullref_in(get_main_ref_store(the_repository),
+						 "", NULL, cb, cb_data);
 	}

Is this merely "while at it", or was there a reason why refs_* variant must be used here?

Good question. This changes later in the series (via "refs/packed-backend.c: implement jump lists to avoid excluded pattern(s)") to pass the excluded patterns from the ref_filter.

There's no need to change it in this patch, since the functionality at this point is equivalent in the pre- and post-image. I think this staging is a consequence of having written much of this series before committing anything, and then trying to segment it out into meaningful patches after the fact.

Quoted 3 lines

It is curious that we do not teach the exclude_patterns to some functions like for_each_fullref_in() while adding exclude_patterns to others, making the API surface uneven.

We could plumb it through in more places, but my preference would be to modify the API if/as new callers need to pass a list of excluded patterns. The API has an enormous amount of surface area (and many functions which take a ton of arguments), so I'd rather keep it small as long as possible, even at the expense of some unevenness in its interface.

Thanks,
Taylor

#78 lore raw

Re: [PATCH v3 09/16] refs/packed-backend.c: implement jump lists to avoid excluded pattern(s)

In reply to Junio C Hamano
Quoted 10 lines · Junio C Hamano
Quoted 2 lines

- Clean up the results from the previous step: discard empty regions, and combine adjacent regions.

Say something like

The resulting list of regions that would never contain refs that are not excluded is called the "jump list".

here, probably. Otherwise the first reference to "jump list" we see later feels a bit too abrupt.

Good suggestion. I phrased it slightly differently in the version that I'll send shortly, but the spirit is the same.

Quoted 14 lines
Quoted 8 lines

There are a few other gotchas worth considering. First, note that the jump list is sorted, so once we jump past a region, we can avoid considering it (or any regions preceding it) again. The member jump_pos is used to track the first next-possible region to jump through.

Second, note that the exclusion list is best-effort, since we do not handle loose references, and because of the meta-character issue above.

I found this a bit misleading; a natural reading of "exclusion list" is that the phrase refers to the list of exclude patterns given from the command line, and users would be upset if the processing of it is best effort.

Quoted 4 lines

I think what you meant to say was that optimization to avoid full scan using the jump list does not aim for perfection, and entries that are not skipped using the jump list may still be excluded by the exclude patterns.

Yep, exactly. I think this was from an earlier version from before this was called "jump lists", and I missed it when find-and-replacing through the patch messages. Thanks for spotting.

Quoted 8 lines
Quoted 1 line

In repositories with a large number of hidden references, the speed-up

"hidden" -> "excluded". Your final objective to implement the feature to exclude refs using patterns and optimize it using the jump list data may be to implement "hidden references", but that hasn't be talked about in the above. All we have been hearing was that we are optimizing the walk over packed-refs using exclude patterns.

Yep, thanks.

Quoted 13 lines

refs/packed-backend.c static struct ref_iterator_vtable packed_ref_iterator_vtable = { · line 1003

Quoted 6 lines
refs/packed-backend.c +6 -0
+static const char *ptr_max(const char *x, const char *y)
+{
+	if (x > y)
+		return x;
+	return y;
+}

Hopefully the compiler would inline the function without being told.

These pointers point into the same mmapped region of contiguous memory that holds the contents of the packed-refs file, so comparison between them is always defined. Good.

Quoted 7 lines

I wondered if

	return (x > y) ? x : y;

is easier to read, simply because it treats both cases more equally (in other words, as written, (x>y) appears more "special"), but that is minor.

Yeah, I think that any reasonable compiler would almost certainly inline this, especially at higher optimization levels. But I agree with your suggestion nonetheless, thanks.

Quoted 21 lines

refs/packed-backend.c static struct ref_iterator_vtable packed_ref_iterator_vtable = { · line 1028

Quoted 10 lines
refs/packed-backend.c +10 -0
+		/*
+		 * We can't feed any excludes with globs in them to the
+		 * refs machinery.  It only understands prefix matching.
+		 * We likewise can't even feed the string leading up to
+		 * the first meta-character, as something like "foo[a]"
+		 * should not exclude "foobar" (but the prefix "foo"
+		 * would match that and mark it for exclusion).
+		 */
+		if (has_glob_special(*pattern))
+			continue;

OK. When we have a "literal" exclude pattern and another "glob" exclude pattern, we can afford to ignore the "glob" one when building the jump list and include only the "literal" one, because the jump list is used only to skip over entries that obviously can never be in the result, *and* --exclude are additive (i.e. being on jump list because of the "literal" pattern is a reason enough to be excluded from the result; matching or not matching the other patterns does not affect the fate of the ref that got excluded due to matching the "literal" pattern).

Quoted 1 line

Makes sense.

Exactly.

Thanks,
Taylor

#82 lore raw

Re: [PATCH v3 10/16] refs/packed-backend.c: add trace2 counters for jump list

In reply to Junio C Hamano
Quoted 10 lines · Junio C Hamano

t/t1419-exclude-refs.sh for_each_ref () { · line 26

Quoted 6 lines · Taylor Blau
t/t1419-exclude-refs.sh +6 -0
+assert_jumps () {
+	local nr="$1"
+	local trace="$2"
+
+	grep -q "name:jumps_made value:$nr" $trace
+}

You may expect 1 hit and the test will pass if trace had 10 or 12 hits.

Oops. Well spotted, thanks.

Thanks,
Taylor

#49 lore raw

Re: [PATCH v2 09/16] refs/packed-backend.c: implement jump lists to avoid excluded pattern(s)

In reply to Patrick Steinhardt
Quoted 4 lines · Patrick Steinhardt
Quoted 4 lines
refs/packed-backend.c +0 -0
@@ -785,6 +802,13 @@ struct packed_ref_iterator {
 	/* The end of the part of the buffer that will be iterated over: */
 	const char *eof;
 
Quoted 11 lines

refs/packed-backend.c struct packed_ref_iterator {

Quoted 7 lines
refs/packed-backend.c +6 -0
+	struct jump_list_entry {
+		const char *start;
+		const char *end;
+	} *jump;
+	size_t jump_nr, jump_alloc;
+	size_t jump_pos;
 

Nit: I had some trouble with jump_pos given that it sounds so similar to iter->pos, and thus you tend to think that they both apply to the position in the packed-refs file. jump_curr or jump_idx might help to avoid this confusion.

Very fair, thanks for observing. I went with "jump_cur" (as a shorthand for "cursor").

Quoted 15 lines
Quoted 15 lines
refs/packed-backend.c +15 -0
+	for (pattern = excluded_patterns; *pattern; pattern++) {
+		struct jump_list_entry *e;
+
+		/*
+		 * We can't feed any excludes with globs in them to the
+		 * refs machinery.  It only understands prefix matching.
+		 * We likewise can't even feed the string leading up to
+		 * the first meta-character, as something like "foo[a]"
+		 * should not exclude "foobar" (but the prefix "foo"
+		 * would match that and mark it for exclusion).
+		 */
+		if (has_glob_special(*pattern))
+			continue;
+
+		ALLOC_GROW(iter->jump, iter->jump_nr + 1, iter->jump_alloc);
Quoted 8 lines

refs/packed-backend.c static struct ref_iterator_vtable packed_ref_iterator_vtable = { · line 1034

Quoted 4 lines
refs/packed-backend.c +4 -0
+
+		e = &iter->jump[iter->jump_nr++];
+		e->start = find_reference_location(snapshot, *pattern, 0);
+		e->end = find_reference_location_end(snapshot, *pattern, 0);

Nit: we could detect the non-matching case here already, which would allow us to skip an allocation. It's probably pre-mature optimization though, so please feel free to ignore.

Probably so, this allocation is so lightweight in comparison to all of the other things that for-each-ref does throughout its execution that I think it's probably negligible to shave off a few allocations.

Quoted 21 lines
Quoted 21 lines
refs/packed-backend.c +21 -0
+	}
+
+	if (!iter->jump_nr) {
+		/*
+		 * Every entry in exclude_patterns has a meta-character,
+		 * nothing to do here.
+		 */
+		return;
+	}
+
+	QSORT(iter->jump, iter->jump_nr, jump_list_entry_cmp);
+
+	/*
+	 * As an optimization, merge adjacent entries in the jump list
+	 * to jump forwards as far as possible when entering a skipped
+	 * region.
+	 *
+	 * For example, if we have two skipped regions:
+	 *
+	 *	[[A, B], [B, C]]
+	 *
Quoted 8 lines

refs/packed-backend.c static struct ref_iterator_vtable packed_ref_iterator_vtable = { · line 1059

Quoted 4 lines
refs/packed-backend.c +4 -0
+	 * we want to combine that into a single entry jumping from A to
+	 * C.
+	 */
+	last_disjoint = iter->jump;

Nit: if we initialized j = 0, then last_disjoint would always be equal to iter->jump[j]. We could then declare the variable inside of the loop to make it a bit easier to understand.

Sure, though we would then need to assign iter->jump_nr = j + 1, which I think adds more confusion than inlining the variable is worth.

Thanks,
Taylor

#53 lore raw

Re: [PATCH v2 11/16] revision.h: store hidden refs in a `strvec`

In reply to Patrick Steinhardt
Quoted 19 lines · Patrick Steinhardt
Quoted 18 lines · Taylor Blau

In subsequent commits, it will be convenient to have a 'const char **' of hidden refs (matching transfer.hiderefs, uploadpack.hideRefs, etc.), instead of a string_list.

Convert spots throughout the tree that store the list of hidden refs from a string_list to a strvec.

Note that in parse_hide_refs_config() there is an ugly const-cast used to avoid an extra copy of each value before trimming any trailing slash characters. This could instead be written as:

    ref = xstrdup(value);
    len = strlen(ref);
    while (len && ref[len - 1] == '/')
            ref[--len] = '\0';
    strvec_push(hide_refs, ref);
    free(ref);
Quoted 7 lines
Quoted 2 lines

but the double-copy (once when calling xstrdup(), and another via strvec_push()) is wasteful.

I guess the proper way to fix this would be to introduce something like a strvec_push_nodup() function that takes ownership. And in fact this helper exists already, but it's declared as static. So we could get around the ugly cast with a simple change to expose the helper function.

We could, but I'd prefer to explore doing so outside of this series, since I think strvec_push_nodup() is a little bit of a footgun unless you are thinking carefully about ownership. So making the function part of the exposed API may be controversial.

Thanks,
Taylor

#57 lore raw

Re: [PATCH v2 13/16] refs.h: let `for_each_namespaced_ref()` take excluded patterns

In reply to Patrick Steinhardt
Quoted 6 lines · Patrick Steinhardt
Quoted 2 lines · Taylor Blau

The following commit will want to call for_each_namespaced_ref() with a list of excluded patterns.

This statement isn't quite true as the following commit touches git-receive-pack(1), which doesn't use for_each_namespaced_ref_().

Oops, well spotted. This was true when I wrote the series, but of course I decided to swap the order of patches 14 and 15 with one another, making this statement false. Good eyes ;-).

Thanks,
Taylor

#62 lore raw

Re: [PATCH v2 00/16] refs: implement jump lists for packed backend

In reply to Patrick Steinhardt
Quoted 20 lines · Patrick Steinhardt
Quoted 19 lines · Taylor Blau

Here is a reroll of my series to implement jump (née skip) lists for the packed refs backend.

Not a ton has changed since last time, but some notable things that have changed include:

- Renaming "skip lists" to "jump lists" to clarify that this implementation does not use the skip list data structure.
- Patch reorganization, splitting out find_reference_location_end() more sensibly, rewording patch messages, etc.
- Addresses feedback from Junio and Patrick Steinhardt's helpful reviews.

As usual, a range-diff is included below for convenience.

Given that we are expecting -rc0 today, we should aim to not let review of this topic direct our attention away from testing the release candidates. We can get more serious about it on the other side of 2.41.

Quoted 6 lines
Quoted 1 line

Thanks in advance for another look.

I didn't have many comments in this round. Personally though I'd split up this patch series into two in order to land the individual parts faster, where the first part introduces git for-each-ref --exclude and the second part introduces the jump list for the packed-refs backend.

Thanks for reviewing, and for the good suggestion. I think that splitting this series in two could be worthwhile, but I'm not sure I want to make this change for v4.

You could imagine splitting the series there, where the first half would implement the naive version of for-each-ref --exclude, and the second half would implement jump lists, and make upload- and receive-pack take advantage of it.

But I think that makes the first half trivial and leaves all of the complexity in this series to the latter half. I suppose that makes it easier to graduate the first six or so patches earlier, but they aren't really all that useful without the remaining patches.

Another split might be:

- the naive implementation of for-each-ref --exclude
- jump lists, and using them within for-each-ref
- teaching upload- and receive-pack to use the jump list

But juggling this series as three sub-topics feels like it would be burdensome to the maintainer ;-). So I think I'd rather leave it as-is, unless you feel strongly.

Thanks,
Taylor

#91 lore raw

v4 cover refs: implement jump lists for packed backend

In reply to Taylor Blau

Here is another reroll of my series to implement jump (née skip) lists for the packed refs backend, based on top of the current 'master'.

Most changes are minor, limited to changing variable names, moving changes around between patches and tweaking commit messages for clarity. I think that the first 9 or so patches are stable, but we may want some more eyes on the remainder.

As usual, a range-diff is included below for convenience.

Thanks in advance for your review.

Jeff King (5):
refs.c: rename ref_filter
ref-filter.h: provide REF_FILTER_INIT
ref-filter: clear reachable list pointers after freeing
ref-filter: add ref_filter_clear() ref-filter.c: parameterize match functions over patterns

Taylor Blau (11):
builtin/for-each-ref.c: add --exclude option
refs: plumb exclude_patterns argument throughout refs/packed-backend.c: refactor find_reference_location() refs/packed-backend.c: implement jump lists to avoid excluded
pattern(s)
refs/packed-backend.c: add trace2 counters for jump list revision.h: store hidden refs in a strvec refs/packed-backend.c: ignore complicated hidden refs rules refs.h: let for_each_namespaced_ref() take excluded patterns builtin/receive-pack.c: avoid enumerating hidden references upload-pack.c: avoid enumerating hidden refs where possible ls-refs.c: avoid enumerating hidden refs where possible

 Documentation/git-for-each-ref.txt |   6 +
 builtin/branch.c                   |   4 +-
 builtin/for-each-ref.c             |   7 +-
 builtin/receive-pack.c             |   7 +-
 builtin/tag.c                      |   4 +-
 http-backend.c                     |   2 +-
 ls-refs.c                          |   8 +-
 ref-filter.c                       |  63 +++++++--
 ref-filter.h                       |  12 ++
 refs.c                             |  61 ++++----
 refs.h                             |  15 +-
 refs/debug.c                       |   5 +-
 refs/files-backend.c               |   5 +-
 refs/packed-backend.c              | 220 ++++++++++++++++++++++++++---
 refs/refs-internal.h               |   7 +-
 revision.c                         |   4 +-
 revision.h                         |   5 +-
 t/helper/test-reach.c              |   2 +-
 t/helper/test-ref-store.c          |  10 ++
 t/t0041-usage.sh                   |   1 +
 t/t1419-exclude-refs.sh            | 131 +++++++++++++++++
 t/t3402-rebase-merge.sh            |   1 +
 t/t6300-for-each-ref.sh            |  35 +++++
 trace2.h                           |   2 +
 trace2/tr2_ctr.c                   |   5 +
 upload-pack.c                      |  43 ++++--
 26 files changed, 559 insertions(+), 106 deletions(-)
 create mode 100755 t/t1419-exclude-refs.sh

Range-diff against v3:

 1:  afac948f04 =  1:  c12def5a30 refs.c: rename `ref_filter`
 2:  b9336e3b77 =  2:  7ce82b6a5a ref-filter.h: provide `REF_FILTER_INIT`
 3:  fc28b5caaa =  3:  7e6bf7766d ref-filter: clear reachable list pointers after freeing
 4:  bc5356fe4b =  4:  777e71004d ref-filter: add `ref_filter_clear()`
 5:  1988ca4c0a =  5:  39e9b0f50d ref-filter.c: parameterize match functions over patterns
 6:  60d85aa4ad =  6:  c4fd47fd75 builtin/for-each-ref.c: add `--exclude` option
 7:  c4fe9a1893 !  7:  e6b50c5021 refs: plumb `exclude_patterns` argument throughout

@@ ls-refs.c: int ls_refs(struct repository *r, struct packet_reader *request)
## ref-filter.c ##
@@ ref-filter.c: static int for_each_fullref_in_pattern(struct ref_filter *filter,

- if (!filter->name_patterns[0]) {
- /* no patterns; we have to look at everything */
-- return for_each_fullref_in("", cb, cb_data);
-+ return refs_for_each_fullref_in(get_main_ref_store(the_repository),
-+ "", NULL, cb, cb_data);
- }
-
return refs_for_each_fullref_in_prefixes(get_main_ref_store(the_repository),
NULL, filter->name_patterns,
- cb, cb_data);

 8:  9cab5e0699 =  8:  a0990b2916 refs/packed-backend.c: refactor `find_reference_location()`
 9:  8066858bf5 !  9:  386ed468fa refs/packed-backend.c: implement jump lists to avoid excluded pattern(s)
    @@ Commit message
             matches.
     
           - Clean up the results from the previous step: discard empty regions,
    -        and combine adjacent regions.
    +        and combine adjacent regions. The set of regions which remains is
    +        referred to as the "jump list", and never contains any references
    +        which should be included in the result set.
     
         Then when iterating through the `packed-refs` file, if `iter->pos` is
         ever contained in one of the regions from the previous steps, advance
    @@ Commit message
         `jump_pos` is used to track the first next-possible region to jump
         through.
     
    -    Second, note that the exclusion list is best-effort, since we do not
    -    handle loose references, and because of the meta-character issue above.
    +    Second, note that the jump list is best-effort, since we do not handle
    +    loose references, and because of the meta-character issue above. The
    +    jump list may not skip past all references which won't appear in the
    +    results, but will never skip over a reference which does appear in the
    +    result set.
     
         In repositories with a large number of hidden references, the speed-up
         can be significant. Tests here are done with a copy of linux.git with a
    @@ Commit message
     
      ## ref-filter.c ##
     @@ ref-filter.c: static int for_each_fullref_in_pattern(struct ref_filter *filter,
    + 
      	if (!filter->name_patterns[0]) {
      		/* no patterns; we have to look at everything */
    - 		return refs_for_each_fullref_in(get_main_ref_store(the_repository),
    --						 "", NULL, cb, cb_data);
    +-		return for_each_fullref_in("", cb, cb_data);
    ++		return refs_for_each_fullref_in(get_main_ref_store(the_repository),
     +						 "", filter->exclude.v, cb, cb_data);
      	}
      
    @@ refs/packed-backend.c: struct packed_ref_iterator {
     +		const char *end;
     +	} *jump;
     +	size_t jump_nr, jump_alloc;
    -+	size_t jump_pos;
    ++	size_t jump_cur;
     +
      	/* Scratch space for current values: */
      	struct object_id oid, peeled;
    @@ refs/packed-backend.c: struct packed_ref_iterator {
     +	 * Note that each skipped region is considered at most once,
     +	 * since they are ordered based on their starting position.
     +	 */
    -+	while (iter->jump_pos < iter->jump_nr) {
    -+		struct jump_list_entry *curr = &iter->jump[iter->jump_pos];
    ++	while (iter->jump_cur < iter->jump_nr) {
    ++		struct jump_list_entry *curr = &iter->jump[iter->jump_cur];
     +		if (iter->pos < curr->start)
     +			break; /* not to the next jump yet */
     +
    -+		iter->jump_pos++;
    ++		iter->jump_cur++;
     +		if (iter->pos < curr->end) {
     +			iter->pos = curr->end;
     +			break;
    @@ refs/packed-backend.c: static struct ref_iterator_vtable packed_ref_iterator_vta
     +	return 0;
     +}
     +
    -+static const char *ptr_max(const char *x, const char *y)
    -+{
    -+	if (x > y)
    -+		return x;
    -+	return y;
    -+}
    -+
     +static void populate_excluded_jump_list(struct packed_ref_iterator *iter,
     +					struct snapshot *snapshot,
     +					const char **excluded_patterns)
    @@ refs/packed-backend.c: static struct ref_iterator_vtable packed_ref_iterator_vta
     +			continue;
     +		} else if (ours->start <= last_disjoint->end) {
     +			/* overlapping regions extend the previous one */
    -+			last_disjoint->end = ptr_max(last_disjoint->end, ours->end);
    ++			last_disjoint->end = last_disjoint->end > ours->end
    ++				? last_disjoint->end : ours->end;
     +		} else {
     +			/* otherwise, insert a new region */
     +			iter->jump[j++] = *ours;
    @@ refs/packed-backend.c: static struct ref_iterator_vtable packed_ref_iterator_vta
     +	}
     +
     +	iter->jump_nr = j;
    -+	iter->jump_pos = 0;
    ++	iter->jump_cur = 0;
     +}
     +
      static struct ref_iterator *packed_ref_iterator_begin(
10:  3c045076a9 ! 10:  49c8f5173a refs/packed-backend.c: add trace2 counters for jump list
    @@ refs/packed-backend.c
      enum mmap_strategy {
      	/*
     @@ refs/packed-backend.c: static int next_record(struct packed_ref_iterator *iter)
    - 		iter->jump_pos++;
    + 		iter->jump_cur++;
      		if (iter->pos < curr->end) {
      			iter->pos = curr->end;
     +			trace2_counter_add(TRACE2_COUNTER_ID_PACKED_REFS_JUMPS, 1);
    @@ t/t1419-exclude-refs.sh: for_each_ref () {
     +	local nr="$1"
     +	local trace="$2"
     +
    -+	grep -q "name:jumps_made value:$nr" $trace
    ++	grep -q "name:jumps_made value:$nr$" $trace
     +}
     +
     +assert_no_jumps () {
    @@ t/t1419-exclude-refs.sh: test_expect_success 'setup' '
      
     -	test_cmp expect actual
     +	test_cmp expect actual &&
    -+	assert_no_jumps
    ++	assert_no_jumps perf
      '
      
      test_expect_success 'meta-characters are discarded' '
    @@ t/t1419-exclude-refs.sh: test_expect_success 'setup' '
      
     -	test_cmp expect actual
     +	test_cmp expect actual &&
    -+	assert_no_jumps
    ++	assert_no_jumps perf
      '
      
      test_done
11:  0ff542eaef = 11:  dd856a3982 revision.h: store hidden refs in a `strvec`
12:  ca006b2c3f ! 12:  845904853e refs/packed-backend.c: ignore complicated hidden refs rules

@@ Commit message
refs/packed-backend.c: ignore complicated hidden refs rules

In subsequent commits, we'll teach receive-pack and upload-pack to
- use the new skip-list feature in the packed-refs iterator by ignoring
+ use the new jump list feature in the packed-refs iterator by ignoring
references which are mentioned via its respective hideRefs lists.

- However, the packed-ref skip lists cannot handle un-hiding rules (that
+ However, the packed-ref jump lists cannot handle un-hiding rules (that
begin with '!'), or namespace comparisons (that begin with '^'). Detect and avoid these cases by falling back to the normal enumeration without
- a skip list when such patterns exist.
+ a jump list when such patterns exist.

@@ refs/packed-backend.c: static void populate_excluded_jump_list(struct packed_ref

## t/t1419-exclude-refs.sh ##
@@ t/t1419-exclude-refs.sh: test_expect_success 'meta-characters are discarded' '
- assert_no_jumps
+ assert_no_jumps perf
'

+test_expect_success 'complex hidden ref rules are discarded' '
13: cae703a425 ! 13: 8d4d7cc22e refs.h: let for_each_namespaced_ref() take excluded patterns
@@ Metadata
## Commit message ##
refs.h: let for_each_namespaced_ref() take excluded patterns

- The following commit will want to call for_each_namespaced_ref() with
+ A future commit will want to call for_each_namespaced_ref() with
a list of excluded patterns.

We could introduce a variant of that function, say,
@@ Commit message
introducing a new function.

For now, supply NULL for the list of excluded patterns at all callers to
- avoid changing behavior, which we will do in the subsequent commit.
+ avoid changing behavior, which we will do in a future change.

14:  1db10b76ea = 14:  49c665f9f8 builtin/receive-pack.c: avoid enumerating hidden references
15:  014243ebe6 = 15:  19bf4a52d6 upload-pack.c: avoid enumerating hidden refs where possible
16:  e02fe93379 = 16:  ea6cbaf292 ls-refs.c: avoid enumerating hidden refs where possible
Signature · 1 line
2.41.0.44.gf2359540d2
#92 lore raw

PATCH v4 1/16 refs.c: rename `ref_filter`

From: Jeff King <[email protected]>

The refs machinery has its own implementation of a ref_filter (used by for-each-ref), which is distinct from the ref-filler.h API (also used by for-each-ref, among other things).

Rename the one within refs.c to more clearly indicate its purpose.

---
 refs.c | 15 ++++++++-------
 1 file changed, 8 insertions(+), 7 deletions(-)
refs.c +8 -7
index 881a0da65c..ba63b69090 100644
@@ -375,8 +375,8 @@ char *resolve_refdup(const char *refname, int resolve_flags,
 				   oid, flags);
 }
 
-/* The argument to filter_refs */
-struct ref_filter {
+/* The argument to for_each_filter_refs */
+struct for_each_ref_filter {
 	const char *pattern;
 	const char *prefix;
 	each_ref_fn *fn;
@@ -409,10 +409,11 @@ int ref_exists(const char *refname)
 	return refs_ref_exists(get_main_ref_store(the_repository), refname);
 }
 
-static int filter_refs(const char *refname, const struct object_id *oid,
-			   int flags, void *data)
+static int for_each_filter_refs(const char *refname,
+				const struct object_id *oid,
+				int flags, void *data)
 {
-	struct ref_filter *filter = (struct ref_filter *)data;
+	struct for_each_ref_filter *filter = data;
 
 	if (wildmatch(filter->pattern, refname, 0))
 		return 0;
@@ -569,7 +570,7 @@ int for_each_glob_ref_in(each_ref_fn fn, const char *pattern,
 	const char *prefix, void *cb_data)
 {
 	struct strbuf real_pattern = STRBUF_INIT;
-	struct ref_filter filter;
+	struct for_each_ref_filter filter;
 	int ret;
 
 	if (!prefix && !starts_with(pattern, "refs/"))
@@ -589,7 +590,7 @@ int for_each_glob_ref_in(each_ref_fn fn, const char *pattern,
 	filter.prefix = prefix;
 	filter.fn = fn;
 	filter.cb_data = cb_data;
-	ret = for_each_ref(filter_refs, &filter);
+	ret = for_each_ref(for_each_filter_refs, &filter);
 
 	strbuf_release(&real_pattern);
 	return ret;
Signature · 1 line
2.41.0.44.gf2359540d2
#94 lore raw

PATCH v4 2/16 ref-filter.h: provide `REF_FILTER_INIT`

In reply to Taylor Blau

From: Jeff King <[email protected]>

Provide a sane initialization value for struct ref_filter, which in a subsequent patch will be used to initialize a new field.

In the meantime, fix a case in test-reach.c where its ref_filter is not even zero-initialized.

---
 builtin/branch.c       | 3 +--
 builtin/for-each-ref.c | 3 +--
 builtin/tag.c          | 3 +--
 ref-filter.h           | 3 +++
 t/helper/test-reach.c  | 2 +-
 5 files changed, 7 insertions(+), 7 deletions(-)
builtin/branch.c +1 -2
index e6c2655af6..7891dec361 100644
@@ -707,7 +707,7 @@ int cmd_branch(int argc, const char **argv, const char *prefix)
 	int reflog = 0, quiet = 0, icase = 0, force = 0,
 	    recurse_submodules_explicit = 0;
 	enum branch_track track;
-	struct ref_filter filter;
+	struct ref_filter filter = REF_FILTER_INIT;
 	static struct ref_sorting *sorting;
 	struct string_list sorting_options = STRING_LIST_INIT_DUP;
 	struct ref_format format = REF_FORMAT_INIT;
@@ -765,7 +765,6 @@ int cmd_branch(int argc, const char **argv, const char *prefix)
 
 	setup_ref_filter_porcelain_msg();
 
-	memset(&filter, 0, sizeof(filter));
 	filter.kind = FILTER_REFS_BRANCHES;
 	filter.abbrev = -1;
 
builtin/for-each-ref.c +1 -2
index 695fc8f4a5..99ccb73518 100644
@@ -24,7 +24,7 @@ int cmd_for_each_ref(int argc, const char **argv, const char *prefix)
 	struct string_list sorting_options = STRING_LIST_INIT_DUP;
 	int maxcount = 0, icase = 0, omit_empty = 0;
 	struct ref_array array;
-	struct ref_filter filter;
+	struct ref_filter filter = REF_FILTER_INIT;
 	struct ref_format format = REF_FORMAT_INIT;
 	struct strbuf output = STRBUF_INIT;
 	struct strbuf err = STRBUF_INIT;
@@ -61,7 +61,6 @@ int cmd_for_each_ref(int argc, const char **argv, const char *prefix)
 	};
 
 	memset(&array, 0, sizeof(array));
-	memset(&filter, 0, sizeof(filter));
 
 	format.format = "%(objectname) %(objecttype)\t%(refname)";
 
builtin/tag.c +1 -2
index 49b64c7a28..ec778ba860 100644
@@ -437,7 +437,7 @@ int cmd_tag(int argc, const char **argv, const char *prefix)
 	struct msg_arg msg = { .buf = STRBUF_INIT };
 	struct ref_transaction *transaction;
 	struct strbuf err = STRBUF_INIT;
-	struct ref_filter filter;
+	struct ref_filter filter = REF_FILTER_INIT;
 	struct ref_sorting *sorting;
 	struct string_list sorting_options = STRING_LIST_INIT_DUP;
 	struct ref_format format = REF_FORMAT_INIT;
@@ -496,7 +496,6 @@ int cmd_tag(int argc, const char **argv, const char *prefix)
 	git_config(git_tag_config, &sorting_options);
 
 	memset(&opt, 0, sizeof(opt));
-	memset(&filter, 0, sizeof(filter));
 	filter.lines = -1;
 	opt.sign = -1;
 
ref-filter.h +3 -0
index 430701cfb7..a920f73b29 100644
@@ -92,6 +92,9 @@ struct ref_format {
 	struct string_list bases;
 };
 
+#define REF_FILTER_INIT { \
+	.points_at = OID_ARRAY_INIT, \
+}
 #define REF_FORMAT_INIT {             \
 	.use_color = -1,              \
 	.bases = STRING_LIST_INIT_DUP, \
t/helper/test-reach.c +1 -1
index 5b6f217441..ef58f10c2d 100644
@@ -139,7 +139,7 @@ int cmd__reach(int ac, const char **av)
 
 		printf("%s(X,_,_,0,0):%d\n", av[1], can_all_from_reach_with_flag(&X_obj, 2, 4, 0, 0));
 	} else if (!strcmp(av[1], "commit_contains")) {
-		struct ref_filter filter;
+		struct ref_filter filter = REF_FILTER_INIT;
 		struct contains_cache cache;
 		init_contains_cache(&cache);
 
Signature · 1 line
2.41.0.44.gf2359540d2
#97 lore raw

PATCH v4 3/16 ref-filter: clear reachable list pointers after freeing

In reply to Taylor Blau

From: Jeff King <[email protected]>

In reach_filter(), we pop all commits from the reachable lists, leaving them empty. But because we're operating on a list pointer that was passed by value, the original filter.reachable_from pointer is left dangling.

---
 ref-filter.c | 14 +++++++-------
 1 file changed, 7 insertions(+), 7 deletions(-)
ref-filter.c +7 -7
index 4991cd4f7a..048d277cbf 100644
@@ -2418,13 +2418,13 @@ void ref_array_clear(struct ref_array *array)
 #define EXCLUDE_REACHED 0
 #define INCLUDE_REACHED 1
 static void reach_filter(struct ref_array *array,
-			 struct commit_list *check_reachable,
+			 struct commit_list **check_reachable,
 			 int include_reached)
 {
 	int i, old_nr;
 	struct commit **to_clear;
 
-	if (!check_reachable)
+	if (!*check_reachable)
 		return;
 
 	CALLOC_ARRAY(to_clear, array->nr);
@@ -2434,7 +2434,7 @@ static void reach_filter(struct ref_array *array,
 	}
 
 	tips_reachable_from_bases(the_repository,
-				  check_reachable,
+				  *check_reachable,
 				  to_clear, array->nr,
 				  UNINTERESTING);
 
@@ -2455,8 +2455,8 @@ static void reach_filter(struct ref_array *array,
 
 	clear_commit_marks_many(old_nr, to_clear, ALL_REV_FLAGS);
 
-	while (check_reachable) {
-		struct commit *merge_commit = pop_commit(&check_reachable);
+	while (*check_reachable) {
+		struct commit *merge_commit = pop_commit(check_reachable);
 		clear_commit_marks(merge_commit, ALL_REV_FLAGS);
 	}
 
@@ -2553,8 +2553,8 @@ int filter_refs(struct ref_array *array, struct ref_filter *filter, unsigned int
 	clear_contains_cache(&ref_cbdata.no_contains_cache);
 
 	/*  Filters that need revision walking */
-	reach_filter(array, filter->reachable_from, INCLUDE_REACHED);
-	reach_filter(array, filter->unreachable_from, EXCLUDE_REACHED);
+	reach_filter(array, &filter->reachable_from, INCLUDE_REACHED);
+	reach_filter(array, &filter->unreachable_from, EXCLUDE_REACHED);
 
 	save_commit_buffer = save_commit_buffer_orig;
 	return ret;
Signature · 1 line
2.41.0.44.gf2359540d2
#99 lore raw

PATCH v4 4/16 ref-filter: add `ref_filter_clear()`

In reply to Taylor Blau

From: Jeff King <[email protected]>

We did not bother to clean up at all in git branch or git tag, and git for-each-ref only cleans up a couple of members.

Add and call ref_filter_clear() when cleaning up a struct ref_filter. Running this patch (without any test changes) indicates a couple of now leak-free tests. This was found by running:

    $ make SANITIZE=leak
    $ make -C t GIT_TEST_PASSING_SANITIZE_LEAK=check GIT_TEST_OPTS=--immediate

(Note that the reachable_from and unreachable_from lists should be cleaned as they are used. So this is just covering any case where we might bail before running the reachability check.)

---
 builtin/branch.c        |  1 +
 builtin/for-each-ref.c  |  3 +--
 builtin/tag.c           |  1 +
 ref-filter.c            | 16 ++++++++++++++++
 ref-filter.h            |  3 +++
 t/t0041-usage.sh        |  1 +
 t/t3402-rebase-merge.sh |  1 +
 7 files changed, 24 insertions(+), 2 deletions(-)
builtin/branch.c +1 -0
index 7891dec361..07ee874617 100644
@@ -858,6 +858,7 @@ int cmd_branch(int argc, const char **argv, const char *prefix)
 		print_columns(&output, colopts, NULL);
 		string_list_clear(&output, 0);
 		ref_sorting_release(sorting);
+		ref_filter_clear(&filter);
 		return 0;
 	} else if (edit_description) {
 		const char *branch_name;
builtin/for-each-ref.c +1 -2
index 99ccb73518..c01fa6fefe 100644
@@ -120,8 +120,7 @@ int cmd_for_each_ref(int argc, const char **argv, const char *prefix)
 	strbuf_release(&err);
 	strbuf_release(&output);
 	ref_array_clear(&array);
-	free_commit_list(filter.with_commit);
-	free_commit_list(filter.no_commit);
+	ref_filter_clear(&filter);
 	ref_sorting_release(sorting);
 	strvec_clear(&vec);
 	return 0;
builtin/tag.c +1 -0
index ec778ba860..2895ff0e45 100644
@@ -651,6 +651,7 @@ int cmd_tag(int argc, const char **argv, const char *prefix)
 
 cleanup:
 	ref_sorting_release(sorting);
+	ref_filter_clear(&filter);
 	strbuf_release(&buf);
 	strbuf_release(&ref);
 	strbuf_release(&reflog_msg);
ref-filter.c +16 -0
index 048d277cbf..d32f426898 100644
@@ -2866,3 +2866,19 @@ int parse_opt_merge_filter(const struct option *opt, const char *arg, int unset)
 
 	return 0;
 }
+
+void ref_filter_init(struct ref_filter *filter)
+{
+	struct ref_filter blank = REF_FILTER_INIT;
+	memcpy(filter, &blank, sizeof(blank));
+}
+
+void ref_filter_clear(struct ref_filter *filter)
+{
+	oid_array_clear(&filter->points_at);
+	free_commit_list(filter->with_commit);
+	free_commit_list(filter->no_commit);
+	free_commit_list(filter->reachable_from);
+	free_commit_list(filter->unreachable_from);
+	ref_filter_init(filter);
+}
ref-filter.h +3 -0
index a920f73b29..160b807224 100644
@@ -170,4 +170,7 @@ void filter_ahead_behind(struct repository *r,
 			 struct ref_format *format,
 			 struct ref_array *array);
 
+void ref_filter_init(struct ref_filter *filter);
+void ref_filter_clear(struct ref_filter *filter);
+
 #endif /*  REF_FILTER_H  */
t/t0041-usage.sh +1 -0
index c4fc34eb18..9ea974b0c6 100755
@@ -5,6 +5,7 @@ test_description='Test commands behavior when given invalid argument value'
 GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main
 export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME
 
+TEST_PASSES_SANITIZE_LEAK=true
 . ./test-lib.sh
 
 test_expect_success 'setup ' '
t/t3402-rebase-merge.sh +1 -0
index 79b0640c00..e9e03ca4b5 100755
@@ -8,6 +8,7 @@ test_description='git rebase --merge test'
 GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main
 export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME
 
+TEST_PASSES_SANITIZE_LEAK=true
 . ./test-lib.sh
 
 T="A quick brown fox
Signature · 1 line
2.41.0.44.gf2359540d2
#103 lore raw

PATCH v4 5/16 ref-filter.c: parameterize match functions over patterns

In reply to Taylor Blau

From: Jeff King <[email protected]>

match_pattern() and match_name_as_path() both take a struct ref_filter *, and then store a stack variable patterns pointing at filter->patterns.

The subsequent patch will add a new array of patterns to match over (the excluded patterns, via a new git for-each-ref --exclude option), treating the return value of these functions differently depending on which patterns are being used to match.

Tweak match_pattern() and match_name_as_path() to take an array of patterns to prepare for passing either in.

Once we start passing either in, match_pattern() will have little to do with a particular struct ref_filter * instance. To clarify this, drop it from the argument list, and replace it with the only bit of the ref_filter that we care about (filter->ignore_case).

---
 ref-filter.c | 16 +++++++++-------
 1 file changed, 9 insertions(+), 7 deletions(-)
ref-filter.c +9 -7
index d32f426898..6d91c7cb0d 100644
@@ -2104,12 +2104,12 @@ static int get_ref_atom_value(struct ref_array_item *ref, int atom,
  * matches a pattern "refs/heads/mas") or a wildcard (e.g. the same ref
  * matches "refs/heads/mas*", too).
  */
-static int match_pattern(const struct ref_filter *filter, const char *refname)
+static int match_pattern(const char **patterns, const char *refname,
+			 const int ignore_case)
 {
-	const char **patterns = filter->name_patterns;
 	unsigned flags = 0;
 
-	if (filter->ignore_case)
+	if (ignore_case)
 		flags |= WM_CASEFOLD;
 
 	/*
@@ -2134,9 +2134,10 @@ static int match_pattern(const struct ref_filter *filter, const char *refname)
  * matches a pattern "refs/heads/" but not "refs/heads/m") or a
  * wildcard (e.g. the same ref matches "refs/heads/m*", too).
  */
-static int match_name_as_path(const struct ref_filter *filter, const char *refname)
+static int match_name_as_path(const struct ref_filter *filter,
+			      const char **pattern,
+			      const char *refname)
 {
-	const char **pattern = filter->name_patterns;
 	int namelen = strlen(refname);
 	unsigned flags = WM_PATHNAME;
 
@@ -2165,8 +2166,9 @@ static int filter_pattern_match(struct ref_filter *filter, const char *refname)
 	if (!*filter->name_patterns)
 		return 1; /* No pattern always matches */
 	if (filter->match_as_path)
-		return match_name_as_path(filter, refname);
-	return match_pattern(filter, refname);
+		return match_name_as_path(filter, filter->name_patterns, refname);
+	return match_pattern(filter->name_patterns, refname,
+			     filter->ignore_case);
 }
 
 /*
Signature · 1 line
2.41.0.44.gf2359540d2
#108 lore raw

PATCH v4 6/16 builtin/for-each-ref.c: add `--exclude` option

In reply to Taylor Blau

When using for-each-ref, it is sometimes convenient for the caller to be able to exclude certain parts of the references.

For example, if there are many refs/__hidden__/* references, the caller may want to emit all references *except* the hidden ones. Currently, the only way to do this is to post-process the output, like:

    $ git for-each-ref --format='%(refname)' | grep -v '^refs/hidden/'

Which is do-able, but requires processing a potentially large quantity of references.

Teach git for-each-ref a new --exclude=<pattern> option, which excludes references from the results if they match one or more excluded patterns.

This patch provides a naive implementation where the ref_filter still sees all references (including ones that it will discard) and is left to check whether each reference matches any excluded pattern(s) before emitting them.

By culling out references we know the caller doesn't care about, we can avoid allocating memory for their storage, as well as spending time sorting the output (among other things). Even the naive implementation provides a significant speed-up on a modified copy of linux.git (that has a hidden ref pointing at each commit):

    $ hyperfine \
      'git.compile for-each-ref --format="%(objectname) %(refname)" | grep -vE "[0-9a-f]{40} refs/pull/"' \
      'git.compile for-each-ref --format="%(objectname) %(refname)" --exclude refs/pull/'
    Benchmark 1: git.compile for-each-ref --format="%(objectname) %(refname)" | grep -vE "[0-9a-f]{40} refs/pull/"
      Time (mean ± σ):     820.1 ms ±   2.0 ms    [User: 703.7 ms, System: 152.0 ms]
      Range (min … max):   817.7 ms … 823.3 ms    10 runs
    Benchmark 2: git.compile for-each-ref --format="%(objectname) %(refname)" --exclude refs/pull/
      Time (mean ± σ):     106.6 ms ±   1.1 ms    [User: 99.4 ms, System: 7.1 ms]
      Range (min … max):   104.7 ms … 109.1 ms    27 runs

    Summary
      'git.compile for-each-ref --format="%(objectname) %(refname)" --exclude refs/pull/' ran
        7.69 ± 0.08 times faster than 'git.compile for-each-ref --format="%(objectname) %(refname)" | grep -vE "[0-9a-f]{40} refs/pull/"'

Subsequent patches will improve on this by avoiding visiting excluded sections of the packed-refs file in certain cases.

---
 Documentation/git-for-each-ref.txt |  6 +++++
 builtin/for-each-ref.c             |  1 +
 ref-filter.c                       | 13 +++++++++++
 ref-filter.h                       |  6 +++++
 t/t6300-for-each-ref.sh            | 35 ++++++++++++++++++++++++++++++
 5 files changed, 61 insertions(+)
Documentation/git-for-each-ref.txt +6 -0
index 1e215d4e73..5743eb5def 100644
@@ -14,6 +14,7 @@ SYNOPSIS
 		   [--points-at=<object>]
 		   [--merged[=<object>]] [--no-merged[=<object>]]
 		   [--contains[=<object>]] [--no-contains[=<object>]]
+		   [--exclude=<pattern> ...]
 
 DESCRIPTION
 -----------
@@ -102,6 +103,11 @@ OPTIONS
 	Do not print a newline after formatted refs where the format expands
 	to the empty string.
 
+--exclude=<pattern>::
+	If one or more patterns are given, only refs which do not match
+	any excluded pattern(s) are shown. Matching is done using the
+	same rules as `<pattern>` above.
+
 FIELD NAMES
 -----------
 
builtin/for-each-ref.c +1 -0
index c01fa6fefe..3384987428 100644
@@ -47,6 +47,7 @@ int cmd_for_each_ref(int argc, const char **argv, const char *prefix)
 		OPT_INTEGER( 0 , "count", &maxcount, N_("show only <n> matched refs")),
 		OPT_STRING(  0 , "format", &format.format, N_("format"), N_("format to use for the output")),
 		OPT__COLOR(&format.use_color, N_("respect format colors")),
+		OPT_REF_FILTER_EXCLUDE(&filter),
 		OPT_REF_SORT(&sorting_options),
 		OPT_CALLBACK(0, "points-at", &filter.points_at,
 			     N_("object"), N_("print only refs which points at the given object"),
ref-filter.c +13 -0
index 6d91c7cb0d..d44418efb7 100644
@@ -2171,6 +2171,15 @@ static int filter_pattern_match(struct ref_filter *filter, const char *refname)
 			     filter->ignore_case);
 }
 
+static int filter_exclude_match(struct ref_filter *filter, const char *refname)
+{
+	if (!filter->exclude.nr)
+		return 0;
+	if (filter->match_as_path)
+		return match_name_as_path(filter, filter->exclude.v, refname);
+	return match_pattern(filter->exclude.v, refname, filter->ignore_case);
+}
+
 /*
  * This is the same as for_each_fullref_in(), but it tries to iterate
  * only over the patterns we'll care about. Note that it _doesn't_ do a full
@@ -2338,6 +2347,9 @@ static int ref_filter_handler(const char *refname, const struct object_id *oid,
 	if (!filter_pattern_match(filter, refname))
 		return 0;
 
+	if (filter_exclude_match(filter, refname))
+		return 0;
+
 	if (filter->points_at.nr && !match_points_at(&filter->points_at, oid, refname))
 		return 0;
 
@@ -2877,6 +2889,7 @@ void ref_filter_init(struct ref_filter *filter)
 
 void ref_filter_clear(struct ref_filter *filter)
 {
+	strvec_clear(&filter->exclude);
 	oid_array_clear(&filter->points_at);
 	free_commit_list(filter->with_commit);
 	free_commit_list(filter->no_commit);
ref-filter.h +6 -0
index 160b807224..1524bc463a 100644
@@ -6,6 +6,7 @@
 #include "refs.h"
 #include "commit.h"
 #include "string-list.h"
+#include "strvec.h"
 
 /* Quoting styles */
 #define QUOTE_NONE 0
@@ -59,6 +60,7 @@ struct ref_array {
 
 struct ref_filter {
 	const char **name_patterns;
+	struct strvec exclude;
 	struct oid_array points_at;
 	struct commit_list *with_commit;
 	struct commit_list *no_commit;
@@ -94,6 +96,7 @@ struct ref_format {
 
 #define REF_FILTER_INIT { \
 	.points_at = OID_ARRAY_INIT, \
+	.exclude = STRVEC_INIT, \
 }
 #define REF_FORMAT_INIT {             \
 	.use_color = -1,              \
@@ -112,6 +115,9 @@ struct ref_format {
 #define OPT_REF_SORT(var) \
 	OPT_STRING_LIST(0, "sort", (var), \
 			N_("key"), N_("field name to sort on"))
+#define OPT_REF_FILTER_EXCLUDE(var) \
+	OPT_STRVEC(0, "exclude", &(var)->exclude, \
+		   N_("pattern"), N_("exclude refs which match pattern"))
 
 /*
  * API for filtering a set of refs. Based on the type of refs the user
t/t6300-for-each-ref.sh +35 -0
index 5c00607608..7e8d578522 100755
@@ -447,6 +447,41 @@ test_expect_success 'exercise glob patterns with prefixes' '
 	test_cmp expected actual
 '
 
+cat >expected <<\EOF
+refs/tags/bar
+refs/tags/baz
+refs/tags/testtag
+EOF
+
+test_expect_success 'exercise patterns with prefix exclusions' '
+	for tag in foo/one foo/two foo/three bar baz
+	do
+		git tag "$tag" || return 1
+	done &&
+	test_when_finished "git tag -d foo/one foo/two foo/three bar baz" &&
+	git for-each-ref --format="%(refname)" \
+		refs/tags/ --exclude=refs/tags/foo >actual &&
+	test_cmp expected actual
+'
+
+cat >expected <<\EOF
+refs/tags/bar
+refs/tags/baz
+refs/tags/foo/one
+refs/tags/testtag
+EOF
+
+test_expect_success 'exercise patterns with pattern exclusions' '
+	for tag in foo/one foo/two foo/three bar baz
+	do
+		git tag "$tag" || return 1
+	done &&
+	test_when_finished "git tag -d foo/one foo/two foo/three bar baz" &&
+	git for-each-ref --format="%(refname)" \
+		refs/tags/ --exclude="refs/tags/foo/t*" >actual &&
+	test_cmp expected actual
+'
+
 cat >expected <<\EOF
 'refs/heads/main'
 'refs/remotes/origin/main'
Signature · 1 line
2.41.0.44.gf2359540d2
#109 lore raw

PATCH v4 7/16 refs: plumb `exclude_patterns` argument throughout

In reply to Taylor Blau

The subsequent patch will want to access an optional excluded_patterns array within refs/packed-backend.c that will cull out certain references matching any of the given patterns on a best-effort basis.

To do so, the refs subsystem needs to be updated to pass this value across a number of different locations.

Prepare for a future patch by introducing this plumbing now, passing NULLs at top-level APIs in order to make that patch less noisy and more easily readable.

---
 ls-refs.c             |  2 +-
 ref-filter.c          |  2 +-
 refs.c                | 32 +++++++++++++++++++-------------
 refs.h                |  8 +++++++-
 refs/debug.c          |  5 +++--
 refs/files-backend.c  |  5 +++--
 refs/packed-backend.c |  5 +++--
 refs/refs-internal.h  |  7 ++++---
 revision.c            |  2 +-
 9 files changed, 42 insertions(+), 26 deletions(-)
ls-refs.c +1 -1
index f385938b64..6f490b2d9c 100644
@@ -193,7 +193,7 @@ int ls_refs(struct repository *r, struct packet_reader *request)
 		strvec_push(&data.prefixes, "");
 	refs_for_each_fullref_in_prefixes(get_main_ref_store(r),
 					  get_git_namespace(), data.prefixes.v,
-					  send_ref, &data);
+					  NULL, send_ref, &data);
 	packet_fflush(stdout);
 	strvec_clear(&data.prefixes);
 	strbuf_release(&data.buf);
ref-filter.c +1 -1
index d44418efb7..5e7ed204dc 100644
@@ -2214,7 +2214,7 @@ static int for_each_fullref_in_pattern(struct ref_filter *filter,
 
 	return refs_for_each_fullref_in_prefixes(get_main_ref_store(the_repository),
 						 NULL, filter->name_patterns,
-						 cb, cb_data);
+						 NULL, cb, cb_data);
 }
 
 /*
refs.c +19 -13
index ba63b69090..b4b7165fc0 100644
@@ -1526,7 +1526,9 @@ int head_ref(each_ref_fn fn, void *cb_data)
 
 struct ref_iterator *refs_ref_iterator_begin(
 		struct ref_store *refs,
-		const char *prefix, int trim,
+		const char *prefix,
+		const char **exclude_patterns,
+		int trim,
 		enum do_for_each_ref_flags flags)
 {
 	struct ref_iterator *iter;
@@ -1542,8 +1544,7 @@ struct ref_iterator *refs_ref_iterator_begin(
 		}
 	}
 
-	iter = refs->be->iterator_begin(refs, prefix, flags);
-
+	iter = refs->be->iterator_begin(refs, prefix, exclude_patterns, flags);
 	/*
 	 * `iterator_begin()` already takes care of prefix, but we
 	 * might need to do some trimming:
@@ -1577,7 +1578,7 @@ static int do_for_each_repo_ref(struct repository *r, const char *prefix,
 	if (!refs)
 		return 0;
 
-	iter = refs_ref_iterator_begin(refs, prefix, trim, flags);
+	iter = refs_ref_iterator_begin(refs, prefix, NULL, trim, flags);
 
 	return do_for_each_repo_ref_iterator(r, iter, fn, cb_data);
 }
@@ -1599,6 +1600,7 @@ static int do_for_each_ref_helper(struct repository *r,
 }
 
 static int do_for_each_ref(struct ref_store *refs, const char *prefix,
+			   const char **exclude_patterns,
 			   each_ref_fn fn, int trim,
 			   enum do_for_each_ref_flags flags, void *cb_data)
 {
@@ -1608,7 +1610,8 @@ static int do_for_each_ref(struct ref_store *refs, const char *prefix,
 	if (!refs)
 		return 0;
 
-	iter = refs_ref_iterator_begin(refs, prefix, trim, flags);
+	iter = refs_ref_iterator_begin(refs, prefix, exclude_patterns, trim,
+				       flags);
 
 	return do_for_each_repo_ref_iterator(the_repository, iter,
 					do_for_each_ref_helper, &hp);
@@ -1616,7 +1619,7 @@ static int do_for_each_ref(struct ref_store *refs, const char *prefix,
 
 int refs_for_each_ref(struct ref_store *refs, each_ref_fn fn, void *cb_data)
 {
-	return do_for_each_ref(refs, "", fn, 0, 0, cb_data);
+	return do_for_each_ref(refs, "", NULL, fn, 0, 0, cb_data);
 }
 
 int for_each_ref(each_ref_fn fn, void *cb_data)
@@ -1627,7 +1630,7 @@ int for_each_ref(each_ref_fn fn, void *cb_data)
 int refs_for_each_ref_in(struct ref_store *refs, const char *prefix,
 			 each_ref_fn fn, void *cb_data)
 {
-	return do_for_each_ref(refs, prefix, fn, strlen(prefix), 0, cb_data);
+	return do_for_each_ref(refs, prefix, NULL, fn, strlen(prefix), 0, cb_data);
 }
 
 int for_each_ref_in(const char *prefix, each_ref_fn fn, void *cb_data)
@@ -1638,13 +1641,14 @@ int for_each_ref_in(const char *prefix, each_ref_fn fn, void *cb_data)
 int for_each_fullref_in(const char *prefix, each_ref_fn fn, void *cb_data)
 {
 	return do_for_each_ref(get_main_ref_store(the_repository),
-			       prefix, fn, 0, 0, cb_data);
+			       prefix, NULL, fn, 0, 0, cb_data);
 }
 
 int refs_for_each_fullref_in(struct ref_store *refs, const char *prefix,
+			     const char **exclude_patterns,
 			     each_ref_fn fn, void *cb_data)
 {
-	return do_for_each_ref(refs, prefix, fn, 0, 0, cb_data);
+	return do_for_each_ref(refs, prefix, exclude_patterns, fn, 0, 0, cb_data);
 }
 
 int for_each_replace_ref(struct repository *r, each_repo_ref_fn fn, void *cb_data)
@@ -1661,14 +1665,14 @@ int for_each_namespaced_ref(each_ref_fn fn, void *cb_data)
 	int ret;
 	strbuf_addf(&buf, "%srefs/", get_git_namespace());
 	ret = do_for_each_ref(get_main_ref_store(the_repository),
-			      buf.buf, fn, 0, 0, cb_data);
+			      buf.buf, NULL, fn, 0, 0, cb_data);
 	strbuf_release(&buf);
 	return ret;
 }
 
 int refs_for_each_rawref(struct ref_store *refs, each_ref_fn fn, void *cb_data)
 {
-	return do_for_each_ref(refs, "", fn, 0,
+	return do_for_each_ref(refs, "", NULL, fn, 0,
 			       DO_FOR_EACH_INCLUDE_BROKEN, cb_data);
 }
 
@@ -1738,6 +1742,7 @@ static void find_longest_prefixes(struct string_list *out,
 int refs_for_each_fullref_in_prefixes(struct ref_store *ref_store,
 				      const char *namespace,
 				      const char **patterns,
+				      const char **exclude_patterns,
 				      each_ref_fn fn, void *cb_data)
 {
 	struct string_list prefixes = STRING_LIST_INIT_DUP;
@@ -1753,7 +1758,8 @@ int refs_for_each_fullref_in_prefixes(struct ref_store *ref_store,
 
 	for_each_string_list_item(prefix, &prefixes) {
 		strbuf_addstr(&buf, prefix->string);
-		ret = refs_for_each_fullref_in(ref_store, buf.buf, fn, cb_data);
+		ret = refs_for_each_fullref_in(ref_store, buf.buf,
+					       exclude_patterns, fn, cb_data);
 		if (ret)
 			break;
 		strbuf_setlen(&buf, namespace_len);
@@ -2408,7 +2414,7 @@ int refs_verify_refname_available(struct ref_store *refs,
 	strbuf_addstr(&dirname, refname + dirname.len);
 	strbuf_addch(&dirname, '/');
 
-	iter = refs_ref_iterator_begin(refs, dirname.buf, 0,
+	iter = refs_ref_iterator_begin(refs, dirname.buf, NULL, 0,
 				       DO_FOR_EACH_INCLUDE_BROKEN);
 	while ((ok = ref_iterator_advance(iter)) == ITER_OK) {
 		if (skip &&
refs.h +7 -1
index 933fdebe58..3c03b035a3 100644
@@ -344,6 +344,7 @@ int for_each_ref(each_ref_fn fn, void *cb_data);
 int for_each_ref_in(const char *prefix, each_ref_fn fn, void *cb_data);
 
 int refs_for_each_fullref_in(struct ref_store *refs, const char *prefix,
+			     const char **exclude_patterns,
 			     each_ref_fn fn, void *cb_data);
 int for_each_fullref_in(const char *prefix, each_ref_fn fn, void *cb_data);
 
@@ -351,10 +352,15 @@ int for_each_fullref_in(const char *prefix, each_ref_fn fn, void *cb_data);
  * iterate all refs in "patterns" by partitioning patterns into disjoint sets
  * and iterating the longest-common prefix of each set.
  *
+ * references matching any pattern in "exclude_patterns" are omitted from the
+ * result set on a best-effort basis.
+ *
  * callers should be prepared to ignore references that they did not ask for.
  */
 int refs_for_each_fullref_in_prefixes(struct ref_store *refs,
-				      const char *namespace, const char **patterns,
+				      const char *namespace,
+				      const char **patterns,
+				      const char **exclude_patterns,
 				      each_ref_fn fn, void *cb_data);
 
 /**
refs/debug.c +3 -2
index c0fa707a1d..b7ffc4ce67 100644
@@ -229,11 +229,12 @@ static struct ref_iterator_vtable debug_ref_iterator_vtable = {
 
 static struct ref_iterator *
 debug_ref_iterator_begin(struct ref_store *ref_store, const char *prefix,
-			 unsigned int flags)
+			 const char **exclude_patterns, unsigned int flags)
 {
 	struct debug_ref_store *drefs = (struct debug_ref_store *)ref_store;
 	struct ref_iterator *res =
-		drefs->refs->be->iterator_begin(drefs->refs, prefix, flags);
+		drefs->refs->be->iterator_begin(drefs->refs, prefix,
+						exclude_patterns, flags);
 	struct debug_ref_iterator *diter = xcalloc(1, sizeof(*diter));
 	base_ref_iterator_init(&diter->base, &debug_ref_iterator_vtable, 1);
 	diter->iter = res;
refs/files-backend.c +3 -2
index 9a8333c0d0..0a60037530 100644
@@ -831,7 +831,8 @@ static struct ref_iterator_vtable files_ref_iterator_vtable = {
 
 static struct ref_iterator *files_ref_iterator_begin(
 		struct ref_store *ref_store,
-		const char *prefix, unsigned int flags)
+		const char *prefix, const char **exclude_patterns,
+		unsigned int flags)
 {
 	struct files_ref_store *refs;
 	struct ref_iterator *loose_iter, *packed_iter, *overlay_iter;
@@ -876,7 +877,7 @@ static struct ref_iterator *files_ref_iterator_begin(
 	 * the packed and loose references.
 	 */
 	packed_iter = refs_ref_iterator_begin(
-			refs->packed_ref_store, prefix, 0,
+			refs->packed_ref_store, prefix, exclude_patterns, 0,
 			DO_FOR_EACH_INCLUDE_BROKEN);
 
 	overlay_iter = overlay_ref_iterator_begin(loose_iter, packed_iter);
refs/packed-backend.c +3 -2
index 291e53f5cf..6855c9a237 100644
@@ -924,7 +924,8 @@ static struct ref_iterator_vtable packed_ref_iterator_vtable = {
 
 static struct ref_iterator *packed_ref_iterator_begin(
 		struct ref_store *ref_store,
-		const char *prefix, unsigned int flags)
+		const char *prefix, const char **exclude_patterns,
+		unsigned int flags)
 {
 	struct packed_ref_store *refs;
 	struct snapshot *snapshot;
@@ -1149,7 +1150,7 @@ static int write_with_updates(struct packed_ref_store *refs,
 	 * list of refs is exhausted, set iter to NULL. When the list
 	 * of updates is exhausted, leave i set to updates->nr.
 	 */
-	iter = packed_ref_iterator_begin(&refs->base, "",
+	iter = packed_ref_iterator_begin(&refs->base, "", NULL,
 					 DO_FOR_EACH_INCLUDE_BROKEN);
 	if ((ok = ref_iterator_advance(iter)) != ITER_OK)
 		iter = NULL;
refs/refs-internal.h +4 -3
index f72b7be894..9db8aec4da 100644
@@ -367,8 +367,8 @@ int is_empty_ref_iterator(struct ref_iterator *ref_iterator);
  */
 struct ref_iterator *refs_ref_iterator_begin(
 		struct ref_store *refs,
-		const char *prefix, int trim,
-		enum do_for_each_ref_flags flags);
+		const char *prefix, const char **exclude_patterns,
+		int trim, enum do_for_each_ref_flags flags);
 
 /*
  * A callback function used to instruct merge_ref_iterator how to
@@ -571,7 +571,8 @@ typedef int copy_ref_fn(struct ref_store *ref_store,
  */
 typedef struct ref_iterator *ref_iterator_begin_fn(
 		struct ref_store *ref_store,
-		const char *prefix, unsigned int flags);
+		const char *prefix, const char **exclude_patterns,
+		unsigned int flags);
 
 /* reflog functions */
 
revision.c +1 -1
index b33cc1d106..89953592f9 100644
@@ -2670,7 +2670,7 @@ static int for_each_bisect_ref(struct ref_store *refs, each_ref_fn fn,
 	struct strbuf bisect_refs = STRBUF_INIT;
 	int status;
 	strbuf_addf(&bisect_refs, "refs/bisect/%s", term);
-	status = refs_for_each_fullref_in(refs, bisect_refs.buf, fn, cb_data);
+	status = refs_for_each_fullref_in(refs, bisect_refs.buf, NULL, fn, cb_data);
 	strbuf_release(&bisect_refs);
 	return status;
 }
Signature · 1 line
2.41.0.44.gf2359540d2
#110 lore raw

PATCH v4 8/16 refs/packed-backend.c: refactor `find_reference_location()`

In reply to Taylor Blau

The function find_reference_location() is used to perform a binary search-like function over the contents of a repository's $GIT_DIR/packed-refs file.

The search it implements is unlike a standard binary search in that the records it searches over are not of a fixed width, so the comparison must locate the end of a record before comparing it.

Extract the core routine of find_reference_location() in order to implement a function in the following patch which will find the first location in the packed-refs file that *doesn't* match the given pattern.

The behavior of find_reference_location() is unchanged.

---
 refs/packed-backend.c | 38 ++++++++++++++++++++++----------------
 1 file changed, 22 insertions(+), 16 deletions(-)
refs/packed-backend.c +22 -16
index 6855c9a237..d9b61d9e03 100644
@@ -527,22 +527,8 @@ static int load_contents(struct snapshot *snapshot)
 	return 1;
 }
 
-/*
- * Find the place in `snapshot->buf` where the start of the record for
- * `refname` starts. If `mustexist` is true and the reference doesn't
- * exist, then return NULL. If `mustexist` is false and the reference
- * doesn't exist, then return the point where that reference would be
- * inserted, or `snapshot->eof` (which might be NULL) if it would be
- * inserted at the end of the file. In the latter mode, `refname`
- * doesn't have to be a proper reference name; for example, one could
- * search for "refs/replace/" to find the start of any replace
- * references.
- *
- * The record is sought using a binary search, so `snapshot->buf` must
- * be sorted.
- */
-static const char *find_reference_location(struct snapshot *snapshot,
-					   const char *refname, int mustexist)
+static const char *find_reference_location_1(struct snapshot *snapshot,
+					     const char *refname, int mustexist)
 {
 	/*
 	 * This is not *quite* a garden-variety binary search, because
@@ -588,6 +574,26 @@ static const char *find_reference_location(struct snapshot *snapshot,
 		return lo;
 }
 
+/*
+ * Find the place in `snapshot->buf` where the start of the record for
+ * `refname` starts. If `mustexist` is true and the reference doesn't
+ * exist, then return NULL. If `mustexist` is false and the reference
+ * doesn't exist, then return the point where that reference would be
+ * inserted, or `snapshot->eof` (which might be NULL) if it would be
+ * inserted at the end of the file. In the latter mode, `refname`
+ * doesn't have to be a proper reference name; for example, one could
+ * search for "refs/replace/" to find the start of any replace
+ * references.
+ *
+ * The record is sought using a binary search, so `snapshot->buf` must
+ * be sorted.
+ */
+static const char *find_reference_location(struct snapshot *snapshot,
+					   const char *refname, int mustexist)
+{
+	return find_reference_location_1(snapshot, refname, mustexist);
+}
+
 /*
  * Create a newly-allocated `snapshot` of the `packed-refs` file in
  * its current state and return it. The return value will already have
Signature · 1 line
2.41.0.44.gf2359540d2
#111 lore raw

PATCH v4 9/16 refs/packed-backend.c: implement jump lists to avoid excluded pattern(s)

In reply to Taylor Blau

When iterating through the packed-refs file in order to answer a query like:

    $ git for-each-ref --exclude=refs/__hidden__

it would be useful to avoid walking over all of the entries in refs/__hidden__/* when possible, since we know that the ref-filter code is going to throw them away anyways.

In certain circumstances, doing so is possible. The algorithm for doing so is as follows:

- For each excluded pattern, find the first record that matches it, and the first record that *doesn't* match it (i.e. the location you'd next want to consider when excluding that pattern).

- Sort the set of excluded regions from the previous step in ascending order of the first location within the packed-refs file that matches.

- Clean up the results from the previous step: discard empty regions, and combine adjacent regions. The set of regions which remains is referred to as the "jump list", and never contains any references which should be included in the result set.

Then when iterating through the packed-refs file, if iter->pos is ever contained in one of the regions from the previous steps, advance iter->pos past the end of that region, and continue enumeration.

Note that we only perform this optimization when none of the excluded pattern(s) have special meta-characters in them. For a pattern like "refs/foo[ac]", the excluded regions ("refs/fooa", "refs/fooc", and everything underneath them) are not connected. A future implementation that handles this case may split the character class (pretending as if two patterns were excluded: "refs/fooa", and "refs/fooc").

There are a few other gotchas worth considering. First, note that the jump list is sorted, so once we jump past a region, we can avoid considering it (or any regions preceding it) again. The member jump_pos is used to track the first next-possible region to jump through.

Second, note that the jump list is best-effort, since we do not handle loose references, and because of the meta-character issue above. The jump list may not skip past all references which won't appear in the results, but will never skip over a reference which does appear in the result set.

In repositories with a large number of hidden references, the speed-up can be significant. Tests here are done with a copy of linux.git with a reference "refs/pull/N" pointing at every commit, as in:

    $ git rev-list HEAD | awk '{ print "create refs/pull/" NR " " $0 }' |
        git update-ref --stdin
    $ git pack-refs --all

, it is significantly faster to have for-each-ref jump over the excluded references, as opposed to filtering them out after the fact:

    $ hyperfine \
      'git for-each-ref --format="%(objectname) %(refname)" | grep -vE "^[0-9a-f]{40} refs/pull/"' \
      'git.compile for-each-ref --format="%(objectname) %(refname)" --exclude="refs/pull"'
    Benchmark 1: git for-each-ref --format="%(objectname) %(refname)" | grep -vE "^[0-9a-f]{40} refs/pull/"
      Time (mean ± σ):     802.7 ms ±   2.1 ms    [User: 691.6 ms, System: 147.0 ms]
      Range (min … max):   800.0 ms … 807.7 ms    10 runs
    Benchmark 2: git.compile for-each-ref --format="%(objectname) %(refname)" --exclude="refs/pull"
      Time (mean ± σ):       4.7 ms ±   0.3 ms    [User: 0.7 ms, System: 4.0 ms]
      Range (min … max):     4.3 ms …   6.7 ms    422 runs

    Summary
      'git.compile for-each-ref --format="%(objectname) %(refname)" --exclude="refs/pull"' ran
      172.03 ± 9.60 times faster than 'git for-each-ref --format="%(objectname) %(refname)" | grep -vE "^[0-9a-f]{40} refs/pull/"'

Using the jump list is fairly straightforward (see the changes to refs/packed-backend.c::next_record()), but constructing the list is not. To ensure that the construction is correct, add a new suite of tests in t1419 covering various corner cases (overlapping regions, partially overlapping regions, adjacent regions, etc.).

---
 ref-filter.c              |   6 +-
 refs/packed-backend.c     | 160 ++++++++++++++++++++++++++++++++++++--
 t/helper/test-ref-store.c |  10 +++
 t/t1419-exclude-refs.sh   | 101 ++++++++++++++++++++++++
 4 files changed, 269 insertions(+), 8 deletions(-)
 create mode 100755 t/t1419-exclude-refs.sh
ref-filter.c +4 -2
index 5e7ed204dc..ddc7f5204f 100644
@@ -2209,12 +2209,14 @@ static int for_each_fullref_in_pattern(struct ref_filter *filter,
 
 	if (!filter->name_patterns[0]) {
 		/* no patterns; we have to look at everything */
-		return for_each_fullref_in("", cb, cb_data);
+		return refs_for_each_fullref_in(get_main_ref_store(the_repository),
+						 "", filter->exclude.v, cb, cb_data);
 	}
 
 	return refs_for_each_fullref_in_prefixes(get_main_ref_store(the_repository),
 						 NULL, filter->name_patterns,
-						 NULL, cb, cb_data);
+						 filter->exclude.v,
+						 cb, cb_data);
 }
 
 /*
refs/packed-backend.c +154 -6

227 lines not on this page · raw message

t/helper/test-ref-store.c +10 -0
index a6977b5e83..08b6d5a86c 100644
@@ -184,6 +184,15 @@ static int cmd_for_each_ref(struct ref_store *refs, const char **argv)
 	return refs_for_each_ref_in(refs, prefix, each_ref, NULL);
 }
 
+static int cmd_for_each_ref__exclude(struct ref_store *refs, const char **argv)
+{
+	const char *prefix = notnull(*argv++, "prefix");
+	const char **exclude_patterns = argv;
+
+	return refs_for_each_fullref_in(refs, prefix, exclude_patterns, each_ref,
+					NULL);
+}
+
 static int cmd_resolve_ref(struct ref_store *refs, const char **argv)
 {
 	struct object_id oid = *null_oid();
@@ -316,6 +325,7 @@ static struct command commands[] = {
 	{ "delete-refs", cmd_delete_refs },
 	{ "rename-ref", cmd_rename_ref },
 	{ "for-each-ref", cmd_for_each_ref },
+	{ "for-each-ref--exclude", cmd_for_each_ref__exclude },
 	{ "resolve-ref", cmd_resolve_ref },
 	{ "verify-ref", cmd_verify_ref },
 	{ "for-each-reflog", cmd_for_each_reflog },
t/t1419-exclude-refs.sh new file +101 -0

101 lines not on this page · raw message

Signature · 1 line
2.41.0.44.gf2359540d2
#114 lore raw

PATCH v4 10/16 refs/packed-backend.c: add trace2 counters for jump list

In reply to Taylor Blau

The previous commit added low-level tests to ensure that the packed-refs iterator did not enumerate excluded sections of the refspace.

However, there was no guarantee that these sections weren't being visited, only that they were being suppressed from the output. To harden these tests, add a trace2 counter which tracks the number of regions skipped by the packed-refs iterator, and assert on its value.

---
 refs/packed-backend.c   |  2 ++
 t/t1419-exclude-refs.sh | 59 ++++++++++++++++++++++++++++-------------
 trace2.h                |  2 ++
 trace2/tr2_ctr.c        |  5 ++++
 4 files changed, 49 insertions(+), 19 deletions(-)
refs/packed-backend.c +2 -0
index f624c9921a..80b877e00c 100644
@@ -12,6 +12,7 @@
 #include "../chdir-notify.h"
 #include "../wrapper.h"
 #include "../write-or-die.h"
+#include "../trace2.h"
 
 enum mmap_strategy {
 	/*
@@ -845,6 +846,7 @@ static int next_record(struct packed_ref_iterator *iter)
 		iter->jump_cur++;
 		if (iter->pos < curr->end) {
 			iter->pos = curr->end;
+			trace2_counter_add(TRACE2_COUNTER_ID_PACKED_REFS_JUMPS, 1);
 			break;
 		}
 	}
t/t1419-exclude-refs.sh +40 -19

120 lines not on this page · raw message

trace2.h +2 -0
index 4ced30c0db..9452e291f5 100644
@@ -551,6 +551,8 @@ enum trace2_counter_id {
 	TRACE2_COUNTER_ID_TEST1 = 0, /* emits summary event only */
 	TRACE2_COUNTER_ID_TEST2,     /* emits summary and thread events */
 
+	TRACE2_COUNTER_ID_PACKED_REFS_JUMPS, /* counts number of jumps */
+
 	/* Add additional counter definitions before here. */
 	TRACE2_NUMBER_OF_COUNTERS
 };
trace2/tr2_ctr.c +5 -0
index b342d3b1a3..50570d0165 100644
@@ -27,6 +27,11 @@ static struct tr2_counter_metadata tr2_counter_metadata[TRACE2_NUMBER_OF_COUNTER
 		.name = "test2",
 		.want_per_thread_events = 1,
 	},
+	[TRACE2_COUNTER_ID_PACKED_REFS_JUMPS] = {
+		.category = "packed-refs",
+		.name = "jumps_made",
+		.want_per_thread_events = 0,
+	},
 
 	/* Add additional metadata before here. */
 };
Signature · 1 line
2.41.0.44.gf2359540d2
#115 lore raw

PATCH v4 11/16 revision.h: store hidden refs in a `strvec`

In reply to Taylor Blau

In subsequent commits, it will be convenient to have a 'const char **' of hidden refs (matching transfer.hiderefs, uploadpack.hideRefs, etc.), instead of a string_list.

Convert spots throughout the tree that store the list of hidden refs from a string_list to a strvec.

Note that in parse_hide_refs_config() there is an ugly const-cast used to avoid an extra copy of each value before trimming any trailing slash characters. This could instead be written as:

    ref = xstrdup(value);
    len = strlen(ref);
    while (len && ref[len - 1] == '/')
            ref[--len] = '\0';
    strvec_push(hide_refs, ref);
    free(ref);

but the double-copy (once when calling xstrdup(), and another via strvec_push()) is wasteful.

---
 builtin/receive-pack.c |  4 ++--
 ls-refs.c              |  6 +++---
 refs.c                 | 11 ++++++-----
 refs.h                 |  4 ++--
 revision.c             |  2 +-
 revision.h             |  5 +++--
 upload-pack.c          | 10 +++++-----
 7 files changed, 22 insertions(+), 20 deletions(-)
builtin/receive-pack.c +2 -2
index 1a31a58367..1a8472eddc 100644
@@ -90,7 +90,7 @@ static struct object_id push_cert_oid;
 static struct signature_check sigcheck;
 static const char *push_cert_nonce;
 static const char *cert_nonce_seed;
-static struct string_list hidden_refs = STRING_LIST_INIT_DUP;
+static struct strvec hidden_refs = STRVEC_INIT;
 
 static const char *NONCE_UNSOLICITED = "UNSOLICITED";
 static const char *NONCE_BAD = "BAD";
@@ -2619,7 +2619,7 @@ int cmd_receive_pack(int argc, const char **argv, const char *prefix)
 		packet_flush(1);
 	oid_array_clear(&shallow);
 	oid_array_clear(&ref);
-	string_list_clear(&hidden_refs, 0);
+	strvec_clear(&hidden_refs);
 	free((void *)push_cert_nonce);
 	return 0;
 }
ls-refs.c +3 -3
index 6f490b2d9c..8c3181d051 100644
@@ -72,7 +72,7 @@ struct ls_refs_data {
 	unsigned symrefs;
 	struct strvec prefixes;
 	struct strbuf buf;
-	struct string_list hidden_refs;
+	struct strvec hidden_refs;
 	unsigned unborn : 1;
 };
 
@@ -155,7 +155,7 @@ int ls_refs(struct repository *r, struct packet_reader *request)
 	memset(&data, 0, sizeof(data));
 	strvec_init(&data.prefixes);
 	strbuf_init(&data.buf, 0);
-	string_list_init_dup(&data.hidden_refs);
+	strvec_init(&data.hidden_refs);
 
 	git_config(ls_refs_config, &data);
 
@@ -197,7 +197,7 @@ int ls_refs(struct repository *r, struct packet_reader *request)
 	packet_fflush(stdout);
 	strvec_clear(&data.prefixes);
 	strbuf_release(&data.buf);
-	string_list_clear(&data.hidden_refs, 0);
+	strvec_clear(&data.hidden_refs);
 	return 0;
 }
 
refs.c +6 -5
index b4b7165fc0..1f01fdf9e8 100644
@@ -1427,7 +1427,7 @@ char *shorten_unambiguous_ref(const char *refname, int strict)
 }
 
 int parse_hide_refs_config(const char *var, const char *value, const char *section,
-			   struct string_list *hide_refs)
+			   struct strvec *hide_refs)
 {
 	const char *key;
 	if (!strcmp("transfer.hiderefs", var) ||
@@ -1438,22 +1438,23 @@ int parse_hide_refs_config(const char *var, const char *value, const char *secti
 
 		if (!value)
 			return config_error_nonbool(var);
-		ref = xstrdup(value);
+
+		/* drop const to remove trailing '/' characters */
+		ref = (char *)strvec_push(hide_refs, value);
 		len = strlen(ref);
 		while (len && ref[len - 1] == '/')
 			ref[--len] = '\0';
-		string_list_append_nodup(hide_refs, ref);
 	}
 	return 0;
 }
 
 int ref_is_hidden(const char *refname, const char *refname_full,
-		  const struct string_list *hide_refs)
+		  const struct strvec *hide_refs)
 {
 	int i;
 
 	for (i = hide_refs->nr - 1; i >= 0; i--) {
-		const char *match = hide_refs->items[i].string;
+		const char *match = hide_refs->v[i];
 		const char *subject;
 		int neg = 0;
 		const char *p;
refs.h +2 -2
index 3c03b035a3..f091741bfa 100644
@@ -816,7 +816,7 @@ int update_ref(const char *msg, const char *refname,
 	       unsigned int flags, enum action_on_err onerr);
 
 int parse_hide_refs_config(const char *var, const char *value, const char *,
-			   struct string_list *);
+			   struct strvec *);
 
 /*
  * Check whether a ref is hidden. If no namespace is set, both the first and
@@ -826,7 +826,7 @@ int parse_hide_refs_config(const char *var, const char *value, const char *,
  * the ref is outside that namespace, the first parameter is NULL. The second
  * parameter always points to the full ref name.
  */
-int ref_is_hidden(const char *, const char *, const struct string_list *);
+int ref_is_hidden(const char *, const char *, const struct strvec *);
 
 /* Is this a per-worktree ref living in the refs/ namespace? */
 int is_per_worktree_ref(const char *refname);
revision.c +1 -1
index 89953592f9..7c9367a266 100644
@@ -1558,7 +1558,7 @@ void init_ref_exclusions(struct ref_exclusions *exclusions)
 void clear_ref_exclusions(struct ref_exclusions *exclusions)
 {
 	string_list_clear(&exclusions->excluded_refs, 0);
-	string_list_clear(&exclusions->hidden_refs, 0);
+	strvec_clear(&exclusions->hidden_refs);
 	exclusions->hidden_refs_configured = 0;
 }
 
revision.h +3 -2
index 25776af381..7f219cde62 100644
@@ -10,6 +10,7 @@
 #include "decorate.h"
 #include "ident.h"
 #include "list-objects-filter-options.h"
+#include "strvec.h"
 
 /**
  * The revision walking API offers functions to build a list of revisions
@@ -95,7 +96,7 @@ struct ref_exclusions {
 	 * Hidden refs is a list of patterns that is to be hidden via
 	 * `ref_is_hidden()`.
 	 */
-	struct string_list hidden_refs;
+	struct strvec hidden_refs;
 
 	/*
 	 * Indicates whether hidden refs have been configured. This is to
@@ -110,7 +111,7 @@ struct ref_exclusions {
  */
 #define REF_EXCLUSIONS_INIT { \
 	.excluded_refs = STRING_LIST_INIT_DUP, \
-	.hidden_refs = STRING_LIST_INIT_DUP, \
+	.hidden_refs = STRVEC_INIT, \
 }
 
 struct oidset;
upload-pack.c +5 -5
index d3312006a3..1a213ed775 100644
@@ -69,7 +69,7 @@ struct upload_pack_data {
 	struct object_array have_obj;
 	struct oid_array haves;					/* v2 only */
 	struct string_list wanted_refs;				/* v2 only */
-	struct string_list hidden_refs;
+	struct strvec hidden_refs;
 
 	struct object_array shallows;
 	struct string_list deepen_not;
@@ -127,7 +127,7 @@ static void upload_pack_data_init(struct upload_pack_data *data)
 {
 	struct string_list symref = STRING_LIST_INIT_DUP;
 	struct string_list wanted_refs = STRING_LIST_INIT_DUP;
-	struct string_list hidden_refs = STRING_LIST_INIT_DUP;
+	struct strvec hidden_refs = STRVEC_INIT;
 	struct object_array want_obj = OBJECT_ARRAY_INIT;
 	struct object_array have_obj = OBJECT_ARRAY_INIT;
 	struct oid_array haves = OID_ARRAY_INIT;
@@ -162,7 +162,7 @@ static void upload_pack_data_clear(struct upload_pack_data *data)
 {
 	string_list_clear(&data->symref, 1);
 	string_list_clear(&data->wanted_refs, 1);
-	string_list_clear(&data->hidden_refs, 0);
+	strvec_clear(&data->hidden_refs);
 	object_array_clear(&data->want_obj);
 	object_array_clear(&data->have_obj);
 	oid_array_clear(&data->haves);
@@ -1170,7 +1170,7 @@ static void receive_needs(struct upload_pack_data *data,
 
 /* return non-zero if the ref is hidden, otherwise 0 */
 static int mark_our_ref(const char *refname, const char *refname_full,
-			const struct object_id *oid, const struct string_list *hidden_refs)
+			const struct object_id *oid, const struct strvec *hidden_refs)
 {
 	struct object *o = lookup_unknown_object(the_repository, oid);
 
@@ -1465,7 +1465,7 @@ static int parse_want(struct packet_writer *writer, const char *line,
 
 static int parse_want_ref(struct packet_writer *writer, const char *line,
 			  struct string_list *wanted_refs,
-			  struct string_list *hidden_refs,
+			  struct strvec *hidden_refs,
 			  struct object_array *want_obj)
 {
 	const char *refname_nons;
Signature · 1 line
2.41.0.44.gf2359540d2
#117 lore raw

PATCH v4 12/16 refs/packed-backend.c: ignore complicated hidden refs rules

In reply to Taylor Blau

In subsequent commits, we'll teach receive-pack and upload-pack to use the new jump list feature in the packed-refs iterator by ignoring references which are mentioned via its respective hideRefs lists.

However, the packed-ref jump lists cannot handle un-hiding rules (that begin with '!'), or namespace comparisons (that begin with '^'). Detect and avoid these cases by falling back to the normal enumeration without a jump list when such patterns exist.

---
 refs/packed-backend.c   | 19 +++++++++++++++++++
 t/t1419-exclude-refs.sh |  9 +++++++++
 2 files changed, 28 insertions(+)
refs/packed-backend.c +19 -0
index 80b877e00c..2aeec5c601 100644
@@ -1008,6 +1008,25 @@ static void populate_excluded_jump_list(struct packed_ref_iterator *iter,
 	if (!excluded_patterns)
 		return;
 
+	for (pattern = excluded_patterns; *pattern; pattern++) {
+		/*
+		 * We also can't feed any excludes from hidden refs
+		 * config sections, since later rules may override
+		 * previous ones. For example, with rules "refs/foo" and
+		 * "!refs/foo/bar", we should show "refs/foo/bar" (and
+		 * everything underneath it), but the earlier exclusion
+		 * would cause us to skip all of "refs/foo". We likewise
+		 * don't implement the namespace stripping required for
+		 * '^' rules.
+		 *
+		 * Both are possible to do, but complicated, so avoid
+		 * populating the jump list at all if we see either of
+		 * these patterns.
+		 */
+		if (**pattern == '!' || **pattern == '^')
+			return;
+	}
+
 	for (pattern = excluded_patterns; *pattern; pattern++) {
 		struct jump_list_entry *e;
 
t/t1419-exclude-refs.sh +9 -0
index 5d8c86b657..f8abf75ab8 100755
@@ -119,4 +119,13 @@ test_expect_success 'meta-characters are discarded' '
 	assert_no_jumps perf
 '
 
+test_expect_success 'complex hidden ref rules are discarded' '
+	for_each_ref__exclude refs/heads refs/heads/foo "!refs/heads/foo/1" \
+		>actual 2>perf &&
+	for_each_ref >expect &&
+
+	test_cmp expect actual &&
+	assert_no_jumps
+'
+
 test_done
Signature · 1 line
2.41.0.44.gf2359540d2
#120 lore raw

PATCH v4 13/16 refs.h: let `for_each_namespaced_ref()` take excluded patterns

In reply to Taylor Blau

A future commit will want to call for_each_namespaced_ref() with a list of excluded patterns.

We could introduce a variant of that function, say, for_each_namespaced_ref_exclude() which takes the extra parameter, and reimplement the original function in terms of that. But all but one caller (in http-backend.c) will supply the new parameter, so add the new parameter to for_each_namespaced_ref() itself instead of introducing a new function.

For now, supply NULL for the list of excluded patterns at all callers to avoid changing behavior, which we will do in a future change.

---
 http-backend.c | 2 +-
 refs.c         | 5 +++--
 refs.h         | 3 ++-
 upload-pack.c  | 6 +++---
 4 files changed, 9 insertions(+), 7 deletions(-)
http-backend.c +1 -1
index ac146d85c5..ad500683c8 100644
@@ -559,7 +559,7 @@ static void get_info_refs(struct strbuf *hdr, char *arg UNUSED)
 
 	} else {
 		select_getanyfile(hdr);
-		for_each_namespaced_ref(show_text_ref, &buf);
+		for_each_namespaced_ref(NULL, show_text_ref, &buf);
 		send_strbuf(hdr, "text/plain", &buf);
 	}
 	strbuf_release(&buf);
refs.c +3 -2
index 1f01fdf9e8..8613184703 100644
@@ -1660,13 +1660,14 @@ int for_each_replace_ref(struct repository *r, each_repo_ref_fn fn, void *cb_dat
 				    DO_FOR_EACH_INCLUDE_BROKEN, cb_data);
 }
 
-int for_each_namespaced_ref(each_ref_fn fn, void *cb_data)
+int for_each_namespaced_ref(const char **exclude_patterns,
+			    each_ref_fn fn, void *cb_data)
 {
 	struct strbuf buf = STRBUF_INIT;
 	int ret;
 	strbuf_addf(&buf, "%srefs/", get_git_namespace());
 	ret = do_for_each_ref(get_main_ref_store(the_repository),
-			      buf.buf, NULL, fn, 0, 0, cb_data);
+			      buf.buf, exclude_patterns, fn, 0, 0, cb_data);
 	strbuf_release(&buf);
 	return ret;
 }
refs.h +2 -1
index f091741bfa..27d341d282 100644
@@ -378,7 +378,8 @@ int for_each_glob_ref_in(each_ref_fn fn, const char *pattern,
 			 const char *prefix, void *cb_data);
 
 int head_ref_namespaced(each_ref_fn fn, void *cb_data);
-int for_each_namespaced_ref(each_ref_fn fn, void *cb_data);
+int for_each_namespaced_ref(const char **exclude_patterns,
+			    each_ref_fn fn, void *cb_data);
 
 /* can be used to learn about broken ref and symref */
 int refs_for_each_rawref(struct ref_store *refs, each_ref_fn fn, void *cb_data);
upload-pack.c +3 -3
index 1a213ed775..99d216938c 100644
@@ -855,7 +855,7 @@ static void deepen(struct upload_pack_data *data, int depth)
 		 * marked with OUR_REF.
 		 */
 		head_ref_namespaced(check_ref, data);
-		for_each_namespaced_ref(check_ref, data);
+		for_each_namespaced_ref(NULL, check_ref, data);
 
 		get_reachable_list(data, &reachable_shallows);
 		result = get_shallow_commits(&reachable_shallows,
@@ -1386,7 +1386,7 @@ void upload_pack(const int advertise_refs, const int stateless_rpc,
 		if (advertise_refs)
 			data.no_done = 1;
 		head_ref_namespaced(send_ref, &data);
-		for_each_namespaced_ref(send_ref, &data);
+		for_each_namespaced_ref(NULL, send_ref, &data);
 		if (!data.sent_capabilities) {
 			const char *refname = "capabilities^{}";
 			write_v0_ref(&data, refname, refname, null_oid());
@@ -1400,7 +1400,7 @@ void upload_pack(const int advertise_refs, const int stateless_rpc,
 		packet_flush(1);
 	} else {
 		head_ref_namespaced(check_ref, &data);
-		for_each_namespaced_ref(check_ref, &data);
+		for_each_namespaced_ref(NULL, check_ref, &data);
 	}
 
 	if (!advertise_refs) {
Signature · 1 line
2.41.0.44.gf2359540d2
#121 lore raw

PATCH v4 14/16 builtin/receive-pack.c: avoid enumerating hidden references

In reply to Taylor Blau

Now that refs_for_each_fullref_in() has the ability to avoid enumerating references matching certain pattern(s), use that to avoid visiting hidden refs when constructing the ref advertisement via receive-pack.

Note that since this exclusion is best-effort, we still need show_ref_cb() to check whether or not each reference is hidden or not before including it in the advertisement.

As was the case when applying this same optimization to upload-pack, receive-pack's reference advertisement phase can proceed much quicker by avoiding enumerating references that will not be part of the advertisement.

(Below, we're still using linux.git with one hidden refs/pull/N ref per commit):

    $ hyperfine -L v ,.compile 'git{v} -c transfer.hideRefs=refs/pull receive-pack --advertise-refs .git'
    Benchmark 1: git -c transfer.hideRefs=refs/pull receive-pack --advertise-refs .git
      Time (mean ± σ):      89.1 ms ±   1.7 ms    [User: 82.0 ms, System: 7.0 ms]
      Range (min … max):    87.7 ms …  95.5 ms    31 runs
    Benchmark 2: git.compile -c transfer.hideRefs=refs/pull receive-pack --advertise-refs .git
      Time (mean ± σ):       4.5 ms ±   0.2 ms    [User: 0.5 ms, System: 3.9 ms]
      Range (min … max):     4.1 ms …   5.6 ms    508 runs

    Summary
      'git.compile -c transfer.hideRefs=refs/pull receive-pack --advertise-refs .git' ran
       20.00 ± 1.05 times faster than 'git -c transfer.hideRefs=refs/pull receive-pack --advertise-refs .git'
---
 builtin/receive-pack.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
builtin/receive-pack.c +2 -1
index 1a8472eddc..bd5bcc375f 100644
@@ -337,7 +337,8 @@ static void write_head_info(void)
 {
 	static struct oidset seen = OIDSET_INIT;
 
-	for_each_ref(show_ref_cb, &seen);
+	refs_for_each_fullref_in(get_main_ref_store(the_repository), "",
+				 hidden_refs.v, show_ref_cb, &seen);
 	for_each_alternate_ref(show_one_alternate_ref, &seen);
 	oidset_clear(&seen);
 	if (!sent_capabilities)
Signature · 1 line
2.41.0.44.gf2359540d2
#122 lore raw

PATCH v4 15/16 upload-pack.c: avoid enumerating hidden refs where possible

In reply to Taylor Blau

In a similar fashion as a previous commit, teach upload-pack to avoid enumerating hidden references where possible.

Note, however, that there are certain cases where cannot avoid enumerating even hidden references, in particular when either of:

- uploadpack.allowTipSHA1InWant, or
- uploadpack.allowReachableSHA1InWant

are set, corresponding to ALLOW_TIP_SHA1 and ALLOW_REACHABLE_SHA1, respectively.

When either of these bits are set, upload-pack's is_our_ref() function needs to consider the HIDDEN_REF bit of the referent's object flags. So we must visit all references, including the hidden ones, in order to mark their referents with the HIDDEN_REF bit.

When neither ALLOW_TIP_SHA1 nor ALLOW_REACHABLE_SHA1 are set, the is_our_ref() function considers only the OUR_REF bit, and not the HIDDEN_REF one. OUR_REF is applied via mark_our_ref(), and only to objects at the tips of non-hidden references, so we do not need to visit hidden references in this case.

When neither of those bits are set, upload-pack can potentially avoid enumerating a large number of references. In the same example as a previous commit (linux.git with one hidden reference per commit, "refs/pull/N"):

    $ printf 0000 >in
    $ hyperfine --warmup=1 \
      'git -c transfer.hideRefs=refs/pull upload-pack . <in' \
      'git.compile -c transfer.hideRefs=refs/pull -c uploadpack.allowTipSHA1InWant upload-pack . <in' \
      'git.compile -c transfer.hideRefs=refs/pull upload-pack . <in'
    Benchmark 1: git -c transfer.hideRefs=refs/pull upload-pack . <in
      Time (mean ± σ):     406.9 ms ±   1.1 ms    [User: 357.3 ms, System: 49.5 ms]
      Range (min … max):   405.7 ms … 409.2 ms    10 runs

Benchmark 2: git.compile -c transfer.hideRefs=refs/pull -c uploadpack.allowTipSHA1InWant upload-pack . <in
Time (mean ± σ): 406.5 ms ± 1.3 ms [User: 356.5 ms, System: 49.9 ms]
Range (min … max): 404.6 ms … 408.8 ms 10 runs

Benchmark 3: git.compile -c transfer.hideRefs=refs/pull upload-pack . <in
Time (mean ± σ): 4.7 ms ± 0.2 ms [User: 0.7 ms, System: 3.9 ms]
Range (min … max): 4.3 ms … 6.1 ms 472 runs

Summary
'git.compile -c transfer.hideRefs=refs/pull upload-pack . <in' ran
86.62 ± 4.33 times faster than 'git.compile -c transfer.hideRefs=refs/pull -c uploadpack.allowTipSHA1InWant upload-pack . <in' 86.70 ± 4.33 times faster than 'git -c transfer.hideRefs=refs/pull upload-pack . <in'

As above, we must visit every reference when uploadPack.allowTipSHA1InWant is set. But when it is unset, we can visit far fewer references.

---
 upload-pack.c | 33 +++++++++++++++++++++++++++------
 1 file changed, 27 insertions(+), 6 deletions(-)
upload-pack.c +27 -6
index 99d216938c..366a101d8d 100644
@@ -602,11 +602,32 @@ static int get_common_commits(struct upload_pack_data *data,
 	}
 }
 
+static int allow_hidden_refs(enum allow_uor allow_uor)
+{
+	return allow_uor & (ALLOW_TIP_SHA1 | ALLOW_REACHABLE_SHA1);
+}
+
+static void for_each_namespaced_ref_1(each_ref_fn fn,
+				      struct upload_pack_data *data)
+{
+	/*
+	 * If `data->allow_uor` allows fetching hidden refs, we need to
+	 * mark all references (including hidden ones), to check in
+	 * `is_our_ref()` below.
+	 *
+	 * Otherwise, we only care about whether each reference's object
+	 * has the OUR_REF bit set or not, so do not need to visit
+	 * hidden references.
+	 */
+	if (allow_hidden_refs(data->allow_uor))
+		for_each_namespaced_ref(NULL, fn, data);
+	else
+		for_each_namespaced_ref(data->hidden_refs.v, fn, data);
+}
+
 static int is_our_ref(struct object *o, enum allow_uor allow_uor)
 {
-	int allow_hidden_ref = (allow_uor &
-				(ALLOW_TIP_SHA1 | ALLOW_REACHABLE_SHA1));
-	return o->flags & ((allow_hidden_ref ? HIDDEN_REF : 0) | OUR_REF);
+	return o->flags & ((allow_hidden_refs(allow_uor) ? HIDDEN_REF : 0) | OUR_REF);
 }
 
 /*
@@ -855,7 +876,7 @@ static void deepen(struct upload_pack_data *data, int depth)
 		 * marked with OUR_REF.
 		 */
 		head_ref_namespaced(check_ref, data);
-		for_each_namespaced_ref(NULL, check_ref, data);
+		for_each_namespaced_ref_1(check_ref, data);
 
 		get_reachable_list(data, &reachable_shallows);
 		result = get_shallow_commits(&reachable_shallows,
@@ -1386,7 +1407,7 @@ void upload_pack(const int advertise_refs, const int stateless_rpc,
 		if (advertise_refs)
 			data.no_done = 1;
 		head_ref_namespaced(send_ref, &data);
-		for_each_namespaced_ref(NULL, send_ref, &data);
+		for_each_namespaced_ref_1(send_ref, &data);
 		if (!data.sent_capabilities) {
 			const char *refname = "capabilities^{}";
 			write_v0_ref(&data, refname, refname, null_oid());
@@ -1400,7 +1421,7 @@ void upload_pack(const int advertise_refs, const int stateless_rpc,
 		packet_flush(1);
 	} else {
 		head_ref_namespaced(check_ref, &data);
-		for_each_namespaced_ref(NULL, check_ref, &data);
+		for_each_namespaced_ref_1(check_ref, &data);
 	}
 
 	if (!advertise_refs) {
Signature · 1 line
2.41.0.44.gf2359540d2
#125 lore raw

PATCH v4 16/16 ls-refs.c: avoid enumerating hidden refs where possible

In reply to Taylor Blau

In a similar fashion as in previous commits, teach ls-refs to avoid enumerating hidden references where possible.

As before, this is linux.git with one hidden reference per commit.

    $ hyperfine -L v ,.compile 'git{v} -c protocol.version=2 ls-remote .'
    Benchmark 1: git -c protocol.version=2 ls-remote .
      Time (mean ± σ):      89.8 ms ±   0.6 ms    [User: 84.3 ms, System: 5.7 ms]
      Range (min … max):    88.8 ms …  91.3 ms    32 runs

Benchmark 2: git.compile -c protocol.version=2 ls-remote .
Time (mean ± σ): 6.5 ms ± 0.1 ms [User: 2.4 ms, System: 4.3 ms]
Range (min … max): 6.2 ms … 8.3 ms 397 runs

Summary
'git.compile -c protocol.version=2 ls-remote .' ran
13.85 ± 0.33 times faster than 'git -c protocol.version=2 ls-remote .'

---
 ls-refs.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
ls-refs.c +1 -1
index 8c3181d051..c9a723ba89 100644
@@ -193,7 +193,7 @@ int ls_refs(struct repository *r, struct packet_reader *request)
 		strvec_push(&data.prefixes, "");
 	refs_for_each_fullref_in_prefixes(get_main_ref_store(r),
 					  get_git_namespace(), data.prefixes.v,
-					  NULL, send_ref, &data);
+					  data.hidden_refs.v, send_ref, &data);
 	packet_fflush(stdout);
 	strvec_clear(&data.prefixes);
 	strbuf_release(&data.buf);
Signature · 1 line
2.41.0.44.gf2359540d2
#79 lore raw

Re: [PATCH v3 09/16] refs/packed-backend.c: implement jump lists to avoid excluded pattern(s)

In reply to Taylor Blau
Quoted 17 lines · Taylor Blau
Quoted 17 lines

refs/packed-backend.c static struct ref_iterator_vtable packed_ref_iterator_vtable = { · line 1003

Quoted 6 lines
refs/packed-backend.c +6 -0
+static const char *ptr_max(const char *x, const char *y)
+{
+	if (x > y)
+		return x;
+	return y;
+}

Hopefully the compiler would inline the function without being told.

These pointers point into the same mmapped region of contiguous memory that holds the contents of the packed-refs file, so comparison between them is always defined. Good.

I wondered if

	return (x > y) ? x : y;
Quoted 7 lines
Quoted 3 lines

is easier to read, simply because it treats both cases more equally (in other words, as written, (x>y) appears more "special"), but that is minor.

Yeah, I think that any reasonable compiler would almost certainly inline this, especially at higher optimization levels. But I agree with your suggestion nonetheless, thanks.

Having seen how this is used (only at a single callsite), I actually think that special casing (x>y) is the right thing to do, especially if you inline it in the caller. That is,

	if (last_disjoint->end < ours->end)
		last_disjoint->end = ours->end;

reads much more naturally than

	last_disjoint->end = (last_disjoint->end > ours->end)
		? last_disjoint->end : ours_end;

as a way to say "if ours is larger, record it as the largest position we have seen so far".

#93 lore raw

Re: [PATCH v4 01/16] refs.c: rename `ref_filter`

In reply to Taylor Blau
Quoted 5 lines · Taylor Blau

From: Jeff King <[email protected]>

The refs machinery has its own implementation of a ref_filter (used by for-each-ref), which is distinct from the ref-filler.h API (also used by for-each-ref, among other things).

Small typo here: s/filler/filter/

-Peff

#95 lore raw

Re: [PATCH v4 02/16] ref-filter.h: provide `REF_FILTER_INIT`

In reply to Taylor Blau
Quoted 7 lines · Taylor Blau

From: Jeff King <[email protected]>

Provide a sane initialization value for struct ref_filter, which in a subsequent patch will be used to initialize a new field.

In the meantime, fix a case in test-reach.c where its ref_filter is not even zero-initialized.

This test-reach case scared me, but it happens to work now because commit_contains() only looks at the one field that we set. So we're not fixing a bug, but more like a bug waiting to happen. :)

-Peff

#98 lore raw

Re: [PATCH v4 03/16] ref-filter: clear reachable list pointers after freeing

In reply to Taylor Blau
Quoted 6 lines · Taylor Blau

From: Jeff King <[email protected]>

In reach_filter(), we pop all commits from the reachable lists, leaving them empty. But because we're operating on a list pointer that was passed by value, the original filter.reachable_from pointer is left dangling.

Yep. This isn't a bug (yet) because nobody looks at the now-dangling pointer. So as with the last patch, we're future-proofing ourselves against dangerous situations.

-Peff

#100 lore raw

Re: [PATCH v4 04/16] ref-filter: add `ref_filter_clear()`

In reply to Taylor Blau
Quoted 9 lines · Taylor Blau

From: Jeff King <[email protected]>

We did not bother to clean up at all in git branch or git tag, and git for-each-ref only cleans up a couple of members.

Add and call ref_filter_clear() when cleaning up a struct ref_filter. Running this patch (without any test changes) indicates a couple of now leak-free tests. This was found by running:

Quoted 6 lines
    $ make SANITIZE=leak
    $ make -C t GIT_TEST_PASSING_SANITIZE_LEAK=check GIT_TEST_OPTS=--immediate

(Note that the reachable_from and unreachable_from lists should be cleaned as they are used. So this is just covering any case where we might bail before running the reachability check.)

And this is the one that benefits from the earlier future-proofing. :)

(In case anyone is wondering why I am reviewing my own commits, it's because Taylor and I worked on this topic together off-list, but he wrote the commit messages after I dumped a bunch of cleanups on him).

ref-filter.c int parse_opt_merge_filter(const struct option *opt, const char *arg, int unset) · line 2874

Quoted 5 lines
ref-filter.c +5 -0
+void ref_filter_init(struct ref_filter *filter)
+{
+	struct ref_filter blank = REF_FILTER_INIT;
+	memcpy(filter, &blank, sizeof(blank));
+}

I was a little surprised by adding init() here, but we need it at the end of clear(). So this is an OK place for it (the other option would be in the earlier INIT patch, but it would be unused until now).

-Peff

#104 lore raw

Re: [PATCH v4 05/16] ref-filter.c: parameterize match functions over patterns

In reply to Taylor Blau
Quoted 4 lines · Taylor Blau

Once we start passing either in, match_pattern() will have little to do with a particular struct ref_filter * instance. To clarify this, drop it from the argument list, and replace it with the only bit of the ref_filter that we care about (filter->ignore_case).

Makes sense, but...

Review static int match_pattern(const struct ref_filter *filter, const char *refname) · line 2139

Quoted 8 lines
(quoted diff) +3 -1
@@ -2134,9 +2134,10 @@ static int match_pattern(const struct ref_filter *filter, const char *refname)
  * matches a pattern "refs/heads/" but not "refs/heads/m") or a
  * wildcard (e.g. the same ref matches "refs/heads/m*", too).
  */
-static int match_name_as_path(const struct ref_filter *filter, const char *refname)
+static int match_name_as_path(const struct ref_filter *filter,
+			      const char **pattern,
+			      const char *refname)

...wouldn't we then want to do the same for match_name_as_path()?

I.e., this:

ref-filter.c +5 -5
index 6aacb99be7..cf10c753e2 100644
@@ -2134,14 +2134,13 @@ static int match_pattern(const char **patterns, const char *refname,
  * matches a pattern "refs/heads/" but not "refs/heads/m") or a
  * wildcard (e.g. the same ref matches "refs/heads/m*", too).
  */
-static int match_name_as_path(const struct ref_filter *filter,
-			      const char **pattern,
-			      const char *refname)
+static int match_name_as_path(const char **pattern, const char *refname,
+			      int ignore_case)
 {
 	int namelen = strlen(refname);
 	unsigned flags = WM_PATHNAME;
 
-	if (filter->ignore_case)
+	if (ignore_case)
 		flags |= WM_CASEFOLD;
 
 	for (; *pattern; pattern++) {
@@ -2166,7 +2165,8 @@ static int filter_pattern_match(struct ref_filter *filter, const char *refname)
 	if (!*filter->name_patterns)
 		return 1; /* No pattern always matches */
 	if (filter->match_as_path)
-		return match_name_as_path(filter, filter->name_patterns, refname);
+		return match_name_as_path(filter->name_patterns, refname,
+					  filter->ignore_case);
 	return match_pattern(filter->name_patterns, refname,
 			     filter->ignore_case);
 }

Also, I noticed that you declared it as "const int ignore_case" in match_pattern(). That's not wrong, but we usually do not bother (it is passed by value, so const-ness is irrelevant to the caller, and the compiler can see inside the function that the value is not changed and optimize appropriately).

-Peff

#112 lore raw

Re: [PATCH v4 09/16] refs/packed-backend.c: implement jump lists to avoid excluded pattern(s)

In reply to Taylor Blau
Quoted 5 lines · Taylor Blau

Second, note that the jump list is best-effort, since we do not handle loose references, and because of the meta-character issue above. The jump list may not skip past all references which won't appear in the results, but will never skip over a reference which does appear in the result set.

I wonder if we should be advertising this in a docstring comment above the relevant function. The problem may be that there are several such functions. I just think that it's a gotcha that may affect somebody who wants to call the function, and they're not going to think to dig up this commit message.

Quoted 7 lines
    $ hyperfine \
      'git for-each-ref --format="%(objectname) %(refname)" | grep -vE "^[0-9a-f]{40} refs/pull/"' \
      'git.compile for-each-ref --format="%(objectname) %(refname)" --exclude="refs/pull"'
    Benchmark 1: git for-each-ref --format="%(objectname) %(refname)" | grep -vE "^[0-9a-f]{40} refs/pull/"
      Time (mean ± σ):     802.7 ms ±   2.1 ms    [User: 691.6 ms, System: 147.0 ms]
      Range (min … max):   800.0 ms … 807.7 ms    10 runs
Quoted 7 lines
    Benchmark 2: git.compile for-each-ref --format="%(objectname) %(refname)" --exclude="refs/pull"
      Time (mean ± σ):       4.7 ms ±   0.3 ms    [User: 0.7 ms, System: 4.0 ms]
      Range (min … max):     4.3 ms …   6.7 ms    422 runs

    Summary
      'git.compile for-each-ref --format="%(objectname) %(refname)" --exclude="refs/pull"' ran
      172.03 ± 9.60 times faster than 'git for-each-ref --format="%(objectname) %(refname)" | grep -vE "^[0-9a-f]{40} refs/pull/"'

This measurement is cheating a little, I think, because the earlier patch to implement --exclude sped that up from ~800ms to ~100ms (because we avoid writing and all of the ref-filter malloc slowness for the excluded entries). So the better comparison is between two invocations with "--exclude", but before/after this patch. You should still see a 20x speedup (100ms down to 5).

Quoted 17 lines
refs/packed-backend.c +9 -1
@@ -802,14 +826,34 @@ struct packed_ref_iterator {
  */
 static int next_record(struct packed_ref_iterator *iter)
 {
-	const char *p = iter->pos, *eol;
+	const char *p, *eol;
 
 	strbuf_reset(&iter->refname_buf);
 
+	/*
+	 * If iter->pos is contained within a skipped region, jump past
+	 * it.
+	 *
+	 * Note that each skipped region is considered at most once,
+	 * since they are ordered based on their starting position.
+	 */
+	while (iter->jump_cur < iter->jump_nr) {

refs/packed-backend.c struct packed_ref_iterator { · line 850

Quoted 10 lines
refs/packed-backend.c +10 -0
+		struct jump_list_entry *curr = &iter->jump[iter->jump_cur];
+		if (iter->pos < curr->start)
+			break; /* not to the next jump yet */
+
+		iter->jump_cur++;
+		if (iter->pos < curr->end) {
+			iter->pos = curr->end;
+			break;
+		}
+	}

It took me a minute to convince myself that this second "break" was right. If we get to it, we know that iter->pos (the current record we are looking at) is in the current jump region. So it makes sense to advance to curr->end. But might we hit another jump region immediately?

I guess not, because earlier we would have coalesced the jump regions. So either there is a non-excluded entry there _or_ we would have coalesced the later region into a single larger region. So breaking is the right thing to do.

refs/packed-backend.c static struct ref_iterator_vtable packed_ref_iterator_vtable = { · line 1021

Quoted 13 lines
refs/packed-backend.c +13 -0
+	for (pattern = excluded_patterns; *pattern; pattern++) {
+		struct jump_list_entry *e;
+
+		/*
+		 * We can't feed any excludes with globs in them to the
+		 * refs machinery.  It only understands prefix matching.
+		 * We likewise can't even feed the string leading up to
+		 * the first meta-character, as something like "foo[a]"
+		 * should not exclude "foobar" (but the prefix "foo"
+		 * would match that and mark it for exclusion).
+		 */
+		if (has_glob_special(*pattern))
+			continue;

OK, and here's where we could split "foo[ac]" into "fooa" and "foob" if we wanted. But I think it is a very good idea to leave that out of this initial patch. :)

Quoted 10 lines
refs/packed-backend.c +10 -0
+	/*
+	 * As an optimization, merge adjacent entries in the jump list
+	 * to jump forwards as far as possible when entering a skipped
+	 * region.
+	 *
+	 * For example, if we have two skipped regions:
+	 *
+	 *	[[A, B], [B, C]]
+	 *
+	 * we want to combine that into a single entry jumping from A to

refs/packed-backend.c static struct ref_iterator_vtable packed_ref_iterator_vtable = { · line 1059

Quoted 10 lines
refs/packed-backend.c +10 -0
+	 * C.
+	 */
+	last_disjoint = iter->jump;
+
+	for (i = 1, j = 1; i < iter->jump_nr; i++) {
+		struct jump_list_entry *ours = &iter->jump[i];
+
+		if (ours->start == ours->end) {
+			/* ignore empty regions (no matching entries) */
+			continue;

Dropping empty regions makes sense, but our iteration starts at "1" (because the rest of the checks are inherently looking at last_disjoint before deciding if each region is worth keeping). So we'd fail to throw away iter->jump[0] if it is empty, I think.

That could be fixed here by iterating from 0 and checking for a NULL last_disjoint, but maybe it would be easier to avoid allocating at all in the earlier loop, when we find that start == end?

refs/packed-backend.c static struct ref_iterator_vtable packed_ref_iterator_vtable = { · line 1063

Quoted 4 lines
refs/packed-backend.c +4 -0
+		} else if (ours->start <= last_disjoint->end) {
+			/* overlapping regions extend the previous one */
+			last_disjoint->end = last_disjoint->end > ours->end
+				? last_disjoint->end : ours->end;

OK, this covers both ([A,C],[B,D]) via "<" and ([A,B],[B,C]) via "=". Good.

refs/packed-backend.c static struct ref_iterator_vtable packed_ref_iterator_vtable = { · line 1069

Quoted 6 lines
refs/packed-backend.c +6 -0
+		} else {
+			/* otherwise, insert a new region */
+			iter->jump[j++] = *ours;
+			last_disjoint = ours;
+
+		}

And this is the rest. Good. There's an extra blank line here before the closing brace.

-Peff

#116 lore raw

Re: [PATCH v4 11/16] revision.h: store hidden refs in a `strvec`

In reply to Taylor Blau
Quoted 18 lines · Taylor Blau

In subsequent commits, it will be convenient to have a 'const char **' of hidden refs (matching transfer.hiderefs, uploadpack.hideRefs, etc.), instead of a string_list.

Convert spots throughout the tree that store the list of hidden refs from a string_list to a strvec.

Note that in parse_hide_refs_config() there is an ugly const-cast used to avoid an extra copy of each value before trimming any trailing slash characters. This could instead be written as:

    ref = xstrdup(value);
    len = strlen(ref);
    while (len && ref[len - 1] == '/')
            ref[--len] = '\0';
    strvec_push(hide_refs, ref);
    free(ref);
Quoted 2 lines

but the double-copy (once when calling xstrdup(), and another via strvec_push()) is wasteful.

I saw strvec_push_nodup() suggested here. I'm OK leaving it like this for now, but I do think we'll want that in the long run.

-Peff

#118 lore raw

Re: [PATCH v4 12/16] refs/packed-backend.c: ignore complicated hidden refs rules

In reply to Taylor Blau
Quoted 8 lines · Taylor Blau

In subsequent commits, we'll teach receive-pack and upload-pack to use the new jump list feature in the packed-refs iterator by ignoring references which are mentioned via its respective hideRefs lists.

However, the packed-ref jump lists cannot handle un-hiding rules (that begin with '!'), or namespace comparisons (that begin with '^'). Detect and avoid these cases by falling back to the normal enumeration without a jump list when such patterns exist.

I'm a fan of punting on such cases to keep things simple and incremental. But the location here seems weird to me:

Quoted 17 lines
refs/packed-backend.c +9 -0
index 80b877e00c..2aeec5c601 100644
@@ -1008,6 +1008,25 @@ static void populate_excluded_jump_list(struct packed_ref_iterator *iter,
 	if (!excluded_patterns)
 		return;
 
+	for (pattern = excluded_patterns; *pattern; pattern++) {
+		/*
+		 * We also can't feed any excludes from hidden refs
+		 * config sections, since later rules may override
+		 * previous ones. For example, with rules "refs/foo" and
+		 * "!refs/foo/bar", we should show "refs/foo/bar" (and
+		 * everything underneath it), but the earlier exclusion
+		 * would cause us to skip all of "refs/foo". We likewise
+		 * don't implement the namespace stripping required for

refs/packed-backend.c static void populate_excluded_jump_list(struct packed_ref_iterator *iter, · line 1029

Quoted 10 lines
refs/packed-backend.c +10 -0
+		 * '^' rules.
+		 *
+		 * Both are possible to do, but complicated, so avoid
+		 * populating the jump list at all if we see either of
+		 * these patterns.
+		 */
+		if (**pattern == '!' || **pattern == '^')
+			return;
+	}
+

This is deep in the packed-refs code, but the magic of "!" and "^" are specific to ref_is_hidden().

So if I did:

  git for-each-ref --exclude='!refs/heads/foo'

my understanding is that "!" would _not_ have an affect normally, but now it is turning off this optimization.

The point may be somewhat academic for "^", as it is not allowed in a refname anyway. But I don't think "!" is forbidden (as stupid as it would be to include it in this way), is it?

It feels like the hiderefs code should be the one checking for these, and then feeding only non-adorned refnames to the "exclude" list (though there is no need to un-adorn them; once we see any with either form of magic, we know we cannot use this "exclude" feature at all).

Something along the lines of (you'd want a similar tweak for upload-pack):

builtin/receive-pack.c +2 -1
index 2b2faa5d18..80a6b11c90 100644
@@ -339,7 +339,8 @@ static void write_head_info(void)
 	static struct oidset seen = OIDSET_INIT;
 
 	refs_for_each_fullref_in(get_main_ref_store(the_repository), "",
-				 hidden_refs.v, show_ref_cb, &seen);
+				 hidden_refs_to_excludes(&hidden_refs),
+				 show_ref_cb, &seen);
 	for_each_alternate_ref(show_one_alternate_ref, &seen);
 	oidset_clear(&seen);
 	if (!sent_capabilities)
refs.c +25 -0
index 3065e514fd..213412efd4 100644
@@ -1482,6 +1482,31 @@ int ref_is_hidden(const char *refname, const char *refname_full,
 	return 0;
 }
 
+const char **hidden_refs_to_excludes(const struct strvec *hide_refs)
+{
+	const char **pattern;
+
+	for (pattern = hide_refs->v; *pattern; pattern++) {
+		/*
+		 * We also can't feed any excludes from hidden refs
+		 * config sections, since later rules may override
+		 * previous ones. For example, with rules "refs/foo" and
+		 * "!refs/foo/bar", we should show "refs/foo/bar" (and
+		 * everything underneath it), but the earlier exclusion
+		 * would cause us to skip all of "refs/foo". We likewise
+		 * don't implement the namespace stripping required for
+		 * '^' rules.
+		 *
+		 * Both are possible to do, but complicated, so avoid
+		 * populating the jump list at all if we see either of
+		 * these patterns.
+		 */
+		if (**pattern == '!' || **pattern == '^')
+			return NULL;
+	}
+	return hide_refs->v;
+}
+
 const char *find_descendant_ref(const char *dirname,
 				const struct string_list *extras,
 				const struct string_list *skip)
refs.h +2 -0
index 27d341d282..50c92d1f55 100644
@@ -829,6 +829,8 @@ int parse_hide_refs_config(const char *var, const char *value, const char *,
  */
 int ref_is_hidden(const char *, const char *, const struct strvec *);
 
+const char **hidden_refs_to_excludes(const struct strvec *hide_refs);
+
 /* Is this a per-worktree ref living in the refs/ namespace? */
 int is_per_worktree_ref(const char *refname);
 
refs/packed-backend.c +0 -19
index 9dd1795bf2..59c3fe9d91 100644
@@ -1009,25 +1009,6 @@ static void populate_excluded_jump_list(struct packed_ref_iterator *iter,
 	if (!excluded_patterns)
 		return;
 
-	for (pattern = excluded_patterns; *pattern; pattern++) {
-		/*
-		 * We also can't feed any excludes from hidden refs
-		 * config sections, since later rules may override
-		 * previous ones. For example, with rules "refs/foo" and
-		 * "!refs/foo/bar", we should show "refs/foo/bar" (and
-		 * everything underneath it), but the earlier exclusion
-		 * would cause us to skip all of "refs/foo". We likewise
-		 * don't implement the namespace stripping required for
-		 * '^' rules.
-		 *
-		 * Both are possible to do, but complicated, so avoid
-		 * populating the jump list at all if we see either of
-		 * these patterns.
-		 */
-		if (**pattern == '!' || **pattern == '^')
-			return;
-	}
-
 	for (pattern = excluded_patterns; *pattern; pattern++) {
 		struct jump_list_entry *e;
 
#123 lore raw

Re: [PATCH v4 15/16] upload-pack.c: avoid enumerating hidden refs where possible

In reply to Taylor Blau
Quoted 17 lines · Taylor Blau

In a similar fashion as a previous commit, teach upload-pack to avoid enumerating hidden references where possible.

Note, however, that there are certain cases where cannot avoid enumerating even hidden references, in particular when either of:

- uploadpack.allowTipSHA1InWant, or
- uploadpack.allowReachableSHA1InWant

are set, corresponding to ALLOW_TIP_SHA1 and ALLOW_REACHABLE_SHA1, respectively.

When either of these bits are set, upload-pack's is_our_ref() function needs to consider the HIDDEN_REF bit of the referent's object flags. So we must visit all references, including the hidden ones, in order to mark their referents with the HIDDEN_REF bit.

Quoted 5 lines

When neither ALLOW_TIP_SHA1 nor ALLOW_REACHABLE_SHA1 are set, the is_our_ref() function considers only the OUR_REF bit, and not the HIDDEN_REF one. OUR_REF is applied via mark_our_ref(), and only to objects at the tips of non-hidden references, so we do not need to visit hidden references in this case.

Both of these are noops if uploadpack.allowAnySHA1InWant is set, or if we are using the v2 protocol. Setting both allowAny and allowTip is sufficiently stupid that I don't care much that they lose out on an optimization. But I could see somebody setting one of those for the benefit of v0 clients, but then also serving v2 clients (which effectively ignore those restrictions).

I guess v2 clients don't hit this code at all (they are handled by ls-refs, which comes in the next patch). So that just leaves the case that allowAny is set. By itself the optimization should kick in (good). With allowTip or allowReachable it would not, but that is the "this is stupid" case in which it is OK to fall back to the existing behavior (even though we _could_ enable the optimization). OTOH, it would be easy to check it, as it's just another bit in allow_uor.

I'm OK going either way.

-Peff

#126 lore raw

Re: [PATCH v4 16/16] ls-refs.c: avoid enumerating hidden refs where possible

In reply to Taylor Blau
Quoted 5 lines · Taylor Blau

In a similar fashion as in previous commits, teach ls-refs to avoid enumerating hidden references where possible.

As before, this is linux.git with one hidden reference per commit.

Quoted 8 lines
    $ hyperfine -L v ,.compile 'git{v} -c protocol.version=2 ls-remote .'
    Benchmark 1: git -c protocol.version=2 ls-remote .
      Time (mean ± σ):      89.8 ms ±   0.6 ms    [User: 84.3 ms, System: 5.7 ms]
      Range (min … max):    88.8 ms …  91.3 ms    32 runs

Benchmark 2: git.compile -c protocol.version=2 ls-remote .
Time (mean ± σ): 6.5 ms ± 0.1 ms [User: 2.4 ms, System: 4.3 ms]
Range (min … max): 6.2 ms … 8.3 ms 397 runs

Very nice. I think this may have big real-world consequences for certain repositories on forges (where they may accrue a large number of hidden metadata like refs/pull).

-Peff

#127 lore raw

Re: [PATCH v4 00/16] refs: implement jump lists for packed backend

In reply to Taylor Blau
Quoted 7 lines · Taylor Blau

Here is another reroll of my series to implement jump (née skip) lists for the packed refs backend, based on top of the current 'master'.

Most changes are minor, limited to changing variable names, moving changes around between patches and tweaking commit messages for clarity. I think that the first 9 or so patches are stable, but we may want some more eyes on the remainder.

I had been avoiding reading this series too carefully, as I was involved in many of the early patches. But now it's been long enough that I mostly forgot everything, so I could look at it with fresh eyes. ;) Plus the hidden-refs bits at the end were totally new to me.

It looks good to me overall. I left a few small comments, some of which I think probably justify a re-roll.

-Peff

#96 lore raw

Re: [PATCH v4 02/16] ref-filter.h: provide `REF_FILTER_INIT`

In reply to Jeff King
Quoted 6 lines · Jeff King
Quoted 2 lines

In the meantime, fix a case in test-reach.c where its ref_filter is not even zero-initialized.

This test-reach case scared me, but it happens to work now because commit_contains() only looks at the one field that we set. So we're not fixing a bug, but more like a bug waiting to happen. :)

Good point, I updated the commit message to more clearly signal that this isn't fixing a bug, but rather preventing one.

Thanks,
Taylor

#101 lore raw

Re: [PATCH v4 04/16] ref-filter: add `ref_filter_clear()`

In reply to Jeff King
Quoted 9 lines · Jeff King

ref-filter.c int parse_opt_merge_filter(const struct option *opt, const char *arg, int unset) · line 2874

Quoted 5 lines
ref-filter.c +5 -0
+void ref_filter_init(struct ref_filter *filter)
+{
+	struct ref_filter blank = REF_FILTER_INIT;
+	memcpy(filter, &blank, sizeof(blank));
+}

I was a little surprised by adding init() here, but we need it at the end of clear(). So this is an OK place for it (the other option would be in the earlier INIT patch, but it would be unused until now).

I used to write more patches in this style where I would add as much of a new API as possible as early as possible in the series. But I think reviewers seem to have an easier time reviewing API additions in the same patch that adds their caller.

So I tend to agree that this patch is probably a good spot to introduce ref_filter_init(). But if you feel strongly, I'm happy to drag it around.

Thanks,
Taylor

#105 lore raw

Re: [PATCH v4 05/16] ref-filter.c: parameterize match functions over patterns

In reply to Jeff King
Quoted 9 lines · Jeff King
Quoted 4 lines · Taylor Blau

Once we start passing either in, match_pattern() will have little to do with a particular struct ref_filter * instance. To clarify this, drop it from the argument list, and replace it with the only bit of the ref_filter that we care about (filter->ignore_case).

Makes sense, but...

Quoted 10 lines

Review static int match_pattern(const struct ref_filter *filter, const char *refname) · line 2139

Quoted 8 lines
(quoted diff) +3 -1
@@ -2134,9 +2134,10 @@ static int match_pattern(const struct ref_filter *filter, const char *refname)
  * matches a pattern "refs/heads/" but not "refs/heads/m") or a
  * wildcard (e.g. the same ref matches "refs/heads/m*", too).
  */
-static int match_name_as_path(const struct ref_filter *filter, const char *refname)
+static int match_name_as_path(const struct ref_filter *filter,
+			      const char **pattern,
+			      const char *refname)

...wouldn't we then want to do the same for match_name_as_path()?

Yes, definitely :-). I'm not sure how I missed this, since the patch message even says that match_name_as_path() gets the same treatment as the other function.

But in any case, I obviously agree (and the diff below makes sense to me). Applied.

Quoted 5 lines

Also, I noticed that you declared it as "const int ignore_case" in match_pattern(). That's not wrong, but we usually do not bother (it is passed by value, so const-ness is irrelevant to the caller, and the compiler can see inside the function that the value is not changed and optimize appropriately).

Indeed :-).

Thanks,
Taylor

#106 lore raw

Re: [PATCH v4 05/16] ref-filter.c: parameterize match functions over patterns

Quoted 21 lines · Taylor Blau
Quoted 9 lines · Jeff King
Quoted 4 lines · Taylor Blau

Once we start passing either in, match_pattern() will have little to do with a particular struct ref_filter * instance. To clarify this, drop it from the argument list, and replace it with the only bit of the ref_filter that we care about (filter->ignore_case).

Makes sense, but...

Quoted 10 lines

Review static int match_pattern(const struct ref_filter *filter, const char *refname) · line 2139

Quoted 8 lines
(quoted diff) +3 -1
@@ -2134,9 +2134,10 @@ static int match_pattern(const struct ref_filter *filter, const char *refname)
  * matches a pattern "refs/heads/" but not "refs/heads/m") or a
  * wildcard (e.g. the same ref matches "refs/heads/m*", too).
  */
-static int match_name_as_path(const struct ref_filter *filter, const char *refname)
+static int match_name_as_path(const struct ref_filter *filter,
+			      const char **pattern,
+			      const char *refname)

...wouldn't we then want to do the same for match_name_as_path()?

Quoted 6 lines

Yes, definitely :-). I'm not sure how I missed this, since the patch message even says that match_name_as_path() gets the same treatment as the other function.

But in any case, I obviously agree (and the diff below makes sense to me). Applied.

Ah, this is missing one more spot (that we wouldn't complain about during a non-DEVELOPER build). This needs to go on top, but I otherwise agree:

--- 8< ---

ref-filter.c +2 -2
index de85904b8d..845173a904 100644
@@ -2176,7 +2176,8 @@ static int filter_exclude_match(struct ref_filter *filter, const char *refname)
 	if (!filter->exclude.nr)
 		return 0;
 	if (filter->match_as_path)
-		return match_name_as_path(filter, filter->exclude.v, refname);
+		return match_name_as_path(filter->exclude.v, refname,
+					  filter->ignore_case);
 	return match_pattern(filter->exclude.v, refname, filter->ignore_case);
 }
--- >8 ---

Thanks,
Taylor

#102 lore raw

Re: [PATCH v4 04/16] ref-filter: add `ref_filter_clear()`

In reply to Taylor Blau
Quoted 11 lines · Taylor Blau
Quoted 9 lines · Jeff King

ref-filter.c int parse_opt_merge_filter(const struct option *opt, const char *arg, int unset) · line 2874

Quoted 5 lines
ref-filter.c +5 -0
+void ref_filter_init(struct ref_filter *filter)
+{
+	struct ref_filter blank = REF_FILTER_INIT;
+	memcpy(filter, &blank, sizeof(blank));
+}

I was a little surprised by adding init() here, but we need it at the end of clear(). So this is an OK place for it (the other option would be in the earlier INIT patch, but it would be unused until now).

Quoted 8 lines

I used to write more patches in this style where I would add as much of a new API as possible as early as possible in the series. But I think reviewers seem to have an easier time reviewing API additions in the same patch that adds their caller.

So I tend to agree that this patch is probably a good spot to introduce ref_filter_init(). But if you feel strongly, I'm happy to drag it around.

Nope, I don't feel strongly at all. Let's leave it as you have it.

-Peff

#107 lore raw

Re: [PATCH v4 05/16] ref-filter.c: parameterize match functions over patterns

In reply to Taylor Blau
Quoted 26 lines · Taylor Blau
Quoted 25 lines · Taylor Blau
Quoted 9 lines · Jeff King
Quoted 4 lines · Taylor Blau

Once we start passing either in, match_pattern() will have little to do with a particular struct ref_filter * instance. To clarify this, drop it from the argument list, and replace it with the only bit of the ref_filter that we care about (filter->ignore_case).

Makes sense, but...

Quoted 10 lines

Review static int match_pattern(const struct ref_filter *filter, const char *refname) · line 2139

Quoted 8 lines
(quoted diff) +3 -1
@@ -2134,9 +2134,10 @@ static int match_pattern(const struct ref_filter *filter, const char *refname)
  * matches a pattern "refs/heads/" but not "refs/heads/m") or a
  * wildcard (e.g. the same ref matches "refs/heads/m*", too).
  */
-static int match_name_as_path(const struct ref_filter *filter, const char *refname)
+static int match_name_as_path(const struct ref_filter *filter,
+			      const char **pattern,
+			      const char *refname)

...wouldn't we then want to do the same for match_name_as_path()?

Yes, definitely :-). I'm not sure how I missed this, since the patch message even says that match_name_as_path() gets the same treatment as the other function.

Quoted 6 lines
Quoted 2 lines

But in any case, I obviously agree (and the diff below makes sense to me). Applied.

Ah, this is missing one more spot (that we wouldn't complain about during a non-DEVELOPER build). This needs to go on top, but I otherwise agree:

Heh, yes. I was applying them in order, and had to make the same fixup on top of the next patch. I think that was the only other fallout as I applied the rest, though.

-Peff

#113 lore raw

Re: [PATCH v4 09/16] refs/packed-backend.c: implement jump lists to avoid excluded pattern(s)

In reply to Jeff King
Quoted 8 lines · Jeff King
Quoted 5 lines · Taylor Blau

Second, note that the jump list is best-effort, since we do not handle loose references, and because of the meta-character issue above. The jump list may not skip past all references which won't appear in the results, but will never skip over a reference which does appear in the result set.

Quoted 5 lines

I wonder if we should be advertising this in a docstring comment above the relevant function. The problem may be that there are several such functions. I just think that it's a gotcha that may affect somebody who wants to call the function, and they're not going to think to dig up this commit message.

Good idea, thanks.

Quoted 15 lines
Quoted 7 lines
    $ hyperfine \
      'git for-each-ref --format="%(objectname) %(refname)" | grep -vE "^[0-9a-f]{40} refs/pull/"' \
      'git.compile for-each-ref --format="%(objectname) %(refname)" --exclude="refs/pull"'
    Benchmark 1: git for-each-ref --format="%(objectname) %(refname)" | grep -vE "^[0-9a-f]{40} refs/pull/"
      Time (mean ± σ):     802.7 ms ±   2.1 ms    [User: 691.6 ms, System: 147.0 ms]
      Range (min … max):   800.0 ms … 807.7 ms    10 runs
Quoted 7 lines
    Benchmark 2: git.compile for-each-ref --format="%(objectname) %(refname)" --exclude="refs/pull"
      Time (mean ± σ):       4.7 ms ±   0.3 ms    [User: 0.7 ms, System: 4.0 ms]
      Range (min … max):     4.3 ms …   6.7 ms    422 runs

    Summary
      'git.compile for-each-ref --format="%(objectname) %(refname)" --exclude="refs/pull"' ran
      172.03 ± 9.60 times faster than 'git for-each-ref --format="%(objectname) %(refname)" | grep -vE "^[0-9a-f]{40} refs/pull/"'
Quoted 6 lines

This measurement is cheating a little, I think, because the earlier patch to implement --exclude sped that up from ~800ms to ~100ms (because we avoid writing and all of the ref-filter malloc slowness for the excluded entries). So the better comparison is between two invocations with "--exclude", but before/after this patch. You should still see a 20x speedup (100ms down to 5).

I agree. I included a build from the previous commit in this benchmark. As expected, it's around ~100ms, but at least it gives readers a clearer picture of how performance changes as a result of this patch. (

Quoted 33 lines
Quoted 17 lines
refs/packed-backend.c +9 -1
@@ -802,14 +826,34 @@ struct packed_ref_iterator {
  */
 static int next_record(struct packed_ref_iterator *iter)
 {
-	const char *p = iter->pos, *eol;
+	const char *p, *eol;
 
 	strbuf_reset(&iter->refname_buf);
 
+	/*
+	 * If iter->pos is contained within a skipped region, jump past
+	 * it.
+	 *
+	 * Note that each skipped region is considered at most once,
+	 * since they are ordered based on their starting position.
+	 */
+	while (iter->jump_cur < iter->jump_nr) {

refs/packed-backend.c struct packed_ref_iterator { · line 850

Quoted 10 lines
refs/packed-backend.c +10 -0
+		struct jump_list_entry *curr = &iter->jump[iter->jump_cur];
+		if (iter->pos < curr->start)
+			break; /* not to the next jump yet */
+
+		iter->jump_cur++;
+		if (iter->pos < curr->end) {
+			iter->pos = curr->end;
+			break;
+		}
+	}

It took me a minute to convince myself that this second "break" was right. If we get to it, we know that iter->pos (the current record we are looking at) is in the current jump region. So it makes sense to advance to curr->end. But might we hit another jump region immediately?

Quoted 4 lines

I guess not, because earlier we would have coalesced the jump regions. So either there is a non-excluded entry there _or_ we would have coalesced the later region into a single larger region. So breaking is the right thing to do.

Exactly. I added a short comment to this effect to hopefully avoid any confusion here.

Quoted 9 lines
Quoted 9 lines
refs/packed-backend.c +9 -0
+	for (pattern = excluded_patterns; *pattern; pattern++) {
+		struct jump_list_entry *e;
+
+		/*
+		 * We can't feed any excludes with globs in them to the
+		 * refs machinery.  It only understands prefix matching.
+		 * We likewise can't even feed the string leading up to
+		 * the first meta-character, as something like "foo[a]"
+		 * should not exclude "foobar" (but the prefix "foo"
Quoted 8 lines

refs/packed-backend.c static struct ref_iterator_vtable packed_ref_iterator_vtable = { · line 1021

Quoted 4 lines
refs/packed-backend.c +4 -0
+		 * would match that and mark it for exclusion).
+		 */
+		if (has_glob_special(*pattern))
+			continue;

OK, and here's where we could split "foo[ac]" into "fooa" and "foob" if we wanted. But I think it is a very good idea to leave that out of this initial patch. :)

Oh, definitely ;-).

Quoted 26 lines
Quoted 10 lines
refs/packed-backend.c +10 -0
+	/*
+	 * As an optimization, merge adjacent entries in the jump list
+	 * to jump forwards as far as possible when entering a skipped
+	 * region.
+	 *
+	 * For example, if we have two skipped regions:
+	 *
+	 *	[[A, B], [B, C]]
+	 *
+	 * we want to combine that into a single entry jumping from A to

refs/packed-backend.c static struct ref_iterator_vtable packed_ref_iterator_vtable = { · line 1059

Quoted 10 lines
refs/packed-backend.c +10 -0
+	 * C.
+	 */
+	last_disjoint = iter->jump;
+
+	for (i = 1, j = 1; i < iter->jump_nr; i++) {
+		struct jump_list_entry *ours = &iter->jump[i];
+
+		if (ours->start == ours->end) {
+			/* ignore empty regions (no matching entries) */
+			continue;

Dropping empty regions makes sense, but our iteration starts at "1" (because the rest of the checks are inherently looking at last_disjoint before deciding if each region is worth keeping). So we'd fail to throw away iter->jump[0] if it is empty, I think.

Quoted 3 lines

That could be fixed here by iterating from 0 and checking for a NULL last_disjoint, but maybe it would be easier to avoid allocating at all in the earlier loop, when we find that start == end?

Yeah, I agree with this. I think Patrick made a similar suggestion in an earlier response, and I decided not to take it since it makes the patch more verbose.

But I think that avoiding the empty region special case is worth it. Thanks, both :-).

Thanks,
Taylor

#119 lore raw

Re: [PATCH v4 12/16] refs/packed-backend.c: ignore complicated hidden refs rules

In reply to Jeff King
Quoted 45 lines · Jeff King
Quoted 8 lines · Taylor Blau

In subsequent commits, we'll teach receive-pack and upload-pack to use the new jump list feature in the packed-refs iterator by ignoring references which are mentioned via its respective hideRefs lists.

However, the packed-ref jump lists cannot handle un-hiding rules (that begin with '!'), or namespace comparisons (that begin with '^'). Detect and avoid these cases by falling back to the normal enumeration without a jump list when such patterns exist.

I'm a fan of punting on such cases to keep things simple and incremental. But the location here seems weird to me:

Quoted 17 lines
refs/packed-backend.c +9 -0
index 80b877e00c..2aeec5c601 100644
@@ -1008,6 +1008,25 @@ static void populate_excluded_jump_list(struct packed_ref_iterator *iter,
 	if (!excluded_patterns)
 		return;
 
+	for (pattern = excluded_patterns; *pattern; pattern++) {
+		/*
+		 * We also can't feed any excludes from hidden refs
+		 * config sections, since later rules may override
+		 * previous ones. For example, with rules "refs/foo" and
+		 * "!refs/foo/bar", we should show "refs/foo/bar" (and
+		 * everything underneath it), but the earlier exclusion
+		 * would cause us to skip all of "refs/foo". We likewise
+		 * don't implement the namespace stripping required for

refs/packed-backend.c static void populate_excluded_jump_list(struct packed_ref_iterator *iter, · line 1029

Quoted 10 lines
refs/packed-backend.c +10 -0
+		 * '^' rules.
+		 *
+		 * Both are possible to do, but complicated, so avoid
+		 * populating the jump list at all if we see either of
+		 * these patterns.
+		 */
+		if (**pattern == '!' || **pattern == '^')
+			return;
+	}
+

This is deep in the packed-refs code, but the magic of "!" and "^" are specific to ref_is_hidden().

Quoted 6 lines

So if I did:

  git for-each-ref --exclude='!refs/heads/foo'

my understanding is that "!" would _not_ have an affect normally, but now it is turning off this optimization.

Yeah, that makes sense -- I agree that it is silly to have a reference with "!" at the beginning, but since it's allowed we should absolutely support it.

Quoted 2 lines

Something along the lines of (you'd want a similar tweak for upload-pack):

Yep. Here's the extra tweak for upload-pack:

--- 8< ---
commit 5a8902731b91a8fc6900586968a79ebc6272e502

Author: Taylor Blau <[email protected]>
Date:   Tue Jul 4 14:21:22 2023 -0400

fixup! upload-pack.c: avoid enumerating hidden refs where possible

upload-pack.c +6 -4
index 3a176a7209..ef2ca36feb 100644
@@ -610,6 +610,7 @@ static int allow_hidden_refs(enum allow_uor allow_uor)
 static void for_each_namespaced_ref_1(each_ref_fn fn,
 				      struct upload_pack_data *data)
 {
+	const char **excludes = NULL;
 	/*
 	 * If `data->allow_uor` allows fetching hidden refs, we need to
 	 * mark all references (including hidden ones), to check in
@@ -619,12 +620,13 @@ static void for_each_namespaced_ref_1(each_ref_fn fn,
 	 * has the OUR_REF bit set or not, so do not need to visit
 	 * hidden references.
 	 */
-	if (allow_hidden_refs(data->allow_uor))
-		for_each_namespaced_ref(NULL, fn, data);
-	else
-		for_each_namespaced_ref(data->hidden_refs.v, fn, data);
+	if (!allow_hidden_refs(data->allow_uor))
+		excludes = hidden_refs_to_excludes(&data->hidden_refs);
+
+	for_each_namespaced_ref(excludes, fn, data);
 }
 
+
 static int is_our_ref(struct object *o, enum allow_uor allow_uor)
 {
 	return o->flags & ((allow_hidden_refs(allow_uor) ? HIDDEN_REF : 0) | OUR_REF);

--- >8 ---

Thanks,
Taylor

#124 lore raw

Re: [PATCH v4 15/16] upload-pack.c: avoid enumerating hidden refs where possible

In reply to Jeff King
Quoted 9 lines · Jeff King

I guess v2 clients don't hit this code at all (they are handled by ls-refs, which comes in the next patch). So that just leaves the case that allowAny is set. By itself the optimization should kick in (good). With allowTip or allowReachable it would not, but that is the "this is stupid" case in which it is OK to fall back to the existing behavior (even though we _could_ enable the optimization). OTOH, it would be easy to check it, as it's just another bit in allow_uor.

I'm OK going either way.

As am I, though I agree that checking for it is not hard. The return value of allow_hidden_refs() as currently written is confusing, so here's an inversion of that function with this suggestion on top:

--- >8 ---

upload-pack.c +5 -3
index ef2ca36feb..5a0fa028c6 100644
@@ -604,7 +604,9 @@ static int get_common_commits(struct upload_pack_data *data,
 
 static int allow_hidden_refs(enum allow_uor allow_uor)
 {
-	return allow_uor & (ALLOW_TIP_SHA1 | ALLOW_REACHABLE_SHA1);
+	if ((allow_uor & ALLOW_ANY_SHA1) == ALLOW_ANY_SHA1)
+		return 1;
+	return !(allow_uor & (ALLOW_TIP_SHA1 | ALLOW_REACHABLE_SHA1));
 }
 
 static void for_each_namespaced_ref_1(each_ref_fn fn,
@@ -620,7 +622,7 @@ static void for_each_namespaced_ref_1(each_ref_fn fn,
 	 * has the OUR_REF bit set or not, so do not need to visit
 	 * hidden references.
 	 */
-	if (!allow_hidden_refs(data->allow_uor))
+	if (allow_hidden_refs(data->allow_uor))
 		excludes = hidden_refs_to_excludes(&data->hidden_refs);
 
 	for_each_namespaced_ref(excludes, fn, data);
@@ -629,7 +631,7 @@ static void for_each_namespaced_ref_1(each_ref_fn fn,
 
 static int is_our_ref(struct object *o, enum allow_uor allow_uor)
 {
-	return o->flags & ((allow_hidden_refs(allow_uor) ? HIDDEN_REF : 0) | OUR_REF);
+	return o->flags & ((allow_hidden_refs(allow_uor) ? 0 : HIDDEN_REF) | OUR_REF);
 }
 
 /*

--- 8< ---

Thanks,
Taylor