Skip to contentSkip to conversation

upload-pack: custom allowed object filters

PATCH Superseded Taylor Blau 4 participants 25 messages Jul 2, 2020 – Jul 23, 2020 Documentation

v1 of 4 Superseded Latest is v4 →

lore
Share & tools
lore
#0 lore raw

cover upload-pack: custom allowed object filters

Hi,

Here are some patches that I sent a while ago [1] as an RFC, but are now ready for an actual review. Not much has changed from the original patches, but a few things have. Namely:

- the 'uploadpack.filter.tree.maxDepth' configuration is new,

- they are rebased forward on top of Christian's various upload-pack clean-ups, so they should apply cleanly to 'master' (in fact, they were rebased to the tip of master, not just Christian's various series), and

- the patches have been substantially cleaned up and now have Signed-off-by trailer

As a reminder, these are the patches that we have been using at GitHub to allow only certain kinds of object filters when serving partial clones. If it is of interest, our configuration looks something like the following:

[uploadpack]
allowAnySHA1InWant = true
allowFilter = true
[uploadpack "filter"]
allow = false
[uploadpack "filter.blob:limit"]
allow = true
[uploadpack "filter.blob:none"]
allow = true
[uploadpack "filter.tree"]
allow = true
maxDepth = 0

Thanks in advance for your review (and thanks also to Peff, Christian, Eric, and Junio for their help and discussion already).

[1]: https://lore.kernel.org/git/[email protected]/

Thanks,
Taylor

Taylor Blau (4):
list_objects_filter_options: introduce 'list_object_filter_config_name' upload-pack.c: allow banning certain object filter(s) upload-pack.c: pass 'struct list_objects_filter_options *' upload-pack.c: introduce 'uploadpack.filter.tree.maxDepth'

 Documentation/config/uploadpack.txt |  22 ++++++
 list-objects-filter-options.c       |  23 ++++++
 list-objects-filter-options.h       |   6 ++
 t/t5616-partial-clone.sh            |  34 +++++++++
 upload-pack.c                       | 104 ++++++++++++++++++++++++++++
 5 files changed, 189 insertions(+)

--
2.27.0.225.g9fa765a71d

#1 lore raw

PATCH 1/4 list_objects_filter_options: introduce 'list_object_filter_config_name'

In a subsequent commit, we will add configuration options that are specific to each kind of object filter, in which case it is handy to have a function that translates between 'enum list_objects_filter_choice' and an appropriate configuration-friendly string.

---
 list-objects-filter-options.c | 23 +++++++++++++++++++++++
 list-objects-filter-options.h |  6 ++++++
 2 files changed, 29 insertions(+)
list-objects-filter-options.c +23 -0
index 3553ad7b0a..92b408c0c8 100644
@@ -15,6 +15,29 @@ static int parse_combine_filter(
 	const char *arg,
 	struct strbuf *errbuf);
 
+const char *list_object_filter_config_name(enum list_objects_filter_choice c)
+{
+	switch (c) {
+	case LOFC_DISABLED:
+		/* we have no name for "no filter at all" */
+		break;
+	case LOFC_BLOB_NONE:
+		return "blob:none";
+	case LOFC_BLOB_LIMIT:
+		return "blob:limit";
+	case LOFC_TREE_DEPTH:
+		return "tree";
+	case LOFC_SPARSE_OID:
+		return "sparse:oid";
+	case LOFC_COMBINE:
+		return "combine";
+	case LOFC__COUNT:
+		/* not a real filter type; just the count of all filters */
+		break;
+	}
+	BUG("list_object_filter_choice_name: invalid argument '%d'", c);
+}
+
 /*
  * Parse value of the argument to the "filter" keyword.
  * On the command line this looks like:
list-objects-filter-options.h +6 -0
index 73fffa4ad7..01767c3c96 100644
@@ -17,6 +17,12 @@ enum list_objects_filter_choice {
 	LOFC__COUNT /* must be last */
 };
 
+/*
+ * Returns a configuration key suitable for describing the given object filter,
+ * e.g.: "blob:none", "combine", etc.
+ */
+const char *list_object_filter_config_name(enum list_objects_filter_choice c);
+
 struct list_objects_filter_options {
 	/*
 	 * 'filter_spec' is the raw argument value given on the command line
Signature · 1 line
2.27.0.225.g9fa765a71d
#2 lore raw

PATCH 2/4 upload-pack.c: allow banning certain object filter(s)

In reply to Taylor Blau

Git clients may ask the server for a partial set of objects, where the set of objects being requested is refined by one or more object filters. Server administrators can configure 'git upload-pack' to allow or ban these filters by setting the 'uploadpack.allowFilter' variable to 'true' or 'false', respectively.

However, administrators using bitmaps may wish to allow certain kinds of object filters, but ban others. Specifically, they may wish to allow object filters that can be optimized by the use of bitmaps, while rejecting other object filters which aren't and represent a perceived performance degradation (as well as an increased load factor on the server).

Allow configuring 'git upload-pack' to support object filters on a case-by-case basis by introducing two new configuration variables:

- 'uploadpack.filter.allow'
- 'uploadpack.filter.<kind>.allow'

where '<kind>' may be one of 'blobNone', 'blobLimit', 'tree', and so on.

Setting the second configuration variable for any valid value of '<kind>' explicitly allows or disallows restricting that kind of object filter.

If a client requests the object filter <kind> and the respective configuration value is not set, 'git upload-pack' will default to the value of 'uploadpack.filter.allow', which itself defaults to 'true' to maintain backwards compatibility. Note that this differs from 'uploadpack.allowfilter', which controls whether or not the 'filter' capability is advertised.

---
 Documentation/config/uploadpack.txt | 16 ++++++
 t/t5616-partial-clone.sh            | 26 ++++++++++
 upload-pack.c                       | 78 +++++++++++++++++++++++++++++
 3 files changed, 120 insertions(+)
Documentation/config/uploadpack.txt +16 -0
index ed1c835695..fd4970306c 100644
@@ -57,6 +57,22 @@ uploadpack.allowFilter::
 	If this option is set, `upload-pack` will support partial
 	clone and partial fetch object filtering.
 
+uploadpack.filter.allow::
+	Provides a default value for unspecified object filters (see: the
+	below configuration variable).
+	Defaults to `true`.
+
+uploadpack.filter.<filter>.allow::
+	Explicitly allow or ban the object filter corresponding to
+	`<filter>`, where `<filter>` may be one of: `blob:none`,
+	`blob:limit`, `tree`, `sparse:oid`, or `combine`. If using
+	combined filters, both `combine` and all of the nested filter
+	kinds must be allowed.  Defaults to `uploadpack.filter.allow`.
++
+Note that the dot between 'filter' and '<filter>' is both non-standard
+and intentional. This is done to avoid a parsing ambiguity when
+specifying this configuration as an argument to Git's top-level `-c`.
+
 uploadpack.allowRefInWant::
 	If this option is set, `upload-pack` will support the `ref-in-want`
 	feature of the protocol version 2 `fetch` command.  This feature
t/t5616-partial-clone.sh +26 -0
index 8a27452a51..5dcd0b5656 100755
@@ -235,6 +235,32 @@ test_expect_success 'implicitly construct combine: filter with repeated flags' '
 	test_cmp unique_types.expected unique_types.actual
 '
 
+test_expect_success 'upload-pack fails banned object filters' '
+	# Test case-insensitivity by intentional use of "blob:None" rather than
+	# "blob:none".
+	test_config -C srv.bare uploadpack.filter.blob:None.allow false &&
+	test_must_fail git clone --no-checkout --filter=blob:none \
+		"file://$(pwd)/srv.bare" pc3 2>err &&
+	test_i18ngrep "filter '\''blob:none'\'' not supported" err
+'
+
+test_expect_success 'upload-pack fails banned combine object filters' '
+	test_config -C srv.bare uploadpack.filter.allow false &&
+	test_config -C srv.bare uploadpack.filter.combine.allow true &&
+	test_config -C srv.bare uploadpack.filter.tree.allow true &&
+	test_config -C srv.bare uploadpack.filter.blob:none.allow false &&
+	test_must_fail git clone --no-checkout --filter=tree:1 \
+		--filter=blob:none "file://$(pwd)/srv.bare" pc3 2>err &&
+	test_i18ngrep "filter '\''blob:none'\'' not supported" err
+'
+
+test_expect_success 'upload-pack fails banned object filters with fallback' '
+	test_config -C srv.bare uploadpack.filter.allow false &&
+	test_must_fail git clone --no-checkout --filter=blob:none \
+		"file://$(pwd)/srv.bare" pc3 2>err &&
+	test_i18ngrep "filter '\''blob:none'\'' not supported" err
+'
+
 test_expect_success 'partial clone fetches blobs pointed to by refs even if normally filtered out' '
 	rm -rf src dst &&
 	git init src &&
upload-pack.c +78 -0

138 lines not on this page · raw message

Signature · 1 line
2.27.0.225.g9fa765a71d
#17 lore raw

PATCH 3/4 upload-pack.c: pass 'struct list_objects_filter_options *'

In reply to Taylor Blau

The 'allows_filter_choice' function used to take an 'enum list_objects_filter_choice', but in a future commit it will be more convenient for it to accept the whole 'struct list_objects_filter_options', for e.g., to inspect the value of '->tree_exclude_depth'.

---
 upload-pack.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
upload-pack.c +3 -3
index a5f56d73cc..a014ae23a9 100644
@@ -990,9 +990,9 @@ static int process_deepen_not(const char *line, struct string_list *deepen_not,
 }
 
 static int allows_filter_choice(struct upload_pack_data *data,
-				enum list_objects_filter_choice c)
+				struct list_objects_filter_options *opts)
 {
-	const char *key = list_object_filter_config_name(c);
+	const char *key = list_object_filter_config_name(opts->choice);
 	struct string_list_item *item = string_list_lookup(&data->allowed_filters,
 							   key);
 	if (item)
@@ -1006,7 +1006,7 @@ static struct list_objects_filter_options *banned_filter(
 {
 	size_t i;
 
-	if (!allows_filter_choice(data, opts->choice))
+	if (!allows_filter_choice(data, opts))
 		return opts;
 
 	if (opts->choice == LOFC_COMBINE)
Signature · 1 line
2.27.0.225.g9fa765a71d
#18 lore raw

PATCH 4/4 upload-pack.c: introduce 'uploadpack.filter.tree.maxDepth'

In reply to Taylor Blau

In b79cf959b2 (upload-pack.c: allow banning certain object filter(s), 2020-02-26), we introduced functionality to disallow certain object filters from being chosen from within 'git upload-pack'. Traditionally, administrators use this functionality to disallow filters that are known to perform slowly, for e.g., those that do not have bitmap-level filtering.

In the past, the '--filter=tree:<n>' was one such filter that does not have bitmap-level filtering support, and so was likely to be banned by administrators.

However, in the previous couple of commits, we introduced bitmap-level filtering for the case when 'n' is equal to '0', i.e., as if we had a '--filter=tree:none' choice.

While it would be sufficient to simply write

  $ git config uploadpack.filter.tree.allow true

(since it would allow all values of 'n'), we would like to be able to allow this filter for certain values of 'n', i.e., those no greater than some pre-specified maximum.

In order to do this, introduce a new configuration key, as follows:

  $ git config uploadpack.filter.tree.maxDepth <m>

where '<m>' specifies the maximum allowed value of 'n' in the filter 'tree:n'. Administrators who wish to allow for only the value '0' can write:

  $ git config uploadpack.filter.tree.allow true
  $ git config uploadpack.filter.tree.maxDepth 0

which allows '--filter=tree:0', but no other values.

Unfortunately, since the tree depth is an unsigned long, we can't use, say, -1 as a sentinel value, and so we must also keep track of "have we set this" as well as "to what value".

---
 Documentation/config/uploadpack.txt |  6 ++++++
 t/t5616-partial-clone.sh            |  8 ++++++++
 upload-pack.c                       | 32 ++++++++++++++++++++++++++---
 3 files changed, 43 insertions(+), 3 deletions(-)
Documentation/config/uploadpack.txt +6 -0
index fd4970306c..3671b62e4c 100644
@@ -73,6 +73,12 @@ Note that the dot between 'filter' and '<filter>' is both non-standard
 and intentional. This is done to avoid a parsing ambiguity when
 specifying this configuration as an argument to Git's top-level `-c`.
 
+uploadpack.filter.tree.maxDepth::
+	Only allow `--filter=tree=<n>` when `n` is no more than the value of
+	`uploadpack.filter.tree.maxDepth`. If set, this also implies
+	`uploadpack.filter.tree.allow=true`, unless this configuration
+	variable had already been set. Has no effect if unset.
+
 uploadpack.allowRefInWant::
 	If this option is set, `upload-pack` will support the `ref-in-want`
 	feature of the protocol version 2 `fetch` command.  This feature
t/t5616-partial-clone.sh +8 -0
index 5dcd0b5656..8781a24cfe 100755
@@ -261,6 +261,14 @@ test_expect_success 'upload-pack fails banned object filters with fallback' '
 	test_i18ngrep "filter '\''blob:none'\'' not supported" err
 '
 
+test_expect_success 'upload-pack limits tree depth filters' '
+	test_config -C srv.bare uploadpack.filter.allow false &&
+	test_config -C srv.bare uploadpack.filter.tree.allow true &&
+	test_config -C srv.bare uploadpack.filter.tree.maxDepth 0 &&
+	test_must_fail ok=sigpipe git clone --no-checkout --filter=tree:1 \
+		"file://$(pwd)/srv.bare" pc3
+'
+
 test_expect_success 'partial clone fetches blobs pointed to by refs even if normally filtered out' '
 	rm -rf src dst &&
 	git init src &&
upload-pack.c +29 -3
index a014ae23a9..8db1745b86 100644
@@ -105,6 +105,7 @@ struct upload_pack_data {
 	unsigned use_include_tag : 1;
 	unsigned allow_filter : 1;
 	unsigned allow_filter_fallback : 1;
+	unsigned long tree_filter_max_depth;
 
 	unsigned done : 1;					/* v2 only */
 	unsigned allow_ref_in_want : 1;				/* v2 only */
@@ -136,6 +137,7 @@ static void upload_pack_data_init(struct upload_pack_data *data)
 	data->extra_edge_obj = extra_edge_obj;
 	data->allowed_filters = allowed_filters;
 	data->allow_filter_fallback = 1;
+	data->tree_filter_max_depth = ULONG_MAX;
 	packet_writer_init(&data->writer, 1);
 
 	data->keepalive = 5;
@@ -995,8 +997,17 @@ static int allows_filter_choice(struct upload_pack_data *data,
 	const char *key = list_object_filter_config_name(opts->choice);
 	struct string_list_item *item = string_list_lookup(&data->allowed_filters,
 							   key);
+	int allowed = -1;
 	if (item)
-		return (intptr_t) item->util;
+		allowed = (intptr_t) item->util;
+
+	if (allowed != 0 &&
+	    opts->choice == LOFC_TREE_DEPTH &&
+	    opts->tree_exclude_depth > data->tree_filter_max_depth)
+		return 0;
+
+	if (allowed > -1)
+		return allowed;
 	return data->allow_filter_fallback;
 }
 
@@ -1022,11 +1033,22 @@ static void die_if_using_banned_filter(struct upload_pack_data *data)
 {
 	struct list_objects_filter_options *banned = banned_filter(data,
 								   &data->filter_options);
+	struct strbuf buf = STRBUF_INIT;
 	if (!banned)
 		return;
 
-	packet_writer_error(&data->writer, _("filter '%s' not supported\n"),
-			    list_object_filter_config_name(banned->choice));
+	strbuf_addf(&buf, _("filter '%s' not supported"),
+		    list_object_filter_config_name(banned->choice));
+	if (banned->choice == LOFC_TREE_DEPTH &&
+	    data->tree_filter_max_depth != ULONG_MAX)
+		strbuf_addf(&buf, _(" (maximum depth: %lu, but got: %lu)"),
+			    data->tree_filter_max_depth,
+			    banned->tree_exclude_depth);
+
+	packet_writer_error(&data->writer, "%s\n", buf.buf);
+
+	strbuf_release(&buf);
+
 	die(_("git upload-pack: banned object filter requested"));
 }
 
@@ -1239,6 +1261,10 @@ static void parse_object_filter_config(const char *var, const char *value,
 				(void *)(intptr_t)git_config_bool(var, value);
 		else
 			data->allow_filter_fallback = git_config_bool(var, value);
+	} else if (!strcmp(spec.buf, "tree") && !strcmp(key, "maxdepth")) {
+		string_list_insert(&data->allowed_filters, "tree")->util
+			= (void *) (intptr_t) 1;
+		data->tree_filter_max_depth = git_config_ulong(var, value);
 	}
 
 	strbuf_release(&spec);
Signature · 1 line
2.27.0.225.g9fa765a71d
#20 lore raw
In reply to Taylor Blau
Quoted 4 lines · Taylor Blau

[uploadpack]
allowAnySHA1InWant = true
allowFilter = true
[uploadpack "filter"]

Quoted 8 lines

allow = false
[uploadpack "filter.blob:limit"]
allow = true
[uploadpack "filter.blob:none"]
allow = true
[uploadpack "filter.tree"]
allow = true
maxDepth = 0

I thought the previous discussion landed on:

uploadpackfilter.blob:none.allow

etc, to avoid the confusing appearance of four-level keys (and the weird case sensitivity implications that would cause).

-Peff

#3 lore raw

Re: [PATCH 2/4] upload-pack.c: allow banning certain object filter(s)

In reply to Taylor Blau

upload-pack.c static int find_symref(const char *refname, const struct object_id *oid, · line 1230

Quoted 11 lines · Taylor Blau
upload-pack.c +11 -0
+static void parse_object_filter_config(const char *var, const char *value,
+				       struct upload_pack_data *data)
+{
+	struct strbuf spec = STRBUF_INIT;
+	const char *sub, *key;
+	size_t sub_len;
+
+	if (parse_config_key(var, "uploadpack", &sub, &sub_len, &key))
+		return;
+	if (!sub || !skip_prefix(sub, "filter.", &sub))
+		return;

Just while I'm thinking about the config name and case-sensitivity (from the cover letter): if we did want to use this scheme, then skip_iprefix() would make this behave more like a regular part of the section name.

But I'd prefer to just do away with it by using a scheme that doesn't have the extra layer of dots.

upload-pack.c static int find_symref(const char *refname, const struct object_id *oid, · line 1234

Quoted 3 lines
upload-pack.c +3 -0
+	if (sub != key)
+		strbuf_add(&spec, sub, key - sub - 1);
+	strbuf_tolower(&spec);

On the flip side, I'd actually consider _not_ matching the filter name case-insensitively. We don't do so elsewhere (e.g., "git rev-list --filter=BLOB:NONE" will complain).

-Peff

#5 lore raw

Re: [PATCH 2/4] upload-pack.c: allow banning certain object filter(s)

In reply to Taylor Blau
Quoted 23 lines · Taylor Blau
t/t5616-partial-clone.sh +15 -0
index 8a27452a51..5dcd0b5656 100755
@@ -235,6 +235,32 @@ test_expect_success 'implicitly construct combine: filter with repeated flags' '
 	test_cmp unique_types.expected unique_types.actual
 '
 
+test_expect_success 'upload-pack fails banned object filters' '
+	# Test case-insensitivity by intentional use of "blob:None" rather than
+	# "blob:none".
+	test_config -C srv.bare uploadpack.filter.blob:None.allow false &&
+	test_must_fail git clone --no-checkout --filter=blob:none \
+		"file://$(pwd)/srv.bare" pc3 2>err &&
+	test_i18ngrep "filter '\''blob:none'\'' not supported" err
+'
+
+test_expect_success 'upload-pack fails banned combine object filters' '
+	test_config -C srv.bare uploadpack.filter.allow false &&
+	test_config -C srv.bare uploadpack.filter.combine.allow true &&
+	test_config -C srv.bare uploadpack.filter.tree.allow true &&
+	test_config -C srv.bare uploadpack.filter.blob:none.allow false &&
+	test_must_fail git clone --no-checkout --filter=tree:1 \

t/t5616-partial-clone.sh test_expect_success 'implicitly construct combine: filter with repeated flags' ' · line 262

Quoted 10 lines
t/t5616-partial-clone.sh +10 -0
+		--filter=blob:none "file://$(pwd)/srv.bare" pc3 2>err &&
+	test_i18ngrep "filter '\''blob:none'\'' not supported" err
+'
+
+test_expect_success 'upload-pack fails banned object filters with fallback' '
+	test_config -C srv.bare uploadpack.filter.allow false &&
+	test_must_fail git clone --no-checkout --filter=blob:none \
+		"file://$(pwd)/srv.bare" pc3 2>err &&
+	test_i18ngrep "filter '\''blob:none'\'' not supported" err
+'

These three tests are very flaky: 'git upload-pack' can error out while clone is still sending packets (usually the 'done' line), resulting in SIGPIPE and frequent CI failures. Running this test script with '-r 1,2,17-19 --stress' tends to fail in a couple of seconds.

Using 'test_must_fail ok=sigpipe', as you did in the test in the last patch, avoids the test failure caused by SIGPIPE, of course, but, unfortunately, all three tests remain flaky, because the expected error message sometimes doesn't make it to 'git clone's stderr, e.g.:

  expecting success of 5616.19 'upload-pack fails banned object filters with fallback': 
          test_config -C srv.bare uploadpack.filter.allow false &&
          test_must_fail ok=sigpipe git clone --no-checkout --filter=blob:none \
                  "file://$(pwd)/srv.bare" pc3 2>err &&
          test_i18ngrep "filter 'blob:none' not supported" err

+ test_config -C srv.bare uploadpack.filter.allow false
+ pwd
+ test_must_fail ok=sigpipe git clone --no-checkout --filter=blob:none file:///home/szeder/src/git/t/trash directory.t5616-partial-clone.stress-2/srv.bare pc3
+ test_i18ngrep filter 'blob:none' not supported err
error: 'grep filter 'blob:none' not supported err' didn't find a match in:
Cloning into 'pc3'...
fatal: git upload-pack: banned object filter requested
error: last command exited with $?=1
not ok 19 - upload-pack fails banned object filters with fallback

Once upon a time I had a PoC patch to deal with 'git upload-pack' aborting while 'git fetch' is still send_request()-ing, by catching the write error to the closed connection and trying read any pending ERR packets; Christian cleaned it up and submitted it with a proper commit message in

https://public-inbox.org/git/[email protected]/

but it haven't been picked up yet. Disappointingly, that patch doesn't solve these issues... I haven't looked what's going on (perhaps 'git clone' does something differently than 'git fetch'? no idea)

#19 lore raw

Re: [PATCH 4/4] upload-pack.c: introduce 'uploadpack.filter.tree.maxDepth'

In reply to Taylor Blau
Quoted 5 lines · Taylor Blau
t/t5616-partial-clone.sh +0 -0
index 5dcd0b5656..8781a24cfe 100755

t/t5616-partial-clone.sh test_expect_success 'upload-pack fails banned object filters with fallback' ' · line 270

Quoted 10 lines
t/t5616-partial-clone.sh +7 -0
@@ -261,6 +261,14 @@ test_expect_success 'upload-pack fails banned object filters with fallback' '
 	test_i18ngrep "filter '\''blob:none'\'' not supported" err
 '
 
+test_expect_success 'upload-pack limits tree depth filters' '
+	test_config -C srv.bare uploadpack.filter.allow false &&
+	test_config -C srv.bare uploadpack.filter.tree.allow true &&
+	test_config -C srv.bare uploadpack.filter.tree.maxDepth 0 &&
+	test_must_fail ok=sigpipe git clone --no-checkout --filter=tree:1 \
+		"file://$(pwd)/srv.bare" pc3
+'

Unlike in the other three tests added earlier in this series, here you do use 'test_must_fail ok=sigpipe', but you don't check that the command died with the right error message. Saving stderr and adding

test_i18ngrep "filter '\''tree'\'' not supported (maximum depth: 0, but got: 1)" err

makes this test flaky, too, like the other three:

expecting success of 5616.20 'upload-pack limits tree depth filters':

          test_config -C srv.bare uploadpack.filter.allow false &&
          test_config -C srv.bare uploadpack.filter.tree.allow true &&
          test_config -C srv.bare uploadpack.filter.tree.maxDepth 0 &&
          test_must_fail ok=sigpipe git clone --no-checkout --filter=tree:1 \
                  "file://$(pwd)/srv.bare" pc3 2>err &&
          test_i18ngrep "filter 'tree' not supported (maximum depth: 0, but got: 1)" err

+ test_config -C srv.bare uploadpack.filter.allow false
+ test_config -C srv.bare uploadpack.filter.tree.allow true
+ test_config -C srv.bare uploadpack.filter.tree.maxDepth 0
+ pwd
+ test_must_fail ok=sigpipe git clone --no-checkout --filter=tree:1 file:///home/szeder/src/git/t/trash directory.t5616-partial-clone.stress-4/srv.bare pc3
+ test_i18ngrep filter 'tree' not supported (maximum depth: 0, but got: 1) err
error: 'grep filter 'tree' not supported (maximum depth: 0, but got: 1) err' didn't find a match in:
Cloning into 'pc3'...
fatal: git upload-pack: banned object filter requested
error: last command exited with $?=1
not ok 20 - upload-pack limits tree depth filters

#6 lore raw

Re: [PATCH 2/4] upload-pack.c: allow banning certain object filter(s)

In reply to SZEDER Gábor
Quoted 6 lines · SZEDER Gábor

Once upon a time I had a PoC patch to deal with 'git upload-pack' aborting while 'git fetch' is still send_request()-ing, by catching the write error to the closed connection and trying read any pending ERR packets; Christian cleaned it up and submitted it with a proper commit message in

Quoted 6 lines

https://public-inbox.org/git/[email protected]/

but it haven't been picked up yet. Disappointingly, that patch doesn't solve these issues... I haven't looked what's going on (perhaps 'git clone' does something differently than 'git fetch'? no idea)

I suspect it is that fetch ignores SIGPIPE, but clone does not. So even when we see a 141 exit code from fetch, it is probably generated synthetically from exit(141) after we saw EPIPE. And your patch works there because we have a chance to pump the read-side of the pipe, whereas in git-clone we die immediately via the signal.

Probably git-clone should ignore SIGPIPE during the network transfer portion of the process for the same reasons given in 143588949c (fetch:
ignore SIGPIPE during network operation, 2019-03-03).

-Peff

#4 lore raw

Re: [PATCH 2/4] upload-pack.c: allow banning certain object filter(s)

In reply to Jeff King
Quoted 19 lines · Jeff King

upload-pack.c static int find_symref(const char *refname, const struct object_id *oid, · line 1230

Quoted 11 lines · Taylor Blau
upload-pack.c +11 -0
+static void parse_object_filter_config(const char *var, const char *value,
+				       struct upload_pack_data *data)
+{
+	struct strbuf spec = STRBUF_INIT;
+	const char *sub, *key;
+	size_t sub_len;
+
+	if (parse_config_key(var, "uploadpack", &sub, &sub_len, &key))
+		return;
+	if (!sub || !skip_prefix(sub, "filter.", &sub))
+		return;

Just while I'm thinking about the config name and case-sensitivity (from the cover letter): if we did want to use this scheme, then skip_iprefix() would make this behave more like a regular part of the section name.

Quoted 2 lines

But I'd prefer to just do away with it by using a scheme that doesn't have the extra layer of dots.

Yeah. I definitely flip-flopped on this when preparing this for the list. I still feel like 'uploadpackfilter' is gross, so I was hoping to send it with 'uploadpack.filter' (which reads nicely, but forces us to write some gross code), but both options are frustrating to me for different reasons.

Playing around with it more, though, I think that uploadpackfilter is our best bet. I'd hate to introduce a string manipulation bug by munging the reuslt of 'parse_config_key()', which is so clearly designed for something like 'uploadpackfilter[.<filter>].allow'.

Quoted 7 lines

upload-pack.c static int find_symref(const char *refname, const struct object_id *oid, · line 1234

Quoted 3 lines
upload-pack.c +3 -0
+	if (sub != key)
+		strbuf_add(&spec, sub, key - sub - 1);
+	strbuf_tolower(&spec);

On the flip side, I'd actually consider _not_ matching the filter name case-insensitively. We don't do so elsewhere (e.g., "git rev-list --filter=BLOB:NONE" will complain).

I dropped the case-insensitive matching in the latest revision. I don't think that we need to be overly accommodating here.

Quoted 1 line

-Peff

Thanks,
Taylor

#7 lore raw

Re: [PATCH 2/4] upload-pack.c: allow banning certain object filter(s)

In reply to Jeff King
Quoted 15 lines · Jeff King
Quoted 6 lines · SZEDER Gábor

Once upon a time I had a PoC patch to deal with 'git upload-pack' aborting while 'git fetch' is still send_request()-ing, by catching the write error to the closed connection and trying read any pending ERR packets; Christian cleaned it up and submitted it with a proper commit message in

Quoted 6 lines

https://public-inbox.org/git/[email protected]/

but it haven't been picked up yet. Disappointingly, that patch doesn't solve these issues... I haven't looked what's going on (perhaps 'git clone' does something differently than 'git fetch'? no idea)

Quoted 5 lines

I suspect it is that fetch ignores SIGPIPE, but clone does not. So even when we see a 141 exit code from fetch, it is probably generated synthetically from exit(141) after we saw EPIPE. And your patch works there because we have a chance to pump the read-side of the pipe, whereas in git-clone we die immediately via the signal.

Heh. I was hoping to be rid of those errors with Christian's patches, but it sounds like the problem is coming from outside of 'upload-pack' and instead in 'clone'.

That reasoning seems sound to me, but I'd rather not touch clone in this patch series if I don't have to. What I'd rather do is something like:

- Introduce this patch series with the 'test_must_fail ok=sigpipe', and no error checking.

- Modify clone to swallow these errors and eat a packet or two.

- Then, drop the 'ok=sigpipe' from t5616 after 'git clone' is a little bit smarter here.

Maybe more steps than is strictly necessary, but I think it keeps the scope of the review on this series reasonable, which is a tradeoff that I'm willing to make.

Quoted 5 lines

Probably git-clone should ignore SIGPIPE during the network transfer portion of the process for the same reasons given in 143588949c (fetch:
ignore SIGPIPE during network operation, 2019-03-03).

-Peff

Thanks,
Taylor

#21 lore raw
In reply to Jeff King
Quoted 15 lines · Jeff King
Quoted 4 lines · Taylor Blau

[uploadpack]
allowAnySHA1InWant = true
allowFilter = true
[uploadpack "filter"]

Quoted 8 lines

allow = false
[uploadpack "filter.blob:limit"]
allow = true
[uploadpack "filter.blob:none"]
allow = true
[uploadpack "filter.tree"]
allow = true
maxDepth = 0

Quoted 6 lines

I thought the previous discussion landed on:

uploadpackfilter.blob:none.allow

etc, to avoid the confusing appearance of four-level keys (and the weird case sensitivity implications that would cause).

Yup, I agree. I added a little bit more detail to this response in another email that I posted shortly above in this thread.

Rest assured that I haven't forgotten about this series, I've just been busy and not feeling too pressured to get back to it since we're well into the release-candidate phase, and shouldn't be worrying about this until after 2.28.0.

If you're interested, I have what I will eventually call 'v2' of this series at:

https://github.com/ttaylorr/git/tree/tb/upload-pack-filter-config

Quoted 1 line

-Peff

Thanks,
Taylor

#8 lore raw

Re: [PATCH 2/4] upload-pack.c: allow banning certain object filter(s)

In reply to Taylor Blau
Quoted 3 lines · Taylor Blau

Heh. I was hoping to be rid of those errors with Christian's patches, but it sounds like the problem is coming from outside of 'upload-pack' and instead in 'clone'.

Yes, it's definitely on the client side.

Quoted 6 lines

That reasoning seems sound to me, but I'd rather not touch clone in this patch series if I don't have to. What I'd rather do is something like:

- Introduce this patch series with the 'test_must_fail ok=sigpipe', and no error checking.

Quoted 8 lines

- Modify clone to swallow these errors and eat a packet or two.

- Then, drop the 'ok=sigpipe' from t5616 after 'git clone' is a little bit smarter here.

Maybe more steps than is strictly necessary, but I think it keeps the scope of the review on this series reasonable, which is a tradeoff that I'm willing to make.

I think that's reasonable.

It doesn't look like the "pump the read side after EPIPE" patch is even on the "seen" branch yet, and you'd need to further extend it to clone. It's definitely a big enough change that it's better not to lump it in with this series.

-Peff

#22 lore raw
In reply to Taylor Blau
Quoted 13 lines · Taylor Blau

Taylor Blau (4):
list_objects_filter_options: introduce 'list_object_filter_config_name' upload-pack.c: allow banning certain object filter(s) upload-pack.c: pass 'struct list_objects_filter_options *' upload-pack.c: introduce 'uploadpack.filter.tree.maxDepth'

 Documentation/config/uploadpack.txt |  22 ++++++
 list-objects-filter-options.c       |  23 ++++++
 list-objects-filter-options.h       |   6 ++
 t/t5616-partial-clone.sh            |  34 +++++++++
 upload-pack.c                       | 104 ++++++++++++++++++++++++++++
 5 files changed, 189 insertions(+)
Quoted 2 lines

--
2.27.0.225.g9fa765a71d

With this series (I do not know which one of them is the culprit) in the 'seen' branch, we seem to consistently get a segfault while running t5616 on macOS clang build [*1*]

[Footnote]

*1* https://travis-ci.org/github/git/git/jobs/710504820 has the
topic at the tip of 'seen' that fails. Without that merge,
https://travis-ci.org/github/git/git/jobs/710342598 seems to pass on all archs.

#23 lore raw
Quoted 15 lines · Junio C Hamano
Quoted 13 lines · Taylor Blau

Taylor Blau (4):
list_objects_filter_options: introduce 'list_object_filter_config_name' upload-pack.c: allow banning certain object filter(s) upload-pack.c: pass 'struct list_objects_filter_options *' upload-pack.c: introduce 'uploadpack.filter.tree.maxDepth'

 Documentation/config/uploadpack.txt |  22 ++++++
 list-objects-filter-options.c       |  23 ++++++
 list-objects-filter-options.h       |   6 ++
 t/t5616-partial-clone.sh            |  34 +++++++++
 upload-pack.c                       | 104 ++++++++++++++++++++++++++++
 5 files changed, 189 insertions(+)
Quoted 6 lines
Quoted 2 lines

--
2.27.0.225.g9fa765a71d

With this series (I do not know which one of them is the culprit) in the 'seen' branch, we seem to consistently get a segfault while running t5616 on macOS clang build [*1*]

Aye. Am I reading that correctly that it's git clone dying with a SIGPIPE instead of a segfault? If so, this is what Szeder pointed out a little lower in the thread (tl;dr is that 'git clone' is not resilient to this whereas 'git fetch' is, and so we still need an ok=sigpipe somewhere in t5616).

The new version that I'm preparing has these appropriately, so feel free to discard this until I send a new version your way...

Quoted 6 lines

[Footnote]

*1* https://travis-ci.org/github/git/git/jobs/710504820 has the
topic at the tip of 'seen' that fails. Without that merge,
https://travis-ci.org/github/git/git/jobs/710342598 seems to pass on all archs.

Thanks,
Taylor

#24 lore raw
Quoted 8 lines · Taylor Blau

Aye. Am I reading that correctly that it's git clone dying with a SIGPIPE instead of a segfault? If so, this is what Szeder pointed out a little lower in the thread (tl;dr is that 'git clone' is not resilient to this whereas 'git fetch' is, and so we still need an ok=sigpipe somewhere in t5616).

The new version that I'm preparing has these appropriately, so feel free to discard this until I send a new version your way...

If it is known and under control, I have nothing to worry about ;-)

Thanks.

#9 lore raw

Re: [PATCH 2/4] upload-pack.c: allow banning certain object filter(s)

In reply to Taylor Blau
Quoted 4 lines · Taylor Blau

What I'd rather do is something like:

- Introduce this patch series with the 'test_must_fail ok=sigpipe', and no error checking.

die_if_using_banned_filter() shows two error messages: a "fancy" one is sent to the client in the ERR packet, including which particular filter is not supported/allowed, and a simple

  die(_("git upload-pack: banned object filter requested"));

If this die() were to show the same fancy error message as in the ERR packet, then it would always make it to 'git clone's stderr in the tests, so the tests could reliably check that 'git upload-pack' died for the expected reason.

Quoted 17 lines

- Modify clone to swallow these errors and eat a packet or two.

- Then, drop the 'ok=sigpipe' from t5616 after 'git clone' is a little bit smarter here.

Maybe more steps than is strictly necessary, but I think it keeps the scope of the review on this series reasonable, which is a tradeoff that I'm willing to make.

Quoted 5 lines

Probably git-clone should ignore SIGPIPE during the network transfer portion of the process for the same reasons given in 143588949c (fetch:
ignore SIGPIPE during network operation, 2019-03-03).

-Peff

Thanks,
Taylor

#13 lore raw

Re: [PATCH 2/4] upload-pack.c: allow banning certain object filter(s)

In reply to Taylor Blau

upload-pack.c static int process_deepen_not(const char *line, struct string_list *deepen_not, · line 1028

Quoted 8 lines · Taylor Blau
upload-pack.c +8 -0
+static void die_if_using_banned_filter(struct upload_pack_data *data)
+{
+	struct list_objects_filter_options *banned = banned_filter(data,
+								   &data->filter_options);
+	if (!banned)
+		return;
+
+	packet_writer_error(&data->writer, _("filter '%s' not supported\n"),

Should we really translate the content of the ERR packet?

Quoted 7 lines
upload-pack.c +4 -0
+			    list_object_filter_config_name(banned->choice));
+	die(_("git upload-pack: banned object filter requested"));
+}
+
 static void receive_needs(struct upload_pack_data *data,
 			  struct packet_reader *reader)
 {
#10 lore raw

Re: [PATCH 2/4] upload-pack.c: allow banning certain object filter(s)

In reply to SZEDER Gábor
Quoted 10 lines · SZEDER Gábor
Quoted 4 lines · Taylor Blau

What I'd rather do is something like:

- Introduce this patch series with the 'test_must_fail ok=sigpipe', and no error checking.

die_if_using_banned_filter() shows two error messages: a "fancy" one is sent to the client in the ERR packet, including which particular filter is not supported/allowed, and a simple

Quoted 6 lines
  die(_("git upload-pack: banned object filter requested"));

If this die() were to show the same fancy error message as in the ERR packet, then it would always make it to 'git clone's stderr in the tests, so the tests could reliably check that 'git upload-pack' died for the expected reason.

Beautiful idea. I changed this in my fork, and I'll send it to this thread after 2.28 is out, since I don't want to create a distraction in the meantime.

Quoted 9 lines
Quoted 9 lines

- Modify clone to swallow these errors and eat a packet or two.

- Then, drop the 'ok=sigpipe' from t5616 after 'git clone' is a little bit smarter here.

Maybe more steps than is strictly necessary, but I think it keeps the scope of the review on this series reasonable, which is a tradeoff that I'm willing to make.

Quoted 8 lines
Quoted 8 lines
Quoted 5 lines

Probably git-clone should ignore SIGPIPE during the network transfer portion of the process for the same reasons given in 143588949c (fetch:
ignore SIGPIPE during network operation, 2019-03-03).

-Peff

Thanks,
Taylor

Thanks,
Taylor

#14 lore raw

Re: [PATCH 2/4] upload-pack.c: allow banning certain object filter(s)

In reply to SZEDER Gábor
Quoted 11 lines · SZEDER Gábor

upload-pack.c static int process_deepen_not(const char *line, struct string_list *deepen_not, · line 1028

Quoted 8 lines · Taylor Blau
upload-pack.c +8 -0
+static void die_if_using_banned_filter(struct upload_pack_data *data)
+{
+	struct list_objects_filter_options *banned = banned_filter(data,
+								   &data->filter_options);
+	if (!banned)
+		return;
+
+	packet_writer_error(&data->writer, _("filter '%s' not supported\n"),

Should we really translate the content of the ERR packet?

s/ERR packet/dying message after your other piece of advice, but I think so. This is eventually going to make its way to users, so translating it seems to make sense to me.

I guess they would get it in the localization of the server, not the client, but that doesn't sway me away from this, either.

Quoted 7 lines

upload-pack.c static int process_deepen_not(const char *line, struct string_list *deepen_not, · line 1035

Quoted 7 lines
upload-pack.c +4 -0
+			    list_object_filter_config_name(banned->choice));
+	die(_("git upload-pack: banned object filter requested"));
+}
+
 static void receive_needs(struct upload_pack_data *data,
 			  struct packet_reader *reader)
 {

Thanks,
Taylor

#11 lore raw

Re: [PATCH 2/4] upload-pack.c: allow banning certain object filter(s)

In reply to Taylor Blau
Quoted 8 lines · Taylor Blau
Quoted 4 lines

If this die() were to show the same fancy error message as in the ERR packet, then it would always make it to 'git clone's stderr in the tests, so the tests could reliably check that 'git upload-pack' died for the expected reason.

Beautiful idea. I changed this in my fork, and I'll send it to this thread after 2.28 is out, since I don't want to create a distraction in the meantime.

Thanks.

As long as everybody understands that "distraction" will immediately gets backburnered when a regression relevant to the upcoming release is found, however, I think performing work as usual is a good thing. After all, we often find glitches in the current code by trying to build on it, in addition to using it.

#12 lore raw

Re: [PATCH 2/4] upload-pack.c: allow banning certain object filter(s)

Quoted 11 lines · Junio C Hamano
Quoted 8 lines · Taylor Blau
Quoted 4 lines

If this die() were to show the same fancy error message as in the ERR packet, then it would always make it to 'git clone's stderr in the tests, so the tests could reliably check that 'git upload-pack' died for the expected reason.

Beautiful idea. I changed this in my fork, and I'll send it to this thread after 2.28 is out, since I don't want to create a distraction in the meantime.

Quoted 7 lines

Thanks.

As long as everybody understands that "distraction" will immediately gets backburnered when a regression relevant to the upcoming release is found, however, I think performing work as usual is a good thing. After all, we often find glitches in the current code by trying to build on it, in addition to using it.

Fair enough ;-). I sent v2 a few minutes ago, but forgot to attach it to this thread as a reply. So, I suppose we can pick up the review there, instead.

Thanks,
Taylor

#15 lore raw

Re: [PATCH 2/4] upload-pack.c: allow banning certain object filter(s)

In reply to Taylor Blau
Quoted 13 lines · Taylor Blau
Quoted 11 lines · SZEDER Gábor

upload-pack.c static int process_deepen_not(const char *line, struct string_list *deepen_not, · line 1028

Quoted 8 lines · Taylor Blau
upload-pack.c +8 -0
+static void die_if_using_banned_filter(struct upload_pack_data *data)
+{
+	struct list_objects_filter_options *banned = banned_filter(data,
+								   &data->filter_options);
+	if (!banned)
+		return;
+
+	packet_writer_error(&data->writer, _("filter '%s' not supported\n"),

Should we really translate the content of the ERR packet?

Quoted 6 lines

s/ERR packet/dying message after your other piece of advice, but I think so. This is eventually going to make its way to users, so translating it seems to make sense to me.

I guess they would get it in the localization of the server, not the client, but that doesn't sway me away from this, either.

Please note that the contents of other ERR packets are not translated, either.

$ git grep -A1 -h packet_writer_error origin/seen:upload-pack.c
			packet_writer_error(&data->writer,
					    "upload-pack: not our ref %s",
--
			packet_writer_error(&data->writer,
					    "upload-pack: not our ref %s",
--
			packet_writer_error(writer,
					    "upload-pack: not our ref %s",
--
			packet_writer_error(writer, "unknown ref %s", arg);
			die("unknown ref %s", arg);
Quoted 10 lines
Quoted 7 lines

upload-pack.c static int process_deepen_not(const char *line, struct string_list *deepen_not, · line 1035

Quoted 7 lines
upload-pack.c +4 -0
+			    list_object_filter_config_name(banned->choice));
+	die(_("git upload-pack: banned object filter requested"));
+}
+
 static void receive_needs(struct upload_pack_data *data,
 			  struct packet_reader *reader)
 {

Thanks,
Taylor

#16 lore raw

Re: [PATCH 2/4] upload-pack.c: allow banning certain object filter(s)

Quoted 24 lines · SZEDER Gábor
Quoted 13 lines · Taylor Blau
Quoted 11 lines · SZEDER Gábor

upload-pack.c static int process_deepen_not(const char *line, struct string_list *deepen_not, · line 1028

Quoted 8 lines · Taylor Blau
upload-pack.c +8 -0
+static void die_if_using_banned_filter(struct upload_pack_data *data)
+{
+	struct list_objects_filter_options *banned = banned_filter(data,
+								   &data->filter_options);
+	if (!banned)
+		return;
+
+	packet_writer_error(&data->writer, _("filter '%s' not supported\n"),

Should we really translate the content of the ERR packet?

Quoted 6 lines

s/ERR packet/dying message after your other piece of advice, but I think so. This is eventually going to make its way to users, so translating it seems to make sense to me.

I guess they would get it in the localization of the server, not the client, but that doesn't sway me away from this, either.

Please note that the contents of other ERR packets are not translated, either.

Quoted 12 lines
$ git grep -A1 -h packet_writer_error origin/seen:upload-pack.c
			packet_writer_error(&data->writer,
					    "upload-pack: not our ref %s",
--
			packet_writer_error(&data->writer,
					    "upload-pack: not our ref %s",
--
			packet_writer_error(writer,
					    "upload-pack: not our ref %s",
--
			packet_writer_error(writer, "unknown ref %s", arg);
			die("unknown ref %s", arg);

Fair enough... if there are other changes to make I'll drop the translation, otherwise this should be easy enough to apply when queuing.

Junio, if you'd rather I reroll the series again, too, that's fine as well.

Quoted 11 lines
Quoted 11 lines
Quoted 8 lines

upload-pack.c static int process_deepen_not(const char *line, struct string_list *deepen_not, · line 1035

Quoted 8 lines
upload-pack.c +4 -0
+			    list_object_filter_config_name(banned->choice));
+	die(_("git upload-pack: banned object filter requested"));
+}
+
 static void receive_needs(struct upload_pack_data *data,
 			  struct packet_reader *reader)
 {

Thanks,
Taylor

Thanks,
Taylor