Skip to contentSkip to conversation

upload-pack.c: treat want-ref relative to namespace

PATCH Superseded Kim Altintop 6 participants 35 messages Jul 30, 2021 – Aug 16, 2021 Documentation

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 upload-pack.c: treat want-ref relative to namespace

When 'upload-pack' runs within the context of a git namespace, treat any 'want-ref' lines the client sends as relative to that namespace.

Also check if the wanted ref is hidden via 'hideRefs', and respond with an error otherwise. It was previously possible to request any ref, but note that this is still the case unless 'hideRefs' is in effect.

---

Please excuse my newbie ness.

 t/t5703-upload-pack-ref-in-want.sh | 77 ++++++++++++++++++++++++++++++
 upload-pack.c                      | 15 +++---
 2 files changed, 86 insertions(+), 6 deletions(-)
t/t5703-upload-pack-ref-in-want.sh +77 -0
index e9e471621d..9fb16848bc 100755
@@ -298,6 +298,83 @@ test_expect_success 'fetching with wildcard that matches multiple refs' '
 	grep "want-ref refs/heads/o/bar" log
 '
 
+REPO="$(pwd)/repo-ns"
+
+test_expect_success 'setup namespaced repo' '
+	(
+		git init -b main "$REPO" &&
+		cd "$REPO" &&
+		test_commit a &&
+		test_commit b &&
+		git checkout a &&
+		test_commit c &&
+		git checkout a &&
+		test_commit d &&
+		git update-ref refs/heads/ns-no b &&
+		git update-ref refs/namespaces/ns/refs/heads/ns-yes c &&
+		git update-ref refs/namespaces/ns/refs/heads/hidden d
+	) &&
+    git -C "$REPO" config uploadpack.allowRefInWant true &&
+    git -C "$REPO" config transfer.hideRefs refs/heads/hidden
+'
+
+test_expect_success 'want-ref with namespaces' '
+	oid=$(git -C "$REPO" rev-parse c) &&
+	cat >expected_refs <<-EOF &&
+	$oid refs/heads/ns-yes
+	EOF
+	>expected_commits &&
+
+	oid=$(git -C "$REPO" rev-parse c) &&
+	test-tool pkt-line pack >in <<-EOF &&
+	$(write_command fetch)
+	0001
+	no-progress
+	want-ref refs/heads/ns-yes
+	have $oid
+	done
+	0000
+	EOF
+
+	GIT_NAMESPACE=ns && export GIT_NAMESPACE &&
+	test-tool -C "$REPO" serve-v2 --stateless-rpc >out <in &&
+	check_output
+'
+
+test_expect_success 'want-ref outside namespace' '
+	oid=$(git -C "$REPO" rev-parse c) &&
+	test-tool pkt-line pack >in <<-EOF &&
+	$(write_command fetch)
+	0001
+	no-progress
+	want-ref refs/heads/ns-no
+	have $oid
+	done
+	0000
+	EOF
+
+	GIT_NAMESPACE=ns && export GIT_NAMESPACE &&
+	test_must_fail test-tool -C "$REPO" serve-v2 --stateless-rpc >out <in &&
+	grep "unknown ref" out
+'
+
+test_expect_success 'hideRefs with namespaces' '
+	oid=$(git -C "$REPO" rev-parse c) &&
+	test-tool pkt-line pack >in <<-EOF &&
+	$(write_command fetch)
+	0001
+	no-progress
+	want-ref refs/heads/hidden
+	have $oid
+	done
+	0000
+	EOF
+
+	GIT_NAMESPACE=ns && export GIT_NAMESPACE &&
+	test_must_fail test-tool -C "$REPO" serve-v2 --stateless-rpc >out <in &&
+	grep "unknown ref" out
+'
+
 . "$TEST_DIRECTORY"/lib-httpd.sh
 start_httpd
 
upload-pack.c +9 -6
index 297b76fcb4..008ac75125 100644
@@ -1417,21 +1417,24 @@ static int parse_want_ref(struct packet_writer *writer, const char *line,
 			  struct string_list *wanted_refs,
 			  struct object_array *want_obj)
 {
-	const char *arg;
+	const char *refname_nons;
 	if (skip_prefix(line, "want-ref ", &arg)) {
 		struct object_id oid;
 		struct string_list_item *item;
 		struct object *o;
+		struct strbuf refname = STRBUF_INIT;
 
-		if (read_ref(arg, &oid)) {
-			packet_writer_error(writer, "unknown ref %s", arg);
-			die("unknown ref %s", arg);
+		strbuf_addf(&refname, "%s%s", get_git_namespace(), refname_nons);
+		if (ref_is_hidden(refname_nons, refname.buf) ||
+		    read_ref(refname.buf, &oid)) {
+			packet_writer_error(writer, "unknown ref %s", refname_nons);
+			die("unknown ref %s", refname.buf);
 		}
 
-		item = string_list_append(wanted_refs, arg);
+		item = string_list_append(wanted_refs, refname_nons);
 		item->util = oiddup(&oid);
 
-		o = parse_object_or_die(&oid, arg);
+		o = parse_object_or_die(&oid, refname);
 		if (!(o->flags & WANTED)) {
 			o->flags |= WANTED;
 			add_object_array(o, NULL, want_obj);

--
2.32.0

#1 lore raw
Quoted 1 line

Please excuse my newbie ness.

Oops I'm sorry. send-email remembered this from a dry run.

Jonathan: I took the test code from your original patch introducing ref-in-want, but modified it substantially. Let me know if it is conventional to credit you anyway, and by which trailer.

#2 lore raw
In reply to Kim Altintop
Quoted 9 lines · Kim Altintop

When 'upload-pack' runs within the context of a git namespace, treat any 'want-ref' lines the client sends as relative to that namespace.

Also check if the wanted ref is hidden via 'hideRefs', and respond with an error otherwise. It was previously possible to request any ref, but note that this is still the case unless 'hideRefs' is in effect.

---

Nicely described. I have a question on the last sentence, though. Do you mean that any ref can be requested when a namespace is in use, as long as 'hideRefs' is not in effect? What does "any ref" exactly mean---even thouse outside the given namespace (and if so how?) I wonder if the last sentence is making the description more confusing without adding any clarity. In other words, would this work as a replacement for the second paragraph, or does it say something different from what you wanted to say?

Requests for any ref, even those that are marked to be hidden via the 'transfer.hideRefs' configuration, were allowed but it is problematic for such and such reasons. Respond with an error if a requested ref is to be hidden.

I couldn't tell why you thought it was problematic, so left "for such and such reasons" to be filled in, but there still may be an issue.

How does the error response look like? We shouldn't be saying "you requested for the hidden/x branch, but you are not allowed to do so, as that is hidden". To hide something, we should pretend that the thing does not exist, so that we can hide even the fact that we are hiding it.

To help future readers of "git log" who find this change from you, we should clarify the "respond with an error" part of your proposed log message (e.g. "pretend that the wanted ref does not exist when it is hidden via the 'transfer.hiderefs' configuration" or something else).

Quoted 7 lines
t/t5703-upload-pack-ref-in-want.sh +7 -0
+test_expect_success 'setup namespaced repo' '
+	(
+		git init -b main "$REPO" &&
+		cd "$REPO" &&
+		test_commit a &&
+		test_commit b &&
+		git checkout a &&

t/t5703-upload-pack-ref-in-want.sh test_expect_success 'fetching with wildcard that matches multiple refs' ' · line 319

Quoted 10 lines
t/t5703-upload-pack-ref-in-want.sh +10 -0
+		test_commit c &&
+		git checkout a &&
+		test_commit d &&
+		git update-ref refs/heads/ns-no b &&
+		git update-ref refs/namespaces/ns/refs/heads/ns-yes c &&
+		git update-ref refs/namespaces/ns/refs/heads/hidden d
+	) &&
+    git -C "$REPO" config uploadpack.allowRefInWant true &&
+    git -C "$REPO" config transfer.hideRefs refs/heads/hidden
+'

I wonder why the last two are outside the subshell? IOW, you could have configured the newly created repository while you were still in there.

Quoted 12 lines
t/t5703-upload-pack-ref-in-want.sh +12 -0
+test_expect_success 'want-ref with namespaces' '
+	oid=$(git -C "$REPO" rev-parse c) &&
+	cat >expected_refs <<-EOF &&
+	$oid refs/heads/ns-yes
+	EOF
+	>expected_commits &&
+
+	oid=$(git -C "$REPO" rev-parse c) &&
+	test-tool pkt-line pack >in <<-EOF &&
+	$(write_command fetch)
+	0001
+	no-progress

t/t5703-upload-pack-ref-in-want.sh test_expect_success 'fetching with wildcard that matches multiple refs' ' · line 342

Quoted 10 lines
t/t5703-upload-pack-ref-in-want.sh +10 -0
+	want-ref refs/heads/ns-yes
+	have $oid
+	done
+	0000
+	EOF
+
+	GIT_NAMESPACE=ns && export GIT_NAMESPACE &&
+	test-tool -C "$REPO" serve-v2 --stateless-rpc >out <in &&
+	check_output
+'

Unless you mean to make all subsequent tests to be done inside the 'ns' namespace, and even when you do, you do not want to do this in order to keep each test as independent as possible (iow, make some of them skippable without affecting the later tests). Run the final test in a subshell, e.g.

	oid=$(git -C "$REPO" rev-parse c) &&
	test-tool pkt-line pack >in <<-EOF &&
	...
	EOF

	(
        	export GIT_NAMESPACE=ns &&
		test-tool ... >out <in
	) &&
	check_output

or if the command you want to run with a custom environment variable is a single external executable like this case, do

	oid=$(git -C "$REPO" rev-parse c) &&
	test-tool pkt-line pack >in <<-EOF &&
	...
	EOF
	GIT_NAMESPACE=ns test-tool ... >out <in &&
	check_output

That way, the environment will be kept clean without GIT_NAMESPACE outside the invocation of test-tool.

Note that you cannot use this technique directly with test_must_fail which is *not* an external executable but is a shell function.

test_must_fail env GIT_NAMESPACE=ns test-tool ...

would be the way to write a step that must fail.

upload-pack.c static int parse_want_ref(struct packet_writer *writer, const char *line, · line 1421

Quoted 11 lines
upload-pack.c +1 -1
index 297b76fcb4..008ac75125 100644
@@ -1417,21 +1417,24 @@ static int parse_want_ref(struct packet_writer *writer, const char *line,
 			  struct string_list *wanted_refs,
 			  struct object_array *want_obj)
 {
-	const char *arg;
+	const char *refname_nons;
 	if (skip_prefix(line, "want-ref ", &arg)) {

Don't you receive the result in refname_nons here, as arg is no longer there?

Quoted 4 lines
upload-pack.c +1 -0
 		struct object_id oid;
 		struct string_list_item *item;
 		struct object *o;
+		struct strbuf refname = STRBUF_INIT;

upload-pack.c static int parse_want_ref(struct packet_writer *writer, const char *line, · line 1432

Quoted 10 lines
upload-pack.c +5 -3
 
-		if (read_ref(arg, &oid)) {
-			packet_writer_error(writer, "unknown ref %s", arg);
-			die("unknown ref %s", arg);
+		strbuf_addf(&refname, "%s%s", get_git_namespace(), refname_nons);
+		if (ref_is_hidden(refname_nons, refname.buf) ||
+		    read_ref(refname.buf, &oid)) {
+			packet_writer_error(writer, "unknown ref %s", refname_nons);
+			die("unknown ref %s", refname.buf);
 		}

OK. Assuming that it makes sense for the hideRefs mechanism to kick in here (which I would prefer to hear from others who've worked with this code, say Jonathan Tan?), the updated code makes sense.

Thanks.

Quoted 11 lines
upload-pack.c +2 -3
-		item = string_list_append(wanted_refs, arg);
+		item = string_list_append(wanted_refs, refname_nons);
 		item->util = oiddup(&oid);
 
-		o = parse_object_or_die(&oid, arg);
+		o = parse_object_or_die(&oid, refname);
 		if (!(o->flags & WANTED)) {
 			o->flags |= WANTED;
 			add_object_array(o, NULL, want_obj);
--

2.32.0

#3 lore raw
Quoted 2 lines · Junio C Hamano

Kim Altintop <[email protected]> writes:

[..]

Quoted 3 lines
Quoted 3 lines

Also check if the wanted ref is hidden via 'hideRefs', and respond with an error otherwise. It was previously possible to request any ref, but note that this is still the case unless 'hideRefs' is in effect.

[..]

Quoted 5 lines

Nicely described. I have a question on the last sentence, though. Do you mean that any ref can be requested when a namespace is in use, as long as 'hideRefs' is not in effect? What does "any ref" exactly mean---even thouse outside the given namespace (and if so how?)

Thank you. It seems like I got confused for a moment when writing the commit message. It's not possible to get a ref outside the namespace anymore. I removed that sentence.

Quoted 13 lines
Quoted 13 lines
t/t5703-upload-pack-ref-in-want.sh +13 -0
+test_expect_success 'setup namespaced repo' '
+	(
+		git init -b main "$REPO" &&
+		cd "$REPO" &&
+		test_commit a &&
+		test_commit b &&
+		git checkout a &&
+		test_commit c &&
+		git checkout a &&
+		test_commit d &&
+		git update-ref refs/heads/ns-no b &&
+		git update-ref refs/namespaces/ns/refs/heads/ns-yes c &&
+		git update-ref refs/namespaces/ns/refs/heads/hidden d
Quoted 8 lines

t/t5703-upload-pack-ref-in-want.sh test_expect_success 'fetching with wildcard that matches multiple refs' ' · line 319

Quoted 4 lines
t/t5703-upload-pack-ref-in-want.sh +4 -0
+	) &&
+    git -C "$REPO" config uploadpack.allowRefInWant true &&
+    git -C "$REPO" config transfer.hideRefs refs/heads/hidden
+'

I wonder why the last two are outside the subshell? IOW, you could have configured the newly created repository while you were still in there.

To be honest, I don't know. I did that because the other repo setups in that file follow the same pattern, I suppose that qualifies as "cargo culting". Happy to remove the subshell, unless others point out that there is some specific reason for it.

Quoted 30 lines

Unless you mean to make all subsequent tests to be done inside the 'ns' namespace, and even when you do, you do not want to do this in order to keep each test as independent as possible (iow, make some of them skippable without affecting the later tests). Run the final test in a subshell, e.g.

oid=$(git -C "$REPO" rev-parse c) &&
test-tool pkt-line pack >in <<-EOF &&
...
EOF

(
export GIT_NAMESPACE=ns &&
test-tool ... >out <in
) &&
check_output

or if the command you want to run with a custom environment variable is a single external executable like this case, do

oid=$(git -C "$REPO" rev-parse c) &&
test-tool pkt-line pack >in <<-EOF &&
...
EOF
GIT_NAMESPACE=ns test-tool ... >out <in &&
check_output

That way, the environment will be kept clean without GIT_NAMESPACE outside the invocation of test-tool.

Quoted 6 lines

Note that you cannot use this technique directly with test_must_fail which is *not* an external executable but is a shell function.

test_must_fail env GIT_NAMESPACE=ns test-tool ...

would be the way to write a step that must fail.

Ah thanks! I had tried

...
GIT_NAMESPACE=ns test-tool ... >out <in

but the linter complained about this without giving a hint as to what the fix would be. It turns out that "env" works, ie.

...
env GIT_NAMESPACE=ns test-tool ...
test_must_fail env GIT_NAMESPACE=ns test-tool ...

Quoted 15 lines

upload-pack.c static int parse_want_ref(struct packet_writer *writer, const char *line, · line 1421

Quoted 12 lines
upload-pack.c +1 -1
index 297b76fcb4..008ac75125 100644
@@ -1417,21 +1417,24 @@ static int parse_want_ref(struct packet_writer *writer, const char *line,
 			  struct string_list *wanted_refs,
 			  struct object_array *want_obj)
 {
-	const char *arg;
+	const char *refname_nons;
 	if (skip_prefix(line, "want-ref ", &arg)) {

Don't you receive the result in refname_nons here, as arg is no longer there?

Ouch. Will fix.

Quoted 6 lines
Quoted 6 lines
upload-pack.c +1 -0
 		struct object_id oid;
 		struct string_list_item *item;
 		struct object *o;
+		struct strbuf refname = STRBUF_INIT;
 
Quoted 13 lines

upload-pack.c static int parse_want_ref(struct packet_writer *writer, const char *line, · line 1432

Quoted 9 lines
upload-pack.c +5 -3
-		if (read_ref(arg, &oid)) {
-			packet_writer_error(writer, "unknown ref %s", arg);
-			die("unknown ref %s", arg);
+		strbuf_addf(&refname, "%s%s", get_git_namespace(), refname_nons);
+		if (ref_is_hidden(refname_nons, refname.buf) ||
+		    read_ref(refname.buf, &oid)) {
+			packet_writer_error(writer, "unknown ref %s", refname_nons);
+			die("unknown ref %s", refname.buf);
 		}

OK. Assuming that it makes sense for the hideRefs mechanism to kick in here (which I would prefer to hear from others who've worked with this code, say Jonathan Tan?), the updated code makes sense.

I have also updated the code for the v2 to use refname_nons for any die() calls, as I realised that this may be transmitted to the client via sideband (is that correct?).

#4 lore raw

PATCH v2 1/1 upload-pack.c: treat want-ref relative to namespace

In reply to Kim Altintop

When 'upload-pack' runs within the context of a git namespace, treat any 'want-ref' lines the client sends as relative to that namespace.

Also check if the wanted ref is hidden via 'hideRefs'. If it is hidden, respond with an error as if the ref didn't exist.

---

Changes from v1:

* Amend commit message
* upload-pack.c: fix variable renaming (how could this even work?)
* upload-pack.c: hide namespace in all output, including die()
* t5703: don't use subshell in repo setup
* t5703: use "env" keyword to correctly scope GIT_NAMESPACE

 t/t5703-upload-pack-ref-in-want.sh | 72 ++++++++++++++++++++++++++++++
 upload-pack.c                      | 17 ++++---
 2 files changed, 82 insertions(+), 7 deletions(-)
t/t5703-upload-pack-ref-in-want.sh +72 -0
index e9e471621d..96df3073d1 100755
@@ -298,6 +298,78 @@ test_expect_success 'fetching with wildcard that matches multiple refs' '
 	grep "want-ref refs/heads/o/bar" log
 '
 
+REPO="$(pwd)/repo-ns"
+
+test_expect_success 'setup namespaced repo' '
+	git init -b main "$REPO" &&
+	cd "$REPO" &&
+	test_commit a &&
+	test_commit b &&
+	git checkout a &&
+	test_commit c &&
+	git checkout a &&
+	test_commit d &&
+	git update-ref refs/heads/ns-no b &&
+	git update-ref refs/namespaces/ns/refs/heads/ns-yes c &&
+	git update-ref refs/namespaces/ns/refs/heads/hidden d &&
+	git -C "$REPO" config uploadpack.allowRefInWant true &&
+	git -C "$REPO" config transfer.hideRefs refs/heads/hidden
+'
+
+test_expect_success 'want-ref with namespaces' '
+	oid=$(git -C "$REPO" rev-parse c) &&
+	cat >expected_refs <<-EOF &&
+	$oid refs/heads/ns-yes
+	EOF
+	>expected_commits &&
+
+	oid=$(git -C "$REPO" rev-parse c) &&
+	test-tool pkt-line pack >in <<-EOF &&
+	$(write_command fetch)
+	0001
+	no-progress
+	want-ref refs/heads/ns-yes
+	have $oid
+	done
+	0000
+	EOF
+
+	env GIT_NAMESPACE=ns test-tool -C "$REPO" serve-v2 --stateless-rpc >out <in &&
+	check_output
+'
+
+test_expect_success 'want-ref outside namespace' '
+	oid=$(git -C "$REPO" rev-parse c) &&
+	test-tool pkt-line pack >in <<-EOF &&
+	$(write_command fetch)
+	0001
+	no-progress
+	want-ref refs/heads/ns-no
+	have $oid
+	done
+	0000
+	EOF
+
+	test_must_fail env GIT_NAMESPACE=ns test-tool -C "$REPO" serve-v2 --stateless-rpc >out <in &&
+	grep "unknown ref" out
+'
+
+test_expect_success 'hideRefs with namespaces' '
+	oid=$(git -C "$REPO" rev-parse c) &&
+	test-tool pkt-line pack >in <<-EOF &&
+	$(write_command fetch)
+	0001
+	no-progress
+	want-ref refs/heads/hidden
+	have $oid
+	done
+	0000
+	EOF
+
+	test_must_fail env GIT_NAMESPACE=ns test-tool -C "$REPO" serve-v2 --stateless-rpc >out <in &&
+	grep "unknown ref" out
+'
+
 . "$TEST_DIRECTORY"/lib-httpd.sh
 start_httpd
 
upload-pack.c +10 -7
index 297b76fcb4..c897802f1c 100644
@@ -1417,21 +1417,24 @@ static int parse_want_ref(struct packet_writer *writer, const char *line,
 			  struct string_list *wanted_refs,
 			  struct object_array *want_obj)
 {
-	const char *arg;
-	if (skip_prefix(line, "want-ref ", &arg)) {
+	const char *refname_nons;
+	if (skip_prefix(line, "want-ref ", &refname_nons)) {
 		struct object_id oid;
 		struct string_list_item *item;
 		struct object *o;
+		struct strbuf refname = STRBUF_INIT;
 
-		if (read_ref(arg, &oid)) {
-			packet_writer_error(writer, "unknown ref %s", arg);
-			die("unknown ref %s", arg);
+		strbuf_addf(&refname, "%s%s", get_git_namespace(), refname_nons);
+		if (ref_is_hidden(refname_nons, refname.buf) ||
+		    read_ref(refname.buf, &oid)) {
+			packet_writer_error(writer, "unknown ref %s", refname_nons);
+			die("unknown ref %s", refname_nons);
 		}
 
-		item = string_list_append(wanted_refs, arg);
+		item = string_list_append(wanted_refs, refname_nons);
 		item->util = oiddup(&oid);
 
-		o = parse_object_or_die(&oid, arg);
+		o = parse_object_or_die(&oid, refname_nons);
 		if (!(o->flags & WANTED)) {
 			o->flags |= WANTED;
 			add_object_array(o, NULL, want_obj);

--
2.32.0

#5 lore raw
Quoted 15 lines
t/t5703-upload-pack-ref-in-want.sh +7 -0
index e9e471621d..96df3073d1 100755
@@ -298,6 +298,78 @@ test_expect_success 'fetching with wildcard that matches multiple refs' '
 	grep "want-ref refs/heads/o/bar" log
 '
 
+REPO="$(pwd)/repo-ns"
+
+test_expect_success 'setup namespaced repo' '
+	git init -b main "$REPO" &&
+	cd "$REPO" &&
+	test_commit a &&
+	test_commit b &&

t/t5703-upload-pack-ref-in-want.sh test_expect_success 'fetching with wildcard that matches multiple refs' ' · line 317

Quoted 10 lines
t/t5703-upload-pack-ref-in-want.sh +10 -0
+	git checkout a &&
+	test_commit c &&
+	git checkout a &&
+	test_commit d &&
+	git update-ref refs/heads/ns-no b &&
+	git update-ref refs/namespaces/ns/refs/heads/ns-yes c &&
+	git update-ref refs/namespaces/ns/refs/heads/hidden d &&
+	git -C "$REPO" config uploadpack.allowRefInWant true &&
+	git -C "$REPO" config transfer.hideRefs refs/heads/hidden
+'

If you're not using a subshell to set up the repo, you should add '-C "$REPO"' to all the "git" commands (like you do in the last 2 lines) instead of "cd"-ing halfway through the test. The helper function test_commit also has that facility ('test_commit -C "$REPO" a', for example).

Quoted 4 lines
t/t5703-upload-pack-ref-in-want.sh +4 -0
+test_expect_success 'want-ref with namespaces' '
+	oid=$(git -C "$REPO" rev-parse c) &&
+	cat >expected_refs <<-EOF &&
+	$oid refs/heads/ns-yes

t/t5703-upload-pack-ref-in-want.sh test_expect_success 'fetching with wildcard that matches multiple refs' ' · line 332

Quoted 10 lines
t/t5703-upload-pack-ref-in-want.sh +10 -0
+	EOF
+	>expected_commits &&
+
+	oid=$(git -C "$REPO" rev-parse c) &&
+	test-tool pkt-line pack >in <<-EOF &&
+	$(write_command fetch)
+	0001
+	no-progress
+	want-ref refs/heads/ns-yes
+	have $oid

Do we need this "have" line? I think we can just omit it, since what the client has is irrelevant to the test. (Same for the other tests.)

Quoted 13 lines
t/t5703-upload-pack-ref-in-want.sh +13 -0
+	done
+	0000
+	EOF
+
+	env GIT_NAMESPACE=ns test-tool -C "$REPO" serve-v2 --stateless-rpc >out <in &&
+	check_output
+'
+
+test_expect_success 'want-ref outside namespace' '
+	oid=$(git -C "$REPO" rev-parse c) &&
+	test-tool pkt-line pack >in <<-EOF &&
+	$(write_command fetch)
+	0001

t/t5703-upload-pack-ref-in-want.sh test_expect_success 'fetching with wildcard that matches multiple refs' ' · line 355

Quoted 10 lines
t/t5703-upload-pack-ref-in-want.sh +10 -0
+	no-progress
+	want-ref refs/heads/ns-no
+	have $oid
+	done
+	0000
+	EOF
+
+	test_must_fail env GIT_NAMESPACE=ns test-tool -C "$REPO" serve-v2 --stateless-rpc >out <in &&
+	grep "unknown ref" out
+'

For the failure tests, it's safer to write them in pairs - one that succeeds and one that fails. Here, a typo in "ns-no" (e.g. if I wrote "ns-noo" instead) would cause the exact same result, but if we were to write a pair of tests, we wouldn't have this problem.

To do this, you can bundle the same code into a function and call them from both tests. E.g.:

  setup_want_ns_no () {
    (common code)
  }

test_expect_success 'want-ref without namespace works...' '

    setup_want_ref_outside_namespace &&
    test-tool ... &&
    check_output

'

test_expect_success '...but, with namespace, does not work' '

    setup_want_ref_outside_namespace &&
    test_must_fail env GIT_NAMESPACE=ns ... &&
    grep "unknown ref" out

'

The first test_expect_success does seem redundant, but I can't think of a better way to ensure that the helper function (setup_want_ns_no in this case) is written correctly.

Quoted 6 lines
t/t5703-upload-pack-ref-in-want.sh +6 -0
+test_expect_success 'hideRefs with namespaces' '
+	oid=$(git -C "$REPO" rev-parse c) &&
+	test-tool pkt-line pack >in <<-EOF &&
+	$(write_command fetch)
+	0001
+	no-progress

t/t5703-upload-pack-ref-in-want.sh test_expect_success 'fetching with wildcard that matches multiple refs' ' · line 372

Quoted 10 lines
t/t5703-upload-pack-ref-in-want.sh +10 -0
+	want-ref refs/heads/hidden
+	have $oid
+	done
+	0000
+	EOF
+
+	test_must_fail env GIT_NAMESPACE=ns test-tool -C "$REPO" serve-v2 --stateless-rpc >out <in &&
+	grep "unknown ref" out
+'
+

Same for this.

Quoted 6 lines
upload-pack.c +0 -0
index 297b76fcb4..c897802f1c 100644
@@ -1417,21 +1417,24 @@ static int parse_want_ref(struct packet_writer *writer, const char *line,
 			  struct string_list *wanted_refs,

upload-pack.c static int parse_want_ref(struct packet_writer *writer, const char *line, · line 1425

Quoted 10 lines
upload-pack.c +3 -2
 			  struct object_array *want_obj)
 {
-	const char *arg;
-	if (skip_prefix(line, "want-ref ", &arg)) {
+	const char *refname_nons;
+	if (skip_prefix(line, "want-ref ", &refname_nons)) {
 		struct object_id oid;
 		struct string_list_item *item;
 		struct object *o;
+		struct strbuf refname = STRBUF_INIT;

"refname" needs to be released somewhere.

Quoted 10 lines
upload-pack.c +5 -3
-		if (read_ref(arg, &oid)) {
-			packet_writer_error(writer, "unknown ref %s", arg);
-			die("unknown ref %s", arg);
+		strbuf_addf(&refname, "%s%s", get_git_namespace(), refname_nons);
+		if (ref_is_hidden(refname_nons, refname.buf) ||
+		    read_ref(refname.buf, &oid)) {
+			packet_writer_error(writer, "unknown ref %s", refname_nons);
+			die("unknown ref %s", refname_nons);
 		}

upload-pack.c static int parse_want_ref(struct packet_writer *writer, const char *line, · line 1440

Quoted 10 lines
upload-pack.c +2 -2
 
-		item = string_list_append(wanted_refs, arg);
+		item = string_list_append(wanted_refs, refname_nons);
 		item->util = oiddup(&oid);
 
-		o = parse_object_or_die(&oid, arg);
+		o = parse_object_or_die(&oid, refname_nons);
 		if (!(o->flags & WANTED)) {
 			o->flags |= WANTED;
 			add_object_array(o, NULL, want_obj);

Besides my comments about the tests, I think that this patch looks good. Junio had a relevant question [1]:

Quoted 3 lines

OK. Assuming that it makes sense for the hideRefs mechanism to kick in here (which I would prefer to hear from others who've worked with this code, say Jonathan Tan?), the updated code makes sense.

I think it makes sense here. I checked my old code that Kim linked [2], and in that patch I put it somewhere else (specifically, the part that prints out the "wanted-ref" lines), but it makes sense that it is different. In my version, "want-ref" supports globs and is allowed to match 0 refs, but in Brandon's final version, "want-ref" does not support globs and must match the ref, so checking it upon parse (as is done in this commit) is the most reasonable.

Questions from Kim:

Quoted 3 lines

Jonathan: I took the test code from your original patch introducing ref-in-want, but modified it substantially. Let me know if it is conventional to credit you anyway, and by which trailer.

I don't think there's a convention, but you can use the "Helped-by:" trailer if you want.

Quoted 3 lines

I have also updated the code for the v2 to use refname_nons for any die() calls, as I realised that this may be transmitted to the client via sideband (is that correct?).

I believe that only packet_writer_error() output is sent via sideband, but it still makes sense for the server-side output to print out the ref as read from the client verbatim.

[1] https://lore.kernel.org/git/[email protected]/
[2] https://lore.kernel.org/git/d0d42b3bb4cf755f122591e191354c53848f197d.1485381677.git.jonathantanmy@google.com/

#6 lore raw
Quoted 13 lines · Jonathan Tan
Quoted 13 lines
t/t5703-upload-pack-ref-in-want.sh +13 -0
+test_expect_success 'setup namespaced repo' '
+	git init -b main "$REPO" &&
+	cd "$REPO" &&
+	test_commit a &&
+	test_commit b &&
+	git checkout a &&
+	test_commit c &&
+	git checkout a &&
+	test_commit d &&
+	git update-ref refs/heads/ns-no b &&
+	git update-ref refs/namespaces/ns/refs/heads/ns-yes c &&
+	git update-ref refs/namespaces/ns/refs/heads/hidden d &&
+	git -C "$REPO" config uploadpack.allowRefInWant true &&
Quoted 8 lines

Review

Quoted 2 lines
(quoted diff) +2 -0
+	git -C "$REPO" config transfer.hideRefs refs/heads/hidden
+'

If you're not using a subshell to set up the repo, you should add '-C "$REPO"' to all the "git" commands (like you do in the last 2 lines) instead of "cd"-ing halfway through the test. The helper function test_commit also has that facility ('test_commit -C "$REPO" a', for example).

Ah, that answers the question raised by Junio in the first review. I'll revert to using a subshell, as that seems clearer and is used throughout the file.

Quoted 9 lines
Quoted 9 lines
t/t5703-upload-pack-ref-in-want.sh +9 -0
+test_expect_success 'want-ref with namespaces' '
+	oid=$(git -C "$REPO" rev-parse c) &&
+	cat >expected_refs <<-EOF &&
+	$oid refs/heads/ns-yes
+	EOF
+	>expected_commits &&
+
+	oid=$(git -C "$REPO" rev-parse c) &&
+	test-tool pkt-line pack >in <<-EOF &&
Quoted 8 lines

t/t5703-upload-pack-ref-in-want.sh test_expect_success 'fetching with wildcard that matches multiple refs' ' · line 332

Quoted 5 lines
t/t5703-upload-pack-ref-in-want.sh +5 -0
+	$(write_command fetch)
+	0001
+	no-progress
+	want-ref refs/heads/ns-yes
+	have $oid

Do we need this "have" line? I think we can just omit it, since what the client has is irrelevant to the test. (Same for the other tests.)

Hm no. I think I misread how this works.

Quoted 12 lines
Quoted 12 lines
t/t5703-upload-pack-ref-in-want.sh +12 -0
+test_expect_success 'want-ref outside namespace' '
+	oid=$(git -C "$REPO" rev-parse c) &&
+	test-tool pkt-line pack >in <<-EOF &&
+	$(write_command fetch)
+	0001
+	no-progress
+	want-ref refs/heads/ns-no
+	have $oid
+	done
+	0000
+	EOF
+
Quoted 8 lines

t/t5703-upload-pack-ref-in-want.sh test_expect_success 'fetching with wildcard that matches multiple refs' ' · line 355

Quoted 3 lines
t/t5703-upload-pack-ref-in-want.sh +3 -0
+	test_must_fail env GIT_NAMESPACE=ns test-tool -C "$REPO" serve-v2 --stateless-rpc >out <in &&
+	grep "unknown ref" out
+'

For the failure tests, it's safer to write them in pairs - one that succeeds and one that fails. Here, a typo in "ns-no" (e.g. if I wrote "ns-noo" instead) would cause the exact same result, but if we were to write a pair of tests, we wouldn't have this problem.

That's a good suggestion. However, I'm having some difficulties finding just the right amount of common code to extract due to

a. having to pass the namespace via env instead of --namespace (and the
different position for the success / test_must_fail cases), and
b. having to rely on _persistent_ config for hideRefs, as opposed to being able
to pass it via -c to upload-pack (ie. I'm worried about tests breaking if they get reordered)

So I'm not exactly happy with what I came up with for v3, but I'd also be reluctant to add --namespace / -c support to test-tool as part of this patch. Let me know what you think.

Quoted 10 lines
Quoted 10 lines
upload-pack.c +0 -2
index 297b76fcb4..c897802f1c 100644
@@ -1417,21 +1417,24 @@ static int parse_want_ref(struct packet_writer *writer, const char *line,
 			  struct string_list *wanted_refs,
 			  struct object_array *want_obj)
 {
-	const char *arg;
-	if (skip_prefix(line, "want-ref ", &arg)) {
Quoted 8 lines

upload-pack.c static int parse_want_ref(struct packet_writer *writer, const char *line, · line 1425

Quoted 6 lines
upload-pack.c +3 -0
+	const char *refname_nons;
+	if (skip_prefix(line, "want-ref ", &refname_nons)) {
 		struct object_id oid;
 		struct string_list_item *item;
 		struct object *o;
+		struct strbuf refname = STRBUF_INIT;

"refname" needs to be released somewhere.

Yeap :/

#7 lore raw

PATCH v3 1/1 upload-pack.c: treat want-ref relative to namespace

In reply to Kim Altintop

When 'upload-pack' runs within the context of a git namespace, treat any 'want-ref' lines the client sends as relative to that namespace.

Also check if the wanted ref is hidden via 'hideRefs'. If it is hidden, respond with an error as if the ref didn't exist.

---

Changes from v2:

* upload-pack.c: release strbuf
* t5730: revert to scoping to $REPO via subshell in setup
* t5730: add "cross-check" tests as per review comments from Jonathan

 t/t5703-upload-pack-ref-in-want.sh | 128 +++++++++++++++++++++++++++++
 upload-pack.c                      |  18 ++--
 2 files changed, 139 insertions(+), 7 deletions(-)
t/t5703-upload-pack-ref-in-want.sh +128 -0
index e9e471621d..4a828042bb 100755
@@ -298,6 +298,134 @@ test_expect_success 'fetching with wildcard that matches multiple refs' '
 	grep "want-ref refs/heads/o/bar" log
 '
 
+REPO="$(pwd)/repo-ns"
+
+write_fetch_want_ref() {
+	local wanted_ref="$1"
+
+	write_command fetch
+	echo "0001"
+	echo "no-progress"
+	echo "want-ref $1"
+	echo "done"
+	echo "0000"
+}
+
+test_expect_success 'setup namespaced repo' '
+	(
+		git init -b main "$REPO" &&
+		cd "$REPO" &&
+		test_commit a &&
+		test_commit b &&
+		git checkout a &&
+		test_commit c &&
+		git checkout a &&
+		test_commit d &&
+		git update-ref refs/heads/ns-no b &&
+		git update-ref refs/namespaces/ns/refs/heads/ns-yes c &&
+		git update-ref refs/namespaces/ns/refs/heads/hidden d
+	) &&
+	git -C "$REPO" config uploadpack.allowRefInWant true
+'
+
+test_expect_success 'with namespace: want-ref is considered relative to namespace' '
+	wanted_ref=refs/heads/ns-yes &&
+
+	oid=$(git -C "$REPO" rev-parse $wanted_ref) &&
+	cat >expected_refs <<-EOF &&
+	$oid $wanted_ref
+	EOF
+	git -C "$REPO" rev-parse a $wanted_ref >expected_commits &&
+
+	test-tool pkt-line pack >in <<-EOF &&
+	$(write_fetch_want_ref $wanted_ref)
+	EOF
+
+	env GIT_NAMESPACE=ns test-tool -C "$REPO" serve-v2 --stateless-rpc >out <in &&
+	check_output
+'
+
+test_expect_success 'with namespace: want-ref outside namespace is unknown' '
+	wanted_ref=refs/heads/ns-no &&
+
+	test-tool pkt-line pack >in <<-EOF &&
+	$(write_fetch_want_ref $wanted_ref)
+	EOF
+
+	test_must_fail env GIT_NAMESPACE=ns \
+		test-tool -C "$REPO" serve-v2 --stateless-rpc >out <in &&
+	grep "unknown ref" out
+'
+
+# Cross-check refs/heads/ns-no indeed exists
+test_expect_success 'without namespace: want-ref outside namespace succeeds' '
+	wanted_ref=refs/heads/ns-no &&
+
+	oid=$(git -C "$REPO" rev-parse $wanted_ref) &&
+	cat >expected_refs <<-EOF &&
+	$oid $wanted_ref
+	EOF
+	git -C "$REPO" rev-parse a $wanted_ref >expected_commits &&
+
+	test-tool pkt-line pack >in <<-EOF &&
+	$(write_fetch_want_ref $wanted_ref)
+	EOF
+
+	test-tool -C "$REPO" serve-v2 --stateless-rpc >out <in &&
+	check_output
+'
+
+test_expect_success 'with namespace: hideRefs is matched, relative to namespace' '
+	wanted_ref=refs/heads/hidden &&
+	git -C "$REPO" config transfer.hideRefs $wanted_ref &&
+
+	test-tool pkt-line pack >in <<-EOF &&
+	$(write_fetch_want_ref $wanted_ref)
+	EOF
+
+	test_must_fail env GIT_NAMESPACE=ns \
+		test-tool -C "$REPO" serve-v2 --stateless-rpc >out <in &&
+	grep "unknown ref" out
+'
+
+# Cross-check refs/heads/hidden indeed exists
+test_expect_success 'with namespace: want-ref succeeds if hideRefs is removed' '
+	wanted_ref=refs/heads/hidden &&
+	git -C "$REPO" config --unset transfer.hideRefs $wanted_ref &&
+
+	oid=$(git -C "$REPO" rev-parse $wanted_ref) &&
+	cat >expected_refs <<-EOF &&
+	$oid $wanted_ref
+	EOF
+	git -C "$REPO" rev-parse a $wanted_ref >expected_commits &&
+
+	test-tool pkt-line pack >in <<-EOF &&
+	$(write_fetch_want_ref $wanted_ref)
+	EOF
+
+	env GIT_NAMESPACE=ns test-tool -C "$REPO" serve-v2 --stateless-rpc >out <in &&
+	check_output
+'
+
+test_expect_success 'without namespace: relative hideRefs does not match' '
+	wanted_ref=refs/namespaces/ns/refs/heads/hidden &&
+	git -C "$REPO" config transfer.hideRefs refs/heads/hidden &&
+
+	oid=$(git -C "$REPO" rev-parse $wanted_ref) &&
+	cat >expected_refs <<-EOF &&
+	$oid $wanted_ref
+	EOF
+	git -C "$REPO" rev-parse a $wanted_ref >expected_commits &&
+
+	test-tool pkt-line pack >in <<-EOF &&
+	$(write_fetch_want_ref $wanted_ref)
+	EOF
+
+	env GIT_NAMESPACE=ns test-tool -C "$REPO" serve-v2 --stateless-rpc >out <in &&
+	check_output
+'
+
+
 . "$TEST_DIRECTORY"/lib-httpd.sh
 start_httpd
 
upload-pack.c +11 -7
index 297b76fcb4..6ce07231d3 100644
@@ -1417,21 +1417,25 @@ static int parse_want_ref(struct packet_writer *writer, const char *line,
 			  struct string_list *wanted_refs,
 			  struct object_array *want_obj)
 {
-	const char *arg;
-	if (skip_prefix(line, "want-ref ", &arg)) {
+	const char *refname_nons;
+	if (skip_prefix(line, "want-ref ", &refname_nons)) {
 		struct object_id oid;
 		struct string_list_item *item;
 		struct object *o;
+		struct strbuf refname = STRBUF_INIT;
 
-		if (read_ref(arg, &oid)) {
-			packet_writer_error(writer, "unknown ref %s", arg);
-			die("unknown ref %s", arg);
+		strbuf_addf(&refname, "%s%s", get_git_namespace(), refname_nons);
+		if (ref_is_hidden(refname_nons, refname.buf) ||
+		    read_ref(refname.buf, &oid)) {
+			packet_writer_error(writer, "unknown ref %s", refname_nons);
+			die("unknown ref %s", refname_nons);
 		}
+		strbuf_release(&refname);
 
-		item = string_list_append(wanted_refs, arg);
+		item = string_list_append(wanted_refs, refname_nons);
 		item->util = oiddup(&oid);
 
-		o = parse_object_or_die(&oid, arg);
+		o = parse_object_or_die(&oid, refname_nons);
 		if (!(o->flags & WANTED)) {
 			o->flags |= WANTED;
 			add_object_array(o, NULL, want_obj);

--
2.32.0

#8 lore raw

PATCH v4 1/1 upload-pack.c: treat want-ref relative to namespace

When 'upload-pack' runs within the context of a git namespace, treat any 'want-ref' lines the client sends as relative to that namespace.

Also check if the wanted ref is hidden via 'hideRefs'. If it is hidden, respond with an error as if the ref didn't exist.

---

Changes from v3:

* t5703: make tests actually pass :)

 t/t5703-upload-pack-ref-in-want.sh | 140 +++++++++++++++++++++++++++++
 upload-pack.c                      |  18 ++--
 2 files changed, 151 insertions(+), 7 deletions(-)
t/t5703-upload-pack-ref-in-want.sh +140 -0
index e9e471621d..4da86d771b 100755
@@ -298,6 +298,146 @@ test_expect_success 'fetching with wildcard that matches multiple refs' '
 	grep "want-ref refs/heads/o/bar" log
 '
 
+REPO="$(pwd)/repo-ns"
+
+write_fetch_want_ref() {
+	local wanted_ref="$1"
+
+	write_command fetch
+	echo "0001"
+	echo "no-progress"
+	echo "want-ref $1"
+	echo "done"
+	echo "0000"
+}
+
+test_expect_success 'setup namespaced repo' '
+	(
+		git init -b main "$REPO" &&
+		cd "$REPO" &&
+		test_commit a &&
+		test_commit b &&
+		git checkout a &&
+		test_commit c &&
+		git checkout a &&
+		test_commit d &&
+		git update-ref refs/heads/ns-no b &&
+		git update-ref refs/namespaces/ns/refs/heads/ns-yes c &&
+		git update-ref refs/namespaces/ns/refs/heads/hidden d
+	) &&
+	git -C "$REPO" config uploadpack.allowRefInWant true
+'
+
+test_expect_success 'with namespace: want-ref is considered relative to namespace' '
+	wanted_ref=refs/heads/ns-yes &&
+
+	oid=$(git -C "$REPO" rev-parse "refs/namespaces/ns/$wanted_ref") &&
+	cat >expected_refs <<-EOF &&
+	$oid $wanted_ref
+	EOF
+	cat >expected_commits <<-EOF &&
+	$oid
+	$(git -C "$REPO" rev-parse a)
+	EOF
+
+	test-tool pkt-line pack >in <<-EOF &&
+	$(write_fetch_want_ref $wanted_ref)
+	EOF
+
+	env GIT_NAMESPACE=ns test-tool -C "$REPO" serve-v2 --stateless-rpc >out <in &&
+	check_output
+'
+
+test_expect_success 'with namespace: want-ref outside namespace is unknown' '
+	wanted_ref=refs/heads/ns-no &&
+
+	test-tool pkt-line pack >in <<-EOF &&
+	$(write_fetch_want_ref $wanted_ref)
+	EOF
+
+	test_must_fail env GIT_NAMESPACE=ns \
+		test-tool -C "$REPO" serve-v2 --stateless-rpc >out <in &&
+	grep "unknown ref" out
+'
+
+# Cross-check refs/heads/ns-no indeed exists
+test_expect_success 'without namespace: want-ref outside namespace succeeds' '
+	wanted_ref=refs/heads/ns-no &&
+
+	oid=$(git -C "$REPO" rev-parse $wanted_ref) &&
+	cat >expected_refs <<-EOF &&
+	$oid $wanted_ref
+	EOF
+	cat >expected_commits <<-EOF &&
+	$oid
+	$(git -C "$REPO" rev-parse a)
+	EOF
+
+	test-tool pkt-line pack >in <<-EOF &&
+	$(write_fetch_want_ref $wanted_ref)
+	EOF
+
+	test-tool -C "$REPO" serve-v2 --stateless-rpc >out <in &&
+	check_output
+'
+
+test_expect_success 'with namespace: hideRefs is matched, relative to namespace' '
+	wanted_ref=refs/heads/hidden &&
+	git -C "$REPO" config transfer.hideRefs $wanted_ref &&
+
+	test-tool pkt-line pack >in <<-EOF &&
+	$(write_fetch_want_ref $wanted_ref)
+	EOF
+
+	test_must_fail env GIT_NAMESPACE=ns \
+		test-tool -C "$REPO" serve-v2 --stateless-rpc >out <in &&
+	grep "unknown ref" out
+'
+
+# Cross-check refs/heads/hidden indeed exists
+test_expect_success 'with namespace: want-ref succeeds if hideRefs is removed' '
+	wanted_ref=refs/heads/hidden &&
+	git -C "$REPO" config --unset transfer.hideRefs $wanted_ref &&
+
+	oid=$(git -C "$REPO" rev-parse "refs/namespaces/ns/$wanted_ref") &&
+	cat >expected_refs <<-EOF &&
+	$oid $wanted_ref
+	EOF
+	cat >expected_commits <<-EOF &&
+	$oid
+	$(git -C "$REPO" rev-parse a)
+	EOF
+
+	test-tool pkt-line pack >in <<-EOF &&
+	$(write_fetch_want_ref $wanted_ref)
+	EOF
+
+	env GIT_NAMESPACE=ns test-tool -C "$REPO" serve-v2 --stateless-rpc >out <in &&
+	check_output
+'
+
+test_expect_success 'without namespace: relative hideRefs does not match' '
+	wanted_ref=refs/namespaces/ns/refs/heads/hidden &&
+	git -C "$REPO" config transfer.hideRefs refs/heads/hidden &&
+
+	oid=$(git -C "$REPO" rev-parse $wanted_ref) &&
+	cat >expected_refs <<-EOF &&
+	$oid $wanted_ref
+	EOF
+	cat >expected_commits <<-EOF &&
+	$oid
+	$(git -C "$REPO" rev-parse a)
+	EOF
+
+	test-tool pkt-line pack >in <<-EOF &&
+	$(write_fetch_want_ref $wanted_ref)
+	EOF
+
+	test-tool -C "$REPO" serve-v2 --stateless-rpc >out <in &&
+	check_output
+'
+
+
 . "$TEST_DIRECTORY"/lib-httpd.sh
 start_httpd
 
upload-pack.c +11 -7
index 297b76fcb4..6ce07231d3 100644
@@ -1417,21 +1417,25 @@ static int parse_want_ref(struct packet_writer *writer, const char *line,
 			  struct string_list *wanted_refs,
 			  struct object_array *want_obj)
 {
-	const char *arg;
-	if (skip_prefix(line, "want-ref ", &arg)) {
+	const char *refname_nons;
+	if (skip_prefix(line, "want-ref ", &refname_nons)) {
 		struct object_id oid;
 		struct string_list_item *item;
 		struct object *o;
+		struct strbuf refname = STRBUF_INIT;
 
-		if (read_ref(arg, &oid)) {
-			packet_writer_error(writer, "unknown ref %s", arg);
-			die("unknown ref %s", arg);
+		strbuf_addf(&refname, "%s%s", get_git_namespace(), refname_nons);
+		if (ref_is_hidden(refname_nons, refname.buf) ||
+		    read_ref(refname.buf, &oid)) {
+			packet_writer_error(writer, "unknown ref %s", refname_nons);
+			die("unknown ref %s", refname_nons);
 		}
+		strbuf_release(&refname);
 
-		item = string_list_append(wanted_refs, arg);
+		item = string_list_append(wanted_refs, refname_nons);
 		item->util = oiddup(&oid);
 
-		o = parse_object_or_die(&oid, arg);
+		o = parse_object_or_die(&oid, refname_nons);
 		if (!(o->flags & WANTED)) {
 			o->flags |= WANTED;
 			add_object_array(o, NULL, want_obj);

--
2.32.0

#9 lore raw

cover upload-pack: treat want-ref relative to namespace

Fifth reroll of [0].

The patch has been split into three commits, where the first is a small refactoring of the test file, the second is the original patch + tests, and the last one attempts to express the transfer.hideRefs behaviour more accurately in the docs.

Thanks for your help and patience!

This version of the patch is also available at [1].

[0]: https://lore.kernel.org/git/[email protected]/
[1]: https://github.com/kim/git/tree/ka/namespaced-want-ref-v5

Kim Altintop (3):
t5730: introduce fetch command helper
upload-pack.c: treat want-ref relative to namespace
docs: clarify the interaction of transfer.hideRefs and namespaces

 Documentation/config/transfer.txt  |  17 ++-
 t/t5703-upload-pack-ref-in-want.sh | 236 +++++++++++++++++++++++++----
 upload-pack.c                      |  18 ++-
 3 files changed, 224 insertions(+), 47 deletions(-)

--
2.32.0

#10 lore raw

PATCH 1/3 t5730: introduce fetch command helper

Assembling a "raw" fetch command to be fed directly to "test-tool serve-v2" is extracted into a test helper.

---
 t/t5703-upload-pack-ref-in-want.sh | 107 ++++++++++++++++++++---------
 1 file changed, 74 insertions(+), 33 deletions(-)
t/t5703-upload-pack-ref-in-want.sh +74 -33
index e9e471621d..cd4744b016 100755
@@ -40,6 +40,54 @@ write_command () {
 	fi
 }
 
+# Write a complete fetch command to stdout, suitable for use with `test-tool
+# pkt-line`. "want-ref", "want", and "have" values can be given in this order,
+# with sections separated by "--".
+#
+# Examples:
+#
+# write_fetch_command refs/heads/main
+#
+# write_fetch_command \
+#	refs/heads/main \
+#	-- \
+#	-- \
+#	$(git rev-parse x)
+#
+# write_fetch_command \
+#	--
+#	$(git rev-parse a) \
+#	--
+#	$(git rev-parse b)
+write_fetch_command () {
+	write_command fetch &&
+	echo "0001" &&
+	echo "no-progress" || return
+    while :
+	do
+		case $# in 0) break ;; esac &&
+		case "$1" in --) shift; break ;; esac &&
+		echo "want-ref $1" &&
+		shift || return
+	done &&
+    while :
+	do
+		case $# in 0) break ;; esac &&
+		case "$1" in --) shift; break ;; esac &&
+		echo "want $1" &&
+		shift || return
+	done &&
+    while :
+	do
+		case $# in 0) break ;; esac &&
+		case "$1" in --) shift; break ;; esac &&
+		echo "have $1" &&
+		shift || return
+	done &&
+	echo "done" &&
+	echo "0000"
+}
+
 # c(o/foo) d(o/bar)
 #        \ /
 #         b   e(baz)  f(main)
@@ -97,15 +145,13 @@ test_expect_success 'basic want-ref' '
 	EOF
 	git rev-parse f >expected_commits &&
 
-	oid=$(git rev-parse a) &&
 	test-tool pkt-line pack >in <<-EOF &&
-	$(write_command fetch)
-	0001
-	no-progress
-	want-ref refs/heads/main
-	have $oid
-	done
-	0000
+	$(write_fetch_command \
+		refs/heads/main \
+		-- \
+		-- \
+		$(git rev-parse a) \
+	)
 	EOF
 
 	test-tool serve-v2 --stateless-rpc >out <in &&
@@ -121,16 +167,14 @@ test_expect_success 'multiple want-ref lines' '
 	EOF
 	git rev-parse c d >expected_commits &&
 
-	oid=$(git rev-parse b) &&
 	test-tool pkt-line pack >in <<-EOF &&
-	$(write_command fetch)
-	0001
-	no-progress
-	want-ref refs/heads/o/foo
-	want-ref refs/heads/o/bar
-	have $oid
-	done
-	0000
+	$(write_fetch_command \
+		refs/heads/o/foo \
+		refs/heads/o/bar \
+		-- \
+		-- \
+		$(git rev-parse b) \
+	)
 	EOF
 
 	test-tool serve-v2 --stateless-rpc >out <in &&
@@ -145,14 +189,13 @@ test_expect_success 'mix want and want-ref' '
 	git rev-parse e f >expected_commits &&
 
 	test-tool pkt-line pack >in <<-EOF &&
-	$(write_command fetch)
-	0001
-	no-progress
-	want-ref refs/heads/main
-	want $(git rev-parse e)
-	have $(git rev-parse a)
-	done
-	0000
+	$(write_fetch_command \
+		refs/heads/main \
+		-- \
+		$(git rev-parse e) \
+		-- \
+		$(git rev-parse a) \
+	)
 	EOF
 
 	test-tool serve-v2 --stateless-rpc >out <in &&
@@ -166,15 +209,13 @@ test_expect_success 'want-ref with ref we already have commit for' '
 	EOF
 	>expected_commits &&
 
-	oid=$(git rev-parse c) &&
 	test-tool pkt-line pack >in <<-EOF &&
-	$(write_command fetch)
-	0001
-	no-progress
-	want-ref refs/heads/o/foo
-	have $oid
-	done
-	0000
+	$(write_fetch_command \
+		refs/heads/o/foo \
+		-- \
+		-- \
+		$(git rev-parse c) \
+	)
 	EOF
 
 	test-tool serve-v2 --stateless-rpc >out <in &&

--
2.32.0

#11 lore raw

Re: [PATCH 1/3] t5730: introduce fetch command helper

Quoted 26 lines · Kim Altintop

Assembling a "raw" fetch command to be fed directly to "test-tool serve-v2" is extracted into a test helper.

---
 t/t5703-upload-pack-ref-in-want.sh | 107 ++++++++++++++++++++---------
 1 file changed, 74 insertions(+), 33 deletions(-)
t/t5703-upload-pack-ref-in-want.sh +9 -0
index e9e471621d..cd4744b016 100755
@@ -40,6 +40,54 @@ write_command () {
 	fi
 }
 
+# Write a complete fetch command to stdout, suitable for use with `test-tool
+# pkt-line`. "want-ref", "want", and "have" values can be given in this order,
+# with sections separated by "--".
+#
+# Examples:
+#
+# write_fetch_command refs/heads/main
+#
+# write_fetch_command \

t/t5703-upload-pack-ref-in-want.sh write_command () { · line 61

Quoted 10 lines
t/t5703-upload-pack-ref-in-want.sh +10 -0
+#	refs/heads/main \
+#	-- \
+#	-- \
+#	$(git rev-parse x)
+#
+# write_fetch_command \
+#	--
+#	$(git rev-parse a) \
+#	--
+#	$(git rev-parse b)

Have a blank line here (or a line with "#" and nothing else) and it would become easier to read.

t/t5703-upload-pack-ref-in-want.sh write_command () { · line 65

Quoted 4 lines
t/t5703-upload-pack-ref-in-want.sh +4 -0
+write_fetch_command () {
+	write_command fetch &&
+	echo "0001" &&
+	echo "no-progress" || return

The "while :" in this helper function are indented with 4 spaces, not a single tab.

Quoted 14 lines
t/t5703-upload-pack-ref-in-want.sh +14 -0
+    while :
+	do
+		case $# in 0) break ;; esac &&
+		case "$1" in --) shift; break ;; esac &&
+		echo "want-ref $1" &&
+		shift || return
+	done &&
+    while :
+	do
+		case $# in 0) break ;; esac &&
+		case "$1" in --) shift; break ;; esac &&
+		echo "want $1" &&
+		shift || return
+	done &&

t/t5703-upload-pack-ref-in-want.sh write_command () { · line 89

Quoted 10 lines
t/t5703-upload-pack-ref-in-want.sh +10 -0
+    while :
+	do
+		case $# in 0) break ;; esac &&
+		case "$1" in --) shift; break ;; esac &&
+		echo "have $1" &&
+		shift || return
+	done &&
+	echo "done" &&
+	echo "0000"
+}

The error checking of the helper function seems to be reasonable, but ...

Quoted 6 lines
t/t5703-upload-pack-ref-in-want.sh +0 -0
 # c(o/foo) d(o/bar)
 #        \ /
 #         b   e(baz)  f(main)
@@ -97,15 +145,13 @@ test_expect_success 'basic want-ref' '
 	EOF
 	git rev-parse f >expected_commits &&

t/t5703-upload-pack-ref-in-want.sh test_expect_success 'basic want-ref' ' · line 156

Quoted 10 lines
t/t5703-upload-pack-ref-in-want.sh +6 -0
 
 	test-tool pkt-line pack >in <<-EOF &&
+	$(write_fetch_command \
+		refs/heads/main \
+		-- \
+		-- \
+		$(git rev-parse a) \
+	)
 	EOF
 	test-tool serve-v2 --stateless-rpc >out <in &&

... the code that uses the helper needs rewriting to make use of it. A failure in $(write_fetch_command) here will not cause the caller to stop here. If we had any "git" command used in the helper, I would recommand restructuring the caller to do something like:

    write_fetch_command >pkt-src \
	refs/heads/main \
	-- \
	-- \
	$(git rev-parse a) &&
    test-tool pkt-line pack <pkt-src >in &&
    test-tool serve-v2 --stateless-rpc >out <in &&

but it probably may not be necessary (we only use "echo" there, and it probably is not worth worrying about 'echo' failing).

The call to $(git rev-parse a) might fail when rev-parse gets broken, and I think the rewritten version would catch such a breakage while the one inside $(write_fetch_command) in the here-doc would not be.

#12 lore raw

Re: [PATCH 1/3] t5730: introduce fetch command helper

Quoted 15 lines · Junio C Hamano
Quoted 15 lines
t/t5703-upload-pack-ref-in-want.sh +14 -0
+# Write a complete fetch command to stdout, suitable for use with `test-tool
+# pkt-line`. "want-ref", "want", and "have" values can be given in this order,
+# with sections separated by "--".
+#
+# Examples:
+#
+# write_fetch_command refs/heads/main
+#
+# write_fetch_command \
+#	refs/heads/main \
+#	-- \
+#	-- \
+#	$(git rev-parse x)
+#
Quoted 8 lines

t/t5703-upload-pack-ref-in-want.sh write_command () { · line 61

Quoted 5 lines
t/t5703-upload-pack-ref-in-want.sh +5 -0
+# write_fetch_command \
+#	--
+#	$(git rev-parse a) \
+#	--
+#	$(git rev-parse b)

Have a blank line here (or a line with "#" and nothing else) and it would become easier to read.

Noted.

Quoted 8 lines

t/t5703-upload-pack-ref-in-want.sh write_command () { · line 65

Quoted 5 lines
t/t5703-upload-pack-ref-in-want.sh +4 -0
+write_fetch_command () {
+	write_command fetch &&
+	echo "0001" &&
+	echo "no-progress" || return

The "while :" in this helper function are indented with 4 spaces, not a single tab.

Noted, sorry.

I can't seem to be able teach git diff --check to exit non-zero on whitespace errors.

Quoted 31 lines
Quoted 6 lines

# c(o/foo) d(o/bar)
# \ /
# b e(baz) f(main)

t/t5703-upload-pack-ref-in-want.sh +0 -0
@@ -97,15 +145,13 @@ test_expect_success 'basic want-ref' '
 	EOF
 	git rev-parse f >expected_commits &&

t/t5703-upload-pack-ref-in-want.sh test_expect_success 'basic want-ref' ' · line 156

Quoted 10 lines
t/t5703-upload-pack-ref-in-want.sh +6 -0
 
 	test-tool pkt-line pack >in <<-EOF &&
+	$(write_fetch_command \
+		refs/heads/main \
+		-- \
+		-- \
+		$(git rev-parse a) \
+	)
 	EOF
 	test-tool serve-v2 --stateless-rpc >out <in &&

... the code that uses the helper needs rewriting to make use of it. A failure in $(write_fetch_command) here will not cause the caller to stop here. If we had any "git" command used in the helper, I would recommand restructuring the caller to do something like:

write_fetch_command >pkt-src \
	refs/heads/main \
	-- \
	-- \
	$(git rev-parse a) &&

test-tool pkt-line pack <pkt-src >in &&
test-tool serve-v2 --stateless-rpc >out <in &&

Quoted 7 lines

but it probably may not be necessary (we only use "echo" there, and it probably is not worth worrying about 'echo' failing).

The call to $(git rev-parse a) might fail when rev-parse gets broken, and I think the rewritten version would catch such a breakage while the one inside $(write_fetch_command) in the here-doc would not be.

Heh, this is turning into an exercise to forget everything about bash :/

I didn't know that an error in a nested command substitution would not cause the outer one to fail. As we can't use a pipe either (as Jonathan Nieder suggests), I think a redirection like this is indeed necessary. The only other option I could think of would be assigning rev-parse to a variable, which arguably is less readable:

    have=$(git rew-parse a) &&
    test-tool pkt-line pack >in <<-EOF &&
    $(write_fetch_command \
	    refs/heads/main \
	    -- \
	    -- \
	    $have)
    EOF

Thanks

#13 lore raw

Re: [PATCH 1/3] t5730: introduce fetch command helper

In reply to Kim Altintop

Hi,

Quoted 8 lines · Kim Altintop

Assembling a "raw" fetch command to be fed directly to "test-tool serve-v2" is extracted into a test helper.

---
 t/t5703-upload-pack-ref-in-want.sh | 107 ++++++++++++++++++++---------
 1 file changed, 74 insertions(+), 33 deletions(-)

Thanks! Interesting.

Quoted 18 lines
t/t5703-upload-pack-ref-in-want.sh +10 -0
index e9e471621d..cd4744b016 100755
@@ -40,6 +40,54 @@ write_command () {
 	fi
 }
 
+# Write a complete fetch command to stdout, suitable for use with `test-tool
+# pkt-line`. "want-ref", "want", and "have" values can be given in this order,
+# with sections separated by "--".
+#
+# Examples:
+#
+# write_fetch_command refs/heads/main
+#
+# write_fetch_command \
+#	refs/heads/main \

t/t5703-upload-pack-ref-in-want.sh write_command () { · line 62

Quoted 10 lines
t/t5703-upload-pack-ref-in-want.sh +10 -0
+#	-- \
+#	-- \
+#	$(git rev-parse x)
+#
+# write_fetch_command \
+#	--
+#	$(git rev-parse a) \
+#	--
+#	$(git rev-parse b)
+write_fetch_command () {

Hm, for comparison let me see what this looks like without the helper:
after some prior step

	object_format=$(test_oid algo) &&	# probably just once in a setup step
	x=$(git rev-parse x) &&

we can write

	cat <<-EOF &&
		command=fetch
		object-format=$object_format
		0001
		no-progress
		want-ref refs/heads/main
		have $x
		done
		0000
	EOF

I find that a little _easier_ to read than a write_fetch_command call, because I don't have to chase the definition and x is labeled as a 'have'.

Is there some additional motivation for this helper?

t/t5703-upload-pack-ref-in-want.sh write_command () { · line 66

Quoted 4 lines
t/t5703-upload-pack-ref-in-want.sh +4 -0
+	write_command fetch &&
+	echo "0001" &&
+	echo "no-progress" || return
+    while :

Whitespace seems off. Junio covers this in his review so I'll ignore other instances.

[...]

Quoted 34 lines
t/t5703-upload-pack-ref-in-want.sh +6 -15
@@ -97,15 +145,13 @@ test_expect_success 'basic want-ref' '
 	EOF
 	git rev-parse f >expected_commits &&
 
-	oid=$(git rev-parse a) &&
 	test-tool pkt-line pack >in <<-EOF &&
-	$(write_command fetch)
-	0001
-	no-progress
-	want-ref refs/heads/main
-	have $oid
-	done
-	0000
+	$(write_fetch_command \
+		refs/heads/main \
+		-- \
+		-- \
+		$(git rev-parse a) \
+	)
 	EOF
 
 	test-tool serve-v2 --stateless-rpc >out <in &&
@@ -121,16 +167,14 @@ test_expect_success 'multiple want-ref lines' '
 	EOF
 	git rev-parse c d >expected_commits &&
 
-	oid=$(git rev-parse b) &&
 	test-tool pkt-line pack >in <<-EOF &&
-	$(write_command fetch)
-	0001
-	no-progress
-	want-ref refs/heads/o/foo
-	want-ref refs/heads/o/bar
-	have $oid

t/t5703-upload-pack-ref-in-want.sh test_expect_success 'multiple want-ref lines' ' · line 178

Quoted 10 lines
t/t5703-upload-pack-ref-in-want.sh +7 -2
-	done
-	0000
+	$(write_fetch_command \
+		refs/heads/o/foo \
+		refs/heads/o/bar \
+		-- \
+		-- \
+		$(git rev-parse b) \
+	)
 	EOF

Here the entirety of the input to "test-tool pkt-line pack" is the entirety of the output from write_fetch_command, which would suggest either

a. making write_fetch_command pipe its output to "test-tool pkt-line
pack", or

b. using a pipe instead of a command substitution, like
"write_fetch_command ... | test-tool pkt-line pack >in"

(although as mentioned above, I think it's simpler to inline the write_fetch_command and even the write_command as well).

Thanks and hope that helps,
Jonathan

#14 lore raw

Re: [PATCH 1/3] t5730: introduce fetch command helper

Quoted 4 lines · Jonathan Nieder

Here the entirety of the input to "test-tool pkt-line pack" is the entirety of the output from write_fetch_command, which would suggest either

Quoted 8 lines

a. making write_fetch_command pipe its output to "test-tool pkt-line
pack", or

b. using a pipe instead of a command substitution, like
"write_fetch_command ... | test-tool pkt-line pack >in"

(although as mentioned above, I think it's simpler to inline the write_fetch_command and even the write_command as well).

Yeah, write_command was not saving all that much to begin with. I was hoping that there were a lot more commonality that we can save some typing.

#15 lore raw

Re: [PATCH 1/3] t5730: introduce fetch command helper

In reply to Jonathan Nieder

Thanks for chiming in!

Quoted 19 lines · Jonathan Nieder

Hm, for comparison let me see what this looks like without the helper:
after some prior step

	 object_format=$(test_oid algo) && # probably just once in a setup step
	 x=$(git rev-parse x) &&

we can write

	 cat <<-EOF &&
	 command=fetch
	 object-format=$object_format
	 0001
	 no-progress
	 want-ref refs/heads/main
	 have $x
	 done
	 0000
	 EOF
Quoted 5 lines

I find that a little _easier_ to read than a write_fetch_command call, because I don't have to chase the definition and x is labeled as a 'have'.

Is there some additional motivation for this helper?

It was suggested in earlier review rounds. I think it does improve readability as quite some lines need to be repeated everywhere a fetch command is assembled. I agree though that not having some sort of "named arguments" is a bit detrimental.

Quoted 29 lines
Quoted 14 lines
t/t5703-upload-pack-ref-in-want.sh +0 -7
 	test-tool serve-v2 --stateless-rpc >out <in &&
@@ -121,16 +167,14 @@ test_expect_success 'multiple want-ref lines' '
 	EOF
 	git rev-parse c d >expected_commits &&
 
-	oid=$(git rev-parse b) &&
 	test-tool pkt-line pack >in <<-EOF &&
-	$(write_command fetch)
-	0001
-	no-progress
-	want-ref refs/heads/o/foo
-	want-ref refs/heads/o/bar
-	have $oid

t/t5703-upload-pack-ref-in-want.sh test_expect_success 'multiple want-ref lines' ' · line 178

Quoted 10 lines
t/t5703-upload-pack-ref-in-want.sh +7 -2
-	done
-	0000
+	$(write_fetch_command \
+		refs/heads/o/foo \
+		refs/heads/o/bar \
+		-- \
+		-- \
+		$(git rev-parse b) \
+	)
 	EOF

Here the entirety of the input to "test-tool pkt-line pack" is the entirety of the output from write_fetch_command, which would suggest either

Quoted 8 lines

a. making write_fetch_command pipe its output to "test-tool pkt-line pack", or

b. using a pipe instead of a command substitution, like "write_fetch_command ... | test-tool pkt-line pack >in"

(although as mentioned above, I think it's simpler to inline the write_fetch_command and even the write_command as well).

Yes, although I believe a pipe cannot be used as we don't have bash's set -o pipefail (ie. the exit status will always be the status of the last command in the pipe, even if an earlier one failed).

Perhaps an alternative would be:

	write_fetch_command () {
		write_command fetch &&
		echo "0001" &&
		echo "no-progress" &&
		cat /dev/stdin &&
		echo "done" &&
		echo "0000"

}

Which would then be called like so:

	write_fetch_command >pkt_cmd <<-EOF &&
	want-ref refs/heads/main
	have $(git rev-parse a)
	EOF
	test-tool pkt-line pack <pkt_cmd >in &&
	test-tool serve-v2 --stateless-rpc >out <in &&

I'm not sure how portable that is, though. Maybe using while read -r instead of cat /dev/stdin?

#16 lore raw

Re: [PATCH 1/3] t5730: introduce fetch command helper

Quoted 22 lines · Kim Altintop

Perhaps an alternative would be:

	write_fetch_command () {
		write_command fetch &&
		echo "0001" &&
		echo "no-progress" &&
		cat /dev/stdin &&
		echo "done" &&
		echo "0000"

}

Which would then be called like so:

	write_fetch_command >pkt_cmd <<-EOF &&
	want-ref refs/heads/main
	have $(git rev-parse a)
	EOF
	test-tool pkt-line pack <pkt_cmd >in &&
	test-tool serve-v2 --stateless-rpc >out <in &&
Quoted 2 lines

I'm not sure how portable that is, though. Maybe using while read -r instead of cat /dev/stdin?

If you drop /dev/stdin, the result would be emimently portable. "cat" without any argument reads from the standard input stream and copies it to the standard output stream.

#17 lore raw

Re: [PATCH 1/3] t5730: introduce fetch command helper

Quoted 24 lines · Junio C Hamano
Quoted 22 lines · Kim Altintop

Perhaps an alternative would be:

	write_fetch_command () {
		write_command fetch &&
		echo "0001" &&
		echo "no-progress" &&
		cat /dev/stdin &&
		echo "done" &&
		echo "0000"

}

Which would then be called like so:

	write_fetch_command >pkt_cmd <<-EOF &&
	want-ref refs/heads/main
	have $(git rev-parse a)
	EOF
	test-tool pkt-line pack <pkt_cmd >in &&
	test-tool serve-v2 --stateless-rpc >out <in &&
Quoted 6 lines
Quoted 2 lines

I'm not sure how portable that is, though. Maybe using while read -r instead of cat /dev/stdin?

If you drop /dev/stdin, the result would be emimently portable. "cat" without any argument reads from the standard input stream and copies it to the standard output stream.

Yep, with that tweak (using "cat" instead of "cat /dev/stdin") this looks like a pleasant enough helper.

Jonathan

#18 lore raw

Re: [PATCH 1/3] t5730: introduce fetch command helper

Quoted 29 lines · Jonathan Nieder
Quoted 27 lines · Junio C Hamano
Quoted 24 lines · Kim Altintop

Perhaps an alternative would be:

	write_fetch_command () {
		write_command fetch &&
		echo "0001" &&
		echo "no-progress" &&
		cat /dev/stdin &&
		echo "done" &&
		echo "0000"

}

Which would then be called like so:

	write_fetch_command >pkt_cmd <<-EOF &&
	want-ref refs/heads/main
	have $(git rev-parse a)
	EOF
	test-tool pkt-line pack <pkt_cmd >in &&
	test-tool serve-v2 --stateless-rpc >out <in &&

I'm not sure how portable that is, though. Maybe using while read -r instead of cat /dev/stdin?

Quoted 6 lines
Quoted 3 lines

If you drop /dev/stdin, the result would be emimently portable. "cat" without any argument reads from the standard input stream and copies it to the standard output stream.

Yep, with that tweak (using "cat" instead of "cat /dev/stdin") this looks like a pleasant enough helper.

Great, I'll reroll like this then. Thanks!

#19 lore raw

PATCH 2/3 upload-pack.c: treat want-ref relative to namespace

In reply to Kim Altintop

When 'upload-pack' runs within the context of a git namespace, treat any 'want-ref' lines the client sends as relative to that namespace.

Also check if the wanted ref is hidden via 'hideRefs'. If it is hidden, respond with an error as if the ref didn't exist.

---
 t/t5703-upload-pack-ref-in-want.sh | 129 +++++++++++++++++++++++++++++
 upload-pack.c                      |  18 ++--
 2 files changed, 140 insertions(+), 7 deletions(-)
t/t5703-upload-pack-ref-in-want.sh +129 -0
index cd4744b016..3e1e0b8fe1 100755
@@ -339,6 +339,135 @@ test_expect_success 'fetching with wildcard that matches multiple refs' '
 	grep "want-ref refs/heads/o/bar" log
 '
 
+REPO="$(pwd)/repo-ns"
+
+test_expect_success 'setup namespaced repo' '
+	(
+		git init -b main "$REPO" &&
+		cd "$REPO" &&
+		test_commit a &&
+		test_commit b &&
+		git checkout a &&
+		test_commit c &&
+		git checkout a &&
+		test_commit d &&
+		git update-ref refs/heads/ns-no b &&
+		git update-ref refs/namespaces/ns/refs/heads/ns-yes c &&
+		git update-ref refs/namespaces/ns/refs/heads/hidden d
+	) &&
+	git -C "$REPO" config uploadpack.allowRefInWant true
+'
+
+test_expect_success 'with namespace: want-ref is considered relative to namespace' '
+	wanted_ref=refs/heads/ns-yes &&
+
+	oid=$(git -C "$REPO" rev-parse "refs/namespaces/ns/$wanted_ref") &&
+	cat >expected_refs <<-EOF &&
+	$oid $wanted_ref
+	EOF
+	cat >expected_commits <<-EOF &&
+	$oid
+	$(git -C "$REPO" rev-parse a)
+	EOF
+
+	test-tool pkt-line pack >in <<-EOF &&
+	$(write_fetch_command $wanted_ref)
+	EOF
+
+	GIT_NAMESPACE=ns test-tool -C "$REPO" serve-v2 --stateless-rpc >out <in &&
+	check_output
+'
+
+test_expect_success 'with namespace: want-ref outside namespace is unknown' '
+	wanted_ref=refs/heads/ns-no &&
+
+	test-tool pkt-line pack >in <<-EOF &&
+	$(write_fetch_command $wanted_ref)
+	EOF
+
+	test_must_fail env GIT_NAMESPACE=ns \
+		test-tool -C "$REPO" serve-v2 --stateless-rpc >out <in &&
+	grep "unknown ref" out
+'
+
+# Cross-check refs/heads/ns-no indeed exists
+test_expect_success 'without namespace: want-ref outside namespace succeeds' '
+	wanted_ref=refs/heads/ns-no &&
+
+	oid=$(git -C "$REPO" rev-parse $wanted_ref) &&
+	cat >expected_refs <<-EOF &&
+	$oid $wanted_ref
+	EOF
+	cat >expected_commits <<-EOF &&
+	$oid
+	$(git -C "$REPO" rev-parse a)
+	EOF
+
+	test-tool pkt-line pack >in <<-EOF &&
+	$(write_fetch_command $wanted_ref)
+	EOF
+
+	test-tool -C "$REPO" serve-v2 --stateless-rpc >out <in &&
+	check_output
+'
+
+test_expect_success 'with namespace: hideRefs is matched, relative to namespace' '
+	wanted_ref=refs/heads/hidden &&
+	git -C "$REPO" config transfer.hideRefs $wanted_ref &&
+
+	test-tool pkt-line pack >in <<-EOF &&
+	$(write_fetch_command $wanted_ref)
+	EOF
+
+	test_must_fail env GIT_NAMESPACE=ns \
+		test-tool -C "$REPO" serve-v2 --stateless-rpc >out <in &&
+	grep "unknown ref" out
+'
+
+# Cross-check refs/heads/hidden indeed exists
+test_expect_success 'with namespace: want-ref succeeds if hideRefs is removed' '
+	wanted_ref=refs/heads/hidden &&
+	git -C "$REPO" config --unset transfer.hideRefs $wanted_ref &&
+
+	oid=$(git -C "$REPO" rev-parse "refs/namespaces/ns/$wanted_ref") &&
+	cat >expected_refs <<-EOF &&
+	$oid $wanted_ref
+	EOF
+	cat >expected_commits <<-EOF &&
+	$oid
+	$(git -C "$REPO" rev-parse a)
+	EOF
+
+	test-tool pkt-line pack >in <<-EOF &&
+	$(write_fetch_command $wanted_ref)
+	EOF
+
+	GIT_NAMESPACE=ns test-tool -C "$REPO" serve-v2 --stateless-rpc >out <in &&
+	check_output
+'
+
+test_expect_success 'without namespace: relative hideRefs does not match' '
+	wanted_ref=refs/namespaces/ns/refs/heads/hidden &&
+	git -C "$REPO" config transfer.hideRefs refs/heads/hidden &&
+
+	oid=$(git -C "$REPO" rev-parse $wanted_ref) &&
+	cat >expected_refs <<-EOF &&
+	$oid $wanted_ref
+	EOF
+	cat >expected_commits <<-EOF &&
+	$oid
+	$(git -C "$REPO" rev-parse a)
+	EOF
+
+	test-tool pkt-line pack >in <<-EOF &&
+	$(write_fetch_command $wanted_ref)
+	EOF
+
+	test-tool -C "$REPO" serve-v2 --stateless-rpc >out <in &&
+	check_output
+'
+
+
 . "$TEST_DIRECTORY"/lib-httpd.sh
 start_httpd
 
upload-pack.c +11 -7
index 297b76fcb4..6ce07231d3 100644
@@ -1417,21 +1417,25 @@ static int parse_want_ref(struct packet_writer *writer, const char *line,
 			  struct string_list *wanted_refs,
 			  struct object_array *want_obj)
 {
-	const char *arg;
-	if (skip_prefix(line, "want-ref ", &arg)) {
+	const char *refname_nons;
+	if (skip_prefix(line, "want-ref ", &refname_nons)) {
 		struct object_id oid;
 		struct string_list_item *item;
 		struct object *o;
+		struct strbuf refname = STRBUF_INIT;
 
-		if (read_ref(arg, &oid)) {
-			packet_writer_error(writer, "unknown ref %s", arg);
-			die("unknown ref %s", arg);
+		strbuf_addf(&refname, "%s%s", get_git_namespace(), refname_nons);
+		if (ref_is_hidden(refname_nons, refname.buf) ||
+		    read_ref(refname.buf, &oid)) {
+			packet_writer_error(writer, "unknown ref %s", refname_nons);
+			die("unknown ref %s", refname_nons);
 		}
+		strbuf_release(&refname);
 
-		item = string_list_append(wanted_refs, arg);
+		item = string_list_append(wanted_refs, refname_nons);
 		item->util = oiddup(&oid);
 
-		o = parse_object_or_die(&oid, arg);
+		o = parse_object_or_die(&oid, refname_nons);
 		if (!(o->flags & WANTED)) {
 			o->flags |= WANTED;
 			add_object_array(o, NULL, want_obj);

--
2.32.0

#20 lore raw

PATCH 3/3 docs: clarify the interaction of transfer.hideRefs and namespaces

In reply to Kim Altintop

Expand the section about namespaces in the documentation of transfer.hideRefs to point out the subtle differences between upload-pack and receive-pack.

9bedd82017 (upload-pack.c: treat want-ref relative to namespace, 2021-07-30) taught upload-pack to reject want-refs for hidden refs, which is now documented.

---
 Documentation/config/transfer.txt | 17 ++++++++++-------
 1 file changed, 10 insertions(+), 7 deletions(-)
Documentation/config/transfer.txt +10 -7
index 505126a780..09ebb399ce 100644
@@ -52,13 +52,16 @@ If you have multiple hideRefs values, later entries override earlier ones
 (and entries in more-specific config files override less-specific ones).
 +
 If a namespace is in use, the namespace prefix is stripped from each
-reference before it is matched against `transfer.hiderefs` patterns.
-For example, if `refs/heads/master` is specified in `transfer.hideRefs` and
-the current namespace is `foo`, then `refs/namespaces/foo/refs/heads/master`
-is omitted from the advertisements but `refs/heads/master` and
-`refs/namespaces/bar/refs/heads/master` are still advertised as so-called
-"have" lines. In order to match refs before stripping, add a `^` in front of
-the ref name. If you combine `!` and `^`, `!` must be specified first.
+reference before it is matched against `transfer.hiderefs` patterns. For
+example, if `refs/heads/master` is specified in `transfer.hideRefs` and the
+current namespace is `foo`, then `refs/namespaces/foo/refs/heads/master` is
+omitted from the advertisements. If `uploadpack.allowRefInWant` is set,
+`upload-pack` will treat `want-ref refs/heads/master` in a protocol v2
+`fetch` command as if `refs/heads/master` was unknown. Note, however, that
+`receive-pack` will still advertise the object id `refs/heads/master` is
+pointing to, but will conceil the name of the ref. In order to match refs
+before stripping, add a `^` in front of the ref name. If you combine `!` and
+`^`, `!` must be specified first.
 +
 Even if you hide refs, a client may still be able to steal the target
 objects via the techniques described in the "SECURITY" section of the

--
2.32.0

#21 lore raw

Re: [PATCH 3/3] docs: clarify the interaction of transfer.hideRefs and namespaces

Quoted 18 lines
Documentation/config/transfer.txt +3 -7
index 505126a780..09ebb399ce 100644
@@ -52,13 +52,16 @@ If you have multiple hideRefs values, later entries override earlier ones
 (and entries in more-specific config files override less-specific ones).
 +
 If a namespace is in use, the namespace prefix is stripped from each
-reference before it is matched against `transfer.hiderefs` patterns.
-For example, if `refs/heads/master` is specified in `transfer.hideRefs` and
-the current namespace is `foo`, then `refs/namespaces/foo/refs/heads/master`
-is omitted from the advertisements but `refs/heads/master` and
-`refs/namespaces/bar/refs/heads/master` are still advertised as so-called
-"have" lines. In order to match refs before stripping, add a `^` in front of
-the ref name. If you combine `!` and `^`, `!` must be specified first.
+reference before it is matched against `transfer.hiderefs` patterns. For
+example, if `refs/heads/master` is specified in `transfer.hideRefs` and the
+current namespace is `foo`, then `refs/namespaces/foo/refs/heads/master` is

Documentation/config/transfer.txt If you have multiple hideRefs values, later entries override earlier ones · line 67

Quoted 10 lines
Documentation/config/transfer.txt +7 -0
+omitted from the advertisements. If `uploadpack.allowRefInWant` is set,
+`upload-pack` will treat `want-ref refs/heads/master` in a protocol v2
+`fetch` command as if `refs/heads/master` was unknown. Note, however, that
+`receive-pack` will still advertise the object id `refs/heads/master` is
+pointing to, but will conceil the name of the ref. In order to match refs
+before stripping, add a `^` in front of the ref name. If you combine `!` and
+`^`, `!` must be specified first.
 +
 Even if you hide refs, a client may still be able to steal the target
 objects via the techniques described in the "SECURITY" section of the

I'd appreciate some feedback on this one before rerolling. Having looked at the code many times recently, it makes sense to me, but that could be different for someone with less fresh memory. Thanks!

#22 lore raw

v6 cover upload-pack: treat want-ref relative to namespace

In reply to Kim Altintop

Round six:

* simplifies the test helper in 1/3 as per review discussion
* rephrases the doc change in 3/3 to make it read less dense
* 2/3 only updates the tests to use the revised helper

CC'ing Johannes Schindelin as suggested by git-contacts.

Published-As: https://github.com/kim/git/tree/ka/namespaced-want-ref-v6

Kim Altintop (3):
t5730: introduce fetch command helper
upload-pack.c: treat want-ref relative to namespace
docs: clarify the interaction of transfer.hideRefs and namespaces

 Documentation/config/transfer.txt  |  14 +-
 t/t5703-upload-pack-ref-in-want.sh | 208 ++++++++++++++++++++++++-----
 upload-pack.c                      |  18 ++-
 3 files changed, 192 insertions(+), 48 deletions(-)

--
2.32.0

#23 lore raw

Re: [PATCH v6 0/3] upload-pack: treat want-ref relative to namespace

Hi Kim,

Quoted 1 line · Kim Altintop

CC'ing Johannes Schindelin as suggested by git-contacts.

git-contacts wouldn't know that there are better experts on the namespace matter.

My only comment is that I would find the diff to upload-pack.c much easier to parse if the arg variable hadn't been renamed.

Ciao,
Johannes

#24 lore raw

Re: [PATCH v6 0/3] upload-pack: treat want-ref relative to namespace

Quoted 2 lines · Johannes Schindelin

My only comment is that I would find the diff to upload-pack.c much easier to parse if the arg variable hadn't been renamed.

If we only look at the patch text, perhaps.

What is left in the resulting file is what people will be reading in the coming years, and because instead of just one string (whose name did not matter and a short-and-sweet arg was a perfect name for it), we now have to deal with two strings (one is the full and real name, and the other is the name without namespace prefix) we need to be clear which is which, the rename is an essential part of the patch to keep the result easier to understand.

When I apply a patch to my tree to review it (e-mailed patch review is merely an efficient way to transmit the change---I do not expect that I can always review all non-trivial patches without stepping out of my MUA). My review, after noticing typos, style glitches and design issues, of code and its correctness will begin after that.

"git show" can be driven with different options and in some cases use of certaion options helps seeing what is going on quite well. "--diff-algorithm" shifts which preimage lines match which postimage lines and sometimes helps reduce the clutter greatly. "--word-diff" is a great tool to see where in reflowed documentation patch actual changes are. "--color-moved" lets reviewer skip pure code movement, various options to "ignore space" sometimes helps declutter the patch. etc.

I wonder if we already have an option (or if there isn't, if we can design such a new option cleanly), that would help in this case. Essentially, what you want is an "I know what used to be arg is renamed to refname_nons in this patch; take advantage of that knowledge and show a diff that is less cluttered" option.

Thanks for a comment, anyway.

#25 lore raw

Re: [PATCH v6 0/3] upload-pack: treat want-ref relative to namespace

In reply to Johannes Schindelin

Hi Johannes,

thanks for your response. This is my very first patch to git.git, and a lot of it is learning about the conventions and expectations towards contributors, as well as getting familiar with the tooling while trying to avoid silly mistakes. So please bear with me.

Quoted 4 lines · On Sat, 14 Aug 2021 Johannes Schindelin wrote:
Quoted 1 line

CC'ing Johannes Schindelin as suggested by git-contacts.

git-contacts wouldn't know that there are better experts on the namespace matter.

I can see now that this could come across weird. I should've written:

"CC'ing Johannes Schindelin, who started to turn up in git-contacts output, although I couldn't quite infer why. I haven't received any feedback about the documentation change yet, and didn't have much success trying to find reviewers by inspecting the history (parts of the file where moved). I am assuming that git-contacts is better than me at this, and Johannes' name shows up because of touching the documentation. Johannes: feel free to ignore if this assumption is wrong."

With this said, if you have any suggestions about finding reviewers for specific parts of a patch, or who are the experts on a more cross-cutting topic, I would appreciate if you'd share them!

Quoted 2 lines

My only comment is that I would find the diff to upload-pack.c much easier to parse if the arg variable hadn't been renamed.

Can you explain why? Just because the diff would be smaller? I can see that in a larger patch it might have been preferable to put the rename into a separate commit, but in a hunk-sized change it seemed fine. It is also that this particular naming ("refname_nons") is used in other places in upload-pack.c, so it seemed obvious that, if I introduce namespace handling where it was previously missing, the terminology (if you will) should be the same.

From you comment, it seems like the proposer of a patch should assume that the reviewers only look at the diff as sent in the email, and not any context. Junio's response suggests something else, but I guess it's fair that if someone feels like they got CC'ed by mistake, they're not going to spend too much time.

So my question from above stands: are there better ways to find the right people to CC, especially for newbies?

Thanks,
Kim

#26 lore raw

Re: [PATCH v6 0/3] upload-pack: treat want-ref relative to namespace

Hi Kim,

Quoted 6 lines · Kim Altintop
Quoted 2 lines · On Sat, 14 Aug 2021 Johannes Schindelin wrote:

My only comment is that I would find the diff to upload-pack.c much easier to parse if the arg variable hadn't been renamed.

Can you explain why?

Yes. I prefer patches to be really obvious. That way, it is really easy to spot bugs.

In this instance, the same patch that introduces a conditional block _also_ renames an involved variable.

To satisfy myself that the patch does what is intended, I therefore have to virtually split the patch into the rename part and the added-conditional-block part.

It would be easier for me if the modifications were presented as two separate patches. And I could imagine that I am not alone in this: you yourself might also have an easier time looking at the commits in six months from now if those concerns are separated into their own commit.

Quoted 8 lines

Just because the diff would be smaller? I can see that in a larger patch it might have been preferable to put the rename into a separate commit, but in a hunk-sized change it seemed fine. It is also that this particular naming ("refname_nons") is used in other places in upload-pack.c, so it seemed obvious that, if I introduce namespace handling where it was previously missing, the terminology (if you will) should be the same.

Quoted 5 lines

From you comment, it seems like the proposer of a patch should assume that the reviewers only look at the diff as sent in the email, and not any context. Junio's response suggests something else, but I guess it's fair that if someone feels like they got CC'ed by mistake, they're not going to spend too much time.

In this instance, I indeed did not spend more time than on reviewing the patch, simply because I am (currently, at least) not all that familiar with the upload-pack.c machinery. I probably touched it in the past, but for the moment, all I can comment on is the shape of the patch series, which is what I did.

Quoted 2 lines

So my question from above stands: are there better ways to find the right people to CC, especially for newbies?

When I look for reviewers in projects other than the ones where I know the usual reviewers' special areas of interest, I like to pick a function at the center of my contribution, then look at git log -L :<function>:<file> and try to figure out who was the last person to implement non-stylistic changes on that function. This has worked relatively well for me, in the past. Maybe it can help you here, too?

Ciao,
Dscho

#27 lore raw

PATCH v6 1/3 t5730: introduce fetch command helper

In reply to Kim Altintop

Assembling a "raw" fetch command to be fed directly to "test-tool serve-v2" is extracted into a test helper.

---
 t/t5703-upload-pack-ref-in-want.sh | 73 +++++++++++++++---------------
 1 file changed, 37 insertions(+), 36 deletions(-)
t/t5703-upload-pack-ref-in-want.sh +37 -36
index e9e471621d..3dad21ff45 100755
@@ -40,6 +40,30 @@ write_command () {
 	fi
 }
 
+# Write a complete fetch command to stdout, suitable for use with `test-tool
+# pkt-line`. "want-ref", "want", and "have" lines are read from stdin.
+#
+# Examples:
+#
+# write_fetch_command <<-EOF
+# want-ref refs/heads/main
+# have $(git rev-parse a)
+# EOF
+#
+# write_fetch_command <<-EOF
+# want $(git rev-parse b)
+# have $(git rev-parse a)
+# EOF
+#
+write_fetch_command () {
+	write_command fetch &&
+	echo "0001" &&
+	echo "no-progress" &&
+	cat &&
+	echo "done" &&
+	echo "0000"
+}
+
 # c(o/foo) d(o/bar)
 #        \ /
 #         b   e(baz)  f(main)
@@ -77,15 +101,11 @@ test_expect_success 'config controls ref-in-want advertisement' '
 '
 
 test_expect_success 'invalid want-ref line' '
-	test-tool pkt-line pack >in <<-EOF &&
-	$(write_command fetch)
-	0001
-	no-progress
+	write_fetch_command >pkt <<-EOF &&
 	want-ref refs/heads/non-existent
-	done
-	0000
 	EOF
 
+	test-tool pkt-line pack <pkt >in &&
 	test_must_fail test-tool serve-v2 --stateless-rpc 2>out <in &&
 	grep "unknown ref" out
 '
@@ -97,16 +117,11 @@ test_expect_success 'basic want-ref' '
 	EOF
 	git rev-parse f >expected_commits &&
 
-	oid=$(git rev-parse a) &&
-	test-tool pkt-line pack >in <<-EOF &&
-	$(write_command fetch)
-	0001
-	no-progress
+	write_fetch_command >pkt <<-EOF &&
 	want-ref refs/heads/main
-	have $oid
-	done
-	0000
+	have $(git rev-parse a)
 	EOF
+	test-tool pkt-line pack <pkt >in &&
 
 	test-tool serve-v2 --stateless-rpc >out <in &&
 	check_output
@@ -121,17 +136,12 @@ test_expect_success 'multiple want-ref lines' '
 	EOF
 	git rev-parse c d >expected_commits &&
 
-	oid=$(git rev-parse b) &&
-	test-tool pkt-line pack >in <<-EOF &&
-	$(write_command fetch)
-	0001
-	no-progress
+	write_fetch_command >pkt <<-EOF &&
 	want-ref refs/heads/o/foo
 	want-ref refs/heads/o/bar
-	have $oid
-	done
-	0000
+	have $(git rev-parse b)
 	EOF
+	test-tool pkt-line pack <pkt >in &&
 
 	test-tool serve-v2 --stateless-rpc >out <in &&
 	check_output
@@ -144,16 +154,12 @@ test_expect_success 'mix want and want-ref' '
 	EOF
 	git rev-parse e f >expected_commits &&
 
-	test-tool pkt-line pack >in <<-EOF &&
-	$(write_command fetch)
-	0001
-	no-progress
+	write_fetch_command >pkt <<-EOF &&
 	want-ref refs/heads/main
 	want $(git rev-parse e)
 	have $(git rev-parse a)
-	done
-	0000
 	EOF
+	test-tool pkt-line pack <pkt >in &&
 
 	test-tool serve-v2 --stateless-rpc >out <in &&
 	check_output
@@ -166,16 +172,11 @@ test_expect_success 'want-ref with ref we already have commit for' '
 	EOF
 	>expected_commits &&
 
-	oid=$(git rev-parse c) &&
-	test-tool pkt-line pack >in <<-EOF &&
-	$(write_command fetch)
-	0001
-	no-progress
+	write_fetch_command >pkt <<-EOF &&
 	want-ref refs/heads/o/foo
-	have $oid
-	done
-	0000
+	have $(git rev-parse c)
 	EOF
+	test-tool pkt-line pack <pkt >in &&
 
 	test-tool serve-v2 --stateless-rpc >out <in &&
 	check_output

--
2.32.0

#28 lore raw

PATCH v6 2/3 upload-pack.c: treat want-ref relative to namespace

In reply to Kim Altintop

When 'upload-pack' runs within the context of a git namespace, treat any 'want-ref' lines the client sends as relative to that namespace.

Also check if the wanted ref is hidden via 'hideRefs'. If it is hidden, respond with an error as if the ref didn't exist.

---
 t/t5703-upload-pack-ref-in-want.sh | 135 +++++++++++++++++++++++++++++
 upload-pack.c                      |  18 ++--
 2 files changed, 146 insertions(+), 7 deletions(-)
t/t5703-upload-pack-ref-in-want.sh +135 -0
index 3dad21ff45..220098523a 100755
@@ -299,6 +299,141 @@ test_expect_success 'fetching with wildcard that matches multiple refs' '
 	grep "want-ref refs/heads/o/bar" log
 '
 
+REPO="$(pwd)/repo-ns"
+
+test_expect_success 'setup namespaced repo' '
+	(
+		git init -b main "$REPO" &&
+		cd "$REPO" &&
+		test_commit a &&
+		test_commit b &&
+		git checkout a &&
+		test_commit c &&
+		git checkout a &&
+		test_commit d &&
+		git update-ref refs/heads/ns-no b &&
+		git update-ref refs/namespaces/ns/refs/heads/ns-yes c &&
+		git update-ref refs/namespaces/ns/refs/heads/hidden d
+	) &&
+	git -C "$REPO" config uploadpack.allowRefInWant true
+'
+
+test_expect_success 'with namespace: want-ref is considered relative to namespace' '
+	wanted_ref=refs/heads/ns-yes &&
+
+	oid=$(git -C "$REPO" rev-parse "refs/namespaces/ns/$wanted_ref") &&
+	cat >expected_refs <<-EOF &&
+	$oid $wanted_ref
+	EOF
+	cat >expected_commits <<-EOF &&
+	$oid
+	$(git -C "$REPO" rev-parse a)
+	EOF
+
+	write_fetch_command >pkt <<-EOF &&
+	want-ref $wanted_ref
+	EOF
+	test-tool pkt-line pack <pkt >in &&
+
+	GIT_NAMESPACE=ns test-tool -C "$REPO" serve-v2 --stateless-rpc >out <in &&
+	check_output
+'
+
+test_expect_success 'with namespace: want-ref outside namespace is unknown' '
+	wanted_ref=refs/heads/ns-no &&
+
+	write_fetch_command >pkt <<-EOF &&
+	want-ref $wanted_ref
+	EOF
+	test-tool pkt-line pack <pkt >in &&
+
+	test_must_fail env GIT_NAMESPACE=ns \
+		test-tool -C "$REPO" serve-v2 --stateless-rpc >out <in &&
+	grep "unknown ref" out
+'
+
+# Cross-check refs/heads/ns-no indeed exists
+test_expect_success 'without namespace: want-ref outside namespace succeeds' '
+	wanted_ref=refs/heads/ns-no &&
+
+	oid=$(git -C "$REPO" rev-parse $wanted_ref) &&
+	cat >expected_refs <<-EOF &&
+	$oid $wanted_ref
+	EOF
+	cat >expected_commits <<-EOF &&
+	$oid
+	$(git -C "$REPO" rev-parse a)
+	EOF
+
+	write_fetch_command >pkt <<-EOF &&
+	want-ref $wanted_ref
+	EOF
+	test-tool pkt-line pack <pkt >in &&
+
+	test-tool -C "$REPO" serve-v2 --stateless-rpc >out <in &&
+	check_output
+'
+
+test_expect_success 'with namespace: hideRefs is matched, relative to namespace' '
+	wanted_ref=refs/heads/hidden &&
+	git -C "$REPO" config transfer.hideRefs $wanted_ref &&
+
+	write_fetch_command >pkt <<-EOF &&
+	want-ref $wanted_ref
+	EOF
+	test-tool pkt-line pack <pkt >in &&
+
+	test_must_fail env GIT_NAMESPACE=ns \
+		test-tool -C "$REPO" serve-v2 --stateless-rpc >out <in &&
+	grep "unknown ref" out
+'
+
+# Cross-check refs/heads/hidden indeed exists
+test_expect_success 'with namespace: want-ref succeeds if hideRefs is removed' '
+	wanted_ref=refs/heads/hidden &&
+	git -C "$REPO" config --unset transfer.hideRefs $wanted_ref &&
+
+	oid=$(git -C "$REPO" rev-parse "refs/namespaces/ns/$wanted_ref") &&
+	cat >expected_refs <<-EOF &&
+	$oid $wanted_ref
+	EOF
+	cat >expected_commits <<-EOF &&
+	$oid
+	$(git -C "$REPO" rev-parse a)
+	EOF
+
+	write_fetch_command >pkt <<-EOF &&
+	want-ref $wanted_ref
+	EOF
+	test-tool pkt-line pack <pkt >in &&
+
+	GIT_NAMESPACE=ns test-tool -C "$REPO" serve-v2 --stateless-rpc >out <in &&
+	check_output
+'
+
+test_expect_success 'without namespace: relative hideRefs does not match' '
+	wanted_ref=refs/namespaces/ns/refs/heads/hidden &&
+	git -C "$REPO" config transfer.hideRefs refs/heads/hidden &&
+
+	oid=$(git -C "$REPO" rev-parse $wanted_ref) &&
+	cat >expected_refs <<-EOF &&
+	$oid $wanted_ref
+	EOF
+	cat >expected_commits <<-EOF &&
+	$oid
+	$(git -C "$REPO" rev-parse a)
+	EOF
+
+	write_fetch_command >pkt <<-EOF &&
+	want-ref $wanted_ref
+	EOF
+	test-tool pkt-line pack <pkt >in &&
+
+	test-tool -C "$REPO" serve-v2 --stateless-rpc >out <in &&
+	check_output
+'
+
+
 . "$TEST_DIRECTORY"/lib-httpd.sh
 start_httpd
 
upload-pack.c +11 -7
index 297b76fcb4..6ce07231d3 100644
@@ -1417,21 +1417,25 @@ static int parse_want_ref(struct packet_writer *writer, const char *line,
 			  struct string_list *wanted_refs,
 			  struct object_array *want_obj)
 {
-	const char *arg;
-	if (skip_prefix(line, "want-ref ", &arg)) {
+	const char *refname_nons;
+	if (skip_prefix(line, "want-ref ", &refname_nons)) {
 		struct object_id oid;
 		struct string_list_item *item;
 		struct object *o;
+		struct strbuf refname = STRBUF_INIT;
 
-		if (read_ref(arg, &oid)) {
-			packet_writer_error(writer, "unknown ref %s", arg);
-			die("unknown ref %s", arg);
+		strbuf_addf(&refname, "%s%s", get_git_namespace(), refname_nons);
+		if (ref_is_hidden(refname_nons, refname.buf) ||
+		    read_ref(refname.buf, &oid)) {
+			packet_writer_error(writer, "unknown ref %s", refname_nons);
+			die("unknown ref %s", refname_nons);
 		}
+		strbuf_release(&refname);
 
-		item = string_list_append(wanted_refs, arg);
+		item = string_list_append(wanted_refs, refname_nons);
 		item->util = oiddup(&oid);
 
-		o = parse_object_or_die(&oid, arg);
+		o = parse_object_or_die(&oid, refname_nons);
 		if (!(o->flags & WANTED)) {
 			o->flags |= WANTED;
 			add_object_array(o, NULL, want_obj);

--
2.32.0

#29 lore raw

PATCH v6 3/3 docs: clarify the interaction of transfer.hideRefs and namespaces

In reply to Kim Altintop

Expand the section about namespaces in the documentation of transfer.hideRefs to point out the subtle differences between upload-pack and receive-pack.

ffcfb68176 (upload-pack.c: treat want-ref relative to namespace, 2021-07-30) taught upload-pack to reject want-refs for hidden refs, which is now mentioned. It is clarified that at no point the name of a hidden ref is revealed, but the object id it points to may.

---
 Documentation/config/transfer.txt | 14 +++++++++-----
 1 file changed, 9 insertions(+), 5 deletions(-)
Documentation/config/transfer.txt +9 -5
index 505126a780..b49429eb4d 100644
@@ -52,13 +52,17 @@ If you have multiple hideRefs values, later entries override earlier ones
 (and entries in more-specific config files override less-specific ones).
 +
 If a namespace is in use, the namespace prefix is stripped from each
-reference before it is matched against `transfer.hiderefs` patterns.
+reference before it is matched against `transfer.hiderefs` patterns. In
+order to match refs before stripping, add a `^` in front of the ref name. If
+you combine `!` and `^`, `!` must be specified first.
++
 For example, if `refs/heads/master` is specified in `transfer.hideRefs` and
 the current namespace is `foo`, then `refs/namespaces/foo/refs/heads/master`
-is omitted from the advertisements but `refs/heads/master` and
-`refs/namespaces/bar/refs/heads/master` are still advertised as so-called
-"have" lines. In order to match refs before stripping, add a `^` in front of
-the ref name. If you combine `!` and `^`, `!` must be specified first.
+is omitted from the advertisements. If `uploadpack.allowRefInWant` is set,
+`upload-pack` will treat `want-ref refs/heads/master` in a protocol v2
+`fetch` command as if `refs/namespaces/foo/refs/heads/master` did not exist.
+`receive-pack`, on the other hand, will still advertise the object id the
+ref is pointing to without mentioning its name (a so-called ".have" line).
 +
 Even if you hide refs, a client may still be able to steal the target
 objects via the techniques described in the "SECURITY" section of the

--
2.32.0

#30 lore raw
In reply to Kim Altintop

t/t5703-upload-pack-ref-in-want.sh test_expect_success 'fetching with wildcard that matches multiple refs' ' · line 303

Quoted 11 lines · Kim Altintop
t/t5703-upload-pack-ref-in-want.sh +3 -0
index e9e471621d..4a828042bb 100755
@@ -298,6 +298,134 @@ test_expect_success 'fetching with wildcard that matches multiple refs' '
 	grep "want-ref refs/heads/o/bar" log
 '
 
+REPO="$(pwd)/repo-ns"
+
+write_fetch_want_ref() {

Style; missing SP before ().

t/t5703-upload-pack-ref-in-want.sh test_expect_success 'fetching with wildcard that matches multiple refs' ' · line 312

Quoted 9 lines
t/t5703-upload-pack-ref-in-want.sh +9 -0
+	local wanted_ref="$1"
+
+	write_command fetch
+	echo "0001"
+	echo "no-progress"
+	echo "want-ref $1"
+	echo "done"
+	echo "0000"
+}

All other helper shell functions in this file are defined near the top. If we compare them with this, we notice that this does not check for any errors. We should string these steps together with "&&".

Also, having this in the middle looks a bit out of place. If this helper is useful only in the tests that are being added by this patch, that may be OK, but we may want to have a preliminary clean-up step before the main patch that adds this helper function near the top (perhaps after "write_command" is defined), and use it in existing tests. It seems that 'invalid want-ref line', 'basic want-ref', and 'multiple want-ref lines' tests among others may benefit from a slightly expanded variant, something along the lines of ...

    write_fetch_command () {
	write_command fetch &&
	echo "0001" &&
	echo "no-progress" || return
        # want-ref ...
	# --
	# want ...
	# --
	# have ...
	# --
        while :
	do
		case $# in 0) break ;; esac &&
		case "$1" in --) shift; break ;; esac &&
		echo "want-ref $1" &&
		shift || return
	done &&
        while :
	do
		case $# in 0) break ;; esac &&
		case "$1" in --) shift; break ;; esac &&
		echo "want $1" &&
		shift || return
	done &&
        while :
	do
		case $# in 0) break ;; esac &&
		case "$1" in --) shift; break ;; esac &&
		echo "have $1" &&
		shift || return
	done &&
	echo "done" &&
	echo "0000"
    }

and with something like that, an existing test like this one:

test_expect_success 'mix want and want-ref' '

	oid=$(git rev-parse f) &&
	cat >expected_refs <<-EOF &&
	$oid refs/heads/main
	EOF
	git rev-parse e f >expected_commits &&

	test-tool pkt-line pack >in <<-EOF &&
	$(write_command fetch)
	0001
	no-progress
	want-ref refs/heads/main
	want $(git rev-parse e)
	have $(git rev-parse a)
	done
	0000
	EOF

	test-tool serve-v2 --stateless-rpc >out <in &&
	check_output

'

may become

test_expect_success 'mix want and want-ref' '

	oid=$(git rev-parse f) &&
	cat >expected_refs <<-EOF &&
	$oid refs/heads/main
	EOF
	git rev-parse e f >expected_commits &&

	test-tool pkt-line pack >in <<-EOF &&
	$(write_fetch_command
		refs/heads/main
		--
		$(git rev-parse e)
		--
		$(git rev-parse a)
	)
	EOF

	test-tool serve-v2 --stateless-rpc >out <in &&
	check_output

'

And after preparing a reusable helper like that, we can add namespace tests using the same helper in the main patch (so it would become an at least 2-patch series).

Quoted 6 lines
t/t5703-upload-pack-ref-in-want.sh +6 -0
+
+test_expect_success 'setup namespaced repo' '
+	(
+		git init -b main "$REPO" &&
+		cd "$REPO" &&
+		test_commit a &&

t/t5703-upload-pack-ref-in-want.sh test_expect_success 'fetching with wildcard that matches multiple refs' ' · line 328

Quoted 10 lines
t/t5703-upload-pack-ref-in-want.sh +10 -0
+		test_commit b &&
+		git checkout a &&
+		test_commit c &&
+		git checkout a &&
+		test_commit d &&
+		git update-ref refs/heads/ns-no b &&
+		git update-ref refs/namespaces/ns/refs/heads/ns-yes c &&
+		git update-ref refs/namespaces/ns/refs/heads/hidden d
+	) &&
+	git -C "$REPO" config uploadpack.allowRefInWant true

I do not see a reason why the last step wants to be done outside the subshell, which we would be using anyway. Does it clarify something?

Quoted 4 lines
t/t5703-upload-pack-ref-in-want.sh +4 -0
+test_expect_success 'with namespace: want-ref is considered relative to namespace' '
+	wanted_ref=refs/heads/ns-yes &&
+
+	oid=$(git -C "$REPO" rev-parse $wanted_ref) &&

t/t5703-upload-pack-ref-in-want.sh test_expect_success 'fetching with wildcard that matches multiple refs' ' · line 344

Quoted 10 lines
t/t5703-upload-pack-ref-in-want.sh +10 -0
+	cat >expected_refs <<-EOF &&
+	$oid $wanted_ref
+	EOF
+	git -C "$REPO" rev-parse a $wanted_ref >expected_commits &&
+
+	test-tool pkt-line pack >in <<-EOF &&
+	$(write_fetch_want_ref $wanted_ref)
+	EOF
+
+	env GIT_NAMESPACE=ns test-tool -C "$REPO" serve-v2 --stateless-rpc >out <in &&

I am not sure why we want "env" in front (it does not hurt, but it should be unnecessary, as test-tool is a plain-vanilla binary executable, not a shell function). Is this a workaround for a buggy test linter or something?

t/t5703-upload-pack-ref-in-want.sh test_expect_success 'fetching with wildcard that matches multiple refs' ' · line 356

Quoted 12 lines
t/t5703-upload-pack-ref-in-want.sh +12 -0
+	check_output
+'
+
+test_expect_success 'with namespace: want-ref outside namespace is unknown' '
+	wanted_ref=refs/heads/ns-no &&
+
+	test-tool pkt-line pack >in <<-EOF &&
+	$(write_fetch_want_ref $wanted_ref)
+	EOF
+
+	test_must_fail env GIT_NAMESPACE=ns \
+		test-tool -C "$REPO" serve-v2 --stateless-rpc >out <in &&

Note that this one does need "env" here.

Quoted 15 lines
upload-pack.c +2 -2
index 297b76fcb4..6ce07231d3 100644
@@ -1417,21 +1417,25 @@ static int parse_want_ref(struct packet_writer *writer, const char *line,
 			  struct string_list *wanted_refs,
 			  struct object_array *want_obj)
 {
-	const char *arg;
-	if (skip_prefix(line, "want-ref ", &arg)) {
+	const char *refname_nons;
+	if (skip_prefix(line, "want-ref ", &refname_nons)) {
 		struct object_id oid;
 		struct string_list_item *item;
 		struct object *o;

upload-pack.c static int parse_want_ref(struct packet_writer *writer, const char *line, · line 1431

Quoted 10 lines
upload-pack.c +6 -3
+		struct strbuf refname = STRBUF_INIT;
 
-		if (read_ref(arg, &oid)) {
-			packet_writer_error(writer, "unknown ref %s", arg);
-			die("unknown ref %s", arg);
+		strbuf_addf(&refname, "%s%s", get_git_namespace(), refname_nons);
+		if (ref_is_hidden(refname_nons, refname.buf) ||
+		    read_ref(refname.buf, &oid)) {
+			packet_writer_error(writer, "unknown ref %s", refname_nons);
+			die("unknown ref %s", refname_nons);

Looks good.

Quoted 14 lines
upload-pack.c +3 -3
 		}
+		strbuf_release(&refname);
 
-		item = string_list_append(wanted_refs, arg);
+		item = string_list_append(wanted_refs, refname_nons);
 		item->util = oiddup(&oid);
 
-		o = parse_object_or_die(&oid, arg);
+		o = parse_object_or_die(&oid, refname_nons);
 		if (!(o->flags & WANTED)) {
 			o->flags |= WANTED;
 			add_object_array(o, NULL, want_obj);
--

2.32.0

#31 lore raw
Quoted 120 lines · Junio C Hamano

t/t5703-upload-pack-ref-in-want.sh test_expect_success 'fetching with wildcard that matches multiple refs' ' · line 303

Quoted 11 lines · Kim Altintop
t/t5703-upload-pack-ref-in-want.sh +3 -0
index e9e471621d..4a828042bb 100755
@@ -298,6 +298,134 @@ test_expect_success 'fetching with wildcard that matches multiple refs' '
 	grep "want-ref refs/heads/o/bar" log
 '
 
+REPO="$(pwd)/repo-ns"
+
+write_fetch_want_ref() {

Style; missing SP before ().

t/t5703-upload-pack-ref-in-want.sh test_expect_success 'fetching with wildcard that matches multiple refs' ' · line 312

Quoted 9 lines
t/t5703-upload-pack-ref-in-want.sh +9 -0
+	local wanted_ref="$1"
+
+	write_command fetch
+	echo "0001"
+	echo "no-progress"
+	echo "want-ref $1"
+	echo "done"
+	echo "0000"
+}

All other helper shell functions in this file are defined near the top. If we compare them with this, we notice that this does not check for any errors. We should string these steps together with "&&".

Also, having this in the middle looks a bit out of place. If this helper is useful only in the tests that are being added by this patch, that may be OK, but we may want to have a preliminary clean-up step before the main patch that adds this helper function near the top (perhaps after "write_command" is defined), and use it in existing tests. It seems that 'invalid want-ref line', 'basic want-ref', and 'multiple want-ref lines' tests among others may benefit from a slightly expanded variant, something along the lines of ...

write_fetch_command () {
write_command fetch &&
echo "0001" &&
echo "no-progress" || return

# want-ref ...
# --
# want ...
# --
# have ...
# --
while :
do
case $# in 0) break ;; esac &&
case "$1" in --) shift; break ;; esac &&
echo "want-ref $1" &&
shift || return
done &&
while :
do
case $# in 0) break ;; esac &&
case "$1" in --) shift; break ;; esac &&
echo "want $1" &&
shift || return
done &&
while :
do
case $# in 0) break ;; esac &&
case "$1" in --) shift; break ;; esac &&
echo "have $1" &&
shift || return
done &&
echo "done" &&
echo "0000"
}

and with something like that, an existing test like this one:

test_expect_success 'mix want and want-ref' '
oid=$(git rev-parse f) &&
cat >expected_refs <<-EOF &&
$oid refs/heads/main
EOF
git rev-parse e f >expected_commits &&

test-tool pkt-line pack >in <<-EOF &&
$(write_command fetch)
0001
no-progress
want-ref refs/heads/main
want $(git rev-parse e)
have $(git rev-parse a)
done
0000
EOF

test-tool serve-v2 --stateless-rpc >out <in &&
check_output
'

may become

test_expect_success 'mix want and want-ref' '
oid=$(git rev-parse f) &&
cat >expected_refs <<-EOF &&
$oid refs/heads/main
EOF
git rev-parse e f >expected_commits &&

test-tool pkt-line pack >in <<-EOF &&
$(write_fetch_command
refs/heads/main
--
$(git rev-parse e)
--
$(git rev-parse a)
)
EOF

Quoted 7 lines

test-tool serve-v2 --stateless-rpc >out <in &&
check_output
'

And after preparing a reusable helper like that, we can add namespace tests using the same helper in the main patch (so it would become an at least 2-patch series).

Ok got it. I did not want to come forward with a grand refactoring of that test in my very first patch, but since reviewers seem to think it's in order I'll give it a spin.

Quoted 11 lines
Quoted 11 lines
t/t5703-upload-pack-ref-in-want.sh +11 -0
+
+test_expect_success 'setup namespaced repo' '
+	(
+		git init -b main "$REPO" &&
+		cd "$REPO" &&
+		test_commit a &&
+		test_commit b &&
+		git checkout a &&
+		test_commit c &&
+		git checkout a &&
+		test_commit d &&
Quoted 8 lines

t/t5703-upload-pack-ref-in-want.sh test_expect_success 'fetching with wildcard that matches multiple refs' ' · line 328

Quoted 5 lines
t/t5703-upload-pack-ref-in-want.sh +5 -0
+		git update-ref refs/heads/ns-no b &&
+		git update-ref refs/namespaces/ns/refs/heads/ns-yes c &&
+		git update-ref refs/namespaces/ns/refs/heads/hidden d
+	) &&
+	git -C "$REPO" config uploadpack.allowRefInWant true

I do not see a reason why the last step wants to be done outside the subshell, which we would be using anyway. Does it clarify something?

I agree, but was sticking to the style that was already there. I'll then revise the other setup steps as part of the refactor.

Quoted 11 lines
Quoted 11 lines
t/t5703-upload-pack-ref-in-want.sh +11 -0
+test_expect_success 'with namespace: want-ref is considered relative to namespace' '
+	wanted_ref=refs/heads/ns-yes &&
+
+	oid=$(git -C "$REPO" rev-parse $wanted_ref) &&
+	cat >expected_refs <<-EOF &&
+	$oid $wanted_ref
+	EOF
+	git -C "$REPO" rev-parse a $wanted_ref >expected_commits &&
+
+	test-tool pkt-line pack >in <<-EOF &&
+	$(write_fetch_want_ref $wanted_ref)
Quoted 8 lines

t/t5703-upload-pack-ref-in-want.sh test_expect_success 'fetching with wildcard that matches multiple refs' ' · line 344

Quoted 3 lines
t/t5703-upload-pack-ref-in-want.sh +3 -0
+	EOF
+
+	env GIT_NAMESPACE=ns test-tool -C "$REPO" serve-v2 --stateless-rpc >out <in &&

I am not sure why we want "env" in front (it does not hurt, but it should be unnecessary, as test-tool is a plain-vanilla binary executable, not a shell function). Is this a workaround for a buggy test linter or something?

The linter did indeed ask me to write GIT_NAMESPACE=ns && export GIT_NAMESPACE && test-tool ... in v1 of the patch, but now it doesn't... nevermind, I must have held something the wrong way.

#32 lore raw
Quoted 4 lines · Kim Altintop
Quoted 3 lines · Junio C Hamano
Quoted 1 line · Kim Altintop

+ env GIT_NAMESPACE=ns test-tool -C "$REPO" serve-v2 --stateless-rpc >out <in &&

Quoted 8 lines
Quoted 4 lines

I am not sure why we want "env" in front (it does not hurt, but it should be unnecessary, as test-tool is a plain-vanilla binary executable, not a shell function). Is this a workaround for a buggy test linter or something?

The linter did indeed ask me to write GIT_NAMESPACE=ns && export GIT_NAMESPACE && test-tool ... in v1 of the patch, but now it doesn't... nevermind, I must have held something the wrong way.

The linter will complain about unportable:

export FOO=bar

and ask you to write it as:

    FOO=bar && export FOO

which is probably what you encountered.

It will also complain about

FOO=bar some-command

which should be rewritten as:

env FOO=bar some-command

if, and only if, some-command is a shell function. If not a shell function, then:

FOO=bar some-command

is just fine.

#33 lore raw
In reply to Kim Altintop
Quoted 8 lines · Kim Altintop
Quoted 5 lines

t/t5703-upload-pack-ref-in-want.sh test_expect_success 'fetching with wildcard that matches multiple refs' ' · line 328

Quoted 2 lines
t/t5703-upload-pack-ref-in-want.sh +2 -0
+	) &&
+	git -C "$REPO" config uploadpack.allowRefInWant true

I do not see a reason why the last step wants to be done outside the subshell, which we would be using anyway. Does it clarify something?

I agree, but was sticking to the style that was already there. I'll then revise the other setup steps as part of the refactor.

Touching existing tests only for this, unlike the creation of a helper out of existing test to improve reuse, does not improve anything. I was curious if there was a good reason to create in a subshell and configure in a separate step, and "others do the same" is good enough to satisfy the curiousity.

Thanks.

#34 lore raw
In reply to Kim Altintop
Quoted 9 lines · Kim Altintop

...

Quoted 8 lines

test-tool pkt-line pack >in <<-EOF &&
$(write_fetch_command
refs/heads/main
--
$(git rev-parse e)
--
$(git rev-parse a)
)

Sorry, I missed all the necessary backslashes at the end of these lines.