Skip to contentSkip to conversation

transport-helper: enforce atomic in push_refs_with_push

PATCH Superseded Emily Shaffer 8 participants 45 messages Jul 2, 2019 – Jul 27, 2019 Documentation

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 transport-helper: enforce atomic in push_refs_with_push

Teach transport-helper how to notice if skipping a ref during push would violate atomicity on the client side. We notice that a ref would be rejected, and choose not to send it, but don't notice that if the client has asked for --atomic we are violating atomicity if all the other pushes we are sending would succeed. Asking the server end to uphold atomicity wouldn't work here as the server doesn't have any idea that we tried to update a ref that's broken.

The added test-case is a succinct way to reproduce this issue that fails today. The same steps work fine when we aren't using a transport-helper to get to the upstream, i.e. when we've added a local repository as a remote:

git remote add ~/upstream upstream

---
 t/t5541-http-push-smart.sh | 58 ++++++++++++++++++++++++++++++++++++++
 transport-helper.c         |  6 ++++
 transport.c                | 15 +++++++++-
 3 files changed, 78 insertions(+), 1 deletion(-)
t/t5541-http-push-smart.sh +58 -0
index 8ef8763e06..b57f6d480f 100755
@@ -177,6 +177,64 @@ test_expect_success 'push (chunked)' '
 	 test $HEAD = $(git rev-parse --verify HEAD))
 '
 
+test_expect_success 'push --atomic also prevents branch creation' '
+	# Make up/master
+	d=$HTTPD_DOCUMENT_ROOT_PATH/atomic-branches.git &&
+	git init --bare "$d" &&
+	git --git-dir="$d" config http.receivepack true &&
+	up="$HTTPD_URL"/smart/atomic-branches.git &&
+	test_commit atomic1 &&
+	test_commit atomic2 &&
+	git push "$up" master &&
+	# Make master incompatible with up/master
+	git reset --hard HEAD^ &&
+	# Add a new branch
+	git branch atomic &&
+	# --atomic should roll back creation of up/atomic
+	test_must_fail git push --atomic "$up" master atomic &&
+	git ls-remote "$up" >up-remotes &&
+	test_must_fail grep atomic up-remotes
+'
+
+test_expect_success 'push --atomic shows all failed refs' '
+	# Make up/master, up/allrefs
+	d=$HTTPD_DOCUMENT_ROOT_PATH/atomic-failed-refs.git &&
+	git init --bare "$d" &&
+	git --git-dir="$d" config http.receivepack true &&
+	up="$HTTPD_URL"/smart/atomic-failed-refs.git &&
+	test_commit allrefs1 &&
+	test_commit allrefs2 &&
+	git branch allrefs &&
+	git push "$up" master allrefs &&
+	# Make master and allrefs incompatible with up/master, up/allrefs
+	git checkout allrefs &&
+	git reset --hard HEAD^ &&
+	git checkout master &&
+	git reset --hard HEAD^ &&
+	# --atomic should complain about both master and allrefs
+	test_must_fail git push --atomic "$up" master allrefs >&output &&
+	grep master output &&
+	grep allrefs output
+'
+
+test_expect_success 'push --atomic indicates collateral failures' '
+	# Make up/master, up/collateral
+	d=$HTTPD_DOCUMENT_ROOT_PATH/atomic-collateral.git &&
+	git init --bare "$d" &&
+	git --git-dir="$d" config http.receivepack true &&
+	up="$HTTPD_URL"/smart/atomic-collateral.git &&
+	test_commit collateral1 &&
+	test_commit collateral2 &&
+	git branch collateral &&
+	git push "$up" master collateral &&
+	# Make master incompatible with up/master
+	git reset --hard HEAD^ &&
+	# --atomic should mention collateral was OK but failed anyway
+	test_must_fail git push --atomic "$up" master collateral >&output &&
+	grep "master -> master" output &&
+	grep "collateral -> collateral" output
+'
+
 test_expect_success 'push --all can push to empty repo' '
 	d=$HTTPD_DOCUMENT_ROOT_PATH/empty-all.git &&
 	git init --bare "$d" &&
transport-helper.c +6 -0
index c7e17ec9cb..6b05a88faf 100644
@@ -853,6 +853,7 @@ static int push_refs_with_push(struct transport *transport,
 {
 	int force_all = flags & TRANSPORT_PUSH_FORCE;
 	int mirror = flags & TRANSPORT_PUSH_MIRROR;
+	int atomic = flags & TRANSPORT_PUSH_ATOMIC;
 	struct helper_data *data = transport->data;
 	struct strbuf buf = STRBUF_INIT;
 	struct ref *ref;
@@ -872,6 +873,11 @@ static int push_refs_with_push(struct transport *transport,
 		case REF_STATUS_REJECT_NONFASTFORWARD:
 		case REF_STATUS_REJECT_STALE:
 		case REF_STATUS_REJECT_ALREADY_EXISTS:
+			if (atomic) {
+				string_list_clear(&cas_options, 0);
+				return 0;
+			} else
+				continue;
 		case REF_STATUS_UPTODATE:
 			continue;
 		default:
transport.c +14 -1
index f1fcd2c4b0..f4d6b38f9d 100644
@@ -1226,10 +1226,23 @@ int transport_push(struct repository *r,
 		err = push_had_errors(remote_refs);
 		ret = push_ret | err;
 
-		if (!quiet || err)
+		if ((flags & TRANSPORT_PUSH_ATOMIC) && err) {
+			for (struct ref *it = remote_refs; it; it = it->next)
+				switch (it->status) {
+				case REF_STATUS_NONE:
+				case REF_STATUS_UPTODATE:
+				case REF_STATUS_OK:
+					it->status = REF_STATUS_ATOMIC_PUSH_FAILED;
+				default:
+					continue;
+				}
+		}
+
+		if (!quiet || err) {
 			transport_print_push_status(transport->url, remote_refs,
 					verbose | porcelain, porcelain,
 					reject_reasons);
+		}
 
 		if (flags & TRANSPORT_PUSH_SET_UPSTREAM)
 			set_upstreams(transport, remote_refs, pretend);
Signature · 1 line
2.22.0.410.gd8fdbe21b5-goog
#1 lore raw

Hi Emily,

Quoted 7 lines · Emily Shaffer

Teach transport-helper how to notice if skipping a ref during push would violate atomicity on the client side. We notice that a ref would be rejected, and choose not to send it, but don't notice that if the client has asked for --atomic we are violating atomicity if all the other pushes we are sending would succeed. Asking the server end to uphold atomicity wouldn't work here as the server doesn't have any idea that we tried to update a ref that's broken.

Makes sense.

Quoted 19 lines

The added test-case is a succinct way to reproduce this issue that fails today. The same steps work fine when we aren't using a transport-helper to get to the upstream, i.e. when we've added a local repository as a remote:

git remote add ~/upstream upstream

---
 t/t5541-http-push-smart.sh | 58 ++++++++++++++++++++++++++++++++++++++
 transport-helper.c         |  6 ++++
 transport.c                | 15 +++++++++-
 3 files changed, 78 insertions(+), 1 deletion(-)
t/t5541-http-push-smart.sh +0 -0
index 8ef8763e06..b57f6d480f 100755

t/t5541-http-push-smart.sh test_expect_success 'push (chunked)' ' · line 184

Quoted 8 lines
t/t5541-http-push-smart.sh +5 -0
@@ -177,6 +177,64 @@ test_expect_success 'push (chunked)' '
 	 test $HEAD = $(git rev-parse --verify HEAD))
 '
 
+test_expect_success 'push --atomic also prevents branch creation' '
+	# Make up/master
+	d=$HTTPD_DOCUMENT_ROOT_PATH/atomic-branches.git &&
+	git init --bare "$d" &&
+	git --git-dir="$d" config http.receivepack true &&

Why not -C "$d"? And why not test_config?

t/t5541-http-push-smart.sh test_expect_success 'push (chunked)' ' · line 188

Quoted 4 lines
t/t5541-http-push-smart.sh +4 -0
+	up="$HTTPD_URL"/smart/atomic-branches.git &&
+	test_commit atomic1 &&
+	test_commit atomic2 &&
+	git push "$up" master &&

It would be more succinct to do a git clone --bare . "$d" here, instead of a git init --bare and a git push no?

t/t5541-http-push-smart.sh test_expect_success 'push (chunked)' ' · line 196

Quoted 8 lines
t/t5541-http-push-smart.sh +8 -0
+	# Make master incompatible with up/master
+	git reset --hard HEAD^ &&
+	# Add a new branch
+	git branch atomic &&
+	# --atomic should roll back creation of up/atomic
+	test_must_fail git push --atomic "$up" master atomic &&
+	git ls-remote "$up" >up-remotes &&
+	test_must_fail grep atomic up-remotes

Why not test_must_fail git -C "$d" rev-parse refs/heads/atomic?

Quoted 12 lines
t/t5541-http-push-smart.sh +12 -0
+'
+
+test_expect_success 'push --atomic shows all failed refs' '
+	# Make up/master, up/allrefs
+	d=$HTTPD_DOCUMENT_ROOT_PATH/atomic-failed-refs.git &&
+	git init --bare "$d" &&
+	git --git-dir="$d" config http.receivepack true &&
+	up="$HTTPD_URL"/smart/atomic-failed-refs.git &&
+	test_commit allrefs1 &&
+	test_commit allrefs2 &&
+	git branch allrefs &&
+	git push "$up" master allrefs &&

t/t5541-http-push-smart.sh test_expect_success 'push (chunked)' ' · line 218

Quoted 10 lines
t/t5541-http-push-smart.sh +10 -0
+	# Make master and allrefs incompatible with up/master, up/allrefs
+	git checkout allrefs &&
+	git reset --hard HEAD^ &&
+	git checkout master &&
+	git reset --hard HEAD^ &&
+	# --atomic should complain about both master and allrefs
+	test_must_fail git push --atomic "$up" master allrefs >&output &&
+	grep master output &&
+	grep allrefs output
+'

I have the impression that the setup these two new test cases perform are _very_ similar, making it most likely that a combined test case would be more succinct, yet still complete and easily readable.

Quoted 8 lines
t/t5541-http-push-smart.sh +8 -0
+
+test_expect_success 'push --atomic indicates collateral failures' '
+	# Make up/master, up/collateral
+	d=$HTTPD_DOCUMENT_ROOT_PATH/atomic-collateral.git &&
+	git init --bare "$d" &&
+	git --git-dir="$d" config http.receivepack true &&
+	up="$HTTPD_URL"/smart/atomic-collateral.git &&
+	test_commit collateral1 &&

t/t5541-http-push-smart.sh test_expect_success 'push (chunked)' ' · line 236

Quoted 10 lines
t/t5541-http-push-smart.sh +10 -0
+	test_commit collateral2 &&
+	git branch collateral &&
+	git push "$up" master collateral &&
+	# Make master incompatible with up/master
+	git reset --hard HEAD^ &&
+	# --atomic should mention collateral was OK but failed anyway
+	test_must_fail git push --atomic "$up" master collateral >&output &&
+	grep "master -> master" output &&
+	grep "collateral -> collateral" output
+'

Same here.

Quoted 33 lines
t/t5541-http-push-smart.sh +1 -0
+
 test_expect_success 'push --all can push to empty repo' '
 	d=$HTTPD_DOCUMENT_ROOT_PATH/empty-all.git &&
 	git init --bare "$d" &&
transport-helper.c +6 -0
index c7e17ec9cb..6b05a88faf 100644
@@ -853,6 +853,7 @@ static int push_refs_with_push(struct transport *transport,
 {
 	int force_all = flags & TRANSPORT_PUSH_FORCE;
 	int mirror = flags & TRANSPORT_PUSH_MIRROR;
+	int atomic = flags & TRANSPORT_PUSH_ATOMIC;
 	struct helper_data *data = transport->data;
 	struct strbuf buf = STRBUF_INIT;
 	struct ref *ref;
@@ -872,6 +873,11 @@ static int push_refs_with_push(struct transport *transport,
 		case REF_STATUS_REJECT_NONFASTFORWARD:
 		case REF_STATUS_REJECT_STALE:
 		case REF_STATUS_REJECT_ALREADY_EXISTS:
+			if (atomic) {
+				string_list_clear(&cas_options, 0);
+				return 0;
+			} else
+				continue;
 		case REF_STATUS_UPTODATE:
 			continue;
 		default:
transport.c +0 -0
index f1fcd2c4b0..f4d6b38f9d 100644

transport.c int transport_push(struct repository *r, · line 1229

Quoted 5 lines
transport.c +1 -1
@@ -1226,10 +1226,23 @@ int transport_push(struct repository *r,
 		err = push_had_errors(remote_refs);
 		ret = push_ret | err;
 
-		if (!quiet || err)
+		if ((flags & TRANSPORT_PUSH_ATOMIC) && err) {

This looks funny. And it does so only...

transport.c int transport_push(struct repository *r, · line 1241

Quoted 12 lines
transport.c +12 -0
+			for (struct ref *it = remote_refs; it; it = it->next)
+				switch (it->status) {
+				case REF_STATUS_NONE:
+				case REF_STATUS_UPTODATE:
+				case REF_STATUS_OK:
+					it->status = REF_STATUS_ATOMIC_PUSH_FAILED;
+				default:
+					continue;
+				}
+		}
+
+		if (!quiet || err) {

... because a curly was introduced around a single-liner. This adds unnecessary noise to the patch.

This easily distracts reviewers like myself from more important questions such as: why was this conditional switch added before this conditional block, does it intend to influence the printed push status? Ah, yes, of course, even if it->status is changed, it actually modifies the data to which remote_refs points. So yes, this has to be done here.

transport.c int transport_push(struct repository *r,

Quoted 8 lines
transport.c +1 -1
 			transport_print_push_status(transport->url, remote_refs,
 					verbose | porcelain, porcelain,
 					reject_reasons);
+		}
 
 		if (flags & TRANSPORT_PUSH_SET_UPSTREAM)
 			set_upstreams(transport, remote_refs, pretend);
--

Apart from minor nits, I like it. Thanks,
Dscho

Quoted 3 lines

2.22.0.410.gd8fdbe21b5-goog

#2 lore raw
Quoted 10 lines · Johannes Schindelin

t/t5541-http-push-smart.sh test_expect_success 'push (chunked)' ' · line 196

Quoted 8 lines
t/t5541-http-push-smart.sh +8 -0
+	# Make master incompatible with up/master
+	git reset --hard HEAD^ &&
+	# Add a new branch
+	git branch atomic &&
+	# --atomic should roll back creation of up/atomic
+	test_must_fail git push --atomic "$up" master atomic &&
+	git ls-remote "$up" >up-remotes &&
+	test_must_fail grep atomic up-remotes

Why not test_must_fail git -C "$d" rev-parse refs/heads/atomic?

They check different expectations.

Between making sure that the state observable in the resulting repository matches what should have been left and checking what the command says it did to its output, I agree that the former is a more robust way to test the commands.

We obviously could test both, though ;-)

#3 lore raw
In reply to Johannes Schindelin
Quoted 7 lines

t/t5541-http-push-smart.sh test_expect_success 'push (chunked)' ' · line 184

Quoted 5 lines
t/t5541-http-push-smart.sh +5 -0
+test_expect_success 'push --atomic also prevents branch creation' '
+	# Make up/master
+	d=$HTTPD_DOCUMENT_ROOT_PATH/atomic-branches.git &&
+	git init --bare "$d" &&
+	git --git-dir="$d" config http.receivepack true &&

Why not -C "$d"?

The example I had found below the new ones used --git-dir, but yeah, there's no reason not to use -C instead. Changing.

Quoted 1 line

And why not test_config?

Done, didn't know about it and it's not used in the test I referred to while writing this one ('push --all can push to empty repo'). Thanks, I learned something new.

Quoted 8 lines

t/t5541-http-push-smart.sh test_expect_success 'push (chunked)' ' · line 188

Quoted 5 lines
t/t5541-http-push-smart.sh +4 -0
+	up="$HTTPD_URL"/smart/atomic-branches.git &&
+	test_commit atomic1 &&
+	test_commit atomic2 &&
+	git push "$up" master &&

It would be more succinct to do a git clone --bare . "$d" here, instead of a git init --bare and a git push no?

I'm not sure I would say "more succinct." This leaves the test with the same number of lines, but now it says:

Make some commits
Name a Git directory
Clone to the new Git directory
Do some config on the new Git directory
Name a remote URL
Change some commits
...

In my opinion, it's more readable the way it is now:

{Do some setup stuff.}
Name a Git directory
Init it
Config it
Name the remote URL
{Do the test stuff.}
Make some commits
Push some commits
Change some commits
...

I did add another comment to separate "Make 'up'" and "Make up/master", which I hope expresses my intent in organizing it this way.

Quoted 11 lines

t/t5541-http-push-smart.sh test_expect_success 'push (chunked)' ' · line 196

Quoted 9 lines
t/t5541-http-push-smart.sh +8 -0
+	# Make master incompatible with up/master
+	git reset --hard HEAD^ &&
+	# Add a new branch
+	git branch atomic &&
+	# --atomic should roll back creation of up/atomic
+	test_must_fail git push --atomic "$up" master atomic &&
+	git ls-remote "$up" >up-remotes &&
+	test_must_fail grep atomic up-remotes

Why not test_must_fail git -C "$d" rev-parse refs/heads/atomic?

Sure, changed.

Quoted 18 lines
Quoted 18 lines
Quoted 1 line

+'

t/t5541-http-push-smart.sh +17 -0
+
+test_expect_success 'push --atomic shows all failed refs' '
+	# Make up/master, up/allrefs
+	d=$HTTPD_DOCUMENT_ROOT_PATH/atomic-failed-refs.git &&
+	git init --bare "$d" &&
+	git --git-dir="$d" config http.receivepack true &&
+	up="$HTTPD_URL"/smart/atomic-failed-refs.git &&
+	test_commit allrefs1 &&
+	test_commit allrefs2 &&
+	git branch allrefs &&
+	git push "$up" master allrefs &&
+	# Make master and allrefs incompatible with up/master, up/allrefs
+	git checkout allrefs &&
+	git reset --hard HEAD^ &&
+	git checkout master &&
+	git reset --hard HEAD^ &&
+	# --atomic should complain about both master and allrefs
Quoted 8 lines

t/t5541-http-push-smart.sh test_expect_success 'push (chunked)' ' · line 218

Quoted 4 lines
t/t5541-http-push-smart.sh +4 -0
+	test_must_fail git push --atomic "$up" master allrefs >&output &&
+	grep master output &&
+	grep allrefs output
+'

I have the impression that the setup these two new test cases perform are _very_ similar, making it most likely that a combined test case would be more succinct, yet still complete and easily readable.

(Junio replied to this downthread... I have more to ask too.)

Quoted 79 lines
Quoted 9 lines
t/t5541-http-push-smart.sh +8 -0
+
+test_expect_success 'push --atomic indicates collateral failures' '
+	# Make up/master, up/collateral
+	d=$HTTPD_DOCUMENT_ROOT_PATH/atomic-collateral.git &&
+	git init --bare "$d" &&
+	git --git-dir="$d" config http.receivepack true &&
+	up="$HTTPD_URL"/smart/atomic-collateral.git &&
+	test_commit collateral1 &&

t/t5541-http-push-smart.sh test_expect_success 'push (chunked)' ' · line 236

Quoted 10 lines
t/t5541-http-push-smart.sh +10 -0
+	test_commit collateral2 &&
+	git branch collateral &&
+	git push "$up" master collateral &&
+	# Make master incompatible with up/master
+	git reset --hard HEAD^ &&
+	# --atomic should mention collateral was OK but failed anyway
+	test_must_fail git push --atomic "$up" master collateral >&output &&
+	grep "master -> master" output &&
+	grep "collateral -> collateral" output
+'

Same here.

Quoted 33 lines
t/t5541-http-push-smart.sh +1 -0
+
 test_expect_success 'push --all can push to empty repo' '
 	d=$HTTPD_DOCUMENT_ROOT_PATH/empty-all.git &&
 	git init --bare "$d" &&
transport-helper.c +6 -0
index c7e17ec9cb..6b05a88faf 100644
@@ -853,6 +853,7 @@ static int push_refs_with_push(struct transport *transport,
 {
 	int force_all = flags & TRANSPORT_PUSH_FORCE;
 	int mirror = flags & TRANSPORT_PUSH_MIRROR;
+	int atomic = flags & TRANSPORT_PUSH_ATOMIC;
 	struct helper_data *data = transport->data;
 	struct strbuf buf = STRBUF_INIT;
 	struct ref *ref;
@@ -872,6 +873,11 @@ static int push_refs_with_push(struct transport *transport,
 		case REF_STATUS_REJECT_NONFASTFORWARD:
 		case REF_STATUS_REJECT_STALE:
 		case REF_STATUS_REJECT_ALREADY_EXISTS:
+			if (atomic) {
+				string_list_clear(&cas_options, 0);
+				return 0;
+			} else
+				continue;
 		case REF_STATUS_UPTODATE:
 			continue;
 		default:
transport.c +0 -0
index f1fcd2c4b0..f4d6b38f9d 100644

transport.c int transport_push(struct repository *r, · line 1229

Quoted 5 lines
transport.c +1 -1
@@ -1226,10 +1226,23 @@ int transport_push(struct repository *r,
 		err = push_had_errors(remote_refs);
 		ret = push_ret | err;
 
-		if (!quiet || err)
+		if ((flags & TRANSPORT_PUSH_ATOMIC) && err) {

This looks funny. And it does so only...

transport.c int transport_push(struct repository *r, · line 1241

Quoted 12 lines
transport.c +12 -0
+			for (struct ref *it = remote_refs; it; it = it->next)
+				switch (it->status) {
+				case REF_STATUS_NONE:
+				case REF_STATUS_UPTODATE:
+				case REF_STATUS_OK:
+					it->status = REF_STATUS_ATOMIC_PUSH_FAILED;
+				default:
+					continue;
+				}
+		}
+
+		if (!quiet || err) {

... because a curly was introduced around a single-liner. This adds unnecessary noise to the patch.

Quoted 5 lines

This easily distracts reviewers like myself from more important questions such as: why was this conditional switch added before this conditional block, does it intend to influence the printed push status? Ah, yes, of course, even if it->status is changed, it actually modifies the data to which remote_refs points. So yes, this has to be done here.

Oops, I thought I had omitted the new braces when I was staging the changes. Really sorry for the distraction. You're right that it makes the diff look weird.

Quoted 16 lines

transport.c int transport_push(struct repository *r,

Quoted 9 lines
transport.c +1 -1
 			transport_print_push_status(transport->url, remote_refs,
 					verbose | porcelain, porcelain,
 					reject_reasons);
+		}
 
 		if (flags & TRANSPORT_PUSH_SET_UPSTREAM)
 			set_upstreams(transport, remote_refs, pretend);
--

Apart from minor nits, I like it. Thanks,
Dscho

Quoted 3 lines

2.22.0.410.gd8fdbe21b5-goog

#4 lore raw
Quoted 27 lines
Quoted 18 lines
Quoted 18 lines
Quoted 1 line

+'

t/t5541-http-push-smart.sh +17 -0
+
+test_expect_success 'push --atomic shows all failed refs' '
+	# Make up/master, up/allrefs
+	d=$HTTPD_DOCUMENT_ROOT_PATH/atomic-failed-refs.git &&
+	git init --bare "$d" &&
+	git --git-dir="$d" config http.receivepack true &&
+	up="$HTTPD_URL"/smart/atomic-failed-refs.git &&
+	test_commit allrefs1 &&
+	test_commit allrefs2 &&
+	git branch allrefs &&
+	git push "$up" master allrefs &&
+	# Make master and allrefs incompatible with up/master, up/allrefs
+	git checkout allrefs &&
+	git reset --hard HEAD^ &&
+	git checkout master &&
+	git reset --hard HEAD^ &&
+	# --atomic should complain about both master and allrefs
Quoted 8 lines

t/t5541-http-push-smart.sh test_expect_success 'push (chunked)' ' · line 218

Quoted 4 lines
t/t5541-http-push-smart.sh +4 -0
+	test_must_fail git push --atomic "$up" master allrefs >&output &&
+	grep master output &&
+	grep allrefs output
+'

I have the impression that the setup these two new test cases perform are _very_ similar, making it most likely that a combined test case would be more succinct, yet still complete and easily readable.

Quoted 1 line

(Junio replied to this downthread... I have more to ask too.)

Oops, I misremembered which comment he had replied to. Yeah, I'll think about this and reword.

Reroll to come shortly. Thanks, all.

- Emily

#5 lore raw
In reply to Emily Shaffer

Hi Emily,

Quoted 10 lines · Emily Shaffer
Quoted 7 lines

t/t5541-http-push-smart.sh test_expect_success 'push (chunked)' ' · line 188

Quoted 4 lines
t/t5541-http-push-smart.sh +4 -0
+	up="$HTTPD_URL"/smart/atomic-branches.git &&
+	test_commit atomic1 &&
+	test_commit atomic2 &&
+	git push "$up" master &&

It would be more succinct to do a git clone --bare . "$d" here, instead of a git init --bare and a git push no?

I'm not sure I would say "more succinct." This leaves the test with the same number of lines,

No, it does not, as git clone --bare . "$d" does _both_ the initializing and the object transfer.

It only saves one line, of course, but do keep in mind that anybody running into any kind of regression with your test case needs to understand what it does. And from experience I can tell you that reading any test case longer than 5 lines is quite annoying when you actually only care about fixing the regression, and not so much about the wonderful story the test case tells.

So in a sense, I guess I would even suggest to move as much of the setup for your test cases outside, preferably into an initial setup test case that initializes the minimal scenario required by the regression test case.

Thanks,
Dscho

#6 lore raw
Quoted 18 lines · Johannes Schindelin

Hi Emily,

Quoted 10 lines · Emily Shaffer
Quoted 7 lines

t/t5541-http-push-smart.sh test_expect_success 'push (chunked)' ' · line 188

Quoted 4 lines
t/t5541-http-push-smart.sh +4 -0
+	up="$HTTPD_URL"/smart/atomic-branches.git &&
+	test_commit atomic1 &&
+	test_commit atomic2 &&
+	git push "$up" master &&

It would be more succinct to do a git clone --bare . "$d" here, instead of a git init --bare and a git push no?

I'm not sure I would say "more succinct." This leaves the test with the same number of lines,

No, it does not, as git clone --bare . "$d" does _both_ the initializing and the object transfer.

Quoted 6 lines

It only saves one line, of course, but do keep in mind that anybody running into any kind of regression with your test case needs to understand what it does. And from experience I can tell you that reading any test case longer than 5 lines is quite annoying when you actually only care about fixing the regression, and not so much about the wonderful story the test case tells.

I suppose I'm confused, then, as I understood you were asking me to combine my three test cases into one, which naturally makes the test itself more complex and longer to read. Which do you prefer?

Quoted 8 lines

So in a sense, I guess I would even suggest to move as much of the setup for your test cases outside, preferably into an initial setup test case that initializes the minimal scenario required by the regression test case.

Thanks,
Dscho

#7 lore raw

Hi Emily,

Quoted 25 lines · Emily Shaffer
Quoted 16 lines · Johannes Schindelin
Quoted 10 lines · Emily Shaffer
Quoted 7 lines

t/t5541-http-push-smart.sh test_expect_success 'push (chunked)' ' · line 188

Quoted 4 lines
t/t5541-http-push-smart.sh +4 -0
+	up="$HTTPD_URL"/smart/atomic-branches.git &&
+	test_commit atomic1 &&
+	test_commit atomic2 &&
+	git push "$up" master &&

It would be more succinct to do a git clone --bare . "$d" here, instead of a git init --bare and a git push no?

I'm not sure I would say "more succinct." This leaves the test with the same number of lines,

No, it does not, as git clone --bare . "$d" does _both_ the initializing and the object transfer.

Quoted 6 lines

It only saves one line, of course, but do keep in mind that anybody running into any kind of regression with your test case needs to understand what it does. And from experience I can tell you that reading any test case longer than 5 lines is quite annoying when you actually only care about fixing the regression, and not so much about the wonderful story the test case tells.

Quoted 3 lines

I suppose I'm confused, then, as I understood you were asking me to combine my three test cases into one, which naturally makes the test itself more complex and longer to read. Which do you prefer?

If I were tasked with developing this further, I would try to move as much of the setup into the initial test case (if there is already a setup test case; otherwise I would create one). In fact, I would try to reuse as much of the existing setup test case as possible, and only add commands if really necessary. Then, I would try to combine the three test cases in the patch into a single one, structuring it by white space and using comments to clarify what is happening.

In my mind, even just adding an empty line before the comments like "Make master incompatible with up/master" would make it much easier for me to read, were I to analyze a test breakage.

I have to admit that I have a hard time understanding what the intention of those three test cases is because I get confused: where does the set-up end, where is the code that is expected to fail, where are the expectations tested?

Also, I get confused by how similar the test cases look, and have a hard time discerning what the differences are (i.e. what are the interesting bits, where the entropy comes from).

I could imagine that I would have had an easier time reading something like this:

test_expect_success 'push --atomic' '

	: set up two branches, one which will require a force push &&
	git checkout -b fast-forwarding master &&
	test_commit push-atomic &&
	git checkout -b non-fast-forwarding &&

	: now, initialize the bare repository to push to &&
	d=$HTTPD_DOCUMENT_ROOT_PATH/atomic.git &&
	git clone --bare . $d &&

	: modify the two branches and create a third one &&
	git reset --hard HEAD^ &&
	git checkout fast-forwarding &&
	test_commit no-need-to-force &&
	git branch new-branch &&

	: now the atomic push must fail &&
	test_must_fail git push --atomic "$HTTPD_URL"/smart/atomic.git \
		fast-forwarding non-fast-forwarding new-branch 2>err &&

	: verify that new-branch was not pushed &&
	test_must_fail git -C $d rev-parse --verify refs/heads/new-branch &&

	: fast-forwarding should be mentioned even if it would have been OK &&
	grep fast-forwarding err

'

Of course, everybody has their preferences, and their personal style. I do not want you to imitate my style just to "pacify" me. That's not the point of this example.

The point is that I need some structure to walk along, especially when I am a bit annoyed at a test case that shows that I introduced a regression and all I want is to understand as quickly as possible what I did wrong again so that I can fix it and move on.

The point is that I want a regression test case to _not_ distract me from the essential part, ideally I should be able to ignore all the set-up code and deduce from the command-line of the failing command what is going on.

For example, if the test case fails in the line grep fast-forwarding err above, that command-line alone does not tell me anything, it just frustrates me. If there is a comment above that line (which is ideally part of the -x trace, that's why I used : instead of #) that states the intention in what I consider to be clear, simple English, it is a lot easier to figure out what the heck is going wrong.

Of course, it would be even better if we had a function, say, must_contain that runs that grep and shows the file contents and the regular expression if that grep failed. That's of course outside of the concern of your patch. But this idea illustrates what I want in a regression test case: I want it to _help_ me figure out things when it breaks.

Ciao,
Dscho

P.S.: Please note that the many test cases _I_ introduced into Git's test suite mostly do not conform to what I wrote above. I learned _quite_ a lot of how I want regression test cases to look like in the past six months, not from writing them, but from analyzing literally hundreds of Azure Pipelines builds. And your question forced me to think about my learnings to formulate the above (hopefully clear) explanation of what I want in a regression test, so: thank you.

#8 lore raw

Firstly, sorry for the delay, I wasn't working for national holiday from the 4th til yesterday.

Quoted 49 lines

If I were tasked with developing this further, I would try to move as much of the setup into the initial test case (if there is already a setup test case; otherwise I would create one). In fact, I would try to reuse as much of the existing setup test case as possible, and only add commands if really necessary. Then, I would try to combine the three test cases in the patch into a single one, structuring it by white space and using comments to clarify what is happening.

In my mind, even just adding an empty line before the comments like "Make master incompatible with up/master" would make it much easier for me to read, were I to analyze a test breakage.

I have to admit that I have a hard time understanding what the intention of those three test cases is because I get confused: where does the set-up end, where is the code that is expected to fail, where are the expectations tested?

Also, I get confused by how similar the test cases look, and have a hard time discerning what the differences are (i.e. what are the interesting bits, where the entropy comes from).

I could imagine that I would have had an easier time reading something like this:

test_expect_success 'push --atomic' '

	: set up two branches, one which will require a force push &&
	git checkout -b fast-forwarding master &&
	test_commit push-atomic &&
	git checkout -b non-fast-forwarding &&

	: now, initialize the bare repository to push to &&
	d=$HTTPD_DOCUMENT_ROOT_PATH/atomic.git &&
	git clone --bare . $d &&

	: modify the two branches and create a third one &&
	git reset --hard HEAD^ &&
	git checkout fast-forwarding &&
	test_commit no-need-to-force &&
	git branch new-branch &&

	: now the atomic push must fail &&
	test_must_fail git push --atomic "$HTTPD_URL"/smart/atomic.git \
		fast-forwarding non-fast-forwarding new-branch 2>err &&

	: verify that new-branch was not pushed &&
	test_must_fail git -C $d rev-parse --verify refs/heads/new-branch &&

	: fast-forwarding should be mentioned even if it would have been OK &&
	grep fast-forwarding err
Quoted 1 line

'

After I read your initial comment this is close to what my rewrite started to look like; I suppose we're only the same page after all :)

Quoted 31 lines

Of course, everybody has their preferences, and their personal style. I do not want you to imitate my style just to "pacify" me. That's not the point of this example.

The point is that I need some structure to walk along, especially when I am a bit annoyed at a test case that shows that I introduced a regression and all I want is to understand as quickly as possible what I did wrong again so that I can fix it and move on.

The point is that I want a regression test case to _not_ distract me from the essential part, ideally I should be able to ignore all the set-up code and deduce from the command-line of the failing command what is going on.

For example, if the test case fails in the line grep fast-forwarding err above, that command-line alone does not tell me anything, it just frustrates me. If there is a comment above that line (which is ideally part of the -x trace, that's why I used : instead of #) that states the intention in what I consider to be clear, simple English, it is a lot easier to figure out what the heck is going wrong.

Of course, it would be even better if we had a function, say, must_contain that runs that grep and shows the file contents and the regular expression if that grep failed. That's of course outside of the concern of your patch. But this idea illustrates what I want in a regression test case: I want it to _help_ me figure out things when it breaks.

Ciao,
Dscho

Quoted 8 lines

P.S.: Please note that the many test cases _I_ introduced into Git's test suite mostly do not conform to what I wrote above. I learned _quite_ a lot of how I want regression test cases to look like in the past six months, not from writing them, but from analyzing literally hundreds of Azure Pipelines builds. And your question forced me to think about my learnings to formulate the above (hopefully clear) explanation of what I want in a regression test, so: thank you.

This writeup is great, and I think your example is very clear and readable. I'll push a reroll with my similar-looking reformat soon. Thanks, Johannes.

- Emily

#9 lore raw
In reply to Emily Shaffer
Quoted 13 lines · Emily Shaffer
transport-helper.c +1 -0
index c7e17ec9cb..6b05a88faf 100644
@@ -853,6 +853,7 @@ static int push_refs_with_push(struct transport *transport,
 {
 	int force_all = flags & TRANSPORT_PUSH_FORCE;
 	int mirror = flags & TRANSPORT_PUSH_MIRROR;
+	int atomic = flags & TRANSPORT_PUSH_ATOMIC;
 	struct helper_data *data = transport->data;
 	struct strbuf buf = STRBUF_INIT;
 	struct ref *ref;

transport-helper.c static int push_refs_with_push(struct transport *transport, · line 880

Quoted 8 lines
transport-helper.c +5 -0
@@ -872,6 +873,11 @@ static int push_refs_with_push(struct transport *transport,
 		case REF_STATUS_REJECT_NONFASTFORWARD:
 		case REF_STATUS_REJECT_STALE:
 		case REF_STATUS_REJECT_ALREADY_EXISTS:
+			if (atomic) {
+				string_list_clear(&cas_options, 0);
+				return 0;
+			} else
+				continue;

Ah, this looks vaguely familiar. Thanks for resurrecting the topic.

The clearing is merely to avoid leaks, and the primary change to the function is to immediately return 0.

transport.c int transport_push(struct repository *r, · line 1227

Quoted 10 lines
transport-helper.c +0 -0
 		case REF_STATUS_UPTODATE:
 			continue;
 		default:
transport.c +0 -0
index f1fcd2c4b0..f4d6b38f9d 100644
@@ -1226,10 +1226,23 @@ int transport_push(struct repository *r,
 		err = push_had_errors(remote_refs);
 		ret = push_ret | err;

Here, before reporting the push result, when we are doing ATOMIC, we tweak the result we are going to report to atomic-push-failed.

transport.c int transport_push(struct repository *r, · line 1239

Quoted 11 lines
transport.c +11 -0
+		if ((flags & TRANSPORT_PUSH_ATOMIC) && err) {
+			for (struct ref *it = remote_refs; it; it = it->next)
+				switch (it->status) {
+				case REF_STATUS_NONE:
+				case REF_STATUS_UPTODATE:
+				case REF_STATUS_OK:
+					it->status = REF_STATUS_ATOMIC_PUSH_FAILED;
+				default:
+					continue;
+				}
+		}

This roughly corresponds to what send-pack.c::atomic_push_failure() does. Here, we avoid overwriting a status that already signals a failure. The list of "good" statuses used here match what is used at the end of send_pack.c::send_pack(), which decides the final outcome of "git push" for the native transport.

Looks good.

By the way, I rearranged the patch as I happen to agree with Dscho that the additional {} was unwarranted and made it harder to review. It is clear that we need to tweak the status before reporting.

Quoted 8 lines
transport.c +0 -0
 		if (!quiet || err)
 			transport_print_push_status(transport->url, remote_refs,
 					verbose | porcelain, porcelain,
 					reject_reasons);
 
 		if (flags & TRANSPORT_PUSH_SET_UPSTREAM)
 			set_upstreams(transport, remote_refs, pretend);
#10 lore raw
In reply to Emily Shaffer

transport.c int transport_push(struct repository *r, · line 1239

Quoted 11 lines · Emily Shaffer
transport.c +11 -0
+		if ((flags & TRANSPORT_PUSH_ATOMIC) && err) {
+			for (struct ref *it = remote_refs; it; it = it->next)
+				switch (it->status) {
+				case REF_STATUS_NONE:
+				case REF_STATUS_UPTODATE:
+				case REF_STATUS_OK:
+					it->status = REF_STATUS_ATOMIC_PUSH_FAILED;
+				default:
+					continue;
+				}
+		}

Let's write this more like so

		if ((flags & TRANSPORT_PUSH_ATOMIC) && err) {
			for (struct ref *it = remote_refs; it; it = it->next)
				switch (it->status) {
				case REF_STATUS_NONE:
				case REF_STATUS_UPTODATE:
				case REF_STATUS_OK:
					it->status = REF_STATUS_ATOMIC_PUSH_FAILED;
					break;
				default:
					break;
				}
		}

to prevent compilers from giving "implicit fallthru" warnings.

#11 lore raw
Quoted 30 lines · Junio C Hamano

transport.c int transport_push(struct repository *r, · line 1239

Quoted 11 lines · Emily Shaffer
transport.c +11 -0
+		if ((flags & TRANSPORT_PUSH_ATOMIC) && err) {
+			for (struct ref *it = remote_refs; it; it = it->next)
+				switch (it->status) {
+				case REF_STATUS_NONE:
+				case REF_STATUS_UPTODATE:
+				case REF_STATUS_OK:
+					it->status = REF_STATUS_ATOMIC_PUSH_FAILED;
+				default:
+					continue;
+				}
+		}

Let's write this more like so

		if ((flags & TRANSPORT_PUSH_ATOMIC) && err) {
			for (struct ref *it = remote_refs; it; it = it->next)
				switch (it->status) {
				case REF_STATUS_NONE:
				case REF_STATUS_UPTODATE:
				case REF_STATUS_OK:
					it->status = REF_STATUS_ATOMIC_PUSH_FAILED;
					break;
				default:
					break;
				}
		}
Quoted 1 line

to prevent compilers from giving "implicit fallthru" warnings.

Done, thanks.

#12 lore raw
In reply to Emily Shaffer
Quoted 37 lines · Emily Shaffer

Teach transport-helper how to notice if skipping a ref during push would violate atomicity on the client side. We notice that a ref would be rejected, and choose not to send it, but don't notice that if the client has asked for --atomic we are violating atomicity if all the other pushes we are sending would succeed. Asking the server end to uphold atomicity wouldn't work here as the server doesn't have any idea that we tried to update a ref that's broken.

The added test-case is a succinct way to reproduce this issue that fails today. The same steps work fine when we aren't using a transport-helper to get to the upstream, i.e. when we've added a local repository as a remote:

git remote add ~/upstream upstream

---
 t/t5541-http-push-smart.sh | 58 ++++++++++++++++++++++++++++++++++++++
 transport-helper.c         |  6 ++++
 transport.c                | 15 +++++++++-
 3 files changed, 78 insertions(+), 1 deletion(-)
t/t5541-http-push-smart.sh +7 -0
index 8ef8763e06..b57f6d480f 100755
@@ -177,6 +177,64 @@ test_expect_success 'push (chunked)' '
 	 test $HEAD = $(git rev-parse --verify HEAD))
 '
 
+test_expect_success 'push --atomic also prevents branch creation' '
+	# Make up/master
+	d=$HTTPD_DOCUMENT_ROOT_PATH/atomic-branches.git &&
+	git init --bare "$d" &&
+	git --git-dir="$d" config http.receivepack true &&
+	up="$HTTPD_URL"/smart/atomic-branches.git &&
+	test_commit atomic1 &&

t/t5541-http-push-smart.sh test_expect_success 'push (chunked)' ' · line 196

Quoted 10 lines
t/t5541-http-push-smart.sh +10 -0
+	test_commit atomic2 &&
+	git push "$up" master &&
+	# Make master incompatible with up/master
+	git reset --hard HEAD^ &&
+	# Add a new branch
+	git branch atomic &&
+	# --atomic should roll back creation of up/atomic
+	test_must_fail git push --atomic "$up" master atomic &&
+	git ls-remote "$up" >up-remotes &&
+	test_must_fail grep atomic up-remotes

Don't use test_must_fail on non-git things. We are not in the business of catching segfaulting system programs.

Quoted 9 lines
t/t5541-http-push-smart.sh +9 -0
+'
+
+test_expect_success 'push --atomic shows all failed refs' '
+	# Make up/master, up/allrefs
+	d=$HTTPD_DOCUMENT_ROOT_PATH/atomic-failed-refs.git &&
+	git init --bare "$d" &&
+	git --git-dir="$d" config http.receivepack true &&
+	up="$HTTPD_URL"/smart/atomic-failed-refs.git &&
+	test_commit allrefs1 &&

t/t5541-http-push-smart.sh test_expect_success 'push (chunked)' ' · line 215

Quoted 10 lines
t/t5541-http-push-smart.sh +10 -0
+	test_commit allrefs2 &&
+	git branch allrefs &&
+	git push "$up" master allrefs &&
+	# Make master and allrefs incompatible with up/master, up/allrefs
+	git checkout allrefs &&
+	git reset --hard HEAD^ &&
+	git checkout master &&
+	git reset --hard HEAD^ &&
+	# --atomic should complain about both master and allrefs
+	test_must_fail git push --atomic "$up" master allrefs >&output &&

Don't rely on ">&output", which is an unnecessary bash-ism here. It breaks test run under shells like dash.

	>output 2>&1

should be OK.

Quoted 8 lines
t/t5541-http-push-smart.sh +8 -0
+	grep master output &&
+	grep allrefs output
+'
+
+test_expect_success 'push --atomic indicates collateral failures' '
+	# Make up/master, up/collateral
+	d=$HTTPD_DOCUMENT_ROOT_PATH/atomic-collateral.git &&
+	git init --bare "$d" &&

t/t5541-http-push-smart.sh test_expect_success 'push (chunked)' ' · line 233

Quoted 10 lines
t/t5541-http-push-smart.sh +10 -0
+	git --git-dir="$d" config http.receivepack true &&
+	up="$HTTPD_URL"/smart/atomic-collateral.git &&
+	test_commit collateral1 &&
+	test_commit collateral2 &&
+	git branch collateral &&
+	git push "$up" master collateral &&
+	# Make master incompatible with up/master
+	git reset --hard HEAD^ &&
+	# --atomic should mention collateral was OK but failed anyway
+	test_must_fail git push --atomic "$up" master collateral >&output &&

Ditto.

Quoted 4 lines
t/t5541-http-push-smart.sh +4 -0
+	grep "master -> master" output &&
+	grep "collateral -> collateral" output
+'
+
#13 lore raw
Quoted 13 lines · Junio C Hamano
Quoted 13 lines
t/t5541-http-push-smart.sh +12 -0
+test_expect_success 'push --atomic also prevents branch creation' '
+	# Make up/master
+	d=$HTTPD_DOCUMENT_ROOT_PATH/atomic-branches.git &&
+	git init --bare "$d" &&
+	git --git-dir="$d" config http.receivepack true &&
+	up="$HTTPD_URL"/smart/atomic-branches.git &&
+	test_commit atomic1 &&
+	test_commit atomic2 &&
+	git push "$up" master &&
+	# Make master incompatible with up/master
+	git reset --hard HEAD^ &&
+	# Add a new branch
Quoted 8 lines

t/t5541-http-push-smart.sh test_expect_success 'push (chunked)' ' · line 196

Quoted 5 lines
t/t5541-http-push-smart.sh +5 -0
+	git branch atomic &&
+	# --atomic should roll back creation of up/atomic
+	test_must_fail git push --atomic "$up" master atomic &&
+	git ls-remote "$up" >up-remotes &&
+	test_must_fail grep atomic up-remotes

Don't use test_must_fail on non-git things. We are not in the business of catching segfaulting system programs.

Done:
! grep atomic up-remotes

Quoted 24 lines
Quoted 10 lines

+'

t/t5541-http-push-smart.sh +8 -0
+
+test_expect_success 'push --atomic shows all failed refs' '
+	# Make up/master, up/allrefs
+	d=$HTTPD_DOCUMENT_ROOT_PATH/atomic-failed-refs.git &&
+	git init --bare "$d" &&
+	git --git-dir="$d" config http.receivepack true &&
+	up="$HTTPD_URL"/smart/atomic-failed-refs.git &&
+	test_commit allrefs1 &&

t/t5541-http-push-smart.sh test_expect_success 'push (chunked)' ' · line 215

Quoted 10 lines
t/t5541-http-push-smart.sh +10 -0
+	test_commit allrefs2 &&
+	git branch allrefs &&
+	git push "$up" master allrefs &&
+	# Make master and allrefs incompatible with up/master, up/allrefs
+	git checkout allrefs &&
+	git reset --hard HEAD^ &&
+	git checkout master &&
+	git reset --hard HEAD^ &&
+	# --atomic should complain about both master and allrefs
+	test_must_fail git push --atomic "$up" master allrefs >&output &&

Don't rely on ">&output", which is an unnecessary bash-ism here. It breaks test run under shells like dash.

Quoted 3 lines
	>output 2>&1

should be OK.

Done.

Quoted 13 lines
Quoted 13 lines
t/t5541-http-push-smart.sh +12 -0
+	grep master output &&
+	grep allrefs output
+'
+
+test_expect_success 'push --atomic indicates collateral failures' '
+	# Make up/master, up/collateral
+	d=$HTTPD_DOCUMENT_ROOT_PATH/atomic-collateral.git &&
+	git init --bare "$d" &&
+	git --git-dir="$d" config http.receivepack true &&
+	up="$HTTPD_URL"/smart/atomic-collateral.git &&
+	test_commit collateral1 &&
+	test_commit collateral2 &&
Quoted 8 lines

t/t5541-http-push-smart.sh test_expect_success 'push (chunked)' ' · line 233

Quoted 6 lines
t/t5541-http-push-smart.sh +6 -0
+	git branch collateral &&
+	git push "$up" master collateral &&
+	# Make master incompatible with up/master
+	git reset --hard HEAD^ &&
+	# --atomic should mention collateral was OK but failed anyway
+	test_must_fail git push --atomic "$up" master collateral >&output &&

Ditto.

Done.

- Emily

#14 lore raw
In reply to Junio C Hamano
Quoted 21 lines · Junio C Hamano
Quoted 7 lines
t/t5541-http-push-smart.sh +7 -0
+test_expect_success 'push --atomic shows all failed refs' '
+	# Make up/master, up/allrefs
+	d=$HTTPD_DOCUMENT_ROOT_PATH/atomic-failed-refs.git &&
+	git init --bare "$d" &&
+	git --git-dir="$d" config http.receivepack true &&
+	up="$HTTPD_URL"/smart/atomic-failed-refs.git &&
+	test_commit allrefs1 &&

t/t5541-http-push-smart.sh test_expect_success 'push (chunked)' ' · line 215

Quoted 10 lines
t/t5541-http-push-smart.sh +10 -0
+	test_commit allrefs2 &&
+	git branch allrefs &&
+	git push "$up" master allrefs &&
+	# Make master and allrefs incompatible with up/master, up/allrefs
+	git checkout allrefs &&
+	git reset --hard HEAD^ &&
+	git checkout master &&
+	git reset --hard HEAD^ &&
+	# --atomic should complain about both master and allrefs
+	test_must_fail git push --atomic "$up" master allrefs >&output &&

Don't rely on ">&output", which is an unnecessary bash-ism here. It breaks test run under shells like dash.

Quoted 3 lines
	>output 2>&1

should be OK.

'2>output' would be a tad better, because those refs should be printed to stderr.

#15 lore raw
Quoted 22 lines · SZEDER Gábor
Quoted 21 lines · Junio C Hamano
Quoted 7 lines
t/t5541-http-push-smart.sh +7 -0
+test_expect_success 'push --atomic shows all failed refs' '
+	# Make up/master, up/allrefs
+	d=$HTTPD_DOCUMENT_ROOT_PATH/atomic-failed-refs.git &&
+	git init --bare "$d" &&
+	git --git-dir="$d" config http.receivepack true &&
+	up="$HTTPD_URL"/smart/atomic-failed-refs.git &&
+	test_commit allrefs1 &&

t/t5541-http-push-smart.sh test_expect_success 'push (chunked)' ' · line 215

Quoted 10 lines
t/t5541-http-push-smart.sh +10 -0
+	test_commit allrefs2 &&
+	git branch allrefs &&
+	git push "$up" master allrefs &&
+	# Make master and allrefs incompatible with up/master, up/allrefs
+	git checkout allrefs &&
+	git reset --hard HEAD^ &&
+	git checkout master &&
+	git reset --hard HEAD^ &&
+	# --atomic should complain about both master and allrefs
+	test_must_fail git push --atomic "$up" master allrefs >&output &&

Don't rely on ">&output", which is an unnecessary bash-ism here. It breaks test run under shells like dash.

Quoted 6 lines
Quoted 3 lines
	>output 2>&1

should be OK.

'2>output' would be a tad better, because those refs should be printed to stderr.

Yeah; there are many existing uses of ">output 2>&1" in the same script and I was following the suit. There also are 2>err and I agree that it is more appropriate in this case.

#16 lore raw
Quoted 26 lines · Junio C Hamano
Quoted 24 lines · SZEDER Gábor
Quoted 23 lines · Junio C Hamano
Quoted 7 lines
t/t5541-http-push-smart.sh +7 -0
+test_expect_success 'push --atomic shows all failed refs' '
+	# Make up/master, up/allrefs
+	d=$HTTPD_DOCUMENT_ROOT_PATH/atomic-failed-refs.git &&
+	git init --bare "$d" &&
+	git --git-dir="$d" config http.receivepack true &&
+	up="$HTTPD_URL"/smart/atomic-failed-refs.git &&
+	test_commit allrefs1 &&

t/t5541-http-push-smart.sh test_expect_success 'push (chunked)' ' · line 215

Quoted 10 lines
t/t5541-http-push-smart.sh +10 -0
+	test_commit allrefs2 &&
+	git branch allrefs &&
+	git push "$up" master allrefs &&
+	# Make master and allrefs incompatible with up/master, up/allrefs
+	git checkout allrefs &&
+	git reset --hard HEAD^ &&
+	git checkout master &&
+	git reset --hard HEAD^ &&
+	# --atomic should complain about both master and allrefs
+	test_must_fail git push --atomic "$up" master allrefs >&output &&

Don't rely on ">&output", which is an unnecessary bash-ism here. It breaks test run under shells like dash.

	>output 2>&1
Quoted 8 lines
Quoted 4 lines
Quoted 1 line

should be OK.

'2>output' would be a tad better, because those refs should be printed to stderr.

Yeah; there are many existing uses of ">output 2>&1" in the same script and I was following the suit. There also are 2>err and I agree that it is more appropriate in this case.

Oh, this is a good point. I'll change it, thanks both.

#17 lore raw

PATCH v2 1/1 transport-helper: enforce atomic in push_refs_with_push

In reply to Emily Shaffer

Teach transport-helper how to notice if skipping a ref during push would violate atomicity on the client side. We notice that a ref would be rejected, and choose not to send it, but don't notice that if the client has asked for --atomic we are violating atomicity if all the other pushes we are sending would succeed. Asking the server end to uphold atomicity wouldn't work here as the server doesn't have any idea that we tried to update a ref that's broken.

The added test-case is a succinct way to reproduce this issue that fails today. The same steps work fine when we aren't using a transport-helper to get to the upstream, i.e. when we've added a local repository as a remote:

git remote add ~/upstream upstream

---
Since v2, I combined the test cases into one. Unfortunately the new test case is checking a few things at once, but I hope the comments make it clear enough. Johannes, I still left the setup at the top; for me in the tradeoff between "one less line" and "setup happens first", the latter wins.

I also removed the "try pushing two bad branches at the same time" case; I don't think it's really covered by atomicity. So now we check that if we push one bad ref, collateral damage in the form of 1) a new branch and 2) a perfectly good push of another branch are both rejected and reported correctly.

Also, got rid of the accidental new curly braces around unrelated part of the code.

Thanks all.

- Emily

 t/t5541-http-push-smart.sh | 39 ++++++++++++++++++++++++++++++++++++++
 transport-helper.c         |  6 ++++++
 transport.c                | 13 +++++++++++++
 3 files changed, 58 insertions(+)
t/t5541-http-push-smart.sh +39 -0
index 8ef8763e06..7f9ae1ef3f 100755
@@ -177,6 +177,45 @@ test_expect_success 'push (chunked)' '
 	 test $HEAD = $(git rev-parse --verify HEAD))
 '
 
+test_expect_success 'push --atomic also prevents branch creation, reports collateral' '
+	# Setup upstream repo - empty for now
+	d=$HTTPD_DOCUMENT_ROOT_PATH/atomic-branches.git &&
+	git init --bare "$d" &&
+	test_config -C "$d" http.receivepack true &&
+	up="$HTTPD_URL"/smart/atomic-branches.git &&
+
+	# Tell up about two branches for now
+	test_commit atomic1 &&
+	test_commit atomic2 &&
+	git branch collateral &&
+	git push "$up" master collateral &&
+
+	# collateral is a valid push, but should be failed by atomic push
+	git checkout collateral &&
+	test_commit collateral1 &&
+
+	# Make master incompatible with upstream to provoke atomic
+	git checkout master &&
+	git reset --hard HEAD^ &&
+
+	# Add a new branch which should be failed by atomic push. This is a
+	# regression case.
+	git branch atomic &&
+
+	# --atomic should cause entire push to be rejected
+	test_must_fail git push --atomic "$up" master atomic collateral 2>output &&
+
+	# the new branch should not have been created upstream
+	test_must_fail git -C "$d" rev-parse refs/heads/atomic &&
+
+	# the failed refs should be indicated
+	grep "master -> master" output | grep rejected &&
+
+	# the collateral failure refs should be indicated
+	grep "atomic -> atomic" output | grep "atomic push failed" &&
+	grep "collateral -> collateral" output | grep "atomic push failed"
+'
+
 test_expect_success 'push --all can push to empty repo' '
 	d=$HTTPD_DOCUMENT_ROOT_PATH/empty-all.git &&
 	git init --bare "$d" &&
transport-helper.c +6 -0
index c7e17ec9cb..6b05a88faf 100644
@@ -853,6 +853,7 @@ static int push_refs_with_push(struct transport *transport,
 {
 	int force_all = flags & TRANSPORT_PUSH_FORCE;
 	int mirror = flags & TRANSPORT_PUSH_MIRROR;
+	int atomic = flags & TRANSPORT_PUSH_ATOMIC;
 	struct helper_data *data = transport->data;
 	struct strbuf buf = STRBUF_INIT;
 	struct ref *ref;
@@ -872,6 +873,11 @@ static int push_refs_with_push(struct transport *transport,
 		case REF_STATUS_REJECT_NONFASTFORWARD:
 		case REF_STATUS_REJECT_STALE:
 		case REF_STATUS_REJECT_ALREADY_EXISTS:
+			if (atomic) {
+				string_list_clear(&cas_options, 0);
+				return 0;
+			} else
+				continue;
 		case REF_STATUS_UPTODATE:
 			continue;
 		default:
transport.c +13 -0
index f1fcd2c4b0..d768bc275e 100644
@@ -1226,6 +1226,19 @@ int transport_push(struct repository *r,
 		err = push_had_errors(remote_refs);
 		ret = push_ret | err;
 
+		if ((flags & TRANSPORT_PUSH_ATOMIC) && err) {
+			for (struct ref *it = remote_refs; it; it = it->next)
+				switch (it->status) {
+				case REF_STATUS_NONE:
+				case REF_STATUS_UPTODATE:
+				case REF_STATUS_OK:
+					it->status = REF_STATUS_ATOMIC_PUSH_FAILED;
+					break;
+				default:
+					break;
+				}
+		}
+
 		if (!quiet || err)
 			transport_print_push_status(transport->url, remote_refs,
 					verbose | porcelain, porcelain,
Signature · 1 line
2.22.0.410.gd8fdbe21b5-goog
#18 lore raw

t/t5541-http-push-smart.sh test_expect_success 'push (chunked)' ' · line 187

Quoted 8 lines · Emily Shaffer
t/t5541-http-push-smart.sh +8 -0
+test_expect_success 'push --atomic also prevents branch creation, reports collateral' '
+	# Setup upstream repo - empty for now
+	d=$HTTPD_DOCUMENT_ROOT_PATH/atomic-branches.git &&
+	git init --bare "$d" &&
+	test_config -C "$d" http.receivepack true &&
+	up="$HTTPD_URL"/smart/atomic-branches.git &&
+
+	# Tell up about two branches for now

-ECANTPARSE "Tell up" part.

t/t5541-http-push-smart.sh test_expect_success 'push (chunked)' ' · line 191

Quoted 4 lines
t/t5541-http-push-smart.sh +4 -0
+	test_commit atomic1 &&
+	test_commit atomic2 &&
+	git branch collateral &&
+	git push "$up" master collateral &&

OK, so an initially empty directory $d that appears to network clients as $up now has two branches, 'master' and 'collateral', both pointing at the same history that ends with two commits, atomic2 whose parent is atomic1.

t/t5541-http-push-smart.sh test_expect_success 'push (chunked)' ' · line 199

Quoted 7 lines
t/t5541-http-push-smart.sh +7 -0
+	# collateral is a valid push, but should be failed by atomic push
+	git checkout collateral &&
+	test_commit collateral1 &&
+
+	# Make master incompatible with upstream to provoke atomic
+	git checkout master &&
+	git reset --hard HEAD^ &&

collateral grows, master rewinds.

t/t5541-http-push-smart.sh test_expect_success 'push (chunked)' ' · line 203

Quoted 3 lines
t/t5541-http-push-smart.sh +3 -0
+	# Add a new branch which should be failed by atomic push. This is a
+	# regression case.
+	git branch atomic &&

Another branch atomic is added

t/t5541-http-push-smart.sh test_expect_success 'push (chunked)' ' · line 206

Quoted 2 lines
t/t5541-http-push-smart.sh +2 -0
+	# --atomic should cause entire push to be rejected
+	test_must_fail git push --atomic "$up" master atomic collateral 2>output &&

Attempt to push all three: collateral alone would be OK, so is atomic, but because master rewinds, we expect none of the three to go through.

t/t5541-http-push-smart.sh test_expect_success 'push (chunked)' ' · line 209

Quoted 2 lines
t/t5541-http-push-smart.sh +2 -0
+	# the new branch should not have been created upstream
+	test_must_fail git -C "$d" rev-parse refs/heads/atomic &&

The new branch should not have been created; if this rev-parse succeeded, it would be a bug.

Up to point, I have no possible improvements to offer ;-) Very well done.

t/t5541-http-push-smart.sh test_expect_success 'push (chunked)' ' · line 212

Quoted 2 lines
t/t5541-http-push-smart.sh +2 -0
+	# the failed refs should be indicated
+	grep "master -> master" output | grep rejected &&

I'd rather see the effect, i.e. what the command did that can be observed externally, than the report, i.e. what the command claims to have done, if it is equally straight-forward to verify either.

That can be done by making sure that the output from "git -C "$d" rev-parse refs/heads/master" match output from "git rev-parse atomic2", no? That ensures 'master' in the receiving end stayed the same.

t/t5541-http-push-smart.sh test_expect_success 'push (chunked)' ' · line 216

Quoted 3 lines
t/t5541-http-push-smart.sh +3 -0
+	# the collateral failure refs should be indicated
+	grep "atomic -> atomic" output | grep "atomic push failed" &&
+	grep "collateral -> collateral" output | grep "atomic push failed"

Likewise for the other two.

FWIW, these three can further lose a process each, i.e.

	grep "^ ! .*rejected.* master -> master" output

even if we for some reason do not want to check the effect and take the claim by the command being tested at the face value (which I do not think is a good idea).

Thanks.

#19 lore raw
Quoted 5 lines · Junio C Hamano

t/t5541-http-push-smart.sh test_expect_success 'push (chunked)' ' · line 209

Quoted 2 lines
t/t5541-http-push-smart.sh +2 -0
+	# the new branch should not have been created upstream
+	test_must_fail git -C "$d" rev-parse refs/heads/atomic &&

The new branch should not have been created; if this rev-parse succeeded, it would be a bug.

One thing I forgot. If refs/heads/atomic did not exist, but say refs/tags/refs/heads/atomic did, the rev-parse would succeed, which is a rather unfortunate source of confusion.

Of course, we know we have never touched "$d" to cause such a funny tag, so rev-parse is good enough in practice, but

    git -C "$d" show-ref --verify refs/heads/atomic

would not dwim (its --verify mode was invented specifically for rectifying this issue with rev-parse, if I recall correctly), and is more appropriate best-practice version to write here, especially if we anticipate that future developers and Git users would treat this line as an example to mimic.

Quoted 2 lines

Up to point, I have no possible improvements to offer ;-) Very well done.

So, I lied, but still the tests looked quite well done.

#20 lore raw
Quoted 23 lines · Junio C Hamano
Quoted 5 lines · Junio C Hamano

t/t5541-http-push-smart.sh test_expect_success 'push (chunked)' ' · line 209

Quoted 2 lines
t/t5541-http-push-smart.sh +2 -0
+	# the new branch should not have been created upstream
+	test_must_fail git -C "$d" rev-parse refs/heads/atomic &&

The new branch should not have been created; if this rev-parse succeeded, it would be a bug.

One thing I forgot. If refs/heads/atomic did not exist, but say refs/tags/refs/heads/atomic did, the rev-parse would succeed, which is a rather unfortunate source of confusion.

Of course, we know we have never touched "$d" to cause such a funny tag, so rev-parse is good enough in practice, but

    git -C "$d" show-ref --verify refs/heads/atomic

would not dwim (its --verify mode was invented specifically for rectifying this issue with rev-parse, if I recall correctly), and is more appropriate best-practice version to write here, especially if we anticipate that future developers and Git users would treat this line as an example to mimic.

Quoted 4 lines
Quoted 2 lines

Up to point, I have no possible improvements to offer ;-) Very well done.

So, I lied, but still the tests looked quite well done.

Oh, that's very interesting! Thanks for pointing it out. :)

Reroll coming in a few. Thanks, Junio.

- Emily

#21 lore raw
In reply to Junio C Hamano
Quoted 12 lines · Junio C Hamano

t/t5541-http-push-smart.sh test_expect_success 'push (chunked)' ' · line 187

Quoted 8 lines · Emily Shaffer
t/t5541-http-push-smart.sh +8 -0
+test_expect_success 'push --atomic also prevents branch creation, reports collateral' '
+	# Setup upstream repo - empty for now
+	d=$HTTPD_DOCUMENT_ROOT_PATH/atomic-branches.git &&
+	git init --bare "$d" &&
+	test_config -C "$d" http.receivepack true &&
+	up="$HTTPD_URL"/smart/atomic-branches.git &&
+
+	# Tell up about two branches for now

-ECANTPARSE "Tell up" part.

s/up/"$&"/ - thanks.

[snip]

Quoted 2 lines

Up to point, I have no possible improvements to offer ;-) Very well done.

This is nice to hear, thanks! :)

Quoted 6 lines

t/t5541-http-push-smart.sh test_expect_success 'push (chunked)' ' · line 212

Quoted 2 lines
t/t5541-http-push-smart.sh +2 -0
+	# the failed refs should be indicated
+	grep "master -> master" output | grep rejected &&

I'd rather see the effect, i.e. what the command did that can be observed externally, than the report, i.e. what the command claims to have done, if it is equally straight-forward to verify either.

Hmm. I'd like to argue that part of the requirement is to show the user what happened; for example, in an earlier iteration I was not successfully reporting the "collateral damage" refs to the user, even though they were not being pushed. To that end, I'd rather check both.

Quoted 5 lines

That can be done by making sure that the output from "git -C "$d" rev-parse refs/heads/master" match output from "git rev-parse atomic2", no? That ensures 'master' in the receiving end stayed the same.

Sure, I agree.

Quoted 7 lines

t/t5541-http-push-smart.sh test_expect_success 'push (chunked)' ' · line 216

Quoted 4 lines
t/t5541-http-push-smart.sh +3 -0
+	# the collateral failure refs should be indicated
+	grep "atomic -> atomic" output | grep "atomic push failed" &&
+	grep "collateral -> collateral" output | grep "atomic push failed"

Likewise for the other two.

Quoted 7 lines

FWIW, these three can further lose a process each, i.e.

	grep "^ ! .*rejected.* master -> master" output

even if we for some reason do not want to check the effect and take the claim by the command being tested at the face value (which I do not think is a good idea).

Will swap, wasn't sure on preference between regex or process count. Thanks.

- Emily

#22 lore raw
Quoted 4 lines · Emily Shaffer

Hmm. I'd like to argue that part of the requirement is to show the user what happened; for example, in an earlier iteration I was not successfully reporting the "collateral damage" refs to the user, even though they were not being pushed. To that end, I'd rather check both.

If we can afford to check both, making sure that we behave correctly and report what we did correctly would of course be the best.

;-)

#23 lore raw

PATCH v3 1/1 transport-helper: enforce atomic in push_refs_with_push

In reply to Emily Shaffer

Teach transport-helper how to notice if skipping a ref during push would violate atomicity on the client side. We notice that a ref would be rejected, and choose not to send it, but don't notice that if the client has asked for --atomic we are violating atomicity if all the other pushes we are sending would succeed. Asking the server end to uphold atomicity wouldn't work here as the server doesn't have any idea that we tried to update a ref that's broken.

The added test-case is a succinct way to reproduce this issue that fails today. The same steps work fine when we aren't using a transport-helper to get to the upstream, i.e. when we've added a local repository as a remote:

git remote add ~/upstream upstream

---
Since the previous version, made a couple changes based on Junio's comments on the testcase.

- Use show-ref instead of rev-parse to avoid a corner case where a tag is named the same thing as the branch we hope not to create
- Reduce number of grep processes by using a longer regex instead
- Check the real contents of the upstream repo instead of just relying on the client side's reporting
- The check on the client side remains, however, to ensure the user is made aware of why their perfectly good ref failed to push.

Thanks, all.

 t/t5541-http-push-smart.sh | 49 ++++++++++++++++++++++++++++++++++++++
 transport-helper.c         |  6 +++++
 transport.c                | 13 ++++++++++
 3 files changed, 68 insertions(+)
t/t5541-http-push-smart.sh +49 -0
index 8ef8763e06..92bac43257 100755
@@ -177,6 +177,55 @@ test_expect_success 'push (chunked)' '
 	 test $HEAD = $(git rev-parse --verify HEAD))
 '
 
+test_expect_success 'push --atomic also prevents branch creation, reports collateral' '
+	# Setup upstream repo - empty for now
+	d=$HTTPD_DOCUMENT_ROOT_PATH/atomic-branches.git &&
+	git init --bare "$d" &&
+	test_config -C "$d" http.receivepack true &&
+	up="$HTTPD_URL"/smart/atomic-branches.git &&
+
+	# Tell "$up" about two branches for now
+	test_commit atomic1 &&
+	test_commit atomic2 &&
+	git branch collateral &&
+	git push "$up" master collateral &&
+
+	# collateral is a valid push, but should be failed by atomic push
+	git checkout collateral &&
+	test_commit collateral1 &&
+
+	# Make master incompatible with upstream to provoke atomic
+	git checkout master &&
+	git reset --hard HEAD^ &&
+
+	# Add a new branch which should be failed by atomic push. This is a
+	# regression case.
+	git branch atomic &&
+
+	# --atomic should cause entire push to be rejected
+	test_must_fail git push --atomic "$up" master atomic collateral 2>output &&
+
+	# the new branch should not have been created upstream
+	test_must_fail git -C "$d" show-ref --verify refs/heads/atomic &&
+
+	# upstream should still reflect atomic2, the last thing we pushed
+	# successfully
+	git rev-parse atomic2 >expected &&
+	# on master...
+	git -C "$d" rev-parse refs/heads/master >actual &&
+	test_cmp expected actual &&
+	# ...and collateral.
+	git -C "$d" rev-parse refs/heads/collateral >actual &&
+	test_cmp expected actual &&
+
+	# the failed refs should be indicated to the user
+	grep "^ ! .*rejected.* master -> master" output &&
+
+	# the collateral failure refs should be indicated to the user
+	grep "^ ! .*rejected.* atomic -> atomic .*atomic push failed" output &&
+	grep "^ ! .*rejected.* collateral -> collateral .*atomic push failed" output
+'
+
 test_expect_success 'push --all can push to empty repo' '
 	d=$HTTPD_DOCUMENT_ROOT_PATH/empty-all.git &&
 	git init --bare "$d" &&
transport-helper.c +6 -0
index c7e17ec9cb..6b05a88faf 100644
@@ -853,6 +853,7 @@ static int push_refs_with_push(struct transport *transport,
 {
 	int force_all = flags & TRANSPORT_PUSH_FORCE;
 	int mirror = flags & TRANSPORT_PUSH_MIRROR;
+	int atomic = flags & TRANSPORT_PUSH_ATOMIC;
 	struct helper_data *data = transport->data;
 	struct strbuf buf = STRBUF_INIT;
 	struct ref *ref;
@@ -872,6 +873,11 @@ static int push_refs_with_push(struct transport *transport,
 		case REF_STATUS_REJECT_NONFASTFORWARD:
 		case REF_STATUS_REJECT_STALE:
 		case REF_STATUS_REJECT_ALREADY_EXISTS:
+			if (atomic) {
+				string_list_clear(&cas_options, 0);
+				return 0;
+			} else
+				continue;
 		case REF_STATUS_UPTODATE:
 			continue;
 		default:
transport.c +13 -0
index f1fcd2c4b0..d768bc275e 100644
@@ -1226,6 +1226,19 @@ int transport_push(struct repository *r,
 		err = push_had_errors(remote_refs);
 		ret = push_ret | err;
 
+		if ((flags & TRANSPORT_PUSH_ATOMIC) && err) {
+			for (struct ref *it = remote_refs; it; it = it->next)
+				switch (it->status) {
+				case REF_STATUS_NONE:
+				case REF_STATUS_UPTODATE:
+				case REF_STATUS_OK:
+					it->status = REF_STATUS_ATOMIC_PUSH_FAILED;
+					break;
+				default:
+					break;
+				}
+		}
+
 		if (!quiet || err)
 			transport_print_push_status(transport->url, remote_refs,
 					verbose | porcelain, porcelain,
Signature · 1 line
2.22.0.410.gd8fdbe21b5-goog
#24 lore raw

t/t5541-http-push-smart.sh test_expect_success 'push (chunked)' ' · line 187

Quoted 8 lines · Emily Shaffer
t/t5541-http-push-smart.sh +8 -0
+test_expect_success 'push --atomic also prevents branch creation, reports collateral' '
+	# Setup upstream repo - empty for now
+	d=$HTTPD_DOCUMENT_ROOT_PATH/atomic-branches.git &&
+	git init --bare "$d" &&
+	test_config -C "$d" http.receivepack true &&
+	up="$HTTPD_URL"/smart/atomic-branches.git &&
+
+	# Tell "$up" about two branches for now

Ahh, that's what you meant ;-) Makes sense.

#25 lore raw
In reply to Emily Shaffer

transport.c int transport_push(struct repository *r, · line 1230

Quoted 11 lines · Emily Shaffer
transport.c +2 -0
index f1fcd2c4b0..d768bc275e 100644
@@ -1226,6 +1226,19 @@ int transport_push(struct repository *r,
                err = push_had_errors(remote_refs);
                ret = push_ret | err;
 
+               if ((flags & TRANSPORT_PUSH_ATOMIC) && err) {
+                       for (struct ref *it = remote_refs; it; it = it->next)

moving "struct ref it" out of the loop, allows for building with ancient compilers that don't support C90 (even if only by default) as I found out while building pu in a Centos 6 box

Carlo

#26 lore raw
Quoted 10 lines · Carlo Arenas
Quoted 9 lines · Emily Shaffer
transport.c +0 -0
index f1fcd2c4b0..d768bc275e 100644
@@ -1226,6 +1226,19 @@ int transport_push(struct repository *r,
                err = push_had_errors(remote_refs);
                ret = push_ret | err;
 
Quoted 6 lines

transport.c int transport_push(struct repository *r, · line 1230

Quoted 2 lines
transport.c +2 -0
+               if ((flags & TRANSPORT_PUSH_ATOMIC) && err) {
+                       for (struct ref *it = remote_refs; it; it = it->next)

moving "struct ref it" out of the loop, allows for building with ancient compilers that don't support C90 (even if only by default) as I found out while building pu in a Centos 6 box

Does everything else compile OK with your rather old compiler on Centos 6? I was historically rather pedantic to stick to C89 but for past several years we've been experimenting with a bit more modern features of the language to see if anybody screams (namely, designated initializers in structs and arrays, and trailing comma in enum definition) but I think we still reject variable definition in for loop control (we saw and rewrote another patch that tried to use it late last year).

Apparently, this one slipped our review process.

Thanks.

cf. https://public-inbox.org/git/[email protected]/
cf. https://public-inbox.org/git/[email protected]/

#27 lore raw

[RFC/PATCH] CodingGuidelines: spell out post-C89 rules

Even though we have been sticking to C89, there are a few handy features we borrow from more recent C language in our codebase after trying them in weather balloons and saw that nobody screamed.

Spell them out.

While at it, extend the existing variable declaration rule a bit to read better with the newly spelled out rule for the for loop.

---
 Documentation/CodingGuidelines | 20 +++++++++++++++++---
 1 file changed, 17 insertions(+), 3 deletions(-)
Documentation/CodingGuidelines +17 -3
index 1169ff6c8e..53903b14c8 100644
@@ -195,10 +195,24 @@ For C programs:
    by e.g. "echo DEVELOPER=1 >>config.mak".
 
  - We try to support a wide range of C compilers to compile Git with,
-   including old ones. That means that you should not use C99
-   initializers, even if a lot of compilers grok it.
+   including old ones. That means that you should not use certain C99
+   features, even if your compiler groks it.  There are a few
+   exceptions:
 
- - Variables have to be declared at the beginning of the block.
+   . since early 2012 with e1327023ea, we have been using an enum
+     definition whose last element is followed by a comma.
+
+   . since mid 2017 with cbc0f81d and 512f41cf, we have been using
+     designated initializers for struct and array.
+
+   These used to be forbidden, but we have not heard breakage report,
+   so they are assumed to be safe.
+
+ - Variables have to be declared at the beginning of the block, before
+   the first statement (i.e. -Wdeclaration-after-statement).
+
+ - Declaring a variable in the for loop "for (int i = 0; i < 10; i++)"
+   is still not allowed in this codebase.
 
  - NULL pointers shall be written as NULL, not as 0.
 
Reviewed-by
#28 lore raw

Re: [RFC/PATCH] CodingGuidelines: spell out post-C89 rules

Quoted 5 lines · Junio C Hamano

Even though we have been sticking to C89, there are a few handy features we borrow from more recent C language in our codebase after trying them in weather balloons and saw that nobody screamed.

Spell them out.

Thanks for this. It gives a place to advertise future weather balloons, too.

[...]

Quoted 4 lines
Documentation/CodingGuidelines +0 -0
@@ -195,10 +195,24 @@ For C programs:
    by e.g. "echo DEVELOPER=1 >>config.mak".

Documentation/CodingGuidelines For C programs: · line 203

Quoted 10 lines
Documentation/CodingGuidelines +6 -2
 
  - We try to support a wide range of C compilers to compile Git with,
-   including old ones. That means that you should not use C99
-   initializers, even if a lot of compilers grok it.
+   including old ones. That means that you should not use certain C99
+   features, even if your compiler groks it.  There are a few
+   exceptions:
+
+   . since early 2012 with e1327023ea, we have been using an enum
+     definition whose last element is followed by a comma.

This is an interesting one: it's super convenient, but we have received patches every 10 years or so to remove the trailing comma --- e.g.
https://public-inbox.org/git/[email protected]/

I *think* these were motivated by wanting to be able to build Git with old compilers with pedantic warnings on, and certainly the last seven years of silence on the subject suggests it's okay. Should we be even more prescriptive and say that the last element should always be followed by a comma, for ease of later patching?

Documentation/CodingGuidelines For C programs: · line 206

Quoted 3 lines
Documentation/CodingGuidelines +3 -0
+
+   . since mid 2017 with cbc0f81d and 512f41cf, we have been using
+     designated initializers for struct and array.

Can this include an example for the benefit of readers that don't know what a designated initializer is? E.g.

. since mid 2017 with cb0f81d and 512f41cf, we have been using
designated initializers for struct members ("{ .alloc = 1 }") and array members ("[5] = 0").

Documentation/CodingGuidelines For C programs: · line 209

Quoted 3 lines
Documentation/CodingGuidelines +3 -0
+
+   These used to be forbidden, but we have not heard breakage report,
+   so they are assumed to be safe.

nit: missing article "any" before "breakage reports".

Documentation/CodingGuidelines For C programs: · line 215

Quoted 7 lines
Documentation/CodingGuidelines +5 -1
- - Variables have to be declared at the beginning of the block.
+ - Variables have to be declared at the beginning of the block, before
+   the first statement (i.e. -Wdeclaration-after-statement).
+
+ - Declaring a variable in the for loop "for (int i = 0; i < 10; i++)"
+   is still not allowed in this codebase.

Nice.

Thanks.

#29 lore raw

Re: [RFC/PATCH] CodingGuidelines: spell out post-C89 rules

Quoted 17 lines · Jonathan Nieder
Quoted 5 lines · Junio C Hamano

Even though we have been sticking to C89, there are a few handy features we borrow from more recent C language in our codebase after trying them in weather balloons and saw that nobody screamed.

Spell them out.

Thanks for this. It gives a place to advertise future weather balloons, too.

[...]

Quoted 5 lines
Documentation/CodingGuidelines +0 -0
@@ -195,10 +195,24 @@ For C programs:
    by e.g. "echo DEVELOPER=1 >>config.mak".
 
Quoted 5 lines

Documentation/CodingGuidelines For C programs: · line 199

Quoted 5 lines
Documentation/CodingGuidelines +2 -2
  - We try to support a wide range of C compilers to compile Git with,
-   including old ones. That means that you should not use C99
-   initializers, even if a lot of compilers grok it.
+   including old ones. That means that you should not use certain C99
+   features, even if your compiler groks it.  There are a few

s/it/them/ I think.

Quoted 8 lines

Documentation/CodingGuidelines For C programs: · line 203

Quoted 4 lines
Documentation/CodingGuidelines +4 -0
+   exceptions:
+
+   . since early 2012 with e1327023ea, we have been using an enum
+     definition whose last element is followed by a comma.

... Should we be even
more prescriptive and say that the last element should always be followed by a comma, for ease of later patching?

That is an interesting tangent. We have been taking advantage of the trailing comma for array initializers (e.g. options[] fed to parse_options() API)---we would gain similar convenience with it.

Quoted 6 lines

Documentation/CodingGuidelines For C programs: · line 206

Quoted 3 lines
Documentation/CodingGuidelines +3 -0
+
+   . since mid 2017 with cbc0f81d and 512f41cf, we have been using
+     designated initializers for struct and array.

Can this include an example for the benefit of readers that don't know what a designated initializer is?

I thought about it, but decided against it, because those who really want to are already given the commit object names to learn from, and in larger context.

But if we are to split this into two bullet points (one for struct, one for array), I am OK to add an example to it while at it. Let me see how it looks like.

Quoted 5 lines

Documentation/CodingGuidelines For C programs: · line 209

Quoted 3 lines
Documentation/CodingGuidelines +3 -0
+
+   These used to be forbidden, but we have not heard breakage report,
+   so they are assumed to be safe.

nit: missing article "any" before "breakage reports".

Thanks.

-- >8 --

Even though we have been sticking to C89, there are a few handy features we borrow from more recent C language in our codebase after trying them in weather balloons and saw that nobody screamed.

Spell them out.

While at it, extend the existing variable declaration rule a bit to read better with the newly spelled out rule for the for loop.

---
 Documentation/CodingGuidelines | 25 ++++++++++++++++++++++---
 1 file changed, 22 insertions(+), 3 deletions(-)
 Documentation/CodingGuidelines | 26 +++++++++++++++++++++++---
 1 file changed, 23 insertions(+), 3 deletions(-)
Documentation/CodingGuidelines +23 -3
index 32210a4386..bd4b7905d0 100644
@@ -195,10 +195,30 @@ For C programs:
    by e.g. "echo DEVELOPER=1 >>config.mak".
 
  - We try to support a wide range of C compilers to compile Git with,
-   including old ones. That means that you should not use C99
-   initializers, even if a lot of compilers grok it.
+   including old ones.  You should not use features from newer C
+   standard, even if your compiler groks them.  
 
- - Variables have to be declared at the beginning of the block.
+   There are a few exceptions to this guideline:
+
+   . since early 2012 with e1327023ea, we have been using an enum
+     definition whose last element is followed by a comma.  This, like
+     an array initializer that ends with a trailing comma, can be used
+     to reduce the patch noise when adding a new identifer at the end.
+
+   . since mid 2017 with cbc0f81d, we have been using designated
+     initializers for struct (e.g. "struct t v = { .val = 'a' };").
+
+   . since mid 2017 with 512f41cf, we have been using designated
+     initializers for array (e.g. "int array[10] = { [5] = 2 }").
+
+   These used to be forbidden, but we have not heard any breakage
+   report, and they are assumed to be safe.
+
+ - Variables have to be declared at the beginning of the block, before
+   the first statement (i.e. -Wdeclaration-after-statement).
+
+ - Declaring a variable in the for loop "for (int i = 0; i < 10; i++)"
+   is still not allowed in this codebase.
 
  - NULL pointers shall be written as NULL, not as 0.
 
Reviewed-by
#30 lore raw

Re: [RFC/PATCH] CodingGuidelines: spell out post-C89 rules

Quoted 7 lines · Junio C Hamano

-- >8 --
Even though we have been sticking to C89, there are a few handy features we borrow from more recent C language in our codebase after trying them in weather balloons and saw that nobody screamed.

Spell them out.

Quoted 7 lines

While at it, extend the existing variable declaration rule a bit to read better with the newly spelled out rule for the for loop.

---
 Documentation/CodingGuidelines | 25 ++++++++++++++++++++++---
 1 file changed, 22 insertions(+), 3 deletions(-)

Thanks.

#31 lore raw

Re: [RFC/PATCH] CodingGuidelines: spell out post-C89 rules

In reply to Junio C Hamano
Quoted 22 lines · Junio C Hamano

Even though we have been sticking to C89, there are a few handy features we borrow from more recent C language in our codebase after trying them in weather balloons and saw that nobody screamed.

Spell them out.

While at it, extend the existing variable declaration rule a bit to read better with the newly spelled out rule for the for loop.

---
 Documentation/CodingGuidelines | 20 +++++++++++++++++---
 1 file changed, 17 insertions(+), 3 deletions(-)
Documentation/CodingGuidelines +0 -0
index 1169ff6c8e..53903b14c8 100644
@@ -195,10 +195,24 @@ For C programs:
    by e.g. "echo DEVELOPER=1 >>config.mak".
 

Documentation/CodingGuidelines For C programs: · line 203

Quoted 10 lines
Documentation/CodingGuidelines +5 -3
  - We try to support a wide range of C compilers to compile Git with,
-   including old ones. That means that you should not use C99
-   initializers, even if a lot of compilers grok it.
+   including old ones. That means that you should not use certain C99
+   features, even if your compiler groks it.  There are a few
+   exceptions:
 
- - Variables have to be declared at the beginning of the block.
+   . since early 2012 with e1327023ea, we have been using an enum
+     definition whose last element is followed by a comma.

Is there a significance to the leading . here versus a leading - below?

Documentation/CodingGuidelines For C programs: · line 209

Quoted 6 lines
Documentation/CodingGuidelines +6 -0
+
+   . since mid 2017 with cbc0f81d and 512f41cf, we have been using
+     designated initializers for struct and array.
+
+   These used to be forbidden, but we have not heard breakage report,
+   so they are assumed to be safe.

With the placement here, is it possible that someone might read the “These used to be forbidden” as applying to the items that follow after it, rather than the items that preceded it? Put a different way, could there be some value in having some additional verbiage here that indicates something along the lines of “Aside from those exceptions, other C99 features are not allowed. Some common examples are:”

Just a thought. (Pardon the suggestion from the peanut gallery!)

Quoted 10 lines
Documentation/CodingGuidelines +6 -0
+
+ - Variables have to be declared at the beginning of the block, before
+   the first statement (i.e. -Wdeclaration-after-statement).
+
+ - Declaring a variable in the for loop "for (int i = 0; i < 10; i++)"
+   is still not allowed in this codebase.
 
  - NULL pointers shall be written as NULL, not as 0.
 
#32 lore raw

Re: [RFC/PATCH] CodingGuidelines: spell out post-C89 rules

Quoted 30 lines · Bryan Turner
Quoted 29 lines · Junio C Hamano

Even though we have been sticking to C89, there are a few handy features we borrow from more recent C language in our codebase after trying them in weather balloons and saw that nobody screamed.

Spell them out.

While at it, extend the existing variable declaration rule a bit to read better with the newly spelled out rule for the for loop.

---
 Documentation/CodingGuidelines | 20 +++++++++++++++++---
 1 file changed, 17 insertions(+), 3 deletions(-)
Documentation/CodingGuidelines +3 -2
index 1169ff6c8e..53903b14c8 100644
@@ -195,10 +195,24 @@ For C programs:
    by e.g. "echo DEVELOPER=1 >>config.mak".
 
  - We try to support a wide range of C compilers to compile Git with,
-   including old ones. That means that you should not use C99
-   initializers, even if a lot of compilers grok it.
+   including old ones. That means that you should not use certain C99
+   features, even if your compiler groks it.  There are a few
+   exceptions:
 
Quoted 4 lines

Documentation/CodingGuidelines For C programs: · line 203

Quoted 2 lines
Documentation/CodingGuidelines +2 -0
+   . since early 2012 with e1327023ea, we have been using an enum
+     definition whose last element is followed by a comma.

Is there a significance to the leading . here versus a leading - below?

Absolutely.

- Item 1's description

- Item 2's description
. subitem a of 2
. subitem b of 2
These two subitems are exceptions.

- Item 3's description

was what I meant.

#33 lore raw
In reply to Junio C Hamano
Quoted 3 lines · Junio C Hamano

Does everything else compile OK with your rather old compiler on Centos 6?

yes, they were a few -Wmaybe-uninitialized warnings but they were most likely false positives.

gcc 4.4.7 aborts the build (even without -Werror) with the following message:
transport.c: In function 'transport_push':
transport.c:1232: error: 'for' loop initial declarations are only allowed in C99 mode
transport.c:1232: note: use option -std=c99 or -std=gnu99 to compile your code
make: *** [transport.o] Error 1

Carlo

#34 lore raw

PATCH 1/1 transport-helper: avoid var decl in for () loop control

In reply to Junio C Hamano

We do allow a few selected C99 constructs in our codebase these days, but this is not among them (yet).

---
 transport.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
transport.c +2 -1
index d768bc275e..453de8f704 100644
@@ -1227,7 +1227,8 @@ int transport_push(struct repository *r,
 		ret = push_ret | err;
 
 		if ((flags & TRANSPORT_PUSH_ATOMIC) && err) {
-			for (struct ref *it = remote_refs; it; it = it->next)
+			struct ref *it;
+			for (it = remote_refs; it; it = it->next)
 				switch (it->status) {
 				case REF_STATUS_NONE:
 				case REF_STATUS_UPTODATE:
Reviewed-by on v1 1/1
#35 lore raw

Re: [PATCH] transport-helper: avoid var decl in for () loop control

Quoted 8 lines · Junio C Hamano

We do allow a few selected C99 constructs in our codebase these days, but this is not among them (yet).

---
 transport.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

Yes, gcc 4.8 fails to build without this:

transport.c:1234:4: error: ‘for’ loop initial declarations are only allowed in C99 mode

     for (struct ref *it = remote_refs; it; it = it->next)
     ^

transport.c:1234:4: note: use option -std=c99 or -std=gnu99 to compile your code

Arguably it would be nice to use -std=gnu99 for better consistency between gcc versions, but it's moot here: avoiding the declaration in for loop initializer is more consistent with -Wdeclaration-after-statement anyway.

Thanks.

#36 lore raw
In reply to Junio C Hamano
Quoted 8 lines · Junio C Hamano
Quoted 5 lines · Carlo Arenas
Quoted 1 line

+ for (struct ref *it = remote_refs; it; it = it->next)

moving "struct ref it" out of the loop, allows for building with ancient compilers that don't support C90 (even if only by default) as I found out while building pu in a Centos 6 box

Quoted 5 lines

but I think we still reject variable definition in
for loop control (we saw and rewrote another patch that tried to use it late last year).

Apparently, this one slipped our review process.

I expected that this will eventually happen after Travis CI's default Linux image recently changed from Ubuntu 14.04 to 16.04; explanation in the commit message below.

With that patch issues like this could be caught earlier, while they are only in 'pu' but not yet in 'next'. But do we really want to do that, is that the right tradeoff? Dunno... Adding a dedicated CI job just to check that there are no 'for' loop initial declarations seems kind of excessive, even if it only builds but doesn't run the test suite. And I don't know whether there are any other undesired ("too new") constructs that GCC 4.8 would catch but later compilers quietly accept.

--- >8 ---

Subject: [PATCH] travis-ci: build with GCC 4.8 as well

C99 'for' loop initial declaration, i.e. 'for (int i = 0; i < n; i++)', is not allowed in Git's codebase yet, to maintain compatibility with some older compilers.

Our Travis CI builds used to catch 'for' loop initial declarations, because the GETTEXT_POISON job has always built Git with the default 'cc', which in Travis CI's previous default Linux image (based on Ubuntu 14.04 Trusty) is GCC 4.8, and that GCC version errors out on this construct (not only with DEVELOPER=1, but with our default CFLAGS as well). Alas, that's not the case anymore, becase after 14.04's EOL Travis CI's current default Linux image is based on Ubuntu 16.04 Xenial [1] and its default 'cc' is now GCC 5.4, which, just like all later GCC and Clang versions, simply accepts this construct, even if we don't explicitly specify '-std=c99'.

Ideally we would adjust our CFLAGS used with DEVELOPER=1 to catch this undesired construct already when contributors build Git on their own machines. Unfortunately, however, there seems to be no compiler option that would catch only this particular construct without choking on many other things, e.g. while a later compiler with '-std=c90' and/or '-ansi' does catch this construct, it can't build Git because of several screenfulls of other errors.

Add the 'linux-gcc-4.8' job to Travis CI, in order to build Git with GCC 4.8, and thus to timely catch any 'for' loop initial declarations. To catch those it's sufficient to only build Git with GCC 4.8, so don't run the test suite in this job, because 'make test' takes rather long [2], and it's already run five times in other jobs, so we wouldn't get our time's worth.

[1] The Azure Pipelines builds have been using Ubuntu 16.04 images from the start, so I belive they never caught 'for' loop initial declarations.

[2] On Travis CI 'make test' alone would take about 9 minutes in this new job (without running httpd, Subversion, and P4 tests). For comparison, starting the job and building Git with GCC 4.8 takes only about 2 minutes.

---
 .travis.yml               |  4 ++++
 ci/run-build-and-tests.sh | 17 +++++++++++++----
 2 files changed, 17 insertions(+), 4 deletions(-)
.travis.yml +4 -0
index ffb1bc46f2..fc5730b085 100644
@@ -21,6 +21,10 @@ matrix:
       compiler:
       addons:
       before_install:
+    - env: jobname=linux-gcc-4.8
+      os: linux
+      dist: trusty
+      compiler:
     - env: jobname=Linux32
       os: linux
       compiler:
ci/run-build-and-tests.sh +13 -4
index cdd2913440..ff0ef7f08e 100755
@@ -11,9 +11,9 @@ windows*) cmd //c mklink //j t\\.prove "$(cygpath -aw "$cache_dir/.prove")";;
 esac
 
 make
-make test
-if test "$jobname" = "linux-gcc"
-then
+case "$jobname" in
+linux-gcc)
+	make test
 	export GIT_TEST_SPLIT_INDEX=yes
 	export GIT_TEST_FULL_IN_PACK_ARRAY=true
 	export GIT_TEST_OE_SIZE=10
@@ -21,7 +21,16 @@ then
 	export GIT_TEST_COMMIT_GRAPH=1
 	export GIT_TEST_MULTI_PACK_INDEX=1
 	make test
-fi
+	;;
+linux-gcc-4.8)
+	# Don't run the tests; we only care about whether Git can be
+	# built with GCC 4.8, as it errors out on some undesired (C99)
+	# constructs that newer compilers seem to quietly accept.
+	;;
+*)
+	make test
+	;;
+esac
 
 check_unignored_build_artifacts
 
Signature · 1 line
2.22.0.810.g50207c7d84
#37 lore raw
Quoted 3 lines · SZEDER Gábor

With that patch issues like this could be caught earlier, while they are only in 'pu' but not yet in 'next'. But do we really want to do that, is that the right tradeoff?

I am sort of in favor of having at least one build with an older compiler without "-std=c99", like the set-up you are proposing. I got an impression from an earlier message that Jonathan's preference is the opposite. I'd prefer to hear opinions from others, too.

The main reason why we accept other new features like trailing comma in enum def and designated initializers while rejecting for loop init is because there apparently are those who build Git with compilers that accept & reject these combinations of features and who care enough to report compilation failure from their build. And apparently gcc4.8 can serve as a representative "old" compiler, so...

#38 lore raw
Quoted 8 lines · Junio C Hamano
Quoted 3 lines · SZEDER Gábor

With that patch issues like this could be caught earlier, while they are only in 'pu' but not yet in 'next'. But do we really want to do that, is that the right tradeoff?

I am sort of in favor of having at least one build with an older compiler without "-std=c99", like the set-up you are proposing.

Alternatively, we could sort-of restore the old state of the GETTEXT_POISON job, i.e. build Git with GCC 4.8 in that job. "Sort-of", because we don't necessarily have to go back to the Ubuntu 14.04 based Linux image, because GCC 4.8 is packaged in 16.04 as well, so it's just an 'apt-get install gcc-4.8' away.

#39 lore raw
In reply to SZEDER Gábor
Quoted 7 lines · SZEDER Gábor

C99 'for' loop initial declaration, i.e. 'for (int i = 0; i < n; i++)', is not allowed in Git's codebase yet, to maintain compatibility with some older compilers.
[...]
[1] The Azure Pipelines builds have been using Ubuntu 16.04 images from the start, so I belive they never caught 'for' loop initial declarations.

s/belive/believe/

Quoted 6 lines

[2] On Travis CI 'make test' alone would take about 9 minutes in this new job (without running httpd, Subversion, and P4 tests). For comparison, starting the job and building Git with GCC 4.8 takes only about 2 minutes.

Reviewed-by
#40 lore raw
In reply to SZEDER Gábor
Quoted 4 lines · SZEDER Gábor

I expected that this will eventually happen after Travis CI's default Linux image recently changed from Ubuntu 14.04 to 16.04; explanation in the commit message below.

Quoted 8 lines

With that patch issues like this could be caught earlier, while they are only in 'pu' but not yet in 'next'. But do we really want to do that, is that the right tradeoff? Dunno... Adding a dedicated CI job just to check that there are no 'for' loop initial declarations seems kind of excessive, even if it only builds but doesn't run the test suite. And I don't know whether there are any other undesired ("too new") constructs that GCC 4.8 would catch but later compilers quietly accept.

This makes sense to me. Not really for the 'for' loop declaration
aspect: for that, I'd want some more specialized tool that allows turning on such a check specifically. But more because Ubuntu trusty is still a platform that some people use (though hopefully not for long), so it's helpful as a representative old platform to see if we break the build on it.

[...]

Quoted 4 lines

[2] On Travis CI 'make test' alone would take about 9 minutes in this new job (without running httpd, Subversion, and P4 tests). For comparison, starting the job and building Git with GCC 4.8 takes only about 2 minutes.

Nice. In an ideal world there would be some kind of "make fasttest" that runs some fast subset of tests. But this seems pretty safe.

Quoted 5 lines
---
 .travis.yml               |  4 ++++
 ci/run-build-and-tests.sh | 17 +++++++++++++----
 2 files changed, 17 insertions(+), 4 deletions(-)

Thanks.

[...]

Quoted 20 lines
ci/run-build-and-tests.sh +3 -4
@@ -11,9 +11,9 @@ windows*) cmd //c mklink //j t\\.prove "$(cygpath -aw "$cache_dir/.prove")";;
 esac
 
 make
-make test
-if test "$jobname" = "linux-gcc"
-then
+case "$jobname" in
+linux-gcc)
+	make test
 	export GIT_TEST_SPLIT_INDEX=yes
 	export GIT_TEST_FULL_IN_PACK_ARRAY=true
 	export GIT_TEST_OE_SIZE=10
@@ -21,7 +21,16 @@ then
 	export GIT_TEST_COMMIT_GRAPH=1
 	export GIT_TEST_MULTI_PACK_INDEX=1
 	make test
-fi

ci/run-build-and-tests.sh then · line 33

Quoted 10 lines
ci/run-build-and-tests.sh +10 -0
+	;;
+linux-gcc-4.8)
+	# Don't run the tests; we only care about whether Git can be
+	# built with GCC 4.8, as it errors out on some undesired (C99)
+	# constructs that newer compilers seem to quietly accept.
+	;;
+*)
+	make test
+	;;
+esac

Does what it says on the tin.

#41 lore raw
Quoted 7 lines · Jonathan Nieder

This makes sense to me. Not really for the 'for' loop declaration
aspect: for that, I'd want some more specialized tool that allows turning on such a check specifically. But more because Ubuntu trusty is still a platform that some people use (though hopefully not for long), so it's helpful as a representative old platform to see if we break the build on it.

FWIW this also breaks Centos 7 using gcc 4.8.5, as well as the one originally reported (Centos 6), and anything else that uses gcc 4 (tested up to 4.9.4)

Carlo

#42 lore raw
Quoted 9 lines · Carlo Arenas
Quoted 7 lines · Jonathan Nieder

This makes sense to me. Not really for the 'for' loop declaration
aspect: for that, I'd want some more specialized tool that allows turning on such a check specifically. But more because Ubuntu trusty is still a platform that some people use (though hopefully not for long), so it's helpful as a representative old platform to see if we break the build on it.

Quoted 3 lines

FWIW this also breaks Centos 7 using gcc 4.8.5, as well as the one originally reported (Centos 6), and anything else that uses gcc 4 (tested up to 4.9.4)

Thanks.

#43 lore raw
In reply to SZEDER Gábor

Junio,

Quoted 1 line · SZEDER Gábor

Subject: [PATCH] travis-ci: build with GCC 4.8 as well

This patch conflicts with topic 'js/trace2-json-schema', and the current conflict resolution in 'pu' is not quite correct.

Quoted 5 lines
ci/run-build-and-tests.sh +0 -0
index cdd2913440..ff0ef7f08e 100755

ci/run-build-and-tests.sh windows*) cmd //c mklink //j t\\.prove "$(cygpath -aw "$cache_dir/.prove")";; · line 16

Quoted 9 lines
ci/run-build-and-tests.sh +3 -3
@@ -11,9 +11,9 @@ windows*) cmd //c mklink //j t\\.prove "$(cygpath -aw "$cache_dir/.prove")";;
 esac
 
 make
-make test
-if test "$jobname" = "linux-gcc"
-then
+case "$jobname" in
+linux-gcc)
+	make test

This 'make test' here is important, but the confict resolution accidentally removed it.

Quoted 25 lines
ci/run-build-and-tests.sh +10 -1
 	export GIT_TEST_SPLIT_INDEX=yes
 	export GIT_TEST_FULL_IN_PACK_ARRAY=true
 	export GIT_TEST_OE_SIZE=10
@@ -21,7 +21,16 @@ then
 	export GIT_TEST_COMMIT_GRAPH=1
 	export GIT_TEST_MULTI_PACK_INDEX=1
 	make test
-fi
+	;;
+linux-gcc-4.8)
+	# Don't run the tests; we only care about whether Git can be
+	# built with GCC 4.8, as it errors out on some undesired (C99)
+	# constructs that newer compilers seem to quietly accept.
+	;;
+*)
+	make test
+	;;
+esac
 
 check_unignored_build_artifacts
 
Signature · 1 line
2.22.0.810.g50207c7d84
#44 lore raw
Quoted 7 lines · SZEDER Gábor

Junio,

Quoted 1 line · SZEDER Gábor

Subject: [PATCH] travis-ci: build with GCC 4.8 as well

This patch conflicts with topic 'js/trace2-json-schema', and the current conflict resolution in 'pu' is not quite correct.

Thanks.

"git diff ...MERGE_HEAD" during a merge of js/trace2-json-schema gives me this patch:

ci/run-build-and-tests.sh +6 -0
index cdd2913440..ec38bf379a 100755
@@ -14,6 +14,8 @@ make
 make test
 if test "$jobname" = "linux-gcc"
 then
+	make -C t/trace_schema_validator
+	export GIT_TRACE2_EVENT=$(mktemp)
 	export GIT_TEST_SPLIT_INDEX=yes
 	export GIT_TEST_FULL_IN_PACK_ARRAY=true
 	export GIT_TEST_OE_SIZE=10
@@ -21,6 +23,10 @@ then
 	export GIT_TEST_COMMIT_GRAPH=1
 	export GIT_TEST_MULTI_PACK_INDEX=1
 	make test
+	t/trace_schema_validator/trace_schema_validator \
+		--trace2_event_file=${GIT_TRACE2_EVENT} \
+		--schema_file=t/trace_schema_validator/strict_schema.json \
+		--progress=10000
 fi
 
 check_unignored_build_artifacts

i.e. they want to run an extra make in that validator directory, export another environment variable, and then run the validator *after* running the normal "make test", in linux-gcc job.

Quoted 7 lines
Quoted 7 lines
ci/run-build-and-tests.sh +0 -0
index cdd2913440..ff0ef7f08e 100755
@@ -11,9 +11,9 @@ windows*) cmd //c mklink //j t\\.prove "$(cygpath -aw "$cache_dir/.prove")";;
 esac
 
Quoted 10 lines

ci/run-build-and-tests.sh windows*) cmd //c mklink //j t\\.prove "$(cygpath -aw "$cache_dir/.prove")";; · line 16

Quoted 7 lines
ci/run-build-and-tests.sh +3 -3
 make
-make test
-if test "$jobname" = "linux-gcc"
-then
+case "$jobname" in
+linux-gcc)
+	make test

This 'make test' here is important, but the confict resolution accidentally removed it.

Right. Thanks for spotting.

I can see in "git diff pu~2 pu~1" that indeed the first 'make test' that we want to run without any of these environment variables is lost in the merge. We want to run two tests, with or without these environment variables, and the validator wants to piggyback on the second one.

Will fix in the meantime, but I was expecting that this "validator in CI" business to be redone differently, perhaps with a different validator implementation and either in a separate job or just part of an existing job but trace enabled only for some subset of the tests and/or only for new tests specifically written for trace coverage, so after that happens this may turn into a moot point.

The change the merge brings in to the file now reads like this.

Thanks, again.

ci/run-build-and-tests.sh +8 -0
index ff0ef7f08e..35ff4d3038 100755
@@ -14,6 +14,9 @@ make
 case "$jobname" in
 linux-gcc)
 	make test
+
+	make -C t/trace_schema_validator
+	export GIT_TRACE2_EVENT=$(mktemp)
 	export GIT_TEST_SPLIT_INDEX=yes
 	export GIT_TEST_FULL_IN_PACK_ARRAY=true
 	export GIT_TEST_OE_SIZE=10
@@ -21,6 +24,11 @@ linux-gcc)
 	export GIT_TEST_COMMIT_GRAPH=1
 	export GIT_TEST_MULTI_PACK_INDEX=1
 	make test
+
+	t/trace_schema_validator/trace_schema_validator \
+		--trace2_event_file=${GIT_TRACE2_EVENT} \
+		--schema_file=t/trace_schema_validator/strict_schema.json \
+		--progress=10000
 	;;
 linux-gcc-4.8)
 	# Don't run the tests; we only care about whether Git can be