tests: fix and add lint for non-portable head -c N

The "head -c BYTES" option is non-portable (not in POSIX[1]). Change such invocations to use the test_copy_bytes wrapper added in 48860819e8 ("t9300: factor out portable "head -c" replacement", 2016-06-30).

This fixes a test added in 9d2e330b17 ("ewah_read_mmap: bounds-check mmap reads", 2018-06-14), which has been breaking t5310-pack-bitmaps.sh on OpenBSD since 2.18.0. The OpenBSD ports already have a similar workaround after their upgrade to 2.18.0[2].

I have not tested this on IRIX, but according to 4de0bbd898 ("t9300:
use perl "head -c" clone in place of "dd bs=1 count=16000" kluge", 2010-12-13) this invocation would have broken things there too.

Also, change a valgrind-specific codepath in test-lib.sh to use this wrapper. Given where valgrind runs I don't think this would ever become a portability issue in practice, but it's easier to just use the wrapper than introduce some exception for the "make test-lint" check being added here.

1. http://pubs.opengroup.org/onlinepubs/9699919799/utilities/head.html
2. https://github.com/openbsd/ports/commit/08d5d82eaefe5cf2f125ecc0c6a57df9cf91350c#diff-f7d3c4fabeed1691620d608f1534f5e5

---

Junio: Even though this isn't a 2.19.0-rc0 regression I think it makes sense for 2.19.0. The fix is trivial, and it'll unbreak (at least some of) the tests on stock git on OpenBSD.

OpenBSD guys: If you CC the git mailing list when you find you need to apply patches like these, we're happy to fix this more pro-actively. I just happened to be testing the upcoming 2.19.0 on OpenBSD and spotted this.

 t/check-non-portable-shell.pl | 1 +
 t/t5310-pack-bitmaps.sh       | 2 +-
 t/test-lib.sh                 | 4 ++--
 3 files changed, 4 insertions(+), 3 deletions(-)

index d5823f71d8..94a7e6165e 100755
@@ -35,6 +35,7 @@ sub err {
 		chomp;
 	}
 
+	/\bhead\s+-c\b/ and err 'head -c is not portable (use test_copy_bytes BYTES <file >out)';
 	/\bsed\s+-i/ and err 'sed -i is not portable';
 	/\becho\s+-[neE]/ and err 'echo with option is not portable (use printf)';
 	/^\s*declare\s+/ and err 'arrays/declare not portable';

index 557bd0d0c0..7bff7923f2 100755
@@ -335,7 +335,7 @@ test_expect_success 'truncated bitmap fails gracefully' '
 	git rev-list --use-bitmap-index --count --all >expect &&
 	bitmap=$(ls .git/objects/pack/*.bitmap) &&
 	test_when_finished "rm -f $bitmap" &&
-	head -c 512 <$bitmap >$bitmap.tmp &&
+	test_copy_bytes 512 <$bitmap >$bitmap.tmp &&
 	mv -f $bitmap.tmp $bitmap &&
 	git rev-list --use-bitmap-index --count --all >actual 2>stderr &&
 	test_cmp expect actual &&

index 8bb0f4348e..44288cbb59 100644
@@ -867,7 +867,7 @@ then
 		# handle only executables, unless they are shell libraries that
 		# need to be in the exec-path.
 		test -x "$1" ||
-		test "# " = "$(head -c 2 <"$1")" ||
+		test "# " = "$(test_copy_bytes 2 <"$1")" ||
 		return;
 
 		base=$(basename "$1")

@@ -882,7 +882,7 @@ then
 		# do not override scripts
 		if test -x "$symlink_target" &&
 		    test ! -d "$symlink_target" &&
-		    test "#!" != "$(head -c 2 < "$symlink_target")"
+		    test "#!" != "$(test_copy_bytes 2 <"$symlink_target")"
 		then
 			symlink_target=../valgrind.sh
 		fi

2.18.0.865.gffc8e1a3cd6

This is on OpenBSD 6.2 amd64 with my "tests: fix and add lint for non-portable head -c N" patch (which fixes one failure).

    $ for t in t1305-config-include.sh t1308-config-set.sh t5004-archive-corner-cases.sh t5552-skipping-fetch-negotiator.sh; do ./$t --no-color -v -x 2>&1 | grep -B10 "^not ok"; done

+ cd bar
+ echo [includeIf "gitdir:bar/"]path=bar7
+ >> .git/config
+ echo [test]seven=7
+ > .git/bar7
+ echo 7
+ > expect
+ git config test.seven
+ > actual
error: last command exited with $?=1
not ok 27 - conditional include, gitdir matching symlink --
+ cd bar
+ echo [includeIf "gitdir/i:BAR/"]path=bar8
+ >> .git/config
+ echo [test]eight=8
+ > .git/bar8
+ echo 8
+ > expect
+ git config test.eight
+ > actual
error: last command exited with $?=1
not ok 28 - conditional include, gitdir matching symlink, icase
test_cmp expect actual

+ echo Error (-1) reading configuration file a-directory.
+ > expect
+ mkdir a-directory
+ test_expect_code 2 test-tool config configset_get_value foo.bar a-directory
+ 2> output
Value not found for "foo.bar"
test_expect_code: command exited with 1, we wanted 2 test-tool config configset_get_value foo.bar a-directory
error: last command exited with $?=1
not ok 23 - proper error on directory "files"
+ make_dir extract
+ tar xf subtree-all.tar -C extract
tar: Cannot identify format. Searching...
tar: End of archive volume 1 reached
tar: Sorry, unable to determine archive format.
error: last command exited with $?=1
+ rm -rf extract
+ exit 1
+ eval_ret=1
+ :
not ok 9 - archive empty subtree with no pathspec
--
+ make_dir extract
+ tar xf subtree-path.tar -C extract
tar: Cannot identify format. Searching...
tar: End of archive volume 1 reached
tar: Sorry, unable to determine archive format.
error: last command exited with $?=1
+ rm -rf extract
+ exit 1
+ eval_ret=1
+ :
not ok 10 - archive empty subtree by direct pathspec
'git <command> [<revision>...] -- [<file>...]'
fatal: ambiguous argument 'c7': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions, like this:
'git <command> [<revision>...] -- [<file>...]'
No have c7 (c7)
error: last command exited with $?=1
+ test_unconfig -C client fetch.negotiationalgorithm
+ exit 1
+ eval_ret=1
+ :
not ok 1 - commits with no parents are sent regardless of skip distance
--
Author: A U Thor <[email protected]>
1 file changed, 1 insertion(+)
create mode 100644 to_fetch.t
+ git init client
Initialized empty Git repository in /home/avar/g/git/t/trash directory.t5552-skipping-fetch-negotiator/client/.git/
+ seq 11
./t5552-skipping-fetch-negotiator.sh: seq: not found
+ git -C client checkout c5
error: pathspec 'c5' did not match any file(s) known to git
error: last command exited with $?=1
not ok 3 - when two skips collide, favor the larger one
--
+ git init client
Initialized empty Git repository in /home/avar/g/git/t/trash directory.t5552-skipping-fetch-negotiator/client/.git/
+ seq 8
./t5552-skipping-fetch-negotiator.sh: seq: not found
+ seq 19
./t5552-skipping-fetch-negotiator.sh: seq: not found
+ pwd
+ git -C server fetch --no-tags /home/avar/g/git/t/trash directory.t5552-skipping-fetch-negotiator/client b1:refs/heads/b1
fatal: Couldn't find remote ref b1
error: last command exited with $?=128
not ok 6 - do not send "have" with ancestors of commits that server ACKed

Full output at https://gitlab.com/snippets/1747801

Some of this, like the t5552-skipping-fetch-negotiator.sh failures are new in 2.19.0 (those go away with s/seq/test_seq). But some are older. E.g. the archive corner cases failure is becuse that test wants unzip & GNU tar, which OpenBSD upstream has patched already:
https://github.com/openbsd/ports/blob/master/devel/git/patches/patch-t_test-lib_sh

A blind guess. is this "BSD allows opening a FILE* on a directory and reading bytes from it" aka FREAD_READS_DIRECTORIES?

I wonder why FreeBSD sets it but not OpenBSD in config.mak.uname.

Perhaps modern tar elements like extended headers are not supported on the platform-supplied "tar"? "make TAR=gnutar"?

This must use test_seq.

 t/t5552-skipping-fetch-negotiator.sh | 12 ++++++------
 1 file changed, 6 insertions(+), 6 deletions(-)

index 5ad5bece55..30857b84a8 100755
@@ -46,7 +46,7 @@ test_expect_success 'commits with no parents are sent regardless of skip distanc
 	test_commit -C server to_fetch &&
 
 	git init client &&
-	for i in $(seq 7)
+	for i in $(test_seq 7)
 	do
 		test_commit -C client c$i
 	done &&

@@ -89,7 +89,7 @@ test_expect_success 'when two skips collide, favor the larger one' '
 	test_commit -C server to_fetch &&
 
 	git init client &&
-	for i in $(seq 11)
+	for i in $(test_seq 11)
 	do
 		test_commit -C client c$i
 	done &&

@@ -168,14 +168,14 @@ test_expect_success 'do not send "have" with ancestors of commits that server AC
 	test_commit -C server to_fetch &&
 
 	git init client &&
-	for i in $(seq 8)
+	for i in $(test_seq 8)
 	do
 		git -C client checkout --orphan b$i &&
 		test_commit -C client b$i.c0
 	done &&
-	for j in $(seq 19)
+	for j in $(test_seq 19)
 	do
-		for i in $(seq 8)
+		for i in $(test_seq 8)
 		do
 			git -C client checkout b$i &&
 			test_commit -C client b$i.c$j

@@ -205,7 +205,7 @@ test_expect_success 'do not send "have" with ancestors of commits that server AC
 
 	# fetch-pack should thus not send any more commits in the b1 branch, but
 	# should still send the others (in this test, just check b2).
-	for i in $(seq 0 8)
+	for i in $(test_seq 0 8)
 	do
 		have_not_sent b1.c$i
 	done &&

The "head -c BYTES" option is non-portable (not in POSIX[1]). Change such invocations to use the test_copy_bytes wrapper added in 48860819e8 ("t9300: factor out portable "head -c" replacement", 2016-06-30).

This fixes a test added in 9d2e330b17 ("ewah_read_mmap: bounds-check mmap reads", 2018-06-14), which has been breaking t5310-pack-bitmaps.sh on OpenBSD since 2.18.0. The OpenBSD ports already have a similar workaround after their upgrade to 2.18.0[2].

I have not tested this on IRIX, but according to 4de0bbd898 ("t9300:
use perl "head -c" clone in place of "dd bs=1 count=16000" kluge", 2010-12-13) this invocation would have broken things there too.

Also, change a valgrind-specific codepath in test-lib.sh to use this wrapper. Given where valgrind runs I don't think this would ever become a portability issue in practice, but it's easier to just use the wrapper than introduce some exception for the "make test-lint" check being added here.

1. http://pubs.opengroup.org/onlinepubs/9699919799/utilities/head.html
2. https://github.com/openbsd/ports/commit/08d5d82eaefe5cf2f125ecc0c6a57df9cf91350c#diff-f7d3c4fabeed1691620d608f1534f5e5

---
 t/check-non-portable-shell.pl | 1 +
 t/t5310-pack-bitmaps.sh       | 2 +-
 t/test-lib.sh                 | 4 ++--
 3 files changed, 4 insertions(+), 3 deletions(-)

index d5823f71d8..c8f10d40a1 100755
@@ -41,6 +41,7 @@ sub err {
 	/^\s*[^#]\s*which\s/ and err 'which is not portable (use type)';
 	/\btest\s+[^=]*==/ and err '"test a == b" is not portable (use =)';
 	/\bwc -l.*"\s*=/ and err '`"$(wc -l)"` is not portable (use test_line_count)';
+	/\bhead\s+-c\b/ and err 'head -c is not portable (use test_copy_bytes BYTES <file >out)';
 	/\bexport\s+[A-Za-z0-9_]*=/ and err '"export FOO=bar" is not portable (use FOO=bar && export FOO)';
 	/^\s*([A-Z0-9_]+=(\w+|(["']).*?\3)\s+)+(\w+)/ and exists($func{$4}) and
 		err '"FOO=bar shell_func" assignment extends beyond "shell_func"';

index 557bd0d0c0..7bff7923f2 100755
@@ -335,7 +335,7 @@ test_expect_success 'truncated bitmap fails gracefully' '
 	git rev-list --use-bitmap-index --count --all >expect &&
 	bitmap=$(ls .git/objects/pack/*.bitmap) &&
 	test_when_finished "rm -f $bitmap" &&
-	head -c 512 <$bitmap >$bitmap.tmp &&
+	test_copy_bytes 512 <$bitmap >$bitmap.tmp &&
 	mv -f $bitmap.tmp $bitmap &&
 	git rev-list --use-bitmap-index --count --all >actual 2>stderr &&
 	test_cmp expect actual &&

index 8bb0f4348e..44288cbb59 100644
@@ -867,7 +867,7 @@ then
 		# handle only executables, unless they are shell libraries that
 		# need to be in the exec-path.
 		test -x "$1" ||
-		test "# " = "$(head -c 2 <"$1")" ||
+		test "# " = "$(test_copy_bytes 2 <"$1")" ||
 		return;
 
 		base=$(basename "$1")

@@ -882,7 +882,7 @@ then
 		# do not override scripts
 		if test -x "$symlink_target" &&
 		    test ! -d "$symlink_target" &&
-		    test "#!" != "$(head -c 2 < "$symlink_target")"
+		    test "#!" != "$(test_copy_bytes 2 <"$symlink_target")"
 		then
 			symlink_target=../valgrind.sh
 		fi

2.18.0.865.gffc8e1a3cd6

Heh, I even considered using this when writing that test. But the reason I introduced test_copy_bytes is not because the target platform did not have "head -c" at all, but because some tests need very specific buffering guarantees when reading from a shared pipe.

That said, if OpenBSD's "head" doesn't have "-c" at all, I'm fine with this as a fix (and it sounds like we know that IRIX lacks it, too).

When working on 9d2e330b17, I recall finding these other "head -c" invocations in the test suite when I did 9d2e330b17 and took them as evidence that it was OK to use in vanilla cases. So even if these sites don't affect any platforms in practice, I think it's worth it to ban "head -c" completely.

Patch itself looks good to me. Thanks.

-Peff

GNU seq is not a POSIX command, and doesn't exist on
e.g. OpenBSD. We've had the test_seq wrapper since d17cf5f3a3 ("tests:
Introduce test_seq", 2012-08-04), but use of it keeps coming back, e.g. in the recently added "fetch negotiator" tests being added here.

So let's also add a check to "make test-lint". The regex is aiming to capture the likes of $(seq ..) and "seq" as a stand-alone command, without capturing some existing cases where we e.g. have files called "seq".

---

Now with a fix & check in v2 for the seq issue mentioned in
https://public-inbox.org/git/[email protected]/

 t/check-non-portable-shell.pl        |  1 +
 t/t5552-skipping-fetch-negotiator.sh | 12 ++++++------
 t/t5703-upload-pack-ref-in-want.sh   |  4 ++--
 3 files changed, 9 insertions(+), 8 deletions(-)

index c8f10d40a1..75f38298d7 100755
@@ -42,6 +42,7 @@ sub err {
 	/\btest\s+[^=]*==/ and err '"test a == b" is not portable (use =)';
 	/\bwc -l.*"\s*=/ and err '`"$(wc -l)"` is not portable (use test_line_count)';
 	/\bhead\s+-c\b/ and err 'head -c is not portable (use test_copy_bytes BYTES <file >out)';
+	/(?:\$\(seq|^\s*seq\b)/ and err 'seq is not portable (use test_seq)';
 	/\bexport\s+[A-Za-z0-9_]*=/ and err '"export FOO=bar" is not portable (use FOO=bar && export FOO)';
 	/^\s*([A-Z0-9_]+=(\w+|(["']).*?\3)\s+)+(\w+)/ and exists($func{$4}) and
 		err '"FOO=bar shell_func" assignment extends beyond "shell_func"';

index 5ad5bece55..30857b84a8 100755
@@ -46,7 +46,7 @@ test_expect_success 'commits with no parents are sent regardless of skip distanc
 	test_commit -C server to_fetch &&
 
 	git init client &&
-	for i in $(seq 7)
+	for i in $(test_seq 7)
 	do
 		test_commit -C client c$i
 	done &&

@@ -89,7 +89,7 @@ test_expect_success 'when two skips collide, favor the larger one' '
 	test_commit -C server to_fetch &&
 
 	git init client &&
-	for i in $(seq 11)
+	for i in $(test_seq 11)
 	do
 		test_commit -C client c$i
 	done &&

@@ -168,14 +168,14 @@ test_expect_success 'do not send "have" with ancestors of commits that server AC
 	test_commit -C server to_fetch &&
 
 	git init client &&
-	for i in $(seq 8)
+	for i in $(test_seq 8)
 	do
 		git -C client checkout --orphan b$i &&
 		test_commit -C client b$i.c0
 	done &&
-	for j in $(seq 19)
+	for j in $(test_seq 19)
 	do
-		for i in $(seq 8)
+		for i in $(test_seq 8)
 		do
 			git -C client checkout b$i &&
 			test_commit -C client b$i.c$j

@@ -205,7 +205,7 @@ test_expect_success 'do not send "have" with ancestors of commits that server AC
 
 	# fetch-pack should thus not send any more commits in the b1 branch, but
 	# should still send the others (in this test, just check b2).
-	for i in $(seq 0 8)
+	for i in $(test_seq 0 8)
 	do
 		have_not_sent b1.c$i
 	done &&

index a73c55a47e..d1ccc22331 100755
@@ -176,7 +176,7 @@ test_expect_success 'setup repos for change-while-negotiating test' '
 		git clone "http://127.0.0.1:$LIB_HTTPD_PORT/smart/repo" "$LOCAL_PRISTINE" &&
 		cd "$LOCAL_PRISTINE" &&
 		git checkout -b side &&
-		for i in $(seq 1 33); do test_commit s$i; done &&
+		for i in $(test_seq 1 33); do test_commit s$i; done &&
 
 		# Add novel commits to upstream
 		git checkout master &&

@@ -289,7 +289,7 @@ test_expect_success 'setup repos for fetching with ref-in-want tests' '
 		git clone "file://$REPO" "$LOCAL_PRISTINE" &&
 		cd "$LOCAL_PRISTINE" &&
 		git checkout -b side &&
-		for i in $(seq 1 33); do test_commit s$i; done &&
+		for i in $(test_seq 1 33); do test_commit s$i; done &&
 
 		# Add novel commits to upstream
 		git checkout master &&

2.18.0.865.gffc8e1a3cd6

s/GNU //; the command did not even originate there, but came from V8 and/or Plan9 IIRC.

Thanks.

As alluded to in the commit message that will find cases like:

    $ git grep -P '\bseq\b' -- t | grep -P -v '(?:\$\(seq|^\s*seq\b)'
    t/perf/p3400-rebase.sh:16:              seq 1000 >>unrelated-file$i &&
    t/perf/p3400-rebase.sh:21:              seq 1000 | tac >>unrelated-file$i &&
    t/t3404-rebase-interactive.sh:1227:             test_seq 5 | sed "s/$double/&&/" >seq &&
    t/t3404-rebase-interactive.sh:1228:             git add seq &&
    t/t3404-rebase-interactive.sh:1230:             git commit -m seq-$double
    t/t3404-rebase-interactive.sh:1232:     git tag seq-onto &&
    t/t3404-rebase-interactive.sh:1234:     git cherry-pick seq-onto &&
    t/t3404-rebase-interactive.sh:1236:     test_must_fail env FAKE_LINES= git rebase -i seq-onto &&
    t/t3404-rebase-interactive.sh:1239:     git diff --exit-code seq-onto &&
    t/t4022-diff-rewrite.sh:69:     test_seq 1 99 >seq &&
    t/t4022-diff-rewrite.sh:70:     printf 100 >>seq &&
    t/t4022-diff-rewrite.sh:71:     git add seq &&
    t/t4022-diff-rewrite.sh:72:     git commit seq -m seq
    t/t4022-diff-rewrite.sh:76:     test_seq 1 5 >seq &&
    t/t4022-diff-rewrite.sh:77:     test_seq 9331 9420 >>seq &&
    t/t4022-diff-rewrite.sh:78:     test_seq 96 100 >>seq
    t/t4022-diff-rewrite.sh:82:     git diff -B seq >res &&
    t/t4022-diff-rewrite.sh:87:     git diff seq >res &&
    t/t4022-diff-rewrite.sh:93:     git diff -B seq >res &&
    t/t4150-am.sh:933:      git format-patch -2 --stdout >seq.patch &&
    t/t4150-am.sh:945:      test_must_fail git am -3 seq.patch &&
    t/t4150-am.sh:955:      test_must_fail git am -3 seq.patch &&
    t/t5520-pull.sh:295:    git checkout -b seq &&
    t/t5520-pull.sh:296:    test_seq 5 >seq.txt &&
    t/t5520-pull.sh:297:    git add seq.txt &&
    t/t5520-pull.sh:299:    git commit -m "Add seq.txt" &&
    t/t5520-pull.sh:300:    echo 6 >>seq.txt &&
    t/t5520-pull.sh:302:    git commit -m "Append to seq.txt" seq.txt &&
    t/t5520-pull.sh:304:    echo conflicting >>seq.txt &&
    t/t5520-pull.sh:306:    git commit -m "Create conflict" seq.txt &&
    t/t5520-pull.sh:307:    test_must_fail git pull --rebase . seq 2>err >out &&

This grew a bit more. I'm going to stop poking at this for now. The tests are still broken on OpenBSD (3-5 broken) and on AIX something like 20-30 are broken, but this makes it slightly better.

Ævar Arnfjörð Bjarmason (5):
tests: fix and add lint for non-portable head -c N
tests: fix and add lint for non-portable seq
tests: use shorter here-docs in chainlint.sed for AIX sed
tests: fix version-specific portability issue in Perl JSON
tests: fix and add lint for non-portable grep --file

 t/chainlint.sed                      |  8 ++++----
 t/check-non-portable-shell.pl        |  3 +++
 t/t0019/parse_json.perl              |  3 +++
 t/t5310-pack-bitmaps.sh              |  2 +-
 t/t5318-commit-graph.sh              |  2 +-
 t/t5552-skipping-fetch-negotiator.sh | 12 ++++++------
 t/t5703-upload-pack-ref-in-want.sh   |  4 ++--
 t/test-lib.sh                        |  4 ++--
 8 files changed, 22 insertions(+), 16 deletions(-)

2.18.0.865.gffc8e1a3cd6

The "head -c BYTES" option is non-portable (not in POSIX[1]). Change such invocations to use the test_copy_bytes wrapper added in 48860819e8 ("t9300: factor out portable "head -c" replacement", 2016-06-30).

This fixes a test added in 9d2e330b17 ("ewah_read_mmap: bounds-check mmap reads", 2018-06-14), which has been breaking t5310-pack-bitmaps.sh on OpenBSD since 2.18.0. The OpenBSD ports already have a similar workaround after their upgrade to 2.18.0[2].

I have not tested this on IRIX, but according to 4de0bbd898 ("t9300:
use perl "head -c" clone in place of "dd bs=1 count=16000" kluge", 2010-12-13) this invocation would have broken things there too.

Also, change a valgrind-specific codepath in test-lib.sh to use this wrapper. Given where valgrind runs I don't think this would ever become a portability issue in practice, but it's easier to just use the wrapper than introduce some exception for the "make test-lint" check being added here.

1. http://pubs.opengroup.org/onlinepubs/9699919799/utilities/head.html
2. https://github.com/openbsd/ports/commit/08d5d82eaefe5cf2f125ecc0c6a57df9cf91350c#diff-f7d3c4fabeed1691620d608f1534f5e5

---
 t/check-non-portable-shell.pl | 1 +
 t/t5310-pack-bitmaps.sh       | 2 +-
 t/test-lib.sh                 | 4 ++--
 3 files changed, 4 insertions(+), 3 deletions(-)

index d5823f71d8..c8f10d40a1 100755
@@ -41,6 +41,7 @@ sub err {
 	/^\s*[^#]\s*which\s/ and err 'which is not portable (use type)';
 	/\btest\s+[^=]*==/ and err '"test a == b" is not portable (use =)';
 	/\bwc -l.*"\s*=/ and err '`"$(wc -l)"` is not portable (use test_line_count)';
+	/\bhead\s+-c\b/ and err 'head -c is not portable (use test_copy_bytes BYTES <file >out)';
 	/\bexport\s+[A-Za-z0-9_]*=/ and err '"export FOO=bar" is not portable (use FOO=bar && export FOO)';
 	/^\s*([A-Z0-9_]+=(\w+|(["']).*?\3)\s+)+(\w+)/ and exists($func{$4}) and
 		err '"FOO=bar shell_func" assignment extends beyond "shell_func"';

index 557bd0d0c0..7bff7923f2 100755
@@ -335,7 +335,7 @@ test_expect_success 'truncated bitmap fails gracefully' '
 	git rev-list --use-bitmap-index --count --all >expect &&
 	bitmap=$(ls .git/objects/pack/*.bitmap) &&
 	test_when_finished "rm -f $bitmap" &&
-	head -c 512 <$bitmap >$bitmap.tmp &&
+	test_copy_bytes 512 <$bitmap >$bitmap.tmp &&
 	mv -f $bitmap.tmp $bitmap &&
 	git rev-list --use-bitmap-index --count --all >actual 2>stderr &&
 	test_cmp expect actual &&

index 8bb0f4348e..44288cbb59 100644
@@ -867,7 +867,7 @@ then
 		# handle only executables, unless they are shell libraries that
 		# need to be in the exec-path.
 		test -x "$1" ||
-		test "# " = "$(head -c 2 <"$1")" ||
+		test "# " = "$(test_copy_bytes 2 <"$1")" ||
 		return;
 
 		base=$(basename "$1")

@@ -882,7 +882,7 @@ then
 		# do not override scripts
 		if test -x "$symlink_target" &&
 		    test ! -d "$symlink_target" &&
-		    test "#!" != "$(head -c 2 < "$symlink_target")"
+		    test "#!" != "$(test_copy_bytes 2 <"$symlink_target")"
 		then
 			symlink_target=../valgrind.sh
 		fi

2.18.0.865.gffc8e1a3cd6

The seq command is not in POSIX, and doesn't exist on
e.g. OpenBSD. We've had the test_seq wrapper since d17cf5f3a3 ("tests:
Introduce test_seq", 2012-08-04), but use of it keeps coming back, e.g. in the recently added "fetch negotiator" tests being added here.

So let's also add a check to "make test-lint". The regex is aiming to capture the likes of $(seq ..) and "seq" as a stand-alone command, without capturing some existing cases where we e.g. have files called "seq", as \bseq\b would do.

---
 t/check-non-portable-shell.pl        |  1 +
 t/t5552-skipping-fetch-negotiator.sh | 12 ++++++------
 t/t5703-upload-pack-ref-in-want.sh   |  4 ++--
 3 files changed, 9 insertions(+), 8 deletions(-)

index c8f10d40a1..75f38298d7 100755
@@ -42,6 +42,7 @@ sub err {
 	/\btest\s+[^=]*==/ and err '"test a == b" is not portable (use =)';
 	/\bwc -l.*"\s*=/ and err '`"$(wc -l)"` is not portable (use test_line_count)';
 	/\bhead\s+-c\b/ and err 'head -c is not portable (use test_copy_bytes BYTES <file >out)';
+	/(?:\$\(seq|^\s*seq\b)/ and err 'seq is not portable (use test_seq)';
 	/\bexport\s+[A-Za-z0-9_]*=/ and err '"export FOO=bar" is not portable (use FOO=bar && export FOO)';
 	/^\s*([A-Z0-9_]+=(\w+|(["']).*?\3)\s+)+(\w+)/ and exists($func{$4}) and
 		err '"FOO=bar shell_func" assignment extends beyond "shell_func"';

index 5ad5bece55..30857b84a8 100755
@@ -46,7 +46,7 @@ test_expect_success 'commits with no parents are sent regardless of skip distanc
 	test_commit -C server to_fetch &&
 
 	git init client &&
-	for i in $(seq 7)
+	for i in $(test_seq 7)
 	do
 		test_commit -C client c$i
 	done &&

@@ -89,7 +89,7 @@ test_expect_success 'when two skips collide, favor the larger one' '
 	test_commit -C server to_fetch &&
 
 	git init client &&
-	for i in $(seq 11)
+	for i in $(test_seq 11)
 	do
 		test_commit -C client c$i
 	done &&

@@ -168,14 +168,14 @@ test_expect_success 'do not send "have" with ancestors of commits that server AC
 	test_commit -C server to_fetch &&
 
 	git init client &&
-	for i in $(seq 8)
+	for i in $(test_seq 8)
 	do
 		git -C client checkout --orphan b$i &&
 		test_commit -C client b$i.c0
 	done &&
-	for j in $(seq 19)
+	for j in $(test_seq 19)
 	do
-		for i in $(seq 8)
+		for i in $(test_seq 8)
 		do
 			git -C client checkout b$i &&
 			test_commit -C client b$i.c$j

@@ -205,7 +205,7 @@ test_expect_success 'do not send "have" with ancestors of commits that server AC
 
 	# fetch-pack should thus not send any more commits in the b1 branch, but
 	# should still send the others (in this test, just check b2).
-	for i in $(seq 0 8)
+	for i in $(test_seq 0 8)
 	do
 		have_not_sent b1.c$i
 	done &&

index a73c55a47e..d1ccc22331 100755
@@ -176,7 +176,7 @@ test_expect_success 'setup repos for change-while-negotiating test' '
 		git clone "http://127.0.0.1:$LIB_HTTPD_PORT/smart/repo" "$LOCAL_PRISTINE" &&
 		cd "$LOCAL_PRISTINE" &&
 		git checkout -b side &&
-		for i in $(seq 1 33); do test_commit s$i; done &&
+		for i in $(test_seq 1 33); do test_commit s$i; done &&
 
 		# Add novel commits to upstream
 		git checkout master &&

@@ -289,7 +289,7 @@ test_expect_success 'setup repos for fetching with ref-in-want tests' '
 		git clone "file://$REPO" "$LOCAL_PRISTINE" &&
 		cd "$LOCAL_PRISTINE" &&
 		git checkout -b side &&
-		for i in $(seq 1 33); do test_commit s$i; done &&
+		for i in $(test_seq 1 33); do test_commit s$i; done &&
 
 		# Add novel commits to upstream
 		git checkout master &&

2.18.0.865.gffc8e1a3cd6

Improve the portability of chainlint by using shorter here-docs. On AIX sed will complain about:

sed: 0602-417 The label :hereslurp is greater than eight characters

As noted in [1] there's still a remaining recently introduced portability issue also introduced in 878f988350 ("t/test-lib: teach --chain-lint to detect broken &&-chains in subshells", 2018-07-11), so under AIX the tests must be run with GIT_TEST_CHAIN_LINT=0.

I don't know how to solve the other issue, and this gets us some of the way to GIT_TEST_CHAIN_LINT=1 working again on AIX.

1. https://public-inbox.org/git/[email protected]/

---
 t/chainlint.sed | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

index 8544df38df..2333705b27 100644
@@ -97,11 +97,11 @@
 /<<[ 	]*[-\\']*[A-Za-z0-9_]/ {
 	s/^\(.*\)<<[ 	]*[-\\']*\([A-Za-z0-9_][A-Za-z0-9_]*\)'*/<\2>\1<</
 	s/[ 	]*<<//
-	:hereslurp
+	:hered
 	N
 	/^<\([^>]*\)>.*\n[ 	]*\1[ 	]*$/!{
 		s/\n.*$//
-		bhereslurp
+		bhered
 	}
 	s/^<[^>]*>//
 	s/\n.*$//

@@ -283,11 +283,11 @@ bfolded
 :heredoc
 s/^\(.*\)<<[ 	]*[-\\']*\([A-Za-z0-9_][A-Za-z0-9_]*\)'*/<\2>\1<</
 s/[ 	]*<<//
-:hereslurpsub
+:heredsub
 N
 /^<\([^>]*\)>.*\n[ 	]*\1[ 	]*$/!{
 	s/\n.*$//
-	bhereslurpsub
+	bheredsub
 }
 s/^<[^>]*>//
 s/\n.*$//

2.18.0.865.gffc8e1a3cd6

I'll globally do s/here-doc/label/ while queueing.

POSIX says "The implementation shall support label arguments recognized as unique up to at least 8 bytes", so replacing these labels to shorter strings makes perfect sense.

Will queue; thanks.

Does unindenting the comment, as I suggested in [1], fix the remaining problem for you?

[1]: https://public-inbox.org/git/CAPig+cTTbU5HFMKgNyrxTp3+kcK46-Fn=4ZH6zDt1oQChAc3KA@mail.gmail.com/

I didn't notice Eric's E-mail before I sent my v3, so going back and testing this revealed two things:

1) Yes, his suggestion works
2) AIX sed will complain about one thing at a time, so we had a lot more of these "labels too long" problems once I got past fixing that issue.

So here's a version, which as noted in 4/6 makes GIT_TEST_CHAIN_LINT=1 fully work on AIX again.

As an aside, the reason I have access to AIX is because I requested access to the GCC compile farm as suggested by someone on-list here the other day: https://cfarm.tetaneutral.net/

They accepted my account request on the basis that I was going to hack on git & perl on those boxes, so if anyone else here is interested in testing stuff for portability...

Ævar Arnfjörð Bjarmason (6):
tests: fix and add lint for non-portable head -c N
tests: fix and add lint for non-portable seq
tests: fix comment syntax in chainlint.sed for AIX sed
tests: use shorter here-docs in chainlint.sed for AIX sed
tests: fix version-specific portability issue in Perl JSON
tests: fix and add lint for non-portable grep --file

 t/chainlint.sed                      | 59 ++++++++++++++--------------
 t/check-non-portable-shell.pl        |  3 ++
 t/t0019/parse_json.perl              |  3 ++
 t/t5310-pack-bitmaps.sh              |  2 +-
 t/t5318-commit-graph.sh              |  2 +-
 t/t5552-skipping-fetch-negotiator.sh | 12 +++---
 t/t5703-upload-pack-ref-in-want.sh   |  4 +-
 t/test-lib.sh                        |  4 +-
 8 files changed, 47 insertions(+), 42 deletions(-)

2.18.0.865.gffc8e1a3cd6

The "head -c BYTES" option is non-portable (not in POSIX[1]). Change such invocations to use the test_copy_bytes wrapper added in 48860819e8 ("t9300: factor out portable "head -c" replacement", 2016-06-30).

This fixes a test added in 9d2e330b17 ("ewah_read_mmap: bounds-check mmap reads", 2018-06-14), which has been breaking t5310-pack-bitmaps.sh on OpenBSD since 2.18.0. The OpenBSD ports already have a similar workaround after their upgrade to 2.18.0[2].

I have not tested this on IRIX, but according to 4de0bbd898 ("t9300:
use perl "head -c" clone in place of "dd bs=1 count=16000" kluge", 2010-12-13) this invocation would have broken things there too.

Also, change a valgrind-specific codepath in test-lib.sh to use this wrapper. Given where valgrind runs I don't think this would ever become a portability issue in practice, but it's easier to just use the wrapper than introduce some exception for the "make test-lint" check being added here.

1. http://pubs.opengroup.org/onlinepubs/9699919799/utilities/head.html
2. https://github.com/openbsd/ports/commit/08d5d82eaefe5cf2f125ecc0c6a57df9cf91350c#diff-f7d3c4fabeed1691620d608f1534f5e5

---
 t/check-non-portable-shell.pl | 1 +
 t/t5310-pack-bitmaps.sh       | 2 +-
 t/test-lib.sh                 | 4 ++--
 3 files changed, 4 insertions(+), 3 deletions(-)

index d5823f71d8..c8f10d40a1 100755
@@ -41,6 +41,7 @@ sub err {
 	/^\s*[^#]\s*which\s/ and err 'which is not portable (use type)';
 	/\btest\s+[^=]*==/ and err '"test a == b" is not portable (use =)';
 	/\bwc -l.*"\s*=/ and err '`"$(wc -l)"` is not portable (use test_line_count)';
+	/\bhead\s+-c\b/ and err 'head -c is not portable (use test_copy_bytes BYTES <file >out)';
 	/\bexport\s+[A-Za-z0-9_]*=/ and err '"export FOO=bar" is not portable (use FOO=bar && export FOO)';
 	/^\s*([A-Z0-9_]+=(\w+|(["']).*?\3)\s+)+(\w+)/ and exists($func{$4}) and
 		err '"FOO=bar shell_func" assignment extends beyond "shell_func"';

index 557bd0d0c0..7bff7923f2 100755
@@ -335,7 +335,7 @@ test_expect_success 'truncated bitmap fails gracefully' '
 	git rev-list --use-bitmap-index --count --all >expect &&
 	bitmap=$(ls .git/objects/pack/*.bitmap) &&
 	test_when_finished "rm -f $bitmap" &&
-	head -c 512 <$bitmap >$bitmap.tmp &&
+	test_copy_bytes 512 <$bitmap >$bitmap.tmp &&
 	mv -f $bitmap.tmp $bitmap &&
 	git rev-list --use-bitmap-index --count --all >actual 2>stderr &&
 	test_cmp expect actual &&

index 8bb0f4348e..44288cbb59 100644
@@ -867,7 +867,7 @@ then
 		# handle only executables, unless they are shell libraries that
 		# need to be in the exec-path.
 		test -x "$1" ||
-		test "# " = "$(head -c 2 <"$1")" ||
+		test "# " = "$(test_copy_bytes 2 <"$1")" ||
 		return;
 
 		base=$(basename "$1")

@@ -882,7 +882,7 @@ then
 		# do not override scripts
 		if test -x "$symlink_target" &&
 		    test ! -d "$symlink_target" &&
-		    test "#!" != "$(head -c 2 < "$symlink_target")"
+		    test "#!" != "$(test_copy_bytes 2 <"$symlink_target")"
 		then
 			symlink_target=../valgrind.sh
 		fi

2.18.0.865.gffc8e1a3cd6

The seq command is not in POSIX, and doesn't exist on
e.g. OpenBSD. We've had the test_seq wrapper since d17cf5f3a3 ("tests:
Introduce test_seq", 2012-08-04), but use of it keeps coming back, e.g. in the recently added "fetch negotiator" tests being added here.

So let's also add a check to "make test-lint". The regex is aiming to capture the likes of $(seq ..) and "seq" as a stand-alone command, without capturing some existing cases where we e.g. have files called "seq", as \bseq\b would do.

---
 t/check-non-portable-shell.pl        |  1 +
 t/t5552-skipping-fetch-negotiator.sh | 12 ++++++------
 t/t5703-upload-pack-ref-in-want.sh   |  4 ++--
 3 files changed, 9 insertions(+), 8 deletions(-)

index c8f10d40a1..75f38298d7 100755
@@ -42,6 +42,7 @@ sub err {
 	/\btest\s+[^=]*==/ and err '"test a == b" is not portable (use =)';
 	/\bwc -l.*"\s*=/ and err '`"$(wc -l)"` is not portable (use test_line_count)';
 	/\bhead\s+-c\b/ and err 'head -c is not portable (use test_copy_bytes BYTES <file >out)';
+	/(?:\$\(seq|^\s*seq\b)/ and err 'seq is not portable (use test_seq)';
 	/\bexport\s+[A-Za-z0-9_]*=/ and err '"export FOO=bar" is not portable (use FOO=bar && export FOO)';
 	/^\s*([A-Z0-9_]+=(\w+|(["']).*?\3)\s+)+(\w+)/ and exists($func{$4}) and
 		err '"FOO=bar shell_func" assignment extends beyond "shell_func"';

index 5ad5bece55..30857b84a8 100755
@@ -46,7 +46,7 @@ test_expect_success 'commits with no parents are sent regardless of skip distanc
 	test_commit -C server to_fetch &&
 
 	git init client &&
-	for i in $(seq 7)
+	for i in $(test_seq 7)
 	do
 		test_commit -C client c$i
 	done &&

@@ -89,7 +89,7 @@ test_expect_success 'when two skips collide, favor the larger one' '
 	test_commit -C server to_fetch &&
 
 	git init client &&
-	for i in $(seq 11)
+	for i in $(test_seq 11)
 	do
 		test_commit -C client c$i
 	done &&

@@ -168,14 +168,14 @@ test_expect_success 'do not send "have" with ancestors of commits that server AC
 	test_commit -C server to_fetch &&
 
 	git init client &&
-	for i in $(seq 8)
+	for i in $(test_seq 8)
 	do
 		git -C client checkout --orphan b$i &&
 		test_commit -C client b$i.c0
 	done &&
-	for j in $(seq 19)
+	for j in $(test_seq 19)
 	do
-		for i in $(seq 8)
+		for i in $(test_seq 8)
 		do
 			git -C client checkout b$i &&
 			test_commit -C client b$i.c$j

@@ -205,7 +205,7 @@ test_expect_success 'do not send "have" with ancestors of commits that server AC
 
 	# fetch-pack should thus not send any more commits in the b1 branch, but
 	# should still send the others (in this test, just check b2).
-	for i in $(seq 0 8)
+	for i in $(test_seq 0 8)
 	do
 		have_not_sent b1.c$i
 	done &&

index a73c55a47e..d1ccc22331 100755
@@ -176,7 +176,7 @@ test_expect_success 'setup repos for change-while-negotiating test' '
 		git clone "http://127.0.0.1:$LIB_HTTPD_PORT/smart/repo" "$LOCAL_PRISTINE" &&
 		cd "$LOCAL_PRISTINE" &&
 		git checkout -b side &&
-		for i in $(seq 1 33); do test_commit s$i; done &&
+		for i in $(test_seq 1 33); do test_commit s$i; done &&
 
 		# Add novel commits to upstream
 		git checkout master &&

@@ -289,7 +289,7 @@ test_expect_success 'setup repos for fetching with ref-in-want tests' '
 		git clone "file://$REPO" "$LOCAL_PRISTINE" &&
 		cd "$LOCAL_PRISTINE" &&
 		git checkout -b side &&
-		for i in $(seq 1 33); do test_commit s$i; done &&
+		for i in $(test_seq 1 33); do test_commit s$i; done &&
 
 		# Add novel commits to upstream
 		git checkout master &&

2.18.0.865.gffc8e1a3cd6

Change a comment in chainlint.sed to appease AIX sed, which would previously print this error:

sed: # stash for later printing is not a recognized function

1. https://public-inbox.org/git/CAPig+cTTbU5HFMKgNyrxTp3+kcK46-Fn=4ZH6zDt1oQChAc3KA@mail.gmail.com/

---
 t/chainlint.sed | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

index 8544df38df..dcb4b333ed 100644
@@ -131,9 +131,8 @@ b
 b
 
 :subshell
-# bare "(" line?
+# bare "(" line? -- stash for later printing
 /^[ 	]*([	]*$/ {
-	# stash for later printing
 	h
 	bnextline
 }

2.18.0.865.gffc8e1a3cd6

It would have been nice for the commit message to explain that the problem was caused by the comment being indented, since it's otherwise not otherwise obvious to the reader without chasing the link to the email exchange, but probably not worth a re-roll.

FWIW,

Improve the portability of chainlint by using shorter here-docs. On AIX sed will complain about:

sed: 0602-417 The label :hereslurp is greater than eight characters

This, in combination with the previous fix to this file makes GIT_TEST_CHAIN_LINT=1 (which is the default) working again on AIX without issues, and the "gmake check-chainlint" test also passes.

---
 t/chainlint.sed | 56 ++++++++++++++++++++++++-------------------------
 1 file changed, 28 insertions(+), 28 deletions(-)

index dcb4b333ed..c80d2fad7a 100644
@@ -97,11 +97,11 @@
 /<<[ 	]*[-\\']*[A-Za-z0-9_]/ {
 	s/^\(.*\)<<[ 	]*[-\\']*\([A-Za-z0-9_][A-Za-z0-9_]*\)'*/<\2>\1<</
 	s/[ 	]*<<//
-	:hereslurp
+	:hered
 	N
 	/^<\([^>]*\)>.*\n[ 	]*\1[ 	]*$/!{
 		s/\n.*$//
-		bhereslurp
+		bhered
 	}
 	s/^<[^>]*>//
 	s/\n.*$//

@@ -149,7 +149,7 @@ s/.*\n//
 
 :slurp
 # incomplete line "...\"
-/\\$/bincomplete
+/\\$/bicmplte
 # multi-line quoted string "...\n..."?
 /"/bdqstring
 # multi-line quoted string '...\n...'? (but not contraction in string "it's")

@@ -171,7 +171,7 @@ s/.*\n//
 	/"[^"]*#[^"]*"/!s/[ 	]#.*$//
 }
 # one-liner "case ... esac"
-/^[ 	]*case[ 	]*..*esac/bcheckchain
+/^[ 	]*case[ 	]*..*esac/bchkchn
 # multi-line "case ... esac"
 /^[ 	]*case[ 	]..*[ 	]in/bcase
 # multi-line "for ... done" or "while ... done"

@@ -200,32 +200,32 @@ s/.*\n//
 /^[ 	]*fi[ 	]*[<>|]/bdone
 /^[ 	]*fi[ 	]*)/bdone
 # nested one-liner "(...) &&"
-/^[ 	]*(.*)[ 	]*&&[ 	]*$/bcheckchain
+/^[ 	]*(.*)[ 	]*&&[ 	]*$/bchkchn
 # nested one-liner "(...)"
-/^[ 	]*(.*)[ 	]*$/bcheckchain
+/^[ 	]*(.*)[ 	]*$/bchkchn
 # nested one-liner "(...) >x" (or "2>x" or "<x" or "|x")
-/^[ 	]*(.*)[ 	]*[0-9]*[<>|]/bcheckchain
+/^[ 	]*(.*)[ 	]*[0-9]*[<>|]/bchkchn
 # nested multi-line "(...\n...)"
 /^[ 	]*(/bnest
 # multi-line "{...\n...}"
 /^[ 	]*{/bblock
 # closing ")" on own line -- exit subshell
-/^[ 	]*)/bclosesolo
+/^[ 	]*)/bclssolo
 # "$((...))" -- arithmetic expansion; not closing ")"
-/\$(([^)][^)]*))[^)]*$/bcheckchain
+/\$(([^)][^)]*))[^)]*$/bchkchn
 # "$(...)" -- command substitution; not closing ")"
-/\$([^)][^)]*)[^)]*$/bcheckchain
+/\$([^)][^)]*)[^)]*$/bchkchn
 # multi-line "$(...\n...)" -- command substitution; treat as nested subshell
 /\$([^)]*$/bnest
 # "=(...)" -- Bash array assignment; not closing ")"
-/=(/bcheckchain
+/=(/bchkchn
 # closing "...) &&"
 /)[ 	]*&&[ 	]*$/bclose
 # closing "...)"
 /)[ 	]*$/bclose
 # closing "...) >x" (or "2>x" or "<x" or "|x")
 /)[ 	]*[<>|]/bclose
-:checkchain
+:chkchn
 # mark suspect if line uses ";" internally rather than "&&" (but not ";" in a
 # string and not ";;" in one-liner "case...esac")
 /;/{

@@ -244,7 +244,7 @@ n
 bslurp
 
 # found incomplete line "...\" -- slurp up next line
-:incomplete
+:icmplte
 N
 s/\\\n//
 bslurp

@@ -282,11 +282,11 @@ bfolded
 :heredoc
 s/^\(.*\)<<[ 	]*[-\\']*\([A-Za-z0-9_][A-Za-z0-9_]*\)'*/<\2>\1<</
 s/[ 	]*<<//
-:hereslurpsub
+:heredsub
 N
 /^<\([^>]*\)>.*\n[ 	]*\1[ 	]*$/!{
 	s/\n.*$//
-	bhereslurpsub
+	bheredsub
 }
 s/^<[^>]*>//
 s/\n.*$//

@@ -316,43 +316,43 @@ x
 # is 'done' or 'fi' cuddled with ")" to close subshell?
 /done.*)/bclose
 /fi.*)/bclose
-bcheckchain
+bchkchn
 
 # found nested multi-line "(...\n...)" -- pass through untouched
 :nest
 x
-:nestslurp
+:nstslurp
 n
 # closing ")" on own line -- stop nested slurp
-/^[ 	]*)/bnestclose
+/^[ 	]*)/bnstclose
 # comment -- not closing ")" if in comment
-/^[ 	]*#/bnestcontinue
+/^[ 	]*#/bnstcnt
 # "$((...))" -- arithmetic expansion; not closing ")"
-/\$(([^)][^)]*))[^)]*$/bnestcontinue
+/\$(([^)][^)]*))[^)]*$/bnstcnt
 # "$(...)" -- command substitution; not closing ")"
-/\$([^)][^)]*)[^)]*$/bnestcontinue
+/\$([^)][^)]*)[^)]*$/bnstcnt
 # closing "...)" -- stop nested slurp
-/)/bnestclose
-:nestcontinue
+/)/bnstclose
+:nstcnt
 x
-bnestslurp
-:nestclose
+bnstslurp
+:nstclose
 s/^/>>/
 # is it "))" which closes nested and parent subshells?
 /)[ 	]*)/bslurp
-bcheckchain
+bchkchn
 
 # found multi-line "{...\n...}" block -- pass through untouched
 :block
 x
 n
 # closing "}" -- stop block slurp
-/}/bcheckchain
+/}/bchkchn
 bblock
 
 # found closing ")" on own line -- drop "suspect" from final line of subshell
 # since that line legitimately lacks "&&" and exit subshell loop
-:closesolo
+:clssolo
 x
 s/?!AMP?!//
 p

2.18.0.865.gffc8e1a3cd6

Shortening the names makes them ugly and often unreadable. That's not a complaint with this patch; just a general observation regarding 8-byte limitation with this platform's "sed" (and POSIX). See a few suggested improvements below, but probably not worth a re-roll.

FWIW,

":fold" (for "fold out NL")

":nesteat"

":nestcont"

":nestend" or ":endnest"

":endsolo"

I was meaning to get to this with a re-roll, but since this is already in next & these label renames seem cosmetic, it seems better to do this after 2.19, i.e. the compiler doesn't care about the specific names, but shortening them to <=8 fixes the bug.

Sounds fine. There's no rush. Thanks.

Remind me again not to forget doing s/here-doc/label/ on this patch before queueing. Other than that, looks good, together with 3/6 for the the indented comment issue.

The test guarded by PERLJSON added in 75459410ed ("json_writer: new routines to create JSON data", 2018-07-13) assumed that a JSON boolean value like "true" or "false" would be represented as "1" or "0" in Perl.

This behavior can't be relied upon, e.g. with JSON.pm 2.50 and JSON::PP A JSON::PP::Boolean object will be represented as "true" or "false". To work around this let's check if we have any refs left after we check for hashes and arrays, assume those are JSON objects, and coerce them to a known boolean value.

The behavior of this test still looks odd to me. Why implement our own ad-hoc encoder just for some one-off test, as opposed to say Perl's own Data::Dumper with Sortkeys et al? But with this change it works, so let's leave it be.

---
 t/t0019/parse_json.perl | 3 +++
 1 file changed, 3 insertions(+)

index ca4e5bfa78..fea87fb81b 100644
@@ -34,6 +34,9 @@ sub dump_item {
     } elsif (ref($value) eq 'HASH') {
 	print "$label_in hash\n";
 	dump_hash($label_in, $value);
+    } elsif (ref $value) {
+	my $bool = $value ? 1 : 0;
+	print "$label_in $bool\n";
     } elsif (defined $value) {
 	print "$label_in $value\n";
     } else {

2.18.0.865.gffc8e1a3cd6

s/PP A/PP. A/

(Not worth a re-roll.)

The --file option to grep isn't in POSIX[1], but -f is[1]. Let's check for that in the future, and fix the portability regression in f237c8b6fe ("commit-graph: implement git-commit-graph write", 2018-04-02) that broke e.g. AIX.

1. http://pubs.opengroup.org/onlinepubs/009695399/utilities/grep.html

---
 t/check-non-portable-shell.pl | 1 +
 t/t5318-commit-graph.sh       | 2 +-
 2 files changed, 2 insertions(+), 1 deletion(-)

index 75f38298d7..b45bdac688 100755
@@ -43,6 +43,7 @@ sub err {
 	/\bwc -l.*"\s*=/ and err '`"$(wc -l)"` is not portable (use test_line_count)';
 	/\bhead\s+-c\b/ and err 'head -c is not portable (use test_copy_bytes BYTES <file >out)';
 	/(?:\$\(seq|^\s*seq\b)/ and err 'seq is not portable (use test_seq)';
+	/\bgrep\b.*--file\b/ and err 'grep --file FILE is not portable (use grep -f FILE)';
 	/\bexport\s+[A-Za-z0-9_]*=/ and err '"export FOO=bar" is not portable (use FOO=bar && export FOO)';
 	/^\s*([A-Z0-9_]+=(\w+|(["']).*?\3)\s+)+(\w+)/ and exists($func{$4}) and
 		err '"FOO=bar shell_func" assignment extends beyond "shell_func"';

index 3c1ffad491..0c500f7ca2 100755
@@ -134,7 +134,7 @@ test_expect_success 'Add one more commit' '
 	git branch commits/8 &&
 	ls $objdir/pack | grep idx >existing-idx &&
 	git repack &&
-	ls $objdir/pack| grep idx | grep -v --file=existing-idx >new-idx
+	ls $objdir/pack| grep idx | grep -v -f existing-idx >new-idx
 '
 
 # Current graph structure:

2.18.0.865.gffc8e1a3cd6

The test guarded by PERLJSON added in 75459410ed ("json_writer: new routines to create JSON data", 2018-07-13) assumed that a JSON boolean value like "true" or "false" would be represented as "1" or "0" in Perl.

This behavior can't be relied upon, e.g. with JSON.pm 2.50 and JSON::PP A JSON::PP::Boolean object will be represented as "true" or "false". To work around this let's check if we have any refs left after we check for hashes and arrays, assume those are JSON objects, and coerce them to a known boolean value.

The behavior of this test still looks odd to me. Why implement our own ad-hoc encoder just for some one-off test, as opposed to say Perl's own Data::Dumper with Sortkeys et al? But with this change it works, so let's leave it be.

---
 t/t0019/parse_json.perl | 3 +++
 1 file changed, 3 insertions(+)

index ca4e5bfa78..fea87fb81b 100644
@@ -34,6 +34,9 @@ sub dump_item {
     } elsif (ref($value) eq 'HASH') {
 	print "$label_in hash\n";
 	dump_hash($label_in, $value);
+    } elsif (ref $value) {
+	my $bool = $value ? 1 : 0;
+	print "$label_in $bool\n";
     } elsif (defined $value) {
 	print "$label_in $value\n";
     } else {

2.18.0.865.gffc8e1a3cd6

The --file option to grep isn't in POSIX[1], but -f is[1]. Let's check for that in the future, and fix the portability regression in f237c8b6fe ("commit-graph: implement git-commit-graph write", 2018-04-02) that broke e.g. AIX.

1. http://pubs.opengroup.org/onlinepubs/009695399/utilities/grep.html

---
 t/check-non-portable-shell.pl | 1 +
 t/t5318-commit-graph.sh       | 2 +-
 2 files changed, 2 insertions(+), 1 deletion(-)

index 75f38298d7..b45bdac688 100755
@@ -43,6 +43,7 @@ sub err {
 	/\bwc -l.*"\s*=/ and err '`"$(wc -l)"` is not portable (use test_line_count)';
 	/\bhead\s+-c\b/ and err 'head -c is not portable (use test_copy_bytes BYTES <file >out)';
 	/(?:\$\(seq|^\s*seq\b)/ and err 'seq is not portable (use test_seq)';
+	/\bgrep\b.*--file\b/ and err 'grep --file FILE is not portable (use grep -f FILE)';
 	/\bexport\s+[A-Za-z0-9_]*=/ and err '"export FOO=bar" is not portable (use FOO=bar && export FOO)';
 	/^\s*([A-Z0-9_]+=(\w+|(["']).*?\3)\s+)+(\w+)/ and exists($func{$4}) and
 		err '"FOO=bar shell_func" assignment extends beyond "shell_func"';

index 3c1ffad491..0c500f7ca2 100755
@@ -134,7 +134,7 @@ test_expect_success 'Add one more commit' '
 	git branch commits/8 &&
 	ls $objdir/pack | grep idx >existing-idx &&
 	git repack &&
-	ls $objdir/pack| grep idx | grep -v --file=existing-idx >new-idx
+	ls $objdir/pack| grep idx | grep -v -f existing-idx >new-idx
 '
 
 # Current graph structure:

2.18.0.865.gffc8e1a3cd6

Thanks.

Yes, thanks! I'll keep this in mind for the future.

I would have been a bit more sympathetic if this were recent regression (say, 2.18 or so), even if it is not new in this cycle, and the patch applied to the target track cleanly (say, maint or maint-2.17), but seeing that some of these are quite old, dating back to 2009, I am not sure I am enthused.

The changes certainly look trivial. Let's apply.

Thanks.

Yeah it's not a big deal if it's not in 2.19.

Note that those parts are the "while I'm at it" valgrind-specific parts, i.e. parts of the code that likely never ran on anything except a GNU toolchain.

Here are the topics that have been cooking. Commits prefixed with '-' are only in 'pu' (proposed updates) while commits prefixed with '+' are in 'next'. The ones marked with '.' do not appear in any of the integration branches, but I am still holding onto them.

Git 2.19-rc2 is out. Hopefully the tip of 'master' is more or less identical to the final one without needing much updates.

You can find the changes described here in the integration branches of the repositories listed at

http://git-blame.blogspot.com/p/git-public-repositories.html

--------------------------------------------------
[Graduated to "master"]

* ab/portable (2018-08-27) 6 commits
(merged to 'next' on 2018-08-27 at 37640e66ef)
+ tests: fix and add lint for non-portable grep --file
+ tests: fix version-specific portability issue in Perl JSON
+ tests: use shorter labels in chainlint.sed for AIX sed
+ tests: fix comment syntax in chainlint.sed for AIX sed
+ tests: fix and add lint for non-portable seq
+ tests: fix and add lint for non-portable head -c N (this branch is used by ab/portable-more.)

Portability fix.

* ab/portable-more (2018-08-29) 2 commits (merged to 'next' on 2018-08-31 at d7b44993e4)
+ tests: fix non-portable iconv invocation
+ tests: fix non-portable "${var:-"str"}" construct (this branch uses ab/portable.)

Portability fix.

* ds/commit-graph-lockfile-fix (2018-08-30) 1 commit (merged to 'next' on 2018-09-04 at 0876e6ddcc)
+ commit: don't use generation numbers if not needed

"git merge-base" in 2.19-rc1 has performance regression when the (experimental) commit-graph feature is in use, which has been mitigated.

* en/directory-renames-nothanks (2018-08-30) 3 commits
(merged to 'next' on 2018-08-31 at 91d663d688)
+ am: avoid directory rename detection when calling recursive merge machinery
+ merge-recursive: add ability to turn off directory rename detection
+ t3401: add another directory rename testcase for rebase and am

Recent addition of "directory rename" heuristics to the merge-recursive backend makes the command susceptible to false positives and false negatives. In the context of "git am -3", which does not know about surrounding unmodified paths and thus cannot inform the merge machinery about the full trees involved, this risk is particularly severe. As such, the heuristic is disabled for "git am -3" to keep the machinery "more stupid but predictable".

* es/chain-lint-more (2018-08-29) 1 commit
(merged to 'next' on 2018-08-31 at d456090b62)
+ chainlint: match "quoted" here-doc tags

The test linter code has learned that the end of here-doc mark "EOF" can be quoted in a double-quote pair, not just in a single-quote pair.

* es/freebsd-iconv-portability (2018-08-31) 1 commit (merged to 'next' on 2018-09-04 at 52baa37dd7)
+ config.mak.uname: resolve FreeBSD iconv-related compilation warning

Build fix.

* pw/rebase-i-author-script-fix (2018-08-07) 2 commits (merged to 'next' on 2018-08-31 at 7b9f485407)
+ sequencer: fix quoting in write_author_script
+ sequencer: handle errors from read_author_ident()

Recent "git rebase -i" update started to write bogusly formatted author-script, with a matching broken reading code. These are fixed.

--------------------------------------------------
[New Topics]

* ab/fetch-tags-noclobber (2018-08-31) 9 commits
- fetch: stop clobbering existing tags without --force
- fetch: document local ref updates with/without --force
- push doc: correct lies about how push refspecs work
- push doc: move mention of "tag <tag>" later in the prose
- push doc: remove confusing mention of remote merger
- fetch tests: add a test for clobbering tag behavior
- push tests: use spaces in interpolated string
- push tests: make use of unused $1 in test description
- fetch: change "branch" to "reference" in --force -h output

The rules used by "git push" and "git fetch" to determine if a ref can or cannot be updated were inconsistent; specifically, fetching to update existing tags were allowed even though tags are supposed to be unmoving anchoring points. "git fetch" was taught to forbid updates to existing tags without the "--force" option.

Will merge to and cook in 'next'.
This is a backward incompatible change but in a good way; it may still need to be treated carefully.

* es/worktree-forced-ops-fix (2018-08-30) 9 commits
- worktree: delete .git/worktrees if empty after 'remove'
- worktree: teach 'remove' to override lock when --force given twice
- worktree: teach 'move' to override lock when --force given twice
- worktree: teach 'add' to respect --force for registered but missing path
- worktree: disallow adding same path multiple times
- worktree: prepare for more checks of whether path can become worktree
- worktree: generalize delete_git_dir() to reduce code duplication
- worktree: move delete_git_dir() earlier in file for upcoming new callers
- worktree: don't die() in library function find_worktree()

Various subcommands of "git worktree" take '--force' but did not behave sensibly, which has been corrected.

* jk/patch-corrupted-delta-fix (2018-08-30) 6 commits
- t5303: use printf to generate delta bases
- patch-delta: handle truncated copy parameters
- patch-delta: consistently report corruption
- patch-delta: fix oob read
- t5303: test some corrupt deltas
- test-delta: read input into a heap buffer

Malformed or crafted data in packstream can make our code attempt to read or write past the allocated buffer and abort, instead of reporting an error, which has been fixed.

Will merge to 'next'.

* jc/rebase-in-c-9-fixes (2018-09-04) 1 commit
- rebase: re-add forgotten -k that stands for --keep-empty
(this branch uses ag/rebase-i-in-c, js/rebase-in-c-5.5-work-with-rebase-i-in-c, pk/rebase-in-c, pk/rebase-in-c-2-basic, pk/rebase-in-c-3-acts, pk/rebase-in-c-4-opts, pk/rebase-in-c-5-test and pk/rebase-in-c-6-final.)

* jk/pack-objects-with-bitmap-fix (2018-09-04) 4 commits
- pack-bitmap: drop "loaded" flag
- traverse_bitmap_commit_list(): don't free result
- t5310: test delta reuse with bitmaps
- bitmap_has_sha1_in_uninteresting(): drop BUG check (this branch uses jk/pack-delta-reuse-with-bitmap.)

Hotfix of the base topic. It may not be a bad idea to squash these in when the next development cycle opens.

* js/rebase-i-autosquash-fix (2018-09-04) 2 commits
- rebase -i: be careful to wrap up fixup/squash chains
- rebase -i --autosquash: demonstrate a problem skipping the last squash

"git rebase -i" did not clear the state files correctly when a run of "squash/fixup" is aborted and then the user manually amended the commit instead, which has been corrected.

Will merge to 'next'.

* nd/bisect-show-list-fix (2018-09-04) 1 commit
- bisect.c: make show_list() build again

Debugging aid update.

Will merge to and cook in 'next'.

* nd/optim-reading-index-v4 (2018-09-04) 1 commit
- read-cache.c: optimize reading index format v4

The v4 format of the index file uses prefix compression to store the pathnames to save file size. The codepath to read such a file has been optimized.

Will merge to and cook in 'next'.

* sg/doc-trace-appends (2018-09-04) 1 commit
- Documentation/git.txt: clarify that GIT_TRACE=/path appends

Docfix.

Will merge to 'next'.

--------------------------------------------------
[Stalled]

* sl/commit-dry-run-with-short-output-fix (2018-07-30) 4 commits . commit: fix exit code when doing a dry run
. wt-status: teach wt_status_collect about merges in progress . wt-status: rename commitable to committable
. t7501: add coverage for flags which imply dry runs

"git commit --dry-run" gave a correct exit status even during a conflict resolution toward a merge, but it did not with the "--short" option, which has been corrected.

Seems to break 7512, 3404 and 7060 in 'pu'.

* ma/wrapped-info (2018-05-28) 2 commits
- usage: prefix all lines in vreportf(), not just the first
- usage: extract prefix_suffix_lines() from advise()

An attempt to help making multi-line messages fed to warning(), error(), and friends more easily translatable.

Will discard and wait for a cleaned-up rewrite. cf. <[email protected]>

* hn/bisect-first-parent (2018-04-21) 1 commit
- bisect: create 'bisect_flags' parameter in find_bisection() (this branch is used by tb/bisect-first-parent.)

Preliminary code update to allow passing more flags down the bisection codepath in the future.

We do not add random code that does not have real users to our codebase, so let's have it wait until such a real code materializes before too long.

* av/fsmonitor-updates (2018-01-04) 6 commits
- fsmonitor: use fsmonitor data in git diff
- fsmonitor: remove debugging lines from t/t7519-status-fsmonitor.sh
- fsmonitor: make output of test-dump-fsmonitor more concise
- fsmonitor: update helper tool, now that flags are filled later
- fsmonitor: stop inline'ing mark_fsmonitor_valid / _invalid
- dir.c: update comments to match argument name

Code clean-up on fsmonitor integration, plus optional utilization of the fsmonitor data in diff-files.

Waiting for an update.
cf. <alpine.DEB.2.21.1.1801042335130.32@MININT-6BKU6QN.europe.corp.microsoft.com>

* pb/bisect-helper-2 (2018-07-23) 8 commits
- t6030: make various test to pass GETTEXT_POISON tests
- bisect--helper: bisect_start shell function partially in C
- bisect--helper: get_terms & bisect_terms shell function in C
- bisect--helper: bisect_next_check shell function in C
- bisect--helper: check_and_set_terms shell function in C
- wrapper: move is_empty_file() and rename it as is_empty_or_missing_file()
- bisect--helper: bisect_write shell function in C
- bisect--helper: bisect_reset shell function in C

Expecting a reroll.
cf. <0102015f5e5ee171-f30f4868-886f-47a1-a4e4-b4936afc545d-000000@eu-west-1.amazonses.com>

I just rebased the topic to a newer base as it did not build standalone with the base I originally queued the topic on, but otherwise there is no update to address any of the review comments in the thread above---we are still waiting for a reroll.

* jk/drop-ancient-curl (2017-08-09) 5 commits
- http: #error on too-old curl
- curl: remove ifdef'd code never used with curl >=7.19.4
- http: drop support for curl < 7.19.4
- http: drop support for curl < 7.16.0
- http: drop support for curl < 7.11.1

Some code in http.c that has bitrot is being removed.

Expecting a reroll.

* mk/use-size-t-in-zlib (2017-08-10) 1 commit
. zlib.c: use size_t for size

The wrapper to call into zlib followed our long tradition to use "unsigned long" for sizes of regions in memory, which have been updated to use "size_t".

Needs resurrecting by making sure the fix is good and still applies (or adjusted to today's codebase).

--------------------------------------------------
[Cooking]

* ds/format-commit-graph-docs (2018-08-21) 2 commits
- commit-graph.txt: improve formatting for asciidoc
- Docs: Add commit-graph tech docs to Makefile

Design docs for the commit-graph machinery is now made into HTML as well as text.

* jk/diff-rendered-docs (2018-08-31) 5 commits
- doc/Makefile: drop doc-diff worktree and temporary files on "make clean"
- doc-diff: add --clean mode to remove temporary working gunk
- doc-diff: fix non-portable 'man' invocation
- doc-diff: always use oids inside worktree
(merged to 'next' on 2018-08-22 at dd7a2b71cd)
+ SubmittingPatches: mention doc-diff

Dev doc update.

Will merge to 'next'.

* jk/pack-delta-reuse-with-bitmap (2018-08-21) 6 commits (merged to 'next' on 2018-08-22 at fc50b59dab)
+ pack-objects: reuse on-disk deltas for thin "have" objects
+ pack-bitmap: save "have" bitmap from walk
+ t/perf: add perf tests for fetches from a bitmapped server
+ t/perf: add infrastructure for measuring sizes
+ t/perf: factor out percent calculations
+ t/perf: factor boilerplate out of test_perf (this branch is used by jk/pack-objects-with-bitmap-fix.)

When creating a thin pack, which allows objects to be made into a delta against another object that is not in the resulting pack but is known to be present on the receiving end, the code learned to take advantage of the reachability bitmap; this allows the server to send a delta against a base beyond the "boundary" commit.

Will cook in 'next'.

* jk/rev-list-stdin-noop-is-ok (2018-08-22) 1 commit (merged to 'next' on 2018-08-27 at d5916f7bc1)
+ rev-list: make empty --stdin not an error

"git rev-list --stdin </dev/null" used to be an error; it now shows no output without an error. "git rev-list --stdin --default HEAD" still falls back to the given default when nothing is given on the standard input.

Will cook in 'next'.

* js/rebase-in-c-5.5-work-with-rebase-i-in-c (2018-08-29) 2 commits
- builtin rebase: prepare for builtin rebase -i
- Merge branch 'ag/rebase-i-in-c' into js/rebase-in-c-5.5-work-with-rebase-i-in-c
(this branch is used by jc/rebase-in-c-9-fixes and pk/rebase-in-c-6-final; uses ag/rebase-i-in-c, pk/rebase-in-c, pk/rebase-in-c-2-basic, pk/rebase-in-c-3-acts, pk/rebase-in-c-4-opts and pk/rebase-in-c-5-test.)

"rebase" that has been rewritten learns the new calling convention used by "rebase -i" that was rewritten in C, tying the loose end between two GSoC topics that stomped on each other's toes.

* jk/trailer-fixes (2018-08-23) 8 commits
(merged to 'next' on 2018-08-27 at 93b671b8c6)
+ append_signoff: use size_t for string offsets
+ sequencer: ignore "---" divider when parsing trailers
+ pretty, ref-filter: format %(trailers) with no_divider option
+ interpret-trailers: allow suppressing "---" divider
+ interpret-trailers: tighten check for "---" patch boundary
+ trailer: pass process_trailer_opts to trailer_info_get()
+ trailer: use size_t for iterating trailer list
+ trailer: use size_t for string offsets

"git interpret-trailers" and its underlying machinery had a buggy code that attempted to ignore patch text after commit log message, which triggered in various codepaths that will always get the log message alone and never get such an input.

Will cook in 'next'.

* tg/rerere-doc-updates (2018-08-29) 2 commits (merged to 'next' on 2018-08-31 at ce4fef1a97)
+ rerere: add note about files with existing conflict markers
+ rerere: mention caveat about unmatched conflict markers (this branch uses tg/rerere.)

Clarify a part of technical documentation for rerere.

Will cook in 'next'.

* ds/commit-graph-tests (2018-08-29) 1 commit
- commit-graph: define GIT_TEST_COMMIT_GRAPH

We can now optionally run tests with commit-graph enabled.

Will merge to 'next'.

* jk/cocci (2018-08-29) 9 commits
(merged to 'next' on 2018-08-31 at 914b4f17ce)
+ show_dirstat: simplify same-content check
+ read-cache: use oideq() in ce_compare functions
+ convert hashmap comparison functions to oideq()
+ convert "hashcmp() != 0" to "!hasheq()"
+ convert "oidcmp() != 0" to "!oideq()"
+ convert "hashcmp() == 0" to hasheq()
+ convert "oidcmp() == 0" to oideq()
+ introduce hasheq() and oideq()
+ coccinelle: use <...> for function exclusion

spatch transformation to replace boolean uses of !hashcmp() to newly introduced oideq() is added, and applied, to regain performance lost due to support of multiple hash algorithms.

Will cook in 'next'.

* js/add-i-coalesce-after-editing-hunk (2018-08-28) 1 commit
- add -p: coalesce hunks before testing applicability

Applicability check after a patch is edited in a "git add -i/p" session has been improved.

Will hold.
cf. <[email protected]>

* rs/mailinfo-format-flowed (2018-08-29) 1 commit
(merged to 'next' on 2018-08-31 at 9e8b92176c)
+ mailinfo: support format=flowed

"git mailinfo" used in "git am" learned to make a best-effort recovery of a patch corrupted by MUA that sends text/plain with format=flawed option.

Will cook in 'next'.

* sb/submodule-move-head-with-corruption (2018-08-28) 2 commits
- submodule.c: warn about missing submodule git directories
- t2013: add test for missing but active submodule

A corner case in switching to a branch that needs to "check out" a submodule is handled a bit better now.

Expecting a reroll.

* tg/conflict-marker-size (2018-08-29) 1 commit (merged to 'next' on 2018-08-31 at 12099161f0)
+ .gitattributes: add conflict-marker-size for relevant files

Developer aid.

Will cook in 'next'.

* ts/doc-build-manpage-xsl-quietly (2018-08-29) 1 commit (merged to 'next' on 2018-08-31 at 7527e0f8d3)
+ Documentation/Makefile: make manpage-base-url.xsl generation quieter

Build tweak.

Will cook in 'next'.

* bp/checkout-new-branch-optim (2018-08-16) 1 commit (merged to 'next' on 2018-08-27 at e69bfd115f)
+ checkout: optimize "git checkout -b <new_branch>"

"git checkout -b newbranch [HEAD]" should not have to do as much as checking out a commit different from HEAD. An attempt is made to optimize this special case.

Will cook in 'next'.

* ao/submodule-wo-gitmodules-checked-out (2018-08-14) 7 commits
- submodule: support reading .gitmodules even when it's not checked out
- t7506: clean up .gitmodules properly before setting up new scenario
- submodule: use the 'submodule--helper config' command
- submodule--helper: add a new 'config' subcommand
- t7411: be nicer to future tests and really clean things up
- submodule: factor out a config_set_in_gitmodules_file_gently function
- submodule: add a print_config_from_gitmodules() helper

The submodule support has been updated to read from the blob at HEAD:.gitmodules when the .gitmodules file is missing from the working tree.

Expecting a reroll.

I find the design a bit iffy in that our usual "missing in the working tree? let's use the latest blob" fallback is to take it from the index, not from the HEAD.

* bw/submodule-name-to-dir (2018-08-10) 2 commits (merged to 'next' on 2018-08-22 at c17f83be24)
+ submodule: munge paths to submodule git directories
+ submodule: create helper to build paths to submodule gitdirs

In modern repository layout, the real body of a cloned submodule repository is held in .git/modules/ of the superproject, indexed by the submodule name. URLencode the submodule name before computing the name of the directory to make sure they form a flat namespace.

Expecting further work on the topic.
cf. <CAGZ79kYnbjaPoWdda0SM_-_X77mVyYC7JO61OV8nm2yj3Q1OvQ@mail.gmail.com>

* cc/delta-islands (2018-08-16) 7 commits
(merged to 'next' on 2018-08-27 at cf3d7bd93f)
+ pack-objects: move 'layer' into 'struct packing_data'
+ pack-objects: move tree_depth into 'struct packing_data'
+ t5320: tests for delta islands
+ repack: add delta-islands support
+ pack-objects: add delta-islands support
+ pack-objects: refactor code into compute_layer_order()
+ Add delta-islands.{c,h}

Lift code from GitHub to restrict delta computation so that an object that exists in one fork is not made into a delta against another object that does not appear in the same forked repository.

Will cook in 'next'.

* md/filter-trees (2018-09-04) 7 commits
- list-objects-filter: implement filter tree:0
- list-objects-filter: use BUG rather than die
- revision: mark non-user-given objects instead
- rev-list: handle missing tree objects properly
- list-objects: always parse trees gently
- list-objects: refactor to process_tree_contents
- list-objects: store common func args in struct

The "rev-list --filter" feature learned to exclude all trees via "tree:0" filter.

Will merge to 'next'.

* ng/status-i-short-for-ignored (2018-08-09) 1 commit
- status: -i shorthand for --ignored command line option

"git status --ignored" gained a shorthand "git status -i".

What's the list opinion on this one? It is Meh to me, but obviously the author cared enough to write a patch, so...

* pk/rebase-in-c-2-basic (2018-08-10) 11 commits
- builtin rebase: support git rebase <upstream> <switch-to>
- builtin rebase: only store fully-qualified refs in options.head_name
- builtin rebase: start a new rebase only if none is in progress
- builtin rebase: support --force-rebase
- builtin rebase: try to fast forward when possible
- builtin rebase: require a clean worktree
- builtin rebase: support the verbose and diffstat options
- builtin rebase: support --quiet
- builtin rebase: handle the pre-rebase hook (and add --no-verify)
- builtin rebase: support git rebase --onto A...B
- builtin rebase: support --onto
(this branch is used by jc/rebase-in-c-9-fixes, js/rebase-in-c-5.5-work-with-rebase-i-in-c, pk/rebase-in-c-3-acts, pk/rebase-in-c-4-opts, pk/rebase-in-c-5-test and pk/rebase-in-c-6-final; uses pk/rebase-in-c.)

* pk/rebase-in-c-3-acts (2018-08-10) 7 commits
- builtin rebase: stop if git am is in progress
- builtin rebase: actions require a rebase in progress
- builtin rebase: support --edit-todo and --show-current-patch
- builtin rebase: support --quit
- builtin rebase: support --abort
- builtin rebase: support --skip
- builtin rebase: support --continue
(this branch is used by jc/rebase-in-c-9-fixes, js/rebase-in-c-5.5-work-with-rebase-i-in-c, pk/rebase-in-c-4-opts, pk/rebase-in-c-5-test and pk/rebase-in-c-6-final; uses pk/rebase-in-c and pk/rebase-in-c-2-basic.)

* pk/rebase-in-c-4-opts (2018-08-10) 18 commits
- builtin rebase: support --root
- builtin rebase: add support for custom merge strategies
- builtin rebase: support fork-point option
- merge-base --fork-point: extract libified function
- builtin rebase: support --rebase-merges[=[no-]rebase-cousins]
- builtin rebase: support --allow-empty-message option
- builtin rebase: support --exec
- builtin rebase: support --autostash option
- builtin rebase: support -C and --whitespace=<type>
- builtin rebase: support --gpg-sign option
- builtin rebase: support --autosquash
- builtin rebase: support keep-empty option
- builtin rebase: support ignore-date option
- builtin rebase: support ignore-whitespace option
- builtin rebase: support --committer-date-is-author-date
- builtin rebase: support --rerere-autoupdate
- builtin rebase: support --signoff
- builtin rebase: allow selecting the rebase "backend"
(this branch is used by jc/rebase-in-c-9-fixes, js/rebase-in-c-5.5-work-with-rebase-i-in-c, pk/rebase-in-c-5-test and pk/rebase-in-c-6-final; uses pk/rebase-in-c, pk/rebase-in-c-2-basic and pk/rebase-in-c-3-acts.)

* pk/rebase-in-c-5-test (2018-08-10) 6 commits
- builtin rebase: error out on incompatible option/mode combinations
- builtin rebase: use no-op editor when interactive is "implied"
- builtin rebase: show progress when connected to a terminal
- builtin rebase: fast-forward to onto if it is a proper descendant
- builtin rebase: optionally pass custom reflogs to reset_head()
- builtin rebase: optionally auto-detect the upstream
(this branch is used by jc/rebase-in-c-9-fixes, js/rebase-in-c-5.5-work-with-rebase-i-in-c and pk/rebase-in-c-6-final; uses pk/rebase-in-c, pk/rebase-in-c-2-basic, pk/rebase-in-c-3-acts and pk/rebase-in-c-4-opts.)

* pk/rebase-in-c-6-final (2018-08-29) 1 commit
- rebase: default to using the builtin rebase
(this branch is used by jc/rebase-in-c-9-fixes; uses ag/rebase-i-in-c, js/rebase-in-c-5.5-work-with-rebase-i-in-c, pk/rebase-in-c, pk/rebase-in-c-2-basic, pk/rebase-in-c-3-acts, pk/rebase-in-c-4-opts and pk/rebase-in-c-5-test.)

* ps/stash-in-c (2018-08-31) 20 commits
- stash: replace all write-tree child processes with API calls
- stash: optimize get_untracked_files() and check_changes()
- stash: convert stash--helper.c into stash.c
- stash: convert save to builtin
- stash: make push -q quiet
- stash: convert push to builtin
- stash: convert create to builtin
- stash: convert store to builtin
- stash: mention options in show synopsis
- stash: convert show to builtin
- stash: convert list to builtin
- stash: convert pop to builtin
- stash: convert branch to builtin
- stash: convert drop and clear to builtin
- stash: convert apply to builtin
- stash: add tests for git stash show config
- stash: rename test cases to be more descriptive
- stash: update test cases conform to coding guidelines
- stash: improve option parsing test coverage
- sha1-name.c: add get_oidf() which acts like get_oid()

* nd/clone-case-smashing-warning (2018-08-17) 1 commit (merged to 'next' on 2018-08-22 at eedae40a8d)
+ clone: report duplicate entries on case-insensitive filesystems

Running "git clone" against a project that contain two files with pathnames that differ only in cases on a case insensitive filesystem would result in one of the files lost because the underlying filesystem is incapable of holding both at the same time. An attempt is made to detect such a case and warn.

Will cook in 'next'.

* nd/unpack-trees-with-cache-tree (2018-08-27) 8 commits (merged to 'next' on 2018-08-27 at b1d841d034)
+ Document update for nd/unpack-trees-with-cache-tree (merged to 'next' on 2018-08-22 at 138b902673)
+ cache-tree: verify valid cache-tree in the test suite
+ unpack-trees: add missing cache invalidation
+ unpack-trees: reuse (still valid) cache-tree from src_index
+ unpack-trees: reduce malloc in cache-tree walk
+ unpack-trees: optimize walking same trees with cache-tree
+ unpack-trees: add performance tracing
+ trace.h: support nested performance tracing

The unpack_trees() API used in checking out a branch and merging walks one or more trees along with the index. When the cache-tree in the index tells us that we are walking a tree whose flattened contents is known (i.e. matches a span in the index), as linearly scanning a span in the index is much more efficient than having to open tree objects recursively and listing their entries, the walk can be optimized, which is done in this topic.

Will cook in 'next'.

* sb/range-diff-colors (2018-08-20) 11 commits (merged to 'next' on 2018-08-22 at eb7ed4fca3)
+ range-diff: indent special lines as context
+ range-diff: make use of different output indicators
+ diff.c: add --output-indicator-{new, old, context}
+ diff.c: rewrite emit_line_0 more understandably
+ diff.c: omit check for line prefix in emit_line_0
+ diff: use emit_line_0 once per line
+ diff.c: add set_sign to emit_line_0
+ diff.c: reorder arguments for emit_line_ws_markup
+ diff.c: simplify caller of emit_line_0
+ t3206: add color test for range-diff --dual-color
+ test_decode_color: understand FAINT and ITALIC

The color output support for recently introduced "range-diff" command got tweaked a bit.

Will cook in 'next'.

* sg/t1404-update-ref-test-timeout (2018-08-01) 1 commit (merged to 'next' on 2018-08-22 at f3cd37b5ea)
+ t1404: increase core.packedRefsTimeout to avoid occasional test failure

An attempt to unflake a test a bit.

Will cook in 'next'.

* pw/add-p-select (2018-07-26) 4 commits
- add -p: optimize line selection for short hunks
- add -p: allow line selection to be inverted
- add -p: select modified lines correctly
- add -p: select individual hunk lines

"git add -p" interactive interface learned to let users choose individual added/removed lines to be used in the operation, instead of accepting or rejecting a whole hunk.

Will hold.
cf. <[email protected]> I found the feature to be hard to explain, and may result in more end-user complaints, but let's see.

* ds/commit-graph-with-grafts (2018-08-21) 8 commits
- commit-graph: close_commit_graph before shallow walk
- commit-graph: not compatible with uninitialized repo
- commit-graph: not compatible with grafts
- commit-graph: not compatible with replace objects
- test-repository: properly init repo
- commit-graph: update design document
- refs.c: upgrade for_each_replace_ref to be a each_repo_ref_fn callback
- refs.c: migrate internal ref iteration to pass thru repository argument

The recently introduced commit-graph auxiliary data is incompatible with mechanisms such as replace & grafts that "breaks" immutable nature of the object reference relationship. Disable optimizations based on its use (and updating existing commit-graph) when these incompatible features are in use in the repository.

Replaced with a newer version.

* ds/reachable (2018-08-28) 19 commits (merged to 'next' on 2018-08-28 at b1634b371d)
+ commit-reach: correct accidental #include of C file (merged to 'next' on 2018-08-22 at 17f3275afb)
+ commit-reach: use can_all_from_reach
+ commit-reach: make can_all_from_reach... linear
+ commit-reach: replace ref_newer logic
+ test-reach: test commit_contains
+ test-reach: test can_all_from_reach_with_flags
+ test-reach: test reduce_heads
+ test-reach: test get_merge_bases_many
+ test-reach: test is_descendant_of
+ test-reach: test in_merge_bases
+ test-reach: create new test tool for ref_newer
+ commit-reach: move can_all_from_reach_with_flags
+ upload-pack: generalize commit date cutoff
+ upload-pack: refactor ok_to_give_up()
+ upload-pack: make reachable() more generic
+ commit-reach: move commit_contains from ref-filter
+ commit-reach: move ref_newer from remote.c
+ commit.h: remove method declarations
+ commit-reach: move walk methods from commit.c

The code for computing history reachability has been shuffled, obtained a bunch of new tests to cover them, and then being improved.

Will cook in 'next'.

* es/format-patch-interdiff (2018-07-23) 6 commits
(merged to 'next' on 2018-08-31 at 63927e0227)
+ format-patch: allow --interdiff to apply to a lone-patch
+ log-tree: show_log: make commentary block delimiting reusable
+ interdiff: teach show_interdiff() to indent interdiff
+ format-patch: teach --interdiff to respect -v/--reroll-count
+ format-patch: add --interdiff option to embed diff in cover letter
+ format-patch: allow additional generated content in make_cover_letter() (this branch is used by es/format-patch-rangediff.)

"git format-patch" learned a new "--interdiff" option to explain the difference between this version and the previous atttempt in the cover letter (or after the tree-dashes as a comment).

Will cook in 'next'.

* es/format-patch-rangediff (2018-08-14) 10 commits
(merged to 'next' on 2018-08-31 at 65627afece)
+ format-patch: allow --range-diff to apply to a lone-patch
+ format-patch: add --creation-factor tweak for --range-diff
+ format-patch: teach --range-diff to respect -v/--reroll-count
+ format-patch: extend --range-diff to accept revision range
+ format-patch: add --range-diff option to embed diff in cover letter
+ range-diff: relieve callers of low-level configuration burden
+ range-diff: publish default creation factor
+ range-diff: respect diff_option.file rather than assuming 'stdout'
+ Merge branch 'es/format-patch-interdiff' into es/format-patch-rangediff
+ Merge branch 'js/range-diff' into es/format-patch-rangediff (this branch uses es/format-patch-interdiff.)

"git format-patch" learned a new "--range-diff" option to explain the difference between this version and the previous attempt in the cover letter (or after the tree-dashes as a comment).

Will cook in 'next'.

* jn/gc-auto (2018-07-17) 3 commits
- gc: do not return error for prior errors in daemonized mode
- gc: exit with status 128 on failure
- gc: improve handling of errors reading gc.log

"gc --auto" ended up calling exit(-1) upon error, which has been corrected to use exit(1). Also the error reporting behaviour when daemonized has been updated to exit with zero status when stopping due to a previously discovered error (which implies there is no point running gc to improve the situation); we used to exit with failure in such a case.

What's the donness of this one?
cf. <[email protected]>

* sb/submodule-update-in-c (2018-08-14) 7 commits
(merged to 'next' on 2018-08-17 at 23c81e5ff7)
+ submodule--helper: introduce new update-module-mode helper
+ submodule--helper: replace connect-gitdir-workingtree by ensure-core-worktree
+ builtin/submodule--helper: factor out method to update a single submodule
+ builtin/submodule--helper: store update_clone information in a struct
+ builtin/submodule--helper: factor out submodule updating
+ git-submodule.sh: rename unused variables
+ git-submodule.sh: align error reporting for update mode to use path

"git submodule update" is getting rewritten piece-by-piece into C.

Will cook in 'next'.

* tg/rerere (2018-08-06) 11 commits
(merged to 'next' on 2018-08-17 at 919a958cdc)
+ rerere: recalculate conflict ID when unresolved conflict is committed
+ rerere: teach rerere to handle nested conflicts
+ rerere: return strbuf from handle path
+ rerere: factor out handle_conflict function
+ rerere: only return whether a path has conflicts or not
+ rerere: fix crash with files rerere can't handle
+ rerere: add documentation for conflict normalization
+ rerere: mark strings for translation
+ rerere: wrap paths in output in sq
+ rerere: lowercase error messages
+ rerere: unify error messages when read_cache fails
(this branch is used by tg/rerere-doc-updates.)

Fixes to "git rerere" corner cases, especially when conflict markers cannot be parsed in the file.

Will cook in 'next'.

* ag/rebase-i-in-c (2018-08-29) 20 commits
- rebase -i: move rebase--helper modes to rebase--interactive
- rebase -i: remove git-rebase--interactive.sh
- rebase--interactive2: rewrite the submodes of interactive rebase in C
- rebase -i: implement the main part of interactive rebase as a builtin
- rebase -i: rewrite init_basic_state() in C
- rebase -i: rewrite write_basic_state() in C
- rebase -i: rewrite the rest of init_revisions_and_shortrevisions() in C
- rebase -i: implement the logic to initialize $revisions in C
- rebase -i: remove unused modes and functions
- rebase -i: rewrite complete_action() in C
- t3404: todo list with commented-out commands only aborts
- sequencer: change the way skip_unnecessary_picks() returns its result
- sequencer: refactor append_todo_help() to write its message to a buffer
- rebase -i: rewrite checkout_onto() in C
- rebase -i: rewrite setup_reflog_action() in C
- sequencer: add a new function to silence a command, except if it fails
- rebase -i: rewrite the edit-todo functionality in C
- editor: add a function to launch the sequence editor
- rebase -i: rewrite append_todo_help() in C
- sequencer: make three functions and an enum from sequencer.c public
(this branch is used by jc/rebase-in-c-9-fixes, js/rebase-in-c-5.5-work-with-rebase-i-in-c and pk/rebase-in-c-6-final.)

Rewrite of the remaining "rebase -i" machinery in C.

* lt/date-human (2018-07-09) 1 commit
- Add 'human' date format

A new date format "--date=human" that morphs its output depending on how far the time is from the current time has been introduced. "--date=auto" can be used to use this new format when the output is goint to the pager or to the terminal and otherwise the default format.

* pk/rebase-in-c (2018-08-06) 3 commits
- builtin/rebase: support running "git rebase <upstream>"
- rebase: refactor common shell functions into their own file
- rebase: start implementing it as a builtin
(this branch is used by jc/rebase-in-c-9-fixes, js/rebase-in-c-5.5-work-with-rebase-i-in-c, pk/rebase-in-c-2-basic, pk/rebase-in-c-3-acts, pk/rebase-in-c-4-opts, pk/rebase-in-c-5-test and pk/rebase-in-c-6-final.)

Rewrite of the "rebase" machinery in C.

* jk/branch-l-1-repurpose (2018-08-30) 2 commits (merged to 'next' on 2018-08-31 at cfa73bbfcb)
+ doc/git-branch: remove obsolete "-l" references (merged to 'next' on 2018-08-08 at d2a08dd08e)
+ branch: make "-l" a synonym for "--list"

Updated plan to repurpose the "-l" option to "git branch".

Will cook in 'next'.

* ds/multi-pack-index (2018-08-20) 33 commits (merged to 'next' on 2018-08-21 at d15e8cadd4)
+ pack-objects: consider packs in multi-pack-index
+ midx: test a few commands that use get_all_packs
+ treewide: use get_all_packs
+ packfile: add all_packs list
+ midx: fix bug that skips midx with alternates
+ midx: stop reporting garbage
+ midx: mark bad packed objects
+ multi-pack-index: store local property
+ multi-pack-index: provide more helpful usage info
+ Sync 'ds/multi-pack-index' to v2.19.0-rc0 (merged to 'next' on 2018-08-08 at 1a56c52967)
+ midx: clear midx on repack
+ packfile: skip loading index if in multi-pack-index
+ midx: prevent duplicate packfile loads
+ midx: use midx in approximate_object_count
+ midx: use existing midx when writing new one
+ midx: use midx in abbreviation calculations
+ midx: read objects from multi-pack-index
+ config: create core.multiPackIndex setting
+ midx: write object offsets
+ midx: write object id fanout chunk
+ midx: write object ids in a chunk
+ midx: sort and deduplicate objects from packfiles
+ midx: read pack names into array
+ multi-pack-index: write pack names in chunk
+ multi-pack-index: read packfile list
+ packfile: generalize pack directory list
+ t5319: expand test data
+ multi-pack-index: load into memory
+ midx: write header information to lockfile
+ multi-pack-index: add 'write' verb
+ multi-pack-index: add builtin
+ multi-pack-index: add format details
+ multi-pack-index: add design document

When there are too many packfiles in a repository (which is not recommended), looking up an object in these would require consulting many pack .idx files; a new mechanism to have a single file that consolidates all of these .idx files is introduced.

Will cook in 'next'.

--------------------------------------------------
[Discarded]

* am/sequencer-author-script-fix (2018-07-18) 1 commit . sequencer.c: terminate the last line of author-script properly

The author-script that records the author information created by the sequencer machinery lacked the closing single quote on the last entry.

Superseded by another topic.

* jc/push-cas-opt-comment (2018-08-01) 1 commit . push: comment on a funny unbalanced option help

Code clarification.

Superseded by another topic.

* cc/remote-odb (2018-08-02) 9 commits
. Documentation/config: add odb.<name>.promisorRemote
. t0410: test fetching from many promisor remotes
. Use odb.origin.partialclonefilter instead of core.partialclonefilter . Use remote_odb_get_direct() and has_remote_odb()
. remote-odb: add remote_odb_reinit()
. remote-odb: implement remote_odb_get_many_direct()
. remote-odb: implement remote_odb_get_direct()
. Add initial remote odb support
. fetch-object: make functions return an error code

Implement lazy fetches of missing objects to complement the experimental partial clone feature.

Ejected; seems to break existing repositories that use partialclone repository extension.

* jh/structured-logging (2018-08-28) 26 commits
. SQUASH??? spatch fix
. structured-logging: add config data facility
. structured-logging: t0420 tests for interacitve child_summary . structured-logging: t0420 tests for child process detail events . structured-logging: add child process classification
. structured-logging: add detail-events for child processes . structured-logging: add structured logging to remote-curl . structured-logging: t0420 tests for aux-data
. structured-logging: add aux-data for size of sparse-checkout file . structured-logging: add aux-data for index size
. structured-logging: add aux-data facility
. structured-logging: t0420 tests for timers
. structured-logging: add timer around preload_index
. structured-logging: add timer around wt-status functions . structured-logging: add timer around do_write_index
. structured-logging: add timer around do_read_index
. structured-logging: add timer facility
. structured-logging: add detail-event for lazy_init_name_hash . structured-logging: add detail-event facility
. structured-logging: t0420 basic tests
. structured-logging: set sub_command field for checkout command . structured-logging: set sub_command field for branch command . structured-logging: add session-id to log events
. structured-logging: add structured logging framework
. structured-logging: add STRUCTURED_LOGGING=1 to Makefile . structured-logging: design document

Being rerolled with an updated tracing API.

I recently gained access to a Solaris 10 SPARC (5.10) box and discovered that the chainlint.sed implementation in 2.19.0 has more sed portability issues.

First, whoever implemented the /bin/sed on Solaris apparently read the POSIX requirements for a max length label of 8 to mean that 8 characters should include the colon, so a bunch of things fail because of that, but are fixed with a shorter 7 character label.

Then GIT_TEST_CHAIN_LINT=1 still fails because 878f988350 ("t/test-lib:
teach --chain-lint to detect broken &&-chains in subshells", 2018-07-11) added a "grep -q" invocation. The /bin/grep on that version of Solaris doesn't have -q. We fixed a similar issue way back in 80700fde91 ("t/t1304: make a second colon optional in the mask ACL check", 2010-03-15) by redirecting to /dev/null instead.

A bunch of other tests in the test suite rely on "grep -q", but nothing as central as chainlint, so it makes everything break. Do we want to away with "grep -q" entirely because of old Solaris /bin/grep?

At this point those familiar with Solaris are screaming ("why are you using anything in /bin!"). Okey fine, but it mostly worked before, so are we OK with breaking it? "Mostly" here is "test suite would fail 20-30 tests for various reasons, but at least no failures in test-lib.sh and the like".

However, if as config.mak.uname does we run the tests with PATH=/usr/xpg6/bin:/usr/xpg4/bin:$PATH, at this point sed is fine with 8 character labels, but starts complaining about this (also in chainlint.sed):

sed: Too many commands, last: s/\n//

As with other sed issues I fixed recently in chainlint.sed this one is just the tip of the iceberg. Once you "fix" one (just remove it, I have no idea how to rewrite it) others appear.

I was hoping this would just be a Solaris 10 issue, but it's also an issue in Solaris 11 (5.11 11.3).

So, do we want to chase this down or just do this?:

    diff --git a/Makefile b/Makefile
    index 5a969f5..f125dc5 100644
    --- a/Makefile
    +++ b/Makefile
    @@ -2602,6 +2602,9 @@ endif
     ifdef TEST_GIT_INDEX_VERSION
            @echo TEST_GIT_INDEX_VERSION=\''$(subst ','\'',$(subst ','\'',$(TEST_GIT_INDEX_VERSION)))'\' >>$@+
     endif
    +ifdef GIT_TEST_CHAIN_LINT
    +       @echo GIT_TEST_CHAIN_LINT=\''$(subst ','\'',$(subst ','\'',$(GIT_TEST_CHAIN_LINT)))'\' >>$@+
    +endif
            @if cmp $@+ $@ >/dev/null 2>&1; then $(RM) $@+; else mv $@+ $@; fi

     ### Detect Python interpreter path changes
    diff --git a/config.mak.uname b/config.mak.uname
    index e47af72..2b02a2b 100644
    --- a/config.mak.uname
    +++ b/config.mak.uname
    @@ -163,6 +163,10 @@ ifeq ($(uname_S),SunOS)
            INSTALL = /usr/ucb/install
            TAR = gtar
            BASIC_CFLAGS += -D__EXTENSIONS__ -D__sun__
    +       # t/chainlint.sed is hopelessly broken all known (tested
    +       # Solaris 10 & 11) versions of Solaris, both /bin/sed and
    +       # /usr/xpg4/bin/sed
    +       GIT_TEST_CHAIN_LINT = 0
     endif
     ifeq ($(uname_O),Cygwin)
            ifeq ($(shell expr "$(uname_R)" : '1\.[1-6]\.'),4)

If I could wave a magic wand I'd say "let's just rewrite chainlint.sed in perl, at least that's portable", but that's a lot of effort (and I doubt Eric wants to). It slightly sucks to not have chainlint on Solaris, but it would also suck to revert chainlint.sed back to 2.18.0 (there were some big improvements). So I think the patch above is the best way forward, especially since we're on rc2. What do you think?

I'm pretty sure that Solaris 'sed' predates POSIX by a good bit, but that's neither here nor there.

I knew that '-q' was potentially problematic on some platforms, so I checked and saw that existing tests were already using it, thus went ahead and used it. Dropping '-q' here and redirecting stderr to /dev/null is a perfectly fine alternative.

I count 132 instances in existing tests (though, I may have missed some).

So, if you run the tests via 'make test' (or whatever), then you get /usr/xpg?/bin in PATH, but if you run an individual test script (and haven't set your PATH appropriately), then you encounter the problems you report above?

Erm. Any additional context to help narrow this down?

Keeping in mind that the main goal of 'chainlint' is to prevent _new_ breakage from entering the test suite, disabling 'chainlint' on Solaris is an entirely reasonable way forward. In present day, it seems quite unlikely that we'll see someone develop new tests on Solaris, so having 'chainlint' disabled there isn't likely to be a big deal. Moreover, if someone does write a new test on Solaris which has a broken &&-chain in a subshell, that breakage will be caught very quickly once the test is run by anyone on Linux or MacOS (or Windows or BSD or AIX), so it's not like the broken test will make it into the project undetected.

You need to manually set the PATH before running the tests, the SANE_TOOL_PATH just affects git-sh-setup. We should probably fix that, i.e. now if you compile git our shellscripts will use the fixed paths, but not the tests.

I tried playing around with it a bit, but honestly don't understand enough of this advanced sed syntax to make any headway, it's complaining about e.g. the "check for multi-line double-quoted string" rule, but then if you remove the s/\n// from there it complains about the next rule in that format.

If you want to dig into this yourself I think the best way forward is the last two paragraphs of
https://public-inbox.org/git/[email protected]/

Since writing my initial message I see that the CSW package packaging git on Solaris just uses GNU userland:
https://sourceforge.net/p/gar/code/HEAD/tree/csw/mgar/pkg/git/trunk/Makefile

I.e. it puts /opt/csw/gnu in its $PATH, so this whole thing is probably a non-issue. I.e. if the near-as-you-can-get "official" package just compiles git against GNU tooling, perhaps we should stop caring about Solaris-native tooling.

That does also mean that something like my patch above isn't OK, since we shouldn't be disabling GIT_TEST_CHAIN_LINT based on the uname, if it then turns out that GNU tooling is being used.

So I think for the purposes of v2.19.0-rc2..v2.19.0 it's fine to just leave this as it is in master right now.

It's somewhat of a shame that we're drifting to not supporing some of these more obscure POSIX systems "natively" though, but maybe that's the right move at this point, and maybe it isn't.

When I was porting C code to Solaris ~10 years ago I'd often get SunCC turning up some interesting issues that were *real* issues, same with compiling with IBM xlc on AIX.

Now when I'm re-visiting these systems much later I've yet to turn up some issue like that, just boring compatibility issues with their old tooling. E.g. one outstanding issue is that some test of ours fail on AIX because we use "readlink", but that command isn't in POSIX (only the C function is), so AIX doesn't implement it.

SunCC used to be ahead of GCC & Clang when it came to certain classes of warnings, but e.g. now everything it complains about is because it doesn't understand C as well, e.g. we have quite a few compile warnings due to code like this, which it claims is unreachable (but isn't):
https://github.com/git/git/blob/v2.19.0-rc2/read-cache.c#L950-L955

Correction. It is still useful for something:
https://public-inbox.org/git/[email protected]/

[]

Wait a second - even if the compiler claims something (wrong)... there a still 1+1/2 questions from my side:

int verify_path(const char *path, unsigned mode)

{
	char c;
	     ^
	/* Q1: should  "c" be initialized like this: */
	char c = *path;
        
	if (has_dos_drive_prefix(path))
		return 0;

	goto inside;
	^^^^^^^^^^^^ /* Q2: and why do we need the "goto" here ? */
	for (;;) {
		if (!c)
			return 1;
		if (is_dir_sep(c)) {

inside:

After some re-reading,
I think that the "goto inside" was just hard to read....

Out of interest:
would the following make the compiler happy ?

index 49add63fe1..d574d58b9d 100644
@@ -951,17 +951,15 @@ static int verify_dotfile(const char *rest, unsigned mode)
 
 int verify_path(const char *path, unsigned mode)
 {
-	char c;
+	char c = *path ? '/' : '\0';
 
 	if (has_dos_drive_prefix(path))
 		return 0;
 
-	goto inside;
 	for (;;) {
 		if (!c)
 			return 1;
 		if (is_dir_sep(c)) {
-inside:
 			if (protect_hfs) {
 				if (is_hfs_dotgit(path))
 					return 0;

I thought that we long time ago declared that it is a lost cause to use Solaris with only tools in /bin and /usr/bin and instead depend on /usr/xpg[46]/bin; did something happen recently to make us reconsider the position? If not, let's not waste time worrying about /bin/sed over there.

This description mischaracterizes what these changes are about. The primary intent of this series is to fix a bug in which the same path can be registered under multiple worktree entries.

As for --force, it worked perfectly fine for the couple git-worktree subcommands which accepted it. The patches in this series dealing with --force are merely extending it to other subcommands or to other use-cases.

So, perhaps rewrite this description like this:

Fix a bug in which the same path could be registered under multiple worktree entries if the patch was missing (for instance, was removed manually).

Also, as a convenience, expand the number of cases in which --force is applicable.

Thanks for a replacement text; very much appreciated.

Bah, typo: s/patch/path/

This series is missing a patch[1] from Peff which he wanted placed at the end of the series. It was probably overlooked since he embedded it as a reply in that thread rather than sending it as a standalone patch.

[1]: https://public-inbox.org/git/[email protected]/

Yeah, I'm not sure which is easier for Junio. I figured by replying inline, it makes it easy to pick up on top of the others (since it really does depend on them and should be in the same topic). But it is easier to overlook as "meh, just more discussion".

Either way, yes, I'd be happy to see that patch on top.

-Peff

Thanks.

Hi Junio,

Quite frankly, I'd rather you pick up my two v2 series. I promised to take over from Pratik, who is busy with exams, and I did. I even announced as much in that mail where I pointed out the missing short option originally.

So there is no need to do add-on patches, as if the rebase-in-c patch series had already been accepted into next.

Ciao,
Dscho

Yup, I'm planning to replace the large pile of patches with the new ones you'd send once the final is done. The topic above is just a stop-gap measure to keep the tip of 'pu' testable, to be dropped when the series it fixes is replaced with a corrected one.

I can't reproduce it here (with either 64 or 32 bit builds on x86). Not denying the problem, just a quick update. I'll need more time, maybe over weekend to have a closer look.

Duy

It's not a regression, it's as old as the split index feature itself.

I have the patches almost ready, only the commit messages need some touching up.