t/README: make the 'Skipping tests' section less confusing

index 28711cc508..b6ec28f634 100644
@@ -494,7 +494,7 @@ And here are the "don'ts:"
 
    The harness will catch this as a programming error of the test.
    Use test_done instead if you need to stop the tests early (see
-   "Skipping tests" below).
+   "Using test prerequisites" below).
 
  - Don't use '! git cmd' when you want to make sure the git command
    exits with failure in a controlled way by calling "die()".  Instead,

@@ -587,28 +587,28 @@ And here are the "don'ts:"
    it'll complain if anything is amiss.
 
 
-Skipping tests
---------------
+Using test prerequisites
+------------------------
 
-If you need to skip tests you should do so by using the three-arg form
-of the test_* functions (see the "Test harness library" section
-below), e.g.:
+Tests that depend on something which may not be present on the system
+should use the three-arg form of the test_* functions (see the "Test
+harness library" section below), e.g.:
 
-    test_expect_success PERL 'I need Perl' '
-        perl -e "hlagh() if unf_unf()"
+    test_expect_success SYMLINKS 'setup' '
+        ln -s target link
     '
 
 The advantage of skipping tests like this is that platforms that don't
-have the PERL and other optional dependencies get an indication of how
+have the SYMLINKS and other optional dependencies get an indication of how
 many tests they're missing.
 
 If the test code is too hairy for that (i.e. does a lot of setup work
 outside test assertions) you can also skip all remaining tests by
 setting skip_all and immediately call test_done:
 
-	if ! test_have_prereq PERL
+	if ! test_have_prereq SYMLINKS
 	then
-	    skip_all='skipping perl interface tests, perl not available'
+	    skip_all="skipping symlink tests, the filesystem doesn't support it"
 	    test_done
 	fi
 

index b6ec28f634..3139f4330a 100644
@@ -202,20 +202,21 @@ GIT_TEST_EXEC_PATH defaults to `$GIT_TEST_INSTALLED/git --exec-path`.
 Skipping Tests
 --------------
 
-In some environments, certain tests have no way of succeeding
-due to platform limitation, such as lack of 'unzip' program, or
-filesystem that do not allow arbitrary sequence of non-NUL bytes
-as pathnames.
+Certain tests may fail intermittently or entirely. These should
+ideally be reported as bugs and fixed, or guarded by a prerequisite
+(see "Using test prerequisites" below). But until then they can be
+skipped.
 
-You should be able to say something like
+To skip tests, set the GIT_SKIP_TESTS variable. Individual tests can
+be skipped:
 
     $ GIT_SKIP_TESTS=t9200.8 sh ./t9200-git-cvsexport-commit.sh
 
-and even:
+Or tests matching a glob:
 
     $ GIT_SKIP_TESTS='t[0-4]??? t91?? t9200.8' make
 
-to omit such tests.  The value of the environment variable is a
+The value of the environment variable is a
 SP separated list of patterns that tells which tests to skip,
 and either can match the "t[0-9]{4}" part to skip the whole
 test, or t[0-9]{4} followed by ".$number" to say which

index 3139f4330a..c03b268813 100644
@@ -216,11 +216,10 @@ Or tests matching a glob:
 
     $ GIT_SKIP_TESTS='t[0-4]??? t91?? t9200.8' make
 
-The value of the environment variable is a
-SP separated list of patterns that tells which tests to skip,
-and either can match the "t[0-9]{4}" part to skip the whole
-test, or t[0-9]{4} followed by ".$number" to say which
-particular test to skip.
+The value of the environment variable is a SP separated list of
+patterns that tells which tests to skip, and either can match the
+"t[0-9]{4}" part to skip the whole test, or t[0-9]{4} followed by
+".$number" to say which particular test to skip.
 
 For an individual test suite --run could be used to specify that
 only some tests should be run or that some tests should be

index b6566003dd..b87a8f18c2 100755
@@ -393,6 +393,23 @@ test_expect_success 'GIT_SKIP_TESTS sh pattern' "
 	)
 "
 
+test_expect_success 'GIT_SKIP_TESTS entire file' "
+	(
+		GIT_SKIP_TESTS='git' && export GIT_SKIP_TESTS &&
+		run_sub_test_lib_test git-skip-tests-entire-file \
+			'GIT_SKIP_TESTS' <<-\\EOF &&
+		for i in 1 2 3
+		do
+			test_expect_success \"passing test #\$i\" 'true'
+		done
+		test_done
+		EOF
+		check_sub_test_lib_test git-skip-tests-entire-file <<-\\EOF
+		1..0 # SKIP skip all tests in git
+		EOF
+	)
+"
+
 test_expect_success '--run basic' "
 	run_sub_test_lib_test run-basic \
 		'--run basic' --run='1 3 5' <<-\\EOF &&

index 69dff4d1ec..ad8290bfdb 100755
@@ -121,7 +121,7 @@ osx-clang|osx-gcc)
 	# t9810 occasionally fails on Travis CI OS X
 	# t9816 occasionally fails with "TAP out of sequence errors" on
 	# Travis CI OS X
-	export GIT_SKIP_TESTS="t9810 t9816"
+	export GIT_TODO_TESTS="t9810 t9816"
 	;;
 GIT_TEST_GETTEXT_POISON)
 	export GIT_TEST_GETTEXT_POISON=YesPlease

index c03b268813..922b4fb3bf 100644
@@ -207,8 +207,19 @@ ideally be reported as bugs and fixed, or guarded by a prerequisite
 (see "Using test prerequisites" below). But until then they can be
 skipped.
 
-To skip tests, set the GIT_SKIP_TESTS variable. Individual tests can
-be skipped:
+To skip tests, set either the GIT_SKIP_TESTS or GIT_TODO_TESTS
+variables. The difference is that with SKIP the tests won't be run at
+all, whereas they will be run with TODO, but in success or failure
+annotated as a TODO test.
+
+It's usually preferrable to use TODO, since the test suite will report
+those tests that unexpectedly succeed, which may indicate that
+whatever bug broke the test in the past has been fixed, and the test
+should be un-TODO'd. There's no such feedback loop with
+GIT_SKIP_TESTS.
+
+The GIT_SKIP_TESTS and GIT_TODO_TESTS take the same values. Individual
+tests can be skipped:
 
     $ GIT_SKIP_TESTS=t9200.8 sh ./t9200-git-cvsexport-commit.sh
 

@@ -223,7 +234,8 @@ patterns that tells which tests to skip, and either can match the
 
 For an individual test suite --run could be used to specify that
 only some tests should be run or that some tests should be
-excluded from a run.
+excluded from a run. The --run option is a shorthand for setting
+a GIT_SKIP_TESTS pattern.
 
 The argument for --run is a list of individual test numbers or
 ranges with an optional negation prefix that define what tests in

131 lines not on this page · raw message

index 6c6c0af7a1..ccd316e7bd 100644
@@ -456,6 +456,8 @@ test_count=0
 test_fixed=0
 test_broken=0
 test_success=0
+test_todo_success=0
+test_todo_failure=0
 
 test_external_has_tap=0
 

@@ -482,13 +484,29 @@ trap 'exit $?' INT
 # the test_expect_* functions instead.
 
 test_ok_ () {
+	todo_blurb=
+	if match_pattern_list $this_test.$test_count $GIT_TODO_TESTS ||
+		    match_pattern_list $this_test $GIT_TODO_TESTS
+	then
+		todo_blurb=' # TODO (GIT_TODO_TESTS)'
+		test_todo_success=$(($test_todo_success + 1))
+	fi
 	test_success=$(($test_success + 1))
-	say_color "" "ok $test_count - $@"
+	say_color "" "ok $test_count - $@$todo_blurb"
 }
 
 test_failure_ () {
-	test_failure=$(($test_failure + 1))
-	say_color error "not ok $test_count - $1"
+	todo_blurb=
+	if match_pattern_list $this_test.$test_count $GIT_TODO_TESTS ||
+		    match_pattern_list $this_test $GIT_TODO_TESTS
+	then
+		test_success=$(($test_success + 1))
+		test_todo_failure=$(($test_todo_failure + 1))
+		todo_blurb=' # TODO (GIT_TODO_TESTS)'
+	else
+		test_failure=$(($test_failure + 1))
+	fi
+	say_color error "not ok $test_count - $1$todo_blurb"
 	shift
 	printf '%s\n' "$*" | sed -e 's/^/#	/'
 	test "$immediate" = "" || { GIT_EXIT_OK=t; exit 1; }

@@ -845,13 +863,18 @@ test_done () {
 		test_remaining=$test_count
 		msg="$test_count test(s)"
 	fi
+	msg_todo=
+	if test "$test_todo_success" != 0 || test "$test_todo_failure" != 0
+	then
+		msg_todo=" (including $test_todo_success/$test_todo_failure ok/failing TODO test(s))"
+	fi
 	case "$test_failure" in
 	0)
 		if test $test_external_has_tap -eq 0
 		then
 			if test $test_remaining -gt 0
 			then
-				say_color pass "# passed all $msg"
+				say_color pass "# passed all $msg$msg_todo"
 			fi
 
 			# Maybe print SKIP message