Introduce clar testing framework

Hi,

there's been some discussion around extending our unit testing framework to avoid duplication when declaring test functions. Right now, a testing function has to be declared and then wired up via the test's main function, which can be a bit annoying. In the thread, René proposes an alternative that gets rid of this duplication by using macros. And while that does solve the issue, there were some concerns about things being too much "magic" while at the same time not being flexible enough.

Part of the discussion revolved around whether we maybe want to have a proper unit testing framework in our codebase instead of reinventing the wheel. As I quite liked the "clar" [2] testing framework from back when I was still developing libgit2 I proposed it as a possible alternative. This patch series wires up the clar framework as a proof of concept and converts the strvec test suite to use it.

The magic to avoid the above code duplication is quite self-contained in a "generate.py" file. This script extracts function declarations from all unit test suites and then writes those into a "clar.suite" header file. All that one needs to do is thus to declare a function with a specific name "test_<suite>__<name>" and then everything else gets wired up automatically.

Whether this is better than the solution proposed by René is probably a matter of taste. While I'm not a huge fan of the macro-based solution, I don't want to veto it either (not that I'd have that power anyway). So please, you should rather read this as a proof of concept to see how alternatives could look like such that we have a better picture of where we want to end up.

Some random thoughts:

- The mandated Python dependency is suboptimal in my opinion. Rewriting the script in e.g. Perl should be easy enough though, it's no rocket science.

- I prefer that the proposed solution results in a single binary as compared to one binary per test system.

- The clar gives us the ability to pick which tests to run via command line parameters, which I personally like more than picking the specific binary to run.

- The clar replaces some test assertions that we already have. They feel a bit more mature, but overall there aren't all that many assertions available. If we wanted to pick it up, then we'd likely have to add some more wrappers.

- The clar uses longjmp instead of manually having to return from functions in case there was an assertion failure. This is easier to work with in my opinion.

Also, note that I only tested this on my Linux machine. I have no clue whether this works as-is on Windows, but I do know that libgit2 tests run on Linux, macOS and Windows. So it should work in theory, it's just a matter of polishing this series.

I'm happy to hear your thoughts on this, even if it ultimately ends up being shot down.

Patrick

[1]: <[email protected]> [2]: https://github.com/clar-test/clar

Patrick Steinhardt (3):
t: import the clar unit testing framework
Makefile: wire up the clar unit testing framework t/unit-tests: convert strvec tests to use clar

 .gitignore                                 |   1 +
 Makefile                                   |  36 +-
 t/Makefile                                 |   1 +
 t/unit-tests/.gitignore                    |   3 +
 t/unit-tests/clar/.github/workflows/ci.yml |  23 +
 t/unit-tests/clar/COPYING                  |  15 +
 t/unit-tests/clar/README.md                | 329 ++++++++
 t/unit-tests/clar/clar.c                   | 842 +++++++++++++++++++++
 t/unit-tests/clar/clar.h                   | 173 +++++
 t/unit-tests/clar/clar/fixtures.h          |  50 ++
 t/unit-tests/clar/clar/fs.h                | 522 +++++++++++++
 t/unit-tests/clar/clar/print.h             | 211 ++++++
 t/unit-tests/clar/clar/sandbox.h           | 154 ++++
 t/unit-tests/clar/clar/summary.h           | 143 ++++
 t/unit-tests/clar/generate.py              | 267 +++++++
 t/unit-tests/clar/test/.gitignore          |   5 +
 t/unit-tests/clar/test/Makefile            |  39 +
 t/unit-tests/clar/test/clar_test.h         |  16 +
 t/unit-tests/clar/test/main.c              |  40 +
 t/unit-tests/clar/test/main.c.sample       |  27 +
 t/unit-tests/clar/test/resources/test/file |   1 +
 t/unit-tests/clar/test/sample.c            |  84 ++
 t/unit-tests/{t-strvec.c => strvec.c}      | 124 ++-
 t/unit-tests/unit-test.c                   |  16 +
 t/unit-tests/unit-test.h                   |   3 +
 25 files changed, 3041 insertions(+), 84 deletions(-)
 create mode 100644 t/unit-tests/clar/.github/workflows/ci.yml
 create mode 100644 t/unit-tests/clar/COPYING
 create mode 100644 t/unit-tests/clar/README.md
 create mode 100644 t/unit-tests/clar/clar.c
 create mode 100644 t/unit-tests/clar/clar.h
 create mode 100644 t/unit-tests/clar/clar/fixtures.h
 create mode 100644 t/unit-tests/clar/clar/fs.h
 create mode 100644 t/unit-tests/clar/clar/print.h
 create mode 100644 t/unit-tests/clar/clar/sandbox.h
 create mode 100644 t/unit-tests/clar/clar/summary.h
 create mode 100755 t/unit-tests/clar/generate.py
 create mode 100644 t/unit-tests/clar/test/.gitignore
 create mode 100644 t/unit-tests/clar/test/Makefile
 create mode 100644 t/unit-tests/clar/test/clar_test.h
 create mode 100644 t/unit-tests/clar/test/main.c
 create mode 100644 t/unit-tests/clar/test/main.c.sample
 create mode 100644 t/unit-tests/clar/test/resources/test/file
 create mode 100644 t/unit-tests/clar/test/sample.c
 rename t/unit-tests/{t-strvec.c => strvec.c} (54%)
 create mode 100644 t/unit-tests/unit-test.c
 create mode 100644 t/unit-tests/unit-test.h

2.46.0.dirty

Import the clar unit testing framework at commit faa8419 (Merge pull request #93 from clar-test/ethomson/fixtures, 2023-12-14). The framework will be wired up in subsequent commits.

---
 t/unit-tests/clar/.github/workflows/ci.yml |  23 +
 t/unit-tests/clar/COPYING                  |  15 +
 t/unit-tests/clar/README.md                | 329 ++++++++
 t/unit-tests/clar/clar.c                   | 842 +++++++++++++++++++++
 t/unit-tests/clar/clar.h                   | 173 +++++
 t/unit-tests/clar/clar/fixtures.h          |  50 ++
 t/unit-tests/clar/clar/fs.h                | 522 +++++++++++++
 t/unit-tests/clar/clar/print.h             | 211 ++++++
 t/unit-tests/clar/clar/sandbox.h           | 154 ++++
 t/unit-tests/clar/clar/summary.h           | 143 ++++
 t/unit-tests/clar/generate.py              | 267 +++++++
 t/unit-tests/clar/test/.gitignore          |   5 +
 t/unit-tests/clar/test/Makefile            |  39 +
 t/unit-tests/clar/test/clar_test.h         |  16 +
 t/unit-tests/clar/test/main.c              |  40 +
 t/unit-tests/clar/test/main.c.sample       |  27 +
 t/unit-tests/clar/test/resources/test/file |   1 +
 t/unit-tests/clar/test/sample.c            |  84 ++
 18 files changed, 2941 insertions(+)
 create mode 100644 t/unit-tests/clar/.github/workflows/ci.yml
 create mode 100644 t/unit-tests/clar/COPYING
 create mode 100644 t/unit-tests/clar/README.md
 create mode 100644 t/unit-tests/clar/clar.c
 create mode 100644 t/unit-tests/clar/clar.h
 create mode 100644 t/unit-tests/clar/clar/fixtures.h
 create mode 100644 t/unit-tests/clar/clar/fs.h
 create mode 100644 t/unit-tests/clar/clar/print.h
 create mode 100644 t/unit-tests/clar/clar/sandbox.h
 create mode 100644 t/unit-tests/clar/clar/summary.h
 create mode 100755 t/unit-tests/clar/generate.py
 create mode 100644 t/unit-tests/clar/test/.gitignore
 create mode 100644 t/unit-tests/clar/test/Makefile
 create mode 100644 t/unit-tests/clar/test/clar_test.h
 create mode 100644 t/unit-tests/clar/test/main.c
 create mode 100644 t/unit-tests/clar/test/main.c.sample
 create mode 100644 t/unit-tests/clar/test/resources/test/file
 create mode 100644 t/unit-tests/clar/test/sample.c

new file mode 100644
index 0000000000..b1ac2de460
@@ -0,0 +1,23 @@
+name: CI
+
+on:
+  push:
+    branches: [ main ]
+  pull_request:
+    branches: [ main ]
+
+jobs:
+  build:
+    strategy:
+      matrix:
+        os: [ ubuntu-latest, macos-latest ]
+
+    runs-on: ${{ matrix.os }}
+
+    steps:
+    - name: Check out
+      uses: actions/checkout@v2
+    - name: Build
+      run: |
+        cd test
+        make

new file mode 100644
index 0000000000..8983817f0c
@@ -0,0 +1,15 @@
+ISC License
+
+Copyright (c) 2011-2015 Vicent Marti
+
+Permission to use, copy, modify, and/or distribute this software for any
+purpose with or without fee is hereby granted, provided that the above
+copyright notice and this permission notice appear in all copies.
+
+THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
+WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
+MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
+ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
+WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
+ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
+OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.

new file mode 100644
index 0000000000..3e8ae0a3a5
@@ -0,0 +1,5 @@
+clar.suite
+.clarcache
+clar_test
+*.o
+

new file mode 100644
index 0000000000..93c6b2ad32
@@ -0,0 +1,39 @@
+#
+# Copyright (c) Vicent Marti. All rights reserved.
+#
+# This file is part of clar, distributed under the ISC license.
+# For full terms see the included COPYING file.
+#
+
+#
+# Set up the path to the clar sources and to the fixtures directory
+#
+# The fixture path needs to be an absolute path so it can be used
+# even after we have chdir'ed into the test directory while testing.
+#
+CURRENT_MAKEFILE  := $(word $(words $(MAKEFILE_LIST)),$(MAKEFILE_LIST))
+TEST_DIRECTORY    := $(abspath $(dir $(CURRENT_MAKEFILE)))
+CLAR_PATH         := $(dir $(TEST_DIRECTORY))
+CLAR_FIXTURE_PATH := $(TEST_DIRECTORY)/resources/
+
+CFLAGS=-g -I.. -I. -Wall -DCLAR_FIXTURE_PATH=\"$(CLAR_FIXTURE_PATH)\"
+
+.PHONY: clean
+
+# list the objects that go into our test
+objects = main.o sample.o
+
+# build the test executable itself
+clar_test: $(objects) clar_test.h clar.suite $(CLAR_PATH)clar.c
+	$(CC) $(CFLAGS) -o $@ "$(CLAR_PATH)clar.c" $(objects)
+
+# test object files depend on clar macros
+$(objects) : $(CLAR_PATH)clar.h
+
+# build the clar.suite file of test metadata
+clar.suite:
+	python "$(CLAR_PATH)generate.py" .
+
+# remove all generated files
+clean:
+	$(RM) -rf *.o clar.suite .clarcache clar_test clar_test.dSYM

new file mode 100644
index 0000000000..0fcaa639aa
@@ -0,0 +1,16 @@
+/*
+ * Copyright (c) Vicent Marti. All rights reserved.
+ *
+ * This file is part of clar, distributed under the ISC license.
+ * For full terms see the included COPYING file.
+ */
+#ifndef __CLAR_TEST__
+#define __CLAR_TEST__
+
+/* Import the standard clar helper functions */
+#include "clar.h"
+
+/* Your custom shared includes / defines here */
+extern int global_test_counter;
+
+#endif

new file mode 100644
index 0000000000..59e56ad255
@@ -0,0 +1,40 @@
+/*
+ * Copyright (c) Vicent Marti. All rights reserved.
+ *
+ * This file is part of clar, distributed under the ISC license.
+ * For full terms see the included COPYING file.
+ */
+
+#include "clar_test.h"
+
+/*
+ * Sample main() for clar tests.
+ *
+ * You should write your own main routine for clar tests that does specific
+ * setup and teardown as necessary for your application.  The only required
+ * line is the call to `clar_test(argc, argv)`, which will execute the test
+ * suite.  If you want to check the return value of the test application,
+ * your main() should return the same value returned by clar_test().
+ */
+
+int global_test_counter = 0;
+
+#ifdef _WIN32
+int __cdecl main(int argc, char *argv[])
+#else
+int main(int argc, char *argv[])
+#endif
+{
+	int ret;
+
+	/* Your custom initialization here */
+	global_test_counter = 0;
+
+	/* Run the test suite */
+	ret = clar_test(argc, argv);
+
+	/* Your custom cleanup here */
+	cl_assert_equal_i(8, global_test_counter);
+
+	return ret;
+}

new file mode 100644
index 0000000000..a4d91b72fa
@@ -0,0 +1,27 @@
+/*
+ * Copyright (c) Vicent Marti. All rights reserved.
+ *
+ * This file is part of clar, distributed under the ISC license.
+ * For full terms see the included COPYING file.
+ */
+
+#include "clar_test.h"
+
+/*
+ * Minimal main() for clar tests.
+ *
+ * Modify this with any application specific setup or teardown that you need.
+ * The only required line is the call to `clar_test(argc, argv)`, which will
+ * execute the test suite.  If you want to check the return value of the test
+ * application, main() should return the same value returned by clar_test().
+ */
+
+#ifdef _WIN32
+int __cdecl main(int argc, char *argv[])
+#else
+int main(int argc, char *argv[])
+#endif
+{
+	/* Run the test suite */
+	return clar_test(argc, argv);
+}

new file mode 100644
index 0000000000..220f4aa98a
@@ -0,0 +1 @@
+File

2.46.0.dirty

Wire up the clar unit testing framework by introducing a new "unit-tests" executable. In contrast to the existing framework, this will result in a single executable for all test suites. The ability to pick specific tests to execute is retained via functionality built into the clar itself.

Note that we need to be a bit careful about how we need to invalidate our Makefile rules. While we obviously have to regenerate the clar suite when our test suites change, we also have to invalidate it in case any of the test suites gets removed. We do so by using our typical pattern of creating a GIT-TEST-SUITES file that gets updated whenever the set of test suites changes, so that we can easily depend on that file.

Another specialty is that we generate a "clar-decls.h" file. The test functions are neither static, nor do they have external declarations. This is because they are getting parsed via "generate.py", which then creates the external generations that get populated into an array. These declarations are only seen by the main function though.

The consequence is that we will get a bunch of "missing prototypes" errors from our compiler for each of these test functions. To fix those errors, we extract the extern declarations from "clar.suite" and put them into a standalone header that then gets included by each of our unit tests. This gets rid of compiler warnings for every function which has been extracted by "generate.py". More importantly though, it does _not_ get rid of warnings in case a function really isn't being used by anything. Thus, it would cause a compiler error if a function name was mistyped and thus not picked up by "generate.py".

---
 .gitignore               |  1 +
 Makefile                 | 34 ++++++++++++++++++++++++++++++----
 t/Makefile               |  1 +
 t/unit-tests/.gitignore  |  3 +++
 t/unit-tests/unit-test.c | 16 ++++++++++++++++
 t/unit-tests/unit-test.h |  3 +++
 6 files changed, 54 insertions(+), 4 deletions(-)
 create mode 100644 t/unit-tests/unit-test.c
 create mode 100644 t/unit-tests/unit-test.h

index 8caf3700c2..6687bd6db4 100644
@@ -9,6 +9,7 @@
 /GIT-PYTHON-VARS
 /GIT-SCRIPT-DEFINES
 /GIT-SPATCH-DEFINES
+/GIT-TEST-SUITES
 /GIT-USER-AGENT
 /GIT-VERSION-FILE
 /bin-wrappers/

index d6479092a0..658acb4d48 100644
@@ -1332,6 +1332,11 @@ THIRD_PARTY_SOURCES += compat/regex/%
 THIRD_PARTY_SOURCES += sha1collisiondetection/%
 THIRD_PARTY_SOURCES += sha1dc/%
 
+UNIT_TESTS_PROG = $(UNIT_TEST_BIN)/unit-tests$(X)
+UNIT_TESTS_OBJS = $(patsubst %,$(UNIT_TEST_DIR)/%.o,$(UNIT_TESTS_SUITES))
+UNIT_TESTS_OBJS += $(UNIT_TEST_DIR)/clar/clar.o
+UNIT_TESTS_OBJS += $(UNIT_TEST_DIR)/unit-test.o
+
 UNIT_TEST_PROGRAMS += t-ctype
 UNIT_TEST_PROGRAMS += t-example-decorate
 UNIT_TEST_PROGRAMS += t-hash

@@ -2711,6 +2716,7 @@ OBJECTS += $(XDIFF_OBJS)
 OBJECTS += $(FUZZ_OBJS)
 OBJECTS += $(REFTABLE_OBJS) $(REFTABLE_TEST_OBJS)
 OBJECTS += $(UNIT_TEST_OBJS)
+OBJECTS += $(UNIT_TESTS_OBJS)
 
 ifndef NO_CURL
 	OBJECTS += http.o http-walker.o remote-curl.o

@@ -3213,7 +3219,7 @@ endif
 
 test_bindir_programs := $(patsubst %,bin-wrappers/%,$(BINDIR_PROGRAMS_NEED_X) $(BINDIR_PROGRAMS_NO_X) $(TEST_PROGRAMS_NEED_X))
 
-all:: $(TEST_PROGRAMS) $(test_bindir_programs) $(UNIT_TEST_PROGS)
+all:: $(TEST_PROGRAMS) $(test_bindir_programs) $(UNIT_TEST_PROGS) $(UNIT_TESTS_PROG)
 
 bin-wrappers/%: wrap-for-bin.sh
 	$(call mkdir_p_parent_template)

@@ -3644,7 +3650,7 @@ endif
 
 artifacts-tar:: $(ALL_COMMANDS_TO_INSTALL) $(SCRIPT_LIB) $(OTHER_PROGRAMS) \
 		GIT-BUILD-OPTIONS $(TEST_PROGRAMS) $(test_bindir_programs) \
-		$(UNIT_TEST_PROGS) $(MOFILES)
+		$(UNIT_TEST_PROGS) $(UNIT_TESTS_PROG) $(MOFILES)
 	$(QUIET_SUBDIR0)templates $(QUIET_SUBDIR1) \
 		SHELL_PATH='$(SHELL_PATH_SQ)' PERL_PATH='$(PERL_PATH_SQ)'
 	test -n "$(ARTIFACTS_DIRECTORY)"

@@ -3700,6 +3706,7 @@ cocciclean:
 
 clean: profile-clean coverage-clean cocciclean
 	$(RM) -r .build $(UNIT_TEST_BIN)
+	$(RM) $(UNIT_TEST_DIR)/clar.suite $(UNIT_TEST_DIR)/.clarcache
 	$(RM) po/git.pot po/git-core.pot
 	$(RM) git.res
 	$(RM) $(OBJECTS)

@@ -3859,7 +3866,26 @@ $(UNIT_TEST_PROGS): $(UNIT_TEST_BIN)/%$X: $(UNIT_TEST_DIR)/%.o \
 	$(QUIET_LINK)$(CC) $(ALL_CFLAGS) -o $@ $(ALL_LDFLAGS) \
 		$(filter %.o,$^) $(filter %.a,$^) $(LIBS)
 
+GIT-TEST-SUITES: FORCE
+	@FLAGS='$(UNIT_TESTS_SUITES)'; \
+	    if test x"$$FLAGS" != x"`cat GIT-TEST-SUITES 2>/dev/null`" ; then \
+		echo >&2 "    * new test suites"; \
+		echo "$$FLAGS" >GIT-TEST-SUITES; \
+		rm -f $(UNIT_TESTS_DIR)/.clarcache; \
+            fi
+
+$(UNIT_TEST_DIR)/clar.suite: $(patsubst %,$(UNIT_TEST_DIR)/%.c,$(UNIT_TESTS_SUITES)) GIT-TEST-SUITES
+	$(QUIET_GEN)$(UNIT_TEST_DIR)/clar/generate.py $(UNIT_TEST_DIR) >/dev/null
+	@touch $@
+$(UNIT_TEST_DIR)/clar-decls.h: $(UNIT_TEST_DIR)/clar.suite
+	$(QUIET_GEN)grep '^extern void' $^ >$@
+$(UNIT_TESTS_OBJS): $(UNIT_TEST_DIR)/clar-decls.h
+$(UNIT_TESTS_OBJS): EXTRA_CPPFLAGS = -I$(UNIT_TEST_DIR) -I$(UNIT_TEST_DIR)/clar
+$(UNIT_TESTS_PROG): $(UNIT_TEST_DIR)/clar.suite $(UNIT_TESTS_OBJS) $(GITLIBS) GIT-LDFLAGS
+	$(call mkdir_p_parent_template)
+	$(QUIET_LINK)$(CC) $(ALL_CFLAGS) -o $@ $(ALL_LDFLAGS) $(filter %.o,$^) $(LIBS)
+
 .PHONY: build-unit-tests unit-tests
-build-unit-tests: $(UNIT_TEST_PROGS)
-unit-tests: $(UNIT_TEST_PROGS) t/helper/test-tool$X
+build-unit-tests: $(UNIT_TEST_PROGS) $(UNIT_TESTS_PROG)
+unit-tests: $(UNIT_TEST_PROGS) $(UNIT_TESTS_PROG) t/helper/test-tool$X
 	$(MAKE) -C t/ unit-tests

index 4c30e7c06f..55f740ae29 100644
@@ -48,6 +48,7 @@ CHAINLINTTESTS = $(sort $(patsubst chainlint/%.test,%,$(wildcard chainlint/*.tes
 CHAINLINT = '$(PERL_PATH_SQ)' chainlint.pl
 UNIT_TEST_SOURCES = $(wildcard unit-tests/t-*.c)
 UNIT_TEST_PROGRAMS = $(patsubst unit-tests/%.c,unit-tests/bin/%$(X),$(UNIT_TEST_SOURCES))
+UNIT_TEST_PROGRAMS += unit-tests/bin/unit-tests$(X)
 UNIT_TESTS = $(sort $(UNIT_TEST_PROGRAMS))
 UNIT_TESTS_NO_DIR = $(notdir $(UNIT_TESTS))
 

index 5e56e040ec..b8d46f7bb1 100644
@@ -1 +1,4 @@
 /bin
+/.clarcache
+/clar.suite
+/clar-decls.h

new file mode 100644
index 0000000000..d7eecc384c
@@ -0,0 +1,16 @@
+#include "unit-test.h"
+
+int cmd_main(int argc, const char **argv)
+{
+	const char **args;
+	int ret;
+
+	/* Append the "-t" flag such that the tests generate TAP output. */
+	DUP_ARRAY(args, argv, argc + 1);
+	args[argc++] = "-t";
+
+	ret = clar_test(argc, (char **) args);
+
+	free(args);
+	return ret;
+}

new file mode 100644
index 0000000000..99d59df1b0
@@ -0,0 +1,3 @@
+#include "git-compat-util.h"
+#include "clar.h"
+#include "clar-decls.h"

2.46.0.dirty

Convert the strvec tests to use the new clar unit testing framework. This is a first test balloon that demonstrates how the testing infra for clar-based tests looks like.

The tests are part of the "t/unit-tests/bin/unit-tests" binary. When running that binary, it generates TAP output:

    # ./t/unit-tests/bin/unit-tests
    TAP version 13
    # start of suite 1: strvec
    ok 1 - strvec::init
    ok 2 - strvec::dynamic_init
    ok 3 - strvec::clear
    ok 4 - strvec::push
    ok 5 - strvec::pushft_pushf
    ok 6 - strvec::pushl
    ok 7 - strvec::pushv
    ok 8 - strvec::replace_at_head
    ok 9 - strvec::replace_at_tail
    ok 10 - strvec::replace_in_between
    ok 11 - strvec::replace_with_substring
    ok 12 - strvec::remove_at_head
    ok 13 - strvec::remove_at_tail
    ok 14 - strvec::remove_in_between
    ok 15 - strvec::pop_empty_array
    ok 16 - strvec::pop_non_empty_array
    ok 17 - strvec::split_empty_string
    ok 18 - strvec::split_single_item
    ok 19 - strvec::split_multiple_items
    ok 20 - strvec::split_whitespace_only
    ok 21 - strvec::split_multiple_consecutive_whitespaces
    ok 22 - strvec::detach
    1..22

The binary also supports some parameters that allow us to run only a subset of unit tests or alter the output:

    $ ./t/unit-tests/bin/unit-tests -h
    Usage: ./t/unit-tests/bin/unit-tests [options]

Options:
-sname Run only the suite with name (can go to individual test name)
-iname Include the suite with name
-xname Exclude the suite with name
-v Increase verbosity (show suite names)
-q Only report tests that had an error
-Q Quit as soon as a test fails
-t Display results in tap format
-l Print suite names
-r[filename] Write summary file (to the optional filename)

Furthermore, running make unit-tests runs the binary along with all the other unit tests we have.

---
 Makefile                              |   2 +-
 t/unit-tests/{t-strvec.c => strvec.c} | 124 ++++++++++----------------
 2 files changed, 46 insertions(+), 80 deletions(-)
 rename t/unit-tests/{t-strvec.c => strvec.c} (54%)

index 658acb4d48..8ebcbdc95a 100644
@@ -1332,6 +1332,7 @@ THIRD_PARTY_SOURCES += compat/regex/%
 THIRD_PARTY_SOURCES += sha1collisiondetection/%
 THIRD_PARTY_SOURCES += sha1dc/%
 
+UNIT_TESTS_SUITES += strvec
 UNIT_TESTS_PROG = $(UNIT_TEST_BIN)/unit-tests$(X)
 UNIT_TESTS_OBJS = $(patsubst %,$(UNIT_TEST_DIR)/%.o,$(UNIT_TESTS_SUITES))
 UNIT_TESTS_OBJS += $(UNIT_TEST_DIR)/clar/clar.o

@@ -1348,7 +1349,6 @@ UNIT_TEST_PROGRAMS += t-reftable-basics
 UNIT_TEST_PROGRAMS += t-reftable-record
 UNIT_TEST_PROGRAMS += t-strbuf
 UNIT_TEST_PROGRAMS += t-strcmp-offset
-UNIT_TEST_PROGRAMS += t-strvec
 UNIT_TEST_PROGRAMS += t-trailer
 UNIT_TEST_PROGS = $(patsubst %,$(UNIT_TEST_BIN)/%$X,$(UNIT_TEST_PROGRAMS))
 UNIT_TEST_OBJS = $(patsubst %,$(UNIT_TEST_DIR)/%.o,$(UNIT_TEST_PROGRAMS))

2.46.0.dirty

One thing I am very unhappy about the current t/unit-tests/ is that the GIT_SKIP_TESTS mechanism is not effective at all. If we can wrap clar's test selection syntax inside t/Makefile to work with GIT_SKIP_TESTS (or its superset equivalent), that would be a great plus.

That is a slight bummer, as importing an externally developed one is with the hope that we won't have to enhance or maintain it, but we'll see how much burden it will be.

Thanks for getting the ball going. Let's see how fast and far it rolls.

I'm sorry for being so behind the curve... what is clar and where does it run?

Am 31.07.24 um 11:04 schrieb Patrick Steinhardt:

---
 .gitignore                |  1 -
 Makefile                  | 20 ++++++-----------
 t/unit-tests/.gitignore   |  1 -
 t/unit-tests/generate.awk | 47 +++++++++++++++++++++++++++++++++++++++
 4 files changed, 54 insertions(+), 15 deletions(-)
 create mode 100644 t/unit-tests/generate.awk

index 6687bd6db4..8caf3700c2 100644
@@ -9,7 +9,6 @@
 /GIT-PYTHON-VARS
 /GIT-SCRIPT-DEFINES
 /GIT-SPATCH-DEFINES
-/GIT-TEST-SUITES
 /GIT-USER-AGENT
 /GIT-VERSION-FILE
 /bin-wrappers/

index 8ebcbdc95a..1ffde38de5 100644
@@ -3706,7 +3706,7 @@ cocciclean:
 
 clean: profile-clean coverage-clean cocciclean
 	$(RM) -r .build $(UNIT_TEST_BIN)
-	$(RM) $(UNIT_TEST_DIR)/clar.suite $(UNIT_TEST_DIR)/.clarcache
+	$(RM) $(UNIT_TEST_DIR)/clar.suite $(UNIT_TEST_DIR)/clar-decls.h
 	$(RM) po/git.pot po/git-core.pot
 	$(RM) git.res
 	$(RM) $(OBJECTS)

@@ -3866,19 +3866,13 @@ $(UNIT_TEST_PROGS): $(UNIT_TEST_BIN)/%$X: $(UNIT_TEST_DIR)/%.o \
 	$(QUIET_LINK)$(CC) $(ALL_CFLAGS) -o $@ $(ALL_LDFLAGS) \
 		$(filter %.o,$^) $(filter %.a,$^) $(LIBS)
 
-GIT-TEST-SUITES: FORCE
-	@FLAGS='$(UNIT_TESTS_SUITES)'; \
-	    if test x"$$FLAGS" != x"`cat GIT-TEST-SUITES 2>/dev/null`" ; then \
-		echo >&2 "    * new test suites"; \
-		echo "$$FLAGS" >GIT-TEST-SUITES; \
-		rm -f $(UNIT_TESTS_DIR)/.clarcache; \
-            fi
+$(UNIT_TEST_DIR)/clar.suite: $(UNIT_TEST_DIR)/clar-decls.h $(UNIT_TEST_DIR)/generate.awk
+	$(QUIET_GEN)awk -f $(UNIT_TEST_DIR)/generate.awk $(UNIT_TEST_DIR)/clar-decls.h >$@
+$(UNIT_TEST_DIR)/clar-decls.h: $(patsubst %,$(UNIT_TEST_DIR)/%.c,$(UNIT_TESTS_SUITES))
+	$(QUIET_GEN)for suite in $(UNIT_TESTS_SUITES); do \
+		sed -ne "s/^\(void test_$${suite}__[a-zA-Z_0-9][a-zA-Z_0-9]*(void)$$\)/extern \1;/p" $(UNIT_TEST_DIR)/$$suite.c; \
+	done >$@
 
-$(UNIT_TEST_DIR)/clar.suite: $(patsubst %,$(UNIT_TEST_DIR)/%.c,$(UNIT_TESTS_SUITES)) GIT-TEST-SUITES
-	$(QUIET_GEN)$(UNIT_TEST_DIR)/clar/generate.py $(UNIT_TEST_DIR) >/dev/null
-	@touch $@
-$(UNIT_TEST_DIR)/clar-decls.h: $(UNIT_TEST_DIR)/clar.suite
-	$(QUIET_GEN)grep '^extern void' $^ >$@
 $(UNIT_TESTS_OBJS): $(UNIT_TEST_DIR)/clar-decls.h
 $(UNIT_TESTS_OBJS): EXTRA_CPPFLAGS = -I$(UNIT_TEST_DIR) -I$(UNIT_TEST_DIR)/clar
 $(UNIT_TESTS_PROG): $(UNIT_TEST_DIR)/clar.suite $(UNIT_TESTS_OBJS) $(GITLIBS) GIT-LDFLAGS

index b8d46f7bb1..d0632ec7f9 100644
@@ -1,4 +1,3 @@
 /bin
-/.clarcache
 /clar.suite
 /clar-decls.h

We have t/unit-test/test-lib.[ch] that are our home-grown unit test framework. A handful of tests have been written to use it, when you say "make test", or "(cd t && make)", unit tests binaries linked with the home-grown unit test framework run.

clar is a _potential_ replacement for our home-grown framework, suggested here because it would be nicer if we can use off-the-shelf component instead of having to enhance and maintain our own.

Where and how it runs does not change even after clar turns out to be good enough for our purpose and we commit to replace our home-grown unit test framework with it.

A trivial fix-up to be squashed into this step.

 Makefile | 1 +
 1 file changed, 1 insertion(+)

index 8ebcbdc95a..d561789582 100644
@@ -3735,6 +3735,7 @@ ifndef NO_TCLTK
 	$(MAKE) -C git-gui clean
 endif
 	$(RM) GIT-VERSION-FILE GIT-CFLAGS GIT-LDFLAGS GIT-BUILD-OPTIONS
+	$(RM) GIT-TEST-SUITES $(UNIT_TEST_DIR)/clar-decls.h
 	$(RM) GIT-USER-AGENT GIT-PREFIX
 	$(RM) GIT-SCRIPT-DEFINES GIT-PERL-DEFINES GIT-PERL-HEADER GIT-PYTHON-VARS
 ifdef MSVC

Rather than forking our own copy of clar, could we just add it as a submodule instead?

Does clar allow running test functions in parallel? With multiple binaries, we can at least run independent tests in parallel (although right now the unit tests are fewer and so much faster than the shell tests that it's hardly noticeable).

Yes this is a nice improvement.

As part of the original unit-test series, I wrote a comparison between different frameworks: Documentation/technical/unit-tests.txt, poorly rendered at [1]. Could you add a row to the table evaluating clar on the individual points there?

[1] https://git-scm.com/docs/unit-tests#framework-selection

I doubt that forking was Patrick's intention to begin with. Convenience is.

It certainly is a possibility to bind it as a submodule, but it would be easier for everybody to futz with and experiment, at least while it is in its RFC phase.

Thanks.

submodule

What are the requirements to build/use this?

Well... I would like to be able to see whether this can be built/used on NonStop just so I can stay ahead of the curve or be far enough in advance of it to request any required fixes to make it work on platform.

After checking out clar, I would say it has potential, but needs to be slightly more portable to depend on more than gcc CFLAGS (specifically -Wall). I will ask that of that team. It does look like it might be possible, but it should integrate with the config.mak.uname settings, so c99 or c11 (or soon c17) could be used with other CFLAGS from that file so maintainers only have to worry about one location.

--Randall

I won't be able to address this today, but "make sparse" and other auxiliary targets also seem to break, due to their lack of dependence on the generated clar.suite file.

Ah, that's something I didn't think of. clar does not support running tests in parallel.

As you say, I guess for now that is fine and I'd claim that it likely is faster to just run all tests sequentially with a single binary if you also include build times. If that claim isn't true, or if we eventually grow a huge body of tests, then we should likely revert to having separate binaries.

Ah, I wasn't aware of this document. I can update it depending on how the discussion goes overall :)

Patrick

Yeah, I guess that shouldn't be too hard.

The clar exposes generic helpers like cl_assert, but also comparison functions like cl_assert_equal_$t for pointers, integers and strings. If anything is missing, it also exposes the building blocks to add project-specific assertions so that it would be easy to add for example cl_assert_equal_oid.

One thing I have been missing is non-equality comparisons like cl_assert_lt_i/cl_assert_gt_i. But adding that to the clar itself and upstreaming it should be easy enough, also because we know the people maintaining it.

Other than that I think it's mostly fine and should serve as a good baseline.

Patrick

I'm not sure I follow what you're saying. We don't use the Makefile of clar at all but integrate it into our own build system, so honoring config.mak.uname should come for free, I think.

In any case, if there are C constructs in clar that do not work right now, then I'd be happy to address those issues. The clar is overall quite small, so it shouldn't be all that involved.

Patrick

In its current form, Python is a dependency due to "generate.py". I mentioned elsewhere though that I'd be happy to port it to our language of choice.

Other than that it should only require a C89 compiler.

Patrick

Indeed. If we decide to adopt it I'd like to carry any required changes upstream.

I'm not too fond of using submodules, but I'm also not strictly against using them. I'll do whatever works best for folks here and adapt accordingly.

Patrick

Neat! I would certainly prefer to not have a dependency on Python, and I think awk(1) is a good alternative here that we already require anyway.

Also, another benefit of having our own script is that it allows us to be more flexible with how exactly our tests are structured.

We could even integrate the generation of clar-decls.h with the AWK script.

I'll play around a bit with what you have, thanks a lot for working on it!

Patrick

Yeah, as said in the cover letter, this was in quite an unpolished state, so it is not surprising at all that things break. I just wanted to provide an unfinished PoC to demonstrate how things would roughly look like so that we have something to discuss. Definitely not material for seen yet.

Given that initial feedback didn't seem to be all that negative I'm happy to polish things a bit more.

Patrick

mentioned

Unfortunately, it needs gcc, but that is just because of its Makefile using cc -Wall (not portable).

cc -Wall

It looks like clar needs CC=c99 when running generate.py, which is fine. But there is no current way to
suppress -Wall without hand-modifying clar/test/Makefile. The options like that should either be in
CFLAGS?=... instead of CFLAGS=... or we could introduce CFLAGS_DBG?= and put -Wall in there - it is
only for reporting warnings which is done different in the c99 I have. Ideally, the CFLAGS should come
from ./git/config.mak.uname/

As mentioned in another mail, we do not use its Makefile at all. Did you check whether the version I have proposed here works when running make test?

Patrick

choice.

check

That is the commit I have been trying to use. make test in clar or git?

In Git itself. make test builds and runs our unit tests, and that now also includes unit tests based on clar with this patch series. The clar Makefile exists only because I did a 1:1 import of the upstream dependency. We could just as well remove it altogether, including other bits that we don't end up using.

Patrick

also includes

only because I

it

I see. Well, the 2.46.0 test passes. I ran 'seen' 5 days ago and 'next' is going now. Would that catch it?

No, as the patches in this thread are only up for discussion right now and have not been merged to any of the branches. You'd have to apply them on top of v2.46.0 first :)

Patrick

test`?

git?

have not

v2.46.0

Do you happen to have a public fork?
Regards,
Randall

You can pull the branch pks-clar-unit-tests from
https://gitlab.com/gitlab-org/git.

Patrick

https://gitlab.com/gitlab-org/git.

Two issues so far:

    CC t/unit-tests/clar/clar.o
                        assert(!"Unexpected commandline argument!");
                        ^
"/home/randall/git-clar/t/unit-tests/clar/clar.c", line 547: warning(317):
          controlling expression is constant

        if (mkdtemp(_clar_path) == NULL)
            ^

"/home/randall/git-clar/t/unit-tests/clar/clar/sandbox.h", line 132:
error(114):
identifier "mkdtemp" is undefined

mkdtemp is not portable. assert was not #included. Thoughts?

https://gitlab.com/gitlab-org/git.

Fix #1:

b/t/unit-tests/clar/clar/sandbox.h
index 0ba1479620..81a40f26b8 100644

@@ -116,7 +116,7 @@ static int build_sandbox_path(void)
 
        strncpy(_clar_path + len, path_tail, sizeof(_clar_path) - len);
 
-#if defined(__MINGW32__)
+#if defined(__MINGW32__) || defined(__TANDEM)
        if (_mktemp(_clar_path) == NULL)
                return -1;
 

That gets past the clar build problem. Probably more to come as I encounter issues. The assert issue was just a warning about a constant expression that will always fail.

https://gitlab.com/gitlab-org/git.

This fix is better (and actually works unlike the previous):

b/t/unit-tests/clar/clar/sandbox.h
index 0ba1479620..1b14dbfd76 100644

@@ -120,6 +120,12 @@ static int build_sandbox_path(void)
        if (_mktemp(_clar_path) == NULL)
                return -1;
 
+       if (mkdir(_clar_path, 0700) != 0)
+               return -1;
+#elif defined(__TANDEM)
+       if (mktemp(_clar_path) == NULL)
+               return -1;
+
        if (mkdir(_clar_path, 0700) != 0)
                return -1;
 #elif defined(_WIN32)

This is based on the above repo/branch. Basically a separate block for __TANDEM for temp directories. Build is done. I'm running the full test now and may report more or success.
Regards,
Randall

Am 01.08.24 um 19:43 schrieb [email protected]:

"make unit-tests" would suffice.

René

Well, that takes a whole lot less time, thanks. 100% pass after the prior fix was applied.

Thank you.
Randall

Hi,

this is the second version of my RFC patch series that introduces the clar testing framework into our unit tests. The intent is to not have to hand-craft all features of a proper unit testing framework, while still not painting us into a corner. As such, the clar itself is small and extensible while still bringing some nice features to the table.

Changes compared to v1:

- Convert the ctypes unit tests to use clar, as I had it lying around anyway and René was asking for it.

- Adopt the AWK-based script by René, replacing the Python-based one. This gets rid of the mandatory Python dependency and at the same time also allows us to be more flexible going forward.

- Wire up the third party sources in our Makefiles such that they do not get linted via hdr-check or Coccinelle.

- Fix t/Makefile to not pass GIT_TEST_OPTS to our unit tests. They don't know how to handle those options, and clar-based tests error out when they see unknown options.

- Adapt Documentation/technical/unit-tests.txt to mention the clar.

- Rebase the series onto the latest master at 406f326d27 (The second

    batch, 2024-08-01) to avoid some conflicts.

- Cherry-pick a whitespace fix that otherwise makes git-apply(1) unhappy. The CI job is still broken because the first patch that imports clar continues to be broken. I've created an upstream PR.

- Cherry-pick a fix by Randall to make clar work on HP NonStop. The fix has been proposed upstream in a PR.

With these changes, the CI jobs at GitLab all pass, except for the whitespace and clang-format checks.

I've also Cc'd Ed, one of the maintainers of clar. I also noticed that I'm a maintainer of that project back from my libgit2 times, so I think it should be relatively easy to land changes upstream.

Thanks!

Patrick

Patrick Steinhardt (7):
t: do not pass GIT_TEST_OPTS to unit tests with prove
t: import the clar unit testing framework t/clar: fix whitespace errors
t/clar: fix compatibility with NonStop
Makefile: wire up the clar unit testing framework t/unit-tests: convert strvec tests to use clar t/unit-tests: convert ctype tests to use clar

 .gitignore                                 |   1 +
 Documentation/technical/unit-tests.txt     |   2 +
 Makefile                                   |  42 +-
 t/Makefile                                 |   4 +-
 t/run-test.sh                              |   2 +-
 t/unit-tests/.gitignore                    |   2 +
 t/unit-tests/clar-generate.awk             |  50 ++
 t/unit-tests/clar/.github/workflows/ci.yml |  23 +
 t/unit-tests/clar/COPYING                  |  15 +
 t/unit-tests/clar/README.md                | 329 ++++++++
 t/unit-tests/clar/clar.c                   | 842 +++++++++++++++++++++
 t/unit-tests/clar/clar.h                   | 173 +++++
 t/unit-tests/clar/clar/fixtures.h          |  50 ++
 t/unit-tests/clar/clar/fs.h                | 522 +++++++++++++
 t/unit-tests/clar/clar/print.h             | 211 ++++++
 t/unit-tests/clar/clar/sandbox.h           | 159 ++++
 t/unit-tests/clar/clar/summary.h           | 143 ++++
 t/unit-tests/clar/generate.py              | 266 +++++++
 t/unit-tests/clar/test/.gitignore          |   4 +
 t/unit-tests/clar/test/Makefile            |  39 +
 t/unit-tests/clar/test/clar_test.h         |  16 +
 t/unit-tests/clar/test/main.c              |  40 +
 t/unit-tests/clar/test/main.c.sample       |  27 +
 t/unit-tests/clar/test/resources/test/file |   1 +
 t/unit-tests/clar/test/sample.c            |  84 ++
 t/unit-tests/{t-ctype.c => ctype.c}        |  71 +-
 t/unit-tests/{t-strvec.c => strvec.c}      | 119 ++-
 t/unit-tests/unit-test.c                   |  17 +
 t/unit-tests/unit-test.h                   |   3 +
 29 files changed, 3159 insertions(+), 98 deletions(-)
 create mode 100644 t/unit-tests/clar-generate.awk
 create mode 100644 t/unit-tests/clar/.github/workflows/ci.yml
 create mode 100644 t/unit-tests/clar/COPYING
 create mode 100644 t/unit-tests/clar/README.md
 create mode 100644 t/unit-tests/clar/clar.c
 create mode 100644 t/unit-tests/clar/clar.h
 create mode 100644 t/unit-tests/clar/clar/fixtures.h
 create mode 100644 t/unit-tests/clar/clar/fs.h
 create mode 100644 t/unit-tests/clar/clar/print.h
 create mode 100644 t/unit-tests/clar/clar/sandbox.h
 create mode 100644 t/unit-tests/clar/clar/summary.h
 create mode 100755 t/unit-tests/clar/generate.py
 create mode 100644 t/unit-tests/clar/test/.gitignore
 create mode 100644 t/unit-tests/clar/test/Makefile
 create mode 100644 t/unit-tests/clar/test/clar_test.h
 create mode 100644 t/unit-tests/clar/test/main.c
 create mode 100644 t/unit-tests/clar/test/main.c.sample
 create mode 100644 t/unit-tests/clar/test/resources/test/file
 create mode 100644 t/unit-tests/clar/test/sample.c
 rename t/unit-tests/{t-ctype.c => ctype.c} (71%)
 rename t/unit-tests/{t-strvec.c => strvec.c} (54%)
 create mode 100644 t/unit-tests/unit-test.c
 create mode 100644 t/unit-tests/unit-test.h

Range-diff against v1:

-:  ---------- > 1:  78a9cc1162 t: do not pass GIT_TEST_OPTS to unit tests with prove
1:  4e3862991a ! 2:  6a88cf22a5 t: import the clar unit testing framework

@@ Commit message

+ ## Documentation/technical/unit-tests.txt ##
+@@ Documentation/technical/unit-tests.txt: GitHub / GitLab stars to estimate this.
+ :criterion: https://github.com/Snaipe/Criterion[Criterion]
+ :c-tap: https://github.com/rra/c-tap-harness/[C TAP]
+ :check: https://libcheck.github.io/check/[Check]
++:clar: https://github.com/clar-test/clar[Clar]
+
+ [format="csv",options="header",width="33%",subs="specialcharacters,attributes,quotes,macros"]
+ |=====
+@@ Documentation/technical/unit-tests.txt: Framework,"<<license,License>>","<<vendorable-or-ubiquitous,Vendorable or ubiqui
+ {criterion},{mit},{false},{partial},{true},{true},{true},{true},{true},{false},{true},19,1800
+ {c-tap},{expat},{true},{partial},{partial},{true},{false},{true},{false},{false},{false},4,33
+ {check},{lgpl},{false},{partial},{true},{true},{true},{false},{false},{false},{true},17,973 ++{clar},{lgpl},{false},{partial},{true},{true},{true},{true},{false},{false},{true},1,192
+ |=====
+
+ === Additional framework candidates
+
+ ## Makefile ##
+@@ Makefile: THIRD_PARTY_SOURCES += compat/poll/%
+ THIRD_PARTY_SOURCES += compat/regex/%
+ THIRD_PARTY_SOURCES += sha1collisiondetection/%
+ THIRD_PARTY_SOURCES += sha1dc/%
++THIRD_PARTY_SOURCES += $(UNIT_TEST_DIR)/clar/%
++THIRD_PARTY_SOURCES += $(UNIT_TEST_DIR)/clar/clar/%
+
+ UNIT_TEST_PROGRAMS += t-ctype
+ UNIT_TEST_PROGRAMS += t-example-decorate
+@@ Makefile: $(SP_OBJ): %.sp: %.c %.o
+ .PHONY: sparse
+ sparse: $(SP_OBJ)
+
+-EXCEPT_HDRS := $(GENERATED_H) unicode-width.h compat/% xdiff/%
++EXCEPT_HDRS := $(GENERATED_H) unicode-width.h compat/% xdiff/% $(UNIT_TEST_DIR)/clar/% $(UNIT_TEST_DIR)/clar/clar/%
+ ifndef OPENSSL_SHA1
+ EXCEPT_HDRS += sha1/openssl.h
+ endif
+
## t/unit-tests/clar/.github/workflows/ci.yml (new) ##
@@
+name: CI

-:  ---------- > 3:  a52ee59bf4 t/clar: fix whitespace errors
-:  ---------- > 4:  02fb86dfbc t/clar: fix compatibility with NonStop
2:  7a5dfd5065 ! 5:  848dc673c4 Makefile: wire up the clar unit testing framework
    @@ .gitignore
      /bin-wrappers/
     
      ## Makefile ##
    -@@ Makefile: THIRD_PARTY_SOURCES += compat/regex/%
    - THIRD_PARTY_SOURCES += sha1collisiondetection/%
    - THIRD_PARTY_SOURCES += sha1dc/%
    +@@ Makefile: THIRD_PARTY_SOURCES += sha1dc/%
    + THIRD_PARTY_SOURCES += $(UNIT_TEST_DIR)/clar/%
    + THIRD_PARTY_SOURCES += $(UNIT_TEST_DIR)/clar/clar/%
      
     +UNIT_TESTS_PROG = $(UNIT_TEST_BIN)/unit-tests$(X)
     +UNIT_TESTS_OBJS = $(patsubst %,$(UNIT_TEST_DIR)/%.o,$(UNIT_TESTS_SUITES))
    @@ Makefile: endif
      
      bin-wrappers/%: wrap-for-bin.sh
      	$(call mkdir_p_parent_template)
    +@@ Makefile: $(SP_OBJ): %.sp: %.c %.o
    + .PHONY: sparse
    + sparse: $(SP_OBJ)
    + 
    +-EXCEPT_HDRS := $(GENERATED_H) unicode-width.h compat/% xdiff/% $(UNIT_TEST_DIR)/clar/% $(UNIT_TEST_DIR)/clar/clar/%
    ++EXCEPT_HDRS := $(GENERATED_H) unicode-width.h compat/% xdiff/% $(UNIT_TEST_DIR)/unit-test.h $(UNIT_TEST_DIR)/clar/% $(UNIT_TEST_DIR)/clar/clar/%
    + ifndef OPENSSL_SHA1
    + 	EXCEPT_HDRS += sha1/openssl.h
    + endif
     @@ Makefile: endif
      
      artifacts-tar:: $(ALL_COMMANDS_TO_INSTALL) $(SCRIPT_LIB) $(OTHER_PROGRAMS) \
    @@ Makefile: cocciclean:
      
      clean: profile-clean coverage-clean cocciclean
      	$(RM) -r .build $(UNIT_TEST_BIN)
    -+	$(RM) $(UNIT_TEST_DIR)/clar.suite $(UNIT_TEST_DIR)/.clarcache
    ++	$(RM) GIT-TEST-SUITES $(UNIT_TEST_DIR)/clar.suite $(UNIT_TEST_DIR)/clar-decls.h
      	$(RM) po/git.pot po/git-core.pot
      	$(RM) git.res
      	$(RM) $(OBJECTS)
    @@ Makefile: $(UNIT_TEST_PROGS): $(UNIT_TEST_BIN)/%$X: $(UNIT_TEST_DIR)/%.o \
     +	    if test x"$$FLAGS" != x"`cat GIT-TEST-SUITES 2>/dev/null`" ; then \
     +		echo >&2 "    * new test suites"; \
     +		echo "$$FLAGS" >GIT-TEST-SUITES; \
    -+		rm -f $(UNIT_TESTS_DIR)/.clarcache; \
     +            fi
     +
    -+$(UNIT_TEST_DIR)/clar.suite: $(patsubst %,$(UNIT_TEST_DIR)/%.c,$(UNIT_TESTS_SUITES)) GIT-TEST-SUITES
    -+	$(QUIET_GEN)$(UNIT_TEST_DIR)/clar/generate.py $(UNIT_TEST_DIR) >/dev/null
    -+	@touch $@
    -+$(UNIT_TEST_DIR)/clar-decls.h: $(UNIT_TEST_DIR)/clar.suite
    -+	$(QUIET_GEN)grep '^extern void' $^ >$@
    ++$(UNIT_TEST_DIR)/clar-decls.h: $(patsubst %,$(UNIT_TEST_DIR)/%.c,$(UNIT_TESTS_SUITES)) GIT-TEST-SUITES
    ++	$(QUIET_GEN)for suite in $(UNIT_TESTS_SUITES); do \
    ++		sed -ne "s/^\(void test_$${suite}__[a-zA-Z_0-9][a-zA-Z_0-9]*(void)$$\)/extern \1;/p" $(UNIT_TEST_DIR)/$$suite.c; \
    ++	done >$@
    ++$(UNIT_TEST_DIR)/clar.suite: $(UNIT_TEST_DIR)/clar-decls.h
    ++	$(QUIET_GEN)awk -f $(UNIT_TEST_DIR)/clar-generate.awk $< >$(UNIT_TEST_DIR)/clar.suite
     +$(UNIT_TESTS_OBJS): $(UNIT_TEST_DIR)/clar-decls.h
     +$(UNIT_TESTS_OBJS): EXTRA_CPPFLAGS = -I$(UNIT_TEST_DIR) -I$(UNIT_TEST_DIR)/clar
     +$(UNIT_TESTS_PROG): $(UNIT_TEST_DIR)/clar.suite $(UNIT_TESTS_OBJS) $(GITLIBS) GIT-LDFLAGS
    @@ t/Makefile: CHAINLINTTESTS = $(sort $(patsubst chainlint/%.test,%,$(wildcard cha
      ## t/unit-tests/.gitignore ##
     @@
      /bin
    -+/.clarcache
     +/clar.suite
     +/clar-decls.h
     
    + ## t/unit-tests/clar-generate.awk (new) ##
    +@@
    ++function add_suite(suite, initialize, cleanup, count) {
    ++       if (!suite) return
    ++       suite_count++
    ++       callback_count += count
    ++       suites = suites "    {\n"
    ++       suites = suites "        \"" suite "\",\n"
    ++       suites = suites "        " initialize ",\n"
    ++       suites = suites "        " cleanup ",\n"
    ++       suites = suites "        _clar_cb_" suite ", " count ", 1\n"
    ++       suites = suites "    },\n"
    ++}
    ++
    ++BEGIN {
    ++       suites = "static struct clar_suite _clar_suites[] = {\n"
    ++}
    ++
    ++{
    ++       print
    ++       name = $3; sub(/\(.*$/, "", name)
    ++       suite = name; sub(/^test_/, "", suite); sub(/__.*$/, "", suite)
    ++       short_name = name; sub(/^.*__/, "", short_name)
    ++       cb = "{ \"" short_name "\", &" name " }"
    ++       if (suite != prev_suite) {
    ++               add_suite(prev_suite, initialize, cleanup, count)
    ++               if (callbacks) callbacks = callbacks "};\n"
    ++               callbacks = callbacks "static const struct clar_func _clar_cb_" suite "[] = {\n"
    ++               initialize = "{ NULL, NULL }"
    ++               cleanup = "{ NULL, NULL }"
    ++               count = 0
    ++               prev_suite = suite
    ++       }
    ++       if (short_name == "initialize") {
    ++               initialize = cb
    ++       } else if (short_name == "cleanup") {
    ++               cleanup = cb
    ++       } else {
    ++               callbacks = callbacks "    " cb ",\n"
    ++               count++
    ++       }
    ++}
    ++
    ++END {
    ++       add_suite(suite, initialize, cleanup, count)
    ++       suites = suites "};"
    ++       if (callbacks) callbacks = callbacks "};"
    ++       print callbacks
    ++       print suites
    ++       print "static const size_t _clar_suite_count = " suite_count ";"
    ++       print "static const size_t _clar_callback_count = " callback_count ";"
    ++}
    +
      ## t/unit-tests/unit-test.c (new) ##
     @@
     +#include "unit-test.h"
     +
     +int cmd_main(int argc, const char **argv)
     +{
    -+	const char **args;
    ++	const char **argv_copy;
     +	int ret;
     +
     +	/* Append the "-t" flag such that the tests generate TAP output. */
    -+	DUP_ARRAY(args, argv, argc + 1);
    -+	args[argc++] = "-t";
    ++	ALLOC_ARRAY(argv_copy, argc + 2);
    ++	COPY_ARRAY(argv_copy, argv, argc);
    ++	argv_copy[argc++] = "-t";
    ++	argv_copy[argc] = NULL;
     +
    -+	ret = clar_test(argc, (char **) args);
    ++	ret = clar_test(argc, (char **) argv_copy);
     +
    -+	free(args);
    ++	free(argv_copy);
     +	return ret;
     +}

3: 1c2a510547 ! 6: 578e657269 t/unit-tests: convert strvec tests to use clar

    @@ Commit message
         Signed-off-by: Patrick Steinhardt <[email protected]>
     
      ## Makefile ##
    -@@ Makefile: THIRD_PARTY_SOURCES += compat/regex/%
    - THIRD_PARTY_SOURCES += sha1collisiondetection/%
    - THIRD_PARTY_SOURCES += sha1dc/%
    +@@ Makefile: THIRD_PARTY_SOURCES += sha1dc/%
    + THIRD_PARTY_SOURCES += $(UNIT_TEST_DIR)/clar/%
    + THIRD_PARTY_SOURCES += $(UNIT_TEST_DIR)/clar/clar/%
      
     +UNIT_TESTS_SUITES += strvec
      UNIT_TESTS_PROG = $(UNIT_TEST_BIN)/unit-tests$(X)
      UNIT_TESTS_OBJS = $(patsubst %,$(UNIT_TEST_DIR)/%.o,$(UNIT_TESTS_SUITES))
      UNIT_TESTS_OBJS += $(UNIT_TEST_DIR)/clar/clar.o
    -@@ Makefile: UNIT_TEST_PROGRAMS += t-reftable-basics
    +@@ Makefile: UNIT_TEST_PROGRAMS += t-reftable-merged
      UNIT_TEST_PROGRAMS += t-reftable-record
      UNIT_TEST_PROGRAMS += t-strbuf
      UNIT_TEST_PROGRAMS += t-strcmp-offset
    @@ t/unit-tests/t-strvec.c => t/unit-tests/strvec.c
      #include "strvec.h"
      
      #define check_strvec(vec, ...) \
    --	check_strvec_loc(TEST_LOCATION(), vec, __VA_ARGS__)
    -+	check_strvec_loc(__FILE__, __func__, __LINE__, vec, __VA_ARGS__)
    - LAST_ARG_MUST_BE_NULL
    --static void check_strvec_loc(const char *loc, struct strvec *vec, ...)
    -+static void check_strvec_loc(const char *file, const char *func, size_t line, struct strvec *vec, ...)
    - {
    - 	va_list ap;
    - 	size_t nr = 0;
    -@@ t/unit-tests/strvec.c: static void check_strvec_loc(const char *loc, struct strvec *vec, ...)
    - 		if (!str)
    - 			break;
    - 
    --		if (!check_uint(vec->nr, >, nr) ||
    --		    !check_uint(vec->alloc, >, nr) ||
    --		    !check_str(vec->v[nr], str)) {
    --			struct strbuf msg = STRBUF_INIT;
    --			strbuf_addf(&msg, "strvec index %"PRIuMAX, (uintmax_t) nr);
    --			test_assert(loc, msg.buf, 0);
    --			strbuf_release(&msg);
    --			va_end(ap);
    --			return;
    --		}
    -+		clar__assert(vec->nr > nr, file, func, line, "Vector size too small", NULL, 1);
    -+		clar__assert(vec->alloc > nr, file, func, line, "Vector allocation too small", NULL, 1);
    -+		cl_assert_equal_s(vec->v[nr], str);
    - 
    - 		nr++;
    - 	}
    - 	va_end(ap);
    - 
    --	check_uint(vec->nr, ==, nr);
    --	check_uint(vec->alloc, >=, nr);
    --	check_pointer_eq(vec->v[nr], NULL);
    -+	cl_assert(vec->nr == nr);
    -+	cl_assert(vec->alloc >= nr);
    -+	cl_assert_equal_p(vec->v[nr], NULL);
    - }
    + 	do { \
    + 		const char *expect[] = { __VA_ARGS__ }; \
    +-		if (check_uint(ARRAY_SIZE(expect), >, 0) && \
    +-		    check_pointer_eq(expect[ARRAY_SIZE(expect) - 1], NULL) && \
    +-		    check_uint((vec)->nr, ==, ARRAY_SIZE(expect) - 1) && \
    +-		    check_uint((vec)->nr, <=, (vec)->alloc)) { \
    +-			for (size_t i = 0; i < ARRAY_SIZE(expect); i++) { \
    +-				if (!check_str((vec)->v[i], expect[i])) { \
    +-					test_msg("      i: %"PRIuMAX, \
    +-						 (uintmax_t)i); \
    +-					break; \
    +-				} \
    +-			} \
    +-		} \
    ++		cl_assert(ARRAY_SIZE(expect) > 0); \
    ++		cl_assert_equal_p(expect[ARRAY_SIZE(expect) - 1], NULL); \
    ++		cl_assert_equal_i((vec)->nr, ARRAY_SIZE(expect) - 1); \
    ++		cl_assert((vec)->nr <= (vec)->alloc); \
    ++		for (size_t i = 0; i < ARRAY_SIZE(expect); i++) \
    ++			cl_assert_equal_s((vec)->v[i], expect[i]); \
    + 	} while (0)
      
     -static void t_static_init(void)
     +void test_strvec__init(void)
    @@ t/unit-tests/strvec.c: static void t_push(void)
      }
      
     -static void t_pushf(void)
    -+void test_strvec__pushft_pushf(void)
    ++void test_strvec__pushf(void)
      {
      	struct strvec vec = STRVEC_INIT;
      	strvec_pushf(&vec, "foo: %d", 1);
    @@ t/unit-tests/strvec.c: static void t_detach(void)
     -	TEST(t_detach(), "detach");
     -	return test_done();
     -}
    +
    + ## t/unit-tests/unit-test.c ##
    +@@ t/unit-tests/unit-test.c: int cmd_main(int argc, const char **argv)
    + 	int ret;
    + 
    + 	/* Append the "-t" flag such that the tests generate TAP output. */
    +-	ALLOC_ARRAY(argv_copy, argc + 2);
    ++	ALLOC_ARRAY(argv_copy, argc + 1);
    + 	COPY_ARRAY(argv_copy, argv, argc);
    + 	argv_copy[argc++] = "-t";
    +-	argv_copy[argc] = NULL;
    + 
    + 	ret = clar_test(argc, (char **) argv_copy);
    + 

-: ---------- > 7: 238de33b93 t/unit-tests: convert ctype tests to use clar

2.46.0.dirty

When using the prove target, we append GIT_TEST_OPTS to the arguments that we execute each of the tests with. This doesn't only include the intended test scripts, but also ends up passing the arguments to our unit tests. This is unintentional though as they do not even know to interpret those arguments, and is inconsistent with how we execute unit tests without prove.

This isn't much of an issue because our current set of unit tests mostly ignore their arguments anyway. With the introduction of clar-based unit tests this is about to become an issue though, as these do parse their command line argument to alter behaviour.

Prepare for this by passing GIT_TEST_OPTS to "run-test.sh" via an environment variable. Like this, we can conditionally forward it to our test scripts, only.

---
 t/Makefile    | 3 ++-
 t/run-test.sh | 2 +-
 2 files changed, 3 insertions(+), 2 deletions(-)

index 4c30e7c06f..d2212de0b7 100644
@@ -68,7 +68,8 @@ failed:
 	test -z "$$failed" || $(MAKE) $$failed
 
 prove: pre-clean check-chainlint $(TEST_LINT)
-	@echo "*** prove (shell & unit tests) ***"; $(CHAINLINTSUPPRESS) TEST_SHELL_PATH='$(TEST_SHELL_PATH_SQ)' $(PROVE) --exec ./run-test.sh $(GIT_PROVE_OPTS) $(T) $(UNIT_TESTS) :: $(GIT_TEST_OPTS)
+	@echo "*** prove (shell & unit tests) ***"
+	@$(CHAINLINTSUPPRESS) TEST_OPTIONS='$(GIT_TEST_OPTS)' TEST_SHELL_PATH='$(TEST_SHELL_PATH_SQ)' $(PROVE) --exec ./run-test.sh $(GIT_PROVE_OPTS) $(T) $(UNIT_TESTS)
 	$(MAKE) clean-except-prove-cache
 
 $(T):

index 13c353b91b..63328ac630 100755
@@ -10,7 +10,7 @@ case "$1" in
 		echo >&2 "ERROR: TEST_SHELL_PATH is empty or not set"
 		exit 1
 	fi
-	exec "${TEST_SHELL_PATH}" "$@"
+	exec "${TEST_SHELL_PATH}" "$@" ${TEST_OPTIONS}
 	;;
 *)
 	exec "$@"

2.46.0.dirty

Import the clar unit testing framework at commit faa8419 (Merge pull request #93 from clar-test/ethomson/fixtures, 2023-12-14). The framework will be wired up in subsequent commits.

---
 Documentation/technical/unit-tests.txt     |   2 +
 Makefile                                   |   4 +-
 t/unit-tests/clar/.github/workflows/ci.yml |  23 +
 t/unit-tests/clar/COPYING                  |  15 +
 t/unit-tests/clar/README.md                | 329 ++++++++
 t/unit-tests/clar/clar.c                   | 842 +++++++++++++++++++++
 t/unit-tests/clar/clar.h                   | 173 +++++
 t/unit-tests/clar/clar/fixtures.h          |  50 ++
 t/unit-tests/clar/clar/fs.h                | 522 +++++++++++++
 t/unit-tests/clar/clar/print.h             | 211 ++++++
 t/unit-tests/clar/clar/sandbox.h           | 154 ++++
 t/unit-tests/clar/clar/summary.h           | 143 ++++
 t/unit-tests/clar/generate.py              | 267 +++++++
 t/unit-tests/clar/test/.gitignore          |   5 +
 t/unit-tests/clar/test/Makefile            |  39 +
 t/unit-tests/clar/test/clar_test.h         |  16 +
 t/unit-tests/clar/test/main.c              |  40 +
 t/unit-tests/clar/test/main.c.sample       |  27 +
 t/unit-tests/clar/test/resources/test/file |   1 +
 t/unit-tests/clar/test/sample.c            |  84 ++
 20 files changed, 2946 insertions(+), 1 deletion(-)
 create mode 100644 t/unit-tests/clar/.github/workflows/ci.yml
 create mode 100644 t/unit-tests/clar/COPYING
 create mode 100644 t/unit-tests/clar/README.md
 create mode 100644 t/unit-tests/clar/clar.c
 create mode 100644 t/unit-tests/clar/clar.h
 create mode 100644 t/unit-tests/clar/clar/fixtures.h
 create mode 100644 t/unit-tests/clar/clar/fs.h
 create mode 100644 t/unit-tests/clar/clar/print.h
 create mode 100644 t/unit-tests/clar/clar/sandbox.h
 create mode 100644 t/unit-tests/clar/clar/summary.h
 create mode 100755 t/unit-tests/clar/generate.py
 create mode 100644 t/unit-tests/clar/test/.gitignore
 create mode 100644 t/unit-tests/clar/test/Makefile
 create mode 100644 t/unit-tests/clar/test/clar_test.h
 create mode 100644 t/unit-tests/clar/test/main.c
 create mode 100644 t/unit-tests/clar/test/main.c.sample
 create mode 100644 t/unit-tests/clar/test/resources/test/file
 create mode 100644 t/unit-tests/clar/test/sample.c

index 206037ffb1..fc201d4b3d 100644
@@ -203,6 +203,7 @@ GitHub / GitLab stars to estimate this.
 :criterion: https://github.com/Snaipe/Criterion[Criterion]
 :c-tap: https://github.com/rra/c-tap-harness/[C TAP]
 :check: https://libcheck.github.io/check/[Check]
+:clar: https://github.com/clar-test/clar[Clar]
 
 [format="csv",options="header",width="33%",subs="specialcharacters,attributes,quotes,macros"]
 |=====

@@ -212,6 +213,7 @@ Framework,"<<license,License>>","<<vendorable-or-ubiquitous,Vendorable or ubiqui
 {criterion},{mit},{false},{partial},{true},{true},{true},{true},{true},{false},{true},19,1800
 {c-tap},{expat},{true},{partial},{partial},{true},{false},{true},{false},{false},{false},4,33
 {check},{lgpl},{false},{partial},{true},{true},{true},{false},{false},{false},{true},17,973
+{clar},{lgpl},{false},{partial},{true},{true},{true},{true},{false},{false},{true},1,192
 |=====
 
 === Additional framework candidates

index 3863e60b66..674b0ac4e1 100644
@@ -1331,6 +1331,8 @@ THIRD_PARTY_SOURCES += compat/poll/%
 THIRD_PARTY_SOURCES += compat/regex/%
 THIRD_PARTY_SOURCES += sha1collisiondetection/%
 THIRD_PARTY_SOURCES += sha1dc/%
+THIRD_PARTY_SOURCES += $(UNIT_TEST_DIR)/clar/%
+THIRD_PARTY_SOURCES += $(UNIT_TEST_DIR)/clar/clar/%
 
 UNIT_TEST_PROGRAMS += t-ctype
 UNIT_TEST_PROGRAMS += t-example-decorate

@@ -3260,7 +3262,7 @@ $(SP_OBJ): %.sp: %.c %.o
 .PHONY: sparse
 sparse: $(SP_OBJ)
 
-EXCEPT_HDRS := $(GENERATED_H) unicode-width.h compat/% xdiff/%
+EXCEPT_HDRS := $(GENERATED_H) unicode-width.h compat/% xdiff/% $(UNIT_TEST_DIR)/clar/% $(UNIT_TEST_DIR)/clar/clar/%
 ifndef OPENSSL_SHA1
 	EXCEPT_HDRS += sha1/openssl.h
 endif

new file mode 100644
index 0000000000..b1ac2de460
@@ -0,0 +1,23 @@
+name: CI
+
+on:
+  push:
+    branches: [ main ]
+  pull_request:
+    branches: [ main ]
+
+jobs:
+  build:
+    strategy:
+      matrix:
+        os: [ ubuntu-latest, macos-latest ]
+
+    runs-on: ${{ matrix.os }}
+
+    steps:
+    - name: Check out
+      uses: actions/checkout@v2
+    - name: Build
+      run: |
+        cd test
+        make

new file mode 100644
index 0000000000..8983817f0c
@@ -0,0 +1,15 @@
+ISC License
+
+Copyright (c) 2011-2015 Vicent Marti
+
+Permission to use, copy, modify, and/or distribute this software for any
+purpose with or without fee is hereby granted, provided that the above
+copyright notice and this permission notice appear in all copies.
+
+THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
+WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
+MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
+ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
+WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
+ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
+OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.

new file mode 100644
index 0000000000..3e8ae0a3a5
@@ -0,0 +1,5 @@
+clar.suite
+.clarcache
+clar_test
+*.o
+

new file mode 100644
index 0000000000..93c6b2ad32
@@ -0,0 +1,39 @@
+#
+# Copyright (c) Vicent Marti. All rights reserved.
+#
+# This file is part of clar, distributed under the ISC license.
+# For full terms see the included COPYING file.
+#
+
+#
+# Set up the path to the clar sources and to the fixtures directory
+#
+# The fixture path needs to be an absolute path so it can be used
+# even after we have chdir'ed into the test directory while testing.
+#
+CURRENT_MAKEFILE  := $(word $(words $(MAKEFILE_LIST)),$(MAKEFILE_LIST))
+TEST_DIRECTORY    := $(abspath $(dir $(CURRENT_MAKEFILE)))
+CLAR_PATH         := $(dir $(TEST_DIRECTORY))
+CLAR_FIXTURE_PATH := $(TEST_DIRECTORY)/resources/
+
+CFLAGS=-g -I.. -I. -Wall -DCLAR_FIXTURE_PATH=\"$(CLAR_FIXTURE_PATH)\"
+
+.PHONY: clean
+
+# list the objects that go into our test
+objects = main.o sample.o
+
+# build the test executable itself
+clar_test: $(objects) clar_test.h clar.suite $(CLAR_PATH)clar.c
+	$(CC) $(CFLAGS) -o $@ "$(CLAR_PATH)clar.c" $(objects)
+
+# test object files depend on clar macros
+$(objects) : $(CLAR_PATH)clar.h
+
+# build the clar.suite file of test metadata
+clar.suite:
+	python "$(CLAR_PATH)generate.py" .
+
+# remove all generated files
+clean:
+	$(RM) -rf *.o clar.suite .clarcache clar_test clar_test.dSYM

new file mode 100644
index 0000000000..0fcaa639aa
@@ -0,0 +1,16 @@
+/*
+ * Copyright (c) Vicent Marti. All rights reserved.
+ *
+ * This file is part of clar, distributed under the ISC license.
+ * For full terms see the included COPYING file.
+ */
+#ifndef __CLAR_TEST__
+#define __CLAR_TEST__
+
+/* Import the standard clar helper functions */
+#include "clar.h"
+
+/* Your custom shared includes / defines here */
+extern int global_test_counter;
+
+#endif

new file mode 100644
index 0000000000..59e56ad255
@@ -0,0 +1,40 @@
+/*
+ * Copyright (c) Vicent Marti. All rights reserved.
+ *
+ * This file is part of clar, distributed under the ISC license.
+ * For full terms see the included COPYING file.
+ */
+
+#include "clar_test.h"
+
+/*
+ * Sample main() for clar tests.
+ *
+ * You should write your own main routine for clar tests that does specific
+ * setup and teardown as necessary for your application.  The only required
+ * line is the call to `clar_test(argc, argv)`, which will execute the test
+ * suite.  If you want to check the return value of the test application,
+ * your main() should return the same value returned by clar_test().
+ */
+
+int global_test_counter = 0;
+
+#ifdef _WIN32
+int __cdecl main(int argc, char *argv[])
+#else
+int main(int argc, char *argv[])
+#endif
+{
+	int ret;
+
+	/* Your custom initialization here */
+	global_test_counter = 0;
+
+	/* Run the test suite */
+	ret = clar_test(argc, argv);
+
+	/* Your custom cleanup here */
+	cl_assert_equal_i(8, global_test_counter);
+
+	return ret;
+}

new file mode 100644
index 0000000000..a4d91b72fa
@@ -0,0 +1,27 @@
+/*
+ * Copyright (c) Vicent Marti. All rights reserved.
+ *
+ * This file is part of clar, distributed under the ISC license.
+ * For full terms see the included COPYING file.
+ */
+
+#include "clar_test.h"
+
+/*
+ * Minimal main() for clar tests.
+ *
+ * Modify this with any application specific setup or teardown that you need.
+ * The only required line is the call to `clar_test(argc, argv)`, which will
+ * execute the test suite.  If you want to check the return value of the test
+ * application, main() should return the same value returned by clar_test().
+ */
+
+#ifdef _WIN32
+int __cdecl main(int argc, char *argv[])
+#else
+int main(int argc, char *argv[])
+#endif
+{
+	/* Run the test suite */
+	return clar_test(argc, argv);
+}

new file mode 100644
index 0000000000..220f4aa98a
@@ -0,0 +1 @@
+File

2.46.0.dirty

Fix whitespace errors in the clar that make git-apply(1) unhappy. This has been cherry-picked from the upstream pull request at [1].

[1]: https://github.com/clar-test/clar/pull/97

---
 t/unit-tests/clar/clar/sandbox.h  | 5 ++---
 t/unit-tests/clar/generate.py     | 1 -
 t/unit-tests/clar/test/.gitignore | 1 -
 3 files changed, 2 insertions(+), 5 deletions(-)

index 0ba1479620..7c177f3525 100644
@@ -25,9 +25,9 @@ find_tmp_path(char *buffer, size_t length)
 	static const size_t var_count = 5;
 	static const char *env_vars[] = {
 		"CLAR_TMP", "TMPDIR", "TMP", "TEMP", "USERPROFILE"
- 	};
+	};
 
- 	size_t i;
+	size_t i;
 
 	for (i = 0; i < var_count; ++i) {
 		const char *env = getenv(env_vars[i]);

@@ -151,4 +151,3 @@ const char *clar_sandbox_path(void)
 {
 	return _clar_path;
 }
-

index 931b4d60d9..80996ac3e7 100755
@@ -264,4 +264,3 @@ def write(self):
     suite.disable(options.excluded)
     if suite.write():
         print("Written `clar.suite` (%d tests in %d suites)" % (suite.callback_count(), suite.suite_count()))
-

index 3e8ae0a3a5..a477d0c40c 100644
@@ -2,4 +2,3 @@ clar.suite
 .clarcache
 clar_test
 *.o
-

2.46.0.dirty

The NonStop platform does not have mkdtemp() available, which we rely on in build_sandbox_path(). Fix this issue by using mktemp() and mkdir() instead on this platform.

This has been cherry-picked from the upstream pull request at [1].

[1]: https://github.com/clar-test/clar/pull/96

---
 t/unit-tests/clar/clar/sandbox.h | 6 ++++++
 1 file changed, 6 insertions(+)

index 7c177f3525..e25057b7c4 100644
@@ -120,6 +120,12 @@ static int build_sandbox_path(void)
 	if (_mktemp(_clar_path) == NULL)
 		return -1;
 
+	if (mkdir(_clar_path, 0700) != 0)
+		return -1;
+#elif defined(__TANDEM)
+	if (mktemp(_clar_path) == NULL)
+		return -1;
+
 	if (mkdir(_clar_path, 0700) != 0)
 		return -1;
 #elif defined(_WIN32)

2.46.0.dirty

Wire up the clar unit testing framework by introducing a new "unit-tests" executable. In contrast to the existing framework, this will result in a single executable for all test suites. The ability to pick specific tests to execute is retained via functionality built into the clar itself.

Note that we need to be a bit careful about how we need to invalidate our Makefile rules. While we obviously have to regenerate the clar suite when our test suites change, we also have to invalidate it in case any of the test suites gets removed. We do so by using our typical pattern of creating a GIT-TEST-SUITES file that gets updated whenever the set of test suites changes, so that we can easily depend on that file.

Another specialty is that we generate a "clar-decls.h" file. The test functions are neither static, nor do they have external declarations. This is because they are getting parsed via "generate.py", which then creates the external generations that get populated into an array. These declarations are only seen by the main function though.

The consequence is that we will get a bunch of "missing prototypes" errors from our compiler for each of these test functions. To fix those errors, we extract the extern declarations from "clar.suite" and put them into a standalone header that then gets included by each of our unit tests. This gets rid of compiler warnings for every function which has been extracted by "generate.py". More importantly though, it does _not_ get rid of warnings in case a function really isn't being used by anything. Thus, it would cause a compiler error if a function name was mistyped and thus not picked up by "generate.py".

---
 .gitignore                     |  1 +
 Makefile                       | 36 ++++++++++++++++++++----
 t/Makefile                     |  1 +
 t/unit-tests/.gitignore        |  2 ++
 t/unit-tests/clar-generate.awk | 50 ++++++++++++++++++++++++++++++++++
 t/unit-tests/unit-test.c       | 18 ++++++++++++
 t/unit-tests/unit-test.h       |  3 ++
 7 files changed, 106 insertions(+), 5 deletions(-)
 create mode 100644 t/unit-tests/clar-generate.awk
 create mode 100644 t/unit-tests/unit-test.c
 create mode 100644 t/unit-tests/unit-test.h

index 8caf3700c2..6687bd6db4 100644
@@ -9,6 +9,7 @@
 /GIT-PYTHON-VARS
 /GIT-SCRIPT-DEFINES
 /GIT-SPATCH-DEFINES
+/GIT-TEST-SUITES
 /GIT-USER-AGENT
 /GIT-VERSION-FILE
 /bin-wrappers/

index 674b0ac4e1..f56b14ad6d 100644
@@ -1334,6 +1334,11 @@ THIRD_PARTY_SOURCES += sha1dc/%
 THIRD_PARTY_SOURCES += $(UNIT_TEST_DIR)/clar/%
 THIRD_PARTY_SOURCES += $(UNIT_TEST_DIR)/clar/clar/%
 
+UNIT_TESTS_PROG = $(UNIT_TEST_BIN)/unit-tests$(X)
+UNIT_TESTS_OBJS = $(patsubst %,$(UNIT_TEST_DIR)/%.o,$(UNIT_TESTS_SUITES))
+UNIT_TESTS_OBJS += $(UNIT_TEST_DIR)/clar/clar.o
+UNIT_TESTS_OBJS += $(UNIT_TEST_DIR)/unit-test.o
+
 UNIT_TEST_PROGRAMS += t-ctype
 UNIT_TEST_PROGRAMS += t-example-decorate
 UNIT_TEST_PROGRAMS += t-hash

@@ -2713,6 +2718,7 @@ OBJECTS += $(XDIFF_OBJS)
 OBJECTS += $(FUZZ_OBJS)
 OBJECTS += $(REFTABLE_OBJS) $(REFTABLE_TEST_OBJS)
 OBJECTS += $(UNIT_TEST_OBJS)
+OBJECTS += $(UNIT_TESTS_OBJS)
 
 ifndef NO_CURL
 	OBJECTS += http.o http-walker.o remote-curl.o

@@ -3215,7 +3221,7 @@ endif
 
 test_bindir_programs := $(patsubst %,bin-wrappers/%,$(BINDIR_PROGRAMS_NEED_X) $(BINDIR_PROGRAMS_NO_X) $(TEST_PROGRAMS_NEED_X))
 
-all:: $(TEST_PROGRAMS) $(test_bindir_programs) $(UNIT_TEST_PROGS)
+all:: $(TEST_PROGRAMS) $(test_bindir_programs) $(UNIT_TEST_PROGS) $(UNIT_TESTS_PROG)
 
 bin-wrappers/%: wrap-for-bin.sh
 	$(call mkdir_p_parent_template)

@@ -3262,7 +3268,7 @@ $(SP_OBJ): %.sp: %.c %.o
 .PHONY: sparse
 sparse: $(SP_OBJ)
 
-EXCEPT_HDRS := $(GENERATED_H) unicode-width.h compat/% xdiff/% $(UNIT_TEST_DIR)/clar/% $(UNIT_TEST_DIR)/clar/clar/%
+EXCEPT_HDRS := $(GENERATED_H) unicode-width.h compat/% xdiff/% $(UNIT_TEST_DIR)/unit-test.h $(UNIT_TEST_DIR)/clar/% $(UNIT_TEST_DIR)/clar/clar/%
 ifndef OPENSSL_SHA1
 	EXCEPT_HDRS += sha1/openssl.h
 endif

@@ -3646,7 +3652,7 @@ endif
 
 artifacts-tar:: $(ALL_COMMANDS_TO_INSTALL) $(SCRIPT_LIB) $(OTHER_PROGRAMS) \
 		GIT-BUILD-OPTIONS $(TEST_PROGRAMS) $(test_bindir_programs) \
-		$(UNIT_TEST_PROGS) $(MOFILES)
+		$(UNIT_TEST_PROGS) $(UNIT_TESTS_PROG) $(MOFILES)
 	$(QUIET_SUBDIR0)templates $(QUIET_SUBDIR1) \
 		SHELL_PATH='$(SHELL_PATH_SQ)' PERL_PATH='$(PERL_PATH_SQ)'
 	test -n "$(ARTIFACTS_DIRECTORY)"

@@ -3702,6 +3708,7 @@ cocciclean:
 
 clean: profile-clean coverage-clean cocciclean
 	$(RM) -r .build $(UNIT_TEST_BIN)
+	$(RM) GIT-TEST-SUITES $(UNIT_TEST_DIR)/clar.suite $(UNIT_TEST_DIR)/clar-decls.h
 	$(RM) po/git.pot po/git-core.pot
 	$(RM) git.res
 	$(RM) $(OBJECTS)

@@ -3861,7 +3868,26 @@ $(UNIT_TEST_PROGS): $(UNIT_TEST_BIN)/%$X: $(UNIT_TEST_DIR)/%.o \
 	$(QUIET_LINK)$(CC) $(ALL_CFLAGS) -o $@ $(ALL_LDFLAGS) \
 		$(filter %.o,$^) $(filter %.a,$^) $(LIBS)
 
+GIT-TEST-SUITES: FORCE
+	@FLAGS='$(UNIT_TESTS_SUITES)'; \
+	    if test x"$$FLAGS" != x"`cat GIT-TEST-SUITES 2>/dev/null`" ; then \
+		echo >&2 "    * new test suites"; \
+		echo "$$FLAGS" >GIT-TEST-SUITES; \
+            fi
+
+$(UNIT_TEST_DIR)/clar-decls.h: $(patsubst %,$(UNIT_TEST_DIR)/%.c,$(UNIT_TESTS_SUITES)) GIT-TEST-SUITES
+	$(QUIET_GEN)for suite in $(UNIT_TESTS_SUITES); do \
+		sed -ne "s/^\(void test_$${suite}__[a-zA-Z_0-9][a-zA-Z_0-9]*(void)$$\)/extern \1;/p" $(UNIT_TEST_DIR)/$$suite.c; \
+	done >$@
+$(UNIT_TEST_DIR)/clar.suite: $(UNIT_TEST_DIR)/clar-decls.h
+	$(QUIET_GEN)awk -f $(UNIT_TEST_DIR)/clar-generate.awk $< >$(UNIT_TEST_DIR)/clar.suite
+$(UNIT_TESTS_OBJS): $(UNIT_TEST_DIR)/clar-decls.h
+$(UNIT_TESTS_OBJS): EXTRA_CPPFLAGS = -I$(UNIT_TEST_DIR) -I$(UNIT_TEST_DIR)/clar
+$(UNIT_TESTS_PROG): $(UNIT_TEST_DIR)/clar.suite $(UNIT_TESTS_OBJS) $(GITLIBS) GIT-LDFLAGS
+	$(call mkdir_p_parent_template)
+	$(QUIET_LINK)$(CC) $(ALL_CFLAGS) -o $@ $(ALL_LDFLAGS) $(filter %.o,$^) $(LIBS)
+
 .PHONY: build-unit-tests unit-tests
-build-unit-tests: $(UNIT_TEST_PROGS)
-unit-tests: $(UNIT_TEST_PROGS) t/helper/test-tool$X
+build-unit-tests: $(UNIT_TEST_PROGS) $(UNIT_TESTS_PROG)
+unit-tests: $(UNIT_TEST_PROGS) $(UNIT_TESTS_PROG) t/helper/test-tool$X
 	$(MAKE) -C t/ unit-tests

index d2212de0b7..131ffd778f 100644
@@ -48,6 +48,7 @@ CHAINLINTTESTS = $(sort $(patsubst chainlint/%.test,%,$(wildcard chainlint/*.tes
 CHAINLINT = '$(PERL_PATH_SQ)' chainlint.pl
 UNIT_TEST_SOURCES = $(wildcard unit-tests/t-*.c)
 UNIT_TEST_PROGRAMS = $(patsubst unit-tests/%.c,unit-tests/bin/%$(X),$(UNIT_TEST_SOURCES))
+UNIT_TEST_PROGRAMS += unit-tests/bin/unit-tests$(X)
 UNIT_TESTS = $(sort $(UNIT_TEST_PROGRAMS))
 UNIT_TESTS_NO_DIR = $(notdir $(UNIT_TESTS))
 

index 5e56e040ec..d0632ec7f9 100644
@@ -1 +1,3 @@
 /bin
+/clar.suite
+/clar-decls.h

new file mode 100644
index 0000000000..32a81299e9
@@ -0,0 +1,18 @@
+#include "unit-test.h"
+
+int cmd_main(int argc, const char **argv)
+{
+	const char **argv_copy;
+	int ret;
+
+	/* Append the "-t" flag such that the tests generate TAP output. */
+	ALLOC_ARRAY(argv_copy, argc + 2);
+	COPY_ARRAY(argv_copy, argv, argc);
+	argv_copy[argc++] = "-t";
+	argv_copy[argc] = NULL;
+
+	ret = clar_test(argc, (char **) argv_copy);
+
+	free(argv_copy);
+	return ret;
+}

new file mode 100644
index 0000000000..99d59df1b0
@@ -0,0 +1,3 @@
+#include "git-compat-util.h"
+#include "clar.h"
+#include "clar-decls.h"

2.46.0.dirty

Convert the strvec tests to use the new clar unit testing framework. This is a first test balloon that demonstrates how the testing infra for clar-based tests looks like.

The tests are part of the "t/unit-tests/bin/unit-tests" binary. When running that binary, it generates TAP output:

    # ./t/unit-tests/bin/unit-tests
    TAP version 13
    # start of suite 1: strvec
    ok 1 - strvec::init
    ok 2 - strvec::dynamic_init
    ok 3 - strvec::clear
    ok 4 - strvec::push
    ok 5 - strvec::pushft_pushf
    ok 6 - strvec::pushl
    ok 7 - strvec::pushv
    ok 8 - strvec::replace_at_head
    ok 9 - strvec::replace_at_tail
    ok 10 - strvec::replace_in_between
    ok 11 - strvec::replace_with_substring
    ok 12 - strvec::remove_at_head
    ok 13 - strvec::remove_at_tail
    ok 14 - strvec::remove_in_between
    ok 15 - strvec::pop_empty_array
    ok 16 - strvec::pop_non_empty_array
    ok 17 - strvec::split_empty_string
    ok 18 - strvec::split_single_item
    ok 19 - strvec::split_multiple_items
    ok 20 - strvec::split_whitespace_only
    ok 21 - strvec::split_multiple_consecutive_whitespaces
    ok 22 - strvec::detach
    1..22

The binary also supports some parameters that allow us to run only a subset of unit tests or alter the output:

    $ ./t/unit-tests/bin/unit-tests -h
    Usage: ./t/unit-tests/bin/unit-tests [options]

Options:
-sname Run only the suite with name (can go to individual test name)
-iname Include the suite with name
-xname Exclude the suite with name
-v Increase verbosity (show suite names)
-q Only report tests that had an error
-Q Quit as soon as a test fails
-t Display results in tap format
-l Print suite names
-r[filename] Write summary file (to the optional filename)

Furthermore, running make unit-tests runs the binary along with all the other unit tests we have.

---
 Makefile                              |   2 +-
 t/unit-tests/{t-strvec.c => strvec.c} | 119 ++++++++++----------------
 t/unit-tests/unit-test.c              |   3 +-
 3 files changed, 45 insertions(+), 79 deletions(-)
 rename t/unit-tests/{t-strvec.c => strvec.c} (54%)

index f56b14ad6d..cf76f9b353 100644
@@ -1334,6 +1334,7 @@ THIRD_PARTY_SOURCES += sha1dc/%
 THIRD_PARTY_SOURCES += $(UNIT_TEST_DIR)/clar/%
 THIRD_PARTY_SOURCES += $(UNIT_TEST_DIR)/clar/clar/%
 
+UNIT_TESTS_SUITES += strvec
 UNIT_TESTS_PROG = $(UNIT_TEST_BIN)/unit-tests$(X)
 UNIT_TESTS_OBJS = $(patsubst %,$(UNIT_TEST_DIR)/%.o,$(UNIT_TESTS_SUITES))
 UNIT_TESTS_OBJS += $(UNIT_TEST_DIR)/clar/clar.o

@@ -1351,7 +1352,6 @@ UNIT_TEST_PROGRAMS += t-reftable-merged
 UNIT_TEST_PROGRAMS += t-reftable-record
 UNIT_TEST_PROGRAMS += t-strbuf
 UNIT_TEST_PROGRAMS += t-strcmp-offset
-UNIT_TEST_PROGRAMS += t-strvec
 UNIT_TEST_PROGRAMS += t-trailer
 UNIT_TEST_PROGS = $(patsubst %,$(UNIT_TEST_BIN)/%$X,$(UNIT_TEST_PROGRAMS))
 UNIT_TEST_OBJS = $(patsubst %,$(UNIT_TEST_DIR)/%.o,$(UNIT_TEST_PROGRAMS))

index 32a81299e9..82b7635e6a 100644
@@ -6,10 +6,9 @@ int cmd_main(int argc, const char **argv)
 	int ret;
 
 	/* Append the "-t" flag such that the tests generate TAP output. */
-	ALLOC_ARRAY(argv_copy, argc + 2);
+	ALLOC_ARRAY(argv_copy, argc + 1);
 	COPY_ARRAY(argv_copy, argv, argc);
 	argv_copy[argc++] = "-t";
-	argv_copy[argc] = NULL;
 
 	ret = clar_test(argc, (char **) argv_copy);
 

2.46.0.dirty

Convert the ctype tests to use the new clar unit testing framework.

---
 Makefile                            |  2 +-
 t/unit-tests/{t-ctype.c => ctype.c} | 71 +++++++++++++++++++++++------
 2 files changed, 59 insertions(+), 14 deletions(-)
 rename t/unit-tests/{t-ctype.c => ctype.c} (71%)

index cf76f9b353..aa2bd5cec1 100644
@@ -1334,13 +1334,13 @@ THIRD_PARTY_SOURCES += sha1dc/%
 THIRD_PARTY_SOURCES += $(UNIT_TEST_DIR)/clar/%
 THIRD_PARTY_SOURCES += $(UNIT_TEST_DIR)/clar/clar/%
 
+UNIT_TESTS_SUITES += ctype
 UNIT_TESTS_SUITES += strvec
 UNIT_TESTS_PROG = $(UNIT_TEST_BIN)/unit-tests$(X)
 UNIT_TESTS_OBJS = $(patsubst %,$(UNIT_TEST_DIR)/%.o,$(UNIT_TESTS_SUITES))
 UNIT_TESTS_OBJS += $(UNIT_TEST_DIR)/clar/clar.o
 UNIT_TESTS_OBJS += $(UNIT_TEST_DIR)/unit-test.o
 
-UNIT_TEST_PROGRAMS += t-ctype
 UNIT_TEST_PROGRAMS += t-example-decorate
 UNIT_TEST_PROGRAMS += t-hash
 UNIT_TEST_PROGRAMS += t-mem-pool

2.46.0.dirty

Yes it is, and that's also what I said in other mails. Guess this here was a copy-paste error, thanks for spotting it!

Patrick

No, there isn't. Let me adapt these indeed, thanks!

Patrick

Hi,

this is the third version of my RFC patch series that introduces the clar testing framework into our unit tests. The intent is to not have to hand-craft all features of a proper unit testing framework, while still not painting us into a corner. As such, the clar itself is small and extensible while still bringing some nice features to the table.

Changes compared to v2:

- Fix a copy/paste error for the clar license. It's ISC, not LGPL.

- Include "clar.h" via "clar/clar.h" such that we do not have to add "clar/" as in preprocessor include directive.

- Adapt strvec unit test to use cl_assert_equal_i() instead of

    `cl_assert()`.

Thanks!

Patrick

Patrick Steinhardt (7):
t: do not pass GIT_TEST_OPTS to unit tests with prove
t: import the clar unit testing framework t/clar: fix whitespace errors
t/clar: fix compatibility with NonStop
Makefile: wire up the clar unit testing framework t/unit-tests: convert strvec tests to use clar t/unit-tests: convert ctype tests to use clar

 .gitignore                                 |   1 +
 Documentation/technical/unit-tests.txt     |   2 +
 Makefile                                   |  42 +-
 t/Makefile                                 |   4 +-
 t/run-test.sh                              |   2 +-
 t/unit-tests/.gitignore                    |   2 +
 t/unit-tests/clar-generate.awk             |  50 ++
 t/unit-tests/clar/.github/workflows/ci.yml |  23 +
 t/unit-tests/clar/COPYING                  |  15 +
 t/unit-tests/clar/README.md                | 329 ++++++++
 t/unit-tests/clar/clar.c                   | 842 +++++++++++++++++++++
 t/unit-tests/clar/clar.h                   | 173 +++++
 t/unit-tests/clar/clar/fixtures.h          |  50 ++
 t/unit-tests/clar/clar/fs.h                | 522 +++++++++++++
 t/unit-tests/clar/clar/print.h             | 211 ++++++
 t/unit-tests/clar/clar/sandbox.h           | 159 ++++
 t/unit-tests/clar/clar/summary.h           | 143 ++++
 t/unit-tests/clar/generate.py              | 266 +++++++
 t/unit-tests/clar/test/.gitignore          |   4 +
 t/unit-tests/clar/test/Makefile            |  39 +
 t/unit-tests/clar/test/clar_test.h         |  16 +
 t/unit-tests/clar/test/main.c              |  40 +
 t/unit-tests/clar/test/main.c.sample       |  27 +
 t/unit-tests/clar/test/resources/test/file |   1 +
 t/unit-tests/clar/test/sample.c            |  84 ++
 t/unit-tests/{t-ctype.c => ctype.c}        |  71 +-
 t/unit-tests/{t-strvec.c => strvec.c}      | 119 ++-
 t/unit-tests/unit-test.c                   |  17 +
 t/unit-tests/unit-test.h                   |   3 +
 29 files changed, 3159 insertions(+), 98 deletions(-)
 create mode 100644 t/unit-tests/clar-generate.awk
 create mode 100644 t/unit-tests/clar/.github/workflows/ci.yml
 create mode 100644 t/unit-tests/clar/COPYING
 create mode 100644 t/unit-tests/clar/README.md
 create mode 100644 t/unit-tests/clar/clar.c
 create mode 100644 t/unit-tests/clar/clar.h
 create mode 100644 t/unit-tests/clar/clar/fixtures.h
 create mode 100644 t/unit-tests/clar/clar/fs.h
 create mode 100644 t/unit-tests/clar/clar/print.h
 create mode 100644 t/unit-tests/clar/clar/sandbox.h
 create mode 100644 t/unit-tests/clar/clar/summary.h
 create mode 100755 t/unit-tests/clar/generate.py
 create mode 100644 t/unit-tests/clar/test/.gitignore
 create mode 100644 t/unit-tests/clar/test/Makefile
 create mode 100644 t/unit-tests/clar/test/clar_test.h
 create mode 100644 t/unit-tests/clar/test/main.c
 create mode 100644 t/unit-tests/clar/test/main.c.sample
 create mode 100644 t/unit-tests/clar/test/resources/test/file
 create mode 100644 t/unit-tests/clar/test/sample.c
 rename t/unit-tests/{t-ctype.c => ctype.c} (71%)
 rename t/unit-tests/{t-strvec.c => strvec.c} (54%)
 create mode 100644 t/unit-tests/unit-test.c
 create mode 100644 t/unit-tests/unit-test.h

Range-diff against v2:

1:  78a9cc1162 = 1:  78a9cc1162 t: do not pass GIT_TEST_OPTS to unit tests with prove
2:  6a88cf22a5 ! 2:  b6c066ee4e t: import the clar unit testing framework
    @@ Documentation/technical/unit-tests.txt: Framework,"<<license,License>>","<<vendo
      {criterion},{mit},{false},{partial},{true},{true},{true},{true},{true},{false},{true},19,1800
      {c-tap},{expat},{true},{partial},{partial},{true},{false},{true},{false},{false},{false},4,33
      {check},{lgpl},{false},{partial},{true},{true},{true},{false},{false},{false},{true},17,973
    -+{clar},{lgpl},{false},{partial},{true},{true},{true},{true},{false},{false},{true},1,192
    ++{clar},{isc},{false},{partial},{true},{true},{true},{true},{false},{false},{true},1,192
      |=====
      
      === Additional framework candidates
3:  a52ee59bf4 = 3:  35682b7686 t/clar: fix whitespace errors
4:  02fb86dfbc = 4:  7a76c21bcb t/clar: fix compatibility with NonStop
5:  848dc673c4 ! 5:  68b3c65951 Makefile: wire up the clar unit testing framework
    @@ Makefile: endif
      
      bin-wrappers/%: wrap-for-bin.sh
      	$(call mkdir_p_parent_template)
    -@@ Makefile: $(SP_OBJ): %.sp: %.c %.o
    - .PHONY: sparse
    - sparse: $(SP_OBJ)
    - 
    --EXCEPT_HDRS := $(GENERATED_H) unicode-width.h compat/% xdiff/% $(UNIT_TEST_DIR)/clar/% $(UNIT_TEST_DIR)/clar/clar/%
    -+EXCEPT_HDRS := $(GENERATED_H) unicode-width.h compat/% xdiff/% $(UNIT_TEST_DIR)/unit-test.h $(UNIT_TEST_DIR)/clar/% $(UNIT_TEST_DIR)/clar/clar/%
    - ifndef OPENSSL_SHA1
    - 	EXCEPT_HDRS += sha1/openssl.h
    - endif
     @@ Makefile: endif
      
      artifacts-tar:: $(ALL_COMMANDS_TO_INSTALL) $(SCRIPT_LIB) $(OTHER_PROGRAMS) \
    @@ Makefile: $(UNIT_TEST_PROGS): $(UNIT_TEST_BIN)/%$X: $(UNIT_TEST_DIR)/%.o \
     +$(UNIT_TEST_DIR)/clar.suite: $(UNIT_TEST_DIR)/clar-decls.h
     +	$(QUIET_GEN)awk -f $(UNIT_TEST_DIR)/clar-generate.awk $< >$(UNIT_TEST_DIR)/clar.suite
     +$(UNIT_TESTS_OBJS): $(UNIT_TEST_DIR)/clar-decls.h
    -+$(UNIT_TESTS_OBJS): EXTRA_CPPFLAGS = -I$(UNIT_TEST_DIR) -I$(UNIT_TEST_DIR)/clar
    ++$(UNIT_TESTS_OBJS): EXTRA_CPPFLAGS = -I$(UNIT_TEST_DIR)
     +$(UNIT_TESTS_PROG): $(UNIT_TEST_DIR)/clar.suite $(UNIT_TESTS_OBJS) $(GITLIBS) GIT-LDFLAGS
     +	$(call mkdir_p_parent_template)
     +	$(QUIET_LINK)$(CC) $(ALL_CFLAGS) -o $@ $(ALL_LDFLAGS) $(filter %.o,$^) $(LIBS)
    @@ t/unit-tests/unit-test.c (new)
      ## t/unit-tests/unit-test.h (new) ##
     @@
     +#include "git-compat-util.h"
    -+#include "clar.h"
    ++#include "clar/clar.h"
     +#include "clar-decls.h"
6:  578e657269 ! 6:  4a0888380e t/unit-tests: convert strvec tests to use clar
    @@ t/unit-tests/t-strvec.c => t/unit-tests/strvec.c
     -	check_uint(vec.nr, ==, 0);
     -	check_uint(vec.alloc, ==, 0);
     +	cl_assert_equal_p(vec.v, empty_strvec);
    -+	cl_assert(vec.nr == 0);
    -+	cl_assert(vec.alloc == 0);
    ++	cl_assert_equal_i(vec.nr, 0);
    ++	cl_assert_equal_i(vec.alloc, 0);
      }
      
     -static void t_dynamic_init(void)
    @@ t/unit-tests/t-strvec.c => t/unit-tests/strvec.c
     -	check_uint(vec.nr, ==, 0);
     -	check_uint(vec.alloc, ==, 0);
     +	cl_assert_equal_p(vec.v, empty_strvec);
    -+	cl_assert(vec.nr == 0);
    -+	cl_assert(vec.alloc == 0);
    ++	cl_assert_equal_i(vec.nr, 0);
    ++	cl_assert_equal_i(vec.alloc, 0);
      }
      
     -static void t_clear(void)
    @@ t/unit-tests/t-strvec.c => t/unit-tests/strvec.c
     -	check_uint(vec.nr, ==, 0);
     -	check_uint(vec.alloc, ==, 0);
     +	cl_assert_equal_p(vec.v, empty_strvec);
    -+	cl_assert(vec.nr == 0);
    -+	cl_assert(vec.alloc == 0);
    ++	cl_assert_equal_i(vec.nr, 0);
    ++	cl_assert_equal_i(vec.alloc, 0);
      }
      
     -static void t_push(void)
    @@ t/unit-tests/strvec.c: static void t_detach(void)
     -	check_uint(vec.nr, ==, 0);
     -	check_uint(vec.alloc, ==, 0);
     +	cl_assert_equal_p(vec.v, empty_strvec);
    -+	cl_assert(vec.nr == 0);
    -+	cl_assert(vec.alloc == 0);
    ++	cl_assert_equal_i(vec.nr, 0);
    ++	cl_assert_equal_i(vec.alloc, 0);
      
      	free((char *) detached[0]);
      	free(detached);

7: 238de33b93 = 7: f423b01c05 t/unit-tests: convert ctype tests to use clar

2.46.0.dirty

When using the prove target, we append GIT_TEST_OPTS to the arguments that we execute each of the tests with. This doesn't only include the intended test scripts, but also ends up passing the arguments to our unit tests. This is unintentional though as they do not even know to interpret those arguments, and is inconsistent with how we execute unit tests without prove.

This isn't much of an issue because our current set of unit tests mostly ignore their arguments anyway. With the introduction of clar-based unit tests this is about to become an issue though, as these do parse their command line argument to alter behaviour.

Prepare for this by passing GIT_TEST_OPTS to "run-test.sh" via an environment variable. Like this, we can conditionally forward it to our test scripts, only.

---
 t/Makefile    | 3 ++-
 t/run-test.sh | 2 +-
 2 files changed, 3 insertions(+), 2 deletions(-)

index 4c30e7c06f..d2212de0b7 100644
@@ -68,7 +68,8 @@ failed:
 	test -z "$$failed" || $(MAKE) $$failed
 
 prove: pre-clean check-chainlint $(TEST_LINT)
-	@echo "*** prove (shell & unit tests) ***"; $(CHAINLINTSUPPRESS) TEST_SHELL_PATH='$(TEST_SHELL_PATH_SQ)' $(PROVE) --exec ./run-test.sh $(GIT_PROVE_OPTS) $(T) $(UNIT_TESTS) :: $(GIT_TEST_OPTS)
+	@echo "*** prove (shell & unit tests) ***"
+	@$(CHAINLINTSUPPRESS) TEST_OPTIONS='$(GIT_TEST_OPTS)' TEST_SHELL_PATH='$(TEST_SHELL_PATH_SQ)' $(PROVE) --exec ./run-test.sh $(GIT_PROVE_OPTS) $(T) $(UNIT_TESTS)
 	$(MAKE) clean-except-prove-cache
 
 $(T):

index 13c353b91b..63328ac630 100755
@@ -10,7 +10,7 @@ case "$1" in
 		echo >&2 "ERROR: TEST_SHELL_PATH is empty or not set"
 		exit 1
 	fi
-	exec "${TEST_SHELL_PATH}" "$@"
+	exec "${TEST_SHELL_PATH}" "$@" ${TEST_OPTIONS}
 	;;
 *)
 	exec "$@"

2.46.0.dirty

Import the clar unit testing framework at commit faa8419 (Merge pull request #93 from clar-test/ethomson/fixtures, 2023-12-14). The framework will be wired up in subsequent commits.

---
 Documentation/technical/unit-tests.txt     |   2 +
 Makefile                                   |   4 +-
 t/unit-tests/clar/.github/workflows/ci.yml |  23 +
 t/unit-tests/clar/COPYING                  |  15 +
 t/unit-tests/clar/README.md                | 329 ++++++++
 t/unit-tests/clar/clar.c                   | 842 +++++++++++++++++++++
 t/unit-tests/clar/clar.h                   | 173 +++++
 t/unit-tests/clar/clar/fixtures.h          |  50 ++
 t/unit-tests/clar/clar/fs.h                | 522 +++++++++++++
 t/unit-tests/clar/clar/print.h             | 211 ++++++
 t/unit-tests/clar/clar/sandbox.h           | 154 ++++
 t/unit-tests/clar/clar/summary.h           | 143 ++++
 t/unit-tests/clar/generate.py              | 267 +++++++
 t/unit-tests/clar/test/.gitignore          |   5 +
 t/unit-tests/clar/test/Makefile            |  39 +
 t/unit-tests/clar/test/clar_test.h         |  16 +
 t/unit-tests/clar/test/main.c              |  40 +
 t/unit-tests/clar/test/main.c.sample       |  27 +
 t/unit-tests/clar/test/resources/test/file |   1 +
 t/unit-tests/clar/test/sample.c            |  84 ++
 20 files changed, 2946 insertions(+), 1 deletion(-)
 create mode 100644 t/unit-tests/clar/.github/workflows/ci.yml
 create mode 100644 t/unit-tests/clar/COPYING
 create mode 100644 t/unit-tests/clar/README.md
 create mode 100644 t/unit-tests/clar/clar.c
 create mode 100644 t/unit-tests/clar/clar.h
 create mode 100644 t/unit-tests/clar/clar/fixtures.h
 create mode 100644 t/unit-tests/clar/clar/fs.h
 create mode 100644 t/unit-tests/clar/clar/print.h
 create mode 100644 t/unit-tests/clar/clar/sandbox.h
 create mode 100644 t/unit-tests/clar/clar/summary.h
 create mode 100755 t/unit-tests/clar/generate.py
 create mode 100644 t/unit-tests/clar/test/.gitignore
 create mode 100644 t/unit-tests/clar/test/Makefile
 create mode 100644 t/unit-tests/clar/test/clar_test.h
 create mode 100644 t/unit-tests/clar/test/main.c
 create mode 100644 t/unit-tests/clar/test/main.c.sample
 create mode 100644 t/unit-tests/clar/test/resources/test/file
 create mode 100644 t/unit-tests/clar/test/sample.c

index 206037ffb1..5a432b7b29 100644
@@ -203,6 +203,7 @@ GitHub / GitLab stars to estimate this.
 :criterion: https://github.com/Snaipe/Criterion[Criterion]
 :c-tap: https://github.com/rra/c-tap-harness/[C TAP]
 :check: https://libcheck.github.io/check/[Check]
+:clar: https://github.com/clar-test/clar[Clar]
 
 [format="csv",options="header",width="33%",subs="specialcharacters,attributes,quotes,macros"]
 |=====

@@ -212,6 +213,7 @@ Framework,"<<license,License>>","<<vendorable-or-ubiquitous,Vendorable or ubiqui
 {criterion},{mit},{false},{partial},{true},{true},{true},{true},{true},{false},{true},19,1800
 {c-tap},{expat},{true},{partial},{partial},{true},{false},{true},{false},{false},{false},4,33
 {check},{lgpl},{false},{partial},{true},{true},{true},{false},{false},{false},{true},17,973
+{clar},{isc},{false},{partial},{true},{true},{true},{true},{false},{false},{true},1,192
 |=====
 
 === Additional framework candidates

index 3863e60b66..674b0ac4e1 100644
@@ -1331,6 +1331,8 @@ THIRD_PARTY_SOURCES += compat/poll/%
 THIRD_PARTY_SOURCES += compat/regex/%
 THIRD_PARTY_SOURCES += sha1collisiondetection/%
 THIRD_PARTY_SOURCES += sha1dc/%
+THIRD_PARTY_SOURCES += $(UNIT_TEST_DIR)/clar/%
+THIRD_PARTY_SOURCES += $(UNIT_TEST_DIR)/clar/clar/%
 
 UNIT_TEST_PROGRAMS += t-ctype
 UNIT_TEST_PROGRAMS += t-example-decorate

@@ -3260,7 +3262,7 @@ $(SP_OBJ): %.sp: %.c %.o
 .PHONY: sparse
 sparse: $(SP_OBJ)
 
-EXCEPT_HDRS := $(GENERATED_H) unicode-width.h compat/% xdiff/%
+EXCEPT_HDRS := $(GENERATED_H) unicode-width.h compat/% xdiff/% $(UNIT_TEST_DIR)/clar/% $(UNIT_TEST_DIR)/clar/clar/%
 ifndef OPENSSL_SHA1
 	EXCEPT_HDRS += sha1/openssl.h
 endif

new file mode 100644
index 0000000000..b1ac2de460
@@ -0,0 +1,23 @@
+name: CI
+
+on:
+  push:
+    branches: [ main ]
+  pull_request:
+    branches: [ main ]
+
+jobs:
+  build:
+    strategy:
+      matrix:
+        os: [ ubuntu-latest, macos-latest ]
+
+    runs-on: ${{ matrix.os }}
+
+    steps:
+    - name: Check out
+      uses: actions/checkout@v2
+    - name: Build
+      run: |
+        cd test
+        make

new file mode 100644
index 0000000000..8983817f0c
@@ -0,0 +1,15 @@
+ISC License
+
+Copyright (c) 2011-2015 Vicent Marti
+
+Permission to use, copy, modify, and/or distribute this software for any
+purpose with or without fee is hereby granted, provided that the above
+copyright notice and this permission notice appear in all copies.
+
+THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
+WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
+MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
+ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
+WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
+ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
+OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.

new file mode 100644
index 0000000000..3e8ae0a3a5
@@ -0,0 +1,5 @@
+clar.suite
+.clarcache
+clar_test
+*.o
+

new file mode 100644
index 0000000000..93c6b2ad32
@@ -0,0 +1,39 @@
+#
+# Copyright (c) Vicent Marti. All rights reserved.
+#
+# This file is part of clar, distributed under the ISC license.
+# For full terms see the included COPYING file.
+#
+
+#
+# Set up the path to the clar sources and to the fixtures directory
+#
+# The fixture path needs to be an absolute path so it can be used
+# even after we have chdir'ed into the test directory while testing.
+#
+CURRENT_MAKEFILE  := $(word $(words $(MAKEFILE_LIST)),$(MAKEFILE_LIST))
+TEST_DIRECTORY    := $(abspath $(dir $(CURRENT_MAKEFILE)))
+CLAR_PATH         := $(dir $(TEST_DIRECTORY))
+CLAR_FIXTURE_PATH := $(TEST_DIRECTORY)/resources/
+
+CFLAGS=-g -I.. -I. -Wall -DCLAR_FIXTURE_PATH=\"$(CLAR_FIXTURE_PATH)\"
+
+.PHONY: clean
+
+# list the objects that go into our test
+objects = main.o sample.o
+
+# build the test executable itself
+clar_test: $(objects) clar_test.h clar.suite $(CLAR_PATH)clar.c
+	$(CC) $(CFLAGS) -o $@ "$(CLAR_PATH)clar.c" $(objects)
+
+# test object files depend on clar macros
+$(objects) : $(CLAR_PATH)clar.h
+
+# build the clar.suite file of test metadata
+clar.suite:
+	python "$(CLAR_PATH)generate.py" .
+
+# remove all generated files
+clean:
+	$(RM) -rf *.o clar.suite .clarcache clar_test clar_test.dSYM

new file mode 100644
index 0000000000..0fcaa639aa
@@ -0,0 +1,16 @@
+/*
+ * Copyright (c) Vicent Marti. All rights reserved.
+ *
+ * This file is part of clar, distributed under the ISC license.
+ * For full terms see the included COPYING file.
+ */
+#ifndef __CLAR_TEST__
+#define __CLAR_TEST__
+
+/* Import the standard clar helper functions */
+#include "clar.h"
+
+/* Your custom shared includes / defines here */
+extern int global_test_counter;
+
+#endif

new file mode 100644
index 0000000000..59e56ad255
@@ -0,0 +1,40 @@
+/*
+ * Copyright (c) Vicent Marti. All rights reserved.
+ *
+ * This file is part of clar, distributed under the ISC license.
+ * For full terms see the included COPYING file.
+ */
+
+#include "clar_test.h"
+
+/*
+ * Sample main() for clar tests.
+ *
+ * You should write your own main routine for clar tests that does specific
+ * setup and teardown as necessary for your application.  The only required
+ * line is the call to `clar_test(argc, argv)`, which will execute the test
+ * suite.  If you want to check the return value of the test application,
+ * your main() should return the same value returned by clar_test().
+ */
+
+int global_test_counter = 0;
+
+#ifdef _WIN32
+int __cdecl main(int argc, char *argv[])
+#else
+int main(int argc, char *argv[])
+#endif
+{
+	int ret;
+
+	/* Your custom initialization here */
+	global_test_counter = 0;
+
+	/* Run the test suite */
+	ret = clar_test(argc, argv);
+
+	/* Your custom cleanup here */
+	cl_assert_equal_i(8, global_test_counter);
+
+	return ret;
+}

new file mode 100644
index 0000000000..a4d91b72fa
@@ -0,0 +1,27 @@
+/*
+ * Copyright (c) Vicent Marti. All rights reserved.
+ *
+ * This file is part of clar, distributed under the ISC license.
+ * For full terms see the included COPYING file.
+ */
+
+#include "clar_test.h"
+
+/*
+ * Minimal main() for clar tests.
+ *
+ * Modify this with any application specific setup or teardown that you need.
+ * The only required line is the call to `clar_test(argc, argv)`, which will
+ * execute the test suite.  If you want to check the return value of the test
+ * application, main() should return the same value returned by clar_test().
+ */
+
+#ifdef _WIN32
+int __cdecl main(int argc, char *argv[])
+#else
+int main(int argc, char *argv[])
+#endif
+{
+	/* Run the test suite */
+	return clar_test(argc, argv);
+}

new file mode 100644
index 0000000000..220f4aa98a
@@ -0,0 +1 @@
+File

2.46.0.dirty

Fix whitespace errors in the clar that make git-apply(1) unhappy. This has been cherry-picked from the upstream pull request at [1].

[1]: https://github.com/clar-test/clar/pull/97

---
 t/unit-tests/clar/clar/sandbox.h  | 5 ++---
 t/unit-tests/clar/generate.py     | 1 -
 t/unit-tests/clar/test/.gitignore | 1 -
 3 files changed, 2 insertions(+), 5 deletions(-)

index 0ba1479620..7c177f3525 100644
@@ -25,9 +25,9 @@ find_tmp_path(char *buffer, size_t length)
 	static const size_t var_count = 5;
 	static const char *env_vars[] = {
 		"CLAR_TMP", "TMPDIR", "TMP", "TEMP", "USERPROFILE"
- 	};
+	};
 
- 	size_t i;
+	size_t i;
 
 	for (i = 0; i < var_count; ++i) {
 		const char *env = getenv(env_vars[i]);

@@ -151,4 +151,3 @@ const char *clar_sandbox_path(void)
 {
 	return _clar_path;
 }
-

index 931b4d60d9..80996ac3e7 100755
@@ -264,4 +264,3 @@ def write(self):
     suite.disable(options.excluded)
     if suite.write():
         print("Written `clar.suite` (%d tests in %d suites)" % (suite.callback_count(), suite.suite_count()))
-

index 3e8ae0a3a5..a477d0c40c 100644
@@ -2,4 +2,3 @@ clar.suite
 .clarcache
 clar_test
 *.o
-

2.46.0.dirty

The NonStop platform does not have mkdtemp() available, which we rely on in build_sandbox_path(). Fix this issue by using mktemp() and mkdir() instead on this platform.

This has been cherry-picked from the upstream pull request at [1].

[1]: https://github.com/clar-test/clar/pull/96

---
 t/unit-tests/clar/clar/sandbox.h | 6 ++++++
 1 file changed, 6 insertions(+)

index 7c177f3525..e25057b7c4 100644
@@ -120,6 +120,12 @@ static int build_sandbox_path(void)
 	if (_mktemp(_clar_path) == NULL)
 		return -1;
 
+	if (mkdir(_clar_path, 0700) != 0)
+		return -1;
+#elif defined(__TANDEM)
+	if (mktemp(_clar_path) == NULL)
+		return -1;
+
 	if (mkdir(_clar_path, 0700) != 0)
 		return -1;
 #elif defined(_WIN32)

2.46.0.dirty

Wire up the clar unit testing framework by introducing a new "unit-tests" executable. In contrast to the existing framework, this will result in a single executable for all test suites. The ability to pick specific tests to execute is retained via functionality built into the clar itself.

Note that we need to be a bit careful about how we need to invalidate our Makefile rules. While we obviously have to regenerate the clar suite when our test suites change, we also have to invalidate it in case any of the test suites gets removed. We do so by using our typical pattern of creating a GIT-TEST-SUITES file that gets updated whenever the set of test suites changes, so that we can easily depend on that file.

Another specialty is that we generate a "clar-decls.h" file. The test functions are neither static, nor do they have external declarations. This is because they are getting parsed via "generate.py", which then creates the external generations that get populated into an array. These declarations are only seen by the main function though.

The consequence is that we will get a bunch of "missing prototypes" errors from our compiler for each of these test functions. To fix those errors, we extract the extern declarations from "clar.suite" and put them into a standalone header that then gets included by each of our unit tests. This gets rid of compiler warnings for every function which has been extracted by "generate.py". More importantly though, it does _not_ get rid of warnings in case a function really isn't being used by anything. Thus, it would cause a compiler error if a function name was mistyped and thus not picked up by "generate.py".

---
 .gitignore                     |  1 +
 Makefile                       | 34 ++++++++++++++++++++---
 t/Makefile                     |  1 +
 t/unit-tests/.gitignore        |  2 ++
 t/unit-tests/clar-generate.awk | 50 ++++++++++++++++++++++++++++++++++
 t/unit-tests/unit-test.c       | 18 ++++++++++++
 t/unit-tests/unit-test.h       |  3 ++
 7 files changed, 105 insertions(+), 4 deletions(-)
 create mode 100644 t/unit-tests/clar-generate.awk
 create mode 100644 t/unit-tests/unit-test.c
 create mode 100644 t/unit-tests/unit-test.h

index 8caf3700c2..6687bd6db4 100644
@@ -9,6 +9,7 @@
 /GIT-PYTHON-VARS
 /GIT-SCRIPT-DEFINES
 /GIT-SPATCH-DEFINES
+/GIT-TEST-SUITES
 /GIT-USER-AGENT
 /GIT-VERSION-FILE
 /bin-wrappers/

index 674b0ac4e1..d343703215 100644
@@ -1334,6 +1334,11 @@ THIRD_PARTY_SOURCES += sha1dc/%
 THIRD_PARTY_SOURCES += $(UNIT_TEST_DIR)/clar/%
 THIRD_PARTY_SOURCES += $(UNIT_TEST_DIR)/clar/clar/%
 
+UNIT_TESTS_PROG = $(UNIT_TEST_BIN)/unit-tests$(X)
+UNIT_TESTS_OBJS = $(patsubst %,$(UNIT_TEST_DIR)/%.o,$(UNIT_TESTS_SUITES))
+UNIT_TESTS_OBJS += $(UNIT_TEST_DIR)/clar/clar.o
+UNIT_TESTS_OBJS += $(UNIT_TEST_DIR)/unit-test.o
+
 UNIT_TEST_PROGRAMS += t-ctype
 UNIT_TEST_PROGRAMS += t-example-decorate
 UNIT_TEST_PROGRAMS += t-hash

@@ -2713,6 +2718,7 @@ OBJECTS += $(XDIFF_OBJS)
 OBJECTS += $(FUZZ_OBJS)
 OBJECTS += $(REFTABLE_OBJS) $(REFTABLE_TEST_OBJS)
 OBJECTS += $(UNIT_TEST_OBJS)
+OBJECTS += $(UNIT_TESTS_OBJS)
 
 ifndef NO_CURL
 	OBJECTS += http.o http-walker.o remote-curl.o

@@ -3215,7 +3221,7 @@ endif
 
 test_bindir_programs := $(patsubst %,bin-wrappers/%,$(BINDIR_PROGRAMS_NEED_X) $(BINDIR_PROGRAMS_NO_X) $(TEST_PROGRAMS_NEED_X))
 
-all:: $(TEST_PROGRAMS) $(test_bindir_programs) $(UNIT_TEST_PROGS)
+all:: $(TEST_PROGRAMS) $(test_bindir_programs) $(UNIT_TEST_PROGS) $(UNIT_TESTS_PROG)
 
 bin-wrappers/%: wrap-for-bin.sh
 	$(call mkdir_p_parent_template)

@@ -3646,7 +3652,7 @@ endif
 
 artifacts-tar:: $(ALL_COMMANDS_TO_INSTALL) $(SCRIPT_LIB) $(OTHER_PROGRAMS) \
 		GIT-BUILD-OPTIONS $(TEST_PROGRAMS) $(test_bindir_programs) \
-		$(UNIT_TEST_PROGS) $(MOFILES)
+		$(UNIT_TEST_PROGS) $(UNIT_TESTS_PROG) $(MOFILES)
 	$(QUIET_SUBDIR0)templates $(QUIET_SUBDIR1) \
 		SHELL_PATH='$(SHELL_PATH_SQ)' PERL_PATH='$(PERL_PATH_SQ)'
 	test -n "$(ARTIFACTS_DIRECTORY)"

@@ -3702,6 +3708,7 @@ cocciclean:
 
 clean: profile-clean coverage-clean cocciclean
 	$(RM) -r .build $(UNIT_TEST_BIN)
+	$(RM) GIT-TEST-SUITES $(UNIT_TEST_DIR)/clar.suite $(UNIT_TEST_DIR)/clar-decls.h
 	$(RM) po/git.pot po/git-core.pot
 	$(RM) git.res
 	$(RM) $(OBJECTS)

@@ -3861,7 +3868,26 @@ $(UNIT_TEST_PROGS): $(UNIT_TEST_BIN)/%$X: $(UNIT_TEST_DIR)/%.o \
 	$(QUIET_LINK)$(CC) $(ALL_CFLAGS) -o $@ $(ALL_LDFLAGS) \
 		$(filter %.o,$^) $(filter %.a,$^) $(LIBS)
 
+GIT-TEST-SUITES: FORCE
+	@FLAGS='$(UNIT_TESTS_SUITES)'; \
+	    if test x"$$FLAGS" != x"`cat GIT-TEST-SUITES 2>/dev/null`" ; then \
+		echo >&2 "    * new test suites"; \
+		echo "$$FLAGS" >GIT-TEST-SUITES; \
+            fi
+
+$(UNIT_TEST_DIR)/clar-decls.h: $(patsubst %,$(UNIT_TEST_DIR)/%.c,$(UNIT_TESTS_SUITES)) GIT-TEST-SUITES
+	$(QUIET_GEN)for suite in $(UNIT_TESTS_SUITES); do \
+		sed -ne "s/^\(void test_$${suite}__[a-zA-Z_0-9][a-zA-Z_0-9]*(void)$$\)/extern \1;/p" $(UNIT_TEST_DIR)/$$suite.c; \
+	done >$@
+$(UNIT_TEST_DIR)/clar.suite: $(UNIT_TEST_DIR)/clar-decls.h
+	$(QUIET_GEN)awk -f $(UNIT_TEST_DIR)/clar-generate.awk $< >$(UNIT_TEST_DIR)/clar.suite
+$(UNIT_TESTS_OBJS): $(UNIT_TEST_DIR)/clar-decls.h
+$(UNIT_TESTS_OBJS): EXTRA_CPPFLAGS = -I$(UNIT_TEST_DIR)
+$(UNIT_TESTS_PROG): $(UNIT_TEST_DIR)/clar.suite $(UNIT_TESTS_OBJS) $(GITLIBS) GIT-LDFLAGS
+	$(call mkdir_p_parent_template)
+	$(QUIET_LINK)$(CC) $(ALL_CFLAGS) -o $@ $(ALL_LDFLAGS) $(filter %.o,$^) $(LIBS)
+
 .PHONY: build-unit-tests unit-tests
-build-unit-tests: $(UNIT_TEST_PROGS)
-unit-tests: $(UNIT_TEST_PROGS) t/helper/test-tool$X
+build-unit-tests: $(UNIT_TEST_PROGS) $(UNIT_TESTS_PROG)
+unit-tests: $(UNIT_TEST_PROGS) $(UNIT_TESTS_PROG) t/helper/test-tool$X
 	$(MAKE) -C t/ unit-tests

index d2212de0b7..131ffd778f 100644
@@ -48,6 +48,7 @@ CHAINLINTTESTS = $(sort $(patsubst chainlint/%.test,%,$(wildcard chainlint/*.tes
 CHAINLINT = '$(PERL_PATH_SQ)' chainlint.pl
 UNIT_TEST_SOURCES = $(wildcard unit-tests/t-*.c)
 UNIT_TEST_PROGRAMS = $(patsubst unit-tests/%.c,unit-tests/bin/%$(X),$(UNIT_TEST_SOURCES))
+UNIT_TEST_PROGRAMS += unit-tests/bin/unit-tests$(X)
 UNIT_TESTS = $(sort $(UNIT_TEST_PROGRAMS))
 UNIT_TESTS_NO_DIR = $(notdir $(UNIT_TESTS))
 

index 5e56e040ec..d0632ec7f9 100644
@@ -1 +1,3 @@
 /bin
+/clar.suite
+/clar-decls.h

new file mode 100644
index 0000000000..32a81299e9
@@ -0,0 +1,18 @@
+#include "unit-test.h"
+
+int cmd_main(int argc, const char **argv)
+{
+	const char **argv_copy;
+	int ret;
+
+	/* Append the "-t" flag such that the tests generate TAP output. */
+	ALLOC_ARRAY(argv_copy, argc + 2);
+	COPY_ARRAY(argv_copy, argv, argc);
+	argv_copy[argc++] = "-t";
+	argv_copy[argc] = NULL;
+
+	ret = clar_test(argc, (char **) argv_copy);
+
+	free(argv_copy);
+	return ret;
+}

new file mode 100644
index 0000000000..66ec2387cc
@@ -0,0 +1,3 @@
+#include "git-compat-util.h"
+#include "clar/clar.h"
+#include "clar-decls.h"

2.46.0.dirty

Convert the strvec tests to use the new clar unit testing framework. This is a first test balloon that demonstrates how the testing infra for clar-based tests looks like.

The tests are part of the "t/unit-tests/bin/unit-tests" binary. When running that binary, it generates TAP output:

    # ./t/unit-tests/bin/unit-tests
    TAP version 13
    # start of suite 1: strvec
    ok 1 - strvec::init
    ok 2 - strvec::dynamic_init
    ok 3 - strvec::clear
    ok 4 - strvec::push
    ok 5 - strvec::pushft_pushf
    ok 6 - strvec::pushl
    ok 7 - strvec::pushv
    ok 8 - strvec::replace_at_head
    ok 9 - strvec::replace_at_tail
    ok 10 - strvec::replace_in_between
    ok 11 - strvec::replace_with_substring
    ok 12 - strvec::remove_at_head
    ok 13 - strvec::remove_at_tail
    ok 14 - strvec::remove_in_between
    ok 15 - strvec::pop_empty_array
    ok 16 - strvec::pop_non_empty_array
    ok 17 - strvec::split_empty_string
    ok 18 - strvec::split_single_item
    ok 19 - strvec::split_multiple_items
    ok 20 - strvec::split_whitespace_only
    ok 21 - strvec::split_multiple_consecutive_whitespaces
    ok 22 - strvec::detach
    1..22

The binary also supports some parameters that allow us to run only a subset of unit tests or alter the output:

    $ ./t/unit-tests/bin/unit-tests -h
    Usage: ./t/unit-tests/bin/unit-tests [options]

Options:
-sname Run only the suite with name (can go to individual test name)
-iname Include the suite with name
-xname Exclude the suite with name
-v Increase verbosity (show suite names)
-q Only report tests that had an error
-Q Quit as soon as a test fails
-t Display results in tap format
-l Print suite names
-r[filename] Write summary file (to the optional filename)

Furthermore, running make unit-tests runs the binary along with all the other unit tests we have.

---
 Makefile                              |   2 +-
 t/unit-tests/{t-strvec.c => strvec.c} | 119 ++++++++++----------------
 t/unit-tests/unit-test.c              |   3 +-
 3 files changed, 45 insertions(+), 79 deletions(-)
 rename t/unit-tests/{t-strvec.c => strvec.c} (54%)

index d343703215..ec90b8ac35 100644
@@ -1334,6 +1334,7 @@ THIRD_PARTY_SOURCES += sha1dc/%
 THIRD_PARTY_SOURCES += $(UNIT_TEST_DIR)/clar/%
 THIRD_PARTY_SOURCES += $(UNIT_TEST_DIR)/clar/clar/%
 
+UNIT_TESTS_SUITES += strvec
 UNIT_TESTS_PROG = $(UNIT_TEST_BIN)/unit-tests$(X)
 UNIT_TESTS_OBJS = $(patsubst %,$(UNIT_TEST_DIR)/%.o,$(UNIT_TESTS_SUITES))
 UNIT_TESTS_OBJS += $(UNIT_TEST_DIR)/clar/clar.o

@@ -1351,7 +1352,6 @@ UNIT_TEST_PROGRAMS += t-reftable-merged
 UNIT_TEST_PROGRAMS += t-reftable-record
 UNIT_TEST_PROGRAMS += t-strbuf
 UNIT_TEST_PROGRAMS += t-strcmp-offset
-UNIT_TEST_PROGRAMS += t-strvec
 UNIT_TEST_PROGRAMS += t-trailer
 UNIT_TEST_PROGS = $(patsubst %,$(UNIT_TEST_BIN)/%$X,$(UNIT_TEST_PROGRAMS))
 UNIT_TEST_OBJS = $(patsubst %,$(UNIT_TEST_DIR)/%.o,$(UNIT_TEST_PROGRAMS))

index 32a81299e9..82b7635e6a 100644
@@ -6,10 +6,9 @@ int cmd_main(int argc, const char **argv)
 	int ret;
 
 	/* Append the "-t" flag such that the tests generate TAP output. */
-	ALLOC_ARRAY(argv_copy, argc + 2);
+	ALLOC_ARRAY(argv_copy, argc + 1);
 	COPY_ARRAY(argv_copy, argv, argc);
 	argv_copy[argc++] = "-t";
-	argv_copy[argc] = NULL;
 
 	ret = clar_test(argc, (char **) argv_copy);
 

2.46.0.dirty

Convert the ctype tests to use the new clar unit testing framework.

---
 Makefile                            |  2 +-
 t/unit-tests/{t-ctype.c => ctype.c} | 71 +++++++++++++++++++++++------
 2 files changed, 59 insertions(+), 14 deletions(-)
 rename t/unit-tests/{t-ctype.c => ctype.c} (71%)

index ec90b8ac35..fa7c540bc0 100644
@@ -1334,13 +1334,13 @@ THIRD_PARTY_SOURCES += sha1dc/%
 THIRD_PARTY_SOURCES += $(UNIT_TEST_DIR)/clar/%
 THIRD_PARTY_SOURCES += $(UNIT_TEST_DIR)/clar/clar/%
 
+UNIT_TESTS_SUITES += ctype
 UNIT_TESTS_SUITES += strvec
 UNIT_TESTS_PROG = $(UNIT_TEST_BIN)/unit-tests$(X)
 UNIT_TESTS_OBJS = $(patsubst %,$(UNIT_TEST_DIR)/%.o,$(UNIT_TESTS_SUITES))
 UNIT_TESTS_OBJS += $(UNIT_TEST_DIR)/clar/clar.o
 UNIT_TESTS_OBJS += $(UNIT_TEST_DIR)/unit-test.o
 
-UNIT_TEST_PROGRAMS += t-ctype
 UNIT_TEST_PROGRAMS += t-example-decorate
 UNIT_TEST_PROGRAMS += t-hash
 UNIT_TEST_PROGRAMS += t-mem-pool

2.46.0.dirty

I'm generally in favor of this change, but I'm still unsure what our plan is for importing this from upstream clar. Are we going to vendor our own copy here and (hopefully) someone will pay attention to upstream fixes and apply them to our copy? Or will we replace this with a submodule?

is for

and

our

I think we have our own copy of clar, customized to integrate with the existing make
infrastructure.

As long as we do not have to make any changes to the "vendored" code ourselves, that would not matter. We will not randomly update the gitlink that specifies "we want to use _this_ version and not other version of upstream clar" without good reasons if you are using it as a submodule, and we would need to justify why we are updating the hierarchy if we import the hierarchy as vendored source. So the hassle of "updating from upstream" is pretty much the same.

For something as small as "clar", I think it is fine to start with the currently proposed layout and see what happens. If we can keep going without touching the imported part of the sources at all, and the system proves to be useful and stable, that is a good time to suggest moving it out and binding the selected version of the upstream as a submodule.

Thanks.

I think we already have a copy customized for git's use. The main clar repo on its own has portability issues. I have contributed a few fixes, but they need work.

Yup, but as long as the changes we make are all upstreamable, the story does not change all that much. Changes like "#ifdef TANDEM" would be totally uncontroversial thing for them to accept and we should be able to upstream them fairly easily, and once we thin our local customization down to zero, we'd reach the state I outlined.

Starting out with a local copy helps us making these portability and other changes without much friction, regardless of how responsive the upstream is, and the request upstream would see is "here are the changes to make it available on more platforms and/or making it generally more useful. all of these changes have been used and battle-tested in the context of the Git project for N months, please apply."

Yeah, agreed. My intention certainly is to upstream all required changes. Right now, it's only two minor fixes that we need, one for NonStop and one for whitespace fixes.

While I'm still one of the maintainers of the clar, I first want to connect with Ed before merging any of these fixes. I would otherwise consider it a bit overreaching to just go in and merge things after having been absent from that project for such a long time. But when things are sorted out I think the upstreaming process should be easy enough.

Meanwhile, I think keeping this in the tree is fine. We can then optionally convert the code to a submodule once the necessary changes are upstream.

Patrick

A funny thing about this step is that the copy I carried for the past few days was an empty patch, because out of habit I use the "--whitespace=fix" option while running "git am". As that fixes the whitespace breakage this step addresses while applying the previous step, this one becomes empty.

I'll requeue these patches with --whitespace=warn instead.

Thanks.

Please forgive the third ask, but I really want to test the build early. Suggestions?

Sorry, but I do not follow. If you have "git am" and "git clone" working, and if you have enough C toolchain to build the rest of Git, wouldn't it be sufficient to also build this addition? I think the 7 patches cleanly apply to any recent tip of 'master'.

I am confused. My bad. The unit test works fine with the custom clar off seen and next.

Oops, it turns out that this is insufficient. I'll add the following as a separate patch immediately after this step.

Thanks.

 t/unit-tests/clar/clar/fs.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

index a6eda5e5dc..3e39890bd3 100644
@@ -146,7 +146,7 @@ fs_rm_wait(WCHAR *_wpath)
 			ERROR_PATH_NOT_FOUND == last_error)
 			return 0;
 
-		Sleep(RM_RETRY_DELAY * retries * retries);	
+		Sleep(RM_RETRY_DELAY * retries * retries);
 	}
 	while (retries++ <= RM_RETRY_COUNT);
 

Hi,

this is the fourth version of my patch series that introduces the clar testing framework for our unit tests.

Changes compared to v3:

- Seeing that the overall feedback seems to be rather positive, I've now dropped the "RFC" prefix.

- Fix another whitespace-damaged line in the clar unit test. I've adapted the upstream pull request accordingly.

- Fix make hdr-check, which failed due to the missing dependency on

    "clar-decls.h".

Thanks!

Patrick

Patrick Steinhardt (7):
t: do not pass GIT_TEST_OPTS to unit tests with prove
t: import the clar unit testing framework t/clar: fix whitespace errors
t/clar: fix compatibility with NonStop
Makefile: wire up the clar unit testing framework t/unit-tests: convert strvec tests to use clar t/unit-tests: convert ctype tests to use clar

 .gitignore                                 |   1 +
 Documentation/technical/unit-tests.txt     |   2 +
 Makefile                                   |  43 +-
 t/Makefile                                 |   4 +-
 t/run-test.sh                              |   2 +-
 t/unit-tests/.gitignore                    |   2 +
 t/unit-tests/clar-generate.awk             |  50 ++
 t/unit-tests/clar/.github/workflows/ci.yml |  23 +
 t/unit-tests/clar/COPYING                  |  15 +
 t/unit-tests/clar/README.md                | 329 ++++++++
 t/unit-tests/clar/clar.c                   | 842 +++++++++++++++++++++
 t/unit-tests/clar/clar.h                   | 173 +++++
 t/unit-tests/clar/clar/fixtures.h          |  50 ++
 t/unit-tests/clar/clar/fs.h                | 522 +++++++++++++
 t/unit-tests/clar/clar/print.h             | 211 ++++++
 t/unit-tests/clar/clar/sandbox.h           | 159 ++++
 t/unit-tests/clar/clar/summary.h           | 143 ++++
 t/unit-tests/clar/generate.py              | 266 +++++++
 t/unit-tests/clar/test/.gitignore          |   4 +
 t/unit-tests/clar/test/Makefile            |  39 +
 t/unit-tests/clar/test/clar_test.h         |  16 +
 t/unit-tests/clar/test/main.c              |  40 +
 t/unit-tests/clar/test/main.c.sample       |  27 +
 t/unit-tests/clar/test/resources/test/file |   1 +
 t/unit-tests/clar/test/sample.c            |  84 ++
 t/unit-tests/{t-ctype.c => ctype.c}        |  71 +-
 t/unit-tests/{t-strvec.c => strvec.c}      | 119 ++-
 t/unit-tests/unit-test.c                   |  17 +
 t/unit-tests/unit-test.h                   |   3 +
 29 files changed, 3160 insertions(+), 98 deletions(-)
 create mode 100644 t/unit-tests/clar-generate.awk
 create mode 100644 t/unit-tests/clar/.github/workflows/ci.yml
 create mode 100644 t/unit-tests/clar/COPYING
 create mode 100644 t/unit-tests/clar/README.md
 create mode 100644 t/unit-tests/clar/clar.c
 create mode 100644 t/unit-tests/clar/clar.h
 create mode 100644 t/unit-tests/clar/clar/fixtures.h
 create mode 100644 t/unit-tests/clar/clar/fs.h
 create mode 100644 t/unit-tests/clar/clar/print.h
 create mode 100644 t/unit-tests/clar/clar/sandbox.h
 create mode 100644 t/unit-tests/clar/clar/summary.h
 create mode 100755 t/unit-tests/clar/generate.py
 create mode 100644 t/unit-tests/clar/test/.gitignore
 create mode 100644 t/unit-tests/clar/test/Makefile
 create mode 100644 t/unit-tests/clar/test/clar_test.h
 create mode 100644 t/unit-tests/clar/test/main.c
 create mode 100644 t/unit-tests/clar/test/main.c.sample
 create mode 100644 t/unit-tests/clar/test/resources/test/file
 create mode 100644 t/unit-tests/clar/test/sample.c
 rename t/unit-tests/{t-ctype.c => ctype.c} (71%)
 rename t/unit-tests/{t-strvec.c => strvec.c} (54%)
 create mode 100644 t/unit-tests/unit-test.c
 create mode 100644 t/unit-tests/unit-test.h

Range-diff against v3:

1:  78a9cc1162 = 1:  086dd728a7 t: do not pass GIT_TEST_OPTS to unit tests with prove
2:  b6c066ee4e = 2:  5c22e0b3b9 t: import the clar unit testing framework
3:  35682b7686 ! 3:  e0f99874cc t/clar: fix whitespace errors

@@ Commit message

+ ## t/unit-tests/clar/clar/fs.h ##
+@@ t/unit-tests/clar/clar/fs.h: fs_rm_wait(WCHAR *_wpath)
+ ERROR_PATH_NOT_FOUND == last_error)
+ return 0;
+
+- Sleep(RM_RETRY_DELAY * retries * retries);
++ Sleep(RM_RETRY_DELAY * retries * retries);
+ }
+ while (retries++ <= RM_RETRY_COUNT);
+
+
## t/unit-tests/clar/clar/sandbox.h ##
@@ t/unit-tests/clar/clar/sandbox.h: find_tmp_path(char *buffer, size_t length)
static const size_t var_count = 5;

4:  7a76c21bcb = 4:  75e097dfa4 t/clar: fix compatibility with NonStop
5:  68b3c65951 ! 5:  5b8a64ae79 Makefile: wire up the clar unit testing framework

@@ Makefile: endif

bin-wrappers/%: wrap-for-bin.sh
$(call mkdir_p_parent_template)
+@@ Makefile: CHK_HDRS = $(filter-out $(EXCEPT_HDRS),$(LIB_H))
+ HCO = $(patsubst %.h,%.hco,$(CHK_HDRS))
+ HCC = $(HCO:hco=hcc)
+
++$(UNIT_TEST_DIR)/unit-test.hcc: $(UNIT_TEST_DIR)/unit-test.h $(UNIT_TEST_DIR)/clar-decls.h
+ %.hcc: %.h
+ @echo '#include "git-compat-util.h"' >$@
+ @echo '#include "$<"' >>$@
@@ Makefile: endif

artifacts-tar:: $(ALL_COMMANDS_TO_INSTALL) $(SCRIPT_LIB) $(OTHER_PROGRAMS) \

6:  4a0888380e = 6:  bc4e23d666 t/unit-tests: convert strvec tests to use clar
7:  f423b01c05 = 7:  0a7fe8775a t/unit-tests: convert ctype tests to use clar

2.46.0.46.g406f326d27.dirty

Import the clar unit testing framework at commit faa8419 (Merge pull request #93 from clar-test/ethomson/fixtures, 2023-12-14). The framework will be wired up in subsequent commits.

---
 Documentation/technical/unit-tests.txt     |   2 +
 Makefile                                   |   4 +-
 t/unit-tests/clar/.github/workflows/ci.yml |  23 +
 t/unit-tests/clar/COPYING                  |  15 +
 t/unit-tests/clar/README.md                | 329 ++++++++
 t/unit-tests/clar/clar.c                   | 842 +++++++++++++++++++++
 t/unit-tests/clar/clar.h                   | 173 +++++
 t/unit-tests/clar/clar/fixtures.h          |  50 ++
 t/unit-tests/clar/clar/fs.h                | 522 +++++++++++++
 t/unit-tests/clar/clar/print.h             | 211 ++++++
 t/unit-tests/clar/clar/sandbox.h           | 154 ++++
 t/unit-tests/clar/clar/summary.h           | 143 ++++
 t/unit-tests/clar/generate.py              | 267 +++++++
 t/unit-tests/clar/test/.gitignore          |   5 +
 t/unit-tests/clar/test/Makefile            |  39 +
 t/unit-tests/clar/test/clar_test.h         |  16 +
 t/unit-tests/clar/test/main.c              |  40 +
 t/unit-tests/clar/test/main.c.sample       |  27 +
 t/unit-tests/clar/test/resources/test/file |   1 +
 t/unit-tests/clar/test/sample.c            |  84 ++
 20 files changed, 2946 insertions(+), 1 deletion(-)
 create mode 100644 t/unit-tests/clar/.github/workflows/ci.yml
 create mode 100644 t/unit-tests/clar/COPYING
 create mode 100644 t/unit-tests/clar/README.md
 create mode 100644 t/unit-tests/clar/clar.c
 create mode 100644 t/unit-tests/clar/clar.h
 create mode 100644 t/unit-tests/clar/clar/fixtures.h
 create mode 100644 t/unit-tests/clar/clar/fs.h
 create mode 100644 t/unit-tests/clar/clar/print.h
 create mode 100644 t/unit-tests/clar/clar/sandbox.h
 create mode 100644 t/unit-tests/clar/clar/summary.h
 create mode 100755 t/unit-tests/clar/generate.py
 create mode 100644 t/unit-tests/clar/test/.gitignore
 create mode 100644 t/unit-tests/clar/test/Makefile
 create mode 100644 t/unit-tests/clar/test/clar_test.h
 create mode 100644 t/unit-tests/clar/test/main.c
 create mode 100644 t/unit-tests/clar/test/main.c.sample
 create mode 100644 t/unit-tests/clar/test/resources/test/file
 create mode 100644 t/unit-tests/clar/test/sample.c

index 206037ffb1..5a432b7b29 100644
@@ -203,6 +203,7 @@ GitHub / GitLab stars to estimate this.
 :criterion: https://github.com/Snaipe/Criterion[Criterion]
 :c-tap: https://github.com/rra/c-tap-harness/[C TAP]
 :check: https://libcheck.github.io/check/[Check]
+:clar: https://github.com/clar-test/clar[Clar]
 
 [format="csv",options="header",width="33%",subs="specialcharacters,attributes,quotes,macros"]
 |=====

@@ -212,6 +213,7 @@ Framework,"<<license,License>>","<<vendorable-or-ubiquitous,Vendorable or ubiqui
 {criterion},{mit},{false},{partial},{true},{true},{true},{true},{true},{false},{true},19,1800
 {c-tap},{expat},{true},{partial},{partial},{true},{false},{true},{false},{false},{false},4,33
 {check},{lgpl},{false},{partial},{true},{true},{true},{false},{false},{false},{true},17,973
+{clar},{isc},{false},{partial},{true},{true},{true},{true},{false},{false},{true},1,192
 |=====
 
 === Additional framework candidates

index 3863e60b66..674b0ac4e1 100644
@@ -1331,6 +1331,8 @@ THIRD_PARTY_SOURCES += compat/poll/%
 THIRD_PARTY_SOURCES += compat/regex/%
 THIRD_PARTY_SOURCES += sha1collisiondetection/%
 THIRD_PARTY_SOURCES += sha1dc/%
+THIRD_PARTY_SOURCES += $(UNIT_TEST_DIR)/clar/%
+THIRD_PARTY_SOURCES += $(UNIT_TEST_DIR)/clar/clar/%
 
 UNIT_TEST_PROGRAMS += t-ctype
 UNIT_TEST_PROGRAMS += t-example-decorate

@@ -3260,7 +3262,7 @@ $(SP_OBJ): %.sp: %.c %.o
 .PHONY: sparse
 sparse: $(SP_OBJ)
 
-EXCEPT_HDRS := $(GENERATED_H) unicode-width.h compat/% xdiff/%
+EXCEPT_HDRS := $(GENERATED_H) unicode-width.h compat/% xdiff/% $(UNIT_TEST_DIR)/clar/% $(UNIT_TEST_DIR)/clar/clar/%
 ifndef OPENSSL_SHA1
 	EXCEPT_HDRS += sha1/openssl.h
 endif

new file mode 100644
index 0000000000..b1ac2de460
@@ -0,0 +1,23 @@
+name: CI
+
+on:
+  push:
+    branches: [ main ]
+  pull_request:
+    branches: [ main ]
+
+jobs:
+  build:
+    strategy:
+      matrix:
+        os: [ ubuntu-latest, macos-latest ]
+
+    runs-on: ${{ matrix.os }}
+
+    steps:
+    - name: Check out
+      uses: actions/checkout@v2
+    - name: Build
+      run: |
+        cd test
+        make

new file mode 100644
index 0000000000..8983817f0c
@@ -0,0 +1,15 @@
+ISC License
+
+Copyright (c) 2011-2015 Vicent Marti
+
+Permission to use, copy, modify, and/or distribute this software for any
+purpose with or without fee is hereby granted, provided that the above
+copyright notice and this permission notice appear in all copies.
+
+THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
+WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
+MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
+ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
+WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
+ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
+OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.

new file mode 100644
index 0000000000..3e8ae0a3a5
@@ -0,0 +1,5 @@
+clar.suite
+.clarcache
+clar_test
+*.o
+

new file mode 100644
index 0000000000..93c6b2ad32
@@ -0,0 +1,39 @@
+#
+# Copyright (c) Vicent Marti. All rights reserved.
+#
+# This file is part of clar, distributed under the ISC license.
+# For full terms see the included COPYING file.
+#
+
+#
+# Set up the path to the clar sources and to the fixtures directory
+#
+# The fixture path needs to be an absolute path so it can be used
+# even after we have chdir'ed into the test directory while testing.
+#
+CURRENT_MAKEFILE  := $(word $(words $(MAKEFILE_LIST)),$(MAKEFILE_LIST))
+TEST_DIRECTORY    := $(abspath $(dir $(CURRENT_MAKEFILE)))
+CLAR_PATH         := $(dir $(TEST_DIRECTORY))
+CLAR_FIXTURE_PATH := $(TEST_DIRECTORY)/resources/
+
+CFLAGS=-g -I.. -I. -Wall -DCLAR_FIXTURE_PATH=\"$(CLAR_FIXTURE_PATH)\"
+
+.PHONY: clean
+
+# list the objects that go into our test
+objects = main.o sample.o
+
+# build the test executable itself
+clar_test: $(objects) clar_test.h clar.suite $(CLAR_PATH)clar.c
+	$(CC) $(CFLAGS) -o $@ "$(CLAR_PATH)clar.c" $(objects)
+
+# test object files depend on clar macros
+$(objects) : $(CLAR_PATH)clar.h
+
+# build the clar.suite file of test metadata
+clar.suite:
+	python "$(CLAR_PATH)generate.py" .
+
+# remove all generated files
+clean:
+	$(RM) -rf *.o clar.suite .clarcache clar_test clar_test.dSYM

new file mode 100644
index 0000000000..0fcaa639aa
@@ -0,0 +1,16 @@
+/*
+ * Copyright (c) Vicent Marti. All rights reserved.
+ *
+ * This file is part of clar, distributed under the ISC license.
+ * For full terms see the included COPYING file.
+ */
+#ifndef __CLAR_TEST__
+#define __CLAR_TEST__
+
+/* Import the standard clar helper functions */
+#include "clar.h"
+
+/* Your custom shared includes / defines here */
+extern int global_test_counter;
+
+#endif

new file mode 100644
index 0000000000..59e56ad255
@@ -0,0 +1,40 @@
+/*
+ * Copyright (c) Vicent Marti. All rights reserved.
+ *
+ * This file is part of clar, distributed under the ISC license.
+ * For full terms see the included COPYING file.
+ */
+
+#include "clar_test.h"
+
+/*
+ * Sample main() for clar tests.
+ *
+ * You should write your own main routine for clar tests that does specific
+ * setup and teardown as necessary for your application.  The only required
+ * line is the call to `clar_test(argc, argv)`, which will execute the test
+ * suite.  If you want to check the return value of the test application,
+ * your main() should return the same value returned by clar_test().
+ */
+
+int global_test_counter = 0;
+
+#ifdef _WIN32
+int __cdecl main(int argc, char *argv[])
+#else
+int main(int argc, char *argv[])
+#endif
+{
+	int ret;
+
+	/* Your custom initialization here */
+	global_test_counter = 0;
+
+	/* Run the test suite */
+	ret = clar_test(argc, argv);
+
+	/* Your custom cleanup here */
+	cl_assert_equal_i(8, global_test_counter);
+
+	return ret;
+}

new file mode 100644
index 0000000000..a4d91b72fa
@@ -0,0 +1,27 @@
+/*
+ * Copyright (c) Vicent Marti. All rights reserved.
+ *
+ * This file is part of clar, distributed under the ISC license.
+ * For full terms see the included COPYING file.
+ */
+
+#include "clar_test.h"
+
+/*
+ * Minimal main() for clar tests.
+ *
+ * Modify this with any application specific setup or teardown that you need.
+ * The only required line is the call to `clar_test(argc, argv)`, which will
+ * execute the test suite.  If you want to check the return value of the test
+ * application, main() should return the same value returned by clar_test().
+ */
+
+#ifdef _WIN32
+int __cdecl main(int argc, char *argv[])
+#else
+int main(int argc, char *argv[])
+#endif
+{
+	/* Run the test suite */
+	return clar_test(argc, argv);
+}

new file mode 100644
index 0000000000..220f4aa98a
@@ -0,0 +1 @@
+File

2.46.0.46.g406f326d27.dirty

The NonStop platform does not have mkdtemp() available, which we rely on in build_sandbox_path(). Fix this issue by using mktemp() and mkdir() instead on this platform.

This has been cherry-picked from the upstream pull request at [1].

[1]: https://github.com/clar-test/clar/pull/96

---
 t/unit-tests/clar/clar/sandbox.h | 6 ++++++
 1 file changed, 6 insertions(+)

index 7c177f3525..e25057b7c4 100644
@@ -120,6 +120,12 @@ static int build_sandbox_path(void)
 	if (_mktemp(_clar_path) == NULL)
 		return -1;
 
+	if (mkdir(_clar_path, 0700) != 0)
+		return -1;
+#elif defined(__TANDEM)
+	if (mktemp(_clar_path) == NULL)
+		return -1;
+
 	if (mkdir(_clar_path, 0700) != 0)
 		return -1;
 #elif defined(_WIN32)

2.46.0.46.g406f326d27.dirty

Convert the strvec tests to use the new clar unit testing framework. This is a first test balloon that demonstrates how the testing infra for clar-based tests looks like.

The tests are part of the "t/unit-tests/bin/unit-tests" binary. When running that binary, it generates TAP output:

    # ./t/unit-tests/bin/unit-tests
    TAP version 13
    # start of suite 1: strvec
    ok 1 - strvec::init
    ok 2 - strvec::dynamic_init
    ok 3 - strvec::clear
    ok 4 - strvec::push
    ok 5 - strvec::pushft_pushf
    ok 6 - strvec::pushl
    ok 7 - strvec::pushv
    ok 8 - strvec::replace_at_head
    ok 9 - strvec::replace_at_tail
    ok 10 - strvec::replace_in_between
    ok 11 - strvec::replace_with_substring
    ok 12 - strvec::remove_at_head
    ok 13 - strvec::remove_at_tail
    ok 14 - strvec::remove_in_between
    ok 15 - strvec::pop_empty_array
    ok 16 - strvec::pop_non_empty_array
    ok 17 - strvec::split_empty_string
    ok 18 - strvec::split_single_item
    ok 19 - strvec::split_multiple_items
    ok 20 - strvec::split_whitespace_only
    ok 21 - strvec::split_multiple_consecutive_whitespaces
    ok 22 - strvec::detach
    1..22

The binary also supports some parameters that allow us to run only a subset of unit tests or alter the output:

    $ ./t/unit-tests/bin/unit-tests -h
    Usage: ./t/unit-tests/bin/unit-tests [options]

Options:
-sname Run only the suite with name (can go to individual test name)
-iname Include the suite with name
-xname Exclude the suite with name
-v Increase verbosity (show suite names)
-q Only report tests that had an error
-Q Quit as soon as a test fails
-t Display results in tap format
-l Print suite names
-r[filename] Write summary file (to the optional filename)

Furthermore, running make unit-tests runs the binary along with all the other unit tests we have.

---
 Makefile                              |   2 +-
 t/unit-tests/{t-strvec.c => strvec.c} | 119 ++++++++++----------------
 t/unit-tests/unit-test.c              |   3 +-
 3 files changed, 45 insertions(+), 79 deletions(-)
 rename t/unit-tests/{t-strvec.c => strvec.c} (54%)

index 1311bbbbed..ebc12b2bf1 100644
@@ -1334,6 +1334,7 @@ THIRD_PARTY_SOURCES += sha1dc/%
 THIRD_PARTY_SOURCES += $(UNIT_TEST_DIR)/clar/%
 THIRD_PARTY_SOURCES += $(UNIT_TEST_DIR)/clar/clar/%
 
+UNIT_TESTS_SUITES += strvec
 UNIT_TESTS_PROG = $(UNIT_TEST_BIN)/unit-tests$(X)
 UNIT_TESTS_OBJS = $(patsubst %,$(UNIT_TEST_DIR)/%.o,$(UNIT_TESTS_SUITES))
 UNIT_TESTS_OBJS += $(UNIT_TEST_DIR)/clar/clar.o

@@ -1351,7 +1352,6 @@ UNIT_TEST_PROGRAMS += t-reftable-merged
 UNIT_TEST_PROGRAMS += t-reftable-record
 UNIT_TEST_PROGRAMS += t-strbuf
 UNIT_TEST_PROGRAMS += t-strcmp-offset
-UNIT_TEST_PROGRAMS += t-strvec
 UNIT_TEST_PROGRAMS += t-trailer
 UNIT_TEST_PROGS = $(patsubst %,$(UNIT_TEST_BIN)/%$X,$(UNIT_TEST_PROGRAMS))
 UNIT_TEST_OBJS = $(patsubst %,$(UNIT_TEST_DIR)/%.o,$(UNIT_TEST_PROGRAMS))

index 32a81299e9..82b7635e6a 100644
@@ -6,10 +6,9 @@ int cmd_main(int argc, const char **argv)
 	int ret;
 
 	/* Append the "-t" flag such that the tests generate TAP output. */
-	ALLOC_ARRAY(argv_copy, argc + 2);
+	ALLOC_ARRAY(argv_copy, argc + 1);
 	COPY_ARRAY(argv_copy, argv, argc);
 	argv_copy[argc++] = "-t";
-	argv_copy[argc] = NULL;
 
 	ret = clar_test(argc, (char **) argv_copy);
 

2.46.0.46.g406f326d27.dirty

Convert the ctype tests to use the new clar unit testing framework.

---
 Makefile                            |  2 +-
 t/unit-tests/{t-ctype.c => ctype.c} | 71 +++++++++++++++++++++++------
 2 files changed, 59 insertions(+), 14 deletions(-)
 rename t/unit-tests/{t-ctype.c => ctype.c} (71%)

index ebc12b2bf1..d28faabbf4 100644
@@ -1334,13 +1334,13 @@ THIRD_PARTY_SOURCES += sha1dc/%
 THIRD_PARTY_SOURCES += $(UNIT_TEST_DIR)/clar/%
 THIRD_PARTY_SOURCES += $(UNIT_TEST_DIR)/clar/clar/%
 
+UNIT_TESTS_SUITES += ctype
 UNIT_TESTS_SUITES += strvec
 UNIT_TESTS_PROG = $(UNIT_TEST_BIN)/unit-tests$(X)
 UNIT_TESTS_OBJS = $(patsubst %,$(UNIT_TEST_DIR)/%.o,$(UNIT_TESTS_SUITES))
 UNIT_TESTS_OBJS += $(UNIT_TEST_DIR)/clar/clar.o
 UNIT_TESTS_OBJS += $(UNIT_TEST_DIR)/unit-test.o
 
-UNIT_TEST_PROGRAMS += t-ctype
 UNIT_TEST_PROGRAMS += t-example-decorate
 UNIT_TEST_PROGRAMS += t-hash
 UNIT_TEST_PROGRAMS += t-mem-pool

2.46.0.46.g406f326d27.dirty

When using the prove target, we append GIT_TEST_OPTS to the arguments that we execute each of the tests with. This doesn't only include the intended test scripts, but also ends up passing the arguments to our unit tests. This is unintentional though as they do not even know to interpret those arguments, and is inconsistent with how we execute unit tests without prove.

This isn't much of an issue because our current set of unit tests mostly ignore their arguments anyway. With the introduction of clar-based unit tests this is about to become an issue though, as these do parse their command line argument to alter behaviour.

Prepare for this by passing GIT_TEST_OPTS to "run-test.sh" via an environment variable. Like this, we can conditionally forward it to our test scripts, only.

---
 t/Makefile    | 3 ++-
 t/run-test.sh | 2 +-
 2 files changed, 3 insertions(+), 2 deletions(-)

index 4c30e7c06f..d2212de0b7 100644
@@ -68,7 +68,8 @@ failed:
 	test -z "$$failed" || $(MAKE) $$failed
 
 prove: pre-clean check-chainlint $(TEST_LINT)
-	@echo "*** prove (shell & unit tests) ***"; $(CHAINLINTSUPPRESS) TEST_SHELL_PATH='$(TEST_SHELL_PATH_SQ)' $(PROVE) --exec ./run-test.sh $(GIT_PROVE_OPTS) $(T) $(UNIT_TESTS) :: $(GIT_TEST_OPTS)
+	@echo "*** prove (shell & unit tests) ***"
+	@$(CHAINLINTSUPPRESS) TEST_OPTIONS='$(GIT_TEST_OPTS)' TEST_SHELL_PATH='$(TEST_SHELL_PATH_SQ)' $(PROVE) --exec ./run-test.sh $(GIT_PROVE_OPTS) $(T) $(UNIT_TESTS)
 	$(MAKE) clean-except-prove-cache
 
 $(T):

index 13c353b91b..63328ac630 100755
@@ -10,7 +10,7 @@ case "$1" in
 		echo >&2 "ERROR: TEST_SHELL_PATH is empty or not set"
 		exit 1
 	fi
-	exec "${TEST_SHELL_PATH}" "$@"
+	exec "${TEST_SHELL_PATH}" "$@" ${TEST_OPTIONS}
 	;;
 *)
 	exec "$@"

2.46.0.46.g406f326d27.dirty

Fix whitespace errors in the clar that make git-apply(1) unhappy. This has been cherry-picked from the upstream pull request at [1].

[1]: https://github.com/clar-test/clar/pull/97

---
 t/unit-tests/clar/clar/fs.h       | 2 +-
 t/unit-tests/clar/clar/sandbox.h  | 5 ++---
 t/unit-tests/clar/generate.py     | 1 -
 t/unit-tests/clar/test/.gitignore | 1 -
 4 files changed, 3 insertions(+), 6 deletions(-)

index a6eda5e5dc..3e39890bd3 100644
@@ -146,7 +146,7 @@ fs_rm_wait(WCHAR *_wpath)
 			ERROR_PATH_NOT_FOUND == last_error)
 			return 0;
 
-		Sleep(RM_RETRY_DELAY * retries * retries);	
+		Sleep(RM_RETRY_DELAY * retries * retries);
 	}
 	while (retries++ <= RM_RETRY_COUNT);
 

index 0ba1479620..7c177f3525 100644
@@ -25,9 +25,9 @@ find_tmp_path(char *buffer, size_t length)
 	static const size_t var_count = 5;
 	static const char *env_vars[] = {
 		"CLAR_TMP", "TMPDIR", "TMP", "TEMP", "USERPROFILE"
- 	};
+	};
 
- 	size_t i;
+	size_t i;
 
 	for (i = 0; i < var_count; ++i) {
 		const char *env = getenv(env_vars[i]);