refs: introduce reftable backend

Hi,

this is the fourth version of my patch series that introduces the reftable backend.

Changes compared to v3:

- kn/for-all-refs has been reverted due to discussions around its user interface. I have thus evicted the patch series as a dependency and dropped handling of DO_FOR_EACH_INCLUDE_ALL_REFS.

- ps/reftable-compacted-tables-permission-fix has been merged to "master" now, fixing permissions on "tables.list" when compacting reftables. I've marked the corresponding test as succeeding.

- jc/reftable-core-fsync has been merged to "master" now, which adds fsyncing logic to reftable. I've marked the corresponding test as succeeding.

- I noticed that the fsync tests fail on macOS because there we use a different fsync method by default. I fixed that by explicitly saying which fsync method should be used in the corresponding tests.

- I also noticed that the second fsync test reused "trace2.txt" from the first fsync test because it only appends to the file. Thus, we saw two fsync events instead of one. I fixed that by truncating the file.

The patch series is based on the current "master" branch at 235986be82 (The fourteenth batch, 2024-02-06).

Thanks!

Patrick

Patrick Steinhardt (2):
refs: introduce reftable backend
ci: add jobs to test with the reftable backend

 .github/workflows/main.yml                    |    9 +
 .gitlab-ci.yml                                |    9 +
 Documentation/ref-storage-format.txt          |    2 +
 .../technical/repository-version.txt          |    5 +-
 Makefile                                      |    1 +
 ci/lib.sh                                     |    2 +-
 ci/run-build-and-tests.sh                     |    3 +
 contrib/workdir/git-new-workdir               |    2 +-
 path.c                                        |    2 +-
 path.h                                        |    1 +
 refs.c                                        |    1 +
 refs/refs-internal.h                          |    1 +
 refs/reftable-backend.c                       | 2297 +++++++++++++++++
 repository.h                                  |    5 +-
 t/t0610-reftable-basics.sh                    |  887 +++++++
 t/t0611-reftable-httpd.sh                     |   26 +
 t/test-lib.sh                                 |    2 +
 17 files changed, 3248 insertions(+), 7 deletions(-)
 create mode 100644 refs/reftable-backend.c
 create mode 100755 t/t0610-reftable-basics.sh
 create mode 100755 t/t0611-reftable-httpd.sh

Range-diff against v3:
1: d83e66e980 ! 1: 5de60d46bd refs: introduce reftable backend

    @@ refs/reftable-backend.c (new)
     +			break;
     +
     +		/*
    -+		 * Unless DO_FOR_EACH_INCLUDE_ALL_REFS is set, we only list
    -+		 * refs starting with "refs/" to mimic the "files" backend.
    ++		 * The files backend only lists references contained in
    ++		 * "refs/". We emulate the same behaviour here and thus skip
    ++		 * all references that don't start with this prefix.
     +		 */
    -+		if (!(iter->flags & DO_FOR_EACH_INCLUDE_ALL_REFS) &&
    -+		    !starts_with(iter->ref.refname, "refs/"))
    ++		if (!starts_with(iter->ref.refname, "refs/"))
     +			continue;
     +
     +		if (iter->prefix &&
    @@ t/t0610-reftable-basics.sh (new)
     +	test_cmp expect actual
     +}
     +
    -+# A fix for this is in flight via jc/reftable-core-fsync.
    -+test_expect_failure 'ref transaction: writes are synced' '
    ++test_expect_success 'ref transaction: writes are synced' '
     +	test_when_finished "rm -rf repo" &&
     +	git init repo &&
     +	test_commit -C repo initial &&
     +
     +	GIT_TRACE2_EVENT="$(pwd)/trace2.txt" \
     +	GIT_TEST_FSYNC=true \
    -+		git -C repo -c core.fsync=reference update-ref refs/heads/branch HEAD &&
    ++		git -C repo -c core.fsync=reference \
    ++		-c core.fsyncMethod=fsync update-ref refs/heads/branch HEAD &&
     +	check_fsync_events trace2.txt <<-EOF
     +	"name":"hardware-flush","count":2
     +	EOF
    @@ t/t0610-reftable-basics.sh (new)
     +
     +for umask in 002 022
     +do
    -+	# A fix for this is in flight via ps/reftable-compacted-tables-permission-fix.
    -+	test_expect_failure POSIXPERM 'pack-refs: honors core.sharedRepository' '
    ++	test_expect_success POSIXPERM 'pack-refs: honors core.sharedRepository' '
     +		test_when_finished "rm -rf repo" &&
     +		(
     +			umask $umask &&
    @@ t/t0610-reftable-basics.sh (new)
     +	'
     +done
     +
    -+# A fix for this is in flight.
    -+test_expect_failure 'packed-refs: writes are synced' '
    ++test_expect_success 'packed-refs: writes are synced' '
     +	test_when_finished "rm -rf repo" &&
     +	git init repo &&
     +	test_commit -C repo initial &&
     +	test_line_count = 2 table-files &&
     +
    ++	: >trace2.txt &&
     +	GIT_TRACE2_EVENT="$(pwd)/trace2.txt" \
     +	GIT_TEST_FSYNC=true \
    -+		git -C repo -c core.fsync=reference pack-refs &&
    ++		git -C repo -c core.fsync=reference \
    ++		-c core.fsyncMethod=fsync pack-refs &&
     +	check_fsync_events trace2.txt <<-EOF
     +	"name":"hardware-flush","count":2
     +	EOF

2: 146bb95c03 = 2: 30e5feb28c ci: add jobs to test with the reftable backend

base-commit: 235986be822c9f8689be2e9a0b7804d0b1b6d821

2.43.GIT