scalar: integrate into core Git

This series completes the initial implementation of the Scalar command as a core component of Git. For readers new to the topic of Scalar, the roadmap/design doc [1] provides some background information including how the project started & evolved, as well as its current intent & high-level design.

Changes since V1
================

* Updated 'scalar.txt' description from "Associated with [git]" -> "Part of [git]".
* Fixed Dscho's "Signed-off-by" and author emails on patches 3 & 4.
* Added 'scalar' to 'git help -a' command list.
* Removed translation tag for literal 'scalar help' usage string.
* Updated 'scalar clone' tests to clone to unique targets and clean up only when test passes.
* Changed 'p9210' perf tests to invoke 'git' and 'scalar' with '-C', rather than creating a subshell to run commands.
* Fixed use of 'test_env_bool' with 'GIT_PERF_USE_SCALAR', locally tested to make sure it was working as-intended.

Goals
=====

The goal of this series is, as noted in the roadmap linked earlier, to integrate Scalar as a core user-facing component of Git. There's no individual existing precedent that the integration is 100% based on; it's more of an amalgam of the approaches used with builtins installed directly to the bindir (e.g. 'git-upload-pack'), contributed tools (e.g. 'gitk'), and the 'git' executable itself. The resulting behavior is:

* Scalar is built & installed unconditionally (like a builtin).
* Scalar is included in 'bin-wrappers/' and is installed into the 'bin/' directory (like 'git-upload-pack').
* Scalar's documentation is installed as 'man1' docs (like 'gitk').
* Scalar is not invoked via the 'git' executable (it's called directly, like 'git' itself).

The move out of 'contrib/' is also intended to mean that the relationship between Git and Scalar should no longer be a strict one-way dependency. In this series, this architectural change is what permits making a change to 'help.c' specifically for the sake of Scalar. Future contributions can and should (where appropriate!) share code between Scalar and non-Scalar parts of Git.

Organization
============

The series is broken up as follows:

1. Fix a CI-breaking documentation error in 'scalar.txt' (patch 1).
2. Move Scalar code out of 'contrib/' and into core Git, including changes in build definition(s) to build & install Scalar and its resources (patch 2).
3. Make 'git help scalar' open the 'scalar' command documentation, introduce 'scalar help' as an alias. This is done after moving out of 'contrib/' due to it involving Scalar-related changes to the non-Scalar file 'help.c' (patches 3-4).
4. Expand test coverage, including perf testing (patches 5-6)
5. Add a perf test configuration allowing users to register standard perf test repos with Scalar (patch 7).
6. Remove the internally-focused Scalar roadmap from 'Documentation/technical/scalar.txt', include it as one of the HTML technical docs (patch 8).

Prior series
============

* Add 'scalar' command in 'contrib/':
https://lore.kernel.org/git/[email protected]/
* Introduce 'scalar diagnose':
https://lore.kernel.org/git/[email protected]/
* Add '-c/-C' compatibility:
https://lore.kernel.org/git/[email protected]/
* [DROPPED] Integrate Scalar into CI builds:
https://lore.kernel.org/git/[email protected]/
* Document Scalar's role in Git & plan remaining work:
https://lore.kernel.org/git/[email protected]/
* Generalize 'scalar diagnose' into 'git diagnose' builtin & 'git bugreport --diagnose':
https://lore.kernel.org/git/[email protected]/
* Add FSMonitor support to Scalar, refactor enlistment search:
https://lore.kernel.org/git/[email protected]/

Thanks!

* Victoria

[1]
https://lore.kernel.org/git/65eda9755d2a29e222bf7e6cc719921b24377cc6.1657584367.git.gitgitgadget@gmail.com/

Johannes Schindelin (2):
git help: special-case scalar
scalar: implement the help subcommand

Victoria Dye (7):
scalar: fix command documentation section header
scalar: include in standard Git build & installation
scalar: add to 'git help -a' command list
scalar-clone: add test coverage
t/perf: add Scalar performance tests
t/perf: add 'GIT_PERF_USE_SCALAR' run option Documentation/technical: include Scalar technical doc

 .gitignore                                    |   1 +
 Documentation/Makefile                        |   2 +
 Documentation/cmd-list.perl                   |   2 +-
 {contrib/scalar => Documentation}/scalar.txt  |   4 +-
 Documentation/technical/scalar.txt            |  61 -------
 Makefile                                      |  31 ++--
 builtin/help.c                                |   2 +
 command-list.txt                              |   1 +
 contrib/buildsystems/CMakeLists.txt           |   9 +-
 contrib/scalar/.gitignore                     |   2 -
 contrib/scalar/Makefile                       |  35 ----
 contrib/scalar/t/Makefile                     |  81 ----------
 contrib/scalar/scalar.c => scalar.c           |  20 +++
 t/perf/README                                 |   4 +
 t/perf/p9210-scalar.sh                        |  39 +++++
 t/perf/perf-lib.sh                            |  13 +-
 t/perf/run                                    |   3 +
 .../t/t9099-scalar.sh => t/t9210-scalar.sh    |  10 +-
 t/t9211-scalar-clone.sh                       | 151 ++++++++++++++++++
 19 files changed, 264 insertions(+), 207 deletions(-)
 rename {contrib/scalar => Documentation}/scalar.txt (99%)
 delete mode 100644 contrib/scalar/.gitignore
 delete mode 100644 contrib/scalar/Makefile
 delete mode 100644 contrib/scalar/t/Makefile
 rename contrib/scalar/scalar.c => scalar.c (98%)
 create mode 100755 t/perf/p9210-scalar.sh
 rename contrib/scalar/t/t9099-scalar.sh => t/t9210-scalar.sh (96%)
 create mode 100755 t/t9211-scalar-clone.sh

base-commit: d42b38dfb5edf1a7fddd9542d722f91038407819
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-1341%2Fvdye%2Ffeature%2Fscalar-toplevel-v2
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1341/vdye/feature/scalar-toplevel-v2
Pull-Request: https://github.com/gitgitgadget/git/pull/1341

Range-diff vs v1:

  1:  fcf73e77574 !  1:  614cbdc629d scalar: fix command documentation section header
     @@ contrib/scalar/scalar.txt: SEE ALSO
      -Scalar
      +GIT
       ---
     - Associated with the linkgit:git[1] suite
     +-Associated with the linkgit:git[1] suite
     ++Part of the linkgit:git[1] suite
  2:  4d69e5eaccb !  2:  bc2092a7a7a scalar: include in standard Git build & installation
     @@ Makefile: XGETTEXT_FLAGS_SH = $(XGETTEXT_FLAGS) --language=Shell \
       	--keyword=__ --keyword=N__ --keyword="__n:1,2"
       MSGMERGE_FLAGS = --add-location --backup=off --update
      -LOCALIZED_C = $(sort $(FOUND_C_SOURCES) $(FOUND_H_SOURCES) $(SCALAR_SOURCES) \
     -+LOCALIZED_C = $(sort $(FOUND_C_SOURCES) $(FOUND_H_SOURCES) \
     - 	        $(GENERATED_H))
     +-	        $(GENERATED_H))
     ++LOCALIZED_C = $(sort $(FOUND_C_SOURCES) $(FOUND_H_SOURCES) $(GENERATED_H))
       LOCALIZED_SH = $(sort $(SCRIPT_SH) git-sh-setup.sh)
       LOCALIZED_PERL = $(sort $(SCRIPT_PERL))
     + 
      @@ Makefile: bin-wrappers/%: wrap-for-bin.sh
       	$(call mkdir_p_parent_template)
       	$(QUIET_GEN)sed -e '1s|#!.*/sh|#!$(SHELL_PATH_SQ)|' \
  3:  969160650aa =  3:  eb8663e0300 git help: special-case `scalar`
  4:  46d0fddfe8f !  4:  55aba052c34 scalar: implement the `help` subcommand
     @@
       ## Metadata ##
     -Author: Johannes Schindelin <[email protected]>
     +Author: Johannes Schindelin <[email protected]>

       ## Commit message ##
          scalar: implement the `help` subcommand
      
          It is merely handing off to `git help scalar`.
      
     -    Signed-off-by: Johannes Schindelin <[email protected]>
     +    Signed-off-by: Johannes Schindelin <[email protected]>
          Signed-off-by: Victoria Dye <[email protected]>
      
       ## scalar.c ##
     @@ scalar.c: static int cmd_delete(int argc, const char **argv)
      +		OPT_END(),
      +	};
      +	const char * const usage[] = {
     -+		N_("scalar help"),
     ++		"scalar help",
      +		NULL
      +	};
      +
  -:  ----------- >  5:  070f195f027 scalar: add to 'git help -a' command list
  5:  a3b7cb0a3bd !  6:  15acf20274b scalar-clone: add test coverage
     @@ Commit message
          scalar-clone: add test coverage
      
          Create a new test file ('t9211-scalar-clone.sh') to exercise the options and
     -    behavior of the 'scalar clone' command.
     +    behavior of the 'scalar clone' command. Each test clones to a unique target
     +    location and cleans up the cloned repo only when the test passes. This
     +    ensures that failed tests' artifacts are captured in CI artifacts for
     +    further debugging.
      
     +    Helped-by: Johannes Schindelin <[email protected]>
          Signed-off-by: Victoria Dye <[email protected]>
      
       ## t/t9211-scalar-clone.sh (new) ##
     @@ t/t9211-scalar-clone.sh (new)
      +}
      +
      +test_expect_success 'creates content in enlistment root' '
     -+	test_when_finished cleanup_clone cloned &&
     ++	enlistment=cloned &&
      +
     -+	scalar clone "file://$(pwd)/to-clone" cloned &&
     -+	ls -A cloned >enlistment-root &&
     ++	scalar clone "file://$(pwd)/to-clone" $enlistment &&
     ++	ls -A $enlistment >enlistment-root &&
      +	test_line_count = 1 enlistment-root &&
     -+	test_path_is_dir cloned/src &&
     -+	test_path_is_dir cloned/src/.git
     ++	test_path_is_dir $enlistment/src &&
     ++	test_path_is_dir $enlistment/src/.git &&
     ++
     ++	cleanup_clone $enlistment
      +'
      +
      +test_expect_success 'with spaces' '
     -+	test_when_finished cleanup_clone "cloned with space" &&
     ++	enlistment="cloned with space" &&
     ++
     ++	scalar clone "file://$(pwd)/to-clone" "$enlistment" &&
     ++	test_path_is_dir "$enlistment" &&
     ++	test_path_is_dir "$enlistment/src" &&
     ++	test_path_is_dir "$enlistment/src/.git" &&
      +
     -+	scalar clone "file://$(pwd)/to-clone" "cloned with space" &&
     -+	test_path_is_dir "cloned with space" &&
     -+	test_path_is_dir "cloned with space/src" &&
     -+	test_path_is_dir "cloned with space/src/.git"
     ++	cleanup_clone "$enlistment"
      +'
      +
      +test_expect_success 'partial clone if supported by server' '
     -+	test_when_finished cleanup_clone cloned &&
     ++	enlistment=partial-clone &&
      +
     -+	scalar clone "file://$(pwd)/to-clone" cloned &&
     ++	scalar clone "file://$(pwd)/to-clone" $enlistment &&
      +
      +	(
     -+		cd cloned/src &&
     ++		cd $enlistment/src &&
      +
      +		# Two promisor packs: one for refs, the other for blobs
      +		ls .git/objects/pack/pack-*.promisor >promisorlist &&
      +		test_line_count = 2 promisorlist
     -+	)
     ++	) &&
     ++
     ++	cleanup_clone $enlistment
      +'
      +
      +test_expect_success 'fall back on full clone if partial unsupported' '
     -+	test_when_finished cleanup_clone cloned &&
     ++	enlistment=no-partial-support &&
      +
      +	test_config -C to-clone uploadpack.allowfilter false &&
      +	test_config -C to-clone uploadpack.allowanysha1inwant false &&
      +
     -+	scalar clone "file://$(pwd)/to-clone" cloned 2>err &&
     ++	scalar clone "file://$(pwd)/to-clone" $enlistment 2>err &&
      +	grep "filtering not recognized by server, ignoring" err &&
      +
      +	(
     -+		cd cloned/src &&
     ++		cd $enlistment/src &&
      +
      +		# Still get a refs promisor file, but none for blobs
      +		ls .git/objects/pack/pack-*.promisor >promisorlist &&
      +		test_line_count = 1 promisorlist
     -+	)
     ++	) &&
     ++
     ++	cleanup_clone $enlistment
      +'
      +
      +test_expect_success 'initializes sparse-checkout by default' '
     -+	test_when_finished cleanup_clone cloned &&
     ++	enlistment=sparse &&
      +
     -+	scalar clone "file://$(pwd)/to-clone" cloned &&
     ++	scalar clone "file://$(pwd)/to-clone" $enlistment &&
      +	(
     -+		cd cloned/src &&
     ++		cd $enlistment/src &&
      +		test_cmp_config true core.sparseCheckout &&
      +		test_cmp_config true core.sparseCheckoutCone
     -+	)
     ++	) &&
     ++
     ++	cleanup_clone $enlistment
      +'
      +
      +test_expect_success '--full-clone does not create sparse-checkout' '
     -+	test_when_finished cleanup_clone cloned &&
     ++	enlistment=full-clone &&
      +
     -+	scalar clone --full-clone "file://$(pwd)/to-clone" cloned &&
     ++	scalar clone --full-clone "file://$(pwd)/to-clone" $enlistment &&
      +	(
     -+		cd cloned/src &&
     ++		cd $enlistment/src &&
      +		test_cmp_config "" --default "" core.sparseCheckout &&
      +		test_cmp_config "" --default "" core.sparseCheckoutCone
     -+	)
     ++	) &&
     ++
     ++	cleanup_clone $enlistment
      +'
      +
      +test_expect_success '--single-branch clones HEAD only' '
     -+	test_when_finished cleanup_clone cloned &&
     ++	enlistment=single-branch &&
      +
     -+	scalar clone --single-branch "file://$(pwd)/to-clone" cloned &&
     ++	scalar clone --single-branch "file://$(pwd)/to-clone" $enlistment &&
      +	(
     -+		cd cloned/src &&
     ++		cd $enlistment/src &&
      +		git for-each-ref refs/remotes/origin >out &&
      +		test_line_count = 1 out &&
      +		grep "refs/remotes/origin/base" out
     -+	)
     ++	) &&
     ++
     ++	cleanup_clone $enlistment
      +'
      +
      +test_expect_success '--no-single-branch clones all branches' '
     -+	test_when_finished cleanup_clone cloned &&
     ++	enlistment=no-single-branch &&
      +
     -+	scalar clone --no-single-branch "file://$(pwd)/to-clone" cloned &&
     ++	scalar clone --no-single-branch "file://$(pwd)/to-clone" $enlistment &&
      +	(
     -+		cd cloned/src &&
     ++		cd $enlistment/src &&
      +		git for-each-ref refs/remotes/origin >out &&
      +		test_line_count = 2 out &&
      +		grep "refs/remotes/origin/base" out &&
      +		grep "refs/remotes/origin/parallel" out
     -+	)
     ++	) &&
     ++
     ++	cleanup_clone $enlistment
      +'
      +
      +test_done
  6:  42ab39f2121 !  7:  d5b68c65163 t/perf: add Scalar performance tests
     @@ Commit message
          9210.6: test_commit --append --no-tag A (scalar)       0.08(0.02+0.03)
          9210.7: test_commit --append --no-tag A (non-scalar)   0.13(0.03+0.11)
      
     +    Helped-by: Johannes Schindelin <[email protected]>
          Signed-off-by: Victoria Dye <[email protected]>
      
       ## t/perf/p9210-scalar.sh (new) ##
     @@ t/perf/p9210-scalar.sh (new)
      +'
      +
      +test_compare_perf () {
     -+	command="$@"
     -+	test_perf "$command (scalar)" "
     -+		(
     -+			cd scalar-clone/src &&
     -+			$command
     -+		)
     ++	command=$1
     ++	shift
     ++	args=$*
     ++	test_perf "$command $args (scalar)" "
     ++		$command -C scalar-clone/src $args
      +	"
      +
     -+	test_perf "$command (non-scalar)" "
     -+		(
     -+			cd git-clone &&
     -+			$command
     -+		)
     ++	test_perf "$command $args (non-scalar)" "
     ++		$command -C git-clone $args
      +	"
      +}
      +
  7:  96e08a95265 !  8:  2e6dd03beba t/perf: add 'GIT_PERF_USE_SCALAR' run option
     @@ t/perf/perf-lib.sh: test_perf_create_repo_from () {
       			# a copy it's fine to remove the lock.
       			rm .git/index.lock
      +		fi &&
     -+		if test_bool_env "$GIT_PERF_USE_SCALAR" false
     ++		if test_bool_env GIT_PERF_USE_SCALAR false
      +		then
      +			"$MODERN_SCALAR" register
       		fi
     @@ t/perf/perf-lib.sh: test_perf_fresh_repo () {
       		cd "$repo" &&
      -		test_perf_do_repo_symlink_config_
      +		test_perf_do_repo_symlink_config_ &&
     -+		if test_bool_env "$GIT_PERF_USE_SCALAR" false
     ++		if test_bool_env GIT_PERF_USE_SCALAR false
      +		then
      +			"$MODERN_SCALAR" register
      +		fi

8: 8bbe913dccf = 9: 6bedd80c398 Documentation/technical: include Scalar technical doc

gitgitgadget