Skip to contentSkip to conversation

Upstreaming the Scalar command

RFC PATCH Superseded Johannes Schindelin via GitGitGadget 11 participants 303 messages Aug 30, 2021 – Dec 14, 2021 1 message not in the archive Page 2 of 3

Subsystems (2)
lore
Share & tools
lore
#133 lore raw

PATCH v4 12/15 scalar: teach 'reconfigure' to optionally handle all registered enlistments

In reply to Johannes Schindelin via GitGitGadget

From: Johannes Schindelin <[email protected]>

After a Scalar upgrade, it can come in really handy if there is an easy way to reconfigure all Scalar enlistments. This new option offers this functionality.

---
 contrib/scalar/scalar.c          | 61 ++++++++++++++++++++++++++++++--
 contrib/scalar/scalar.txt        |  9 +++--
 contrib/scalar/t/t9099-scalar.sh |  3 ++
 3 files changed, 67 insertions(+), 6 deletions(-)
contrib/scalar/scalar.c +58 -3
index 1fff7eb7c12..67fa5305225 100644
@@ -494,22 +494,77 @@ static int cmd_register(int argc, const char **argv)
 	return register_dir();
 }
 
+static int get_scalar_repos(const char *key, const char *value, void *data)
+{
+	struct string_list *list = data;
+
+	if (!strcmp(key, "scalar.repo"))
+		string_list_append(list, value);
+
+	return 0;
+}
+
 static int cmd_reconfigure(int argc, const char **argv)
 {
+	int all = 0;
 	struct option options[] = {
+		OPT_BOOL('a', "all", &all,
+			 N_("reconfigure all registered enlistments")),
 		OPT_END(),
 	};
 	const char * const usage[] = {
-		N_("scalar reconfigure [<enlistment>]"),
+		N_("scalar reconfigure [--all | <enlistment>]"),
 		NULL
 	};
+	struct string_list scalar_repos = STRING_LIST_INIT_DUP;
+	int i, res = 0;
+	struct repository r = { NULL };
+	struct strbuf commondir = STRBUF_INIT, gitdir = STRBUF_INIT;
 
 	argc = parse_options(argc, argv, NULL, options,
 			     usage, 0);
 
-	setup_enlistment_directory(argc, argv, usage, options, NULL);
+	if (!all) {
+		setup_enlistment_directory(argc, argv, usage, options, NULL);
+
+		return set_recommended_config(1);
+	}
+
+	if (argc > 0)
+		usage_msg_opt(_("--all or <enlistment>, but not both"),
+			      usage, options);
+
+	git_config(get_scalar_repos, &scalar_repos);
 
-	return set_recommended_config(1);
+	for (i = 0; i < scalar_repos.nr; i++) {
+		const char *dir = scalar_repos.items[i].string;
+
+		strbuf_reset(&commondir);
+		strbuf_reset(&gitdir);
+
+		if (chdir(dir) < 0) {
+			warning_errno(_("could not switch to '%s'"), dir);
+			res = -1;
+		} else if (discover_git_directory(&commondir, &gitdir) < 0) {
+			warning_errno(_("git repository gone in '%s'"), dir);
+			res = -1;
+		} else {
+			git_config_clear();
+
+			the_repository = &r;
+			r.commondir = commondir.buf;
+			r.gitdir = gitdir.buf;
+
+			if (set_recommended_config(1) < 0)
+				res = -1;
+		}
+	}
+
+	string_list_clear(&scalar_repos, 1);
+	strbuf_release(&commondir);
+	strbuf_release(&gitdir);
+
+	return res;
 }
 
 static int cmd_run(int argc, const char **argv)
contrib/scalar/scalar.txt +6 -3
index f4e4686e8c8..2fa96fcabc6 100644
@@ -13,7 +13,7 @@ scalar list
 scalar register [<enlistment>]
 scalar unregister [<enlistment>]
 scalar run ( all | config | commit-graph | fetch | loose-objects | pack-files ) [<enlistment>]
-scalar reconfigure <enlistment>
+scalar reconfigure [ --all | <enlistment> ]
 
 DESCRIPTION
 -----------
@@ -32,8 +32,8 @@ an existing Git worktree with Scalar whose name is not `src`, the enlistment
 will be identical to the worktree.
 
 The `scalar` command implements various subcommands, and different options
-depending on the subcommand. With the exception of `clone` and `list`, all
-subcommands expect to be run in an enlistment.
+depending on the subcommand. With the exception of `clone`, `list` and
+`reconfigure --all`, all subcommands expect to be run in an enlistment.
 
 COMMANDS
 --------
@@ -124,6 +124,9 @@ After a Scalar upgrade, or when the configuration of a Scalar enlistment
 was somehow corrupted or changed by mistake, this subcommand allows to
 reconfigure the enlistment.
 
+With the `--all` option, all enlistments currently registered with Scalar
+will be reconfigured. Use this option after each Scalar upgrade.
+
 SEE ALSO
 --------
 linkgit:git-clone[1], linkgit:git-maintenance[1].
contrib/scalar/t/t9099-scalar.sh +3 -0
index e6d74a06ca0..5fe7fabd0e5 100755
@@ -70,6 +70,9 @@ test_expect_success 'scalar reconfigure' '
 	scalar register one &&
 	git -C one/src config core.preloadIndex false &&
 	scalar reconfigure one &&
+	test true = "$(git -C one/src config core.preloadIndex)" &&
+	git -C one/src config core.preloadIndex false &&
+	scalar reconfigure -a &&
 	test true = "$(git -C one/src config core.preloadIndex)"
 '
 
Signature · 1 line
gitgitgadget
#134 lore raw

PATCH v4 13/15 scalar: implement the `delete` command

In reply to Johannes Schindelin via GitGitGadget

From: Matthew John Cheetham <[email protected]>

Delete an enlistment by first unregistering the repository and then deleting the enlistment directory (usually the directory containing the worktree src/ directory).

On Windows, if the current directory is inside the enlistment's directory, change to the parent of the enlistment directory, to allow us to delete the enlistment (directories used by processes e.g. as current working directories cannot be deleted on Windows).

---
 contrib/scalar/scalar.c          | 55 ++++++++++++++++++++++++++++++++
 contrib/scalar/scalar.txt        |  8 +++++
 contrib/scalar/t/t9099-scalar.sh |  9 ++++++
 3 files changed, 72 insertions(+)
contrib/scalar/scalar.c +55 -0
index 67fa5305225..00bedb0bf66 100644
@@ -8,6 +8,7 @@
 #include "config.h"
 #include "run-command.h"
 #include "refs.h"
+#include "dir.h"
 
 /*
  * Remove the deepest subdirectory in the provided path string. Path must not
@@ -334,6 +335,33 @@ static char *remote_default_branch(const char *url)
 	return NULL;
 }
 
+static int delete_enlistment(struct strbuf *enlistment)
+{
+#ifdef WIN32
+	struct strbuf parent = STRBUF_INIT;
+#endif
+
+	if (unregister_dir())
+		die(_("failed to unregister repository"));
+
+#ifdef WIN32
+	/*
+	 * Change the current directory to one outside of the enlistment so
+	 * that we may delete everything underneath it.
+	 */
+	strbuf_addbuf(&parent, enlistment);
+	strbuf_parent_directory(&parent);
+	if (chdir(parent.buf) < 0)
+		die_errno(_("could not switch to '%s'"), parent.buf);
+	strbuf_release(&parent);
+#endif
+
+	if (remove_dir_recursively(enlistment, 0))
+		die(_("failed to delete enlistment directory"));
+
+	return 0;
+}
+
 static int cmd_clone(int argc, const char **argv)
 {
 	const char *branch = NULL;
@@ -694,6 +722,32 @@ static int cmd_unregister(int argc, const char **argv)
 	return unregister_dir();
 }
 
+static int cmd_delete(int argc, const char **argv)
+{
+	struct option options[] = {
+		OPT_END(),
+	};
+	const char * const usage[] = {
+		N_("scalar delete <enlistment>"),
+		NULL
+	};
+	struct strbuf enlistment = STRBUF_INIT;
+	int res = 0;
+
+	argc = parse_options(argc, argv, NULL, options,
+			     usage, 0);
+
+	if (argc != 1)
+		usage_with_options(usage, options);
+
+	setup_enlistment_directory(argc, argv, usage, options, &enlistment);
+
+	res = delete_enlistment(&enlistment);
+	strbuf_release(&enlistment);
+
+	return res;
+}
+
 static struct {
 	const char *name;
 	int (*fn)(int, const char **);
@@ -704,6 +758,7 @@ static struct {
 	{ "unregister", cmd_unregister },
 	{ "run", cmd_run },
 	{ "reconfigure", cmd_reconfigure },
+	{ "delete", cmd_delete },
 	{ NULL, NULL},
 };
 
contrib/scalar/scalar.txt +8 -0
index 2fa96fcabc6..6fc57707718 100644
@@ -14,6 +14,7 @@ scalar register [<enlistment>]
 scalar unregister [<enlistment>]
 scalar run ( all | config | commit-graph | fetch | loose-objects | pack-files ) [<enlistment>]
 scalar reconfigure [ --all | <enlistment> ]
+scalar delete <enlistment>
 
 DESCRIPTION
 -----------
@@ -127,6 +128,13 @@ reconfigure the enlistment.
 With the `--all` option, all enlistments currently registered with Scalar
 will be reconfigured. Use this option after each Scalar upgrade.
 
+Delete
+~~~~~~
+
+delete <enlistment>::
+	This subcommand lets you delete an existing Scalar enlistment from your
+	local file system, unregistering the repository.
+
 SEE ALSO
 --------
 linkgit:git-clone[1], linkgit:git-maintenance[1].
contrib/scalar/t/t9099-scalar.sh +9 -0
index 5fe7fabd0e5..7e8771d0eff 100755
@@ -76,4 +76,13 @@ test_expect_success 'scalar reconfigure' '
 	test true = "$(git -C one/src config core.preloadIndex)"
 '
 
+test_expect_success 'scalar delete without enlistment shows a usage' '
+	test_expect_code 129 scalar delete
+'
+
+test_expect_success 'scalar delete with enlistment' '
+	scalar delete cloned &&
+	test_path_is_missing cloned
+'
+
 test_done
Signature · 1 line
gitgitgadget
#138 lore raw

PATCH v4 14/15 scalar: implement the `version` command

In reply to Johannes Schindelin via GitGitGadget

From: Johannes Schindelin <[email protected]>

The .NET version of Scalar has a version command. This was necessary because it was versioned independently of Git.

Since Scalar is now tightly coupled with Git, it does not make sense for them to show different versions. Therefore, it shows the same output as git version. For backwards-compatibility with the .NET version, scalar version prints to stderr, though (git version prints to stdout instead).

---
 contrib/scalar/scalar.c | 39 +++++++++++++++++++++++++++++++++++++++
 1 file changed, 39 insertions(+)
contrib/scalar/scalar.c +39 -0
index 00bedb0bf66..728166aa97a 100644
@@ -9,6 +9,7 @@
 #include "run-command.h"
 #include "refs.h"
 #include "dir.h"
+#include "help.h"
 
 /*
  * Remove the deepest subdirectory in the provided path string. Path must not
@@ -362,6 +363,15 @@ static int delete_enlistment(struct strbuf *enlistment)
 	return 0;
 }
 
+/*
+ * Dummy implementation; Using `get_version_info()` would cause a link error
+ * without this.
+ */
+void load_builtin_commands(const char *prefix, struct cmdnames *cmds)
+{
+	die("not implemented");
+}
+
 static int cmd_clone(int argc, const char **argv)
 {
 	const char *branch = NULL;
@@ -748,6 +758,34 @@ static int cmd_delete(int argc, const char **argv)
 	return res;
 }
 
+static int cmd_version(int argc, const char **argv)
+{
+	int verbose = 0, build_options = 0;
+	struct option options[] = {
+		OPT__VERBOSE(&verbose, N_("include Git version")),
+		OPT_BOOL(0, "build-options", &build_options,
+			 N_("include Git's build options")),
+		OPT_END(),
+	};
+	const char * const usage[] = {
+		N_("scalar verbose [-v | --verbose] [--build-options]"),
+		NULL
+	};
+	struct strbuf buf = STRBUF_INIT;
+
+	argc = parse_options(argc, argv, NULL, options,
+			     usage, 0);
+
+	if (argc != 0)
+		usage_with_options(usage, options);
+
+	get_version_info(&buf, build_options);
+	fprintf(stderr, "%s\n", buf.buf);
+	strbuf_release(&buf);
+
+	return 0;
+}
+
 static struct {
 	const char *name;
 	int (*fn)(int, const char **);
@@ -759,6 +797,7 @@ static struct {
 	{ "run", cmd_run },
 	{ "reconfigure", cmd_reconfigure },
 	{ "delete", cmd_delete },
+	{ "version", cmd_version },
 	{ NULL, NULL},
 };
 
Signature · 1 line
gitgitgadget
#139 lore raw

PATCH v4 15/15 scalar: accept -C and -c options before the subcommand

In reply to Johannes Schindelin via GitGitGadget

From: Johannes Schindelin <[email protected]>

The git executable has these two very useful options:

-C <directory>:
switch to the specified directory before performing any actions

-c <key>=<value>:
temporarily configure this setting for the duration of the specified scalar subcommand

With this commit, we teach the scalar executable the same trick.

---
 contrib/scalar/scalar.c   | 22 +++++++++++++++++++++-
 contrib/scalar/scalar.txt | 10 ++++++++++
 2 files changed, 31 insertions(+), 1 deletion(-)
contrib/scalar/scalar.c +21 -1
index 728166aa97a..76a77ca1ed0 100644
@@ -806,6 +806,25 @@ int cmd_main(int argc, const char **argv)
 	struct strbuf scalar_usage = STRBUF_INIT;
 	int i;
 
+	while (argc > 1 && *argv[1] == '-') {
+		if (!strcmp(argv[1], "-C")) {
+			if (argc < 3)
+				die(_("-C requires a <directory>"));
+			if (chdir(argv[2]) < 0)
+				die_errno(_("could not change to '%s'"),
+					  argv[2]);
+			argc -= 2;
+			argv += 2;
+		} else if (!strcmp(argv[1], "-c")) {
+			if (argc < 3)
+				die(_("-c requires a <key>=<value> argument"));
+			git_config_push_parameter(argv[2]);
+			argc -= 2;
+			argv += 2;
+		} else
+			break;
+	}
+
 	if (argc > 1) {
 		argv++;
 		argc--;
@@ -816,7 +835,8 @@ int cmd_main(int argc, const char **argv)
 	}
 
 	strbuf_addstr(&scalar_usage,
-		      N_("scalar <command> [<options>]\n\nCommands:\n"));
+		      N_("scalar [-C <directory>] [-c <key>=<value>] "
+			 "<command> [<options>]\n\nCommands:\n"));
 	for (i = 0; builtins[i].name; i++)
 		strbuf_addf(&scalar_usage, "\t%s\n", builtins[i].name);
 
contrib/scalar/scalar.txt +10 -0
index 6fc57707718..3a80f829edc 100644
@@ -36,6 +36,16 @@ The `scalar` command implements various subcommands, and different options
 depending on the subcommand. With the exception of `clone`, `list` and
 `reconfigure --all`, all subcommands expect to be run in an enlistment.
 
+The following options can be specified _before_ the subcommand:
+
+-C <directory>::
+	Before running the subcommand, change the working directory. This
+	option imitates the same option of linkgit:git[1].
+
+-c <key>=<value>::
+	For the duration of running the specified subcommand, configure this
+	setting. This option imitates the same option of linkgit:git[1].
+
 COMMANDS
 --------
 
Signature · 1 line
gitgitgadget
#110 lore raw

v4 cover Upstreaming the Scalar command

In reply to Johannes Schindelin via GitGitGadget

tl;dr: This series contributes the Scalar command to the Git project. This command provides an opinionated way to create and configure repositories with a focus on very large repositories.

Changes since v3:

* Moved the "Changes since" section to the top, to make it easier to see what changed.
* Reworded the commit message of the first patch.
* Removed the [RFC] prefix because I did not hear any objections against putting this into contrib/.

Changes since v2:

* Adjusted the description of the list command in the manual page , as suggested by Bagas.
* Addressed two style nits in cmd_run().
* The documentation of git reconfigure -a was improved.

Changes since v1:

* A couple typos were fixed
* The code parsing the output of ls-remote was made more readable
* The indentation used in scalar.txt now consistently uses tabs
* We no longer hard-code core.bare = false when registering with Scalar

Background
==========

Years ago, Microsoft wanted to move the source code of the Windows operating system to Git. The challenge there was to prove that Git could scale to massive monorepos. The VFS for Git (formerly GVFS) project was born to take up that challenge.

The final solution included a virtual filesystem (with both user-mode and kernel components) and a customized fork of Git for Windows. This solution contained several key concepts, such as only populating a portion of the working directory, demand-fetching blobs, and performing periodic repo maintenance in the background. However, the required kernel drivers made it difficult to port the solution to other platforms.

But it was realized that many of these key concepts were independent of the actual VFS and its projection of the working directory. The Scalar project was created to make that separation, refine the key concepts, and then extract those features into the new Scalar command.

The present
===========

The Scalar project provides a completely functional non-virtual experience for monorepos. But why stop there. The Scalar project was designed to be a self-destructing vehicle to allow those key concepts to be moved into core Git itself for the benefit of all. For example, partial clone, sparse-checkout, and background maintenance have already been upstreamed and removed from Scalar proper. This patch series provides a C-based implementation of the final remaining portions of the Scalar command. This will make it easier for users to experiment with the Scalar command. It will also make it substantially easier to experiment with moving functionality from Scalar into core Git, while maintaining backwards-compatibility for existing Scalar users.

The C-based Scalar has been shipped to Scalar users, and can be tested by any interested reader:
https://github.com/microsoft/git/releases/tag/v2.33.0.vfs.0.0 (it offers a Git for Windows installer, a macOS package and an Ubuntu package).

Opportunities
=============

Apart from providing the Scalar command, this contribution is intended to serve as a basis for further mailing list discussions on moving (some of) these key concepts into the main Git commands.

For example, we previously discussed the idea of a "git big-clone" that does much of what "scalar clone" is doing. This patch series is a step to make such functionality exist in the Git code base while we simmer on what such a "git big-clone" command-line interface would look like.

This is one of many possible ways to do this. Creating a 'git big-clone' could lock Git into backwards compatibility concerns so it is necessary to approach such an endeavor with caution. As a discussion starter, the scalar clone <url> command does roughly this:

1. git clone --sparse --filter=blob:none /src
2. git -C /src sparse-checkout init --cone
3. git -C /src config (many times)
4. git -C /src maintenance start

It is my hope inspire discussions about what parts of Scalar could go into core Git, and where, and in which form. While we wish to maintain backwards-compatibility of Scalar's command-line interface (because it is already in use), by having the Scalar code in the same code base as Git's, it will be much easier to move functionality without having to maintain loose version coupling between independently-versioned Scalar and Git. The tight version-coupling, along with having access to libgit.a also allows the C-based implementation of Scalar to be much smaller than the original .NET version.

For example, we might choose in the future to implement, say, git clone --scale=partial,cone to initialize a partial clone with a cone-sparse checkout, that would not only be totally doable, and not only would we already have precedent and data to prove that this actually makes engineers happy who have to work on ginormous repositories, but we could then also implement it by moving parts of contrib/scalar/ to builtin/ (where contrib/scalar/ would then call the built-ins accordingly rather than hard-coding the defaults itself).

We now also have the opportunity to discuss the merits of Scalar's clone caching, which is not actually part of this patch series because it is a bit coupled with the GVFS parts of microsoft/git for the moment, where clones automatically get registered with a populated alternate repository that is identified by the URL, meaning: subsequent clones of the same repository are vastly faster than the first one because they do not actually download the already-received objects again, they access the cache instead.

Another thing that I could imagine to be discussed at length is the distinction between enlistment and worktree (where the latter is the actual Git worktree and usually lives in the src/ subdirectory of the former). This encourages untracked and ignored files to be placed outside the worktree, making Git's job much easier. This idea, too, might find its way in one way or another into Git proper.

These are just a few concepts in Scalar that do not yet have equivalents in Git. By putting this initial implementation into contrib/, we create a foundation for future discussions of these concepts.

We plan on updating the recommended config settings in scalar register as new Git features are available (such as builtin FSMonitor and sparse-index, when ready). To facilitate upgrading existing Scalar enlistments, their paths are automatically added to the [scalar] section of the global Git config, and the scalar reconfigure --all command will process all of them.

Epilogue
========

Now, to address some questions that I imagine every reader has who made it this far:

* Why not put the Scalar functionality directly into a built-in? Creating a Git builtin requires scrutiny over every aspect of the feature, which is difficult to do while also maintaining the command-line interface contract and expected behavior of the Scalar command (there are existing users, after all). By having the Scalar command in contrib/, we present a simple option for users to have these features in the short term while the Git contributor community decides which bits to absorb into Git built-ins.
* Why implement the Scalar command in the Git codebase? We ported Scalar to the microsoft/git fork for several reasons. First, we realized it was possible now that the core features exist inside Git itself. Second, compiling Scalar directly within a version of Git allows us to remove a version compatibility check from each config option that might or might not apply based on the installed Git version. Finally, this new location has greatly simplified our release process and the installation process for users. We now have ways to install Scalar with microsoft/git via winget, brew, and apt-get. This has been the case since we shipped v2.32.0 to our users, read: this setup has served us well already.
* Why contribute Scalar to the Git project? We are biased, of course, yet we do have evidence that the Scalar command is a helpful tool that offers an simple way to handle huge repositories with ease. By contributing it to the core Git project, we are able to share it with more users, especially some users who do not want to install the microsoft/git fork. We intend to include Scalar as a component in git-for-windows/git, but are contributing it here first. Further, we think there is benefit to the Git developer community as this presents an example of how to set certain defaults that work for large repositories.
* Does this integrate with the built-in FSMonitor yet? No, not yet. I do have a couple of add-on patch series lined up, one of them being the integration with the built-in FSMonitor, which obviously has to wait until the FSMonitor patch series advances further.

Derrick Stolee (4):
scalar: 'register' sets recommended config and starts maintenance
scalar: 'unregister' stops background maintenance
scalar: implement 'scalar list'
scalar: implement the run command

Johannes Schindelin (10):
scalar: create a rudimentary executable
scalar: start documenting the command
scalar: create test infrastructure
scalar: let 'unregister' handle a deleted enlistment directory
gracefully
scalar: implement the clone subcommand
scalar: teach 'clone' to support the --single-branch option
scalar: allow reconfiguring an existing enlistment
scalar: teach 'reconfigure' to optionally handle all registered
enlistments
scalar: implement the version command
scalar: accept -C and -c options before the subcommand

Matthew John Cheetham (1):
scalar: implement the delete command

 Makefile                         |   8 +
 contrib/scalar/.gitignore        |   5 +
 contrib/scalar/Makefile          |  57 +++
 contrib/scalar/scalar.c          | 844 +++++++++++++++++++++++++++++++
 contrib/scalar/scalar.txt        | 154 ++++++
 contrib/scalar/t/Makefile        |  78 +++
 contrib/scalar/t/t9099-scalar.sh |  88 ++++
 7 files changed, 1234 insertions(+)
 create mode 100644 contrib/scalar/.gitignore
 create mode 100644 contrib/scalar/Makefile
 create mode 100644 contrib/scalar/scalar.c
 create mode 100644 contrib/scalar/scalar.txt
 create mode 100644 contrib/scalar/t/Makefile
 create mode 100755 contrib/scalar/t/t9099-scalar.sh

base-commit: ebf3c04b262aa27fbb97f8a0156c2347fecafafb
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-1005%2Fdscho%2Fscalar-the-beginning-v4
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1005/dscho/scalar-the-beginning-v4
Pull-Request: https://github.com/gitgitgadget/git/pull/1005

Range-diff vs v3:

1: b8c7d3f8450 ! 1: 852ec003109 scalar: create a rudimentary executable
@@ Commit message
with the intention to facilitate integrating the functionality into core Git all while maintaining backwards-compatibility for existing Scalar users (which will be much easier when both live in the same worktree).
- It was always to plan to contribute all of the proven strategies back to
- core Git.
+ It has always been the plan to contribute all of the proven strategies
+ back to core Git.

For example, while the virtual filesystem provided by VFS for Git helped the team developing the Windows operating system to move onto Git, while
@@ Commit message
repository in a healthy state.

With partial clone, sparse checkout and git maintenance having been
- upstreamed, there is little left that scalar.exe does that which
- git.exe cannot do. One such thing is that scalar clone <url> will
+ upstreamed, there is little left that scalar.exe does which git.exe
+ cannot do. One such thing is that scalar clone <url> will
automatically set up a partial, sparse clone, and configure known-helpful settings from the start.

  2:  4f886575dcf =  2:  6ab9c7195da scalar: start documenting the command
  3:  bcfde9bc765 =  3:  14992033d7c scalar: create test infrastructure
  4:  ee3e26a0c4e =  4:  bbbc4c33390 scalar: 'register' sets recommended config and starts maintenance
  5:  6142f75875b =  5:  eadcddb2a9b scalar: 'unregister' stops background maintenance
  6:  82dd253154f =  6:  c3c2f3a4971 scalar: let 'unregister' handle a deleted enlistment directory gracefully
  7:  d291d3723a6 =  7:  90ef9b826b2 scalar: implement 'scalar list'
  8:  40dbf61771e =  8:  79cde4417d8 scalar: implement the `clone` subcommand
  9:  414dbe7d859 =  9:  0acdaeb7396 scalar: teach 'clone' to support the --single-branch option
 10:  76de416a643 = 10:  64e3403ac12 scalar: implement the `run` command
 11:  655a902b9df = 11:  ada242c7c8c scalar: allow reconfiguring an existing enlistment
 12:  2d1987bfcda = 12:  5c11117da51 scalar: teach 'reconfigure' to optionally handle all registered enlistments
 13:  c67938299ee = 13:  914c16c7fcd scalar: implement the `delete` command
 14:  d2cd2b7094b = 14:  06995770120 scalar: implement the `version` command
 15:  7ccc4f8b9b0 = 15:  7539725bb4f scalar: accept -C and -c options before the subcommand
Signature · 1 line
gitgitgadget
#140 lore raw

Hi,

Quoted 11 lines · Johannes Schindelin via GitGitGadget

tl;dr: This series contributes the Scalar command to the Git project. This command provides an opinionated way to create and configure repositories with a focus on very large repositories.

Changes since v3:

* Moved the "Changes since" section to the top, to make it easier to see what changed.
* Reworded the commit message of the first patch.
* Removed the [RFC] prefix because I did not hear any objections against putting this into contrib/.

Forgot to say:

* Sent this out to reflect my latest state before taking off for two weeks.

Ciao,
Dscho

#99 lore raw

Re: Train station analogy, was Re: [PATCH v3 00/15] [RFC] Upstreaming the Scalar command

In reply to Johannes Schindelin
Quoted 14 lines · Johannes Schindelin

Okay, let's try an analogy.

Imagine that a person is asking for directions to the train station. And the other person is replying by asking "did you know that this train station was built in 1878? It is actually quite interesting a story... [and then goes on to describe the history and what excites them about it]". Now, the first person tries again to ask for directions, again does not get an answer to that question, and is slowly starting to look at their watch. The second person, being completely oblivious to all of this, goes on with their wonderful story about the train station and its cultural heritage. So the first person walks a bit further to ask a third person, but the second person is not done yet and says "but you haven't heard me out! That's disrespectful!".

Quoted 5 lines

Just imagine for a minute how you would feel if you were the first person.

And that is how I feel asking for reviews about the Scalar patch series and then being forcefully dragged into that tangent about the build process.

At least to me, how this Makefile for Scalar should interact with the overall build process does not mesh well with the story about hwo direction to and history of the station are unrelated. If we plan to start from contrib/ and eventually want to make it a part of the core Git (i.e. "git scalar <subcmd> ..." becomes just like "git bisect <subcmd> ..."), we would eventually need to see the recipe needed for including "bisect" and "scalar" work the same way, no?

I am getting the impression that such a unified build process is Ævar wants to see at the end, I am not even sure if you do from the above "analogy". Cool down a bit, perhaps?

The following assumes that you share the goal of making "git scalar" just like "git bisect"---another first class citizen of Git toolbox, the user can choose to use it or the user may not have a need to interact with it, but it exists there by default and is not an opt-in add-on component.

I would understand it if your plan is to convert to a unified build procedure at the very end of the upstreaming process, and not while you populate contrib/ with more and more scalar stuff, because the Makefile bits for the entire scalar, while not yet upstreamed, has already been written as a separate procedure and having to convert the whole thing upfront before you can start trickle parts would mean you need to (re)start the process. And I would even be sympathetic if you felt it like a distraction.

But at least I view it as a step that needs to happen sometime between now and at the end. I do not yet have an opinion on which one is more pleasant, between (1) having to deal with a single Makefile that needs to be aware of two different locations *.[ch] lives in, and (2) having to deal with two Makefiles that duplicates definitions and risks them needlessly diverging.

I also would understand it if the reason why you want to keep the top-level Makefile as intact as possible because you sense a high probability that scalar will stay in contrib/ and even turn out to be a failure. Keeping the build procedure separated certainly will keep it easier to yank it out later. But I do not think such a case is quite likely.

Thanks.

#141 lore raw
In reply to Johannes Schindelin
Quoted 16 lines · Johannes Schindelin

Hi,

Quoted 11 lines · Johannes Schindelin via GitGitGadget

tl;dr: This series contributes the Scalar command to the Git project. This command provides an opinionated way to create and configure repositories with a focus on very large repositories.

Changes since v3:

* Moved the "Changes since" section to the top, to make it easier to see what changed.
* Reworded the commit message of the first patch.
* Removed the [RFC] prefix because I did not hear any objections against putting this into contrib/.

Quoted 4 lines

Forgot to say:

* Sent this out to reflect my latest state before taking off for two weeks.

Thanks, will queue.

Have fun.

#100 lore raw

Re: Train station analogy, was Re: [PATCH v3 00/15] [RFC] Upstreaming the Scalar command

In reply to Junio C Hamano
Quoted 50 lines · Junio C Hamano
Quoted 14 lines · Johannes Schindelin

Okay, let's try an analogy.

Imagine that a person is asking for directions to the train station. And the other person is replying by asking "did you know that this train station was built in 1878? It is actually quite interesting a story... [and then goes on to describe the history and what excites them about it]". Now, the first person tries again to ask for directions, again does not get an answer to that question, and is slowly starting to look at their watch. The second person, being completely oblivious to all of this, goes on with their wonderful story about the train station and its cultural heritage. So the first person walks a bit further to ask a third person, but the second person is not done yet and says "but you haven't heard me out! That's disrespectful!".

Quoted 5 lines

Just imagine for a minute how you would feel if you were the first person.

And that is how I feel asking for reviews about the Scalar patch series and then being forcefully dragged into that tangent about the build process.

At least to me, how this Makefile for Scalar should interact with the overall build process does not mesh well with the story about hwo direction to and history of the station are unrelated. If we plan to start from contrib/ and eventually want to make it a part of the core Git (i.e. "git scalar <subcmd> ..." becomes just like "git bisect <subcmd> ..."), we would eventually need to see the recipe needed for including "bisect" and "scalar" work the same way, no?

I am getting the impression that such a unified build process is Ævar wants to see at the end, I am not even sure if you do from the above "analogy". Cool down a bit, perhaps?

The following assumes that you share the goal of making "git scalar" just like "git bisect"---another first class citizen of Git toolbox, the user can choose to use it or the user may not have a need to interact with it, but it exists there by default and is not an opt-in add-on component.

I would understand it if your plan is to convert to a unified build procedure at the very end of the upstreaming process, and not while you populate contrib/ with more and more scalar stuff, because the Makefile bits for the entire scalar, while not yet upstreamed, has already been written as a separate procedure and having to convert the whole thing upfront before you can start trickle parts would mean you need to (re)start the process. And I would even be sympathetic if you felt it like a distraction.

Quoted 6 lines

But at least I view it as a step that needs to happen sometime between now and at the end. I do not yet have an opinion on which one is more pleasant, between (1) having to deal with a single Makefile that needs to be aware of two different locations *.[ch] lives in, and (2) having to deal with two Makefiles that duplicates definitions and risks them needlessly diverging.

For what it's worth what I had on top of this is not (1) or (2), but a (0): I.e. there isn't a contrib/scalar anymore, I moved:

    contrib/scalar/scalar.c -> scalar
    contrib/scalar/scalar.txt -> Documentation/scalar.txt
    contrib/scalar/t9099-scalar.sh -> t/t9099-scalar.sh

We build, test, and otherwise check (e.g. "make check-docs") it by default, what we don't do is install it unless you ask. You need to run:

# Or any other install* target
make install install-doc INSTALL_SCALAR=YesPlease

It could be be kept in contrib/scalar/ even with that sort of approach, and it would still be simpler than the two-Makefile approach.

But just moving the code, tests and documentation where everything else lives cuts down an all sorts of special cases, file globs in various places (e.g. doc lints) will just work and won't need adjustment.

Quoted 6 lines

I also would understand it if the reason why you want to keep the top-level Makefile as intact as possible because you sense a high probability that scalar will stay in contrib/ and even turn out to be a failure. Keeping the build procedure separated certainly will keep it easier to yank it out later. But I do not think such a case is quite likely.

For what it's worth the WIP patch(es) I have on top of it will probably make such a thing even easier, not that removing it from the tree would be much of a problem in either case. It's mostly a few lines added to lists in various places in the Makfile.

If I were to clean this up properly most of the changes would be teaching the Makefile that it can build N number of named top-level "special" commands that get dropped into bin/, not just the "git" we hardcode now.

If you're interested I could try to clean that up and send something on top, but given the tense-ness of the discussion & unrelated but relevant patch queue I've got outstanding wouldn't do so otherwise...

#109 lore raw

Re: Train station analogy, was Re: [PATCH v3 00/15] [RFC] Upstreaming the Scalar command

In reply to Johannes Schindelin
Quoted 41 lines · Johannes Schindelin

Hi Ævar,

Quoted 30 lines · Ævar Arnfjörð Bjarmason
Quoted 28 lines · Johannes Schindelin
Quoted 10 lines · Ævar Arnfjörð Bjarmason
Quoted 8 lines · Ævar Arnfjörð Bjarmason

In the summary I had on v1->v2 points 1-3 are for v2->v3, respectively, outstanding, addressed, outstanding:

    https://lore.kernel.org/git/[email protected]/

In addition the discussion ending here:
https://lore.kernel.org/git/[email protected]/

Quoted 8 lines
Quoted 4 lines

For that point: I think it's fair enough not to properly handle the cleanup case in "scalar clone", but perhaps add a note in the commit message that unlike "git clone" this is known not to clean after itself properly on ctrl+c?

Seeing [1] about the planned re-roll I have the above a shot a few days ago, see the original discussion at [2] (indirectly linked above).

There is a good reason why I did not engage in that tangent about deviating from the established contrib/*/Makefile paradigm: I find it particularly unrelated to what this here patch series is trying to accomplish, and I cannot bring myself to be interested in the proposed build system changes, either, because I do not see any benefit in the changes, only downsides.

Quoted 4 lines
Quoted 1 line

I find the distraction unnecessary.

Perhaps I'm reading too much between the lines here, so forgive any undue knee-jerk reaction.

Okay, let's try an analogy.

Quoted 11 lines

Imagine that a person is asking for directions to the train station. And the other person is replying by asking "did you know that this train station was built in 1878? It is actually quite interesting a story... [and then goes on to describe the history and what excites them about it]". Now, the first person tries again to ask for directions, again does not get an answer to that question, and is slowly starting to look at their watch. The second person, being completely oblivious to all of this, goes on with their wonderful story about the train station and its cultural heritage. So the first person walks a bit further to ask a third person, but the second person is not done yet and says "but you haven't heard me out! That's disrespectful!".

To be clear that's not what I said or meant. I wasn't saying you had to exhaustively hear out some argument or participate in a discussion you're not interested in.

I am saying that if you're soliciting feedback and you get some, and the person giving you the feedback sends you pings back, as I did here:

    https://lore.kernel.org/git/[email protected]/
    https://lore.kernel.org/git/[email protected]/
    https://lore.kernel.org/git/[email protected]/
    https://lore.kernel.org/git/[email protected]/

That it would be nice to at least reply with some brief comment to the effect that you're not personally interested in improving this area.

Now, shortly after you send this you re-rolled the v3
(https://lore.kernel.org/git/[email protected]/)
with a note of:

* Removed the [RFC] prefix because I did not hear any objections against putting this into contrib/.

I don't know if you wrote that after this reply, or really didn't see any of above, but that's a really inaccurate/misleading comment considering that context.

Quoted 17 lines

Just imagine for a minute how you would feel if you were the first person.

And that is how I feel asking for reviews about the Scalar patch series and then being forcefully dragged into that tangent about the build process.

I find the well-established paradigm to keep contrib/'s build procedures as confined to their own directory as possible the most reasonable way to handle the build by virtue of _not_ polluting the top-level Makefile unnecessarily. All of your objections strike me simply as personal viewpoints, not as technical arguments, and they fail to address this "pollution of the top-level Makefile" problem. I therefore strongly disagree with your suggestion that the build system should be changed, I would even argue that your suggestion should been dismissed on purely technical grounds, and I wish you hadn't forced me to say this as forcefully.

Quoted 3 lines

And even if I looked more favorably on your suggestion to change the build procedure, I find this distraction about the build as little constructive as the explanations about the train station's history above.

If we're indulging each other, here's my version of that train analogy:

We're in the business of selling a sugary drink that comes in a red can.

We've got a big factory with an attached train station, and all the trains that move our product are also red.

Now, some of our customers want the new purple colored sugary drink we've cooked up. A new purple factory's all set up for making them.

But for some reason the part of the business and distribution plans call for building a new purple train station parallel to our existing one.

All purple sugary drinks are expected to be moved on purple trains, all our conductors will need some slight re-training and switchover time to flip-flop between red and purple trains. Issues with purple production floors and purple workflows will need to be ironed out.

We did a survey of our customers and most of them weren't even aware that there was such a thing as train. Or perhaps they've seen other trains, but most haven't seen our trains.

Trains occasionally need field servicing, luckily our fleet of red trains is set up to carry its own spare parts. Purple trains can only be serviced with the assistance of a red train.

A product survey asking customers whether their enjoyment of the red or purple sugary drinks might be impacted by the color of the train they were shipped on only resulted in puzzled blank looks from the participants.

The current state of the purple train station and its fleet of purple trains is that it somewhat works with some hiccups. The currently built purple train station omitted any sort of train track for leaving the station though. It should be easy to add one, but...

The manager of the purple train project has been asked whether we can't just have our red machines in our red factory make the purple sugary drink, which we can then load on our fleet of red trains. Why do we need a new parallel rail system when we really care about customers drinking our delicious sugary drinks?

In case it's not obvious:

{red,purple} sugary drink = /usr/bin/git & /usr/bin/scalar {red,purple} train station = ./Makefile & ./contrib/scalar/Makefile train track for leaving the station = make install (AFAICT your
current patches have no installation mechanism)

Quoted 13 lines

Those
suggestions do succeed in derailing the conversation about how Git could scale better, how Scalar _does_ teach Git how to scale better, and about how to teach Git itself more and more of Scalar's tricks.

If you have ideas how to teach, say, git clone to perform a couple of Scalar's tricks, by all means, let's hear them, or even better, let's see those patches. If you want to change the build system, still, I cannot stop you from sending patches to that end to the Git mailing list, but please expect me to be uninterested in them in any way, and to prefer to spend my efforts to improve Git elsewhere. If you have other ideas how to improve on Scalar in a user-perceptible way, however, I am all ears again.

Quoted 2 lines

I hope this clarifies it, without the need to read between the lines, Johannes

Whatever the end goal of the patches you're sending part of them is proposing a given approach to go from A to B.

I find it odd to be claiming that the end-state is so important that we can't spare time to discuss some of the implementation details.

Particularly in this case, where I've sent a mostly working patch-on-top which I think should be clear from context I'd be willing to finish up, so it's not like it's just pointless nitpicking with no end-goal of a code improvement in sight.

#101 lore raw

Re: Train station analogy, was Re: [PATCH v3 00/15] [RFC] Upstreaming the Scalar command

In reply to Ævar Arnfjörð Bjarmason
Quoted 5 lines · Ævar Arnfjörð Bjarmason
Quoted 2 lines · Junio C Hamano

Johannes Schindelin <[email protected]> writes:

Quoted 8 lines
Quoted 8 lines

At least to me, how this Makefile for Scalar should interact with the overall build process does not mesh well with the story about hwo direction to and history of the station are unrelated. If we plan to start from contrib/ and eventually want to make it a part of the core Git (i.e. "git scalar <subcmd> ..." becomes just like "git bisect <subcmd> ..."), we would eventually need to see the recipe needed for including "bisect" and "scalar" work the same way, no?

We should definitely work to find a better way to describe our vision for how _the ideas in Scalar_ can be adopted into Git proper.

Before this series, we were adding functionality to Git that allowed Scalar to simplify to just a CLI that configures Git features. This submission allows that CLI to be available via an opt-in compile flag. This should allow more users to try out the ideas and perhaps we find the things that really work for people (and almost more importantly, the ideas that are _too_ opinionated).

But the way I see Scalar being fully incorporated into Git is not as a "git scalar <foo>" command or even having "scalar" be included by default. Instead, perhaps a new builtin would need to be created and its CLI would need to be presented and reviewed with significant attention to long-term support in the Git project. Having Scalar as a testing ground for these ideas seems like a positive way forward.

This is a big reason why we think that contrib/ is a good place for it to exist.

Quoted 3 lines
Quoted 3 lines

I am getting the impression that such a unified build process is Ævar wants to see at the end, I am not even sure if you do from the above "analogy". Cool down a bit, perhaps?

I agree that the temperature of this thread has gotten a bit heated. I think there is something valuable to be gained from each perspective, but not in a way that either has presented it.

Quoted 15 lines
Quoted 15 lines

The following assumes that you share the goal of making "git scalar" just like "git bisect"---another first class citizen of Git toolbox, the user can choose to use it or the user may not have a need to interact with it, but it exists there by default and is not an opt-in add-on component.

I would understand it if your plan is to convert to a unified build procedure at the very end of the upstreaming process, and not while you populate contrib/ with more and more scalar stuff, because the Makefile bits for the entire scalar, while not yet upstreamed, has already been written as a separate procedure and having to convert the whole thing upfront before you can start trickle parts would mean you need to (re)start the process. And I would even be sympathetic if you felt it like a distraction.

Quoted 6 lines
Quoted 6 lines

But at least I view it as a step that needs to happen sometime between now and at the end. I do not yet have an opinion on which one is more pleasant, between (1) having to deal with a single Makefile that needs to be aware of two different locations *.[ch] lives in, and (2) having to deal with two Makefiles that duplicates definitions and risks them needlessly diverging.

Since we already need to modify the root Makefile, I think having the root Makefile add the files from contrib/scalar from an optional flag is a great way to reduce duplication across multiple Makefiles while also maintaining the Scalar is compiled optionally.

One big goal is to minimize how often we need to update Scalar. I can see things like adjusting the recommended config once per release cycle based on which new features are available. I don't really want to be spending time updating the Makefile to match a contribution that was already carefully reviewed and tested. I also don't want to put the burden of updating contrib/scalar upon those contributors.

Quoted 6 lines

For what it's worth what I had on top of this is not (1) or (2), but a (0): I.e. there isn't a contrib/scalar anymore, I moved:

    contrib/scalar/scalar.c -> scalar>     contrib/scalar/scalar.txt -> Documentation/scalar.txt
    contrib/scalar/t9099-scalar.sh -> t/t9099-scalar.sh
Quoted 8 lines

We build, test, and otherwise check (e.g. "make check-docs") it by default, what we don't do is install it unless you ask. You need to run:

# Or any other install* target
make install install-doc INSTALL_SCALAR=YesPlease

It could be be kept in contrib/scalar/ even with that sort of approach, and it would still be simpler than the two-Makefile approach.

I think keeping it in contrib/scalar is best for now. But I do agree that a single Makefile has benefits.

One early suggestion from a while back was to modify git.c to handle the "scalar" executable as well as the "git" executable, specifically to reduce duplication handling options such as

-c config.key=value
-C worktree
--exec-path

and similar commands. While our duplication of the "-c" option does add similar code in a second place, these other options are less critical for Scalar, especially in its current version. I think refactoring the code in git.c to cater to the "scalar" executable is at least premature. If we want to pursue these other options in the future, then that refactoring could happen as a separate discussion after the rest of the build system and CLI have been figured out.

_Perhaps_ Johannes still had that level of integration in his head when responding to the single-Makefile recommendations.

Quoted 11 lines

But just moving the code, tests and documentation where everything else lives cuts down an all sorts of special cases, file globs in various places (e.g. doc lints) will just work and won't need adjustment.

Quoted 6 lines

I also would understand it if the reason why you want to keep the top-level Makefile as intact as possible because you sense a high probability that scalar will stay in contrib/ and even turn out to be a failure. Keeping the build procedure separated certainly will keep it easier to yank it out later. But I do not think such a case is quite likely.

Quoted 4 lines

For what it's worth the WIP patch(es) I have on top of it will probably make such a thing even easier, not that removing it from the tree would be much of a problem in either case. It's mostly a few lines added to lists in various places in the Makfile.

Do you have a version of these patches available for adaptation into this series? I'd like to take a look and see what it would look like to squash them into this series. Forgive me if I just missed the link. (I see the diff you posted earlier in this thread.)

Quoted 4 lines

If I were to clean this up properly most of the changes would be teaching the Makefile that it can build N number of named top-level "special" commands that get dropped into bin/, not just the "git" we hardcode now.

This is an interesting idea for revamping how adjacent tools are compiled and shipped with Git from contrib/ (or possibly elsewhere if we decided to start including more things as "blessed helpers".

As a complete aside: I'm interested in using the sparse-checkout feature as I work on the Git codebase, just to make sure I hit pain points before any other user.

This is the best that I could do for my purposes:

$ git sparse-checkout list
.github
Documentation
builtin
compat
contrib/scalar
ewah
git-gui
gitk-git
gitk-gui
gitweb
mergetools
negotiator
perl
po
refs
sha1dc
sha256
t
templates
trace2
xdiff

And 'git status' reports that this includes 97% of the tracked files. Perhaps there are ways to make this be smaller by having make skip building things like git-gui if the directory doesn't exist. Another idea would be to skip any logic around translating messages if the 'po' directory is missing.

The reason I bring this up is that I'm interested in finding ways to make our build system be streamlined a bit using the presence of directories as a way to opt in/out of certain build outputs. Since Scalar is being added as a new component, this is a good opportunity to establish a pattern that works for this effort, too.

Thanks,
-Stolee

#104 lore raw

Re: Train station analogy, was Re: [PATCH v3 00/15] [RFC] Upstreaming the Scalar command

In reply to Junio C Hamano
Quoted 6 lines · Junio C Hamano

But at least I view it as a step that needs to happen sometime between now and at the end. I do not yet have an opinion on which one is more pleasant, between (1) having to deal with a single Makefile that needs to be aware of two different locations *.[ch] lives in, and (2) having to deal with two Makefiles that duplicates definitions and risks them needlessly diverging.

FWIW, I am leaning towards the latter, with the assumption that this may take more than a cycle to cook in contrib/.

Adding the Makefile bits to the top-level and keeping the topic in 'next' means all the topics that pass the pipeline will need to be written in such a way that its Makefile change works well with and without the unified Makefile bits from this topic, an additional burden on other topics this topic would impose.

So it is understandable to want to keep the changes to the top-level Makefile to the minimum, even if it may mean that it requires more effort in the end to clean things up when the topic graduates.

An alternative would be to bypass the contrib/ phase and start as a new subcommand that is first-class citizen from day one and let it spend as much time as it needs to mature. It would burden the topics that pass the pipeline while this is cooking the same way as having a unified build procedure in the top-level Makefile, of course, though.

#102 lore raw

Re: Train station analogy, was Re: [PATCH v3 00/15] [RFC] Upstreaming the Scalar command

In reply to Derrick Stolee
Quoted 8 lines · Derrick Stolee

We should definitely work to find a better way to describe our vision for how _the ideas in Scalar_ can be adopted into Git proper. ..

But the way I see Scalar being fully incorporated into Git is not as a "git scalar <foo>" command or even having "scalar" be included by default.

I agree 1000%. It may be convenient for existing scalar users to have a way they can stick with the CLI that they are used to, but before we add this functionality to git proper, let's please make sure git users get a CLI which makes sense and not dictated by history.

We already have enough people who complain that the git interface design is hard to understand but which can't be changed because we bias our UI design in favor of existing users as opposed new users. Given that I'm an existing user, I'm actually not complaining, but I do recognize the validity of the complaints that for example, git reset does three different, mostly unrelated things.

Given that the existing scalar users are used to "scalar <foo>" and not "git scalar <foo">, I'd gently suggest that it's better that there be an existing compatibility program which translates "scalar <foo>" to whatever git command(s) makes sense, as opposed to optimizing for the simplicity of said program so that all forms of "scalar <foo>" should get translated to "git scalar <foo>".

So for example, if we are inside a mono repro, it would seem to me that "git commit" should automatically do the right thing, as opposed to imposing cognitive load on the user to know when they are supposed to type "git commit ..." versus "git scalar commit ..." versus "scalar commit".

Please, let's design the UI for scalar integration into git with deep sympathy for the users, and *not* the convenience of the compatibility script for the existing scalar CLI users.

- Ted

#103 lore raw

Re: Train station analogy, was Re: [PATCH v3 00/15] [RFC] Upstreaming the Scalar command

In reply to Derrick Stolee
Quoted 18 lines · Derrick Stolee
Quoted 5 lines · Ævar Arnfjörð Bjarmason
Quoted 2 lines · Junio C Hamano

Johannes Schindelin <[email protected]> writes:

Quoted 8 lines
Quoted 8 lines

At least to me, how this Makefile for Scalar should interact with the overall build process does not mesh well with the story about hwo direction to and history of the station are unrelated. If we plan to start from contrib/ and eventually want to make it a part of the core Git (i.e. "git scalar <subcmd> ..." becomes just like "git bisect <subcmd> ..."), we would eventually need to see the recipe needed for including "bisect" and "scalar" work the same way, no?

We should definitely work to find a better way to describe our vision for how _the ideas in Scalar_ can be adopted into Git proper.

Quoted 6 lines

Before this series, we were adding functionality to Git that allowed Scalar to simplify to just a CLI that configures Git features. This submission allows that CLI to be available via an opt-in compile flag. This should allow more users to try out the ideas and perhaps we find the things that really work for people (and almost more importantly, the ideas that are _too_ opinionated).

Yeah that makes sense. I think it looks like a useful command & it's already useful to users.

I haven't been suggesting any changes to what gets installed here, FWIW I think we could be even more aggressive on that front, e.g. shipping it unconditionally in libexec, maybe with an optional switch for /usr/bin/scalar, or to ship "scalar" symlinked to "git" and have it route to the top-level scalar command depending on argv.

I dabbled in that a bit locally, FWIW it seems if anything even easier to do than the approaches we've discussed so far, but I wanted to focus on providing the same behavior in terms of build system maintenance.

Quoted 6 lines

But the way I see Scalar being fully incorporated into Git is not as a "git scalar <foo>" command or even having "scalar" be included by default. Instead, perhaps a new builtin would need to be created and its CLI would need to be presented and reviewed with significant attention to long-term support in the Git project. Having Scalar as a testing ground for these ideas seems like a positive way forward.

*Nod*

Quoted 2 lines

This is a big reason why we think that contrib/ is a good place for it to exist.

Here's where you and Johannes lose me. There's some rationale in your minds for why sticking it in contrib is the obvious way to go. So far you've been describing how it'll look to users etc, how it's arranged in our source tree only matters to git.git developers.

I think the actual reason is to carve in advance some subjective ownership/apartness or whatever for this thing, if that's the case I think documentation/commit messages would also work.

I really don't care much if something lives in contrib or not in the abstract, but various integration around builds in Makefile makes that much easier in practice, and if "make install" looks the same...

Quoted 7 lines
Quoted 3 lines
Quoted 3 lines

I am getting the impression that such a unified build process is Ævar wants to see at the end, I am not even sure if you do from the above "analogy". Cool down a bit, perhaps?

I agree that the temperature of this thread has gotten a bit heated. I think there is something valuable to be gained from each perspective, but not in a way that either has presented it.

Thanks, hopefully we can keep it jovial going forward. If you've got any (either on-list or off-list) feedback about how I can improve my side of that it would be most welcome.

Quoted 27 lines
Quoted 15 lines
Quoted 15 lines

The following assumes that you share the goal of making "git scalar" just like "git bisect"---another first class citizen of Git toolbox, the user can choose to use it or the user may not have a need to interact with it, but it exists there by default and is not an opt-in add-on component.

I would understand it if your plan is to convert to a unified build procedure at the very end of the upstreaming process, and not while you populate contrib/ with more and more scalar stuff, because the Makefile bits for the entire scalar, while not yet upstreamed, has already been written as a separate procedure and having to convert the whole thing upfront before you can start trickle parts would mean you need to (re)start the process. And I would even be sympathetic if you felt it like a distraction.

Quoted 6 lines
Quoted 6 lines

But at least I view it as a step that needs to happen sometime between now and at the end. I do not yet have an opinion on which one is more pleasant, between (1) having to deal with a single Makefile that needs to be aware of two different locations *.[ch] lives in, and (2) having to deal with two Makefiles that duplicates definitions and risks them needlessly diverging.

Since we already need to modify the root Makefile, I think having the root Makefile add the files from contrib/scalar from an optional flag is a great way to reduce duplication across multiple Makefiles while also maintaining the Scalar is compiled optionally.

Quoted 7 lines

One big goal is to minimize how often we need to update Scalar. I can see things like adjusting the recommended config once per release cycle based on which new features are available. I don't really want to be spending time updating the Makefile to match a contribution that was already carefully reviewed and tested. I also don't want to put the burden of updating contrib/scalar upon those contributors.

I'd think not having large parts of t/Makefile & Makefile should ease that maintenance burden for you & others.

Quoted 9 lines
Quoted 9 lines

For what it's worth what I had on top of this is not (1) or (2), but a (0): I.e. there isn't a contrib/scalar anymore, I moved:

    contrib/scalar/scalar.c -> scalar>     contrib/scalar/scalar.txt -> Documentation/scalar.txt
    contrib/scalar/t9099-scalar.sh -> t/t9099-scalar.sh

We build, test, and otherwise check (e.g. "make check-docs") it by default, what we don't do is install it unless you ask. You need to run:

Quoted 8 lines
Quoted 5 lines

# Or any other install* target
make install install-doc INSTALL_SCALAR=YesPlease

It could be be kept in contrib/scalar/ even with that sort of approach, and it would still be simpler than the two-Makefile approach.

I think keeping it in contrib/scalar is best for now. But I do agree that a single Makefile has benefits.

I noted the "why contrib" above.

Quoted 8 lines

One early suggestion from a while back was to modify git.c to handle the "scalar" executable as well as the "git" executable, specifically to reduce duplication handling options such as

-c config.key=value
-C worktree
--exec-path

Quoted 8 lines

and similar commands. While our duplication of the "-c" option does add similar code in a second place, these other options are less critical for Scalar, especially in its current version. I think refactoring the code in git.c to cater to the "scalar" executable is at least premature. If we want to pursue these other options in the future, then that refactoring could happen as a separate discussion after the rest of the build system and CLI have been figured out.

As noted above that seems like a sensible way forward, I hadn't noticed how much of git.c's setup was copied into scalar.c.

It seems to me that it wouldn't be that hard, on the order of the existing setup code or less. I.e. just make "git.c" learn that it may be running some arbitrary command name, and do some options parsing, but and finally dispatch to a cmd_scalar(). IOW mostly like a built-in.

Quoted 19 lines

_Perhaps_ Johannes still had that level of integration in his head when responding to the single-Makefile recommendations.

Quoted 11 lines

But just moving the code, tests and documentation where everything else lives cuts down an all sorts of special cases, file globs in various places (e.g. doc lints) will just work and won't need adjustment.

Quoted 6 lines

I also would understand it if the reason why you want to keep the top-level Makefile as intact as possible because you sense a high probability that scalar will stay in contrib/ and even turn out to be a failure. Keeping the build procedure separated certainly will keep it easier to yank it out later. But I do not think such a case is quite likely.

Quoted 4 lines

For what it's worth the WIP patch(es) I have on top of it will probably make such a thing even easier, not that removing it from the tree would be much of a problem in either case. It's mostly a few lines added to lists in various places in the Makfile.

Quoted 4 lines

Do you have a version of these patches available for adaptation into this series? I'd like to take a look and see what it would look like to squash them into this series. Forgive me if I just missed the link. (I see the diff you posted earlier in this thread.)

I've got it at
https://github.com/avar/git/tree/avar-dscho/scalar-the-beginning-normalize-Makefile

Not very ML-ready, and soft-depends on some other Makefile cleanups I thought I'd do & still haven't untangled and submitted. Soft-depends as in this can easily be done on master, but some of the variable names etc. are quite confusing there.

But you should be able to check it out, it'll build, test and install if you run "install" with "INSTALL_SCALAR=Y".

You may run into on everly eager new (but unrelated to this, I just merged it on top) Makefile assertion I'm experimenting with, just comment out the relevant line in the Makefile if that happens, i.e. something like this error:

Makefile:3608: *** "please sort and de-duplicate BUILT_INS_EXTRA!". Stop.

Quoted 38 lines
Quoted 4 lines

If I were to clean this up properly most of the changes would be teaching the Makefile that it can build N number of named top-level "special" commands that get dropped into bin/, not just the "git" we hardcode now.

This is an interesting idea for revamping how adjacent tools are compiled and shipped with Git from contrib/ (or possibly elsewhere if we decided to start including more things as "blessed helpers".

As a complete aside: I'm interested in using the sparse-checkout feature as I work on the Git codebase, just to make sure I hit pain points before any other user.

This is the best that I could do for my purposes:

$ git sparse-checkout list
.github
Documentation
builtin
compat
contrib/scalar
ewah
git-gui
gitk-git
gitk-gui
gitweb
mergetools
negotiator
perl
po
refs
sha1dc
sha256
t
templates
trace2
xdiff
Quoted 5 lines

And 'git status' reports that this includes 97% of the tracked files. Perhaps there are ways to make this be smaller by having make skip building things like git-gui if the directory doesn't exist. Another idea would be to skip any logic around translating messages if the 'po' directory is missing.

For git-gui in particular NO_TCLTK=Y should do it.

Quoted 6 lines

The reason I bring this up is that I'm interested in finding ways to make our build system be streamlined a bit using the presence of directories as a way to opt in/out of certain build outputs. Since Scalar is being added as a new component, this is a good opportunity to establish a pattern that works for this effort, too.

Sure, the hard part isn't that you can't grep out nonexisting files or directories when building where we now use a glob.

It's that everything downstream of that, i.e. tests, installation etc. is going to have to work properly in the face of arbitrary parts of what the developer who tested the code expected going missing.

Which is why we've generally carved out very specific things, usually along the boundaries of installed dependencies.

#112 lore raw

Re: [PATCH v4 01/15] scalar: create a rudimentary executable

In reply to Johannes Schindelin via GitGitGadget
Quoted 11 lines · Johannes Schindelin via GitGitGadget
contrib/scalar/scalar.c new file +11 -0
+int cmd_main(int argc, const char **argv)
+{
+	struct strbuf scalar_usage = STRBUF_INIT;
+	int i;
+
+	if (argc > 1) {
+		argv++;
+		argc--;
+
+		for (i = 0; builtins[i].name; i++)
+			if (!strcmp(builtins[i].name, argv[0]))

contrib/scalar/scalar.c line 36

Quoted 10 lines
contrib/scalar/scalar.c new file +10 -0
+				return !!builtins[i].fn(argc, argv);
+	}
+
+	strbuf_addstr(&scalar_usage,
+		      N_("scalar <command> [<options>]\n\nCommands:\n"));
+	for (i = 0; builtins[i].name; i++)
+		strbuf_addf(&scalar_usage, "\t%s\n", builtins[i].name);
+
+	usage(scalar_usage.buf);
+}

In 04/15 you continue and use the parse-options.c API, but here it's the usage.c API, which is generally being phased out. Any reason for the difference? It's preferrable not to add new usage() users if we can help it, I think we'll eventually want to remove it.

#113 lore raw

Re: [PATCH v4 01/15] scalar: create a rudimentary executable

Quoted 2 lines · Ævar Arnfjörð Bjarmason

... the
usage.c API, which is generally being phased out.

That is news to me. Any reason why you think so?

#114 lore raw

Re: [PATCH v4 01/15] scalar: create a rudimentary executable

Quoted 6 lines · Junio C Hamano
Quoted 2 lines · Ævar Arnfjörð Bjarmason

... the
usage.c API, which is generally being phased out.

That is news to me. Any reason why you think so?

Perhaps better phrased as "generally going unused where we're using parse-options.c", although some quick historical trends I ran as an ad-hoc show usage() standing still since v1.6.0 in number of builtin*.c files[1], v.s. a 3x growth in usage_with_options() since then[2].

But in this case we're using an ad-hoc parser in cmd_main(), seemingly because it ends up copy/pasting a very small part of git.ci functionality over there, which is something that's been discussed as something we should move over to parse_options() sooner than later.

It looks like a better approach to just use parse_options() consistently in scalar.c, as e.g. commit-graph.c, stash.c, multi-pack-index.c etc. that all implement a similar cmd/subcommand pattern do.

The end-state of duplicating the "-C" and "-c" options from git.c can then easily be handled by parse-options.c, IIRC the stumbling point in migrating over git.c was some of the statefulness of other parts potentially needing incremenatl parsing (i.e. via parse_options_step() and friends).

1. $ parallel "printf "%s: " {} && git grep -l '\busage\(' {} -- 'builtin*.c' | wc -l" ::: v1.{1..9}.0 v2.{0..32}.0 v1.1.0:0
v1.2.0:0
v1.3.0:0
v1.4.0:20
v1.5.0:51
v1.7.0:35
v1.6.0:40
v1.8.0:34
v2.1.0:32
v1.9.0:33
v2.0.0:33
v2.2.0:32
v2.3.0:32
v2.4.0:32
v2.5.0:32
v2.6.0:31
v2.7.0:31
v2.9.0:29
v2.8.0:30
v2.11.0:29
v2.10.0:29
v2.12.0:29
v2.13.0:29
v2.14.0:31
v2.15.0:31
v2.16.0:31
v2.17.0:31
v2.19.0:32
v2.18.0:31
v2.22.0:31
v2.21.0:32
v2.24.0:31
v2.23.0:31
v2.20.0:32
v2.25.0:30
v2.26.0:30
v2.27.0:30
v2.32.0:29
v2.31.0:30
v2.29.0:31
v2.28.0:29
v2.30.0:31

2. $ parallel "printf "%s: " {} && git grep -l '\busage_with_options\(' {} -- 'builtin*.c' | wc -l" ::: v1.{1..9}.0 v2.{0..32}.0 v1.1.0:0
v1.2.0:0
v1.3.0:0
v1.4.0:0
v1.5.0:0
v1.6.0:19
v1.7.0:33
v1.8.0:42
v2.0.0:42
v1.9.0:42
v2.1.0:43
v2.2.0:43
v2.3.0:43
v2.4.0:43
v2.5.0:44
v2.6.0:45
v2.8.0:44
v2.10.0:45
v2.7.0:44
v2.9.0:45
v2.11.0:45
v2.12.0:45
v2.13.0:46
v2.14.0:47
v2.15.0:47
v2.16.0:47
v2.17.0:47
v2.19.0:50
v2.18.0:49
v2.20.0:52
v2.21.0:52
v2.22.0:53
v2.24.0:54
v2.23.0:54
v2.25.0:56
v2.26.0:55
v2.27.0:55
v2.28.0:55
v2.30.0:58
v2.32.0:60
v2.29.0:58
v2.31.0:58

#115 lore raw

Re: [PATCH v4 01/15] scalar: create a rudimentary executable

Quoted 11 lines · Ævar Arnfjörð Bjarmason
Quoted 6 lines · Junio C Hamano
Quoted 2 lines · Ævar Arnfjörð Bjarmason

... the
usage.c API, which is generally being phased out.

That is news to me. Any reason why you think so?

Perhaps better phrased as "generally going unused where we're using parse-options.c", ...

Ah, if it was merely an observation of the general trend, then I agree. My reaction was primarily "Oh, why is somebody all of a sudden setting a project decision unilaterally, especially when even I do not do so very often myself? Did I miss recent discussion that resulted in an update to Documentation/ meant for developers?"

Thanks.

#119 lore raw

Re: [PATCH v4 04/15] scalar: 'register' sets recommended config and starts maintenance

In reply to Derrick Stolee via GitGitGadget

On Tue, Sep 14, 2021 at 7:39 AM Derrick Stolee via GitGitGadget <[email protected]> wrote:
...

Quoted 17 lines
contrib/scalar/scalar.c +17 -0
+static int set_recommended_config(void)
+{
+       struct {
+               const char *key;
+               const char *value;
+       } config[] = {
+               { "am.keepCR", "true" },
+               { "core.FSCache", "true" },
+               { "core.multiPackIndex", "true" },
+               { "core.preloadIndex", "true" },
+#ifndef WIN32
+               { "core.untrackedCache", "true" },
+#else
+               /*
+                * Unfortunately, Scalar's Functional Tests demonstrated
+                * that the untracked cache feature is unreliable on Windows
+                * (which is a bummer because that platform would benefit the

contrib/scalar/scalar.c line 143

Quoted 10 lines
contrib/scalar/scalar.c +10 -0
+                * most from it). For some reason, freshly created files seem
+                * not to update the directory's `lastModified` time
+                * immediately, but the untracked cache would need to rely on
+                * that.
+                *
+                * Therefore, with a sad heart, we disable this very useful
+                * feature on Windows.
+                */
+               { "core.untrackedCache", "false" },
+#endif

Interesting. (I'm somewhat leery of the untrackedCache just knowing that it used to operate despite an exponential number of visits to files (exponential in depth of directories) and getting different answers with different visits, making me feel like it was black magic that it ever worked and wondering what kind of corner case issues still lurk with it. See e.g.
https://lore.kernel.org/git/CABPp-BFiwzzUgiTj_zu+vF5x20L0=1cf25cHwk7KZQj2YkVzXw@mail.gmail.com/)

contrib/scalar/scalar.c line 145

Quoted 2 lines
contrib/scalar/scalar.c +2 -0
+               { "core.logAllRefUpdates", "true" },
+               { "credential.https://dev.azure.com.useHttpPath", "true" },

Not only opinionated, but special configuration for certain sites? I'm not complaining, just slightly surprised.

contrib/scalar/scalar.c line 150

Quoted 5 lines
contrib/scalar/scalar.c +5 -0
+               { "credential.validate", "false" }, /* GCM4W-only */
+               { "gc.auto", "0" },
+               { "gui.GCWarning", "false" },
+               { "index.threads", "true" },
+               { "index.version", "4" },

I take it your users don't make use of jgit? (Users aren't using jgit directly here, at least not to my knowledge, but multiple gradle plugins do.) I tried turning this on a while back, and quickly got multiple reports of problems because jgit didn't understand the index. I had to turn it off and send out various PSAs on how to recover.

contrib/scalar/scalar.c line 152

Quoted 2 lines
contrib/scalar/scalar.c +2 -0
+               { "merge.stat", "false" },
+               { "merge.renames", "false" },

Is this just historical and not needed anymore, is it here just for a little longer and you are planning on transitioning away from this, or are you still set on this setting?

Quoted 1 line

+ { "pack.useBitmaps", "false" },

I don't understand anything bitmap related, but I thought they were performance related, so I'm surprised by this one. Is there a reason for this one? (Is it handled by maintenance instead?)

contrib/scalar/scalar.c line 157

Quoted 4 lines
contrib/scalar/scalar.c +4 -0
+               { "pack.useSparse", "true" },
+               { "receive.autoGC", "false" },
+               { "reset.quiet", "true" },
+               { "feature.manyFiles", "false" },

If you simply set core.untrackedCache to false _after_ setting feature.manyFiles to true, would it make sense to switch this? (Or does it matter, since you've already individually set all the config settings that this one would set?)

contrib/scalar/scalar.c line 169

Quoted 12 lines
contrib/scalar/scalar.c +12 -0
+               { "feature.experimental", "false" },
+               { "fetch.unpackLimit", "1" },
+               { "fetch.writeCommitGraph", "false" },
+#ifdef WIN32
+               { "http.sslBackend", "schannel" },
+#endif
+               { "status.aheadBehind", "false" },
+               { "commitGraph.generationVersion", "1" },
+               { "core.autoCRLF", "false" },
+               { "core.safeCRLF", "false" },
+               { NULL, NULL },
+       };

Are there easy-ish ways for other groups of users to adopt scalar but change the list of config settings (e.g. index.version and merge.renames) in some common way for all those users?

#122 lore raw

Re: [PATCH v4 04/15] scalar: 'register' sets recommended config and starts maintenance

In reply to Derrick Stolee via GitGitGadget

Sorry, one more thing...

On Tue, Sep 14, 2021 at 7:39 AM Derrick Stolee via GitGitGadget <[email protected]> wrote:
...

contrib/scalar/scalar.c line 53

Quoted 3 lines
contrib/scalar/scalar.c +3 -0
+               /* check if currently in enlistment root with src/ workdir */
+               strbuf_addstr(&path, "/src/.git");
+               if (is_git_directory(path.buf)) {

...and...

contrib/scalar/scalar.c line 68

Quoted 3 lines
contrib/scalar/scalar.c +3 -0
+               /* check if currently in workdir */
+               strbuf_addstr(&path, "/.git");
+               if (is_git_directory(path.buf)) {

Do these two checks suggest that only a primary worktree can be enlisted with scalar? (Is git-worktree generally incompatible?)

#25 lore raw

Re: [PATCH 08/15] scalar: implement the `clone` subcommand

In reply to Johannes Schindelin via GitGitGadget
Quoted 1 line · Johannes Schindelin via GitGitGadget

...

Quoted 7 lines

COMMANDS
--------

contrib/scalar/scalar.txt +4 -0
+Clone
+~~~~~
+
+clone [<options>] <url> [<enlistment>]::

contrib/scalar/scalar.txt an existing Git worktree with Scalar whose name is not `src`, the enlistment · line 52

Quoted 10 lines
contrib/scalar/scalar.txt +10 -0
+    Clones the specified repository, similar to linkgit:git-clone[1]. By
+    default, only commit and tree objects are cloned. Once finished, the
+    worktree is located at `<enlistment>/src`.
++
+The sparse-checkout feature is enabled (except when run with `--full-clone`)
+and the only files present are those in the top-level directory. Use
+`git sparse-checkout set` to expand the set of directories you want to see,
+or `git sparse-checkout disable` to expand to all files (see
+linkgit:git-sparse-checkout[1] for more details). You can explore the
+subdirectories outside your sparse-checkout by using `git ls-tree HEAD`.

Should this be git ls-tree [-r] HEAD? Do you expect people to just add directories that are found immediately under the toplevel, rather than some that are a bit deeper?

#131 lore raw

Re: [PATCH v4 11/15] scalar: allow reconfiguring an existing enlistment

In reply to Johannes Schindelin via GitGitGadget
Quoted 83 lines · Johannes Schindelin via GitGitGadget

From: Johannes Schindelin <[email protected]>

This comes in handy during Scalar upgrades, or when config settings were messed up by mistake.

---
 contrib/scalar/scalar.c          | 79 +++++++++++++++++++++-----------
 contrib/scalar/scalar.txt        |  8 ++++
 contrib/scalar/t/t9099-scalar.sh |  8 ++++
 3 files changed, 67 insertions(+), 28 deletions(-)
contrib/scalar/scalar.c +24 -24
index 8a11f390251..1fff7eb7c12 100644
@@ -115,18 +115,20 @@ static int run_git(const char *arg, ...)
        return res;
 }
 
-static int set_recommended_config(void)
+static int set_recommended_config(int reconfigure)
 {
        struct {
                const char *key;
                const char *value;
+               int overwrite_on_reconfigure;
        } config[] = {
-               { "am.keepCR", "true" },
-               { "core.FSCache", "true" },
-               { "core.multiPackIndex", "true" },
-               { "core.preloadIndex", "true" },
+               /* Required */
+               { "am.keepCR", "true", 1 },
+               { "core.FSCache", "true", 1 },
+               { "core.multiPackIndex", "true", 1 },
+               { "core.preloadIndex", "true", 1 },
 #ifndef WIN32
-               { "core.untrackedCache", "true" },
+               { "core.untrackedCache", "true", 1 },
 #else
                /*
                 * Unfortunately, Scalar's Functional Tests demonstrated
@@ -140,28 +142,29 @@ static int set_recommended_config(void)
                 * Therefore, with a sad heart, we disable this very useful
                 * feature on Windows.
                 */
-               { "core.untrackedCache", "false" },
+               { "core.untrackedCache", "false", 1 },
 #endif
-               { "core.logAllRefUpdates", "true" },
-               { "credential.https://dev.azure.com.useHttpPath", "true" },
-               { "credential.validate", "false" }, /* GCM4W-only */
-               { "gc.auto", "0" },
-               { "gui.GCWarning", "false" },
-               { "index.threads", "true" },
-               { "index.version", "4" },
-               { "merge.stat", "false" },
-               { "merge.renames", "false" },
-               { "pack.useBitmaps", "false" },
-               { "pack.useSparse", "true" },
-               { "receive.autoGC", "false" },
-               { "reset.quiet", "true" },
-               { "feature.manyFiles", "false" },
-               { "feature.experimental", "false" },
-               { "fetch.unpackLimit", "1" },
-               { "fetch.writeCommitGraph", "false" },
+               { "core.logAllRefUpdates", "true", 1 },
+               { "credential.https://dev.azure.com.useHttpPath", "true", 1 },
+               { "credential.validate", "false", 1 }, /* GCM4W-only */
+               { "gc.auto", "0", 1 },
+               { "gui.GCWarning", "false", 1 },
+               { "index.threads", "true", 1 },
+               { "index.version", "4", 1 },
+               { "merge.stat", "false", 1 },
+               { "merge.renames", "false", 1 },
+               { "pack.useBitmaps", "false", 1 },
+               { "pack.useSparse", "true", 1 },
+               { "receive.autoGC", "false", 1 },
+               { "reset.quiet", "true", 1 },
+               { "feature.manyFiles", "false", 1 },
+               { "feature.experimental", "false", 1 },

contrib/scalar/scalar.c static int set_recommended_config(void) · line 170

Quoted 10 lines
contrib/scalar/scalar.c +4 -1
+               { "fetch.unpackLimit", "1", 1 },
+               { "fetch.writeCommitGraph", "false", 1 },
 #ifdef WIN32
-               { "http.sslBackend", "schannel" },
+               { "http.sslBackend", "schannel", 1 },
 #endif
+               /* Optional */
                { "status.aheadBehind", "false" },
                { "commitGraph.generationVersion", "1" },
                { "core.autoCRLF", "false" },

Now you have optional settings...but index.version and merge.renames aren't among them?? Why are those required? (...and to go a step further; should merge.renames even be off in a merge-ort world?)

#135 lore raw

Re: [PATCH v4 13/15] scalar: implement the `delete` command

In reply to Matthew John Cheetham via GitGitGadget
Quoted 11 lines · Matthew John Cheetham via GitGitGadget

From: Matthew John Cheetham <[email protected]>

Delete an enlistment by first unregistering the repository and then deleting the enlistment directory (usually the directory containing the worktree src/ directory).

On Windows, if the current directory is inside the enlistment's directory, change to the parent of the enlistment directory, to allow us to delete the enlistment (directories used by processes e.g. as current working directories cannot be deleted on Windows).

But if the current directory is inside the enlistment's directory, didn't that happen because the parent process' current directory was inside the enlistment directory? Or was there some kind of directory switching that scalar itself was doing causing it to be inside the enlistment directory?

If the the current directory was inside the enlistment's directory because it inherited a parent process' current directory, wouldn't that also prevent deleting it? If so, should there be a special check for that case and pre-emptively returning an error rather than attempting the recursive directory deletion and just spitting out an error when it fails?

(Also seems slightly related to
https://github.com/gitgitgadget/git/pull/1037, which I'll submit as soon as en/removing_untracked_fixes hits next.)

Quoted 143 lines
---
 contrib/scalar/scalar.c          | 55 ++++++++++++++++++++++++++++++++
 contrib/scalar/scalar.txt        |  8 +++++
 contrib/scalar/t/t9099-scalar.sh |  9 ++++++
 3 files changed, 72 insertions(+)
contrib/scalar/scalar.c +55 -0
index 67fa5305225..00bedb0bf66 100644
@@ -8,6 +8,7 @@
 #include "config.h"
 #include "run-command.h"
 #include "refs.h"
+#include "dir.h"
 
 /*
  * Remove the deepest subdirectory in the provided path string. Path must not
@@ -334,6 +335,33 @@ static char *remote_default_branch(const char *url)
        return NULL;
 }
 
+static int delete_enlistment(struct strbuf *enlistment)
+{
+#ifdef WIN32
+       struct strbuf parent = STRBUF_INIT;
+#endif
+
+       if (unregister_dir())
+               die(_("failed to unregister repository"));
+
+#ifdef WIN32
+       /*
+        * Change the current directory to one outside of the enlistment so
+        * that we may delete everything underneath it.
+        */
+       strbuf_addbuf(&parent, enlistment);
+       strbuf_parent_directory(&parent);
+       if (chdir(parent.buf) < 0)
+               die_errno(_("could not switch to '%s'"), parent.buf);
+       strbuf_release(&parent);
+#endif
+
+       if (remove_dir_recursively(enlistment, 0))
+               die(_("failed to delete enlistment directory"));
+
+       return 0;
+}
+
 static int cmd_clone(int argc, const char **argv)
 {
        const char *branch = NULL;
@@ -694,6 +722,32 @@ static int cmd_unregister(int argc, const char **argv)
        return unregister_dir();
 }
 
+static int cmd_delete(int argc, const char **argv)
+{
+       struct option options[] = {
+               OPT_END(),
+       };
+       const char * const usage[] = {
+               N_("scalar delete <enlistment>"),
+               NULL
+       };
+       struct strbuf enlistment = STRBUF_INIT;
+       int res = 0;
+
+       argc = parse_options(argc, argv, NULL, options,
+                            usage, 0);
+
+       if (argc != 1)
+               usage_with_options(usage, options);
+
+       setup_enlistment_directory(argc, argv, usage, options, &enlistment);
+
+       res = delete_enlistment(&enlistment);
+       strbuf_release(&enlistment);
+
+       return res;
+}
+
 static struct {
        const char *name;
        int (*fn)(int, const char **);
@@ -704,6 +758,7 @@ static struct {
        { "unregister", cmd_unregister },
        { "run", cmd_run },
        { "reconfigure", cmd_reconfigure },
+       { "delete", cmd_delete },
        { NULL, NULL},
 };
 
contrib/scalar/scalar.txt +8 -0
index 2fa96fcabc6..6fc57707718 100644
@@ -14,6 +14,7 @@ scalar register [<enlistment>]
 scalar unregister [<enlistment>]
 scalar run ( all | config | commit-graph | fetch | loose-objects | pack-files ) [<enlistment>]
 scalar reconfigure [ --all | <enlistment> ]
+scalar delete <enlistment>
 
 DESCRIPTION
 -----------
@@ -127,6 +128,13 @@ reconfigure the enlistment.
 With the `--all` option, all enlistments currently registered with Scalar
 will be reconfigured. Use this option after each Scalar upgrade.
 
+Delete
+~~~~~~
+
+delete <enlistment>::
+       This subcommand lets you delete an existing Scalar enlistment from your
+       local file system, unregistering the repository.
+
 SEE ALSO
 --------
 linkgit:git-clone[1], linkgit:git-maintenance[1].
contrib/scalar/t/t9099-scalar.sh +9 -1
index 5fe7fabd0e5..7e8771d0eff 100755
@@ -76,4 +76,13 @@ test_expect_success 'scalar reconfigure' '
        test true = "$(git -C one/src config core.preloadIndex)"
 '
 
+test_expect_success 'scalar delete without enlistment shows a usage' '
+       test_expect_code 129 scalar delete
+'
+
+test_expect_success 'scalar delete with enlistment' '
+       scalar delete cloned &&
+       test_path_is_missing cloned
+'
+
 test_done
--

gitgitgadget

#120 lore raw

Re: [PATCH v4 04/15] scalar: 'register' sets recommended config and starts maintenance

In reply to Elijah Newren
Quoted 8 lines · Elijah Newren

On Tue, Sep 14, 2021 at 7:39 AM Derrick Stolee via GitGitGadget <[email protected]> wrote:
...

Quoted 1 line

+ { "pack.useBitmaps", "false" },

I don't understand anything bitmap related, but I thought they were performance related, so I'm surprised by this one. Is there a reason for this one? (Is it handled by maintenance instead?)

I don't know why Derrick did this, but there's still AFAIK cases where bitmaps are worse than not in the context of a client (the scalar use-case), see the rabbit hole starting at:
https://lore.kernel.org/git/[email protected]/

#105 lore raw

Re: Train station analogy, was Re: [PATCH v3 00/15] [RFC] Upstreaming the Scalar command

In reply to Junio C Hamano

Hi Junio,

Quoted 3 lines · Junio C Hamano

An alternative would be to bypass the contrib/ phase and start as a new subcommand that is first-class citizen from day one and let it spend as much time as it needs to mature.

I don't think that there is a lot of sense in that. The main benefits of scalar are in the register and the clone part, and the most natural end game would hence be for git init and git clone to sprout new options to support Scalar's features, in a Git-native way.

As I have explained earlier, the scalar command has existing users, and therefore its command-line interface is not up for discussion (for example, turning scalar into git scalar would be a usability disaster). Scalar's _functionality_, however, should make it into Git proper. Into existing built-ins, that is.

So I don't think that the contrib/ phase can be by-passed. It would not make sense to port Scalar to a new builtin. To the contrary, contrib/scalar/ should be the final destination for the scalar command. And you can't bypass a final destination. That simply makes no sense.

So why bother with contrib/ at all? you may ask. The reason is that it makes it substantially easier for me to move the features into core Git, as I can incrementally implement those new options for Git's built-ins, use them in contrib/scalar/ instead of duplicating the functionality, and then make use of Scalar's Functional Test suite for a much more comprehensive testing (which has served us already really well in the past). It also doesn't hurt that this way, my day job will be very happy because Scalar users directly benefit from that work.

Of course, these suggestions to integrate Scalar more into the core part of Git (missing the point that the final destination for the functionality is not a new built-in, but rather new options for existing built-ins) made everything much more cumbersome for me instead, for no gain that would be apparent to me, impeding on aforementioned ease to move the features into core Git (which has not happened yet, as a consequence), but hopefully this will soon be a thing of the past.

So I would like to request that we close the discussion about the question whether to integrate Scalar more into the top-level Makefile or into git.c, and instead go ahead with keeping the scalar command in contrib/scalar/. The freed-up time can then be used to focus on the much more rewarding project of upstreaming Scalar's functionality such as teaching git clone a short-and-sweet option that Just Makes Sense for large monorepos (i.e. that imitates at least a large part of what scalar clone does right now).

Ciao,
Dscho

#106 lore raw

Re: Train station analogy, was Re: [PATCH v3 00/15] [RFC] Upstreaming the Scalar command

Quoted 4 lines · Johannes Schindelin

I don't think that there is a lot of sense in that. The main benefits of scalar are in the register and the clone part, and the most natural end game would hence be for git init and git clone to sprout new options to support Scalar's features, in a Git-native way.

Yes, that is even better. An endgame where everybody benefits natively would be highly desirable.

Now you are back, do you think we can have the "no more preserve merges backend" topic graduate to 'master', or do you prefer to cook it over the cycle (or even two)?

Thanks.

#121 lore raw

Re: [PATCH v4 04/15] scalar: 'register' sets recommended config and starts maintenance

In reply to Elijah Newren

Hi Elijah,

Quoted 31 lines · Elijah Newren

On Tue, Sep 14, 2021 at 7:39 AM Derrick Stolee via GitGitGadget <[email protected]> wrote:
...

Quoted 17 lines
contrib/scalar/scalar.c +17 -0
+static int set_recommended_config(void)
+{
+       struct {
+               const char *key;
+               const char *value;
+       } config[] = {
+               { "am.keepCR", "true" },
+               { "core.FSCache", "true" },
+               { "core.multiPackIndex", "true" },
+               { "core.preloadIndex", "true" },
+#ifndef WIN32
+               { "core.untrackedCache", "true" },
+#else
+               /*
+                * Unfortunately, Scalar's Functional Tests demonstrated
+                * that the untracked cache feature is unreliable on Windows
+                * (which is a bummer because that platform would benefit the

contrib/scalar/scalar.c line 143

Quoted 10 lines
contrib/scalar/scalar.c +10 -0
+                * most from it). For some reason, freshly created files seem
+                * not to update the directory's `lastModified` time
+                * immediately, but the untracked cache would need to rely on
+                * that.
+                *
+                * Therefore, with a sad heart, we disable this very useful
+                * feature on Windows.
+                */
+               { "core.untrackedCache", "false" },
+#endif
Quoted 7 lines

Interesting. (I'm somewhat leery of the untrackedCache just knowing that it used to operate despite an exponential number of visits to files (exponential in depth of directories) and getting different answers with different visits, making me feel like it was black magic that it ever worked and wondering what kind of corner case issues still lurk with it. See e.g.
https://lore.kernel.org/git/CABPp-BFiwzzUgiTj_zu+vF5x20L0=1cf25cHwk7KZQj2YkVzXw@mail.gmail.com/)

The implementation of the untracked cache certainly is quite a challenge to wrap one's head around, for sure. However, it does manage to speed up operations substantially (when it works).

The real fun starts when you turn on the FSMonitor, though. Then it is reliable, all of a sudden! The reason seems to be some sort of delayed lastModified (AKA mtime) evaluation which is somehow triggered by FSMonitor ;-)

So in microsoft/git, where we include FSMonitor and turn it on as part of scalar clone, we also enable the untracked cache, for noticeably happier users.

Quoted 5 lines

contrib/scalar/scalar.c line 145

Quoted 2 lines
contrib/scalar/scalar.c +2 -0
+               { "core.logAllRefUpdates", "true" },
+               { "credential.https://dev.azure.com.useHttpPath", "true" },

Not only opinionated, but special configuration for certain sites? I'm not complaining, just slightly surprised.

Yes. I am not aware of other sites where you would want to use different credentials depending on the URL path, but Azure DevOps definitely is such a site, and therefore needs useHttpPath. Rather than requiring users to know this, we set it for them.

Quoted 7 lines

contrib/scalar/scalar.c line 150

Quoted 5 lines
contrib/scalar/scalar.c +5 -0
+               { "credential.validate", "false" }, /* GCM4W-only */
+               { "gc.auto", "0" },
+               { "gui.GCWarning", "false" },
+               { "index.threads", "true" },
+               { "index.version", "4" },

I take it your users don't make use of jgit?

Nope ;-) I doubt that the features we use to make Git scalable are implemented in JGit.

Quoted 5 lines

(Users aren't using jgit directly here, at least not to my knowledge, but multiple gradle plugins do.) I tried turning this on a while back, and quickly got multiple reports of problems because jgit didn't understand the index. I had to turn it off and send out various PSAs on how to recover.

TBH it gives me shivers of dread thinking about large repositories/worktrees being handled within a Java VM. The amount of, let's call it "non-canonical" code, required by JGit to make it somewhat performant, is staggering. Just think about the way you have to emulate mmap()ing part of a packfile and interpreting it as a packed C struct. I forgot the details, of course, and I am quite glad that I did.

Quoted 6 lines

contrib/scalar/scalar.c line 152

Quoted 2 lines
contrib/scalar/scalar.c +2 -0
+               { "merge.stat", "false" },
+               { "merge.renames", "false" },

Is this just historical and not needed anymore, is it here just for a little longer and you are planning on transitioning away from this, or are you still set on this setting?

It is here mostly for historical reasons.

Quoted 5 lines
Quoted 1 line

+ { "pack.useBitmaps", "false" },

I don't understand anything bitmap related, but I thought they were performance related, so I'm surprised by this one. Is there a reason for this one? (Is it handled by maintenance instead?)

Again, this is here for historical reasons. Scalar sets this, and my goal with this patch series is to port it from .NET to C. So I did not question the reasoning.

My _guess_ however is that bitmaps really only work well when everything is in one single pack. Which is rather not the case with Scalar
enlistments: they are way too large to be repacked all the time.

Quoted 9 lines

contrib/scalar/scalar.c line 157

Quoted 4 lines
contrib/scalar/scalar.c +4 -0
+               { "pack.useSparse", "true" },
+               { "receive.autoGC", "false" },
+               { "reset.quiet", "true" },
+               { "feature.manyFiles", "false" },

If you simply set core.untrackedCache to false _after_ setting feature.manyFiles to true, would it make sense to switch this? (Or does it matter, since you've already individually set all the config settings that this one would set?)

Frankly, I was a bit puzzled why feature.manyFiles was set to false. The rationale is explained in
https://github.com/microsoft/scalar/commit/2fc84dba9c95:

The feature.* config settings change the defaults for some other config settings. We already monitor config settings pretty carefully, so let's disable these.

As to switching this, it shouldn't matter. The idea of feature.* is to set defaults, but not override any explicitly configured settings.

Quoted 8 lines
Quoted 8 lines
contrib/scalar/scalar.c +8 -0
+               { "feature.experimental", "false" },
+               { "fetch.unpackLimit", "1" },
+               { "fetch.writeCommitGraph", "false" },
+#ifdef WIN32
+               { "http.sslBackend", "schannel" },
+#endif
+               { "status.aheadBehind", "false" },
+               { "commitGraph.generationVersion", "1" },
Quoted 8 lines

contrib/scalar/scalar.c line 169

Quoted 4 lines
contrib/scalar/scalar.c +4 -0
+               { "core.autoCRLF", "false" },
+               { "core.safeCRLF", "false" },
+               { NULL, NULL },
+       };

Are there easy-ish ways for other groups of users to adopt scalar but change the list of config settings (e.g. index.version and merge.renames) in some common way for all those users?

Not in Scalar.

I would hope, however, that we could figure out ways to make this more configurable when re-implementing this functionality in core Git. I have a couple ideas, but nothing fleshed out, and besides, I do not want to think too far ahead, I already made that mistake and then got bogged down in discussions about minimal vs non-minimal changes in the top-level Makefile ;-)

So yeah, good point, but it's probably not a good time yet to discuss this tangent.

Thank you for reviewing,
Dscho

#123 lore raw

Re: [PATCH v4 04/15] scalar: 'register' sets recommended config and starts maintenance

In reply to Elijah Newren

Hi Elijah,

Quoted 9 lines · Elijah Newren

Sorry, one more thing...

On Tue, Sep 14, 2021 at 7:39 AM Derrick Stolee via GitGitGadget <[email protected]> wrote:
...

contrib/scalar/scalar.c line 53

Quoted 3 lines
contrib/scalar/scalar.c +3 -0
+               /* check if currently in enlistment root with src/ workdir */
+               strbuf_addstr(&path, "/src/.git");
+               if (is_git_directory(path.buf)) {
Quoted 8 lines

...and...

contrib/scalar/scalar.c line 68

Quoted 3 lines
contrib/scalar/scalar.c +3 -0
+               /* check if currently in workdir */
+               strbuf_addstr(&path, "/.git");
+               if (is_git_directory(path.buf)) {

Do these two checks suggest that only a primary worktree can be enlisted with scalar? (Is git-worktree generally incompatible?)

Good point! I think we'll need to use is_nonbare_repository_dir() instead.

This also has the additional benefit of doing away with quite a bit of /.git appending and undoing it. I.e. it simplifies the code dramatically.

Ciao,
Dscho

#26 lore raw

Re: [PATCH 08/15] scalar: implement the `clone` subcommand

In reply to Elijah Newren

Hi Elijah,

Quoted 17 lines · Elijah Newren
Quoted 1 line · Johannes Schindelin via GitGitGadget

...

Quoted 13 lines

COMMANDS
--------

contrib/scalar/scalar.txt +10 -0
+Clone
+~~~~~
+
+clone [<options>] <url> [<enlistment>]::
+    Clones the specified repository, similar to linkgit:git-clone[1]. By
+    default, only commit and tree objects are cloned. Once finished, the
+    worktree is located at `<enlistment>/src`.
++
+The sparse-checkout feature is enabled (except when run with `--full-clone`)
+and the only files present are those in the top-level directory. Use
Quoted 8 lines

contrib/scalar/scalar.txt an existing Git worktree with Scalar whose name is not `src`, the enlistment · line 52

Quoted 4 lines

+git sparse-checkout set to expand the set of directories you want to see,

contrib/scalar/scalar.txt +3 -0
+or `git sparse-checkout disable` to expand to all files (see
+linkgit:git-sparse-checkout[1] for more details). You can explore the
+subdirectories outside your sparse-checkout by using `git ls-tree HEAD`.

Should this be git ls-tree [-r] HEAD? Do you expect people to just add directories that are found immediately under the toplevel, rather than some that are a bit deeper?

I fear that git ls-tree -r HEAD in any monorepo might be a bit too overwhelming for any reader.

But I agree that just looking at HEAD is probably not enough. Maybe we should use git ls-tree HEAD[:<dir>]?

Ciao,
Dscho

#132 lore raw

Re: [PATCH v4 11/15] scalar: allow reconfiguring an existing enlistment

In reply to Elijah Newren

Hi Elijah,

Quoted 91 lines · Elijah Newren
Quoted 89 lines · Johannes Schindelin via GitGitGadget

From: Johannes Schindelin <[email protected]>

This comes in handy during Scalar upgrades, or when config settings were messed up by mistake.

---
 contrib/scalar/scalar.c          | 79 +++++++++++++++++++++-----------
 contrib/scalar/scalar.txt        |  8 ++++
 contrib/scalar/t/t9099-scalar.sh |  8 ++++
 3 files changed, 67 insertions(+), 28 deletions(-)
contrib/scalar/scalar.c +27 -25
index 8a11f390251..1fff7eb7c12 100644
@@ -115,18 +115,20 @@ static int run_git(const char *arg, ...)
        return res;
 }
 
-static int set_recommended_config(void)
+static int set_recommended_config(int reconfigure)
 {
        struct {
                const char *key;
                const char *value;
+               int overwrite_on_reconfigure;
        } config[] = {
-               { "am.keepCR", "true" },
-               { "core.FSCache", "true" },
-               { "core.multiPackIndex", "true" },
-               { "core.preloadIndex", "true" },
+               /* Required */
+               { "am.keepCR", "true", 1 },
+               { "core.FSCache", "true", 1 },
+               { "core.multiPackIndex", "true", 1 },
+               { "core.preloadIndex", "true", 1 },
 #ifndef WIN32
-               { "core.untrackedCache", "true" },
+               { "core.untrackedCache", "true", 1 },
 #else
                /*
                 * Unfortunately, Scalar's Functional Tests demonstrated
@@ -140,28 +142,29 @@ static int set_recommended_config(void)
                 * Therefore, with a sad heart, we disable this very useful
                 * feature on Windows.
                 */
-               { "core.untrackedCache", "false" },
+               { "core.untrackedCache", "false", 1 },
 #endif
-               { "core.logAllRefUpdates", "true" },
-               { "credential.https://dev.azure.com.useHttpPath", "true" },
-               { "credential.validate", "false" }, /* GCM4W-only */
-               { "gc.auto", "0" },
-               { "gui.GCWarning", "false" },
-               { "index.threads", "true" },
-               { "index.version", "4" },
-               { "merge.stat", "false" },
-               { "merge.renames", "false" },
-               { "pack.useBitmaps", "false" },
-               { "pack.useSparse", "true" },
-               { "receive.autoGC", "false" },
-               { "reset.quiet", "true" },
-               { "feature.manyFiles", "false" },
-               { "feature.experimental", "false" },
-               { "fetch.unpackLimit", "1" },
-               { "fetch.writeCommitGraph", "false" },
+               { "core.logAllRefUpdates", "true", 1 },
+               { "credential.https://dev.azure.com.useHttpPath", "true", 1 },
+               { "credential.validate", "false", 1 }, /* GCM4W-only */
+               { "gc.auto", "0", 1 },
+               { "gui.GCWarning", "false", 1 },
+               { "index.threads", "true", 1 },
+               { "index.version", "4", 1 },
+               { "merge.stat", "false", 1 },
+               { "merge.renames", "false", 1 },
+               { "pack.useBitmaps", "false", 1 },
+               { "pack.useSparse", "true", 1 },
+               { "receive.autoGC", "false", 1 },
+               { "reset.quiet", "true", 1 },
+               { "feature.manyFiles", "false", 1 },
+               { "feature.experimental", "false", 1 },
+               { "fetch.unpackLimit", "1", 1 },
+               { "fetch.writeCommitGraph", "false", 1 },
 #ifdef WIN32
-               { "http.sslBackend", "schannel" },
+               { "http.sslBackend", "schannel", 1 },
 #endif
Quoted 8 lines

contrib/scalar/scalar.c static int set_recommended_config(void) · line 170

Quoted 4 lines
contrib/scalar/scalar.c +1 -0
+               /* Optional */
                { "status.aheadBehind", "false" },
                { "commitGraph.generationVersion", "1" },
                { "core.autoCRLF", "false" },

Now you have optional settings...but index.version and merge.renames aren't among them?? Why are those required? (...and to go a step further; should merge.renames even be off in a merge-ort world?)

I think the idea here is that they are required so that a scalar reconfigure will set them, even if the current enlistment had been created by a previous Scalar version that had _not_ set those.

And yes, in a merge-ort world, merge.renames should probably be forced to true, again because it is in the "Required" section.

Ciao,
Dscho

#137 lore raw

Re: [PATCH v4 13/15] scalar: implement the `delete` command

In reply to a message not in the archive

Hi Matthew and Elijah,

Quoted 18 lines · Matthew Cheetham
Quoted 5 lines · Elijah Newren

But if the current directory is inside the enlistment's directory, didn't that happen because the parent process' current directory was inside the enlistment directory? Or was there some kind of directory switching that scalar itself was doing causing it to be inside the enlistment directory?

Yes to the latter. setup_enlistment_directory changes the current directory much like setup_git_directory.

Quoted 6 lines

If the the current directory was inside the enlistment's directory because it inherited a parent process' current directory, wouldn't that also prevent deleting it? If so, should there be a special check for that case and pre-emptively returning an error rather than attempting the recursive directory deletion and just spitting out an error when it fails?

Quoted 2 lines

You are correct. Speaking to Johannes about this I believe he is looking to add a check/error in a new patch series revision.

Indeed, I did notice GGG#1037, and I changed the code so that it detects whether scalar delete was called from within the enlistment, and refuses to run in that case. Users will have to call scalar delete <path> from outside the enlistment.

Ciao,
Dscho

#108 lore raw

Re: Train station analogy, was Re: [PATCH v3 00/15] [RFC] Upstreaming the Scalar command

In reply to Johannes Schindelin
Quoted 11 lines · Johannes Schindelin
Quoted 3 lines · Junio C Hamano

An alternative would be to bypass the contrib/ phase and start as a new subcommand that is first-class citizen from day one and let it spend as much time as it needs to mature.

I don't think that there is a lot of sense in that. The main benefits of scalar are in the register and the clone part, and the most natural end game would hence be for git init and git clone to sprout new options to support Scalar's features, in a Git-native way.

Quoted 5 lines

As I have explained earlier, the scalar command has existing users, and therefore its command-line interface is not up for discussion (for example, turning scalar into git scalar would be a usability disaster). Scalar's _functionality_, however, should make it into Git proper. Into existing built-ins, that is.

Given the sub-thread this seems like it's trying to be an indirect reply to me, but I haven't been advocating changing the UX of "scalar" in any way, or that we should have a "git scalar".

I have in fact been advocating exactly what you're advocating here. So we're in violent agreement. Yes the CLI UI shouldn't change, that's the whole point of having a "scalar" in git.git, not a "git scalar" or whatever.

I've been suggesting we can simplify the *build system* git.git uses, something no user will ever see.

Quoted 4 lines

So I don't think that the contrib/ phase can be by-passed. It would not make sense to port Scalar to a new builtin. To the contrary, contrib/scalar/ should be the final destination for the scalar command. And you can't bypass a final destination. That simply makes no sense.

I'm right now using a "scalar" installed on my system based on the diff at the end of this E-Mail that changes the *build system* without any user-facing changes (see
https://lore.kernel.org/git/[email protected]/) for a previous reference.

    $ which scalar
    /home/avar/local/bin/scalar
    $ scalar -h 2>&1 | head -n 1
    usage: scalar [-C <directory>] [-c <key>=<value>] <command> [<options>]
    $ man scalar
    [...]
    NAME
           scalar - an opinionated repository management tool
    $ git help scalar
    No manual entry for gitscalar

Except that I think your patches as they stand (correct me if I'm wrong) don't have any way to install it or its documentation, just to build it in-place.

Quoted 8 lines

So why bother with contrib/ at all? you may ask. The reason is that it makes it substantially easier for me to move the features into core Git, as I can incrementally implement those new options for Git's built-ins, use them in contrib/scalar/ instead of duplicating the functionality, and then make use of Scalar's Functional Test suite for a much more comprehensive testing (which has served us already really well in the past). It also doesn't hurt that this way, my day job will be very happy because Scalar users directly benefit from that work.

I think all of that summarizes "why have this live in git.git", which I 100% agree with. It's not a counter-argument to a working solution for simplifying your proposed build system integration.

Quoted 7 lines

Of course, these suggestions to integrate Scalar more into the core part of Git (missing the point that the final destination for the functionality is not a new built-in, but rather new options for existing built-ins) made everything much more cumbersome for me instead, for no gain that would be apparent to me, impeding on aforementioned ease to move the features into core Git (which has not happened yet, as a consequence), but hopefully this will soon be a thing of the past.

The diff below doesn't make scalar a built-in.

Quoted 8 lines

So I would like to request that we close the discussion about the question whether to integrate Scalar more into the top-level Makefile or into git.c, and instead go ahead with keeping the scalar command in contrib/scalar/. The freed-up time can then be used to focus on the much more rewarding project of upstreaming Scalar's functionality such as teaching git clone a short-and-sweet option that Just Makes Sense for large monorepos (i.e. that imitates at least a large part of what scalar clone does right now).

The reason I care about this is because duplicating this as a one-off may be easier for you now, but it creates a lot of maintenance burden for others down the line.

For example, I've been fixing memory leaks recently and have a linux-leaks CI job that's now running on GitHub. Scalar would benefit from having a t/t9099-scalar.sh run as part of that.

Yes you can special-case it somehow and teach ci/, or even "make test" to do the same thing. But there's a lot of little things both current and future that are going to be like that. Here's another one:

    $ make -C contrib/scalar scalar
    $ echo bad >cache.h
    $ make -C contrib/scalar scalar
    make: 'scalar' is up to date.

I.e. because it's using its own Makefile it's not getting the very basics of the dependency graph right. My version?

    $ make scalar
    $ touch cache.h
    $ make scalar
    [... Works exactly as well as doing the same with "git"...]

Yes we could fix that and other things etc, but why not just get all that for free in around 1/4 lines of Makefile boilerplate (and less than that in complexity)?

 .gitignore                                   |  1 +
 Documentation/Makefile                       |  3 ++
 {contrib/scalar => Documentation}/scalar.txt |  4 +-
 Makefile                                     | 44 +++++++++++-----
 contrib/scalar/.gitignore                    |  5 --
 contrib/scalar/Makefile                      | 57 --------------------
 contrib/scalar/t/Makefile                    | 78 ----------------------------
 contrib/scalar/scalar.c => scalar.c          |  0
 {contrib/scalar/t => t}/t9099-scalar.sh      |  8 +--
 9 files changed, 37 insertions(+), 163 deletions(-)

[Note that this probably doesn't fully apply to your series/master, not because of any great textual/semantic conflict, but just because I find it convenient to stack my WIP/unsubmitted serieses in related areas on top of one another. This is on top of other Makefile changes I've got unsubmitted, but it would be easy/trivial to re-apply on master+your series].

.gitignore +1 -0
index 68ecb7f7e9c..ec9771e3532 100644
@@ -217,6 +217,7 @@
 /configure
 /.vscode/
 /tags
+/scalar
 /TAGS
 /cscope*
 /compile_commands.json
Documentation/Makefile +3 -0
index f5605b7767f..57b72e1999a 100644
@@ -18,6 +18,9 @@ MAN1_TXT += $(filter-out \
 MAN1_TXT += git.txt
 MAN1_TXT += gitk.txt
 MAN1_TXT += gitweb.txt
+ifndef NO_INSTALL_SCALAR_DOC
+MAN1_TXT += scalar.txt
+endif
 
 # man5 / man7 guides (note: new guides should also be added to command-list.txt)
 MAN5_TXT += gitattributes.txt
Documentation/scalar.txt renamed from contrib/scalar/scalar.txt +2 -2
similarity index 99%
rename from contrib/scalar/scalar.txt
rename to Documentation/scalar.txt
index 3a80f829edc..f287ab18c31 100644
@@ -149,6 +149,6 @@ SEE ALSO
 --------
 linkgit:git-clone[1], linkgit:git-maintenance[1].
 
-Scalar
+GIT
 ---
-Associated with the linkgit:git[1] suite
+Part of the linkgit:git[1] suite
Makefile +30 -14

121 lines not on this page · raw message

contrib/scalar/.gitignore deleted +0 -5
deleted file mode 100644
index 00441073f59..00000000000
@@ -1,5 +0,0 @@
-/*.xml
-/*.1
-/*.html
-/*.exe
-/scalar
contrib/scalar/Makefile deleted +0 -57

57 lines not on this page · raw message

contrib/scalar/t/Makefile deleted +0 -78

78 lines not on this page · raw message

scalar.c renamed from contrib/scalar/scalar.c +0 -0
similarity index 100%
rename from contrib/scalar/scalar.c
rename to scalar.c
t/t9099-scalar.sh renamed from contrib/scalar/t/t9099-scalar.sh +1 -7
similarity index 94%
rename from contrib/scalar/t/t9099-scalar.sh
rename to t/t9099-scalar.sh
index 7e8771d0eff..f686ba5d84c 100755
@@ -2,13 +2,7 @@
 
 test_description='test the `scalar` command'
 
-TEST_DIRECTORY=$PWD/../../../t
-export TEST_DIRECTORY
-
-# Make it work with --no-bin-wrappers
-PATH=$PWD/..:$PATH
-
-. ../../../t/test-lib.sh
+. ./test-lib.sh
 
 GIT_TEST_MAINT_SCHEDULER="crontab:test-tool crontab ../cron.txt"
 export GIT_TEST_MAINT_SCHEDULER
#107 lore raw

Re: Train station analogy, was Re: [PATCH v3 00/15] [RFC] Upstreaming the Scalar command

In reply to Junio C Hamano

Hi Junio,

Quoted 9 lines · Junio C Hamano
Quoted 4 lines · Johannes Schindelin

I don't think that there is a lot of sense in that. The main benefits of scalar are in the register and the clone part, and the most natural end game would hence be for git init and git clone to sprout new options to support Scalar's features, in a Git-native way.

Yes, that is even better. An endgame where everybody benefits natively would be highly desirable.

I am glad that this question is now resolved.

Quoted 3 lines

Now you are back, do you think we can have the "no more preserve merges backend" topic graduate to 'master', or do you prefer to cook it over the cycle (or even two)?

I did not see _any_ problems with it, so I'd be in favor of promoting it to master. But if you want to play the game more carefully, I would be fine with that, too.

Ciao,
Dscho

#142 lore raw

v5 cover Upstreaming the Scalar command

In reply to Johannes Schindelin via GitGitGadget

tl;dr: This series contributes the Scalar command to the Git project. This command provides an opinionated way to create and configure repositories with a focus on very large repositories.

Changes since v4:

* scalar delete now refuses to delete anything if it was started from within the enlistment.
* scalar delete releases any handles to the object store before deleting the enlistment.
* The OBJECTS list in the Makefile will now include Scalar.
* scalar register now supports secondary worktrees, in addition to the primary worktree.

Changes since v3:

* Moved the "Changes since" section to the top, to make it easier to see what changed.
* Reworded the commit message of the first patch.
* Removed the [RFC] prefix because I did not hear any objections against putting this into contrib/.

Changes since v2:

* Adjusted the description of the list command in the manual page , as suggested by Bagas.
* Addressed two style nits in cmd_run().
* The documentation of git reconfigure -a was improved.

Changes since v1:

* A couple typos were fixed
* The code parsing the output of ls-remote was made more readable
* The indentation used in scalar.txt now consistently uses tabs
* We no longer hard-code core.bare = false when registering with Scalar

Background
==========

Years ago, Microsoft wanted to move the source code of the Windows operating system to Git. The challenge there was to prove that Git could scale to massive monorepos. The VFS for Git (formerly GVFS) project was born to take up that challenge.

The final solution included a virtual filesystem (with both user-mode and kernel components) and a customized fork of Git for Windows. This solution contained several key concepts, such as only populating a portion of the working directory, demand-fetching blobs, and performing periodic repo maintenance in the background. However, the required kernel drivers made it difficult to port the solution to other platforms.

But it was realized that many of these key concepts were independent of the actual VFS and its projection of the working directory. The Scalar project was created to make that separation, refine the key concepts, and then extract those features into the new Scalar command.

The present
===========

The Scalar project provides a completely functional non-virtual experience for monorepos. But why stop there. The Scalar project was designed to be a self-destructing vehicle to allow those key concepts to be moved into core Git itself for the benefit of all. For example, partial clone, sparse-checkout, and background maintenance have already been upstreamed and removed from Scalar proper. This patch series provides a C-based implementation of the final remaining portions of the Scalar command. This will make it easier for users to experiment with the Scalar command. It will also make it substantially easier to experiment with moving functionality from Scalar into core Git, while maintaining backwards-compatibility for existing Scalar users.

The C-based Scalar has been shipped to Scalar users, and can be tested by any interested reader:
https://github.com/microsoft/git/releases/tag/v2.33.0.vfs.0.0 (it offers a Git for Windows installer, a macOS package and an Ubuntu package).

Opportunities
=============

Apart from providing the Scalar command, this contribution is intended to serve as a basis for further mailing list discussions on moving (some of) these key concepts into the main Git commands.

For example, we previously discussed the idea of a "git big-clone" that does much of what "scalar clone" is doing. This patch series is a step to make such functionality exist in the Git code base while we simmer on what such a "git big-clone" command-line interface would look like.

This is one of many possible ways to do this. Creating a 'git big-clone' could lock Git into backwards compatibility concerns so it is necessary to approach such an endeavor with caution. As a discussion starter, the scalar clone <url> command does roughly this:

1. git clone --sparse --filter=blob:none /src
2. git -C /src sparse-checkout init --cone
3. git -C /src config (many times)
4. git -C /src maintenance start

It is my hope inspire discussions about what parts of Scalar could go into core Git, and where, and in which form. While we wish to maintain backwards-compatibility of Scalar's command-line interface (because it is already in use), by having the Scalar code in the same code base as Git's, it will be much easier to move functionality without having to maintain loose version coupling between independently-versioned Scalar and Git. The tight version-coupling, along with having access to libgit.a also allows the C-based implementation of Scalar to be much smaller than the original .NET version.

For example, we might choose in the future to implement, say, git clone --scale=partial,cone to initialize a partial clone with a cone-sparse checkout, that would not only be totally doable, and not only would we already have precedent and data to prove that this actually makes engineers happy who have to work on ginormous repositories, but we could then also implement it by moving parts of contrib/scalar/ to builtin/ (where contrib/scalar/ would then call the built-ins accordingly rather than hard-coding the defaults itself).

We now also have the opportunity to discuss the merits of Scalar's clone caching, which is not actually part of this patch series because it is a bit coupled with the GVFS parts of microsoft/git for the moment, where clones automatically get registered with a populated alternate repository that is identified by the URL, meaning: subsequent clones of the same repository are vastly faster than the first one because they do not actually download the already-received objects again, they access the cache instead.

Another thing that I could imagine to be discussed at length is the distinction between enlistment and worktree (where the latter is the actual Git worktree and usually lives in the src/ subdirectory of the former). This encourages untracked and ignored files to be placed outside the worktree, making Git's job much easier. This idea, too, might find its way in one way or another into Git proper.

These are just a few concepts in Scalar that do not yet have equivalents in Git. By putting this initial implementation into contrib/, we create a foundation for future discussions of these concepts.

We plan on updating the recommended config settings in scalar register as new Git features are available (such as builtin FSMonitor and sparse-index, when ready). To facilitate upgrading existing Scalar enlistments, their paths are automatically added to the [scalar] section of the global Git config, and the scalar reconfigure --all command will process all of them.

Epilogue
========

Now, to address some questions that I imagine every reader has who made it this far:

* Why not put the Scalar functionality directly into a built-in? Creating a Git builtin requires scrutiny over every aspect of the feature, which is difficult to do while also maintaining the command-line interface contract and expected behavior of the Scalar command (there are existing users, after all). By having the Scalar command in contrib/, we present a simple option for users to have these features in the short term while the Git contributor community decides which bits to absorb into Git built-ins.
* Why implement the Scalar command in the Git codebase? We ported Scalar to the microsoft/git fork for several reasons. First, we realized it was possible now that the core features exist inside Git itself. Second, compiling Scalar directly within a version of Git allows us to remove a version compatibility check from each config option that might or might not apply based on the installed Git version. Finally, this new location has greatly simplified our release process and the installation process for users. We now have ways to install Scalar with microsoft/git via winget, brew, and apt-get. This has been the case since we shipped v2.32.0 to our users, read: this setup has served us well already.
* Why contribute Scalar to the Git project? We are biased, of course, yet we do have evidence that the Scalar command is a helpful tool that offers an simple way to handle huge repositories with ease. By contributing it to the core Git project, we are able to share it with more users, especially some users who do not want to install the microsoft/git fork. We intend to include Scalar as a component in git-for-windows/git, but are contributing it here first. Further, we think there is benefit to the Git developer community as this presents an example of how to set certain defaults that work for large repositories.
* Does this integrate with the built-in FSMonitor yet? No, not yet. I do have a couple of add-on patch series lined up, one of them being the integration with the built-in FSMonitor, which obviously has to wait until the FSMonitor patch series advances further.

Derrick Stolee (4):
scalar: 'register' sets recommended config and starts maintenance
scalar: 'unregister' stops background maintenance
scalar: implement 'scalar list'
scalar: implement the run command

Johannes Schindelin (10):
scalar: create a rudimentary executable
scalar: start documenting the command
scalar: create test infrastructure
scalar: let 'unregister' handle a deleted enlistment directory
gracefully
scalar: implement the clone subcommand
scalar: teach 'clone' to support the --single-branch option
scalar: allow reconfiguring an existing enlistment
scalar: teach 'reconfigure' to optionally handle all registered
enlistments
scalar: implement the version command
scalar: accept -C and -c options before the subcommand

Matthew John Cheetham (1):
scalar: implement the delete command

 Makefile                         |   9 +
 contrib/scalar/.gitignore        |   5 +
 contrib/scalar/Makefile          |  57 +++
 contrib/scalar/scalar.c          | 845 +++++++++++++++++++++++++++++++
 contrib/scalar/scalar.txt        | 154 ++++++
 contrib/scalar/t/Makefile        |  78 +++
 contrib/scalar/t/t9099-scalar.sh |  88 ++++
 7 files changed, 1236 insertions(+)
 create mode 100644 contrib/scalar/.gitignore
 create mode 100644 contrib/scalar/Makefile
 create mode 100644 contrib/scalar/scalar.c
 create mode 100644 contrib/scalar/scalar.txt
 create mode 100644 contrib/scalar/t/Makefile
 create mode 100755 contrib/scalar/t/t9099-scalar.sh

base-commit: ebf3c04b262aa27fbb97f8a0156c2347fecafafb
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-1005%2Fdscho%2Fscalar-the-beginning-v5
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1005/dscho/scalar-the-beginning-v5
Pull-Request: https://github.com/gitgitgadget/git/pull/1005

Range-diff vs v4:

1: 852ec003109 ! 1: 7119a8efc21 scalar: create a rudimentary executable
@@ Commit message

## Makefile ##
-@@ Makefile: endif
- .PHONY: objects
- objects: $(OBJECTS)
-
+@@ Makefile: OBJECTS += $(FUZZ_OBJS)
+ ifndef NO_CURL
+ OBJECTS += http.o http-walker.o remote-curl.o
+ endif
++
+SCALAR_SOURCES := contrib/scalar/scalar.c +SCALAR_OBJECTS := $(SCALAR_SOURCES:c=o)
+OBJECTS += $(SCALAR_OBJECTS)
+
- dep_files := $(foreach f,$(OBJECTS),$(dir $f).depend/$(notdir $f).d)
- dep_dirs := $(addsuffix .depend,$(sort $(dir $(OBJECTS))))
+ .PHONY: objects
+ objects: $(OBJECTS)

@@ Makefile: $(REMOTE_CURL_PRIMARY): remote-curl.o http.o http-walker.o GIT-LDFLAGS $(GITLIBS
$(QUIET_LINK)$(CC) $(ALL_CFLAGS) -o $@ $(ALL_LDFLAGS) $(filter %.o,$^) \

  2:  6ab9c7195da =  2:  edc1e5f73bd scalar: start documenting the command
  3:  14992033d7c =  3:  9eeb66d5b5d scalar: create test infrastructure
  4:  bbbc4c33390 !  4:  8ebfa51ae15 scalar: 'register' sets recommended config and starts maintenance
     @@ contrib/scalar/scalar.c
      +		const size_t len = path.len;
      +
      +		/* check if currently in enlistment root with src/ workdir */
     -+		strbuf_addstr(&path, "/src/.git");
     -+		if (is_git_directory(path.buf)) {
     -+			strbuf_strip_suffix(&path, "/.git");
     -+
     ++		strbuf_addstr(&path, "/src");
     ++		if (is_nonbare_repository_dir(&path)) {
      +			if (enlistment_root)
      +				strbuf_add(enlistment_root, path.buf, len);
      +
     @@ contrib/scalar/scalar.c
      +		strbuf_setlen(&path, len);
      +
      +		/* check if currently in workdir */
     -+		strbuf_addstr(&path, "/.git");
     -+		if (is_git_directory(path.buf)) {
     -+			strbuf_setlen(&path, len);
     -+
     ++		if (is_nonbare_repository_dir(&path)) {
      +			if (enlistment_root) {
      +				/*
      +				 * If the worktree's directory's name is `src`, the enlistment is the
     @@ contrib/scalar/scalar.c
      +			enlistment_found = 1;
      +			break;
      +		}
     -+
     -+		strbuf_setlen(&path, len);
      +	} while (strbuf_parent_directory(&path));
      +
      +	if (!enlistment_found)
  5:  eadcddb2a9b =  5:  51b5fc577c9 scalar: 'unregister' stops background maintenance
  6:  c3c2f3a4971 =  6:  0041477374e scalar: let 'unregister' handle a deleted enlistment directory gracefully
  7:  90ef9b826b2 =  7:  12efa86e4cb scalar: implement 'scalar list'
  8:  79cde4417d8 =  8:  670891d14cc scalar: implement the `clone` subcommand
  9:  0acdaeb7396 =  9:  03b8385ebb5 scalar: teach 'clone' to support the --single-branch option
 10:  64e3403ac12 = 10:  5a0b3843f98 scalar: implement the `run` command
 11:  ada242c7c8c = 11:  6d9aaeb05fa scalar: allow reconfiguring an existing enlistment
 12:  5c11117da51 = 12:  2967d7f1425 scalar: teach 'reconfigure' to optionally handle all registered enlistments
 13:  914c16c7fcd ! 13:  8069cc536fe scalar: implement the `delete` command
     @@ contrib/scalar/scalar.c
       #include "run-command.h"
       #include "refs.h"
      +#include "dir.h"
     ++#include "packfile.h"
       
       /*
        * Remove the deepest subdirectory in the provided path string. Path must not
     @@ contrib/scalar/scalar.c: static int cmd_unregister(int argc, const char **argv)
       
      +static int cmd_delete(int argc, const char **argv)
      +{
     ++	char *cwd = xgetcwd();
      +	struct option options[] = {
      +		OPT_END(),
      +	};
     @@ contrib/scalar/scalar.c: static int cmd_unregister(int argc, const char **argv)
      +
      +	setup_enlistment_directory(argc, argv, usage, options, &enlistment);
      +
     -+	res = delete_enlistment(&enlistment);
     ++	if (dir_inside_of(cwd, enlistment.buf) >= 0)
     ++		res = error(_("refusing to delete current working directory"));
     ++	else {
     ++		close_object_store(the_repository->objects);
     ++		res = delete_enlistment(&enlistment);
     ++	}
      +	strbuf_release(&enlistment);
     ++	free(cwd);
      +
      +	return res;
      +}
 14:  06995770120 ! 14:  2ecfaa5d0fe scalar: implement the `version` command
     @@ Commit message
      
       ## contrib/scalar/scalar.c ##
      @@
     - #include "run-command.h"
       #include "refs.h"
       #include "dir.h"
     + #include "packfile.h"
      +#include "help.h"
       
       /*

15: 7539725bb4f = 15: f81e8b3bcf1 scalar: accept -C and -c options before the subcommand

Signature · 1 line
gitgitgadget
#143 lore raw

PATCH v5 1/15 scalar: create a rudimentary executable

From: Johannes Schindelin <[email protected]>

The idea of Scalar (https://github.com/microsoft/scalar), and before that, of VFS for Git, has always been to prove that Git _can_ scale, and to upstream whatever strategies have been demonstrated to help.

With this patch, we start the journey from that C# project to move what is left to Git's own contrib/ directory, reimplementing it in pure C, with the intention to facilitate integrating the functionality into core Git all while maintaining backwards-compatibility for existing Scalar users (which will be much easier when both live in the same worktree). It has always been the plan to contribute all of the proven strategies back to core Git.

For example, while the virtual filesystem provided by VFS for Git helped the team developing the Windows operating system to move onto Git, while trying to upstream it we realized that it cannot be done: getting the virtual filesystem to work (which we only managed to implement fully on Windows, but not on, say, macOS or Linux), and the required server-side support for the GVFS protocol, made this not quite feasible.

The Scalar project learned from that and tackled the problem with different tactics: instead of pretending to Git that the working directory is fully populated, it _specifically_ teaches Git about partial clone (which is based on VFS for Git's cache server), about sparse checkout (which VFS for Git tried to do transparently, in the file system layer), and regularly runs maintenance tasks to keep the repository in a healthy state.

With partial clone, sparse checkout and git maintenance having been upstreamed, there is little left that scalar.exe does which git.exe cannot do. One such thing is that scalar clone <url> will automatically set up a partial, sparse clone, and configure known-helpful settings from the start.

So let's bring this convenience into Git's tree.

The idea here is that you can (optionally) build Scalar via

make -C contrib/scalar/Makefile

This will build the scalar executable and put it into the contrib/scalar/ subdirectory.

The slightly awkward addition of the contrib/scalar/* bits to the top-level Makefile are actually really required: we want to link to libgit.a, which means that we will need to use the very same CFLAGS and LDFLAGS as the rest of Git.

An early development version of this patch tried to replicate all the conditional code in contrib/scalar/Makefile (e.g. NO_POLL) just like contrib/svn-fe/Makefile used to do before it was retired. It turned out to be quite the whack-a-mole game: the SHA-1-related flags, the flags enabling/disabling compat/poll/, compat/regex/, compat/win32mmap.c & friends depending on the current platform... To put it mildly: it was a major mess.

Instead, this patch makes minimal changes to the top-level Makefile so that the bits in contrib/scalar/ can be compiled and linked, and adds a contrib/scalar/Makefile that uses the top-level Makefile in a most minimal way to do the actual compiling.

Note: With this commit, we only establish the infrastructure, no Scalar functionality is implemented yet; We will do that incrementally over the next few commits.

---
 Makefile                  |  9 +++++++++
 contrib/scalar/.gitignore |  2 ++
 contrib/scalar/Makefile   | 34 ++++++++++++++++++++++++++++++++++
 contrib/scalar/scalar.c   | 36 ++++++++++++++++++++++++++++++++++++
 4 files changed, 81 insertions(+)
 create mode 100644 contrib/scalar/.gitignore
 create mode 100644 contrib/scalar/Makefile
 create mode 100644 contrib/scalar/scalar.c
Makefile +9 -0
index c3565fc0f8f..a12cac1b68b 100644
@@ -2444,6 +2444,11 @@ OBJECTS += $(FUZZ_OBJS)
 ifndef NO_CURL
 	OBJECTS += http.o http-walker.o remote-curl.o
 endif
+
+SCALAR_SOURCES := contrib/scalar/scalar.c
+SCALAR_OBJECTS := $(SCALAR_SOURCES:c=o)
+OBJECTS += $(SCALAR_OBJECTS)
+
 .PHONY: objects
 objects: $(OBJECTS)
 
@@ -2586,6 +2591,10 @@ $(REMOTE_CURL_PRIMARY): remote-curl.o http.o http-walker.o GIT-LDFLAGS $(GITLIBS
 	$(QUIET_LINK)$(CC) $(ALL_CFLAGS) -o $@ $(ALL_LDFLAGS) $(filter %.o,$^) \
 		$(CURL_LIBCURL) $(EXPAT_LIBEXPAT) $(LIBS)
 
+contrib/scalar/scalar$X: $(SCALAR_OBJECTS) GIT-LDFLAGS $(GITLIBS)
+	$(QUIET_LINK)$(CC) $(ALL_CFLAGS) -o $@ $(ALL_LDFLAGS) \
+		$(filter %.o,$^) $(LIBS)
+
 $(LIB_FILE): $(LIB_OBJS)
 	$(QUIET_AR)$(RM) $@ && $(AR) $(ARFLAGS) $@ $^
 
contrib/scalar/.gitignore new file +2 -0
new file mode 100644
index 00000000000..ff3d47e84d0
@@ -0,0 +1,2 @@
+/*.exe
+/scalar
contrib/scalar/Makefile new file +34 -0
new file mode 100644
index 00000000000..40c03ad10e1
@@ -0,0 +1,34 @@
+QUIET_SUBDIR0  = +$(MAKE) -C # space to separate -C and subdir
+QUIET_SUBDIR1  =
+
+ifneq ($(findstring s,$(MAKEFLAGS)),s)
+ifndef V
+	QUIET_SUBDIR0  = +@subdir=
+	QUIET_SUBDIR1  = ;$(NO_SUBDIR) echo '   ' SUBDIR $$subdir; \
+			 $(MAKE) $(PRINT_DIR) -C $$subdir
+else
+	export V
+endif
+endif
+
+all:
+
+include ../../config.mak.uname
+-include ../../config.mak.autogen
+-include ../../config.mak
+
+TARGETS = scalar$(X) scalar.o
+GITLIBS = ../../common-main.o ../../libgit.a ../../xdiff/lib.a
+
+all: scalar$X
+
+$(GITLIBS):
+	$(QUIET_SUBDIR0)../.. $(QUIET_SUBDIR1) $(subst ../../,,$@)
+
+$(TARGETS): $(GITLIBS) scalar.c
+	$(QUIET_SUBDIR0)../.. $(QUIET_SUBDIR1) $(patsubst %,contrib/scalar/%,$@)
+
+clean:
+	$(RM) $(TARGETS)
+
+.PHONY: all clean FORCE
contrib/scalar/scalar.c new file +36 -0
new file mode 100644
index 00000000000..7cff29e0fcd
@@ -0,0 +1,36 @@
+/*
+ * The Scalar command-line interface.
+ */
+
+#include "cache.h"
+#include "gettext.h"
+#include "parse-options.h"
+
+static struct {
+	const char *name;
+	int (*fn)(int, const char **);
+} builtins[] = {
+	{ NULL, NULL},
+};
+
+int cmd_main(int argc, const char **argv)
+{
+	struct strbuf scalar_usage = STRBUF_INIT;
+	int i;
+
+	if (argc > 1) {
+		argv++;
+		argc--;
+
+		for (i = 0; builtins[i].name; i++)
+			if (!strcmp(builtins[i].name, argv[0]))
+				return !!builtins[i].fn(argc, argv);
+	}
+
+	strbuf_addstr(&scalar_usage,
+		      N_("scalar <command> [<options>]\n\nCommands:\n"));
+	for (i = 0; builtins[i].name; i++)
+		strbuf_addf(&scalar_usage, "\t%s\n", builtins[i].name);
+
+	usage(scalar_usage.buf);
+}
Signature · 1 line
gitgitgadget
#144 lore raw

PATCH v5 2/15 scalar: start documenting the command

In reply to Johannes Schindelin via GitGitGadget

From: Johannes Schindelin <[email protected]>

This commit establishes the infrastructure to build the manual page for the scalar command.

---
 contrib/scalar/.gitignore |  3 +++
 contrib/scalar/Makefile   | 14 +++++++++++++-
 contrib/scalar/scalar.txt | 38 ++++++++++++++++++++++++++++++++++++++
 3 files changed, 54 insertions(+), 1 deletion(-)
 create mode 100644 contrib/scalar/scalar.txt
contrib/scalar/.gitignore +3 -0
index ff3d47e84d0..00441073f59 100644
@@ -1,2 +1,5 @@
+/*.xml
+/*.1
+/*.html
 /*.exe
 /scalar
contrib/scalar/Makefile +13 -1
index 40c03ad10e1..85c186634e9 100644
@@ -6,6 +6,7 @@ ifndef V
 	QUIET_SUBDIR0  = +@subdir=
 	QUIET_SUBDIR1  = ;$(NO_SUBDIR) echo '   ' SUBDIR $$subdir; \
 			 $(MAKE) $(PRINT_DIR) -C $$subdir
+	QUIET          = @
 else
 	export V
 endif
@@ -30,5 +31,16 @@ $(TARGETS): $(GITLIBS) scalar.c
 
 clean:
 	$(RM) $(TARGETS)
+	$(RM) scalar.1 scalar.html scalar.xml
 
-.PHONY: all clean FORCE
+docs: scalar.html scalar.1
+
+scalar.html: | scalar.1 # prevent them from trying to build `doc.dep` in parallel
+
+scalar.html scalar.1: scalar.txt
+	$(QUIET_SUBDIR0)../../Documentation$(QUIET_SUBDIR1) \
+		MAN_TXT=../contrib/scalar/scalar.txt \
+		../contrib/scalar/$@
+	$(QUIET)test scalar.1 != "$@" || mv ../../Documentation/$@ .
+
+.PHONY: all clean docs FORCE
contrib/scalar/scalar.txt new file +38 -0
new file mode 100644
index 00000000000..5f7131861a5
@@ -0,0 +1,38 @@
+scalar(1)
+=========
+
+NAME
+----
+scalar - an opinionated repository management tool
+
+SYNOPSIS
+--------
+[verse]
+scalar <command> [<options>]
+
+DESCRIPTION
+-----------
+
+Scalar is an opinionated repository management tool. By creating new
+repositories or registering existing repositories with Scalar, your Git
+experience will speed up. Scalar sets advanced Git config settings,
+maintains your repositories in the background, and helps reduce data sent
+across the network.
+
+An important Scalar concept is the enlistment: this is the top-level directory
+of the project. It usually contains the subdirectory `src/` which is a Git
+worktree. This encourages the separation between tracked files (inside `src/`)
+and untracked files, such as build artifacts (outside `src/`). When registering
+an existing Git worktree with Scalar whose name is not `src`, the enlistment
+will be identical to the worktree.
+
+The `scalar` command implements various subcommands, and different options
+depending on the subcommand.
+
+SEE ALSO
+--------
+linkgit:git-maintenance[1].
+
+Scalar
+---
+Associated with the linkgit:git[1] suite
Signature · 1 line
gitgitgadget
#145 lore raw

PATCH v5 3/15 scalar: create test infrastructure

In reply to Johannes Schindelin via GitGitGadget

From: Johannes Schindelin <[email protected]>

To test the Scalar command, create a test script in contrib/scalar/t that is executed as make -C contrib/scalar test. Since Scalar has no meaningful capabilities yet, the only test is rather simple. We will add more tests in subsequent commits that introduce corresponding, new functionality.

Note: this test script is intended to test scalar only lightly, even after all of the functionality is implemented.

A more comprehensive functional (or: integration) test suite can be found at https://github.com/microsoft/scalar; It is used in the workflow
https://github.com/microsoft/git/blob/HEAD/.github/workflows/scalar-functional-tests.yml
in Microsoft's Git fork. This test suite performs end-to-end tests with a real remote repository, and is run as part of the regular CI builds. Since those tests require some functionality supported only by Microsoft's Git fork ("GVFS protocol"), there is no intention to port that fuller test suite to contrib/scalar/.

---
 contrib/scalar/Makefile          | 17 +++++--
 contrib/scalar/t/Makefile        | 78 ++++++++++++++++++++++++++++++++
 contrib/scalar/t/t9099-scalar.sh | 17 +++++++
 3 files changed, 109 insertions(+), 3 deletions(-)
 create mode 100644 contrib/scalar/t/Makefile
 create mode 100755 contrib/scalar/t/t9099-scalar.sh
contrib/scalar/Makefile +14 -3
index 85c186634e9..8620042f281 100644
@@ -3,6 +3,7 @@ QUIET_SUBDIR1  =
 
 ifneq ($(findstring s,$(MAKEFLAGS)),s)
 ifndef V
+	QUIET_GEN      = @echo '   ' GEN $@;
 	QUIET_SUBDIR0  = +@subdir=
 	QUIET_SUBDIR1  = ;$(NO_SUBDIR) echo '   ' SUBDIR $$subdir; \
 			 $(MAKE) $(PRINT_DIR) -C $$subdir
@@ -21,7 +22,7 @@ include ../../config.mak.uname
 TARGETS = scalar$(X) scalar.o
 GITLIBS = ../../common-main.o ../../libgit.a ../../xdiff/lib.a
 
-all: scalar$X
+all: scalar$X ../../bin-wrappers/scalar
 
 $(GITLIBS):
 	$(QUIET_SUBDIR0)../.. $(QUIET_SUBDIR1) $(subst ../../,,$@)
@@ -30,9 +31,19 @@ $(TARGETS): $(GITLIBS) scalar.c
 	$(QUIET_SUBDIR0)../.. $(QUIET_SUBDIR1) $(patsubst %,contrib/scalar/%,$@)
 
 clean:
-	$(RM) $(TARGETS)
+	$(RM) $(TARGETS) ../../bin-wrappers/scalar
 	$(RM) scalar.1 scalar.html scalar.xml
 
+../../bin-wrappers/scalar: ../../wrap-for-bin.sh Makefile
+	@mkdir -p ../../bin-wrappers
+	$(QUIET_GEN)sed -e '1s|#!.*/sh|#!$(SHELL_PATH_SQ)|' \
+	     -e 's|@@BUILD_DIR@@|$(shell cd ../.. && pwd)|' \
+	     -e 's|@@PROG@@|contrib/scalar/scalar$(X)|' < $< > $@ && \
+	chmod +x $@
+
+test: all
+	$(MAKE) -C t
+
 docs: scalar.html scalar.1
 
 scalar.html: | scalar.1 # prevent them from trying to build `doc.dep` in parallel
@@ -43,4 +54,4 @@ scalar.html scalar.1: scalar.txt
 		../contrib/scalar/$@
 	$(QUIET)test scalar.1 != "$@" || mv ../../Documentation/$@ .
 
-.PHONY: all clean docs FORCE
+.PHONY: all clean docs test FORCE
contrib/scalar/t/Makefile new file +78 -0

78 lines not on this page · raw message

contrib/scalar/t/t9099-scalar.sh new file +17 -0
new file mode 100755
index 00000000000..16f2b72b126
@@ -0,0 +1,17 @@
+#!/bin/sh
+
+test_description='test the `scalar` command'
+
+TEST_DIRECTORY=$PWD/../../../t
+export TEST_DIRECTORY
+
+# Make it work with --no-bin-wrappers
+PATH=$PWD/..:$PATH
+
+. ../../../t/test-lib.sh
+
+test_expect_success 'scalar shows a usage' '
+	test_expect_code 129 scalar -h
+'
+
+test_done
Signature · 1 line
gitgitgadget
#146 lore raw

PATCH v5 4/15 scalar: 'register' sets recommended config and starts maintenance

In reply to Johannes Schindelin via GitGitGadget

From: Derrick Stolee <[email protected]>

Let's start implementing the register command. With this commit, recommended settings are configured upon scalar register, and Git's background maintenance is started.

The recommended config settings may very well change in the future. For example, once the built-in FSMonitor is available, we will want to enable it upon scalar register. For that reason, we explicitly support running scalar register in an already-registered enlistment.

---
 contrib/scalar/scalar.c   | 248 ++++++++++++++++++++++++++++++++++++++
 contrib/scalar/scalar.txt |  18 ++-
 2 files changed, 265 insertions(+), 1 deletion(-)
contrib/scalar/scalar.c +248 -0
index 7cff29e0fcd..38721d671ba 100644
@@ -5,11 +5,259 @@
 #include "cache.h"
 #include "gettext.h"
 #include "parse-options.h"
+#include "config.h"
+#include "run-command.h"
+
+/*
+ * Remove the deepest subdirectory in the provided path string. Path must not
+ * include a trailing path separator. Returns 1 if parent directory found,
+ * otherwise 0.
+ */
+static int strbuf_parent_directory(struct strbuf *buf)
+{
+	size_t len = buf->len;
+	size_t offset = offset_1st_component(buf->buf);
+	char *path_sep = find_last_dir_sep(buf->buf + offset);
+	strbuf_setlen(buf, path_sep ? path_sep - buf->buf : offset);
+
+	return buf->len < len;
+}
+
+static void setup_enlistment_directory(int argc, const char **argv,
+				       const char * const *usagestr,
+				       const struct option *options,
+				       struct strbuf *enlistment_root)
+{
+	struct strbuf path = STRBUF_INIT;
+	char *root;
+	int enlistment_found = 0;
+
+	if (startup_info->have_repository)
+		BUG("gitdir already set up?!?");
+
+	if (argc > 1)
+		usage_with_options(usagestr, options);
+
+	/* find the worktree, determine its corresponding root */
+	if (argc == 1)
+		strbuf_add_absolute_path(&path, argv[0]);
+	else if (strbuf_getcwd(&path) < 0)
+		die(_("need a working directory"));
+
+	strbuf_trim_trailing_dir_sep(&path);
+	do {
+		const size_t len = path.len;
+
+		/* check if currently in enlistment root with src/ workdir */
+		strbuf_addstr(&path, "/src");
+		if (is_nonbare_repository_dir(&path)) {
+			if (enlistment_root)
+				strbuf_add(enlistment_root, path.buf, len);
+
+			enlistment_found = 1;
+			break;
+		}
+
+		/* reset to original path */
+		strbuf_setlen(&path, len);
+
+		/* check if currently in workdir */
+		if (is_nonbare_repository_dir(&path)) {
+			if (enlistment_root) {
+				/*
+				 * If the worktree's directory's name is `src`, the enlistment is the
+				 * parent directory, otherwise it is identical to the worktree.
+				 */
+				root = strip_path_suffix(path.buf, "src");
+				strbuf_addstr(enlistment_root, root ? root : path.buf);
+				free(root);
+			}
+
+			enlistment_found = 1;
+			break;
+		}
+	} while (strbuf_parent_directory(&path));
+
+	if (!enlistment_found)
+		die(_("could not find enlistment root"));
+
+	if (chdir(path.buf) < 0)
+		die_errno(_("could not switch to '%s'"), path.buf);
+
+	strbuf_release(&path);
+	setup_git_directory();
+}
+
+static int run_git(const char *arg, ...)
+{
+	struct strvec argv = STRVEC_INIT;
+	va_list args;
+	const char *p;
+	int res;
+
+	va_start(args, arg);
+	strvec_push(&argv, arg);
+	while ((p = va_arg(args, const char *)))
+		strvec_push(&argv, p);
+	va_end(args);
+
+	res = run_command_v_opt(argv.v, RUN_GIT_CMD);
+
+	strvec_clear(&argv);
+	return res;
+}
+
+static int set_recommended_config(void)
+{
+	struct {
+		const char *key;
+		const char *value;
+	} config[] = {
+		{ "am.keepCR", "true" },
+		{ "core.FSCache", "true" },
+		{ "core.multiPackIndex", "true" },
+		{ "core.preloadIndex", "true" },
+#ifndef WIN32
+		{ "core.untrackedCache", "true" },
+#else
+		/*
+		 * Unfortunately, Scalar's Functional Tests demonstrated
+		 * that the untracked cache feature is unreliable on Windows
+		 * (which is a bummer because that platform would benefit the
+		 * most from it). For some reason, freshly created files seem
+		 * not to update the directory's `lastModified` time
+		 * immediately, but the untracked cache would need to rely on
+		 * that.
+		 *
+		 * Therefore, with a sad heart, we disable this very useful
+		 * feature on Windows.
+		 */
+		{ "core.untrackedCache", "false" },
+#endif
+		{ "core.logAllRefUpdates", "true" },
+		{ "credential.https://dev.azure.com.useHttpPath", "true" },
+		{ "credential.validate", "false" }, /* GCM4W-only */
+		{ "gc.auto", "0" },
+		{ "gui.GCWarning", "false" },
+		{ "index.threads", "true" },
+		{ "index.version", "4" },
+		{ "merge.stat", "false" },
+		{ "merge.renames", "false" },
+		{ "pack.useBitmaps", "false" },
+		{ "pack.useSparse", "true" },
+		{ "receive.autoGC", "false" },
+		{ "reset.quiet", "true" },
+		{ "feature.manyFiles", "false" },
+		{ "feature.experimental", "false" },
+		{ "fetch.unpackLimit", "1" },
+		{ "fetch.writeCommitGraph", "false" },
+#ifdef WIN32
+		{ "http.sslBackend", "schannel" },
+#endif
+		{ "status.aheadBehind", "false" },
+		{ "commitGraph.generationVersion", "1" },
+		{ "core.autoCRLF", "false" },
+		{ "core.safeCRLF", "false" },
+		{ NULL, NULL },
+	};
+	int i;
+	char *value;
+
+	for (i = 0; config[i].key; i++) {
+		if (git_config_get_string(config[i].key, &value)) {
+			trace2_data_string("scalar", the_repository, config[i].key, "created");
+			if (git_config_set_gently(config[i].key,
+						  config[i].value) < 0)
+				return error(_("could not configure %s=%s"),
+					     config[i].key, config[i].value);
+		} else {
+			trace2_data_string("scalar", the_repository, config[i].key, "exists");
+			free(value);
+		}
+	}
+
+	/*
+	 * The `log.excludeDecoration` setting is special because it allows
+	 * for multiple values.
+	 */
+	if (git_config_get_string("log.excludeDecoration", &value)) {
+		trace2_data_string("scalar", the_repository,
+				   "log.excludeDecoration", "created");
+		if (git_config_set_multivar_gently("log.excludeDecoration",
+						   "refs/prefetch/*",
+						   CONFIG_REGEX_NONE, 0))
+			return error(_("could not configure "
+				       "log.excludeDecoration"));
+	} else {
+		trace2_data_string("scalar", the_repository,
+				   "log.excludeDecoration", "exists");
+		free(value);
+	}
+
+	return 0;
+}
+
+static int start_maintenance(void)
+{
+	return run_git("maintenance", "start", NULL);
+}
+
+static int add_enlistment(void)
+{
+	int res;
+
+	if (!the_repository->worktree)
+		die(_("Scalar enlistments require a worktree"));
+
+	res = run_git("config", "--global", "--get", "--fixed-value",
+		      "scalar.repo", the_repository->worktree, NULL);
+
+	/*
+	 * If the setting is already there, then do nothing.
+	 */
+	if (!res)
+		return 0;
+
+	return run_git("config", "--global", "--add",
+		       "scalar.repo", the_repository->worktree, NULL);
+}
+
+static int register_dir(void)
+{
+	int res = add_enlistment();
+
+	if (!res)
+		res = set_recommended_config();
+
+	if (!res)
+		res = start_maintenance();
+
+	return res;
+}
+
+static int cmd_register(int argc, const char **argv)
+{
+	struct option options[] = {
+		OPT_END(),
+	};
+	const char * const usage[] = {
+		N_("scalar register [<enlistment>]"),
+		NULL
+	};
+
+	argc = parse_options(argc, argv, NULL, options,
+			     usage, 0);
+
+	setup_enlistment_directory(argc, argv, usage, options, NULL);
+
+	return register_dir();
+}
 
 static struct {
 	const char *name;
 	int (*fn)(int, const char **);
 } builtins[] = {
+	{ "register", cmd_register },
 	{ NULL, NULL},
 };
 
contrib/scalar/scalar.txt +17 -1
index 5f7131861a5..568987064b2 100644
@@ -8,7 +8,7 @@ scalar - an opinionated repository management tool
 SYNOPSIS
 --------
 [verse]
-scalar <command> [<options>]
+scalar register [<enlistment>]
 
 DESCRIPTION
 -----------
@@ -29,6 +29,22 @@ will be identical to the worktree.
 The `scalar` command implements various subcommands, and different options
 depending on the subcommand.
 
+COMMANDS
+--------
+
+Register
+~~~~~~~~
+
+register [<enlistment>]::
+	Adds the enlistment's repository to the list of registered repositories
+	and starts background maintenance. If `<enlistment>` is not provided,
+	then the enlistment associated with the current working directory is
+	registered.
++
+Note: when this subcommand is called in a worktree that is called `src/`, its
+parent directory is considered to be the Scalar enlistment. If the worktree is
+_not_ called `src/`, it itself will be considered to be the Scalar enlistment.
+
 SEE ALSO
 --------
 linkgit:git-maintenance[1].
Signature · 1 line
gitgitgadget
#147 lore raw

PATCH v5 5/15 scalar: 'unregister' stops background maintenance

In reply to Johannes Schindelin via GitGitGadget

From: Derrick Stolee <[email protected]>

Just like scalar register starts the scheduled background maintenance, scalar unregister stops it. Note that we use git maintenance start in scalar register, but we do not use git maintenance stop in scalar unregister: this would stop maintenance for _all_ repositories, not just for the one we want to unregister.

The unregister command also removes the corresponding entry from the [scalar] section in the global Git config.

---
 contrib/scalar/scalar.c   | 50 ++++++++++++++++++++++++++++++++-------
 contrib/scalar/scalar.txt |  8 +++++++
 2 files changed, 50 insertions(+), 8 deletions(-)
contrib/scalar/scalar.c +42 -8
index 38721d671ba..fc55404230d 100644
@@ -197,12 +197,12 @@ static int set_recommended_config(void)
 	return 0;
 }
 
-static int start_maintenance(void)
+static int toggle_maintenance(int enable)
 {
-	return run_git("maintenance", "start", NULL);
+	return run_git("maintenance", enable ? "start" : "unregister", NULL);
 }
 
-static int add_enlistment(void)
+static int add_or_remove_enlistment(int add)
 {
 	int res;
 
@@ -213,24 +213,39 @@ static int add_enlistment(void)
 		      "scalar.repo", the_repository->worktree, NULL);
 
 	/*
-	 * If the setting is already there, then do nothing.
+	 * If we want to add and the setting is already there, then do nothing.
+	 * If we want to remove and the setting is not there, then do nothing.
 	 */
-	if (!res)
+	if ((add && !res) || (!add && res))
 		return 0;
 
-	return run_git("config", "--global", "--add",
+	return run_git("config", "--global", add ? "--add" : "--unset",
+		       add ? "--no-fixed-value" : "--fixed-value",
 		       "scalar.repo", the_repository->worktree, NULL);
 }
 
 static int register_dir(void)
 {
-	int res = add_enlistment();
+	int res = add_or_remove_enlistment(1);
 
 	if (!res)
 		res = set_recommended_config();
 
 	if (!res)
-		res = start_maintenance();
+		res = toggle_maintenance(1);
+
+	return res;
+}
+
+static int unregister_dir(void)
+{
+	int res = 0;
+
+	if (toggle_maintenance(0) < 0)
+		res = -1;
+
+	if (add_or_remove_enlistment(0) < 0)
+		res = -1;
 
 	return res;
 }
@@ -253,11 +268,30 @@ static int cmd_register(int argc, const char **argv)
 	return register_dir();
 }
 
+static int cmd_unregister(int argc, const char **argv)
+{
+	struct option options[] = {
+		OPT_END(),
+	};
+	const char * const usage[] = {
+		N_("scalar unregister [<enlistment>]"),
+		NULL
+	};
+
+	argc = parse_options(argc, argv, NULL, options,
+			     usage, 0);
+
+	setup_enlistment_directory(argc, argv, usage, options, NULL);
+
+	return unregister_dir();
+}
+
 static struct {
 	const char *name;
 	int (*fn)(int, const char **);
 } builtins[] = {
 	{ "register", cmd_register },
+	{ "unregister", cmd_unregister },
 	{ NULL, NULL},
 };
 
contrib/scalar/scalar.txt +8 -0
index 568987064b2..d9a79984492 100644
@@ -9,6 +9,7 @@ SYNOPSIS
 --------
 [verse]
 scalar register [<enlistment>]
+scalar unregister [<enlistment>]
 
 DESCRIPTION
 -----------
@@ -45,6 +46,13 @@ Note: when this subcommand is called in a worktree that is called `src/`, its
 parent directory is considered to be the Scalar enlistment. If the worktree is
 _not_ called `src/`, it itself will be considered to be the Scalar enlistment.
 
+Unregister
+~~~~~~~~~~
+
+unregister [<enlistment>]::
+	Remove the specified repository from the list of repositories
+	registered with Scalar and stop the scheduled background maintenance.
+
 SEE ALSO
 --------
 linkgit:git-maintenance[1].
Signature · 1 line
gitgitgadget
#148 lore raw

PATCH v5 6/15 scalar: let 'unregister' handle a deleted enlistment directory gracefully

In reply to Johannes Schindelin via GitGitGadget

From: Johannes Schindelin <[email protected]>

When a user deleted an enlistment manually, let's be generous and _still_ unregister it.

---
 contrib/scalar/scalar.c          | 46 ++++++++++++++++++++++++++++++++
 contrib/scalar/t/t9099-scalar.sh | 15 +++++++++++
 2 files changed, 61 insertions(+)
contrib/scalar/scalar.c +46 -0
index fc55404230d..dc84ce0d5b2 100644
@@ -268,6 +268,24 @@ static int cmd_register(int argc, const char **argv)
 	return register_dir();
 }
 
+static int remove_deleted_enlistment(struct strbuf *path)
+{
+	int res = 0;
+	strbuf_realpath_forgiving(path, path->buf, 1);
+
+	if (run_git("config", "--global",
+		    "--unset", "--fixed-value",
+		    "scalar.repo", path->buf, NULL) < 0)
+		res = -1;
+
+	if (run_git("config", "--global",
+		    "--unset", "--fixed-value",
+		    "maintenance.repo", path->buf, NULL) < 0)
+		res = -1;
+
+	return res;
+}
+
 static int cmd_unregister(int argc, const char **argv)
 {
 	struct option options[] = {
@@ -281,6 +299,34 @@ static int cmd_unregister(int argc, const char **argv)
 	argc = parse_options(argc, argv, NULL, options,
 			     usage, 0);
 
+	/*
+	 * Be forgiving when the enlistment or worktree does not even exist any
+	 * longer; This can be the case if a user deleted the worktree by
+	 * mistake and _still_ wants to unregister the thing.
+	 */
+	if (argc == 1) {
+		struct strbuf src_path = STRBUF_INIT, workdir_path = STRBUF_INIT;
+
+		strbuf_addf(&src_path, "%s/src/.git", argv[0]);
+		strbuf_addf(&workdir_path, "%s/.git", argv[0]);
+		if (!is_directory(src_path.buf) && !is_directory(workdir_path.buf)) {
+			/* remove possible matching registrations */
+			int res = -1;
+
+			strbuf_strip_suffix(&src_path, "/.git");
+			res = remove_deleted_enlistment(&src_path) && res;
+
+			strbuf_strip_suffix(&workdir_path, "/.git");
+			res = remove_deleted_enlistment(&workdir_path) && res;
+
+			strbuf_release(&src_path);
+			strbuf_release(&workdir_path);
+			return res;
+		}
+		strbuf_release(&src_path);
+		strbuf_release(&workdir_path);
+	}
+
 	setup_enlistment_directory(argc, argv, usage, options, NULL);
 
 	return unregister_dir();
contrib/scalar/t/t9099-scalar.sh +15 -0
index 16f2b72b126..ef0e8d680d5 100755
@@ -14,4 +14,19 @@ test_expect_success 'scalar shows a usage' '
 	test_expect_code 129 scalar -h
 '
 
+test_expect_success 'scalar unregister' '
+	git init vanish/src &&
+	scalar register vanish/src &&
+	git config --get --global --fixed-value \
+		maintenance.repo "$(pwd)/vanish/src" &&
+	scalar list >scalar.repos &&
+	grep -F "$(pwd)/vanish/src" scalar.repos &&
+	rm -rf vanish/src/.git &&
+	scalar unregister vanish &&
+	test_must_fail git config --get --global --fixed-value \
+		maintenance.repo "$(pwd)/vanish/src" &&
+	scalar list >scalar.repos &&
+	! grep -F "$(pwd)/vanish/src" scalar.repos
+'
+
 test_done
Signature · 1 line
gitgitgadget
#149 lore raw

PATCH v5 7/15 scalar: implement 'scalar list'

In reply to Johannes Schindelin via GitGitGadget

From: Derrick Stolee <[email protected]>

The produced list simply consists of those repositories registered under the multi-valued scalar.repo config setting in the user's Git config.

---
 contrib/scalar/scalar.c   | 11 +++++++++++
 contrib/scalar/scalar.txt | 11 ++++++++++-
 2 files changed, 21 insertions(+), 1 deletion(-)
contrib/scalar/scalar.c +11 -0
index dc84ce0d5b2..d13eb951c3d 100644
@@ -250,6 +250,16 @@ static int unregister_dir(void)
 	return res;
 }
 
+static int cmd_list(int argc, const char **argv)
+{
+	if (argc != 1)
+		die(_("`scalar list` does not take arguments"));
+
+	if (run_git("config", "--global", "--get-all", "scalar.repo", NULL) < 0)
+		return -1;
+	return 0;
+}
+
 static int cmd_register(int argc, const char **argv)
 {
 	struct option options[] = {
@@ -336,6 +346,7 @@ static struct {
 	const char *name;
 	int (*fn)(int, const char **);
 } builtins[] = {
+	{ "list", cmd_list },
 	{ "register", cmd_register },
 	{ "unregister", cmd_unregister },
 	{ NULL, NULL},
contrib/scalar/scalar.txt +10 -1
index d9a79984492..f93e3d00efd 100644
@@ -8,6 +8,7 @@ scalar - an opinionated repository management tool
 SYNOPSIS
 --------
 [verse]
+scalar list
 scalar register [<enlistment>]
 scalar unregister [<enlistment>]
 
@@ -28,11 +29,19 @@ an existing Git worktree with Scalar whose name is not `src`, the enlistment
 will be identical to the worktree.
 
 The `scalar` command implements various subcommands, and different options
-depending on the subcommand.
+depending on the subcommand. With the exception of `list`, all subcommands
+expect to be run in an enlistment.
 
 COMMANDS
 --------
 
+List
+~~~~
+
+list::
+	List enlistments that are currently registered by Scalar. This
+	subcommand does not need to be run inside an enlistment.
+
 Register
 ~~~~~~~~
 
Signature · 1 line
gitgitgadget
#150 lore raw

PATCH v5 8/15 scalar: implement the `clone` subcommand

In reply to Johannes Schindelin via GitGitGadget

From: Johannes Schindelin <[email protected]>

This implements Scalar's opinionated clone command: it tries to use a partial clone and sets up a sparse checkout by default. In contrast to git clone, scalar clone sets up the worktree in the src/ subdirectory, to encourage a separation between the source files and the build output (which helps Git tremendously because it avoids untracked files that have to be specifically ignored when refreshing the index).

Also, it registers the repository for regular, scheduled maintenance, and configures a flurry of configuration settings based on the experience and experiments of the Microsoft Windows and the Microsoft Office development teams.

Note: since the scalar clone command is by far the most commonly called scalar subcommand, we document it at the top of the manual page.

---
 contrib/scalar/scalar.c          | 201 +++++++++++++++++++++++++++++++
 contrib/scalar/scalar.txt        |  31 ++++-
 contrib/scalar/t/t9099-scalar.sh |  32 +++++
 3 files changed, 261 insertions(+), 3 deletions(-)
contrib/scalar/scalar.c +201 -0
index d13eb951c3d..0401462b9b3 100644
@@ -7,6 +7,7 @@
 #include "parse-options.h"
 #include "config.h"
 #include "run-command.h"
+#include "refs.h"
 
 /*
  * Remove the deepest subdirectory in the provided path string. Path must not
@@ -250,6 +251,205 @@ static int unregister_dir(void)
 	return res;
 }
 
+/* printf-style interface, expects `<key>=<value>` argument */
+static int set_config(const char *fmt, ...)
+{
+	struct strbuf buf = STRBUF_INIT;
+	char *value;
+	int res;
+	va_list args;
+
+	va_start(args, fmt);
+	strbuf_vaddf(&buf, fmt, args);
+	va_end(args);
+
+	value = strchr(buf.buf, '=');
+	if (value)
+		*(value++) = '\0';
+	res = git_config_set_gently(buf.buf, value);
+	strbuf_release(&buf);
+
+	return res;
+}
+
+static char *remote_default_branch(const char *url)
+{
+	struct child_process cp = CHILD_PROCESS_INIT;
+	struct strbuf out = STRBUF_INIT;
+
+	cp.git_cmd = 1;
+	strvec_pushl(&cp.args, "ls-remote", "--symref", url, "HEAD", NULL);
+	if (!pipe_command(&cp, NULL, 0, &out, 0, NULL, 0)) {
+		const char *line = out.buf;
+
+		while (*line) {
+			const char *eol = strchrnul(line, '\n'), *p;
+			size_t len = eol - line;
+			char *branch;
+
+			if (!skip_prefix(line, "ref: ", &p) ||
+			    !strip_suffix_mem(line, &len, "\tHEAD")) {
+				line = eol + (*eol == '\n');
+				continue;
+			}
+
+			eol = line + len;
+			if (skip_prefix(p, "refs/heads/", &p)) {
+				branch = xstrndup(p, eol - p);
+				strbuf_release(&out);
+				return branch;
+			}
+
+			error(_("remote HEAD is not a branch: '%.*s'"),
+			      (int)(eol - p), p);
+			strbuf_release(&out);
+			return NULL;
+		}
+	}
+	warning(_("failed to get default branch name from remote; "
+		  "using local default"));
+	strbuf_reset(&out);
+
+	child_process_init(&cp);
+	cp.git_cmd = 1;
+	strvec_pushl(&cp.args, "symbolic-ref", "--short", "HEAD", NULL);
+	if (!pipe_command(&cp, NULL, 0, &out, 0, NULL, 0)) {
+		strbuf_trim(&out);
+		return strbuf_detach(&out, NULL);
+	}
+
+	strbuf_release(&out);
+	error(_("failed to get default branch name"));
+	return NULL;
+}
+
+static int cmd_clone(int argc, const char **argv)
+{
+	const char *branch = NULL;
+	int full_clone = 0;
+	struct option clone_options[] = {
+		OPT_STRING('b', "branch", &branch, N_("<branch>"),
+			   N_("branch to checkout after clone")),
+		OPT_BOOL(0, "full-clone", &full_clone,
+			 N_("when cloning, create full working directory")),
+		OPT_END(),
+	};
+	const char * const clone_usage[] = {
+		N_("scalar clone [<options>] [--] <repo> [<dir>]"),
+		NULL
+	};
+	const char *url;
+	char *enlistment = NULL, *dir = NULL;
+	struct strbuf buf = STRBUF_INIT;
+	int res;
+
+	argc = parse_options(argc, argv, NULL, clone_options, clone_usage, 0);
+
+	if (argc == 2) {
+		url = argv[0];
+		enlistment = xstrdup(argv[1]);
+	} else if (argc == 1) {
+		url = argv[0];
+
+		strbuf_addstr(&buf, url);
+		/* Strip trailing slashes, if any */
+		while (buf.len > 0 && is_dir_sep(buf.buf[buf.len - 1]))
+			strbuf_setlen(&buf, buf.len - 1);
+		/* Strip suffix `.git`, if any */
+		strbuf_strip_suffix(&buf, ".git");
+
+		enlistment = find_last_dir_sep(buf.buf);
+		if (!enlistment) {
+			die(_("cannot deduce worktree name from '%s'"), url);
+		}
+		enlistment = xstrdup(enlistment + 1);
+	} else {
+		usage_msg_opt(_("You must specify a repository to clone."),
+			      clone_usage, clone_options);
+	}
+
+	if (is_directory(enlistment))
+		die(_("directory '%s' exists already"), enlistment);
+
+	dir = xstrfmt("%s/src", enlistment);
+
+	strbuf_reset(&buf);
+	if (branch)
+		strbuf_addf(&buf, "init.defaultBranch=%s", branch);
+	else {
+		char *b = repo_default_branch_name(the_repository, 1);
+		strbuf_addf(&buf, "init.defaultBranch=%s", b);
+		free(b);
+	}
+
+	if ((res = run_git("-c", buf.buf, "init", "--", dir, NULL)))
+		goto cleanup;
+
+	if (chdir(dir) < 0) {
+		res = error_errno(_("could not switch to '%s'"), dir);
+		goto cleanup;
+	}
+
+	setup_git_directory();
+
+	/* common-main already logs `argv` */
+	trace2_def_repo(the_repository);
+
+	if (!branch && !(branch = remote_default_branch(url))) {
+		res = error(_("failed to get default branch for '%s'"), url);
+		goto cleanup;
+	}
+
+	if (set_config("remote.origin.url=%s", url) ||
+	    set_config("remote.origin.fetch="
+		       "+refs/heads/*:refs/remotes/origin/*") ||
+	    set_config("remote.origin.promisor=true") ||
+	    set_config("remote.origin.partialCloneFilter=blob:none")) {
+		res = error(_("could not configure remote in '%s'"), dir);
+		goto cleanup;
+	}
+
+	if (!full_clone &&
+	    (res = run_git("sparse-checkout", "init", "--cone", NULL)))
+		goto cleanup;
+
+	if (set_recommended_config())
+		return error(_("could not configure '%s'"), dir);
+
+	if ((res = run_git("fetch", "--quiet", "origin", NULL))) {
+		warning(_("partial clone failed; attempting full clone"));
+
+		if (set_config("remote.origin.promisor") ||
+		    set_config("remote.origin.partialCloneFilter")) {
+			res = error(_("could not configure for full clone"));
+			goto cleanup;
+		}
+
+		if ((res = run_git("fetch", "--quiet", "origin", NULL)))
+			goto cleanup;
+	}
+
+	if ((res = set_config("branch.%s.remote=origin", branch)))
+		goto cleanup;
+	if ((res = set_config("branch.%s.merge=refs/heads/%s",
+			      branch, branch)))
+		goto cleanup;
+
+	strbuf_reset(&buf);
+	strbuf_addf(&buf, "origin/%s", branch);
+	res = run_git("checkout", "-f", "-t", buf.buf, NULL);
+	if (res)
+		goto cleanup;
+
+	res = register_dir();
+
+cleanup:
+	free(enlistment);
+	free(dir);
+	strbuf_release(&buf);
+	return res;
+}
+
 static int cmd_list(int argc, const char **argv)
 {
 	if (argc != 1)
@@ -346,6 +546,7 @@ static struct {
 	const char *name;
 	int (*fn)(int, const char **);
 } builtins[] = {
+	{ "clone", cmd_clone },
 	{ "list", cmd_list },
 	{ "register", cmd_register },
 	{ "unregister", cmd_unregister },
contrib/scalar/scalar.txt +28 -3
index f93e3d00efd..d65fb5f1491 100644
@@ -8,6 +8,7 @@ scalar - an opinionated repository management tool
 SYNOPSIS
 --------
 [verse]
+scalar clone [--branch <main-branch>] [--full-clone] <url> [<enlistment>]
 scalar list
 scalar register [<enlistment>]
 scalar unregister [<enlistment>]
@@ -29,12 +30,36 @@ an existing Git worktree with Scalar whose name is not `src`, the enlistment
 will be identical to the worktree.
 
 The `scalar` command implements various subcommands, and different options
-depending on the subcommand. With the exception of `list`, all subcommands
-expect to be run in an enlistment.
+depending on the subcommand. With the exception of `clone` and `list`, all
+subcommands expect to be run in an enlistment.
 
 COMMANDS
 --------
 
+Clone
+~~~~~
+
+clone [<options>] <url> [<enlistment>]::
+	Clones the specified repository, similar to linkgit:git-clone[1]. By
+	default, only commit and tree objects are cloned. Once finished, the
+	worktree is located at `<enlistment>/src`.
++
+The sparse-checkout feature is enabled (except when run with `--full-clone`)
+and the only files present are those in the top-level directory. Use
+`git sparse-checkout set` to expand the set of directories you want to see,
+or `git sparse-checkout disable` to expand to all files (see
+linkgit:git-sparse-checkout[1] for more details). You can explore the
+subdirectories outside your sparse-checkout by using `git ls-tree HEAD`.
+
+-b <name>::
+--branch <name>::
+	Instead of checking out the branch pointed to by the cloned
+	repository's HEAD, check out the `<name>` branch instead.
+
+--[no-]full-clone::
+	A sparse-checkout is initialized by default. This behavior can be
+	turned off via `--full-clone`.
+
 List
 ~~~~
 
@@ -64,7 +89,7 @@ unregister [<enlistment>]::
 
 SEE ALSO
 --------
-linkgit:git-maintenance[1].
+linkgit:git-clone[1], linkgit:git-maintenance[1].
 
 Scalar
 ---
contrib/scalar/t/t9099-scalar.sh +32 -0
index ef0e8d680d5..295398f62cc 100755
@@ -10,6 +10,9 @@ PATH=$PWD/..:$PATH
 
 . ../../../t/test-lib.sh
 
+GIT_TEST_MAINT_SCHEDULER="crontab:test-tool crontab ../cron.txt"
+export GIT_TEST_MAINT_SCHEDULER
+
 test_expect_success 'scalar shows a usage' '
 	test_expect_code 129 scalar -h
 '
@@ -29,4 +32,33 @@ test_expect_success 'scalar unregister' '
 	! grep -F "$(pwd)/vanish/src" scalar.repos
 '
 
+test_expect_success 'set up repository to clone' '
+	test_commit first &&
+	test_commit second &&
+	test_commit third &&
+	git switch -c parallel first &&
+	mkdir -p 1/2 &&
+	test_commit 1/2/3 &&
+	git config uploadPack.allowFilter true &&
+	git config uploadPack.allowAnySHA1InWant true
+'
+
+test_expect_success 'scalar clone' '
+	second=$(git rev-parse --verify second:second.t) &&
+	scalar clone "file://$(pwd)" cloned &&
+	(
+		cd cloned/src &&
+
+		git config --get --global --fixed-value maintenance.repo \
+			"$(pwd)" &&
+
+		test_path_is_missing 1/2 &&
+		test_must_fail git rev-list --missing=print $second &&
+		git rev-list $second &&
+		git cat-file blob $second >actual &&
+		echo "second" >expect &&
+		test_cmp expect actual
+	)
+'
+
 test_done
Signature · 1 line
gitgitgadget
#152 lore raw

PATCH v5 10/15 scalar: implement the `run` command

In reply to Johannes Schindelin via GitGitGadget

From: Derrick Stolee <[email protected]>

Note: this subcommand is provided primarily for backwards-compatibility, for existing Scalar uses. It is mostly just a shim for git maintenance, mapping task names from the way Scalar called them to the way Git calls them.

The reason why those names differ? The background maintenance was first implemented in Scalar, and when it was contributed as a patch series implementing the git maintenance command, reviewers suggested better names, those suggestions were accepted before the patches were integrated into core Git.

---
 contrib/scalar/scalar.c   | 64 +++++++++++++++++++++++++++++++++++++++
 contrib/scalar/scalar.txt | 19 ++++++++++++
 2 files changed, 83 insertions(+)
contrib/scalar/scalar.c +64 -0
index 754e19d781d..d9631287e12 100644
@@ -483,6 +483,69 @@ static int cmd_register(int argc, const char **argv)
 	return register_dir();
 }
 
+static int cmd_run(int argc, const char **argv)
+{
+	struct option options[] = {
+		OPT_END(),
+	};
+	struct {
+		const char *arg, *task;
+	} tasks[] = {
+		{ "config", NULL },
+		{ "commit-graph", "commit-graph" },
+		{ "fetch", "prefetch" },
+		{ "loose-objects", "loose-objects" },
+		{ "pack-files", "incremental-repack" },
+		{ NULL, NULL }
+	};
+	struct strbuf buf = STRBUF_INIT;
+	const char *usagestr[] = { NULL, NULL };
+	int i;
+
+	strbuf_addstr(&buf, N_("scalar run <task> [<enlistment>]\nTasks:\n"));
+	for (i = 0; tasks[i].arg; i++)
+		strbuf_addf(&buf, "\t%s\n", tasks[i].arg);
+	usagestr[0] = buf.buf;
+
+	argc = parse_options(argc, argv, NULL, options,
+			     usagestr, 0);
+
+	if (!argc)
+		usage_with_options(usagestr, options);
+
+	if (!strcmp("all", argv[0])) {
+		i = -1;
+	} else {
+		for (i = 0; tasks[i].arg && strcmp(tasks[i].arg, argv[0]); i++)
+			; /* keep looking for the task */
+
+		if (i > 0 && !tasks[i].arg) {
+			error(_("no such task: '%s'"), argv[0]);
+			usage_with_options(usagestr, options);
+		}
+	}
+
+	argc--;
+	argv++;
+	setup_enlistment_directory(argc, argv, usagestr, options, NULL);
+	strbuf_release(&buf);
+
+	if (i == 0)
+		return register_dir();
+
+	if (i > 0)
+		return run_git("maintenance", "run",
+			       "--task", tasks[i].task, NULL);
+
+	if (register_dir())
+		return -1;
+	for (i = 1; tasks[i].arg; i++)
+		if (run_git("maintenance", "run",
+			    "--task", tasks[i].task, NULL))
+			return -1;
+	return 0;
+}
+
 static int remove_deleted_enlistment(struct strbuf *path)
 {
 	int res = 0;
@@ -555,6 +618,7 @@ static struct {
 	{ "list", cmd_list },
 	{ "register", cmd_register },
 	{ "unregister", cmd_unregister },
+	{ "run", cmd_run },
 	{ NULL, NULL},
 };
 
contrib/scalar/scalar.txt +19 -0
index 46999cf7c84..f139a14445d 100644
@@ -12,6 +12,7 @@ scalar clone [--single-branch] [--branch <main-branch>] [--full-clone] <url> [<e
 scalar list
 scalar register [<enlistment>]
 scalar unregister [<enlistment>]
+scalar run ( all | config | commit-graph | fetch | loose-objects | pack-files ) [<enlistment>]
 
 DESCRIPTION
 -----------
@@ -97,6 +98,24 @@ unregister [<enlistment>]::
 	Remove the specified repository from the list of repositories
 	registered with Scalar and stop the scheduled background maintenance.
 
+Run
+~~~
+
+scalar run ( all | config | commit-graph | fetch | loose-objects | pack-files ) [<enlistment>]::
+	Run the given maintenance task (or all tasks, if `all` was specified).
+	Except for `all` and `config`, this subcommand simply hands off to
+	linkgit:git-maintenance[1] (mapping `fetch` to `prefetch` and
+	`pack-files` to `incremental-repack`).
++
+These tasks are run automatically as part of the scheduled maintenance,
+as soon as the repository is registered with Scalar. It should therefore
+not be necessary to run this subcommand manually.
++
+The `config` task is specific to Scalar and configures all those
+opinionated default settings that make Git work more efficiently with
+large repositories. As this task is run as part of `scalar clone`
+automatically, explicit invocations of this task are rarely needed.
+
 SEE ALSO
 --------
 linkgit:git-clone[1], linkgit:git-maintenance[1].
Signature · 1 line
gitgitgadget
#151 lore raw

PATCH v5 9/15 scalar: teach 'clone' to support the --single-branch option

In reply to Johannes Schindelin via GitGitGadget

From: Johannes Schindelin <[email protected]>

Just like git clone, the scalar clone command now also offers to restrict the clone to a single branch.

---
 contrib/scalar/scalar.c          |  9 +++++++--
 contrib/scalar/scalar.txt        | 12 +++++++++++-
 contrib/scalar/t/t9099-scalar.sh |  6 +++++-
 3 files changed, 23 insertions(+), 4 deletions(-)
contrib/scalar/scalar.c +7 -2
index 0401462b9b3..754e19d781d 100644
@@ -326,12 +326,15 @@ static char *remote_default_branch(const char *url)
 static int cmd_clone(int argc, const char **argv)
 {
 	const char *branch = NULL;
-	int full_clone = 0;
+	int full_clone = 0, single_branch = 0;
 	struct option clone_options[] = {
 		OPT_STRING('b', "branch", &branch, N_("<branch>"),
 			   N_("branch to checkout after clone")),
 		OPT_BOOL(0, "full-clone", &full_clone,
 			 N_("when cloning, create full working directory")),
+		OPT_BOOL(0, "single-branch", &single_branch,
+			 N_("only download metadata for the branch that will "
+			    "be checked out")),
 		OPT_END(),
 	};
 	const char * const clone_usage[] = {
@@ -402,7 +405,9 @@ static int cmd_clone(int argc, const char **argv)
 
 	if (set_config("remote.origin.url=%s", url) ||
 	    set_config("remote.origin.fetch="
-		       "+refs/heads/*:refs/remotes/origin/*") ||
+		       "+refs/heads/%s:refs/remotes/origin/%s",
+		       single_branch ? branch : "*",
+		       single_branch ? branch : "*") ||
 	    set_config("remote.origin.promisor=true") ||
 	    set_config("remote.origin.partialCloneFilter=blob:none")) {
 		res = error(_("could not configure remote in '%s'"), dir);
contrib/scalar/scalar.txt +11 -1
index d65fb5f1491..46999cf7c84 100644
@@ -8,7 +8,7 @@ scalar - an opinionated repository management tool
 SYNOPSIS
 --------
 [verse]
-scalar clone [--branch <main-branch>] [--full-clone] <url> [<enlistment>]
+scalar clone [--single-branch] [--branch <main-branch>] [--full-clone] <url> [<enlistment>]
 scalar list
 scalar register [<enlistment>]
 scalar unregister [<enlistment>]
@@ -56,6 +56,16 @@ subdirectories outside your sparse-checkout by using `git ls-tree HEAD`.
 	Instead of checking out the branch pointed to by the cloned
 	repository's HEAD, check out the `<name>` branch instead.
 
+--[no-]single-branch::
+	Clone only the history leading to the tip of a single branch, either
+	specified by the `--branch` option or the primary branch remote's
+	`HEAD` points at.
++
+Further fetches into the resulting repository will only update the
+remote-tracking branch for the branch this option was used for the initial
+cloning. If the HEAD at the remote did not point at any branch when
+`--single-branch` clone was made, no remote-tracking branch is created.
+
 --[no-]full-clone::
 	A sparse-checkout is initialized by default. This behavior can be
 	turned off via `--full-clone`.
contrib/scalar/t/t9099-scalar.sh +5 -1
index 295398f62cc..9a35ab4fde6 100755
@@ -45,13 +45,17 @@ test_expect_success 'set up repository to clone' '
 
 test_expect_success 'scalar clone' '
 	second=$(git rev-parse --verify second:second.t) &&
-	scalar clone "file://$(pwd)" cloned &&
+	scalar clone "file://$(pwd)" cloned --single-branch &&
 	(
 		cd cloned/src &&
 
 		git config --get --global --fixed-value maintenance.repo \
 			"$(pwd)" &&
 
+		git for-each-ref --format="%(refname)" refs/remotes/origin/ >actual &&
+		echo "refs/remotes/origin/parallel" >expect &&
+		test_cmp expect actual &&
+
 		test_path_is_missing 1/2 &&
 		test_must_fail git rev-list --missing=print $second &&
 		git rev-list $second &&
Signature · 1 line
gitgitgadget
#153 lore raw

PATCH v5 11/15 scalar: allow reconfiguring an existing enlistment

In reply to Johannes Schindelin via GitGitGadget

From: Johannes Schindelin <[email protected]>

This comes in handy during Scalar upgrades, or when config settings were messed up by mistake.

---
 contrib/scalar/scalar.c          | 79 +++++++++++++++++++++-----------
 contrib/scalar/scalar.txt        |  8 ++++
 contrib/scalar/t/t9099-scalar.sh |  8 ++++
 3 files changed, 67 insertions(+), 28 deletions(-)
contrib/scalar/scalar.c +51 -28
index d9631287e12..6bebba0b51f 100644
@@ -108,18 +108,20 @@ static int run_git(const char *arg, ...)
 	return res;
 }
 
-static int set_recommended_config(void)
+static int set_recommended_config(int reconfigure)
 {
 	struct {
 		const char *key;
 		const char *value;
+		int overwrite_on_reconfigure;
 	} config[] = {
-		{ "am.keepCR", "true" },
-		{ "core.FSCache", "true" },
-		{ "core.multiPackIndex", "true" },
-		{ "core.preloadIndex", "true" },
+		/* Required */
+		{ "am.keepCR", "true", 1 },
+		{ "core.FSCache", "true", 1 },
+		{ "core.multiPackIndex", "true", 1 },
+		{ "core.preloadIndex", "true", 1 },
 #ifndef WIN32
-		{ "core.untrackedCache", "true" },
+		{ "core.untrackedCache", "true", 1 },
 #else
 		/*
 		 * Unfortunately, Scalar's Functional Tests demonstrated
@@ -133,28 +135,29 @@ static int set_recommended_config(void)
 		 * Therefore, with a sad heart, we disable this very useful
 		 * feature on Windows.
 		 */
-		{ "core.untrackedCache", "false" },
+		{ "core.untrackedCache", "false", 1 },
 #endif
-		{ "core.logAllRefUpdates", "true" },
-		{ "credential.https://dev.azure.com.useHttpPath", "true" },
-		{ "credential.validate", "false" }, /* GCM4W-only */
-		{ "gc.auto", "0" },
-		{ "gui.GCWarning", "false" },
-		{ "index.threads", "true" },
-		{ "index.version", "4" },
-		{ "merge.stat", "false" },
-		{ "merge.renames", "false" },
-		{ "pack.useBitmaps", "false" },
-		{ "pack.useSparse", "true" },
-		{ "receive.autoGC", "false" },
-		{ "reset.quiet", "true" },
-		{ "feature.manyFiles", "false" },
-		{ "feature.experimental", "false" },
-		{ "fetch.unpackLimit", "1" },
-		{ "fetch.writeCommitGraph", "false" },
+		{ "core.logAllRefUpdates", "true", 1 },
+		{ "credential.https://dev.azure.com.useHttpPath", "true", 1 },
+		{ "credential.validate", "false", 1 }, /* GCM4W-only */
+		{ "gc.auto", "0", 1 },
+		{ "gui.GCWarning", "false", 1 },
+		{ "index.threads", "true", 1 },
+		{ "index.version", "4", 1 },
+		{ "merge.stat", "false", 1 },
+		{ "merge.renames", "false", 1 },
+		{ "pack.useBitmaps", "false", 1 },
+		{ "pack.useSparse", "true", 1 },
+		{ "receive.autoGC", "false", 1 },
+		{ "reset.quiet", "true", 1 },
+		{ "feature.manyFiles", "false", 1 },
+		{ "feature.experimental", "false", 1 },
+		{ "fetch.unpackLimit", "1", 1 },
+		{ "fetch.writeCommitGraph", "false", 1 },
 #ifdef WIN32
-		{ "http.sslBackend", "schannel" },
+		{ "http.sslBackend", "schannel", 1 },
 #endif
+		/* Optional */
 		{ "status.aheadBehind", "false" },
 		{ "commitGraph.generationVersion", "1" },
 		{ "core.autoCRLF", "false" },
@@ -165,7 +168,8 @@ static int set_recommended_config(void)
 	char *value;
 
 	for (i = 0; config[i].key; i++) {
-		if (git_config_get_string(config[i].key, &value)) {
+		if ((reconfigure && config[i].overwrite_on_reconfigure) ||
+		    git_config_get_string(config[i].key, &value)) {
 			trace2_data_string("scalar", the_repository, config[i].key, "created");
 			if (git_config_set_gently(config[i].key,
 						  config[i].value) < 0)
@@ -230,7 +234,7 @@ static int register_dir(void)
 	int res = add_or_remove_enlistment(1);
 
 	if (!res)
-		res = set_recommended_config();
+		res = set_recommended_config(0);
 
 	if (!res)
 		res = toggle_maintenance(1);
@@ -418,7 +422,7 @@ static int cmd_clone(int argc, const char **argv)
 	    (res = run_git("sparse-checkout", "init", "--cone", NULL)))
 		goto cleanup;
 
-	if (set_recommended_config())
+	if (set_recommended_config(0))
 		return error(_("could not configure '%s'"), dir);
 
 	if ((res = run_git("fetch", "--quiet", "origin", NULL))) {
@@ -483,6 +487,24 @@ static int cmd_register(int argc, const char **argv)
 	return register_dir();
 }
 
+static int cmd_reconfigure(int argc, const char **argv)
+{
+	struct option options[] = {
+		OPT_END(),
+	};
+	const char * const usage[] = {
+		N_("scalar reconfigure [<enlistment>]"),
+		NULL
+	};
+
+	argc = parse_options(argc, argv, NULL, options,
+			     usage, 0);
+
+	setup_enlistment_directory(argc, argv, usage, options, NULL);
+
+	return set_recommended_config(1);
+}
+
 static int cmd_run(int argc, const char **argv)
 {
 	struct option options[] = {
@@ -619,6 +641,7 @@ static struct {
 	{ "register", cmd_register },
 	{ "unregister", cmd_unregister },
 	{ "run", cmd_run },
+	{ "reconfigure", cmd_reconfigure },
 	{ NULL, NULL},
 };
 
contrib/scalar/scalar.txt +8 -0
index f139a14445d..f4e4686e8c8 100644
@@ -13,6 +13,7 @@ scalar list
 scalar register [<enlistment>]
 scalar unregister [<enlistment>]
 scalar run ( all | config | commit-graph | fetch | loose-objects | pack-files ) [<enlistment>]
+scalar reconfigure <enlistment>
 
 DESCRIPTION
 -----------
@@ -116,6 +117,13 @@ opinionated default settings that make Git work more efficiently with
 large repositories. As this task is run as part of `scalar clone`
 automatically, explicit invocations of this task are rarely needed.
 
+Reconfigure
+~~~~~~~~~~~
+
+After a Scalar upgrade, or when the configuration of a Scalar enlistment
+was somehow corrupted or changed by mistake, this subcommand allows to
+reconfigure the enlistment.
+
 SEE ALSO
 --------
 linkgit:git-clone[1], linkgit:git-maintenance[1].
contrib/scalar/t/t9099-scalar.sh +8 -0
index 9a35ab4fde6..e6d74a06ca0 100755
@@ -65,4 +65,12 @@ test_expect_success 'scalar clone' '
 	)
 '
 
+test_expect_success 'scalar reconfigure' '
+	git init one/src &&
+	scalar register one &&
+	git -C one/src config core.preloadIndex false &&
+	scalar reconfigure one &&
+	test true = "$(git -C one/src config core.preloadIndex)"
+'
+
 test_done
Signature · 1 line
gitgitgadget
#156 lore raw

PATCH v5 14/15 scalar: implement the `version` command

In reply to Johannes Schindelin via GitGitGadget

From: Johannes Schindelin <[email protected]>

The .NET version of Scalar has a version command. This was necessary because it was versioned independently of Git.

Since Scalar is now tightly coupled with Git, it does not make sense for them to show different versions. Therefore, it shows the same output as git version. For backwards-compatibility with the .NET version, scalar version prints to stderr, though (git version prints to stdout instead).

---
 contrib/scalar/scalar.c | 39 +++++++++++++++++++++++++++++++++++++++
 1 file changed, 39 insertions(+)
contrib/scalar/scalar.c +39 -0
index 8aaeca7cc64..b2e92cf63b5 100644
@@ -10,6 +10,7 @@
 #include "refs.h"
 #include "dir.h"
 #include "packfile.h"
+#include "help.h"
 
 /*
  * Remove the deepest subdirectory in the provided path string. Path must not
@@ -356,6 +357,15 @@ static int delete_enlistment(struct strbuf *enlistment)
 	return 0;
 }
 
+/*
+ * Dummy implementation; Using `get_version_info()` would cause a link error
+ * without this.
+ */
+void load_builtin_commands(const char *prefix, struct cmdnames *cmds)
+{
+	die("not implemented");
+}
+
 static int cmd_clone(int argc, const char **argv)
 {
 	const char *branch = NULL;
@@ -749,6 +759,34 @@ static int cmd_delete(int argc, const char **argv)
 	return res;
 }
 
+static int cmd_version(int argc, const char **argv)
+{
+	int verbose = 0, build_options = 0;
+	struct option options[] = {
+		OPT__VERBOSE(&verbose, N_("include Git version")),
+		OPT_BOOL(0, "build-options", &build_options,
+			 N_("include Git's build options")),
+		OPT_END(),
+	};
+	const char * const usage[] = {
+		N_("scalar verbose [-v | --verbose] [--build-options]"),
+		NULL
+	};
+	struct strbuf buf = STRBUF_INIT;
+
+	argc = parse_options(argc, argv, NULL, options,
+			     usage, 0);
+
+	if (argc != 0)
+		usage_with_options(usage, options);
+
+	get_version_info(&buf, build_options);
+	fprintf(stderr, "%s\n", buf.buf);
+	strbuf_release(&buf);
+
+	return 0;
+}
+
 static struct {
 	const char *name;
 	int (*fn)(int, const char **);
@@ -760,6 +798,7 @@ static struct {
 	{ "run", cmd_run },
 	{ "reconfigure", cmd_reconfigure },
 	{ "delete", cmd_delete },
+	{ "version", cmd_version },
 	{ NULL, NULL},
 };
 
Signature · 1 line
gitgitgadget
#154 lore raw

PATCH v5 12/15 scalar: teach 'reconfigure' to optionally handle all registered enlistments

In reply to Johannes Schindelin via GitGitGadget

From: Johannes Schindelin <[email protected]>

After a Scalar upgrade, it can come in really handy if there is an easy way to reconfigure all Scalar enlistments. This new option offers this functionality.

---
 contrib/scalar/scalar.c          | 61 ++++++++++++++++++++++++++++++--
 contrib/scalar/scalar.txt        |  9 +++--
 contrib/scalar/t/t9099-scalar.sh |  3 ++
 3 files changed, 67 insertions(+), 6 deletions(-)
contrib/scalar/scalar.c +58 -3
index 6bebba0b51f..234a7dce479 100644
@@ -487,22 +487,77 @@ static int cmd_register(int argc, const char **argv)
 	return register_dir();
 }
 
+static int get_scalar_repos(const char *key, const char *value, void *data)
+{
+	struct string_list *list = data;
+
+	if (!strcmp(key, "scalar.repo"))
+		string_list_append(list, value);
+
+	return 0;
+}
+
 static int cmd_reconfigure(int argc, const char **argv)
 {
+	int all = 0;
 	struct option options[] = {
+		OPT_BOOL('a', "all", &all,
+			 N_("reconfigure all registered enlistments")),
 		OPT_END(),
 	};
 	const char * const usage[] = {
-		N_("scalar reconfigure [<enlistment>]"),
+		N_("scalar reconfigure [--all | <enlistment>]"),
 		NULL
 	};
+	struct string_list scalar_repos = STRING_LIST_INIT_DUP;
+	int i, res = 0;
+	struct repository r = { NULL };
+	struct strbuf commondir = STRBUF_INIT, gitdir = STRBUF_INIT;
 
 	argc = parse_options(argc, argv, NULL, options,
 			     usage, 0);
 
-	setup_enlistment_directory(argc, argv, usage, options, NULL);
+	if (!all) {
+		setup_enlistment_directory(argc, argv, usage, options, NULL);
+
+		return set_recommended_config(1);
+	}
+
+	if (argc > 0)
+		usage_msg_opt(_("--all or <enlistment>, but not both"),
+			      usage, options);
+
+	git_config(get_scalar_repos, &scalar_repos);
 
-	return set_recommended_config(1);
+	for (i = 0; i < scalar_repos.nr; i++) {
+		const char *dir = scalar_repos.items[i].string;
+
+		strbuf_reset(&commondir);
+		strbuf_reset(&gitdir);
+
+		if (chdir(dir) < 0) {
+			warning_errno(_("could not switch to '%s'"), dir);
+			res = -1;
+		} else if (discover_git_directory(&commondir, &gitdir) < 0) {
+			warning_errno(_("git repository gone in '%s'"), dir);
+			res = -1;
+		} else {
+			git_config_clear();
+
+			the_repository = &r;
+			r.commondir = commondir.buf;
+			r.gitdir = gitdir.buf;
+
+			if (set_recommended_config(1) < 0)
+				res = -1;
+		}
+	}
+
+	string_list_clear(&scalar_repos, 1);
+	strbuf_release(&commondir);
+	strbuf_release(&gitdir);
+
+	return res;
 }
 
 static int cmd_run(int argc, const char **argv)
contrib/scalar/scalar.txt +6 -3
index f4e4686e8c8..2fa96fcabc6 100644
@@ -13,7 +13,7 @@ scalar list
 scalar register [<enlistment>]
 scalar unregister [<enlistment>]
 scalar run ( all | config | commit-graph | fetch | loose-objects | pack-files ) [<enlistment>]
-scalar reconfigure <enlistment>
+scalar reconfigure [ --all | <enlistment> ]
 
 DESCRIPTION
 -----------
@@ -32,8 +32,8 @@ an existing Git worktree with Scalar whose name is not `src`, the enlistment
 will be identical to the worktree.
 
 The `scalar` command implements various subcommands, and different options
-depending on the subcommand. With the exception of `clone` and `list`, all
-subcommands expect to be run in an enlistment.
+depending on the subcommand. With the exception of `clone`, `list` and
+`reconfigure --all`, all subcommands expect to be run in an enlistment.
 
 COMMANDS
 --------
@@ -124,6 +124,9 @@ After a Scalar upgrade, or when the configuration of a Scalar enlistment
 was somehow corrupted or changed by mistake, this subcommand allows to
 reconfigure the enlistment.
 
+With the `--all` option, all enlistments currently registered with Scalar
+will be reconfigured. Use this option after each Scalar upgrade.
+
 SEE ALSO
 --------
 linkgit:git-clone[1], linkgit:git-maintenance[1].
contrib/scalar/t/t9099-scalar.sh +3 -0
index e6d74a06ca0..5fe7fabd0e5 100755
@@ -70,6 +70,9 @@ test_expect_success 'scalar reconfigure' '
 	scalar register one &&
 	git -C one/src config core.preloadIndex false &&
 	scalar reconfigure one &&
+	test true = "$(git -C one/src config core.preloadIndex)" &&
+	git -C one/src config core.preloadIndex false &&
+	scalar reconfigure -a &&
 	test true = "$(git -C one/src config core.preloadIndex)"
 '
 
Signature · 1 line
gitgitgadget
#155 lore raw

PATCH v5 13/15 scalar: implement the `delete` command

In reply to Johannes Schindelin via GitGitGadget

From: Matthew John Cheetham <[email protected]>

Delete an enlistment by first unregistering the repository and then deleting the enlistment directory (usually the directory containing the worktree src/ directory).

On Windows, if the current directory is inside the enlistment's directory, change to the parent of the enlistment directory, to allow us to delete the enlistment (directories used by processes e.g. as current working directories cannot be deleted on Windows).

---
 contrib/scalar/scalar.c          | 63 ++++++++++++++++++++++++++++++++
 contrib/scalar/scalar.txt        |  8 ++++
 contrib/scalar/t/t9099-scalar.sh |  9 +++++
 3 files changed, 80 insertions(+)
contrib/scalar/scalar.c +63 -0
index 234a7dce479..8aaeca7cc64 100644
@@ -8,6 +8,8 @@
 #include "config.h"
 #include "run-command.h"
 #include "refs.h"
+#include "dir.h"
+#include "packfile.h"
 
 /*
  * Remove the deepest subdirectory in the provided path string. Path must not
@@ -327,6 +329,33 @@ static char *remote_default_branch(const char *url)
 	return NULL;
 }
 
+static int delete_enlistment(struct strbuf *enlistment)
+{
+#ifdef WIN32
+	struct strbuf parent = STRBUF_INIT;
+#endif
+
+	if (unregister_dir())
+		die(_("failed to unregister repository"));
+
+#ifdef WIN32
+	/*
+	 * Change the current directory to one outside of the enlistment so
+	 * that we may delete everything underneath it.
+	 */
+	strbuf_addbuf(&parent, enlistment);
+	strbuf_parent_directory(&parent);
+	if (chdir(parent.buf) < 0)
+		die_errno(_("could not switch to '%s'"), parent.buf);
+	strbuf_release(&parent);
+#endif
+
+	if (remove_dir_recursively(enlistment, 0))
+		die(_("failed to delete enlistment directory"));
+
+	return 0;
+}
+
 static int cmd_clone(int argc, const char **argv)
 {
 	const char *branch = NULL;
@@ -687,6 +716,39 @@ static int cmd_unregister(int argc, const char **argv)
 	return unregister_dir();
 }
 
+static int cmd_delete(int argc, const char **argv)
+{
+	char *cwd = xgetcwd();
+	struct option options[] = {
+		OPT_END(),
+	};
+	const char * const usage[] = {
+		N_("scalar delete <enlistment>"),
+		NULL
+	};
+	struct strbuf enlistment = STRBUF_INIT;
+	int res = 0;
+
+	argc = parse_options(argc, argv, NULL, options,
+			     usage, 0);
+
+	if (argc != 1)
+		usage_with_options(usage, options);
+
+	setup_enlistment_directory(argc, argv, usage, options, &enlistment);
+
+	if (dir_inside_of(cwd, enlistment.buf) >= 0)
+		res = error(_("refusing to delete current working directory"));
+	else {
+		close_object_store(the_repository->objects);
+		res = delete_enlistment(&enlistment);
+	}
+	strbuf_release(&enlistment);
+	free(cwd);
+
+	return res;
+}
+
 static struct {
 	const char *name;
 	int (*fn)(int, const char **);
@@ -697,6 +759,7 @@ static struct {
 	{ "unregister", cmd_unregister },
 	{ "run", cmd_run },
 	{ "reconfigure", cmd_reconfigure },
+	{ "delete", cmd_delete },
 	{ NULL, NULL},
 };
 
contrib/scalar/scalar.txt +8 -0
index 2fa96fcabc6..6fc57707718 100644
@@ -14,6 +14,7 @@ scalar register [<enlistment>]
 scalar unregister [<enlistment>]
 scalar run ( all | config | commit-graph | fetch | loose-objects | pack-files ) [<enlistment>]
 scalar reconfigure [ --all | <enlistment> ]
+scalar delete <enlistment>
 
 DESCRIPTION
 -----------
@@ -127,6 +128,13 @@ reconfigure the enlistment.
 With the `--all` option, all enlistments currently registered with Scalar
 will be reconfigured. Use this option after each Scalar upgrade.
 
+Delete
+~~~~~~
+
+delete <enlistment>::
+	This subcommand lets you delete an existing Scalar enlistment from your
+	local file system, unregistering the repository.
+
 SEE ALSO
 --------
 linkgit:git-clone[1], linkgit:git-maintenance[1].
contrib/scalar/t/t9099-scalar.sh +9 -0
index 5fe7fabd0e5..7e8771d0eff 100755
@@ -76,4 +76,13 @@ test_expect_success 'scalar reconfigure' '
 	test true = "$(git -C one/src config core.preloadIndex)"
 '
 
+test_expect_success 'scalar delete without enlistment shows a usage' '
+	test_expect_code 129 scalar delete
+'
+
+test_expect_success 'scalar delete with enlistment' '
+	scalar delete cloned &&
+	test_path_is_missing cloned
+'
+
 test_done
Signature · 1 line
gitgitgadget
#157 lore raw

PATCH v5 15/15 scalar: accept -C and -c options before the subcommand

In reply to Johannes Schindelin via GitGitGadget

From: Johannes Schindelin <[email protected]>

The git executable has these two very useful options:

-C <directory>:
switch to the specified directory before performing any actions

-c <key>=<value>:
temporarily configure this setting for the duration of the specified scalar subcommand

With this commit, we teach the scalar executable the same trick.

---
 contrib/scalar/scalar.c   | 22 +++++++++++++++++++++-
 contrib/scalar/scalar.txt | 10 ++++++++++
 2 files changed, 31 insertions(+), 1 deletion(-)
contrib/scalar/scalar.c +21 -1
index b2e92cf63b5..6c496318bd4 100644
@@ -807,6 +807,25 @@ int cmd_main(int argc, const char **argv)
 	struct strbuf scalar_usage = STRBUF_INIT;
 	int i;
 
+	while (argc > 1 && *argv[1] == '-') {
+		if (!strcmp(argv[1], "-C")) {
+			if (argc < 3)
+				die(_("-C requires a <directory>"));
+			if (chdir(argv[2]) < 0)
+				die_errno(_("could not change to '%s'"),
+					  argv[2]);
+			argc -= 2;
+			argv += 2;
+		} else if (!strcmp(argv[1], "-c")) {
+			if (argc < 3)
+				die(_("-c requires a <key>=<value> argument"));
+			git_config_push_parameter(argv[2]);
+			argc -= 2;
+			argv += 2;
+		} else
+			break;
+	}
+
 	if (argc > 1) {
 		argv++;
 		argc--;
@@ -817,7 +836,8 @@ int cmd_main(int argc, const char **argv)
 	}
 
 	strbuf_addstr(&scalar_usage,
-		      N_("scalar <command> [<options>]\n\nCommands:\n"));
+		      N_("scalar [-C <directory>] [-c <key>=<value>] "
+			 "<command> [<options>]\n\nCommands:\n"));
 	for (i = 0; builtins[i].name; i++)
 		strbuf_addf(&scalar_usage, "\t%s\n", builtins[i].name);
 
contrib/scalar/scalar.txt +10 -0
index 6fc57707718..3a80f829edc 100644
@@ -36,6 +36,16 @@ The `scalar` command implements various subcommands, and different options
 depending on the subcommand. With the exception of `clone`, `list` and
 `reconfigure --all`, all subcommands expect to be run in an enlistment.
 
+The following options can be specified _before_ the subcommand:
+
+-C <directory>::
+	Before running the subcommand, change the working directory. This
+	option imitates the same option of linkgit:git[1].
+
+-c <key>=<value>::
+	For the duration of running the specified subcommand, configure this
+	setting. This option imitates the same option of linkgit:git[1].
+
 COMMANDS
 --------
 
Signature · 1 line
gitgitgadget
#158 lore raw
In reply to Johannes Schindelin via GitGitGadget
Quoted 1 line · Johannes Schindelin via GitGitGadget

* The OBJECTS list in the Makefile will now include Scalar.

So that looks like a partial fix for what I brought up in [1] [...]

Quoted 16 lines

Range-diff vs v4:

1: 852ec003109 ! 1: 7119a8efc21 scalar: create a rudimentary executable
@@ Commit message

## Makefile ##
-@@ Makefile: endif
- .PHONY: objects
- objects: $(OBJECTS)
-
+@@ Makefile: OBJECTS += $(FUZZ_OBJS)
+ ifndef NO_CURL
+ OBJECTS += http.o http-walker.o remote-curl.o
+ endif
++

Quoted 8 lines

+SCALAR_SOURCES := contrib/scalar/scalar.c +SCALAR_OBJECTS := $(SCALAR_SOURCES:c=o)
+OBJECTS += $(SCALAR_OBJECTS)
+
- dep_files := $(foreach f,$(OBJECTS),$(dir $f).depend/$(notdir $f).d)
- dep_dirs := $(addsuffix .depend,$(sort $(dir $(OBJECTS))))
+ .PHONY: objects
+ objects: $(OBJECTS)

Except that this & contrib/scalar/Makefile is still broken in multiple ways. We now have two Makefiles that can build contrib/scalar/scalar:

touch advice.h; make -j8 contrib/scalar/scalar

But try:

    $ touch advice.h; (cd contrib/scalar 2>/dev/null && make scalar)
    make: 'scalar' is up to date.

I.e. (I'm presuming in response to what I brought up in [1]) the depenency graph in the top-level Makefile is correct in this specific area. But it understands the ".depends" files (depending on COMPUTE_HEADER_DEPENDENCIES), your sub-Makefile doesn't.

There's similar whack-a-mole issues in other areas, e.g.:

make -C contrib/scalar/ test

Will break or not depending on whether you've built the top-level git.

I noticed at least one other subtle breakage (first thing I checked after those two).

I'm happy to send you a working patch to integrate that fixes all these issues, it also integrates with "make install", this series leaves us with a "scalar" binary, but no way to install it, if we just piggy-back on the existing installation procedure.

The side-thread on the v3[3] that you most recently replied to is conflating some suggestion of shipping this as a built-in, with the purely build-system implementation details I'm suggesting here.

I did mention using it as a built-in in [4], but for the semi-related issue of scalar.c copy/pasting less code from git.c. But that was in the context of such a thing being purely a non-visible implementation detail. I.e. it would still be "scalar", not "git scalar".

*That* suggestion is just a side-musing about whether it would be easier to teach git.c to inspects its argv and have a special-case for dispatching to cmd_scalar(), a user would never know the difference. I think that might also be worthwhile, but I care *way* less about that than making maintaining the Makefile a hassle, and it's an entirely orthogonal suggestion.

1. https://lore.kernel.org/git/[email protected]/
2. https://lore.kernel.org/git/[email protected]/
3. https://lore.kernel.org/git/[email protected]
4. https://lore.kernel.org/git/[email protected]/

#27 lore raw

Re: [PATCH 08/15] scalar: implement the `clone` subcommand

In reply to Johannes Schindelin
Quoted 31 lines · Johannes Schindelin

Hi Elijah,

Quoted 17 lines · Elijah Newren
Quoted 1 line · Johannes Schindelin via GitGitGadget

...

Quoted 13 lines

COMMANDS
--------

contrib/scalar/scalar.txt +10 -0
+Clone
+~~~~~
+
+clone [<options>] <url> [<enlistment>]::
+    Clones the specified repository, similar to linkgit:git-clone[1]. By
+    default, only commit and tree objects are cloned. Once finished, the
+    worktree is located at `<enlistment>/src`.
++
+The sparse-checkout feature is enabled (except when run with `--full-clone`)
+and the only files present are those in the top-level directory. Use
Quoted 8 lines

contrib/scalar/scalar.txt an existing Git worktree with Scalar whose name is not `src`, the enlistment · line 52

Quoted 4 lines

+git sparse-checkout set to expand the set of directories you want to see,

contrib/scalar/scalar.txt +3 -0
+or `git sparse-checkout disable` to expand to all files (see
+linkgit:git-sparse-checkout[1] for more details). You can explore the
+subdirectories outside your sparse-checkout by using `git ls-tree HEAD`.

Should this be git ls-tree [-r] HEAD? Do you expect people to just add directories that are found immediately under the toplevel, rather than some that are a bit deeper?

Quoted 2 lines

I fear that git ls-tree -r HEAD in any monorepo might be a bit too overwhelming for any reader.

Oh, right...

Quoted 2 lines

But I agree that just looking at HEAD is probably not enough. Maybe we should use git ls-tree HEAD[:<dir>]?

Sounds good.

#159 lore raw

v6 cover Upstreaming the Scalar command

In reply to Johannes Schindelin via GitGitGadget

tl;dr: This series contributes the core part of the Scalar command to the Git project. This command provides an opinionated way to create and configure Git repositories with a focus on very large repositories.

Changes since v5:

* Fixed the commit message talking about make -C contrib/scalar/Makefile.
* Fixed the git ls-tree invocation suggested in the manual for scalar clone.
* Invoking make -C contrib/scalar, then changing a source file of libgit.a and then immediately invoking make -C contrib/scalar again will now implicitly rebuild libgit.a.

Changes since v4:

* scalar delete now refuses to delete anything if it was started from within the enlistment.
* scalar delete releases any handles to the object store before deleting the enlistment.
* The OBJECTS list in the Makefile will now include Scalar.
* scalar register now supports secondary worktrees, in addition to the primary worktree.

Changes since v3:

* Moved the "Changes since" section to the top, to make it easier to see what changed.
* Reworded the commit message of the first patch.
* Removed the [RFC] prefix because I did not hear any objections against putting this into contrib/.

Changes since v2:

* Adjusted the description of the list command in the manual page , as suggested by Bagas.
* Addressed two style nits in cmd_run().
* The documentation of git reconfigure -a was improved.

Changes since v1:

* A couple typos were fixed
* The code parsing the output of ls-remote was made more readable
* The indentation used in scalar.txt now consistently uses tabs
* We no longer hard-code core.bare = false when registering with Scalar

Background
==========

Years ago, Microsoft wanted to move the source code of the Windows operating system to Git. The challenge there was to prove that Git could scale to massive monorepos. The VFS for Git (formerly GVFS) project was born to take up that challenge.

The final solution included a virtual filesystem (with both user-mode and kernel components) and a customized fork of Git for Windows. This solution contained several key concepts, such as only populating a portion of the working directory, demand-fetching blobs, and performing periodic repo maintenance in the background. However, the required kernel drivers made it difficult to port the solution to other platforms.

But it was realized that many of these key concepts were independent of the actual VFS and its projection of the working directory. The Scalar project was created to make that separation, refine the key concepts, and then extract those features into the new Scalar command.

The present
===========

The Scalar project provides a completely functional non-virtual experience for monorepos. But why stop there. The Scalar project was designed to be a self-destructing vehicle to allow those key concepts to be moved into core Git itself for the benefit of all. For example, partial clone, sparse-checkout, and background maintenance have already been upstreamed and removed from Scalar proper. This patch series provides a C-based implementation of the final remaining portions of the Scalar command. This will make it easier for users to experiment with the Scalar command. It will also make it substantially easier to experiment with moving functionality from Scalar into core Git, while maintaining backwards-compatibility for existing Scalar users.

The C-based Scalar has been shipped to Scalar users, and can be tested by any interested reader: https://github.com/microsoft/git/releases/ (it offers a Git for Windows installer, a macOS package and an Ubuntu package, Scalar has been included since v2.33.0.vfs.0.0).

Next steps
==========

I have lined up a few more patch series on top of this one:

1. Implement a scalar diagnose command.
2. Use the built-in FSMonitor (that patch series obviously needs to wait for FSMonitor to be integrated).
3. Modify the config machinery to be more generous about concurrent writes, say, to the user-wide config.
4. A few patches to optionally build and install scalar as part of a regular Git install (also teaching git help scalar to find the Scalar documentation

These are included in my vfs-with-scalar branch thicket
[https://github.com/dscho/git/commits/vfs-with-scalar]. On top of that, this branch thicket also includes patches I do not plan on upstreaming, mainly because they are too specific either to VFS for Git, or they support Azure Repos (which does not offer partial clones but speaks the GVFS protocol, which can be used to emulate partial clones).

One other thing is very interesting about that vfs-with-scalar branch
thicket: it contains a GitHub workflow which will run Scalar's quite extensive Functional Tests suite. This test suite is quite comprehensive and caught us a lot of bugs in the past, not only in the Scalar code, but also core Git.

Epilogue
========

Now, to address some questions that I imagine every reader has who made it this far:

* Why not put the Scalar functionality directly into core Git, even a built-in? Creating a Git builtin requires scrutiny over every aspect of the feature, which is difficult to do while also maintaining the command-line interface contract and expected behavior of the Scalar command (there are existing users, after all). By having the Scalar command in contrib/, we present a simple option for users to have these features in the short term while the Git contributor community decides which bits to absorb into Git built-ins.
* Why implement the Scalar command in the Git codebase? We ported Scalar to the microsoft/git fork for several reasons. First, we realized it was possible in the first place, now that the core features exist inside Git itself. Second, compiling Scalar directly within a version of Git allows us to remove a version compatibility check from each config option that might or might not apply based on the installed Git version. Finally, this new location has greatly simplified our release process and the installation process for users. We now have ways to install Scalar with microsoft/git via winget, brew, and apt-get. This has been the case since we shipped v2.32.0 to our users, read: this setup has served us well already.
* Why contribute Scalar to the Git project? We are biased, of course, yet we do have evidence that the Scalar command is a helpful tool that offers an simple way to handle huge repositories with ease. By contributing it to the core Git project, we are able to share it with more users, especially some users who do not want to install the microsoft/git fork. We intend to include Scalar as a component in git-for-windows/git, but are contributing it here first. Further, we think there is benefit to the Git developer community as this presents an example of how to set certain defaults that work for large repositories.

Derrick Stolee (4):
scalar: 'register' sets recommended config and starts maintenance
scalar: 'unregister' stops background maintenance
scalar: implement 'scalar list'
scalar: implement the run command

Johannes Schindelin (10):
scalar: create a rudimentary executable
scalar: start documenting the command
scalar: create test infrastructure
scalar: let 'unregister' handle a deleted enlistment directory
gracefully
scalar: implement the clone subcommand
scalar: teach 'clone' to support the --single-branch option
scalar: allow reconfiguring an existing enlistment
scalar: teach 'reconfigure' to optionally handle all registered
enlistments
scalar: implement the version command
scalar: accept -C and -c options before the subcommand

Matthew John Cheetham (1):
scalar: implement the delete command

 Makefile                         |   9 +
 contrib/scalar/.gitignore        |   5 +
 contrib/scalar/Makefile          |  57 +++
 contrib/scalar/scalar.c          | 845 +++++++++++++++++++++++++++++++
 contrib/scalar/scalar.txt        | 155 ++++++
 contrib/scalar/t/Makefile        |  78 +++
 contrib/scalar/t/t9099-scalar.sh |  88 ++++
 7 files changed, 1237 insertions(+)
 create mode 100644 contrib/scalar/.gitignore
 create mode 100644 contrib/scalar/Makefile
 create mode 100644 contrib/scalar/scalar.c
 create mode 100644 contrib/scalar/scalar.txt
 create mode 100644 contrib/scalar/t/Makefile
 create mode 100755 contrib/scalar/t/t9099-scalar.sh

base-commit: ebf3c04b262aa27fbb97f8a0156c2347fecafafb
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-1005%2Fdscho%2Fscalar-the-beginning-v6
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1005/dscho/scalar-the-beginning-v6
Pull-Request: https://github.com/gitgitgadget/git/pull/1005

Range-diff vs v5:

1: 7119a8efc21 ! 1: 9b0b00438ec scalar: create a rudimentary executable

     @@ Commit message
      
          The idea here is that you can (optionally) build Scalar via
      
     -            make -C contrib/scalar/Makefile
     +            make -C contrib/scalar/
      
          This will build the `scalar` executable and put it into the
          contrib/scalar/ subdirectory.
     @@ contrib/scalar/Makefile (new)
      +TARGETS = scalar$(X) scalar.o
      +GITLIBS = ../../common-main.o ../../libgit.a ../../xdiff/lib.a
      +
     -+all: scalar$X
     ++all: scalar$(X)
      +
      +$(GITLIBS):
      +	$(QUIET_SUBDIR0)../.. $(QUIET_SUBDIR1) $(subst ../../,,$@)
     @@ contrib/scalar/Makefile (new)
      +clean:
      +	$(RM) $(TARGETS)
      +
     -+.PHONY: all clean FORCE
     ++.PHONY: $(GITLIBS) all clean FORCE
      
       ## contrib/scalar/scalar.c (new) ##
      @@
  2:  edc1e5f73bd !  2:  40fee75968e scalar: start documenting the command
     @@ contrib/scalar/Makefile: $(TARGETS): $(GITLIBS) scalar.c
       	$(RM) $(TARGETS)
      +	$(RM) scalar.1 scalar.html scalar.xml
       
     --.PHONY: all clean FORCE
     +-.PHONY: $(GITLIBS) all clean FORCE
      +docs: scalar.html scalar.1
      +
      +scalar.html: | scalar.1 # prevent them from trying to build `doc.dep` in parallel
     @@ contrib/scalar/Makefile: $(TARGETS): $(GITLIBS) scalar.c
      +		../contrib/scalar/$@
      +	$(QUIET)test scalar.1 != "$@" || mv ../../Documentation/$@ .
      +
     -+.PHONY: all clean docs FORCE
     ++.PHONY: $(GITLIBS) all clean docs FORCE
      
       ## contrib/scalar/scalar.txt (new) ##
      @@
  3:  9eeb66d5b5d !  3:  e3507c2d5f7 scalar: create test infrastructure
     @@ contrib/scalar/Makefile: include ../../config.mak.uname
       TARGETS = scalar$(X) scalar.o
       GITLIBS = ../../common-main.o ../../libgit.a ../../xdiff/lib.a
       
     --all: scalar$X
     -+all: scalar$X ../../bin-wrappers/scalar
     +-all: scalar$(X)
     ++all: scalar$(X) ../../bin-wrappers/scalar
       
       $(GITLIBS):
       	$(QUIET_SUBDIR0)../.. $(QUIET_SUBDIR1) $(subst ../../,,$@)
     @@ contrib/scalar/Makefile: scalar.html scalar.1: scalar.txt
       		../contrib/scalar/$@
       	$(QUIET)test scalar.1 != "$@" || mv ../../Documentation/$@ .
       
     --.PHONY: all clean docs FORCE
     -+.PHONY: all clean docs test FORCE
     +-.PHONY: $(GITLIBS) all clean docs FORCE
     ++.PHONY: $(GITLIBS) all clean docs test FORCE
      
       ## contrib/scalar/t/Makefile (new) ##
      @@
  4:  8ebfa51ae15 =  4:  53224e506ea scalar: 'register' sets recommended config and starts maintenance
  5:  51b5fc577c9 =  5:  3591e53700b scalar: 'unregister' stops background maintenance
  6:  0041477374e =  6:  fe04ae7ec66 scalar: let 'unregister' handle a deleted enlistment directory gracefully
  7:  12efa86e4cb =  7:  136aec439fd scalar: implement 'scalar list'
  8:  670891d14cc !  8:  ed24a5e9b4c scalar: implement the `clone` subcommand

@@ contrib/scalar/scalar.txt: an existing Git worktree with Scalar whose name is no
+git sparse-checkout set to expand the set of directories you want to see,
+or git sparse-checkout disable to expand to all files (see +linkgit:git-sparse-checkout[1] for more details). You can explore the
-+subdirectories outside your sparse-checkout by using git ls-tree HEAD. ++subdirectories outside your sparse-checkout by using git ls-tree ++HEAD[:<directory>].
+
+-b <name>::
+--branch <name>::
9: 03b8385ebb5 ! 9: 5ee54a9e25f scalar: teach 'clone' to support the --single-branch option
@@ contrib/scalar/scalar.txt: scalar - an opinionated repository management tool
scalar list
scalar register [<enlistment>]
scalar unregister [<enlistment>]
-@@ contrib/scalar/scalar.txt: subdirectories outside your sparse-checkout by using git ls-tree HEAD. +@@ contrib/scalar/scalar.txt: HEAD[:<directory>]`.
Instead of checking out the branch pointed to by the cloned
repository's HEAD, check out the <name> branch instead.

 10:  5a0b3843f98 = 10:  3361eb8f091 scalar: implement the `run` command
 11:  6d9aaeb05fa = 11:  c4778c1dc5f scalar: allow reconfiguring an existing enlistment
 12:  2967d7f1425 = 12:  cd824e9e483 scalar: teach 'reconfigure' to optionally handle all registered enlistments
 13:  8069cc536fe = 13:  843026d5481 scalar: implement the `delete` command
 14:  2ecfaa5d0fe = 14:  5ca169b3f3a scalar: implement the `version` command
 15:  f81e8b3bcf1 = 15:  e3a6eea0534 scalar: accept -C and -c options before the subcommand
Signature · 1 line
gitgitgadget
#160 lore raw

PATCH v6 1/15 scalar: create a rudimentary executable

From: Johannes Schindelin <[email protected]>

The idea of Scalar (https://github.com/microsoft/scalar), and before that, of VFS for Git, has always been to prove that Git _can_ scale, and to upstream whatever strategies have been demonstrated to help.

With this patch, we start the journey from that C# project to move what is left to Git's own contrib/ directory, reimplementing it in pure C, with the intention to facilitate integrating the functionality into core Git all while maintaining backwards-compatibility for existing Scalar users (which will be much easier when both live in the same worktree). It has always been the plan to contribute all of the proven strategies back to core Git.

For example, while the virtual filesystem provided by VFS for Git helped the team developing the Windows operating system to move onto Git, while trying to upstream it we realized that it cannot be done: getting the virtual filesystem to work (which we only managed to implement fully on Windows, but not on, say, macOS or Linux), and the required server-side support for the GVFS protocol, made this not quite feasible.

The Scalar project learned from that and tackled the problem with different tactics: instead of pretending to Git that the working directory is fully populated, it _specifically_ teaches Git about partial clone (which is based on VFS for Git's cache server), about sparse checkout (which VFS for Git tried to do transparently, in the file system layer), and regularly runs maintenance tasks to keep the repository in a healthy state.

With partial clone, sparse checkout and git maintenance having been upstreamed, there is little left that scalar.exe does which git.exe cannot do. One such thing is that scalar clone <url> will automatically set up a partial, sparse clone, and configure known-helpful settings from the start.

So let's bring this convenience into Git's tree.

The idea here is that you can (optionally) build Scalar via

make -C contrib/scalar/

This will build the scalar executable and put it into the contrib/scalar/ subdirectory.

The slightly awkward addition of the contrib/scalar/* bits to the top-level Makefile are actually really required: we want to link to libgit.a, which means that we will need to use the very same CFLAGS and LDFLAGS as the rest of Git.

An early development version of this patch tried to replicate all the conditional code in contrib/scalar/Makefile (e.g. NO_POLL) just like contrib/svn-fe/Makefile used to do before it was retired. It turned out to be quite the whack-a-mole game: the SHA-1-related flags, the flags enabling/disabling compat/poll/, compat/regex/, compat/win32mmap.c & friends depending on the current platform... To put it mildly: it was a major mess.

Instead, this patch makes minimal changes to the top-level Makefile so that the bits in contrib/scalar/ can be compiled and linked, and adds a contrib/scalar/Makefile that uses the top-level Makefile in a most minimal way to do the actual compiling.

Note: With this commit, we only establish the infrastructure, no Scalar functionality is implemented yet; We will do that incrementally over the next few commits.

---
 Makefile                  |  9 +++++++++
 contrib/scalar/.gitignore |  2 ++
 contrib/scalar/Makefile   | 34 ++++++++++++++++++++++++++++++++++
 contrib/scalar/scalar.c   | 36 ++++++++++++++++++++++++++++++++++++
 4 files changed, 81 insertions(+)
 create mode 100644 contrib/scalar/.gitignore
 create mode 100644 contrib/scalar/Makefile
 create mode 100644 contrib/scalar/scalar.c
Makefile +9 -0
index c3565fc0f8f..a12cac1b68b 100644
@@ -2444,6 +2444,11 @@ OBJECTS += $(FUZZ_OBJS)
 ifndef NO_CURL
 	OBJECTS += http.o http-walker.o remote-curl.o
 endif
+
+SCALAR_SOURCES := contrib/scalar/scalar.c
+SCALAR_OBJECTS := $(SCALAR_SOURCES:c=o)
+OBJECTS += $(SCALAR_OBJECTS)
+
 .PHONY: objects
 objects: $(OBJECTS)
 
@@ -2586,6 +2591,10 @@ $(REMOTE_CURL_PRIMARY): remote-curl.o http.o http-walker.o GIT-LDFLAGS $(GITLIBS
 	$(QUIET_LINK)$(CC) $(ALL_CFLAGS) -o $@ $(ALL_LDFLAGS) $(filter %.o,$^) \
 		$(CURL_LIBCURL) $(EXPAT_LIBEXPAT) $(LIBS)
 
+contrib/scalar/scalar$X: $(SCALAR_OBJECTS) GIT-LDFLAGS $(GITLIBS)
+	$(QUIET_LINK)$(CC) $(ALL_CFLAGS) -o $@ $(ALL_LDFLAGS) \
+		$(filter %.o,$^) $(LIBS)
+
 $(LIB_FILE): $(LIB_OBJS)
 	$(QUIET_AR)$(RM) $@ && $(AR) $(ARFLAGS) $@ $^
 
contrib/scalar/.gitignore new file +2 -0
new file mode 100644
index 00000000000..ff3d47e84d0
@@ -0,0 +1,2 @@
+/*.exe
+/scalar
contrib/scalar/Makefile new file +34 -0
new file mode 100644
index 00000000000..f6f0036f0fa
@@ -0,0 +1,34 @@
+QUIET_SUBDIR0  = +$(MAKE) -C # space to separate -C and subdir
+QUIET_SUBDIR1  =
+
+ifneq ($(findstring s,$(MAKEFLAGS)),s)
+ifndef V
+	QUIET_SUBDIR0  = +@subdir=
+	QUIET_SUBDIR1  = ;$(NO_SUBDIR) echo '   ' SUBDIR $$subdir; \
+			 $(MAKE) $(PRINT_DIR) -C $$subdir
+else
+	export V
+endif
+endif
+
+all:
+
+include ../../config.mak.uname
+-include ../../config.mak.autogen
+-include ../../config.mak
+
+TARGETS = scalar$(X) scalar.o
+GITLIBS = ../../common-main.o ../../libgit.a ../../xdiff/lib.a
+
+all: scalar$(X)
+
+$(GITLIBS):
+	$(QUIET_SUBDIR0)../.. $(QUIET_SUBDIR1) $(subst ../../,,$@)
+
+$(TARGETS): $(GITLIBS) scalar.c
+	$(QUIET_SUBDIR0)../.. $(QUIET_SUBDIR1) $(patsubst %,contrib/scalar/%,$@)
+
+clean:
+	$(RM) $(TARGETS)
+
+.PHONY: $(GITLIBS) all clean FORCE
contrib/scalar/scalar.c new file +36 -0
new file mode 100644
index 00000000000..7cff29e0fcd
@@ -0,0 +1,36 @@
+/*
+ * The Scalar command-line interface.
+ */
+
+#include "cache.h"
+#include "gettext.h"
+#include "parse-options.h"
+
+static struct {
+	const char *name;
+	int (*fn)(int, const char **);
+} builtins[] = {
+	{ NULL, NULL},
+};
+
+int cmd_main(int argc, const char **argv)
+{
+	struct strbuf scalar_usage = STRBUF_INIT;
+	int i;
+
+	if (argc > 1) {
+		argv++;
+		argc--;
+
+		for (i = 0; builtins[i].name; i++)
+			if (!strcmp(builtins[i].name, argv[0]))
+				return !!builtins[i].fn(argc, argv);
+	}
+
+	strbuf_addstr(&scalar_usage,
+		      N_("scalar <command> [<options>]\n\nCommands:\n"));
+	for (i = 0; builtins[i].name; i++)
+		strbuf_addf(&scalar_usage, "\t%s\n", builtins[i].name);
+
+	usage(scalar_usage.buf);
+}
Signature · 1 line
gitgitgadget
#161 lore raw

PATCH v6 2/15 scalar: start documenting the command

In reply to Johannes Schindelin via GitGitGadget

From: Johannes Schindelin <[email protected]>

This commit establishes the infrastructure to build the manual page for the scalar command.

---
 contrib/scalar/.gitignore |  3 +++
 contrib/scalar/Makefile   | 14 +++++++++++++-
 contrib/scalar/scalar.txt | 38 ++++++++++++++++++++++++++++++++++++++
 3 files changed, 54 insertions(+), 1 deletion(-)
 create mode 100644 contrib/scalar/scalar.txt
contrib/scalar/.gitignore +3 -0
index ff3d47e84d0..00441073f59 100644
@@ -1,2 +1,5 @@
+/*.xml
+/*.1
+/*.html
 /*.exe
 /scalar
contrib/scalar/Makefile +13 -1
index f6f0036f0fa..e862d3ad8fa 100644
@@ -6,6 +6,7 @@ ifndef V
 	QUIET_SUBDIR0  = +@subdir=
 	QUIET_SUBDIR1  = ;$(NO_SUBDIR) echo '   ' SUBDIR $$subdir; \
 			 $(MAKE) $(PRINT_DIR) -C $$subdir
+	QUIET          = @
 else
 	export V
 endif
@@ -30,5 +31,16 @@ $(TARGETS): $(GITLIBS) scalar.c
 
 clean:
 	$(RM) $(TARGETS)
+	$(RM) scalar.1 scalar.html scalar.xml
 
-.PHONY: $(GITLIBS) all clean FORCE
+docs: scalar.html scalar.1
+
+scalar.html: | scalar.1 # prevent them from trying to build `doc.dep` in parallel
+
+scalar.html scalar.1: scalar.txt
+	$(QUIET_SUBDIR0)../../Documentation$(QUIET_SUBDIR1) \
+		MAN_TXT=../contrib/scalar/scalar.txt \
+		../contrib/scalar/$@
+	$(QUIET)test scalar.1 != "$@" || mv ../../Documentation/$@ .
+
+.PHONY: $(GITLIBS) all clean docs FORCE
contrib/scalar/scalar.txt new file +38 -0
new file mode 100644
index 00000000000..5f7131861a5
@@ -0,0 +1,38 @@
+scalar(1)
+=========
+
+NAME
+----
+scalar - an opinionated repository management tool
+
+SYNOPSIS
+--------
+[verse]
+scalar <command> [<options>]
+
+DESCRIPTION
+-----------
+
+Scalar is an opinionated repository management tool. By creating new
+repositories or registering existing repositories with Scalar, your Git
+experience will speed up. Scalar sets advanced Git config settings,
+maintains your repositories in the background, and helps reduce data sent
+across the network.
+
+An important Scalar concept is the enlistment: this is the top-level directory
+of the project. It usually contains the subdirectory `src/` which is a Git
+worktree. This encourages the separation between tracked files (inside `src/`)
+and untracked files, such as build artifacts (outside `src/`). When registering
+an existing Git worktree with Scalar whose name is not `src`, the enlistment
+will be identical to the worktree.
+
+The `scalar` command implements various subcommands, and different options
+depending on the subcommand.
+
+SEE ALSO
+--------
+linkgit:git-maintenance[1].
+
+Scalar
+---
+Associated with the linkgit:git[1] suite
Signature · 1 line
gitgitgadget
#162 lore raw

PATCH v6 3/15 scalar: create test infrastructure

In reply to Johannes Schindelin via GitGitGadget

From: Johannes Schindelin <[email protected]>

To test the Scalar command, create a test script in contrib/scalar/t that is executed as make -C contrib/scalar test. Since Scalar has no meaningful capabilities yet, the only test is rather simple. We will add more tests in subsequent commits that introduce corresponding, new functionality.

Note: this test script is intended to test scalar only lightly, even after all of the functionality is implemented.

A more comprehensive functional (or: integration) test suite can be found at https://github.com/microsoft/scalar; It is used in the workflow
https://github.com/microsoft/git/blob/HEAD/.github/workflows/scalar-functional-tests.yml
in Microsoft's Git fork. This test suite performs end-to-end tests with a real remote repository, and is run as part of the regular CI builds. Since those tests require some functionality supported only by Microsoft's Git fork ("GVFS protocol"), there is no intention to port that fuller test suite to contrib/scalar/.

---
 contrib/scalar/Makefile          | 17 +++++--
 contrib/scalar/t/Makefile        | 78 ++++++++++++++++++++++++++++++++
 contrib/scalar/t/t9099-scalar.sh | 17 +++++++
 3 files changed, 109 insertions(+), 3 deletions(-)
 create mode 100644 contrib/scalar/t/Makefile
 create mode 100755 contrib/scalar/t/t9099-scalar.sh
contrib/scalar/Makefile +14 -3
index e862d3ad8fa..44796572ef4 100644
@@ -3,6 +3,7 @@ QUIET_SUBDIR1  =
 
 ifneq ($(findstring s,$(MAKEFLAGS)),s)
 ifndef V
+	QUIET_GEN      = @echo '   ' GEN $@;
 	QUIET_SUBDIR0  = +@subdir=
 	QUIET_SUBDIR1  = ;$(NO_SUBDIR) echo '   ' SUBDIR $$subdir; \
 			 $(MAKE) $(PRINT_DIR) -C $$subdir
@@ -21,7 +22,7 @@ include ../../config.mak.uname
 TARGETS = scalar$(X) scalar.o
 GITLIBS = ../../common-main.o ../../libgit.a ../../xdiff/lib.a
 
-all: scalar$(X)
+all: scalar$(X) ../../bin-wrappers/scalar
 
 $(GITLIBS):
 	$(QUIET_SUBDIR0)../.. $(QUIET_SUBDIR1) $(subst ../../,,$@)
@@ -30,9 +31,19 @@ $(TARGETS): $(GITLIBS) scalar.c
 	$(QUIET_SUBDIR0)../.. $(QUIET_SUBDIR1) $(patsubst %,contrib/scalar/%,$@)
 
 clean:
-	$(RM) $(TARGETS)
+	$(RM) $(TARGETS) ../../bin-wrappers/scalar
 	$(RM) scalar.1 scalar.html scalar.xml
 
+../../bin-wrappers/scalar: ../../wrap-for-bin.sh Makefile
+	@mkdir -p ../../bin-wrappers
+	$(QUIET_GEN)sed -e '1s|#!.*/sh|#!$(SHELL_PATH_SQ)|' \
+	     -e 's|@@BUILD_DIR@@|$(shell cd ../.. && pwd)|' \
+	     -e 's|@@PROG@@|contrib/scalar/scalar$(X)|' < $< > $@ && \
+	chmod +x $@
+
+test: all
+	$(MAKE) -C t
+
 docs: scalar.html scalar.1
 
 scalar.html: | scalar.1 # prevent them from trying to build `doc.dep` in parallel
@@ -43,4 +54,4 @@ scalar.html scalar.1: scalar.txt
 		../contrib/scalar/$@
 	$(QUIET)test scalar.1 != "$@" || mv ../../Documentation/$@ .
 
-.PHONY: $(GITLIBS) all clean docs FORCE
+.PHONY: $(GITLIBS) all clean docs test FORCE
contrib/scalar/t/Makefile new file +78 -0

78 lines not on this page · raw message

contrib/scalar/t/t9099-scalar.sh new file +17 -0
new file mode 100755
index 00000000000..16f2b72b126
@@ -0,0 +1,17 @@
+#!/bin/sh
+
+test_description='test the `scalar` command'
+
+TEST_DIRECTORY=$PWD/../../../t
+export TEST_DIRECTORY
+
+# Make it work with --no-bin-wrappers
+PATH=$PWD/..:$PATH
+
+. ../../../t/test-lib.sh
+
+test_expect_success 'scalar shows a usage' '
+	test_expect_code 129 scalar -h
+'
+
+test_done
Signature · 1 line
gitgitgadget
#163 lore raw

PATCH v6 4/15 scalar: 'register' sets recommended config and starts maintenance

In reply to Johannes Schindelin via GitGitGadget

From: Derrick Stolee <[email protected]>

Let's start implementing the register command. With this commit, recommended settings are configured upon scalar register, and Git's background maintenance is started.

The recommended config settings may very well change in the future. For example, once the built-in FSMonitor is available, we will want to enable it upon scalar register. For that reason, we explicitly support running scalar register in an already-registered enlistment.

---
 contrib/scalar/scalar.c   | 248 ++++++++++++++++++++++++++++++++++++++
 contrib/scalar/scalar.txt |  18 ++-
 2 files changed, 265 insertions(+), 1 deletion(-)
contrib/scalar/scalar.c +248 -0
index 7cff29e0fcd..38721d671ba 100644
@@ -5,11 +5,259 @@
 #include "cache.h"
 #include "gettext.h"
 #include "parse-options.h"
+#include "config.h"
+#include "run-command.h"
+
+/*
+ * Remove the deepest subdirectory in the provided path string. Path must not
+ * include a trailing path separator. Returns 1 if parent directory found,
+ * otherwise 0.
+ */
+static int strbuf_parent_directory(struct strbuf *buf)
+{
+	size_t len = buf->len;
+	size_t offset = offset_1st_component(buf->buf);
+	char *path_sep = find_last_dir_sep(buf->buf + offset);
+	strbuf_setlen(buf, path_sep ? path_sep - buf->buf : offset);
+
+	return buf->len < len;
+}
+
+static void setup_enlistment_directory(int argc, const char **argv,
+				       const char * const *usagestr,
+				       const struct option *options,
+				       struct strbuf *enlistment_root)
+{
+	struct strbuf path = STRBUF_INIT;
+	char *root;
+	int enlistment_found = 0;
+
+	if (startup_info->have_repository)
+		BUG("gitdir already set up?!?");
+
+	if (argc > 1)
+		usage_with_options(usagestr, options);
+
+	/* find the worktree, determine its corresponding root */
+	if (argc == 1)
+		strbuf_add_absolute_path(&path, argv[0]);
+	else if (strbuf_getcwd(&path) < 0)
+		die(_("need a working directory"));
+
+	strbuf_trim_trailing_dir_sep(&path);
+	do {
+		const size_t len = path.len;
+
+		/* check if currently in enlistment root with src/ workdir */
+		strbuf_addstr(&path, "/src");
+		if (is_nonbare_repository_dir(&path)) {
+			if (enlistment_root)
+				strbuf_add(enlistment_root, path.buf, len);
+
+			enlistment_found = 1;
+			break;
+		}
+
+		/* reset to original path */
+		strbuf_setlen(&path, len);
+
+		/* check if currently in workdir */
+		if (is_nonbare_repository_dir(&path)) {
+			if (enlistment_root) {
+				/*
+				 * If the worktree's directory's name is `src`, the enlistment is the
+				 * parent directory, otherwise it is identical to the worktree.
+				 */
+				root = strip_path_suffix(path.buf, "src");
+				strbuf_addstr(enlistment_root, root ? root : path.buf);
+				free(root);
+			}
+
+			enlistment_found = 1;
+			break;
+		}
+	} while (strbuf_parent_directory(&path));
+
+	if (!enlistment_found)
+		die(_("could not find enlistment root"));
+
+	if (chdir(path.buf) < 0)
+		die_errno(_("could not switch to '%s'"), path.buf);
+
+	strbuf_release(&path);
+	setup_git_directory();
+}
+
+static int run_git(const char *arg, ...)
+{
+	struct strvec argv = STRVEC_INIT;
+	va_list args;
+	const char *p;
+	int res;
+
+	va_start(args, arg);
+	strvec_push(&argv, arg);
+	while ((p = va_arg(args, const char *)))
+		strvec_push(&argv, p);
+	va_end(args);
+
+	res = run_command_v_opt(argv.v, RUN_GIT_CMD);
+
+	strvec_clear(&argv);
+	return res;
+}
+
+static int set_recommended_config(void)
+{
+	struct {
+		const char *key;
+		const char *value;
+	} config[] = {
+		{ "am.keepCR", "true" },
+		{ "core.FSCache", "true" },
+		{ "core.multiPackIndex", "true" },
+		{ "core.preloadIndex", "true" },
+#ifndef WIN32
+		{ "core.untrackedCache", "true" },
+#else
+		/*
+		 * Unfortunately, Scalar's Functional Tests demonstrated
+		 * that the untracked cache feature is unreliable on Windows
+		 * (which is a bummer because that platform would benefit the
+		 * most from it). For some reason, freshly created files seem
+		 * not to update the directory's `lastModified` time
+		 * immediately, but the untracked cache would need to rely on
+		 * that.
+		 *
+		 * Therefore, with a sad heart, we disable this very useful
+		 * feature on Windows.
+		 */
+		{ "core.untrackedCache", "false" },
+#endif
+		{ "core.logAllRefUpdates", "true" },
+		{ "credential.https://dev.azure.com.useHttpPath", "true" },
+		{ "credential.validate", "false" }, /* GCM4W-only */
+		{ "gc.auto", "0" },
+		{ "gui.GCWarning", "false" },
+		{ "index.threads", "true" },
+		{ "index.version", "4" },
+		{ "merge.stat", "false" },
+		{ "merge.renames", "false" },
+		{ "pack.useBitmaps", "false" },
+		{ "pack.useSparse", "true" },
+		{ "receive.autoGC", "false" },
+		{ "reset.quiet", "true" },
+		{ "feature.manyFiles", "false" },
+		{ "feature.experimental", "false" },
+		{ "fetch.unpackLimit", "1" },
+		{ "fetch.writeCommitGraph", "false" },
+#ifdef WIN32
+		{ "http.sslBackend", "schannel" },
+#endif
+		{ "status.aheadBehind", "false" },
+		{ "commitGraph.generationVersion", "1" },
+		{ "core.autoCRLF", "false" },
+		{ "core.safeCRLF", "false" },
+		{ NULL, NULL },
+	};
+	int i;
+	char *value;
+
+	for (i = 0; config[i].key; i++) {
+		if (git_config_get_string(config[i].key, &value)) {
+			trace2_data_string("scalar", the_repository, config[i].key, "created");
+			if (git_config_set_gently(config[i].key,
+						  config[i].value) < 0)
+				return error(_("could not configure %s=%s"),
+					     config[i].key, config[i].value);
+		} else {
+			trace2_data_string("scalar", the_repository, config[i].key, "exists");
+			free(value);
+		}
+	}
+
+	/*
+	 * The `log.excludeDecoration` setting is special because it allows
+	 * for multiple values.
+	 */
+	if (git_config_get_string("log.excludeDecoration", &value)) {
+		trace2_data_string("scalar", the_repository,
+				   "log.excludeDecoration", "created");
+		if (git_config_set_multivar_gently("log.excludeDecoration",
+						   "refs/prefetch/*",
+						   CONFIG_REGEX_NONE, 0))
+			return error(_("could not configure "
+				       "log.excludeDecoration"));
+	} else {
+		trace2_data_string("scalar", the_repository,
+				   "log.excludeDecoration", "exists");
+		free(value);
+	}
+
+	return 0;
+}
+
+static int start_maintenance(void)
+{
+	return run_git("maintenance", "start", NULL);
+}
+
+static int add_enlistment(void)
+{
+	int res;
+
+	if (!the_repository->worktree)
+		die(_("Scalar enlistments require a worktree"));
+
+	res = run_git("config", "--global", "--get", "--fixed-value",
+		      "scalar.repo", the_repository->worktree, NULL);
+
+	/*
+	 * If the setting is already there, then do nothing.
+	 */
+	if (!res)
+		return 0;
+
+	return run_git("config", "--global", "--add",
+		       "scalar.repo", the_repository->worktree, NULL);
+}
+
+static int register_dir(void)
+{
+	int res = add_enlistment();
+
+	if (!res)
+		res = set_recommended_config();
+
+	if (!res)
+		res = start_maintenance();
+
+	return res;
+}
+
+static int cmd_register(int argc, const char **argv)
+{
+	struct option options[] = {
+		OPT_END(),
+	};
+	const char * const usage[] = {
+		N_("scalar register [<enlistment>]"),
+		NULL
+	};
+
+	argc = parse_options(argc, argv, NULL, options,
+			     usage, 0);
+
+	setup_enlistment_directory(argc, argv, usage, options, NULL);
+
+	return register_dir();
+}
 
 static struct {
 	const char *name;
 	int (*fn)(int, const char **);
 } builtins[] = {
+	{ "register", cmd_register },
 	{ NULL, NULL},
 };
 
contrib/scalar/scalar.txt +17 -1
index 5f7131861a5..568987064b2 100644
@@ -8,7 +8,7 @@ scalar - an opinionated repository management tool
 SYNOPSIS
 --------
 [verse]
-scalar <command> [<options>]
+scalar register [<enlistment>]
 
 DESCRIPTION
 -----------
@@ -29,6 +29,22 @@ will be identical to the worktree.
 The `scalar` command implements various subcommands, and different options
 depending on the subcommand.
 
+COMMANDS
+--------
+
+Register
+~~~~~~~~
+
+register [<enlistment>]::
+	Adds the enlistment's repository to the list of registered repositories
+	and starts background maintenance. If `<enlistment>` is not provided,
+	then the enlistment associated with the current working directory is
+	registered.
++
+Note: when this subcommand is called in a worktree that is called `src/`, its
+parent directory is considered to be the Scalar enlistment. If the worktree is
+_not_ called `src/`, it itself will be considered to be the Scalar enlistment.
+
 SEE ALSO
 --------
 linkgit:git-maintenance[1].
Signature · 1 line
gitgitgadget
#164 lore raw

PATCH v6 5/15 scalar: 'unregister' stops background maintenance

In reply to Johannes Schindelin via GitGitGadget

From: Derrick Stolee <[email protected]>

Just like scalar register starts the scheduled background maintenance, scalar unregister stops it. Note that we use git maintenance start in scalar register, but we do not use git maintenance stop in scalar unregister: this would stop maintenance for _all_ repositories, not just for the one we want to unregister.

The unregister command also removes the corresponding entry from the [scalar] section in the global Git config.

---
 contrib/scalar/scalar.c   | 50 ++++++++++++++++++++++++++++++++-------
 contrib/scalar/scalar.txt |  8 +++++++
 2 files changed, 50 insertions(+), 8 deletions(-)
contrib/scalar/scalar.c +42 -8
index 38721d671ba..fc55404230d 100644
@@ -197,12 +197,12 @@ static int set_recommended_config(void)
 	return 0;
 }
 
-static int start_maintenance(void)
+static int toggle_maintenance(int enable)
 {
-	return run_git("maintenance", "start", NULL);
+	return run_git("maintenance", enable ? "start" : "unregister", NULL);
 }
 
-static int add_enlistment(void)
+static int add_or_remove_enlistment(int add)
 {
 	int res;
 
@@ -213,24 +213,39 @@ static int add_enlistment(void)
 		      "scalar.repo", the_repository->worktree, NULL);
 
 	/*
-	 * If the setting is already there, then do nothing.
+	 * If we want to add and the setting is already there, then do nothing.
+	 * If we want to remove and the setting is not there, then do nothing.
 	 */
-	if (!res)
+	if ((add && !res) || (!add && res))
 		return 0;
 
-	return run_git("config", "--global", "--add",
+	return run_git("config", "--global", add ? "--add" : "--unset",
+		       add ? "--no-fixed-value" : "--fixed-value",
 		       "scalar.repo", the_repository->worktree, NULL);
 }
 
 static int register_dir(void)
 {
-	int res = add_enlistment();
+	int res = add_or_remove_enlistment(1);
 
 	if (!res)
 		res = set_recommended_config();
 
 	if (!res)
-		res = start_maintenance();
+		res = toggle_maintenance(1);
+
+	return res;
+}
+
+static int unregister_dir(void)
+{
+	int res = 0;
+
+	if (toggle_maintenance(0) < 0)
+		res = -1;
+
+	if (add_or_remove_enlistment(0) < 0)
+		res = -1;
 
 	return res;
 }
@@ -253,11 +268,30 @@ static int cmd_register(int argc, const char **argv)
 	return register_dir();
 }
 
+static int cmd_unregister(int argc, const char **argv)
+{
+	struct option options[] = {
+		OPT_END(),
+	};
+	const char * const usage[] = {
+		N_("scalar unregister [<enlistment>]"),
+		NULL
+	};
+
+	argc = parse_options(argc, argv, NULL, options,
+			     usage, 0);
+
+	setup_enlistment_directory(argc, argv, usage, options, NULL);
+
+	return unregister_dir();
+}
+
 static struct {
 	const char *name;
 	int (*fn)(int, const char **);
 } builtins[] = {
 	{ "register", cmd_register },
+	{ "unregister", cmd_unregister },
 	{ NULL, NULL},
 };
 
contrib/scalar/scalar.txt +8 -0
index 568987064b2..d9a79984492 100644
@@ -9,6 +9,7 @@ SYNOPSIS
 --------
 [verse]
 scalar register [<enlistment>]
+scalar unregister [<enlistment>]
 
 DESCRIPTION
 -----------
@@ -45,6 +46,13 @@ Note: when this subcommand is called in a worktree that is called `src/`, its
 parent directory is considered to be the Scalar enlistment. If the worktree is
 _not_ called `src/`, it itself will be considered to be the Scalar enlistment.
 
+Unregister
+~~~~~~~~~~
+
+unregister [<enlistment>]::
+	Remove the specified repository from the list of repositories
+	registered with Scalar and stop the scheduled background maintenance.
+
 SEE ALSO
 --------
 linkgit:git-maintenance[1].
Signature · 1 line
gitgitgadget
#165 lore raw

PATCH v6 6/15 scalar: let 'unregister' handle a deleted enlistment directory gracefully

In reply to Johannes Schindelin via GitGitGadget

From: Johannes Schindelin <[email protected]>

When a user deleted an enlistment manually, let's be generous and _still_ unregister it.

---
 contrib/scalar/scalar.c          | 46 ++++++++++++++++++++++++++++++++
 contrib/scalar/t/t9099-scalar.sh | 15 +++++++++++
 2 files changed, 61 insertions(+)
contrib/scalar/scalar.c +46 -0
index fc55404230d..dc84ce0d5b2 100644
@@ -268,6 +268,24 @@ static int cmd_register(int argc, const char **argv)
 	return register_dir();
 }
 
+static int remove_deleted_enlistment(struct strbuf *path)
+{
+	int res = 0;
+	strbuf_realpath_forgiving(path, path->buf, 1);
+
+	if (run_git("config", "--global",
+		    "--unset", "--fixed-value",
+		    "scalar.repo", path->buf, NULL) < 0)
+		res = -1;
+
+	if (run_git("config", "--global",
+		    "--unset", "--fixed-value",
+		    "maintenance.repo", path->buf, NULL) < 0)
+		res = -1;
+
+	return res;
+}
+
 static int cmd_unregister(int argc, const char **argv)
 {
 	struct option options[] = {
@@ -281,6 +299,34 @@ static int cmd_unregister(int argc, const char **argv)
 	argc = parse_options(argc, argv, NULL, options,
 			     usage, 0);
 
+	/*
+	 * Be forgiving when the enlistment or worktree does not even exist any
+	 * longer; This can be the case if a user deleted the worktree by
+	 * mistake and _still_ wants to unregister the thing.
+	 */
+	if (argc == 1) {
+		struct strbuf src_path = STRBUF_INIT, workdir_path = STRBUF_INIT;
+
+		strbuf_addf(&src_path, "%s/src/.git", argv[0]);
+		strbuf_addf(&workdir_path, "%s/.git", argv[0]);
+		if (!is_directory(src_path.buf) && !is_directory(workdir_path.buf)) {
+			/* remove possible matching registrations */
+			int res = -1;
+
+			strbuf_strip_suffix(&src_path, "/.git");
+			res = remove_deleted_enlistment(&src_path) && res;
+
+			strbuf_strip_suffix(&workdir_path, "/.git");
+			res = remove_deleted_enlistment(&workdir_path) && res;
+
+			strbuf_release(&src_path);
+			strbuf_release(&workdir_path);
+			return res;
+		}
+		strbuf_release(&src_path);
+		strbuf_release(&workdir_path);
+	}
+
 	setup_enlistment_directory(argc, argv, usage, options, NULL);
 
 	return unregister_dir();
contrib/scalar/t/t9099-scalar.sh +15 -0
index 16f2b72b126..ef0e8d680d5 100755
@@ -14,4 +14,19 @@ test_expect_success 'scalar shows a usage' '
 	test_expect_code 129 scalar -h
 '
 
+test_expect_success 'scalar unregister' '
+	git init vanish/src &&
+	scalar register vanish/src &&
+	git config --get --global --fixed-value \
+		maintenance.repo "$(pwd)/vanish/src" &&
+	scalar list >scalar.repos &&
+	grep -F "$(pwd)/vanish/src" scalar.repos &&
+	rm -rf vanish/src/.git &&
+	scalar unregister vanish &&
+	test_must_fail git config --get --global --fixed-value \
+		maintenance.repo "$(pwd)/vanish/src" &&
+	scalar list >scalar.repos &&
+	! grep -F "$(pwd)/vanish/src" scalar.repos
+'
+
 test_done
Signature · 1 line
gitgitgadget
#166 lore raw

PATCH v6 7/15 scalar: implement 'scalar list'

In reply to Johannes Schindelin via GitGitGadget

From: Derrick Stolee <[email protected]>

The produced list simply consists of those repositories registered under the multi-valued scalar.repo config setting in the user's Git config.

---
 contrib/scalar/scalar.c   | 11 +++++++++++
 contrib/scalar/scalar.txt | 11 ++++++++++-
 2 files changed, 21 insertions(+), 1 deletion(-)
contrib/scalar/scalar.c +11 -0
index dc84ce0d5b2..d13eb951c3d 100644
@@ -250,6 +250,16 @@ static int unregister_dir(void)
 	return res;
 }
 
+static int cmd_list(int argc, const char **argv)
+{
+	if (argc != 1)
+		die(_("`scalar list` does not take arguments"));
+
+	if (run_git("config", "--global", "--get-all", "scalar.repo", NULL) < 0)
+		return -1;
+	return 0;
+}
+
 static int cmd_register(int argc, const char **argv)
 {
 	struct option options[] = {
@@ -336,6 +346,7 @@ static struct {
 	const char *name;
 	int (*fn)(int, const char **);
 } builtins[] = {
+	{ "list", cmd_list },
 	{ "register", cmd_register },
 	{ "unregister", cmd_unregister },
 	{ NULL, NULL},
contrib/scalar/scalar.txt +10 -1
index d9a79984492..f93e3d00efd 100644
@@ -8,6 +8,7 @@ scalar - an opinionated repository management tool
 SYNOPSIS
 --------
 [verse]
+scalar list
 scalar register [<enlistment>]
 scalar unregister [<enlistment>]
 
@@ -28,11 +29,19 @@ an existing Git worktree with Scalar whose name is not `src`, the enlistment
 will be identical to the worktree.
 
 The `scalar` command implements various subcommands, and different options
-depending on the subcommand.
+depending on the subcommand. With the exception of `list`, all subcommands
+expect to be run in an enlistment.
 
 COMMANDS
 --------
 
+List
+~~~~
+
+list::
+	List enlistments that are currently registered by Scalar. This
+	subcommand does not need to be run inside an enlistment.
+
 Register
 ~~~~~~~~
 
Signature · 1 line
gitgitgadget
#167 lore raw

PATCH v6 8/15 scalar: implement the `clone` subcommand

In reply to Johannes Schindelin via GitGitGadget

From: Johannes Schindelin <[email protected]>

This implements Scalar's opinionated clone command: it tries to use a partial clone and sets up a sparse checkout by default. In contrast to git clone, scalar clone sets up the worktree in the src/ subdirectory, to encourage a separation between the source files and the build output (which helps Git tremendously because it avoids untracked files that have to be specifically ignored when refreshing the index).

Also, it registers the repository for regular, scheduled maintenance, and configures a flurry of configuration settings based on the experience and experiments of the Microsoft Windows and the Microsoft Office development teams.

Note: since the scalar clone command is by far the most commonly called scalar subcommand, we document it at the top of the manual page.

---
 contrib/scalar/scalar.c          | 201 +++++++++++++++++++++++++++++++
 contrib/scalar/scalar.txt        |  32 ++++-
 contrib/scalar/t/t9099-scalar.sh |  32 +++++
 3 files changed, 262 insertions(+), 3 deletions(-)
contrib/scalar/scalar.c +201 -0
index d13eb951c3d..0401462b9b3 100644
@@ -7,6 +7,7 @@
 #include "parse-options.h"
 #include "config.h"
 #include "run-command.h"
+#include "refs.h"
 
 /*
  * Remove the deepest subdirectory in the provided path string. Path must not
@@ -250,6 +251,205 @@ static int unregister_dir(void)
 	return res;
 }
 
+/* printf-style interface, expects `<key>=<value>` argument */
+static int set_config(const char *fmt, ...)
+{
+	struct strbuf buf = STRBUF_INIT;
+	char *value;
+	int res;
+	va_list args;
+
+	va_start(args, fmt);
+	strbuf_vaddf(&buf, fmt, args);
+	va_end(args);
+
+	value = strchr(buf.buf, '=');
+	if (value)
+		*(value++) = '\0';
+	res = git_config_set_gently(buf.buf, value);
+	strbuf_release(&buf);
+
+	return res;
+}
+
+static char *remote_default_branch(const char *url)
+{
+	struct child_process cp = CHILD_PROCESS_INIT;
+	struct strbuf out = STRBUF_INIT;
+
+	cp.git_cmd = 1;
+	strvec_pushl(&cp.args, "ls-remote", "--symref", url, "HEAD", NULL);
+	if (!pipe_command(&cp, NULL, 0, &out, 0, NULL, 0)) {
+		const char *line = out.buf;
+
+		while (*line) {
+			const char *eol = strchrnul(line, '\n'), *p;
+			size_t len = eol - line;
+			char *branch;
+
+			if (!skip_prefix(line, "ref: ", &p) ||
+			    !strip_suffix_mem(line, &len, "\tHEAD")) {
+				line = eol + (*eol == '\n');
+				continue;
+			}
+
+			eol = line + len;
+			if (skip_prefix(p, "refs/heads/", &p)) {
+				branch = xstrndup(p, eol - p);
+				strbuf_release(&out);
+				return branch;
+			}
+
+			error(_("remote HEAD is not a branch: '%.*s'"),
+			      (int)(eol - p), p);
+			strbuf_release(&out);
+			return NULL;
+		}
+	}
+	warning(_("failed to get default branch name from remote; "
+		  "using local default"));
+	strbuf_reset(&out);
+
+	child_process_init(&cp);
+	cp.git_cmd = 1;
+	strvec_pushl(&cp.args, "symbolic-ref", "--short", "HEAD", NULL);
+	if (!pipe_command(&cp, NULL, 0, &out, 0, NULL, 0)) {
+		strbuf_trim(&out);
+		return strbuf_detach(&out, NULL);
+	}
+
+	strbuf_release(&out);
+	error(_("failed to get default branch name"));
+	return NULL;
+}
+
+static int cmd_clone(int argc, const char **argv)
+{
+	const char *branch = NULL;
+	int full_clone = 0;
+	struct option clone_options[] = {
+		OPT_STRING('b', "branch", &branch, N_("<branch>"),
+			   N_("branch to checkout after clone")),
+		OPT_BOOL(0, "full-clone", &full_clone,
+			 N_("when cloning, create full working directory")),
+		OPT_END(),
+	};
+	const char * const clone_usage[] = {
+		N_("scalar clone [<options>] [--] <repo> [<dir>]"),
+		NULL
+	};
+	const char *url;
+	char *enlistment = NULL, *dir = NULL;
+	struct strbuf buf = STRBUF_INIT;
+	int res;
+
+	argc = parse_options(argc, argv, NULL, clone_options, clone_usage, 0);
+
+	if (argc == 2) {
+		url = argv[0];
+		enlistment = xstrdup(argv[1]);
+	} else if (argc == 1) {
+		url = argv[0];
+
+		strbuf_addstr(&buf, url);
+		/* Strip trailing slashes, if any */
+		while (buf.len > 0 && is_dir_sep(buf.buf[buf.len - 1]))
+			strbuf_setlen(&buf, buf.len - 1);
+		/* Strip suffix `.git`, if any */
+		strbuf_strip_suffix(&buf, ".git");
+
+		enlistment = find_last_dir_sep(buf.buf);
+		if (!enlistment) {
+			die(_("cannot deduce worktree name from '%s'"), url);
+		}
+		enlistment = xstrdup(enlistment + 1);
+	} else {
+		usage_msg_opt(_("You must specify a repository to clone."),
+			      clone_usage, clone_options);
+	}
+
+	if (is_directory(enlistment))
+		die(_("directory '%s' exists already"), enlistment);
+
+	dir = xstrfmt("%s/src", enlistment);
+
+	strbuf_reset(&buf);
+	if (branch)
+		strbuf_addf(&buf, "init.defaultBranch=%s", branch);
+	else {
+		char *b = repo_default_branch_name(the_repository, 1);
+		strbuf_addf(&buf, "init.defaultBranch=%s", b);
+		free(b);
+	}
+
+	if ((res = run_git("-c", buf.buf, "init", "--", dir, NULL)))
+		goto cleanup;
+
+	if (chdir(dir) < 0) {
+		res = error_errno(_("could not switch to '%s'"), dir);
+		goto cleanup;
+	}
+
+	setup_git_directory();
+
+	/* common-main already logs `argv` */
+	trace2_def_repo(the_repository);
+
+	if (!branch && !(branch = remote_default_branch(url))) {
+		res = error(_("failed to get default branch for '%s'"), url);
+		goto cleanup;
+	}
+
+	if (set_config("remote.origin.url=%s", url) ||
+	    set_config("remote.origin.fetch="
+		       "+refs/heads/*:refs/remotes/origin/*") ||
+	    set_config("remote.origin.promisor=true") ||
+	    set_config("remote.origin.partialCloneFilter=blob:none")) {
+		res = error(_("could not configure remote in '%s'"), dir);
+		goto cleanup;
+	}
+
+	if (!full_clone &&
+	    (res = run_git("sparse-checkout", "init", "--cone", NULL)))
+		goto cleanup;
+
+	if (set_recommended_config())
+		return error(_("could not configure '%s'"), dir);
+
+	if ((res = run_git("fetch", "--quiet", "origin", NULL))) {
+		warning(_("partial clone failed; attempting full clone"));
+
+		if (set_config("remote.origin.promisor") ||
+		    set_config("remote.origin.partialCloneFilter")) {
+			res = error(_("could not configure for full clone"));
+			goto cleanup;
+		}
+
+		if ((res = run_git("fetch", "--quiet", "origin", NULL)))
+			goto cleanup;
+	}
+
+	if ((res = set_config("branch.%s.remote=origin", branch)))
+		goto cleanup;
+	if ((res = set_config("branch.%s.merge=refs/heads/%s",
+			      branch, branch)))
+		goto cleanup;
+
+	strbuf_reset(&buf);
+	strbuf_addf(&buf, "origin/%s", branch);
+	res = run_git("checkout", "-f", "-t", buf.buf, NULL);
+	if (res)
+		goto cleanup;
+
+	res = register_dir();
+
+cleanup:
+	free(enlistment);
+	free(dir);
+	strbuf_release(&buf);
+	return res;
+}
+
 static int cmd_list(int argc, const char **argv)
 {
 	if (argc != 1)
@@ -346,6 +546,7 @@ static struct {
 	const char *name;
 	int (*fn)(int, const char **);
 } builtins[] = {
+	{ "clone", cmd_clone },
 	{ "list", cmd_list },
 	{ "register", cmd_register },
 	{ "unregister", cmd_unregister },
contrib/scalar/scalar.txt +29 -3
index f93e3d00efd..e8730967f16 100644
@@ -8,6 +8,7 @@ scalar - an opinionated repository management tool
 SYNOPSIS
 --------
 [verse]
+scalar clone [--branch <main-branch>] [--full-clone] <url> [<enlistment>]
 scalar list
 scalar register [<enlistment>]
 scalar unregister [<enlistment>]
@@ -29,12 +30,37 @@ an existing Git worktree with Scalar whose name is not `src`, the enlistment
 will be identical to the worktree.
 
 The `scalar` command implements various subcommands, and different options
-depending on the subcommand. With the exception of `list`, all subcommands
-expect to be run in an enlistment.
+depending on the subcommand. With the exception of `clone` and `list`, all
+subcommands expect to be run in an enlistment.
 
 COMMANDS
 --------
 
+Clone
+~~~~~
+
+clone [<options>] <url> [<enlistment>]::
+	Clones the specified repository, similar to linkgit:git-clone[1]. By
+	default, only commit and tree objects are cloned. Once finished, the
+	worktree is located at `<enlistment>/src`.
++
+The sparse-checkout feature is enabled (except when run with `--full-clone`)
+and the only files present are those in the top-level directory. Use
+`git sparse-checkout set` to expand the set of directories you want to see,
+or `git sparse-checkout disable` to expand to all files (see
+linkgit:git-sparse-checkout[1] for more details). You can explore the
+subdirectories outside your sparse-checkout by using `git ls-tree
+HEAD[:<directory>]`.
+
+-b <name>::
+--branch <name>::
+	Instead of checking out the branch pointed to by the cloned
+	repository's HEAD, check out the `<name>` branch instead.
+
+--[no-]full-clone::
+	A sparse-checkout is initialized by default. This behavior can be
+	turned off via `--full-clone`.
+
 List
 ~~~~
 
@@ -64,7 +90,7 @@ unregister [<enlistment>]::
 
 SEE ALSO
 --------
-linkgit:git-maintenance[1].
+linkgit:git-clone[1], linkgit:git-maintenance[1].
 
 Scalar
 ---
contrib/scalar/t/t9099-scalar.sh +32 -0
index ef0e8d680d5..295398f62cc 100755
@@ -10,6 +10,9 @@ PATH=$PWD/..:$PATH
 
 . ../../../t/test-lib.sh
 
+GIT_TEST_MAINT_SCHEDULER="crontab:test-tool crontab ../cron.txt"
+export GIT_TEST_MAINT_SCHEDULER
+
 test_expect_success 'scalar shows a usage' '
 	test_expect_code 129 scalar -h
 '
@@ -29,4 +32,33 @@ test_expect_success 'scalar unregister' '
 	! grep -F "$(pwd)/vanish/src" scalar.repos
 '
 
+test_expect_success 'set up repository to clone' '
+	test_commit first &&
+	test_commit second &&
+	test_commit third &&
+	git switch -c parallel first &&
+	mkdir -p 1/2 &&
+	test_commit 1/2/3 &&
+	git config uploadPack.allowFilter true &&
+	git config uploadPack.allowAnySHA1InWant true
+'
+
+test_expect_success 'scalar clone' '
+	second=$(git rev-parse --verify second:second.t) &&
+	scalar clone "file://$(pwd)" cloned &&
+	(
+		cd cloned/src &&
+
+		git config --get --global --fixed-value maintenance.repo \
+			"$(pwd)" &&
+
+		test_path_is_missing 1/2 &&
+		test_must_fail git rev-list --missing=print $second &&
+		git rev-list $second &&
+		git cat-file blob $second >actual &&
+		echo "second" >expect &&
+		test_cmp expect actual
+	)
+'
+
 test_done
Signature · 1 line
gitgitgadget
#168 lore raw

PATCH v6 9/15 scalar: teach 'clone' to support the --single-branch option

In reply to Johannes Schindelin via GitGitGadget

From: Johannes Schindelin <[email protected]>

Just like git clone, the scalar clone command now also offers to restrict the clone to a single branch.

---
 contrib/scalar/scalar.c          |  9 +++++++--
 contrib/scalar/scalar.txt        | 12 +++++++++++-
 contrib/scalar/t/t9099-scalar.sh |  6 +++++-
 3 files changed, 23 insertions(+), 4 deletions(-)
contrib/scalar/scalar.c +7 -2
index 0401462b9b3..754e19d781d 100644
@@ -326,12 +326,15 @@ static char *remote_default_branch(const char *url)
 static int cmd_clone(int argc, const char **argv)
 {
 	const char *branch = NULL;
-	int full_clone = 0;
+	int full_clone = 0, single_branch = 0;
 	struct option clone_options[] = {
 		OPT_STRING('b', "branch", &branch, N_("<branch>"),
 			   N_("branch to checkout after clone")),
 		OPT_BOOL(0, "full-clone", &full_clone,
 			 N_("when cloning, create full working directory")),
+		OPT_BOOL(0, "single-branch", &single_branch,
+			 N_("only download metadata for the branch that will "
+			    "be checked out")),
 		OPT_END(),
 	};
 	const char * const clone_usage[] = {
@@ -402,7 +405,9 @@ static int cmd_clone(int argc, const char **argv)
 
 	if (set_config("remote.origin.url=%s", url) ||
 	    set_config("remote.origin.fetch="
-		       "+refs/heads/*:refs/remotes/origin/*") ||
+		       "+refs/heads/%s:refs/remotes/origin/%s",
+		       single_branch ? branch : "*",
+		       single_branch ? branch : "*") ||
 	    set_config("remote.origin.promisor=true") ||
 	    set_config("remote.origin.partialCloneFilter=blob:none")) {
 		res = error(_("could not configure remote in '%s'"), dir);
contrib/scalar/scalar.txt +11 -1
index e8730967f16..56f744a4aa9 100644
@@ -8,7 +8,7 @@ scalar - an opinionated repository management tool
 SYNOPSIS
 --------
 [verse]
-scalar clone [--branch <main-branch>] [--full-clone] <url> [<enlistment>]
+scalar clone [--single-branch] [--branch <main-branch>] [--full-clone] <url> [<enlistment>]
 scalar list
 scalar register [<enlistment>]
 scalar unregister [<enlistment>]
@@ -57,6 +57,16 @@ HEAD[:<directory>]`.
 	Instead of checking out the branch pointed to by the cloned
 	repository's HEAD, check out the `<name>` branch instead.
 
+--[no-]single-branch::
+	Clone only the history leading to the tip of a single branch, either
+	specified by the `--branch` option or the primary branch remote's
+	`HEAD` points at.
++
+Further fetches into the resulting repository will only update the
+remote-tracking branch for the branch this option was used for the initial
+cloning. If the HEAD at the remote did not point at any branch when
+`--single-branch` clone was made, no remote-tracking branch is created.
+
 --[no-]full-clone::
 	A sparse-checkout is initialized by default. This behavior can be
 	turned off via `--full-clone`.
contrib/scalar/t/t9099-scalar.sh +5 -1
index 295398f62cc..9a35ab4fde6 100755
@@ -45,13 +45,17 @@ test_expect_success 'set up repository to clone' '
 
 test_expect_success 'scalar clone' '
 	second=$(git rev-parse --verify second:second.t) &&
-	scalar clone "file://$(pwd)" cloned &&
+	scalar clone "file://$(pwd)" cloned --single-branch &&
 	(
 		cd cloned/src &&
 
 		git config --get --global --fixed-value maintenance.repo \
 			"$(pwd)" &&
 
+		git for-each-ref --format="%(refname)" refs/remotes/origin/ >actual &&
+		echo "refs/remotes/origin/parallel" >expect &&
+		test_cmp expect actual &&
+
 		test_path_is_missing 1/2 &&
 		test_must_fail git rev-list --missing=print $second &&
 		git rev-list $second &&
Signature · 1 line
gitgitgadget
#169 lore raw

PATCH v6 10/15 scalar: implement the `run` command

In reply to Johannes Schindelin via GitGitGadget

From: Derrick Stolee <[email protected]>

Note: this subcommand is provided primarily for backwards-compatibility, for existing Scalar uses. It is mostly just a shim for git maintenance, mapping task names from the way Scalar called them to the way Git calls them.

The reason why those names differ? The background maintenance was first implemented in Scalar, and when it was contributed as a patch series implementing the git maintenance command, reviewers suggested better names, those suggestions were accepted before the patches were integrated into core Git.

---
 contrib/scalar/scalar.c   | 64 +++++++++++++++++++++++++++++++++++++++
 contrib/scalar/scalar.txt | 19 ++++++++++++
 2 files changed, 83 insertions(+)
contrib/scalar/scalar.c +64 -0
index 754e19d781d..d9631287e12 100644
@@ -483,6 +483,69 @@ static int cmd_register(int argc, const char **argv)
 	return register_dir();
 }
 
+static int cmd_run(int argc, const char **argv)
+{
+	struct option options[] = {
+		OPT_END(),
+	};
+	struct {
+		const char *arg, *task;
+	} tasks[] = {
+		{ "config", NULL },
+		{ "commit-graph", "commit-graph" },
+		{ "fetch", "prefetch" },
+		{ "loose-objects", "loose-objects" },
+		{ "pack-files", "incremental-repack" },
+		{ NULL, NULL }
+	};
+	struct strbuf buf = STRBUF_INIT;
+	const char *usagestr[] = { NULL, NULL };
+	int i;
+
+	strbuf_addstr(&buf, N_("scalar run <task> [<enlistment>]\nTasks:\n"));
+	for (i = 0; tasks[i].arg; i++)
+		strbuf_addf(&buf, "\t%s\n", tasks[i].arg);
+	usagestr[0] = buf.buf;
+
+	argc = parse_options(argc, argv, NULL, options,
+			     usagestr, 0);
+
+	if (!argc)
+		usage_with_options(usagestr, options);
+
+	if (!strcmp("all", argv[0])) {
+		i = -1;
+	} else {
+		for (i = 0; tasks[i].arg && strcmp(tasks[i].arg, argv[0]); i++)
+			; /* keep looking for the task */
+
+		if (i > 0 && !tasks[i].arg) {
+			error(_("no such task: '%s'"), argv[0]);
+			usage_with_options(usagestr, options);
+		}
+	}
+
+	argc--;
+	argv++;
+	setup_enlistment_directory(argc, argv, usagestr, options, NULL);
+	strbuf_release(&buf);
+
+	if (i == 0)
+		return register_dir();
+
+	if (i > 0)
+		return run_git("maintenance", "run",
+			       "--task", tasks[i].task, NULL);
+
+	if (register_dir())
+		return -1;
+	for (i = 1; tasks[i].arg; i++)
+		if (run_git("maintenance", "run",
+			    "--task", tasks[i].task, NULL))
+			return -1;
+	return 0;
+}
+
 static int remove_deleted_enlistment(struct strbuf *path)
 {
 	int res = 0;
@@ -555,6 +618,7 @@ static struct {
 	{ "list", cmd_list },
 	{ "register", cmd_register },
 	{ "unregister", cmd_unregister },
+	{ "run", cmd_run },
 	{ NULL, NULL},
 };
 
contrib/scalar/scalar.txt +19 -0
index 56f744a4aa9..39143b08324 100644
@@ -12,6 +12,7 @@ scalar clone [--single-branch] [--branch <main-branch>] [--full-clone] <url> [<e
 scalar list
 scalar register [<enlistment>]
 scalar unregister [<enlistment>]
+scalar run ( all | config | commit-graph | fetch | loose-objects | pack-files ) [<enlistment>]
 
 DESCRIPTION
 -----------
@@ -98,6 +99,24 @@ unregister [<enlistment>]::
 	Remove the specified repository from the list of repositories
 	registered with Scalar and stop the scheduled background maintenance.
 
+Run
+~~~
+
+scalar run ( all | config | commit-graph | fetch | loose-objects | pack-files ) [<enlistment>]::
+	Run the given maintenance task (or all tasks, if `all` was specified).
+	Except for `all` and `config`, this subcommand simply hands off to
+	linkgit:git-maintenance[1] (mapping `fetch` to `prefetch` and
+	`pack-files` to `incremental-repack`).
++
+These tasks are run automatically as part of the scheduled maintenance,
+as soon as the repository is registered with Scalar. It should therefore
+not be necessary to run this subcommand manually.
++
+The `config` task is specific to Scalar and configures all those
+opinionated default settings that make Git work more efficiently with
+large repositories. As this task is run as part of `scalar clone`
+automatically, explicit invocations of this task are rarely needed.
+
 SEE ALSO
 --------
 linkgit:git-clone[1], linkgit:git-maintenance[1].
Signature · 1 line
gitgitgadget
#170 lore raw

PATCH v6 11/15 scalar: allow reconfiguring an existing enlistment

In reply to Johannes Schindelin via GitGitGadget

From: Johannes Schindelin <[email protected]>

This comes in handy during Scalar upgrades, or when config settings were messed up by mistake.

---
 contrib/scalar/scalar.c          | 79 +++++++++++++++++++++-----------
 contrib/scalar/scalar.txt        |  8 ++++
 contrib/scalar/t/t9099-scalar.sh |  8 ++++
 3 files changed, 67 insertions(+), 28 deletions(-)
contrib/scalar/scalar.c +51 -28
index d9631287e12..6bebba0b51f 100644
@@ -108,18 +108,20 @@ static int run_git(const char *arg, ...)
 	return res;
 }
 
-static int set_recommended_config(void)
+static int set_recommended_config(int reconfigure)
 {
 	struct {
 		const char *key;
 		const char *value;
+		int overwrite_on_reconfigure;
 	} config[] = {
-		{ "am.keepCR", "true" },
-		{ "core.FSCache", "true" },
-		{ "core.multiPackIndex", "true" },
-		{ "core.preloadIndex", "true" },
+		/* Required */
+		{ "am.keepCR", "true", 1 },
+		{ "core.FSCache", "true", 1 },
+		{ "core.multiPackIndex", "true", 1 },
+		{ "core.preloadIndex", "true", 1 },
 #ifndef WIN32
-		{ "core.untrackedCache", "true" },
+		{ "core.untrackedCache", "true", 1 },
 #else
 		/*
 		 * Unfortunately, Scalar's Functional Tests demonstrated
@@ -133,28 +135,29 @@ static int set_recommended_config(void)
 		 * Therefore, with a sad heart, we disable this very useful
 		 * feature on Windows.
 		 */
-		{ "core.untrackedCache", "false" },
+		{ "core.untrackedCache", "false", 1 },
 #endif
-		{ "core.logAllRefUpdates", "true" },
-		{ "credential.https://dev.azure.com.useHttpPath", "true" },
-		{ "credential.validate", "false" }, /* GCM4W-only */
-		{ "gc.auto", "0" },
-		{ "gui.GCWarning", "false" },
-		{ "index.threads", "true" },
-		{ "index.version", "4" },
-		{ "merge.stat", "false" },
-		{ "merge.renames", "false" },
-		{ "pack.useBitmaps", "false" },
-		{ "pack.useSparse", "true" },
-		{ "receive.autoGC", "false" },
-		{ "reset.quiet", "true" },
-		{ "feature.manyFiles", "false" },
-		{ "feature.experimental", "false" },
-		{ "fetch.unpackLimit", "1" },
-		{ "fetch.writeCommitGraph", "false" },
+		{ "core.logAllRefUpdates", "true", 1 },
+		{ "credential.https://dev.azure.com.useHttpPath", "true", 1 },
+		{ "credential.validate", "false", 1 }, /* GCM4W-only */
+		{ "gc.auto", "0", 1 },
+		{ "gui.GCWarning", "false", 1 },
+		{ "index.threads", "true", 1 },
+		{ "index.version", "4", 1 },
+		{ "merge.stat", "false", 1 },
+		{ "merge.renames", "false", 1 },
+		{ "pack.useBitmaps", "false", 1 },
+		{ "pack.useSparse", "true", 1 },
+		{ "receive.autoGC", "false", 1 },
+		{ "reset.quiet", "true", 1 },
+		{ "feature.manyFiles", "false", 1 },
+		{ "feature.experimental", "false", 1 },
+		{ "fetch.unpackLimit", "1", 1 },
+		{ "fetch.writeCommitGraph", "false", 1 },
 #ifdef WIN32
-		{ "http.sslBackend", "schannel" },
+		{ "http.sslBackend", "schannel", 1 },
 #endif
+		/* Optional */
 		{ "status.aheadBehind", "false" },
 		{ "commitGraph.generationVersion", "1" },
 		{ "core.autoCRLF", "false" },
@@ -165,7 +168,8 @@ static int set_recommended_config(void)
 	char *value;
 
 	for (i = 0; config[i].key; i++) {
-		if (git_config_get_string(config[i].key, &value)) {
+		if ((reconfigure && config[i].overwrite_on_reconfigure) ||
+		    git_config_get_string(config[i].key, &value)) {
 			trace2_data_string("scalar", the_repository, config[i].key, "created");
 			if (git_config_set_gently(config[i].key,
 						  config[i].value) < 0)
@@ -230,7 +234,7 @@ static int register_dir(void)
 	int res = add_or_remove_enlistment(1);
 
 	if (!res)
-		res = set_recommended_config();
+		res = set_recommended_config(0);
 
 	if (!res)
 		res = toggle_maintenance(1);
@@ -418,7 +422,7 @@ static int cmd_clone(int argc, const char **argv)
 	    (res = run_git("sparse-checkout", "init", "--cone", NULL)))
 		goto cleanup;
 
-	if (set_recommended_config())
+	if (set_recommended_config(0))
 		return error(_("could not configure '%s'"), dir);
 
 	if ((res = run_git("fetch", "--quiet", "origin", NULL))) {
@@ -483,6 +487,24 @@ static int cmd_register(int argc, const char **argv)
 	return register_dir();
 }
 
+static int cmd_reconfigure(int argc, const char **argv)
+{
+	struct option options[] = {
+		OPT_END(),
+	};
+	const char * const usage[] = {
+		N_("scalar reconfigure [<enlistment>]"),
+		NULL
+	};
+
+	argc = parse_options(argc, argv, NULL, options,
+			     usage, 0);
+
+	setup_enlistment_directory(argc, argv, usage, options, NULL);
+
+	return set_recommended_config(1);
+}
+
 static int cmd_run(int argc, const char **argv)
 {
 	struct option options[] = {
@@ -619,6 +641,7 @@ static struct {
 	{ "register", cmd_register },
 	{ "unregister", cmd_unregister },
 	{ "run", cmd_run },
+	{ "reconfigure", cmd_reconfigure },
 	{ NULL, NULL},
 };
 
contrib/scalar/scalar.txt +8 -0
index 39143b08324..89fd7901585 100644
@@ -13,6 +13,7 @@ scalar list
 scalar register [<enlistment>]
 scalar unregister [<enlistment>]
 scalar run ( all | config | commit-graph | fetch | loose-objects | pack-files ) [<enlistment>]
+scalar reconfigure <enlistment>
 
 DESCRIPTION
 -----------
@@ -117,6 +118,13 @@ opinionated default settings that make Git work more efficiently with
 large repositories. As this task is run as part of `scalar clone`
 automatically, explicit invocations of this task are rarely needed.
 
+Reconfigure
+~~~~~~~~~~~
+
+After a Scalar upgrade, or when the configuration of a Scalar enlistment
+was somehow corrupted or changed by mistake, this subcommand allows to
+reconfigure the enlistment.
+
 SEE ALSO
 --------
 linkgit:git-clone[1], linkgit:git-maintenance[1].
contrib/scalar/t/t9099-scalar.sh +8 -0
index 9a35ab4fde6..e6d74a06ca0 100755
@@ -65,4 +65,12 @@ test_expect_success 'scalar clone' '
 	)
 '
 
+test_expect_success 'scalar reconfigure' '
+	git init one/src &&
+	scalar register one &&
+	git -C one/src config core.preloadIndex false &&
+	scalar reconfigure one &&
+	test true = "$(git -C one/src config core.preloadIndex)"
+'
+
 test_done
Signature · 1 line
gitgitgadget
#171 lore raw

PATCH v6 12/15 scalar: teach 'reconfigure' to optionally handle all registered enlistments

In reply to Johannes Schindelin via GitGitGadget

From: Johannes Schindelin <[email protected]>

After a Scalar upgrade, it can come in really handy if there is an easy way to reconfigure all Scalar enlistments. This new option offers this functionality.

---
 contrib/scalar/scalar.c          | 61 ++++++++++++++++++++++++++++++--
 contrib/scalar/scalar.txt        |  9 +++--
 contrib/scalar/t/t9099-scalar.sh |  3 ++
 3 files changed, 67 insertions(+), 6 deletions(-)
contrib/scalar/scalar.c +58 -3
index 6bebba0b51f..234a7dce479 100644
@@ -487,22 +487,77 @@ static int cmd_register(int argc, const char **argv)
 	return register_dir();
 }
 
+static int get_scalar_repos(const char *key, const char *value, void *data)
+{
+	struct string_list *list = data;
+
+	if (!strcmp(key, "scalar.repo"))
+		string_list_append(list, value);
+
+	return 0;
+}
+
 static int cmd_reconfigure(int argc, const char **argv)
 {
+	int all = 0;
 	struct option options[] = {
+		OPT_BOOL('a', "all", &all,
+			 N_("reconfigure all registered enlistments")),
 		OPT_END(),
 	};
 	const char * const usage[] = {
-		N_("scalar reconfigure [<enlistment>]"),
+		N_("scalar reconfigure [--all | <enlistment>]"),
 		NULL
 	};
+	struct string_list scalar_repos = STRING_LIST_INIT_DUP;
+	int i, res = 0;
+	struct repository r = { NULL };
+	struct strbuf commondir = STRBUF_INIT, gitdir = STRBUF_INIT;
 
 	argc = parse_options(argc, argv, NULL, options,
 			     usage, 0);
 
-	setup_enlistment_directory(argc, argv, usage, options, NULL);
+	if (!all) {
+		setup_enlistment_directory(argc, argv, usage, options, NULL);
+
+		return set_recommended_config(1);
+	}
+
+	if (argc > 0)
+		usage_msg_opt(_("--all or <enlistment>, but not both"),
+			      usage, options);
+
+	git_config(get_scalar_repos, &scalar_repos);
 
-	return set_recommended_config(1);
+	for (i = 0; i < scalar_repos.nr; i++) {
+		const char *dir = scalar_repos.items[i].string;
+
+		strbuf_reset(&commondir);
+		strbuf_reset(&gitdir);
+
+		if (chdir(dir) < 0) {
+			warning_errno(_("could not switch to '%s'"), dir);
+			res = -1;
+		} else if (discover_git_directory(&commondir, &gitdir) < 0) {
+			warning_errno(_("git repository gone in '%s'"), dir);
+			res = -1;
+		} else {
+			git_config_clear();
+
+			the_repository = &r;
+			r.commondir = commondir.buf;
+			r.gitdir = gitdir.buf;
+
+			if (set_recommended_config(1) < 0)
+				res = -1;
+		}
+	}
+
+	string_list_clear(&scalar_repos, 1);
+	strbuf_release(&commondir);
+	strbuf_release(&gitdir);
+
+	return res;
 }
 
 static int cmd_run(int argc, const char **argv)
contrib/scalar/scalar.txt +6 -3
index 89fd7901585..737cf563c1a 100644
@@ -13,7 +13,7 @@ scalar list
 scalar register [<enlistment>]
 scalar unregister [<enlistment>]
 scalar run ( all | config | commit-graph | fetch | loose-objects | pack-files ) [<enlistment>]
-scalar reconfigure <enlistment>
+scalar reconfigure [ --all | <enlistment> ]
 
 DESCRIPTION
 -----------
@@ -32,8 +32,8 @@ an existing Git worktree with Scalar whose name is not `src`, the enlistment
 will be identical to the worktree.
 
 The `scalar` command implements various subcommands, and different options
-depending on the subcommand. With the exception of `clone` and `list`, all
-subcommands expect to be run in an enlistment.
+depending on the subcommand. With the exception of `clone`, `list` and
+`reconfigure --all`, all subcommands expect to be run in an enlistment.
 
 COMMANDS
 --------
@@ -125,6 +125,9 @@ After a Scalar upgrade, or when the configuration of a Scalar enlistment
 was somehow corrupted or changed by mistake, this subcommand allows to
 reconfigure the enlistment.
 
+With the `--all` option, all enlistments currently registered with Scalar
+will be reconfigured. Use this option after each Scalar upgrade.
+
 SEE ALSO
 --------
 linkgit:git-clone[1], linkgit:git-maintenance[1].
contrib/scalar/t/t9099-scalar.sh +3 -0
index e6d74a06ca0..5fe7fabd0e5 100755
@@ -70,6 +70,9 @@ test_expect_success 'scalar reconfigure' '
 	scalar register one &&
 	git -C one/src config core.preloadIndex false &&
 	scalar reconfigure one &&
+	test true = "$(git -C one/src config core.preloadIndex)" &&
+	git -C one/src config core.preloadIndex false &&
+	scalar reconfigure -a &&
 	test true = "$(git -C one/src config core.preloadIndex)"
 '
 
Signature · 1 line
gitgitgadget
#172 lore raw

PATCH v6 13/15 scalar: implement the `delete` command

In reply to Johannes Schindelin via GitGitGadget

From: Matthew John Cheetham <[email protected]>

Delete an enlistment by first unregistering the repository and then deleting the enlistment directory (usually the directory containing the worktree src/ directory).

On Windows, if the current directory is inside the enlistment's directory, change to the parent of the enlistment directory, to allow us to delete the enlistment (directories used by processes e.g. as current working directories cannot be deleted on Windows).

---
 contrib/scalar/scalar.c          | 63 ++++++++++++++++++++++++++++++++
 contrib/scalar/scalar.txt        |  8 ++++
 contrib/scalar/t/t9099-scalar.sh |  9 +++++
 3 files changed, 80 insertions(+)
contrib/scalar/scalar.c +63 -0
index 234a7dce479..8aaeca7cc64 100644
@@ -8,6 +8,8 @@
 #include "config.h"
 #include "run-command.h"
 #include "refs.h"
+#include "dir.h"
+#include "packfile.h"
 
 /*
  * Remove the deepest subdirectory in the provided path string. Path must not
@@ -327,6 +329,33 @@ static char *remote_default_branch(const char *url)
 	return NULL;
 }
 
+static int delete_enlistment(struct strbuf *enlistment)
+{
+#ifdef WIN32
+	struct strbuf parent = STRBUF_INIT;
+#endif
+
+	if (unregister_dir())
+		die(_("failed to unregister repository"));
+
+#ifdef WIN32
+	/*
+	 * Change the current directory to one outside of the enlistment so
+	 * that we may delete everything underneath it.
+	 */
+	strbuf_addbuf(&parent, enlistment);
+	strbuf_parent_directory(&parent);
+	if (chdir(parent.buf) < 0)
+		die_errno(_("could not switch to '%s'"), parent.buf);
+	strbuf_release(&parent);
+#endif
+
+	if (remove_dir_recursively(enlistment, 0))
+		die(_("failed to delete enlistment directory"));
+
+	return 0;
+}
+
 static int cmd_clone(int argc, const char **argv)
 {
 	const char *branch = NULL;
@@ -687,6 +716,39 @@ static int cmd_unregister(int argc, const char **argv)
 	return unregister_dir();
 }
 
+static int cmd_delete(int argc, const char **argv)
+{
+	char *cwd = xgetcwd();
+	struct option options[] = {
+		OPT_END(),
+	};
+	const char * const usage[] = {
+		N_("scalar delete <enlistment>"),
+		NULL
+	};
+	struct strbuf enlistment = STRBUF_INIT;
+	int res = 0;
+
+	argc = parse_options(argc, argv, NULL, options,
+			     usage, 0);
+
+	if (argc != 1)
+		usage_with_options(usage, options);
+
+	setup_enlistment_directory(argc, argv, usage, options, &enlistment);
+
+	if (dir_inside_of(cwd, enlistment.buf) >= 0)
+		res = error(_("refusing to delete current working directory"));
+	else {
+		close_object_store(the_repository->objects);
+		res = delete_enlistment(&enlistment);
+	}
+	strbuf_release(&enlistment);
+	free(cwd);
+
+	return res;
+}
+
 static struct {
 	const char *name;
 	int (*fn)(int, const char **);
@@ -697,6 +759,7 @@ static struct {
 	{ "unregister", cmd_unregister },
 	{ "run", cmd_run },
 	{ "reconfigure", cmd_reconfigure },
+	{ "delete", cmd_delete },
 	{ NULL, NULL},
 };
 
contrib/scalar/scalar.txt +8 -0
index 737cf563c1a..f416d637289 100644
@@ -14,6 +14,7 @@ scalar register [<enlistment>]
 scalar unregister [<enlistment>]
 scalar run ( all | config | commit-graph | fetch | loose-objects | pack-files ) [<enlistment>]
 scalar reconfigure [ --all | <enlistment> ]
+scalar delete <enlistment>
 
 DESCRIPTION
 -----------
@@ -128,6 +129,13 @@ reconfigure the enlistment.
 With the `--all` option, all enlistments currently registered with Scalar
 will be reconfigured. Use this option after each Scalar upgrade.
 
+Delete
+~~~~~~
+
+delete <enlistment>::
+	This subcommand lets you delete an existing Scalar enlistment from your
+	local file system, unregistering the repository.
+
 SEE ALSO
 --------
 linkgit:git-clone[1], linkgit:git-maintenance[1].
contrib/scalar/t/t9099-scalar.sh +9 -0
index 5fe7fabd0e5..7e8771d0eff 100755
@@ -76,4 +76,13 @@ test_expect_success 'scalar reconfigure' '
 	test true = "$(git -C one/src config core.preloadIndex)"
 '
 
+test_expect_success 'scalar delete without enlistment shows a usage' '
+	test_expect_code 129 scalar delete
+'
+
+test_expect_success 'scalar delete with enlistment' '
+	scalar delete cloned &&
+	test_path_is_missing cloned
+'
+
 test_done
Signature · 1 line
gitgitgadget
#173 lore raw

PATCH v6 14/15 scalar: implement the `version` command

In reply to Johannes Schindelin via GitGitGadget

From: Johannes Schindelin <[email protected]>

The .NET version of Scalar has a version command. This was necessary because it was versioned independently of Git.

Since Scalar is now tightly coupled with Git, it does not make sense for them to show different versions. Therefore, it shows the same output as git version. For backwards-compatibility with the .NET version, scalar version prints to stderr, though (git version prints to stdout instead).

---
 contrib/scalar/scalar.c | 39 +++++++++++++++++++++++++++++++++++++++
 1 file changed, 39 insertions(+)
contrib/scalar/scalar.c +39 -0
index 8aaeca7cc64..b2e92cf63b5 100644
@@ -10,6 +10,7 @@
 #include "refs.h"
 #include "dir.h"
 #include "packfile.h"
+#include "help.h"
 
 /*
  * Remove the deepest subdirectory in the provided path string. Path must not
@@ -356,6 +357,15 @@ static int delete_enlistment(struct strbuf *enlistment)
 	return 0;
 }
 
+/*
+ * Dummy implementation; Using `get_version_info()` would cause a link error
+ * without this.
+ */
+void load_builtin_commands(const char *prefix, struct cmdnames *cmds)
+{
+	die("not implemented");
+}
+
 static int cmd_clone(int argc, const char **argv)
 {
 	const char *branch = NULL;
@@ -749,6 +759,34 @@ static int cmd_delete(int argc, const char **argv)
 	return res;
 }
 
+static int cmd_version(int argc, const char **argv)
+{
+	int verbose = 0, build_options = 0;
+	struct option options[] = {
+		OPT__VERBOSE(&verbose, N_("include Git version")),
+		OPT_BOOL(0, "build-options", &build_options,
+			 N_("include Git's build options")),
+		OPT_END(),
+	};
+	const char * const usage[] = {
+		N_("scalar verbose [-v | --verbose] [--build-options]"),
+		NULL
+	};
+	struct strbuf buf = STRBUF_INIT;
+
+	argc = parse_options(argc, argv, NULL, options,
+			     usage, 0);
+
+	if (argc != 0)
+		usage_with_options(usage, options);
+
+	get_version_info(&buf, build_options);
+	fprintf(stderr, "%s\n", buf.buf);
+	strbuf_release(&buf);
+
+	return 0;
+}
+
 static struct {
 	const char *name;
 	int (*fn)(int, const char **);
@@ -760,6 +798,7 @@ static struct {
 	{ "run", cmd_run },
 	{ "reconfigure", cmd_reconfigure },
 	{ "delete", cmd_delete },
+	{ "version", cmd_version },
 	{ NULL, NULL},
 };
 
Signature · 1 line
gitgitgadget
#174 lore raw

PATCH v6 15/15 scalar: accept -C and -c options before the subcommand

In reply to Johannes Schindelin via GitGitGadget

From: Johannes Schindelin <[email protected]>

The git executable has these two very useful options:

-C <directory>:
switch to the specified directory before performing any actions

-c <key>=<value>:
temporarily configure this setting for the duration of the specified scalar subcommand

With this commit, we teach the scalar executable the same trick.

---
 contrib/scalar/scalar.c   | 22 +++++++++++++++++++++-
 contrib/scalar/scalar.txt | 10 ++++++++++
 2 files changed, 31 insertions(+), 1 deletion(-)
contrib/scalar/scalar.c +21 -1
index b2e92cf63b5..6c496318bd4 100644
@@ -807,6 +807,25 @@ int cmd_main(int argc, const char **argv)
 	struct strbuf scalar_usage = STRBUF_INIT;
 	int i;
 
+	while (argc > 1 && *argv[1] == '-') {
+		if (!strcmp(argv[1], "-C")) {
+			if (argc < 3)
+				die(_("-C requires a <directory>"));
+			if (chdir(argv[2]) < 0)
+				die_errno(_("could not change to '%s'"),
+					  argv[2]);
+			argc -= 2;
+			argv += 2;
+		} else if (!strcmp(argv[1], "-c")) {
+			if (argc < 3)
+				die(_("-c requires a <key>=<value> argument"));
+			git_config_push_parameter(argv[2]);
+			argc -= 2;
+			argv += 2;
+		} else
+			break;
+	}
+
 	if (argc > 1) {
 		argv++;
 		argc--;
@@ -817,7 +836,8 @@ int cmd_main(int argc, const char **argv)
 	}
 
 	strbuf_addstr(&scalar_usage,
-		      N_("scalar <command> [<options>]\n\nCommands:\n"));
+		      N_("scalar [-C <directory>] [-c <key>=<value>] "
+			 "<command> [<options>]\n\nCommands:\n"));
 	for (i = 0; builtins[i].name; i++)
 		strbuf_addf(&scalar_usage, "\t%s\n", builtins[i].name);
 
contrib/scalar/scalar.txt +10 -0
index f416d637289..cf4e5b889cc 100644
@@ -36,6 +36,16 @@ The `scalar` command implements various subcommands, and different options
 depending on the subcommand. With the exception of `clone`, `list` and
 `reconfigure --all`, all subcommands expect to be run in an enlistment.
 
+The following options can be specified _before_ the subcommand:
+
+-C <directory>::
+	Before running the subcommand, change the working directory. This
+	option imitates the same option of linkgit:git[1].
+
+-c <key>=<value>::
+	For the duration of running the specified subcommand, configure this
+	setting. This option imitates the same option of linkgit:git[1].
+
 COMMANDS
 --------
 
Signature · 1 line
gitgitgadget
#175 lore raw
In reply to Johannes Schindelin via GitGitGadget
Quoted 3 lines · Johannes Schindelin via GitGitGadget

tl;dr: This series contributes the core part of the Scalar command to the Git project. This command provides an opinionated way to create and configure Git repositories with a focus on very large repositories.

I started a discussion [1] to help answer some big questions about this series, but to separate those thoughts from the concrete patches. If you are interested, please contribute feedback there.

[1] https://lore.kernel.org/git/[email protected]/

Thanks,
-Stolee

#176 lore raw

v7 cover Upstreaming the Scalar command

In reply to Johannes Schindelin via GitGitGadget

tl;dr: This series contributes the core part of the Scalar command to the Git project. This command provides an opinionated way to create and configure Git repositories with a focus on very large repositories.

Changes since v6:

* Rebased on top of v2.34.0.
* Inserted a commit that adds contrib/scalar/README.md, containing the roadmap of what I have planned for Scalar.
* The Scalar test's definition of GIT_TEST_MAINT_SCHEDULER has been adjusted to accommodate for a change in v2.32.0..v2.34.0.
* The config setting defaults now include fetch.showForcedUpdates=false, which has been identified as helping with a performance issue in large repositories.
* To avoid mistaking the current patch series for being feature-complete enough to unleash onto end users, I moved the Makefile rules to build HTML/manual pages to a later patch series.
* The patch that adds support for -c <key>=<value> and -C <directory> was moved to its own add-on patch series: While it is obvious that those options are valuable to have, an open question is whether there are other "pre-command" options in git that would be useful, too, and I would like to postpone that discussion to that date.
* I added two patches that I had planned on keeping in an add-on patch series for later, to build and test Scalar as part of the CI. I am still not 100% certain that it is a good idea to do so already now, but let's see what the reviewers have to say.

Changes since v5:

* Fixed the commit message talking about make -C contrib/scalar/Makefile.
* Fixed the git ls-tree invocation suggested in the manual for scalar clone.
* Invoking make -C contrib/scalar, then changing a source file of libgit.a and then immediately invoking make -C contrib/scalar again will now implicitly rebuild libgit.a.

Changes since v4:

* scalar delete now refuses to delete anything if it was started from within the enlistment.
* scalar delete releases any handles to the object store before deleting the enlistment.
* The OBJECTS list in the Makefile will now include Scalar.
* scalar register now supports secondary worktrees, in addition to the primary worktree.

Changes since v3:

* Moved the "Changes since" section to the top, to make it easier to see what changed.
* Reworded the commit message of the first patch.
* Removed the [RFC] prefix because I did not hear any objections against putting this into contrib/.

Changes since v2:

* Adjusted the description of the list command in the manual page , as suggested by Bagas.
* Addressed two style nits in cmd_run().
* The documentation of git reconfigure -a was improved.

Changes since v1:

* A couple typos were fixed
* The code parsing the output of ls-remote was made more readable
* The indentation used in scalar.txt now consistently uses tabs
* We no longer hard-code core.bare = false when registering with Scalar

Background
==========

Microsoft invested a lot of effort into scaling Git to the needs of the Windows operating system source code. Based on the experience of the first approach, VFS for Git, the Scalar project was started. Scalar specifically has as its core goal to funnel all improvements into core Git.

The present
===========

The Scalar project provides a completely functional non-virtual experience for monorepos. But why stop there. The Scalar project was designed to be a self-destructing vehicle to allow those key concepts to be moved into core Git itself for the benefit of all. For example, partial clone, sparse-checkout, and scheduled background maintenance have already been upstreamed and removed from Scalar proper. This patch series provides a C-based implementation of the final remaining portions of the Scalar command. This will make it easier for users to experiment with the Scalar command. It will also make it substantially easier to experiment with moving functionality from Scalar into core Git, while maintaining backwards-compatibility for existing Scalar users.

The C-based Scalar has been shipped to Scalar users, and can be tested by any interested reader: https://github.com/microsoft/git/releases/ (it offers a Git for Windows installer, a macOS package and an Ubuntu package, Scalar has been included since v2.33.0.vfs.0.0).

Next steps
==========

Since there are existing Scalar users, I want to ensure backwards-compatibility with its existing command-line interface. Keeping that in mind, everything in this series is up for discussion.

I obviously believe that Scalar brings a huge benefit, and think that it would be ideal for all of Scalar's learnings to end up in git clone/git init/git maintenance eventually. It is also conceivable, however, that the scalar command could graduate to be a core part of Git at some stage in the future (such a decision would probably depend highly on users' feedback). See also the discussion about the architecture of Scalar
[https://lore.kernel.org/git/[email protected]/],
kicked off by Stolee.

On top of this patch series, I have lined up a few more:

1. Implement a scalar diagnose command.
2. Use the built-in FSMonitor (that patch series obviously needs to wait for FSMonitor to be integrated).
3. Modify the config machinery to be more generous about concurrent writes, say, to the user-wide config.
4. A few patches to optionally build and install scalar as part of a regular Git install (also teaching git help scalar to find the Scalar documentation

These are included in my vfs-with-scalar branch thicket
[https://github.com/dscho/git/commits/vfs-with-scalar]. On top of that, this branch thicket also includes patches I do not plan on upstreaming, mainly because they are too specific either to VFS for Git, or they support Azure Repos (which does not offer partial clones but speaks the GVFS protocol, which can be used to emulate partial clones).

One other thing is very interesting about that vfs-with-scalar branch
thicket: it contains a GitHub workflow which will run Scalar's quite extensive Functional Tests suite. This test suite is quite comprehensive and caught us a lot of bugs in the past, not only in the Scalar code, but also core Git.

Epilogue
========

Now, to address some questions that I imagine every reader has who made it this far:

* Why not put the Scalar functionality directly into core Git, even a built-in? I wanted to provide an easy way for Git contributors to "play with" Scalar, without forcing a new top-level command into Git.
* Why implement the Scalar command in the Git code base? Apart from simplifying Scalar maintenance in the Microsoft port of Git, the tight version coupling between Git and Scalar reduces the maintenance burden even further. Besides, I believe that it will make it much easier to shift functionality from Scalar into core Git, once we took the hurdle of accepting the Scalar code into the code base.
* Why contribute Scalar to the Git project? We are biased, of course, yet our data-driven approach provides evidence that Scalar helps handling huge repositories with ease. By contributing it to the core Git project, we are able to share it with more users, especially some users who do not want to install Microsoft's fork of Git. We also hope that a lot of Scalar (maybe all of it) will end up in core Git, to benefit even more users.

Derrick Stolee (4):
scalar: 'register' sets recommended config and starts maintenance
scalar: 'unregister' stops background maintenance
scalar: implement 'scalar list'
scalar: implement the run command

Johannes Schindelin (12):
scalar: add a README with a roadmap
scalar: create a rudimentary executable
scalar: start documenting the command
scalar: create test infrastructure
cmake: optionally build scalar, too
ci: also run the scalar tests
scalar: let 'unregister' handle a deleted enlistment directory
gracefully
scalar: implement the clone subcommand
scalar: teach 'clone' to support the --single-branch option
scalar: allow reconfiguring an existing enlistment
scalar: teach 'reconfigure' to optionally handle all registered
enlistments
scalar: implement the version command

Matthew John Cheetham (1):
scalar: implement the delete command

 .github/workflows/main.yml          |  15 +
 Makefile                            |   9 +
 ci/run-build-and-tests.sh           |   1 +
 ci/run-test-slice.sh                |   5 +
 contrib/buildsystems/CMakeLists.txt |  14 +
 contrib/scalar/.gitignore           |   2 +
 contrib/scalar/Makefile             |  45 ++
 contrib/scalar/README.md            |  71 +++
 contrib/scalar/scalar.c             | 826 ++++++++++++++++++++++++++++
 contrib/scalar/scalar.txt           | 145 +++++
 contrib/scalar/t/Makefile           |  78 +++
 contrib/scalar/t/t9099-scalar.sh    |  88 +++
 12 files changed, 1299 insertions(+)
 create mode 100644 contrib/scalar/.gitignore
 create mode 100644 contrib/scalar/Makefile
 create mode 100644 contrib/scalar/README.md
 create mode 100644 contrib/scalar/scalar.c
 create mode 100644 contrib/scalar/scalar.txt
 create mode 100644 contrib/scalar/t/Makefile
 create mode 100755 contrib/scalar/t/t9099-scalar.sh

base-commit: cd3e606211bb1cf8bc57f7d76bab98cc17a150bc
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-1005%2Fdscho%2Fscalar-the-beginning-v7
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1005/dscho/scalar-the-beginning-v7
Pull-Request: https://github.com/gitgitgadget/git/pull/1005

Range-diff vs v6:

  -:  ----------- >  1:  3aa095dc824 scalar: add a README with a roadmap
  1:  9b0b00438ec =  2:  e0693cc713c scalar: create a rudimentary executable
  2:  40fee75968e !  3:  d80627615f8 scalar: start documenting the command

@@ Metadata
## Commit message ##
scalar: start documenting the command

- This commit establishes the infrastructure to build the manual page for
- the scalar command.
+ Let's build up the documentation for the Scalar command along with the
+ patches that implement its functionality.

- Signed-off-by: Johannes Schindelin <[email protected]>
-
- ## contrib/scalar/.gitignore ##
-@@
-+/*.xml
-+/*.1
-+/*.html
- /*.exe
- /scalar
+ Note: To discourage the feature-incomplete documentation from being
+ mistaken for the complete thing, we do not yet provide any way to build
+ HTML or manual pages from the text file.

- ## contrib/scalar/Makefile ##
-@@ contrib/scalar/Makefile: ifndef V
- QUIET_SUBDIR0 = +@subdir=
- QUIET_SUBDIR1 = ;$(NO_SUBDIR) echo ' ' SUBDIR $$subdir; \
- $(MAKE) $(PRINT_DIR) -C $$subdir
-+ QUIET = @
- else
- export V
- endif
-@@ contrib/scalar/Makefile: $(TARGETS): $(GITLIBS) scalar.c
-
- clean:
- $(RM) $(TARGETS)
-+ $(RM) scalar.1 scalar.html scalar.xml
-
--.PHONY: $(GITLIBS) all clean FORCE
-+docs: scalar.html scalar.1
-+
-+scalar.html: | scalar.1 # prevent them from trying to build doc.dep in parallel -+
-+scalar.html scalar.1: scalar.txt
-+ $(QUIET_SUBDIR0)../../Documentation$(QUIET_SUBDIR1) \
-+ MAN_TXT=../contrib/scalar/scalar.txt \
-+ ../contrib/scalar/$@
-+ $(QUIET)test scalar.1 != "$@" || mv ../../Documentation/$@ .
-+
-+.PHONY: $(GITLIBS) all clean docs FORCE
+ Signed-off-by: Johannes Schindelin <[email protected]>

## contrib/scalar/scalar.txt (new) ##
@@
3: e3507c2d5f7 ! 4: 9da1616849e scalar: create test infrastructure
@@ Commit message
more tests in subsequent commits that introduce corresponding, new functionality.

- Note: this test script is intended to test scalar only lightly, even
+ Note: This test script is intended to test scalar only lightly, even
after all of the functionality is implemented.

A more comprehensive functional (or: integration) test suite can be found at https://github.com/microsoft/scalar; It is used in the workflow
https://github.com/microsoft/git/blob/HEAD/.github/workflows/scalar-functional-tests.yml
in Microsoft's Git fork. This test suite performs end-to-end tests with
- a real remote repository, and is run as part of the regular CI builds.
+ a real remote repository, and is run as part of the regular CI and PR
+ builds in that fork.
+
Since those tests require some functionality supported only by Microsoft's Git fork ("GVFS protocol"), there is no intention to port that fuller test suite to contrib/scalar/.
@@ contrib/scalar/Makefile: $(TARGETS): $(GITLIBS) scalar.c
clean:
- $(RM) $(TARGETS)
+ $(RM) $(TARGETS) ../../bin-wrappers/scalar
- $(RM) scalar.1 scalar.html scalar.xml

+-.PHONY: $(GITLIBS) all clean FORCE
+../../bin-wrappers/scalar: ../../wrap-for-bin.sh Makefile
+ @mkdir -p ../../bin-wrappers
+ $(QUIET_GEN)sed -e '1s|#!.*/sh|#!$(SHELL_PATH_SQ)|' \
@@ contrib/scalar/Makefile: $(TARGETS): $(GITLIBS) scalar.c
+test: all
+ $(MAKE) -C t
+
- docs: scalar.html scalar.1
-
- scalar.html: | scalar.1 # prevent them from trying to build doc.dep in parallel -@@ contrib/scalar/Makefile: scalar.html scalar.1: scalar.txt
- ../contrib/scalar/$@
- $(QUIET)test scalar.1 != "$@" || mv ../../Documentation/$@ .
-
--.PHONY: $(GITLIBS) all clean docs FORCE
-+.PHONY: $(GITLIBS) all clean docs test FORCE
++.PHONY: $(GITLIBS) all clean test FORCE

## contrib/scalar/t/Makefile (new) ##
@@

  -:  ----------- >  5:  dbaad4753c1 cmake: optionally build `scalar`, too
  -:  ----------- >  6:  1b0328fa236 ci: also run the `scalar` tests
  4:  53224e506ea !  7:  cca604ef326 scalar: 'register' sets recommended config and starts maintenance
     @@ contrib/scalar/scalar.c
      +		{ "commitGraph.generationVersion", "1" },
      +		{ "core.autoCRLF", "false" },
      +		{ "core.safeCRLF", "false" },
     ++		{ "fetch.showForcedUpdates", "false" },
      +		{ NULL, NULL },
      +	};
      +	int i;
  5:  3591e53700b =  8:  9fea89cd161 scalar: 'unregister' stops background maintenance
  6:  fe04ae7ec66 =  9:  5e077bf892b scalar: let 'unregister' handle a deleted enlistment directory gracefully
  7:  136aec439fd = 10:  dfa0c470989 scalar: implement 'scalar list'
  8:  ed24a5e9b4c ! 11:  febefe39886 scalar: implement the `clone` subcommand

@@ contrib/scalar/t/t9099-scalar.sh: PATH=$PWD/..:$PATH

. ../../../t/test-lib.sh

-+GIT_TEST_MAINT_SCHEDULER="crontab:test-tool crontab ../cron.txt" ++GIT_TEST_MAINT_SCHEDULER="crontab:test-tool crontab ../cron.txt,launchctl:true,schtasks:true"
+export GIT_TEST_MAINT_SCHEDULER
+
test_expect_success 'scalar shows a usage' '

  9:  5ee54a9e25f = 12:  2677bcff335 scalar: teach 'clone' to support the --single-branch option
 10:  3361eb8f091 = 13:  99affb84284 scalar: implement the `run` command
 11:  c4778c1dc5f = 14:  69e2242240b scalar: allow reconfiguring an existing enlistment
 12:  cd824e9e483 = 15:  0068c18aa62 scalar: teach 'reconfigure' to optionally handle all registered enlistments
 13:  843026d5481 = 16:  d5218523a38 scalar: implement the `delete` command
 14:  5ca169b3f3a = 17:  96a803416b5 scalar: implement the `version` command
 15:  e3a6eea0534 <  -:  ----------- scalar: accept -C and -c options before the subcommand
Signature · 1 line
gitgitgadget
#177 lore raw

PATCH v7 1/17 scalar: add a README with a roadmap

From: Johannes Schindelin <[email protected]>

The Scalar command will be contributed incrementally, over a bunch of patch series. Let's document what Scalar is about, and then describe the patch series that are planned.

---
 contrib/scalar/README.md | 71 ++++++++++++++++++++++++++++++++++++++++
 1 file changed, 71 insertions(+)
 create mode 100644 contrib/scalar/README.md
contrib/scalar/README.md new file +71 -0
new file mode 100644
index 00000000000..7898a683ba5
@@ -0,0 +1,71 @@
+# Scalar - an opinionated repository management tool
+
+Scalar is an add-on to Git, helping Git scale to very large repositories and
+worktrees. Originally implemented in C# using .NET Core, based on the learnings
+from the VFS for Git project, most of the techniques developed by the Scalar
+project have been integrated into core Git already:
+
+* partial clone,
+* commit graphs,
+* multi-pack index,
+* sparse checkout (cone mode),
+* scheduled background maintenance,
+* etc
+
+This directory contains the remaining parts of Scalar that are not (yet) in
+core Git.
+
+## Roadmap
+
+The idea is to populate this directory via incremental patch series and
+eventually move to a top-level directory next to `gitk-git/` and to `git-gui/`. The
+current plan involves the following patch series:
+
+- `scalar-the-beginning`: The initial patch series which sets up
+  `contrib/scalar/` and populates it with a minimal `scalar` command that
+  demonstrates the fundamental ideas.
+
+- `scalar-c-and-C`: The `scalar` command learns about two options that can be
+  specified before the command, `-c <key>=<value>` and `-C <directory>`.
+
+- `scalar-diagnose`: The `scalar` command is taught the `diagnose` subcommand.
+
+- `scalar-and-builtin-fsmonitor`: The built-in FSMonitor is enabled in `scalar
+  init` and in `scalar clone`, for an enormous performance boost when working
+  in large worktrees. This patch series necessarily depends on Jeff Hostetler's
+  FSMonitor patch series to be integrated into Git.
+
+- `scalar-gentler-config-locking`: Scalar enlistments are registered in the
+  user's Git config. This usually does not represent any problem because it is
+  rare for a user to register an enlistment. However, in Scalar's functional
+  tests, Scalar enlistments are created galore, and in parallel, which can lead
+  to lock contention. This patch series works around that problem by re-trying
+  to lock the config file in a gentle fashion.
+
+- `scalar-extra-docs`: Add some extensive documentation that has been written
+  in the original Scalar project (all subject to discussion, of course).
+
+- `optionally-install-scalar`: Now that Scalar is feature (and documentation)
+  complete and is verified in CI builds, let's offer to install it.
+
+- `move-scalar-to-toplevel`: Now that Scalar is complete, let's move it next to
+  `gitk-git/` and to `git-gui/`, making it a top-level command.
+
+The following two patch series exist, but there is no plan to integrate them
+into Git's source tree:
+
+- `scalar-with-gvfs`: The primary purpose of this patch series is to support
+  existing Scalar users whose repositories are hosted in Azure Repos (which
+  does not support Git's partial clones, but supports its predecessor, the GVFS
+  protocol, which is used by Scalar to emulate the partial clone).
+
+  Since the GVFS protocol will never be supported by core Git, this patch
+  series will remain in Microsoft's fork of Git.
+
+- `run-scalar-functional-tests`: The Scalar project developed a quite
+  comprehensive set of integration tests (or, "Functional Tests"). They are the
+  sole remaining part of the original C#-based Scalar project, and this patch
+  adds a GitHub workflow that runs them all.
+
+  Since the tests partially depend on features that are only provided in the
+  `scalar-with-gvfs` patch series, this patch cannot be upstreamed.
Signature · 1 line
gitgitgadget
#180 lore raw

PATCH v7 2/17 scalar: create a rudimentary executable

In reply to Johannes Schindelin via GitGitGadget

From: Johannes Schindelin <[email protected]>

The idea of Scalar (https://github.com/microsoft/scalar), and before that, of VFS for Git, has always been to prove that Git _can_ scale, and to upstream whatever strategies have been demonstrated to help.

With this patch, we start the journey from that C# project to move what is left to Git's own contrib/ directory, reimplementing it in pure C, with the intention to facilitate integrating the functionality into core Git all while maintaining backwards-compatibility for existing Scalar users (which will be much easier when both live in the same worktree). It has always been the plan to contribute all of the proven strategies back to core Git.

For example, while the virtual filesystem provided by VFS for Git helped the team developing the Windows operating system to move onto Git, while trying to upstream it we realized that it cannot be done: getting the virtual filesystem to work (which we only managed to implement fully on Windows, but not on, say, macOS or Linux), and the required server-side support for the GVFS protocol, made this not quite feasible.

The Scalar project learned from that and tackled the problem with different tactics: instead of pretending to Git that the working directory is fully populated, it _specifically_ teaches Git about partial clone (which is based on VFS for Git's cache server), about sparse checkout (which VFS for Git tried to do transparently, in the file system layer), and regularly runs maintenance tasks to keep the repository in a healthy state.

With partial clone, sparse checkout and git maintenance having been upstreamed, there is little left that scalar.exe does which git.exe cannot do. One such thing is that scalar clone <url> will automatically set up a partial, sparse clone, and configure known-helpful settings from the start.

So let's bring this convenience into Git's tree.

The idea here is that you can (optionally) build Scalar via

make -C contrib/scalar/

This will build the scalar executable and put it into the contrib/scalar/ subdirectory.

The slightly awkward addition of the contrib/scalar/* bits to the top-level Makefile are actually really required: we want to link to libgit.a, which means that we will need to use the very same CFLAGS and LDFLAGS as the rest of Git.

An early development version of this patch tried to replicate all the conditional code in contrib/scalar/Makefile (e.g. NO_POLL) just like contrib/svn-fe/Makefile used to do before it was retired. It turned out to be quite the whack-a-mole game: the SHA-1-related flags, the flags enabling/disabling compat/poll/, compat/regex/, compat/win32mmap.c & friends depending on the current platform... To put it mildly: it was a major mess.

Instead, this patch makes minimal changes to the top-level Makefile so that the bits in contrib/scalar/ can be compiled and linked, and adds a contrib/scalar/Makefile that uses the top-level Makefile in a most minimal way to do the actual compiling.

Note: With this commit, we only establish the infrastructure, no Scalar functionality is implemented yet; We will do that incrementally over the next few commits.

---
 Makefile                  |  9 +++++++++
 contrib/scalar/.gitignore |  2 ++
 contrib/scalar/Makefile   | 34 ++++++++++++++++++++++++++++++++++
 contrib/scalar/scalar.c   | 36 ++++++++++++++++++++++++++++++++++++
 4 files changed, 81 insertions(+)
 create mode 100644 contrib/scalar/.gitignore
 create mode 100644 contrib/scalar/Makefile
 create mode 100644 contrib/scalar/scalar.c
Makefile +9 -0
index 12be39ac497..fe898aeea08 100644
@@ -2456,6 +2456,11 @@ OBJECTS += $(FUZZ_OBJS)
 ifndef NO_CURL
 	OBJECTS += http.o http-walker.o remote-curl.o
 endif
+
+SCALAR_SOURCES := contrib/scalar/scalar.c
+SCALAR_OBJECTS := $(SCALAR_SOURCES:c=o)
+OBJECTS += $(SCALAR_OBJECTS)
+
 .PHONY: objects
 objects: $(OBJECTS)
 
@@ -2589,6 +2594,10 @@ $(REMOTE_CURL_PRIMARY): remote-curl.o http.o http-walker.o GIT-LDFLAGS $(GITLIBS
 	$(QUIET_LINK)$(CC) $(ALL_CFLAGS) -o $@ $(ALL_LDFLAGS) $(filter %.o,$^) \
 		$(CURL_LIBCURL) $(EXPAT_LIBEXPAT) $(LIBS)
 
+contrib/scalar/scalar$X: $(SCALAR_OBJECTS) GIT-LDFLAGS $(GITLIBS)
+	$(QUIET_LINK)$(CC) $(ALL_CFLAGS) -o $@ $(ALL_LDFLAGS) \
+		$(filter %.o,$^) $(LIBS)
+
 $(LIB_FILE): $(LIB_OBJS)
 	$(QUIET_AR)$(RM) $@ && $(AR) $(ARFLAGS) $@ $^
 
contrib/scalar/.gitignore new file +2 -0
new file mode 100644
index 00000000000..ff3d47e84d0
@@ -0,0 +1,2 @@
+/*.exe
+/scalar
contrib/scalar/Makefile new file +34 -0
new file mode 100644
index 00000000000..f6f0036f0fa
@@ -0,0 +1,34 @@
+QUIET_SUBDIR0  = +$(MAKE) -C # space to separate -C and subdir
+QUIET_SUBDIR1  =
+
+ifneq ($(findstring s,$(MAKEFLAGS)),s)
+ifndef V
+	QUIET_SUBDIR0  = +@subdir=
+	QUIET_SUBDIR1  = ;$(NO_SUBDIR) echo '   ' SUBDIR $$subdir; \
+			 $(MAKE) $(PRINT_DIR) -C $$subdir
+else
+	export V
+endif
+endif
+
+all:
+
+include ../../config.mak.uname
+-include ../../config.mak.autogen
+-include ../../config.mak
+
+TARGETS = scalar$(X) scalar.o
+GITLIBS = ../../common-main.o ../../libgit.a ../../xdiff/lib.a
+
+all: scalar$(X)
+
+$(GITLIBS):
+	$(QUIET_SUBDIR0)../.. $(QUIET_SUBDIR1) $(subst ../../,,$@)
+
+$(TARGETS): $(GITLIBS) scalar.c
+	$(QUIET_SUBDIR0)../.. $(QUIET_SUBDIR1) $(patsubst %,contrib/scalar/%,$@)
+
+clean:
+	$(RM) $(TARGETS)
+
+.PHONY: $(GITLIBS) all clean FORCE
contrib/scalar/scalar.c new file +36 -0
new file mode 100644
index 00000000000..7cff29e0fcd
@@ -0,0 +1,36 @@
+/*
+ * The Scalar command-line interface.
+ */
+
+#include "cache.h"
+#include "gettext.h"
+#include "parse-options.h"
+
+static struct {
+	const char *name;
+	int (*fn)(int, const char **);
+} builtins[] = {
+	{ NULL, NULL},
+};
+
+int cmd_main(int argc, const char **argv)
+{
+	struct strbuf scalar_usage = STRBUF_INIT;
+	int i;
+
+	if (argc > 1) {
+		argv++;
+		argc--;
+
+		for (i = 0; builtins[i].name; i++)
+			if (!strcmp(builtins[i].name, argv[0]))
+				return !!builtins[i].fn(argc, argv);
+	}
+
+	strbuf_addstr(&scalar_usage,
+		      N_("scalar <command> [<options>]\n\nCommands:\n"));
+	for (i = 0; builtins[i].name; i++)
+		strbuf_addf(&scalar_usage, "\t%s\n", builtins[i].name);
+
+	usage(scalar_usage.buf);
+}
Signature · 1 line
gitgitgadget
#181 lore raw

PATCH v7 3/17 scalar: start documenting the command

In reply to Johannes Schindelin via GitGitGadget

From: Johannes Schindelin <[email protected]>

Let's build up the documentation for the Scalar command along with the patches that implement its functionality.

Note: To discourage the feature-incomplete documentation from being mistaken for the complete thing, we do not yet provide any way to build HTML or manual pages from the text file.

---
 contrib/scalar/scalar.txt | 38 ++++++++++++++++++++++++++++++++++++++
 1 file changed, 38 insertions(+)
 create mode 100644 contrib/scalar/scalar.txt
contrib/scalar/scalar.txt new file +38 -0
new file mode 100644
index 00000000000..5f7131861a5
@@ -0,0 +1,38 @@
+scalar(1)
+=========
+
+NAME
+----
+scalar - an opinionated repository management tool
+
+SYNOPSIS
+--------
+[verse]
+scalar <command> [<options>]
+
+DESCRIPTION
+-----------
+
+Scalar is an opinionated repository management tool. By creating new
+repositories or registering existing repositories with Scalar, your Git
+experience will speed up. Scalar sets advanced Git config settings,
+maintains your repositories in the background, and helps reduce data sent
+across the network.
+
+An important Scalar concept is the enlistment: this is the top-level directory
+of the project. It usually contains the subdirectory `src/` which is a Git
+worktree. This encourages the separation between tracked files (inside `src/`)
+and untracked files, such as build artifacts (outside `src/`). When registering
+an existing Git worktree with Scalar whose name is not `src`, the enlistment
+will be identical to the worktree.
+
+The `scalar` command implements various subcommands, and different options
+depending on the subcommand.
+
+SEE ALSO
+--------
+linkgit:git-maintenance[1].
+
+Scalar
+---
+Associated with the linkgit:git[1] suite
Signature · 1 line
gitgitgadget
#182 lore raw

PATCH v7 4/17 scalar: create test infrastructure

In reply to Johannes Schindelin via GitGitGadget

From: Johannes Schindelin <[email protected]>

To test the Scalar command, create a test script in contrib/scalar/t that is executed as make -C contrib/scalar test. Since Scalar has no meaningful capabilities yet, the only test is rather simple. We will add more tests in subsequent commits that introduce corresponding, new functionality.

Note: This test script is intended to test scalar only lightly, even after all of the functionality is implemented.

A more comprehensive functional (or: integration) test suite can be found at https://github.com/microsoft/scalar; It is used in the workflow
https://github.com/microsoft/git/blob/HEAD/.github/workflows/scalar-functional-tests.yml
in Microsoft's Git fork. This test suite performs end-to-end tests with a real remote repository, and is run as part of the regular CI and PR builds in that fork.

Since those tests require some functionality supported only by Microsoft's Git fork ("GVFS protocol"), there is no intention to port that fuller test suite to contrib/scalar/.

---
 contrib/scalar/Makefile          | 17 +++++--
 contrib/scalar/t/Makefile        | 78 ++++++++++++++++++++++++++++++++
 contrib/scalar/t/t9099-scalar.sh | 17 +++++++
 3 files changed, 109 insertions(+), 3 deletions(-)
 create mode 100644 contrib/scalar/t/Makefile
 create mode 100755 contrib/scalar/t/t9099-scalar.sh
contrib/scalar/Makefile +14 -3
index f6f0036f0fa..231b1ee1796 100644
@@ -3,6 +3,7 @@ QUIET_SUBDIR1  =
 
 ifneq ($(findstring s,$(MAKEFLAGS)),s)
 ifndef V
+	QUIET_GEN      = @echo '   ' GEN $@;
 	QUIET_SUBDIR0  = +@subdir=
 	QUIET_SUBDIR1  = ;$(NO_SUBDIR) echo '   ' SUBDIR $$subdir; \
 			 $(MAKE) $(PRINT_DIR) -C $$subdir
@@ -20,7 +21,7 @@ include ../../config.mak.uname
 TARGETS = scalar$(X) scalar.o
 GITLIBS = ../../common-main.o ../../libgit.a ../../xdiff/lib.a
 
-all: scalar$(X)
+all: scalar$(X) ../../bin-wrappers/scalar
 
 $(GITLIBS):
 	$(QUIET_SUBDIR0)../.. $(QUIET_SUBDIR1) $(subst ../../,,$@)
@@ -29,6 +30,16 @@ $(TARGETS): $(GITLIBS) scalar.c
 	$(QUIET_SUBDIR0)../.. $(QUIET_SUBDIR1) $(patsubst %,contrib/scalar/%,$@)
 
 clean:
-	$(RM) $(TARGETS)
+	$(RM) $(TARGETS) ../../bin-wrappers/scalar
 
-.PHONY: $(GITLIBS) all clean FORCE
+../../bin-wrappers/scalar: ../../wrap-for-bin.sh Makefile
+	@mkdir -p ../../bin-wrappers
+	$(QUIET_GEN)sed -e '1s|#!.*/sh|#!$(SHELL_PATH_SQ)|' \
+	     -e 's|@@BUILD_DIR@@|$(shell cd ../.. && pwd)|' \
+	     -e 's|@@PROG@@|contrib/scalar/scalar$(X)|' < $< > $@ && \
+	chmod +x $@
+
+test: all
+	$(MAKE) -C t
+
+.PHONY: $(GITLIBS) all clean test FORCE
contrib/scalar/t/Makefile new file +78 -0

78 lines not on this page · raw message

contrib/scalar/t/t9099-scalar.sh new file +17 -0
new file mode 100755
index 00000000000..16f2b72b126
@@ -0,0 +1,17 @@
+#!/bin/sh
+
+test_description='test the `scalar` command'
+
+TEST_DIRECTORY=$PWD/../../../t
+export TEST_DIRECTORY
+
+# Make it work with --no-bin-wrappers
+PATH=$PWD/..:$PATH
+
+. ../../../t/test-lib.sh
+
+test_expect_success 'scalar shows a usage' '
+	test_expect_code 129 scalar -h
+'
+
+test_done
Signature · 1 line
gitgitgadget
#183 lore raw

PATCH v7 5/17 cmake: optionally build `scalar`, too

In reply to Johannes Schindelin via GitGitGadget

From: Johannes Schindelin <[email protected]>

The CMake configuration unfortunately does not let us encapsulate all (or at least the vast majority) of Scalar's build definition in the contrib/scalar/ subdirectory.

To alleviate that somewhat, we guard the inclusion of Scalar via the INCLUDE_SCALAR environment variable.

---
 contrib/buildsystems/CMakeLists.txt | 14 ++++++++++++++
 1 file changed, 14 insertions(+)
contrib/buildsystems/CMakeLists.txt +14 -0
index fd1399c440f..dd7496b0322 100644
@@ -729,6 +729,13 @@ if(CURL_FOUND)
 	endif()
 endif()
 
+if(DEFINED ENV{INCLUDE_SCALAR} AND NOT ENV{INCLUDE_SCALAR} STREQUAL "")
+	add_executable(scalar ${CMAKE_SOURCE_DIR}/contrib/scalar/scalar.c)
+	target_link_libraries(scalar common-main)
+	set_target_properties(scalar PROPERTIES RUNTIME_OUTPUT_DIRECTORY_DEBUG ${CMAKE_BINARY_DIR}/contrib/scalar)
+	set_target_properties(scalar PROPERTIES RUNTIME_OUTPUT_DIRECTORY_RELEASE ${CMAKE_BINARY_DIR}/contrib/scalar)
+endif()
+
 parse_makefile_for_executables(git_builtin_extra "BUILT_INS")
 
 option(SKIP_DASHED_BUILT_INS "Skip hardlinking the dashed versions of the built-ins")
@@ -953,6 +960,13 @@ string(REPLACE "@@BUILD_DIR@@" "${CMAKE_BINARY_DIR}" content "${content}")
 string(REPLACE "@@PROG@@" "git-cvsserver" content "${content}")
 file(WRITE ${CMAKE_BINARY_DIR}/bin-wrappers/git-cvsserver ${content})
 
+if(DEFINED ENV{INCLUDE_SCALAR} AND NOT ENV{INCLUDE_SCALAR} STREQUAL "")
+	file(STRINGS ${CMAKE_SOURCE_DIR}/wrap-for-bin.sh content NEWLINE_CONSUME)
+	string(REPLACE "@@BUILD_DIR@@" "${CMAKE_BINARY_DIR}" content "${content}")
+	string(REPLACE "@@PROG@@" "contrib/scalar/scalar${EXE_EXTENSION}" content "${content}")
+	file(WRITE ${CMAKE_BINARY_DIR}/bin-wrappers/scalar ${content})
+endif()
+
 #options for configuring test options
 option(PERL_TESTS "Perform tests that use perl" ON)
 option(PYTHON_TESTS "Perform tests that use python" ON)
Signature · 1 line
gitgitgadget
#190 lore raw

PATCH v7 10/17 scalar: implement 'scalar list'

In reply to Johannes Schindelin via GitGitGadget

From: Derrick Stolee <[email protected]>

The produced list simply consists of those repositories registered under the multi-valued scalar.repo config setting in the user's Git config.

---
 contrib/scalar/scalar.c   | 11 +++++++++++
 contrib/scalar/scalar.txt | 11 ++++++++++-
 2 files changed, 21 insertions(+), 1 deletion(-)
contrib/scalar/scalar.c +11 -0
index 097d3bd478b..4feacd8d62b 100644
@@ -251,6 +251,16 @@ static int unregister_dir(void)
 	return res;
 }
 
+static int cmd_list(int argc, const char **argv)
+{
+	if (argc != 1)
+		die(_("`scalar list` does not take arguments"));
+
+	if (run_git("config", "--global", "--get-all", "scalar.repo", NULL) < 0)
+		return -1;
+	return 0;
+}
+
 static int cmd_register(int argc, const char **argv)
 {
 	struct option options[] = {
@@ -337,6 +347,7 @@ static struct {
 	const char *name;
 	int (*fn)(int, const char **);
 } builtins[] = {
+	{ "list", cmd_list },
 	{ "register", cmd_register },
 	{ "unregister", cmd_unregister },
 	{ NULL, NULL},
contrib/scalar/scalar.txt +10 -1
index d9a79984492..f93e3d00efd 100644
@@ -8,6 +8,7 @@ scalar - an opinionated repository management tool
 SYNOPSIS
 --------
 [verse]
+scalar list
 scalar register [<enlistment>]
 scalar unregister [<enlistment>]
 
@@ -28,11 +29,19 @@ an existing Git worktree with Scalar whose name is not `src`, the enlistment
 will be identical to the worktree.
 
 The `scalar` command implements various subcommands, and different options
-depending on the subcommand.
+depending on the subcommand. With the exception of `list`, all subcommands
+expect to be run in an enlistment.
 
 COMMANDS
 --------
 
+List
+~~~~
+
+list::
+	List enlistments that are currently registered by Scalar. This
+	subcommand does not need to be run inside an enlistment.
+
 Register
 ~~~~~~~~
 
Signature · 1 line
gitgitgadget
#188 lore raw

PATCH v7 8/17 scalar: 'unregister' stops background maintenance

In reply to Johannes Schindelin via GitGitGadget

From: Derrick Stolee <[email protected]>

Just like scalar register starts the scheduled background maintenance, scalar unregister stops it. Note that we use git maintenance start in scalar register, but we do not use git maintenance stop in scalar unregister: this would stop maintenance for _all_ repositories, not just for the one we want to unregister.

The unregister command also removes the corresponding entry from the [scalar] section in the global Git config.

---
 contrib/scalar/scalar.c   | 50 ++++++++++++++++++++++++++++++++-------
 contrib/scalar/scalar.txt |  8 +++++++
 2 files changed, 50 insertions(+), 8 deletions(-)
contrib/scalar/scalar.c +42 -8
index 03d5f84c764..bab0271c37d 100644
@@ -198,12 +198,12 @@ static int set_recommended_config(void)
 	return 0;
 }
 
-static int start_maintenance(void)
+static int toggle_maintenance(int enable)
 {
-	return run_git("maintenance", "start", NULL);
+	return run_git("maintenance", enable ? "start" : "unregister", NULL);
 }
 
-static int add_enlistment(void)
+static int add_or_remove_enlistment(int add)
 {
 	int res;
 
@@ -214,24 +214,39 @@ static int add_enlistment(void)
 		      "scalar.repo", the_repository->worktree, NULL);
 
 	/*
-	 * If the setting is already there, then do nothing.
+	 * If we want to add and the setting is already there, then do nothing.
+	 * If we want to remove and the setting is not there, then do nothing.
 	 */
-	if (!res)
+	if ((add && !res) || (!add && res))
 		return 0;
 
-	return run_git("config", "--global", "--add",
+	return run_git("config", "--global", add ? "--add" : "--unset",
+		       add ? "--no-fixed-value" : "--fixed-value",
 		       "scalar.repo", the_repository->worktree, NULL);
 }
 
 static int register_dir(void)
 {
-	int res = add_enlistment();
+	int res = add_or_remove_enlistment(1);
 
 	if (!res)
 		res = set_recommended_config();
 
 	if (!res)
-		res = start_maintenance();
+		res = toggle_maintenance(1);
+
+	return res;
+}
+
+static int unregister_dir(void)
+{
+	int res = 0;
+
+	if (toggle_maintenance(0) < 0)
+		res = -1;
+
+	if (add_or_remove_enlistment(0) < 0)
+		res = -1;
 
 	return res;
 }
@@ -254,11 +269,30 @@ static int cmd_register(int argc, const char **argv)
 	return register_dir();
 }
 
+static int cmd_unregister(int argc, const char **argv)
+{
+	struct option options[] = {
+		OPT_END(),
+	};
+	const char * const usage[] = {
+		N_("scalar unregister [<enlistment>]"),
+		NULL
+	};
+
+	argc = parse_options(argc, argv, NULL, options,
+			     usage, 0);
+
+	setup_enlistment_directory(argc, argv, usage, options, NULL);
+
+	return unregister_dir();
+}
+
 static struct {
 	const char *name;
 	int (*fn)(int, const char **);
 } builtins[] = {
 	{ "register", cmd_register },
+	{ "unregister", cmd_unregister },
 	{ NULL, NULL},
 };
 
contrib/scalar/scalar.txt +8 -0
index 568987064b2..d9a79984492 100644
@@ -9,6 +9,7 @@ SYNOPSIS
 --------
 [verse]
 scalar register [<enlistment>]
+scalar unregister [<enlistment>]
 
 DESCRIPTION
 -----------
@@ -45,6 +46,13 @@ Note: when this subcommand is called in a worktree that is called `src/`, its
 parent directory is considered to be the Scalar enlistment. If the worktree is
 _not_ called `src/`, it itself will be considered to be the Scalar enlistment.
 
+Unregister
+~~~~~~~~~~
+
+unregister [<enlistment>]::
+	Remove the specified repository from the list of repositories
+	registered with Scalar and stop the scheduled background maintenance.
+
 SEE ALSO
 --------
 linkgit:git-maintenance[1].
Signature · 1 line
gitgitgadget
#189 lore raw

PATCH v7 9/17 scalar: let 'unregister' handle a deleted enlistment directory gracefully

In reply to Johannes Schindelin via GitGitGadget

From: Johannes Schindelin <[email protected]>

When a user deleted an enlistment manually, let's be generous and _still_ unregister it.

---
 contrib/scalar/scalar.c          | 46 ++++++++++++++++++++++++++++++++
 contrib/scalar/t/t9099-scalar.sh | 15 +++++++++++
 2 files changed, 61 insertions(+)
contrib/scalar/scalar.c +46 -0
index bab0271c37d..097d3bd478b 100644
@@ -269,6 +269,24 @@ static int cmd_register(int argc, const char **argv)
 	return register_dir();
 }
 
+static int remove_deleted_enlistment(struct strbuf *path)
+{
+	int res = 0;
+	strbuf_realpath_forgiving(path, path->buf, 1);
+
+	if (run_git("config", "--global",
+		    "--unset", "--fixed-value",
+		    "scalar.repo", path->buf, NULL) < 0)
+		res = -1;
+
+	if (run_git("config", "--global",
+		    "--unset", "--fixed-value",
+		    "maintenance.repo", path->buf, NULL) < 0)
+		res = -1;
+
+	return res;
+}
+
 static int cmd_unregister(int argc, const char **argv)
 {
 	struct option options[] = {
@@ -282,6 +300,34 @@ static int cmd_unregister(int argc, const char **argv)
 	argc = parse_options(argc, argv, NULL, options,
 			     usage, 0);
 
+	/*
+	 * Be forgiving when the enlistment or worktree does not even exist any
+	 * longer; This can be the case if a user deleted the worktree by
+	 * mistake and _still_ wants to unregister the thing.
+	 */
+	if (argc == 1) {
+		struct strbuf src_path = STRBUF_INIT, workdir_path = STRBUF_INIT;
+
+		strbuf_addf(&src_path, "%s/src/.git", argv[0]);
+		strbuf_addf(&workdir_path, "%s/.git", argv[0]);
+		if (!is_directory(src_path.buf) && !is_directory(workdir_path.buf)) {
+			/* remove possible matching registrations */
+			int res = -1;
+
+			strbuf_strip_suffix(&src_path, "/.git");
+			res = remove_deleted_enlistment(&src_path) && res;
+
+			strbuf_strip_suffix(&workdir_path, "/.git");
+			res = remove_deleted_enlistment(&workdir_path) && res;
+
+			strbuf_release(&src_path);
+			strbuf_release(&workdir_path);
+			return res;
+		}
+		strbuf_release(&src_path);
+		strbuf_release(&workdir_path);
+	}
+
 	setup_enlistment_directory(argc, argv, usage, options, NULL);
 
 	return unregister_dir();
contrib/scalar/t/t9099-scalar.sh +15 -0
index 16f2b72b126..ef0e8d680d5 100755
@@ -14,4 +14,19 @@ test_expect_success 'scalar shows a usage' '
 	test_expect_code 129 scalar -h
 '
 
+test_expect_success 'scalar unregister' '
+	git init vanish/src &&
+	scalar register vanish/src &&
+	git config --get --global --fixed-value \
+		maintenance.repo "$(pwd)/vanish/src" &&
+	scalar list >scalar.repos &&
+	grep -F "$(pwd)/vanish/src" scalar.repos &&
+	rm -rf vanish/src/.git &&
+	scalar unregister vanish &&
+	test_must_fail git config --get --global --fixed-value \
+		maintenance.repo "$(pwd)/vanish/src" &&
+	scalar list >scalar.repos &&
+	! grep -F "$(pwd)/vanish/src" scalar.repos
+'
+
 test_done
Signature · 1 line
gitgitgadget
#191 lore raw

PATCH v7 11/17 scalar: implement the `clone` subcommand

In reply to Johannes Schindelin via GitGitGadget

From: Johannes Schindelin <[email protected]>

This implements Scalar's opinionated clone command: it tries to use a partial clone and sets up a sparse checkout by default. In contrast to git clone, scalar clone sets up the worktree in the src/ subdirectory, to encourage a separation between the source files and the build output (which helps Git tremendously because it avoids untracked files that have to be specifically ignored when refreshing the index).

Also, it registers the repository for regular, scheduled maintenance, and configures a flurry of configuration settings based on the experience and experiments of the Microsoft Windows and the Microsoft Office development teams.

Note: since the scalar clone command is by far the most commonly called scalar subcommand, we document it at the top of the manual page.

---
 contrib/scalar/scalar.c          | 201 +++++++++++++++++++++++++++++++
 contrib/scalar/scalar.txt        |  32 ++++-
 contrib/scalar/t/t9099-scalar.sh |  32 +++++
 3 files changed, 262 insertions(+), 3 deletions(-)
contrib/scalar/scalar.c +201 -0
index 4feacd8d62b..43f83dde33b 100644
@@ -7,6 +7,7 @@
 #include "parse-options.h"
 #include "config.h"
 #include "run-command.h"
+#include "refs.h"
 
 /*
  * Remove the deepest subdirectory in the provided path string. Path must not
@@ -251,6 +252,205 @@ static int unregister_dir(void)
 	return res;
 }
 
+/* printf-style interface, expects `<key>=<value>` argument */
+static int set_config(const char *fmt, ...)
+{
+	struct strbuf buf = STRBUF_INIT;
+	char *value;
+	int res;
+	va_list args;
+
+	va_start(args, fmt);
+	strbuf_vaddf(&buf, fmt, args);
+	va_end(args);
+
+	value = strchr(buf.buf, '=');
+	if (value)
+		*(value++) = '\0';
+	res = git_config_set_gently(buf.buf, value);
+	strbuf_release(&buf);
+
+	return res;
+}
+
+static char *remote_default_branch(const char *url)
+{
+	struct child_process cp = CHILD_PROCESS_INIT;
+	struct strbuf out = STRBUF_INIT;
+
+	cp.git_cmd = 1;
+	strvec_pushl(&cp.args, "ls-remote", "--symref", url, "HEAD", NULL);
+	if (!pipe_command(&cp, NULL, 0, &out, 0, NULL, 0)) {
+		const char *line = out.buf;
+
+		while (*line) {
+			const char *eol = strchrnul(line, '\n'), *p;
+			size_t len = eol - line;
+			char *branch;
+
+			if (!skip_prefix(line, "ref: ", &p) ||
+			    !strip_suffix_mem(line, &len, "\tHEAD")) {
+				line = eol + (*eol == '\n');
+				continue;
+			}
+
+			eol = line + len;
+			if (skip_prefix(p, "refs/heads/", &p)) {
+				branch = xstrndup(p, eol - p);
+				strbuf_release(&out);
+				return branch;
+			}
+
+			error(_("remote HEAD is not a branch: '%.*s'"),
+			      (int)(eol - p), p);
+			strbuf_release(&out);
+			return NULL;
+		}
+	}
+	warning(_("failed to get default branch name from remote; "
+		  "using local default"));
+	strbuf_reset(&out);
+
+	child_process_init(&cp);
+	cp.git_cmd = 1;
+	strvec_pushl(&cp.args, "symbolic-ref", "--short", "HEAD", NULL);
+	if (!pipe_command(&cp, NULL, 0, &out, 0, NULL, 0)) {
+		strbuf_trim(&out);
+		return strbuf_detach(&out, NULL);
+	}
+
+	strbuf_release(&out);
+	error(_("failed to get default branch name"));
+	return NULL;
+}
+
+static int cmd_clone(int argc, const char **argv)
+{
+	const char *branch = NULL;
+	int full_clone = 0;
+	struct option clone_options[] = {
+		OPT_STRING('b', "branch", &branch, N_("<branch>"),
+			   N_("branch to checkout after clone")),
+		OPT_BOOL(0, "full-clone", &full_clone,
+			 N_("when cloning, create full working directory")),
+		OPT_END(),
+	};
+	const char * const clone_usage[] = {
+		N_("scalar clone [<options>] [--] <repo> [<dir>]"),
+		NULL
+	};
+	const char *url;
+	char *enlistment = NULL, *dir = NULL;
+	struct strbuf buf = STRBUF_INIT;
+	int res;
+
+	argc = parse_options(argc, argv, NULL, clone_options, clone_usage, 0);
+
+	if (argc == 2) {
+		url = argv[0];
+		enlistment = xstrdup(argv[1]);
+	} else if (argc == 1) {
+		url = argv[0];
+
+		strbuf_addstr(&buf, url);
+		/* Strip trailing slashes, if any */
+		while (buf.len > 0 && is_dir_sep(buf.buf[buf.len - 1]))
+			strbuf_setlen(&buf, buf.len - 1);
+		/* Strip suffix `.git`, if any */
+		strbuf_strip_suffix(&buf, ".git");
+
+		enlistment = find_last_dir_sep(buf.buf);
+		if (!enlistment) {
+			die(_("cannot deduce worktree name from '%s'"), url);
+		}
+		enlistment = xstrdup(enlistment + 1);
+	} else {
+		usage_msg_opt(_("You must specify a repository to clone."),
+			      clone_usage, clone_options);
+	}
+
+	if (is_directory(enlistment))
+		die(_("directory '%s' exists already"), enlistment);
+
+	dir = xstrfmt("%s/src", enlistment);
+
+	strbuf_reset(&buf);
+	if (branch)
+		strbuf_addf(&buf, "init.defaultBranch=%s", branch);
+	else {
+		char *b = repo_default_branch_name(the_repository, 1);
+		strbuf_addf(&buf, "init.defaultBranch=%s", b);
+		free(b);
+	}
+
+	if ((res = run_git("-c", buf.buf, "init", "--", dir, NULL)))
+		goto cleanup;
+
+	if (chdir(dir) < 0) {
+		res = error_errno(_("could not switch to '%s'"), dir);
+		goto cleanup;
+	}
+
+	setup_git_directory();
+
+	/* common-main already logs `argv` */
+	trace2_def_repo(the_repository);
+
+	if (!branch && !(branch = remote_default_branch(url))) {
+		res = error(_("failed to get default branch for '%s'"), url);
+		goto cleanup;
+	}
+
+	if (set_config("remote.origin.url=%s", url) ||
+	    set_config("remote.origin.fetch="
+		       "+refs/heads/*:refs/remotes/origin/*") ||
+	    set_config("remote.origin.promisor=true") ||
+	    set_config("remote.origin.partialCloneFilter=blob:none")) {
+		res = error(_("could not configure remote in '%s'"), dir);
+		goto cleanup;
+	}
+
+	if (!full_clone &&
+	    (res = run_git("sparse-checkout", "init", "--cone", NULL)))
+		goto cleanup;
+
+	if (set_recommended_config())
+		return error(_("could not configure '%s'"), dir);
+
+	if ((res = run_git("fetch", "--quiet", "origin", NULL))) {
+		warning(_("partial clone failed; attempting full clone"));
+
+		if (set_config("remote.origin.promisor") ||
+		    set_config("remote.origin.partialCloneFilter")) {
+			res = error(_("could not configure for full clone"));
+			goto cleanup;
+		}
+
+		if ((res = run_git("fetch", "--quiet", "origin", NULL)))
+			goto cleanup;
+	}
+
+	if ((res = set_config("branch.%s.remote=origin", branch)))
+		goto cleanup;
+	if ((res = set_config("branch.%s.merge=refs/heads/%s",
+			      branch, branch)))
+		goto cleanup;
+
+	strbuf_reset(&buf);
+	strbuf_addf(&buf, "origin/%s", branch);
+	res = run_git("checkout", "-f", "-t", buf.buf, NULL);
+	if (res)
+		goto cleanup;
+
+	res = register_dir();
+
+cleanup:
+	free(enlistment);
+	free(dir);
+	strbuf_release(&buf);
+	return res;
+}
+
 static int cmd_list(int argc, const char **argv)
 {
 	if (argc != 1)
@@ -347,6 +547,7 @@ static struct {
 	const char *name;
 	int (*fn)(int, const char **);
 } builtins[] = {
+	{ "clone", cmd_clone },
 	{ "list", cmd_list },
 	{ "register", cmd_register },
 	{ "unregister", cmd_unregister },
contrib/scalar/scalar.txt +29 -3
index f93e3d00efd..e8730967f16 100644
@@ -8,6 +8,7 @@ scalar - an opinionated repository management tool
 SYNOPSIS
 --------
 [verse]
+scalar clone [--branch <main-branch>] [--full-clone] <url> [<enlistment>]
 scalar list
 scalar register [<enlistment>]
 scalar unregister [<enlistment>]
@@ -29,12 +30,37 @@ an existing Git worktree with Scalar whose name is not `src`, the enlistment
 will be identical to the worktree.
 
 The `scalar` command implements various subcommands, and different options
-depending on the subcommand. With the exception of `list`, all subcommands
-expect to be run in an enlistment.
+depending on the subcommand. With the exception of `clone` and `list`, all
+subcommands expect to be run in an enlistment.
 
 COMMANDS
 --------
 
+Clone
+~~~~~
+
+clone [<options>] <url> [<enlistment>]::
+	Clones the specified repository, similar to linkgit:git-clone[1]. By
+	default, only commit and tree objects are cloned. Once finished, the
+	worktree is located at `<enlistment>/src`.
++
+The sparse-checkout feature is enabled (except when run with `--full-clone`)
+and the only files present are those in the top-level directory. Use
+`git sparse-checkout set` to expand the set of directories you want to see,
+or `git sparse-checkout disable` to expand to all files (see
+linkgit:git-sparse-checkout[1] for more details). You can explore the
+subdirectories outside your sparse-checkout by using `git ls-tree
+HEAD[:<directory>]`.
+
+-b <name>::
+--branch <name>::
+	Instead of checking out the branch pointed to by the cloned
+	repository's HEAD, check out the `<name>` branch instead.
+
+--[no-]full-clone::
+	A sparse-checkout is initialized by default. This behavior can be
+	turned off via `--full-clone`.
+
 List
 ~~~~
 
@@ -64,7 +90,7 @@ unregister [<enlistment>]::
 
 SEE ALSO
 --------
-linkgit:git-maintenance[1].
+linkgit:git-clone[1], linkgit:git-maintenance[1].
 
 Scalar
 ---
contrib/scalar/t/t9099-scalar.sh +32 -0
index ef0e8d680d5..984d69e8f75 100755
@@ -10,6 +10,9 @@ PATH=$PWD/..:$PATH
 
 . ../../../t/test-lib.sh
 
+GIT_TEST_MAINT_SCHEDULER="crontab:test-tool crontab ../cron.txt,launchctl:true,schtasks:true"
+export GIT_TEST_MAINT_SCHEDULER
+
 test_expect_success 'scalar shows a usage' '
 	test_expect_code 129 scalar -h
 '
@@ -29,4 +32,33 @@ test_expect_success 'scalar unregister' '
 	! grep -F "$(pwd)/vanish/src" scalar.repos
 '
 
+test_expect_success 'set up repository to clone' '
+	test_commit first &&
+	test_commit second &&
+	test_commit third &&
+	git switch -c parallel first &&
+	mkdir -p 1/2 &&
+	test_commit 1/2/3 &&
+	git config uploadPack.allowFilter true &&
+	git config uploadPack.allowAnySHA1InWant true
+'
+
+test_expect_success 'scalar clone' '
+	second=$(git rev-parse --verify second:second.t) &&
+	scalar clone "file://$(pwd)" cloned &&
+	(
+		cd cloned/src &&
+
+		git config --get --global --fixed-value maintenance.repo \
+			"$(pwd)" &&
+
+		test_path_is_missing 1/2 &&
+		test_must_fail git rev-list --missing=print $second &&
+		git rev-list $second &&
+		git cat-file blob $second >actual &&
+		echo "second" >expect &&
+		test_cmp expect actual
+	)
+'
+
 test_done
Signature · 1 line
gitgitgadget
#192 lore raw

PATCH v7 12/17 scalar: teach 'clone' to support the --single-branch option

In reply to Johannes Schindelin via GitGitGadget

From: Johannes Schindelin <[email protected]>

Just like git clone, the scalar clone command now also offers to restrict the clone to a single branch.

---
 contrib/scalar/scalar.c          |  9 +++++++--
 contrib/scalar/scalar.txt        | 12 +++++++++++-
 contrib/scalar/t/t9099-scalar.sh |  6 +++++-
 3 files changed, 23 insertions(+), 4 deletions(-)
contrib/scalar/scalar.c +7 -2
index 43f83dde33b..516a75be3c4 100644
@@ -327,12 +327,15 @@ static char *remote_default_branch(const char *url)
 static int cmd_clone(int argc, const char **argv)
 {
 	const char *branch = NULL;
-	int full_clone = 0;
+	int full_clone = 0, single_branch = 0;
 	struct option clone_options[] = {
 		OPT_STRING('b', "branch", &branch, N_("<branch>"),
 			   N_("branch to checkout after clone")),
 		OPT_BOOL(0, "full-clone", &full_clone,
 			 N_("when cloning, create full working directory")),
+		OPT_BOOL(0, "single-branch", &single_branch,
+			 N_("only download metadata for the branch that will "
+			    "be checked out")),
 		OPT_END(),
 	};
 	const char * const clone_usage[] = {
@@ -403,7 +406,9 @@ static int cmd_clone(int argc, const char **argv)
 
 	if (set_config("remote.origin.url=%s", url) ||
 	    set_config("remote.origin.fetch="
-		       "+refs/heads/*:refs/remotes/origin/*") ||
+		       "+refs/heads/%s:refs/remotes/origin/%s",
+		       single_branch ? branch : "*",
+		       single_branch ? branch : "*") ||
 	    set_config("remote.origin.promisor=true") ||
 	    set_config("remote.origin.partialCloneFilter=blob:none")) {
 		res = error(_("could not configure remote in '%s'"), dir);
contrib/scalar/scalar.txt +11 -1
index e8730967f16..56f744a4aa9 100644
@@ -8,7 +8,7 @@ scalar - an opinionated repository management tool
 SYNOPSIS
 --------
 [verse]
-scalar clone [--branch <main-branch>] [--full-clone] <url> [<enlistment>]
+scalar clone [--single-branch] [--branch <main-branch>] [--full-clone] <url> [<enlistment>]
 scalar list
 scalar register [<enlistment>]
 scalar unregister [<enlistment>]
@@ -57,6 +57,16 @@ HEAD[:<directory>]`.
 	Instead of checking out the branch pointed to by the cloned
 	repository's HEAD, check out the `<name>` branch instead.
 
+--[no-]single-branch::
+	Clone only the history leading to the tip of a single branch, either
+	specified by the `--branch` option or the primary branch remote's
+	`HEAD` points at.
++
+Further fetches into the resulting repository will only update the
+remote-tracking branch for the branch this option was used for the initial
+cloning. If the HEAD at the remote did not point at any branch when
+`--single-branch` clone was made, no remote-tracking branch is created.
+
 --[no-]full-clone::
 	A sparse-checkout is initialized by default. This behavior can be
 	turned off via `--full-clone`.
contrib/scalar/t/t9099-scalar.sh +5 -1
index 984d69e8f75..f60e086d6f9 100755
@@ -45,13 +45,17 @@ test_expect_success 'set up repository to clone' '
 
 test_expect_success 'scalar clone' '
 	second=$(git rev-parse --verify second:second.t) &&
-	scalar clone "file://$(pwd)" cloned &&
+	scalar clone "file://$(pwd)" cloned --single-branch &&
 	(
 		cd cloned/src &&
 
 		git config --get --global --fixed-value maintenance.repo \
 			"$(pwd)" &&
 
+		git for-each-ref --format="%(refname)" refs/remotes/origin/ >actual &&
+		echo "refs/remotes/origin/parallel" >expect &&
+		test_cmp expect actual &&
+
 		test_path_is_missing 1/2 &&
 		test_must_fail git rev-list --missing=print $second &&
 		git rev-list $second &&
Signature · 1 line
gitgitgadget
#194 lore raw

PATCH v7 14/17 scalar: allow reconfiguring an existing enlistment

In reply to Johannes Schindelin via GitGitGadget

From: Johannes Schindelin <[email protected]>

This comes in handy during Scalar upgrades, or when config settings were messed up by mistake.

---
 contrib/scalar/scalar.c          | 79 +++++++++++++++++++++-----------
 contrib/scalar/scalar.txt        |  8 ++++
 contrib/scalar/t/t9099-scalar.sh |  8 ++++
 3 files changed, 67 insertions(+), 28 deletions(-)
contrib/scalar/scalar.c +51 -28
index ca524576011..b799decbc2f 100644
@@ -108,18 +108,20 @@ static int run_git(const char *arg, ...)
 	return res;
 }
 
-static int set_recommended_config(void)
+static int set_recommended_config(int reconfigure)
 {
 	struct {
 		const char *key;
 		const char *value;
+		int overwrite_on_reconfigure;
 	} config[] = {
-		{ "am.keepCR", "true" },
-		{ "core.FSCache", "true" },
-		{ "core.multiPackIndex", "true" },
-		{ "core.preloadIndex", "true" },
+		/* Required */
+		{ "am.keepCR", "true", 1 },
+		{ "core.FSCache", "true", 1 },
+		{ "core.multiPackIndex", "true", 1 },
+		{ "core.preloadIndex", "true", 1 },
 #ifndef WIN32
-		{ "core.untrackedCache", "true" },
+		{ "core.untrackedCache", "true", 1 },
 #else
 		/*
 		 * Unfortunately, Scalar's Functional Tests demonstrated
@@ -133,28 +135,29 @@ static int set_recommended_config(void)
 		 * Therefore, with a sad heart, we disable this very useful
 		 * feature on Windows.
 		 */
-		{ "core.untrackedCache", "false" },
+		{ "core.untrackedCache", "false", 1 },
 #endif
-		{ "core.logAllRefUpdates", "true" },
-		{ "credential.https://dev.azure.com.useHttpPath", "true" },
-		{ "credential.validate", "false" }, /* GCM4W-only */
-		{ "gc.auto", "0" },
-		{ "gui.GCWarning", "false" },
-		{ "index.threads", "true" },
-		{ "index.version", "4" },
-		{ "merge.stat", "false" },
-		{ "merge.renames", "false" },
-		{ "pack.useBitmaps", "false" },
-		{ "pack.useSparse", "true" },
-		{ "receive.autoGC", "false" },
-		{ "reset.quiet", "true" },
-		{ "feature.manyFiles", "false" },
-		{ "feature.experimental", "false" },
-		{ "fetch.unpackLimit", "1" },
-		{ "fetch.writeCommitGraph", "false" },
+		{ "core.logAllRefUpdates", "true", 1 },
+		{ "credential.https://dev.azure.com.useHttpPath", "true", 1 },
+		{ "credential.validate", "false", 1 }, /* GCM4W-only */
+		{ "gc.auto", "0", 1 },
+		{ "gui.GCWarning", "false", 1 },
+		{ "index.threads", "true", 1 },
+		{ "index.version", "4", 1 },
+		{ "merge.stat", "false", 1 },
+		{ "merge.renames", "false", 1 },
+		{ "pack.useBitmaps", "false", 1 },
+		{ "pack.useSparse", "true", 1 },
+		{ "receive.autoGC", "false", 1 },
+		{ "reset.quiet", "true", 1 },
+		{ "feature.manyFiles", "false", 1 },
+		{ "feature.experimental", "false", 1 },
+		{ "fetch.unpackLimit", "1", 1 },
+		{ "fetch.writeCommitGraph", "false", 1 },
 #ifdef WIN32
-		{ "http.sslBackend", "schannel" },
+		{ "http.sslBackend", "schannel", 1 },
 #endif
+		/* Optional */
 		{ "status.aheadBehind", "false" },
 		{ "commitGraph.generationVersion", "1" },
 		{ "core.autoCRLF", "false" },
@@ -166,7 +169,8 @@ static int set_recommended_config(void)
 	char *value;
 
 	for (i = 0; config[i].key; i++) {
-		if (git_config_get_string(config[i].key, &value)) {
+		if ((reconfigure && config[i].overwrite_on_reconfigure) ||
+		    git_config_get_string(config[i].key, &value)) {
 			trace2_data_string("scalar", the_repository, config[i].key, "created");
 			if (git_config_set_gently(config[i].key,
 						  config[i].value) < 0)
@@ -231,7 +235,7 @@ static int register_dir(void)
 	int res = add_or_remove_enlistment(1);
 
 	if (!res)
-		res = set_recommended_config();
+		res = set_recommended_config(0);
 
 	if (!res)
 		res = toggle_maintenance(1);
@@ -419,7 +423,7 @@ static int cmd_clone(int argc, const char **argv)
 	    (res = run_git("sparse-checkout", "init", "--cone", NULL)))
 		goto cleanup;
 
-	if (set_recommended_config())
+	if (set_recommended_config(0))
 		return error(_("could not configure '%s'"), dir);
 
 	if ((res = run_git("fetch", "--quiet", "origin", NULL))) {
@@ -484,6 +488,24 @@ static int cmd_register(int argc, const char **argv)
 	return register_dir();
 }
 
+static int cmd_reconfigure(int argc, const char **argv)
+{
+	struct option options[] = {
+		OPT_END(),
+	};
+	const char * const usage[] = {
+		N_("scalar reconfigure [<enlistment>]"),
+		NULL
+	};
+
+	argc = parse_options(argc, argv, NULL, options,
+			     usage, 0);
+
+	setup_enlistment_directory(argc, argv, usage, options, NULL);
+
+	return set_recommended_config(1);
+}
+
 static int cmd_run(int argc, const char **argv)
 {
 	struct option options[] = {
@@ -620,6 +642,7 @@ static struct {
 	{ "register", cmd_register },
 	{ "unregister", cmd_unregister },
 	{ "run", cmd_run },
+	{ "reconfigure", cmd_reconfigure },
 	{ NULL, NULL},
 };
 
contrib/scalar/scalar.txt +8 -0
index 39143b08324..89fd7901585 100644
@@ -13,6 +13,7 @@ scalar list
 scalar register [<enlistment>]
 scalar unregister [<enlistment>]
 scalar run ( all | config | commit-graph | fetch | loose-objects | pack-files ) [<enlistment>]
+scalar reconfigure <enlistment>
 
 DESCRIPTION
 -----------
@@ -117,6 +118,13 @@ opinionated default settings that make Git work more efficiently with
 large repositories. As this task is run as part of `scalar clone`
 automatically, explicit invocations of this task are rarely needed.
 
+Reconfigure
+~~~~~~~~~~~
+
+After a Scalar upgrade, or when the configuration of a Scalar enlistment
+was somehow corrupted or changed by mistake, this subcommand allows to
+reconfigure the enlistment.
+
 SEE ALSO
 --------
 linkgit:git-clone[1], linkgit:git-maintenance[1].
contrib/scalar/t/t9099-scalar.sh +8 -0
index f60e086d6f9..fb5e2efee0a 100755
@@ -65,4 +65,12 @@ test_expect_success 'scalar clone' '
 	)
 '
 
+test_expect_success 'scalar reconfigure' '
+	git init one/src &&
+	scalar register one &&
+	git -C one/src config core.preloadIndex false &&
+	scalar reconfigure one &&
+	test true = "$(git -C one/src config core.preloadIndex)"
+'
+
 test_done
Signature · 1 line
gitgitgadget
#187 lore raw

PATCH v7 7/17 scalar: 'register' sets recommended config and starts maintenance

In reply to Johannes Schindelin via GitGitGadget

From: Derrick Stolee <[email protected]>

Let's start implementing the register command. With this commit, recommended settings are configured upon scalar register, and Git's background maintenance is started.

The recommended config settings may very well change in the future. For example, once the built-in FSMonitor is available, we will want to enable it upon scalar register. For that reason, we explicitly support running scalar register in an already-registered enlistment.

---
 contrib/scalar/scalar.c   | 249 ++++++++++++++++++++++++++++++++++++++
 contrib/scalar/scalar.txt |  18 ++-
 2 files changed, 266 insertions(+), 1 deletion(-)
contrib/scalar/scalar.c +249 -0
index 7cff29e0fcd..03d5f84c764 100644
@@ -5,11 +5,260 @@
 #include "cache.h"
 #include "gettext.h"
 #include "parse-options.h"
+#include "config.h"
+#include "run-command.h"
+
+/*
+ * Remove the deepest subdirectory in the provided path string. Path must not
+ * include a trailing path separator. Returns 1 if parent directory found,
+ * otherwise 0.
+ */
+static int strbuf_parent_directory(struct strbuf *buf)
+{
+	size_t len = buf->len;
+	size_t offset = offset_1st_component(buf->buf);
+	char *path_sep = find_last_dir_sep(buf->buf + offset);
+	strbuf_setlen(buf, path_sep ? path_sep - buf->buf : offset);
+
+	return buf->len < len;
+}
+
+static void setup_enlistment_directory(int argc, const char **argv,
+				       const char * const *usagestr,
+				       const struct option *options,
+				       struct strbuf *enlistment_root)
+{
+	struct strbuf path = STRBUF_INIT;
+	char *root;
+	int enlistment_found = 0;
+
+	if (startup_info->have_repository)
+		BUG("gitdir already set up?!?");
+
+	if (argc > 1)
+		usage_with_options(usagestr, options);
+
+	/* find the worktree, determine its corresponding root */
+	if (argc == 1)
+		strbuf_add_absolute_path(&path, argv[0]);
+	else if (strbuf_getcwd(&path) < 0)
+		die(_("need a working directory"));
+
+	strbuf_trim_trailing_dir_sep(&path);
+	do {
+		const size_t len = path.len;
+
+		/* check if currently in enlistment root with src/ workdir */
+		strbuf_addstr(&path, "/src");
+		if (is_nonbare_repository_dir(&path)) {
+			if (enlistment_root)
+				strbuf_add(enlistment_root, path.buf, len);
+
+			enlistment_found = 1;
+			break;
+		}
+
+		/* reset to original path */
+		strbuf_setlen(&path, len);
+
+		/* check if currently in workdir */
+		if (is_nonbare_repository_dir(&path)) {
+			if (enlistment_root) {
+				/*
+				 * If the worktree's directory's name is `src`, the enlistment is the
+				 * parent directory, otherwise it is identical to the worktree.
+				 */
+				root = strip_path_suffix(path.buf, "src");
+				strbuf_addstr(enlistment_root, root ? root : path.buf);
+				free(root);
+			}
+
+			enlistment_found = 1;
+			break;
+		}
+	} while (strbuf_parent_directory(&path));
+
+	if (!enlistment_found)
+		die(_("could not find enlistment root"));
+
+	if (chdir(path.buf) < 0)
+		die_errno(_("could not switch to '%s'"), path.buf);
+
+	strbuf_release(&path);
+	setup_git_directory();
+}
+
+static int run_git(const char *arg, ...)
+{
+	struct strvec argv = STRVEC_INIT;
+	va_list args;
+	const char *p;
+	int res;
+
+	va_start(args, arg);
+	strvec_push(&argv, arg);
+	while ((p = va_arg(args, const char *)))
+		strvec_push(&argv, p);
+	va_end(args);
+
+	res = run_command_v_opt(argv.v, RUN_GIT_CMD);
+
+	strvec_clear(&argv);
+	return res;
+}
+
+static int set_recommended_config(void)
+{
+	struct {
+		const char *key;
+		const char *value;
+	} config[] = {
+		{ "am.keepCR", "true" },
+		{ "core.FSCache", "true" },
+		{ "core.multiPackIndex", "true" },
+		{ "core.preloadIndex", "true" },
+#ifndef WIN32
+		{ "core.untrackedCache", "true" },
+#else
+		/*
+		 * Unfortunately, Scalar's Functional Tests demonstrated
+		 * that the untracked cache feature is unreliable on Windows
+		 * (which is a bummer because that platform would benefit the
+		 * most from it). For some reason, freshly created files seem
+		 * not to update the directory's `lastModified` time
+		 * immediately, but the untracked cache would need to rely on
+		 * that.
+		 *
+		 * Therefore, with a sad heart, we disable this very useful
+		 * feature on Windows.
+		 */
+		{ "core.untrackedCache", "false" },
+#endif
+		{ "core.logAllRefUpdates", "true" },
+		{ "credential.https://dev.azure.com.useHttpPath", "true" },
+		{ "credential.validate", "false" }, /* GCM4W-only */
+		{ "gc.auto", "0" },
+		{ "gui.GCWarning", "false" },
+		{ "index.threads", "true" },
+		{ "index.version", "4" },
+		{ "merge.stat", "false" },
+		{ "merge.renames", "false" },
+		{ "pack.useBitmaps", "false" },
+		{ "pack.useSparse", "true" },
+		{ "receive.autoGC", "false" },
+		{ "reset.quiet", "true" },
+		{ "feature.manyFiles", "false" },
+		{ "feature.experimental", "false" },
+		{ "fetch.unpackLimit", "1" },
+		{ "fetch.writeCommitGraph", "false" },
+#ifdef WIN32
+		{ "http.sslBackend", "schannel" },
+#endif
+		{ "status.aheadBehind", "false" },
+		{ "commitGraph.generationVersion", "1" },
+		{ "core.autoCRLF", "false" },
+		{ "core.safeCRLF", "false" },
+		{ "fetch.showForcedUpdates", "false" },
+		{ NULL, NULL },
+	};
+	int i;
+	char *value;
+
+	for (i = 0; config[i].key; i++) {
+		if (git_config_get_string(config[i].key, &value)) {
+			trace2_data_string("scalar", the_repository, config[i].key, "created");
+			if (git_config_set_gently(config[i].key,
+						  config[i].value) < 0)
+				return error(_("could not configure %s=%s"),
+					     config[i].key, config[i].value);
+		} else {
+			trace2_data_string("scalar", the_repository, config[i].key, "exists");
+			free(value);
+		}
+	}
+
+	/*
+	 * The `log.excludeDecoration` setting is special because it allows
+	 * for multiple values.
+	 */
+	if (git_config_get_string("log.excludeDecoration", &value)) {
+		trace2_data_string("scalar", the_repository,
+				   "log.excludeDecoration", "created");
+		if (git_config_set_multivar_gently("log.excludeDecoration",
+						   "refs/prefetch/*",
+						   CONFIG_REGEX_NONE, 0))
+			return error(_("could not configure "
+				       "log.excludeDecoration"));
+	} else {
+		trace2_data_string("scalar", the_repository,
+				   "log.excludeDecoration", "exists");
+		free(value);
+	}
+
+	return 0;
+}
+
+static int start_maintenance(void)
+{
+	return run_git("maintenance", "start", NULL);
+}
+
+static int add_enlistment(void)
+{
+	int res;
+
+	if (!the_repository->worktree)
+		die(_("Scalar enlistments require a worktree"));
+
+	res = run_git("config", "--global", "--get", "--fixed-value",
+		      "scalar.repo", the_repository->worktree, NULL);
+
+	/*
+	 * If the setting is already there, then do nothing.
+	 */
+	if (!res)
+		return 0;
+
+	return run_git("config", "--global", "--add",
+		       "scalar.repo", the_repository->worktree, NULL);
+}
+
+static int register_dir(void)
+{
+	int res = add_enlistment();
+
+	if (!res)
+		res = set_recommended_config();
+
+	if (!res)
+		res = start_maintenance();
+
+	return res;
+}
+
+static int cmd_register(int argc, const char **argv)
+{
+	struct option options[] = {
+		OPT_END(),
+	};
+	const char * const usage[] = {
+		N_("scalar register [<enlistment>]"),
+		NULL
+	};
+
+	argc = parse_options(argc, argv, NULL, options,
+			     usage, 0);
+
+	setup_enlistment_directory(argc, argv, usage, options, NULL);
+
+	return register_dir();
+}
 
 static struct {
 	const char *name;
 	int (*fn)(int, const char **);
 } builtins[] = {
+	{ "register", cmd_register },
 	{ NULL, NULL},
 };
 
contrib/scalar/scalar.txt +17 -1
index 5f7131861a5..568987064b2 100644
@@ -8,7 +8,7 @@ scalar - an opinionated repository management tool
 SYNOPSIS
 --------
 [verse]
-scalar <command> [<options>]
+scalar register [<enlistment>]
 
 DESCRIPTION
 -----------
@@ -29,6 +29,22 @@ will be identical to the worktree.
 The `scalar` command implements various subcommands, and different options
 depending on the subcommand.
 
+COMMANDS
+--------
+
+Register
+~~~~~~~~
+
+register [<enlistment>]::
+	Adds the enlistment's repository to the list of registered repositories
+	and starts background maintenance. If `<enlistment>` is not provided,
+	then the enlistment associated with the current working directory is
+	registered.
++
+Note: when this subcommand is called in a worktree that is called `src/`, its
+parent directory is considered to be the Scalar enlistment. If the worktree is
+_not_ called `src/`, it itself will be considered to be the Scalar enlistment.
+
 SEE ALSO
 --------
 linkgit:git-maintenance[1].
Signature · 1 line
gitgitgadget
#196 lore raw

PATCH v7 16/17 scalar: implement the `delete` command

In reply to Johannes Schindelin via GitGitGadget

From: Matthew John Cheetham <[email protected]>

Delete an enlistment by first unregistering the repository and then deleting the enlistment directory (usually the directory containing the worktree src/ directory).

On Windows, if the current directory is inside the enlistment's directory, change to the parent of the enlistment directory, to allow us to delete the enlistment (directories used by processes e.g. as current working directories cannot be deleted on Windows).

---
 contrib/scalar/scalar.c          | 63 ++++++++++++++++++++++++++++++++
 contrib/scalar/scalar.txt        |  8 ++++
 contrib/scalar/t/t9099-scalar.sh |  9 +++++
 3 files changed, 80 insertions(+)
contrib/scalar/scalar.c +63 -0
index 71ca573f3af..c53697ad6a0 100644
@@ -8,6 +8,8 @@
 #include "config.h"
 #include "run-command.h"
 #include "refs.h"
+#include "dir.h"
+#include "packfile.h"
 
 /*
  * Remove the deepest subdirectory in the provided path string. Path must not
@@ -328,6 +330,33 @@ static char *remote_default_branch(const char *url)
 	return NULL;
 }
 
+static int delete_enlistment(struct strbuf *enlistment)
+{
+#ifdef WIN32
+	struct strbuf parent = STRBUF_INIT;
+#endif
+
+	if (unregister_dir())
+		die(_("failed to unregister repository"));
+
+#ifdef WIN32
+	/*
+	 * Change the current directory to one outside of the enlistment so
+	 * that we may delete everything underneath it.
+	 */
+	strbuf_addbuf(&parent, enlistment);
+	strbuf_parent_directory(&parent);
+	if (chdir(parent.buf) < 0)
+		die_errno(_("could not switch to '%s'"), parent.buf);
+	strbuf_release(&parent);
+#endif
+
+	if (remove_dir_recursively(enlistment, 0))
+		die(_("failed to delete enlistment directory"));
+
+	return 0;
+}
+
 static int cmd_clone(int argc, const char **argv)
 {
 	const char *branch = NULL;
@@ -688,6 +717,39 @@ static int cmd_unregister(int argc, const char **argv)
 	return unregister_dir();
 }
 
+static int cmd_delete(int argc, const char **argv)
+{
+	char *cwd = xgetcwd();
+	struct option options[] = {
+		OPT_END(),
+	};
+	const char * const usage[] = {
+		N_("scalar delete <enlistment>"),
+		NULL
+	};
+	struct strbuf enlistment = STRBUF_INIT;
+	int res = 0;
+
+	argc = parse_options(argc, argv, NULL, options,
+			     usage, 0);
+
+	if (argc != 1)
+		usage_with_options(usage, options);
+
+	setup_enlistment_directory(argc, argv, usage, options, &enlistment);
+
+	if (dir_inside_of(cwd, enlistment.buf) >= 0)
+		res = error(_("refusing to delete current working directory"));
+	else {
+		close_object_store(the_repository->objects);
+		res = delete_enlistment(&enlistment);
+	}
+	strbuf_release(&enlistment);
+	free(cwd);
+
+	return res;
+}
+
 static struct {
 	const char *name;
 	int (*fn)(int, const char **);
@@ -698,6 +760,7 @@ static struct {
 	{ "unregister", cmd_unregister },
 	{ "run", cmd_run },
 	{ "reconfigure", cmd_reconfigure },
+	{ "delete", cmd_delete },
 	{ NULL, NULL},
 };
 
contrib/scalar/scalar.txt +8 -0
index 737cf563c1a..f416d637289 100644
@@ -14,6 +14,7 @@ scalar register [<enlistment>]
 scalar unregister [<enlistment>]
 scalar run ( all | config | commit-graph | fetch | loose-objects | pack-files ) [<enlistment>]
 scalar reconfigure [ --all | <enlistment> ]
+scalar delete <enlistment>
 
 DESCRIPTION
 -----------
@@ -128,6 +129,13 @@ reconfigure the enlistment.
 With the `--all` option, all enlistments currently registered with Scalar
 will be reconfigured. Use this option after each Scalar upgrade.
 
+Delete
+~~~~~~
+
+delete <enlistment>::
+	This subcommand lets you delete an existing Scalar enlistment from your
+	local file system, unregistering the repository.
+
 SEE ALSO
 --------
 linkgit:git-clone[1], linkgit:git-maintenance[1].
contrib/scalar/t/t9099-scalar.sh +9 -0
index 58af546fd84..2e1502ad45e 100755
@@ -76,4 +76,13 @@ test_expect_success 'scalar reconfigure' '
 	test true = "$(git -C one/src config core.preloadIndex)"
 '
 
+test_expect_success 'scalar delete without enlistment shows a usage' '
+	test_expect_code 129 scalar delete
+'
+
+test_expect_success 'scalar delete with enlistment' '
+	scalar delete cloned &&
+	test_path_is_missing cloned
+'
+
 test_done
Signature · 1 line
gitgitgadget
#186 lore raw

PATCH v7 6/17 ci: also run the `scalar` tests

In reply to Johannes Schindelin via GitGitGadget

From: Johannes Schindelin <[email protected]>

Since Scalar depends on libgit.a, it makes sense to ensure in the CI and the PR builds that it does not get broken in case of industrious refactorings of the core Git code.

---
 .github/workflows/main.yml | 15 +++++++++++++++
 ci/run-build-and-tests.sh  |  1 +
 ci/run-test-slice.sh       |  5 +++++
 3 files changed, 21 insertions(+)
.github/workflows/main.yml +15 -0
index 6ed6a9e8076..6eda6be895d 100644
@@ -89,6 +89,13 @@ jobs:
         HOME: ${{runner.workspace}}
         NO_PERL: 1
       run: . /etc/profile && ci/make-test-artifacts.sh artifacts
+    - name: build Scalar
+      shell: bash
+      run: |
+        make -C contrib/scalar &&
+        mkdir -p artifacts/bin-wrappers artifacts/contrib/scalar &&
+        cp contrib/scalar/scalar.exe artifacts/contrib/scalar/ &&
+        cp bin-wrappers/scalar artifacts/bin-wrappers/
     - name: zip up tracked files
       run: git archive -o artifacts/tracked.tar.gz HEAD
     - name: upload tracked files and build artifacts
@@ -157,6 +164,8 @@ jobs:
       run: compat\vcbuild\vcpkg_copy_dlls.bat release
     - name: generate Visual Studio solution
       shell: bash
+      env:
+        INCLUDE_SCALAR: YesPlease
       run: |
         cmake `pwd`/contrib/buildsystems/ -DCMAKE_PREFIX_PATH=`pwd`/compat/vcbuild/vcpkg/installed/x64-windows \
         -DNO_GETTEXT=YesPlease -DPERL_TESTS=OFF -DPYTHON_TESTS=OFF -DCURL_NO_CURL_CMAKE=ON
@@ -170,6 +179,12 @@ jobs:
       run: |
         mkdir -p artifacts &&
         eval "$(make -n artifacts-tar INCLUDE_DLLS_IN_ARTIFACTS=YesPlease ARTIFACTS_DIRECTORY=artifacts NO_GETTEXT=YesPlease 2>&1 | grep ^tar)"
+    - name: copy Scalar
+      shell: bash
+      run: |
+        mkdir -p artifacts/bin-wrappers artifacts/contrib/scalar &&
+        cp contrib/scalar/scalar.exe artifacts/contrib/scalar/ &&
+        cp bin-wrappers/scalar artifacts/bin-wrappers/
     - name: zip up tracked files
       run: git archive -o artifacts/tracked.tar.gz HEAD
     - name: upload tracked files and build artifacts
ci/run-build-and-tests.sh +1 -0
index cc62616d806..07cedd25ff1 100755
@@ -49,6 +49,7 @@ linux-gcc-4.8|pedantic)
 	make test
 	;;
 esac
+make -C contrib/scalar test
 
 check_unignored_build_artifacts
 
ci/run-test-slice.sh +5 -0
index f8c2c3106a2..b741fd8f361 100755
@@ -14,4 +14,9 @@ make --quiet -C t T="$(cd t &&
 	./helper/test-tool path-utils slice-tests "$1" "$2" t[0-9]*.sh |
 	tr '\n' ' ')"
 
+if test 0 = "$1"
+then
+	make -C contrib/scalar test
+fi
+
 check_unignored_build_artifacts
Signature · 1 line
gitgitgadget
#197 lore raw

PATCH v7 17/17 scalar: implement the `version` command

In reply to Johannes Schindelin via GitGitGadget

From: Johannes Schindelin <[email protected]>

The .NET version of Scalar has a version command. This was necessary because it was versioned independently of Git.

Since Scalar is now tightly coupled with Git, it does not make sense for them to show different versions. Therefore, it shows the same output as git version. For backwards-compatibility with the .NET version, scalar version prints to stderr, though (git version prints to stdout instead).

---
 contrib/scalar/scalar.c | 39 +++++++++++++++++++++++++++++++++++++++
 1 file changed, 39 insertions(+)
contrib/scalar/scalar.c +39 -0
index c53697ad6a0..1fc4965bebb 100644
@@ -10,6 +10,7 @@
 #include "refs.h"
 #include "dir.h"
 #include "packfile.h"
+#include "help.h"
 
 /*
  * Remove the deepest subdirectory in the provided path string. Path must not
@@ -357,6 +358,15 @@ static int delete_enlistment(struct strbuf *enlistment)
 	return 0;
 }
 
+/*
+ * Dummy implementation; Using `get_version_info()` would cause a link error
+ * without this.
+ */
+void load_builtin_commands(const char *prefix, struct cmdnames *cmds)
+{
+	die("not implemented");
+}
+
 static int cmd_clone(int argc, const char **argv)
 {
 	const char *branch = NULL;
@@ -750,6 +760,34 @@ static int cmd_delete(int argc, const char **argv)
 	return res;
 }
 
+static int cmd_version(int argc, const char **argv)
+{
+	int verbose = 0, build_options = 0;
+	struct option options[] = {
+		OPT__VERBOSE(&verbose, N_("include Git version")),
+		OPT_BOOL(0, "build-options", &build_options,
+			 N_("include Git's build options")),
+		OPT_END(),
+	};
+	const char * const usage[] = {
+		N_("scalar verbose [-v | --verbose] [--build-options]"),
+		NULL
+	};
+	struct strbuf buf = STRBUF_INIT;
+
+	argc = parse_options(argc, argv, NULL, options,
+			     usage, 0);
+
+	if (argc != 0)
+		usage_with_options(usage, options);
+
+	get_version_info(&buf, build_options);
+	fprintf(stderr, "%s\n", buf.buf);
+	strbuf_release(&buf);
+
+	return 0;
+}
+
 static struct {
 	const char *name;
 	int (*fn)(int, const char **);
@@ -761,6 +799,7 @@ static struct {
 	{ "run", cmd_run },
 	{ "reconfigure", cmd_reconfigure },
 	{ "delete", cmd_delete },
+	{ "version", cmd_version },
 	{ NULL, NULL},
 };
 
Signature · 1 line
gitgitgadget
#193 lore raw

PATCH v7 13/17 scalar: implement the `run` command

In reply to Johannes Schindelin via GitGitGadget

From: Derrick Stolee <[email protected]>

Note: this subcommand is provided primarily for backwards-compatibility, for existing Scalar uses. It is mostly just a shim for git maintenance, mapping task names from the way Scalar called them to the way Git calls them.

The reason why those names differ? The background maintenance was first implemented in Scalar, and when it was contributed as a patch series implementing the git maintenance command, reviewers suggested better names, those suggestions were accepted before the patches were integrated into core Git.

---
 contrib/scalar/scalar.c   | 64 +++++++++++++++++++++++++++++++++++++++
 contrib/scalar/scalar.txt | 19 ++++++++++++
 2 files changed, 83 insertions(+)
contrib/scalar/scalar.c +64 -0
index 516a75be3c4..ca524576011 100644
@@ -484,6 +484,69 @@ static int cmd_register(int argc, const char **argv)
 	return register_dir();
 }
 
+static int cmd_run(int argc, const char **argv)
+{
+	struct option options[] = {
+		OPT_END(),
+	};
+	struct {
+		const char *arg, *task;
+	} tasks[] = {
+		{ "config", NULL },
+		{ "commit-graph", "commit-graph" },
+		{ "fetch", "prefetch" },
+		{ "loose-objects", "loose-objects" },
+		{ "pack-files", "incremental-repack" },
+		{ NULL, NULL }
+	};
+	struct strbuf buf = STRBUF_INIT;
+	const char *usagestr[] = { NULL, NULL };
+	int i;
+
+	strbuf_addstr(&buf, N_("scalar run <task> [<enlistment>]\nTasks:\n"));
+	for (i = 0; tasks[i].arg; i++)
+		strbuf_addf(&buf, "\t%s\n", tasks[i].arg);
+	usagestr[0] = buf.buf;
+
+	argc = parse_options(argc, argv, NULL, options,
+			     usagestr, 0);
+
+	if (!argc)
+		usage_with_options(usagestr, options);
+
+	if (!strcmp("all", argv[0])) {
+		i = -1;
+	} else {
+		for (i = 0; tasks[i].arg && strcmp(tasks[i].arg, argv[0]); i++)
+			; /* keep looking for the task */
+
+		if (i > 0 && !tasks[i].arg) {
+			error(_("no such task: '%s'"), argv[0]);
+			usage_with_options(usagestr, options);
+		}
+	}
+
+	argc--;
+	argv++;
+	setup_enlistment_directory(argc, argv, usagestr, options, NULL);
+	strbuf_release(&buf);
+
+	if (i == 0)
+		return register_dir();
+
+	if (i > 0)
+		return run_git("maintenance", "run",
+			       "--task", tasks[i].task, NULL);
+
+	if (register_dir())
+		return -1;
+	for (i = 1; tasks[i].arg; i++)
+		if (run_git("maintenance", "run",
+			    "--task", tasks[i].task, NULL))
+			return -1;
+	return 0;
+}
+
 static int remove_deleted_enlistment(struct strbuf *path)
 {
 	int res = 0;
@@ -556,6 +619,7 @@ static struct {
 	{ "list", cmd_list },
 	{ "register", cmd_register },
 	{ "unregister", cmd_unregister },
+	{ "run", cmd_run },
 	{ NULL, NULL},
 };
 
contrib/scalar/scalar.txt +19 -0
index 56f744a4aa9..39143b08324 100644
@@ -12,6 +12,7 @@ scalar clone [--single-branch] [--branch <main-branch>] [--full-clone] <url> [<e
 scalar list
 scalar register [<enlistment>]
 scalar unregister [<enlistment>]
+scalar run ( all | config | commit-graph | fetch | loose-objects | pack-files ) [<enlistment>]
 
 DESCRIPTION
 -----------
@@ -98,6 +99,24 @@ unregister [<enlistment>]::
 	Remove the specified repository from the list of repositories
 	registered with Scalar and stop the scheduled background maintenance.
 
+Run
+~~~
+
+scalar run ( all | config | commit-graph | fetch | loose-objects | pack-files ) [<enlistment>]::
+	Run the given maintenance task (or all tasks, if `all` was specified).
+	Except for `all` and `config`, this subcommand simply hands off to
+	linkgit:git-maintenance[1] (mapping `fetch` to `prefetch` and
+	`pack-files` to `incremental-repack`).
++
+These tasks are run automatically as part of the scheduled maintenance,
+as soon as the repository is registered with Scalar. It should therefore
+not be necessary to run this subcommand manually.
++
+The `config` task is specific to Scalar and configures all those
+opinionated default settings that make Git work more efficiently with
+large repositories. As this task is run as part of `scalar clone`
+automatically, explicit invocations of this task are rarely needed.
+
 SEE ALSO
 --------
 linkgit:git-clone[1], linkgit:git-maintenance[1].
Signature · 1 line
gitgitgadget
#195 lore raw

PATCH v7 15/17 scalar: teach 'reconfigure' to optionally handle all registered enlistments

In reply to Johannes Schindelin via GitGitGadget

From: Johannes Schindelin <[email protected]>

After a Scalar upgrade, it can come in really handy if there is an easy way to reconfigure all Scalar enlistments. This new option offers this functionality.

---
 contrib/scalar/scalar.c          | 61 ++++++++++++++++++++++++++++++--
 contrib/scalar/scalar.txt        |  9 +++--
 contrib/scalar/t/t9099-scalar.sh |  3 ++
 3 files changed, 67 insertions(+), 6 deletions(-)
contrib/scalar/scalar.c +58 -3
index b799decbc2f..71ca573f3af 100644
@@ -488,22 +488,77 @@ static int cmd_register(int argc, const char **argv)
 	return register_dir();
 }
 
+static int get_scalar_repos(const char *key, const char *value, void *data)
+{
+	struct string_list *list = data;
+
+	if (!strcmp(key, "scalar.repo"))
+		string_list_append(list, value);
+
+	return 0;
+}
+
 static int cmd_reconfigure(int argc, const char **argv)
 {
+	int all = 0;
 	struct option options[] = {
+		OPT_BOOL('a', "all", &all,
+			 N_("reconfigure all registered enlistments")),
 		OPT_END(),
 	};
 	const char * const usage[] = {
-		N_("scalar reconfigure [<enlistment>]"),
+		N_("scalar reconfigure [--all | <enlistment>]"),
 		NULL
 	};
+	struct string_list scalar_repos = STRING_LIST_INIT_DUP;
+	int i, res = 0;
+	struct repository r = { NULL };
+	struct strbuf commondir = STRBUF_INIT, gitdir = STRBUF_INIT;
 
 	argc = parse_options(argc, argv, NULL, options,
 			     usage, 0);
 
-	setup_enlistment_directory(argc, argv, usage, options, NULL);
+	if (!all) {
+		setup_enlistment_directory(argc, argv, usage, options, NULL);
+
+		return set_recommended_config(1);
+	}
+
+	if (argc > 0)
+		usage_msg_opt(_("--all or <enlistment>, but not both"),
+			      usage, options);
+
+	git_config(get_scalar_repos, &scalar_repos);
 
-	return set_recommended_config(1);
+	for (i = 0; i < scalar_repos.nr; i++) {
+		const char *dir = scalar_repos.items[i].string;
+
+		strbuf_reset(&commondir);
+		strbuf_reset(&gitdir);
+
+		if (chdir(dir) < 0) {
+			warning_errno(_("could not switch to '%s'"), dir);
+			res = -1;
+		} else if (discover_git_directory(&commondir, &gitdir) < 0) {
+			warning_errno(_("git repository gone in '%s'"), dir);
+			res = -1;
+		} else {
+			git_config_clear();
+
+			the_repository = &r;
+			r.commondir = commondir.buf;
+			r.gitdir = gitdir.buf;
+
+			if (set_recommended_config(1) < 0)
+				res = -1;
+		}
+	}
+
+	string_list_clear(&scalar_repos, 1);
+	strbuf_release(&commondir);
+	strbuf_release(&gitdir);
+
+	return res;
 }
 
 static int cmd_run(int argc, const char **argv)
contrib/scalar/scalar.txt +6 -3
index 89fd7901585..737cf563c1a 100644
@@ -13,7 +13,7 @@ scalar list
 scalar register [<enlistment>]
 scalar unregister [<enlistment>]
 scalar run ( all | config | commit-graph | fetch | loose-objects | pack-files ) [<enlistment>]
-scalar reconfigure <enlistment>
+scalar reconfigure [ --all | <enlistment> ]
 
 DESCRIPTION
 -----------
@@ -32,8 +32,8 @@ an existing Git worktree with Scalar whose name is not `src`, the enlistment
 will be identical to the worktree.
 
 The `scalar` command implements various subcommands, and different options
-depending on the subcommand. With the exception of `clone` and `list`, all
-subcommands expect to be run in an enlistment.
+depending on the subcommand. With the exception of `clone`, `list` and
+`reconfigure --all`, all subcommands expect to be run in an enlistment.
 
 COMMANDS
 --------
@@ -125,6 +125,9 @@ After a Scalar upgrade, or when the configuration of a Scalar enlistment
 was somehow corrupted or changed by mistake, this subcommand allows to
 reconfigure the enlistment.
 
+With the `--all` option, all enlistments currently registered with Scalar
+will be reconfigured. Use this option after each Scalar upgrade.
+
 SEE ALSO
 --------
 linkgit:git-clone[1], linkgit:git-maintenance[1].
contrib/scalar/t/t9099-scalar.sh +3 -0
index fb5e2efee0a..58af546fd84 100755
@@ -70,6 +70,9 @@ test_expect_success 'scalar reconfigure' '
 	scalar register one &&
 	git -C one/src config core.preloadIndex false &&
 	scalar reconfigure one &&
+	test true = "$(git -C one/src config core.preloadIndex)" &&
+	git -C one/src config core.preloadIndex false &&
+	scalar reconfigure -a &&
 	test true = "$(git -C one/src config core.preloadIndex)"
 '
 
Signature · 1 line
gitgitgadget
#178 lore raw

Re: [PATCH v7 01/17] scalar: add a README with a roadmap

In reply to Johannes Schindelin via GitGitGadget
Quoted 18 lines · Johannes Schindelin via GitGitGadget

From: Johannes Schindelin <[email protected]>

The Scalar command will be contributed incrementally, over a bunch of patch series. Let's document what Scalar is about, and then describe the patch series that are planned.

---
 contrib/scalar/README.md | 71 ++++++++++++++++++++++++++++++++++++++++
 1 file changed, 71 insertions(+)
 create mode 100644 contrib/scalar/README.md
contrib/scalar/README.md new file +0 -0
new file mode 100644
index 00000000000..7898a683ba5

contrib/scalar/README.md line 4

Quoted 4 lines
contrib/scalar/README.md new file +4 -0
@@ -0,0 +1,71 @@
+# Scalar - an opinionated repository management tool
+
+Scalar is an add-on to Git, helping Git scale to very large repositories and
+worktrees.

I would rephrase this as "Scalar is an add-on to Git that helps users take advantage of advanced performance features in Git."

Git scales just fine, only it helps to enable some features that are off by default.

Quoted 23 lines

Originally implemented in C# using .NET Core, based on the learnings

contrib/scalar/README.md new file +22 -0
+from the VFS for Git project, most of the techniques developed by the Scalar
+project have been integrated into core Git already:
+
+* partial clone,
+* commit graphs,
+* multi-pack index,
+* sparse checkout (cone mode),
+* scheduled background maintenance,
+* etc
+
+This directory contains the remaining parts of Scalar that are not (yet) in
+core Git.
+
+## Roadmap
+
+The idea is to populate this directory via incremental patch series and
+eventually move to a top-level directory next to `gitk-git/` and to `git-gui/`. The
+current plan involves the following patch series:
+
+- `scalar-the-beginning`: The initial patch series which sets up
+  `contrib/scalar/` and populates it with a minimal `scalar` command that
+  demonstrates the fundamental ideas.

contrib/scalar/README.md line 36

Quoted 10 lines
contrib/scalar/README.md new file +10 -0
+
+- `scalar-c-and-C`: The `scalar` command learns about two options that can be
+  specified before the command, `-c <key>=<value>` and `-C <directory>`.
+
+- `scalar-diagnose`: The `scalar` command is taught the `diagnose` subcommand.
+
+- `scalar-and-builtin-fsmonitor`: The built-in FSMonitor is enabled in `scalar
+  init` and in `scalar clone`, for an enormous performance boost when working
+  in large worktrees. This patch series necessarily depends on Jeff Hostetler's
+  FSMonitor patch series to be integrated into Git.

You say 'scalar init' but do you mean 'scalar register'?

Quoted 5 lines

+- scalar-gentler-config-locking: Scalar enlistments are registered in the

contrib/scalar/README.md new file +4 -0
+  user's Git config. This usually does not represent any problem because it is
+  rare for a user to register an enlistment. However, in Scalar's functional
+  tests, Scalar enlistments are created galore, and in parallel, which can lead
+  to lock contention. This patch series works around that problem by re-trying

contrib/scalar/README.md line 52

Quoted 10 lines
contrib/scalar/README.md new file +10 -0
+  to lock the config file in a gentle fashion.
+
+- `scalar-extra-docs`: Add some extensive documentation that has been written
+  in the original Scalar project (all subject to discussion, of course).
+
+- `optionally-install-scalar`: Now that Scalar is feature (and documentation)
+  complete and is verified in CI builds, let's offer to install it.
+
+- `move-scalar-to-toplevel`: Now that Scalar is complete, let's move it next to
+  `gitk-git/` and to `git-gui/`, making it a top-level command.

This final one is where we can make the final call about where Scalar should exist in the tree and how optional it should be. This would also move the Scalar man pages into Documentation/, along with possibly the docs from 'scalar-extra-docs', and the tests into t/. The benefit of leaving this until the end is that we can see the entirety of Scalar before making a final call.

Quoted 8 lines
contrib/scalar/README.md new file +8 -0
+The following two patch series exist, but there is no plan to integrate them
+into Git's source tree:
+
+- `scalar-with-gvfs`: The primary purpose of this patch series is to support
+  existing Scalar users whose repositories are hosted in Azure Repos (which
+  does not support Git's partial clones, but supports its predecessor, the GVFS
+  protocol, which is used by Scalar to emulate the partial clone).
+

contrib/scalar/README.md line 71

Quoted 10 lines
contrib/scalar/README.md new file +10 -0
+  Since the GVFS protocol will never be supported by core Git, this patch
+  series will remain in Microsoft's fork of Git.
+
+- `run-scalar-functional-tests`: The Scalar project developed a quite
+  comprehensive set of integration tests (or, "Functional Tests"). They are the
+  sole remaining part of the original C#-based Scalar project, and this patch
+  adds a GitHub workflow that runs them all.
+
+  Since the tests partially depend on features that are only provided in the
+  `scalar-with-gvfs` patch series, this patch cannot be upstreamed.

These topics (in some form or another) exist on microsoft/git and are available via GPL, so we don't intend to say "we are withholding these patches" but instead are saying "We don't think the Git community is interested in these patches." There are some interesting ideas there, but the implementation is too specific to Azure Repos to be of much help in general. These still exist mainly because the GVFS protocol is what Azure Repos has instead of partial clone. We are focused instead on improving partial clone.

Thanks,
-Stolee

#184 lore raw

Re: [PATCH v7 05/17] cmake: optionally build `scalar`, too

In reply to Johannes Schindelin via GitGitGadget
Quoted 6 lines · Johannes Schindelin via GitGitGadget

From: Johannes Schindelin <[email protected]>

The CMake configuration unfortunately does not let us encapsulate all (or at least the vast majority) of Scalar's build definition in the contrib/scalar/ subdirectory.

I believe that this isn't fully correct in that you could call add_subdirectory() with
a directory that isn't a subdirectory so long as you provide it an explicit binary directory
which is part of what we're doing here anyways. (at least it works fine for me with
cmake version 3.21)

Quoted 14 lines

To alleviate that somewhat, we guard the inclusion of Scalar via the INCLUDE_SCALAR environment variable.

---
 contrib/buildsystems/CMakeLists.txt | 14 ++++++++++++++
 1 file changed, 14 insertions(+)
contrib/buildsystems/CMakeLists.txt +0 -0
index fd1399c440f..dd7496b0322 100644

contrib/buildsystems/CMakeLists.txt if(CURL_FOUND) · line 737

Quoted 9 lines
contrib/buildsystems/CMakeLists.txt +6 -0
@@ -729,6 +729,13 @@ if(CURL_FOUND)
        endif()
 endif()
 
+if(DEFINED ENV{INCLUDE_SCALAR} AND NOT ENV{INCLUDE_SCALAR} STREQUAL "")
+       add_executable(scalar ${CMAKE_SOURCE_DIR}/contrib/scalar/scalar.c)
+       target_link_libraries(scalar common-main)
+       set_target_properties(scalar PROPERTIES RUNTIME_OUTPUT_DIRECTORY_DEBUG ${CMAKE_BINARY_DIR}/contrib/scalar)
+       set_target_properties(scalar PROPERTIES RUNTIME_OUTPUT_DIRECTORY_RELEASE ${CMAKE_BINARY_DIR}/contrib/scalar)
+endif()

Would it make sense to just mark this with EXCLUDE_FROM_ALL so that way you don't have to have
the INCLUDE_SCALAR environment variable just for this? This way people who don't want
to build scalar can still run cmake --build . and not have scalar built and people who do want
it built can just run cmake --build . --target scalar, alternatively you could also do something like

if(DEFINED ENV{INCLUDE_SCALAR} AND NOT ENV{INCLUDE_SCALAR} STREQUAL "")

    add_executable(scalar ${CMAKE_SOURCE_DIR}/contrib/scalar/scalar.c)

else()
add_executable(scalar EXCLUDE_FROM_ALL
${CMAKE_SOURCE_DIR}/contrib/scalar/scalar.c)
endif()

Just some food for thought,

Matthew Rogers

#185 lore raw

Re: [PATCH v7 05/17] cmake: optionally build `scalar`, too

Hi Matt,

Quoted 9 lines · Matt Rogers
Quoted 6 lines · Johannes Schindelin via GitGitGadget

From: Johannes Schindelin <[email protected]>

The CMake configuration unfortunately does not let us encapsulate all (or at least the vast majority) of Scalar's build definition in the contrib/scalar/ subdirectory.

Quoted 5 lines

I believe that this isn't fully correct in that you could call add_subdirectory() with a directory that isn't a subdirectory so long as you provide it an explicit binary directory which is part of what we're doing here anyways. (at least it works fine for me with cmake version 3.21)

You are of course correct, you _could_ put things into a subdirectory with CMake. However, please note that scalar wants to link to libgit.a (and also link in common-main.o), therefore the subdirectory _still_ has to have some sort of connection to the top-level configuration.

I shall clarify that in the commit message in the next iteration.

Quoted 30 lines
Quoted 13 lines

To alleviate that somewhat, we guard the inclusion of Scalar via the INCLUDE_SCALAR environment variable.

---
 contrib/buildsystems/CMakeLists.txt | 14 ++++++++++++++
 1 file changed, 14 insertions(+)
contrib/buildsystems/CMakeLists.txt +0 -0
index fd1399c440f..dd7496b0322 100644

contrib/buildsystems/CMakeLists.txt if(CURL_FOUND) · line 737

Quoted 9 lines
contrib/buildsystems/CMakeLists.txt +6 -0
@@ -729,6 +729,13 @@ if(CURL_FOUND)
        endif()
 endif()
 
+if(DEFINED ENV{INCLUDE_SCALAR} AND NOT ENV{INCLUDE_SCALAR} STREQUAL "")
+       add_executable(scalar ${CMAKE_SOURCE_DIR}/contrib/scalar/scalar.c)
+       target_link_libraries(scalar common-main)
+       set_target_properties(scalar PROPERTIES RUNTIME_OUTPUT_DIRECTORY_DEBUG ${CMAKE_BINARY_DIR}/contrib/scalar)
+       set_target_properties(scalar PROPERTIES RUNTIME_OUTPUT_DIRECTORY_RELEASE ${CMAKE_BINARY_DIR}/contrib/scalar)
+endif()

Would it make sense to just mark this with EXCLUDE_FROM_ALL so that way you don't have to have the INCLUDE_SCALAR environment variable just for this? This way people who don't want to build scalar can still run cmake --build . and not have scalar built and people who do want it built can just run cmake --build . --target scalar, alternatively you could also do something like

Quoted 6 lines

if(DEFINED ENV{INCLUDE_SCALAR} AND NOT ENV{INCLUDE_SCALAR} STREQUAL "")

    add_executable(scalar ${CMAKE_SOURCE_DIR}/contrib/scalar/scalar.c)

else()
add_executable(scalar EXCLUDE_FROM_ALL
${CMAKE_SOURCE_DIR}/contrib/scalar/scalar.c)
endif()

The entire point of pulling this patch from a much later patch series to the first Scalar patch series is to allow for easy testing in the CI build. To that end, what you propose would complexify the setup too much for my liking.

Since the end game seems to be for Scalar to become a top-level command (at least that's how I read Junio's favorable reply at
https://lore.kernel.org/git/[email protected]/), I suspect that it might not matter whether we stay with INCLUDE_SCALAR or change it to an idempotent alternative, anyway.

Ciao,
Dscho

#179 lore raw

Re: [PATCH v7 01/17] scalar: add a README with a roadmap

In reply to Derrick Stolee

Hi Stolee,

Quoted 13 lines · Derrick Stolee
Quoted 12 lines · Johannes Schindelin via GitGitGadget

From: Johannes Schindelin <[email protected]>

The Scalar command will be contributed incrementally, over a bunch of patch series. Let's document what Scalar is about, and then describe the patch series that are planned.

---
 contrib/scalar/README.md | 71 ++++++++++++++++++++++++++++++++++++++++
 1 file changed, 71 insertions(+)
 create mode 100644 contrib/scalar/README.md
Quoted 13 lines

contrib/scalar/README.md line 4

Quoted 10 lines
contrib/scalar/README.md new file +4 -0
new file mode 100644
index 00000000000..7898a683ba5
@@ -0,0 +1,71 @@
+# Scalar - an opinionated repository management tool
+
+Scalar is an add-on to Git, helping Git scale to very large repositories and
+worktrees.

I would rephrase this as "Scalar is an add-on to Git that helps users take advantage of advanced performance features in Git."

Good idea!

Quoted 30 lines

Git scales just fine, only it helps to enable some features that are off by default.

Quoted 27 lines

Originally implemented in C# using .NET Core, based on the learnings

contrib/scalar/README.md new file +26 -0
+from the VFS for Git project, most of the techniques developed by the Scalar
+project have been integrated into core Git already:
+
+* partial clone,
+* commit graphs,
+* multi-pack index,
+* sparse checkout (cone mode),
+* scheduled background maintenance,
+* etc
+
+This directory contains the remaining parts of Scalar that are not (yet) in
+core Git.
+
+## Roadmap
+
+The idea is to populate this directory via incremental patch series and
+eventually move to a top-level directory next to `gitk-git/` and to `git-gui/`. The
+current plan involves the following patch series:
+
+- `scalar-the-beginning`: The initial patch series which sets up
+  `contrib/scalar/` and populates it with a minimal `scalar` command that
+  demonstrates the fundamental ideas.
+
+- `scalar-c-and-C`: The `scalar` command learns about two options that can be
+  specified before the command, `-c <key>=<value>` and `-C <directory>`.
+
Quoted 8 lines

contrib/scalar/README.md line 36

Quoted 6 lines

+- scalar-diagnose: The scalar command is taught the diagnose subcommand.

contrib/scalar/README.md new file +5 -0
+
+- `scalar-and-builtin-fsmonitor`: The built-in FSMonitor is enabled in `scalar
+  init` and in `scalar clone`, for an enormous performance boost when working
+  in large worktrees. This patch series necessarily depends on Jeff Hostetler's
+  FSMonitor patch series to be integrated into Git.

You say 'scalar init' but do you mean 'scalar register'?

D'oh. Yes, I meant scalar register. I was thinking of git init too much when I wrote that.

Quoted 13 lines
Quoted 13 lines

+- scalar-gentler-config-locking: Scalar enlistments are registered in the

contrib/scalar/README.md new file +12 -0
+  user's Git config. This usually does not represent any problem because it is
+  rare for a user to register an enlistment. However, in Scalar's functional
+  tests, Scalar enlistments are created galore, and in parallel, which can lead
+  to lock contention. This patch series works around that problem by re-trying
+  to lock the config file in a gentle fashion.
+
+- `scalar-extra-docs`: Add some extensive documentation that has been written
+  in the original Scalar project (all subject to discussion, of course).
+
+- `optionally-install-scalar`: Now that Scalar is feature (and documentation)
+  complete and is verified in CI builds, let's offer to install it.
+
Quoted 8 lines
Quoted 2 lines
+- `move-scalar-to-toplevel`: Now that Scalar is complete, let's move it next to
+  `gitk-git/` and to `git-gui/`, making it a top-level command.

This final one is where we can make the final call about where Scalar should exist in the tree and how optional it should be. This would also move the Scalar man pages into Documentation/, along with possibly the docs from 'scalar-extra-docs', and the tests into t/. The benefit of leaving this until the end is that we can see the entirety of Scalar before making a final call.

Yes.

It allows the current patch series to focus on the core functionality of Scalar. The move-scalar-to-toplevel patch series will present an excellent opportunity to discuss the merits and complications of accepting Scalar as a top-level command into Git, without having to delay the current patch series any further.

Quoted 19 lines
Quoted 8 lines
contrib/scalar/README.md new file +8 -0
+The following two patch series exist, but there is no plan to integrate them
+into Git's source tree:
+
+- `scalar-with-gvfs`: The primary purpose of this patch series is to support
+  existing Scalar users whose repositories are hosted in Azure Repos (which
+  does not support Git's partial clones, but supports its predecessor, the GVFS
+  protocol, which is used by Scalar to emulate the partial clone).
+

contrib/scalar/README.md line 71

Quoted 10 lines
contrib/scalar/README.md new file +10 -0
+  Since the GVFS protocol will never be supported by core Git, this patch
+  series will remain in Microsoft's fork of Git.
+
+- `run-scalar-functional-tests`: The Scalar project developed a quite
+  comprehensive set of integration tests (or, "Functional Tests"). They are the
+  sole remaining part of the original C#-based Scalar project, and this patch
+  adds a GitHub workflow that runs them all.
+
+  Since the tests partially depend on features that are only provided in the
+  `scalar-with-gvfs` patch series, this patch cannot be upstreamed.
Quoted 7 lines

These topics (in some form or another) exist on microsoft/git and are available via GPL, so we don't intend to say "we are withholding these patches" but instead are saying "We don't think the Git community is interested in these patches." There are some interesting ideas there, but the implementation is too specific to Azure Repos to be of much help in general. These still exist mainly because the GVFS protocol is what Azure Repos has instead of partial clone. We are focused instead on improving partial clone.

Good point. What I wrote does not fully reflect that. I have changed it.

Do you see anything in the remainder of the patch series that still needs to be improved?

Thanks,
Dscho

#198 lore raw
In reply to Johannes Schindelin via GitGitGadget
Quoted 8 lines · Johannes Schindelin via GitGitGadget

* To avoid mistaking the current patch series for being feature-complete enough to unleash onto end users, I moved the Makefile rules to build HTML/manual pages to a later patch series.
[...]
* I added two patches that I had planned on keeping in an add-on patch series for later, to build and test Scalar as part of the CI. I am still not 100% certain that it is a good idea to do so already now, but let's see what the reviewers have to say.

I don't expect given [1] that I'll get a reply to this, but as noted there I've had outstanding reviews of this series (including proposed patches) for weeks/months now that haven't been addressed, including the patch-on-top I suggested in the previous round of discussion [2].

Looking at the new README.md it seems that the change of plan between v6..v7 is that after insisting that this command must stay decoupled in contrib/scalar/ and have its own build/test etc. infrastructure, the end-goal now eventual full integration at the top-level.

Peeking a branch of yours[3] there's a change[4] in it that's much like my [2], some of the hunks are even the same.

I'm quite happy with that end-state, but still don't understand why the "scalar-extra-docs" and "run-scalar-functional-tests" (in your README.md here) couldn't be squashed into this first step, as suggested in my [2].

The end-state is the same, but doing it that way avoids all this path-based churn, setting up build systems that are then going away in the short-to-medium term etc, as well as the new custom just-for-scalar CI integration in this v7, you'd get that and more for free with my [2].

All of that is apparently to:

Quoted 2 lines

[...]avoid mistaking the current patch series for being feature-complete enough to unleash onto end users[. ...]

Which is fair enough, but doesn't explain why this back-and-forth churn is needed to reach an end-goal we apparently now almost entirely agree on.

I.e. in my [2] it's built & tested by default, but not installed. So end users aren't seeing it. The small bit of proposed in-advanced reference to it in our documentation is there in [2], but that's trivially tweaked, we could defer that too. No end-user would see any of it.

So who's being helped by these intermediate steps?

As far as I can tell based on your previous comments the perceived necessity these intermediate steps comes from a narrow reading of contrib/README,
i.e. that software in this sort of flux *MUST* live in contrib, even if we've got plans & patches in-progress to move it out eventually.

Whereas I think we can just document that it's in flux, and skip the intermediate churn to get to the agreed-upon end-state.

As before I'm happy to engage with you, but I must say at this point I'm not getting my hopes up much.

Cheers.

1. https://lore.kernel.org/git/[email protected]/
2. https://lore.kernel.org/git/[email protected]/
3. https://github.com/dscho/git/tree/vfs-with-scalar
4. https://github.com/dscho/git/commit/f9b8e5d5e7e

#199 lore raw

v8 cover Upstreaming the Scalar command

In reply to Johannes Schindelin via GitGitGadget

tl;dr: This series contributes the core part of the Scalar command to the Git project. This command provides an opinionated way to create and configure Git repositories with a focus on very large repositories.

Changes since v7:

* Clarified in the commit message why we cannot easily encapsulate the Scalar part of the CMake configuration in contrib/scalar/.
* Improved the README.md.

Changes since v6:

* Rebased on top of v2.34.0.
* Inserted a commit that adds contrib/scalar/README.md, containing the roadmap of what I have planned for Scalar.
* The Scalar test's definition of GIT_TEST_MAINT_SCHEDULER has been adjusted to accommodate for a change in v2.32.0..v2.34.0.
* The config setting defaults now include fetch.showForcedUpdates=false, which has been identified as helping with a performance issue in large repositories.
* To avoid mistaking the current patch series for being feature-complete enough to unleash onto end users, I moved the Makefile rules to build HTML/manual pages to a later patch series.
* The patch that adds support for -c <key>=<value> and -C <directory> was moved to its own add-on patch series: While it is obvious that those options are valuable to have, an open question is whether there are other "pre-command" options in git that would be useful, too, and I would like to postpone that discussion to that date.
* I added two patches that I had planned on keeping in an add-on patch series for later, to build and test Scalar as part of the CI. I am still not 100% certain that it is a good idea to do so already now, but let's see what the reviewers have to say.

Changes since v5:

* Fixed the commit message talking about make -C contrib/scalar/Makefile.
* Fixed the git ls-tree invocation suggested in the manual for scalar clone.
* Invoking make -C contrib/scalar, then changing a source file of libgit.a and then immediately invoking make -C contrib/scalar again will now implicitly rebuild libgit.a.

Changes since v4:

* scalar delete now refuses to delete anything if it was started from within the enlistment.
* scalar delete releases any handles to the object store before deleting the enlistment.
* The OBJECTS list in the Makefile will now include Scalar.
* scalar register now supports secondary worktrees, in addition to the primary worktree.

Changes since v3:

* Moved the "Changes since" section to the top, to make it easier to see what changed.
* Reworded the commit message of the first patch.
* Removed the [RFC] prefix because I did not hear any objections against putting this into contrib/.

Changes since v2:

* Adjusted the description of the list command in the manual page , as suggested by Bagas.
* Addressed two style nits in cmd_run().
* The documentation of git reconfigure -a was improved.

Changes since v1:

* A couple typos were fixed
* The code parsing the output of ls-remote was made more readable
* The indentation used in scalar.txt now consistently uses tabs
* We no longer hard-code core.bare = false when registering with Scalar

Background
==========

Microsoft invested a lot of effort into scaling Git to the needs of the Windows operating system source code. Based on the experience of the first approach, VFS for Git, the Scalar project was started. Scalar specifically has as its core goal to funnel all improvements into core Git.

The present
===========

The Scalar project provides a completely functional non-virtual experience for monorepos. But why stop there. The Scalar project was designed to be a self-destructing vehicle to allow those key concepts to be moved into core Git itself for the benefit of all. For example, partial clone, sparse-checkout, and scheduled background maintenance have already been upstreamed and removed from Scalar proper. This patch series provides a C-based implementation of the final remaining portions of the Scalar command. This will make it easier for users to experiment with the Scalar command. It will also make it substantially easier to experiment with moving functionality from Scalar into core Git, while maintaining backwards-compatibility for existing Scalar users.

The C-based Scalar has been shipped to Scalar users, and can be tested by any interested reader: https://github.com/microsoft/git/releases/ (it offers a Git for Windows installer, a macOS package and an Ubuntu package, Scalar has been included since v2.33.0.vfs.0.0).

Next steps
==========

Since there are existing Scalar users, I want to ensure backwards-compatibility with its existing command-line interface. Keeping that in mind, everything in this series is up for discussion.

I obviously believe that Scalar brings a huge benefit, and think that it would be ideal for all of Scalar's learnings to end up in git clone/git init/git maintenance eventually. It is also conceivable, however, that the scalar command could graduate to be a core part of Git at some stage in the future (such a decision would probably depend highly on users' feedback). See also the discussion about the architecture of Scalar
[https://lore.kernel.org/git/[email protected]/],
kicked off by Stolee.

On top of this patch series, I have lined up a few more:

1. Implement a scalar diagnose command.
2. Use the built-in FSMonitor (that patch series obviously needs to wait for FSMonitor to be integrated).
3. Modify the config machinery to be more generous about concurrent writes, say, to the user-wide config.
4. A few patches to optionally build and install scalar as part of a regular Git install (also teaching git help scalar to find the Scalar documentation

These are included in my vfs-with-scalar branch thicket
[https://github.com/dscho/git/commits/vfs-with-scalar]. On top of that, this branch thicket also includes patches I do not plan on upstreaming, mainly because they are too specific either to VFS for Git, or they support Azure Repos (which does not offer partial clones but speaks the GVFS protocol, which can be used to emulate partial clones).

One other thing is very interesting about that vfs-with-scalar branch
thicket: it contains a GitHub workflow which will run Scalar's quite extensive Functional Tests suite. This test suite is quite comprehensive and caught us a lot of bugs in the past, not only in the Scalar code, but also core Git.

Epilogue
========

Now, to address some questions that I imagine every reader has who made it this far:

* Why not put the Scalar functionality directly into core Git, even a built-in? I wanted to provide an easy way for Git contributors to "play with" Scalar, without forcing a new top-level command into Git.
* Why implement the Scalar command in the Git code base? Apart from simplifying Scalar maintenance in the Microsoft port of Git, the tight version coupling between Git and Scalar reduces the maintenance burden even further. Besides, I believe that it will make it much easier to shift functionality from Scalar into core Git, once we took the hurdle of accepting the Scalar code into the code base.
* Why contribute Scalar to the Git project? We are biased, of course, yet our data-driven approach provides evidence that Scalar helps handling huge repositories with ease. By contributing it to the core Git project, we are able to share it with more users, especially some users who do not want to install Microsoft's fork of Git. We also hope that a lot of Scalar (maybe all of it) will end up in core Git, to benefit even more users.

Derrick Stolee (4):
scalar: 'register' sets recommended config and starts maintenance
scalar: 'unregister' stops background maintenance
scalar: implement 'scalar list'
scalar: implement the run command

Johannes Schindelin (12):
scalar: add a README with a roadmap
scalar: create a rudimentary executable
scalar: start documenting the command
scalar: create test infrastructure
cmake: optionally build scalar, too
ci: also run the scalar tests
scalar: let 'unregister' handle a deleted enlistment directory
gracefully
scalar: implement the clone subcommand
scalar: teach 'clone' to support the --single-branch option
scalar: allow reconfiguring an existing enlistment
scalar: teach 'reconfigure' to optionally handle all registered
enlistments
scalar: implement the version command

Matthew John Cheetham (1):
scalar: implement the delete command

 .github/workflows/main.yml          |  15 +
 Makefile                            |   9 +
 ci/run-build-and-tests.sh           |   1 +
 ci/run-test-slice.sh                |   5 +
 contrib/buildsystems/CMakeLists.txt |  14 +
 contrib/scalar/.gitignore           |   2 +
 contrib/scalar/Makefile             |  45 ++
 contrib/scalar/README.md            |  82 +++
 contrib/scalar/scalar.c             | 826 ++++++++++++++++++++++++++++
 contrib/scalar/scalar.txt           | 145 +++++
 contrib/scalar/t/Makefile           |  78 +++
 contrib/scalar/t/t9099-scalar.sh    |  88 +++
 12 files changed, 1310 insertions(+)
 create mode 100644 contrib/scalar/.gitignore
 create mode 100644 contrib/scalar/Makefile
 create mode 100644 contrib/scalar/README.md
 create mode 100644 contrib/scalar/scalar.c
 create mode 100644 contrib/scalar/scalar.txt
 create mode 100644 contrib/scalar/t/Makefile
 create mode 100755 contrib/scalar/t/t9099-scalar.sh

base-commit: cd3e606211bb1cf8bc57f7d76bab98cc17a150bc
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-1005%2Fdscho%2Fscalar-the-beginning-v8
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1005/dscho/scalar-the-beginning-v8
Pull-Request: https://github.com/gitgitgadget/git/pull/1005

Range-diff vs v7:

1: 3aa095dc824 ! 1: 3a2e28275f1 scalar: add a README with a roadmap
@@ contrib/scalar/README.md (new)
@@
+# Scalar - an opinionated repository management tool
+
-+Scalar is an add-on to Git, helping Git scale to very large repositories and -+worktrees. Originally implemented in C# using .NET Core, based on the learnings -+from the VFS for Git project, most of the techniques developed by the Scalar -+project have been integrated into core Git already:
++Scalar is an add-on to Git that helps users take advantage of advanced ++performance features in Git. Originally implemented in C# using .NET Core, ++based on the learnings from the VFS for Git project, most of the techniques ++developed by the Scalar project have been integrated into core Git already:
+
+* partial clone,
+* commit graphs,
@@ contrib/scalar/README.md (new)
+- scalar-diagnose: The scalar command is taught the diagnose subcommand. +
+- scalar-and-builtin-fsmonitor: The built-in FSMonitor is enabled in `scalar
-+ init and in scalar clone`, for an enormous performance boost when working -+ in large worktrees. This patch series necessarily depends on Jeff Hostetler's -+ FSMonitor patch series to be integrated into Git.
++ register and in scalar clone`, for an enormous performance boost when ++ working in large worktrees. This patch series necessarily depends on Jeff ++ Hostetler's FSMonitor patch series to be integrated into Git.
+
+- scalar-gentler-config-locking: Scalar enlistments are registered in the
+ user's Git config. This usually does not represent any problem because it is
@@ contrib/scalar/README.md (new)
+- move-scalar-to-toplevel: Now that Scalar is complete, let's move it next to
+ gitk-git/ and to git-gui/, making it a top-level command.
+
-+The following two patch series exist, but there is no plan to integrate them -+into Git's source tree:
++The following two patch series exist in Microsoft's fork of Git and are ++publicly available. There is no current plan to upstream them, not because I ++want to withhold these patches, but because I don't think the Git community is ++interested in these patches.
++
++There are some interesting ideas there, but the implementation is too specific ++to Azure Repos and/or VFS for Git to be of much help in general (and also: my ++colleagues tried to upstream some patches already and the enthusiasm for ++integrating things related to Azure Repos and VFS for Git can be summarized in ++very, very few words).
++
++These still exist mainly because the GVFS protocol is what Azure Repos has ++instead of partial clone, while Git is focused on improving partial clone:
+
+- scalar-with-gvfs: The primary purpose of this patch series is to support
+ existing Scalar users whose repositories are hosted in Azure Repos (which

  2:  e0693cc713c =  2:  50160d61a41 scalar: create a rudimentary executable
  3:  d80627615f8 =  3:  74cd6410931 scalar: start documenting the command
  4:  9da1616849e =  4:  37231a4dd07 scalar: create test infrastructure
  5:  dbaad4753c1 !  5:  a39b9c81214 cmake: optionally build `scalar`, too

@@ Metadata
## Commit message ##
cmake: optionally build scalar, too

- The CMake configuration unfortunately does not let us encapsulate all
- (or at least the vast majority) of Scalar's build definition in the
- contrib/scalar/ subdirectory.
+ The CMake configuration unfortunately does not let us easily encapsulate
+ Scalar's build definition in the contrib/scalar/ subdirectory: The
+ scalar executable needs to link in libgit.a and common-main.o, for
+ example.
+
+ Also, scalar.c includes Git's header files, which means that
+ scalar.c needs to be compiled with the very same flags as libgit.a
+ lest scalar.o and libgit.a have different ideas of, say,
+ platform_SHA_CTX, which would naturally lead to memory corruption.

To alleviate that somewhat, we guard the inclusion of Scalar via the INCLUDE_SCALAR environment variable.

  6:  1b0328fa236 =  6:  8e3542e43f7 ci: also run the `scalar` tests
  7:  cca604ef326 =  7:  385abdb8d8e scalar: 'register' sets recommended config and starts maintenance
  8:  9fea89cd161 =  8:  64c6a75353e scalar: 'unregister' stops background maintenance
  9:  5e077bf892b =  9:  f7fc1958b9e scalar: let 'unregister' handle a deleted enlistment directory gracefully
 10:  dfa0c470989 = 10:  fd2680bc945 scalar: implement 'scalar list'
 11:  febefe39886 = 11:  4966a43aad9 scalar: implement the `clone` subcommand
 12:  2677bcff335 = 12:  b00d68b37b0 scalar: teach 'clone' to support the --single-branch option
 13:  99affb84284 = 13:  771d826bbb1 scalar: implement the `run` command
 14:  69e2242240b = 14:  a8b2d26a830 scalar: allow reconfiguring an existing enlistment
 15:  0068c18aa62 = 15:  ca284ff34a2 scalar: teach 'reconfigure' to optionally handle all registered enlistments
 16:  d5218523a38 = 16:  9983eb8912c scalar: implement the `delete` command
 17:  96a803416b5 = 17:  889f613ab18 scalar: implement the `version` command
Signature · 1 line
gitgitgadget
#200 lore raw

PATCH v8 1/17 scalar: add a README with a roadmap

From: Johannes Schindelin <[email protected]>

The Scalar command will be contributed incrementally, over a bunch of patch series. Let's document what Scalar is about, and then describe the patch series that are planned.

---
 contrib/scalar/README.md | 82 ++++++++++++++++++++++++++++++++++++++++
 1 file changed, 82 insertions(+)
 create mode 100644 contrib/scalar/README.md
contrib/scalar/README.md new file +82 -0
new file mode 100644
index 00000000000..634b5771ed3
@@ -0,0 +1,82 @@
+# Scalar - an opinionated repository management tool
+
+Scalar is an add-on to Git that helps users take advantage of advanced
+performance features in Git. Originally implemented in C# using .NET Core,
+based on the learnings from the VFS for Git project, most of the techniques
+developed by the Scalar project have been integrated into core Git already:
+
+* partial clone,
+* commit graphs,
+* multi-pack index,
+* sparse checkout (cone mode),
+* scheduled background maintenance,
+* etc
+
+This directory contains the remaining parts of Scalar that are not (yet) in
+core Git.
+
+## Roadmap
+
+The idea is to populate this directory via incremental patch series and
+eventually move to a top-level directory next to `gitk-git/` and to `git-gui/`. The
+current plan involves the following patch series:
+
+- `scalar-the-beginning`: The initial patch series which sets up
+  `contrib/scalar/` and populates it with a minimal `scalar` command that
+  demonstrates the fundamental ideas.
+
+- `scalar-c-and-C`: The `scalar` command learns about two options that can be
+  specified before the command, `-c <key>=<value>` and `-C <directory>`.
+
+- `scalar-diagnose`: The `scalar` command is taught the `diagnose` subcommand.
+
+- `scalar-and-builtin-fsmonitor`: The built-in FSMonitor is enabled in `scalar
+  register` and in `scalar clone`, for an enormous performance boost when
+  working in large worktrees. This patch series necessarily depends on Jeff
+  Hostetler's FSMonitor patch series to be integrated into Git.
+
+- `scalar-gentler-config-locking`: Scalar enlistments are registered in the
+  user's Git config. This usually does not represent any problem because it is
+  rare for a user to register an enlistment. However, in Scalar's functional
+  tests, Scalar enlistments are created galore, and in parallel, which can lead
+  to lock contention. This patch series works around that problem by re-trying
+  to lock the config file in a gentle fashion.
+
+- `scalar-extra-docs`: Add some extensive documentation that has been written
+  in the original Scalar project (all subject to discussion, of course).
+
+- `optionally-install-scalar`: Now that Scalar is feature (and documentation)
+  complete and is verified in CI builds, let's offer to install it.
+
+- `move-scalar-to-toplevel`: Now that Scalar is complete, let's move it next to
+  `gitk-git/` and to `git-gui/`, making it a top-level command.
+
+The following two patch series exist in Microsoft's fork of Git and are
+publicly available. There is no current plan to upstream them, not because I
+want to withhold these patches, but because I don't think the Git community is
+interested in these patches.
+
+There are some interesting ideas there, but the implementation is too specific
+to Azure Repos and/or VFS for Git to be of much help in general (and also: my
+colleagues tried to upstream some patches already and the enthusiasm for
+integrating things related to Azure Repos and VFS for Git can be summarized in
+very, very few words).
+
+These still exist mainly because the GVFS protocol is what Azure Repos has
+instead of partial clone, while Git is focused on improving partial clone:
+
+- `scalar-with-gvfs`: The primary purpose of this patch series is to support
+  existing Scalar users whose repositories are hosted in Azure Repos (which
+  does not support Git's partial clones, but supports its predecessor, the GVFS
+  protocol, which is used by Scalar to emulate the partial clone).
+
+  Since the GVFS protocol will never be supported by core Git, this patch
+  series will remain in Microsoft's fork of Git.
+
+- `run-scalar-functional-tests`: The Scalar project developed a quite
+  comprehensive set of integration tests (or, "Functional Tests"). They are the
+  sole remaining part of the original C#-based Scalar project, and this patch
+  adds a GitHub workflow that runs them all.
+
+  Since the tests partially depend on features that are only provided in the
+  `scalar-with-gvfs` patch series, this patch cannot be upstreamed.
Signature · 1 line
gitgitgadget
#201 lore raw

PATCH v8 2/17 scalar: create a rudimentary executable

In reply to Johannes Schindelin via GitGitGadget

From: Johannes Schindelin <[email protected]>

The idea of Scalar (https://github.com/microsoft/scalar), and before that, of VFS for Git, has always been to prove that Git _can_ scale, and to upstream whatever strategies have been demonstrated to help.

With this patch, we start the journey from that C# project to move what is left to Git's own contrib/ directory, reimplementing it in pure C, with the intention to facilitate integrating the functionality into core Git all while maintaining backwards-compatibility for existing Scalar users (which will be much easier when both live in the same worktree). It has always been the plan to contribute all of the proven strategies back to core Git.

For example, while the virtual filesystem provided by VFS for Git helped the team developing the Windows operating system to move onto Git, while trying to upstream it we realized that it cannot be done: getting the virtual filesystem to work (which we only managed to implement fully on Windows, but not on, say, macOS or Linux), and the required server-side support for the GVFS protocol, made this not quite feasible.

The Scalar project learned from that and tackled the problem with different tactics: instead of pretending to Git that the working directory is fully populated, it _specifically_ teaches Git about partial clone (which is based on VFS for Git's cache server), about sparse checkout (which VFS for Git tried to do transparently, in the file system layer), and regularly runs maintenance tasks to keep the repository in a healthy state.

With partial clone, sparse checkout and git maintenance having been upstreamed, there is little left that scalar.exe does which git.exe cannot do. One such thing is that scalar clone <url> will automatically set up a partial, sparse clone, and configure known-helpful settings from the start.

So let's bring this convenience into Git's tree.

The idea here is that you can (optionally) build Scalar via

make -C contrib/scalar/

This will build the scalar executable and put it into the contrib/scalar/ subdirectory.

The slightly awkward addition of the contrib/scalar/* bits to the top-level Makefile are actually really required: we want to link to libgit.a, which means that we will need to use the very same CFLAGS and LDFLAGS as the rest of Git.

An early development version of this patch tried to replicate all the conditional code in contrib/scalar/Makefile (e.g. NO_POLL) just like contrib/svn-fe/Makefile used to do before it was retired. It turned out to be quite the whack-a-mole game: the SHA-1-related flags, the flags enabling/disabling compat/poll/, compat/regex/, compat/win32mmap.c & friends depending on the current platform... To put it mildly: it was a major mess.

Instead, this patch makes minimal changes to the top-level Makefile so that the bits in contrib/scalar/ can be compiled and linked, and adds a contrib/scalar/Makefile that uses the top-level Makefile in a most minimal way to do the actual compiling.

Note: With this commit, we only establish the infrastructure, no Scalar functionality is implemented yet; We will do that incrementally over the next few commits.

---
 Makefile                  |  9 +++++++++
 contrib/scalar/.gitignore |  2 ++
 contrib/scalar/Makefile   | 34 ++++++++++++++++++++++++++++++++++
 contrib/scalar/scalar.c   | 36 ++++++++++++++++++++++++++++++++++++
 4 files changed, 81 insertions(+)
 create mode 100644 contrib/scalar/.gitignore
 create mode 100644 contrib/scalar/Makefile
 create mode 100644 contrib/scalar/scalar.c
Makefile +9 -0
index 12be39ac497..fe898aeea08 100644
@@ -2456,6 +2456,11 @@ OBJECTS += $(FUZZ_OBJS)
 ifndef NO_CURL
 	OBJECTS += http.o http-walker.o remote-curl.o
 endif
+
+SCALAR_SOURCES := contrib/scalar/scalar.c
+SCALAR_OBJECTS := $(SCALAR_SOURCES:c=o)
+OBJECTS += $(SCALAR_OBJECTS)
+
 .PHONY: objects
 objects: $(OBJECTS)
 
@@ -2589,6 +2594,10 @@ $(REMOTE_CURL_PRIMARY): remote-curl.o http.o http-walker.o GIT-LDFLAGS $(GITLIBS
 	$(QUIET_LINK)$(CC) $(ALL_CFLAGS) -o $@ $(ALL_LDFLAGS) $(filter %.o,$^) \
 		$(CURL_LIBCURL) $(EXPAT_LIBEXPAT) $(LIBS)
 
+contrib/scalar/scalar$X: $(SCALAR_OBJECTS) GIT-LDFLAGS $(GITLIBS)
+	$(QUIET_LINK)$(CC) $(ALL_CFLAGS) -o $@ $(ALL_LDFLAGS) \
+		$(filter %.o,$^) $(LIBS)
+
 $(LIB_FILE): $(LIB_OBJS)
 	$(QUIET_AR)$(RM) $@ && $(AR) $(ARFLAGS) $@ $^
 
contrib/scalar/.gitignore new file +2 -0
new file mode 100644
index 00000000000..ff3d47e84d0
@@ -0,0 +1,2 @@
+/*.exe
+/scalar
contrib/scalar/Makefile new file +34 -0
new file mode 100644
index 00000000000..f6f0036f0fa
@@ -0,0 +1,34 @@
+QUIET_SUBDIR0  = +$(MAKE) -C # space to separate -C and subdir
+QUIET_SUBDIR1  =
+
+ifneq ($(findstring s,$(MAKEFLAGS)),s)
+ifndef V
+	QUIET_SUBDIR0  = +@subdir=
+	QUIET_SUBDIR1  = ;$(NO_SUBDIR) echo '   ' SUBDIR $$subdir; \
+			 $(MAKE) $(PRINT_DIR) -C $$subdir
+else
+	export V
+endif
+endif
+
+all:
+
+include ../../config.mak.uname
+-include ../../config.mak.autogen
+-include ../../config.mak
+
+TARGETS = scalar$(X) scalar.o
+GITLIBS = ../../common-main.o ../../libgit.a ../../xdiff/lib.a
+
+all: scalar$(X)
+
+$(GITLIBS):
+	$(QUIET_SUBDIR0)../.. $(QUIET_SUBDIR1) $(subst ../../,,$@)
+
+$(TARGETS): $(GITLIBS) scalar.c
+	$(QUIET_SUBDIR0)../.. $(QUIET_SUBDIR1) $(patsubst %,contrib/scalar/%,$@)
+
+clean:
+	$(RM) $(TARGETS)
+
+.PHONY: $(GITLIBS) all clean FORCE
contrib/scalar/scalar.c new file +36 -0
new file mode 100644
index 00000000000..7cff29e0fcd
@@ -0,0 +1,36 @@
+/*
+ * The Scalar command-line interface.
+ */
+
+#include "cache.h"
+#include "gettext.h"
+#include "parse-options.h"
+
+static struct {
+	const char *name;
+	int (*fn)(int, const char **);
+} builtins[] = {
+	{ NULL, NULL},
+};
+
+int cmd_main(int argc, const char **argv)
+{
+	struct strbuf scalar_usage = STRBUF_INIT;
+	int i;
+
+	if (argc > 1) {
+		argv++;
+		argc--;
+
+		for (i = 0; builtins[i].name; i++)
+			if (!strcmp(builtins[i].name, argv[0]))
+				return !!builtins[i].fn(argc, argv);
+	}
+
+	strbuf_addstr(&scalar_usage,
+		      N_("scalar <command> [<options>]\n\nCommands:\n"));
+	for (i = 0; builtins[i].name; i++)
+		strbuf_addf(&scalar_usage, "\t%s\n", builtins[i].name);
+
+	usage(scalar_usage.buf);
+}
Signature · 1 line
gitgitgadget
#202 lore raw

PATCH v8 3/17 scalar: start documenting the command

In reply to Johannes Schindelin via GitGitGadget

From: Johannes Schindelin <[email protected]>

Let's build up the documentation for the Scalar command along with the patches that implement its functionality.

Note: To discourage the feature-incomplete documentation from being mistaken for the complete thing, we do not yet provide any way to build HTML or manual pages from the text file.

---
 contrib/scalar/scalar.txt | 38 ++++++++++++++++++++++++++++++++++++++
 1 file changed, 38 insertions(+)
 create mode 100644 contrib/scalar/scalar.txt
contrib/scalar/scalar.txt new file +38 -0
new file mode 100644
index 00000000000..5f7131861a5
@@ -0,0 +1,38 @@
+scalar(1)
+=========
+
+NAME
+----
+scalar - an opinionated repository management tool
+
+SYNOPSIS
+--------
+[verse]
+scalar <command> [<options>]
+
+DESCRIPTION
+-----------
+
+Scalar is an opinionated repository management tool. By creating new
+repositories or registering existing repositories with Scalar, your Git
+experience will speed up. Scalar sets advanced Git config settings,
+maintains your repositories in the background, and helps reduce data sent
+across the network.
+
+An important Scalar concept is the enlistment: this is the top-level directory
+of the project. It usually contains the subdirectory `src/` which is a Git
+worktree. This encourages the separation between tracked files (inside `src/`)
+and untracked files, such as build artifacts (outside `src/`). When registering
+an existing Git worktree with Scalar whose name is not `src`, the enlistment
+will be identical to the worktree.
+
+The `scalar` command implements various subcommands, and different options
+depending on the subcommand.
+
+SEE ALSO
+--------
+linkgit:git-maintenance[1].
+
+Scalar
+---
+Associated with the linkgit:git[1] suite
Signature · 1 line
gitgitgadget
#203 lore raw

PATCH v8 4/17 scalar: create test infrastructure

In reply to Johannes Schindelin via GitGitGadget

From: Johannes Schindelin <[email protected]>

To test the Scalar command, create a test script in contrib/scalar/t that is executed as make -C contrib/scalar test. Since Scalar has no meaningful capabilities yet, the only test is rather simple. We will add more tests in subsequent commits that introduce corresponding, new functionality.

Note: This test script is intended to test scalar only lightly, even after all of the functionality is implemented.

A more comprehensive functional (or: integration) test suite can be found at https://github.com/microsoft/scalar; It is used in the workflow
https://github.com/microsoft/git/blob/HEAD/.github/workflows/scalar-functional-tests.yml
in Microsoft's Git fork. This test suite performs end-to-end tests with a real remote repository, and is run as part of the regular CI and PR builds in that fork.

Since those tests require some functionality supported only by Microsoft's Git fork ("GVFS protocol"), there is no intention to port that fuller test suite to contrib/scalar/.

---
 contrib/scalar/Makefile          | 17 +++++--
 contrib/scalar/t/Makefile        | 78 ++++++++++++++++++++++++++++++++
 contrib/scalar/t/t9099-scalar.sh | 17 +++++++
 3 files changed, 109 insertions(+), 3 deletions(-)
 create mode 100644 contrib/scalar/t/Makefile
 create mode 100755 contrib/scalar/t/t9099-scalar.sh
contrib/scalar/Makefile +14 -3
index f6f0036f0fa..231b1ee1796 100644
@@ -3,6 +3,7 @@ QUIET_SUBDIR1  =
 
 ifneq ($(findstring s,$(MAKEFLAGS)),s)
 ifndef V
+	QUIET_GEN      = @echo '   ' GEN $@;
 	QUIET_SUBDIR0  = +@subdir=
 	QUIET_SUBDIR1  = ;$(NO_SUBDIR) echo '   ' SUBDIR $$subdir; \
 			 $(MAKE) $(PRINT_DIR) -C $$subdir
@@ -20,7 +21,7 @@ include ../../config.mak.uname
 TARGETS = scalar$(X) scalar.o
 GITLIBS = ../../common-main.o ../../libgit.a ../../xdiff/lib.a
 
-all: scalar$(X)
+all: scalar$(X) ../../bin-wrappers/scalar
 
 $(GITLIBS):
 	$(QUIET_SUBDIR0)../.. $(QUIET_SUBDIR1) $(subst ../../,,$@)
@@ -29,6 +30,16 @@ $(TARGETS): $(GITLIBS) scalar.c
 	$(QUIET_SUBDIR0)../.. $(QUIET_SUBDIR1) $(patsubst %,contrib/scalar/%,$@)
 
 clean:
-	$(RM) $(TARGETS)
+	$(RM) $(TARGETS) ../../bin-wrappers/scalar
 
-.PHONY: $(GITLIBS) all clean FORCE
+../../bin-wrappers/scalar: ../../wrap-for-bin.sh Makefile
+	@mkdir -p ../../bin-wrappers
+	$(QUIET_GEN)sed -e '1s|#!.*/sh|#!$(SHELL_PATH_SQ)|' \
+	     -e 's|@@BUILD_DIR@@|$(shell cd ../.. && pwd)|' \
+	     -e 's|@@PROG@@|contrib/scalar/scalar$(X)|' < $< > $@ && \
+	chmod +x $@
+
+test: all
+	$(MAKE) -C t
+
+.PHONY: $(GITLIBS) all clean test FORCE
contrib/scalar/t/Makefile new file +78 -0

78 lines not on this page · raw message

contrib/scalar/t/t9099-scalar.sh new file +17 -0
new file mode 100755
index 00000000000..16f2b72b126
@@ -0,0 +1,17 @@
+#!/bin/sh
+
+test_description='test the `scalar` command'
+
+TEST_DIRECTORY=$PWD/../../../t
+export TEST_DIRECTORY
+
+# Make it work with --no-bin-wrappers
+PATH=$PWD/..:$PATH
+
+. ../../../t/test-lib.sh
+
+test_expect_success 'scalar shows a usage' '
+	test_expect_code 129 scalar -h
+'
+
+test_done
Signature · 1 line
gitgitgadget
#205 lore raw

PATCH v8 5/17 cmake: optionally build `scalar`, too

In reply to Johannes Schindelin via GitGitGadget

From: Johannes Schindelin <[email protected]>

The CMake configuration unfortunately does not let us easily encapsulate Scalar's build definition in the contrib/scalar/ subdirectory: The scalar executable needs to link in libgit.a and common-main.o, for example.

Also, scalar.c includes Git's header files, which means that scalar.c needs to be compiled with the very same flags as libgit.a lest scalar.o and libgit.a have different ideas of, say, platform_SHA_CTX, which would naturally lead to memory corruption.

To alleviate that somewhat, we guard the inclusion of Scalar via the INCLUDE_SCALAR environment variable.

---
 contrib/buildsystems/CMakeLists.txt | 14 ++++++++++++++
 1 file changed, 14 insertions(+)
contrib/buildsystems/CMakeLists.txt +14 -0
index fd1399c440f..dd7496b0322 100644
@@ -729,6 +729,13 @@ if(CURL_FOUND)
 	endif()
 endif()
 
+if(DEFINED ENV{INCLUDE_SCALAR} AND NOT ENV{INCLUDE_SCALAR} STREQUAL "")
+	add_executable(scalar ${CMAKE_SOURCE_DIR}/contrib/scalar/scalar.c)
+	target_link_libraries(scalar common-main)
+	set_target_properties(scalar PROPERTIES RUNTIME_OUTPUT_DIRECTORY_DEBUG ${CMAKE_BINARY_DIR}/contrib/scalar)
+	set_target_properties(scalar PROPERTIES RUNTIME_OUTPUT_DIRECTORY_RELEASE ${CMAKE_BINARY_DIR}/contrib/scalar)
+endif()
+
 parse_makefile_for_executables(git_builtin_extra "BUILT_INS")
 
 option(SKIP_DASHED_BUILT_INS "Skip hardlinking the dashed versions of the built-ins")
@@ -953,6 +960,13 @@ string(REPLACE "@@BUILD_DIR@@" "${CMAKE_BINARY_DIR}" content "${content}")
 string(REPLACE "@@PROG@@" "git-cvsserver" content "${content}")
 file(WRITE ${CMAKE_BINARY_DIR}/bin-wrappers/git-cvsserver ${content})
 
+if(DEFINED ENV{INCLUDE_SCALAR} AND NOT ENV{INCLUDE_SCALAR} STREQUAL "")
+	file(STRINGS ${CMAKE_SOURCE_DIR}/wrap-for-bin.sh content NEWLINE_CONSUME)
+	string(REPLACE "@@BUILD_DIR@@" "${CMAKE_BINARY_DIR}" content "${content}")
+	string(REPLACE "@@PROG@@" "contrib/scalar/scalar${EXE_EXTENSION}" content "${content}")
+	file(WRITE ${CMAKE_BINARY_DIR}/bin-wrappers/scalar ${content})
+endif()
+
 #options for configuring test options
 option(PERL_TESTS "Perform tests that use perl" ON)
 option(PYTHON_TESTS "Perform tests that use python" ON)
Signature · 1 line
gitgitgadget
#206 lore raw

PATCH v8 6/17 ci: also run the `scalar` tests

In reply to Johannes Schindelin via GitGitGadget

From: Johannes Schindelin <[email protected]>

Since Scalar depends on libgit.a, it makes sense to ensure in the CI and the PR builds that it does not get broken in case of industrious refactorings of the core Git code.

---
 .github/workflows/main.yml | 15 +++++++++++++++
 ci/run-build-and-tests.sh  |  1 +
 ci/run-test-slice.sh       |  5 +++++
 3 files changed, 21 insertions(+)
.github/workflows/main.yml +15 -0
index 6ed6a9e8076..6eda6be895d 100644
@@ -89,6 +89,13 @@ jobs:
         HOME: ${{runner.workspace}}
         NO_PERL: 1
       run: . /etc/profile && ci/make-test-artifacts.sh artifacts
+    - name: build Scalar
+      shell: bash
+      run: |
+        make -C contrib/scalar &&
+        mkdir -p artifacts/bin-wrappers artifacts/contrib/scalar &&
+        cp contrib/scalar/scalar.exe artifacts/contrib/scalar/ &&
+        cp bin-wrappers/scalar artifacts/bin-wrappers/
     - name: zip up tracked files
       run: git archive -o artifacts/tracked.tar.gz HEAD
     - name: upload tracked files and build artifacts
@@ -157,6 +164,8 @@ jobs:
       run: compat\vcbuild\vcpkg_copy_dlls.bat release
     - name: generate Visual Studio solution
       shell: bash
+      env:
+        INCLUDE_SCALAR: YesPlease
       run: |
         cmake `pwd`/contrib/buildsystems/ -DCMAKE_PREFIX_PATH=`pwd`/compat/vcbuild/vcpkg/installed/x64-windows \
         -DNO_GETTEXT=YesPlease -DPERL_TESTS=OFF -DPYTHON_TESTS=OFF -DCURL_NO_CURL_CMAKE=ON
@@ -170,6 +179,12 @@ jobs:
       run: |
         mkdir -p artifacts &&
         eval "$(make -n artifacts-tar INCLUDE_DLLS_IN_ARTIFACTS=YesPlease ARTIFACTS_DIRECTORY=artifacts NO_GETTEXT=YesPlease 2>&1 | grep ^tar)"
+    - name: copy Scalar
+      shell: bash
+      run: |
+        mkdir -p artifacts/bin-wrappers artifacts/contrib/scalar &&
+        cp contrib/scalar/scalar.exe artifacts/contrib/scalar/ &&
+        cp bin-wrappers/scalar artifacts/bin-wrappers/
     - name: zip up tracked files
       run: git archive -o artifacts/tracked.tar.gz HEAD
     - name: upload tracked files and build artifacts
ci/run-build-and-tests.sh +1 -0
index cc62616d806..07cedd25ff1 100755
@@ -49,6 +49,7 @@ linux-gcc-4.8|pedantic)
 	make test
 	;;
 esac
+make -C contrib/scalar test
 
 check_unignored_build_artifacts
 
ci/run-test-slice.sh +5 -0
index f8c2c3106a2..b741fd8f361 100755
@@ -14,4 +14,9 @@ make --quiet -C t T="$(cd t &&
 	./helper/test-tool path-utils slice-tests "$1" "$2" t[0-9]*.sh |
 	tr '\n' ' ')"
 
+if test 0 = "$1"
+then
+	make -C contrib/scalar test
+fi
+
 check_unignored_build_artifacts
Signature · 1 line
gitgitgadget
#207 lore raw

PATCH v8 7/17 scalar: 'register' sets recommended config and starts maintenance

In reply to Johannes Schindelin via GitGitGadget

From: Derrick Stolee <[email protected]>

Let's start implementing the register command. With this commit, recommended settings are configured upon scalar register, and Git's background maintenance is started.

The recommended config settings may very well change in the future. For example, once the built-in FSMonitor is available, we will want to enable it upon scalar register. For that reason, we explicitly support running scalar register in an already-registered enlistment.

---
 contrib/scalar/scalar.c   | 249 ++++++++++++++++++++++++++++++++++++++
 contrib/scalar/scalar.txt |  18 ++-
 2 files changed, 266 insertions(+), 1 deletion(-)
contrib/scalar/scalar.c +249 -0
index 7cff29e0fcd..03d5f84c764 100644
@@ -5,11 +5,260 @@
 #include "cache.h"
 #include "gettext.h"
 #include "parse-options.h"
+#include "config.h"
+#include "run-command.h"
+
+/*
+ * Remove the deepest subdirectory in the provided path string. Path must not
+ * include a trailing path separator. Returns 1 if parent directory found,
+ * otherwise 0.
+ */
+static int strbuf_parent_directory(struct strbuf *buf)
+{
+	size_t len = buf->len;
+	size_t offset = offset_1st_component(buf->buf);
+	char *path_sep = find_last_dir_sep(buf->buf + offset);
+	strbuf_setlen(buf, path_sep ? path_sep - buf->buf : offset);
+
+	return buf->len < len;
+}
+
+static void setup_enlistment_directory(int argc, const char **argv,
+				       const char * const *usagestr,
+				       const struct option *options,
+				       struct strbuf *enlistment_root)
+{
+	struct strbuf path = STRBUF_INIT;
+	char *root;
+	int enlistment_found = 0;
+
+	if (startup_info->have_repository)
+		BUG("gitdir already set up?!?");
+
+	if (argc > 1)
+		usage_with_options(usagestr, options);
+
+	/* find the worktree, determine its corresponding root */
+	if (argc == 1)
+		strbuf_add_absolute_path(&path, argv[0]);
+	else if (strbuf_getcwd(&path) < 0)
+		die(_("need a working directory"));
+
+	strbuf_trim_trailing_dir_sep(&path);
+	do {
+		const size_t len = path.len;
+
+		/* check if currently in enlistment root with src/ workdir */
+		strbuf_addstr(&path, "/src");
+		if (is_nonbare_repository_dir(&path)) {
+			if (enlistment_root)
+				strbuf_add(enlistment_root, path.buf, len);
+
+			enlistment_found = 1;
+			break;
+		}
+
+		/* reset to original path */
+		strbuf_setlen(&path, len);
+
+		/* check if currently in workdir */
+		if (is_nonbare_repository_dir(&path)) {
+			if (enlistment_root) {
+				/*
+				 * If the worktree's directory's name is `src`, the enlistment is the
+				 * parent directory, otherwise it is identical to the worktree.
+				 */
+				root = strip_path_suffix(path.buf, "src");
+				strbuf_addstr(enlistment_root, root ? root : path.buf);
+				free(root);
+			}
+
+			enlistment_found = 1;
+			break;
+		}
+	} while (strbuf_parent_directory(&path));
+
+	if (!enlistment_found)
+		die(_("could not find enlistment root"));
+
+	if (chdir(path.buf) < 0)
+		die_errno(_("could not switch to '%s'"), path.buf);
+
+	strbuf_release(&path);
+	setup_git_directory();
+}
+
+static int run_git(const char *arg, ...)
+{
+	struct strvec argv = STRVEC_INIT;
+	va_list args;
+	const char *p;
+	int res;
+
+	va_start(args, arg);
+	strvec_push(&argv, arg);
+	while ((p = va_arg(args, const char *)))
+		strvec_push(&argv, p);
+	va_end(args);
+
+	res = run_command_v_opt(argv.v, RUN_GIT_CMD);
+
+	strvec_clear(&argv);
+	return res;
+}
+
+static int set_recommended_config(void)
+{
+	struct {
+		const char *key;
+		const char *value;
+	} config[] = {
+		{ "am.keepCR", "true" },
+		{ "core.FSCache", "true" },
+		{ "core.multiPackIndex", "true" },
+		{ "core.preloadIndex", "true" },
+#ifndef WIN32
+		{ "core.untrackedCache", "true" },
+#else
+		/*
+		 * Unfortunately, Scalar's Functional Tests demonstrated
+		 * that the untracked cache feature is unreliable on Windows
+		 * (which is a bummer because that platform would benefit the
+		 * most from it). For some reason, freshly created files seem
+		 * not to update the directory's `lastModified` time
+		 * immediately, but the untracked cache would need to rely on
+		 * that.
+		 *
+		 * Therefore, with a sad heart, we disable this very useful
+		 * feature on Windows.
+		 */
+		{ "core.untrackedCache", "false" },
+#endif
+		{ "core.logAllRefUpdates", "true" },
+		{ "credential.https://dev.azure.com.useHttpPath", "true" },
+		{ "credential.validate", "false" }, /* GCM4W-only */
+		{ "gc.auto", "0" },
+		{ "gui.GCWarning", "false" },
+		{ "index.threads", "true" },
+		{ "index.version", "4" },
+		{ "merge.stat", "false" },
+		{ "merge.renames", "false" },
+		{ "pack.useBitmaps", "false" },
+		{ "pack.useSparse", "true" },
+		{ "receive.autoGC", "false" },
+		{ "reset.quiet", "true" },
+		{ "feature.manyFiles", "false" },
+		{ "feature.experimental", "false" },
+		{ "fetch.unpackLimit", "1" },
+		{ "fetch.writeCommitGraph", "false" },
+#ifdef WIN32
+		{ "http.sslBackend", "schannel" },
+#endif
+		{ "status.aheadBehind", "false" },
+		{ "commitGraph.generationVersion", "1" },
+		{ "core.autoCRLF", "false" },
+		{ "core.safeCRLF", "false" },
+		{ "fetch.showForcedUpdates", "false" },
+		{ NULL, NULL },
+	};
+	int i;
+	char *value;
+
+	for (i = 0; config[i].key; i++) {
+		if (git_config_get_string(config[i].key, &value)) {
+			trace2_data_string("scalar", the_repository, config[i].key, "created");
+			if (git_config_set_gently(config[i].key,
+						  config[i].value) < 0)
+				return error(_("could not configure %s=%s"),
+					     config[i].key, config[i].value);
+		} else {
+			trace2_data_string("scalar", the_repository, config[i].key, "exists");
+			free(value);
+		}
+	}
+
+	/*
+	 * The `log.excludeDecoration` setting is special because it allows
+	 * for multiple values.
+	 */
+	if (git_config_get_string("log.excludeDecoration", &value)) {
+		trace2_data_string("scalar", the_repository,
+				   "log.excludeDecoration", "created");
+		if (git_config_set_multivar_gently("log.excludeDecoration",
+						   "refs/prefetch/*",
+						   CONFIG_REGEX_NONE, 0))
+			return error(_("could not configure "
+				       "log.excludeDecoration"));
+	} else {
+		trace2_data_string("scalar", the_repository,
+				   "log.excludeDecoration", "exists");
+		free(value);
+	}
+
+	return 0;
+}
+
+static int start_maintenance(void)
+{
+	return run_git("maintenance", "start", NULL);
+}
+
+static int add_enlistment(void)
+{
+	int res;
+
+	if (!the_repository->worktree)
+		die(_("Scalar enlistments require a worktree"));
+
+	res = run_git("config", "--global", "--get", "--fixed-value",
+		      "scalar.repo", the_repository->worktree, NULL);
+
+	/*
+	 * If the setting is already there, then do nothing.
+	 */
+	if (!res)
+		return 0;
+
+	return run_git("config", "--global", "--add",
+		       "scalar.repo", the_repository->worktree, NULL);
+}
+
+static int register_dir(void)
+{
+	int res = add_enlistment();
+
+	if (!res)
+		res = set_recommended_config();
+
+	if (!res)
+		res = start_maintenance();
+
+	return res;
+}
+
+static int cmd_register(int argc, const char **argv)
+{
+	struct option options[] = {
+		OPT_END(),
+	};
+	const char * const usage[] = {
+		N_("scalar register [<enlistment>]"),
+		NULL
+	};
+
+	argc = parse_options(argc, argv, NULL, options,
+			     usage, 0);
+
+	setup_enlistment_directory(argc, argv, usage, options, NULL);
+
+	return register_dir();
+}
 
 static struct {
 	const char *name;
 	int (*fn)(int, const char **);
 } builtins[] = {
+	{ "register", cmd_register },
 	{ NULL, NULL},
 };
 
contrib/scalar/scalar.txt +17 -1
index 5f7131861a5..568987064b2 100644
@@ -8,7 +8,7 @@ scalar - an opinionated repository management tool
 SYNOPSIS
 --------
 [verse]
-scalar <command> [<options>]
+scalar register [<enlistment>]
 
 DESCRIPTION
 -----------
@@ -29,6 +29,22 @@ will be identical to the worktree.
 The `scalar` command implements various subcommands, and different options
 depending on the subcommand.
 
+COMMANDS
+--------
+
+Register
+~~~~~~~~
+
+register [<enlistment>]::
+	Adds the enlistment's repository to the list of registered repositories
+	and starts background maintenance. If `<enlistment>` is not provided,
+	then the enlistment associated with the current working directory is
+	registered.
++
+Note: when this subcommand is called in a worktree that is called `src/`, its
+parent directory is considered to be the Scalar enlistment. If the worktree is
+_not_ called `src/`, it itself will be considered to be the Scalar enlistment.
+
 SEE ALSO
 --------
 linkgit:git-maintenance[1].
Signature · 1 line
gitgitgadget
#208 lore raw

PATCH v8 8/17 scalar: 'unregister' stops background maintenance

In reply to Johannes Schindelin via GitGitGadget

From: Derrick Stolee <[email protected]>

Just like scalar register starts the scheduled background maintenance, scalar unregister stops it. Note that we use git maintenance start in scalar register, but we do not use git maintenance stop in scalar unregister: this would stop maintenance for _all_ repositories, not just for the one we want to unregister.

The unregister command also removes the corresponding entry from the [scalar] section in the global Git config.

---
 contrib/scalar/scalar.c   | 50 ++++++++++++++++++++++++++++++++-------
 contrib/scalar/scalar.txt |  8 +++++++
 2 files changed, 50 insertions(+), 8 deletions(-)
contrib/scalar/scalar.c +42 -8
index 03d5f84c764..bab0271c37d 100644
@@ -198,12 +198,12 @@ static int set_recommended_config(void)
 	return 0;
 }
 
-static int start_maintenance(void)
+static int toggle_maintenance(int enable)
 {
-	return run_git("maintenance", "start", NULL);
+	return run_git("maintenance", enable ? "start" : "unregister", NULL);
 }
 
-static int add_enlistment(void)
+static int add_or_remove_enlistment(int add)
 {
 	int res;
 
@@ -214,24 +214,39 @@ static int add_enlistment(void)
 		      "scalar.repo", the_repository->worktree, NULL);
 
 	/*
-	 * If the setting is already there, then do nothing.
+	 * If we want to add and the setting is already there, then do nothing.
+	 * If we want to remove and the setting is not there, then do nothing.
 	 */
-	if (!res)
+	if ((add && !res) || (!add && res))
 		return 0;
 
-	return run_git("config", "--global", "--add",
+	return run_git("config", "--global", add ? "--add" : "--unset",
+		       add ? "--no-fixed-value" : "--fixed-value",
 		       "scalar.repo", the_repository->worktree, NULL);
 }
 
 static int register_dir(void)
 {
-	int res = add_enlistment();
+	int res = add_or_remove_enlistment(1);
 
 	if (!res)
 		res = set_recommended_config();
 
 	if (!res)
-		res = start_maintenance();
+		res = toggle_maintenance(1);
+
+	return res;
+}
+
+static int unregister_dir(void)
+{
+	int res = 0;
+
+	if (toggle_maintenance(0) < 0)
+		res = -1;
+
+	if (add_or_remove_enlistment(0) < 0)
+		res = -1;
 
 	return res;
 }
@@ -254,11 +269,30 @@ static int cmd_register(int argc, const char **argv)
 	return register_dir();
 }
 
+static int cmd_unregister(int argc, const char **argv)
+{
+	struct option options[] = {
+		OPT_END(),
+	};
+	const char * const usage[] = {
+		N_("scalar unregister [<enlistment>]"),
+		NULL
+	};
+
+	argc = parse_options(argc, argv, NULL, options,
+			     usage, 0);
+
+	setup_enlistment_directory(argc, argv, usage, options, NULL);
+
+	return unregister_dir();
+}
+
 static struct {
 	const char *name;
 	int (*fn)(int, const char **);
 } builtins[] = {
 	{ "register", cmd_register },
+	{ "unregister", cmd_unregister },
 	{ NULL, NULL},
 };
 
contrib/scalar/scalar.txt +8 -0
index 568987064b2..d9a79984492 100644
@@ -9,6 +9,7 @@ SYNOPSIS
 --------
 [verse]
 scalar register [<enlistment>]
+scalar unregister [<enlistment>]
 
 DESCRIPTION
 -----------
@@ -45,6 +46,13 @@ Note: when this subcommand is called in a worktree that is called `src/`, its
 parent directory is considered to be the Scalar enlistment. If the worktree is
 _not_ called `src/`, it itself will be considered to be the Scalar enlistment.
 
+Unregister
+~~~~~~~~~~
+
+unregister [<enlistment>]::
+	Remove the specified repository from the list of repositories
+	registered with Scalar and stop the scheduled background maintenance.
+
 SEE ALSO
 --------
 linkgit:git-maintenance[1].
Signature · 1 line
gitgitgadget
#209 lore raw

PATCH v8 9/17 scalar: let 'unregister' handle a deleted enlistment directory gracefully

In reply to Johannes Schindelin via GitGitGadget

From: Johannes Schindelin <[email protected]>

When a user deleted an enlistment manually, let's be generous and _still_ unregister it.

---
 contrib/scalar/scalar.c          | 46 ++++++++++++++++++++++++++++++++
 contrib/scalar/t/t9099-scalar.sh | 15 +++++++++++
 2 files changed, 61 insertions(+)
contrib/scalar/scalar.c +46 -0
index bab0271c37d..097d3bd478b 100644
@@ -269,6 +269,24 @@ static int cmd_register(int argc, const char **argv)
 	return register_dir();
 }
 
+static int remove_deleted_enlistment(struct strbuf *path)
+{
+	int res = 0;
+	strbuf_realpath_forgiving(path, path->buf, 1);
+
+	if (run_git("config", "--global",
+		    "--unset", "--fixed-value",
+		    "scalar.repo", path->buf, NULL) < 0)
+		res = -1;
+
+	if (run_git("config", "--global",
+		    "--unset", "--fixed-value",
+		    "maintenance.repo", path->buf, NULL) < 0)
+		res = -1;
+
+	return res;
+}
+
 static int cmd_unregister(int argc, const char **argv)
 {
 	struct option options[] = {
@@ -282,6 +300,34 @@ static int cmd_unregister(int argc, const char **argv)
 	argc = parse_options(argc, argv, NULL, options,
 			     usage, 0);
 
+	/*
+	 * Be forgiving when the enlistment or worktree does not even exist any
+	 * longer; This can be the case if a user deleted the worktree by
+	 * mistake and _still_ wants to unregister the thing.
+	 */
+	if (argc == 1) {
+		struct strbuf src_path = STRBUF_INIT, workdir_path = STRBUF_INIT;
+
+		strbuf_addf(&src_path, "%s/src/.git", argv[0]);
+		strbuf_addf(&workdir_path, "%s/.git", argv[0]);
+		if (!is_directory(src_path.buf) && !is_directory(workdir_path.buf)) {
+			/* remove possible matching registrations */
+			int res = -1;
+
+			strbuf_strip_suffix(&src_path, "/.git");
+			res = remove_deleted_enlistment(&src_path) && res;
+
+			strbuf_strip_suffix(&workdir_path, "/.git");
+			res = remove_deleted_enlistment(&workdir_path) && res;
+
+			strbuf_release(&src_path);
+			strbuf_release(&workdir_path);
+			return res;
+		}
+		strbuf_release(&src_path);
+		strbuf_release(&workdir_path);
+	}
+
 	setup_enlistment_directory(argc, argv, usage, options, NULL);
 
 	return unregister_dir();
contrib/scalar/t/t9099-scalar.sh +15 -0
index 16f2b72b126..ef0e8d680d5 100755
@@ -14,4 +14,19 @@ test_expect_success 'scalar shows a usage' '
 	test_expect_code 129 scalar -h
 '
 
+test_expect_success 'scalar unregister' '
+	git init vanish/src &&
+	scalar register vanish/src &&
+	git config --get --global --fixed-value \
+		maintenance.repo "$(pwd)/vanish/src" &&
+	scalar list >scalar.repos &&
+	grep -F "$(pwd)/vanish/src" scalar.repos &&
+	rm -rf vanish/src/.git &&
+	scalar unregister vanish &&
+	test_must_fail git config --get --global --fixed-value \
+		maintenance.repo "$(pwd)/vanish/src" &&
+	scalar list >scalar.repos &&
+	! grep -F "$(pwd)/vanish/src" scalar.repos
+'
+
 test_done
Signature · 1 line
gitgitgadget
#210 lore raw

PATCH v8 10/17 scalar: implement 'scalar list'

In reply to Johannes Schindelin via GitGitGadget

From: Derrick Stolee <[email protected]>

The produced list simply consists of those repositories registered under the multi-valued scalar.repo config setting in the user's Git config.

---
 contrib/scalar/scalar.c   | 11 +++++++++++
 contrib/scalar/scalar.txt | 11 ++++++++++-
 2 files changed, 21 insertions(+), 1 deletion(-)
contrib/scalar/scalar.c +11 -0
index 097d3bd478b..4feacd8d62b 100644
@@ -251,6 +251,16 @@ static int unregister_dir(void)
 	return res;
 }
 
+static int cmd_list(int argc, const char **argv)
+{
+	if (argc != 1)
+		die(_("`scalar list` does not take arguments"));
+
+	if (run_git("config", "--global", "--get-all", "scalar.repo", NULL) < 0)
+		return -1;
+	return 0;
+}
+
 static int cmd_register(int argc, const char **argv)
 {
 	struct option options[] = {
@@ -337,6 +347,7 @@ static struct {
 	const char *name;
 	int (*fn)(int, const char **);
 } builtins[] = {
+	{ "list", cmd_list },
 	{ "register", cmd_register },
 	{ "unregister", cmd_unregister },
 	{ NULL, NULL},
contrib/scalar/scalar.txt +10 -1
index d9a79984492..f93e3d00efd 100644
@@ -8,6 +8,7 @@ scalar - an opinionated repository management tool
 SYNOPSIS
 --------
 [verse]
+scalar list
 scalar register [<enlistment>]
 scalar unregister [<enlistment>]
 
@@ -28,11 +29,19 @@ an existing Git worktree with Scalar whose name is not `src`, the enlistment
 will be identical to the worktree.
 
 The `scalar` command implements various subcommands, and different options
-depending on the subcommand.
+depending on the subcommand. With the exception of `list`, all subcommands
+expect to be run in an enlistment.
 
 COMMANDS
 --------
 
+List
+~~~~
+
+list::
+	List enlistments that are currently registered by Scalar. This
+	subcommand does not need to be run inside an enlistment.
+
 Register
 ~~~~~~~~
 
Signature · 1 line
gitgitgadget